|Xbox 360 team skipped quality testing of console to beat Sony to market|
Last summer, amidst a flurry of reports from Xbox 360 gamers, DailyTech exposed retailers’ estimates that up to one-third of Xbox 360 consoles experience hardware failures within the first year of ownership. Just days after the report, Microsoft extended its warranty to cover the specific hardware failure for three years from purchase.
Now, six months later, a supposed Microsoft insider confirms that around 30% of Xbox 360 consoles, most based on the original ‘Xenon’ design, fail. “It's around 30 percent, and all will probably fail early,” the source told 8Bit Joystick. “This quarter they are expecting 1M failures, most of those Xenons. Some of those are repeat failures.”
Although Microsoft now covers all Xbox 360 consoles for three years against the Red Ring of Death (RROD) – the sign of a hardware failure – there is no specific time frame for the defect to appear. “Life expectancy is all over the map because the design has very little margin for most of the important parameters,” continued the insider. “That means it's not a fault tolerant design. So a good unit may last a couple of years, while a bad unit can fail in hours.”
Prior to the warranty extensions enacted by Microsoft, the Xbox 360 shipped with only a 90-day warranty. Some of those with failed hardware outside of the warranty period took matters in their own hands and almost unanimously discovered that the failure was due to inadequate cooling of entire system (particularly the GPU), leading to overstressed components.
“RROD is caused by anything that fails in the "digital backbone" on the mother board,” said the source, confirming user findings. “The main design flaw was the excessive heat on the GPU warping the mother board around it. This would stress the solder joints on the GPU and any bad joints would then fail in early life.”
Microsoft quickly attempted to rectify the hardware flaw by incorporating a redesigned heatsink to better cool the GPU. A die-shrink to 65nm would also help solve heat issues, though the much anticipated ‘Falcon’ design only featured a 65nm CPU, while the GPU remained at 90nm. The GPU shrink to 65nm is planned later this year in the ‘Jasper’ redesign.
While the above topic points are generally known by the Xbox 360 user community, what comes more alarmingly from the supposed member of the Xbox 360 project is that Microsoft allegedly launched its console fully aware of a potential issue in quality.
“First, MS has under resourced that product unit in all engineering areas since the very beginning. Especially in engineering support functions like test, quality, manufacturing, and supplier management,” the source wrote in an email. “There just weren't enough people to do the job that needed to be done. The leadership in many of those areas was also lopsided in essential skills and experience. But I hear they are really trying to staff up now based on what has happened, and how cheap staff is compared to a couple of billion in cost of quality.”
Microsoft had to take an over $1 billion USD charge to cover the Red Ring of Death defect warranty, which last year cost the company’s Entertainment and Devices Division a $1.89 billion USD loss. Microsoft CEO Steve Ballmer termed the warranty coverage charge as “painful” to announce.
During the previous generation, Microsoft attributed much of PlayStation 2’s lead over the Xbox to its earlier arrival on the market. Microsoft was determined to beat Sony to the punch for the next generation, as the source explained, “MS was so focused on beating Sony this cycle that the 360 was rushed to market when all indications were that it had serious flaws. The design [quality] testing was insufficient and incomplete when the product was released to production. The manufacturing test equipment had major gaps in test coverage and wasn't reliable or repeatable. Manufacturing processes at [all] levels of suppliers were immature and not in control. Initial end to end yields were in the mid 30%. Low yields always indicate serious design and manufacturing defects.
“Management chose to continue to ship anyways, and keep the lines running while trying to solve problems and bring the yields up. Whenever something failed and there was a question about whether the test result was false, they would remove that test, retest and ship, or see if the unit would boot a game and run briefly and then ship. [The] 360 is too complex of a machine to get away with that.”
In hindsight, the Xbox 360 project team likely wishes that it had paid closer attention to its processes – though the fierce competition of the industry gave them fuelled their desire to take shortcuts. “In the end I think it was fear of failure, ambition to beat Sony, and the arrogance that they could figure anything out, that led to the decision to keep shipping,” the insider revealed. “Plus, they tend to make big decisions like that in terms of dollars. They would rationalize that if the first few million boxes had a high failure rate, a few 10's of millions of dollars would cover it. And contrasting that cost with a big lead on Sony, would pay it in a heartbeat.”
According to the Microsoft insider, the new ‘Falcon’ Xbox 360 hardware is far more reliable than the original ‘Xenon.’ “I've heard that the failure rates for the current design is sub 10%. Much much better, but still too high ... And those designs haven't seen much life yet, so no one knows if that failure rate will hold,” explained the source, adding that future revisions are in the cards. “They will come out with new hardware at least once a year until they retire this design. That's the console financial model. Keep the features and functionality the same, reduce cost and price, and improve quality if needed.”
With the Xbox 360 being the leading console in North America and the choice for gamers looking for a complete online service, Microsoft’s next focus should clearly be on getting consumers to trust the hardware.
Bill Gates said recently that it is now Microsoft’s goal to make the Xbox 360 the “most reliable” console on the market. “We've got incredible reliability on the new work we've done,” said the Microsoft chairman. “Our commitment is that it will be the most reliable video game box out there. People really love the Xbox because of the content, but we've got to make sure that the hardware never stands in the way of that.”