thoughthead

I had an article nearing completion and ready to go and then Microsoft dropped a bomb on us all; they finally owned up to the 3 Red Lights of Death issues happening across a large percentage of the Xbox 360 Consoles. so what brought it to this point?

Yesterday Microsoft released this press release:

REDMOND, Wash. — July 5, 2007 —Microsoft Corp. today announced that it will expand its global Xbox 360™ warranty coverage. Any Xbox 360 customer who experiences a general hardware failure indicated by three flashing red lights will now be covered by a three year warranty from date of purchase. All other existing Xbox 360 warranty policies remain in place.

As a result of what Microsoft views as an unacceptable number of repairs to Xbox 360 consoles, the company conducted extensive investigations into potential sources of general hardware failures. Having identified a number of factors which can cause general hardware failures indicated by three red flashing lights on the console, Microsoft has made improvements to the console and is enhancing its Xbox 360 warranty policy for existing and new customers.

Microsoft stands behind its products and is taking responsibility to repair or replace any Xbox 360 console that experiences the “three flashing red lights” error message within three years from time of purchase free of charge, including shipping costs. Microsoft will take a $1.05 billion to $1.15 billion pre-tax charge to earnings for the quarter ended June 30, 2007 for anticipated costs under its current and enhanced Xbox 360 policies.

“The majority of Xbox 360 owners are having a great experience with their console and have from day one. But, this problem has caused frustration for some of our customers and for that, we sincerely apologize,” said Robbie Bach, president of Microsoft’s Entertainment & Devices Division. “We value our community tremendously and look at this as an investment in our customer base. We look forward to great things to come.”

For any customer who has previously paid for repair expenses related to the three flashing lights error message on the Xbox 360 console, Microsoft will retroactively reimburse them.

Immediately following the Press Release was an Open Letter by Peter Moore.

To our Xbox Community:

You’ve spoken, and we’ve heard you. Good service and a good customer experience are areas of the business that we care deeply about. And frankly, we’ve not been doing a good enough job.

Some of you have expressed frustration with the customer experiences you have had with Xbox 360; frustration with having to return your console for service after receiving the general hardware error message on the console.

The majority of customers who own Xbox 360 consoles have had a terrific experience from their first day, and continue to, day in and day out. But when anyone questions the reliability of our product, or our commitment to our customers, it’s something I take very seriously.

We have been following this issue closely, and with on-going testing have identified several factors that can cause a general hardware failure indicated by three flashing red lights on the console. To address this issue, and as part of our ongoing work, we have already made certain improvements to the console.

We are also implementing some important policy changes intended to keep you in the game, worry-free.

As of today, all Xbox 360 consoles are covered by an enhanced warranty program to address specifically the general hardware failures indicated by the three flashing red lights on the console. This applies to new and previously-sold consoles. While we will still have a general one year console warranty (two years in some countries), we are announcing today a three-year warranty that covers any console that displays a three flashing red lights error message. If a customer has an issue indicated by the three flashing red lights, Microsoft will repair the console free of charge—including shipping—for three years from the console’s purchase date. We will also retroactively reimburse any of you who paid for repairs related to problems indicated by this error message in the past. In doing so, Microsoft stands behind its products and takes responsibility to ensure that every Xbox 360 console owner continues to have a fantastic gaming experience.

If we have let any of you down in the experience you have had with your Xbox 360, we sincerely apologize. We are taking responsibility and are making these changes to ensure that every Xbox 360 owner continues to have a great experience.

This will take a few days to roll out globally, and I appreciate your continued patience as we launch this program. I’ve posted an FAQ that should address some additional questions, and we’ll update it over the next few days.

I want to thank you, on behalf of all us at Microsoft, for your loyalty.

For those of you that have been following the issue the Xbox 360 has garnered the reputation of being a lemon of a console. That is to say there is a high enough percentage of the population experiencing hardware failure that you’re bound to hear the issue come up with in conversations with the unwashed masses where the console is mentioned; it’s synonymous.

Essentially the issue started rearing it’s ugly head almost immediately after the console launched in 2005… It essentially escalated from an “isolated issue” to “below the industry standard of 3%” to “below the industry standard of 5%” and then just a few months ago “a majority of owners are having good experiences”. I think the straw that broke the camels back was the recent article by dailytech that a large amount of anonymous sources from major retailers were seeing anywhere from 25% to 33% return rates due to hardware failure. That’s about the point where I started seeing would-be buyers go from cautiously optimistic to “I’m going to buy something else because I don’t want a $400 door stop”. At that point there was no more benefit of the doubt, no more faith in Microsoft’s claims of a small overall percentage, if you bought three consoles you were likely to see one of them die prematurely.

Xbox-Scene has been pushing the issue to the forefront since the get go. When these sort of issues arise that becomes their modus-operandi. When the Xbox 1 was found to have dangerous and faulty power supplies we pushed the issue HARD and MS eventually recalled the power cord, which wasn’t the desired result but I’d like to think it wouldn’t have even got to that point if we hadn’t done something. The 360′s issues were quite a bit more widespread, and it cut deep into the core of the market on a whole. While I don’t think we played as big a role this time around we put in our time and I think the gaming community is finally seeing from Microsoft today what we’ve wanted to see them admit to in early 2006 when it first became apparent that it was a little more than a small problem.

While I personally have never had my console fail, nor have any of my close friends. I’d like to think that there are few as familiar with the issue as I am. I started writing FAQs on Xbox-Scene mostly for my own benefit because I didn’t want to hunt down the answers to common questions every time someone asked one. Eventually I became sort of a resident FAQ writer on XS’s 360 boards. Shortly after the 360′s launch I got an anonymous email about a “secret code” you could enter into the console while it was flashing red lights and the LEDs would give you a secondary error code. It was the first I’d ever heard of it and I asked a few people to test it out and it actually worked. What started as a small blurb in a hardware FAQ grew into a massive error code FAQ.

Knowing a bit more about the issue there are a few frequent codes that people get but the most common, probably responsible for 90% of the 3RLoD errors people see is a “0102″ code which has been traced back to a problem with Graphics Processor Unit (GPU). The problem occurs in the Ball Grid Array (BGA) which is the interface in which the processor connects the main board. Unlike a traditional pin and slot interface like you’d find on a PC the Xbox 360, like most consoles is soldered directly to the board by way of a matrix of solder pads under the chip. During manufacturing these basically receive carefully measured balls of solder before the chip is placed on top of the array and then the whole main board is baked at a temperature that melts the solder all at once to the chip.

When it was first discovered that this was the culprit a few remedies started appearing, one of the popular one was “the heat-gun trick” which had owners of 3RLoD consoles cracking open the case, pulling off the heat sinks and using a heat gun on their GPU. This essentially re-flowed all of the solder in the BGA. While risky it did work for some though in the long run, it proved to be only a temporary fix. We were treating the symptoms and the not the problem so to speak.

More recently it was discovered WHY the BGA was actually getting screwed up. The heat sinks actually bolt straight through the motherboard and on the opposite side of the motherboard are two springy metal X shaped brackets referred to as the “X-Clamps” the high temperatures of the GPU coupled with the fact that it has a fairly weak heat sink (because it has to fit under the DVD drive) and a weak main board material plus the high pressure produced by the X-Clams was actually causing the main board to warp slightly and disjointing areas of the BGA where the GPU connects the main board. “Cold” solder joints in the BGA happen much more frequently because the Xbox 360 also uses eco-friendly lead free solder (which on a whole doesn’t work all that great). This discovery lead to several X-Clamp Replacement methods for fixing the issue, and these fixes were met with not only a very high percentage of success but consistent long term success. These didn’t really treat the heat issue or the weak board, but it has gamers bolting the heat sink directly to the case chassis which prevents the main board from warping. by relocating the pressure to a non-critical location.

Microsoft has recently implemented their own fix, many of those who have sent their console in for repair have found a more advanced heat sink on the GPU which includes more numerous and thinner fins and a heat pipe leading to a secondary heat sink. Main board scans from the more recent Elite consoles have shown a reinforced main board surrounding the processors, and this change is likely also reflected in newly produced units as well. In addition to both of those we should also see the long awaited move to 65nm processors which will allow the processors to run cooler and more efficiently than before.

This combination of fixes coupled with the recently extended warranty will do much in terms of good will for Microsoft and the Xbox 360. I think we got this announcement later than we should have and the fixes are being implemented in the hardware much later than they should have but I truly appreciate the gesture they’re making. It takes some big balls for a company Microsoft’s size to own up to a major issue like this under their own terms (read: not by court order). On some levels though it was really necessary for them to not lose this console market completely considering the levels in which the issue had been escalated.

I hope you all enjoy your 3 year warranty, considering I bought my console on launch day back in November of 2005 and I still have over a year left to enjoy worry free use of my console, I’m pretty content with the situation.

Microsoft, I salute you for doing the right thing, even if you were a little late to come around.