All of this seemed to change abruptly, as if a switch had been flipped, when Microsoft announced that they would be extending their console’s warranty a full 3 years for RROD errors. I started to see people start quoting that turn around times were going from 1-2 weeks to 8-9 weeks. That’s rather dramatic, though it seemed that maybe it was only growing pains from the hoards of new complains from those that had just been added back under warranty coverage.

My opinion changed when I read an article posted up on Achieve360points.com. Speaker Ender, a moderator there, posted, in detail, his experience with Microsoft’s 1-800-4-MY-XBOX support line. A horrific Ordeal indeed. What followed was an onslaught of others posting their similar experiences with the “new” Microsoft support plan.

What I find most interesting is not just the incompetence on the part of Microsoft but the outright abuse coupled with deliberate organized and widespread “run-around” tactic doled out by the higher level staff. Quite frankly it’s appalling. Even more scary is if even a fraction of these consumers record their experiences it would be perfect fodder for a class-action lawsuit. I realize Microsoft extended their warranty to help prevent such a scenario but in order to prevent it they actually have to make good on their proposal.

I’m posting this here because I like the guys at a360p, and I’d like to see Speaker Ender’s issues resolved. More importantly though, I would like to see this issue resolved for everyone that has seen these kinds of problems when dealing with Microsoft Customer Support. As well as the DRM issues that arise with Xbox Live content when moving to a new console.

As consumers the only way we’ll ever see change is if we make our voices heard. I encourage you to share your experiences with Microsoft Customer Support and spread this message along to anyone you can.

I was deeply disturbed by this situation and took it upon myself to write my state’s representative, and senators. I originally wanted to send my correspondences by physical letter signed by hand to help get my point across, however I was unable to find the appropriate address to use such a method and instead passed on my concerns by email. I didn’t have very high hopes for the outcome of my message as when it comes to the US Government I am somewhat of a pessimist. However I also believe that unless I do my part to make my voice heard I have only myself to blame if nothing comes of it. By speaking my peace I can with good conscience lay the blame on those who have listened and remained idle in their actions to right these wrongs.

Read the letter that I sent to House Representative Paul Hodes, as well as Senator Judd Gregg, and Senator John E. Sununu.

The First Reply I got back was from Senator Sununu. The last time I had written this man it was about Net Neutrality and the response I had received to that was something analogous to: you’re wrong, the bill is right, big Telco told me so. My Fiancee had also written a letter to him concerning Net Neutrality and received the same exact form letter response. Sununu’s reply to my letter about the ICE raids was slightly more positive, albeit not by much. Read it yourself; I got the impression that he’s pretty much saying: I’m not even going to bother with this issue unless there is a bill in my face, and even then I might not even bother taking into account your concerns.

The Second Reply I received was from Senator Gregg. He isn’t even technically the senator for my region of the state but I decided to write to him anyway. I was actually quite pleased with the response I received back from Senator Gregg. The basic impression I got was that he took what I was saying to heart, he admitted to not knowing many details of the case but he was going to make an effort to learn more from ICE and that he would be in contact with me if he needed more details. I was quite pleased with the response, though I’m staying cautiously optimistic about whether or not anything will come of it. I’ve heard replies like this before in the corporate world and while good intentions are there it quite often gets pushed aside in the face of more pressing issues. While this is one of the issues I hold closest to my heart, I have no delusions that this likely very much a low priority for nearly all of congress.

The Third reply I received was from House Representative Paul Hodes. Despite the fact that I received Paul Hodes’ response last he was the only one who sent it via email. If I recall correctly I never received a response from him concerning the Net Neutrality issue so I was pleased to see that at least someone read my letter. Though if you read it yourself you will probably find yourself scratching your head in the same way I was. No I did not link the wrong response, and no I have never contacted this man with any interests outside of Net Neutrality and the ICE raids. I really hope Mr Hodes reads this and feels embarrassed by it. Sir, if you are reading this, you are welcome to send me a real response and I will gladly post it here.

I do hope that others who felt as I did about the actions of ICE wrote their congress persons about it. I would very much like to see some of the replies you may have received or at least your thoughts on the matter.

If you neglected to write your congress persons despite your feeling against the actions of ICE then shame on you. If you’re not doing something to get your voice heard while our peers are having their lives destroyed by something that should not be a crime then you might as well be the one raiding their homes.

We interrupt this weeks regularly scheduled discussion to take a moment and reflect on the recent events in the modding community. Read this article on Xbox-Scene.

Yesterday Microsoft released this press release:

REDMOND, Wash. — July 5, 2007 —Microsoft Corp. today announced that it will expand its global Xbox 360™ warranty coverage. Any Xbox 360 customer who experiences a general hardware failure indicated by three flashing red lights will now be covered by a three year warranty from date of purchase. All other existing Xbox 360 warranty policies remain in place.

As a result of what Microsoft views as an unacceptable number of repairs to Xbox 360 consoles, the company conducted extensive investigations into potential sources of general hardware failures. Having identified a number of factors which can cause general hardware failures indicated by three red flashing lights on the console, Microsoft has made improvements to the console and is enhancing its Xbox 360 warranty policy for existing and new customers.

Microsoft stands behind its products and is taking responsibility to repair or replace any Xbox 360 console that experiences the “three flashing red lights” error message within three years from time of purchase free of charge, including shipping costs. Microsoft will take a $1.05 billion to $1.15 billion pre-tax charge to earnings for the quarter ended June 30, 2007 for anticipated costs under its current and enhanced Xbox 360 policies.

“The majority of Xbox 360 owners are having a great experience with their console and have from day one. But, this problem has caused frustration for some of our customers and for that, we sincerely apologize,” said Robbie Bach, president of Microsoft’s Entertainment & Devices Division. “We value our community tremendously and look at this as an investment in our customer base. We look forward to great things to come.”

For any customer who has previously paid for repair expenses related to the three flashing lights error message on the Xbox 360 console, Microsoft will retroactively reimburse them.

Immediately following the Press Release was an Open Letter by Peter Moore.

To our Xbox Community:

You’ve spoken, and we’ve heard you. Good service and a good customer experience are areas of the business that we care deeply about. And frankly, we’ve not been doing a good enough job.

Some of you have expressed frustration with the customer experiences you have had with Xbox 360; frustration with having to return your console for service after receiving the general hardware error message on the console.

The majority of customers who own Xbox 360 consoles have had a terrific experience from their first day, and continue to, day in and day out. But when anyone questions the reliability of our product, or our commitment to our customers, it’s something I take very seriously.

We have been following this issue closely, and with on-going testing have identified several factors that can cause a general hardware failure indicated by three flashing red lights on the console. To address this issue, and as part of our ongoing work, we have already made certain improvements to the console.

We are also implementing some important policy changes intended to keep you in the game, worry-free.

As of today, all Xbox 360 consoles are covered by an enhanced warranty program to address specifically the general hardware failures indicated by the three flashing red lights on the console. This applies to new and previously-sold consoles. While we will still have a general one year console warranty (two years in some countries), we are announcing today a three-year warranty that covers any console that displays a three flashing red lights error message. If a customer has an issue indicated by the three flashing red lights, Microsoft will repair the console free of charge—including shipping—for three years from the console’s purchase date. We will also retroactively reimburse any of you who paid for repairs related to problems indicated by this error message in the past. In doing so, Microsoft stands behind its products and takes responsibility to ensure that every Xbox 360 console owner continues to have a fantastic gaming experience.

If we have let any of you down in the experience you have had with your Xbox 360, we sincerely apologize. We are taking responsibility and are making these changes to ensure that every Xbox 360 owner continues to have a great experience.

This will take a few days to roll out globally, and I appreciate your continued patience as we launch this program. I’ve posted an FAQ that should address some additional questions, and we’ll update it over the next few days.

I want to thank you, on behalf of all us at Microsoft, for your loyalty.

For those of you that have been following the issue the Xbox 360 has garnered the reputation of being a lemon of a console. That is to say there is a high enough percentage of the population experiencing hardware failure that you’re bound to hear the issue come up with in conversations with the unwashed masses where the console is mentioned; it’s synonymous.

Essentially the issue started rearing it’s ugly head almost immediately after the console launched in 2005… It essentially escalated from an “isolated issue” to “below the industry standard of 3%” to “below the industry standard of 5%” and then just a few months ago “a majority of owners are having good experiences”. I think the straw that broke the camels back was the recent article by dailytech that a large amount of anonymous sources from major retailers were seeing anywhere from 25% to 33% return rates due to hardware failure. That’s about the point where I started seeing would-be buyers go from cautiously optimistic to “I’m going to buy something else because I don’t want a $400 door stop”. At that point there was no more benefit of the doubt, no more faith in Microsoft’s claims of a small overall percentage, if you bought three consoles you were likely to see one of them die prematurely.

Xbox-Scene has been pushing the issue to the forefront since the get go. When these sort of issues arise that becomes their modus-operandi. When the Xbox 1 was found to have dangerous and faulty power supplies we pushed the issue HARD and MS eventually recalled the power cord, which wasn’t the desired result but I’d like to think it wouldn’t have even got to that point if we hadn’t done something. The 360′s issues were quite a bit more widespread, and it cut deep into the core of the market on a whole. While I don’t think we played as big a role this time around we put in our time and I think the gaming community is finally seeing from Microsoft today what we’ve wanted to see them admit to in early 2006 when it first became apparent that it was a little more than a small problem.

While I personally have never had my console fail, nor have any of my close friends. I’d like to think that there are few as familiar with the issue as I am. I started writing FAQs on Xbox-Scene mostly for my own benefit because I didn’t want to hunt down the answers to common questions every time someone asked one. Eventually I became sort of a resident FAQ writer on XS’s 360 boards. Shortly after the 360′s launch I got an anonymous email about a “secret code” you could enter into the console while it was flashing red lights and the LEDs would give you a secondary error code. It was the first I’d ever heard of it and I asked a few people to test it out and it actually worked. What started as a small blurb in a hardware FAQ grew into a massive error code FAQ.

Knowing a bit more about the issue there are a few frequent codes that people get but the most common, probably responsible for 90% of the 3RLoD errors people see is a “0102″ code which has been traced back to a problem with Graphics Processor Unit (GPU). The problem occurs in the Ball Grid Array (BGA) which is the interface in which the processor connects the main board. Unlike a traditional pin and slot interface like you’d find on a PC the Xbox 360, like most consoles is soldered directly to the board by way of a matrix of solder pads under the chip. During manufacturing these basically receive carefully measured balls of solder before the chip is placed on top of the array and then the whole main board is baked at a temperature that melts the solder all at once to the chip.

When it was first discovered that this was the culprit a few remedies started appearing, one of the popular one was “the heat-gun trick” which had owners of 3RLoD consoles cracking open the case, pulling off the heat sinks and using a heat gun on their GPU. This essentially re-flowed all of the solder in the BGA. While risky it did work for some though in the long run, it proved to be only a temporary fix. We were treating the symptoms and the not the problem so to speak.

More recently it was discovered WHY the BGA was actually getting screwed up. The heat sinks actually bolt straight through the motherboard and on the opposite side of the motherboard are two springy metal X shaped brackets referred to as the “X-Clamps” the high temperatures of the GPU coupled with the fact that it has a fairly weak heat sink (because it has to fit under the DVD drive) and a weak main board material plus the high pressure produced by the X-Clams was actually causing the main board to warp slightly and disjointing areas of the BGA where the GPU connects the main board. “Cold” solder joints in the BGA happen much more frequently because the Xbox 360 also uses eco-friendly lead free solder (which on a whole doesn’t work all that great). This discovery lead to several X-Clamp Replacement methods for fixing the issue, and these fixes were met with not only a very high percentage of success but consistent long term success. These didn’t really treat the heat issue or the weak board, but it has gamers bolting the heat sink directly to the case chassis which prevents the main board from warping. by relocating the pressure to a non-critical location.

Microsoft has recently implemented their own fix, many of those who have sent their console in for repair have found a more advanced heat sink on the GPU which includes more numerous and thinner fins and a heat pipe leading to a secondary heat sink. Main board scans from the more recent Elite consoles have shown a reinforced main board surrounding the processors, and this change is likely also reflected in newly produced units as well. In addition to both of those we should also see the long awaited move to 65nm processors which will allow the processors to run cooler and more efficiently than before.

This combination of fixes coupled with the recently extended warranty will do much in terms of good will for Microsoft and the Xbox 360. I think we got this announcement later than we should have and the fixes are being implemented in the hardware much later than they should have but I truly appreciate the gesture they’re making. It takes some big balls for a company Microsoft’s size to own up to a major issue like this under their own terms (read: not by court order). On some levels though it was really necessary for them to not lose this console market completely considering the levels in which the issue had been escalated.

I hope you all enjoy your 3 year warranty, considering I bought my console on launch day back in November of 2005 and I still have over a year left to enjoy worry free use of my console, I’m pretty content with the situation.

Microsoft, I salute you for doing the right thing, even if you were a little late to come around.

Next week I’ll be back with an in depth article on building a home theater. You can read my post on Xbox-Scene to get an idea of where I’m starting from.