If you indulge in Yahoo! Fantasy Football like me, this past weekend was particularly frustrating.
For those who don’t engage in this pastime, the long and short is that their system went down, leaving me and thousands of other fans without access to our team’s information.
It was almost unforgivable…but then I received the following email from Ken Fuchs, the Head of Yahoo! Sports, and it actually makes me admire them.
I appreciate that he said “We’re sorry” without equivocation or excuses.
I appreciate the fact that he told me what happened.
And I appreciate that he gave me something as a concrete expression of his regret.
So that got me thinking about the general experience we have with organizations and leaders. More often than not, we see blame shifting or blunders in honest, timely responsiveness…and that always makes a problem devolve from bad to worse in a nanosecond.
I’d like to see if we can think of more examples like this one where an organization did the right thing, took responsibility and just plain ol’ apologized for their mistake.
Can you think of examples – from leaders and/or organizations – that have admitted when they were wrong and, in the process, moved you from anger to admiration (or, at least, assuaged your initial disappointment)?
I’m open to any examples, but am particularly interested in government scenarios.
(Can government even admit when it’s wrong…or does it hurt citizen trust even more?)
——————————–
Dear Yahoo! Fantasy Users,
As I mentioned in my earlier email, Yahoo! works hard to provide great experiences for our users every day. For you, we understand that Fantasy is a core part of your daily routine during this time of year. On Sunday we experienced technical issues which caused the Fantasy experience to be inaccessible. We’re sorry. Our entire team worked tirelessly to re-establish the service – however, we had a significant outage. We know this created inconvenience and frustration for millions of our fans, and we feel we owe you an explanation.
So what happened? At Yahoo!, we have giant machines called “filers” that process a lot of the real-time data and stats for us and for you. We do millions of calculations every hour for our games, and normally our machines can handle this with no problem. Recently, we discovered a hardware issue in one of the filers that caused the other one to overload. We replaced some hardware, re-configured the setup, and did some testing. However this Sunday – at approximately 12:15 p.m. Eastern – the new configuration failed. This created an overload on storage capacity and took the Fantasy part of our site down.
We had dozens of engineers from various teams working together to try to determine the cause and fix it. One option was to fall back on another data center, but that would have meant shutting the game down and losing scoring data. We wanted to avoid that at all costs. Ultimately, we were able to move our mobile apps to a back-up data center, free up storage to get the PC version of the game working, and get the mobile apps up in a “read-only” state – meaning you could see scores and data, but you still couldn’t set lineups and interact.
We spent Sunday night and most of Monday looking at dozens of potential causes. Monday afternoon, we stress-tested our system. Everything seemed to be in working order, so we turned on all our mobile app functions in time for Monday Night Football. Everything performed as expected and continues to do so. We’ll have all hands on deck this coming Sunday to closely monitor performance and ensure we can respond quickly in case of any abnormal activity.
We know many of you have questions about your leagues and games, so we’ve also set up a FAQ here that will explain how we are going to handle various scenarios.
Finally, we know we failed you this weekend, and there’s no way to fix that. As a token of our appreciation for your patience and continued support, we wanted to give something back. First, we’re offering a complimentary football weekly Scouting Report for the remainder of the year. Hopefully this will help you down the stretch. The report includes weekly expert analysis, rankings and news to help you make smart decisions. Second, we are offering all our users a 20% discount for the rest the year at the Yahoo! Sports Store.
For any updates, we will keep you informed on @YahooSports and facebook.com/yahoosports.
We won’t stop working to provide you with new and improved features that help you enjoy your Sundays.
Thanks again for playing and for your patience.
Best,
Ken Fuchs
Head of Yahoo! Sports