This post was originally published in AOL Gov. To read the entire article by Wyatt Kash, click here.
Federal agencies and regional data center operators, including one operated by Amazon Web Services, are still taking stock of the impact of widespread power outages that began Friday night and continue to leave large swaths of greater Washington, D.C., region without electrical power.
Many federal buildings are still without power as a result of unusually violent wind gusts Friday evening, prompting the U.S. Office of Personnel Management to give federal employees in the Washington, D.C. area the option to take unscheduled leave or work remotely. Among agencies taking advantage of the policy is the U.S. Patent and Trademark Office, where 7,000 of the agency’s 10,000 employees now routinely telework at least one day a week, according to Danette R. Campbell, senior advisor, at PTO’s Telework Office.
What is less clear, however, is how widely the power outages may have affected federal IT and communications systems used to support employees.
Northern Virginia and the greater Washington area are home not only to millions of residents, but some of the nation’s largest data center operations. The storm forced data centers in the region to scramble Friday night, including an Amazon Web Services facility, which began seeing “elevated error rates impacting a limited number” of customers,” according to status reports issued on a company blog and subsequently reported by the New York Times.
In response to the incident, an Amazon spokesperson told AOL Government Monday:
“Severe thunderstorms caused us to lose primary and backup generator power to a portion of a single Availability Zone in our US-East Region Friday night. For perspective, in our US-East Region in Virginia, we have in excess of 10 data centers.
“In the thunderstorm on Friday night, several of our data centers had their utility power impacted, but in only one of them did the redundant power not operate correctly (which ended up impacting a single digit percentage of our Amazon EC2 instances in the US-East Region). We began restoring service to most of the impacted customers Friday night and the remainder were restored on Saturday,” said the spokesperson.
Amazon’s was not the only data center that lost power.
“Our data center in Reston lost external power for a relatively short time,” said Unisys Group Vice President for Federal Systems Peter Gallagher. “Generators of course kicked in – so we were never off-line in Reston despite the storm. Our primary data centers in Salt Lake City and Eagan, Minn., were similarly not in the storm zone so our contingency operations for all customers – federal and commercial – were unaffected,” he said.
But as a growing number of organizations have come to rely on Amazon and others for storage, the type of shutdown that occurred over the weekend continues to demonstrate why cloud computing systems are still not a complete answer for organizations requiring dedicated backup options.
“Having systems in the cloud doesn’t mean that those systems are fault tolerant,” said Jason Lewis, chief scientist, Lookingglass Cyber Solutions.
“Amazon has multiple zones that can be setup for failover if a problem in one (zone) occurs. If a customer only has a presence in one location and a weather event takes down that location, then they didn’t plan for failure. I think there is a misconception that everything in the ‘cloud’ will be fault tolerant and redundant. This isn’t the case,” he said.
“The Dulles area has a huge Internet presence,” Lewis said. “I’d expect a lot of affected customers are looking at expanding their backup/fail-over plans. While recent events are rare, planning for them is part of highly available design.
According to Amazon’s blogs however, the power outage served as a reminder that organizations that rely on large scale data and cloud computing centers may face more than a temporary suspension of service in the event of regional power outages. It may also mean that the data itself can come into question, as appeared to be the case when Amazon reported at 10:36 PM PDT that Elastic Block Store storage volumes “may have inconsistent data” and in effect be “Impaired.”
Amazon, which supports a variety of Web based services for consumer, commercial and government customers, provides regular blog posts about the status of its Elastic Cloud Computing (EC2) and other data centers operations.
Those blog posts captured a glimpse of how the storm impacted Amazon’s Northern Virginia server operation as millions of area residents and business suddenly lost power from the storm: continued here.
For more news and insights on innovations at work in government, please sign up for the AOL Gov newsletter. For the quickest updates, like us on Facebook.
Someone who operates a large data center supporting mission critical apps can prob answer this better than me, but I don’t think this event slows down the cloud adoption trends. Amazon will learn from this event and adapt accordingly.
I have to put this in. First, the storm was bad, but it wasn’t that bad. Storms in the Midwest routinely have 50 mph gust, some even straight 50 mph winds in them and there isn’t half the damage as there was in the DC area. However, the Greater DC area is noted for allowing trees to literally grow over the power lines making them a prime target of falling trees. Also, if you check the soil in the area, it is so sandy, well there just isn’t that much to hold the trees down. You go down about a foot or so and find sand, like play box sand. Also, PEPCO (who supplies DC with power) is notorious for having issues during storms, as is also Old Dominion, SMECO, and BGE, only to lesser degrees. But, let’s just take a look at the area. In the last 10 years there have been hurricanes, massive floods, tornadoes, blizzards, earth quakes, and the list goes on. So let’s just be intelligent and put major data centers in the area, which incidentally is a major target for any terrorist or hostile foreign country to hit. Potentially hit with very devastating devices that if the burst radius doesn’t get them, the EMT pulse would. There are many examples where major data centers could be located, such as the Nevada desert which would eliminate the issues with trees, hurricanes, and blizzards or the salt mines around St. Louis and or all over Kansas, not to mention the old Nuc silos all over the country.
I am in hopes someone will take a long hard look at where they have located these centers and ask if they are really in the best places they could be, or just placed someplace that has prestige, even though the infrastructure of the area is falling apart.