Why the site was down on tuesday.

PG&E’s faulty equipment reveals the Internet’s vulnerability to a disruption of its power source

Verne Kopytoff, Chronicle Staff Writer

Thursday, July 26, 2007

All it took to wipe out some of the Internet's biggest sites Tuesday was some faulty PG&E electrical breakers that caused a blackout in downtown San Francisco.

Some of the Web's hottest destinations - Craigslist, Yelp, Second Life - were suddenly inaccessible from San Mateo to Singapore after back-up generators failed at the facility housing their computer equipment.

Although mostly fixed within 12 hours, the incident shows how easy it is to send major swaths of the online world to the dark ages. Sites that millions of people rely on can be knocked offline by freak accidents, not to mention major catastrophes, and this event served as a wake-up call to the executives that operate them.

"If the data center was that vulnerable to a power outage, what if something really catastrophic happened like an earthquake?" asked Derek Gordon, marketing vice president for Technorati, a search engine of blogs that was brought down for a couple of hours Tuesday after the blackout. "What does that say about the vulnerability of the Internet in the Bay Area?"

The troubles started when 365 Main, a key data center in downtown San Francisco that touts its "state-of-the-art electrical system," failed to get its backup generators started immediately after the power outage hit around 1:45 p.m. A number of companies that house their computer servers in the facility were suddenly offline, setting off a mad scramble to get the Web sites up and running.

Shoppers at RedEnvelope, an online retailer, couldn't buy monogrammed pillow soaps. Hipsters on Yelp, the review site, had to take a break from sharing their reports of fabulous and not-so fabulous restaurants. Users of online classified service Craigslist were out of luck in finding a second-hand futon.

The backup generators were turned on 45 minutes after the blackout started, a delay that 365 Main said it was still investigating yesterday. But it took some of the facility's customers anywhere from another hour to 11 hours to get their servers safely rebooted and their Web sites operational.

What the episode exposed is that some companies operate entirely from one data center, a decision described by some security experts as risky. In emergencies, such companies can't shift traffic to an alternative facility where they keep additional servers.

"There's all kinds of things that can happen from a power outage to a tornado to a backhoe," said Jason Needham, director of product management at F5 Networks, a Seattle company that sells software and equipment for data centers. "All these things seem far-fetched until they happen."

However, Needham said the trend is for companies to put all their eggs in one basket, so to speak, in an effort to save money. In fact, just hours before Tuesday's power outage, 365 Main put out a press release trumpeting the fact that RedEnvelope had moved all its operations to its facility and closed an unneeded center in the Midwest.

Data centers are usually designed with redundant equipment to ensure power during outages, earthquakes and floods. Backup electricity is supposed to kick in within seconds after an outage through a complex system that keeps servers humming without interruption.

Internet companies pay thousands of dollars, minimum, to house a few servers in such a secure environment. In most cases, the precautions work.

"The well-designed data centers, this should be handled pretty seamlessly," said Michael Foust, chief executive of Digital Realty Trust, a San Francisco company that owns 60 data centers globally, including one in San Francisco that was not affected by the outage.

However, the rare snafu or major disaster can create havoc, such as the Sept. 11, 2001, terrorist attacks or the flooding of New Orleans caused by Hurricane Katrina. In those cases, many companies were thrown offline.

Google, Yahoo and eBay all keep multiple data centers across the world, partly because proximity makes Web pages load faster and partly for security.

Jim Buckmaster, chief executive of Craigslist, which got back to normal operations after 11 hours of scrambling, said that his company is in the process of moving to a second center but that it will take a few more weeks.

Gordon, from Technorati, called opening several data centers ruinously expensive for thinly funded Internet startups, of which there are hundreds in the Bay Area. Only profitable companies can afford such an extravagance, he said, though he acknowledged that Technorati, which isn't profitable, is in the process of moving into a second facility.

Tuesday's outage "added to the sense of urgency," Gordon said.

"The lesson here is despite all of your planning and all of your promises, you are vulnerable."

E-mail Verne Kopytoff at [email protected].

This article appeared on page C - 1 of the San Francisco Chronicle