the film forum library tutorial contact DONATE

Data Barns in a Farm Town, Gobbling Power and Flexing Muscle by James Glanz New York Times, September 23, 2012

bluefish recommends:
photo gallery at orginal site.

QUINCY, Wash. -- Set in the dry hills and irrigated farmland of Central Washington, Grant County is known for its robust harvest of apples, potatoes, cherries and beans. But for Microsoft, a prime lure was the region's other valuable resource: cheap electrical power.

The technology giant created a stir here in 2006 when it bought about 75 acres of bean fields to build a giant data center, a digital warehouse to support various Internet services. Its voracious appetite for electricity would be fed by hydroelectric generators that work off the flow of the nearby Columbia River, and Microsoft officials pledged to operate their new enterprise with a focus on energy efficiency and environmental sensitivity.

"You're talking about one of the largest corporations," said Tim Culbertson, who was the general manager of the local utility at the time. "You're talking Microsoft and Bill Gates. Wow!"

But for some in Quincy, the gee-whiz factor of such a prominent high-tech neighbor wore off quickly. First, a citizens group initiated a legal challenge over pollution from some of nearly 40 giant diesel generators that Microsoft's facility -- near an elementary school -- is allowed to use for backup power.

Then came a showdown late last year between the utility and Microsoft, whose hardball tactics shocked some local officials.

In an attempt to erase a $210,000 penalty the utility said the company owed for overestimating its power use, Microsoft proceeded to simply waste millions of watts of electricity, records show. Then it threatened to continue burning power in what it acknowledged was an "unnecessarily wasteful" way until the fine was substantially cut, according to documents obtained by The New York Times.

"For a company of that size and that nature, and with all the 'green' things they advertised to me, that was an insult," said Randall Allred, a utility commissioner and local farmer.

A Microsoft spokeswoman said the episode was "a one-time event that was quickly resolved."

Internet-based industries have honed a reputation for sleek, clean convenience based on the magic they deliver to screens everywhere. At the heart of every Internet enterprise are data centers, which have become more sprawling and ubiquitous as the amount of stored information explodes, sprouting in community after community.

But the Microsoft experience in Quincy shows that when these Internet factories come to town, they can feel a bit more like old-time manufacturing than modern magic.

In Santa Clara, Calif., a hub of technology facilities in Silicon Valley, diesel emissions from generators at a Microsoft data center caught the attention of regulators for potentially threatening the health of workers at nearby businesses. Microsoft, which was notified by state regulators last year, says it has reduced its emissions.

Over the last few years, Quincy has become an unlikely technology outpost, with five data centers and a sixth under construction. Far from the software meccas of Northern California or Seattle, Quincy has barely 6,900 residents, two hardware stores, two supermarkets, no movie theater and a main drag, State Route 28, whose largest buildings are mostly food packers and processors. Its tallest building is a grain elevator.

"A farming community in the middle of a desert," said Warren Morgan, the president of Double Diamond Fruit.

The remarkable scale of the Quincy data centers, and their power demands, have made this town something of a test tube for studying the planet's exploding need to house and process digital information.

The data centers, which include Yahoo and Dell facilities, wound up in Quincy by way of the Columbia. The river flows 1,200 miles from the mountains of British Columbia to the spectacular gorge between Oregon and Washington, where the water crashes into the Pacific Ocean.

Along the way, about a dozen large hydroelectric dams tame the river, providing irrigation for farms and the cheap, plentiful power that has become a magnet for large agricultural operations and heavy industries like aluminum, steel, paper and chemical plants.

When Microsoft was searching the country for a location for its new installation, the Grant County Public Utility District, which owns two of the dams, says it offered the company rates that would range from 2.5 cents to 3.8 cents per kilowatt-hour in its first five years -- far below the national industrial average of 6 cents to 7 cents, according to analysis based on federal figures by the Electric Power Research Institute. The power from dams is also highly reliable, a critical factor for data centers, which can crash with the slightest interruption.

Beyond power, Washington State has awarded the industry lucrative tax breaks, ostensibly to promote growth in rural areas. Although the initial expectations that private fortunes would be made on land sales and housing developments were quickly dashed, Quincy's revenue from property taxes, which data centers do pay, has risen from $815,250 in 2005 to a projected $3.6 million this year, paying for a library and repaved streets, among other benefits, according to Tim Snead, the city administrator.

A New Era Begins

The ribbon cutting on April 16, 2007, for Microsoft's "server farm," as the buildings containing thousands of modular computers or servers are often called, had all the trappings of a proud civic unveiling, with speeches by area dignitaries. Michael Manos, the company's general manager for data center services at the time, walked away with a small bag of beans from the field's final crop. It carried a message: "Preparing the Site for Another Farmer: Microsoft."

"We thought that Microsoft would bring a certain air of class to our town," said Danna Dal Porto, a retired teacher.

Just three days after the ribbon cutting, Microsoft began flexing its muscle. Mr. Manos wrote to the utility commissioners complaining that they were slow in building a substation to provide 48 million watts of electrical capacity to Microsoft. That would be enough to power about 29,000 American homes, according to an analysis based on federal figures conducted by the Electric Power Research Institute -- about four homes for every person in Quincy.

Mr. Manos said the pace of construction "dramatically affects our agility as a business," adding that "our confidence is becoming quite shaky." If construction could not be accelerated, Mr. Manos asked, would Microsoft be eligible for $700,000 in reimbursements?

Some local officials were taken aback at what Mr. Culbertson, the former utility general manager, called "a level of arrogance."

"Microsoft had lot of expectations," he said. "Early on, I don't think it was as cooperative as it could have been."

The stakes for Microsoft were high. According to current and former company employees, its Quincy servers ran Bing -- its challenge to Google's search engine -- the Hotmail service and other so-called cloud functions.

While the term "cloud" is often used loosely to refer to remote memory or other computing services accessed by the Internet, it is hardly some vaporous formation.

"Quite simply, data centers are the cloud," Eric S. Laschever, a Microsoft lawyer, said during the legal challenge to its backup generators. "You've seen it on TV. The heart of the cloud are these data centers, and the data centers are really at the heart of Microsoft's business."

Microsoft's operation has now spread to four buildings and is the largest of Quincy's data centers. Taken together, Microsoft and Yahoo's operations overwhelm all nonindustrial electric usage, utility figures show. All residential and small commercial accounts in Quincy consumed an average of 9.5 million watts last year, while Microsoft and Yahoo used 41.8 million watts, the utility said.

The loads are growing so fast that some local residents and business owners -- particularly irrigation farmers, who also depend on low-cost electricity -- are concerned. With other industries also chasing low electricity prices, the increases could lead to higher prices or even a shortage of available power from the dams.

Sarah Morford, a spokeswoman for the utility, said that it did not expect the capacity of the dams to be exceeded "in the foreseeable future."

Even so, the growing data centers have given the City of Quincy's Web site a new motto: "Where Agriculture Meets Technology!"

A Diesel Dependence

Not long after Microsoft arrived in 2006, Robert Koster, an environmental engineer in the Spokane office of the Washington State Department of Ecology, was assigned to review the company's request for permits for 24 diesel generators.

Such huge backup generators, which can weigh thousands of pounds and stand over 10 feet tall, produce thousands of horsepower -- enough to generate two million to three million watts each.

Back then, Mr. Koster said, he had little experience with data centers and no inkling of the avalanche of servers about to descend on Quincy. "Microsoft was our first indication that we were going to see this kind of project here," Mr. Koster said. "At the time, we were in scramble mode to permit our first one of these data centers."

Although emissions containing diesel particulates are an environmental threat, they were was not yet classified as toxic pollutants in Washington. The original permit did not impose stringent limits, allowing Microsoft to operate its generators for a combined total of more than 6,000 hours a year for "emergency backup electrical power" or unspecified "maintenance purposes."

In 2010, during an expansion of the data center, Microsoft repeatedly ran the center on generator power. The Microsoft spokeswoman, Andrea Platt, said the company was forced to rely on the generators "at certain times" that year because the utility needed to perform work on a substation.

The utility, however, said its documents indicated that Microsoft asked to be disconnected from the grid.

Several experts said in interviews that they found the episode confusing because data center designs typically anticipate expansions and allow construction to take place without disconnecting from the grid.

For example, Yahoo's Quincy data center, which expanded around the same time, ran its generators just 65 hours in 2010, state records show. Microsoft's generators operated for a combined total of 3,615 hours that year, the records show.

In Silicon Valley, Microsoft's heavy use of generators at its data center there drew the attention of the California authorities, who were already regulating diesel particulate emissions because of their carcinogenic properties.

In 2008 and 2009, the Bay Area Air Quality Management District listed Microsoft's Santa Clara data center as one of the largest stationary diesel polluters in the Bay Area, along with facilities including a major rock quarry and Alcatraz Island, which relied solely on diesel generators until it installed solar panels this year.

Microsoft had been granted emergency permission to run its generators more often in 2008 and 2009 while it made repairs to dangerously corroded electrical components.

Christian Belady, Microsoft's general manager for data center services, said the company substantially reduced its generator use after the overhaul. But in October, Microsoft was informed that the increased emissions had prompted a review under California's Air Toxics "Hot Spots" Program. The notice said the potential for diesel emissions to cause cancers among employees in more than a dozen nearby businesses was above the program's threshold.

In responding to the notification, Microsoft pointed out that its emissions had dropped since the emergency repairs and produced computer models of the diesel exhaust that appeared to show a carcinogenic potential below the threshold, said Jane Lundquist, the engineer who conducted the original review for the Bay Area district. The case is still under consideration, said Brian Bateman, the district's health and science officer.

In Quincy, Microsoft's reliance on generators did not draw the same response. But some people noticed.

"When they first start up, a big, huge cloud of black smoke comes up," said Ronald Carden, a forklift driver at a nearby fruit warehouse. "It just kind of makes you nauseous."

While others in town said they were hardly aware of the generators, Patty Martin, a former Quincy mayor and environmental activist, made them an issue. Ms. Martin and Ms. Dal Porto, the retired teacher, formed a group called Microsoft-Yes; Toxic Air Pollution-No. When the State Ecology Department granted Microsoft permits for new backup generators for the expanded center, they appealed the decision to the state's Pollution Control Hearings Board.

During two days of hearings in February, Ms. Martin squared off against a team of lawyers for Microsoft and an assistant attorney general representing the Ecology Department. In July, the board issued a ruling that confirmed the permits allowing Microsoft to increase the number of generators to 37 at its Columbia Data Center. The group has appealed that decision to a Superior Court.

One clause in the new permit, though, indicated that Ms. Martin's opposition had raised concerns. It ordered Microsoft to meet with administrators of the nearby Mountain View Elementary School and "provide the school administrators with a direct telephone contact to one of the Columbia Data Center managers" as well as a schedule of when it would test generators.

Burton Dickerson, the superintendent of the Quincy School District, said that because the school was close to both Microsoft's data center and one run by Dell, he was concerned enough to install particulate monitors. He said that there were no dangerous levels "the nearest we could tell," but that the district did not have the technical expertise to separate out the diesel emissions in the readings.

Mr. Dickerson said that in the end, the district was placing "quite a lot of confidence" in the state "to assure that we're not in any great danger." He added that Microsoft and the other data centers had abided by the permit provisions requiring communication with the school.

Mr. Belady of Microsoft said that generator use at Quincy had dropped in each of the last two years. "From both an operational cost and a system reliability perspective, we seek to minimize the amount of time we operate our backup generators," he said, referring both to Quincy and to Santa Clara.

Even so, as Quincy's data centers grow, so do its diesel generators. The State Ecology Department says enough permits for generating power have been issued in Quincy to eventually pump out 337 million watts -- roughly a third of the output of a major nuclear power plant. Ms. Martin and her group are now challenging generators for an expansion of Yahoo's data center as well as for facilities run by Dell and Sabey.

Those cases are pending, according to the state attorney general's office.

Meanwhile, some Ecology Department officials involved in approving the backup generators now express doubts about the soaring number of them in Quincy.

"I find it hard to believe that this is the best way to store data," Mr. Koster said. "Something's flawed in that thought process."

Burning Off a Fine

In the midst of the tussle over its generators, Microsoft received notice last December that it was likely to face a penalty from the Grant County Public Utility District. Surprisingly, it was for using too little power.

Microsoft was in violation of an obscure provision called Rate Schedule 99.

Utilities perform a constant balancing act, matching the demand from its customers with its supply of power. The Grant County utility must carefully plan for how much its big customers need or risk losing money it could earn by selling the excess power elsewhere -- a cost all of its customers would have to absorb.

After several years of poor forecasts based on input from the large industrial users, the utility instituted the provision. It requires large industrial customers to file load forecasts each fall for the next calendar year and face a penalty if they are off by a significant margin in either direction.

At the end of last year, said Ms. Morford, the utility spokeswoman, just two customers were in line for penalties for using less energy than they had asked the utility to set aside for them: data centers operated by Yahoo and Microsoft.

Yahoo paid its $94,608 penalty, Ms. Morford said. Microsoft took a different tack.

That is when Microsoft threatened to waste tremendous amounts of power by simply running giant heaters for no purpose, according to utility officials who said they were briefed on the matter by Microsoft, unless the penalty was largely forgiven. The idea was to burn the power fast enough to move closer to the forecast before year's end.

Documents related to the case and interviews with utility officials show that Microsoft started burning roughly an additional five million to seven million watts -- well over half of the consumption of the entire town of Quincy -- in mid-December.

On Dec. 16, Microsoft delivered its ultimatum to the utility in a letter from Darrell Amundson, the facility's manager.

The letter stated bluntly that Microsoft "has the alternative available to it of increasing power utilization" in a "commercially unproductive" manner.

"By staff estimate," the letter said, "Microsoft could incur approximately $70,000 in power costs to avoid the $210,000 penalty, resulting in real savings of $140,000. Microsoft must make the decision on continuing to burn $70,000 worth of power in the next three days."

Ms. Morford said Microsoft's power use jumped from 28.5 million watts on Dec. 16 to 34 million watts on Dec. 19.

"It was very well recognized by the board as well as management that this was a very unusual situation," said Kevin Nordt, the utility's chief financial officer. The utility board, he said, focused on avoiding wasted energy without shifting costs to the utility's other customers. In a special weekend session, the board voted 4 to 1 to waive all but $60,000 of the penalty.

Mr. Belady, the Microsoft official, said the board's resolution "eliminated the illogical financial incentive for Microsoft to consume unnecessary power in order to avoid a larger fine."

Mr. Manos, the former Microsoft data center chief who had pledged to operate in an environmentally sensitive way, said he was surprised by the company's response to the penalty. "Those types of decisions would not have been part of the program's initial inception," he said.

The Microsoft spokeswoman, Ms. Platt, said the company remained committed to the environment. "Microsoft's focus on efficiency and resource utilization has not changed," she said.

Aside from a few brief notices in local newspapers, the episode barely registered in the state or in the wider digital world. And among local leaders, Microsoft has mostly retained its support, in part because the data centers are increasing tax revenues.

But Mr. Morgan, the president of Double Diamond Fruit, said the positive impact over all had been far less than many people imagined. As for all the digital services that data centers power around the country, Mr. Morgan said, "I understand that it's a necessary situation for us as a society and the way we want to live."

"But I don't think it's benefiting Quincy," he said. "I think we're taking one for the team, to tell you the truth."

Correction: September 24, 2012 This article has been revised to reflect the following correction:
A previous version of this article misstated why Microsoft wasted millions of watts of electricity, according to records. It was an attempt to erase a $210,000 penalty the utility said the company owed for overestimating its power use, not underestimating its power use.