Boosting Data Center Efficiency

Innovate and Thrive in a Difficult Economy

Since 2005, the power required for every $1,000 of IT equipment (servers, storage, and networks) has grown by four to six times, dramatically raising the “hidden” costs of operating a data center and setting the stage for significant problems. As a result, corporate data centers will have to become much more efficient in terms of both power density and asset utilization.

If you deploy the latest technology without boosting utilization of those assets you’ll potentially waste a huge amount of power (at significant cost to your company). More important, if you fail to raise utilization levels your data center will likely become constrained to the point where you can’t deliver the power and cooling necessary to support the latest technology. In fact, 42–43% of all data centers in Europe and the United States are already power constrained.

In this article, I explore this problem in a bit more detail, discuss how you can enhance your data centers and IT practices to improve efficiency, and describe the implications of these changes for storage administrators.

The Hidden Cost of IT

If you deploy that low-cost server in an enterprise-class, tier-2 data center, the direct costs to power and cool it for a year could amount to as much as $8,300. The associated operational expenses would be about $1,320 annually, covering the depreciation of the capital investment, electricity, and site operations (security, etc.).

Most IT departments do not include these costs in their business justifications for new server deployments. In fact, most IT organizations may never see a power bill until their data centers are out of power—so they have no idea these hidden costs even exist.

Since these costs are not seen, data center managers may not have the incentive to control them. At current growth rates we would need to build more than 10 new power plants in the United States alone, at a cost of $2–$6 billion each, to meet the growing power demands from data centers.

Of U.S. and EU companies surveyed:

• 47% do not monitor server utilization.
• 55% of data center managers are not aware of their monthly power costs.
• 43% of EU data centers are power constrained.
• 42% of U.S. data centers will run out of power in the next 18–24 months.

The key to getting your data center ready for the future as well as controlling hidden costs is to increase power efficiency and boost asset utilization to decrease the amount of equipment you need and slow the rate of future acquisition. You can accomplish this by:

• Designing facilities to support higher power density
• Rearchitecting core networks
• Virtualizing servers and storage so you can scale and move applications more fluidly

At NetApp, we’ve gained a lot of experience with all three of these items through efforts to improve the efficiency of our own data centers.

Redesigning Data Center Facilities for Efficiency and Density

To boost efficiency and density in NetApp data centers, we found that we needed to challenge a lot of old assumptions. Here are some of the things we changed on our way to increased efficiency:

• Facilities management and IT strategy are typically discrete functions, but we found that teaming up results in more energy-efficient solutions.
• Raised floors in a data center are artifacts of mainframe days. A cold aisle/hot aisle layout enables power and cooling delivery from overhead and is much more energy efficient.
• High-tech problems can sometimes be solved with low-tech solutions. A vinyl curtain can create a physical barrier between hot and cold aisles, saving more than 1 million KWh per year.
• Data centers don’t have to be cold. We’ve gradually moved up the supply air temperature from 52º F to 70º F, and we let our hot aisle temperatures go as high as 95º F. This has resulted in less cooling and more hours of free cooling using outside air.

One important measure of data center efficiency is power usage effectiveness (PUE), the ratio of total facility power used to power used directly by IT equipment. Possible sources of electricity consumption are illustrated in Figure 1.

Figure 1) Sources of power consumption.

The current benchmark for good data center design is a PUE of 2.0: Everything else in the data center consumes as much power as the IT equipment. Other important measures of data center efficiency include:

• How much power you can deliver per rack. A typical value today is about 3kW. In other words, the equipment deployed in the rack can draw a maximum of 3kW of power.
• How much space you need per rack. A rack in a typical data center can require as much as 28 square feet of floor space.

By applying techniques such as those mentioned above, NetApp has been able to lower the PUE in its data centers substantially below the 2.0 benchmark while delivering much more power per rack and consuming much less space.

For instance, our second-generation data center design uses variable air volume (VAV) control and a cold aisle damper to achieve a PUE of 1.40 with 801 racks drawing 3.5KW/rack. This saves the company approximately $1.7 million per year versus the same data center operating at a PUE of 2.0.

Figure 2) NetApp second-generation data center design.

Our third-generation design resulted in a data center with 720 racks drawing 8KW/rack with a PUE of 1.30 that will save the company $4.3 million per year versus a PUE of 2.0. We’re also currently deploying a new combined IT and engineering data center that will include 1,800 engineering racks capable of delivering 12KW/rack with a predicted PUE of 1.2. Hot air from the data center is used to heat office space during the winter months.

Figure 3) NetApp third-generation data center design.

You can read more about the strategies that we use to boost data center efficiency in a recent Tech OnTap articleas well as a recent white paper.

Increasing the efficiency and density of your data center will only get you so far. You also need to substantially increase the utilization of the IT assets you deploy in your data centers by rearchitecting your core networks, consolidating as much as possible, and virtualizing both servers and storage. You can read about one approach to full data center virtualization using a combination of the Cisco Unified Compute System (UCS) and NetApp® unified storage in a companion article in this issue.

Rearchitecting Core Networks

In terms of core networks, there are two objectives:

• Standardize on a unified network fabric.
• Rearchitect your networks so that applications aren’t “boxed in.”

The arrival of Converged Network Adapters (CNAs) and Fibre Channel over Ethernet (FCoE) makes it possible to incorporate your existing Fibre Channel devices on a single network fabric. Eliminating separate storage network infrastructures can further reduce your data center power consumption. By wiring once you can connect to any network—SAN, LAN, or HPC—for faster rollout of new applications and services.

Most current networks are designed so that key business applications are isolated for security and performance. Unfortunately, this localizes IT resources and limits utilization. By opening up your networks, you make it possible for every host to mount any storage target. This drives storage consolidation, boosts utilization, and sets the stage for full virtualization of servers and storage.

Virtualize Everything

Rearchitecting your core networks makes it possible to take the greatest advantage of server and storage virtualization. Existing resources can be fully consolidated and pools of resources can be efficiently brought to bear on any application. Because applications are free to move to any set of resources in the data center, or to alternate data centers, downtime can be significantly reduced.

NetApp has moved aggressively to consolidate older servers and storage. For example, a review of our engineering labs identified 4,600 x86 client servers that are candidates for virtualization. These systems result in about $1.4 million in power and cooling costs and consume 192 racks of data center space. Full virtualization of these systems at 20:1 will result in a reduction in the total system count to 230 servers, resulting in about $70,000 in annual power and cooling and occupying only 10 racks. The initial phases of such a project in one engineering lab were described in a recent Tech OnTap article.

Another Tech OnTap article focused specifically on NetApp’s efforts to consolidate storage in one of our corporate data centers, resulting in:

• An increase in storage utilization to an average of 60%
• A reduction in storage footprint from 24.83 racks to 5.48
• Replacement of 50 storage systems with just 10
• A decrease in direct power consumption of 41,184 kWh per month
• Substantial capacity and performance gains

The general approach is described in a white paper on reducing power consumption through storage efficiency.

Many NetApp customers have achieved similar results in their own data centers. For instance, one large telecom customer consolidated 3,103 physical servers down to 134 (23:1) and increased storage utilization from 25% to 70% while saving $2.25 million on power per year and freeing 660 racks and 8,500 ports. The project also resulted in a return on investment in just 8 months, enabled same-day server provisioning, and cut backup time from 96 hours to less than 30 minutes.

Implications for Storage Administration

The move to a massively shared, virtualized ecosystem has some significant implications for storage administrators. Because servers and storage will now need to be accessible from everywhere—even from outside the data center—the way you do things will necessarily change. I asked Jessica Yu, NetApp IT manager for storage, to describe the advantages based on her direct experience:

Unlike dedicated environments, virtualized environments leverage all the resources that are available in the technological landscape. The concerns are usually around performance, storage capacity, scalability, fault tolerance, complexity in application dependency, and how to keep track of applications versus servers versus storage and information.

Benefits of shared storage include:

• Capacity savings. Flexibility in resources allocation, ability to leverage resources throughout the organization, and easier capacity planning all contribute to significant reductions in capacity requirements.
• Ease of administration. Fewer physical machines to manage and maintain also simplifies the overall environment and lowers maintenance costs. A simplified architecture leads to standardization and more and better automation.
• Fault tolerance. The shared storage environment has built-in redundancy when one of the hardware components fails.
• Increased storage efficiency. Virtualization of our storage environment makes it possible to utilize thin provisioning for all applications. Standardization also results in more duplicate data being colocated on the same storage where it can be deduplicated to yield additional space savings.

With shared storage, everything becomes standardized. Therefore, it is much easier and faster to plan, procure, and implement projects requiring storage. We used to have multiple silos of storage. However; using this methodology, data center planning and growth become a nightmare, wasting both data center resources and administrator hours:

• We used to spend weeks talking about adding additional capacity; now we are able to handle capacity planning in a regular monthly meeting.
• Where we used to spend days discussing storage design with project teams, we now spend just a few hours to understand the requirements and review the design template.
• It used to take several hours to locate free capacity across all our storage systems to allocate space for new application implementations. Now we only have to spend a few minutes looking at capacity availability to identify appropriate resources.
• Our virtualized environment makes tasks easier, so a task that would previously have been handled by level 3 support staff can now be delegated to level 2.
• By implementing SnapManager® for Virtual Infrastructure (SMVI) for our VMware® environment, we have been able to delegate many tasks, such as creating Snapshot™ copies and restoring files to system administrators instead of having storage administrators involved in every single task.

All these time savings add up to weeks of man-hours saved across multiple functional groups, and at the same time we are serving our user community better.

Conclusion

Data centers will need to become much more efficient to continue to meet business needs in the face of flat or decreasing IT budgets and increasingly power-hungry IT equipment. I’ve outlined some of the steps you can take to boost the power efficiency of data center infrastructure and maximize utilization of IT resources, as well as the implications of taking these steps.

A variety of additional initiatives are on the horizon that will also seriously impact IT. To learn more about these topics see the companion articles in this special issue of Tech OnTap that discuss cloud storage and a new partnership between Cisco and NetApp for data center infrastructure.

Dave Robbins Chief Technology Officer for IT NetApp As an IT practitioner since 1979, Dave has seen and participated in many technology evolutions that have contributed to his keen understanding of data center efficiency and emerging cloud technologies. At NetApp, he is responsible for identifying and selecting new technologies and establishing adoption road maps for NetApp IT. He works hard to ensure the use of NetApp technologies within NetApp IT, which serve as a model of best practices.