Case Study: Lessons Learned with
Hyper-V and NetApp
At Avanade, the success of our business relies on how well our global consultants address the needs of enterprise clients. To help do that, Avanade has made significant investment in IT applications and services that provide all the digital tools required to achieve our goals. One of the key reasons we originally explored a transition to server virtualization with Microsoft® Hyper-V™ was the flexible disaster recovery and availability scenarios it offered us. When coupled with a shared storage back-end, we believed this type of architecture would allow our applications and services to more smoothly fail over to another local node, and replicate back and forth to a remote data center, thereby protecting us from a more widespread disaster.
While there were other drivers to our Hyper-V transition, the flexibility of restoring and recovering virtual server environments for disaster recovery was what ultimately got our CIO to buy off on the significant up-front investment. Other drivers for our decision for Hyper-V included the need to:
- Reduce our data center costs (rack space, power and cooling) without reducing the quality of service delivery
- Make better use of our more powerful server resources
- More rapidly deploy new application server environments
We hoped that we could make inroads in all of these areas. With that in mind, we signed up as a beta user of Microsoft’s not-yet-released Windows Server® 2008 with Hyper-V. That’s when our real work began.
Our Hyper-V Deployment
Before I describe the nature of our current Hyper-V environment, let me share some background about our IT infrastructure before we implemented Hyper-V. Our environment consists of three data centers (in the U.S., Europe, and Asia), with most application services used by our consultants originating from our U.S. data center in Seattle. When we began the Hyper-V deployment, we had well over 200 distributed servers in use, supporting the efforts of Avanade’s 8,000+ professionals.
Since Avanade is a joint venture between Microsoft and Accenture, our IT environment relies on mainly Microsoft applications to service our field force and our company’s general operations. These applications include a large Microsoft Office SharePoint® Server installation, Microsoft Exchange, Microsoft SQL Server®, Microsoft Team Foundation Server, Microsoft Terminal Services, and many line 120 line of business applications supported by Web front ends.
To get the most from our upcoming virtual server environment, we decided to pair Microsoft Hyper-V with a shared storage infrastructure. We had already been using NetApp® storage with iSCSI to support our SQL Server infrastructure. So it was natural for us to consider NetApp to also support the upcoming Hyper-V deployment. We liked the system’s flexibility and efficiency of technologies like NetApp Snapshot™ and SnapMirror® for data protection and replication. We’d grown used to being able to rapidly provision storage for new applications and had also begun to explore reclaiming some of our capacity by using NetApp deduplication on primary storage.
In this case, we hoped that NetApp would be just as suitable for Hyper-V as it had been for our other environments. Performance tests that we performed with Hyper-V and NetApp over iSCSI confirmed that we would get the high performance we needed in a NetApp/Hyper-V combination.
Since we decided to move forward, our Hyper-V implementation has been deployed in two areas—production as well as our test and development environment. Here are a few highlights of our current deployment.
Production Environment
- Microsoft Hyper-V for Windows Server 2008, Core Installation, Enterprise Edition
- Configured as a four-node Hyper-V Cluster
- 27 child partitions (virtual machines)
- Virtualized applications run under either the Windows Server 2003 or Windows Server 2008 guest operating system
Virtualized Applications
- Team Foundation Server
- Systems Center Operations Manager 2007
- Microsoft Windows Server 2008 Terminal Services (both TS farm and TS Gateway)
Servers and Storage
- Sun Fire X4150 servers, each with two quad core processors, 32GB RAM
- NetApp FAS3040 storage system configured with the iSCSI protocol
Development and Test Environment
- Microsoft Hyper-V for Windows Server 2008, full installation, Enterprise Edition
- One 4-node cluster
- One 2-node cluster
- Three standalone servers
- Over 150 child partitions (virtual machines) across the dev and test Hyper-V hosts
Virtualized Applications
- Varied, but includes applications in production
Servers and Storage
- Dell 2950 servers, each with two quad core processors, 32GB RAM
- NetApp FAS3020 storage system configured with the iSCSI protocol
Decisions on the Road to Hyper-V
As I look back on our deployment experiences, I can see how our approach in a few critical areas has contributed to the overall success of the project. These include:
- Deciding which edition of Microsoft Windows Server 2008 to deploy, The full edition or the thinner Windows Server Core edition
- Deciding how to configure the NetApp system to work with the Hyper-V child partitions’ virtual hard disk (VHD) files
- Developing a process that let us quickly provision and attach a new VHD to an existing Hyper-V node
Deciding on Microsoft Windows Server 2008 Core Edition
It took us a while to figure out whether or not we should use Server Core or the full edition of Windows Server 2008. We knew that Server Core would have a much lighter footprint from a security perspective, which meant that we would probably have to patch it less often. This was important because each host might ultimately have 10, 15, or 20 virtual machines on it. In order to keep the VMs highly available, we wanted to minimize the number of reboots we would have to make on the host servers.
However, Server Core had a command line interface, which was a little intimidating for our first-tier operations personnel. They asked us, “How well will I be able to manage this thing or troubleshoot it?” After negotiating back and forth, we decided to move forward with Server Core. It turned out to be a big win for us. Once it was built, it worked really well, providing a very solid infrastructure. Our support personnel were still able to use standard tools for Windows Server 2008 with it, including MMC snap-ins and remote server administration tools. After this experience, we all became big fans of Server Core.
Configuring NetApp Storage to Work with Hyper-V Child Partitions and VHDs
We knew that NetApp deduplication and Snapshot technology worked at the volume level, so we decided to architect storage for our Hyper-V child partitions to take as much advantage as possible of these features.
Because there is so much duplicate data between instances of the same guest OS, we expected to see significant savings when we ran NetApp deduplication on a development and test volume with Hyper-V VHDs on it. We were happy to report over 50% space savings and now look forward to deploying deduplication in our production environment as well. Given these volume-level features, we adopted the following practices:
- Grouped child partitions running the same guest operating system into the same NetApp flexible volume (FlexVol®).
In our production environment, this meant that our Windows Server 2003 child partitions would be on one volume, while our Windows Server 2008 partitions would be on another volume. (In our development and test environment, this rule is somewhat relaxed, because we can choose to use NetApp Snapshot to quickly back up or restore an overall environment for further testing. In this case, the environment might consist of more than one guest OS.)
- Adopted a 1:1 mapping of VHDs to LUNs.
For every child partition, there is an associated LUN storing that child partition’s virtual hard disk (VHD) files and related data. To reinforce independence between the child partitions, this 1:1 mapping means that no child partition shares a LUN with another child partition. However, many LUNs (and VHDs) still exist on the same NetApp volume.
- Used Microsoft’s Volume GUIDs to assign a unique identifier to each child partition.
Because we have many virtual machines on our Hyper-V clusters, drive letters became scarce very quickly. Using mount points was not an option, because it would create unwanted dependencies between disks. Our strategy was to use Volume GUIDs to identify our disks on our clusters. Rather than store a LUN on a lettered drive (such as the E: drive), we chose not to assign a drive letter at all. When you don’t assign a drive letter, you get a GUID assignment that looks something like this: \\?\Volume{c9612f6f-702e-11dc-b79a-00123f769676}\.
The following sections describe how storage is currently configured for our production, development, and test environments.
Storage Configuration for Production Hyper-V Environment
Two NetApp flexible volumes (FlexVol) store data associated with each child partition. Child partition VHD files are configured in Hyper-V as “fixed VHD” files to maximize performance. We knew we could still gain the space savings with NetApp deduplication technology that customers expect when using dynamic VHDs.
Volume 1:
- Stores data for seven Windows Server 2003 child partitions (virtual machines), where each child partition’s data is stored in its own iSCSI LUN
- Volume is allocated 500GB of overall storage capacity
- Volume is currently 48% used
Volume 2:
- Stores data for 20 Windows Server 2008 child partitions, where each child partition’s data is stored in its own iSCSI LUN
- Volume is allocated 1.25TB of overall storage capacity
- Volume is currently 76% used
Storage Configuration for Development and Testing Environment
Over 30 NetApp flexible volumes (FlexVol) store development or test data associated with child partitions. Child partition VHD files are configured in Hyper-V as “dynamic VHD” files to make it easier to grow and shrink their size during test and development.
Volume 1: Used by Development
- Stores data for 25 Windows Server 2008 child partitions
- Each child partition’s data is stored in its own iSCSI LUN
- LUNs range from 40 to 100GB in size
- Volume is allocated 600GB of overall storage capacity
- Volume is currently 39% used
~32 Volumes: Used by Both the Development and Test Teams
- Volumes collectively store data for over 100 Windows Server 2008 Hyper-V child partitions
- Each child partition’s data is stored in its own iSCSI LUN
- LUNs average 20GB in size
- Each volume averages three LUNs
- Volumes are collectively allocated 2.5TB of overall storage capacity
Quick Provisioning of New LUNs and VHDs
One thing we liked about NetApp is the flexibility on the back-end storage array. We easily developed a way to provision new LUNs and attach a brand new disk (VHD) to a Hyper-V node in five minutes. We developed a set of scripts wrapped around secure shell (ssh) commands. The scripts create the LUN and automatically attach it to our Hyper-V cluster. This process has since allowed us to automate LUN provisioning. Once the process is fully documented, we expect to be able to hand it off to any of our applications personnel so they can do it themselves. The process involves going into the host, running the commands, and it’s attached. It’s that simple!
What’s Still in Store
Based on our experience, we fully expect everything to be virtual from here on out. If you ask me about our destination architecture, I see both our Hyper-V cluster and SQL Server cluster attached to NetApp storage. All of our front-end applications will eventually be on the Hyper-V cluster, which we expect to grow from four nodes up to eight nodes. We are also in the process of upgrading the NetApp firmware on our production system so that we can begin running NetApp deduplication to help reduce the amount of storage space we need as our VMs increase.
Virtualization Plans
We have plans to virtualize our SharePoint implementation and believe that even our SQL Server cluster could be virtualized. Given Microsoft’s growing support for Hyper-V, our virtualization plans may extend to include our edge servers, firewall ISA Servers, and IIS Web front-end servers. In the next 6 to 9 months, we plan to virtualize a lot of the single-box, single-application servers that are currently taking up a lot of rack space in the data center. These are applications being used by maybe 10 to 20 people in a department like HR or finance. We would consider these some of the low-hanging fruit that would bring us easy wins in a move to virtual machines.
We are still evolving many of our Hyper-V practices. Some of the timing on this relates to recent Microsoft releases that more closely integrate Microsoft solutions with Hyper-V, such as Service Center Virtual Machine Manager (SCVMM) and Microsoft Data Protection Manager (DPM). In fact, we have already begun to test SCVMM’s P to V (Physical to Virtual) feature, which looks promising as a fast way for us to convert many of our physical servers into virtual machines. We are also exploring how best to incorporate emerging NetApp data protection technologies that leverage VSS for Microsoft Hyper-V to capture fast snapshots of a base VM. We expect that we could use this type of NetApp technology for management and data protection, in conjunction with Data Protection Manager.
Data Protection and DR Plans
We are also developing processes for tiered data protection.
- Some Hyper-V data would be backed up by using Microsoft Data Protection Manager or some combination of DPM and NetApp data protection technologies built by leveraging VSS.
- We are currently using DPM in our production environment, which requires an agent within each child partition.
- Some data would be backed up by using NetApp Snapshot. (Snapshot is currently in use in our development and test environments.)
- Some critical data would be replicated by using NetApp SnapMirror between our Seattle data center and another off-site DR location at one of our offices in the Eastern U.S.
Conclusion
Our experience working with Microsoft Hyper-V and NetApp has been excellent. We’ve gained valuable disk space (through NetApp deduplication) and can now autoprovision as many as four servers in under an hour. We expect this process to get even faster with SCVMM. Our virtual machines remain highly available. We’ve also seen great performance with Hyper-V and NetApp.
 |
Andy Schneider
Senior Systems Engineer
Avanade
After beginning his Avanade career as a field consultant three and a half years ago, Schneider has spent the last two years putting his field perspective to work in the company’s central IT infrastructure team. He and the team now determine the best use for new technologies in Avanade’s production environment.
|
