Windows Server 2012 R2 Shared VHDX Infrastructure for Private/Service Provider Cloud for Resiliency

Windows Server 2012 R2 introduces Shared VHDX for guest clusters inside Hyper-V. In previous version of Windows Server to create guest clusters in Hyper-V installations you needed to expose raw shared storage to the guest VMs, either through in-guest iSCSI (Windows Server 2008 R2 and above) or through Virtual Fibre Channel HBAs (Windows Server 2012).

There are three fantastic documents that Microsoft have produced for reference IaaS architecture:

  1. Infrastructure-as-a-Service Product Line Architecture Fabric Architecture Guide
  2. Infrastructure-as-a-Service Product Line Architecture Fabric Management Architecture Guide
  3. Infrastructure-as-a-Service Product Line Architecture Deployment Guide

The first deals with your Hyper-V infrastructure and storage, the second deals with the management software (System Center); the third tells you how to do it.

They are very informative and are currently aimed at Windows Server 2012. These documents are very detailed and are very much worth reading. For obvious reasons they do not recommend specific hardware vendors.

Microsoft is yet to release any reference architecture material for Windows Server 2012 R2 (as it is in preview) and these documents got me thinking out how you could protect your guest clusters that use Shared VHDX files.

Known issues with Shared VHDX

Please bear in mind that this information is based on the preview bits of Windows Server 2012 R2… So what are the issues?

  1. Backing up the guest OSs and data is not possible with Hyper-V host level backups. You can backup the operating system aspect (probably not supported) but it will not backup the Shared VHDX file. You need to backup the guest cluster by installing agents inside the cluster – not ideal for service providers that want to offer backup without installing guests and exposing their backup infrastructure – albeit a small part – to tenants
  2. You can’t replicate a Shared VHDX using Hyper-V replica
  3. You can’t hot-resize a Shared VHDX file, you can add more Shared VHDX files but you can’t resize it whilst it’s live (unlike non-Shared VHDX files)

Hyper-V Replica and Shared VHDX

Hyper-V replica is an amazing inbox tool for replicating VMs from one Hyper-V server/cluster to another Hyper-V server/cluster. With Windows Server 2012 R2 you can add another point of replication – so you have tertiary replicas. Great – but what about your Shared VHDX?

In Hyper-V replica you can select the disks you want to replicate to the other server so in this case you would NOT select the Shared VHDX file. So how could you replicate the Shared VHDX? SAN replication (I’ll come back to this in a moment).

SMB Storage Spaces

Using Storage Spaces to store your VHDX files that contain your guest cluster VM operating systems is a no-brainer. Storage Spaces are very cheap to implement (JBODs are cheap) and with Windows Server 2012 R2 you get inbox data tiering (usually something associated with expensive SANs) – essentially moving the blocks of data that are accessed frequently to SSD for super-fast access.  Combine it with RDMA NICs (in my opinion iWARP is probably the best as you can route the traffic but you’ll take a small hit for it) and you’ve got an extremely rapid storage infrastructure. But what about the Shared VHDX? I’m getting there…

The SAN is dead! Long live the SAN!

There has been a lot of chatter about whether or not SANs still have a future in the Microsoft Hyper-V world. On the surface you can see why – Storage Spaces with its tiering, write-back cache, CSV cache, deduplication of CSVs for VDI and all other goodies it brings makes a solid argument.

One key feature missing (at the moment, I can’t believe it’ll be long before this changes) is block level synchronous (or even asynchronous) replication. Sure there is DFS-R but that doesn’t deal with open files which your Shared VHDX would be.

SANs are really the only viable alternative (there is some software out there that can do it but I don’t know enough about them to recommend any) for block level synchronous (or even asynchronous) replication.

SANs are not cheap – that is a fact. If you’ve implemented Storage Spaces for Hyper-V storage then you’ve already saved a lot on your storage budget. So what you could potentially do is buy a “small” SAN that offers the type replication you require and deploy that for Shared VHDX storage. However we all know that Fibre Channel (if that’s your SAN of choice and I’m going to assume it is) HBAs, switches, cables, etc. are not cheap so how can you make it cheaper? Put a Storage Space in front of the SAN!

You’ll need 2 or more physical nodes (up to 8) so you’ve got redundancy, you can then connect each node to the SAN, either directly or through a FC switch, configure all the MPIO, CSVs, etc. and make the storage available via SMB. That way you’ve not had to deploy FC HBAs to all the hosts, put in enough switching infrastructure etc. Also you don’t need storage that will support a large number of SCSI-3 persistent reservations as the only reservations come from the Storage Space servers.

The key issue is that Storage Spaces can be accessed by multiple clusters/hosts; it doesn’t care what cluster a host belongs to (or even if it is a member of a cluster) as long as all the correct Kerberos delegations and ACLs are in place (which System Center Virtual Machine Manager 2012 R2 can do for you). This allows you to move roles between clusters without having to move the storage – and as you can’t move Shared VHDX files this is quite important.

This implementation allows service providers to not break “the red line” between hosts and the storage fabric i.e. you don’t expose your storage to your tenants.

The diagram below shows (very roughly) what this could look like:

Shared VHDX Hyper-V Cluster

I’ve highlighted the fault domains in this implementation and as you can see the each cluster is a fault domain as is each space. So to reduce the impact of a failure of a fault domain you could create N number of Space 1, ensuring that no 2 guest cluster VM operating system disks are kept of the same space.

For example guest cluster A has two VMs and a Shared VHDX file. Server 1 for guest cluster A could reside on Space 1; Server 2 for guest cluster A could reside on Space N+1 – this just leaves the FC SAN as a fault domain (I’m certain the hardware vendor would be able to provide assistance here to reduce the likelihood of a component failure having a serious impact).

Using the Hyper-V Replicas

So you’re not just going to be able to turn on the replicas without making them aware where to find their Shared VHDX file; enter Hyper-V Recovery Manager in windows Azure. This coordinates the recovery of Hyper-V guest VMs, in a planned, unplanned or test manner and has the ability to execute scripts, especially PowerShell scripts… With a PowerShell script you can manipulate a VM’s settings, including where to find its Shared VHDX file… So you’ll need to know the path to Storage Space where the Shared VHDX replica will be (this will be the space in front of your SAN replica).

Job done… In theory…

Le Caveat

I’ve absolutely no idea if any of this will be supported in Windows Server 2012 R2 yet or if will it even work… It’s mainly my internal ramblings on a page. If I had some kit to try it on – I would.


Posted on 23 July, 2013, in Hyper-V, Microsoft, Windows Server 2012 and tagged , , , , . Bookmark the permalink. Leave a comment.

Anything to add? Let me know

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: