Category Archives: Azure

Azure Internal Load Balancer and the Windows Azure Pack

I’ve been working with a customer who have been developing their own custom portal for the Windows Azure Pack (WAP) and wanted to host WAP in Azure as they had two datacentres and wanted to ensure that WAP was hosted elsewhere. So in this design my Inframon colleagues and I decided to use Azure to host WAP, ADFS, SQL AlwaysOn for WAP and a few other components (not SMA and SPF though).

The design called for two servers hosting WAP (to start with) to be deployed in Azure and to have all the different WAP components (Tenant API, Tenant Public API, Admin API, etc.) load balanced using the Azure Internal Load Balancer. This also required that we changed the FQDN for the different WAP components and create a DNS entry pointing to the Azure Load Balancer’s IP address.

The diagram below shows the desired installation:

WAP-ILB

Whilst there is a great deal of documentation from Microsoft about how to use the Azure Internal Load Balancer to create a SQL AlwaysOn cluster – which was very useful as we needed one of those for WAP’s databases – there wasn’t very much about using it for anything else.

After many late nights trying to get the Azure Internal Load Balancer to do what we required we started to understand the issues we were having. We had configured the probe port to use the port that WAP was listening on (we didn’t need to change the default WAP port numbers as the customer was happy for them to remain as default as WAP wasn’t customer facing). For example to load balance the Admin API that is on port 30004 we created an Azure Endpoint using the following PowerShell:


Add-AzureInternalLoadBalancer -InternalLoadBalancerName "WAP" -ServiceName "WAP" -StaticVNetIPAddress 192.168.100.25 -SubnetName "WAP"

$Port = 30004

$WAP1 = Get-AzureVM -ServiceName "WAP" -Name "WAP1"
$WAP2 = Get-AzureVM -ServiceName "WAP" -Name "WAP2"

Add-AzureEndpoint -Name "WAP$Port" -Protocol tcp -LocalPort $Port -PublicPort $Port -DirectServerReturn $false -LBSetName "WAP$Port" -ProbePort $Port -ProbeProtocol http -ProbeIntervalInSeconds 15 -ProbeTimeoutInSeconds 31 -InternalLoadBalancerName WAP -ProbePath / -VM $WAP1

Add-AzureEndpoint -Name "WAP$Port" -Protocol tcp -LocalPort $Port -PublicPort $Port -DirectServerReturn $false -LBSetName "WAP$Port" -ProbePort $Port -ProbeProtocol http -ProbeIntervalInSeconds 15 -ProbeTimeoutInSeconds 31 -InternalLoadBalancerName WAP -ProbePath / -VM $WAP2

$WAP1 | Update-AzureVM
$WAP2 | Update-AzureVM

This refused to work.

Unfortunately there is no way (that I can find) to monitor the Azure Internal Load Balancer to find out what was happening with it and what errors, if any, it is receiving from load balanced servers. After much investigation we discovered that as we were using Server Name Indication in IIS the HTTP probe wasn’t working. This was because the probe wasn’t using the SNI name and was receiving back a HTTP 400 BAD REQUEST ERROR. The Azure Internal Load Balancer correctly interpreted this a service failure so the load balancer wouldn’t work.

To resolve this problem we changed the default web site in IIS to listen on port 40091, ensured there was no SNI configured and altered the PowerShell to this:

$ServiceName = ""
$InternalLoadBalancerName = ""
$InternalLoadBalancerIPAddr = ""
$SubnetName = ""

Add-AzureInternalLoadBalancer -InternalLoadBalancerName $InternalLoadBalancerName -ServiceName $ServiceName -StaticVNetIPAddress $InternalLoadBalancerIPAddr -SubnetName $SubnetName

#Get the VM configuration from Azure
$WAP1 = Get-AzureVM -ServiceName $ServiceName -Name "WAP1"
$WAP2 = Get-AzureVM -ServiceName $ServiceName -Name "WAP2"

#The list of ports to be load balanced
$Ports = @("30004","30005","30006","30020","30022","30071","30072","30081","30091")

#Iterate each port in the list creating a new endpoint within the Azure ILB for each VM and port
ForEach($Port in $Ports){

Add-AzureEndpoint -Name "WAP$Port" -Protocol tcp -LocalPort $Port -PublicPort $Port -DirectServerReturn $false -LBSetName "WAP$Port" -ProbePort 40091 -ProbeProtocol http -ProbeIntervalInSeconds 15 -ProbeTimeoutInSeconds 31 -InternalLoadBalancerName $InternalLoadBalancerName -ProbePath / -VM $WAP1
Add-AzureEndpoint -Name "WAP$Port" -Protocol tcp -LocalPort $Port -PublicPort $Port -DirectServerReturn $false -LBSetName "WAP$Port" -ProbePort 40091 -ProbeProtocol http -ProbeIntervalInSeconds 15 -ProbeTimeoutInSeconds 31 -InternalLoadBalancerName $InternalLoadBalancerName -ProbePath / -VM $WAP2

}

#Update the VMs in Azure with their new configuration
$WAP1 | Update-AzureVM
$WAP2 | Update-AzureVM

This resulted in a happy Azure Internal Load Balancer but not a happy WAP deployment…

As each WAP server had all of the required components, Tenant API, Tenant Public API, Admin API, etc. and the IP address of the FQDN was set to the IP address of Azure Internal Load Balancer WAP was unable to communicate with itself across components. The diagram below shows the problem:

Azure-ILB-Fail

After much deliberation on how to solve this, including moving away from the Azure Internal Load Balancer and using a 3rd party tool in Azure, it was decided that we should put an entry in each WAP server’s hosts file for the WAP FQDN to reference itself. This led to a happy deployment!

So if you’re going to use the Azure Load Internal Load Balancer for anything then make sure you understand that servers that sit behind it can’t communicate with the IP address of it. If we had WAP split into each separate component on different servers, in different subnets behind different Azure Internal Load Balancers then this would have been OK but for this customer it would’ve been too much!

RDS Connection Broker on Azure IaaS (Microsoft patching is a problem)

As part of my up coming change of employment I’ve been asked by my new employer to get up to speed on Windows Server 2012 VDI solutions and to get MCSE: Desktop Infrastructure ASAP!

After looking through the 70-415 syllabus I realised there were some gaps in my knowledge – mainly on the RDS Gateway and the RDS Connection Broker front. In my current environment UAG 2010 deals with the RDS Gateway for me – hence no real experience.

To Azure!

Not having all the capacity in the world (unlike Azure) I went and merrily built a RDS infrastructure on Azure IaaS or so I thought…

The Windows Server 2012 images that Microsoft provide on Azure are patched and unbeknownst to me there is a problem installing the RDS Connection Broker role if you’ve got KB2821895 installed on your Windows Server 2012 instance… The currently available VM images from Microsoft have this patch installed (unsurprisingly).

What to do?

To SCVMM!

So I went to my local SCVMM install, dug out the ISO that contains the Windows Server 2012 RTM files, created a VM, installed Windows Server 2012 Datacenter Edition, SysPrep’d it and attempted to upload the file to Azure using PowerShell. Azure PowerShell is amazing and wonderful and brilliant and didn’t work for me… At first I thought it was a problem with my local firewall/HTTP/HTTPS proxy (it usually is) but no! For once it was happy!

What to do?

To Cerebrata’s Azure Management Studio!

I downloaded the free trial of Cerebrata’s Azure Management Studio and my first thought was “Err… OK…” I ploughed on and found the correct way to upload a PageBlob (which VHDs, not VHDXs as they are not support on Azure, need to be for use as VM images) to my container. It’s not the most amazing user interface I’ve ever seen but it does EXACTLY what you need it to do. They even give MVPs a free copy! (I wish I was a MVP!)

Back to Azure!

Once the VHD was up there it was back to the Azure portal to create the VM image template and the new VM for my RDS Connection Broker. Instead of installing the Windows Internal Database (which a standalone RDS Connection Broker uses) as part of the RDS setup wizard I decided to install that first and then use the RDS setup wizard. Job done!

Summary

There seems to be something going slightly awry at Microsoft with regards to patching at the moment. I can’t remember ever having to revert to a non-patched server OS to install a role before.

There have been recent problems with Hyper-V patches causing BSODs when using VLANS, ADFS has had issues, Exchange 2013 too. Windows 7 is also having issues as well.

It makes me wonder if MS’s new rapid development cycle is causing substandard code, and consequently patches, to be released?

Like most people I try to test patches before rolling them into production however that can be an issue if you don’t have a test environment that matches your production environment. I think we’re heading towards a period where IT Pros are going to reluctant to install patches without someone else doing it first just in case it destroys their environment. This means systems that need patching will remain unpatched until someone bites the bullet and tries – the question is who?

Update 13/9/2013

It would appear that Microsoft have fixed the issue with http://support.microsoft.com/kb/2871777

Windows Azure – What is it?

Azure is Microsoft’s public cloud offering with a wide variety of services available to consume. I’ve not really looked at Azure in the past as not being a developer by trade (and not working for an organisation that developed its own software) there was very little that caught my attention – until now.

With the release of Infrastructure Services (Iaas) Microsoft has squared up to Amazon and basically said “bring it on!” I’ve been waiting for these services to become available for some time and they’ve not disappointed me. The two main things that they offer are:

  1. Virtual Networks – create a Virtual Private Network (VPN) tunnel to your office/data centre from Azure
  2. Virtual Machines – there are several Microsoft operating systems, and non-Microsoft operating systems that you can run (mostly what runs on a local Hyper-V server, but not all)

As I understand it the Azure Virtual Machines (VMs) run on a modified version of Microsoft’s Hyper-V server designed specifically for Azure. This means you can move a Hyper-V based VM to the Azure IaaS platform with very little effort (especially if you’re running System Center 2012 AppController). There are a few caveats but they are quite straight forward to understand (apart from one – guess which one):

  1. Fixed size Virtual Hard Disk (VHD) files only (at the moment the VHDX – the new Windows Server 2012 VHD file format is not supported). If it’s a VHD that contains Operating System files there is a hard limit of 127GB otherwise it’s 1TB.
  2. VMs only have 1 virtual network card (IP addresses are DHCP leased to VMs for 150 years – not a typo! Do not mess with the IP address of your VM or it will become totally inaccessible if you do, the exception is changing DNS settings, just BE CAREFUL)
  3. LICENCING! You need licence mobility if you’re moving your own VMs with server software to Azure (FAQ). Check the latest Product Use Rights (PUR) document. Exchange is not covered – Microsoft’s answer – use Office 365

Virtual Networks

For me this is great – I’ve created a VPN from an unsupported device (shh! Don’t tell support) to the Europe North (Ireland) data centre. Before anyone criticises the European naming of the Azure data centres – blame the UN’s classification. Apparently Ireland is in Northern Europe and the Netherlands is in Western Europe… OK… Right… Someone should buy the UN an atlas.

So I’ve extended my data centre into Azure now giving me unlimited power – as long as the credit limit on my boss’ credit card is high enough! So what do I want to do with it?

Virtual Machines

I’m going to move some of what I’ve got in my perimeter network to Azure. There’s nothing stopping me from controlling what goes through the VPN via my firewall, and the Windows host firewall on the VMs, so it’s just as secure as most in house deployments (just need to get the pesky compliance guys to agree).

It is possible to open endpoints into VMs on Azure so publishing applications – for example port 80/443 for http/https applications, port 21 for FTP, etc. It is possible to open any port with the exception of ICMP traffic (basically pings) – ICMP internally within Azure and across a VPN is fine but anything external either incoming or outgoing is blocked by the Azure firewall.

There are some crucial things to understand about endpoints especially in load balanced applications. Load balancing is done by the Azure load balancer not anything you’ve got internally unless you do some massively complex setup and it probably won’t be supported. The Azure load balancer is not a hardware product; it’s a software load balancer that does things slightly different to traditional load balancers. If you have 2 servers in a load balanced configuration you cannot guarantee that requests will go Server 1, Server 2, Server 1, Server 2, etc. It may well go Server 2, Server 1, Server 2, Server 1, Server 1, Server 1. There is method in there – somewhere!

There’s lots of information on the internet saying it uses round-robin but on my VMs it didn’t and on everyone else’s VMs in Steve Plank’s Windows Azure Camp for the IT Pro it didn’t!

You could, for example, run any software on a Windows Azure VM that you could run on a normal Hyper-V VM guest. SharePoint farm anyone (there’s a template for that)? SQL Server Always-On cluster (there’s a template for that)? Some other random LAMP based application? Just remember to check the licencing! If you run Microsoft operating systems you’ll be able to obtain support from the Azure support team (provided you’ve paid for support – not sure how that works with Software Assurance customers and their “free” incidents) however if you’re running a Linux VM don’t bother calling support. It’s not MS they won’t support it – why would they?

I think the biggest application of Azure IaaS, for me, will be proof-of-concepts. The ability to extend my network and spin up a proof of concept, test it, demo it – all without impacting my production Hyper-V cluster will be invaluable to me. With all the template VMs available in Azure it is quite easy to get concepts going in hours rather than days! The best thing is not to having to worry about the underlying hardware resource (and someone else has done most of the hard work with the SQL installer).

So what’s next on Azure?

Obviously there is no public road map for Azure but there are several features in preview (this doesn’t guarantee they’ll make it to production):

  • Point to site vpn – think traditional end user VPN connections from laptops/desktops etc.
  • Websites – there are wide variety templates available, for example Word Press, Drupal, MODX
  • Mobile Services – backend database for mobile apps
  • HDInsight – Hadoop Big Data on Apache
  • Backup – native Windows backup to Azure and integration with System Center Data Protection Manager (one feature I am very much looking forward to)
  • Hyper-V Recovery Manager – protection for your System Center Virtual Machine Manager private clouds – essentially coordinates hyper-v replicas for you in a more complex fashion

Summary

Azure has been around for quite a while now, starting life as a Platform as a Service infrastructure slowly, if somewhat reluctantly, moving into Infrastructure as a Service. It’s going to become extremely important for Microsoft in the near future as it strives to take back ground from Amazon on the IaaS side. It will take time for the new services to mature but once they do (and the pricing drops – I hope) it may well, one-day, completely replace the on-premise data centre just as Microsoft’s Software as a Service offerings, Office 365 in particular, are starting to replace traditional on-premise deployments.

Creating a VPN between a WatchGuard XTM 510 and Windows Azure Virtual Networks

So there are only two supported hardware vendors for Windows Azure Virtual Networks (VPNs):

  • Cisco:
    • ASA – OS version 8.3
    • ASR – IOS 15.1 (Static routing), IOS 15.2 (dynamic routing)
    • ISR – IOS 15.0 (Static routing), IOS 15.1 (dynamic routing)
  • Juniper
    • SRX – JunOS 10.2 (static), JunOS 11.4 (dynamic)
    • J-Series – JunOS 10.4r9 (static), JunOS 11.4 (dynamic)
    • ISG – ScreenOS 6.3 (static and dynamic)
    • SSG – ScreenOS 6.2 (static and dynamic)

Windows Server 2012 Routing and Remote Access Service is also supported.

Helpfully Microsoft has created some templates for all of the above to help you create the VPN connections.

But if you’ve got something that isn’t supported?

Well you’re on your own! No support from Microsoft at all.

We use WatchGuard XTM 510 firewalls. Whilst not the most user friendly or feature packed, they are quite reliable.

Not being one to be deterred by this I forged on and got it working!

This worked for me; I hope it works for you!

Prerequisites:

  1. A Windows Azure subscription (can be a trial)
  2. A WatchGuard XTM 510 Firewall (will probably work with all XTM series, not sure about the hypervisor based products)
  3. Know what your internal IP address range(s) is that you want to connect to Azure
  4. Know what your external IP address is that you’ll be creating your VPN tunnel from is (remember to check your outbound NAT settings if necessary)
  5. Know what IP subnet you want in Azure
  6. Some time and patience(don’t rush it or you’ll have start all over again)

Step 1

Log in to Azure (https://manage.windowsazure.com)

Go to the bottom left hand corner and select New. Navigate to Networks, Virtual Network, Custom Create

Azure Portal - Create New virtual network

Create new Virtual network

Step 2

In the Virtual Network Details put in your details (remember to pick the appropriate region). Create a new affinity group if necessary or select a pre-existing one.

AzureVPN1

Step 3

If you want your Azure VMs to use your internal DNS servers you must add it here. Select Site-To-Site-VPN and Specify a New Local Network (this is your internal office/data centre network not Azure’s). It should end up like this

AzureVPN2

Step 4

This step and the next are very important and is all about your local network, not the network you’ll be creating in Azure (that comes later). Make sure you enter WatchGuard’s External IP address here as shown below:

AzureVPN4

Step 5

Name the network something meaningful to you. Could be OfficeNetwork as shown, could be CoreServers just make sure it makes sense to you. Add the address space(s) of your local network not the network you’ll be creating in Azure (that comes later).

AzureVPN5

Step 6

This is where you create your IP ranges for your Azure VMs to use. I’d suggest something completely different to what you’re using in house. So if you’re using 192.168.x.x ranges, go for the 10.x.x.x or 172.16.x.x – that way if you see that IP address anywhere in logs, firewall rules/notifications, etc. you’ll know it’s not local to you instantly.

The first 3 or 4 (can’t remember exactly) IP addresses of your range(s) are taken by Azure for “internal purposes” so don’t worry if you’re first VM doesn’t have a 10.x.x.1 IP address. I’d suggest creating a Gateway subnet too, this way you can have multiple subnets in Azure and they can communicate (I think)! If you make it the lower end of the range as I’ve done then you know where you stand (you could of course make it the top end, middle, anywhere you like but I like to keep mine tidy). If you do the same as below you may end up your first VM having a .12 IP address! Don’t worry it’s just an IP address (see Important Notes at the bottom of this post)

AzureVPN6

Step 7

So Azure will then go off and create the Virtual Networks, this will take several minutes so don’t be impatient. Once the network is created you’ll need to create the gateway (this seems odd as you’ve just created the network but MS won’t just give you an IP address straight away you need to ask for one). So click on “Create Gateway” and “Static Routing” at the bottom of the Dashboard view. This will again take some time.

AzureCreateGateway

Step 8

Now Azure should’ve given you your external IP address, in the example below I’ve redacted mine but that’s where it’ll be. Write this down. Be aware that if you ever delete and recreate your Virtual Network Gateway it’ll be different.

AzureReadyforWatchGuard

Step 9

To create the VPN you’ll need the preshared key, without this there’s no VPN for you! So once everything is ready as above click on the “Manage Key” button at the bottom, I’d advise copying and pasting this to notepad as you’ll need it later on.

AzurePSK

Step 10

Open Policy Manager for your WatchGuard firewall and go to VPN -> Branch Office Gateways. You’ll be presented with the New Gateway screen

BlankGateway

Step 11

Enter the preshared key you pasted into notepad earlier (you can paste into this field using the keyboard CTRL+V). Click Add in the Gateway Endpoints section and enter the information as shown below. It may seem wrong to enter the Azure external IP address twice but it worked for me. Other instructions I read said to enter the internal IP address of the Azure Gateway in the “Specify the gateway ID for tunnel authentication” but that didn’t work for me.

GatewayEndpointConfig

Step 12

Once you’ve entered the information click on the Phase 1 Settings tab at the top and set the mode to Main, NAT Traversal to on, IKE Keep-alive on and everything else as shown below. The next step covers the Transform Settings.

GatewayPhase1Settings

Click on Edit to change what is there. Needs to be set SHA1, AES (128-bit), SA Life 8 Hours and key group Diffie-Hellman Group 2.

Phase1Transform

Step 13

Click OK to close the Branch Office Gateway, click Close on the Gateway list. Go to VPN -> Branch Office Tunnels. Click Add.

NewPhase2Tunnel

Step 14

Give your Tunnel a meaningful name (ToAzure for example). Make sure you select the correct Gateway under name. Click Add. In the Local subnet box enter the IP range you told Azure about in step 5. In my case its 192.168.5.0/24 and the Remote range is 10.254.254.0/24. Leave everything else alone (you can amend later if you want – one thing at a time)

TunnelRouteSettings

Step 15

Click on Phase 2 Settings (make sure PFS is OFF). Click Remove to get rid of the default IPsec proposal, click Add. Give the proposal a meaningful name. Make sure the type is ESP, Authentication is SHA1, Encryption is AES (128-bit). On the Force Key Expiration options set to 1 hour and 102400000 kilobytes (seems a lot but that’s what MS want). Click OK.

NewPhase2Proposal

Step 16

Make sure the Multicast settings are off.

TunnelMulticast

Step 17

To check for any errors you can increase the Diagnostic logging level on the Firewall. In Policy Manager go to Setup -> Logging. Click Diagnostic Log Level… Change VPN IKE logging level to Information

LoggingLevel

Step 18

Click OK, Click Close and save you policy to your Firebox.

Step 19

Once your policy is applied go to Firebox System Manager and go to the Traffic Monitor tab; in the filter box type “ike” (no quotes) to watch the tunnel diagnostic output. On the Front Panel tab you should see a new entry under Branch Office VPN Tunnels like this:

PROOFWhat’s Next?

Now that you’ve created your tunnel and you can control what traverses it just like any other tunnel or policy. To further secure your Firebox I’d suggest creating a new alias for the scope in Azure with the subnet.

If you found this helpful please leave me a comment – it’s always good to know I’ve helped at least one person!

Important Notes

Azure VMs are leased IP addresses by the Azure DHCP platform and not any internal DHCP server you may create (don’t create one – there is no point). Azure DHCP leases are for 150 years (that’s not a typo)! If you change your IP address from dynamically assigned to statically assigned (even if you keep the same IP address I believe) your VM will become inaccessible! No RDP, No HTTP, NOTHING. Microsoft’s Configuration Management Database keeps track of every IP address it leases. If your machine isn’t using DHCP it’ll know and kill access to it.

How this will work with technologies like Network Access Protection I don’t know. I’d imagine IPsec enforcement would work as that doesn’t rely on changing DHCP leases like NAP DHCP enforcement does. Maybe I’ll do a post on that on day!

%d bloggers like this: