# Maarten Balliauw {blog}

ASP.NET, ASP.NET MVC, Windows Azure, PHP, ...

Maarten Balliauw is currently employed as a Technical Evangelist at JetBrains. His interests are mainly web applications developed in ASP.NET (C#) or PHP and the Windows Azure cloud platform.
E-mail me

### Disclaimer

The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

## Tales from the trenches: resizing a Windows Azure virtual disk the smooth way

We’ve all been there. Running a virtual machine on Windows Azure and all of a sudden you notice that a virtual disk is running full. Having no access to the hypervisor nor to its storage (directly), there’s no easy way out…

Big disclaimer: use the provided code on your own risk! I’m not responsible if something breaks! The provided code is as-is without warranty! I have tested this on a couple of data disks without any problems. I've tested this on OS disks and this sometimes works, sometimes fails. Be warned.

When searching for a solution to this issue,the typical solution you’ll find is the following:

• Delete the VM
• Delete the original .vhd from blob storage
• Recreate the VM
• Use diskpart to resize the partition

That’s a lot of work. Deleting and re-creating the VM isn’t that bad, it can be done pretty quickly. But doing a download of a 30GB disk, resizing the disk and re-uploading it is a serious PITA! Even if you do this on a temporary VM that sits in the same datacenter as your storage account.

Last saturday, I was in this situation… A decision would have to be made: spend an estimated 3 hours in doing the entire download/resize/upload process or reading up on the VHD file format and finding an easier way. With the possibility of having to fall back to doing the entire process…

Being a bit geeked out, I decided to read up on the VHD file format and download the specs.

Before we dive in: why would I even read up on the VHD file format? Well, since Windows Azure storage is used as the underlying store for Windows Azure Virtual Machine VHD’s and Windows Azure storage supports byte operations without having to download an entire file, it occurred to me that combining both would result in a less-than-one-second VHD resize. Or would it?

Note that if you’re just interested in the bits to “get it done”, check the last section of this post.

## Researching the VHD file format specs

The specs for the VHD file format are publicly available. Which means it shouldn't be to hard to learn how VHD files, the underlying format for virtual disks on Windows Azure Virtual Machines, are structured. Having fear of extremely complex file structures, I started reading and found that a VHD isn’t actually that complicated.

Apparently, VHD files created with Virtual PC 2004 are a bit different from newer VHD files. But hey, Microsoft will probably not use that old beast in their datacenters, right? Using that assumption and the assumption that VHD files for Windows Azure Virtual Machines are always fixed in size, I learnt the following over-generalized lesson:

A fixed-size VHD for Windows Azure Virtual Machines is a bunch of bytes representing the actual disk contents, followed by a 512-byte file footer that holds some metadata.
Maarten Balliauw – last Saturday

A-ha! So in short, if the size of the VHD file is known, the offset to the footer can be calculated and the entire footer can be read. And this footer is just a simple byte array. From the specs:

Let’s see what’s needed to do some dynamic VHD resizing…

## Resizing a VHD file - take 1

My first approach to “fixing” this issue was simple:

• Write null values over it and resize the disk to (desired size + 512 bytes)
• Write the footer in those last 512 bytes

Guess what? I tried mounting an updated VHD file in Windows, without any successful result. Time for some more reading… resulting in the big Eureka! scream: the “current size” field in the footer must be updated!

So I did that… and got failure again. But Eureka! again: the checksum must be updated so that the VHD driver can verify the footer is valid!

So I did that… and found more failure.

## Resizing a VHD file - take 2

Being a persistent developer, I decided to do some more searching. For most problems, at least a partial solution is available out there! And there was: CodePlex holds a library called .NET DiscUtils which supports reading from and writing to a giant load of file container formats such as ISO, VHD, various file systems, Udf, Vdi and much more!

Going through the sources and doing some research, I found the one missing piece from my first attempt: “geometry”. An old class on basic computer principles came to mind where the professor taught us that disks have geometry: cylinder-head-sector or CHS information for the disk driver which can use this info for determining physical data blocks on the disk.

Being lazy, I decided to copy-and-adapt the Footer class from this library. Why reinvent the wheel? Why risk  going sub-zero on the WIfe Acceptance Factor since this was saturday?

So I decided to generate a fresh VHD file in Windows and try to resize that one using this Footer class. Let’s start simple: specify the file to open, the desired new size and open a read/write stream to it.

1 string file = @"c:\temp\path\to\some.vhd";
2 long newSize = 20971520; // resize to 20 MB
3
4 using (Stream stream = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
5 {
6     // code goes here
7 }

Since we know the size of the file we’ve just opened, the footer is at length – 512, the Footer class takes these bytes and creates a .NET object for it:

1 stream.Seek(-512, SeekOrigin.End);
2 var currentFooterPosition = stream.Position;
3
5 var footer = new byte[512];
7
8 var footerInstance = Footer.FromBytes(footer, 0);

Of course, we want to make sure we’re working on a fixed-size disk and that it’s smaller than the requested new size.

1 if (footerInstance.DiskType != FileType.Fixed
2         || footerInstance.CurrentSize >= newSize)
3 {
4     throw new Exception("You are one serious nutcase!");
5 }

If all is well, we can start resizing the disk. Simply writing a series of zeroes in the least optimal way will do:

1 // Write 0 values
2 stream.Seek(currentFooterPosition, SeekOrigin.Begin);
3 while (stream.Length < newSize)
4 {
5     stream.WriteByte(0);
6 }

Now that we have a VHD file that holds the desired new size capacity, there’s one thing left: updating the VHD file footer. Again, the Footer class can help us here by updating the current size, original size, geometry and checksum fields:

1 // Change footer size values
2 footerInstance.CurrentSize = newSize;
3 footerInstance.OriginalSize = newSize;
4 footerInstance.Geometry = Geometry.FromCapacity(newSize);
5
6 footerInstance.UpdateChecksum();

One thing left: writing the footer to our VHD file:

1 footer = new byte[512];
2 footerInstance.ToBytes(footer, 0);
3
4 // Write new footer
5 stream.Write(footer, 0, footer.Length);

That’s it. And my big surprise after running this? Great success! A VHD that doubled in size.

So we can now resize VHD files in under a second. That’s much faster than any VHD resizer tool you find out here! But still: what about the download/upload?

## Resizing a VHD file stored in blob storage

Now that we have the code for resizing a local VHD, porting this to using blob storage and more specifically, the features provided for manipulating page blobs, is pretty straightforward. The Windows Azure Storage SDK gives us access to every single page of 512 bytes of a page blob, meaning we can work with files that span gigabytes of data while only downloading and uploading a couple of bytes…

Let’s give it a try. First of all, our file is now a URL to a blob:

1 var blob = new CloudPageBlob(
2     "http://account.blob.core.windows.net/vhds/some.vhd",
3     new StorageCredentials("accountname", "accountkey));

Next, we can fetch the last page of this blob to read our VHD’s footer:

 1 blob.FetchAttributes();
2 var originalLength = blob.Properties.Length;
3
4 var footer = new byte[512];
5 using (Stream stream = new MemoryStream())
6 {
8     stream.Position = 0;
10     stream.Close();
11 }
12
13 var footerInstance = Footer.FromBytes(footer, 0);

After doing the check on disk type again (fixed and smaller than the desired new size), we can resize the VHD. This time not by writing zeroes to it, but by calling one simple method on the storage SDK.

1 blob.Resize(newSize + 512);

In theory, it’s not required to overwrite the current footer with zeroes, but let’s play it clean:

1 blob.ClearPages(originalLength - 512, 512);

Next, we can change our footer values again:

1 footerInstance.CurrentSize = newSize;
2 footerInstance.OriginalSize = newSize;
3 footerInstance.Geometry = Geometry.FromCapacity(newSize);
4
5 footerInstance.UpdateChecksum();
6
7 footer = new byte[512];
8 footerInstance.ToBytes(footer, 0);

And write them to the last page of our page blob:

1 using (Stream stream = new MemoryStream(footer))
2 {
3     blob.WritePages(stream, newSize);
4 }

And that’s all, folks! Using this code you’ll be able to resize a VHD file stored on blob storage in less than a second without having to download and upload several gigabytes of data.

## Meet WindowsAzureDiskResizer

Since resizing Windows Azure VHD files is a well-known missing feature, I decided to wrap all my code in a console application and share it on GitHub. Feel free to fork, contribute and so on. WindowsAzureDiskResizer takes at least two parameters: the desired new size (in bytes) and a blob URL to the VHD. This can be a URL containing a Shared Access SIgnature.

Now let’s resize a disk. Here are the steps to take:

• Shutdown the VM
• Delete the VM -or- detach the disk if it’s not the OS disk
• In the Windows Azure portal, delete the disk (retain the data!) do that the lease Windows Azure has on it is removed
• Run WindowsAzureDiskResizer
• In the Windows Azure portal, recreate the disk based on the existing blob
• Recreate the VM  -or- reattach the disk if it’s not the OS disk
• Start the VM
• Use diskpart / disk management to resize the partition

Here’s how fast the resizing happens:

Woah! Enjoy!

We’re good for now, at least until Microsoft decides to switch to the newer VHDX file format…

Categories: C# | General | ICT | Software | Windows Azure

Tuesday, January 08, 2013 12:22 AM

What an awesome piece of research and utility! Nice one Maarten.

Friday, January 11, 2013 5:24 AM

Nice utility. I would have expected the size to be in GB rather than bytes. Does anyone need that level of granularity when sizing VHDs?

Friday, January 11, 2013 12:52 PM

Good point. Guess I started with bytes since all internal stuff works in bytes as well. Feel free to create a pull request (should be an easy change)

Monday, January 14, 2013 1:39 PM

I see you already took care of this over the weekend - Thx

Friday, February 01, 2013 1:05 PM

Great utility - please update your screenshots - took a while before i realised that I was dealing in GB not bytes!
Really helped me when I tried to convert a blob to a disk
Add-AzureDisk : HTTP Status Code: BadRequest - HTTP Error Message: The VHD http://xxx.blob.core.win
dows.net/vhds/fixeddisk1.vhd has an unsupported virtual size of 53686402560 bytes.  The size must be a whole number
(in MBs).

Monday, February 04, 2013 7:03 PM

The program crashes for me. I think its has something todo with my blob credentials.  Is it the same email and password i use to login to portal for the blob creds?

Wednesday, March 06, 2013 10:32 PM

Steve, Have you gotten any reply on why the crash?

Thursday, February 14, 2013 8:46 PM

You are so seriously awesome for posting this, it really saved me.  I can't imagine why MS would think it reasonable to default the C drive to 30 gigs when Windows takes up 20+ gigs out of the box... so stupid.  I wonder how many fire drills this has caused... Well anyways I owe you a beer if you're ever in South OC!

Cheers!

Wednesday, March 06, 2013 10:33 PM

That's right Matt! But unfortunately it keeps rejecting my credentials. Which ones are you using?

Wednesday, March 06, 2013 10:30 PM

I keep geting "The specified storage credentials are invalid". Did anyone come accross that and how did you fix it? My blob url does not include SAS. So I am typing the storage account name, the Storage access key as well as the full blob url to my OS disk.

Friday, March 08, 2013 12:45 PM

Just checking: are you calling WindowsAzureDiskResizer.exe <size> <blob> < account>  <key>  ? E.g. WindowsAzureDiskResizer.exe 60 "http://blob.core....."; "foo" "HDHDHDHDHDGD=" ?

Friday, March 08, 2013 4:50 PM

Hi Maarten, Thank you!
I don't know exactly why it happened but I finally got everything working following the syntax. So sweet! I resized my VHD from 30 to 50.

Can't comprehend how Microsoft managers sometimes make their decisions.

Thursday, March 28, 2013 10:34 PM