Maarten Balliauw {blog}

ASP.NET MVC, Microsoft Azure, PHP, web development ...

NAVIGATION - SEARCH

Working with Windows Azure SQL Database in PhpStorm

Disclaimer: My job at JetBrains holds a lot of “exploration of tools”. From time to time I discover things I personally find really cool and blog about those on the JetBrains blogs. If it relates to Windows Azure, I  typically cross-post on my personal blog.

clip_image002PhpStorm provides us the possibility to connect to Windows Azure SQL Database right from within the IDE. In this post, we’ll explore several options that are available for working with Windows Azure SQL Database (or database systems like SQL Server, MySQL, PostgreSQL or Oracle, for that matter):

  • Setting up a database connection
  • Creating a table
  • Inserting and updating data
  • Using the database console
  • Generating a database diagram
  • Database refactoring

If you are familiar with Windows Azure SQL Database, make sure to configure the database firewall correctly so you can connect to it from your current machine.

Setting up a database connection

Database support can be found on the right-hand side of the IDE or by using the Ctrl+Alt+A (Cmd+Alt+A on Mac) and searching for “Database”.

clip_image004

Opening the database pane, we can create a new connection or Data Source. We’ll have to specify the JDBC database driver to be used to connect to our database. Since Windows Azure SQL Database is just “SQL Server” in essence, we can use the SQL Server driver available in the list of drivers. PhpStorm doesn’t ship these drivers but a simple click (on “Click here”) fetches the correct JDBC driver from the Internet.

clip_image006

Next, we’ll have to enter our connection details. As the JDBC driver class, select the com.microsoft.sqlserver.jdbc driver. The Database URL should be a connection string to our SQL Database and typically comes in the following form:

1 jdbc:sqlserver://<servername>.database.windows.net;database=<databasename>

The username to use comes in a different form. Due to a protocol change that was required for Windows Azure SQL Database, we have to suffix the username with the server name.

clip_image007

After filling out the necessary information, we can use the Test Connection button to test the database connection.

clip_image009

Congratulations! Our database connection is a fact and we can store it by closing the Data Source dialog using the Ok button.

Creating a table

If we right click a schema discovered in our Data Source, we can use the New | Table menu item to create a table.

clip_image011

We can use the Create New Table dialog to define columns on our to-be-created table. PhpStorm provides us with a user interface which allows us to graphically specify columns and generates the DDL for us.

clip_image013

Clicking Ok will close the dialog and create the table for us. We can now right-click our table and modify existing columns or add additional columns and generate DDL which alters the table.

Inserting and updating data

After creating a table, we can insert data (or update data from an existing table). Upon connecting to the database, PhpStorm will display a list of all tables and their columns. We can select a table and press F4 (or right-click and use the Table Editor context menu).

clip_image015

We can add new rows and/or edit existing rows by using the + and - buttons in the toolbar. By default, auto-commit is enabled and changes are committed automatically to the database. We can disable this option and manually commit and rollback any changes that have been made in the table editor.

Using the database console

Sometimes there is no better tool than a database console. We can bring up the Console by right-clicking a table and selecting the Console menu item or simply by pressing Ctrl+Shift+F10 (Cmd+Shift+F10 on Mac).

clip_image017

We can enter any SQL statement in the console and run it against our database. As you can see from the screenshot above, we even get autocompletion on table names and column names!

Generating a database diagram

If we have multiple tables with foreign keys between them, we can easily generate a database diagram by selecting the tables to be included in the diagram and selecting Diagrams | Show Visualization... from the context menu or using the Ctrl+Alt+Shift+U (Cmd+Alt+Shift+U on Mac). PhpStorm will then generate a database diagram for these tables, displaying how they relate to each other.

clip_image019

Database refactoring

Renaming a table or column often is tedious. PhpStorm includes a Rename refactoring (Shift-F6) which generates the required SQL code for renaming tables or columns.

clip_image021

As we’ve seen in this post, working with Windows Azure SQL Database is pretty simple from within PhpStorm using the built-in database support.

Global Windows Azure Bootcamp - april 27th

On April 27th, 2013, you’ll have the ability to join a Windows Azure Bootcamp on a location close to you. We’ve started this with the idea of maybe having 10 or 15 locations world wide. We were wrong. Here’s what happened:

Much ocations for our bootcamp!

In short: we now have over 50 locations available where a bootcamp will be organized! This one day deep dive class will get you up to speed on developing for Windows Azure. The class includes a trainer with deep real world experience with Windows Azure, as well as a series of labs so you can practice what you just learned. It’s free, so find your location and join the fun!

Running unit tests when deploying to Windows Azure Web Sites

When deploying an application to Windows Azure Web Sites, a number of deployment steps are executed. For .NET projects, msbuild is triggered. For node.js applications, a list of dependencies is restored. For PHP applications, files are copied from source control to the actual web root which is served publicly. Wouldn’t it be cool if Windows Azure Web Sites refused to deploy fresh source code whenever unit tests fail? In this post, I’ll show you how.

Disclaimer:  I’m using PHP and PHPUnit here but the same approach can be used for node.js. .NET is a bit harder since most test runners out there are not supported by the Windows Azure Web Sites sandbox. I’m confident however that in the near future this issue will be resolved and the same technique can be used for .NET applications.

Our sample application

First of all, let’s create a simple application. Here’s a very simple one using the Silex framework which is similar to frameworks like Sinatra and Nancy.

1 <?php 2 require_once(__DIR__ . '/../vendor/autoload.php'); 3 4 $app = new \Silex\Application(); 5 6 $app->get('/', function (\Silex\Application $app) { 7 return 'Hello, world!'; 8 }); 9 10 $app->run();

Next, we can create some unit tests for this application. Since our app itself isn’t that massive to test, let’s create some dummy tests instead:

1 <?php 2 namespace Jb\Tests; 3 4 class SampleTest 5 extends \PHPUnit_Framework_TestCase { 6 7 public function testFoo() { 8 $this->assertTrue(true); 9 } 10 11 public function testBar() { 12 $this->assertTrue(true); 13 } 14 15 public function testBar2() { 16 $this->assertTrue(true); 17 } 18 }

As we can see from our IDE, the three unit tests run perfectly fine.

Running PHPUnit in PhpStorm

Now let’s see if we can hook them up to Windows Azure Web Sites…

Creating a Windows Azure Web Sites deployment script

Windows Azure Web Sites allows us to customize deployment. Using the azure-cli tools we can issue the following command:

1 azure site deploymentscript

As you can see from the following screenshot, this command allows us to specify some additional options, such as specifying the project type (ASP.NET, PHP, node.js, …) or the script type (batch or bash).

image

Running this command does two things: it creates a .deployment file which tells Windows Azure Web Sites which command should be run during the deployment process and a deploy.cmd (or deploy.sh if you’ve opted for a bash script) which contains the entire deployment process. Let’s first look at the .deployment file:

1 [config] 2 command = bash deploy.sh

This is a very simple file which tells Windows Azure Web Sites to invoke the deploy.sh script using bash as the shell. The default deploy.sh will look like this:

1 #!/bin/bash 2 3 # ---------------------- 4 # KUDU Deployment Script 5 # ---------------------- 6 7 # Helpers 8 # ------- 9 10 exitWithMessageOnError () { 11 if [ ! $? -eq 0 ]; then 12 echo "An error has occured during web site deployment." 13 echo $1 14 exit 1 15 fi 16 } 17 18 # Prerequisites 19 # ------------- 20 21 # Verify node.js installed 22 where node &> /dev/null 23 exitWithMessageOnError "Missing node.js executable, please install node.js, if already installed make sure it can be reached from current environment." 24 25 # Setup 26 # ----- 27 28 SCRIPT_DIR="$( cd -P "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" 29 ARTIFACTS=$SCRIPT_DIR/artifacts 30 31 if [[ ! -n "$DEPLOYMENT_SOURCE" ]]; then 32 DEPLOYMENT_SOURCE=$SCRIPT_DIR 33 fi 34 35 if [[ ! -n "$NEXT_MANIFEST_PATH" ]]; then 36 NEXT_MANIFEST_PATH=$ARTIFACTS/manifest 37 38 if [[ ! -n "$PREVIOUS_MANIFEST_PATH" ]]; then 39 PREVIOUS_MANIFEST_PATH=$NEXT_MANIFEST_PATH 40 fi 41 fi 42 43 if [[ ! -n "$KUDU_SYNC_COMMAND" ]]; then 44 # Install kudu sync 45 echo Installing Kudu Sync 46 npm install kudusync -g --silent 47 exitWithMessageOnError "npm failed" 48 49 KUDU_SYNC_COMMAND="kuduSync" 50 fi 51 52 if [[ ! -n "$DEPLOYMENT_TARGET" ]]; then 53 DEPLOYMENT_TARGET=$ARTIFACTS/wwwroot 54 else 55 # In case we are running on kudu service this is the correct location of kuduSync 56 KUDU_SYNC_COMMAND="$APPDATA\\npm\\node_modules\\kuduSync\\bin\\kuduSync" 57 fi 58 59 ################################################################################################################################## 60 # Deployment 61 # ---------- 62 63 echo Handling Basic Web Site deployment. 64 65 # 1. KuduSync 66 echo Kudu Sync from "$DEPLOYMENT_SOURCE" to "$DEPLOYMENT_TARGET" 67 $KUDU_SYNC_COMMAND -q -f "$DEPLOYMENT_SOURCE" -t "$DEPLOYMENT_TARGET" -n "$NEXT_MANIFEST_PATH" -p "$PREVIOUS_MANIFEST_PATH" -i ".git;.deployment;deploy.sh" 68 exitWithMessageOnError "Kudu Sync failed" 69 70 ################################################################################################################################## 71 72 echo "Finished successfully." 73

This script does two things: setup a bunch of environment variables so our script has all the paths to the source code repository, the target web site root and some well-known commands, Next, it runs the KuduSync executable, a helper which copies files from the source code repository to the web site root using an optimized algorithm which only copies files that have been modified. For .NET, there would be a third action which is done: running msbuild to compile sources into binaries.

Right before the part that reads # Deployment, we can add some additional steps for running unit tests. We can invoke the php.exe executable (located on the D:\ drive in Windows Azure Web Sites) and run phpunit.php passing in the path to the test configuration file:

1 ################################################################################################################################## 2 # Testing 3 # ------- 4 5 echo Running PHPUnit tests. 6 7 # 1. PHPUnit 8 "D:\Program Files (x86)\PHP\v5.4\php.exe" -d auto_prepend_file="$DEPLOYMENT_SOURCE\\vendor\\autoload.php" "$DEPLOYMENT_SOURCE\\vendor\\phpunit\\phpunit\\phpunit.php" --configuration "$DEPLOYMENT_SOURCE\\app\\phpunit.xml" 9 exitWithMessageOnError "PHPUnit tests failed" 10 echo

On a side note, we can also run other commands like issuing a composer update, similar to NuGet package restore in the .NET world:

1 echo Download composer. 2 curl -O https://getcomposer.org/composer.phar > /dev/null 3 4 echo Run composer update. 5 cd "$DEPLOYMENT_SOURCE" 6 "D:\Program Files (x86)\PHP\v5.4\php.exe" composer.phar update --optimize-autoloader 7

Putting our deployment script to the test

All that’s left to do now is commit and push our changes to Windows Azure Web Sites. If everything goes right, the output for the git push command should contain details of running our unit tests:

image

Here’s what happens when a test fails:

image

And even better, the Windows Azure Web Sites portal shows us that the latest sources were commited to the git repository but not deployed because tests failed:

image

As you can see, using deployment scripts we can customize deployment on Windows Azure Web Sites to fit our needs. We can run unit tests, fetch source code from a different location and so on. Enjoy!

Hosting a YouTrack instance on Windows Azure

Note: this is a cross-post from the JetBrains YouTrack blog. Since it is centered around Windows Azure, I thought it is appropriate to post a copy on my own blog as well.


YouTrack, JetBrains’ agile issue tracker, can be installed on different platforms. There is a stand-alone version which can be downloaded and installed on your own server. If you prefer a cloud-hosted solution there’s YouTrack InCloud available for you. There is always a third way as well: why not host YouTrack stand-alone on a virtual machine hosted in Windows Azure?

In this post we’ll walk you through getting a Windows Azure subscription, creating a virtual machine, installing YouTrack and configuring firewalls so we can use our cloud-hosted YouTrack instance from any browser on any location.

Getting a Windows Azure subscription

In order to be able to work with Windows Azure, we’ll need a subscription. Microsoft has several options there but as a first-time user, there is a 90-day free trial which comes with a limited amount of free resources, enough for hosting YouTrack. If you are an MSDN subscriber or BizSpark member, there are some additional benefits that are worth exploring.

On www.windowsazure.com, click the Try it free button to start the subscription wizard. You will be asked for a Windows Live ID and for credit card details, depending on the country you live in. No worries: you will not be charged in this trial unless you explicitly remove the spending cap.

clip_image002

The 90-day trial comes with 750 small compute hours monthly, which means we can host a single core machine with 1.5 GB of memory without being charged. There is 35 GB of storage included, enough to host the standard virtual machines available in the platform. Inbound traffic is free, 25 GB of outbound traffic is included as well. Seems reasonable to give YouTrack on Windows Azure a spin!

Enabling Windows Azure preview features

Before continuing, it is important to know that some features of the Windows Azure platform are still in preview, such as the “infrastructure-as-a-service” virtual machines (VM) we’re going to use in this blog post. After creating a Windows Azure account, make sure to enable these preview features from the administration page.

clip_image004

Once that’s done, we can direct our browser to http://manage.windowsazure.com and start our YouTrack deployment.

Creating a virtual machine

The Windows Azure Management Portal gives us access to all services activated in our subscription. Under Virtual Machines we can manage existing virtual machines or create our own.

When clicking the + New button, we can create a new virtual machine, either by using the Quick create option or by using the From gallery option. We’ll choose the latter as it provides us with some preinstalled virtual machines running a variety of operating systems, both Windows and Linux.

clip_image006

Depending on your preferences, feel free to go with one of the templates available. YouTrack is supported on both Windows and Linux. Let’s go with the latest version of Windows Server 2012 for this blog post.

Following the wizard, we can name our virtual machine and provide the administrator password. The name we’re giving in this screen is the actual hostname, not the DNS name we will be using to access the machine remotely. Note the machine size can also be selected. If you are using the free trial, make sure to use the Small machine size or charges will incur. There is also an Extra Small instance but this has few resources available.

clip_image008

In the next step of the wizard, we have to provide the DNS name for our machine. Pick anything you would like to use, do note it will always end in .cloudapp.net. No worries if you would like to link a custom domain name later since that is supported as well.

We can also select the region where our virtual machine will be located. Microsoft has 8 Windows Azure datacenters globally: 4 in the US, 2 in Europe and 2 in Asia. Pick one that’s close to you since that will reduce network latency.

clip_image010

The last step of the wizard provides us with the option of creating an availability set. Since we’ll be starting off with just one virtual machine this doesn’t really matter. However when hosting multiple virtual machines make sure to add them to the same availability set. Microsoft uses these to plan maintenance and make sure only part of your virtual machines is subject to maintenance at any given time.

After clicking the Complete button, we can relax a bit. Depending on the virtual machine size selected it may take up to 15 minutes before our machine is started. Status of the machine can be inspected through the management portal, as well as some performance indicators like CPU and memory usage.

clip_image012

Every machine has only one open firewall port by default: remote desktop for Windows VM’s (on TCP port 3389) or SSH for Linux VM’s (on TCP port 22). Which is enough to start our YouTrack installation. Using the Connect button or by opening a remote desktop or SSH session to the URL we created in the VM creation wizard, we can connect to our fresh machine as an administrator.

Installing YouTrack

After logging in to the virtual machine using remote desktop, we have a complete server available. There is a browser available on the Windows Server 2012 start screen which can be accessed by moving our mouse to the lower left-hand corner.

clip_image014

From our browser we can navigate to the JetBrains website and download the YouTrack installer. Note that by default, Internet Explorer on Windows Server is being paranoid about any website and will display a security warning. Use the Add button to allow it to access the JetBrains website. If you want to disable this entirely it’s also possible to disable Internet Explorer Enhanced Security.

clip_image016

We can now download the YouTrack installer directly from the JetBrains website. Internet Explorer will probably give us another security warning but we know the drill.

clip_image018

If you wish to save the installer to disk, you may notice that there is both a C:\ and D:\ drive available in a Windows Azure VM. It’s important to know that only the C:\ drive is persistent. The D:\ drive holds the Windows pagefile and can be used as temporary storage. It may get swiped during automated maintenance in the datacenter.

We can install YouTrack like we would do it on any other server: complete the wizard and make sure YouTrack gets installed to the C:\ drive.

clip_image020

The final step of the YouTrack installation wizard requires us to provide the port number on which YouTrack will be available. This can be any port number you want but since we’re only going to use this server to host YouTrack let’s go with the default HTTP port 80.

clip_image022

Once the wizard completes, a browser Window is opened and the initial YouTrack configuration page is loaded. Note that the first start may take a couple of minutes. An important setting to specify, next to the root password, is the system base URL. By default, this will read http://localhost. Since we want to be able to use this YouTrack instance through any browser and have correctly generated URLs in e-mail being sent out, we have to specify the full DNS name to our Windows Azure VM.

clip_image024

Once saved we can start creating a project, add issues, configure the agile board, do time tracking and so on.

clip_image026

Let’s see if we can make our YouTrack instance accessible from the outside world.

Configuring the firewall

By default, every VM can only be accessed remotely through either remote desktop or SSH. To open up access to HTTP port 80 on which YouTrack is running, we have to explicitly open some firewall ports.

Before diving in, it’s important to know that every virtual machine on Windows Azure is sitting behind a load balancer in the datacenter’s network topology. This means we will have to configure the load balancer to send traffic on HTTP port 80 to our virtual machine. Next to that, our virtual machine may have a firewall enabled as well, depending on the selected operating system. Windows Server 2012 blocks all traffic on HTTP port 80 by default which means we have to configure both our machine and the load balancer.

Allowing HTTP traffic on the VM

If you are a command-line person, open up a command console in the remote desktop session and issue the following command:

netsh advfirewall firewall add rule name="YouTrack" dir=in action=allow protocol=TCP localport=80

If not, here’s a crash-course in configuring Windows Firewall. From the remote desktop session to our machine we can bring up Windows Firewall configuration by using the Server Manager (starts with Windows) and clicking Configure this local server and then Windows Firewall.

clip_image028

Next, open Advanced settings.

clip_image030

Next, add a new inbound rule by right-clicking the Inbound Rules node and using the New Rule… menu item. In the wizard that opens, add a Port rule, specify TCP port 80, allow the connection and apply it to all firewall modes. Finally, we can give the rule a descriptive name like “Allow YouTrack”.

clip_image032

Once that’s done, we can configure the Windows Azure load balancer.

Configuring the Windows Azure load balancer

From the Windows Azure management portal, we can navigate to our newly created VM and open the Endpoints tab. Next, click Add Endpoint and open up public TCP port 80 and forward it to private port 80 (or another one if you’ve configured YouTrack differently).

clip_image034

After clicking Complete, the load balancer rules will be updated. This operation typically takes a couple of seconds. Progress will be reported on the Endpoints tab.

clip_image036

Once completed we can use any browser on any Internet-connected machine to use our YouTrack instance. Using the login created earlier, we can create projects and invite users to register with our cloud-hosted YouTrack instance.

clip_image038

Enjoy!

Tales from the trenches: resizing a Windows Azure virtual disk the smooth way

We’ve all been there. Running a virtual machine on Windows Azure and all of a sudden you notice that a virtual disk is running full. Having no access to the hypervisor nor to its storage (directly), there’s no easy way out…

Big disclaimer: use the provided code on your own risk! I’m not responsible if something breaks! The provided code is as-is without warranty! I have tested this on a couple of data disks without any problems. I've tested this on OS disks and this sometimes works, sometimes fails. Be warned.

Download/contribute: on GitHub

When searching for a solution to this issue,the typical solution you’ll find is the following:

  • Delete the VM
  • Download the .vhd
  • Resize the downloaded .vhd
  • Delete the original .vhd from blob storage
  • Upload the resized .vhd
  • Recreate the VM
  • Use diskpart to resize the partition

That’s a lot of work. Deleting and re-creating the VM isn’t that bad, it can be done pretty quickly. But doing a download of a 30GB disk, resizing the disk and re-uploading it is a serious PITA! Even if you do this on a temporary VM that sits in the same datacenter as your storage account.

Last saturday, I was in this situation… A decision would have to be made: spend an estimated 3 hours in doing the entire download/resize/upload process or reading up on the VHD file format and finding an easier way. With the possibility of having to fall back to doing the entire process…

Now what!

Being a bit geeked out, I decided to read up on the VHD file format and download the specs.

Before we dive in: why would I even read up on the VHD file format? Well, since Windows Azure storage is used as the underlying store for Windows Azure Virtual Machine VHD’s and Windows Azure storage supports byte operations without having to download an entire file, it occurred to me that combining both would result in a less-than-one-second VHD resize. Or would it?

Note that if you’re just interested in the bits to “get it done”, check the last section of this post.

Researching the VHD file format specs

The specs for the VHD file format are publicly available. Which means it shouldn't be to hard to learn how VHD files, the underlying format for virtual disks on Windows Azure Virtual Machines, are structured. Having fear of extremely complex file structures, I started reading and found that a VHD isn’t actually that complicated.

Apparently, VHD files created with Virtual PC 2004 are a bit different from newer VHD files. But hey, Microsoft will probably not use that old beast in their datacenters, right? Using that assumption and the assumption that VHD files for Windows Azure Virtual Machines are always fixed in size, I learnt the following over-generalized lesson:

A fixed-size VHD for Windows Azure Virtual Machines is a bunch of bytes representing the actual disk contents, followed by a 512-byte file footer that holds some metadata.
Maarten Balliauw – last Saturday

A-ha! So in short, if the size of the VHD file is known, the offset to the footer can be calculated and the entire footer can be read. And this footer is just a simple byte array. From the specs:

VHD footer specification

Let’s see what’s needed to do some dynamic VHD resizing…

Resizing a VHD file - take 1

My first approach to “fixing” this issue was simple:

  • Read the footer bytes
  • Write null values over it and resize the disk to (desired size + 512 bytes)
  • Write the footer in those last 512 bytes

Guess what? I tried mounting an updated VHD file in Windows, without any successful result. Time for some more reading… resulting in the big Eureka! scream: the “current size” field in the footer must be updated!

So I did that… and got failure again. But Eureka! again: the checksum must be updated so that the VHD driver can verify the footer is valid!

So I did that… and found more failure.

*sigh* – the fallback scenario of download/resize/update came to mind again…

Resizing a VHD file - take 2

Being a persistent developer, I decided to do some more searching. For most problems, at least a partial solution is available out there! And there was: CodePlex holds a library called .NET DiscUtils which supports reading from and writing to a giant load of file container formats such as ISO, VHD, various file systems, Udf, Vdi and much more!

Going through the sources and doing some research, I found the one missing piece from my first attempt: “geometry”. An old class on basic computer principles came to mind where the professor taught us that disks have geometry: cylinder-head-sector or CHS information for the disk driver which can use this info for determining physical data blocks on the disk.

Being lazy, I decided to copy-and-adapt the Footer class from this library. Why reinvent the wheel? Why risk  going sub-zero on the WIfe Acceptance Factor since this was saturday?

So I decided to generate a fresh VHD file in Windows and try to resize that one using this Footer class. Let’s start simple: specify the file to open, the desired new size and open a read/write stream to it.

1 string file = @"c:\temp\path\to\some.vhd"; 2 long newSize = 20971520; // resize to 20 MB 3 4 using (Stream stream = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite)) 5 { 6 // code goes here 7 }

Since we know the size of the file we’ve just opened, the footer is at length – 512, the Footer class takes these bytes and creates a .NET object for it:

1 stream.Seek(-512, SeekOrigin.End); 2 var currentFooterPosition = stream.Position; 3 4 // Read current footer 5 var footer = new byte[512]; 6 stream.Read(footer, 0, 512); 7 8 var footerInstance = Footer.FromBytes(footer, 0);

Of course, we want to make sure we’re working on a fixed-size disk and that it’s smaller than the requested new size.

1 if (footerInstance.DiskType != FileType.Fixed 2 || footerInstance.CurrentSize >= newSize) 3 { 4 throw new Exception("You are one serious nutcase!"); 5 }

If all is well, we can start resizing the disk. Simply writing a series of zeroes in the least optimal way will do:

1 // Write 0 values 2 stream.Seek(currentFooterPosition, SeekOrigin.Begin); 3 while (stream.Length < newSize) 4 { 5 stream.WriteByte(0); 6 }

Now that we have a VHD file that holds the desired new size capacity, there’s one thing left: updating the VHD file footer. Again, the Footer class can help us here by updating the current size, original size, geometry and checksum fields:

1 // Change footer size values 2 footerInstance.CurrentSize = newSize; 3 footerInstance.OriginalSize = newSize; 4 footerInstance.Geometry = Geometry.FromCapacity(newSize); 5 6 footerInstance.UpdateChecksum();

One thing left: writing the footer to our VHD file:

1 footer = new byte[512]; 2 footerInstance.ToBytes(footer, 0); 3 4 // Write new footer 5 stream.Write(footer, 0, footer.Length);

That’s it. And my big surprise after running this? Great success! A VHD that doubled in size.

Resize VHD Windows Azure disk

So we can now resize VHD files in under a second. That’s much faster than any VHD resizer tool you find out here! But still: what about the download/upload?

Resizing a VHD file stored in blob storage

Now that we have the code for resizing a local VHD, porting this to using blob storage and more specifically, the features provided for manipulating page blobs, is pretty straightforward. The Windows Azure Storage SDK gives us access to every single page of 512 bytes of a page blob, meaning we can work with files that span gigabytes of data while only downloading and uploading a couple of bytes…

Let’s give it a try. First of all, our file is now a URL to a blob:

1 var blob = new CloudPageBlob( 2 "http://account.blob.core.windows.net/vhds/some.vhd", 3 new StorageCredentials("accountname", "accountkey));

Next, we can fetch the last page of this blob to read our VHD’s footer:

1 blob.FetchAttributes(); 2 var originalLength = blob.Properties.Length; 3 4 var footer = new byte[512]; 5 using (Stream stream = new MemoryStream()) 6 { 7 blob.DownloadRangeToStream(stream, originalLength - 512, 512); 8 stream.Position = 0; 9 stream.Read(footer, 0, 512); 10 stream.Close(); 11 } 12 13 var footerInstance = Footer.FromBytes(footer, 0);

After doing the check on disk type again (fixed and smaller than the desired new size), we can resize the VHD. This time not by writing zeroes to it, but by calling one simple method on the storage SDK.

1 blob.Resize(newSize + 512);

In theory, it’s not required to overwrite the current footer with zeroes, but let’s play it clean:

1 blob.ClearPages(originalLength - 512, 512);

Next, we can change our footer values again:

1 footerInstance.CurrentSize = newSize; 2 footerInstance.OriginalSize = newSize; 3 footerInstance.Geometry = Geometry.FromCapacity(newSize); 4 5 footerInstance.UpdateChecksum(); 6 7 footer = new byte[512]; 8 footerInstance.ToBytes(footer, 0);

And write them to the last page of our page blob:

1 using (Stream stream = new MemoryStream(footer)) 2 { 3 blob.WritePages(stream, newSize); 4 }

And that’s all, folks! Using this code you’ll be able to resize a VHD file stored on blob storage in less than a second without having to download and upload several gigabytes of data.

Meet WindowsAzureDiskResizer

Since resizing Windows Azure VHD files is a well-known missing feature, I decided to wrap all my code in a console application and share it on GitHub. Feel free to fork, contribute and so on. WindowsAzureDiskResizer takes at least two parameters: the desired new size (in bytes) and a blob URL to the VHD. This can be a URL containing a Shared Access SIgnature.

Resize windows azure VM disk

Now let’s resize a disk. Here are the steps to take:

  • Shutdown the VM
  • Delete the VM -or- detach the disk if it’s not the OS disk
  • In the Windows Azure portal, delete the disk (retain the data!) do that the lease Windows Azure has on it is removed
  • Run WindowsAzureDiskResizer
  • In the Windows Azure portal, recreate the disk based on the existing blob
  • Recreate the VM  -or- reattach the disk if it’s not the OS disk
  • Start the VM
  • Use diskpart / disk management to resize the partition

Here’s how fast the resizing happens:

VhdResizer

Woah! Enjoy!

We’re good for now, at least until Microsoft decides to switch to the newer VHDX file format…

Download/contribute: on GitHub or binaries: WindowsAzureDiskResizer-1.0.0.0.zip (831.69 kb)

Working with Windows Azure command line tools from within Visual Studio

Right after my last post (Working with Windows Azure command line tools from PhpStorm), the obvious question came to mind… Can I do Windows Azure things using the command line tools from within Visual Studio as well? Sure you can! At least if you have the NuGet Package Manager Console installed into your Visual Studio.

For good order: you can use either the PowerShell cmdlets that are available or use the Node-based tools available (how-to). In this post we’ll be using the PowerShell cmdlets. And once those are installed… there’s nothing you have to do to get these working in Visual Studio!

The first thing we’ll have to do before being able to do anything with these cmdlets is making sure we can access the Windows Azure management service. Invoke the Get-AzurePublishSettings command. This will open op a new browser window and generate a .publishsettings. Save it somewhere and remember the full path to it. Next, we’ll have to import that file using the Import-AzurePublishSettingsFile <path to publishsettings file> command.

If everything went according to plan, we’ll now be able to do some really interesting things from inside our NuGet Package Manager console. Let’s see if we can list all Windows Azure Web Sites under our subscription… Get-AzureWebsite should do!

List Windows Azure Web Site from NuGet Package Manager console

And it did. Let’s scale our brewbuddy website and make use of 3 workers.

image

Whoa!

For reference, here’s the full list of supported cmdlets. There’s also Glenn Block’s post on some common recipes you can mash together using these cmdlets. Enjoy!

[edit] Sandrino Di Mattia has a take on this as well: http://fabriccontroller.net/blog/posts/using-the-windows-azure-cli-on-windows-and-from-within-visual-studio/

Working with Windows Azure from within PhpStorm

Working with Windows Azure and my new toy (PhpStorm), I wanted to have support for doing specific actions like creating a new web site or a new database in the IDE. Since I’m not a Java guy, writing a plugin was not an option. Fortunately, PhpStorm (or WebStorm for that matter) provide support for issuing commands from the IDE. Which led me to think that it may be possible to hook up the Windows Azure Command Line Tools in my IDE… Let’s see what we can do…

First of all, we’ll need the ‘azure’ tools. These are available for download for Windows or Mac. If you happen to have Node and NPM installed, simply issue npm install azure-cli -g and we’re good to go.

Next, we’ll have to configure PhpStorm with a custom command so that we can invoke these commands from within our IDE. From the File > Settings menu find the Command Line Tool Support pane and add a new framework:

PhpStorm custom framework

Next, enter the following detail. Note that the tool path may be different on your machine. It should be the full path to the command line tools for Windows Azure, which on my machine is C:\Program Files (x86)\Microsoft SDKs\Windows Azure\CLI\0.6.9\wbin\azure.cmd.

PhpStorm custom framework settings

Click Ok, close the settings dialog and return to your working environment. From there, we can open a command line through the Tools > Run Command menu or by simply using the Ctrl+Shift+X keyboard combo. Let’s invoke the azure command:

Running Windows Azure bash tools in PhpStrom WebStorm

Cool aye? Let’s see if we can actually do some things. The first thing we’ll have to do before being able to do anything with these tools is making sure we can access the Windows Azure management service. Invoke the azure account download command and save the generated .publishsettings file somewhere on your system. Next, we’ll have to import that file using the azure account import <path to publishsettings file> command.

If everything went according to plan, we’ll now be able to do some really interesting things from inside our PhpStorm IDE… How about we create a new Windows Azure Website named “GroovyBaby” in the West US datacenter with Git support and a local clone lined to it? Here’s the command:

azure site create GroovyBaby --git --location "West US"

And here’s the result:

Create a new website in PhpStorm

I seriously love this stuff! For reference, here’s the complete list of available commands. And Glenn Block cooked up some cool commands too.

Windows Azure Websites and PhpStorm

In my new role as Technical Evangelist at JetBrains, I’ve been experimenting with one of our products a lot: PhpStorm. I was kind of curious how this tool would integrate with Windows Azure Web Sites. Now before you quit reading this post because of that acronym: if you are a Node-head you can also use WebStorm to do the same things I will describe in this post. Let’s see if we can get a simple PHP application running on Windows Azure right from within our IDE…

Setting up a Windows Azure Web Site

Let’s go through setting up a Windows Azure Web Site real quickly. If this is the first time you hear about Web Sites and want more detail on getting started, check the Windows Azure website for a detailed step-by-step explanation.

From the management portal, click the big “New” button and create a new web site. Use the “quick create” so you just have to specify a URL and select the datacenter location where you’ll be hosted. Click “Create” and endure the 4 second wait before everything is provisioned.

Create a Windows Azure web site

Next, make sure Git support is enabled. From your newly created web site,click “Enable Git publishing”. This will create a new Git repository for your web site.

Windows Azure git

From now on, we have a choice to make. We can opt to “deploy from GitHub”, which will link the web site to a project on GitHub and deploy fresh code on every change on a specific branch. It’s very easy to do that, but we’ll be taking the other option: let’s use our Windows Azure Web Site as a Git repository instead.

Creating a PhpStorm project

After starting PhpStorm, go to VCS > Checkout from Version Control > Git. For repository URL, enter the repository that is listed in the Windows Azure management portal. It’s probably similar to @.scm.azurewebsites.net/stormy.git">https://<yourusername>@<your web name>.scm.azurewebsites.net/stormy.git.

Windows Azure PHPStorm WebStorm

Once that’s done, simply click “Clone”. PhpStorm will ask for credentials after which it will download the contents of your Windows Azure Web Site. For this post, we started from an empty web site but if we would have started with creating a web site from gallery, PhpStorm would simply download the entire web site’s contents. After the cloning finishes, this should be your PhpStorm project:

PHPStorm clone web site

Let’s add a new file by right-clicking the project and clicking New > File. Name the file “index.php” since that is one of the root documents recognized by Windows Azure Web Sites. If PhpStorm asks you if you want to add he file to the Git repository, answer affirmative. We want this file to end up being deployed some day.

The following code will do:

<?php echo "Hello world!";

Now let’s get this beauty online!

Publishing the application to Windows Azure

To commit the changes we’ve made earlier, press CTRL + K or use the menu VCS > Commit Changes. This will commit the created and modified files to our local copy of the remote Git repository.

Commit VCS changes PHPStorm

On the “Commit” button, click the little arrow and go with Commit and Push. This will make PhpStorm do two things at once: create a changeset containing our modifications and push it to Windows Azure Web Sites. We’ll be asked for a final confirmation:

Push to Windows Azure

After having clicked Push, PhpStorm will send our contents to Windows Azure Web Sites and create a new deployment as you can see from the management portal:

Windows Azure Web Sites deployment from PHPStorm

Guess what this all did? Our web site is now up and running at http://stormy.azurewebsites.net/.

image

A non-Microsoft language on Windows Azure? A non-Microsoft IDE? It all works seamlessly together! Enjoy!

Storing user uploads in Windows Azure blob storage

On one of the mailing lists I follow, an interesting question came up: “We want to write a VSTO plugin for Outlook which copies attachments to blob storage. What’s the best way to do this? What about security?”. Shortly thereafter, an answer came around: “That can be done directly from the client. And storage credentials can be encrypted for use in your VSTO plugin.”

While that’s certainly a solution to the problem, it’s not the best. Let’s try and answer…

What’s the best way to uploads data to blob storage directly from the client?

The first solution that comes to mind is implementing the following flow: the client authenticates and uploads data to your service which then stores the upload on blob storage.

Upload data to blob storage

While that is in fact a valid solution, think about the following: you are creating an expensive layer in your application that just sits there copying data from one network connection to another. If you have to scale this solution, you will have to scale out the service layer in between. If you want redundancy, you need at least two machines for doing this simple copy operation… A better approach would be one where the client authenticates with your service and then uploads the data directly to blob storage.

Upload data to blob storage using shared access signature

This approach allows you to have a “cheap” service layer: it can even run on the free version of Windows Azure Web Sites if you have a low traffic volume. You don’t have to scale out the service layer once your number of clients grows (at least, not for the uploading scenario).But how would you handle uploading to blob storage from a security point of view…

What about security? Shared access signatures!

The first suggested answer on the mailing list was this: “(…) storage credentials can be encrypted for use in your VSTO plugin.” That’s true, but you only have 2 access keys to storage. It’s like giving the master key of your house to someone you don’t know. It’s encrypted, sure, but still, the master key is at the client and that’s a potential risk. The solution? Using a shared access signature!

Shared access signatures (SAS) allow us to separate the code that signs a request from the code that executes it. It basically is a set of query string parameters attached to a blob (or container!) URL that serves as the authentication ticket to blob storage. Of course, these parameters are signed using the real storage access key, so that no-one can change this signature without knowing the master key. And that’s the scenario we want to support…

On the service side, the place where you’ll be authenticating your user, you can create a Web API method (or ASMX or WCF or whatever you feel like) similar to this one:

public class UploadController : ApiController { [Authorize] public string Put(string fileName) { var account = CloudStorageAccount.DevelopmentStorageAccount; var blobClient = account.CreateCloudBlobClient(); var blobContainer = blobClient.GetContainerReference("uploads"); blobContainer.CreateIfNotExists(); var blob = blobContainer.GetBlockBlobReference("customer1-" + fileName); var uriBuilder = new UriBuilder(blob.Uri); uriBuilder.Query = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy { Permissions = SharedAccessBlobPermissions.Write, SharedAccessStartTime = DateTime.UtcNow, SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5) }).Substring(1); return uriBuilder.ToString(); } }

This method does a couple of things:

  • Authenticate the client using your authentication mechanism
  • Create a blob reference (not the actual blob, just a URL)
  • Signs the blob URL with write access, allowed from now until now + 5 minutes. That should give the client 5 minutes to start the upload.

On the client side, in our VSTO plugin, the only thing to do now is call this method with a filename. The web service will create a shared access signature to a non-existing blob and returns that to the client. The VSTO plugin can then use this signed blob URL to perform the upload:

Uri url = new Uri("http://...../uploads/customer1-test.txt?sv=2012-02-12&st=2012-12-18T08%3A11%3A57Z&se=2012-12-18T08%3A16%3A57Z&sr=b&sp=w&sig=Rb5sHlwRAJp7mELGBiog%2F1t0qYcdA9glaJGryFocj88%3D"); var blob = new CloudBlockBlob(url); blob.Properties.ContentType = "test/plain"; using (var data = new MemoryStream( Encoding.UTF8.GetBytes("Hello, world!"))) { blob.UploadFromStream(data); }

Easy, secure and scalable. Enjoy!

Protecting your ASP.NET Web API using OAuth2 and the Windows Azure Access Control Service

An article I wrote a while ago has been posted on DeveloperFusion:

The world in which we live evolves at a vast speed. Today, many applications on the Internet expose an API which can be consumed by everyone using a web browser or a mobile application on their smartphone or tablet. How would you build your API if you want these apps to be a full-fledged front-end to your service without compromising security? In this article, I’ll dive into that. We’ll be using OAuth2 and the Windows Azure Access Control Service to secure our API yet provide access to all those apps out there.

Why would I need an API?

A couple of years ago, having a web-based application was enough. Users would navigate to it using their computer’s browser, do their dance and log out again. Nowadays, a web-based application isn’t enough anymore. People have smartphones, tablets and maybe even a refrigerator with Internet access on which applications can run. Applications or “apps”. We’re moving from the web towards apps.

If you want to expose your data and services to external third-parties, you may want to think about building an API. Having an API gives you a giant advantage on the Internet nowadays. Having an API will allow your web application to reach more users. App developers will jump onto your API and build their app around it. Other websites or apps will integrate with your services by consuming your API. The only thing you have to do is expose an API and get people to know it. Apps will come. Integration will come.

A great example of an API is Twitter. They have a massive data store containing tweets and data related to that. They have user profiles. And a web site. And an API. Are you using www.twitter.com to post tweets? I am using the website, maybe once a year. All other tweets come either from my Windows Phone 7’s Twitter application or through www.hootsuite.com, a third-party Twitter client which provides added value in the form of statistics and scheduling. Both the app on my phone as well as the third-party service are using the Twitter API. By exposing an API, Twitter has created a rich ecosystem which drives adoption of their service, reaches more users and adds to their real value: data which they can analyze and sell.

(…)

Getting to know OAuth2

If you decide that your API isn’t public or specific actions can only be done for a certain user (let that third party web site get me my tweets, Twitter!), you’ll be facing authentication and authorization problems. With ASP.NET Web API, this is simple: add an [Authorize] attribute on top of a controller or action method and you’re done, right? Well, sort of…

When using the out-of-the-box authentication/authorization mechanisms of ASP.NET Web API, you are relying on basic or Windows authentication. Both require the user to log in. While perfectly viable and a good way of securing your API, a good alternative may be to use delegation.

In many cases, typically with public API’s, your API user will not really be your user, but an application acting on behalf of that user. That means that the application should know the user’s credentials. In an ideal world, you would only give your username and password to the service you’re using rather than just trusting the third-party application or website with it. You’ll be delegating access to these third parties. If you look at Facebook for example, many apps and websites redirect you to Facebook to do the login there instead of through the app itself.

Head over to the original article for more! (I’ll also be doing a talk on this on some upcoming conferences)