Logo

Maarten Balliauw {blog}

ASP.NET, ASP.NET MVC, Windows Azure, PHP, ...

About the author

Maarten Balliauw is currently employed as a Technical Evangelist at JetBrains. His interests are mainly web applications developed in ASP.NET (C#) or PHP and the Windows Azure cloud platform.
More about me More about me
Send mail E-mail me


ASP.NET MVC Quickly Pro NuGet Subscribe to my RSS feed Follow me on Twitter! View Maarten Balliauw's profile on LinkedIn
Maarten Balliauw - MVP - Most Valuable Professional
Maarten Balliauw - ASPInsider

Search

Archive

Disclaimer

The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

© Copyright Maarten Balliauw 2013


Running Memcached on Windows Azure for PHP

After three conferences in two weeks with a lot of “airport time”, which typically converts into “let’s code!” time, I think I may have tackled a commonly requested Windows Azure feature for PHP developers. Some sort of distributed caching is always a great thing to have when building scalable services and applications. While Windows Azure offers a distributed caching layer under the form of the Windows Azure Caching, that components currently lacks support for non-.NET technologies. I’ve heard there’s work being done there, but that’s not very interesting if you are building your app today. This blog post will show you how to modify a Windows Azure deployment to run and use Memcached in the easiest possible manner.

Note: this post focuses on PHP but can also be used to setup Memcached on Windows Azure for NodeJS, Java, Ruby, Python, …

Related downloads:
The scaffolder source code: MemcachedScaffolderSource.zip (1.12 mb)
The scaffolder, packaged and ready for use: MemcachedScaffolder.phar (2.87 mb)

The short version: use my scaffolder

As you may know, when working with PHP on Windows Azure and when making use of the Windows Azure SDK, you can use and create scaffolders. The Windows Azure SDK for PHP includes a powerful scaffolding feature that allows users to quickly setup a pre-packaged and configured website ready for Windows Azure.

If you want to use Memcached in your project, do the following:

  • Download my custom MemcacheScaffolder (MemcachedScaffolder.phar (2.87 mb)) and make sure it is located either under the scaffolders folder of the Windows Azure SDK for PHP, or that you remember the path to this scaffolder
  • Run the scaffolder from the command line: (note: best use the latest SVN version of the command line tools)
1 scaffolder run -out="c:\temp\myapp" -s="MemcachedScaffolder"

  • Find the newly created Windows Azure project structure in the folder you’ve used.
  • In your PHP code, simply add require_once 'memcache.inc.php'; to your code, and enjoy the $memcache variable which will hold a preconfigured Memcached client for you to use. This $memcache instance will also be automatically updated when adding more server instances or deleting server instances.
  • 1 require_once 'memcache.inc.php';

    That’s it!

    The long version: what this scaffolder does behind the scenes

    Of course, behind this “developers can simply use 1 line of code” trick a lot of things happen in the background. Let’s go through the places I’ve made changes from the default scaffolder.

    The ServiceDefinition.csdef file

    Let’s start with the beginning: when running Memcached in a Windows Azure instance, you’ll have to specify it with a port number to use. As such, the ServiceDefinition.csdef file which defines what the datacenter configuration for your app should be looks like the following:

    1 <?xml version="1.0" encoding="utf-8"?> 2 <ServiceDefinition name="PhpOnAzure" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"> 3 <WebRole name="PhpOnAzure.Web" enableNativeCodeExecution="true"> 4 <Sites> 5 <Site name="Web" physicalDirectory="./PhpOnAzure.Web"> 6 <Bindings> 7 <Binding name="Endpoint1" endpointName="HttpEndpoint" /> 8 </Bindings> 9 </Site> 10 </Sites> 11 <Startup> 12 <Task commandLine="add-environment-variables.cmd" executionContext="elevated" taskType="simple" /> 13 <Task commandLine="install-php.cmd" executionContext="elevated" taskType="simple"> 14 <Environment> 15 <Variable name="EMULATED"> 16 <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" /> 17 </Variable> 18 </Environment> 19 </Task> 20 <Task commandLine="memcached.cmd" executionContext="elevated" taskType="background" /> 21 <Task commandLine="monitor-environment.cmd" executionContext="elevated" taskType="background" /> 22 </Startup> 23 <Endpoints> 24 <InputEndpoint name="HttpEndpoint" protocol="http" port="80" /> 25 <InternalEndpoint name="MemcachedEndpoint" protocol="tcp" /> 26 </Endpoints> 27 <Imports> 28 <Import moduleName="Diagnostics"/> 29 </Imports> 30 <ConfigurationSettings> 31 </ConfigurationSettings> 32 </WebRole> 33 </ServiceDefinition>

    Note the <InternalEndpoint name="MemcachedEndpoint" protocol="tcp" /> line of code. This one defines that the web role instance should open some TCP port in the firewall with the name MemcachedEndpoint and expose that to the other virtual machines in your deployment. We’ll use this named endpoint later when starting Memcached.

    Something else in this file is noteworthy: the startup tasks under the <Startup> element. With the default scaffolder, the first two tasks (namely add-environment-variables.cmd and install-php.cmd) are also present. These do nothing more than providing some environment information about your deployment in the environment variables. The second one does what its name implies: install PHP on your virtual machine. The latter two scripts added, memcached.cmd and monitor-environment.cmd are used to bootstrap Memcached. Note these two tasks run as background tasks: I wanted to have these two always running to ensure when Memcached crashes the task can simply restart Memcached.

    The php folder

    If you’ve played with the default scaffolder in the Windows Azure SDK for PHP, you probably know that the PHP installation in Windows Azure is a “default” one. This means: no memcached extension is in there. To overcome this, simply copy the correct php_memcache.dll extension into the /php/ext folder and Windows Azure (well, the install-php.cmd script) will know what to do with it.

    Memcached.cmd and Memcached.ps1

    Under the application’s bin folder, I’ve added some additional startup tasks. The one responsible for starting (and maintaining a running instance of) Memcached is, of course, Memcached.cmd. This one simply delegates the call to Memcached.ps1, of which the following is the source code:

    1 [Reflection.Assembly]::LoadWithPartialName("Microsoft.WindowsAzure.ServiceRuntime") 2 3 # Start memcached. To infinity and beyond! 4 while (1) { 5 $p = [diagnostics.process]::Start("memcached.exe", "-m 64 -p " + [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::CurrentRoleInstance.InstanceEndpoints["MemcachedEndpoint"].IPEndpoint.Port) 6 $p.WaitForExit() 7 }

    To be honest, this file is pretty simple. It loads the WindowsAzure ServiceRuntime assembly which contains all kinds of information about the current deployment. Next, I start an infinite loop which continuously starts a new memcached.exe process consuming 64MB of RAM memory and listens on the port specified by the MemcachedEndpoint defined earlier.

    Monitor-environment.cmd and Monitor-environment.ps1

    The monitor-environment.cmd script takes the same approach as the memcached.cmd script: just pass the command along to a PowerShell script in the form of monitor-environment.ps1. I do want to show you the monitor-environment.cmd script however, as there’s one difference in there: I’m changing the file system permissions for my application (the icacls line).

    1 @echo off 2 cd "%~dp0" 3 4 icacls %RoleRoot%\approot /grant "Everyone":F /T 5 6 powershell.exe Set-ExecutionPolicy Unrestricted 7 powershell.exe .\monitor-environment.ps1

    The reason for changing permissions is simple: I want to make sure I can write a PHP script to disk every minute. Yes, you heard me! I’m using PowerShell (in the monitor-environment.ps1 script) to generate PHP code. Here’s the PowerShell:

    1 [Reflection.Assembly]::LoadWithPartialName("Microsoft.WindowsAzure.ServiceRuntime") 2 3 # To infinity and beyond! 4 5 while(1) { 6 ########################################################## 7 # Create memcached include file for PHP 8 ########################################################## 9 10 # Dump all memcached endpoints to ../memcached-servers.php 11 $memcached = "<?php`r`n" 12 $memcached += "`$memcachedServers = array(" 13 14 $currentRolename = [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::CurrentRoleInstance.Role.Name 15 $roles = [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::Roles 16 foreach ($role in $roles.Keys | sort-object) { 17 if ($role -eq $currentRolename) { 18 $instances = $roles[$role].Instances 19 for ($i = 0; $i -lt $instances.Count; $i++) { 20 $endpoints = $instances[$i].InstanceEndpoints 21 foreach ($endpoint in $endpoints.Keys | sort-object) { 22 if ($endpoint -eq "MemcachedEndpoint") { 23 $memcached += "array(`"" 24 $memcached += $endpoints[$endpoint].IPEndpoint.Address 25 $memcached += "`" ," 26 $memcached += $endpoints[$endpoint].IPEndpoint.Port 27 $memcached += "), " 28 } 29 30 31 } 32 } 33 } 34 } 35 36 $memcached += ");" 37 38 Write-Output $memcached | Out-File -Encoding Ascii ../memcached-servers.php 39 40 # Restart the loop in 1 minute 41 Start-Sleep -Seconds 60 42 }

    The output is being written every minute to the memcached-servers.php file. Why every minute? Well, if servers are added or removed I want my application to use the correct set of servers. This leaves a possible gap of one minute where some server may not be available, you can easily catch any error related to this in your PHP code (or add a comment to this blog post telling me what’s a better interval). Anyway, here’s the sample output:

    1 <?php 2 $memcachedServers = array(array('10.0.0.1', 11211), array('10.0.0.2', 11211), );

    All there’s left to do is consume this array. I’ve added a default memcache.inc.php file in the root of the web role to make things easy:

    1 <?php 2 require_once $_SERVER["RoleRoot"] . '\\approot\\memcached-servers.php'; 3 $memcache = new Memcache(); 4 foreach ($memcachedServers as $memcachedServer) { 5 if (strpos($memcachedServer[0], '127.') !== false) { 6 $memcachedServer[0] = 'localhost'; 7 } 8 $memcache->addServer($memcachedServer[0], $memcachedServer[1]); 9 }

    Include this file in your code and you have a full-blown distributed cache available in your Windows Azure deployment! Here’s a sample of some operations that can be done on Memcached:

    1 <?php 2 error_reporting(E_ALL); 3 require_once 'memcache.inc.php'; 4 5 var_dump($memcachedServers); 6 var_dump($memcache->getVersion()); 7 8 $memcache->set('key1', 'value1', false, 30); 9 echo $memcache->get('key1'); 10 11 $memcache->set('var_key', 'some really big variable', MEMCACHE_COMPRESSED, 50); 12 echo $memcache->get('var_key');

    That’s it!

    Conclusion and feedback

    This is just a fun project I’ve been working on when lonely and bored on airports. However, if you think this is valuable and in your opinion should be made available as a standard thing in the Windows Azure SDK for PHP, let me know. I’ll be happy to push this into the main branch and make sure it’s available in a future release.

    Comments or praise? There’s a comment form right below this post!


    Setting up a NuGet repository in seconds: MyGet public feeds

    A few months ago, my colleague Xavier Decoster and I introduced MyGet as a tool where you can create your own, private NuGet feeds. A couple of weeks later we introduced some options to delegate feed privileges to other MyGet users allowing you to make another MyGet user “co-admin” or “contributor” to a feed. Since then we’ve expanded our view on the NuGet ecosystem and moved MyGet from a solution to create your private feeds to a service that allows you to set up a NuGet feed, whether private or public.

    Supporting public feeds allows you to set up a structure similar to www.nuget.org: you can give any user privileges to publish a package to your feed while the user can never manage other packages on your feed. This is great in several scenarios:

    • You run an open source project and want people to contribute modules or plugins to your feed
    • You are a business and you want people to contribute internal packages to your feed whilst prohibiting them from updating or deleting other packages

    Setting up a public feed

    Setting up a public feed on MyGet is similar to setting up a private feed. In fact, both are identical except for the default privileges assigned to users. Navigate to www.myget.org and sign in using an identity provider of choice. Next, create a feed, for example:

    Create a MyGet NuGet feed and host your own NuGet packages

    This new feed may be named “public”, however it is private by obscurity: if someone knows the URL to the feed, he/she can consume packages from it. Let’s change that. Go to the “Feed Security” tab and have a look at the assigned privileges for Everyone. By default, these are set to “Can consume this feed”, meaning that everyone can add the feed URL to Visual Studio and consume packages. Other options are “No access” (requires authentication prior to being able to consume the feed) and “Can contribute own packages to this feed”. This last one is what we want:

    Setting up a NuGet feed

    Assigning the “Can contribute own packages to this feed” privilege to a specific user or to everyone means that the user (or everyone) will be able to contribute packages to the feed, as long as the package id used is not already on the feed and as long as the package id was originally submitted by this user. Exactly the same model as www.nuget.org, that is.

    For reference, all available privileges are:

    • Has no access to this feed (speaks for itself)
    • Can consume this feed (allows the user to use the feed in Visual Studio / NuGet)
    • Can contribute own packages to this feed '(allows the user to contribute packages but can only update and remove his own packages and not those of others)
    • Can manage all packages for this feed (allows the user to add packages to the feed via the website and via the NuGet push API)
    • Can manage users and all packages for this feed (extends the above with feed privilege management capabilities)

    Contributing to a public feed

    Of course, if you have a public feed you may want to have people contributing to it. This is very easy: provide them with a link to your feed editing page (for example, http://www.myget.org/Feed/Edit/public). Users can publish their packages via the MyGet user interface in no time.

    If you want to have users push packages using nuget.exe or NuGet Package Explorer, provide them a link to the feed endpoint (for example, http://www.myget.org/F/public/). Using their API key (which can be found in the MyGet profile for the user) they can push packages to the public feed from any API consumer.

    Enjoy!

     

    PS: We’re working on lots more, but will probably provide that in a MyGet Premium version. Make sure to subscribe to our newsletter on www.myget.org if this is of interest.


    NuGet push... to Windows Azure

    When looking at how people like to deploy their applications to a cloud environment, a large faction seems to prefer being able to use their source control system as a source for their production deployment. While interesting, I see a lot of problems there: your source code may not run immediately and probably has to be compiled. You don’t want to maintain compiled assemblies in source control, right? Also, maybe some QA process is in place where a deployment can only occur after approval. Why not use source control for what it’s there for: source control? And how about using a NuGet repository as the source for our deployment? Meet the Windows Azure NuGetRole.

    Disclaimer/Warning: this is demo material and should probably not be used for real-life deployments without making it bullet proof!

    Download the sample code: NuGetRole.zip (262.22 kb)

    How to use it

    If you compile the source code (download), you have X steps left in getting your NuGetRole running on Windows Azure:

    • Specifying the package source to use
    • Add some packages to the package source feed (which you can easily host on MyGet)
    • Deploy to Windows Azure

    When all these steps have been taken care of, the NuGetRole will download all latest package versions from the package source specified in ServiceConfiguration.cscfg:

    1 <?xml version="1.0" encoding="utf-8"?> 2 <ServiceConfiguration serviceName="NuGetRole.Azure" 3 xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" 4 osFamily="1" 5 osVersion="*"> 6 <Role name="NuGetRole.Web"> 7 <Instances count="1" /> 8 <ConfigurationSettings> 9 <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" /> 10 <Setting name="PackageSource" value="http://www.myget.org/F/nugetrole/" /> 11 </ConfigurationSettings> 12 </Role> 13 </ServiceConfiguration>

    Packages you publish should only contain a content and/or lib folder. Other package contents will currently be ignored by the NuGetRole. If you want to add some web content like a default page to your role, simply publish the following package:

    NuGet Package Explorer MyGet NuGet NuGetRole Azure

    Just push, and watch your Windows Azure web role farm update their contents. Or have your build server push a NuGet package containing your application and have your server farm update itself. Whatever pleases you.

    How it works

    What I did was create a fairly empty Windows Azure project (download).  In this project, one Web role exists. This web role consists of nothing but a Web.config file and a WebRole.cs class which looks like the following:

    1 public class WebRole : RoleEntryPoint 2 { 3 private bool _isSynchronizing; 4 private PackageSynchronizer _packageSynchronizer = null; 5 6 public override bool OnStart() 7 { 8 var localPath = Path.Combine(Environment.GetEnvironmentVariable("RdRoleRoot") + "\\approot"); 9 10 _packageSynchronizer = new PackageSynchronizer( 11 new Uri(RoleEnvironment.GetConfigurationSettingValue("PackageSource")), localPath); 12 13 _packageSynchronizer.SynchronizationStarted += sender => _isSynchronizing = true; 14 _packageSynchronizer.SynchronizationCompleted += sender => _isSynchronizing = false; 15 16 RoleEnvironment.StatusCheck += (sender, args) => 17 { 18 if (_isSynchronizing) 19 { 20 args.SetBusy(); 21 } 22 }; 23 24 return base.OnStart(); 25 } 26 27 public override void Run() 28 { 29 _packageSynchronizer.SynchronizeForever(TimeSpan.FromSeconds(30)); 30 31 base.Run(); 32 } 33 }

    The above code is essentially wiring some configuration values like the local web root and the NuGet package source to use to a second class in this project: the PackageSynchronizer. This class simply checks the specified NuGet package source every few minutes, checks for the latest package versions and if required, updates content and bin files.  Each synchronization run does the following:

    1 public void SynchronizeOnce() 2 { 3 var packages = _packageRepository.GetPackages() 4 .Where(p => p.IsLatestVersion == true).ToList(); 5 6 var touchedFiles = new List<string>(); 7 8 // Deploy new content 9 foreach (var package in packages) 10 { 11 var packageHash = package.GetHash(); 12 var packageFiles = package.GetFiles(); 13 foreach (var packageFile in packageFiles) 14 { 15 // Keep filename 16 var packageFileName = packageFile.Path.Replace("content\\", "").Replace("lib\\", "bin\\"); 17 18 // Mark file as touched 19 touchedFiles.Add(packageFileName); 20 21 // Do not overwrite content that has not been updated 22 if (!_packageFileHash.ContainsKey(packageFileName) || _packageFileHash[packageFileName] != packageHash) 23 { 24 _packageFileHash[packageFileName] = packageHash; 25 26 Deploy(packageFile.GetStream(), packageFileName); 27 } 28 } 29 30 // Remove obsolete content 31 var obsoleteFiles = _packageFileHash.Keys.Except(touchedFiles).ToList(); 32 foreach (var obsoletePath in obsoleteFiles) 33 { 34 _packageFileHash.Remove(obsoletePath); 35 Undeploy(obsoletePath); 36 } 37 } 38 }

    Or in human language:

    • The specified NuGet package source is checked for packages
    • Every package marked “IsLatest” is being downloaded and deployed onto the machine
    • Files that have not been used in the current synchronization step are deleted

    This is probably not a bullet-proof solution, but I wanted to show you how easy it is to use NuGet not only as a package manager inside Visual Studio, but also from your code: NuGet is not just a package manager but in essence a package management protocol. Which you can easily extend.

    One thing to note: I also made the Windows Azure load balancer ignore the role that’s updating itself. This means a roie instance that is synchronizing its contents will never be available in the load balancing pool so no traffic is sent to the role instance during an update.


    A client side Glimpse to your PHP application

    Glimpse for PHPA few months ago, the .NET world was surprised with a magnificent tool called “Glimpse”. Today I’m pleased to release a first draft of a PHP version for Glimpse! Now what is this Glimpse thing… Well: "what Firebug is for the client, Glimpse does for the server... in other words, a client side Glimpse into whats going on in your server."

    For a quick demonstration of what this means, check the video at http://getglimpse.com/. Yes, it’s a .NET based video but the idea behind Glimpse for PHP is the same. And if you do need a PHP-based one, check http://screenr.com/27ds (warning: unedited :-))

    Fundamentally Glimpse is made up of 3 different parts, all of which are extensible and customizable for any platform:

    • Glimpse Server Module
    • Glimpse Client Side Viewer
    • Glimpse Protocol

    This means an server technology that provides support for the Glimpse protocol can provide the Glimpse Client Side Viewer with information. And that’s what I’ve done.

    What can I do with Glimpse?

    A lot of things. The most basic usage of Glimpse would be enabling it and inspecting your requests by hand. Here’s a small view on the information provided:

    Glimpse phpinfo()

    By default, Glimpse offers you a glimpse into the current Ajax requests being made, your PHP Configuration, environment info, request variables, server variables, session variables and a trace viewer. And then there’s the remote tab, Glimpse’s killer feature.

    When configuring Glimpse through www.yoursite.com/?glimpseFile=Config, you can specify a Glimpse session name. If you do that on a separate device, for example a customer’s browser or a mobile device you are working with, you can distinguish remote sessions in the remote tab. This allows debugging requests that are being made live on other devices! A full description is over at http://getglimpse.com/Help/Plugin/Remote.

    PHP debug mobile browser

    Adding Glimpse to your PHP project

    Installing Glimpse in a PHP application is very straightforward. Glimpse is supported starting with PHP 5.2 or higher.

    • For PHP 5.2, copy the source folder of the repository to your server and add <?php include '/path/to/glimpse/index.php'; ?> as early as possible in your PHP script.
    • For PHP 5.3, copy the glimpse.phar file from the build folder of the repository to your server and add <?php include 'phar://path/to/glimpse.phar'; ?> as early as possible in your PHP script.

    Here’s an example of the Hello World page shown above:

    1 <?php 2 require_once 'phar://../build/Glimpse.phar'; 3 ?> 4 <html> 5 <head> 6 <title>Hello world!</title> 7 </head> 8 9 <?php Glimpse_Trace::info('Rendering body...'); ?> 10 <body> 11 <h1>Hello world!</h1> 12 <p>This is just a test.</p> 13 </body> 14 <?php Glimpse_Trace::info('Rendered body.'); ?> 15 </html>

    Enabling Glimpse

    From the moment Glimpse is installed into your web application, navigate to your web application and append the ?glimpseFile=Config query string to enable/disable Glimpse. Optionally, a client name can also be specified to distinguish remote requests.

    Configuring Glimpse for PHP

    After enabling Glimpse, a small “eye” icon will appear in the bottom-right corner of your browser. Click it and behold the magic!

    Now of course: anyone can potentially enable Glimpse. If you don’t want that, ensure you have some conditional mechanism around the <?php require_once 'phar://../build/Glimpse.phar'; ?> statement.

    Creating a first Glimpse plugin

    Not enough information on your screen? Working with Zend Framework and want to have a look at route values? Want to work with Wordpress and view some hidden details about a post through Glimpse? The sky is the limit. All there’s to it is creating a Glimpse plugin and registering it. Implementing Glimpse_Plugin_Interface is enough:

    1 <?php 2 class MyGlimpsePlugin 3 implements Glimpse_Plugin_Interface 4 { 5 public function getData(Glimpse $glimpse) { 6 $data = array( 7 array('Included file path') 8 ); 9 10 foreach (get_included_files() as $includedFile) { 11 $data[] = array($includedFile); 12 } 13 14 return array( 15 "MyGlimpsePlugin" => count($data) > 0 ? $data : null 16 ); 17 } 18 19 public function getHelpUrl() { 20 return null; // or the URL to a help page 21 } 22 } 23 ?>

    To register the plugin, add a call to $glimpse->registerPlugin():

    1 <?php 2 $glimpse->registerPlugin(new MyGlimpsePlugin()); 3 ?>

    And Bob’s your uncle:

    Creating a Glimpse plugin in PHP

    Now what?

    Well, it’s up to you. First of all: all feedback would be welcomed. Second of all: this is on Github (https://github.com/Glimpse/Glimpse.PHP). Feel free to fork and extend! Feel free to contribute plugins, core features, whatever you like! Have a lot of CakePHP projects? Why not contribute a plugin that provides a Glimpse at CakePHP diagnostics?

    ‘Till next time!


    Categories: ASP.NET | C# | General | ICT | PHP | Projects | Software | Webfarm | Glimpse

    Version 4 of the Windows Azure SDK for PHP released

    Only a few months after the Windows Azure SDK for PHP 3.0.0, Microsoft and RealDolmen are proud to present you the next version of the most complete SDK for Windows Azure out there (yes, that is a rant against the .NET SDK!): Windows Azure SDK for PHP. We’ve been working very hard with an expanding globally distributed team on getting this version out.

    The Windows Azure SDK 4 contains some significant feature enhancements. For example, it now incorporates a PHP library for accessing Windows Azure storage, a logging component, a session sharing component and clients for both the Windows Azure and SQL Azure Management API’s. On top of that, all of these API’s are now also available from the command-line both under Windows and Linux. This means you can batch-script a complete datacenter setup including servers, storage, SQL Azure, firewalls, … If that’s not cool, move to the North Pole.

    Here’s the official change log:

    • New feature: Service Management API support for SQL Azure
    • New feature: Service Management API's exposed as command-line tools
    • New feature: MicrosoftWindowsAzureRoleEnvironment for retrieving environment details
    • New feature: Package scaffolders
    • Integration of the Windows Azure command-line packaging tool
    • Expansion of the autoloader class increasing performance
    • Several minor bugfixes and performance tweaks

    Some interesting links on some of the new features:

    Also keep an eye on www.sdn.nl where I’ll be posting an article on scripting a complete application deployment to Windows Azure, including SQL Azure, storage and firewalls.

    And finally: keep an eye on http://azurephp.interoperabilitybridges.com and http://blogs.technet.com/b/port25/. I have a feeling some cool stuff may be coming following this release...


    Copy packages from one NuGet feed to another

    Copy packages from one NuGet feed to another - MyGet NuGet Server

    Yesterday, a funny discussion was going on at the NuGet Discussion Forum on CodePlex. Funny, you say? Well yes. Funny because it was about a feature we envisioned as being a must-have feature for the NuGet ecosystem: copying packages from the NuGet feed to another feed. And funny because we already have that feature present in MyGet. You may wonder why anyone wants to do that? Allow me to explain.

    Scenarios where copying packages makes sense

    The first scenario is feed stability. Imagine you are building a project and expect to always reference a NuGet package from the official feed. That’s OK as long as you have that package present in the NuGet feed, but what happens if someone removes it or updates it without respecting proper versioning? This should not happen, but it can be an unpleasant surprise if it happens. Copying the package to another feed provides stability: the specific package version is available on that other feed and will never change unless you update or remove it. It puts you in control, not the package owner.

    A second scenario: enhanced speed! It’s still much faster to pull packages from a local feed or a feed that’s geographically distributed, like the one MyGet offers (US and Europe at the moment). This is not to bash any carriers or network providers, it’s just physics: electrons don’t travel that fast and it’s better to have them coming from a closer location.

    But… how to do it? Client side

    There are some solutions to this problem/feature. The first one is a hard one: write a script that just pulls packages from the official feed. You’ll find a suggestion on how to do that here. This thing however does not pull along dependencies and forces you to do ugly, user-unfriendly things. Let’s go for beauty :-)

    Rob Reynolds (aka @ferventcoder) added some extension sauce to the NuGet.exe:

    NuGet.exe Install /ExcludeVersion /OutputDir %LocalAppData%\NuGet\Commands AddConsoleExtension NuGet.exe addextension nuget.copy.extension NuGet.exe copy castle.windsor –destination http://myget.org/F/somefeed

    Sweet! And Rob also shared how he created this extension (warning: interesting read!)

    But… how to do it? Server side

    The easiest solution is to just use MyGet! We have a nifty feature in there named “Mirror packages”. It copies the selected package to your private feed, distributes it across our CDN nodes for a fast download and it pulls along all dependencies.

    Mirror a NuGet package - Copy a NuGet package

    Enjoy making NuGet a component of your enterprise workflow! And MyGet of course as well!


    A first look at Windows Azure AppFabric Applications

    After the Windows Azure AppFabric team announced the availability of Windows Azure AppFabric Applications (preview), I signed up for early access immediately and got in. After installing the tools and creating a namespace through the portal, I decided to give it a try to see what it’s all about. Note that Neil Mackenzie also has an extensive post on “WAAFapps” which I recommend you to read as well.

    So what is this Windows Azure AppFabric Applications thing?

    Before answering that question, let’s have a brief look at what Windows Azure is today. According to Microsoft, Windows Azure is a “PaaS” (Platform-as-a-Service) offering. What that means is that Windows Azure offers a series of platform components like compute, storage, caching, authentication, a service bus, a database, a CDN, … to your applications.

    Consuming those components is pretty low level though: in order to use, let’s say, caching, one has to add the required references, make some web.config changes and open up a connection to these things. Ok, an API is provided but it’s not the case that you can seamlessly integrate caching into an application in seconds (in a manner like one would integrate file system access in an application which you literally can do in seconds).

    Meet Windows Azure AppFabric Applications. Windows Azure AppFabric Applications (why such long names, Microsoft!) redefine the concept of Platform-as-a-Service: where Windows Azure out of the box is more like a “Platform API-as-a-Service”, Windows Azure AppFabric Applications  is offering tools and platform support for easily integrating the various Windows Azure components.

    This “redefinition” of Windows Azure introduces some new concepts: in Windows Azure you have roles and role instances. In AppFabric Applications you don’t have that concept: AFA (yes, I managed to abbreviate it!) uses so-called Containers. A Container is a logical unit in which one or more services of an application are hosted. For example, if you have 2 web applications, caching and SQL Azure, you will (by default) have one Container containing 2 web applications + 2 service references: one for caching, one for SQL Azure.

    Containers are not limited to one role or role instance: a container is a set of predeployed role instances on which your applications will run. For example, if you add a WCF service, chances are that this will be part of the same container. Or a different one if you specify otherwise.

    It’s pretty interesting that you can scale containers separately. For example, one can have 2 scale units for the container containing web applications, 3 for the WCF container, … A scale unit is not necessarily just one extra instance: it depends on how many services are in a container? In fact, you shouldn’t care anymore about role instances and virtual machines: with AFA (my abbreviation for Windows Azure AppFabric Applications, remember) one can now truly care about only one thing: the application you are building.

    Hello, Windows Azure AppFabric Applications

    Visual Studio tooling support

    To demonstrate a few concepts, I decided to create a simple web application that uses caching to store the number of visits to the website. After installing the Visual Studio tooling, I started with one of the templates contained in the SDK:

    Creating a Windows Azure AppFabric Application

    This template creates a few things. To start with, 2 projects are created in Visual Studio: one MVC application in which I’ll create my web application, and one Windows Azure AppFabric Application containing a file App.cs which seems to be a DSL for building Windows Azure AppFabric Application. Opening this DSL gives the following canvas in Visual Studio:

    App.cs Windows Azure AppFabric Applications

    As you can see, this is the overview of my application as well as how they interact with each other. For example, the “MVCWebApp” has 1 endpoint (to serve HTTP requests) + 2 service references (to Windows Azure AppFabric caching and SQL Azure). This is an important notion as it will generate integration code for you. For example, in my MVC web application I can find the ServiceReferences.g.cs file containing the following code:

    1 class ServiceReferences 2 { 3 public static Microsoft.ApplicationServer.Caching.DataCache CreateImport1() 4 { 5 return Service.ExecutingService.ResolveImport<Microsoft.ApplicationServer.Caching.DataCache>("Import1"); 6 } 7 8 public static System.Data.SqlClient.SqlConnection CreateImport2() 9 { 10 return Service.ExecutingService.ResolveImport<System.Data.SqlClient.SqlConnection>("Import2"); 11 } 12 }

    Wait a minute… This looks like a cool thing! It’s basically a factory for components that may be hosted elsewhere! Calling ServiceReferences.CreateImport1() will give me a caching client that I can immediately work with! ServiceReferences.CreateImport2() (you can change these names by the way) gives me a connection to SQL Azure. No need to add connection strings in the application itself, no need to configure caching in the application itself. Instead, I can configure these things in the Windows Azure AppFabric Application canvas and just consume them blindly in my code. Awesome!

    Here’s the code for my HomeController where I consume the cache/. Even my grandmother can write this!

    1 [HandleError] 2 public class HomeController : Controller 3 { 4 public ActionResult Index() 5 { 6 var count = 1; 7 var cache = ServiceReferences.CreateImport1(); 8 var countItem = cache.GetCacheItem("visits"); 9 if (countItem != null) 10 { 11 count = ((int)countItem.Value) + 1; 12 } 13 cache.Put("visits", count); 14 15 ViewData["Message"] = string.Format("You are visitor number {0}.", count); 16 17 return View(); 18 } 19 20 public ActionResult About() 21 { 22 return View(); 23 } 24 }

    Now let’s go back to the Windows Azure AppFabric Application canvas, where I can switch to “Deployment View”:

    Windows Azure AppFabric Application Deployment View

    Deployment View basically lets you decide in which container one or more applications will be running and how many scale units a container should span (see the properties window in Visual Studio for this).

    Right-clicking and selecting “Deploy…” deploys my Windows Azure AppFabric Application to the production environment.

    The management portal

    After logging in to http://portal.appfabriclabs.com, I can manage the application I just published:

    Windows Azure AppFabric Application Management Portal

    I’m not going to go in much detail but will highlight some features. The portal enables you to manage your application: deploy/undeploy, scale, monitor, change configuration, …  Basically everything you would expect to be able to do. And more! If you look at the monitoring part, for example, you will see some KPI’s on your application. Here’s what my sample application shows after being deployed for a few minutes:

    Windows Azure AppFabric Applications monitoring and latency

    Pretty slick. It even monitors average latencies etc.!

    Conclusion

    As you can read in this blog post, I’ve been exploring this product and trying out the basics of it. I’m no sure yet if this model will fit every application, but I’m sure a solution like this is where the future of PaaS should be: no longer caring about servers, VM’s or instances, just deploy and let the platform figure everything out. My business value is my application, not the fact that it spans 2 VM’s.

    Now when I say “future of PaaS”, I’m also a bit skeptical… Most customers I work with use COM, require startup scripts to configure the environment, care about the server their application runs on. In fact, some applications will never be able to be deployed on this solution because of that. Where Windows Azure already represents a major shift in terms of development paradigm (a too large step for many!), I thing the step to Windows Azure AppFabric Applications is a bridge too far for most people. At present.

    But then there’s corporations… As corporations always are 10 steps behind, I foresee that this will only become mainstream within the next 5-8 years (for enterpise). Too bad! I wish most corporate environments moved faster…

    If Microsoft wants this thing to succeed I think they need to work even more on shifting minds to the cloud paradigm and more specific to the PaaS paradigm. Perhaps Windows 8 can be a utility to do this: if Windows 8 shifts from “programming for a Windows environment” to “programming for a PaaS environment”, people will start following that direction. What the heck, maybe this is even a great model for Joe Average to create “apps” for Windows 8! Just like one submits an app to AppStore or Marketplace today, he/she can submit an app to “Windows Marketplace” which in the background just drops everything on a technology like Windows Azure AppFabric Applications?


    Delegate feed privileges to other users on MyGet

    MyGetOne of the first features we had envisioned for MyGet and which seemed increasingly popular was the ability to provide other users a means of managing packages on another user’s feed.

    As of today, we’re proud to announce the following new features:

    • Delegating feed privileges to other users – This allows you to make another MyGet user “co-admin” or “contributor” to a feed. This eases management of a private feed as that work can be spread across multiple people.
    • Making private feeds private by requiring authentication – It’s now possible to configure a feed so that nobody can consult its list of packages unless a valid login is provided. This feature is not yet available for use with NuGet 1.4.
    • Global deployment – We’ve updated our deployment so managing feeds can now be done on a server that’s closer to you.

    Now when is Microsoft going to buy us out :-)

    Delegating feed privileges to other users

    MyGet now allows you to make another MyGet user “co-admin” or “contributor” to a feed. This eases management of a private feed as that work can be spread across multiple people. If combined with the “private feeds” option, it’s also possible to give some users read access to the feed while unauthenticated users can not access the feed created.

    To delegate privileges to a user, navigate to the feed details and click the Feed security tab. This tab allows you to change feed privileges for different users. Adding feed privileges can be done by clicking the Add feed privileges… button (duh!).

    Add MyGet feed privileges

    Available privileges are:

    • Has no access to this feed (speaks for itself)
    • Can consume this feed (allows the user to use the feed in Visual Studio / NuGet)
    • Can manage packages for this feed (allows the user to add packages to the feed via the website and via the NuGet push API)
    • Can manage users and packages for this feed (extends the above with feed privilege management capabilities)

    After selecting the privileges, the user receives an e-mail in which he/she can claim the acquired privileges:

    Claim MyGet feed privileges

    Privileges are not granted per direct: after assigning privileges, the user has to claim these privileges by clicking a link in an automated e-mail that has been sent.

    Making private feeds private by requiring authentication

    It’s now possible to configure a feed so that nobody can consult its list of packages unless a valid login is provided. Combined with the feed privilege delegation feature one can granularly control who can and who can not consume a feed from MyGet. Note that his feature is not yet available for use with NuGet 1.4, we hope to see support for this shipping with NuGet 1.5.

    In order to enable this feature, on the Feed security tab change feed privileges for Everyone to Has no access to this feed.

    NuGet feed authentication

    This will instruct MyGet to request for basic authentication when someone accesses a MyGet feed. For example, try our sample feed: http://www.myget.org/F/mygetsample/

    Global deployment

    We’ve updated our deployment so managing feeds can now be done on a server that’s closer to you. Currently we have a deployment running in a European datacenter and one in the US. We hope to expand this further as well as leverage a content delivery network for high-speed distribution of packages.

    image

    We need your opinion!

    As features keep popping into our head, the time we have to work on MyGet in our spare time is not enough. To support some extra development, we are thinking along the lines of introducing a premium version which you can host in your own datacenter or on a dedicated cloud environment. We would love some feedback on the following survey:


    Enabling conditional Basic HTTP authentication on a WCF OData service

    imageYes, a long title, but also something I was not able to find too easily using Google. Here’s the situation: for MyGet, we are implementing basic authentication to the OData feed serving available NuGet packages. If you recall my post Using dynamic WCF service routes, you may have deducted that MyGet uses that technique to have one WCF OData service serving the feeds of all our users. It’s just convenient! Unless you want basic HTTP authentication for some feeds and not for others…

    After doing some research, I thought the easiest way to resolve this was to use WCF intercepting. Convenient, but how would you go about this? And moreover: how to make it extensible so we can use this for other WCF OData (or WebAPi) services in the future?

    The solution to this was to create a message inspector (IDispatchMessageInspector). Here’s the implementation we created for MyGet: (disclaimer: this will only work for OData services and WebApi!)

     

    1 public class ConditionalBasicAuthenticationMessageInspector : IDispatchMessageInspector 2 { 3 protected IBasicAuthenticationCondition Condition { get; private set; } 4 protected IBasicAuthenticationProvider Provider { get; private set; } 5 6 public ConditionalBasicAuthenticationMessageInspector( 7 IBasicAuthenticationCondition condition, IBasicAuthenticationProvider provider) 8 { 9 Condition = condition; 10 Provider = provider; 11 } 12 13 public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) 14 { 15 // Determine HttpContextBase 16 if (HttpContext.Current == null) 17 { 18 return null; 19 } 20 HttpContextBase httpContext = new HttpContextWrapper(HttpContext.Current); 21 22 // Is basic authentication required? 23 if (Condition.Evaluate(httpContext)) 24 { 25 // Extract credentials 26 string[] credentials = ExtractCredentials(request); 27 28 // Are credentials present? If so, is the user authenticated? 29 if (credentials.Length > 0 && Provider.Authenticate(httpContext, credentials[0], credentials[1])) 30 { 31 httpContext.User = new GenericPrincipal( 32 new GenericIdentity(credentials[0]), new string[] { }); 33 return null; 34 } 35 36 // Require authentication 37 HttpContext.Current.Response.StatusCode = 401; 38 HttpContext.Current.Response.StatusDescription = "Unauthorized"; 39 HttpContext.Current.Response.Headers.Add("WWW-Authenticate", string.Format("Basic realm=\"{0}\"", Provider.Realm)); 40 HttpContext.Current.Response.End(); 41 } 42 43 return null; 44 } 45 46 public void BeforeSendReply(ref Message reply, object correlationState) 47 { 48 // Noop 49 } 50 51 private string[] ExtractCredentials(Message requestMessage) 52 { 53 HttpRequestMessageProperty request = (HttpRequestMessageProperty)requestMessage.Properties[HttpRequestMessageProperty.Name]; 54 55 string authHeader = request.Headers["Authorization"]; 56 57 if (authHeader != null && authHeader.StartsWith("Basic")) 58 { 59 string encodedUserPass = authHeader.Substring(6).Trim(); 60 61 Encoding encoding = Encoding.GetEncoding("iso-8859-1"); 62 string userPass = encoding.GetString(Convert.FromBase64String(encodedUserPass)); 63 int separator = userPass.IndexOf(':'); 64 65 string[] credentials = new string[2]; 66 credentials[0] = userPass.Substring(0, separator); 67 credentials[1] = userPass.Substring(separator + 1); 68 69 return credentials; 70 } 71 72 return new string[] { }; 73 } 74 }

    Our ConditionalBasicAuthenticationMessageInspector implements a WCF message inspector that, once a request has been received, checks the HTTP authentication headers to check for a basic username/password. One extra there: since we wanted conditional authentication, we have also implemented an IBasicAuthenticationCondition interface which we have to implement. This interface decides whether to invoke authentication or not. The authentication itself is done by calling into our IBasicAuthenticationProvider. Implementations of these can be found on our CodePlex site.

    If you are getting optimistic: great! But how do you apply this message inspector to a WCF service? No worries: you can create a behavior for that. All you have to do is create a new Attribute and implement IServiceBehavior. In this implementation, you can register the ConditionalBasicAuthenticationMessageInspector on the service endpoint. Here’s the implementation:

    1 [AttributeUsage(AttributeTargets.Class)] 2 public class ConditionalBasicAuthenticationInspectionBehaviorAttribute 3 : Attribute, IServiceBehavior 4 { 5 protected IBasicAuthenticationCondition Condition { get; private set; } 6 protected IBasicAuthenticationProvider Provider { get; private set; } 7 8 public ConditionalBasicAuthenticationInspectionBehaviorAttribute( 9 IBasicAuthenticationCondition condition, IBasicAuthenticationProvider provider) 10 { 11 Condition = condition; 12 Provider = provider; 13 } 14 15 public ConditionalBasicAuthenticationInspectionBehaviorAttribute( 16 Type condition, Type provider) 17 { 18 Condition = Activator.CreateInstance(condition) as IBasicAuthenticationCondition; 19 Provider = Activator.CreateInstance(provider) as IBasicAuthenticationProvider; 20 } 21 22 public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) 23 { 24 // Noop 25 } 26 27 public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) 28 { 29 foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) 30 { 31 foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) 32 { 33 endpointDispatcher.DispatchRuntime.MessageInspectors.Add( 34 new ConditionalBasicAuthenticationMessageInspector(Condition, Provider)); 35 } 36 } 37 } 38 39 public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) 40 { 41 // Noop 42 } 43 }

    One step to do: apply this service behavior to our OData service. Easy! Just add an attribute to the service class and you’re done! Note that we specify the IBasicAuthenticationCondition and IBasicAuthenticationProvider on the attribute.

    1 [ConditionalBasicAuthenticationInspectionBehavior( 2 typeof(MyGetBasicAuthenticationCondition), 3 typeof(MyGetBasicAuthenticationProvider))] 4 public class PackageFeedHandler 5 : DataService<PackageEntities>, 6 IDataServiceStreamProvider, 7 IServiceProvider 8 { 9 }

    Enjoy!


    Categories: ASP.NET | C# | General | ICT | MVC | Projects | Software

    MyGet now supports pushing from the command line

    One of the work items we had opened for MyGet was the ability to push packages to a private feed from the command line. Only a few hours after our initial launch, David Fowler provided us with example code on how to implement NuGet command line pushes on the server side. An evening of coding later, I quickly hacked this into MyGet, which means that we now support pushing packages from the command line!

    For those that did not catch up with my blog post overload of the past week: MyGet offers you the possibility to create your own, private, filtered NuGet feed for use in the Visual Studio Package Manager.  It can contain packages from the official NuGet feed as well as your private packages, hosted on MyGet. Want a sample? Add this feed to your Visual Studio package manager: http://www.myget.org/F/chucknorris

    Pushing a package from the command line to MyGet

    The first thing you’ll be needing is an API key for your private feed. This can be obtained through the “Edit Feed” link, where you’ll see an API key listed as well as a button to regenerate the API key, just in case someone steals it from you while giving a demo of MyGet :-)

    image

    Once you have the API key, it can be stored into the NuGet executable’s settings by running the following command, including your private API key and your private feed URL:

    1 NuGet setApiKey c18673a2-7b57-4207-8b29-7bb57c04f070 -Source http://www.myget.org/F/testfeed

    After that, you can easily push a package to your private feed. The package will automatically show up on the website and your private feed. Do note that this can take a few minutes to propagate.

    1 NuGet push RouteMagic.0.2.2.2.nupkg -Source http://www.myget.org/F/testfeed

    More on the command line can be found on the NuGet documentation wiki.

    Other change: authentication to the website

    Someone on Twitter (@corydeppen) complained he had to login using Windows Live ID. Because we’re using the Windows Azure AppFabric Access Control Service (which I’ll abbreviate to ACS next time), this was in fact a no-brainer. We now support Windows Live ID, Google, Yahoo! and Facebook as authentication mechanisms for MyGet. Enjoy!


    Categories: ASP.NET | Windows Azure | C# | General | ICT | MEF | MVC | Projects | Software