Maarten Balliauw {blog}

ASP.NET, ASP.NET MVC, Windows Azure, PHP, ...

NAVIGATION - SEARCH

Don’t brag about your Visual Studio achievements! (yet?)

imageThe Channel 9 folks seem to have released the first beta of their Visual Studio Achievements project. The idea of Visual Studio Achievements is pretty awesome:

Bring Some Game To Your Code!

A software engineer’s glory so often goes unnoticed. Attention seems to come either when there are bugs or when the final project ships. But rarely is a developer appreciated for all the nuances and subtleties of a piece of code–and all the heroics it took to write it. With Visual Studio Achievements Beta, your talents are recognized as you perform various coding feats, unlock achievements and earn badges.

Find the announcement here and the beta from the Visual Studio Gallery here.

The bad

The idea behind Visual Studio Achievements is awesome! Unfortunately, the current achievements series is pure crap and will get you into trouble. A simple example:

Regional Manager (7 points)

Add 10 regions to a class. Your code is so readable, if I only didn't have to keep collapsing and expanding!

Are they serious? 10 regions in a class means bad code design. It should crash your Visual Studio and only allow you to restart it if you swear you’ll read a book on modern OO design.

Another example:

Job Security (0 points)

Write 20 single letter class level variables in one file. Kudos to you for being cryptic! Uses FxCop

While I’m sure this one is meant to be sarcastic (hence the 0 points), it makes people write unreadable code.

There’s a number of bad coding habits in the list of achievements. And I really hope no-one on my team ever “achieves” some items on that list. If they do, I’m pretty sure that project is doomed.

The good

The good thing is: there are some positive achievements. For example, stimulating people to organize usings. Or to try out some extensions. Unfortunately, there are almost no “good” achievements. What I would like to see is a bunch more extensions that make it fun to discover new features in Visual Studio or learn about good coding habits.

Don’t get me wrong: I do like the idea of achievements very much. In fact, I feel an urge to have the Go To Hell achievement (and delete the code afterwards, promise!), but why not use them to teach people to be better at coding or be more productive? How about achievements that stimulate people to use CTRL + , which a lot of people don’t know about. Or teach people to write a unit test. Heck, you can even become Disposable by correctly implementing IDisposable!

So in conclusion: your resume will look very bad if you are a Regional Manager or gained the Turtles All The Way Down achievement. Don’t brag about those. Come up with some good habits that can be rewarded with achievements and please, ask the Channel 9 guys to include those.

[edit]This one does have positive achievements: https://github.com/jonasswiatek/strokes [/edit]
[edit]http://channel9.msdn.com/niners/maartenba/achievements/visualstudio/GotoAchievement [/edit]

How do you synchronize a million to-do lists?

Not this question, but a similar one, has been asked by one of our customers. An interesting question, isn’t it? Wait. It gets more interesting. I’ll sketch a fake scenario that’s similar to our customer’s question. Imagine you are building mobile applications to manage a simple to-do list. This software is available on Android, iPhone, iPad, Windows Phone 7 and via a web browser. One day, the decision to share to-do lists has been made. Me and my wife should be able to share one to-do list between us, having an up-to-date version of the list on every device we grant access to this to-do list. Now imagine there are a million of those groups, where every partner in the sync relationship has the latest version of the list on his device. In often a disconnected world.

How would you solve this?

My take: Windows Azure Service Bus Topics & Subscriptions

According to the Windows Azure Service Bus product description, it “implements a publish/subscribe pattern that delivers a highly scalable, flexible, and cost-effective way to publish messages from an application and deliver them to multiple subscribers.“ Interesting. I’m not going into the specifics of it (maybe in a next post), but the Windows Azure Service Bus gave me an idea: why not put all actions (add an item, complete a to-do) on a queue, tagged with the appropriate “group” metadata? Here’s the producer side:

Windows Azure Service Bus Topics

On the consumer side, our devices are listening as well. Every device creates its subscription on the service bus topic. These subscriptions are named per device and filtered on the SyncGroup metadata. The Windows Azure Service Bus will take care of duplicating messages to every subscription as well as keeping track of messages that have not been processed: if I’m offline, messages are queued. If I’m online, I receive messages targeted at my device:

Windows Azure Service Bus Subscritpions

The only limitation to this is keeping the number of topics & subscriptions below the limits of Windows Azure Service Bus. But even then: if I just make sure every sync group is on the same bus, I can scale out over multiple service buses.

How would you solve the problem sketched? Comments are very welcomed!

Using SignalR to broadcast a slide deck

imageLast week, I’ve discussed Techniques for real-time client-server communication on the web (SignalR to the rescue). We’ve seen that when building web applications, you often face the fact that HTTP, the foundation of the web, is a request/response protocol. A client issues a request, a server handles this request and sends back a response. All the time, with no relation between the first request and subsequent requests. Also, since it’s request-based, there is no way to send messages from the server to the client without having the client create a request first.

We’ve had a look at how to tackle this problem: using Ajax polling, Long polling an WebSockets. the conclusion was that each of these solutions has it pros and cons. SignalR, an open source project led by some Microsoft developers, is an ASP.NET library which leverages the three techniques described before to create a seamless experience between client and server.

The main idea of SignalR is that the boundary between client and server should become easier to tackle. In this blog post, I will go deeper into how you can use SignalR to achieve real-time communication between client and server over HTTP.

Meet DeckCast

DeckCast is a small sample which I’ve created for this blog post series on SignalR. You can download the source code here: DeckCast.zip (291.58 kb)

The general idea of DeckCast is that a presenter can navigate to http://example.org/Deck/Present/SomeDeck and he can navigate through the slide deck with the arrow keys on his keyboard. One or more clients can then navigate to http://example.org/Deck/View/SomeDeck and view the “presentation”. Note the idea is not to create something you can use to present slide decks over HTTP, instead I’m just showing how to use SignalR to communicate between client and server.

The presenter and viewers will navigate to their URLs:SignalR presentation JavaScript Slide

The presenter will then navigate to a different slide using the arrow keys on his keyboard. All viewers will automatically navigate to the exact same slide at the same time. How much more real-time can it get?

SignalR presentation JavaScript Slide

And just for fun… Wouldn’t it be great if one of these clients was a console application?

SignalR.Client console application

Try doing this today on the web. Chances are you’ll get stuck in a maze of HTML, JavaScript and insanity. We’ll use SignalR to achieve this in a non-complex way, as well as Deck.JS to create nice HTML 5 slides without me having to code too much. Fair deal.

Connections and Hubs

SignalR is built around Connections and Hubs. A Connection is a persistent connection between a client and the server. It represents a communication channel between client and server (and vice-versa, of course). Hubs on the other hand offer an additional abstraction layer and can be used to provide “multicast” connections where multiple clients communicate with the Hub and the Hub can distribute data back to just one specific client or to a group of clients. Or to all, for that matter.

DeckCast would be a perfect example to create a SignalR Hub: clients will connect to the Hub, select the slide deck they want to view and receive slide contents and navigational actions for just that slide deck. The Hub will know which clients are viewing which slide deck and communicate changes and movements to only the intended group of clients.

One additional note: a Connection or a Hub are “transport agnostic”. This means that the SignalR framework will decide which transport is best for each client and server combination. Ajax polling? Long polling? Websockets? A hidden iframe? You don’t have to care about that. The raw connection details are abstracted by SignalR and you get to work with a nice Connection or Hub class. Which we’ll do: let’s create the server side using a Hub.

Creating the server side

First of all: make sure you have NuGet installed and install the SignalR package. Install-Package SignalR will bring down two additional packages: SignalR.Server, the server implementation, and SignalR.Js, the JavaScript libraries for communicating with the server. If your server implementation will not host the JavaScript client as well, installing SignalR.Server will do as well.

We will make use of a SignalR Hub to distribute data between clients and server. A Hub has a type (the class that you create for it), a name (which is, by default, the same as the type) and inherits SignalR’s Hub class. The wireframe for our presentation Hub would look like this:

1 [HubName("presentation")] 2 public class PresentationHub 3 : Hub 4 { 5 }

SignalR’s Hub class will do the heavy lifting for us. It will expose all public methods to any client connecting to the Hub, whether it’s a JavaScript, Console or Silverlight or even Windows Phone client. I see 4 methods for the PresentationHub class: Join, GotoSlide, GotoCurrentSlide and GetDeckContents. The first one serves as the starting point to identify a client is watching a specific slide deck. GotoSlide will be used by the presenter to navigate to slide 0, 1, 2 and so on. GotoCurrentSlide is one used by the viewer to go to the current slide selected by the presenter. GetDeckContents is one which returns the presentation structure to the client: the Title, all slides and their bullets. Let’s translate that to some C#:

1 [HubName("presentation")] 2 public class PresentationHub 3 : Hub 4 { 5 static ConcurrentDictionary<string, int> CurrentSlide { get; set; } 6 7 public void Join(string deckId) 8 { 9 } 10 11 public void GotoSlide(int slideId) 12 { 13 } 14 15 public void GotoCurrentSlide() 16 { 17 } 18 19 public Deck GetDeckContents(string id) 20 { 21 } 22 }

The concurrent dictionary will be used to store which slide is currently being viewed for a specific presentation. All the rest are just standard C# methods. Let’s implement them.

1 [HubName("presentation")] 2 public class PresentationHub 3 : Hub 4 { 5 static ConcurrentDictionary<string, int> DeckLocation { get; set; } 6 7 static PresentationHub() 8 { 9 DeckLocation = new ConcurrentDictionary<string, int>(); 10 } 11 12 public void Join(string deckId) 13 { 14 Caller.DeckId = deckId; 15 16 AddToGroup(deckId); 17 } 18 19 public void GotoSlide(int slideId) 20 { 21 string deckId = Caller.DeckId; 22 DeckLocation.AddOrUpdate(deckId, (k) => slideId, (k, v) => slideId); 23 24 Clients[deckId].showSlide(slideId); 25 } 26 27 public void GotoCurrentSlide() 28 { 29 int slideId = 0; 30 DeckLocation.TryGetValue(Caller.DeckId, out slideId); 31 Caller.showSlide(slideId); 32 } 33 34 public Deck GetDeckContents(string id) 35 { 36 var deckRepository = new DeckRepository(); 37 return deckRepository.GetDeck(id); 38 } 39 }

The code should be pretty straightforward, although there are some things I would like to mention about SignalR Hubs:

  • The Join method does two things: it sets a property on the client calling into this method. That’s right: the server tells the client to set a specific property at the client side. It also adds the client to a group identified by the slide deck id. Reason for this is that we want to group clients based on the slide deck id so that we can broadcast to a group of clients instead of having to broadcast messages to all.
  • The GotoSlide method will be called by the presenter to advance the slide deck. It calls into Clients[deckId]. Remember the grouping we just did? Well, this method returns me the group of clients viewing slide deck deckId. We’re calling the method showSlide n his group. showSlide? That’s a method we will define on the client side! Again, the server is calling the client(s) here.
  • GotoCurrentSlide calls into one client’s showSlide method.
  • GetDeckContents just fetches the Deck from a database and returns the complete object tree to the client.

Let’s continue with the web client side!

Creating the web client side

Assuming you have installed SignalR.JS and that you have already referenced jQuery, add the following two script references to your view:

1 <script type="text/javascript" src="@Url.Content("~/Scripts/jquery.signalR.min.js")"></script> 2 <script type="text/javascript" src="/signalr/hubs"></script>

That first reference is just the SignalR client library. The second reference is a reference to SignalR’s metadata endpoint: it contains information about available Hubs on the server side. The client library will use that metadata to connect to a Hub and maintain the persistent connection between client and server.

The viewer of a presentation will then have to connect to the Hub on the server side. This is probably the easiest piece of code you’ve ever seen (next to Hello World):

1 <script type="text/javascript"> 2 $(function () { 3 // SignalR hub initialization 4 var presentation = $.connection.presentation; 5 $.connection.hub.start(); 6 }); 7 </script>

We’ve just established a connection with the PresentationHub on the server side. We also start the hub connection (one call, even if you are connecting to multiple hubs at once). Of course, the code above will not do a lot. Let’s add some more body.

1 <script type="text/javascript"> 2 $(function () { 3 // SignalR hub initialization 4 var presentation = $.connection.presentation; 5 presentation.showSlide = function (slideId) { 6 $.deck('go', slideId); 7 }; 8 }); 9 </script>

Remember the showSlide method we were calling from the server? This is the one. We’re allowing SignalR to call into the JavaScript method showSlide from the server side. This will call into Deck.JS and advance the presentation. The only thing left to do is tell the server which slide deck we are interested in. We do this immediately after the connection to the hub has been established using a callback function:

1 <script type="text/javascript"> 2 $(function () { 3 // SignalR hub initialization 4 var presentation = $.connection.presentation; 5 presentation.showSlide = function (slideId) { 6 $.deck('go', slideId); 7 }; 8 $.connection.hub.start(function () { 9 // Deck initialization 10 $.extend(true, $.deck.defaults, { 11 keys: { 12 next: 0, 13 previous: 0, 14 goto: 0 15 } 16 }); 17 $.deck('.slide'); 18 19 // Join presentation 20 presentation.join('@Model.DeckId', function () { 21 presentation.gotoCurrentSlide(); 22 }); 23 }); 24 }); 25 </script>

Cool, no? The presenter side of things is very similar, except that it also calls into the server’s GotoSlide method.

Let’s see if we can convert this JavaScript code into come C#as well.  I promised to add a Console application to the mix to show you SignalR is not just about web. It’s also about desktop apps, Silverlight apps and Windows Phone apps. And since it’s open source, I also expect someone to contribute client libraries for Android or iPhone. David, if you are reading this: you have work to do ;-)

Creating the console client side

Install the NuGet package SignalR.Client. This will add the required client library for the console application. If you’re creating a Windows Phone client, SignalR.WP71 will do (Mango).

The first thing to do would be connecting to SignalR’s Hub metadata and creating a proxy for the presentation Hub. Note that I am using the root URL this time (which is enough) and the full type name for the Hub (important!).

1 var connection = new HubConnection("http://localhost:56285/"); 2 var presentationHub = connection.CreateProxy("DeckCast.Hubs.PresentationHub"); 3 dynamic presentation = presentationHub;

Note that I’m also using a dynamic representation of this proxy. It may facilitate the rest of my code later.

Next up: connecting our client to the server. SignalR makes extensive use of the Task Parallel Library (TPL) and there’s no escaping there for you. Start the connection and if something fails, let’s show the client:

1 connection.Start() 2 .ContinueWith(task => 3 { 4 if (task.IsFaulted) 5 { 6 System.Console.ForegroundColor = ConsoleColor.Red; 7 System.Console.WriteLine("There was an error connecting to the DeckCast presentation hub."); 8 } 9 });

Just like with the JavaScript client, we have to join the presentation of our choice. Again, the TPL is used here but I’m explicitly telling it to Wait for the result before continuing my code.

1 presentationHub.Invoke("Join", deckId).Wait();

Because the console does not have any notion about the Deck class and its Slide objects, let’s fetch the slide contents into a dynamic object again. Here’s how:

1 var getDeckContents = presentationHub.Invoke<dynamic>("GetDeckContents", deckId); 2 getDeckContents.Wait(); 3 4 var deck = getDeckContents.Result;

We also want to respond to the showSlide method. Since there’s no means of defining that method on the client in C#in the same fashion as we did it on the JavaScript side, let’s simply use the On method exposed by the hub proxy. It subscribes to a server-side event (such as showSlide) and takes action whenever that occurs. Here’s the code:

1 presentationHub.On<int>("showSlide", slideId => 2 { 3 System.Console.Clear(); 4 System.Console.ForegroundColor = ConsoleColor.White; 5 System.Console.WriteLine(""); 6 System.Console.WriteLine(deck.Slides[slideId].Title); 7 System.Console.WriteLine(""); 8 9 if (deck.Slides[slideId].Bullets != null) 10 { 11 foreach (var bullet in deck.Slides[slideId].Bullets) 12 { 13 System.Console.WriteLine(" * {0}", bullet); 14 System.Console.WriteLine(""); 15 } 16 } 17 18 if (deck.Slides[slideId].Quote != null) 19 { 20 System.Console.WriteLine(" \"{0}\"", deck.Slides[slideId].Quote); 21 } 22 });

We also want to move to the current slide:

1 presentationHub.Invoke("GotoCurrentSlide").Wait();

There we go. The presenter can now switch between slides and all clients, both web and console will be informed and updated accordingly.

SignalR Console Client

Conclusion

SignalR offers a relatively easy to use abstraction over various bidirectional connection paradigms introduced on the web. The fact that its open source and features clients for JavaScript, .NET, Silverlight and Windows Phone in my opinion makes it a viable alternative for applications where you typically use polling or a bidirectional WCF communication channel. Even WCF RIA Services should be a little bit afraid of SignalR as it’s lean and mean!

[edit] There's the Objective-C client: https://github.com/DyKnow/SignalR-ObjC

The sample code for this blog post can be downloaded here: DeckCast.zip (291.58 kb)

Techniques for real-time client-server communication on the web (SignalR to the rescue)

SignalR websockets html5 long pollingWhen building web applications, you often face the fact that HTTP, the foundation of the web, is a request/response protocol. A client issues a request, a server handles this request and sends back a response. All the time, with no relation between the first request and subsequent requests. Also, since it’s request-based, there is no way to send messages from the server to the client without having the client create a request first.

Today users expect that in their projects, sorry, “experiences”, a form of “real time” is available. Questions like “I want this stock ticker to update whenever the price changes” or “I want to view real-time GPS locations of my vehicles on this map”. Or even better: experiences where people collaborate often require live notifications and changes in the browser so that whenever a user triggers a task or event, the other users collaborating immediately are notified. Think Google Spreadsheets where you can work together. Think Facebook chat. Think Twitter where new messages automatically appear. Think your apps with a sprinkle of real-time sauce.

How would you implement this?

But what if the server wants to communicate with the client?

Over the years, web developers have been very inventive working around the request/response nature of the web. Two techniques are being used on different platforms and provide a relatively easy workaround to the “problem” of HTTP’s paradigm where the client initiates any connection: simple polling using Ajax and a variant of that, long polling.

Simple Ajax polling is, well, simple: the client “polls” the server via an Ajax request the server answers if there is data. The client waits for a while and goes through this process again. Schematically, this would be the following:

image

A problem with this is that the server still relies on the client initiating the connection. Whenever during the polling interval the server has data for the client, this data can only be sent to the client when the next polling cycle occurs. This is probably no problem when the polling interval is in the 100ms range, apart from the fact that you’ll be hammering your servers when doing that. From the moment your polling interval goes up to say 2 seconds or even 30 seconds, you lose part of the “real time” communication idea.

This problem has been solved by using a technique called “long polling”. The idea is that the client opens an Ajax-based connection to the server, the server does not reply until it has data. The client just has the false feeling that the request is taking a while, and eventually will have some data coming back from the server. Whenever data is returned, the client immediately opens up a “long polling” connection again. Schematically:

image

There’s no polling interval: as long as the connection is open, the server has the ability to send data back to the client. Awesome, right? Not really… Your servers will not be hammered with billions of requests, but the requests it handles will take a while to complete (the “long poll”). Imagine what your ASP.NET thread pool will do in such case… Well, unless you implement your server-side using an IAsyncHttpHandler or similar. Otherwise, your servers will simply stop accepting requests.

HTML5 to the rescue?

As we’ve seen, both techniques that exist work to simulate full-duplex communication between client and server. However, both of them have some disadvantages if you don’t know what you are doing. Also, the techniques described before are just simulating bi-directional communication. Wouldn’t it be nice to have a solution that works great and was intended to do this? Meet HTML5 WebSockets.

WebSockets offer a real, bi-directional TCP connection between the client and the server. That’s right, a TCP (non-HTTP) connection. To establish a WebSocket connection, the client sends a WebSocket handshake request over HTTP, the server sends a WebSocket handshake response with details on how to open the actual TCP connection. Easy as that! Schematically:

image

Unfortunately, the web does not evolve that fast… WebSockets are still a draft specification (“CTP” or “alpha” as you will). Not all browsers support them. And because they are using a raw TCP connection, many proxy servers used at companies don’t support them, yet. We’ll get there, just not today. Also, if you want to use WebSockets with ASP.NET, you’ll be forced to use the preview release of .NET 4.5.

So what to do in this messed up world? How to achieve real-time, bi-directional communication over the interwebs, today?

SignalR to the rescue!

SignalR is “an asynchronous signaling library for ASP.NET that Microsoft is working on to help build real-time multi-user web applications”.  Let me rephrase that: SignalR is an ASP.NET library which leverages the three techniques I described before to create a seamless experience between client and server.

The main idea of SignalR is that the boundary between client and server should become easier to tackle. A really quick example would be two parts of code. The client side:

1 var helloConnection = $.connection.hello; 2 3 helloConnection.sayHelloToMe = function (message) { 4 alert(message); 5 }; 6 7 $.connection.hub.start(function() { 8 helloConnection.sayHelloToAll("Hello all!"); 9 });

The server side:

1 public class Hello : Hub { 2 public void SayHelloToAll(string message) { 3 Clients.sayHelloToMe(message); 4 } 5 }

Are you already seeing the link? The JavaScript client calls the C# method “SayHelloToAll” as if it were a JavaScript function. The C# side calls all of its clients (meaning the 200.000 browser windows connecting to this service :-)) JavaScript method “sayHelloToMe” as if it were a C# method.

If I add that not only JavaScript clients are supported but also Windows Phone, Silverlight and plain .NET, does this sound of interest? If I add that SignalR can use any of the three techniques described earlier in this post based on what the client and the server support, without you even having to care… does this sound of interest? If the answer is yes, stay tuned for some follow up posts…

Repaving your PC: the easier way

It"’s been a while since I had to repave my laptop. I have a Windows Home Server (WHS) at home which images my PC almost daily and allows restoring it to a given point in time in less than 30 minutes. Which is awesome! And which is how I usually “restore” my PC into a stable state.  Over the past year some hardware changes have been made of which the most noteworthy is the replacement of the existing hard drive with an SSD. A great addition, and it was easy to restore as well: swap the disks and restore the image from WHS. SSD and full system install? 30 minutes.

imageThe downside of restoring an image which came from a non-SSD drive has been bugging me for a while though. My SSD did not feel as fast as it should have felt, resulting in me reinstalling Windows on it just to check if that led to any speed improvements. And it did. And I knew I was in trouble: that would be a load of software to re-install and reconfigure. Here’s a list of what I had on my system before and is absolutely required for me to be able to do my job:

  • Telnet client
  • PDFCreator
  • ZoomIt
  • Win7 SP1
  • Virtual CloneDrive
  • HP Printer Corporate Edition
  • Ccleaner
  • Virus scanner
  • Adobe Flash
  • Adobe PDF
  • Silverlight
  • Office 2010
  • Windows Live Writer
  • Windows Live Mesh
  • WinRAR
  • Office Live Meeting & Communicator
  • VS 2010
  • VS 2010 SP1
  • GhostDoc
  • Resharper
  • Windows Azure Tools
  • WIF tools
  • MVC 3 tools
  • SQL Express R2
  • SQL Express Management Tools
  • Webmatrix
  • IIS Express
  • Firefox
  • Chrome
  • Notepad++
  • NuGet Package Explorer
  • Paint.net
  • Skype
  • TortoiseHg
  • TortoiseSVN
  • Fiddler2
  • Java (sorry :-))
  • Zune

Oh boy… Knowing how '”fast” some of these can be installed, that would cost me a day of clicking and waiting.

[edit]Also checkout https://github.com/chocolatey/chocolatey/issues/46[/edit]

Three tools can save you a lot of that work

Fortunately, we live in this time of computers. A time where some things can be automated and it seems like a PC repave should be relatively easy to do. There are three tools that will save you time:

  • Ninite, which you can find at www.ninite.com. Ninite allows you to download and install some items of the list above in one go. I’ve packaged Flash, Acrobat Reader, Chrome, Firefox, Java, … using Ninite and was able to install these items in one go. Great!
  • Web Platform Installer (Web PI) – command line version. A small executable which is able to pull a lot of software from Microsoft and install it in one go. Things like .NET 4, Silverlight, the ASP.NET MVC 3 tooling, … are all on the Web PI feed and can be downloaded in one go.
  • Chocolatey, available at www.chocolatey.org. Chocolatey is a tool based on NuGet which uses a feed of known software and can install these from the command line. For example, “cinst notepadplusplus” is enough to get NotePad++ running on your system.

Using these three tools, I have created a script which you have to run in a PowerShell administrative console. The scripts consist of calls to the Web PI, Ninite and Chocolatey. I’ll give you an example:

1 # Windows Installer 2 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:WindowsInstaller31" 3 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:WindowsInstaller45" 4 5 # Powershell 6 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:PowerShell" 7 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:PowerShell2" 8 9 # .NET 10 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:NETFramework20SP2" 11 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:NETFramework35" 12 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:NETFramework4" 13 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:JUNEAUNETFX4" 14 15 # Ninite stuff 16 cmd /C "ninite\ninite.exe" 17 18 # Chocolatey stuff 19 iex ((new-object net.webclient).DownloadString("http://bit.ly/psChocInstall")) 20 21 cinst windowstelnet 22 cinst virtualclonedrive 23 cinst sysinternals 24 cinst notepadplusplus 25 cinst adobereader 26 cinst msysgit 27 cinst fiddler 28 cinst filezilla 29 cinst skype 30 cinst paint.net 31 cinst ccleaner 32 cinst tortoisesvn 33 cinst tortoisehg 34 35 # IIS 36 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:IIS7" 37 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:ASPNET" 38 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:BasicAuthentication" 39 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:DefaultDocument" 40 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:DigestAuthentication" 41 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:DirectoryBrowse" 42 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:HTTPErrors" 43 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:HTTPLogging" 44 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:HTTPRedirection" 45 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:IIS7_ExtensionLessURLs" 46 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:IISManagementConsole" 47 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:IPSecurity" 48 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:ISAPIExtensions" 49 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:ISAPIFilters" 50 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:LoggingTools" 51 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:MetabaseAndIIS6Compatibility" 52 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:NETExtensibility" 53 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:RequestFiltering" 54 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:RequestMonitor" 55 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:StaticContent" 56 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:StaticContentCompression" 57 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:Tracing" 58 cmd /C "webpicmdline\webpicmdline.exe /AcceptEula /SuppressReboot /Products:WindowsAuthentication"

For those interested, here’s the set of scripts I have used: Repave.zip (986.66 kb). These contain a number of commands that use the tools mentioned above to do 75% of the install work on my PC. All I had to do was install Office 2010, VS2010 and my scripts did the rest. Not the holy grail yet, but certainly a big relief of a lot of frustration finding software and clicking next-next-finish. And now my PC has been repaved, it’s time for a WHS image again. Enjoy!

Publishing symbol packages for a MyGet feed

MyGet host your NuGet feed serverEver since NuGet 1.2, there is a great way for NuGet package authors to let their users debug into the package’s binaries. With almost no additional effort, package authors can publish their symbols and sources, and package consumers can debug into them from Visual Studio, simply by pushing a symbols package in addition to the standard NuGet package.

SymbolSourceToday, we’re proud to announce MyGet has partnered with SymbolSource.org to offer an easy workflow to publish symbol packages for a private MyGet feed. This means from now on you can publish symbol packages for your private feeds as well!

On a sidenote: we're sharing API keys between both services. If you also want to share the same password with both services, simply go to your MyGet profile page and re-enter your password. We'll keep it in sync after that.

Publishing a symbols package for use with MyGet

As I will assume you are used to publishing packages to NuGet and SymbolSource, here’s what changes. First of all, you will require the URLs to which to publish. Log in to MyGet and browse to your feed details. The Feed Details tab will give you all the information you need, as you can see in the following screenshot:

image

In short, your feed URL remains the same. If you want to consume your private feed in Visual Studio or using the NuGet Package Manager Console, simply add http://www.myget.org/F/yourfeedname as the source. The thing that changed is the publish URL: if you want to publish your packages to MyGet, use the URL http://www.myget.org/F/yourfeedname/api/v1 as the publish URL. For symbol packages, your URL will be in the form of http://nuget.gw.symbolsource.org/MyGet/yourfeedname.

The publish workflow to publish the SamplePackage.1.0.0.nupkg to a MyGet feed, including symbols, would be issuing the following two commands from the console:

1 nuget push SamplePackage.1.0.0.nupkg 00000000-0000-0000-0000-00000000000 -Source http://www.myget.org/F/somefeed/api/v1 2 3 nuget push SamplePackage.1.0.0.Symbols.nupkg 00000000-0000-0000-0000-00000000000 -Source http://nuget.gw.symbolsource.org/MyGet/somefeed

An example of these commands can also be found on the Feed Details tab for your MyGet feed.

Consuming symbol packages in Visual Studio

When logging in to MyGet, you can find the symbols URL compatible with Visual Studio under the Feed Details tab for your MyGet feed. This URL will be the same for all feeds you are allowed to consume, so no need to configure 10+ symbol servers in Visual Studio. Here’s how to configure it.

First of all, Visual Studio typically will only debug your own source code, the source code of the project or projects that are currently opened in Visual Studio. To disable this behavior and to instruct Visual Studio to also try to debug code other than the projects that are currently opened, open the Options dialog (under the menu Tools > Options). Find the Debugging node on the left and click the General node underneath. Turn off the option Enable Just My Code. Also turn on the option Enable source server support. This usually triggers a warning message but it is safe to just click Yes and continue with the settings specified.

MyGet symbol server in Visual Studio

Keep the Options dialog opened and find the Symbols node under the Debugging node on the left. In the dialog shown in Figure 4-14, add the symbol server URL for your MyGet feed: http://srv.symbolsource.org/pdb/MyGet/username/11111111-1111-1111-1111-11111111111. After that, click OK to confirm configuration changes and consume symbols for NuGet packages.

Enjoy!

Rewriting WCF OData Services base URL with load balancing & reverse proxy

When scaling out an application to multiple servers, often a form of load balancing or reverse proxying is used to provide external users access to a web server. For example, one can be in the situation where two servers are hosting a WCF OData Service and are exposed to the Internet through either a load balancer or a reverse proxy. Below is a figure of such setup using a reverse proxy.

WCF OData Services hosted in reverse proxy

As you can see, the external server listens on the URL www.example.com, while both internal servers are listening on their respective host names. Guess what: whenever someone accesses a WCF OData Service through the reverse proxy, the XML generated by one of the two backend servers is slightly invalid:

OData base URL invalid incorrect

While valid XML, the hostname provided to all our clients is wrong. The host name of the backend machine is in there and not the hostname of the reverse proxy URL…

How can this be solved? There are a couple of answers to that, one that popped into our minds was to rewrite the XML on the reverse proxy and simply “string.Replace” the invalid URLs. This will probably work, but it feels… dirty. We chose to create WCF inspector, which simply changes this at the WCF level on each backend node.

Our inspector looks like this: (note I did some hardcoding of the base hostname in here, which obviously should not be done in your code)

1 public class RewriteBaseUrlMessageInspector 2 : IDispatchMessageInspector 3 { 4 public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) 5 { 6 if (WebOperationContext.Current != null && WebOperationContext.Current.IncomingRequest.UriTemplateMatch != null) 7 { 8 UriBuilder baseUriBuilder = new UriBuilder(WebOperationContext.Current.IncomingRequest.UriTemplateMatch.BaseUri); 9 UriBuilder requestUriBuilder = new UriBuilder(WebOperationContext.Current.IncomingRequest.UriTemplateMatch.RequestUri); 10 11 baseUriBuilder.Host = "www.example.com"; 12 requestUriBuilder.Host = baseUriBuilder.Host; 13 14 OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRootUri"] = baseUriBuilder.Uri; 15 OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = requestUriBuilder.Uri; 16 } 17 18 return null; 19 } 20 21 public void BeforeSendReply(ref Message reply, object correlationState) 22 { 23 // Noop 24 } 25 }

There’s not much rocket science in there, although some noteworthy actions are being performed:

  • The current WebOperationContext is queried for the full incoming request URI as well as the base URI. These values are based on the local server, in our example “srvweb01” and “srvweb02”.
  • The Host part of that URI is being replaced with the external hostname, www.example.com
  • These two values are stored in the current OperationContext’s IncomingMessageProperties. Apparently the keys MicrosoftDataServicesRootUri and MicrosoftDataServicesRequestUri affect the URL being generated in the XML feed

To apply this inspector to our WCF OData Service, we’ve created a behavior and applied the inspector to our service channel. Here’s the code for that:

1 [AttributeUsage(AttributeTargets.Class)] 2 public class RewriteBaseUrlBehavior 3 : Attribute, IServiceBehavior 4 { 5 public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) 6 { 7 // Noop 8 } 9 10 public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) 11 { 12 // Noop 13 } 14 15 public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) 16 { 17 foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) 18 { 19 foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) 20 { 21 endpointDispatcher.DispatchRuntime.MessageInspectors.Add( 22 new RewriteBaseUrlMessageInspector()); 23 } 24 } 25 } 26 }

This behavior simply loops all channel dispatchers and their endpoints and applies our inspector to them.

Finally, there’s nothing left to do to fix our reverse proxy issue than to just annotate our WCF OData Service with this behavior attribute:

1 [RewriteBaseUrlBehavior] 2 public class PackageFeedHandler 3 : DataService<PackageEntities> 4 { 5 // ... 6 }

Working with URL routing

A while ago, I posted about Using dynamic WCF service routes. The technique described below is also appropriate for services created using that technique. When working with that implementation, the source code for the inspector would be slightly different.

1 public class RewriteBaseUrlMessageInspector 2 : IDispatchMessageInspector 3 { 4 public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) 5 { 6 if (WebOperationContext.Current != null && WebOperationContext.Current.IncomingRequest.UriTemplateMatch != null) 7 { 8 UriBuilder baseUriBuilder = new UriBuilder(WebOperationContext.Current.IncomingRequest.UriTemplateMatch.BaseUri); 9 UriBuilder requestUriBuilder = new UriBuilder(WebOperationContext.Current.IncomingRequest.UriTemplateMatch.RequestUri); 10 11 var routeData = MyGet.Server.Routing.DynamicServiceRoute.GetCurrentRouteData(); 12 var route = routeData.Route as Route; 13 if (route != null) 14 { 15 string servicePath = route.Url; 16 servicePath = Regex.Replace(servicePath, @"({\*.*})", ""); // strip out catch-all 17 foreach (var routeValue in routeData.Values) 18 { 19 if (routeValue.Value != null) 20 { 21 servicePath = servicePath.Replace("{" + routeValue.Key + "}", routeValue.Value.ToString()); 22 } 23 } 24 25 if (!servicePath.StartsWith("/")) 26 { 27 servicePath = "/" + servicePath; 28 } 29 30 if (!servicePath.EndsWith("/")) 31 { 32 servicePath = servicePath + "/"; 33 } 34 35 requestUriBuilder.Path = requestUriBuilder.Path.Replace(baseUriBuilder.Path, servicePath); 36 requestUriBuilder.Host = baseUriBuilder.Host; 37 baseUriBuilder.Path = servicePath; 38 } 39 40 OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRootUri"] = baseUriBuilder.Uri; 41 OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = requestUriBuilder.Uri; 42 } 43 44 return null; 45 } 46 47 public void BeforeSendReply(ref Message reply, object correlationState) 48 { 49 // Noop 50 } 51 }

The idea is identical, except that we’re updating the incoming URL path for reasons described in the aforementioned blog post.

Enjoy!

Running Memcached on Windows Azure for PHP

After three conferences in two weeks with a lot of “airport time”, which typically converts into “let’s code!” time, I think I may have tackled a commonly requested Windows Azure feature for PHP developers. Some sort of distributed caching is always a great thing to have when building scalable services and applications. While Windows Azure offers a distributed caching layer under the form of the Windows Azure Caching, that components currently lacks support for non-.NET technologies. I’ve heard there’s work being done there, but that’s not very interesting if you are building your app today. This blog post will show you how to modify a Windows Azure deployment to run and use Memcached in the easiest possible manner.

Note: this post focuses on PHP but can also be used to setup Memcached on Windows Azure for NodeJS, Java, Ruby, Python, …

Related downloads:
The scaffolder source code: MemcachedScaffolderSource.zip (1.12 mb)
The scaffolder, packaged and ready for use: MemcachedScaffolder.phar (2.87 mb)

The short version: use my scaffolder

As you may know, when working with PHP on Windows Azure and when making use of the Windows Azure SDK, you can use and create scaffolders. The Windows Azure SDK for PHP includes a powerful scaffolding feature that allows users to quickly setup a pre-packaged and configured website ready for Windows Azure.

If you want to use Memcached in your project, do the following:

  • Download my custom MemcacheScaffolder (MemcachedScaffolder.phar (2.87 mb)) and make sure it is located either under the scaffolders folder of the Windows Azure SDK for PHP, or that you remember the path to this scaffolder
  • Run the scaffolder from the command line: (note: best use the latest SVN version of the command line tools)
1 scaffolder run -out="c:\temp\myapp" -s="MemcachedScaffolder"

  • Find the newly created Windows Azure project structure in the folder you’ve used.
  • In your PHP code, simply add require_once 'memcache.inc.php'; to your code, and enjoy the $memcache variable which will hold a preconfigured Memcached client for you to use. This $memcache instance will also be automatically updated when adding more server instances or deleting server instances.
  • 1 require_once 'memcache.inc.php';

    That’s it!

    The long version: what this scaffolder does behind the scenes

    Of course, behind this “developers can simply use 1 line of code” trick a lot of things happen in the background. Let’s go through the places I’ve made changes from the default scaffolder.

    The ServiceDefinition.csdef file

    Let’s start with the beginning: when running Memcached in a Windows Azure instance, you’ll have to specify it with a port number to use. As such, the ServiceDefinition.csdef file which defines what the datacenter configuration for your app should be looks like the following:

    1 <?xml version="1.0" encoding="utf-8"?> 2 <ServiceDefinition name="PhpOnAzure" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"> 3 <WebRole name="PhpOnAzure.Web" enableNativeCodeExecution="true"> 4 <Sites> 5 <Site name="Web" physicalDirectory="./PhpOnAzure.Web"> 6 <Bindings> 7 <Binding name="Endpoint1" endpointName="HttpEndpoint" /> 8 </Bindings> 9 </Site> 10 </Sites> 11 <Startup> 12 <Task commandLine="add-environment-variables.cmd" executionContext="elevated" taskType="simple" /> 13 <Task commandLine="install-php.cmd" executionContext="elevated" taskType="simple"> 14 <Environment> 15 <Variable name="EMULATED"> 16 <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" /> 17 </Variable> 18 </Environment> 19 </Task> 20 <Task commandLine="memcached.cmd" executionContext="elevated" taskType="background" /> 21 <Task commandLine="monitor-environment.cmd" executionContext="elevated" taskType="background" /> 22 </Startup> 23 <Endpoints> 24 <InputEndpoint name="HttpEndpoint" protocol="http" port="80" /> 25 <InternalEndpoint name="MemcachedEndpoint" protocol="tcp" /> 26 </Endpoints> 27 <Imports> 28 <Import moduleName="Diagnostics"/> 29 </Imports> 30 <ConfigurationSettings> 31 </ConfigurationSettings> 32 </WebRole> 33 </ServiceDefinition>

    Note the <InternalEndpoint name="MemcachedEndpoint" protocol="tcp" /> line of code. This one defines that the web role instance should open some TCP port in the firewall with the name MemcachedEndpoint and expose that to the other virtual machines in your deployment. We’ll use this named endpoint later when starting Memcached.

    Something else in this file is noteworthy: the startup tasks under the <Startup> element. With the default scaffolder, the first two tasks (namely add-environment-variables.cmd and install-php.cmd) are also present. These do nothing more than providing some environment information about your deployment in the environment variables. The second one does what its name implies: install PHP on your virtual machine. The latter two scripts added, memcached.cmd and monitor-environment.cmd are used to bootstrap Memcached. Note these two tasks run as background tasks: I wanted to have these two always running to ensure when Memcached crashes the task can simply restart Memcached.

    The php folder

    If you’ve played with the default scaffolder in the Windows Azure SDK for PHP, you probably know that the PHP installation in Windows Azure is a “default” one. This means: no memcached extension is in there. To overcome this, simply copy the correct php_memcache.dll extension into the /php/ext folder and Windows Azure (well, the install-php.cmd script) will know what to do with it.

    Memcached.cmd and Memcached.ps1

    Under the application’s bin folder, I’ve added some additional startup tasks. The one responsible for starting (and maintaining a running instance of) Memcached is, of course, Memcached.cmd. This one simply delegates the call to Memcached.ps1, of which the following is the source code:

    1 [Reflection.Assembly]::LoadWithPartialName("Microsoft.WindowsAzure.ServiceRuntime") 2 3 # Start memcached. To infinity and beyond! 4 while (1) { 5 $p = [diagnostics.process]::Start("memcached.exe", "-m 64 -p " + [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::CurrentRoleInstance.InstanceEndpoints["MemcachedEndpoint"].IPEndpoint.Port) 6 $p.WaitForExit() 7 }

    To be honest, this file is pretty simple. It loads the WindowsAzure ServiceRuntime assembly which contains all kinds of information about the current deployment. Next, I start an infinite loop which continuously starts a new memcached.exe process consuming 64MB of RAM memory and listens on the port specified by the MemcachedEndpoint defined earlier.

    Monitor-environment.cmd and Monitor-environment.ps1

    The monitor-environment.cmd script takes the same approach as the memcached.cmd script: just pass the command along to a PowerShell script in the form of monitor-environment.ps1. I do want to show you the monitor-environment.cmd script however, as there’s one difference in there: I’m changing the file system permissions for my application (the icacls line).

    1 @echo off 2 cd "%~dp0" 3 4 icacls %RoleRoot%\approot /grant "Everyone":F /T 5 6 powershell.exe Set-ExecutionPolicy Unrestricted 7 powershell.exe .\monitor-environment.ps1

    The reason for changing permissions is simple: I want to make sure I can write a PHP script to disk every minute. Yes, you heard me! I’m using PowerShell (in the monitor-environment.ps1 script) to generate PHP code. Here’s the PowerShell:

    1 [Reflection.Assembly]::LoadWithPartialName("Microsoft.WindowsAzure.ServiceRuntime") 2 3 # To infinity and beyond! 4 5 while(1) { 6 ########################################################## 7 # Create memcached include file for PHP 8 ########################################################## 9 10 # Dump all memcached endpoints to ../memcached-servers.php 11 $memcached = "<?php`r`n" 12 $memcached += "`$memcachedServers = array(" 13 14 $currentRolename = [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::CurrentRoleInstance.Role.Name 15 $roles = [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::Roles 16 foreach ($role in $roles.Keys | sort-object) { 17 if ($role -eq $currentRolename) { 18 $instances = $roles[$role].Instances 19 for ($i = 0; $i -lt $instances.Count; $i++) { 20 $endpoints = $instances[$i].InstanceEndpoints 21 foreach ($endpoint in $endpoints.Keys | sort-object) { 22 if ($endpoint -eq "MemcachedEndpoint") { 23 $memcached += "array(`"" 24 $memcached += $endpoints[$endpoint].IPEndpoint.Address 25 $memcached += "`" ," 26 $memcached += $endpoints[$endpoint].IPEndpoint.Port 27 $memcached += "), " 28 } 29 30 31 } 32 } 33 } 34 } 35 36 $memcached += ");" 37 38 Write-Output $memcached | Out-File -Encoding Ascii ../memcached-servers.php 39 40 # Restart the loop in 1 minute 41 Start-Sleep -Seconds 60 42 }

    The output is being written every minute to the memcached-servers.php file. Why every minute? Well, if servers are added or removed I want my application to use the correct set of servers. This leaves a possible gap of one minute where some server may not be available, you can easily catch any error related to this in your PHP code (or add a comment to this blog post telling me what’s a better interval). Anyway, here’s the sample output:

    1 <?php 2 $memcachedServers = array(array('10.0.0.1', 11211), array('10.0.0.2', 11211), );

    All there’s left to do is consume this array. I’ve added a default memcache.inc.php file in the root of the web role to make things easy:

    1 <?php 2 require_once $_SERVER["RoleRoot"] . '\\approot\\memcached-servers.php'; 3 $memcache = new Memcache(); 4 foreach ($memcachedServers as $memcachedServer) { 5 if (strpos($memcachedServer[0], '127.') !== false) { 6 $memcachedServer[0] = 'localhost'; 7 } 8 $memcache->addServer($memcachedServer[0], $memcachedServer[1]); 9 }

    Include this file in your code and you have a full-blown distributed cache available in your Windows Azure deployment! Here’s a sample of some operations that can be done on Memcached:

    1 <?php 2 error_reporting(E_ALL); 3 require_once 'memcache.inc.php'; 4 5 var_dump($memcachedServers); 6 var_dump($memcache->getVersion()); 7 8 $memcache->set('key1', 'value1', false, 30); 9 echo $memcache->get('key1'); 10 11 $memcache->set('var_key', 'some really big variable', MEMCACHE_COMPRESSED, 50); 12 echo $memcache->get('var_key');

    That’s it!

    Conclusion and feedback

    This is just a fun project I’ve been working on when lonely and bored on airports. However, if you think this is valuable and in your opinion should be made available as a standard thing in the Windows Azure SDK for PHP, let me know. I’ll be happy to push this into the main branch and make sure it’s available in a future release.

    Comments or praise? There’s a comment form right below this post!

    Setting up a NuGet repository in seconds: MyGet public feeds

    A few months ago, my colleague Xavier Decoster and I introduced MyGet as a tool where you can create your own, private NuGet feeds. A couple of weeks later we introduced some options to delegate feed privileges to other MyGet users allowing you to make another MyGet user “co-admin” or “contributor” to a feed. Since then we’ve expanded our view on the NuGet ecosystem and moved MyGet from a solution to create your private feeds to a service that allows you to set up a NuGet feed, whether private or public.

    Supporting public feeds allows you to set up a structure similar to www.nuget.org: you can give any user privileges to publish a package to your feed while the user can never manage other packages on your feed. This is great in several scenarios:

    • You run an open source project and want people to contribute modules or plugins to your feed
    • You are a business and you want people to contribute internal packages to your feed whilst prohibiting them from updating or deleting other packages

    Setting up a public feed

    Setting up a public feed on MyGet is similar to setting up a private feed. In fact, both are identical except for the default privileges assigned to users. Navigate to www.myget.org and sign in using an identity provider of choice. Next, create a feed, for example:

    Create a MyGet NuGet feed and host your own NuGet packages

    This new feed may be named “public”, however it is private by obscurity: if someone knows the URL to the feed, he/she can consume packages from it. Let’s change that. Go to the “Feed Security” tab and have a look at the assigned privileges for Everyone. By default, these are set to “Can consume this feed”, meaning that everyone can add the feed URL to Visual Studio and consume packages. Other options are “No access” (requires authentication prior to being able to consume the feed) and “Can contribute own packages to this feed”. This last one is what we want:

    Setting up a NuGet feed

    Assigning the “Can contribute own packages to this feed” privilege to a specific user or to everyone means that the user (or everyone) will be able to contribute packages to the feed, as long as the package id used is not already on the feed and as long as the package id was originally submitted by this user. Exactly the same model as www.nuget.org, that is.

    For reference, all available privileges are:

    • Has no access to this feed (speaks for itself)
    • Can consume this feed (allows the user to use the feed in Visual Studio / NuGet)
    • Can contribute own packages to this feed '(allows the user to contribute packages but can only update and remove his own packages and not those of others)
    • Can manage all packages for this feed (allows the user to add packages to the feed via the website and via the NuGet push API)
    • Can manage users and all packages for this feed (extends the above with feed privilege management capabilities)

    Contributing to a public feed

    Of course, if you have a public feed you may want to have people contributing to it. This is very easy: provide them with a link to your feed editing page (for example, http://www.myget.org/Feed/Edit/public). Users can publish their packages via the MyGet user interface in no time.

    If you want to have users push packages using nuget.exe or NuGet Package Explorer, provide them a link to the feed endpoint (for example, http://www.myget.org/F/public/). Using their API key (which can be found in the MyGet profile for the user) they can push packages to the public feed from any API consumer.

    Enjoy!

     

    PS: We’re working on lots more, but will probably provide that in a MyGet Premium version. Make sure to subscribe to our newsletter on www.myget.org if this is of interest.

    NuGet push... to Windows Azure

    When looking at how people like to deploy their applications to a cloud environment, a large faction seems to prefer being able to use their source control system as a source for their production deployment. While interesting, I see a lot of problems there: your source code may not run immediately and probably has to be compiled. You don’t want to maintain compiled assemblies in source control, right? Also, maybe some QA process is in place where a deployment can only occur after approval. Why not use source control for what it’s there for: source control? And how about using a NuGet repository as the source for our deployment? Meet the Windows Azure NuGetRole.

    Disclaimer/Warning: this is demo material and should probably not be used for real-life deployments without making it bullet proof!

    Download the sample code: NuGetRole.zip (262.22 kb)

    How to use it

    If you compile the source code (download), you have X steps left in getting your NuGetRole running on Windows Azure:

    • Specifying the package source to use
    • Add some packages to the package source feed (which you can easily host on MyGet)
    • Deploy to Windows Azure

    When all these steps have been taken care of, the NuGetRole will download all latest package versions from the package source specified in ServiceConfiguration.cscfg:

    1 <?xml version="1.0" encoding="utf-8"?> 2 <ServiceConfiguration serviceName="NuGetRole.Azure" 3 xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" 4 osFamily="1" 5 osVersion="*"> 6 <Role name="NuGetRole.Web"> 7 <Instances count="1" /> 8 <ConfigurationSettings> 9 <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" /> 10 <Setting name="PackageSource" value="http://www.myget.org/F/nugetrole/" /> 11 </ConfigurationSettings> 12 </Role> 13 </ServiceConfiguration>

    Packages you publish should only contain a content and/or lib folder. Other package contents will currently be ignored by the NuGetRole. If you want to add some web content like a default page to your role, simply publish the following package:

    NuGet Package Explorer MyGet NuGet NuGetRole Azure

    Just push, and watch your Windows Azure web role farm update their contents. Or have your build server push a NuGet package containing your application and have your server farm update itself. Whatever pleases you.

    How it works

    What I did was create a fairly empty Windows Azure project (download).  In this project, one Web role exists. This web role consists of nothing but a Web.config file and a WebRole.cs class which looks like the following:

    1 public class WebRole : RoleEntryPoint 2 { 3 private bool _isSynchronizing; 4 private PackageSynchronizer _packageSynchronizer = null; 5 6 public override bool OnStart() 7 { 8 var localPath = Path.Combine(Environment.GetEnvironmentVariable("RdRoleRoot") + "\\approot"); 9 10 _packageSynchronizer = new PackageSynchronizer( 11 new Uri(RoleEnvironment.GetConfigurationSettingValue("PackageSource")), localPath); 12 13 _packageSynchronizer.SynchronizationStarted += sender => _isSynchronizing = true; 14 _packageSynchronizer.SynchronizationCompleted += sender => _isSynchronizing = false; 15 16 RoleEnvironment.StatusCheck += (sender, args) => 17 { 18 if (_isSynchronizing) 19 { 20 args.SetBusy(); 21 } 22 }; 23 24 return base.OnStart(); 25 } 26 27 public override void Run() 28 { 29 _packageSynchronizer.SynchronizeForever(TimeSpan.FromSeconds(30)); 30 31 base.Run(); 32 } 33 }

    The above code is essentially wiring some configuration values like the local web root and the NuGet package source to use to a second class in this project: the PackageSynchronizer. This class simply checks the specified NuGet package source every few minutes, checks for the latest package versions and if required, updates content and bin files.  Each synchronization run does the following:

    1 public void SynchronizeOnce() 2 { 3 var packages = _packageRepository.GetPackages() 4 .Where(p => p.IsLatestVersion == true).ToList(); 5 6 var touchedFiles = new List<string>(); 7 8 // Deploy new content 9 foreach (var package in packages) 10 { 11 var packageHash = package.GetHash(); 12 var packageFiles = package.GetFiles(); 13 foreach (var packageFile in packageFiles) 14 { 15 // Keep filename 16 var packageFileName = packageFile.Path.Replace("content\\", "").Replace("lib\\", "bin\\"); 17 18 // Mark file as touched 19 touchedFiles.Add(packageFileName); 20 21 // Do not overwrite content that has not been updated 22 if (!_packageFileHash.ContainsKey(packageFileName) || _packageFileHash[packageFileName] != packageHash) 23 { 24 _packageFileHash[packageFileName] = packageHash; 25 26 Deploy(packageFile.GetStream(), packageFileName); 27 } 28 } 29 30 // Remove obsolete content 31 var obsoleteFiles = _packageFileHash.Keys.Except(touchedFiles).ToList(); 32 foreach (var obsoletePath in obsoleteFiles) 33 { 34 _packageFileHash.Remove(obsoletePath); 35 Undeploy(obsoletePath); 36 } 37 } 38 }

    Or in human language:

    • The specified NuGet package source is checked for packages
    • Every package marked “IsLatest” is being downloaded and deployed onto the machine
    • Files that have not been used in the current synchronization step are deleted

    This is probably not a bullet-proof solution, but I wanted to show you how easy it is to use NuGet not only as a package manager inside Visual Studio, but also from your code: NuGet is not just a package manager but in essence a package management protocol. Which you can easily extend.

    One thing to note: I also made the Windows Azure load balancer ignore the role that’s updating itself. This means a roie instance that is synchronizing its contents will never be available in the load balancing pool so no traffic is sent to the role instance during an update.