Building .NET projects is a world of pain and here’s how we should solve it

During the past few weeks, I’ve been working on and off on setting up a build agent that can build as many open-source .NET projects as possible in an effort to learn how hard it is to do. Allow me to open this blog post with a rant… One which will feel very familiar if you’ve recently installed a build agent yourself.

Setting up a .NET build machine is insane

As the minimal installation of components I started with installing the .NET framework 2.0, 3.0, 3.5, 4.0, 4.0.1 (yes, that exists), 4.0.2, 4.0.3, 4.5, 4.5.1 and their multi-targeting packs on the build agent. Next, I took 100 random C# projects from GitHub that had activity in the last year or so and started building and reading build logs. Great news! There are a lot of self-contained open source projects out there that build happily on this minimal install. Most of these seem to be class libraries, often depending on some NuGet packages that are installed using NuGet package restore.

Unfortunately, there are a great number of projects that do not build on this minimal setup: those that require specific SDK’s and components installed. So I started delving deeper into build logs and tackled project by project with the necessary “headless installs” of SDK’s. In practice, this sometimes means running an installer with specific commands to only install what is required to build projects on it. In other cases it means copying .targets and reference assemblies from my Windows 8.1 machine to the Windows Server 2012 R2 machine that was my build agent (yes, you can build Windows Store apps on Windows Server if you are persistent…). And in other cases (looking at you, Windows Phone SDK!) it meant running the installer in compatibility mode with some registry keys changed to overcome installer checks that do not allow installing that SDK on Windows Server.

In the end, I had to install pretty much the entire world on the build agent, or at least all SDK’s and tools that have been released between Visual Studio 2010 and the latest 2013. Here’s 17.6 GB of sh… dependencies for you.

Installed programs and features

What is the issue?

Well, there isn’t just “one” issue. There are several. Here’s a quick list of issues and questions

  • There is no way to clearly specify dependencies on SDK’s and tooling in .NET projects. The only way to know what is required is to build, read the build log, build, read the build log and someday succeed in finding the right SDK for the job. These dependencies are all implicit and there is no good way of finding out what they are, except trial and error.
  • The fact that I need this amount of SDK’s installed is crazy in itself. Why is this? Most builds simply need a .targets file and some DLLs, not all the other stuff that is in the download of such SDK.
  • Some SDK’s don’t install on every platform. Why is that? Why can’t SDK X install on platform Y?
  • Will I be able to install future versions of the SDK side-by-side so “older” projects build on my machine? Or will I need a machine for every Visual Studio version separately? How to isolate these things?

This is not only Microsoft tooling and SDK’s. Various other SDKs also require installs, prerequisites, configuration, … If only that picture above would allow scrolling so you could see Amazon, Xamarin and many others in that list.

How should we solve this?

Let’s look at the Node.js community and how they manage to do things. Every project, whether an actual application, a library or component, contains an important file: packages.json. It contains a description of the project itself, as well as the dependencies it requires, both id and version. All you need to build or run most of such projects is a node executable and an Internet connection to download dependencies on the fly. Sounds familiar? It does!

We’ve been using NuGet for quite a while now in the .NET space (if you haven’t, look into it now, even for in-house frameworks hosted on private feeds!). We’re distributing open source projects as NuGet packages that we can depend on in our own software. We can publish our own software as a NuGet package so others can depend on it. Awesome! Then why aren’t we doing this with the 17.6 GB of SDK madness we have to install on a build machine?

I do not think we can solve this quickly and change history. But I do think from now on we have to start building SDK’s differently. Most projects only require an MSBuild .targets file and some assemblies, either containing MSBuild tasks or reference assemblies, to do their compilation work. What if… we shipped the minimum files required to succesfully build a project as NuGet packages? The NuGet gallery contains some examples of this, but there are only a few. Another example is the ReSharper SDK which is shipped as a NuGet package. Need a test runner? Wrap the executable in a NuGet package and I’ll bring it down and run it during build. My takeaway: if you have a .targets file and are wrapping it in an MSI, you are doing it wrong.

Does that mean MSI's should disappear? No! They can exist and add tooling or whatever they need to add to a developer machine. All I want is the .targets file and supporting assemblies to be distributed separately as a self-contained package which I can reference explicitly, rather than the implicit way it is done now.

In my ideal world, all .NET projects would have a packages.config file in their root folder in which library dependencies as well as MSBuild dependencies can be described. My build machine would contain the .NET framework and Mono. And during build, all dependencies would be magically brought down for just that build.

P.S.: A lot of the new packages like ASP.NET MVC and WebApi, the OData packages and such are being shipped as NuGet packages which is awesome. The ones that I am missing are those that require additional build targets that are typically shipped in SDK's. Examples are the Windows Azure SDK, database tools and targets, ... I would like those to come aboard the NuGet train and ship their Visual Studio tooling separately from teh artifacts required to run a build.

This is an imported post. It was imported from my old blog using an automated tool and may contain formatting errors and/or broken images.

Leave a Comment

avatar

12 responses

  1. Avatar for Xavier Decoster
    Xavier Decoster April 11th, 2014

    Setting up a build server really is a good indicator of the many "implicit" dependencies we all use. We should all really step away from having implicit dependencies and declare them explicitly, and ensure all these things get pulled it in as needed in a unified way.
    Imho, given the analogies between NuGet, Chocolatey and OneGet (and even the Visual Studio Extension manager in a certain way), there's definitely a way to solve this...

  2. Avatar for Slavo Ingilizov
    Slavo Ingilizov April 11th, 2014

    "All you need to build or run most of such projects is a node executable and an Internet connection to download dependencies on the fly"

    In the proprietary software world, this is a problem. If it happens, Vendors have no control over how their software is distributed and how people pay for it. If everything was free to download things would be much easier. In reality, it isn't.

    I don't think the problem can be solved, unless Microsoft take the lead and do it for their SDKs, like they took the lead on NuGet.

  3. Avatar for Joshua Toon
    Joshua Toon April 11th, 2014

    Could you do a blog post about the hoops you had to jump through to get those SDK's to install?

  4. Avatar for Maarten Balliauw
    Maarten Balliauw April 11th, 2014

    You are trying to get me killed :-) Web Platform Installer is a great help but still.

  5. Avatar for khalidabuhakmeh
    khalidabuhakmeh April 11th, 2014

    I think I remember during a Build talk that the .NET team is moving as much functionality out of the .NET framework as they possibly can and are trying to distribute those changes via NuGet. The teams at Microsoft want to iterate as fast as they can without forcing you to download .NET Framework 4.5.1.14 to do it :)

    You are right, Microsoft and .NET have a long history, and fighting to change the habits of the past from permeating and shaping the future is HARD! That said, it needs to happen, so it's time for all .NET developers to strap in for the ride, it might be bumpy at times.

  6. Avatar for wwb_99
    wwb_99 April 11th, 2014

    One thing that could help would be just packaging more .targets and such with MSBuild directly -- there is no good reason that WebApplication.targets needs to be manually imported if you don't run visual studio for example. And I'm probably grossly violating some license agreement copying it to the build server but a man's gotta do what a man's gotta do.

  7. Avatar for Rudi Larno
    Rudi Larno April 18th, 2014

    If you agree this needs to change, take action and vote: http://visualstudio.uservoi...

  8. Avatar for Maarten Balliauw
    Maarten Balliauw April 18th, 2014

    Done, saw your tweet come by :-)

  9. Avatar for Rudi Larno
    Rudi Larno April 18th, 2014

    It's been almost 3 years, and 'under review' since Aug 2012, about time to make some more noise about this. Would be nice to retweet ;-)

  10. Avatar for Björn Bailleul
    Björn Bailleul April 21st, 2014

    Oh, I feel your pain ... try to mix in Oracle ;)
    Just spent 2 days figuring out all dependencies for a build agent that'll support most of our modules created by different teams.

  11. Avatar for Ubikuity
    Ubikuity April 27th, 2014

    The next step probably is to find a way to run unit tests without external dependencies. For example MsTest which requires Visual Studio or Visual Studio Team Agent on the build server.
    A nuget package for MsTest would be more than welcome.

  12. Avatar for Bennie Kloosteman
    Bennie Kloosteman January 27th, 2016

    1. Biggest pain i have is with linux make files..nothing more annoying than getting the right dll versions ..
    2. Use Visual studio online .. build and done Simplest CI i have seen and done.
    3. Dont use old visual studio .. upgrade your projects.

    KISS