We can already see a number of things from the dump summary: operating system, CLR version, modules that were loaded, but more important: we can see an exception code and exception information. It looks like IIS crashed because of a stack overflow (“The thread used up its stack”). Interesting! Seems we may be able to finish this investigation before we’re out of coffee…
In the Actions pane, we can Debug our dump file. What this will do is load all available information from the dump file, and provide that information via the tool windows we’re used to. Visual Studio can do this for managed code (.NET), unmanaged code, or “mixed”. If we Debug with Managed Only, we’ll be able to look at the stack trace at the moment the exception occurred, and we should be able to find out where this exception originated.
That all sounds great! Let’s click Break and see what we can find. In the Call Stack, we can see there are quite a number of recursive
TelemetryService.CaptureReportStatistics calls - which would explain the stack overflow that crashed our server. That method is quite complex, so it would be good if we were able to pinpoint exactly where the recursive call comes from. Unfortunately, Visual Studio tells us “Symbol loading skipped”, which means all we can see in terms of code would be the disassembly.
Confusing, but we can fix that. All we need is debugger symbols, which can map the execution stack back to our source code. In other words: if we fetch the .pdb files for our application from the build server, we should be able to look at the exact line of code where the exception was thrown. Of course the symbols should match the exact build that was deployed at the time of the crash, but we have that information. Except…
No symbols (.pdb) are to be found on the build server! Only the assemblies (.dll) are there - not the symbols! (insert swearing)
That’s sad - we’re now unable to map the crash dump to our code, making it a much harder task to find that bad line of code. Time for a second coffee… Unless - could we use the dotPeek built-in symbol server to solve our quest?
The IT folks provided us with a crash dump, and our build server still has the application assemblies. Let’s see if we can use dotPeek to provide us with debugger symbols and (decompiled) source code to find the root cause of our application crash.
dotPeek is a (free) decompiler which we can use to load an assembly and then look at its source code. A great feature is a built-in symbol server, which serves those decompiled sources so they can be used in WinDbg, Visual Studio or any other debugging tool being used. We can just launch it from the menu or the tool bar, but before doing that let’s look at the options first.
To prevent dotPeek from decompiling all assemblies it can find, let’s limit it to only decompile assemblies we explicitly open in dotPeek. This should speed up symbol loading in Visual Studio afterwards, as we’ll just be decompiling the assemblies we are interested in. Also note the symbol server URL,
http://localhost:33417/, as we’ll need that one in a bit.
|dotPeek’s symbol server is configured and started, all that’s left to do is load our application assembly which was still available on the build server (**File||Open…**), and then configure the symbol server URL in Visual Studio. We can configure symbol servers and symbols loading from the options, on the **Debugging||Symbols** page. Depending on the scenario, other options can be configured here as well but for this one, we’re totally fine just providing our dotPeek symbol server URL.|
Here comes the magic: we can now load symbols for our application into Visual Studio. From the call stack, we can use the context menu Load symbols on the highest entry that shows our assembly name.
In the dotPeek Project/Pdb Generation Status tool window, we’ll see our assembly being decompiled and PDB’s being generated. And in Visual Studio, we can now double-click our stack frame and see the decompiled code that triggered the stack overflow exception and crashed our application!
The morning started with a fresh coffee, a crash dump and no debuger symbols, and now we are able to see the exact line of code that crashed our application over night. dotPeek helped us in generating the .pdb file and decompiling source code from just our application’s assembly. Time for another coffee, and then fix the code…