InstallUtil and BadImageFormatException – Facepalm

I had a frustrating issue at work this week: one that was easy to fix, but embarrassingly difficult to find. I came pretty close to giving up, which is not a solution I often explore, but in the end we figured it out and got everything working.

A member of our operations team was installing a Windows service I’d built to monitor some stuff in our production environment. I’ve made a few windows services in my day, and installed them many times on many machines. I’d even installed this one on my development machine with no issue. In our staging environment, however, this is what we got:

C:\Install\TheService>C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe TheService.exe
Microsoft (R) .NET Framework Installation utility Version 4.0.30319.1
Copyright (c) Microsoft Corporation. All rights reserved.

Exception occurred while initializing the installation:
System.BadImageFormatException: Could not load file or assembly ‘file:///C:\Monitoring\Service\TheService.exe’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

We checked the likely things: the framework version, the platform the app was built for, even re-copying the files in case they somehow got corrupted. When these didn’t work, we started trying more radical things: forcing all assemblies to 32 bit, even running the service as an executable to see if there was some error in the app.

In my defence, we are both experienced engineers, and I’m not the only person who missed it. Look closely at the command line we used:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe

Long version: Service applications in Visual Studio 2010 are 32 bit by default, and this is a reasonable default for them to have. We were trying to install the 32 bit service with the 64 bit version of InstallUtil. InstallUtil loads the target assembly to access it’s installation instructions, but you can’t load a 32 bit assembly from a 64 bit application (or vice versa). If you try to, you get a BadImageFormatException.

Short version: Two numbers derailed my entire afternoon.

It would have been nice if the error message from InstallUtil was a little more specific, but I suppose this isn’t a common problem. At least I got a good reminder about the importance of checking the small details when the big ones aren’t bearing fruit.

The Average Test

I was conducting technical interviews recently for a senior developer position. I like to talk about a bunch of things, but there is one question I particularly enjoy asking each candidate. Write an implementation for this:

It’s a great question because it’s a simple problem, and anyone that comes through the door should know the formula without having to ask. It’s interesting how people react to it sometimes, but I chalk it up to interview jitters. About half the people interviewed answer like this:

It’s a good first answer too, except that it doesn’t always work.

“What will be the result of your function if it’s called with the values Int32.MaxValue and Int32.MaxValue?”

The answer I want to hear is that it won’t work. Some people think (like I did), that it will throw a System.OverflowException, but it actually returns -1. The default for C# projects is to skip overflow checking for arithmetic.

This is another answer that I get from time to time:

I respect this answer; it indicates that a developer wants to leave low-level details to the framework, which is a perfectly reasonable attitude when developing business-class software. Unfortunately there are two problems with this answer: it defeats the purpose of the test, and it doesn’t exist in the framework. There is the Linq average statement which can be used like this:

After the first attempt, I indicate that I want a function which will work correctly for all values of a and b. The way candidates tackle it tells you a lot about them as a developer. Here are some behaviours I’ve noticed:

  • Very few candidates stop to consider a solution before answering.
  • About 3/4 of candidates try to use an if statement.
  • About 1/4 of candidates write the “if” on the board before deciding what to use it for.
  • Occasional candidates will challenge the necessity for handling high values.

A clean correct answer is hard to pull off on the spot, especially under pressure. I’ve never seen it, and I wouldn’t be surprised if I never do. This doesn’t bother me either. The key thing I’m looking for is the ability to understand questions and discuss the source code. These skills are essential to working in a team.

How did I come up with such a brilliant interview question? I didn’t. Once upon a time in a company far far away, someone asked me the very same thing. I was pretty proud of my answer too, until I tested it. This was what I came up with:

I wrote a test harness in some spare time after the last round of interviews. Here’s how each of the answers measure up:

Results for SimpleAverage():
 Near Zero: 121 of 121 passed.
 End of Range: 0 of 3 passed.
 Failed Values:
 - SimpleAverage(2147483647, 2147483647) returns -1, but should return 2147483647
 - SimpleAverage(-2147483648, -2147483648) returns 0, but should return -2147483648
 - SimpleAverage(2147483647, 2147483645) returns -2, but should return 2147483646

Results for LinqAverage():
 Near Zero: 121 of 121 passed.
 End of Range: 3 of 3 passed.

Results for JessesAverage():
 Near Zero: 109 of 121 passed.
 End of Range: 3 of 3 passed.
 Failed Values:
 - JessesAverage(-4, 1) returns -2, but should return -1
 - JessesAverage(-4, 3) returns -1, but should return 0
 - JessesAverage(-3, 4) returns 1, but should return 0
 - JessesAverage(-2, 1) returns -1, but should return 0
 - JessesAverage(-1, 2) returns 1, but should return 0
 - JessesAverage(-1, 4) returns 2, but should return 1
 - JessesAverage(1, -4) returns -2, but should return -1
 - JessesAverage(1, -2) returns -1, but should return 0
 - JessesAverage(2, -1) returns 1, but should return 0
 - JessesAverage(3, -4) returns -1, but should return 0
 - JessesAverage(4, -3) returns 1, but should return 0
 - JessesAverage(4, -1) returns 2, but should return 1

This was the best answer I could come up with after some experimentation. I’m not sure why, but the need for type conversion bugs me.

Results for ConversionAverage():
 Near Zero: 121 of 121 passed.
 End of Range: 3 of 3 passed.

64-bit IIS vs. 32-bit Assemblies

I found my first 64-bit bug at work. I was moving a windows service built for the ‘Any CPU’ to a 64-bit server. It started fine on the new server, and gave no indication of poor health in the logs, but one key function was malfunctioning. Im not exactly sure what the cause is, but I know that the hash of any binary file was resulting in the same value. The service does some direct memory manipulation which is a likely culprit.

Fixing the problem was easy: Set all the assemblies to compile for the 32-bit platform, check it in, build it, ship it.

Except that my web service now looks like this:

Server Error in ‘/MyApp’ Application.

Could not load file or assembly ‘MyCompany.MyApp.MyAssembly’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

This application has two parts on the server: a windows service that processes emails, and a web service for interacting with them. The two parts share a database and a bunch of business logic, so they also share some assemblies. The windows service works great now that I’ve converted all the assemblies to 32-bit, but the web service wont start.

Although the error message is a bit misleading, it wasnt that hard to find the problem. When IIS is running in 64-bit mode (as it tends to do on 64-bit servers), all of its application pools will be 64-bit, and a 64-bit application pool cant load a 32 bit assembly. These are all the ways I can think of to fix it:

  • IIS 7 can run an application pool in 32-bit mode
  • IIS 6 can be configured to run in 32-bit mode, or you can run a second instance in 32-bit mode, though I dont imagine both instances could share the same ports
  • Write 32-bit and 64-bit versions for any platform-specific operations using conditional compilation, then build two versions of your assemblies
  • Rewrite the malfunctioning parts of your app so you dont have any platform-specific code
  • Compile most of your assemblies for ‘Any CPU’, but compile entry-point assemblies (such as my service executable) for 32-bit

A configuration-based solution isnt ideal based on the number and management practices of our environments. Finding and removing platform dependence would be the best technical solution, but it would be risky and time consuming. Switching some of my assemblies back to ‘Any CPU’ did the job with a minimal amount of impact.