Rory Primrose

Skip to content
Skip to menu

<< Readable testable code | Readonly is in the eye of the beholder >>

Jun 2 2008

Load tests don't like code coverage

Category: .Net — Rory Primrose @ 11:13

I have been pulling my hair out this afternoon trying to figure out why my code is performing so badly in a load test. After playing with the code and running a lot of tests, I didn't have any answers until I looked at my testrunconfig file. Code coverage was the first thing I looked at.

With code coverage turned on, the report looked like this.

After I turned code coverage off, I was much happier.

The Test Response Time graphs (top right) highlight the difference between the test run performance as does the the CPU usage (red line in the bottom right graph). Of the four tests, the one that is doing the most work was averaging 6.76 seconds per test with code coverage which is atrocious. Without code coverage, this test was running an average of 0.00024 seconds per test. Just a little different.

Updated: Changed images so they are more readable and changed the scale of the Test Response Time graph so the difference of the tests can be seen.

Tags:

0d844622-eb2c-4804-a691-068f69f083dc|0|.0

Comments (3)

Code coverage not available when debugging unit tests Yep, this one bit me last week. I had been writing some unit tests and debugging them. When t...Code coverage doesn't like foreach loopsI have an interesting scenario that I have just come across in my code. I have a foreach loop that i...Don't trace the callstack if you don't need toHere is another performance tip with tracing. In configuration, there is the opportunity to define s...

Comments

Ryan Boucher says:
Monday, June 02, 2008 at 7:42 AM

I'm glad you checked code coverage first. Although I do wonder why the default setting is to have it on (not your fault more of a Microsoft thing I would suspect)

Ideally your load testing (or perf/stress testing) configuration should resemble the eventual production configuration. This includes instrumentation (if you only turn it on in prod when needed) and hardware architecture.

Rory Primrose says:
Monday, June 02, 2008 at 3:25 PM

The default setting is to have it off. We have turned it on so that we can get code coverage of the unit tests.

The side issue here is that the code coverage setting is defined in a testrunconfig file which is stored against the solution, not an individual project. You can have multiple testrunconfig files, but only one can be the active config.

This means that in order to run load tests from the IDE, you either need two testrunconfig files and switch the active one to that which doesn't enable code coverage, or manually edit the only testrunconfig to disable the setting. Manual effort either way. It would have been nice if the testrunconfig could be assigned to a project.

We got around this in team build by running all unit tests, specifying the testrunconfig (which is why one config is easier), and then running the load tests as a separate task that doesn't specify a config.