A test guy

. . . is making a false-pass mistake in the test code. After all, you know the code works, so the test should pass. Right?

On the plane ride back from Agile2006, I finished the brain-dead simple unit test for some javascript code. I checked it in this week, and started working on factoring out the test harness. The tests kept failing under the new test harness, and I couldn’t see why.

I wrote more tests for my harness, and these new tests passed. I mixed up the test runs, and the test still failed. I investigated the possibility the failure was due to obscure javascript functionality related to closures and constructors. I even put in print statements because I have not yet tried to set up a debugger.

The print statements did it. I realized that in my original simple harness, I was grouping calls to assertThrows, and reported a pass if any of the calls passed. In my new harness code, the calls were not grouped, so the test failed.

I moved the assertThrows from inside testUpdatePatchThrowsExceptionForZeroSizePatch – now that function is called from within an assertThrows, and the test pass.

Lessons learned:

1. I have a bias when writing tests when the code exists, especially when I trust that the code works.
2. There was a point when the original harness wasn’t working right. I should have looked at all the code, including the calling tests, instead of just trying to get the harness to report “pass”
3. (Update) Don’t code at altitude

(running used as an adjective)

My goal is to have as many of the existing automated tests running in our continuous build system as possible by the end of the month. If the tests fail, the tinderbox will turn orange.

Please look for a post from me in dev-quality (newsgroup google group) later this evening with more details.

Notes:

1. Tests will most likely include, but are not limited to, necko tests, feed parser tests, jsunit-based tests, javascript tests, and my brain-dead simple unit tests

2. I’m not doing this all by myself. Rob Helmer, build team member, will be doing as much (if not more) of the heavy lifting as I am. He’s been working on the code that sent me running screaming from the room.

3. First targets are the trunk tinderboxes building firefox.

4. Yes, if the tests take too long to run, we’ll move them.

5. I want it to be easy to add new tests to the test run. ’cause tests that are never run aren’t worth much