Three years ago, as I was working on the regression test for “Project: Wolowizard [1],” I decided it might be a good idea to make a checklist of the steps required to run the thing, because it's complicated [2]. It's broken into four sections:
As you can see, it's not a short list but that's because I attempted to make it as explicit as possible (and while it could be automated [3], the time it would take to automate would exceed the time saved [4]).
I say “as possible” because there's implicit knowledge I have about the environment and forget to include, like “Oh yeah, I assumed that NFS (Nightmare File System) would be working” and “that's right, we have a new build server and ‘Project: Wolowizard’ has yet to be set up on that” and “oooh, yeah, you need to talk to R before you run the regression test as the system might not be available for testing.”
This is coming up because fellow cow-orker T has been tasked with running the “Project: Wolowizard” regression test (as I've been busy getting the “Project: Sippy-Cup [5]” regression test working) and these issues are coming up as he works his way through running the test.