Generating regression test cases

We miscounted [1]—there are, in fact, 9,697 test cases so far.

My manager, S, has been responsible for generating the test case descriptions, stuff like:

## Section 1
Originating device is XXXXXXXXXX XX XXXXX and is calling a terminator with our client on the device test cases.
> * Terminating device can have the following features: XXXXXXX, XXXXXXXXXXXXX (2 options)
* Originating device can be a sip: [2] or tel: [3] URI (Uniform Resource Locator) (2 options)
* Terminating device can be a sip: or tel: URI (2 options)
* Originating can have XXXXXXX present or not (2 options)
* Terminator's XXXXXXX state is not applicable (1 option)
* Originating device can have two options for XXX (2 options)
* Terminator's XXX state is not applicable (1 option)
* Originating device can have XXXX values of XXXXXXXXXX, XXXXXXXXXXX, XXXXXXXXXXX (3 options)
* Terminator's XXXX state/capability is not applicable (1 option)
* XXXXXXXX XXXX XXXXXXXXXXX is either present or not for the Originating device (2 options)
* Terminator's XXXXXXXX XXXX XXXXXXXXXXX is not applicable (1 option)
* Originating Type of Number can either by CDMA (Code Division Multiple Access) or LTE (Long Term Evolution) (2 options)
* Terminating Type of Number can be: cdma+XXXX, cdma, lte (3 options)
* Terminating XXXXX can be: XXXXXX, XXXXXXX, XXXX (3 options)
* Originators XXXXX is not applicable (1 option)
* Total number of test cases in this section = 2 × 2 × 2 × 2 × 2 × 3 × 2 × 2 × 3 × 3 = 3456.

It's my job to generate the data required to run all these test cases, which means I need to run through the various variables and all their values, generating the data required to run the test and there's no better way of doing this than to brute force it. I could have written the code as:

>
```
-- variable names changed to protect the innocent.
for _,A in ipairs { false, true } do
for _,B in ipairs { 'tel' , 'sip' } do
for _,C in ipairs { 'tel' , 'sip' } do
for _,D in ipairs { false , true } do
for _,E in ipairs { false , true } do
for _, F in pairs { 'opt1' , 'opt2' , 'opt3' } do
...
test_case(A,B,C,E,E,F, ... )
end
end
end
end
end
end
end
```

and while that does the trick, it's verbose and it uses a convention (using '_' to designate a result one doesn't care about, and in this case, I don't care about the first return result from ipairs()) that the other programmers here might not immediately pick up on (as far as I know, I'm the only one using Lua [4] at the Corporation).

No, I feel it's better to show my intent. I want the code to look like:

>
```
for A in bool() do
for B in list { 'tel' , 'sip' } do
for C in list { 'tel' , 'sip' } do
for D in bool() do
for E in bool() do
for F in list { 'opt1' , 'opt2' , 'opt3' } do
...
test_case(A,B,C,D,E,F, ... )
end
end
end
end
end
end
end
```

And that's exactly what I did. The generic for loop in Lua is defined as [5]:

for variable in function, state-variable, initial-value

where function is repeatedly invoked as function(state-variable,initial-value) until the it returns nil; when the result isn't nil, it's assigned to the main loop variable. In the first example, ipairs() (a standard Lua function [6]):

>
```
function ipairs(list)
local function next_value(state,var)
var = var + 1
if var > #state then
return nil
else
return var,state[var]
end
end
return next_value,list,0
end
```

ipairs() returns a function that takes a state variable (in this case, the variable list which should be an array) and an initial value (0) and this is repeatedly called until that function returns nil (here, when the index exceeds the number of items in the array).

For what I wanted, the function list() is easy:

>
```
function list(L)
local function next_item(state)
state.n = state.n + 1
return L[state.n]
end
return next_item,{ n = 0 }
end
```

The function next_item() only cares about the state, which holds an index into the list (it's stored in a table because this is the only way we can modify it) which we increment and return that item from the passed in list L. We only return two values, the function and the state. The missing third value will be nil, which we don't care about. The one for cycling through booleans, bool() looks a bit more complicated:

>
```
function bool()
return function(_,value)
if value == nil then
return false
elseif value == false then
return true
else
return nil
end
end
end
```

Here we just return a function; the other two values for is expecting will then become nil. And since I don't care about any state (the next value is readily apparent from the previous value we've returned), that parameter in the returned function is named _ (the conventional name for ignored parameters). Since we initially gave for a nil value, that's what we get on the first call, so we return false. On the next call, value is false so we return true. On the third call, we call through, returning nil which ends the for loop.

And those two functions make the test case generation code look much better. The intent is clearer, and you can easily match the code against the English description.

[1] /boston/2015/01/22.1

[2] https://www.ietf.org/rfc/rfc3261.txt

[3] https://www.ietf.org/rfc/rfc2806.txt

[4] http://www.lua.org/

[5] http://www.lua.org/manual/5.3/manual.html#3.3.5

[6] http://www.lua.org/manual/5.3/manual.html#pdf-ipairs

Gemini Mention this post

Contact the author