💾 Archived View for thrig.me › blog › 2024 › 04 › 17 › failure-is-an-option.gmi captured on 2024-05-12 at 15:15:29. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-05-10)
-=-=-=-=-=-=-
100% test coverage is a luxury for the idle who have the time to write the necessary tests. Full coverage can be tricky if you need a file descriptor or socket to fail after some number of bytes (or maybe some amount of time or packets) have gone by. One can sometimes use a fake or mock object, or possibly LD_PRELOAD hacks. Some solutions may limit the portability of the tests.
100% test coverage may also be a waste of time, as there can be diminishing returns on covering those last few nooks and crannies—high cost, low benefit. Making comments about what is not tested and why may help future maintainers, possibly a future version of you. How much coverage the code needs depends on the language and how much the compiler checks, and what exactly the code does. Libraries probably need more tests for correctness than a tricky to test game loop. But if you are not testing that game loop automatically, then someone probably does need to try to start the game and do a few things before it gets shipped. This does not appear to be how some in the game industry operate.
Or rather a function that reads two five byte (or possibly character) chunks, and some tests that it does so.
#!/usr/bin/env perl
use 5.36.0;
use Test2::V0;
sub parse ($fh) {
my ( $first, $second );
my $amount = read $fh, $first, 5;
die "read1 $!" unless defined $amount;
die "len1 $amount" if $amount != 5;
$amount = read $fh, $second, 5;
die "read2 $!" unless defined $amount;
die "len2 $amount" if $amount != 5;
return $first, $second;
}
open my $fh, '<', \"helloworld" or die "open $!";
my ( $first, $second ) = parse($fh);
is $first, 'hello';
is $second, 'world';
done_testing;
If we test this,
$ PERL5OPT=-MDevel::Cover perl helloworld.pl
# Seeded srand with seed '20240416' from local date.
ok 1
ok 2
1..2
$ cover -report compilation
...
helloworld.pl 100.0 50.0 n/a 100.0 n/a 100.0 86.8
...
Branch never false at helloworld.pl line 9: defined $amount
Branch never true at helloworld.pl line 10: $amount != 5
Branch never false at helloworld.pl line 13: defined $amount
Branch never true at helloworld.pl line 14: $amount != 5
Branch never false at helloworld.pl line 19: open my $fh, "<", \"helloworld"
Coverage is 87% and who knows what those missing percents are up to. A maintainer would have to review and think about all that code. Or they could ignore it, on account of management having them busy with other things. Meetings, for example.
The last branch is tricky to test as opening an in-memory string as a filehandle would likely only fail in a low memory situation (or maybe some other weird cases?) and to test it you'd need to make perl fail inside the open function. High effort, low reward. The other branches probably should be tested and here we need a mock filehandle that can return an error on demand. The end-of-file condition can be tested by passing a filehandle with not enough bytes (or characters) in it, but EOF is a different branch than an error (should it be?).
In Perl one can tie a filehandle to some arbitrary object, one that returns only some number of bytes (or characters) and then always fails.
#!/usr/bin/env perl
use 5.36.0;
package FailFH {
use Errno qw(EBADF);
sub creat ( $data, $limit ) {
\do { local *H; tie *H, FailFH => $data, $limit; *H }
}
sub TIEHANDLE {
my ( $class, $data, $limit ) = @_;
bless { data => $data, limit => $limit, offset => 0, }, $class;
}
sub READ {
my ( $self, undef, $len ) = @_;
my $remains = $self->{limit} - $self->{offset};
if ( $remains <= 0 ) {
$! = EBADF;
return undef;
}
$len = $remains if $remains < $len;
$_[1] = substr $self->{data}, $self->{offset}, $len;
$self->{offset} += $len;
return $len;
}
}
my $fh = FailFH::creat foobar => 3;
my $buf;
read $fh, $buf, 3;
say $buf; # okay, read 'foo'
read $fh, $buf, 3;
say $!; # not okay, got EBADF
Only three bytes (or maybe characters) can be read before an errno happens:
$ perl tiehandle.pl
foo
Bad file descriptor
This is a woefully incomplete class (a "seek" handler might be good to have, so you can reset the offset for subsequent tests with the same handle) but on the other hand it may be enough to raise the code coverage by some amount. In particular it could be used to trigger the errno conditions (read returns undef) in the parse subroutine up at the top of this document.
Why is the "creat" function missing an "e"? Tradition.
The above code is totally contrived. For actual tests that test whether a MIDI test module (who tests the testers?) works correctly, see the new Perl module
https://thrig.me/src/Test2-Tools-MIDI.git
which allows MIDI files (or strings in memory) to be tested for appropriate headers, tracks, and (some but not all MIDI) events.
On the other hand, LilyPond runs much quicker in OpenBSD 7.5, so this might all have been a glorious waste of time, besides learning a bunch more about the ancient MIDI protocol and ancient filehandle tie methods. Gotta keep my skills from going rusty.
Some code does not need tests. Probably on account of being too short, very interactive, often used, and controlling nothing critical whatsoever.
#!/bin/sh
# brogue - wrapper script to launch Brogue
xdotool search --name Brogue windowactivate 2>/dev/null && exit 0
exec solitary / /usr/local/bin/brogue --size 16 "$@"
Other code really does need formalisms and proven designs. Therac-25 comes to mind. Between mostly unproven and verified design lies a whole lot of code, where full verification would cost too much, or how do you even formal where you only have the vaguest of notions as to what the code may do? (This is almost everything I write.) With experience you may be able to guess (or there could be regulations as to) whether something will need formalisms. In other cases you may end up with prototype code that is too small a sheet for the bed, revealed after tucking the sheet in pulls it off the other side. Such antics might be called a regression, and at this point you'll either need to go formal or add regression tests that check whether a change has pulled the sheet off (that is, created an error in) some other part of code. Or, you could continue to fiddle around with the code, maybe breaking things, and how would you know? All three methods are popular among programmers.
Hence a need for a MIDI test module that can check whether various generated MIDI files contain the expected events. Like maybe you added note tie support, and that broke tempo changes, and thus the sheet was too small for the bed. Getting the MIDI test module to 100% test coverage is probably a needless rabbit hole, but on the other hand, how do you know your test code is doing what it should? Lacking formalisms one might want to have a certain confidence that the code will not summon mallet gnomes.
Mallet gnomes? Those are the tiny gnomes that climb out of your monitor and bonk you when your code does something silly. Programmers can be a mighty superstitions lot.
Maybe some year something like Ada/SPARK or Coq will be common, available, quick, and easy to use?