What does it mean literate testing?
Let’s say you created some program
command. You want to create a set of end-to-end tests for it.
With fhtagn it’s as simple as creating a file
tests.tush with the following content:
$ command --that --should --execute correctly | expected stdout output $ command --that --will --cause error @ expected stderr output ? expected-exit-code
And running it:
In case of success the tool will output nothing (in Unix tradition of being silent when all is OK).
In case if the actual output of a command line doesn’t match the expected output or exit code – the tool will show a
diff of the expected and the actual results.
?. So you can have any other content there, that doesn’t start these symbols, for example description for each test. Alternatively, you can even make test files a markdown and place the tests into code blocks for readability.
I use fhtagn as a testing tool for my projects:
I wrote this tool in AWK. Surprised? Check my article Fascination with AWK.
But my rewrite is simpler (single tiny AWK script) and faster, because:
/dev/shmwhere available instead of
diffto show the difference if they don’t match
mktempbut rather generates random name in the code
Below I want to elaborate a bit on design principles I’ve used to achieve the best speed. Basically there are two of them:
Let’s show on examples.
/usr/bin/awk shebang instead of
/bin/sh with subsequent awk invocation we make it one shell process call less.
Here we do a single shell invocation and get two values from it. It’s faster than doing two separate shell invocations.
Here we do
echo actual_data | diff expected_file -
diff expected_file actual_file
Because this allows to skip creating the
Also note how we combine there the deletion (
rm) of a temp file in the same call.
This change optimizes the temporary files removal. Instead of calling
rm for each test we collect all temp files into
ToDel variable and do the actual removal inside the
On makesure project running the test suite (excluding the tests in
200_update.tush that does network calls)
|Before (tush*)||After (fhtagn)|
|36.1 sec||24.2 sec|
The speedup is 33%!
*) For tush I was using the adolfopa/tush fork.
We test fhtagn with fhtagn! Yes, we eat our own dog food.
Testing fhtag with fhtagn revealed very interesting bug. The test constantly stopped with an error
Testing with fhtagn (./fhtagn.awk)... error reading file: /dev/shm/fhtagn.893568705.out
.out files are created for each
$ line to capture stdout and stderr outputs of the executed command line. For some reason, the files were missing!
Initially I thought that for some mysterious reason, when running fhtagn tests with fhtagn the mentioned files were not created. I spent tons of time debugging this hypothesis. It appears, this was not the case.
Long story short, the files were created, but were then deleted. Let me explain.
When running fhtagn tests with fhtagh what happens is
fhtagn.awk (runner) starts
fhtagn.awk (program under test).
Now, both fhtagn processes generate temporary
.out files with random names.
srand() AWK functions to generate a random name for a file.
It appears, that AWK’s
rand()always generates the same random sequence unless you set a seed with
srand()by default sets the seed number that equals to the current timestamp (with 1 sec precision).
At this point you may have already guessed the culprit.
Since the “runner” fhtagn starts the “program under test” fhtagn in under some milliseconds after own start, the seed appears to be equal for both fhtagn processes, so they generate equal “random” file names!
I solved this problem by introducing the additional source of randomness in the form of the PID of a shell process (
The problem was solved.
Overall the dogfooding practice proved to be really useful to stress-test the application.