Volodymyr Gubarkov

fhtagn – a tiny CLI programs tester written in AWK

June 2023

fhtagn is a tiny CLI tool for literate testing for command-line programs.

What does it mean literate testing?

Let’s say you created some program command. You want to create a set of end-to-end tests for it.

With fhtagn it’s as simple as creating a file tests.tush with the following content:

$ command --that --should --execute correctly
| expected stdout output

$ command --that --will --cause error
@ expected stderr output
? expected-exit-code

And running it:

./fhtagn.awk tests.tush

In case of success the tool will output nothing (in Unix tradition of being silent when all is OK).

In case if the actual output of a command line doesn’t match the expected output or exit code – the tool will show a diff of the expected and the actual results.

Each test file can have multiple such tests (like the example above have two, as you can see).
fhtagn will only process lines starting $, |, @ and ?. So you can have any other content there, that doesn’t start these symbols, for example description for each test. Alternatively, you can even make test files a markdown and place the tests into code blocks for readability.
Command lines can be multiline.

I use fhtagn as a testing tool for my projects:

I wrote this tool in AWK. Surprised? Check my article Fascination with AWK.

tush rewrite

In fact this project is my re-implementation of darius/tush, so all credit for the idea goes to the original author Darius Bacon.

But my rewrite is simpler (single tiny AWK script) and faster, because:

it uses /dev/shm where available instead of /tmp
it compares the expected result with the actual in the code and only calls diff to show the difference if they don’t match
it doesn’t create a sandbox folder for each test
it doesn’t use mktemp but rather generates random name in the code

Design principles

Below I want to elaborate a bit on design principles I’ve used to achieve the best speed. Basically there are two of them:

minimize running external processes
minimize I/O operations

Let’s show on examples.

AWK script, not shell

By using /usr/bin/awk shebang instead of /bin/sh with subsequent awk invocation we make it one shell process call less.

Do more in one shell invocation

Here we do a single shell invocation and get two values from it. It’s faster than doing two separate shell invocations.

Creating one less file by using stdin

Here we do

echo actual_data | diff expected_file -

instead of

diff expected_file actual_file

Because this allows to skip creating the actual_file.

Also note how we combine there the deletion (rm) of a temp file in the same call.

Batching cleanup in a single call

This change optimizes the temporary files removal. Instead of calling rm for each test we collect all temp files into ToDel variable and do the actual removal inside the END block.

By the way, this issue was identified by generating an arbitrary large synthetic test file and was fixed in this issue.

Results

On makesure project running the test suite (excluding the tests in 200_update.tush that does network calls) ./makesure tested_awks:

Before (tush*)	After (fhtagn)
36.1 sec	24.2 sec

The speedup is 33%!

*) For tush I was using the adolfopa/tush fork.

Tricky issue trying to test fhtagn with fhtagn

We test fhtagn with fhtagn! Yes, we eat our own dog food.

In fact, since fhtagn is syntactically fully compatible with tush, for the reference we firstly run the test suite with tush, and then, as an additional check, we run it with fhtagn.

Testing fhtag with fhtagn revealed very interesting bug. The test constantly stopped with an error

Testing with fhtagn (./fhtagn.awk)...
error reading file: /dev/shm/fhtagn.893568705.out

The .err and .out files are created for each $ line to capture stdout and stderr outputs of the executed command line. For some reason, the files were missing!

Initially I thought that for some mysterious reason, when running fhtagn tests with fhtagn the mentioned files were not created. I spent tons of time debugging this hypothesis. It appears, this was not the case.

Long story short, the files were created, but were then deleted. Let me explain.

When running fhtagn tests with fhtagh what happens is

fhtagn.awk (runner) starts fhtagn.awk (program under test).

Now, both fhtagn processes generate temporary .err and .out files with random names.

I used rand() and srand() AWK functions to generate a random name for a file.

It appears, that AWK’s

rand() always generates the same random sequence unless you set a seed with srand()
srand() by default sets the seed number that equals to the current timestamp (with 1 sec precision).

At this point you may have already guessed the culprit.

Since the “runner” fhtagn starts the “program under test” fhtagn in under some milliseconds after own start, the seed appears to be equal for both fhtagn processes, so they generate equal “random” file names!

I solved this problem by introducing the additional source of randomness in the form of the PID of a shell process ($$).

The problem was solved.

Overall the dogfooding practice proved to be really useful to stress-test the application.

If you noticed a typo or have other feedback, please email me at xonixx@gmail.com