Test Cases

The boxes sources include three types of tests:

• Unit tests (“white-box tests”) test individual functions in isolation.
• Sunny-day tests are a suite of tests which simply creates, mends, and removes every box from our official config file. This is to make sure that regular operations work with all of our box designs.
• Black-box tests invoke boxes with all kinds of different arguments, inputs, and configurations in order to test the application as a whole.

Our continuous integration pipeline based on GitHub Actions will execute all three test suites in order to make sure that a change did not break the program.

In order to calculate total test coverage across all of these test suites, execute them in the following order:

make cov             # compile with coverage instrumentation
make utest           # unit tests (no "cov" prefix required)
make covtest-sunny   # sunny-day tests
make covtest         # black-box tests

The final coverage output will be the total coverage value (example, will have changed by now):

Overall coverage rate:
  lines......: 89.0% (3976 of 4465 lines)
  functions..: 98.6% (275 of 279 functions)
  branches...: 76.3% (2573 of 3371 branches)

Unit Tests

Unit tests, also called “white-box tests”, test individual functions in isolation.

In order to invoke the unit test suite, simply call make utest (Linux/UNIX/MacOS) or make win32.utest (Windows).

New unit tests can be added in the utest folder. We use cmocka as a unit test framework.

Sunny-Day Tests

Sunny-day tests are a suite of tests which simply creates, mends, and removes every box from our official config file. This is to make sure that regular operations work with all of our box designs.

In order to run the sunny-day test suite, make sure the TERM and LANG environment variables are set to indicate a color-capable terminal and a system encoding of UTF-8. For example:

export TERM=xterm-color
export LANG=en_US.UTF-8

Then, simply call make test-sunny. This is valid for all operating systems. Calling make cov && make covtest-sunny instead will do the same, but also calculate test coverage.

A new sunny-day test is only required when a box design is added to the official config file. Then, two files must be created:

• test/sunny-day/mydesign.create.txt
  Here, mydesign is the name of your new box design. Create the file with this command:

  out/boxes -f boxes-config -d mydesign \
      test/sunny-day/_input.txt > test/sunny-day/mydesign.create.txt
  

  The new file contains the expected output of the create operation.

• test/sunny-day/mydesign.mend.txt
  Again, mydesign is the name of your new box design. Create the file with this command:

  out/boxes -f boxes-config -d mydesign test/sunny-day/_input.txt \
      | sed -e "s/et /__ET__ /g" \
      | out/boxes -f boxes-config -d mydesign -m > test/sunny-day/mydesign.mend.txt
  

  The first boxes invocation is the same as for “create” above. Then the sed command simulates some editing of the box content, and the second boxes invocation mends the box.
  Make sure that these results are what you expect them to be, and that’s it!

Black-Box Tests

Black-box tests invoke boxes with all kinds of different arguments, inputs, and configurations in order to test the application as a whole.

In order to run the black-box test suite, make sure the TERM and LANG environment variables are set to indicate a color-capable terminal and a system encoding of UTF-8. For example:

export TERM=xterm-color
export LANG=en_US.UTF-8

Then, simply call make && make test from the top level directory. This is valid for all operating systems. Calling make cov && make covtest instead will do the same, but also calculate test coverage.

Run an individual test by calling ./testrunner.sh testcase.txt from the test directory.

Test Case Format

The black-box tests are defined in the test subdirectory.

Each test case is a single file within the test subdirectory. It must follow this naming convention:

nnn_description.txt

where nnn is a three-digit number which uniquely identifies the test case. description is any short text that describes what the test case does. It must not contain spaces; use underscores instead. The file extension is always .txt.

A test case that tests a successful invocation of boxes looks like this:

:DESC
Tests invoking boxes with a fixed result box size.

:ARGS
-s 10x4
:INPUT
foo
:OUTPUT-FILTER
:EXPECTED
/********/
/* foo  */
/*      */
/********/
:EOF

Sections may be empty, e.g. if there are no arguments or there is no input. The :DESC section is optional, so it is valid if the test case starts with an :ARGS section. The order of sections is fixed.

The :EXPECTED section understands an additional argument discard-stderr, which instructs the test runner to, well, discard stderr output instead of including it with the actual test result to be checked.

A test case that makes sure boxes fails under certain conditions looks like this:

:DESC
Tests that the right error message is shown when the boxes
config file cannot be found.

:ARGS
-f nonexistent
:INPUT
:OUTPUT-FILTER
:EXPECTED-ERROR 1
boxes: Couldn't find config file at 'nonexistent'
:EOF

Note that you write :EXPECTED-ERROR instead of just :EXPECTED, and the expected return code is given after a space (in this example, it is 1).

The :OUTPUT-FILTER section can be used to give a sed script which is run on the actual output before comparing it to the expected output (example). This way, differences in output that occur because of platform differences can be filtered out. The general advice is to leave this section empty unless you are facing a situation where there is no other solution.