Mentions légales du service

Skip to content
Snippets Groups Projects

License: LGPL v3 pipeline status coverage report

should -- Test command-line applications through .should files

should is a single-file program to test command-line applications on Unix-like systems. It checks the standard output, or possibly another file, and parses for exact or regular expressions, possibly while counting them and checking their number of occurrences. It outputs reports in .tap format and in JUnit-like XML.

should is written in Python with no external dependencies except from Python >= 3.4 and is intended to work on any command-line application -- should your application outputs something, you can test it!

Download

The archive should-2.0.0.tar.gz also includes this documentation with demo files. In your projetcts, you only need the should.py file.

Basic usage

The demo/hello.should example covers the basic functionality :

# A .should file launches any command it encounters.
# Every line starting with a `#` is a comment.

echo "hello, world"

# Lines containing a `:` are test lines.
# The `test expression` is what is found at the right of the `:`.
# It should be found in the stdout, at least one time.

:world
:lo, wo


# What is at the left of the `:` are modifiers.
# One can specify the exact number of times the test expression has to appear.

1:hello
0:Bye-bye


# Lines beginning by `$` give a name to the following tests

$ Check space before world
1:, wo

$ Check misspellings
0:wrld
0:helo


# The `r` modifier triggers Python regular expressions

$ Two o´s
r: o.*o

$ A more flexible test
r: [Hh]ello,\s+world

The test is then run by calling should on the .should file:

> ./should demo/hello.should
$LAUNCHER=

demo/hello.should
**echo "hello, world"**
  stdout --> 1 lines
  stderr --> 0 lines

ok
ok
ok
ok
ok - Check space before world
ok - Check misspellings
ok - Check misspellings
ok - Two o´s
ok - A more flexible test
ok - Exit code is 0
==> ok - ok:10 total:10 tests

demo/hello.should
==> ok - ok:10 total:10 tests

should can be run on several .should files at once. In this case, it furthers shows statistics on all tests. Here 39 out of 40 tests passed from 8 .should files.

> ./should demo/*.should
(...)

Summary ==> ok - ok:8 total:8 files
Summary ==> ok - ok:39 TODO:1 total:40 tests

Documentation

The documentation is completed by the files in demo/.

The demo/cal.should example shows matching on several lines (l), counting inside lines (w), ignoring whitespace differences (b), expecting a test to fail (f) or allowing a test to fail (a), requiring less than or more than a given number of expressions (</>).

demo/commands.should shows that several commands can be used into a same .should file. Tests are flushed after each set of commands.

demo/exit-codes.should shows how to require a particular exit code with !EXIT_CODE.

demo/variables.should shows how to define and use variables.

demo/launcher.should shows how to define a "launcher" that is appended before every command, for example to launch tools like valgrind on a test set. demo/extra.should show how to add an argument to every tested command.

Options and modifiers

usage: should [-h] [--version] [--cd PATH] [--cd-same] [--launcher CMD]
              [--extra ARG] [--mod MODIFIERS] [--var NAME=value]
              [--timeout TIMEOUT] [--shuffle] [--only-a] [--only-f] [--log]
              [--tap] [--xml] [-v] [-q] [--retry]
              should-file [should-file ...]

Test command-line applications through .should files

positional arguments:
  should-file       input files (.should)

optional arguments:
  -h, --help        show this help message and exit
  --version         show program's version number and exit
  --cd PATH         directory from which to run the test commands
  --cd-same         run the test commands from the same directory as the .should files
  --launcher CMD    launcher preceding each command (or replacing $LAUNCHER)
  --extra ARG       extra argument after the first word of each command (or replacing $EXTRA)
  --mod MODIFIERS   global modifiers (uppercase letters cancel previous modifiers)
                      f/F consider that the test should fail
                      a/A consider that the test is allowed to fail
                      r/R consider as a regular expression
                      w/W count all occurrences, even on a same line
                      i/I ignore case changes
                      b/B ignore whitespace differences as soon as there is at least one space. Implies 'r'
                      l/L search on all the output rather than on every line
                      z/Z keep leading and trailing spaces
                      >   requires that the expression occurs strictly more than the given number
                      <   requires that the expression occurs strictly less than the given number
  --var NAME=value  variable definition (then use $NAME in .should files)
  --timeout TIMEOUT Delay (in seconds) after which the task is stopped (default: 120)
  --shuffle         shuffle the tests
  --retry           launch again the last failed or warned tests

filter options:
  --only-a          launches only 'a' tests
  --only-f          launches only 'f' tests

output options:
  --log             stores the output into .log files
  --tap             outputs .tap files
  --xml             outputs JUnit-like XML into should.xml
  -v, --verbose     increase verbosity
  -q, --quiet       verbosity to zero

Output. By default, should only writes to the standard output. The --log, --tap and --xml options enable to store the actual output of the tested commands as well as .tap files and a JUnit should.xml file.

Exit code. should.py returns 0 when all the tests passed (or have been skipped, or marked as failed-with-ALLOW with a). A As soon as one regular test fails, or as soon as a test marked with TODO pass, should.py returns 1.

Alternatives

See also https://stackoverflow.com/questions/353198/best-way-to-test-command-line-tools.

History

should is a refactor of an earlier shell script that was developed and heavily used since 2013 to test two bioinformatics programs, CRAC (mapping of RNA sequences, discovery of transcriptomic and genomic variants), Vidjil-algo (clustering and analysis of lymphocyte clusters from their DNA sequences, contains 700+ should tests in about 100 files).