Overview¶

Directory trees¶

The pytest-executable plugin deals with multiple directory trees:

the inputs
the outputs
the regression references

The inputs tree contains the files required to run an executable and to check its outcomes for different settings. It is composed of test cases as directories at the leaves of the tree. To create a test case, see Add a test case.

All the directory trees have the same hierarchy, this convention allows pytest-executable to work out what to test and what to check. The outputs tree is automatically created by pytest-executable, inside it, a test case directory typically contains:

symbolic links to the executable input files for the corresponding test case in the inputs tree
a runner shell script to execute executable
the files produced by the execution of executable
eventually, the files produced by the additional test modules

At the beginning, a regression reference tree is generally created from an existing outputs tree. In a regression references tree, a test case directory shall contain all the result files required for performing the comparisons for the regression testing. There can be more than one regression references trees for storing different sets of references, for instance for comparing the results against more than one version of executable.

Execution order¶

The pytest-executable plugin will reorder the execution such that the pytest tests are executed in the following order:

in a test case, the tests defined in the default test module (see --exe-test-module),
any other tests defined in a test case directory, with pytest natural order,
any other tests defined in the parent directories of a test case.

The purposes of this order is to make sure that the runner shell script and the other default tests are executed first before the tests in other modules can be performed on the outcome of the executable. It also allows to create test modules in the parent directory of several test cases to gather their outcomes.

How to use¶

Run the executable only¶

pytest <path/to/tests/inputs> --exe-runner <path/to/runner> -k runner

This command will execute the executable for all the test cases that are found in the input tree under path/to/tests/inputs. A test case is identified by a directory that contains a test-settings.yaml file. For each of the test cases found, pytest-executable will create an output directory with the same directory hierarchy and run the cases in that output directory. By default, the root directory of the output tree is tests-output, this can be changed with the option --exe-output-root. Finally, the -k runner option instructs pytest to only execute the runner shell script and nothing more, see Standard pytest options for more information on doing only some of the processing.

For instance, if the tests input tree contains:

path/to/tests/inputs
├── case-1
│   ├── input
│   └── test-settings.yaml
└── case-2
    ├── input
    └── test-settings.yaml

Then the tests output tree is:

tests-output
├── case-1
│   ├── input -> path/to/tests/inputs/case-1/input
│   ├── output
│   ├── executable.stderr
│   ├── executable.stdout
│   ├── runner.sh
│   ├── runner.sh.stderr
│   └── runner.sh.stdout
└── case-2
    ├── input -> path/to/tests/inputs/case-2/input
    ├── output
    ├── executable.stderr
    ├── executable.stdout
    ├── runner.sh
    ├── runner.sh.stderr
    └── runner.sh.stdout

For a given test case, for instance tests-output/case-1, the output directory contains:

output: the output file produced by the execution of the executable, in practice there can be any number of output files and directories produced.
input: a symbolic link to the file in the test input directory, in practice there can be any number of input files.
executable.stderr: contains the error messages from the executable execution
executable.stdout: contains the log messages from the executable execution
runner.sh: a copy of the runner shell script defined with --exe-runner, eventually modified by pytest-executable for replacing the placeholders. Executing this script directly from a console shall produce the same results as when it is executed by pytest-executable. This script is intended to be as much as possible independent of the execution context such that it can be executed independently of pytest-executable in a reproducible way, i.e. it is self contained and does not depend on the shell context.
runner.sh.stderr: contains the error messages from the runner shell script execution
runner.sh.stdout: contains the log messages from the runner shell script execution

If you need to manually run the executable for a test case, for debugging purposes for instance, just go to its output directory, for instance cd tests-output/case-1, and execute the runner shell script.

Check regressions without running the executable¶

pytest <path/to/tests/inputs> --exe-regression-root <path/to/tests/references> --exe-overwrite-output

We assume that the executable results have already been produced for the test cases considered. This is not enough though because the output directory already exists and pytest-executable will by default prevent the user from silently modifying any existing test output directories. In that case, the option --exe-overwrite-output shall be used. The above command line will compare the results in the default output tree with the references, if the existing executable results are in a different directory then you need to add the path to it with --exe-output-root.

The option --exe-regression-root points to the root directory with the regression references tree . This tree shall have the same hierarchy as the output tree but it only contains the results files that are used for doing the regression checks.

Run the executable and do default regression checks¶

pytest <path/to/tests/inputs> --exe-runner <path/to/runner> --exe-regression-root <path/to/tests/references>

Note

Currently this can only be used when the executable execution is done on the same machine as the one that execute the regression checks, i.e. this will not work when the executable is executed on another machine.

Finally, checks are done on the executable log files to verify that the file executable.stdout exists and is not empty, and that the file executable.stderr exists and is empty.