Usage

The pytest-executable tool can be used in a wide variety of ways, the following sections explain how.

Run executable only

pytest --runner <path/to/runner> <path/to/tests/inputs> -k runner

This command will execute executable for all the test cases that are found in the inputs tree under path/to/tests/inputs. A test case is identified by a directory that contains a test_case.yaml file. For each of the test cases found, pytest-executable will create an output directory with the same directory hierarchy and run the cases in that output directory. By default, the root directory of the output tree tests-output, this can be changed with the option --output-root. Finally, the -k runner option instructs pytest to only execute the executable runner and nothing more, see Standard pytest options for more informations on doing only some of the processing.

For instance, if the test inputs tree contains:

path/to/tests/inputs
├── dir-1
│   ├── case.input
│   └── test_case.yaml
└── dir-2
    ├── case.input
    └── test_case.yaml

Then the output tree is:

tests-output
├── dir-1
│   ├── case.input -> path/to/tests/inputs/dir-1/case.input
│   ├── case.output
│   ├── executable.stderr
│   ├── executable.stdout
│   ├── run_executable.sh
│   ├── run_executable.stderr
│   └── run_executable.stdout
├── dir-2
    ├── case.input -> path/to/tests/inputs/dir-2/case.input
    ├── case.output
    ├── executable.stderr
    ├── executable.stdout
    ├── run_executable.sh
    ├── run_executable.stderr
    └── run_executable.stdout

For a given test case, for instance tests-output/dir-1, the output directory contains:

case.output

the output file produced by the execution of executable, in practice there can be any number of ouput files and directories produced.

case.input

a symbolic link to the file in the test case input directory, in pratice there can be any number of input files.

executable.stderr

contains the error messages from the executable execution

executable.stdout

contains the log messages from the executable execution

run_executable.sh

Executing this script directly from a console shall produce the same results as when it is executed by pytest-executable. This script is intended to be as much as possible independent of the execution context such that it can be executed independently of pytest-executable in a reproductible way, i.e. it is self contained and does not depend on the shell context. run_executable.stderr contains the error messages from the run_executable.sh execution

run_executable.stdout

contains the log messages from the run_executable.sh execution

If you need to manually run executable for a test case, for debugging purposes for instance, just go to its output directory, for instance cd tests-output/dir-1, and execute run_executable.sh.

Do default regression checking without running executable

pytest --regression-root <path/to/tests/references> <path/to/tests/inputs> --overwrite-output

We assume that executable results have already been produced for the test cases considered. This is not enough though because the output directory already exists and pytest-executable will by default prevent the user from silently modifying any existing test output directories. In that case, the option --overwrite-output shall be used. The above command line will compare the results in the default output tree with the references, if the existing executable results are in a different directory then you need to add the path to it with --output-root.

The option --regression-root points to the root directory with the regression references tree . This tree shall have the same hierarchy as the output tree but it only contains the results files that are used for doing the regression checks.

Run executable and do default regression checks

pytest --runner <path/to/runner> --regression-root <path/to/tests/references> <path/to/tests/inputs>

Note

Currently this can only be used when executable execution is done on the same machine as the one that execute the regression checks, i.e. this will not work when executable is submitted through a job scheduler.

Finally, checks are done on the executable log files to verify that the file executable.stdout exists and is not empty, and that the file executable.stderr exists and is empty.