The pytest-executable tool can be used in a wide variety of ways, the following sections explain how.
Run executable only¶
pytest --runner <path/to/runner> <path/to/tests/inputs> -k runner
This command will execute executable for all the test cases that are found in
the inputs tree under
path/to/tests/inputs. A test case is identified
by a directory that contains a
test_case.yaml file. For each of the test cases found,
pytest-executable will create an output directory with the same directory hierarchy and
run the cases in that output directory. By default, the root directory of the
tests-output, this can be changed with the option
--output-root. Finally, the
-k runner option instructs
pytest to only execute the executable runner and nothing more, see
Standard pytest options for more informations on doing only some of the processing.
For instance, if the test inputs tree contains:
path/to/tests/inputs ├── dir-1 │ ├── case.input │ └── test_case.yaml └── dir-2 ├── case.input └── test_case.yaml
Then the output tree is:
tests-output ├── dir-1 │ ├── case.input -> path/to/tests/inputs/dir-1/case.input │ ├── case.output │ ├── executable.stderr │ ├── executable.stdout │ ├── run_executable.sh │ ├── run_executable.stderr │ └── run_executable.stdout ├── dir-2 ├── case.input -> path/to/tests/inputs/dir-2/case.input ├── case.output ├── executable.stderr ├── executable.stdout ├── run_executable.sh ├── run_executable.stderr └── run_executable.stdout
For a given test case, for instance
the output directory contains:
the output file produced by the execution of executable, in practice there can be any number of ouput files and directories produced.
a symbolic link to the file in the test case input directory, in pratice there can be any number of input files.
contains the error messages from the executable execution
contains the log messages from the executable execution
Executing this script directly from a console shall produce the same results as when it is executed by pytest-executable. This script is intended to be as much as possible independent of the execution context such that it can be executed independently of pytest-executable in a reproductible way, i.e. it is self contained and does not depend on the shell context.
run_executable.stderrcontains the error messages from the
contains the log messages from the
If you need to manually run executable for a test case, for debugging purposes
for instance, just go to its output directory, for instance cd
tests-output/dir-1, and execute
Do default regression checking without running executable¶
pytest --regression-root <path/to/tests/references> <path/to/tests/inputs> --overwrite-output
We assume that executable results have already been produced for the test cases
considered. This is not enough though because the output directory already
exists and pytest-executable will by default prevent the user from silently modifying
any existing test output directories. In that case, the option
--overwrite-output shall be used. The above command line will compare
the results in the default output tree with the references, if the existing
executable results are in a different directory then you need to add the path
to it with --output-root.
--regression-root points to the root directory with the
regression references tree . This tree shall have the same hierarchy as the
output tree but it only contains the results files that are used for doing the
Run executable and do default regression checks¶
pytest --runner <path/to/runner> --regression-root <path/to/tests/references> <path/to/tests/inputs>
Currently this can only be used when executable execution is done on the same machine as the one that execute the regression checks, i.e. this will not work when executable is submitted through a job scheduler.
Finally, checks are done on the executable log files to verify that the file
executable.stdout exists and is not empty, and that the file
executable.stderr exists and is empty.