Installation

Install using pip:

pip install pytest-executable

Install using conda:

conda install pytest-executable -c conda-forge

Command line interface

The pytest command line shall be executed from the directory that contains the inputs root directory.

Plugin options

--exe-runner PATH

use the shell script at PATH to run the executable.

This shell script may contain placeholders, such as {{output_path}} or others defined in the Runner section of a test-settings.yaml. A final runner shell script, with replaced placeholders, is written in the output directory of a test case ({{output_path}} is set to this path). This final script is then executed before any other test functions of a test case. See Runner fixture for further information.

If this option is not defined then the runner shell script will not be executed, but all the other test functions will.

A typical runner shell script for running the executable with MPI could be:

#! /usr/bin/env bash

env=/path/to/env/settings
exe=/path/to/executable

source $env

mpirun -np {{nproc}} \
	$exe \
	--options \
	1> executable.stdout \
	2> executable.stderr
--exe-output-root PATH

use PATH as the root for the output directory tree, default: tests-output

--exe-overwrite-output

overwrite existing files in the output directories

--exe-clean-output

clean the output directories before executing the tests

--exe-regression-root PATH

use PATH as the root directory with the references for the regression testing, if omitted then the tests using the regression_path fixture will be skipped

--exe-default-settings PATH

use PATH as the yaml file with the default test settings instead of the built-in ones

--exe-test-module PATH

use PATH as the default test module instead of the built-in one

--exe-report-generator PATH

use PATH as the script to generate the test report

See generate_report.py in the report-conf directory for an example of such a script.

Note

The report generator script may require to install additional dependencies, such as sphinx, which are not install by the pytest-executable plugin.

Standard pytest options

You can get all the standard command line options of pytest by executing pytest -h. In particular, to run only some of the test cases in the inputs tree, or to execute only some of the test functions, you may use one of the following ways:

Use multiple path patterns

Instead of providing the path to the root of the inputs tree, you may provide the path to one or more of its sub-directories, for instance:

pytest --exe-runner <path/to/runner> <path/to/tests/inputs/sub-directory1> <path/to/tests/inputs/sub/sub/sub-directory2>

You may also use shell patterns (with * and ? characters) in the paths like:

pytest --exe-runner <path/to/runner> <path/to/tests/inputs/*/sub-directory?>

Use marks

A test case could be assigned one or more marks in the test-settings.yaml file, see Marks section. Use the -m to execute only the test cases that match a given mark expression. A mark expression is a logical expression that combines marks and yields a truth value. For example, to run only the tests that have the mark1 mark but not the mark2 mark, use -m "mark1 and not mark2". The logical operator or could be used as well.

Use sub-string expression

Like the marks, any part (sub-string) of the name of a test case or of a test function can be used to filter what will be executed. For instance to only execute the tests that have the string transition anywhere in their name, use -k "transition". Or, to execute only the functions that have runner in their names, use -k "runner". Logical expressions could be used to combine more sub-strings as well.

Process last failed tests only

To only execute the tests that previously failed, use --last-failed.

Show the markers

Use --markers to show the available markers without executing the tests.

Show the tests to be executed

Use --collect-only to show the test cases and the test events (functions) selected without executing them. You may combine this option with other options, like the one above to filter the test cases.