Command line interface

The pytest command line shall be executed from the directory that contains the inputs root directory.

Plugin options

--runner PATH

use the shell script at PATH to run executable, if omitted then executable is not run.

This shell script may contain placeholders, such as {{nproc}} and {{output_path}}. The placeholders will be replaced with the parameters determined from the context (either a pytest option or a setting defined in a test case via the test_case.yaml), and a final script is saved for each test cases to be run in their output directories, under the name run_executable.sh. This latter is used to run executable.

A typical script for running executable with MPI could be:

#! /usr/bin/env bash

env=/path/to/env/settings
exe=/path/to/executable

source $env

mpirun -np {{nproc}} \
	$exe \
	--options \
	1> executable.stdout \
	2> executable.stderr
--output-root PATH

use PATH as the root for the output directory tree, default: tests-output

--overwrite-output

overwrite existing files in the output directories

--clean-output

clean the output directories before executing the tests

--regression-root PATH

use PATH as the root directory with the references for the regression testing, if omitted then the tests using the regression_path fixture will be skipped

--default-settings PATH

use PATH as the yaml file with the global default test settings instead of the built-in ones

--report-generator PATH

use PATH as the script to generate the test report

See the report-conf directory for an example of such a script.

Note

The report generator script may require to install additionnal dependencies, such as sphinx, which are not required by the plugin.

Standard pytest options

You can get all the standard command line options of pytest by executing pytest -h. In particular, to run only some of the test cases in the inputs tree, or to execute only some of the test functions, you may use one of the following ways:

Use multiple path patterns

Instead of providing the path to the root of the inputs tree, you may provide the path to one or more of its sub-directories, for instance:

pytest --runner <path/to/runner> <path/to/tests/inputs/sub-directory1> <path/to/tests/inputs/sub/sub/sub-directory2>

You may also use shell patterns (with * and ? characters) in the paths like:

pytest --runner <path/to/runner> <path/to/tests/inputs/*/sub-directory?>

Use marks

A test case could be assigned one or more marks in the test_case.yaml file, then with -m only the test cases that match a given mark expression will be run. A mark expression is a logical expression that combines marks and yields a truth value. For example, to run only the tests that have the mark1 mark but not the mark2 mark, use -m "mark1 and not mark2". The logical operator or could be used as well.

Use substring expression

Like the marks, any part (substring) of the name of a test case or of a test function can be used to filter what will be executed. For instance to only execute the tests that have the string transition anywhere in their name, use -k "transition". Or, to execute only the functions that have runner in their names, use -k "runner". Logical expressions could be used to combine more susbtrings as well.

Process last failed tests only

To only execute the tests that previously failed, use --last-failed.

Show the markers

Use --markers to show the available markers without executing the tests.

Show the tests to be executed

Use --collect-only to show the test cases and the test events (functions) selected without executing them. You may combine this option with other options, like the one above to filter the test cases.