Add a test case

A test case is composed of an input directory with:

  • the input files required by the runner shell script,

  • a test-settings.yaml file with the pytest-executable settings,

  • any optional pytest python modules for performing additional tests.


The input directory of a test case shall not contain any of the files created by the execution of the executable or of the additional python modules, otherwise they may badly interfere with the executions done by pytest-executable. In other words: do not run anything in the input directory of a test case, this directory shall only contain input data.

The test-settings.yaml file is used by pytest-executable for several things. When this file is found, pytest-executable will:

  1. create the output directory of the test case and, if needed, its parents,

  2. execute the tests defined in the default test module,

  3. execute the tests defined in the additional test modules.

  4. execute the tests defined in the parent directories.

The parents of an output directory are created such that the path from the directory where pytest is executed to the input directory of the test case is the same but for the first parent. This way, the directories hierarchy below the first parent of both the inputs and the outputs trees are the same.

If test-settings.yaml is empty, then the default settings are used. If --exe-default-settings is not set, the default settings are the builtin ones:

# Copyright 2020, CS Systemes d'Information,
# This file is part of pytest-executable
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# See the License for the specific language governing permissions and
# limitations under the License.

# Settings for the runner script
runner: {}

# List of keywords used to mark a test case
marks: []

# File path patterns relative to a test case to identify the regression
# reference files used for the regression assertions.
references: []

# Tolerances used for the assertions when comparing the fields, all default
# tolerances are 0.
tolerances: {}

The following gives a description of the contents of test-settings.yaml.


If other settings not described below exist in test-settings.yaml, they will be ignored by pytest-executable. This means that you can use test-settings.yaml to store settings for other purposes than pytest-executable.

Runner section

The purpose of this section is to be able to precisely define how to run the executable for each test case. The runner section contains key-value pairs of settings to be used for replacing placeholders in the runner shell script passed to --exe-runner. For a key to be replaced, the runner shell script shall contain the key between double curly braces.

For instance, if test-settings.yaml of a test case contains:

   nproc: 10

and the runner shell script passed to --exe-runner contains:

mpirun -np {{nproc}} executable

then this line in the actual runner shell script used to run the test case will be:

mpirun -np 10 executable

The runner section may also contain the timeout key to set the maximum duration of the runner shell script execution. When this duration is reached and if the execution is not finished then the execution is failed and likely the other tests that rely on the outcome of the executable. If timeout is not set then there is no duration limit. The duration can be expressed with one or more numbers followed by its unit and separated by a space, for instance:

   timeout: 1h 2m 3s

The available units are:

  • y, year, years

  • m, month, months

  • w, week, weeks

  • d, day, days

  • h, hour, hours

  • min, minute, minutes

  • s, second, seconds

  • ms, millis, millisecond, milliseconds

Reference section

The reference files are used to check for regressions on the files created by the executable. Those checks can be done by comparing the files with a tolerance , see Tolerances section. The references section shall contain a list of paths to the files to be compared. A path shall be defined relatively to the test case output directory, it may use any shell pattern like **, *, ?, for instance:

   - output/file
   - '**/*.txt'

Note that pytest-executable does not know how to check for regression on files, you have to implement the pytest tests by yourself. To get the path to the references files in a test function, use the fixture Regression path fixture.

Tolerances section

A tolerance is used to define how close shall be 2 data to be considered as equal. It can be used when checking for regression by comparing files, see Reference section. To set the tolerances for the data named data-name1 and data-name2:

        abs: 1.
        rel: 0.
        abs: 0.

For a given name, if one of the tolerance value is not defined, like the rel one for the data-name1, then its value will be set to 0..

Note that pytest-executable does not know how to use a tolerance, you have to implement it by yourself in a pytest tests. To get the tolerance in a test function, use the Tolerances fixture.

Marks section

A mark is a pytest feature that allows to select some of the tests to be executed, see Use marks. This is how to add marks to a test case, for instance the slow and big marks:

   - slow
   - big

Such a declared mark will be set to all the test functions in the directory of a test case, either from the default test module or from an additional pytest module.

You can also use the marks that already existing. In particular, the skip and xfail marks provided by pytest can be used. The skip mark tells pytest to record but not execute the built-in test events of a test case. The xfail mark tells pytest to expect that at least one of the built-in test events will fail.

Marks declaration

The marks defined in all test cases shall be declared to pytest in order to be used. This is done in the file pytest.ini that shall be created in the parent folder of the test inputs directory tree, where the pytest command is executed. This file shall have the format:

markers =
    slow: one line explanation of what slow means
    big: one line explanation of what big means