DFREGRESS(8)

NAME
dfregress - an automation test driver and framework

SYNOPSIS
dfregress [-o output_plist] [-t testcase_dir] [-p prepost_dir] runlist_file
dfregress -h

DESCRIPTION
dfregress is a regression test framework and automation driver. It executes a series of testcases as specified in the runlist_file and collects the results.

The path to the testcase collection is specified via the testcase_dir argument. If this argument is not specified, the default is assumed to be the same directory as that of the runlist. For example if the used runlist is /usr/src/test/testcases/sample.run the default testcase directory, unless otherwise specified, will be /usr/src/test/testcases.

Similarly the path to the pre- and post commands is specified via prepost_dir. The prepost_dir only needs to be specified if the runlist contains custom pre or post commands.

The output is saved in plist format to the output_plist file, or if none is specified, to a file with the same base name as the runlist, but in the current working directory and with a .plist extension. For example if the used runlist is /usr/src/test/testcases/sample.run the default output, unless otherwise specified, will be ./sample.plist. Other tools, known as frontends, can parse the plist output into an easier to read form.

The following is a summary of the optional parameters:

-o output_plist
    Specifies the file to which to write the results. The resulting file, output_plist, will be in plist format.

-t testcase_dir
    Specifies the directory in which to find the testcases specified in the runlist.

-p prepost_dir
    Specifies the directory in which to find the pre- and post commands used in the runlist. This argument is only required when the runlist uses custom pre- or post commands.

-h  Prints a short synopsis.

RUNLIST SYNTAX
Testcases are specified one testcase per line, with whitespace separated values. Empty lines and lines beginning with a “#” are ignored.

Runlist lines are of the following format:

    testcase type options arguments ...

The testcase part needs to be a relative path from the testcase base directory specified by the -t argument to the resulting (after building the testcase) testcase executable. The testcase will be executed with the arguments passed as command line arguments to it.

Valid types are userland, kernel and buildonly:

userland
    A userland testcase is a normal userland executable that returns a non-zero exit value on test failure.

kernel
    A kernel testcase is run with the kernel test bridge inside the kernel.

buildonly
    A buildonly test passes when the build for the given testcase succeeds.

Valid options are defaults, interpreter, make, rc, timeout, nobuild, runas, intpre, intpost, pre, and post:

defaults
    This option does nothing. All default options are applied.

interpreter command
    Uses command for running the test case instead of calling it directly. This is useful when you have tests that you want to call with a specific shell like sh(1), and there is no shebang in the testcase file, there are no execution permissions and you don't want or cannot modify the testcase.

make make_command
    Uses make_command instead of make(1) to build the testcase.

rc code
    Sets the return code that will be considered successful. By default 0 is considered success.

timeout timeout
    Sets the timeout for the testcase after which the testcase will be aborted to timeout seconds.

nobuild
    If this option is set, the build stage and cleanup stage of the test case are not run.

runas uid
    Runs the testcase as the user identified by the given uid.

intpre
    Executes the testcase command with the argument “pre” during the pre-run command stage.

intpost
    Executes the testcase command with the argument “post” during the post-run command stage.

pre precmd
    Executes the command specified by precmd during the pre-run command stage.

pre postcmd
    Executes the command specified by postcmd during the post-run command stage.

TESTCASE EXECUTION

1.  CHDIR to the testcase directory. If it fails, the result will be set to RESULT_PREFAIL and the sysbuf buffer will provide further details.

2.  BUILD the testcase using the command specified by the make option or, if not specified, make(1), unless the nobuild option was specified. If there is an internal driver error, the result will be set to RESULT_PREFAIL, the next step to be performed will be CLEANUP and the sysbuf buffer will provide further details. If no internal error occurs, the buildbuf will contain the build log.

3.  RUN PRE COMMAND if intpre or pre are set. If there is an internal driver error, the result will be set to RESULT_PREFAIL, the next step to be performed will be CLEANUP and the sysbuf buffer will provide further details. If the pre command has a non-zero exit value, the result will be set to RESULT_PREFAIL and the next step to be performed will be CLEANUP. In this case and in the case where the command succeeds, the precmd_buf will contain the execution log.

4.  RUN TESTCASE depending on type:
    buildonly
        testcases will get their result set to RESULT_PASS at this point, since the build must have succeeded already.
    userland
        testcases will get executed in a separate child process, possibly as a different user, depending on whether the runas option was specified. The result will be set to RESULT_TIMEOUT if the timeout is reached before the testcase finishes, RESULT_SIGNALLED if the testcase dies because of an unhandled signal (often due to crashing), RESULT_NOTRUN if the testcase could not be executed, RESULT_FAIL if the testcase completes but returns failure or RESULT_PASS if the testcase completes and returns success.
    kernel
        testcases will be executed in kernel space by loading a given module and running the testcase entry point function. The result will be set to RESULT_NOTRUN if the testcase could not be executed. Otherwise the result will be set according to the kernel test case to one of RESULT_TIMEOUT, RESULT_FAIL, or RESULT_PASS.
    The output will be logged separately for stdout and stderr to the stdout_buf and stderr_buf respectively. If the result was RESULT_NOTRUN the sysbuf will contain more information.

5.  RUN POST COMMAND if intpost or post are set. If there is an internal driver error, the result will be set to RESULT_POSTFAIL, the next step to be performed will be CLEANUP and the sysbuf buffer will provide further details. If the post command has a non-zero exit value, the result will be set to RESULT_POSTFAIL and the next step to be performed will be CLEANUP. In this case and in the case where the command succeeds, the postcmd_buf will contain the execution log.

6.  CLEANUP the testcase execution using the command specified by the make option or, if not specified, make(1) with the parameter “clean”, unless the nobuild option was specified. If there is an internal driver error the sysbuf buffer will contain more information. If no internal error occurs, the cleanu_pbuf will contain the cleanup log.

FRONTENDS
The output of dfregress is in the Apple plist serialized object format. This format can be easily parsed by using proplib(3) when using C. Ruby and Python also have parsers for the plist format.

A frontend for dfregress parses the intermediate output plist into a more easily readable format such as plain text or websites.

By default dfregress ships only with the dfr2text(8) text-based frontend.

HOW TO WRITE A TESTCASE
A userland testcase is a simple program that prints some relevant information to stdout and stderr, both of which are captured by the test driver, and returns an exit value of 0 if the test passed, or any other non-zero exit value to signal a failure. The exact exit value is recorded by dfregress. All signals/exceptions not explicitly caught by the testcase will abort the execution of the testcase and the result will be set appropriately and the signal number will be recorded.

A kernel testcase is a simple kernel module that defines two methods: a run() method as well as an optional abort() method. The run() method will be run from a separate kernel thread. The testcase will need to call tbridge(9) functions to record output and to notify of testcase completion. Refer to the tbridge(9) manual page for more details.

For all testcase types the build stage is common. Every testcase should either have the nobuild option set, or have a Makefile or similar in its directory. By default dfregress assumes it is a standard make(1) Makefile. If that is not the case, the build option needs to be adjusted accordingly.

GENERAL ADVICE ON WRITING TESTCASES
Test only one thing at a time, it is not good practice to test multiple things in the same testcase as it makes it less obvious what's going on.

Keep it short, simple and well documented on what the requirements are, what the preconditions need to be, what exactly is being tested, ideally with a reference to a particular bug if that exists, and most importantly what the expected outcomes are.

Make sure your testcase doesn't depend on any other being run previously as well as that it won't hinder any other testcase from running. This effectively means that your testcase should make no assumptions as to what has been run previously unless it has a registered pre-command and that the system should be left as found before your testcase.

EXAMPLES
An example runlist can be found under test/testcases/sample.run.

Several example testcases for both userland and kernel are under test/testcases/sample.

SEE ALSO
dfr2text(8), tbridge(9)

HISTORY
The dfregress utility appeared in DragonFly 2.13.

AUTHORS
Alex Hornung