diff options
Diffstat (limited to 'gnu/llvm/docs/CommandGuide/lit.rst')
| -rw-r--r-- | gnu/llvm/docs/CommandGuide/lit.rst | 482 |
1 files changed, 0 insertions, 482 deletions
diff --git a/gnu/llvm/docs/CommandGuide/lit.rst b/gnu/llvm/docs/CommandGuide/lit.rst deleted file mode 100644 index e0d09ae977d..00000000000 --- a/gnu/llvm/docs/CommandGuide/lit.rst +++ /dev/null @@ -1,482 +0,0 @@ -lit - LLVM Integrated Tester -============================ - -SYNOPSIS --------- - -:program:`lit` [*options*] [*tests*] - -DESCRIPTION ------------ - -:program:`lit` is a portable tool for executing LLVM and Clang style test -suites, summarizing their results, and providing indication of failures. -:program:`lit` is designed to be a lightweight testing tool with as simple a -user interface as possible. - -:program:`lit` should be run with one or more *tests* to run specified on the -command line. Tests can be either individual test files or directories to -search for tests (see :ref:`test-discovery`). - -Each specified test will be executed (potentially in parallel) and once all -tests have been run :program:`lit` will print summary information on the number -of tests which passed or failed (see :ref:`test-status-results`). The -:program:`lit` program will execute with a non-zero exit code if any tests -fail. - -By default :program:`lit` will use a succinct progress display and will only -print summary information for test failures. See :ref:`output-options` for -options controlling the :program:`lit` progress display and output. - -:program:`lit` also includes a number of options for controlling how tests are -executed (specific features may depend on the particular test format). See -:ref:`execution-options` for more information. - -Finally, :program:`lit` also supports additional options for only running a -subset of the options specified on the command line, see -:ref:`selection-options` for more information. - -Users interested in the :program:`lit` architecture or designing a -:program:`lit` testing implementation should see :ref:`lit-infrastructure`. - -GENERAL OPTIONS ---------------- - -.. option:: -h, --help - - Show the :program:`lit` help message. - -.. option:: -j N, --threads=N - - Run ``N`` tests in parallel. By default, this is automatically chosen to - match the number of detected available CPUs. - -.. option:: --config-prefix=NAME - - Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for - test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`. - -.. option:: -D NAME[=VALUE], --param NAME[=VALUE] - - Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty - string if not given). The meaning and use of these parameters is test suite - dependent. - -.. _output-options: - -OUTPUT OPTIONS --------------- - -.. option:: -q, --quiet - - Suppress any output except for test failures. - -.. option:: -s, --succinct - - Show less output, for example don't show information on tests that pass. - -.. option:: -v, --verbose - - Show more information on test failures, for example the entire test output - instead of just the test result. - -.. option:: -vv, --echo-all-commands - - Echo all commands to stdout, as they are being executed. - This can be valuable for debugging test failures, as the last echoed command - will be the one which has failed. - :program:`lit` normally inserts a no-op command (``:`` in the case of bash) - with argument ``'RUN: at line N'`` before each command pipeline, and this - option also causes those no-op commands to be echoed to stdout to help you - locate the source line of the failed command. - This option implies ``--verbose``. - -.. option:: -a, --show-all - - Show more information about all tests, for example the entire test - commandline and output. - -.. option:: --no-progress-bar - - Do not use curses based progress bar. - -.. option:: --show-unsupported - - Show the names of unsupported tests. - -.. option:: --show-xfail - - Show the names of tests that were expected to fail. - -.. _execution-options: - -EXECUTION OPTIONS ------------------ - -.. option:: --path=PATH - - Specify an additional ``PATH`` to use when searching for executables in tests. - -.. option:: --vg - - Run individual tests under valgrind (using the memcheck tool). The - ``--error-exitcode`` argument for valgrind is used so that valgrind failures - will cause the program to exit with a non-zero status. - - When this option is enabled, :program:`lit` will also automatically provide a - "``valgrind``" feature that can be used to conditionally disable (or expect - failure in) certain tests. - -.. option:: --vg-arg=ARG - - When :option:`--vg` is used, specify an additional argument to pass to - :program:`valgrind` itself. - -.. option:: --vg-leak - - When :option:`--vg` is used, enable memory leak checks. When this option is - enabled, :program:`lit` will also automatically provide a "``vg_leak``" - feature that can be used to conditionally disable (or expect failure in) - certain tests. - -.. option:: --time-tests - - Track the wall time individual tests take to execute and includes the results - in the summary output. This is useful for determining which tests in a test - suite take the most time to execute. Note that this option is most useful - with ``-j 1``. - -.. _selection-options: - -SELECTION OPTIONS ------------------ - -.. option:: --max-tests=N - - Run at most ``N`` tests and then terminate. - -.. option:: --max-time=N - - Spend at most ``N`` seconds (approximately) running tests and then terminate. - -.. option:: --shuffle - - Run the tests in a random order. - -.. option:: --num-shards=M - - Divide the set of selected tests into ``M`` equal-sized subsets or - "shards", and run only one of them. Must be used with the - ``--run-shard=N`` option, which selects the shard to run. The environment - variable ``LIT_NUM_SHARDS`` can also be used in place of this - option. These two options provide a coarse mechanism for paritioning large - testsuites, for parallel execution on separate machines (say in a large - testing farm). - -.. option:: --run-shard=N - - Select which shard to run, assuming the ``--num-shards=M`` option was - provided. The two options must be used together, and the value of ``N`` - must be in the range ``1..M``. The environment variable - ``LIT_RUN_SHARD`` can also be used in place of this option. - -.. option:: --filter=REGEXP - - Run only those tests whose name matches the regular expression specified in - ``REGEXP``. The environment variable ``LIT_FILTER`` can be also used in place - of this option, which is especially useful in environments where the call - to ``lit`` is issued indirectly. - -ADDITIONAL OPTIONS ------------------- - -.. option:: --debug - - Run :program:`lit` in debug mode, for debugging configuration issues and - :program:`lit` itself. - -.. option:: --show-suites - - List the discovered test suites and exit. - -.. option:: --show-tests - - List all of the discovered tests and exit. - -EXIT STATUS ------------ - -:program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS -results. Otherwise, it will exit with the status 0. Other exit codes are used -for non-test related failures (for example a user error or an internal program -error). - -.. _test-discovery: - -TEST DISCOVERY --------------- - -The inputs passed to :program:`lit` can be either individual tests, or entire -directories or hierarchies of tests to run. When :program:`lit` starts up, the -first thing it does is convert the inputs into a complete list of tests to run -as part of *test discovery*. - -In the :program:`lit` model, every test must exist inside some *test suite*. -:program:`lit` resolves the inputs specified on the command line to test suites -by searching upwards from the input path until it finds a :file:`lit.cfg` or -:file:`lit.site.cfg` file. These files serve as both a marker of test suites -and as configuration files which :program:`lit` loads in order to understand -how to find and run the tests inside the test suite. - -Once :program:`lit` has mapped the inputs into test suites it traverses the -list of inputs adding tests for individual files and recursively searching for -tests in directories. - -This behavior makes it easy to specify a subset of tests to run, while still -allowing the test suite configuration to control exactly how tests are -interpreted. In addition, :program:`lit` always identifies tests by the test -suite they are in, and their relative path inside the test suite. For -appropriately configured projects, this allows :program:`lit` to provide -convenient and flexible support for out-of-tree builds. - -.. _test-status-results: - -TEST STATUS RESULTS -------------------- - -Each test ultimately produces one of the following six results: - -**PASS** - - The test succeeded. - -**XFAIL** - - The test failed, but that is expected. This is used for test formats which allow - specifying that a test does not currently work, but wish to leave it in the test - suite. - -**XPASS** - - The test succeeded, but it was expected to fail. This is used for tests which - were specified as expected to fail, but are now succeeding (generally because - the feature they test was broken and has been fixed). - -**FAIL** - - The test failed. - -**UNRESOLVED** - - The test result could not be determined. For example, this occurs when the test - could not be run, the test itself is invalid, or the test was interrupted. - -**UNSUPPORTED** - - The test is not supported in this environment. This is used by test formats - which can report unsupported tests. - -Depending on the test format tests may produce additional information about -their status (generally only for failures). See the :ref:`output-options` -section for more information. - -.. _lit-infrastructure: - -LIT INFRASTRUCTURE ------------------- - -This section describes the :program:`lit` testing architecture for users interested in -creating a new :program:`lit` testing implementation, or extending an existing one. - -:program:`lit` proper is primarily an infrastructure for discovering and running -arbitrary tests, and to expose a single convenient interface to these -tests. :program:`lit` itself doesn't know how to run tests, rather this logic is -defined by *test suites*. - -TEST SUITES -~~~~~~~~~~~ - -As described in :ref:`test-discovery`, tests are always located inside a *test -suite*. Test suites serve to define the format of the tests they contain, the -logic for finding those tests, and any additional information to run the tests. - -:program:`lit` identifies test suites as directories containing ``lit.cfg`` or -``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are -initially discovered by recursively searching up the directory hierarchy for -all the input files passed on the command line. You can use -:option:`--show-suites` to display the discovered test suites at startup. - -Once a test suite is discovered, its config file is loaded. Config files -themselves are Python modules which will be executed. When the config file is -executed, two important global variables are predefined: - -**lit_config** - - The global **lit** configuration object (a *LitConfig* instance), which defines - the builtin test formats, global configuration parameters, and other helper - routines for implementing test configurations. - -**config** - - This is the config object (a *TestingConfig* instance) for the test suite, - which the config file is expected to populate. The following variables are also - available on the *config* object, some of which must be set by the config and - others are optional or predefined: - - **name** *[required]* The name of the test suite, for use in reports and - diagnostics. - - **test_format** *[required]* The test format object which will be used to - discover and run tests in the test suite. Generally this will be a builtin test - format available from the *lit.formats* module. - - **test_source_root** The filesystem path to the test suite root. For out-of-dir - builds this is the directory that will be scanned for tests. - - **test_exec_root** For out-of-dir builds, the path to the test suite root inside - the object directory. This is where tests will be run and temporary output files - placed. - - **environment** A dictionary representing the environment to use when executing - tests in the suite. - - **suffixes** For **lit** test formats which scan directories for tests, this - variable is a list of suffixes to identify test files. Used by: *ShTest*. - - **substitutions** For **lit** test formats which substitute variables into a test - script, the list of substitutions to perform. Used by: *ShTest*. - - **unsupported** Mark an unsupported directory, all tests within it will be - reported as unsupported. Used by: *ShTest*. - - **parent** The parent configuration, this is the config object for the directory - containing the test suite, or None. - - **root** The root configuration. This is the top-most :program:`lit` configuration in - the project. - - **pipefail** Normally a test using a shell pipe fails if any of the commands - on the pipe fail. If this is not desired, setting this variable to false - makes the test fail only if the last command in the pipe fails. - - **available_features** A set of features that can be used in `XFAIL`, - `REQUIRES`, and `UNSUPPORTED` directives. - -TEST DISCOVERY -~~~~~~~~~~~~~~ - -Once test suites are located, :program:`lit` recursively traverses the source -directory (following *test_source_root*) looking for tests. When :program:`lit` -enters a sub-directory, it first checks to see if a nested test suite is -defined in that directory. If so, it loads that test suite recursively, -otherwise it instantiates a local test config for the directory (see -:ref:`local-configuration-files`). - -Tests are identified by the test suite they are contained within, and the -relative path inside that suite. Note that the relative path may not refer to -an actual file on disk; some test formats (such as *GoogleTest*) define -"virtual tests" which have a path that contains both the path to the actual -test file and a subpath to identify the virtual test. - -.. _local-configuration-files: - -LOCAL CONFIGURATION FILES -~~~~~~~~~~~~~~~~~~~~~~~~~ - -When :program:`lit` loads a subdirectory in a test suite, it instantiates a -local test configuration by cloning the configuration for the parent directory ---- the root of this configuration chain will always be a test suite. Once the -test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file -in the subdirectory. If present, this file will be loaded and can be used to -specialize the configuration for each individual directory. This facility can -be used to define subdirectories of optional tests, or to change other -configuration parameters --- for example, to change the test format, or the -suffixes which identify test files. - -PRE-DEFINED SUBSTITUTIONS -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:program:`lit` provides various patterns that can be used with the RUN command. -These are defined in TestRunner.py. The base set of substitutions are: - - ========== ============== - Macro Substitution - ========== ============== - %s source path (path to the file currently being run) - %S source dir (directory of the file currently being run) - %p same as %S - %{pathsep} path separator - %t temporary file name unique to the test - %T parent directory of %t (not unique, deprecated, do not use) - %% % - ========== ============== - -Other substitutions are provided that are variations on this base set and -further substitution patterns can be defined by each test module. See the -modules :ref:`local-configuration-files`. - -More detailed information on substitutions can be found in the -:doc:`../TestingGuide`. - -TEST RUN OUTPUT FORMAT -~~~~~~~~~~~~~~~~~~~~~~ - -The :program:`lit` output for a test run conforms to the following schema, in -both short and verbose modes (although in short mode no PASS lines will be -shown). This schema has been chosen to be relatively easy to reliably parse by -a machine (for example in buildbot log scraping), and for other tools to -generate. - -Each test result is expected to appear on a line that matches: - -.. code-block:: none - - <result code>: <test name> (<progress info>) - -where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL, -XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and -REGRESSED are also allowed. - -The ``<test name>`` field can consist of an arbitrary string containing no -newline. - -The ``<progress info>`` field can be used to report progress information such -as (1/300) or can be empty, but even when empty the parentheses are required. - -Each test result may include additional (multiline) log information in the -following format: - -.. code-block:: none - - <log delineator> TEST '(<test name>)' <trailing delineator> - ... log message ... - <log delineator> - -where ``<test name>`` should be the name of a preceding reported test, ``<log -delineator>`` is a string of "*" characters *at least* four characters long -(the recommended length is 20), and ``<trailing delineator>`` is an arbitrary -(unparsed) string. - -The following is an example of a test run output which consists of four tests A, -B, C, and D, and a log message for the failing test C: - -.. code-block:: none - - PASS: A (1 of 4) - PASS: B (2 of 4) - FAIL: C (3 of 4) - ******************** TEST 'C' FAILED ******************** - Test 'C' failed as a result of exit code 1. - ******************** - PASS: D (4 of 4) - -LIT EXAMPLE TESTS -~~~~~~~~~~~~~~~~~ - -The :program:`lit` distribution contains several example implementations of -test suites in the *ExampleTests* directory. - -SEE ALSO --------- - -valgrind(1) |