aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/tools/testing/selftests/kselftest_harness.h (follow)
AgeCommit message (Collapse)AuthorFilesLines
2020-09-17selftests/harness: Flush stdout before forkingMichael Ellerman1-0/+5
The test harness forks() a child to run each test. Both the parent and the child print to stdout using libc functions. That can lead to duplicated (or more) output if the libc buffers are not flushed before forking. It's generally not seen when running programs directly, because stdout will usually be line buffered when it's pointing to a terminal. This was noticed when running the seccomp_bpf test, eg: $ ./seccomp_bpf | tee test.log $ grep -c "TAP version 13" test.log 2 But we only expect the TAP header to appear once. It can be exacerbated using stdbuf to increase the buffer size: $ stdbuf -o 1MB ./seccomp_bpf > test.log $ grep -c "TAP version 13" test.log 13 The fix is simple, we just flush stdout & stderr before fork. Usually stderr is unbuffered, but that can be changed, so flush it as well just to be safe. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Tested-by: Max Filippov <jcmvbkbc@gmail.com> Acked-by: Shuah Khan <skhan@linuxfoundation.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-17selftests/harness: Limit step counter reportingKees Cook1-7/+2
When the selftest "step" counter grew beyond 255, non-fatal warnings were being emitted, which is noisy and pointless. There are selftests with more than 255 steps (especially those in loops, etc). Instead, just cap "steps" to 254 and do not report the saturation. Reported-by: Ralph Campbell <rcampbell@nvidia.com> Tested-by: Ralph Campbell <rcampbell@nvidia.com> Fixes: 9847d24af95c ("selftests/harness: Refactor XFAIL into SKIP") Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-17selftests/harness: Clean up kern-doc for fixturesKees Cook1-7/+8
The FIXTURE*() macro kern-doc examples had the wrong names for the C code examples associated with them. Fix those and clarify that FIXTURE_DATA() usage should be avoided. Cc: Shuah Khan <shuah@kernel.org> Cc: Jakub Kicinski <kuba@kernel.org> Fixes: 74bc7c97fa88 ("kselftest: add fixture variants") Signed-off-by: Kees Cook <keescook@chromium.org> Reviewed-by: Jakub Kicinski <kuba@kernel.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06selftests/harness: Report skip reasonKees Cook1-4/+21
Use a share memory segment to pass string information between forked test and the test runner for the skip reason. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06selftests/harness: Display signed values correctlyKees Cook1-5/+37
Since forever the harness output for signed value tests have reported unsigned values to avoid casting. Instead, actually test the variable types and perform the correct casts and choose the correct format specifiers. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06selftests/harness: Refactor XFAIL into SKIPKees Cook1-16/+48
Plumb the old XFAIL result into a TAP SKIP. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06selftests/harness: Switch to TAP outputKees Cook1-22/+30
Using the kselftest_harness.h would result in non-TAP test reporting, which didn't make much sense given that all the requirements for using the low-level API were met. Switch to using ksft_*() helpers while retaining as much of a human-readability as possible. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-04-28kselftest: add fixture variantsJakub Kicinski1-26/+122
Allow users to build parameterized variants of fixtures. If fixtures want variants, they call FIXTURE_VARIANT() to declare the structure to fill for each variant. Each fixture will be re-run for each of the variants defined by calling FIXTURE_VARIANT_ADD() with the differing parameters initializing the structure. Since tests are being re-run, additional initialization (steps, no_print) is also added. Signed-off-by: Jakub Kicinski <kuba@kernel.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-28kselftest: run tests by fixtureJakub Kicinski1-9/+14
Now that all tests have a fixture object move from a global list of tests to a list of tests per fixture. Order of tests may change as we will now group and run test fixture by fixture, rather than in declaration order. Signed-off-by: Jakub Kicinski <kuba@kernel.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-28kselftest: create fixture objectsJakub Kicinski1-13/+38
Grouping tests by fixture will allow us to parametrize test runs. Create full objects for fixtures. Add a "global" fixture for tests without a fixture. Signed-off-by: Jakub Kicinski <kuba@kernel.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-28kselftest: factor out list manipulation to a helperJakub Kicinski1-18/+24
Kees suggest to factor out the list append code to a macro, since following commits need it, which leads to code duplication. Suggested-by: Kees Cook <keescook@chromium.org> Signed-off-by: Jakub Kicinski <kuba@kernel.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-14selftests/harness: fix spelling mistake "SIGARLM" -> "SIGALRM"Colin Ian King1-3/+3
There a few identical spelling mistakes, fix these. Signed-off-by: Colin Ian King <colin.king@canonical.com> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-03-26selftests/harness: Handle timeouts cleanlyKees Cook1-2/+51
When a selftest would timeout before, the program would just fall over and no accounting of failures would be reported (i.e. it would result in an incomplete TAP report). Instead, add an explicit SIGALRM handler to cleanly catch and report the timeout. Before: [==========] Running 2 tests from 2 test cases. [ RUN ] timeout.finish [ OK ] timeout.finish [ RUN ] timeout.too_long Alarm clock After: [==========] Running 2 tests from 2 test cases. [ RUN ] timeout.finish [ OK ] timeout.finish [ RUN ] timeout.too_long timeout.too_long: Test terminated by timeout [ FAIL ] timeout.too_long [==========] 1 / 2 tests passed. [ FAILED ] Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-03-26selftests/harness: Move test child waiting logicKees Cook1-44/+49
In order to better handle timeout failures, rearrange the child waiting logic into a separate function. This is mostly a copy/paste with an indentation change. To handle pid tracking, a new field is added for the child pid. Also move the alarm() pairing into the function. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 481Thomas Gleixner1-1/+1
Based on 1 normalized pattern(s): use of this source code is governed by the gplv2 license extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 2 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Enrico Weigelt <info@metux.net> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081204.507272547@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-24selftests/harness: Allow test to configure timeoutAlexandre Belloni1-5/+12
Commit a745f7af3cbd ("selftests/harness: Add 30 second timeout per test") adds an hardcoded 30s timeout to all tests. Unfortunately, rtctest has two tests taking up to 60s. Allow for individual tests to define their own timeout. Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Reviewed-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-16selftests/harness: Add 30 second timeout per testKees Cook1-0/+2
In order to keep tests from hanging forever, this adds an alarm signal to each test run. This assumes an individual test doesn't take longer than 30 seconds. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-02-13selftests/harness: Update named initializer syntaxKees Cook1-5/+5
The harness was still using old-style GNU named initializer syntax. Fix this so Clang will stop warning: seccomp_bpf.c:2924:1: warning: use of GNU old-style field designator extension [-Wgnu-designator] ./../kselftest_harness.h:147:25: note: expanded from macro 'TEST' ^ ./../kselftest_harness.h:172:5: note: expanded from macro '__TEST_IMPL' fn: &test_name, termsig: _signal }; \ ^ Signed-off-by: Kees Cook <keescook@chromium.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com> Signed-off-by: Shuah Khan <shuah@kernel.org>
2018-12-10selftests: do not macro-expand failed assertion expressionsDmitry V. Levin1-21/+21
I've stumbled over the current macro-expand behaviour of the test harness: $ gcc -Wall -xc - <<'__EOF__' TEST(macro) { int status = 0; ASSERT_TRUE(WIFSIGNALED(status)); } TEST_HARNESS_MAIN __EOF__ $ ./a.out [==========] Running 1 tests from 1 test cases. [ RUN ] global.macro <stdin>:4:global.macro:Expected 0 (0) != (((signed char) (((status) & 0x7f) + 1) >> 1) > 0) (0) global.macro: Test terminated by assertion [ FAIL ] global.macro [==========] 0 / 1 tests passed. [ FAILED ] With this change the output of the same test looks much more comprehensible: [==========] Running 1 tests from 1 test cases. [ RUN ] global.macro <stdin>:4:global.macro:Expected 0 (0) != WIFSIGNALED(status) (0) global.macro: Test terminated by assertion [ FAIL ] global.macro [==========] 0 / 1 tests passed. [ FAILED ] The issue is very similar to the bug fixed in glibc assert(3) three years ago: https://sourceware.org/bugzilla/show_bug.cgi?id=18604 Cc: Shuah Khan <shuah@kernel.org> Cc: Kees Cook <keescook@chromium.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Will Drewry <wad@chromium.org> Cc: linux-kselftest@vger.kernel.org Signed-off-by: Dmitry V. Levin <ldv@altlinux.org> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <shuah@kernel.org>
2018-03-21selftests/seccomp: Allow get_metadata to XFAILKees Cook1-1/+25
Since seccomp_get_metadata() depends on CHECKPOINT_RESTORE, XFAIL the test if the ptrace reports it as missing. Signed-off-by: Kees Cook <keescook@chromium.org> Acked-by: Tycho Andersen <tycho@tycho.ws> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-09-05selftests: Enhance kselftest_harness.h to print which assert failedMickaël Salaün1-5/+34
When a test process is not able to write to TH_LOG_STREAM, this step mechanism enable to print the assert number which triggered the failure. This can be enabled by setting _metadata->no_print to true at the beginning of the test sequence. Update the seccomp-bpf test to return 0 if a test succeeded. This feature is needed for the Landlock tests. Signed-off-by: Mickaël Salaün <mic@digikod.net> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Kees Cook <keescook@chromium.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Will Drewry <wad@chromium.org> Link: https://lkml.kernel.org/r/CAGXu5j+D-FP8Kt9unNOqKrQJP4DYTpmgkJxWykZyrYiVPz3Y3Q@mail.gmail.com Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-07Documentation/dev-tools: Add kselftest_harness documentationMickaël Salaün1-85/+330
Add ReST metadata to kselftest_harness.h to be able to include the comments in the Sphinx documentation. Signed-off-by: Mickaël Salaün <mic@digikod.net> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kees Cook <keescook@chromium.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Will Drewry <wad@chromium.org> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-07selftests: Remove the TEST_API() wrapper from kselftest_harness.hMickaël Salaün1-205/+150
Remove the TEST_API() wrapper to expose the underlying macro arguments to the documentation tools. Signed-off-by: Mickaël Salaün <mic@digikod.net> Acked-by: Kees Cook <keescook@chromium.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kees Cook <keescook@chromium.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Will Drewry <wad@chromium.org> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-07selftests: Cosmetic renames in kselftest_harness.hMickaël Salaün1-5/+6
Keep the content consistent with the new name. Signed-off-by: Mickaël Salaün <mic@digikod.net> Acked-by: Kees Cook <keescook@chromium.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Shuah Khan <shuah@kernel.org> Cc: Will Drewry <wad@chromium.org> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-07selftests: Make test_harness.h more generally availableMickaël Salaün1-0/+535
The seccomp/test_harness.h file contains useful helpers to build tests. Moving it to the selftest directory should benefit to other test components. Keep seccomp maintainers for this file. Signed-off-by: Mickaël Salaün <mic@digikod.net> Acked-by: Kees Cook <keescook@chromium.org> Acked-by: Will Drewry <wad@chromium.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Shuah Khan <shuah@kernel.org> Link: https://lkml.kernel.org/r/CAGXu5j+8CVz8vL51DRYXqOY=xc3zuKFf=PTENe88XYHzFYidUQ@mail.gmail.com Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Copyright © 1996 – 2023 Jason A. Donenfeld. All Rights Reverse Engineered.