linux-dev/tools/testing/kunit, branch master

kunit: tool: Don't download risc-v opensbi firmware with wget

2022-10-07T16:16:28Z

When running a RISC-V test kernel under QEMU, we need an OpenSBI BIOS file. In the original QEMU support patchset, kunit_tool would optionally download this file from GitHub if it didn't exist, using wget. These days, it can usually be found in the distro's qemu-system-riscv package, and is located in /usr/share/qemu on all the distros I tried (Debian, Arch, OpenSUSE). Use this file, and thereby don't do any downloading in kunit_tool. In addition, we used to shell out to whatever 'wget' was in the path, which could have potentially been used to trick the developer into running another binary. By not using wget at all, we nicely sidestep this issue. Cc: Xu Panda Fixes: 87c9c1631788 ("kunit: tool: add support for QEMU") Reported-by: Zeal Robot Signed-off-by: David Gow Tested-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: rename all_test_uml.config, use it for --alltests

2022-09-30T19:22:55Z

Context: 1. all_tests_uml.config used to be UML specific back when users to manually specify CONFIG_VIRTIO_UML=y to enable CONFIG_PCI=y. 2. --alltests used allyesconfig along with a curated list of options to disable. It's only ever worked for brief periods of time and has perennially been broken due to compile issues. Now all_tests_uml.config should work across ~all architectures. Let's instead use this to implement --alltests. Note: if anyone was using all_tests_uml.config, this change breaks them. I think that's unlikely since it was added in 5.19 and was a lot to type: --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config. We could make it a symlink to the new name, but I don't think the caution is warranted here. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: remove UML specific options from all_tests_uml.config

2022-09-30T19:22:37Z

Commit 6fc3a8636a7b ("kunit: tool: Enable virtio/PCI by default on UML") made it so we enable these options by default for UML. Specifying them here is now redundant. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: add kunit.enable to enable/disable KUnit test

2022-09-30T19:17:39Z

This patch adds the kunit.enable module parameter that will need to be set to true in addition to KUNIT being enabled for KUnit tests to run. The default value is true giving backwards compatibility. However, for the production+testing use case the new config option KUNIT_DEFAULT_ENABLED can be set to N requiring the tester to opt-in by passing kunit.enable=1 to the kernel. Signed-off-by: Joe Fradley Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: make --raw_output=kunit (aka --raw_output) preserve leading spaces

2022-09-30T19:17:17Z

With $ kunit.py run --raw_output=all ... you get the raw output from the kernel, e.g. something like > TAP version 14 > 1..26 > # Subtest: time_test_cases > 1..1 > ok 1 - time64_to_tm_test_date_range > ok 1 - time_test_cases But --raw_output=kunit or equivalently --raw_output, you get > TAP version 14 > 1..26 > # Subtest: time_test_cases > 1..1 > ok 1 - time64_to_tm_test_date_range > ok 1 - time_test_cases It looks less readable in my opinion, and it also isn't "raw output." This is due to sharing code with kunit_parser.py, which wants to strip leading whitespace since it uses anchored regexes. We could update the kunit_parser.py code to tolerate leaading spaces, but this patch takes the easier way out and adds a bool flag. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: Enable virtio/PCI by default on UML

2022-07-08T17:22:29Z

There are several tests which depend on PCI, and hence need a bunch of extra options to run under UML. This makes it awkward to give configuration instructions (whether in documentation, or as part of a .kunitconfig file), as two separate, incompatible sets of config options are required for UML and "most other architectures". For non-UML architectures, it's possible to add default kconfig options via the qemu_config python files, but there's no equivalent for UML. Add a new tools/testing/kunit/configs/arch_uml.config file containing extra kconfig options to use on UML. Tested-by: José Expósito Reviewed-by: Daniel Latypov Signed-off-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Daniel Latypov Signed-off-by: Shuah Khan

kunit: tool: make --kunitconfig repeatable, blindly concat

2022-07-08T17:22:02Z

It's come up a few times that it would be useful to have --kunitconfig be repeatable [1][2]. This could be done before with a bit of shell-fu, e.g. $ find fs/ -name '.kunitconfig' -exec cat {} + | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin or equivalently: $ cat fs/ext4/.kunitconfig fs/fat/.kunitconfig | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin But this can be fairly clunky to use in practice. And having explicit support in kunit.py opens the door to having more config fragments of interest, e.g. options for PCI on UML [1], UML coverage [2], variants of tests [3]. There's another argument to be made that users can just use multiple --kconfig_add's, but this gets very clunky very fast (e.g. [2]). Note: there's a big caveat here that some kconfig options might be incompatible. We try to give a clearish error message in the simple case where the same option appears multiple times with conflicting values, but more subtle ones (e.g. mutually exclusive options) will be potentially very confusing for the user. I don't know we can do better. Note 2: if you want to combine a --kunitconfig with the default, you either have to do to specify the current build_dir > --kunitconfig=.kunit --kunitconfig=additional.config or > --kunitconfig=tools/testing/kunit/configs/default.config --kunitconifg=additional.config each of which have their downsides (former depends on --build_dir, doesn't work if you don't have a .kunitconfig yet), etc. Example with conflicting values: > $ ./tools/testing/kunit/kunit.py config --kunitconfig=lib/kunit --kunitconfig=/dev/stdin < CONFIG_KUNIT_TEST=n > CONFIG_KUNIT=m > EOF > ... > kunit_kernel.ConfigError: Multiple values specified for 2 options in kunitconfig: > CONFIG_KUNIT=y > vs from /dev/stdin > CONFIG_KUNIT=m > > CONFIG_KUNIT_TEST=y > vs from /dev/stdin > # CONFIG_KUNIT_TEST is not set [1] https://lists.freedesktop.org/archives/dri-devel/2022-June/357616.html [2] https://lore.kernel.org/linux-kselftest/CAFd5g45f3X3xF2vz2BkTHRqOC4uW6GZxtUUMaP5mwwbK8uNVtA@mail.gmail.com/ [3] https://lore.kernel.org/linux-kselftest/CANpmjNOdSy6DuO6CYZ4UxhGxqhjzx4tn0sJMbRqo2xRFv9kX6Q@mail.gmail.com/ Signed-off-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: add coverage_uml.config to enable GCOV on UML

2022-07-08T00:06:36Z

Now that kunit.py's --kunitconfig is repeatable, let's create a file to hold the various options needed to enable coverage under UML. This can be used like so: $ ./tools/testing/kunit/kunit.py run \ --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \ --make_options=CC=/usr/bin/gcc-6 which on my system is enough to get coverage working [1]. This is still a clunky command, but far better than before. [1] at the time of this commit, I get: Overall coverage rate: lines......: 11.6% (34112 of 295033 lines) functions..: 15.3% (3721 of 24368 functions) Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: refactor internal kconfig handling, allow overriding

2022-07-08T00:03:30Z

Currently, you cannot ovewrwrite what's in your kunitconfig via --kconfig_add. Nor can you override something in a qemu_config via either means. This patch makes it so we have this level of priority * --kconfig_add * kunitconfig file (the default or the one from --kunitconfig) * qemu_config The rationale for this order is that the more "dynamic" sources of kconfig options should take priority. --kconfig_add is obviously the most dynamic. And for kunitconfig, users probably tweak the file manually or specify --kunitconfig more often than they delve into qemu_config python files. And internally, we convert the kconfigs from a python list into a set or dict fairly often. We should just use a dict internally. We exposed the set transform in the past since we didn't define __eq__, so also take the chance to shore up the kunit_kconfig.Kconfig interface. Example ======= Let's consider the unrealistic example where someone would want to disable CONFIG_KUNIT. I.e. they run $ ./tools/testing/kunit/kunit.py config --kconfig_add=CONFIG_KUNIT=n Before ------ We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: # CONFIG_KUNIT is not set After ----- We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: CONFIG_KUNIT_EXAMPLE_TEST=y, CONFIG_KUNIT_TEST=y, CONFIG_KUNIT_ALL_TESTS=y Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: introduce --qemu_args

2022-07-08T00:00:05Z

Example usage: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 \ --kconfig_add=CONFIG_SMP=y --qemu_args='-smp 8' Looking in the test.log, one can see > smp: Bringing up secondary CPUs ... > .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 > smp: Brought up 1 node, 8 CPUs This flag would allow people to make tweaks like this without having to create custom qemu_config files. For consistency with --kernel_args, we allow users to repeat this argument, e.g. you can tack on a --qemu_args='-m 2048', or you could just append it to the first string ('-smp 8 -m 2048'). Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan