Merge tag 'linux-kselftest-kunit-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
Pull KUnit updates from Shuah Khan:
"Several documentation fixes, UML related cleanups, and a feature to
enable/disable KUnit tests
This includes the change to rename all_test_uml.config, and use it for
'--alltests'. Note: if anyone was using all_tests_uml.config, this
change breaks them.
This change simplifies the usage and eliminates the need to type:
--kunitconfig=tools/testing/kunit/configs/all_tests_uml.config
A simple workaround to create a symlink to the new name can solve the
problem for anyone using all_tests_uml.config.
all_tests_uml.config should work across ~all architectures"
* tag 'linux-kselftest-kunit-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest:
Documentation: Kunit: Use full path to .kunitconfig
kunit: tool: rename all_test_uml.config, use it for --alltests
kunit: tool: remove UML specific options from all_tests_uml.config
lib: stackinit: update reference to kunit-tool
lib: overflow: update reference to kunit-tool
Documentation: KUnit: update links in the index page
Documentation: KUnit: add intro to the getting-started page
Documentation: KUnit: Reword start guide for selecting tests
Documentation: KUnit: add note about mrproper in start.rst
Documentation: KUnit: avoid repeating "kunit.py run" in start.rst
Documentation: KUnit: remove duplicated docs for kunit_tool
Documentation: Kunit: Add ref for other kinds of tests
Documentation: KUnit: Fix non-uml anchor
Documentation: Kunit: Fix inconsistent titles
Documentation: kunit: fix trivial typo
kunit: no longer call module_info(test, "Y") for kunit modules
kunit: add kunit.enable to enable/disable KUnit test
kunit: tool: make --raw_output=kunit (aka --raw_output) preserve leading spaces
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index faa9f41..02ee2e2 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2432,6 +2432,12 @@
0: force disabled
1: force enabled
+ kunit.enable= [KUNIT] Enable executing KUnit tests. Requires
+ CONFIG_KUNIT to be set to be fully enabled. The
+ default value can be overridden via
+ KUNIT_DEFAULT_ENABLED.
+ Default is 1 (enabled)
+
kvm.ignore_msrs=[KVM] Ignore guest accesses to unhandled MSRs.
Default is 0 (don't ignore, but inject #GP)
diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst
index cf9e6e3..8efe792 100644
--- a/Documentation/dev-tools/kunit/architecture.rst
+++ b/Documentation/dev-tools/kunit/architecture.rst
@@ -6,8 +6,8 @@
The KUnit architecture can be divided into two parts:
-- Kernel testing library
-- kunit_tool (Command line test harness)
+- `In-Kernel Testing Framework`_
+- `kunit_tool (Command Line Test Harness)`_
In-Kernel Testing Framework
===========================
diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst
index 172e239..fae426f 100644
--- a/Documentation/dev-tools/kunit/faq.rst
+++ b/Documentation/dev-tools/kunit/faq.rst
@@ -31,13 +31,16 @@
can compile to any architecture. It compiles like just another part of the
kernel and runs when the kernel boots, or when built as a module, when the
module is loaded. However, there is infrastructure, like the KUnit Wrapper
-(``tools/testing/kunit/kunit.py``) that does not support other architectures.
+(``tools/testing/kunit/kunit.py``) that might not support some architectures
+(see :ref:`kunit-on-qemu`).
In short, yes, you can run KUnit on other architectures, but it might require
more work than using KUnit on UML.
For more information, see :ref:`kunit-on-non-uml`.
+.. _kinds-of-tests:
+
What is the difference between a unit test and other kinds of tests?
====================================================================
Most existing tests for the Linux kernel would be categorized as an integration
@@ -95,8 +98,7 @@
seeing. When tests are built-in, they will execute when the kernel boots, and
modules will automatically execute associated tests when loaded. Test results
can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
- can be parsed with ``kunit.py parse``. For more details, see "KUnit on
- non-UML architectures" in Documentation/dev-tools/kunit/usage.rst.
+ can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`.
If none of the above tricks help, you are always welcome to email any issues to
kunit-dev@googlegroups.com.
diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst
index 5952053..f5d13f1 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -13,7 +13,6 @@
run_wrapper
run_manual
usage
- kunit-tool
api/index
style
faq
@@ -29,10 +28,10 @@
unit tests within the Linux kernel. Using KUnit, you can define groups
of test cases called test suites. The tests either run on kernel boot
if built-in, or load as a module. KUnit automatically flags and reports
-failed test cases in the kernel log. The test results appear in `TAP
-(Test Anything Protocol) format <https://testanything.org/>`_. It is inspired by
-JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock (C++ unit testing
-framework).
+failed test cases in the kernel log. The test results appear in
+:doc:`KTAP (Kernel - Test Anything Protocol) format</dev-tools/ktap>`.
+It is inspired by JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock
+(C++ unit testing framework).
KUnit tests are part of the kernel, written in the C (programming)
language, and test parts of the Kernel implementation (example: a C
@@ -46,8 +45,9 @@
restricted to things exposed to user-space.
In addition, KUnit has kunit_tool, a script (``tools/testing/kunit/kunit.py``)
-that configures the Linux kernel, runs KUnit tests under QEMU or UML (`User Mode
-Linux <http://user-mode-linux.sourceforge.net/>`_), parses the test results and
+that configures the Linux kernel, runs KUnit tests under QEMU or UML
+(:doc:`User Mode Linux </virt/uml/user_mode_linux_howto_v2>`),
+parses the test results and
displays them in a user friendly manner.
Features
@@ -95,6 +95,8 @@
- Improves code quality.
- Encourages writing testable code.
+Read also :ref:`kinds-of-tests`.
+
How do I use it?
================
@@ -107,7 +109,5 @@
examples.
* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs
used for testing.
-* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper
- script.
* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and
answers.
diff --git a/Documentation/dev-tools/kunit/kunit-tool.rst b/Documentation/dev-tools/kunit/kunit-tool.rst
deleted file mode 100644
index ae52e0f..0000000
--- a/Documentation/dev-tools/kunit/kunit-tool.rst
+++ /dev/null
@@ -1,232 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0
-
-=================
-kunit_tool How-To
-=================
-
-What is kunit_tool?
-===================
-
-kunit_tool is a script (``tools/testing/kunit/kunit.py``) that aids in building
-the Linux kernel as UML (`User Mode Linux
-<http://user-mode-linux.sourceforge.net/>`_), running KUnit tests, parsing
-the test results and displaying them in a user friendly manner.
-
-kunit_tool addresses the problem of being able to run tests without needing a
-virtual machine or actual hardware with User Mode Linux. User Mode Linux is a
-Linux architecture, like ARM or x86; however, unlike other architectures it
-compiles the kernel as a standalone Linux executable that can be run like any
-other program directly inside of a host operating system. To be clear, it does
-not require any virtualization support: it is just a regular program.
-
-What is a .kunitconfig?
-=======================
-
-It's just a defconfig that kunit_tool looks for in the build directory
-(``.kunit`` by default). kunit_tool uses it to generate a .config as you might
-expect. In addition, it verifies that the generated .config contains the CONFIG
-options in the .kunitconfig; the reason it does this is so that it is easy to
-be sure that a CONFIG that enables a test actually ends up in the .config.
-
-It's also possible to pass a separate .kunitconfig fragment to kunit_tool,
-which is useful if you have several different groups of tests you wish
-to run independently, or if you want to use pre-defined test configs for
-certain subsystems.
-
-Getting Started with kunit_tool
-===============================
-
-If a kunitconfig is present at the root directory, all you have to do is:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run
-
-However, you most likely want to use it with the following options:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
-
-- ``--timeout`` sets a maximum amount of time to allow tests to run.
-- ``--jobs`` sets the number of threads to use to build the kernel.
-
-.. note::
- This command will work even without a .kunitconfig file: if no
- .kunitconfig is present, a default one will be used instead.
-
-If you wish to use a different .kunitconfig file (such as one provided for
-testing a particular subsystem), you can pass it as an option.
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig
-
-For a list of all the flags supported by kunit_tool, you can run:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run --help
-
-Configuring, Building, and Running Tests
-========================================
-
-It's also possible to run just parts of the KUnit build process independently,
-which is useful if you want to make manual changes to part of the process.
-
-A .config can be generated from a .kunitconfig by using the ``config`` argument
-when running kunit_tool:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py config
-
-Similarly, if you just want to build a KUnit kernel from the current .config,
-you can use the ``build`` argument:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py build
-
-And, if you already have a built UML kernel with built-in KUnit tests, you can
-run the kernel and display the test results with the ``exec`` argument:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py exec
-
-The ``run`` command which is discussed above is equivalent to running all three
-of these in sequence.
-
-All of these commands accept a number of optional command-line arguments. The
-``--help`` flag will give a complete list of these, or keep reading this page
-for a guide to some of the more useful ones.
-
-Parsing Test Results
-====================
-
-KUnit tests output their results in TAP (Test Anything Protocol) format.
-kunit_tool will, when running tests, parse this output and print a summary
-which is much more pleasant to read. If you wish to look at the raw test
-results in TAP format, you can pass the ``--raw_output`` argument.
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run --raw_output
-
-The raw output from test runs may contain other, non-KUnit kernel log
-lines. You can see just KUnit output with ``--raw_output=kunit``:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run --raw_output=kunit
-
-If you have KUnit results in their raw TAP format, you can parse them and print
-the human-readable summary with the ``parse`` command for kunit_tool. This
-accepts a filename for an argument, or will read from standard input.
-
-.. code-block:: bash
-
- # Reading from a file
- ./tools/testing/kunit/kunit.py parse /var/log/dmesg
- # Reading from stdin
- dmesg | ./tools/testing/kunit/kunit.py parse
-
-This is very useful if you wish to run tests in a configuration not supported
-by kunit_tool (such as on real hardware, or an unsupported architecture).
-
-Filtering Tests
-===============
-
-It's possible to run only a subset of the tests built into a kernel by passing
-a filter to the ``exec`` or ``run`` commands. For example, if you only wanted
-to run KUnit resource tests, you could use:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run 'kunit-resource*'
-
-This uses the standard glob format for wildcards.
-
-Running Tests on QEMU
-=====================
-
-kunit_tool supports running tests on QEMU as well as via UML (as mentioned
-elsewhere). The default way of running tests on QEMU requires two flags:
-
-``--arch``
- Selects a collection of configs (Kconfig as well as QEMU configs
- options, etc) that allow KUnit tests to be run on the specified
- architecture in a minimal way; this is usually not much slower than
- using UML. The architecture argument is the same as the name of the
- option passed to the ``ARCH`` variable used by Kbuild. Not all
- architectures are currently supported by this flag, but can be handled
- by the ``--qemu_config`` discussed later. If ``um`` is passed (or this
- this flag is ignored) the tests will run via UML. Non-UML architectures,
- e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU.
-
-``--cross_compile``
- Specifies the use of a toolchain by Kbuild. The argument passed here is
- the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a
- reminder this will be the prefix for the toolchain binaries such as gcc
- for example ``sparc64-linux-gnu-`` if you have the sparc toolchain
- installed on your system, or
- ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-``
- if you have downloaded the microblaze toolchain from the 0-day website
- to a directory in your home directory called ``toolchains``.
-
-In many cases it is likely that you may want to run an architecture which is
-not supported by the ``--arch`` flag, or you may want to just run KUnit tests
-on QEMU using a non-default configuration. For this use case, you can write
-your own QemuConfig. These QemuConfigs are written in Python. They must have an
-import line ``from ..qemu_config import QemuArchParams`` at the top of the file
-and the file must contain a variable called ``QEMU_ARCH`` that has an instance
-of ``QemuArchParams`` assigned to it. An example can be seen in
-``tools/testing/kunit/qemu_configs/x86_64.py``.
-
-Once you have a QemuConfig you can pass it into kunit_tool using the
-``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we
-were to do this with the ``x86_64.py`` example from above, the invocation would
-look something like this:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run \
- --timeout=60 \
- --jobs=12 \
- --qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
-
-Other Useful Options
-====================
-
-kunit_tool has a number of other command-line arguments which can be useful
-when adapting it to fit your environment or needs.
-
-Some of the more useful ones are:
-
-``--help``
- Lists all of the available options. Note that different commands
- (``config``, ``build``, ``run``, etc) will have different supported
- options. Place ``--help`` before the command to list common options,
- and after the command for options specific to that command.
-
-``--build_dir``
- Specifies the build directory that kunit_tool will use. This is where
- the .kunitconfig file is located, as well as where the .config and
- compiled kernel will be placed. Defaults to ``.kunit``.
-
-``--make_options``
- Specifies additional options to pass to ``make`` when compiling a
- kernel (with the ``build`` or ``run`` commands). For example, to enable
- compiler warnings, you can pass ``--make_options W=1``.
-
-``--alltests``
- Builds a UML kernel with all config options enabled using ``make
- allyesconfig``. This allows you to run as many tests as is possible,
- but is very slow and prone to breakage as new options are added or
- modified. In most cases, enabling all tests which have satisfied
- dependencies by adding ``CONFIG_KUNIT_ALL_TESTS=1`` to your
- .kunitconfig is preferable.
-
-There are several other options (and new ones are often added), so do check
-``--help`` if you're looking for something not mentioned here.
diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst
index cce2031..6b33caf 100644
--- a/Documentation/dev-tools/kunit/run_wrapper.rst
+++ b/Documentation/dev-tools/kunit/run_wrapper.rst
@@ -1,8 +1,8 @@
.. SPDX-License-Identifier: GPL-2.0
-=========================
-Run Tests with kunit_tool
-=========================
+=============================
+Running tests with kunit_tool
+=============================
We can either run KUnit tests using kunit_tool or can run tests
manually, and then use kunit_tool to parse the results. To run tests
@@ -22,7 +22,7 @@
.. code-block::
- Generating .config...
+ Configuring KUnit Kernel ...
Building KUnit kernel...
Starting KUnit kernel...
@@ -30,7 +30,7 @@
.. code-block::
- ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all
+ ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
- ``--timeout`` sets a maximum amount of time for tests to run.
- ``--jobs`` sets the number of threads to build the kernel.
@@ -58,8 +58,8 @@
./tools/testing/kunit/kunit.py run --help
-Create a ``.kunitconfig`` File
-===============================
+Creating a ``.kunitconfig`` file
+================================
If we want to run a specific set of tests (rather than those listed
in the KUnit ``defconfig``), we can provide Kconfig options in the
@@ -98,8 +98,8 @@
The build dir needs to be set for ``make menuconfig`` to
work, therefore by default use ``make O=.kunit menuconfig``.
-Configure, Build, and Run Tests
-===============================
+Configuring, building, and running tests
+========================================
If we want to make manual changes to the KUnit build process, we
can run part of the KUnit build process independently.
@@ -125,11 +125,11 @@
./tools/testing/kunit/kunit.py exec
-The ``run`` command discussed in section: **Run Tests with kunit_tool**,
+The ``run`` command discussed in section: **Running tests with kunit_tool**,
is equivalent to running the above three commands in sequence.
-Parse Test Results
-==================
+Parsing test results
+====================
KUnit tests output displays results in TAP (Test Anything Protocol)
format. When running tests, kunit_tool parses this output and prints
@@ -152,8 +152,8 @@
# Reading from stdin
dmesg | ./tools/testing/kunit/kunit.py parse
-Run Selected Test Suites
-========================
+Filtering tests
+===============
By passing a bash style glob filter to the ``exec`` or ``run``
commands, we can run a subset of the tests built into a kernel . For
@@ -165,8 +165,10 @@
This uses the standard glob format with wildcard characters.
-Run Tests on qemu
-=================
+.. _kunit-on-qemu:
+
+Running tests on QEMU
+=====================
kunit_tool supports running tests on qemu as well as
via UML. To run tests on qemu, by default it requires two flags:
@@ -229,8 +231,8 @@
--jobs=12 \
--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
-Command-Line Arguments
-======================
+Running command-line arguments
+==============================
kunit_tool has a number of other command-line arguments which can
be useful for our test environment. Below are the most commonly used
diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst
index 867a4bb..f4f504f 100644
--- a/Documentation/dev-tools/kunit/start.rst
+++ b/Documentation/dev-tools/kunit/start.rst
@@ -4,6 +4,10 @@
Getting Started
===============
+This page contains an overview of the kunit_tool and KUnit framework,
+teaching how to run existing tests and then how to write a simple test case,
+and covers common problems users face when using KUnit for the first time.
+
Installing Dependencies
=======================
KUnit has the same dependencies as the Linux kernel. As long as you can
@@ -19,30 +23,53 @@
./tools/testing/kunit/kunit.py run
-For more information on this wrapper, see:
+.. note ::
+ You may see the following error:
+ "The source tree is not clean, please run 'make ARCH=um mrproper'"
+
+ This happens because internally kunit.py specifies ``.kunit``
+ (default option) as the build directory in the command ``make O=output/dir``
+ through the argument ``--build_dir``. Hence, before starting an
+ out-of-tree build, the source tree must be clean.
+
+ There is also the same caveat mentioned in the "Build directory for
+ the kernel" section of the :doc:`admin-guide </admin-guide/README>`,
+ that is, its use, it must be used for all invocations of ``make``.
+ The good news is that it can indeed be solved by running
+ ``make ARCH=um mrproper``, just be aware that this will delete the
+ current configuration and all generated files.
+
+If everything worked correctly, you should see the following:
+
+.. code-block::
+
+ Configuring KUnit Kernel ...
+ Building KUnit Kernel ...
+ Starting KUnit Kernel ...
+
+The tests will pass or fail.
+
+.. note ::
+ Because it is building a lot of sources for the first time,
+ the ``Building KUnit Kernel`` step may take a while.
+
+For detailed information on this wrapper, see:
Documentation/dev-tools/kunit/run_wrapper.rst.
-Creating a ``.kunitconfig``
----------------------------
+Selecting which tests to run
+----------------------------
-By default, kunit_tool runs a selection of tests. However, you can specify which
-unit tests to run by creating a ``.kunitconfig`` file with kernel config options
-that enable only a specific set of tests and their dependencies.
-The ``.kunitconfig`` file contains a list of kconfig options which are required
-to run the desired targets. The ``.kunitconfig`` also contains any other test
-specific config options, such as test dependencies. For example: the
-``FAT_FS`` tests - ``FAT_KUNIT_TEST``, depends on
-``FAT_FS``. ``FAT_FS`` can be enabled by selecting either ``MSDOS_FS``
-or ``VFAT_FS``. To run ``FAT_KUNIT_TEST``, the ``.kunitconfig`` has:
+By default, kunit_tool runs all tests reachable with minimal configuration,
+that is, using default values for most of the kconfig options. However,
+you can select which tests to run by:
-.. code-block:: none
+- `Customizing Kconfig`_ used to compile the kernel, or
+- `Filtering tests by name`_ to select specifically which compiled tests to run.
- CONFIG_KUNIT=y
- CONFIG_MSDOS_FS=y
- CONFIG_FAT_KUNIT_TEST=y
-
-1. A good starting point for the ``.kunitconfig`` is the KUnit default config.
- You can generate it by running:
+Customizing Kconfig
+~~~~~~~~~~~~~~~~~~~
+A good starting point for the ``.kunitconfig`` is the KUnit default config.
+If you didn't run ``kunit.py run`` yet, you can generate it by running:
.. code-block:: bash
@@ -54,48 +81,69 @@
``.kunitconfig`` lives in the ``--build_dir`` used by kunit.py, which is
``.kunit`` by default.
-.. note ::
- You may want to remove CONFIG_KUNIT_ALL_TESTS from the ``.kunitconfig`` as
- it will enable a number of additional tests that you may not want.
-
-2. You can then add any other Kconfig options, for example:
-
-.. code-block:: none
-
- CONFIG_LIST_KUNIT_TEST=y
-
Before running the tests, kunit_tool ensures that all config options
set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn
you if you have not included dependencies for the options used.
-.. note ::
- If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the
+There are many ways to customize the configurations:
+
+a. Edit ``.kunit/.kunitconfig``. The file should contain the list of kconfig
+ options required to run the desired tests, including their dependencies.
+ You may want to remove CONFIG_KUNIT_ALL_TESTS from the ``.kunitconfig`` as
+ it will enable a number of additional tests that you may not want.
+ If you need to run on an architecture other than UML see :ref:`kunit-on-qemu`.
+
+b. Enable additional kconfig options on top of ``.kunit/.kunitconfig``.
+ For example, to include the kernel's linked-list test you can run::
+
+ ./tools/testing/kunit/kunit.py run \
+ --kconfig_add CONFIG_LIST_KUNIT_TEST=y
+
+c. Provide the path of one or more .kunitconfig files from the tree.
+ For example, to run only ``FAT_FS`` and ``EXT4`` tests you can run::
+
+ ./tools/testing/kunit/kunit.py run \
+ --kunitconfig ./fs/fat/.kunitconfig \
+ --kunitconfig ./fs/ext4/.kunitconfig
+
+d. If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the
``.config`` file. But you can edit the ``.config`` file directly or with
tools like ``make menuconfig O=.kunit``. As long as its a superset of
``.kunitconfig``, kunit.py won't overwrite your changes.
-Running Tests (KUnit Wrapper)
------------------------------
-1. To make sure that everything is set up correctly, invoke the Python
- wrapper from your kernel repository:
-
-.. code-block:: bash
-
- ./tools/testing/kunit/kunit.py run
-
-If everything worked correctly, you should see the following:
-
-.. code-block::
-
- Generating .config ...
- Building KUnit Kernel ...
- Starting KUnit Kernel ...
-
-The tests will pass or fail.
.. note ::
- Because it is building a lot of sources for the first time, the
- ``Building KUnit kernel`` may take a while.
+
+ To save a .kunitconfig after finding a satisfactory configuration::
+
+ make savedefconfig O=.kunit
+ cp .kunit/defconfig .kunit/.kunitconfig
+
+Filtering tests by name
+~~~~~~~~~~~~~~~~~~~~~~~
+If you want to be more specific than Kconfig can provide, it is also possible
+to select which tests to execute at boot-time by passing a glob filter
+(read instructions regarding the pattern in the manpage :manpage:`glob(7)`).
+If there is a ``"."`` (period) in the filter, it will be interpreted as a
+separator between the name of the test suite and the test case,
+otherwise, it will be interpreted as the name of the test suite.
+For example, let's assume we are using the default config:
+
+a. inform the name of a test suite, like ``"kunit_executor_test"``,
+ to run every test case it contains::
+
+ ./tools/testing/kunit/kunit.py run "kunit_executor_test"
+
+b. inform the name of a test case prefixed by its test suite,
+ like ``"example.example_simple_test"``, to run specifically that test case::
+
+ ./tools/testing/kunit/kunit.py run "example.example_simple_test"
+
+c. use wildcard characters (``*?[``) to run any test case that matches the pattern,
+ like ``"*.*64*"`` to run test cases containing ``"64"`` in the name inside
+ any test suite::
+
+ ./tools/testing/kunit/kunit.py run "*.*64*"
Running Tests without the KUnit Wrapper
=======================================
@@ -217,7 +265,7 @@
obj-$(CONFIG_MISC_EXAMPLE_TEST) += example_test.o
-4. Add the following lines to ``.kunitconfig``:
+4. Add the following lines to ``.kunit/.kunitconfig``:
.. code-block:: none
@@ -254,7 +302,5 @@
examples.
* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs
used for testing.
-* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper
- script.
* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and
answers.
diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst
index 44158ee..2737863 100644
--- a/Documentation/dev-tools/kunit/usage.rst
+++ b/Documentation/dev-tools/kunit/usage.rst
@@ -165,6 +165,8 @@
For more information, see Documentation/dev-tools/kunit/api/test.rst.
+.. _kunit-on-non-uml:
+
Writing Tests For Other Architectures
-------------------------------------
@@ -544,8 +546,6 @@
{}
};
-.. _kunit-on-non-uml:
-
Exiting Early on Failed Expectations
------------------------------------
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 840a2c3..20cc477 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -228,6 +228,8 @@ static inline void kunit_set_failure(struct kunit *test)
WRITE_ONCE(test->status, KUNIT_FAILURE);
}
+bool kunit_enabled(void);
+
void kunit_init_test(struct kunit *test, const char *name, char *log);
int kunit_run_tests(struct kunit_suite *suite);
@@ -251,7 +253,6 @@ static inline int kunit_run_all_tests(void)
#endif /* IS_BUILTIN(CONFIG_KUNIT) */
#define __kunit_test_suites(unique_array, ...) \
- MODULE_INFO(test, "Y"); \
static struct kunit_suite *unique_array[] \
__aligned(sizeof(struct kunit_suite *)) \
__used __section(".kunit_test_suites") = { __VA_ARGS__ }
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 0b5dfb0..626719b 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -59,4 +59,15 @@
If unsure, say N.
+config KUNIT_DEFAULT_ENABLED
+ bool "Default value of kunit.enable"
+ default y
+ help
+ Sets the default value of kunit.enable. If set to N then KUnit
+ tests will not execute unless kunit.enable=1 is passed to the
+ kernel command line.
+
+ In most cases this should be left as Y. Only if additional opt-in
+ behavior is needed should this be set to N.
+
endif # KUNIT
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 5e22332..9bbc422 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -190,6 +190,10 @@ int kunit_run_all_tests(void)
{
struct suite_set suite_set = {__kunit_suites_start, __kunit_suites_end};
int err = 0;
+ if (!kunit_enabled()) {
+ pr_info("kunit: disabled\n");
+ goto out;
+ }
if (filter_glob_param) {
suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index b73d5bb..1e54373 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -55,6 +55,17 @@ EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
#endif
/*
+ * Enable KUnit tests to run.
+ */
+#ifdef CONFIG_KUNIT_DEFAULT_ENABLED
+static bool enable_param = true;
+#else
+static bool enable_param;
+#endif
+module_param_named(enable, enable_param, bool, 0);
+MODULE_PARM_DESC(enable, "Enable KUnit tests");
+
+/*
* KUnit statistic mode:
* 0 - disabled
* 1 - only when there is more than one subtest
@@ -586,10 +597,20 @@ static void kunit_init_suite(struct kunit_suite *suite)
suite->suite_init_err = 0;
}
+bool kunit_enabled(void)
+{
+ return enable_param;
+}
+
int __kunit_test_suites_init(struct kunit_suite * const * const suites, int num_suites)
{
unsigned int i;
+ if (!kunit_enabled() && num_suites > 0) {
+ pr_info("kunit: disabled\n");
+ return 0;
+ }
+
for (i = 0; i < num_suites; i++) {
kunit_init_suite(suites[i]);
kunit_run_tests(suites[i]);
@@ -607,6 +628,9 @@ void __kunit_test_suites_exit(struct kunit_suite **suites, int num_suites)
{
unsigned int i;
+ if (!kunit_enabled())
+ return;
+
for (i = 0; i < num_suites; i++)
kunit_exit_suite(suites[i]);
diff --git a/lib/overflow_kunit.c b/lib/overflow_kunit.c
index f385ca6..5369634 100644
--- a/lib/overflow_kunit.c
+++ b/lib/overflow_kunit.c
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: GPL-2.0 OR MIT
/*
* Test cases for arithmetic overflow checks. See:
- * https://www.kernel.org/doc/html/latest/dev-tools/kunit/kunit-tool.html#configuring-building-and-running-tests
+ * "Running tests with kunit_tool" at Documentation/dev-tools/kunit/start.rst
* ./tools/testing/kunit/kunit.py run overflow [--raw_output]
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
diff --git a/lib/stackinit_kunit.c b/lib/stackinit_kunit.c
index 35c69aa..4591d6c 100644
--- a/lib/stackinit_kunit.c
+++ b/lib/stackinit_kunit.c
@@ -3,7 +3,7 @@
* Test cases for compiler-based stack variable zeroing via
* -ftrivial-auto-var-init={zero,pattern} or CONFIG_GCC_PLUGIN_STRUCTLEAK*.
* For example, see:
- * https://www.kernel.org/doc/html/latest/dev-tools/kunit/kunit-tool.html#configuring-building-and-running-tests
+ * "Running tests with kunit_tool" at Documentation/dev-tools/kunit/start.rst
* ./tools/testing/kunit/kunit.py run stackinit [--raw_output] \
* --make_option LLVM=1 \
* --kconfig_add CONFIG_INIT_STACK_ALL_ZERO=y
diff --git a/tools/testing/kunit/configs/all_tests_uml.config b/tools/testing/kunit/configs/all_tests.config
similarity index 93%
rename from tools/testing/kunit/configs/all_tests_uml.config
rename to tools/testing/kunit/configs/all_tests.config
index bdee36b..f990cbb 100644
--- a/tools/testing/kunit/configs/all_tests_uml.config
+++ b/tools/testing/kunit/configs/all_tests.config
@@ -16,8 +16,6 @@
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
-CONFIG_VIRTIO_UML=y
-CONFIG_UML_PCI_OVER_VIRTIO=y
CONFIG_PCI=y
CONFIG_USB4=y
diff --git a/tools/testing/kunit/configs/broken_on_uml.config b/tools/testing/kunit/configs/broken_on_uml.config
deleted file mode 100644
index 6908700..0000000
--- a/tools/testing/kunit/configs/broken_on_uml.config
+++ /dev/null
@@ -1,44 +0,0 @@
-# These are currently broken on UML and prevent allyesconfig from building
-# CONFIG_STATIC_LINK is not set
-# CONFIG_UML_NET_VECTOR is not set
-# CONFIG_UML_NET_VDE is not set
-# CONFIG_UML_NET_PCAP is not set
-# CONFIG_NET_PTP_CLASSIFY is not set
-# CONFIG_IP_VS is not set
-# CONFIG_BRIDGE_EBT_BROUTE is not set
-# CONFIG_BRIDGE_EBT_T_FILTER is not set
-# CONFIG_BRIDGE_EBT_T_NAT is not set
-# CONFIG_MTD_NAND_CADENCE is not set
-# CONFIG_MTD_NAND_NANDSIM is not set
-# CONFIG_BLK_DEV_NULL_BLK is not set
-# CONFIG_BLK_DEV_RAM is not set
-# CONFIG_SCSI_DEBUG is not set
-# CONFIG_NET_VENDOR_XILINX is not set
-# CONFIG_NULL_TTY is not set
-# CONFIG_PTP_1588_CLOCK is not set
-# CONFIG_PINCTRL_EQUILIBRIUM is not set
-# CONFIG_DMABUF_SELFTESTS is not set
-# CONFIG_COMEDI is not set
-# CONFIG_XIL_AXIS_FIFO is not set
-# CONFIG_EXFAT_FS is not set
-# CONFIG_STM_DUMMY is not set
-# CONFIG_FSI_MASTER_ASPEED is not set
-# CONFIG_JFS_FS is not set
-# CONFIG_UBIFS_FS is not set
-# CONFIG_CRAMFS is not set
-# CONFIG_CRYPTO_DEV_SAFEXCEL is not set
-# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set
-# CONFIG_KCOV is not set
-# CONFIG_LKDTM is not set
-# CONFIG_REED_SOLOMON_TEST is not set
-# CONFIG_TEST_RHASHTABLE is not set
-# CONFIG_TEST_MEMINIT is not set
-# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
-# CONFIG_DEBUG_INFO_BTF is not set
-# CONFIG_PTP_1588_CLOCK_INES is not set
-# CONFIG_QCOM_CPR is not set
-# CONFIG_RESET_BRCMSTB_RESCAL is not set
-# CONFIG_RESET_INTEL_GW is not set
-# CONFIG_ADI_AXI_ADC is not set
-# CONFIG_DEBUG_PAGEALLOC is not set
-# CONFIG_PAGE_POISONING is not set
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e132b06..4d4663f 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -44,7 +44,6 @@
@dataclass
class KunitBuildRequest(KunitConfigRequest):
jobs: int
- alltests: bool
@dataclass
class KunitParseRequest:
@@ -55,7 +54,6 @@
class KunitExecRequest(KunitParseRequest):
build_dir: str
timeout: int
- alltests: bool
filter_glob: str
kernel_args: Optional[List[str]]
run_isolated: Optional[str]
@@ -90,8 +88,7 @@
stdout.print_with_timestamp('Building KUnit Kernel ...')
build_start = time.time()
- success = linux.build_kernel(request.alltests,
- request.jobs,
+ success = linux.build_kernel(request.jobs,
request.build_dir,
request.make_options)
build_end = time.time()
@@ -118,7 +115,7 @@
args.extend(request.kernel_args)
output = linux.run_kernel(args=args,
- timeout=None if request.alltests else request.timeout,
+ timeout=request.timeout,
filter_glob=request.filter_glob,
build_dir=request.build_dir)
lines = kunit_parser.extract_tap_lines(output)
@@ -165,7 +162,7 @@
test_start = time.time()
run_result = linux.run_kernel(
args=request.kernel_args,
- timeout=None if request.alltests else request.timeout,
+ timeout=request.timeout,
filter_glob=filter_glob,
build_dir=request.build_dir)
@@ -206,7 +203,7 @@
if request.raw_output == 'all':
pass
elif request.raw_output == 'kunit':
- output = kunit_parser.extract_tap_lines(output)
+ output = kunit_parser.extract_tap_lines(output, lstrip=False)
for line in output:
print(line.rstrip())
@@ -288,7 +285,7 @@
help='X=Y make option, can be repeated.',
action='append', metavar='X=Y')
parser.add_argument('--alltests',
- help='Run all KUnit tests through allyesconfig',
+ help='Run all KUnit tests via tools/testing/kunit/configs/all_tests.config',
action='store_true')
parser.add_argument('--kunitconfig',
help='Path to Kconfig fragment that enables KUnit tests.'
@@ -381,8 +378,14 @@
for arg in cli_args.qemu_args:
qemu_args.extend(shlex.split(arg))
+ kunitconfigs = cli_args.kunitconfig if cli_args.kunitconfig else []
+ if cli_args.alltests:
+ # Prepend so user-specified options take prio if we ever allow
+ # --kunitconfig options to have differing options.
+ kunitconfigs = [kunit_kernel.ALL_TESTS_CONFIG_PATH] + kunitconfigs
+
return kunit_kernel.LinuxSourceTree(cli_args.build_dir,
- kunitconfig_paths=cli_args.kunitconfig,
+ kunitconfig_paths=kunitconfigs,
kconfig_add=cli_args.kconfig_add,
arch=cli_args.arch,
cross_compile=cli_args.cross_compile,
@@ -441,7 +444,6 @@
request = KunitRequest(build_dir=cli_args.build_dir,
make_options=cli_args.make_options,
jobs=cli_args.jobs,
- alltests=cli_args.alltests,
raw_output=cli_args.raw_output,
json=cli_args.json,
timeout=cli_args.timeout,
@@ -469,8 +471,7 @@
linux = tree_from_args(cli_args)
request = KunitBuildRequest(build_dir=cli_args.build_dir,
make_options=cli_args.make_options,
- jobs=cli_args.jobs,
- alltests=cli_args.alltests)
+ jobs=cli_args.jobs)
result = config_and_build_tests(linux, request)
stdout.print_with_timestamp((
'Elapsed time: %.3fs\n') % (
@@ -483,7 +484,6 @@
build_dir=cli_args.build_dir,
json=cli_args.json,
timeout=cli_args.timeout,
- alltests=cli_args.alltests,
filter_glob=cli_args.filter_glob,
kernel_args=cli_args.kernel_args,
run_isolated=cli_args.run_isolated)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index f5c26ea8..53e90c3 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -25,7 +25,7 @@
KUNITCONFIG_PATH = '.kunitconfig'
OLD_KUNITCONFIG_PATH = 'last_used_kunitconfig'
DEFAULT_KUNITCONFIG_PATH = 'tools/testing/kunit/configs/default.config'
-BROKEN_ALLCONFIG_PATH = 'tools/testing/kunit/configs/broken_on_uml.config'
+ALL_TESTS_CONFIG_PATH = 'tools/testing/kunit/configs/all_tests.config'
UML_KCONFIG_PATH = 'tools/testing/kunit/configs/arch_uml.config'
OUTFILE_PATH = 'test.log'
ABS_TOOL_PATH = os.path.abspath(os.path.dirname(__file__))
@@ -57,9 +57,6 @@
def make_arch_config(self, base_kunitconfig: kunit_config.Kconfig) -> kunit_config.Kconfig:
return base_kunitconfig
- def make_allyesconfig(self, build_dir: str, make_options) -> None:
- raise ConfigError('Only the "um" arch is supported for alltests')
-
def make_olddefconfig(self, build_dir: str, make_options) -> None:
command = ['make', 'ARCH=' + self._linux_arch, 'O=' + build_dir, 'olddefconfig']
if self._cross_compile:
@@ -144,26 +141,6 @@
kconfig.merge_in_entries(base_kunitconfig)
return kconfig
- def make_allyesconfig(self, build_dir: str, make_options) -> None:
- stdout.print_with_timestamp(
- 'Enabling all CONFIGs for UML...')
- command = ['make', 'ARCH=um', 'O=' + build_dir, 'allyesconfig']
- if make_options:
- command.extend(make_options)
- process = subprocess.Popen(
- command,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.STDOUT)
- process.wait()
- stdout.print_with_timestamp(
- 'Disabling broken configs to run KUnit tests...')
-
- with open(get_kconfig_path(build_dir), 'a') as config:
- with open(BROKEN_ALLCONFIG_PATH, 'r') as disable:
- config.write(disable.read())
- stdout.print_with_timestamp(
- 'Starting Kernel with all configs takes a few minutes...')
-
def start(self, params: List[str], build_dir: str) -> subprocess.Popen:
"""Runs the Linux UML binary. Must be named 'linux'."""
linux_bin = os.path.join(build_dir, 'linux')
@@ -343,10 +320,8 @@
os.remove(kconfig_path)
return self.build_config(build_dir, make_options)
- def build_kernel(self, alltests, jobs, build_dir: str, make_options) -> bool:
+ def build_kernel(self, jobs, build_dir: str, make_options) -> bool:
try:
- if alltests:
- self._ops.make_allyesconfig(build_dir, make_options)
self._ops.make_olddefconfig(build_dir, make_options)
self._ops.make(jobs, build_dir, make_options)
except (ConfigError, BuildError) as e:
@@ -359,6 +334,7 @@
args = []
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
+ args.append('kunit.enable=1')
process = self._ops.start(args, build_dir)
assert process.stdout is not None # tell mypy it's set
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 12d3ec7..1ae873e 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -218,7 +218,7 @@
KTAP_END = re.compile('(List of all partitions:|'
'Kernel panic - not syncing: VFS:|reboot: System halted)')
-def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream:
+def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream:
"""Extracts KTAP lines from the kernel output."""
def isolate_ktap_output(kernel_output: Iterable[str]) \
-> Iterator[Tuple[int, str]]:
@@ -244,9 +244,11 @@
# stop extracting KTAP lines
break
elif started:
- # remove prefix and any indention and yield
- # line with line number
- line = line[prefix_len:].lstrip()
+ # remove the prefix and optionally any leading
+ # whitespace. Our parsing logic relies on this.
+ line = line[prefix_len:]
+ if lstrip:
+ line = line.lstrip()
yield line_num, line
return LineStream(lines=isolate_ktap_output(kernel_output))
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index 446ac43..e2cd2cc 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -549,7 +549,7 @@
def test_build_passes_args_pass(self):
kunit.main(['build'])
self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
- self.linux_source_mock.build_kernel.assert_called_once_with(False, kunit.get_default_jobs(), '.kunit', None)
+ self.linux_source_mock.build_kernel.assert_called_once_with(kunit.get_default_jobs(), '.kunit', None)
self.assertEqual(self.linux_source_mock.run_kernel.call_count, 0)
def test_exec_passes_args_pass(self):
@@ -664,7 +664,7 @@
build_dir = '.kunit'
jobs = kunit.get_default_jobs()
kunit.main(['build', '--build_dir', build_dir])
- self.linux_source_mock.build_kernel.assert_called_once_with(False, jobs, build_dir, None)
+ self.linux_source_mock.build_kernel.assert_called_once_with(jobs, build_dir, None)
def test_exec_builddir(self):
build_dir = '.kunit'
@@ -695,6 +695,18 @@
qemu_config_path=None,
extra_qemu_args=[])
+ def test_config_alltests(self):
+ kunit.main(['config', '--kunitconfig=mykunitconfig', '--alltests'])
+ # Just verify that we parsed and initialized it correctly here.
+ self.mock_linux_init.assert_called_once_with('.kunit',
+ kunitconfig_paths=[kunit_kernel.ALL_TESTS_CONFIG_PATH, 'mykunitconfig'],
+ kconfig_add=None,
+ arch='um',
+ cross_compile=None,
+ qemu_config_path=None,
+ extra_qemu_args=[])
+
+
@mock.patch.object(kunit_kernel, 'LinuxSourceTree')
def test_run_multiple_kunitconfig(self, mock_linux_init):
mock_linux_init.return_value = self.linux_source_mock
@@ -712,7 +724,7 @@
kunit.main(['run', '--kconfig_add=CONFIG_KASAN=y', '--kconfig_add=CONFIG_KCSAN=y'])
# Just verify that we parsed and initialized it correctly here.
self.mock_linux_init.assert_called_once_with('.kunit',
- kunitconfig_paths=None,
+ kunitconfig_paths=[],
kconfig_add=['CONFIG_KASAN=y', 'CONFIG_KCSAN=y'],
arch='um',
cross_compile=None,
@@ -723,7 +735,7 @@
kunit.main(['run', '--arch=x86_64', '--qemu_args', '-m 2048'])
# Just verify that we parsed and initialized it correctly here.
self.mock_linux_init.assert_called_once_with('.kunit',
- kunitconfig_paths=None,
+ kunitconfig_paths=[],
kconfig_add=None,
arch='x86_64',
cross_compile=None,
@@ -742,7 +754,7 @@
self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want
got = kunit._list_tests(self.linux_source_mock,
- kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*', None, 'suite'))
+ kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite'))
self.assertEqual(got, want)
# Should respect the user's filter glob when listing tests.
@@ -757,7 +769,7 @@
# Should respect the user's filter glob when listing tests.
mock_tests.assert_called_once_with(mock.ANY,
- kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*.test*', None, 'suite'))
+ kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite'))
self.linux_source_mock.run_kernel.assert_has_calls([
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300),
mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300),
@@ -770,7 +782,7 @@
# Should respect the user's filter glob when listing tests.
mock_tests.assert_called_once_with(mock.ANY,
- kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*', None, 'test'))
+ kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test'))
self.linux_source_mock.run_kernel.assert_has_calls([
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300),
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300),
