| .. SPDX-License-Identifier: GPL-2.0 |
| |
| .. _cpumasks-header-label: |
| |
| ================== |
| BPF cpumask kfuncs |
| ================== |
| |
| 1. Introduction |
| =============== |
| |
| ``struct cpumask`` is a bitmap data structure in the kernel whose indices |
| reflect the CPUs on the system. Commonly, cpumasks are used to track which CPUs |
| a task is affinitized to, but they can also be used to e.g. track which cores |
| are associated with a scheduling domain, which cores on a machine are idle, |
| etc. |
| |
| BPF provides programs with a set of :ref:`kfuncs-header-label` that can be |
| used to allocate, mutate, query, and free cpumasks. |
| |
| 2. BPF cpumask objects |
| ====================== |
| |
| There are two different types of cpumasks that can be used by BPF programs. |
| |
| 2.1 ``struct bpf_cpumask *`` |
| ---------------------------- |
| |
| ``struct bpf_cpumask *`` is a cpumask that is allocated by BPF, on behalf of a |
| BPF program, and whose lifecycle is entirely controlled by BPF. These cpumasks |
| are RCU-protected, can be mutated, can be used as kptrs, and can be safely cast |
| to a ``struct cpumask *``. |
| |
| 2.1.1 ``struct bpf_cpumask *`` lifecycle |
| ---------------------------------------- |
| |
| A ``struct bpf_cpumask *`` is allocated, acquired, and released, using the |
| following functions: |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_create |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_acquire |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_release |
| |
| For example: |
| |
| .. code-block:: c |
| |
| struct cpumask_map_value { |
| struct bpf_cpumask __kptr * cpumask; |
| }; |
| |
| struct array_map { |
| __uint(type, BPF_MAP_TYPE_ARRAY); |
| __type(key, int); |
| __type(value, struct cpumask_map_value); |
| __uint(max_entries, 65536); |
| } cpumask_map SEC(".maps"); |
| |
| static int cpumask_map_insert(struct bpf_cpumask *mask, u32 pid) |
| { |
| struct cpumask_map_value local, *v; |
| long status; |
| struct bpf_cpumask *old; |
| u32 key = pid; |
| |
| local.cpumask = NULL; |
| status = bpf_map_update_elem(&cpumask_map, &key, &local, 0); |
| if (status) { |
| bpf_cpumask_release(mask); |
| return status; |
| } |
| |
| v = bpf_map_lookup_elem(&cpumask_map, &key); |
| if (!v) { |
| bpf_cpumask_release(mask); |
| return -ENOENT; |
| } |
| |
| old = bpf_kptr_xchg(&v->cpumask, mask); |
| if (old) |
| bpf_cpumask_release(old); |
| |
| return 0; |
| } |
| |
| /** |
| * A sample tracepoint showing how a task's cpumask can be queried and |
| * recorded as a kptr. |
| */ |
| SEC("tp_btf/task_newtask") |
| int BPF_PROG(record_task_cpumask, struct task_struct *task, u64 clone_flags) |
| { |
| struct bpf_cpumask *cpumask; |
| int ret; |
| |
| cpumask = bpf_cpumask_create(); |
| if (!cpumask) |
| return -ENOMEM; |
| |
| if (!bpf_cpumask_full(task->cpus_ptr)) |
| bpf_printk("task %s has CPU affinity", task->comm); |
| |
| bpf_cpumask_copy(cpumask, task->cpus_ptr); |
| return cpumask_map_insert(cpumask, task->pid); |
| } |
| |
| ---- |
| |
| 2.1.1 ``struct bpf_cpumask *`` as kptrs |
| --------------------------------------- |
| |
| As mentioned and illustrated above, these ``struct bpf_cpumask *`` objects can |
| also be stored in a map and used as kptrs. If a ``struct bpf_cpumask *`` is in |
| a map, the reference can be removed from the map with bpf_kptr_xchg(), or |
| opportunistically acquired using RCU: |
| |
| .. code-block:: c |
| |
| /* struct containing the struct bpf_cpumask kptr which is stored in the map. */ |
| struct cpumasks_kfunc_map_value { |
| struct bpf_cpumask __kptr * bpf_cpumask; |
| }; |
| |
| /* The map containing struct cpumasks_kfunc_map_value entries. */ |
| struct { |
| __uint(type, BPF_MAP_TYPE_ARRAY); |
| __type(key, int); |
| __type(value, struct cpumasks_kfunc_map_value); |
| __uint(max_entries, 1); |
| } cpumasks_kfunc_map SEC(".maps"); |
| |
| /* ... */ |
| |
| /** |
| * A simple example tracepoint program showing how a |
| * struct bpf_cpumask * kptr that is stored in a map can |
| * be passed to kfuncs using RCU protection. |
| */ |
| SEC("tp_btf/cgroup_mkdir") |
| int BPF_PROG(cgrp_ancestor_example, struct cgroup *cgrp, const char *path) |
| { |
| struct bpf_cpumask *kptr; |
| struct cpumasks_kfunc_map_value *v; |
| u32 key = 0; |
| |
| /* Assume a bpf_cpumask * kptr was previously stored in the map. */ |
| v = bpf_map_lookup_elem(&cpumasks_kfunc_map, &key); |
| if (!v) |
| return -ENOENT; |
| |
| bpf_rcu_read_lock(); |
| /* Acquire a reference to the bpf_cpumask * kptr that's already stored in the map. */ |
| kptr = v->cpumask; |
| if (!kptr) { |
| /* If no bpf_cpumask was present in the map, it's because |
| * we're racing with another CPU that removed it with |
| * bpf_kptr_xchg() between the bpf_map_lookup_elem() |
| * above, and our load of the pointer from the map. |
| */ |
| bpf_rcu_read_unlock(); |
| return -EBUSY; |
| } |
| |
| bpf_cpumask_setall(kptr); |
| bpf_rcu_read_unlock(); |
| |
| return 0; |
| } |
| |
| ---- |
| |
| 2.2 ``struct cpumask`` |
| ---------------------- |
| |
| ``struct cpumask`` is the object that actually contains the cpumask bitmap |
| being queried, mutated, etc. A ``struct bpf_cpumask`` wraps a ``struct |
| cpumask``, which is why it's safe to cast it as such (note however that it is |
| **not** safe to cast a ``struct cpumask *`` to a ``struct bpf_cpumask *``, and |
| the verifier will reject any program that tries to do so). |
| |
| As we'll see below, any kfunc that mutates its cpumask argument will take a |
| ``struct bpf_cpumask *`` as that argument. Any argument that simply queries the |
| cpumask will instead take a ``struct cpumask *``. |
| |
| 3. cpumask kfuncs |
| ================= |
| |
| Above, we described the kfuncs that can be used to allocate, acquire, release, |
| etc a ``struct bpf_cpumask *``. This section of the document will describe the |
| kfuncs for mutating and querying cpumasks. |
| |
| 3.1 Mutating cpumasks |
| --------------------- |
| |
| Some cpumask kfuncs are "read-only" in that they don't mutate any of their |
| arguments, whereas others mutate at least one argument (which means that the |
| argument must be a ``struct bpf_cpumask *``, as described above). |
| |
| This section will describe all of the cpumask kfuncs which mutate at least one |
| argument. :ref:`cpumasks-querying-label` below describes the read-only kfuncs. |
| |
| 3.1.1 Setting and clearing CPUs |
| ------------------------------- |
| |
| bpf_cpumask_set_cpu() and bpf_cpumask_clear_cpu() can be used to set and clear |
| a CPU in a ``struct bpf_cpumask`` respectively: |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_set_cpu bpf_cpumask_clear_cpu |
| |
| These kfuncs are pretty straightforward, and can be used, for example, as |
| follows: |
| |
| .. code-block:: c |
| |
| /** |
| * A sample tracepoint showing how a cpumask can be queried. |
| */ |
| SEC("tp_btf/task_newtask") |
| int BPF_PROG(test_set_clear_cpu, struct task_struct *task, u64 clone_flags) |
| { |
| struct bpf_cpumask *cpumask; |
| |
| cpumask = bpf_cpumask_create(); |
| if (!cpumask) |
| return -ENOMEM; |
| |
| bpf_cpumask_set_cpu(0, cpumask); |
| if (!bpf_cpumask_test_cpu(0, cast(cpumask))) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| bpf_cpumask_clear_cpu(0, cpumask); |
| if (bpf_cpumask_test_cpu(0, cast(cpumask))) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| /* struct cpumask * pointers such as task->cpus_ptr can also be queried. */ |
| if (bpf_cpumask_test_cpu(0, task->cpus_ptr)) |
| bpf_printk("task %s can use CPU %d", task->comm, 0); |
| |
| release_exit: |
| bpf_cpumask_release(cpumask); |
| return 0; |
| } |
| |
| ---- |
| |
| bpf_cpumask_test_and_set_cpu() and bpf_cpumask_test_and_clear_cpu() are |
| complementary kfuncs that allow callers to atomically test and set (or clear) |
| CPUs: |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_test_and_set_cpu bpf_cpumask_test_and_clear_cpu |
| |
| ---- |
| |
| We can also set and clear entire ``struct bpf_cpumask *`` objects in one |
| operation using bpf_cpumask_setall() and bpf_cpumask_clear(): |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_setall bpf_cpumask_clear |
| |
| 3.1.2 Operations between cpumasks |
| --------------------------------- |
| |
| In addition to setting and clearing individual CPUs in a single cpumask, |
| callers can also perform bitwise operations between multiple cpumasks using |
| bpf_cpumask_and(), bpf_cpumask_or(), and bpf_cpumask_xor(): |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_and bpf_cpumask_or bpf_cpumask_xor |
| |
| The following is an example of how they may be used. Note that some of the |
| kfuncs shown in this example will be covered in more detail below. |
| |
| .. code-block:: c |
| |
| /** |
| * A sample tracepoint showing how a cpumask can be mutated using |
| bitwise operators (and queried). |
| */ |
| SEC("tp_btf/task_newtask") |
| int BPF_PROG(test_and_or_xor, struct task_struct *task, u64 clone_flags) |
| { |
| struct bpf_cpumask *mask1, *mask2, *dst1, *dst2; |
| |
| mask1 = bpf_cpumask_create(); |
| if (!mask1) |
| return -ENOMEM; |
| |
| mask2 = bpf_cpumask_create(); |
| if (!mask2) { |
| bpf_cpumask_release(mask1); |
| return -ENOMEM; |
| } |
| |
| // ...Safely create the other two masks... */ |
| |
| bpf_cpumask_set_cpu(0, mask1); |
| bpf_cpumask_set_cpu(1, mask2); |
| bpf_cpumask_and(dst1, (const struct cpumask *)mask1, (const struct cpumask *)mask2); |
| if (!bpf_cpumask_empty((const struct cpumask *)dst1)) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| bpf_cpumask_or(dst1, (const struct cpumask *)mask1, (const struct cpumask *)mask2); |
| if (!bpf_cpumask_test_cpu(0, (const struct cpumask *)dst1)) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| if (!bpf_cpumask_test_cpu(1, (const struct cpumask *)dst1)) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| bpf_cpumask_xor(dst2, (const struct cpumask *)mask1, (const struct cpumask *)mask2); |
| if (!bpf_cpumask_equal((const struct cpumask *)dst1, |
| (const struct cpumask *)dst2)) |
| /* Should never happen. */ |
| goto release_exit; |
| |
| release_exit: |
| bpf_cpumask_release(mask1); |
| bpf_cpumask_release(mask2); |
| bpf_cpumask_release(dst1); |
| bpf_cpumask_release(dst2); |
| return 0; |
| } |
| |
| ---- |
| |
| The contents of an entire cpumask may be copied to another using |
| bpf_cpumask_copy(): |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_copy |
| |
| ---- |
| |
| .. _cpumasks-querying-label: |
| |
| 3.2 Querying cpumasks |
| --------------------- |
| |
| In addition to the above kfuncs, there is also a set of read-only kfuncs that |
| can be used to query the contents of cpumasks. |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_first bpf_cpumask_first_zero bpf_cpumask_first_and |
| bpf_cpumask_test_cpu bpf_cpumask_weight |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_equal bpf_cpumask_intersects bpf_cpumask_subset |
| bpf_cpumask_empty bpf_cpumask_full |
| |
| .. kernel-doc:: kernel/bpf/cpumask.c |
| :identifiers: bpf_cpumask_any_distribute bpf_cpumask_any_and_distribute |
| |
| ---- |
| |
| Some example usages of these querying kfuncs were shown above. We will not |
| replicate those examples here. Note, however, that all of the aforementioned |
| kfuncs are tested in `tools/testing/selftests/bpf/progs/cpumask_success.c`_, so |
| please take a look there if you're looking for more examples of how they can be |
| used. |
| |
| .. _tools/testing/selftests/bpf/progs/cpumask_success.c: |
| https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c |
| |
| |
| 4. Adding BPF cpumask kfuncs |
| ============================ |
| |
| The set of supported BPF cpumask kfuncs are not (yet) a 1-1 match with the |
| cpumask operations in include/linux/cpumask.h. Any of those cpumask operations |
| could easily be encapsulated in a new kfunc if and when required. If you'd like |
| to support a new cpumask operation, please feel free to submit a patch. If you |
| do add a new cpumask kfunc, please document it here, and add any relevant |
| selftest testcases to the cpumask selftest suite. |
