| ============== |
| BPF Design Q&A |
| ============== |
| |
| BPF extensibility and applicability to networking, tracing, security |
| in the linux kernel and several user space implementations of BPF |
| virtual machine led to a number of misunderstanding on what BPF actually is. |
| This short QA is an attempt to address that and outline a direction |
| of where BPF is heading long term. |
| |
| .. contents:: |
| :local: |
| :depth: 3 |
| |
| Questions and Answers |
| ===================== |
| |
| Q: Is BPF a generic instruction set similar to x64 and arm64? |
| ------------------------------------------------------------- |
| A: NO. |
| |
| Q: Is BPF a generic virtual machine ? |
| ------------------------------------- |
| A: NO. |
| |
| BPF is generic instruction set *with* C calling convention. |
| ----------------------------------------------------------- |
| |
| Q: Why C calling convention was chosen? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| A: Because BPF programs are designed to run in the linux kernel |
| which is written in C, hence BPF defines instruction set compatible |
| with two most used architectures x64 and arm64 (and takes into |
| consideration important quirks of other architectures) and |
| defines calling convention that is compatible with C calling |
| convention of the linux kernel on those architectures. |
| |
| Q: Can multiple return values be supported in the future? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: NO. BPF allows only register R0 to be used as return value. |
| |
| Q: Can more than 5 function arguments be supported in the future? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: NO. BPF calling convention only allows registers R1-R5 to be used |
| as arguments. BPF is not a standalone instruction set. |
| (unlike x64 ISA that allows msft, cdecl and other conventions) |
| |
| Q: Can BPF programs access instruction pointer or return address? |
| ----------------------------------------------------------------- |
| A: NO. |
| |
| Q: Can BPF programs access stack pointer ? |
| ------------------------------------------ |
| A: NO. |
| |
| Only frame pointer (register R10) is accessible. |
| From compiler point of view it's necessary to have stack pointer. |
| For example, LLVM defines register R11 as stack pointer in its |
| BPF backend, but it makes sure that generated code never uses it. |
| |
| Q: Does C-calling convention diminishes possible use cases? |
| ----------------------------------------------------------- |
| A: YES. |
| |
| BPF design forces addition of major functionality in the form |
| of kernel helper functions and kernel objects like BPF maps with |
| seamless interoperability between them. It lets kernel call into |
| BPF programs and programs call kernel helpers with zero overhead, |
| as all of them were native C code. That is particularly the case |
| for JITed BPF programs that are indistinguishable from |
| native kernel C code. |
| |
| Q: Does it mean that 'innovative' extensions to BPF code are disallowed? |
| ------------------------------------------------------------------------ |
| A: Soft yes. |
| |
| At least for now, until BPF core has support for |
| bpf-to-bpf calls, indirect calls, loops, global variables, |
| jump tables, read-only sections, and all other normal constructs |
| that C code can produce. |
| |
| Q: Can loops be supported in a safe way? |
| ---------------------------------------- |
| A: It's not clear yet. |
| |
| BPF developers are trying to find a way to |
| support bounded loops. |
| |
| Q: What are the verifier limits? |
| -------------------------------- |
| A: The only limit known to the user space is BPF_MAXINSNS (4096). |
| It's the maximum number of instructions that the unprivileged bpf |
| program can have. The verifier has various internal limits. |
| Like the maximum number of instructions that can be explored during |
| program analysis. Currently, that limit is set to 1 million. |
| Which essentially means that the largest program can consist |
| of 1 million NOP instructions. There is a limit to the maximum number |
| of subsequent branches, a limit to the number of nested bpf-to-bpf |
| calls, a limit to the number of the verifier states per instruction, |
| a limit to the number of maps used by the program. |
| All these limits can be hit with a sufficiently complex program. |
| There are also non-numerical limits that can cause the program |
| to be rejected. The verifier used to recognize only pointer + constant |
| expressions. Now it can recognize pointer + bounded_register. |
| bpf_lookup_map_elem(key) had a requirement that 'key' must be |
| a pointer to the stack. Now, 'key' can be a pointer to map value. |
| The verifier is steadily getting 'smarter'. The limits are |
| being removed. The only way to know that the program is going to |
| be accepted by the verifier is to try to load it. |
| The bpf development process guarantees that the future kernel |
| versions will accept all bpf programs that were accepted by |
| the earlier versions. |
| |
| |
| Instruction level questions |
| --------------------------- |
| |
| Q: LD_ABS and LD_IND instructions vs C code |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Q: How come LD_ABS and LD_IND instruction are present in BPF whereas |
| C code cannot express them and has to use builtin intrinsics? |
| |
| A: This is artifact of compatibility with classic BPF. Modern |
| networking code in BPF performs better without them. |
| See 'direct packet access'. |
| |
| Q: BPF instructions mapping not one-to-one to native CPU |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| Q: It seems not all BPF instructions are one-to-one to native CPU. |
| For example why BPF_JNE and other compare and jumps are not cpu-like? |
| |
| A: This was necessary to avoid introducing flags into ISA which are |
| impossible to make generic and efficient across CPU architectures. |
| |
| Q: Why BPF_DIV instruction doesn't map to x64 div? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: Because if we picked one-to-one relationship to x64 it would have made |
| it more complicated to support on arm64 and other archs. Also it |
| needs div-by-zero runtime check. |
| |
| Q: Why there is no BPF_SDIV for signed divide operation? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: Because it would be rarely used. llvm errors in such case and |
| prints a suggestion to use unsigned divide instead. |
| |
| Q: Why BPF has implicit prologue and epilogue? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: Because architectures like sparc have register windows and in general |
| there are enough subtle differences between architectures, so naive |
| store return address into stack won't work. Another reason is BPF has |
| to be safe from division by zero (and legacy exception path |
| of LD_ABS insn). Those instructions need to invoke epilogue and |
| return implicitly. |
| |
| Q: Why BPF_JLT and BPF_JLE instructions were not introduced in the beginning? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| A: Because classic BPF didn't have them and BPF authors felt that compiler |
| workaround would be acceptable. Turned out that programs lose performance |
| due to lack of these compare instructions and they were added. |
| These two instructions is a perfect example what kind of new BPF |
| instructions are acceptable and can be added in the future. |
| These two already had equivalent instructions in native CPUs. |
| New instructions that don't have one-to-one mapping to HW instructions |
| will not be accepted. |
| |
| Q: BPF 32-bit subregister requirements |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| Q: BPF 32-bit subregisters have a requirement to zero upper 32-bits of BPF |
| registers which makes BPF inefficient virtual machine for 32-bit |
| CPU architectures and 32-bit HW accelerators. Can true 32-bit registers |
| be added to BPF in the future? |
| |
| A: NO. The first thing to improve performance on 32-bit archs is to teach |
| LLVM to generate code that uses 32-bit subregisters. Then second step |
| is to teach verifier to mark operations where zero-ing upper bits |
| is unnecessary. Then JITs can take advantage of those markings and |
| drastically reduce size of generated code and improve performance. |
| |
| Q: Does BPF have a stable ABI? |
| ------------------------------ |
| A: YES. BPF instructions, arguments to BPF programs, set of helper |
| functions and their arguments, recognized return codes are all part |
| of ABI. However there is one specific exception to tracing programs |
| which are using helpers like bpf_probe_read() to walk kernel internal |
| data structures and compile with kernel internal headers. Both of these |
| kernel internals are subject to change and can break with newer kernels |
| such that the program needs to be adapted accordingly. |
| |
| Q: How much stack space a BPF program uses? |
| ------------------------------------------- |
| A: Currently all program types are limited to 512 bytes of stack |
| space, but the verifier computes the actual amount of stack used |
| and both interpreter and most JITed code consume necessary amount. |
| |
| Q: Can BPF be offloaded to HW? |
| ------------------------------ |
| A: YES. BPF HW offload is supported by NFP driver. |
| |
| Q: Does classic BPF interpreter still exist? |
| -------------------------------------------- |
| A: NO. Classic BPF programs are converted into extend BPF instructions. |
| |
| Q: Can BPF call arbitrary kernel functions? |
| ------------------------------------------- |
| A: NO. BPF programs can only call a set of helper functions which |
| is defined for every program type. |
| |
| Q: Can BPF overwrite arbitrary kernel memory? |
| --------------------------------------------- |
| A: NO. |
| |
| Tracing bpf programs can *read* arbitrary memory with bpf_probe_read() |
| and bpf_probe_read_str() helpers. Networking programs cannot read |
| arbitrary memory, since they don't have access to these helpers. |
| Programs can never read or write arbitrary memory directly. |
| |
| Q: Can BPF overwrite arbitrary user memory? |
| ------------------------------------------- |
| A: Sort-of. |
| |
| Tracing BPF programs can overwrite the user memory |
| of the current task with bpf_probe_write_user(). Every time such |
| program is loaded the kernel will print warning message, so |
| this helper is only useful for experiments and prototypes. |
| Tracing BPF programs are root only. |
| |
| Q: bpf_trace_printk() helper warning |
| ------------------------------------ |
| Q: When bpf_trace_printk() helper is used the kernel prints nasty |
| warning message. Why is that? |
| |
| A: This is done to nudge program authors into better interfaces when |
| programs need to pass data to user space. Like bpf_perf_event_output() |
| can be used to efficiently stream data via perf ring buffer. |
| BPF maps can be used for asynchronous data sharing between kernel |
| and user space. bpf_trace_printk() should only be used for debugging. |
| |
| Q: New functionality via kernel modules? |
| ---------------------------------------- |
| Q: Can BPF functionality such as new program or map types, new |
| helpers, etc be added out of kernel module code? |
| |
| A: NO. |
