wireguard-linux/arch/x86/net, branch stable

x86/bpf: Make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

2026-08-20T15:52:31Z

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1]. The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free(). With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX. The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space. On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA) Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper. Fixes: 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable") Reported-by: Jiri Olsa Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Daniel Borkmann Tested-by: Jiri Olsa Link: https://lore.kernel.org/all/an8r7EODLIL-bZM3@krava Link: https://lore.kernel.org/bpf/20260818130510.3110054-1-rppt@kernel.org

bpf, x86: Fix per-CPU address resolution into an extended register

2026-08-15T06:55:32Z

The destination of the per-CPU address MOV is encoded in ModRM.reg, which is extended by REX.R, but the REX prefix is built with add_1mod(), which sets REX.B. REX.B extends ModRM.rm and SIB.base, and this instruction addresses memory as disp32 with no base, so the bit has no effect at all and the high register bit is simply lost. Every is_ereg() destination therefore resolves to the wrong register, picking whichever one shares the low three bits: R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI With BPF_REG_5, whose reg2hex is 0, the emitted 65 49 03 04 25 add %gs:,%rax adds the per-CPU offset to RAX rather than R8. The destination keeps the unadjusted address and RAX is clobbered, so the program goes on to dereference a pointer that was never made per-CPU: BUG: unable to handle page fault for address: 0000607e386a8894 RIP: bpf_prog_707837aafd2aa9ae_update_percpu_data+0x93/0xc9 Call Trace: __bpf_prog_test_run_raw_tp+0x2dc/0x7d0 __flush_smp_call_function_queue+0x1e9/0xc80 Kernel panic - not syncing: Fatal exception in interrupt R5 is the mildest of the four, aliasing a scratch register and faulting at the store. R7 aliases RBP and would corrupt the frame pointer, R8 and R9 alias the argument registers. Use add_2mod() so the register goes through REX.R, matching how add_2reg() places it in ModRM.reg and how emit_priv_frame_ptr() hardcodes 0x4c for the same instruction with R9. Encodings for the non-extended registers are unchanged. Problem showed up when trying to resurrect BPF_GCC CI (selftests built with BPF_GCC). This has gone unnoticed because clang reloads the address into R1 before each per-CPU access, so the destination is never an extended register. GCC keeps several per-CPU addresses live at once, and test_progs-bpf_gcc panics the kernel in global_percpu_data/init, where the address of a .percpu variable ends up in R5. Fixes: 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs") Signed-off-by: Vineet Gupta Reviewed-by: Eduard Zingerman Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260814220254.3797467-2-vineet.gupta@linux.dev Signed-off-by: Eduard Zingerman

bpf, x86: Clear fetch destination on faulting arena atomic

2026-08-12T17:33:53Z

populate_extable() encodes "there is no destination register to clear" as DONT_CLEAR in the DST_REG field of the exception table metadata, and later ex_handler_bpf() then reuses that very value to derive the direction it reports the fault with is_write = (reg == DONT_CLEAR). The two coincide for a plain load or store, but not for a RMW carrying BPF_FETCH. Such an atomic writes memory, so it has to be reported as a WRITE, and it also reads the old value into a register, src_reg for BPF_ADD | BPF_FETCH and BPF_XCHG, r0 for BPF_CMPXCHG, so that register has to be cleared on fault. A single DONT_CLEAR cannot say both, and the store branch picks it unconditionally: [...] } else { arena_reg = reg2pt_regs[dst_reg]; fixup_reg = DONT_CLEAR; } [...] The reported direction is therefore right, but on a fault over an unmapped arena page the fetch destination keeps whatever it held before the atomic, where every other BPF_PROBE_* access delivers 0. Give the metadata its own ARENA_WRITE bit so that the reported direction no longer depends on whether there is a register to clear, and fill DST_REG in from bpf_atomic_load_reg(). BPF_{AND,OR,XOR} | BPF_FETCH need no handling here, bpf_jit_supports_insn() already rejects those in the arena. Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260811131600.506721-3-daniel@iogearbox.net Signed-off-by: Eduard Zingerman

bpf, x86: Fix stack-passed arguments for indirect trampolines

2026-08-08T10:03:26Z

save_args() reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of a struct_ops callback with more than six argument slots is read one slot off. This has gone unnoticed because no in-tree struct_ops member passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treat BPF_TRAMP_F_INDIRECT the same way. Fixes: 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING") Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Tested-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-12-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf, x86: Convert struct_ops arena arguments in the trampoline

2026-08-08T10:03:26Z

Implement the struct_ops arena argument conversion on x86. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through RAX: movl %esrc, %eax /* truncate and clear the upper 32 bits */ subl $base_lo, %eax movq %rax, ctx_slot A nullable argument tests the full 64-bit kernel pointer first: movq %rsrc, %rax testq %rax, %rax jz 1f subl $base_lo, %eax 1: movq %rax, ctx_slot The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through RAX, so only the subtraction (and the NULL test) is inserted there. Keep arena and nullable classification in btf_func_model. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline; other trampolines pass zero and perform no conversion. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for x86. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-8-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf, x86: JIT __arena kfunc argument rebasing

2026-08-08T10:03:26Z

Implement arena argument rebasing for kfunc calls on x86. R12 already holds kern_vm_start whenever the prog has an arena, so each tagged argument costs two instructions emitted right before the call: movl %eN, %eN /* truncate, clear the upper 32 bits */ addq %r12, %rN A nullable argument tests the truncated value and jumps over the add: movl %eN, %eN testl %eN, %eN jz 1f addq %r12, %rN 1: addq carries a REX prefix for every argument register and is always three bytes, so the jz displacement is constant. The sequence is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees the rebase and needs no special handling. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-7-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf, x86: Fix exception table metadata for arena load-acquire

2026-08-07T12:57:13Z

A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but the entry is filled in as if it were a store, since populate_extable() decides based on instruction class alone and a load-acquire is of BPF_STX class: if (BPF_CLASS(insn->code) == BPF_LDX) { arena_reg = reg2pt_regs[src_reg]; fixup_reg = reg2pt_regs[dst_reg]; } else { arena_reg = reg2pt_regs[dst_reg]; fixup_reg = DONT_CLEAR; } For a load-acquire dst_reg holds the loaded value and src_reg holds the address, so both assignments in the else branch are wrong. On a fault over an unmapped arena page ex_handler_bpf() then: - computes the reported address from the value register instead of the address register - reports the access as a WRITE, since it derives the direction from fixup_reg == DONT_CLEAR - leaves dst_reg untouched, so the program continues with a stale value instead of the 0 that BPF_PROBE_* loads deliver The access itself is emitted correctly, emit_atomic_ld_st_index() uses src_reg as the address, so this is a broken probe contract and a wrong diagnostic rather than a memory safety issue. Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path. Fixes: 5341c9a4d833 ("bpf, x86: Support load-acquire and store-release instructions") Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-3-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi

bpf, x86: Fix trampoline stack size for 128-bit arguments

2026-08-01T01:00:16Z

btf_distill_func_proto() accepts a function argument up to 16 bytes, so a 128-bit scalar such as __int128 reaches the x86 trampoline with arg_size == 16. But the current implementation assumes an __int128 argument only needs one register, so the register save area is under-allocated and save_args() overwrites adjacent stack slots. Compute the register count from arg_size for all arguments to fix it. Fixes: a9c5ad31fbdc ("bpf: x86: Support in-register struct arguments in trampoline programs") Signed-off-by: Yonghong Song Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260729050204.2586457-1-yonghong.song@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi

bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable

2026-07-22T15:27:10Z

arch_bpf_trampoline_size() allocates a buffer to get actual size required for a trampoline. This buffer must be in the module address space because __arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to that buffer. In preparation for enabling ROX mode for EXECMEM_BPF make sure that the allocated memory is writable. Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for buffer allocation in arch_bpf_trampoline_size(). Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Restrict JIT predictor flush to cBPF

2026-07-01T08:33:38Z

Currently predictor flush on memory reuse is done for all BPF JIT allocations, but only cBPF programs can be loaded by an unprivileged user. eBPF is privileged by default, and flushing predictors for all CPUs on every eBPF reuse penalizes the common case for no security benefit. eBPF allocations can be frequent on busy systems, only flush predictors for cBPF programs. Trampoline and dispatcher allocations also skip the flush as they are eBPF-only. Signed-off-by: Pawan Gupta Acked-by: Daniel Borkmann Signed-off-by: Daniel Borkmann