aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/kernel/bpf
AgeCommit message (Collapse)AuthorFilesLines
2026-08-13bpf: Fix func_info_aux desync after dead code eliminationKumar Kartikeya Dwivedi1-1/+5
The verifier keeps per-subprogram metadata in three parallel arrays: subprog_info, func_info, and func_info_aux. Dead code elimination can remove whole subprograms, and adjust_subprog_starts_after_remove() shifts subprog_info and func_info to close the gap, but leaves func_info_aux in place. From that point on, func_info_aux[i] no longer describes subprogram i. Shift func_info_aux together with func_info so the three arrays stay aligned after subprogram removal. Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com
2026-08-13bpf: Introduce global percpu dataLeon Hwang4-3/+84
Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro. For example, in BPF, it is able to define a global percpu variable like: int data SEC(".percpu"); With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs"); to static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs"); This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper. Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps. Links: [1] https://github.com/anakryiko/retsnoop [2] https://github.com/bpfsnoop/bpfsnoop Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
2026-08-13bpf: Factor out check_map_mem_read helper in verifierLeon Hwang1-32/+43
In the next commit, percpu_array map will add map_direct_value_addr support. IOW, it will add a map_type check in the iff condition of the bpf_map_direct_read() code block, which will reduce the code block readability. Hence, factor out check_map_mem_read helper to improve the readability, and the maintainability for the percpu_array map case. Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
2026-08-13bpf: Drop duplicate blank lines in kernel/bpf/Leon Hwang10-30/+0
There are many adjacent blank lines in kernel/bpf/ that have accumulated over time. Drop them for cleanup. No functional changes intended. Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev
2026-08-13bpf: Trim special_kfunc_list in verifierLeon Hwang1-6/+0
The commit 7619a0ee9340 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE") dropped some helpers in verifier, which also eliminated the use of the following kfuncs from the special_kfunc_list: * bpf_arena_reserve_pages * bpf_stream_vprintk * bpf_stream_print_stack So, drop them from the special_kfunc_list. Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-13bpf: Eliminate dup/restore of insn_aux_dataXu Kuohai3-52/+4
The dup/restore of insn_aux_data was introduced to resolve the inconsistency between insnsi and insn_aux_data arrays, which occurs on the failure path where insnsi was rolled back to the original state before constants blinding, while insn_aux_data was not. After JIT failure, there is only one user, bpf_clear_insn_aux_data(), that requires insnsi and insn_aux_data to be synchronized. It accesses both insnsi and insn_aux_data using the same array size and index. However, the access to insnsi in bpf_clear_insn_aux_data() is not necessary. It is checked to skip the second slot of an ldimm64 instruction, whose jt is never set and can be absorbed into the jt check itself. So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a specific length field for insn_aux_data to allow it to have a different length from the insnsi array. Then remove dup/restore of insn_aux_data. Signed-off-by: Xu Kuohai <xukuohai@huawei.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-13bpf: Make bpf_trampoline_multi_detach return voidHui Zhu1-2/+2
bpf_trampoline_multi_detach() always returns 0 and the sole caller ignores the return value. Change it to return void and drop the WARN_ON_ONCE at the call site. Signed-off-by: Hui Zhu <zhuhui@kylinos.cn> Acked-by: Leon Hwang <leon.hwang@linux.dev> Acked-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-13bpf: Fix UAF in bpf_trampoline_multi_attach_free on update failureHui Zhu1-1/+11
When bpf_trampoline_update() fails before modify_fentry_multi()/ unregister_fentry_multi() is called, cur_image is unchanged (cur_image == old_image) and ftrace still calls into it. Freeing old_image in that case causes a UAF. Only free old_image when it differs from cur_image. Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions") Signed-off-by: Hui Zhu <zhuhui@kylinos.cn> Acked-by: Leon Hwang <leon.hwang@linux.dev> Acked-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-12bpf: Show more useful info in stack depth statsKumar Kartikeya Dwivedi1-9/+14
Stack depth statistics list captured depths in subprogram-number order, while per-verification instruction counts are reported separately. Since libbpf determines subprogram numbers, it is hard to associate either statistic with its subprogram name or see where verifier work is spent. Now that self and inclusive instruction counts are available for every subprogram, keep the combined maximum stack depth on its own line and print one uniform record for each subprogram. Represent the main program as subprog 0, then classify each record as main, global, or static before reporting insns_self, insns_total, and stack depth. The aggregate processed count is the sum of all self counts, while each total shows verifier work rooted at that subprogram. When no subprogram name is available, print <unknown>. Keep the existing aggregate "processed ... insns" record unchanged for compatibility. Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-12bpf: Attribute async callback instructions to verification rootsKumar Kartikeya Dwivedi1-0/+10
Asynchronous callbacks are explored as fresh frame-zero verifier states, so normal callee-to-caller accounting cannot propagate their instruction budget to the main or global subprogram whose verification scheduled them. The callback exploration still happens within the same do_check_common() invocation as that independent verification root. Record env->insn_processed at do_check_common() entry and override the root's inclusive count with the delta before returning. This includes all directly and transitively scheduled asynchronous callbacks in the root's total without maintaining a separate accounting call stack. Static subprogram and callback totals remain local to their synchronous call paths. Their self counts continue to account for each processed instruction exactly once. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-12bpf: Track verifier instruction stats for each subprogramKumar Kartikeya Dwivedi1-9/+46
The verifier currently records one instruction count for the main program and each global subprogram checked independently. Static subprograms are explored within callers, so their verification cost cannot be reported separately. Track both self and inclusive instruction counts for every subprogram. Charge each processed instruction as self work to the current subprogram and to a path-local subtotal in its function frame. When a function returns, add the callee subtotal to its inclusive count and to its parent subtotal. Fold any remaining frames when a path terminates or is pruned. Instruction subtotals are accounting state, not semantic verifier state. Clear them when a verifier state is copied so work before a path fork is charged once, rather than again when a saved branch is explored. If copying a saved state fails before all frames are allocated, skip missing frames while folding the current path. This generic frame accounting also records self and inclusive totals when an asynchronous callback starts as a fresh frame-zero state. It does not yet charge that independently explored callback path back to the main or global exploration root which scheduled it. That will be done in subsequent changes. This does not change the verification statistics output format. It only prepares the counters for per-subprogram reporting. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-12bpf: Derive the atomic load register in one placeDaniel Borkmann4-35/+9
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register it reads the old value into, the BPF_STX case of insn_def_regno() open codes the very same mapping a second time, the const folding and the liveness transfer functions a third and a fourth time, and BPF JITs need it as well to know which register a faulting BPF_PROBE_ATOMIC has to clear. Add a small helper so that all of them can share it. No functional change. The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally accepts cannot be seen there as it is only set from bpf_do_misc_fixups(), that is, after const folding has run. arg_track_xfer() keeps its mode test since that also guards the stack clearing next to it. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-12cachefiles,netfs: sunset ondemand modeGao Xiang1-7/+0
It was an effort to enhance fscache as a kernel cache for lazy pulling (at least according to previous Incremental FS discussion [1]) and EROFS over fscache was the in-tree user of this mode. fscache has since evolved to be netfslib-oriented, serving network filesystem inodes via the netfs library, but EROFS never acts as a network filesystem and we need to cache golden filesystem images rather than individual EROFS inodes. Since EROFS over fscache is now removed, clean up netfs/fscache/ cachefiles upstream too. [1] https://lore.kernel.org/r/CAOQ4uxi4dzxArY24YO=+kBCK2gGoq3Ptb8WkzCqSogPgU_R3dQ@mail.gmail.com [dh] Fixed up comments on: https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com Signed-off-by: Gao Xiang <xiang@kernel.org> Signed-off-by: David Howells <dhowells@redhat.com> Link: https://patch.msgid.link/1046393.1786544127@warthog.procyon.org.uk cc: Paulo Alcantara <pc@manguebit.org> cc: netfs@lists.linux.dev cc: linux-erofs@lists.ozlabs.org cc: bpf@vger.kernel.org cc: linux-fsdevel@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
2026-08-11bpf: Compare iterator types during state pruningNing Ding1-1/+2
An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states must not be equal, or the verifier can prune an unsafe path. Compare the pointer type for STACK_ITER slots. Fixes: dfab99df147b ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS") Signed-off-by: Ning Ding <dingning04@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Reject tracing/freplace progs for struct_ops with arena argsKumar Kartikeya Dwivedi1-0/+10
Reject tracing and freplace attachments to a target program with arena context arguments. The struct_ops indirect trampoline converts those arguments before entering the target, so a generic tracing trampoline would otherwise expose arena offsets using the target BTF pointer type. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-14-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Support __arena and __arena__nullable on struct_ops argumentsTejun Heo4-22/+104
A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in. Add the __arena and __arena__nullable stub argument suffixes to make this convenient. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). The trampoline converts the value while saving the arguments into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the program never sees a kernel address and nothing rewrites the ctx after the fact. The converted value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address. __arena converts unconditionally and the kernel caller must not pass NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the program's type state already encompasses NULL-ness, so it is not meaningful to force a NULL check for the program. The composite suffix intentionally ends in __nullable. Classify __arena__nullable before the generic suffix so scalar arena pointees do not take the generic nullable BTF pointer path. This patch adds the generic side. prepare_arg_info() records arena and nullable argument flags in the struct_ops function model, and bpf_tramp_arena_base() returns the arena base for a single-program struct_ops indirect trampoline. Only that trampoline converts: its program's arena is fixed at generation time. Generic trampolines can mix programs with different arenas and reject arena context arguments defensively, which is unreachable today as only struct_ops programs carry them. Architectures that do not implement the conversion are gated out at verification time with bpf_jit_supports_arena_args(). Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Tejun Heo <tj@kernel.org> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-6-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Support __arena and __arena__nullable kfunc argument suffixesTejun Heo3-5/+64
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation. Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments. __arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call. The double separator makes the annotations composable: __arena__nullable also ends in __nullable and naturally follows the common nullable argument path. Plain __arena follows that path too for verifier type checking because both forms accept a constant zero; the function-model flag still determines whether the JIT preserves NULL or rebases it to the arena base. This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data. JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere. Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Tejun Heo <tj@kernel.org> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Collect kfuncs after resolving program resourcesKumar Kartikeya Dwivedi1-5/+6
The kfunc descriptors include argument prototypes generated while calls are collected. Some argument classifications need program auxiliary state derived from referenced maps, such as the arena associated with the program. This avoids a footgun in get_kfunc_arg_type() checks where we do validation on whether program has prog->aux->arena and it hasn't been resolved yet. check_and_resolve_insns() records used maps and populates that state. It must remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations, so that instruction validation and the program tag observe the relocated instruction stream. Move only add_kfuncs() after instruction and resource resolution. Subprogram discovery and validation remain before the full BTF phase because that phase needs the complete subprogram layout. Add a short comment describing the resource resolution phase at the call site. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Reviewed-by: Amery Hung <ameryhung@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-4-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Split subprogram and kfunc collectionKumar Kartikeya Dwivedi1-9/+32
add_subprog_and_kfunc() combines two operations with different ordering requirements. Subprogram discovery must precede validation of func_info and line_info, while kfunc descriptors are only needed by the verifier after its initial program setup is complete. Split the helper into add_subprogs() and add_kfuncs() so each operation can be placed according to its actual dependencies. Keep both calls adjacent and in their existing phase for now, and add short comments describing their roles. No functional change is intended for valid programs. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Reviewed-by: Amery Hung <ameryhung@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-3-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Rename 'early' BTF checking as a preparation phaseKumar Kartikeya Dwivedi2-8/+10
BTF processing is split around subprogram discovery. The first phase gets program BTF and imports func_info because a BTF-tagged exception callback may not be referenced by any instruction. Subprogram discovery needs this metadata to find it. The later phase validates func_info and line_info against the complete subprogram table and applies CO-RE relocations. This split breaks a real dependency cycle rather than merely running the same checks early. Rename bpf_check_btf_info_early() and check_btf_func_early() to preparation names that reflect this role. Add short call-site comments to make the two phases and their responsibilities clear. No functional change is intended. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Reviewed-by: Amery Hung <ameryhung@gmail.com> Link: https://patch.msgid.link/20260808003938.3486067-2-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-08bpf: Simplify the bpf_is_reg64()Eduard Zingerman1-82/+26
After the previous commit bpf_is_reg64() is only used in a context where destination register's property is queried, and only for instructions for which insn_def_regno() >= 0. Hence, simplify the function by: - removing unused parameters; - removing code paths considering BPF_JMP{,32} instructions; - streamlining the condition expressions. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-6-b6c270013c77@gmail.com
2026-08-08bpf: Infer zext_dst based on static register liveness analysisEduard Zingerman3-116/+37
As reported in the thread [1], the verifier's 32-bit operations zero extension logic is broken. This logic is responsible for correct semantics of 32-bit operations on s390 architecture. According to BPF semantics, operation `w1 += 1` is supposed to zero extend the upper half of the register `r1`. On s390 the JIT relies on the verifier emitting explicit zero extension before such operations. The verifier attempts to minimize the amount of zero extensions inserted by tracking whether upper halves of the 64-bit registers are ever used. Previously such tracking worked as follows: - bpf_reg_state->subreg_def field was set by do_check_insn() for each operation defining lower but not the upper halves of the register. - Whenever an operation reading the whole register was verified, the verifier checked register's subreg_def and set bpf_insn_aux_data->zext_dst flag as true via a call to mark_insn_zext() function. - After the verification was complete, a special pass bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations with bpf_insn_aux_data->zext_dst set as true by adding explicit zero extension. Note that the logic above relies on bpf_reg_state->subreg_def, which is a property of a current verifier state. Before the commit [2] two additional steps happened: - The verifier tracked upper and lower register halves' liveness as flags REG_LIVE_READ{32,64} in bpf_reg_state->live. - The function propagate_liveness() called mark_insn_zext() in order to transfer the knowledge about which registers have their upper halves alive (and thus might require zero extension). The commit [2] removed the two steps described above, hence making possible a situation like below: - The register's upper half is set and is used on some verification path P1 and the register happens not to be marked as precise. - The checkpoint C is created while processing some instruction between register initialization and usage. - On some other verification path P2 the register's upper half is not initialized and that path ends hitting the checkpoint C. - In such a case the register's initialization on path P2 would lack zext_dst mark, making it possible for the program to inject an arbitrary value in the register's upper half. This commit replaces subreg_def based logic with computing zext_dst statically, as a part of the bpf_compute_live_registers() analysis: - The analysis now tracks usage of upper and lower halves of the registers separately. - If some instruction defines a 32-bit subregister, but not the whole register, *and* the upper half of the register is alive after that instruction, the instruction is marked as zext_dst. There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict a noticeable performance penalty. [1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/ [2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness") Fixes: 107e16979905 ("bpf: disable and remove registers chain based liveness") Reported-by: Min-gyu Kim <gimm78064@gmail.com> Reported-by: STAR Labs SG <info@starlabs.sg> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/ Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-5-b6c270013c77@gmail.com
2026-08-08bpf: Track upper 32-bit register halves' liveness in compute_live_registers()Eduard Zingerman2-31/+59
Extend compute_live_registers() to track upper and lower register halves' liveness separately. This is mostly straightforward: - use/def masks are extended to track 2 bits per register; - compute_insn_live_regs() is updated to properly track these 2 bits according to the instruction semantics. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-4-b6c270013c77@gmail.com
2026-08-08bpf: Move bpf_is_reg64() to fixups.cEduard Zingerman2-90/+90
The following patches are going to remove bpf_is_reg64() users from everywhere except fixups.c, and also make it dependent on functions local to fixups.c. Move the function before hand to simplify the review. Non functional change. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-3-b6c270013c77@gmail.com
2026-08-08bpf: Extract is_addr_space_cast32() utility functionEduard Zingerman1-9/+26
bpf_do_misc_fixups() converts the following address space cast instructions to 32-bit moves: - cast from address space 1 (user) to address space 0 (kernel) - cast from address space 0 (kernel) to address space 1 (user) iff associated arena map has a BPF_F_NO_USER_CONV flag. Extract a predicate detecting such instructions for use in the following patches. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-2-b6c270013c77@gmail.com
2026-08-08bpf: Do not print a newline after disassembly in bpf_verbose_insn()Eduard Zingerman4-38/+38
At the moment there are more callsites that want bpf_verbose_insn() to not print a newline after the instruction, than callsites that want a newline. Drop '\n' from disasm.c. Non-functional change. The changes in bpftool are verified by writing a bpf program using a variety of instructions and comparing `prog dump xlated` output in the following modes: plain, opcodes, visual, visual opcodes. The output before and after the changes is identical. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Quentin Monnet <qmo@kernel.org> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-1-b6c270013c77@gmail.com
2026-08-08bpf: Fix mmap_lock leak in irq_work pathSanghyun Park3-37/+56
stack_map_get_build_id_offset() introduced a per-CPU irq_work to defer mmap_read_unlock() from NMI context, and bpf_find_vma() later reused the same mmap_unlock_work. Both callers only check whether the work is busy before taking mmap_lock, so a nested caller can reuse the slot before the first caller queues it. Two read locks may then be acquired while only one deferred unlock runs, leaking a read lock and blocking exit_mmap(). Reserve the per-CPU slot before mmap_read_trylock(). Use the same wrapper in stackmap and bpf_find_vma() so both callers release the reservation on trylock failure. Keep rejecting the slot while the irq_work remains busy. Release it after the irq_work callback unlocks the mm. Fixes: eac9153f2b58 ("bpf/stackmap: Fix deadlock with rq_lock in bpf_get_stack()") Reported-by: syzbot+cdd6c0925e12b0af60cc@syzkaller.appspotmail.com Reported-by: sashiko-bot@kernel.org Signed-off-by: Sanghyun Park <sanghyun.park.cnu@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Closes: https://syzkaller.appspot.com/bug?extid=cdd6c0925e12b0af60cc Closes: https://lore.kernel.org/r/20260630033745.B80201F000E9@smtp.kernel.org Link: https://lore.kernel.org/bpf/20260805031425.2157475-2-sanghyun.park.cnu@gmail.com
2026-08-07bpf, cgroup: Fix storage null-ptr-deref after replacing progPu Lehui1-0/+17
Syzkaller reported a storage null-ptr-deref issue after replacing prog. This occurs in the following scenario: 1. prog A, an empty prog, is attached to a cgrp. 2. prog B uses BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE and calls the bpf_get_local_storage helper. 3. link_update is called to replace prog A with prog B. The reason is that __cgroup_bpf_replace fails to alloc and assign the required cgrp storage for the incoming replacement prog. Consequently, the new prog inherits an uninit storage, leading to null-ptr-deref panic when kick the new prog. Fix this by rejecting a link update if new_prog's cgroup storage is incompatible with link->prog. Fixes: 0c991ebc8c69 ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link") Signed-off-by: Pu Lehui <pulehui@huawei.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Reviewed-by: Amery Hung <ameryhung@gmail.com> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/bpf/20260728132336.2857800-1-pulehui@huaweicloud.com [0] Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [1] Link: https://lore.kernel.org/bpf/20260807104403.1013064-1-pulehui@huaweicloud.com
2026-08-07Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf 7.2-rc7Daniel Borkmann2-25/+25
Cross-merge BPF and other fixes after downstream PR. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2026-08-07bpf: Reject load-acquire from pointers requiring fault protectionDaniel Borkmann1-1/+26
A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier, unlike a regular BPF_LDX, so the JIT emits a plain load with no exception table entry and a fault panics the kernel instead of being handled. Reject the source pointer types that a BPF_LDX would have had that fault protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED, PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED. This is reachable e.g. by loading ->mm out of a trusted task_struct yields an untrusted pointer to mm_struct, and it is NULL for a kernel thread: [...] SEC("tp_btf/sched_switch") int BPF_PROG(demo, bool preempt, struct task_struct *prev, struct task_struct *next) { struct mm_struct *mm = next->mm; /* untrusted */ out_ldx = (__u64)mm->pgd; /* BPF_LDX */ out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */ return 0; } [...] Both dereference the same pointer, but only the BPF_LDX is protected (x86-64 JIT, jump targets shown prog-relative): [...] ; out_ldx = (__u64)mm->pgd; 17: movq $-10485760, %r10 1e: movq %rsi, %r11 21: addq $184, %r11 28: subq %r10, %r11 2b: movabsq $140737498841088, %r10 35: cmpq %r10, %r11 38: ja 0x3e <-- kernel addr? 3a: xorl %edi, %edi <-- no: dst = 0, skip the load 3c: jmp 0x45 3e: movq 184(%rsi), %rdi <-- yes: load + extable entry [...] ; load_acquire(&mm->pgd) 53: movq %rsi, %rdi 56: movq 184(%rdi), %rax <-- no check, no extable entry [...] Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as bpf_insn_prepare_dump() rewrites it back to BPF_MEM. A PTR_TRUSTED pointer is deliberately not on the list. Such a load is not converted either, but it does not need to be, since the pointer is guaranteed live, so load-acquire from it stays allowed. The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release error messages are unchanged; writes (RMW / store-release) to such pointers are already rejected elsewhere, so only load-acquire needs this. Fixes: 880442305a39 ("bpf: Introduce load-acquire and store-release instructions") Reported-by: STAR Labs SG <info@starlabs.sg> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260806201047.333389-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-06rqspinlock: Reset tail when preserving queue on deadlockKumar Kartikeya Dwivedi1-2/+3
Currently, the destruction of the waiter queue is suppressed for rqspinlock in cases where a deadlock is detected. Deadlock checks happen relatively frequently (on entry for AA, within 1ms for ABBA), and waiter threads may not be involved in locking scenarios involving deadlocks. Thus, it is useful to not flush the queue and let other waiters take a stab at acquiring the lock after we detect a deadlock and exit. However, we need to follow the same logic as what we did previously for the waitq_timeout label: reset the tail, and if we cannot, signal the next waiter appropriately. In case of deadlocks, this signal would just mark the MCS node as unlocked, and in case of timeouts, it would signal RES_TIMEOUT_VAL. The difference thus is in the value propagated, which decides whether the queue remains active or gets flushed. Not doing the tail reset, and waiting for the next waiter can lead to cases where we are the final waiter, and thus no next waiter arrives, leading to intermittent stalls in this path. Once the next waiter does join, we will be unblocked. In the theoretical case when the next waiter never joins, we risk stalling indefinitely. This can only happen for ABBA deadlocks, since entry into the wait queue is guarded with AA checks. A precise sequence of executions leading up to this scenario can be: CPU 0 holds lock A. CPU 1 holds lock B. CPU 2 attempts lock B, becomes the pending waiter for B. CPU 0 attempts lock B. B has locked+pending bits set, thus CPU 0 queues. CPU 1 attempts lock A. CPU 0 detects an ABBA deadlock. Once deadlock detection happens for CPU 0, it will sit waiting for the next waiter in the queue to populate node->next, which will experience delays until such a waiter arrives. Fix this by adjusting the logic for the check for deadlocks preceding the waitq_timeout label. It would make sense to consolidate code for both cases and use 'ret' to distinguish the value being propagated, but that is left as an exercise for a future refactoring task to avoid diff noise in this patch. Fixes: 7bd6e5ce5be6 ("rqspinlock: Disable queue destruction for deadlocks") Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260802021759.1139457-1-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
2026-08-06bpf: Account for preempt and IRQ state in RCU protectionNing Ding1-4/+10
Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends. Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends. Signed-off-by: Ning Ding <dingning04@gmail.com> Link: https://lore.kernel.org/bpf/20260805233940.3966981-2-dingning04@gmail.com [ kkd: Simplify was_in_rcu_cs on spin unlock and adjust the selftest. ] Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-06bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFEKaitao Cheng4-66/+36
The verifier currently keeps a hard-coded list of kfuncs that may be called while holding a bpf_spin_lock. With KF_SPINLOCK_SAFE available, retaining this list creates two sources of truth and requires verifier changes whenever another lock-safe kfunc is added. Mark every kfunc currently accepted by kfunc_spin_allowed() with KF_SPINLOCK_SAFE. This covers the graph, numeric iterator, resource spin lock, arena, and stream kfuncs. Remove the obsolete category checks and make kfunc_spin_allowed() rely solely on the kfunc registration metadata. This preserves the behavior of existing kfuncs while using the same mechanism for built-in and module kfuncs. Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/bpf/20260805153340.34776-3-kaitao.cheng@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-06bpf: Add KF_SPINLOCK_SAFE flag for kfuncs under bpf_spin_lockKaitao Cheng1-5/+15
Introduce the KF_SPINLOCK_SAFE kfunc metadata flag in BTF so kfuncs may be explicitly marked as safe to call while holding bpf_spin_lock. Allow kfuncs defined in kernel modules to be marked with KF_SPINLOCK_SAFE. Example: BTF_ID_FLAGS(func, $kfunc_name, KF_SPINLOCK_SAFE) Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/bpf/20260805153340.34776-2-kaitao.cheng@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-08-05bpf: Harden bloom filter sizing and indexing on 32-bit kernelsJérémy Jean1-10/+9
bloom_map_alloc() has two 32-bit-specific problems when the computed bitmap reaches the U32_MAX fallback case. First, BITS_TO_BYTES(U32_MAX) is evaluated with 32-bit arithmetic. The addition performed by DIV_ROUND_UP wraps, so the map allocates only the fixed-size bloom filter object while keeping bitset_mask == U32_MAX. Subsequent updates can then write past the allocated object. Second, fixing only the allocation size is not sufficient. The bloom hash is a u32, but set_bit() takes a signed long bit number and x86 test_bit() eventually feeds the index to variable_test_bit(long, ...). On 32-bit kernels, hashes in [0x80000000, U32_MAX] therefore become negative bit offsets. x86 bt/bts with a memory operand interpret those offsets relative to the supplied base, so a map with bitset_mask == U32_MAX can read or write before bloom->bitset even after allocating the full 512 MiB bitmap. Keep the U32_MAX fallback, but split each hash into a word pointer and an in-word bit number before calling test_bit() or set_bit(). The bitops argument is then always in [0, BITS_PER_LONG - 1], while BIT_WORD(h) still selects the intended word in the full bitmap. Compute the bitset size from (u64)bitset_mask + 1 before passing the final size to bpf_map_area_alloc(). This fixes the original under-allocation and keeps the allocated storage consistent with the addressable bitset. Exploitation note: local privilege escalation is possible on a 32-bit x86 kernel using the under-allocation bug from a binary with CAP_BPF. Fixes: 9330986c0300 ("bpf: Add bloom filter map implementation") Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260805060228.2703051-1-Jeremy.Jean@oss.cyber.gouv.fr Assisted-by: Codex:gpt-5
2026-08-05bpf: Fix sleepable check for tracing/lsm progLeon Hwang1-0/+3
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog is allowed to attach to '__x64_'-alike prefix symbols. It is because the verifier does not verify whether the symbol is a kernel function or a bpf prog. That said, a sleepable tracing prog is allowed to attach to a bpf prog target whose name has '__x64_'-alike prefix. For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP prog, and copies buffer from a user pointer with bpf_copy_from_user() helper. After attaching the XDP prog to lo interface, the kernel BUG could be triggered by 'ping -c 1 -W 1 127.0.0.1': [ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324 Fix it by disallowing sleepable prog always when its target btf is not a kernel's btf. Fixes: 16d9c5660692 ("bpf: Always allow sleepable programs on syscalls") Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Viktor Malik <vmalik@redhat.com> Link: https://lore.kernel.org/bpf/20260805150810.34907-2-leon.hwang@linux.dev
2026-08-05bpf: Avoid changing callchain in bpf_get_stackid_peJiri Olsa1-22/+17
There's no need to modify the trace object bpf_get_stackid_pe, we just need to pass the needed callchain length in separate argument. This way we can have callchain pointers const and remove the trace->nr modification and restoration. Assisted-by: Codex:GPT-5.5 Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-13-jolsa@kernel.org
2026-08-05bpf: Avoid changing callchain in bpf_get_stack_peJiri Olsa1-21/+15
There's no need to modify the trace object bpf_get_stack_pe, we just need to pass the needed callchain length in separate argument. This way we can have callchain pointers const and remove the trace->nr modification and restoration. Assisted-by: Codex:GPT-5.5 Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-12-jolsa@kernel.org
2026-08-05bpf: Disable preemption in __bpf_get_stackDaniel Borkmann1-0/+3
get_perf_callchain() returns a per-CPU perf_callchain_entry buffer and releases its recursion slot via put_callchain_entry() before returning, so nothing keeps the entry reserved while __bpf_get_stack() consumes it below. A preemptible BPF program (e.g. a non-sleepable raw tracepoint program on a PREEMPT kernel, which runs under migrate_disable() but not preempt_disable()) can be scheduled out between obtaining the entry and the copy. Another task scheduled on the same CPU then reuses the same per-CPU buffer and overwrites trace->nr with a larger value. copy_len is then computed from the inflated trace->nr and can exceed the caller's buffer, causing an out-of-bounds write in the memcpy() and in the build_id path. The rcu_read_lock() taken here alone does not prevent this. It is only taken on the may_fault path, and under CONFIG_PREEMPT_RCU it does not disable preemption; it merely keeps perf's callchain buffer array alive (freed via call_rcu()) and does nothing to stop another task from reusing the entry. Disable preemption around obtaining the callchain entry and copying it into the caller's buffer, so the entry cannot be reused underneath us and trace->nr stays bounded by max_depth. Build ID resolution may fault and is therefore deferred until after preemption is re-enabled; by then the instruction pointers have already been copied into buf, so it operates only on that private copy. Note, preempt_disable() also subsumes the buffer-lifetime guarantee the rcu_read_lock() provided, since a preempt-disabled section is an RCU read-side critical section for the callchain buffers' call_rcu() reclaim. Fixes: c195651e565a ("bpf: add bpf_get_stack helper") Reported-by: Tao Chen <chen.dylane@linux.dev> Reported-by: STAR Labs SG <info@starlabs.sg> Signed-off-by: Daniel Borkmann <borkmann@iogearbox.net> Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260803210149.296496-11-jolsa@kernel.org Closes: https://lore.kernel.org/bpf/20260206090653.1336687-1-chen.dylane@linux.dev/ [ changed Fixes: commit ]
2026-08-05bpf: Clear buf on error in __bpf_get_task_stackJiri Olsa1-2/+5
Both bpf_get_task_stack and bpf_get_task_stack_sleepable helpers that use __bpf_get_task_stack have buf defined as ARG_PTR_TO_UNINIT_MEM argument and we should initialize the buf on every return path. Adding missing buf memset for __bpf_get_task_stack fail paths. This provides deterministic buffer contents, which is useful when the buffer is used directly as a map key. Fixes: 06ab134ce8ec ("bpf: Refcount task stack in bpf_get_task_stack") Fixes: b992f01e6615 ("bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-10-jolsa@kernel.org
2026-08-05bpf: Remove trace_in argument from __bpf_get_stackJiri Olsa1-14/+35
Now with the new callchain_* helper functions we can process trace_in case directly in bpf_get_stack_pe function and remove it from __bpf_get_stack which makes things easier for preemption fix in following change. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-9-jolsa@kernel.org
2026-08-05bpf: Factor callchain_finalize function from __bpf_get_stackJiri Olsa1-8/+17
The new callchain_finalize function calls the build-id retrieval (if needed) and zeroes the buffer. This makes things easier for preemption fix in following change. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-8-jolsa@kernel.org
2026-08-05bpf: Factor callchain_store function from __bpf_get_stackJiri Olsa1-14/+25
The new callchain_store function stores trace entries buffer into user supplied buffer. It covers both just-ip and buildid data. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-7-jolsa@kernel.org
2026-08-05bpf: Disable preemption in bpf_get_stackidJiri Olsa1-11/+13
The get_perf_callchain call needs disabled preemption plus we need it disabled as long as we access its returned trace entries buffer. Note the bpf_get_stackid_pe function is executed already with preemption disabled. Fixes: d5a3b1f69186 ("bpf: introduce BPF_MAP_TYPE_STACK_TRACE") Reported-by: Tao Chen <chen.dylane@linux.dev> Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260803210149.296496-6-jolsa@kernel.org Closes: https://lore.kernel.org/bpf/20260206090653.1336687-2-chen.dylane@linux.dev/
2026-08-05bpf: Use stack id functions instead of __bpf_get_stackidJiri Olsa1-16/+27
Replacing __bpf_get_stackid calls with sequence of following functions: stackid_fastpath stackid_new_bucket stackid_install This makes code more structured and allows us to easily disable preemption only in bpf_get_stackid in following changes. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-5-jolsa@kernel.org
2026-08-05bpf: Factor stackid_new_bucket from __bpf_get_stackidJiri Olsa1-18/+30
The new stackid_new_bucket allocates the new bucket and initializes it with the trace data. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-4-jolsa@kernel.org
2026-08-05bpf: Factor stackid_fastpath function from __bpf_get_stackidJiri Olsa1-15/+28
The new stackid_fastpath does the fast stack hash and trace check, that does not need new bucket allocation. It covers both just-ip and buildid code paths. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-3-jolsa@kernel.org
2026-08-05bpf: Factor stackid_init function from __bpf_get_stackidJiri Olsa1-36/+59
The new stackid_init function stores all the necessary bits for stackid trace and it will be used by other functions in following changes. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260803210149.296496-2-jolsa@kernel.org
2026-08-05bpf: Inline bpf_iter_num_destroy() as a no-opPuranjay Mohan2-3/+5
Once destroy() returns the stack slot is no longer tracked as iterator state, so zeroing it is dead work. Make the kfunc a no-op and inline the call to a single BPF_JA 0 (the fixup can't drop the instruction outright, so emit a nop; the JITs elide it). Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260804134601.2305303-5-puranjay@kernel.org
2026-08-05bpf: Inline bpf_iter_num_next() kfuncPuranjay Mohan1-0/+17
bpf_iter_num_next() runs on every bpf_for() iteration, so inlining it drops a call from the loop body. R1 points to the iterator; the returned pointer to s->cur is R1 itself, since s->cur is first. s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end is a signed 32-bit compare and the inlined code needs no sign extension. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260804134601.2305303-4-puranjay@kernel.org