| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Pull bpf updates from Daniel Borkmann:
"Major changes:
- Redesign the verifier error reporting: failures now carry source
and instruction annotations along with the causal event history
that led to them, making program rejections far easier to debug and
repair (Kumar Kartikeya Dwivedi)
- Add arena argument support to kfuncs and struct_ops through the new
__arena and __arena__nullable suffixes (Tejun Heo, Puranjay Mohan,
Kumar Kartikeya Dwivedi, Ihor Solodrai)
- Signed BPF program loader rework to accommodate both BPF and
security community needs where the kernel runs the signature
verification at BPF_PROG_LOAD time before the LSM admission hook
(Daniel Borkmann)
- Add a set of ksock kfuncs which let BPF LSM and syscall programs
create, connect and send on UDP sockets in order to emit telemetry
data (Mahe Tardy)
- Unify helper and kfunc call argument verification and classify
kfunc arguments purely from BTF into a generated bpf_func_proto
which is computed once at add-call time (Amery Hung)
Other features and fixes:
- Enable EXECMEM_ROX_CACHE for BPF allocations on x86 (Mike Rapoport)
- Add bidirectional VLAN support to bpf_fib_lookup() through the new
BPF_FIB_LOOKUP_VLAN and BPF_FIB_LOOKUP_VLAN_INPUT flags (Avinash
Duduskar)
- Infer zext_dst from static register liveness analysis to fix 32-bit
zero-extension semantics, and remove the artificial limitations on
pointer types eligible for spilling (Eduard Zingerman)
- Inline the numeric open-coded iterator kfuncs so that bpf_for()
loops no longer pay a kfunc call on every iteration (Puranjay
Mohan)
- Add an arena-based bitmap data structure to libarena along with
serial and parallel selftests (Emil Tsalapatis)
- Teach resolve_btfids to discover kfuncs from the kernel's BTF ID
sets and to emit kfunc BTF decl tags, reducing the kernel build's
dependency on pahole features (Ihor Solodrai)
- Add BPF_F_ADJ_ROOM_DECAP_* flags to bpf_skb_adjust_room() so that
tunnel decapsulation can update the GSO and encapsulation state of
the skb (Nick Hudson)
- Fix the ring buffer pending_pos walk and the available-data
accounting on 32-bit position wrap (Israel Téllez García)
- Add memory usage accounting for arena maps and fix an mmap_lock
deadlock on arena lock failure (Jiayuan Chen)
- Add tracing_multi link info support to the kernel UAPI and bpftool,
and refactor the stack map code to run with preemption disabled
(Jiri Olsa)
- Support BPF_F_EGRESS in bpf_redirect_peer() to emit the skb in the
egress direction of the target's peer device (Jordan Rife)
- Add a KF_SPINLOCK_SAFE kfunc flag so that providers, in particular
modules, can declare kfuncs safe to call under bpf_spin_lock
instead of relying on the verifier's hard-coded allowlist (Kaitao
Cheng)
- Introduce global percpu data for BPF programs with libbpf probing
and bpftool skeleton support, and stop exposing uninitialized
kernel heap memory when copying per-CPU map values (Leon Hwang)
- Add s390 JIT support for load-acquire and store-release
instructions (Maxim Khmelevskii)
- Fix a CFI mismatch in the task work callback and an arm64 KASAN
false positive after bpf_throw() (Mykyta Yatsenko)
- Reject writes through untrusted BTF pointers and bound the
rdonly/rdwr_buf_size kfunc arguments (Nicholas Dudar)
- Invalidate RCU pointers only after the final spin unlock and
account for preempt and IRQ disabled regions as overlapping RCU
protection (Ning Ding)
- Support mixing bpf2bpf calls and tail calls on RV64, add signed
operations and 32-bit atomics to the RV32 JIT, and add timed
may_goto support (Pu Lehui, Kuan-Wei Chiu, Feng Jiang)
- Fix a use-after-free on mm_struct in bpf_find_vma() for foreign
tasks and an mmap_lock leak in the irq_work path (Sanghyun Park)
- Populate mmap-able BPF array map memory lazily which makes mmap()
O(1) instead of proportional to the map size (Song Liu)
- Introduce a jit_required flag and reject programs with inlined
helpers when no JIT is available, where the interpreter would
otherwise jump into an invalid address (Tiezhu Yang)
- Fix the x86 JIT per-CPU address resolution into an extended
register where the REX prefix dropped the high destination register
bit (Vineet Gupta)
- Reject MEM_ALLOC BTF accesses past object bounds, arena frees below
the arena base, and mixed arena and ordinary atomic paths (Yiyang
Chen)
- Fix the trampoline handling of 128-bit arguments and of return
values larger than 8 bytes (Yonghong Song)
- Ensure that any fault prone load is rewritten with exception table
handling, and fix the arena load-acquire and atomic fetch handling
in the x86, arm64, riscv and s390 JITs (Daniel Borkmann)
- Many more fixes and cleanups across the verifier, arena,
trampolines, sockmap, cgroup, ring buffer, x86/arm64/riscv/s390
JITs, libbpf, bpftool, resolve_btfids and selftests"
* tag 'bpf-next-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (373 commits)
selftests/bpf: Add tests for a store on a fault prone qdisc pointer
selftests/bpf: Add tests for fault prone loads out of RCU pointers
selftests/bpf: Add tests for pointer type merge at a shared load
selftests/bpf: Remove duplicate copies of the arena spinlock qnodes
selftests/bpf: Retry stat generation in cgroup_iter_memcg
selftests/bpf: Test pseudo-function policy diagnostics
bpf: Distinguish function references in policy diagnostics
bpf: Preserve source attribution without source text
selftests/bpf: Test kfunc argument diagnostics
bpf: Correct kfunc argument diagnostics
bpf: Use canonical stack argument names in diagnostics
bpf: Preserve R0 lineage across helper calls
selftests/bpf: Exercise negative optlen in cgroup getsockopt hook
bpf: Reject negative optlen in cgroup getsockopt hook
selftests/bpf: tc_tunnel - validate decap GSO and encapsulation state
bpf: Clear decap state on skb_adjust_room shrink path
bpf: Allow new DECAP flags and add guard rails
bpf: Add BPF_F_ADJ_ROOM_DECAP_* flags for tunnel decapsulation
bpf: Refactor masks for ADJ_ROOM flags and encap validation
bpf: Name the enum for BPF_FUNC_skb_adjust_room flags
...
|
|
Pull tracing updates from Steven Rostedt:
- Expose btf_ids to trace events
In order to allow BPF programs to attach to system call trace events
(which are actually pseudo trace events built on top of raw_syscall
events), expose the BTF ID of the events. This will allow BPF
programs better precision in attaching to events.
- Use "u64" to assign to hist_field->type
Instead of using kstrdup("u64", GFP_KERNEL) to assign the
hist_field->type, just point it to "u64" instead. The
hist_field->type is freed via kfree_const().
- Replace kmalloc()/strcpy() with kstrdup() for trace_printk
Instead of having two calls to copy the module format string, just
use kstrdup().
- Use __free() in trace event histograms and triggres where possible
- Use seq_buf in trace event code instead of strcat()
Instead of calculating the size of the buffer to use and filling it
with strcat(), use the seq_buf infrastructure that takes care of
making sure not to overflow the string size.
- Reject invalid preemptirq_delay_test CPU affinity
The preempt_delay_test module can take an invalid CPU affinity mask
and create confusing output. Simply have the module reject invalid
affinity masks.
- Prevent division by zero in ftrace_ops sample module code
If the ftrace_ops sample module code receives the module parameter
nr_function_calls set to zero, it can cause a division by zero error.
- Warn when an event dereferences a parameter in TP_printk()
On boot up and module load, the trace event TP_printk() is scanned
for possible bugs. As the TP_printk() code is executed when the user
reads the "trace" file and processes the data written when the
trace_event executed, the data it reads can be literally days old.
The scan currently checks for dereferencing printk formats like
"%pI6". But it does not check if the parameters themselves have a
dereference like:
TP_printk("offset %08x: value %08x",
(u32)(__entry->addr - __entry->edma->membase), __entry->value)
__entry represents the pointer to the event on the ring buffer. The
__entry->edma->membase is dereferencing a pointer on the ring buffer
to find membase, but the __entry->edma may no longer be a valid
pointer.
Warn on this case too.
- Replace some strcpy() with strscpy()
- Clean up mmiotrace events to use assign_type() macro
The assign_type() macro makes sure the event type is indeed the type
that is being parsed. The mmiotrace trace was written before that
macro was created so it just simply typecasted the pointer.
Replace the typecasting with the macro.
- Have the ENUM processing to numbers only process what is added
The code that converts ENUMs to their numbers in the trace events
scanned all events to do the processing. This was true when a module
was loaded too. That is, instead of processing just the events for
the module, it processed *all* events. Even the builtin ones that
were processed at boot up.
Add a check for the event->module matching mod if it is a module
before processing it.
* tag 'trace-v7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (21 commits)
tracing: Have trace_event_update_all() only handle module that is loading
tracing: Cleanup event_enable_trigger_parse() by using __free()
tracing: Report every TP_printk double dereference
tracing/mmiotrace: Use trace_assign_type() in mmio_print_mark()
tracing: Make per-template BTF id lists file-local
tracing: Use seq_buf for string concatenation
tracing: Use strscpy() instead of strcpy() in trace_sched_switch
tracing: Warn when an event dereferences a pointer in TP_printk()
samples/ftrace: Prevent division by zero when nr_function_calls is zero
tracing: Reject invalid preemptirq_delay_test CPU affinity
fgraph: Use trace_seq_putc() in print_graph_return()
tracing/user_events: Replace a seq_printf() call by seq_puts() in user_seq_show()
tracing/user_events: Use seq_putc() in two functions
tracing: Bound histogram expression strings with seq_buf
tracing: Return ERR_PTR() from expr_str()
tracing: Use __free() for expr_str() buffer
kernel/trace/trace_printk: Use kstrdup() instead of kmalloc() and strcpy()
tracing: Point constant hist field type to string literal
selftests/bpf: Add test for tracepoint btf_ids tracefs file
tracing: Expose tracepoint BTF ids via tracefs
...
|
|
add_subprogs() rejects both BPF-to-BPF calls and BPF_PSEUDO_FUNC loads for
unprivileged programs. The latter loads a subprogram address for use as a
callback, but its Policy report currently describes it as a function call and
suggests avoiding calls that the program does not contain.
Select the operation and suggestion from the instruction kind. Preserve the
existing call wording for BPF_PSEUDO_CALL, and describe BPF_PSEUDO_FUNC as a
BPF function reference.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/bpf/d02e6a6d3b2dc43a207b8ba836ce62497b250dede9252e7409c5212201c794b7@mail.kernel.org
Link: https://lore.kernel.org/bpf/20260816015746.2632990-14-memxor@gmail.com
|
|
GCC emits BTF line records with a file name and line number, but leaves the
source line string empty. bpf_diag_source() currently treats that empty string
as if the complete line record were unavailable, so diagnostics fall back to
an instruction number and discard the function, file, and line attribution.
Print the available source location before deciding whether source context can
be rendered. When source text is absent, omit only the source context and retain
the diagnostic annotation and instruction context.
Fixes: b9c5d822f677 ("bpf: Add source and instruction diagnostic context")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260816015746.2632990-12-memxor@gmail.com
|
|
The Call Type Safety diagnostics mishandle three kfunc argument classes.
BTF type ID 0 represents void, but btf_show_name() also uses zero to end
type traversal. A pointer that resolves to void therefore loses its pointee
name and is rendered as "()". End traversal directly for concrete terminal
types, but resolve referenced types before testing for ID zero, and name the
void terminal type explicitly. Format the complete parameter pointer type
for nullable kfunc arguments, so void pointers are reported as (void *).
Also add the missing structured report when an __szk memory-size argument is
not a verifier-known constant. Describe the generic bpf_refcount_acquire()
contract without deriving an object type from its void pointer prototype.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/668871823f90f69896d3db27b56db2f53e481162.camel@gmail.com
Link: https://lore.kernel.org/bpf/20260816015746.2632990-7-memxor@gmail.com
|
|
The main diagnostic identifies the first outgoing stack slot as stack
argument 1 and the sixth function argument. The causal history instead
labels the same value as stack arg6, making it look like a different slot.
Render causal-history targets in the verifier's canonical stack-argument
location form. The first outgoing slot is now shown as *(R11-8), matching
reg_arg_name(), while the main diagnostic retains its fuller slot and
ordinal description.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/eb1be5327d136b7e5bd6d68e76fef6de20c40790.camel@gmail.com
Link: https://lore.kernel.org/bpf/20260816015746.2632990-6-memxor@gmail.com
|
|
check_helper_call() clears all caller-saved registers before taking the
diagnostic snapshot of R0. This records NOT_INIT as the old state for every
helper return and loses the lineage of the value held in R0 before the call.
bpf_diag_record_caller_saved() deliberately skips R0 because the paired
modification scope is responsible for it.
Open the R0 modification scope before clearing caller-saved registers,
matching the kfunc, ld_abs, and subprogram call paths.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260815073833.A93A91F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/48e6f021b89562f68850fe21ef8c78719819b04cf9c4e4f50bc791937d37ace8@mail.kernel.org
Link: https://lore.kernel.org/bpf/20260816015746.2632990-4-memxor@gmail.com
|
|
A cgroup getsockopt BPF program can shrink ctx->optlen after the
kernel getsockopt handler has run. The kernel-buffer variant, used by
TCP_ZEROCOPY_RECEIVE, only rejects values larger than the original
length.
If BPF writes a negative optlen, that value is accepted and propagated
back to the TCP getsockopt code. It can then be passed to
copy_to_sockptr() as a size_t and trigger the hardened usercopy
bytes > INT_MAX warning.
Reject negative ctx.optlen in __cgroup_bpf_run_filter_getsockopt_kern(),
matching the lower-bound validation already present in the sockptr-based
getsockopt hook.
Fixes: 9cacf81f8161 ("bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE")
Reported-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Junseo Lim <zirajs7@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/187a4d756275aaaee5d65eecb63c1477b3b66554.1786448307.git.zirajs7@gmail.com
|
|
bpf_convert_ctx_accesses() turns a BPF_LDX into a BPF_PROBE_MEM one by
matching the type recorded for the insn against a list of exact pointer
types. The list cannot keep up with the flag combinations the verifier
produces, and a type which is missing from it ends up as a plain load
without an exception table entry, so a bad address panics the kernel
instead of being handled.
Two such types exist today and are reachable:
- PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_ALLOC | NON_OWN_REF
- PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU
Rather than adding the two, just drop the list and state the property
itself in the default case of the switch. This is a superset of what
the list matched, the untrusted PTR_TO_MEM does not have to carry
MEM_RDONLY for it anymore, and it stays in sync with the verifier side
which uses the same match in save_aux_ptr_type() and reg_type_mismatch_ok().
Assert that a fault prone type which does not get the rewrite for whatever
reason is rejected at load time rather than left to fault at runtime to
catch any future cases.
Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Fixes: 6fcd486b3a0a ("bpf: Refactor RCU enforcement in the verifier.")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-4-daniel@iogearbox.net
|
|
check_ptr_to_btf_access() allows the program to store before the default
BTF access path gets to reject a non read access. ac65c710cc64 ("bpf:
Reject writes through untrusted BTF pointers") closed that for a
PTR_UNTRUSTED pointer, but a bare PTR_TO_BTF_ID may fault on a dereference
just the same and is let through.
A BPF_LDX gets the BPF_PROBE_MEM rewrite in bpf_convert_ctx_accesses()
and a bad address is handled, but a BPF_STX does not and cannot, there
is no probed store to rewrite. The store is emitted as a plain one without
an exception table entry and a bad address panics the kernel.
A bpf_qdisc program can reach this, bpf_qdisc_btf_struct_access() permits a
write to Qdisc::limit and Qdisc::next_sched is a plain struct Qdisc pointer
which the walk turns into the compat type:
struct Qdisc *next = sch->next_sched;
next->limit = 1000;
BUG: kernel NULL pointer dereference, address: 0000000000000014
RIP: 0010:bpf_prog_c6e14e7f32c8e325_bpf_fifo_enqueue+0x3a/0x12b
Code: [...] bf e8 03 00 00 <89> 7e 14 41 8b 7f 14 [...]
Kernel panic - not syncing: Fatal exception in interrupt
Fix by widen the check to bpf_may_fault_on_deref() so that it covers both.
Fixes: 27ae7997a661 ("bpf: Introduce BPF_PROG_TYPE_STRUCT_OPS")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-3-daniel@iogearbox.net
|
|
reg_type_mismatch_ok() enumerates the pointer types which must not
silently share a BPF_LDX with a different one, since the type recorded
for the insn drives a rewrite in bpf_convert_ctx_accesses().
f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()") added
PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED as another type in need of one,
namely the BPF_PROBE_MEM rewrite, but did not add it there. Fix it by
adding the missing case to reg_type_mismatch_ok(), so that a PTR_TO_MEM
which may fault on deref is not mismatch ok anymore. The triage in
save_aux_ptr_type() then merges them.
Fixes: f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-2-daniel@iogearbox.net
|
|
When the same BPF_LDX instruction is reached through paths that yield
different pointer types, save_aux_ptr_type() merges them into a single
type which is later used by bpf_convert_ctx_accesses() to decide whether
the load has to be rewritten into a BPF_PROBE_MEM one.
Before f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()")
the merge only accepted two PTR_TO_BTF_ID pointers and unconditionally
fell back to PTR_TO_BTF_ID | PTR_UNTRUSTED, so the merged type was always
one that gets the BPF_PROBE_MEM rewrite. However, the mentioned commit
widened the merge to also cover a PTR_TO_MEM base and replaced the
fallback by a union of the PTR_UNTRUSTED and MEM_RDONLY flags.
A union of flags though cannot express the property the later rewrite
is built upon, some examples:
- PTR_TO_MEM merged with PTR_TO_BTF_ID | PTR_UNTRUSTED gets
PTR_TO_MEM | PTR_UNTRUSTED but only the MEM_RDONLY variant is valid
- PTR_TO_MEM merged with a plain PTR_TO_BTF_ID gets PTR_TO_MEM
dropping the rewrite the latter type would have gotten
- PTR_TO_MEM | MEM_RDONLY merged with a plain PTR_TO_BTF_ID gets
PTR_TO_MEM | MEM_RDONLY which is not rewritten either since only
its PTR_UNTRUSTED variant is
In all three cases a program can take the unsafe path at runtime with a
NULL or otherwise bad pointer and panic the kernel on the faulting load:
BUG: kernel NULL pointer dereference, address: 0000000000000038
RIP: 0010:bpf_prog_77531a87032eeaf1_mixed_mem_btf_id_type+0x4b/0x65
Call Trace:
<TASK>
bpf_test_run+0x20b/0x460
bpf_prog_test_run_skb+0x650/0xbe0
__sys_bpf+0xb96/0x3140
__x64_sys_bpf+0x2c/0x40
do_syscall_64+0xba/0x590
Kernel panic - not syncing: Fatal exception in interrupt
Note that the last two shapes have to be fixed right here, otherwise
the merged type retains nothing which marks the load as fault prone,
thus no rule in bpf_convert_ctx_accesses() can recover it. Fix it by
normalizing the merged type instead.
Reuse it in is_load_acq_unsafe() to avoid open coding, and trim the
overly verbose comment which is more of an implementation detail of
bpf_convert_ctx_accesses() anyway.
Fixes: f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260814215301.709827-1-daniel@iogearbox.net
|
|
Atomic RMW verification records an instruction pointer type only when the
current destination is PTR_TO_ARENA. A second path can therefore reach the
same instruction with an ordinary pointer without comparing it against the
saved arena type.
The post-verification fixup uses the saved type to rewrite the instruction
to BPF_PROBE_ATOMIC for every path. Record the actual destination type for
all atomic RMW paths so the existing mismatch check rejects incompatible
uses of one instruction.
Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-1-4644c1886dbc@mails.tsinghua.edu.cn
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Add BPF kfuncs that allow BPF LSM programs to create and use sockets for
sending data. This provides a mechanism for BPF programs to emit
telemetry. For this first patch set, it's restricted to SOCK_DGRAM
socket types with IPPROTO_UDP protocol but could be easily extended to
SOCK_STREAM and IPPROTO_TCP in the future.
The API consists of five kfuncs:
bpf_ksock_create() - Create a socket (sleepable)
bpf_ksock_connect() - Connect socket to remote address (sleepable)
bpf_ksock_send() - Send data through the socket (sleepable)
bpf_ksock_acquire() - Acquire a reference to a socket context
bpf_ksock_release() - Release a reference (cleanup via
queue_rcu_work since sock_release sleeps)
The setup kfuncs bpf_ksock_create, bpf_ksock_connect, can be called from
SYSCALL programs only. While bpf_ksock_acquire, bpf_ksock_release and
bpf_ksock_send can be called from SYSCALL and LSM programs.
The implementation follows the established kfunc lifecycle pattern
(create/acquire/release with refcounting, kptr map storage, dtor
registration). The kernel socket is wrapped in a refcounted bpf_ksock
struct. Cleanup is deferred via queue_rcu_work() because sock_release()
may sleep.
The kfuncs are only compiled when CONFIG_INET is enabled, as they
specifically support AF_INET and AF_INET6 sockets.
The socket operations go through the expected LSM hooks instead of
by-passing them like many kernel sockets since those are created by BPF
programs and thus system users. Thus, the bpf_ksock_send() kfunc, which
is exposed to LSM progs has a verifier filter protection to avoid
recursion so that the whole bpf_kfunc_set kfunc set cannot be called in
a program attached to security_socket_sendmsg(). Also, because of the
LSM checks, we prevent the use of the kfuncs from asynchronous workqueue
as the current value would then be invalid.
In bpf_ksock_create(), we copy the arg values to avoid TOCTOU races
since the kfunc can sleep and the arg values could be stored in a map
that could be re-written by BPF progs or even userspace programs if the
map is mmaped.
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260813110540.103550-3-mahe.tardy@gmail.com
|
|
Augment selected helper and kfunc allowability failures with Policy reports.
These reports explain which requested operation is forbidden and why, without
adding path history for non-path-dependent policy checks.
Cover unprivileged bpf2bpf and kfunc use, helper program-type restrictions,
GPL-only helpers, helper-specific allow callbacks, kfunc allowability, and
destructive kfunc capability checks.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-15-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected whole-program and subprogram CFG validation failures with
Program Structure reports. These errors are structural rather than
path-dependent, so the reports focus on source and instruction context
instead of causal history.
Cover direct and indirect jumps outside the program or current subprogram,
unprivileged backedges, missing and out-of-range jump tables, targets in the
second half of an ldimm64, unreachable instructions, subprogram fallthrough,
and recursive bpf2bpf call graph edges.
Format long jump-range reasons directly in diagnostics.c, and keep the
fallthrough suggestion aligned with the verifier check by suggesting exit or
explicit jumps.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-14-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected sleepability and critical-section failures with Execution
Context Safety reports. Keep the existing verifier messages and add source
context, path history, and suggestions tied to the active context.
Use the context history recorded earlier to anchor causal paths to lock, IRQ,
RCU, and preempt regions instead of unrelated register updates.
Cover global calls while holding a lock, sleepable global function calls,
sleepable helpers, sleepable kfunc calls from disallowed contexts, operations
that exit while a context is still active, and unmatched context exits.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-13-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected helper and kfunc argument-contract failures with Call Type
Safety reports. Keep the existing terse verifier messages and add reason,
source context, causal register or stack-argument history, and targeted
suggestions.
Cover helper register-type mismatch, helper and kfunc non-NULL pointer
requirements, release-helper ownership requirements, scalar and constant kfunc
arguments, trusted and RCU pointer contracts, kfunc memory arguments,
memory/length pairs, refcounted kptrs, constant strings, and IRQ flag stack
arguments.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-12-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected Resource Lifetime Safety failures with structured diagnostics
while preserving the existing verifier messages.
Report unreleased references from check_reference_leak() using
reference-scoped diagnostic history, and add state reports for dynptr,
iterator, lock, and IRQ-flag lifetime misuse.
IRQ restore mismatch and out-of-order diagnostics use IRQ context-scoped
history when an IRQ-disabled region is active, so retained save/restore context
is still visible after per-state history removal.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-11-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected memory-range verifier failures with Memory Safety reports
while preserving the existing terse verifier messages for compatibility.
Cover stack spill corruption, uninitialized stack reads, variable stack helper
accesses, and check_mem_region_access() range-proof failures. The bounds report
spells out the required offset + access_size <= object_size proof with concrete
values and uses scoped diagnostic history for causal context.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-10-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Augment selected register-state verifier failures with Register Type Safety
reports. The existing verbose verifier messages remain in place; the new
reports add reason, source context, causal path, and suggestions.
Cover invalid pointer dereferences, unreadable registers, missing outgoing
stack arguments for bpf2bpf and kfunc calls, and rejected pointer arithmetic.
Use scoped diagnostic history so reports start from the latest relevant value
change and then show later branch outcomes.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-9-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Record verifier context transitions in the diagnostic history so later reports
can anchor causal paths to the critical section that made an operation invalid.
This covers lock, IRQ, RCU, and preempt regions without adding any new
verifier error reports. Category-specific commits decide where those recorded
events should be rendered.
Use context depth when selecting scoped history so nested regions anchor at the
outer active region, and fall back to the earliest retained event when the
matching entry was pruned.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Add reference acquire and release events to diagnostic history so Resource
Lifetime Safety reports can show the lifetime of a specific reference id along
the path.
Record acquisitions after the verifier assigns the reference id. Record
releases only after release_reference_nomark() succeeds, including the
kptr_xchg RCU conversion path and owning-to-non-owning conversion path that
consume an owning reference.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Record material register and outgoing stack argument changes so diagnostics can
explain how a value reached its current type, bounds, or unreadable state.
Store old and new register types, scalar ranges, tnum value and mask, map and
BTF type identity, and basic operand metadata in the environment-owned
diagnostic event stream.
Record invalidations when packet data moves, references are released, or
borrowed references leave their protected region. Register-scoped history
starts at the latest matching modification and then shows later branch
outcomes.
Also record fixed stack spills and overwrites, and tag register fills from
stack so register-scoped history can follow value flow through spilled stack
slots.
The type_is_map_ptr() helper previously lived as a static function in
kernel/bpf/log.c since commit 0c95c9fdb696 ("bpf: emit map name in register
state if applicable and available"). Move it verbatim to
include/linux/bpf_verifier.h as a static inline, next to the other type
classifiers, so diagnostics.c can reuse it without duplicating the case list.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Save the diagnostic event-log position with each verifier stack entry and
reset the environment-owned stream together with the normal verifier log
when a queued state is popped. Also reset the diagnostic stream after
successful subprogram verification even when level-2 logging preserves the
normal verifier log.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Add an environment-owned diagnostic history for verifier reports. Event
payloads keep the user-facing branch history shape, while storage lives
in bpf_verifier_env and follows the active verifier path.
Grow the event array geometrically up to a 64 MiB limit. Once storage
reaches the limit, or an allocation fails, overwrite the oldest event so
diagnostics retain the newest useful suffix without adding per-event
metadata.
Represent saved positions as absolute logical sequence numbers. A restore
truncates to a retained position. If its prefix has already been evicted,
clear the abandoned suffix and preserve the missing-history position. This
keeps marks stable across rotation without increasing their size.
Add the branch event renderer and branch recording.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Teach verifier diagnostics to annotate an instruction with BTF source
line information and nearby BPF instructions. The renderer keeps source
text in a fixed-width lane and prints instructions in a stable right-hand
gutter.
Wrap annotation text under the source line so long error labels remain
readable while the source and instruction lanes keep their fixed layout.
Keeping source and instruction context in one commit preserves the visual
layout contract that later diagnostic reports rely on.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Add the initial diagnostics renderer for verifier reports and wire it into
the BPF build. The helper emits the common failure header through the
verifier log.
Later patches add prose wrapping, reusable report sections, and source and
instruction context for category-specific diagnostics.
Gate the helpers on normal verifier log output from the start, so
BPF_LOG_STATS-only loads do not collect or render diagnostics.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.
The handler is reached through a new optional ->map_mmap_fault callback.
Maps that provide it get a vm_operations_struct with a .fault handler;
maps that populate their mapping eagerly keep the one they had. Both
share the same open/close callbacks, so the existing VMA accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized
rather than each map installing its own vm_operations_struct.
Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.
Time for one mmap()+munmap() of an 8MiB mmap-able array map:
before after
no MAP_POPULATE, no access 226us 1.1us
no MAP_POPULATE, access all pages 236us 1341us
MAP_POPULATE, no access 312us 493us
MAP_POPULATE, access all pages 318us 519us
Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org
|
|
In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer
and overwrite positions before measuring how much data is available:
return prod_pos - max(cons_pos, over_pos);
max() is an ordering comparison, and consumer_pos, producer_pos and
overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures,
where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the
two positions has wrapped and the other has not, max() returns the older
one: the result is then a modular difference close to 2^32, so the
function reports far more available data than the ring can hold. Pollers
using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be
woken with nothing to read.
Compare distances rather than positions. prod_pos - X is the amount of
data produced since X for either position, wrap or no wrap, so the newer
position is simply the one with the smaller distance, which is also the
value the function wants to return.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap. Found by review of the same class of bug fixed in
"bpf: Fix pending_pos walk on 32-bit ring position wrap".
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com
|
|
The reservation path caches the position of the oldest not-yet-committed
record in rb->pending_pos and advances it past already committed records
on every reservation:
while (pend_pos < prod_pos) {
consumer_pos, producer_pos and pending_pos are unsigned long, i.e.
32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states
that these counters may wrap around there. Every other comparison in the
file is written as a difference, so modular arithmetic keeps them
correct across the wrap. This one is an ordering comparison, and it is
not wrap-safe.
Once producer_pos wraps past 2^32, prod_pos is small while pend_pos
still holds its pre-wrap value, so the loop condition is false and
pending_pos is never advanced again. Reservations keep succeeding for a
while, because bpf_ringbuf_has_space() uses differences, but
new_prod_pos - pend_pos grows as the producer advances, and once it
exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails:
the kernel believes a pending record spans the whole buffer. The ring
never recovers, bpf_ringbuf_output() drops every event from then on, and
nothing is logged.
Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a
tracepoint-based collector with a 512 KiB ring and 160-byte records.
Every one of them stopped delivering after exactly 26846821 records and
4295491360 bytes had passed through the ring, at event rates between 441
and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is
the byte count, not time or load. That figure is 2^32 plus 524064 bytes,
and the excess is one ring's worth of grace period, as expected while
new_prod_pos - pend_pos is still below rb->mask. The last reservation
that fits is the largest record boundary X with X + 160 <= 524287, and
since 2^32 mod 160 = 96 the boundaries after the wrap sit at
X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally
until the producer stopped, then read zero records for good. With this
patch applied, one of the four devices took 10 GiB through the same ring
with no stall, while the three unpatched ones kept wedging at the same
byte count.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap.
Compare the two positions as a difference instead. pending_pos never
runs ahead of producer_pos, so the unsigned difference is the real
distance between them and stays correct across the wrap.
Fixes: cfa1a2329a69 ("bpf: Fix overrunning reservations in ringbuf")
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-2-i.tellez@btesa.com
|
|
arm64 passes zero as the stack pointer while walking BPF frames, so
bpf_throw() leaves stale KASAN stack poison after jumping to the
exception callback.
Use the frame pointer as the fallback stack watermark.
Fixes: e74cb1b42213 ("arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT")
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260812-hello_world-v1-1-c3c2ddcb362d@meta.com
|
|
The verifier keeps per-subprogram metadata in three parallel arrays:
subprog_info, func_info, and func_info_aux. Dead code elimination can
remove whole subprograms, and adjust_subprog_starts_after_remove()
shifts subprog_info and func_info to close the gap, but leaves
func_info_aux in place. From that point on, func_info_aux[i] no longer
describes subprogram i.
Shift func_info_aux together with func_info so the three arrays stay
aligned after subprogram removal.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com
|
|
Introduce global percpu data, inspired by the commit
6316f78306c1 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
|
|
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
|
|
There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.
Drop them for cleanup.
No functional changes intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev
|
|
The commit 7619a0ee9340 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
bpf_trampoline_multi_detach() always returns 0 and the sole caller
ignores the return value. Change it to return void and drop the
WARN_ON_ONCE at the call site.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
When bpf_trampoline_update() fails before modify_fentry_multi()/
unregister_fentry_multi() is called, cur_image is unchanged
(cur_image == old_image) and ftrace still calls into it. Freeing
old_image in that case causes a UAF.
Only free old_image when it differs from cur_image.
Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.
Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
It was an effort to enhance fscache as a kernel cache for lazy
pulling (at least according to previous Incremental FS discussion [1])
and EROFS over fscache was the in-tree user of this mode.
fscache has since evolved to be netfslib-oriented, serving network
filesystem inodes via the netfs library, but EROFS never acts as a
network filesystem and we need to cache golden filesystem images rather
than individual EROFS inodes.
Since EROFS over fscache is now removed, clean up netfs/fscache/
cachefiles upstream too.
[1] https://lore.kernel.org/r/CAOQ4uxi4dzxArY24YO=+kBCK2gGoq3Ptb8WkzCqSogPgU_R3dQ@mail.gmail.com
[dh] Fixed up comments on:
https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com
https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com
Signed-off-by: Gao Xiang <xiang@kernel.org>
Signed-off-by: David Howells <dhowells@redhat.com>
Link: https://patch.msgid.link/1046393.1786544127@warthog.procyon.org.uk
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-erofs@lists.ozlabs.org
cc: bpf@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states
must not be equal, or the verifier can prune an unsafe path.
Compare the pointer type for STACK_ITER slots.
Fixes: dfab99df147b ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS")
Signed-off-by: Ning Ding <dingning04@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Reject tracing and freplace attachments to a target program with arena
context arguments. The struct_ops indirect trampoline converts those
arguments before entering the target, so a generic tracing trampoline
would otherwise expose arena offsets using the target BTF pointer type.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-14-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
A struct_ops callback cannot receive an arena pointer directly, so
passing one takes two steps. The pointer arrives as a bare u64 that the
callback casts, and because the two sides address the arena through
different bases it also has to be rebased by hand on the way in.
Add the __arena and __arena__nullable stub argument suffixes to make this
convenient. The callback declares the parameter as an arena pointer,
receives it as a PTR_TO_ARENA register, and dereferences it directly,
while the kernel caller just passes the natural kernel arena address
(kaddr). The trampoline converts the value while saving the arguments
into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the
program never sees a kernel address and nothing rewrites the ctx after
the fact. The converted value keeps the upper 32 bits clear as the JITs
require of arena pointer registers and behaves like any cast_kern'ed
arena pointer, so cast_user recovers the full user-visible address.
__arena converts unconditionally and the kernel caller must not pass
NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel
pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a
PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the
program's type state already encompasses NULL-ness, so it is not
meaningful to force a NULL check for the program.
The composite suffix intentionally ends in __nullable. Classify
__arena__nullable before the generic suffix so scalar arena pointees do
not take the generic nullable BTF pointer path.
This patch adds the generic side. prepare_arg_info() records arena and
nullable argument flags in the struct_ops function model, and
bpf_tramp_arena_base() returns the arena base for a single-program
struct_ops indirect trampoline. Only that trampoline converts: its
program's arena is fixed at generation time. Generic trampolines can mix
programs with different arenas and reject arena context arguments
defensively, which is unreachable today as only struct_ops programs
carry them. Architectures that do not implement the conversion are
gated out at verification time with bpf_jit_supports_arena_args().
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Passing an arena pointer to a kfunc takes two steps today. There is no
arena pointer argument type, so the pointer crosses the boundary as a
bare scalar, and the kfunc then offsets it by the arena base and casts
it before it can touch the memory. Every such kfunc open-codes the same
translation.
Add the __arena and __arena__nullable argument suffixes to make this more
convenient. The kfunc declares the parameter by its real pointer type
and dereferences it directly, with the JIT rebasing the value at the
call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the
u32 offset stays within the guard-padded arena kernel mapping, and a
fault on an unpopulated page recovers through the per-arena scratch
page. A suffixed argument accepts a PTR_TO_ARENA or scalar register,
matching global subprog arena arguments.
__arena rebases unconditionally, so the kfunc never sees NULL and a
value with zero in the low 32 bits arrives as the arena base.
__arena__nullable preserves NULL for optional arguments by skipping the
rebase when the truncated value, arena offset 0, is zero. Keeping the
plain form NULL-free saves the NULL test on every call.
The double separator makes the annotations composable:
__arena__nullable also ends in __nullable and naturally follows the
common nullable argument path. Plain __arena follows that path too for
verifier type checking because both forms accept a constant zero; the
function-model flag still determines whether the JIT preserves NULL or
rebases it to the arena base.
This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args() and distilled into argument flags in the function
model stored in the kfunc descriptor. JITs retrieve the model while
emitting the call, avoiding per-call state in insn_aux_data.
JITs declare support with bpf_jit_supports_arena_args() and verification
fails with -ENOTSUPP elsewhere.
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The kfunc descriptors include argument prototypes generated while calls are
collected. Some argument classifications need program auxiliary state derived
from referenced maps, such as the arena associated with the program.
This avoids a footgun in get_kfunc_arg_type() checks where we do
validation on whether program has prog->aux->arena and it hasn't been
resolved yet.
check_and_resolve_insns() records used maps and populates that state. It must
remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations,
so that instruction validation and the program tag observe the relocated
instruction stream.
Move only add_kfuncs() after instruction and resource resolution. Subprogram
discovery and validation remain before the full BTF phase because that phase
needs the complete subprogram layout. Add a short comment describing the
resource resolution phase at the call site.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
Linus Torvalds
Kumar Kartikeya Dwivedi
Junseo Lim
Daniel Borkmann
Yiyang Chen
Mahe Tardy
Song Liu
Israel Téllez García
Mykyta Yatsenko
Leon Hwang
Xu Kuohai
Hui Zhu
Gao Xiang
Ning Ding
Tejun Heo