aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/kernel
AgeCommit message (Collapse)AuthorFilesLines
2026-09-04bpf: Clear NON_OWN_REF after RCU protection endsKumar Kartikeya Dwivedi1-4/+14
A local kptr load of an object containing a graph node is marked MEM_RCU and NON_OWN_REF while protected by RCU. When the last RCU read-side critical section ends, invalidate_rcu_protected_refs() removes MEM_RCU and marks the pointer PTR_UNTRUSTED, but leaves NON_OWN_REF set. The stale flag lets graph kfunc argument checks continue treating the pointer as a live borrowed reference. In particular, bpf_rbtree_remove() can accept a pointer after its protection ended and return it as a new owning reference, even though the object may already have been freed. Clear NON_OWN_REF when an RCU-protected pointer is demoted. A spin lock also provides implicit RCU protection, so invalidate non-owning references before demoting RCU-protected pointers when releasing the lock. Otherwise the demotion would clear the flag before invalidate_non_owning_refs() can find and invalidate those aliases. The demoted pointer remains available for fault-protected reads. Exempt such reads from the allocated-object reference-state assertion; writes through a fault-prone pointer are already rejected, and bpf_may_fault_on_deref() makes the surviving loads use BPF_PROBE_MEM. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260904084325.52250-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-04bpf: Keep refcount_acquire nullable for borrowed RCU kptrsNing Ding1-1/+1
bpf_refcount_acquire() is fallible for a borrowed reference because the object may have reached a zero refcount. The verifier therefore keeps KF_RET_NULL on the return value unless the argument is an owning reference. An RCU-protected load of a local kptr is marked MEM_ALLOC, but it only receives NON_OWN_REF when the pointee contains a graph node. A refcounted object without a graph node consequently looks like an owning reference even though the loaded register has no acquired reference state. If the program drops the last real reference while remaining in the RCU critical section, refcount_inc_not_zero() returns NULL while the verifier treats the result as non-NULL. Only classify the argument as owning when it is backed by a verifier-tracked reference. This retains the non-NULL return for pointers from bpf_obj_new(), bpf_kptr_xchg(), or an earlier successful acquisition, while requiring a NULL check for borrowed RCU kptrs. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Ning Ding <dingning04@gmail.com> [ kkd: Rewrote commit log ] Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260904084325.52250-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-04bpf: Require MEM_PERCPU for percpu kptr storesKumar Kartikeya Dwivedi1-0/+7
map_kptr_match_type() treats perm_flags as the set of register type flags that a kptr field permits. Adding MEM_PERCPU to that set for BPF_KPTR_PERCPU does not require the source register to carry it, however. The subset test consequently accepts both a plain bpf_obj_new() allocation and a referenced kernel pointer into a __percpu_kptr map field. Loads from the field are always marked MEM_PERCPU. Consumers then treat the stored value as the cookie returned by bpf_percpu_obj_new(): per-CPU pointer helpers relocate it, and map teardown selects the per-CPU free path. A plain allocation can therefore provide an arbitrary kernel read/write, while a kernel pointer can be relocated into an invalid address or sent through a missing destructor. Require the source MEM_PERCPU flag to match the destination field kind. This preserves valid bpf_percpu_obj_new() stores and rejects both the program-BTF and kernel-BTF variants. Fixes: 36d8bdf75a93 ("bpf: Add alloc/xchg/direct_access support for local percpu kptr") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260904084325.52250-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-04bpf: Mark the zero register precise for a register-form NULL checkEduard Zingerman1-0/+9
check_cond_jmp_op() accepts "if rA <op> rB" as a NULL check for a nullable pointer rA when rB is a scalar known to be zero, lifts PTR_MAYBE_NULL from rA in the corresponding branch and does not mark rB precise. Consider the following program: r0 = bpf_get_prandom_u32(); r6 = 1; /* the r6 == 0 path is explored first */ if (r0 == 0) goto 1f; r6 = 0; 1: r0 = bpf_map_lookup_elem(map, &0); /* absent, NULL at runtime */ if (r0 == r6) goto 2f; /* taken as a NULL check for r0 */ *(u8 *)(r0 + 0); /* verifier: map value; runtime: zero */ 2: return 0; The r6 == 0 path is explored first and the dereference is accepted. The r6 == 1 path is pruned at the checkpoint recorded for (1), so the comparison is never verified with a non-zero r6. At runtime a failed lookup returns NULL, NULL != 1 takes the non-NULL edge and the program dereferences a pointer that is zero. Fixes: 2f4cb53eed44 ("bpf: detect non null pointer with register operand in JEQ/JNE.") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260904083325.2083493-7-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-09-04bpf: Don't predict JMP32 pointer vs zero comparisonsEduard Zingerman1-0/+7
Consider the following program: r1 = map_value; /* low 32 bits are zero at runtime */ r6 = 0xdead000000000000; if w1 != 0 goto l1; l0: r1 += r6; r2 = *(u64 *)(r1 + 0); exit; l1: r6 = 0; goto l0; At the moment is_branch_taken() reports the jump as always taken, because it does not distinguish between BPF_JMP and BPF_JMP32 comparisons when processing 'if w1 != 0 ...'. Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260904083325.2083493-5-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-09-04bpf: Don't resurrect a scalar id dropped by collect_linked_regs()Eduard Zingerman1-4/+10
check_cond_jmp_op() copies the compared registers into env->{false,true}_reg{1,2} before collect_linked_regs() runs and copies those snapshots back into both branch states afterwards. collect_linked_regs() records at most LINKED_REGS_MAX members of a linked registers group in the jump history and calls clear_scalar_id() for every member that does not fit. The compared register is not exempt from that. As a consequence, sync_linked_regs() might adjust ranges for more registers than bpf_bt_sync_linked_regs() can propagate precision to. Collect the linked registers before the snapshots are taken instead. This might lead to some unnecessary clear_scalar_id's, but from previous testing situations with many linked registers are extremely rare. Fixes: ec1d77cb0ee9 ("bpf: Use bpf_verifier_env buffers for reg_set_min_max") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260904083325.2083493-3-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-09-04bpf: Don't infer non-NULL from a pointer with an unbounded offsetEduard Zingerman1-0/+7
reg_not_null() decides that a register holds a non-NULL value by looking at its type alone. For pointer types that allow arithmetic the type only guarantees a non-NULL base, in case of an unbound offset the runtime offset value might still add up to NULL. Consider the followng program: r6 = bpf_map_lookup_elem(map, &0); /* present */ if (r6 == 0) return 0; r7 = bpf_map_lookup_elem(map, &1); /* absent, NULL at runtime */ r8 = r7; r8 -= r6; /* pointer - pointer: unknown scalar, -r6 */ r8 <<= 1; r8 >>= 1; /* any non-negative offset is accepted by */ /* check_reg_sane_offset_ptr() */ r6 += r8; /* verifier: map value; runtime: zero */ if (r7 != r6) return 0; *(u8 *)(r7 + 0); /* r7 is inferred non-NULL, both are zero */ At runtime both registers are zero, the comparison is true and the load faults with NULL pointer dereference. Require the offset to be within +-BPF_MAX_VAR_OFF in reg_not_null(). Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ") Reported-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/bpf/20260904083325.2083493-1-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-09-04exit: hold a reference to thread_pid across proc_flush_pidDaehyeon Ko1-3/+6
Commit 0a36bad01731 ("release_task: kill the no longer needed get/put_pid(thread_pid)") removed the reference around proc_flush_pid(). It assumed that free_pids(post.pids) at the end of release_task() would keep thread_pid alive until then. That assumption is wrong. __change_pid() only records a detached PID in post.pids when pid_has_task() is false for every PIDTYPE. If another task still uses the exiting task's PID as its process group or session ID, __unhash_process() removes the exiting task's PIDTYPE_PID link but leaves the PID out of post.pids. release_task() therefore holds no reference to it after dropping tasklist_lock. The other task can then remove the remaining PIDTYPE links. Its free_pids() call schedules delayed_put_pid(), and the RCU callback can free the PID before the first release_task() reaches proc_flush_pid(). An unprivileged reproducer races wait4(-1) against setsid() to trigger this ordering. Three of three fresh v7.2 KASAN boots reported: BUG: KASAN: slab-use-after-free in proc_invalidate_siblings_dcache+0x3e2/0x3f0 Read of size 8 by task h7_pid_reaper/1921 Call Trace: proc_invalidate_siblings_dcache release_task wait_consider_task __do_wait do_wait kernel_wait4 Freed by task 0: kmem_cache_free put_pid delayed_put_pid rcu_core Last potentially related work creation: __call_rcu_common free_pids ksys_setsid KASAN identified a 144-byte object from the pid cache and located the bad read 80 bytes into the freed object, matching pid->inodes. With an explicit reference, three of three fresh boots completed without a KASAN report. The concurrent RCU callback dropped its reference while proc_flush_pid() was protected, and the balancing put_pid() performed the final free afterward. Take a reference before __unhash_process() clears p->thread_pid and release it after proc_flush_pid() completes. A tested source reproducer is available privately on request. No controlled read or write, information leak, or privilege escalation is claimed. The mainline patch applies directly to v6.19.y and newer; v6.16.y through v6.18.y need a context-adjusted backport. Fixes: 0a36bad01731 ("release_task: kill the no longer needed get/put_pid(thread_pid)") Reported-by: syzbot+0aee5e8066eddbbe7397@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=0aee5e8066eddbbe7397 Reported-by: syzbot+e8b3520b53e78e90034e@syzkaller.appspotmail.com Link: https://syzkaller.appspot.com/bug?extid=e8b3520b53e78e90034e Cc: stable@vger.kernel.org # see patch description, needs adjustments for 6.16.y-6.18.y Signed-off-by: Daehyeon Ko <4ncienth@gmail.com> Link: https://patch.msgid.link/20260831001221.3755948-1-4ncienth@gmail.com Acked-by: Oleg Nesterov <oleg@redhat.com> Reviewed-by: Bradley Morgan <brads@mainlining.org> Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
2026-09-04futex: Prevent rcuwait use-after-free during requeue PIYao Kai1-2/+10
On PREEMPT_RT, FUTEX_CMP_REQUEUE_PI can trigger a KASAN report (slab-out-of-bounds) in futex_requeue_pi_complete() invocation of rcuwait_wake_up(). The futex_q used by futex_wait_requeue_pi() is allocated on the waiter's stack. An early wakeup can race with a PI requeue as follows: waiter requeue task ------ ------------ futex_wait_requeue_pi() futex_do_wait() schedule() futex_requeue futex_proxy_trylock_atomic() futex_requeue_pi_prepare() Q_REQUEUE_PI_NONE -> Q_REQUEUE_PI_IN_PROGRESS * timeout/ signal wakes waiter * futex_requeue_pi_wakeup_sync() Q_REQUEUE_PI_IN_PROGRESS -> Q_REQUEUE_PI_WAIT requeue_pi_wake_futex futex_requeue_pi_complete() cmpxchg Q_REQUEUE_PI_WAIT -> Q_REQUEUE_PI_LOCKED rcuwait_wait_event() if (atomic_read(&q->requeue_state) != Q_REQUEUE_PI_WAIT) break /* no schedule() */ /* q.pi_state->owner == current */ futex_private_hash_put() /* return from syscall */ rcuwait_wake_up(&q->requeue_wait) /* q is gone */ futex_requeue_pi_complete() publishes Q_REQUEUE_PI_LOCKED before calling rcuwait_wake_up(). The waiter observes this state in rcuwait_wait_event() before invoking schedule() in rcuwait_wait_event(). Here, the waiter is free leave the syscall before requeue task can complete the wake. To address this race skip rcuwait_wake_up() in the Q_REQUEUE_PI_LOCKED case. This state is only published by requeue_pi_wake_futex(), which saves q->task before futex_requeue_pi_complete() and wakes the waiter via wake_up_state(). This wake is intended to wake the waiter from its futex_do_wait() sleep. If the waiter is still sleeping there, it can not get into the Q_REQUEUE_PI_WAIT state (and require this removed wake). Should the waiter be woken up from futex_do_wait() by other means (as in this example) and sleep in futex_requeue_pi_wakeup_sync() then the wake_up_state() from requeue_pi_wake_futex() will wake it, too. Should the waiter task terminate before wake_up_state() had a chance to wake the task then the task pointer does not become invalid because the futex_hash_bucket::lock is held and the task pointer is RCU protected. [bigeasy: Updated comment and commit message] Fixes: 07d91ef510fb1 ("futex: Prevent requeue_pi() lock nesting issue on RT") Signed-off-by: Yao Kai <yaokai34@huawei.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260901135453.3121948-3-bigeasy@linutronix.de
2026-09-04futex: Provide rt_mutex_.*_schedule() equivalents for futex schedulingSebastian Andrzej Siewior3-13/+21
There is rt_mutex_{pre|post}_schedule() around rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/ sched_update_worker() is invoked before we schedule out and block on rt_mutex while waiting for it become available. The reason is that blocking on rt_mutex assigns a pi_waiter for the PI chain and sched_submit_work() will also assign a pi_waiter if it blocks on lock but a this point we already have a waiter assigned. We can't skip sched_submit_work() entirely because I/O relies on the fact that I/O queue is flushed while it blocks on a sleeping lock. Therefore sched_submit_work() is moved before we block on the lock. Sleeping lock in this context means mutex or rw_semaphore not spinlock_t on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same abstraction as the futex proxy lock, the futex code ended up using rt_mutex_{pre|post}_schedule(), too. Using it is/ was just to keep the task_struct::sched_rt_mutex assertion happy. Futex proxy lock is used only in the syscall context of a task. At this point it never got any I/O that needs to be flushed and it can't be a workqueue that needs to notify that it will be scheduled out. Therefore sched_submit_work() does nothing here. By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed because in this callchain the lock is (usually) not contended and so rt_mutex_slowlock_block() does not schedule, triggering the assert. Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ) because at this point there is a pi_waiter recorded and invoking sched_submit_work() with a possible lock contention would be wrong. Add rt_mutex_futex_{pre|post}_schedule() which toggles the sched_rt_mutex assert and does not involve sched_submit_work(). Add asserts here to ensure that sched_submit_work() would do nothing. Use it only in futex proxy lock case which is rt_mutex_wait_proxy_lock(). Remove it from futex_lock_pi(). Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers") Reported-by: Yao Kai <yaokai34@huawei.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com
2026-09-03bpf: Reject legacy packet loads from callbacksKumar Kartikeya Dwivedi1-0/+17
check_ld_abs() models a failed BPF_LD_ABS or BPF_LD_IND in a subprogram as an implicit return with R0 set to zero. It calls prepare_func_exit() to explore this synthesized path. When the load is reached directly from a synchronous callback, prepare_func_exit() enforces the callback return contract and marks R0 precise. R0 is not derived from a real instruction on this path, so precision backtracking reaches the callback call with R0 still requested and triggers the "callback unexpected regs" verifier bug. A privileged program loader can therefore cause a verifier warning and an -EFAULT BPF_PROG_LOAD. These legacy packet-load instructions are deprecated. Reject them from callbacks rather than complicating their implicit-return model. Check all active frames before constructing the implicit return so nested static subprograms cannot hide the callback context. Global functions are verified independently with a fresh frame zero, so an active-frame check cannot identify a global function called from a callback. Also check the complete subprogram call graph during stack-depth validation and reject a function containing a legacy load when any caller is a callback. This covers global and static descendants without making has_ld_abs transitive, preserving its per-function BTF return-type check. Ordinary uses outside callbacks remain supported. Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs") Reported-by: Sashiko <sashiko-bot@kernel.org> Link: https://lore.kernel.org/bpf/20260903152147.C0E241F00A3A@smtp.kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903214758.2727663-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Mark faultable stack helpers as sleepableKumar Kartikeya Dwivedi1-0/+2
The faultable variants of bpf_get_stack() and bpf_get_task_stack() pass may_fault=true into the common stack collection code. Resolving user-space build IDs may then call build_id_parse_file() and block on filesystem reads. Neither helper prototype sets might_sleep. Since prototype selection uses the sleepability of the whole program, the verifier can still allow these helpers from a non-sleepable region within that program, such as an explicit RCU or preemption-disabled region. The task-stack helper can also be called from a non-sleepable timer callback of a sleepable program. Mark both faultable prototypes as sleepable. The existing helper context check then rejects these calls while continuing to allow them in genuinely sleepable contexts. Fixes: d4dd9775ec24 ("bpf: wire up sleepable bpf_get_stack() and bpf_get_task_stack() helpers") Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903214758.2727663-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Mark bpf_btf_find_by_name_kind() as sleepableKumar Kartikeya Dwivedi1-0/+1
When bpf_btf_find_by_name_kind() finds a type in module BTF, it returns a new BTF object fd through __btf_new_fd(). This reaches anon_inode_getfd(), which can sleep while allocating or expanding the current task fd table. The helper prototype does not set might_sleep, so the verifier allows the helper in non-sleepable contexts such as BPF timer callbacks. The fd allocation can then sleep in softirq context and install the fd into the interrupted task. Mark the helper as sleepable. This preserves calls from the main body of a sleepable syscall program while rejecting calls from its non-sleepable regions. Fixes: 3d78417b60fb ("bpf: Add bpf_btf_find_by_name_kind() helper.") Reported-by: Sashiko <sashiko-bot@kernel.org> Link: https://lore.kernel.org/bpf/20260903155150.D57251F000E9@smtp.kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903214758.2727663-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Check ancestor frames for rbtree callbacksKumar Kartikeya Dwivedi1-11/+14
bpf_rbtree_add() invokes its comparator while the caller holds the root lock. The native insertion code retains raw parent and link pointers across the callback, so the verifier prohibits unlocking, consuming tree nodes, or changing RCU state from that callback. in_rbtree_lock_required_cb() only checks the innermost verifier frame. Static subprogram calls are permitted while holding a spin lock, and such a call pushes a frame without in_callback_fn set. Consequently, all callback restrictions disappear in the nested frame. The subprogram can unlock the tree, remove and drop the node being compared, then relock. Native insertion resumes with the stale parent pointer and links freed memory into the tree. Walk all active frames for the rbtree callback instead. Benign static subprograms remain permitted, while callback restrictions follow execution into nested frames. Fixes: a44b1334aadd ("bpf: Allow calling static subprogs while holding a bpf_spin_lock") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903214758.2727663-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: don't rewrite bpf_fastcall patterns entered by a jumpEduard Zingerman2-0/+11
mark_fastcall_pattern_for_call() must ensure that matched "spill; call; fill" instruction series is not interrupted by a jump. Otherwise the rewrite applied by bpf_remove_fastcall_spills_fills() is not sound. Record the instructions targeted by jumps in insn_aux_data[*].jump_target when the CFG is built and use this flag to stop growing a pattern at such an instruction. Jumps to the first spill are fine. Note that existing insn_aux_data[*].jmp_point field can't be reused, as it marks subprogram return instructions. Fixes: 5b5f51bff1b6 ("bpf: no_caller_saved_registers attribute for helper calls") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903205820.1743087-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: update disasm.c to print BPF_PROBE_ATOMIC as atomicsEduard Zingerman1-22/+25
bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, this patch adjusts print_bpf_insn() to print such instructions as regular atomics with a 'probe_' prefix (instead of printing them as BUG_XX). Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903171542.1438050-2-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: zero extend the result of an arena 32-bit cmpxchgEduard Zingerman1-1/+6
bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, and it runs before bpf_opt_subreg_zext_lo32_rnd_hi32(). That pass emits an explicit zero extension for a 32-bit cmpxchg even when bpf_jit_needs_zext() is false. This is done because on some architectures 32-bit cmpxchg requires explicit zero extension for the dst register. E.g. on x86-64 'lock cmpxchg' does not change the %eax if comparison is successful, while BPF semantics declare that each operation on a 32-bit register zero extends it's upper half. is_cmpxchg_insn() matches BPF_MODE == BPF_ATOMIC only, so an arena cmpxchg misses said zero extension adjustment. This patch adjusts is_cmpxchg_insn() to match BPF_PROBE_ATOMIC alongside BPF_ATOMIC. Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260903171542.1438050-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03ring-buffer: Add checking nr_subbufs to persistent ring buffer validationSteven Rostedt1-0/+5
Sashiko reported that the code was using meta->nr_subbufs without making sure that it matched the nr_pages + 1 on data that was assuming the two were the same. Add a check to the persistent ring buffer validation code to make sure that the saved nr_subbufs matches what we expect. Link: https://patch.msgid.link/20260903132728.7fb27d34@gandalf.local.home Fixes: f5b95f1fa2ef3 ("ring-buffer: Validate the persistent meta data subbuf array") Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/all/20260901164836.D962D1F000E9@smtp.kernel.org/ Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-03sched_ext: Fix NULL sched deref in kfunc sub-sched error pathsWanwu Li2-4/+18
When the root scheduler has sub-scheds attached, the COMPAT kfunc wrappers scx_bpf_select_cpu_and() and scx_bpf_dsq_insert_vtime() refuse the call and report to @p's scheduler: scx_error(scx_task_sched(p), "... must be used"); The wrappers are reachable with tasks that have no scheduler. scx_bpf_select_cpu_and() is in the select_cpu kfunc group, which scx_kfunc_context_filter() opens to BPF_PROG_TYPE_SYSCALL programs; scx_bpf_dsq_insert_vtime() is in the enqueue_dispatch group, which ops.enqueue() and ops.dispatch() may call with any KF_RCU task -- the group has no kf_tasks validation, and scx_dsq_insert_preamble() checks task ownership with scx_task_on_sched() precisely because @p may be an arbitrary task. scx_task_sched(p) is p->scx.sched, which is NULL for tasks past sched_ext_dead() -- which clears it via scx_disable_and_exit_task() on exit -- and for idle tasks, which the enable paths skip as they are never scheduled through SCX. It is also an rcu_dereference_protected() that expects @p's pi_lock or rq lock, which neither wrapper holds. Passing NULL to scx_error() reaches scx_vexit(), which dereferences sch->exit_info, oopsing the kernel. One concrete trigger exercised while developing the fix: a BPF_PROG_TYPE_SYSCALL program calling the select_cpu_and wrapper on an exited-but-not-reaped task while a sub-scheduler was attached (its pid stays findable while the zombie is unreaped; faulting instruction is the scx_vexit() prologue "mov r15,[rdi+0x398]" with RDI=NULL and 0x398 the offset of sch->exit_info): sched_ext: BPF scheduler "kfunc_subsched_null" enabled sched_ext: BPF sub-scheduler "kfunc_subsched_null" enabled sched_ext: Unassociated program run_select_cpu_ (id 76) BUG: kernel NULL pointer dereference, address: 0000000000000398 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page Oops: Oops: 0000 [#1] SMP NOPTI CPU: 7 UID: 0 PID: 8201 Comm: kfunc_test_runn Tainted: G W RIP: 0010:scx_vexit+0x25/0xa0 Code: ... <4c> 8b bf 98 03 00 00 ... CR2: 0000000000000398 Call Trace: <TASK> __scx_exit+0x4f/0x70 scx_bpf_select_cpu_and+0xab/0xb0 bpf_prog_430ed61a7b66e03a_run_select_cpu_and+0x9c/0xe7 ? __x64_sys_bpf+0x2c/0x40 bpf_prog_test_run_syscall+0x130/0x2f0 __sys_bpf+0x930/0x10d0 ? __x64_sys_bpf+0x2c/0x40 __x64_sys_bpf+0x2c/0x40 do_syscall_64+0xbc/0x460 entry_SYSCALL_64_after_hwframe+0x76/0x7e </TASK> Read @p's scheduler under RCU instead, which the wrappers can do from their guard(rcu)(): fault it when it can be determined, and when it can't be determined -- @p is a task past sched_ext_dead() or an idle task -- there is nothing obviously wrong to report, so just refuse the call as before without faulting any scheduler. These COMPAT wrappers are scheduled for eventual removal once the deprecation grace period elapses, but until then -- and regardless of their removal timeline -- they must not oops the kernel on a task they are handed. Cc: stable@vger.kernel.org # v7.1+ Fixes: a5fa0708cbfd ("sched_ext: Enforce scheduling authority in dispatch and select_cpu operations") Suggested-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Wanwu Li <liwanwu@kylinos.cn> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-09-03bpf: Mark syscall helpers as sleepableKumar Kartikeya Dwivedi1-0/+2
bpf_sys_bpf() executes the bpf(2) syscall body, which can take mutexes, allocate with GFP_KERNEL, and wait for an RCU grace period. bpf_sys_close() reaches close_fd() and filp_close(), which can sleep as well. Both helpers are limited to BPF_PROG_TYPE_SYSCALL, whose main program is sleepable. That does not make every callback sleepable: a syscall program can register a bpf_timer callback, and the verifier checks that callback in a non-sleepable context while retaining the syscall helper set. Without .might_sleep on the prototypes, such a callback can invoke bpf_sys_bpf() from hrtimer softirq context and trigger a scheduling-while-atomic failure. bpf_sys_close() is exposed through the same missing context check. Set .might_sleep on both prototypes so the existing helper-context check rejects them from timer callbacks and other atomic regions. Calls from the sleepable main body remain valid. Fixes: 79a7f8bdb159 ("bpf: Introduce bpf_sys_bpf() helper and program type.") Fixes: 3abea089246f ("bpf: Add bpf_sys_close() helper.") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903144433.1716731-10-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Mark sched_process_wait argument as nullableKumar Kartikeya Dwivedi1-0/+4
do_wait() passes wo->wo_pid to the sched_process_wait tracepoint. kernel_wait4() leaves wo_pid NULL for wait4(-1), and kernel_waitid_prepare() does likewise for waitid(P_ALL). btf_ctx_access() currently types argument 0 as PTR_TO_BTF_ID | PTR_TRUSTED. Without PTR_MAYBE_NULL, the verifier accepts an unchecked dereference. Trusted pointer loads have no fault protection, so a wait for any child can then cause a NULL pointer dereference in JITed BPF code. Add sched_process_wait to raw_tp_null_args[] with argument 0 marked nullable. The verifier rejects an unchecked dereference while preserving access after the program checks the pointer for NULL. Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903144433.1716731-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Reject resilient lock operations in rbtree callbacksKumar Kartikeya Dwivedi1-0/+5
__bpf_rbtree_add() keeps parent and link pointers live across calls to the program-supplied comparison callback. The verifier therefore requires the root's lock to remain held throughout the callback. The helper path enforces this rule for bpf_spin_lock() and bpf_spin_unlock(), but the resilient lock kfunc argument path does not. Since resilient locks may protect BPF rbtree roots, a callback can release the root lock and let another CPU remove and free the node referenced by the in-progress tree walk. The walk then resumes using freed pointers. Reject resilient lock kfuncs in an rbtree comparison callback, matching the existing policy for the spin lock helpers. Resilient-lock-protected trees remain valid when their comparison callbacks leave lock state alone. Fixes: 0de2046137f9 ("bpf: Implement verifier support for rqspinlock") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903144433.1716731-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Reject tail calls directly from callback framesKumar Kartikeya Dwivedi1-0/+11
A tail call from a non-zero frame is modeled as a return from that frame. The verifier makes R0 unknown and calls prepare_func_exit() for the taken branch. When the current frame is a synchronous callback, prepare_func_exit() enforces the callback return-value contract and marks R0 precise. Since the tail-call path synthesized R0 rather than deriving it from an instruction, precision backtracking reaches the callback-calling instruction with R0 still requested and triggers the "callback unexpected regs" verifier bug. A CAP_BPF task can therefore cause a WARN and an -EFAULT BPF_PROG_LOAD. Tail calls reachable from callbacks are already rejected later by check_max_stack_depth(). Reject a tail call made directly by a callback before constructing the inconsistent return state, using the existing diagnostic. Tail calls from ordinary subprograms keep their current behavior. Fixes: e3245f899043 ("bpf: properly verify tail call behavior") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903144433.1716731-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Mark signal tracepoint siginfo arguments as scalarKumar Kartikeya Dwivedi1-0/+3
The signal_generate and signal_deliver tracepoints declare their info argument as a struct kernel_siginfo pointer. btf_ctx_access() therefore treats it as a trusted pointer for tp_btf programs. Signal delivery also uses SEND_SIG_NOINFO and SEND_SIG_PRIV as special values for this argument. Those values are zero and one respectively, and are not pointers. A tp_btf program can currently dereference either value and fault the kernel. In particular, signal_generate can run from timer interrupt context, turning the fault into a kernel panic. Record both tracepoints in raw_tp_null_args[] and mark argument one as a non-pointer. This preserves scalar access to the cookie while rejecting direct and helper-mediated pointer use. Merely marking it nullable would not suffice because SEND_SIG_PRIV is nonzero. Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20260903144433.1716731-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Fix NULL-ptr-deref in btf_var_show()Jiayuan Chen1-1/+9
btf_var_show() calls btf_type_id_resolve() unconditionally, which dereferences btf->resolved_ids. That is NULL for a base BTF - e.g. the vmlinux BTF that bpf_snprintf_btf() renders against - since base BTF is not resolved during parsing. btf_modifier_show() guards this with 'if (btf->resolved_ids)', but btf_var_show() does not. A BPF program that passes the type_id of a BTF_KIND_VAR from the vmlinux BTF to bpf_snprintf_btf() thus NULL-derefs: KASAN: probably user-memory-access in range [0x46638-0x4663f] RIP: 0010:btf_var_show (kernel/bpf/btf.c:2929) Call Trace: <TASK> btf_type_show (kernel/bpf/btf.c:8259) btf_type_snprintf_show (kernel/bpf/btf.c:8329) bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047) bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829) __sys_bpf (kernel/bpf/syscall.c:4804) do_syscall_64 (arch/x86/entry/syscall_64.c:84) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) </TASK> Resolve the var's type directly with btf_type_skip_modifiers() when resolved_ids is NULL, mirroring btf_modifier_show(). Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper") Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev> Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev> Link: https://lore.kernel.org/r/20260901104924.346187-4-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Fix NULL-ptr-deref when showing a void BTF typeJiayuan Chen1-1/+8
btf_modifier_show() resolves the modifier and then calls btf_type_ops(t)->show() unconditionally. For the void type (type_id 0, BTF_KIND_UNKN) kind_ops[] has no entry, so ->show is NULL. A "const void" (a modifier resolving to void) cannot be a map key or value - map_check_btf() rejects it because void has no size - so the map dump path does not reach it. But bpf_snprintf_btf() takes a type_id straight from the BPF program, and passing such a "const void" from the vmlinux BTF NULL-derefs: KASAN: null-ptr-deref in range [0x0000000000000028-0x000000000000002f] RIP: 0010:btf_modifier_show (kernel/bpf/btf.c:2914) Call Trace: <TASK> btf_type_show (kernel/bpf/btf.c:8251) btf_type_snprintf_show (kernel/bpf/btf.c:8321) bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047) bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829) __sys_bpf (kernel/bpf/syscall.c:4804) do_syscall_64 (arch/x86/entry/syscall_64.c:94) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) </TASK> Fall back to btf_df_show() when the resolved type has no show op; it emits the "<unsupported kind:N>" placeholder already used for kinds like FWD and FUNC. bpf_snprintf_btf() then returns the length as usual. Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper") Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev> Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev> Link: https://lore.kernel.org/r/20260901104924.346187-3-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03bpf: Reject key-less BTF for hash mapsJiayuan Chen1-0/+6
map_check_btf() allows a key-less BTF (btf_key_type_id == 0) only for maps that have a ->map_check_btf callback, and leaves the actual decision to that callback. Hash maps used to have no ->map_check_btf, so a key-less BTF was rejected outright. That changed when htab and rhtab gained a ->map_check_btf to register a dtor - htab in commit 1df97a7453ee ("bpf: Register dtor for freeing special fields") and rhtab in commit 6905f8601298 ("bpf: Allow special fields in resizable hashtab"). Neither looks at the key, so a key-less hash map now passes map_check_btf() and gets created. Reading it back through bpffs feeds the key type_id 0 into btf_type_seq_show(); btf_type_by_id() returns the void type, kind_ops[BTF_KIND_UNKN] is NULL, and btf_type_show() dereferences it: RIP: 0010:btf_type_show+0x223/0x2e0 kernel/bpf/btf.c:8232 RSP: 0018:ffffc9000399f868 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000028 RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 R10: ffffc9000399f970 R11: 0000000000000001 R12: ffffffff9b96b140 R13: ffffc9000399f8e0 R14: ffff88803d393c00 R15: 0000000000000003 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000200000000000 CR3: 000000003d213000 CR4: 0000000000352ef0 DR0: 0000000039ae8f55 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: <TASK> btf_type_seq_show_flags+0xca/0x120 kernel/bpf/btf.c:8250 htab_map_seq_show_elem+0x12e/0x350 kernel/bpf/hashtab.c:1669 map_seq_show+0x13d/0x1e0 kernel/bpf/inode.c:293 traverse.part.0.constprop.0+0x107/0x650 fs/seq_file.c:112 traverse fs/seq_file.c:99 [inline] seq_read_iter+0x93f/0x1270 fs/seq_file.c:196 seq_read+0x344/0x4d0 fs/seq_file.c:163 vfs_read+0x1e4/0xb40 fs/read_write.c:572 ksys_pread64 fs/read_write.c:764 [inline] __do_sys_pread64 fs/read_write.c:772 [inline] __se_sys_pread64 fs/read_write.c:769 [inline] __x64_sys_pread64+0x1eb/0x250 fs/read_write.c:769 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x123/0x790 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f Reject a key-less BTF in htab_map_check_btf() and rhtab_map_check_btf(), restoring the previous behavior. Fixes: 1df97a7453ee ("bpf: Register dtor for freeing special fields") Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab") Reported-by: syzbot+37b56485bbbf90ad8489@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a8f4e88.27659fcc.2ceef7.0008.GAE@google.com/T/ Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev> Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev> Link: https://lore.kernel.org/r/20260901104924.346187-2-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03ring-buffer: Allow splice reads on static buffersVincent Donnefort1-9/+2
ring_buffer_read_page() rejects splice (full=1) reads on static buffers (that is user-mapped, persistent or remote) because !read check assumes unread pages must be swapped. However for those buffers we have no other choice than memcpy the data. For the memcpy case, only return an error when the writer is still on the reader page for the splice interface to wait. Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260901155445.1475405-2-vdonnefort@google.com Fixes: 117c39200d9d ("ring-buffer: Introducing ring-buffer mapping functions") Signed-off-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-02bpf: reject BPF_PSEUDO_FUNC reference to the main programEduard Zingerman1-0/+9
fixups.c:jit_subprogs() rewrites BPF_PSEUDO_FUNC loads to contain real function addresses. This function is invoked from bpf_jit_subprogs() only when env->subprog_cnt > 1. Meaning that for any program like below: int main(void *ctx) { void *ptr = main; ... bpf_timer_set_callback(..., ptr); ... } The 'ptr' won't be ever converted to contain an address. In combination with e.g. bpf_timer_set_callback() this would lead to a function call at a bogus address. Instead of complicating the implementation, just assume that no useful program needs main to be a sync or async callback and reject BPF_PSEUDO_FUNC loads for the main subprogram. Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260902233658.1186477-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-03kprobes: Protect kprobe_blacklist with RCUMasami Hiramatsu (Google)1-4/+10
__within_kprobe_blacklist() traverses kprobe_blacklist without holding kprobe_mutex. When a module is unloaded, kprobe_remove_area_blacklist() removes blacklist entries and immediately frees them with kfree(). A concurrent call to within_kprobe_blacklist() can therefore dereference freed memory. Furthermore, within_kprobe_blacklist() can be called in atomic or non-preemptible contexts where the sleeping kprobe_mutex cannot be taken. Protect kprobe_blacklist with RCU. Use guard(rcu)() and list_for_each_entry_rcu() for traversal, list_add_tail_rcu() for insertions, list_del_rcu() for deletions, and kfree_rcu() to reclaim entries safely after a grace period. Link: https://lore.kernel.org/all/178810004323.64882.16493230858653316962.stgit@devnote2/ Fixes: 376e242429bf ("kprobes: Introduce NOKPROBE_SYMBOL() macro to maintain kprobes blacklist") Cc: stable@vger.kernel.org Reported-by: Sashiko <sashiko-bot@kernel.org> Closes: https://lore.kernel.org/all/20260807155802.F06041F000E9@smtp.kernel.org/ Assisted-by: Antigravity:gemini-3.7-flash Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
2026-09-03tracing/probes: Fix use-after-free on field name/type of events with multiple probesHenry Martin2-1/+49
The fields of a probe-based dynamic event (kprobe, uprobe, eprobe and fprobe events) are created in traceprobe_define_arg_fields() by handing the probe_arg name/type strings to trace_define_field(), which only stores the pointers without copying. Those strings are owned by the trace_probe and are freed when that probe is removed. An event can have several probes attached. The field list is defined only once, by the first probe that registers the event, but it is kept alive by any surviving sibling probe. Deleting just that first probe by symbol - # primary A: fields are defined from A's args echo 'p:kprobes/ev vfs_read a1=$arg1' > kprobe_events # append B: shares A's event call echo 'p:kprobes/ev vfs_write a1=$arg1' >> kprobe_events # delete only A (matched by symbol), B survives echo '-:kprobes/ev vfs_read' >> kprobe_events frees A's args (trace_probe_cleanup() -> traceprobe_free_probe_arg()), but trace_probe_unlink() keeps the trace_probe_event because the probe list is not empty. The event call stays registered via B while its fields now reference freed memory. Any field lookup then reads it, e.g. echo 'a1 == 1' > events/kprobes/ev/filter BUG: KASAN: slab-use-after-free in strcmp+0xa7/0xb0 Call Trace: strcmp trace_find_event_field parse_pred process_preds create_filter apply_event_filter event_filter_write field->name references parg->name (kstrdup'd, freed with the probe) and, for array arguments, field->type references parg->fmt (kmalloc'd, freed with the probe) - the scalar type otherwise points at the static fmttype rodata, which is safe. Have traceprobe_define_arg_fields() duplicate the name and type strings and anchor the copies on the trace_probe_event, which embeds the event call and outlives every individual probe; trace_probe_event_free() releases them. The reproducer above triggers reliably; the field lookup and the delete both run under event_mutex, so this is a dangling reference after removal rather than a race. The issue was found by the autokbug dynamic kernel fuzzer at Tencent Yunding Lab. Link: https://lore.kernel.org/all/20260826030009.1855331-1-bsdhenrymartin@gmail.com/ Fixes: ca89bc071d5e4 ("tracing/kprobe: Add multi-probe per event support") Signed-off-by: Henry Martin <bsdhenrymartin@gmail.com> Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
2026-09-03tracing/probes: Fix code indent in get_bitoffset_of_field()Masami Hiramatsu (Google)1-5/+5
Fix code block indentation introduced by commit f21834524025 ("tracing/probes: Support field specifier option for typecast"). Link: https://lore.kernel.org/all/178827252027.123716.7095571176291547259.stgit@devnote2/ Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org> Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-03tracing/probes: Fix BTF kflag check for anonymous struct member accessMasami Hiramatsu (Google)3-11/+16
btf_find_struct_member() traverses into nested anonymous structures and unions to find a struct member. However, get_bitoffset_of_field() in trace_probe.c checked btf_type_kflag(type) using the outer parent type instead of the actual anonymous structure/union that directly contains the found member. If the parent structure and anonymous structure have mismatched kflags (e.g., the parent has kflag=0 while the anonymous structure has kflag=1 because it contains bitfields), the bitfield size encoded in the upper 8 bits of member->offset is erroneously treated as part of the byte/bit offset, corrupting the resolved offset and failing to set last_bitsize. Similarly, btf_find_struct_member() pushed anonymous member offsets onto anon_stack without masking BTF_MEMBER_BIT_OFFSET() when kflag is set. To fix this problem, update btf_find_struct_member() to return actual containing structure/union type via member_type, use appropriate __btf_member_bit_offset() to get bit offset, and use member_type for btf_type_kflag() in get_bitoffset_of_field(). Link: https://lore.kernel.org/all/178827250904.123716.17452648791331881284.stgit@devnote2/ Fixes: c440adfbe302 ("tracing/probes: Support BTF based data structure field access") Cc: stable@vger.kernel.org Reported-by: Sashiko <sashiko-bot@kernel.org> Closes: https://lore.kernel.org/all/20260822095110.0772E1F000E9@smtp.kernel.org/ Assisted-by: Antigravity:gemini-3.7-flash Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org> Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-03tracing/probes: Fix anon_stack check for unnamed bitfields in btf_find_struct_memberMasami Hiramatsu (Google)1-6/+6
btf_find_struct_member() traverses into nested anonymous structures and unions by pushing members with !member->name_off onto anon_stack. However, it does not consider the unnamed bitfields (e.g. `int : 5` or `unsigned int : 0`) which also have member->name_off == 0. If such an unnamed bitfield is pushed to anon_stack, the btf_find_struct_member() return an error even if there are other valid entries in anon_stack. To fix this, only push unnamed struct/union members to anon_stack. Also move the btf_type_is_struct() check to the entry of this function because now it is sure only struct/union are pushed to anon_stack. Link: https://lore.kernel.org/all/178827249775.123716.7813217688423513612.stgit@devnote2/ Fixes: 302db0f5b3d8 ("tracing/probes: Add a function to search a member of a struct/union") Cc: stable@vger.kernel.org Reported-by: Sashiko <sashiko-bot@kernel.org> Closes: https://lore.kernel.org/all/20260830143859.D56991F00A3D@smtp.kernel.org/ Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org> Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-02tracing: Take trace_array reference when opening options fileSteven Rostedt1-4/+63
The options files do not take the trace_array reference for the options they represent. This could cause a use-after-free kernel crash if one of these files is opened by one task and another task removes the instance that the option is for. Because it doesn't take a reference upon opening, it will not stop the removal which will free the options descriptor that is being used. As the options are somewhat dynamic in their creation at boot up, each file represents a flag in the trace_array. The trace_array has an array of indexes to represent each of these flags that is stored in the trace_flags_index array. The address of the index array element is used to pass to the inode->i_private pointer. Then that element is read which holds the index (which represents the flag) and then the index is used to calculate the trace_array descriptor from its trace_flags_index array. One issue is that the index element can not be referenced until the trace_array's reference is taken. To handle this, create a new helper function called: trace_array_options_get() that will iterate all the existing trace_arrays in the ftrace_trace_arrays list (under the trace_types_lock), and compare the passed in address of the index element with the entire array of the trace_array's trace_flags_index array. If it matches, then up the corresponding trace_array's reference and return. Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260902121918.5a9e9d1b@gandalf.local.home Fixes: 577b785f55168 ("tracing: add tracer dependent options to options directory") Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/linux-trace-kernel/20260828135858.2AC501F000E9@smtp.kernel.org/ Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-02bpf: backtracking shouldn't clear outer frame R1-R5 for callbacksEduard Zingerman1-20/+17
When processing calls to bpf_loop() verifier marks R1 (and R4) as precise. R1 tracks loop iterations number and because of the 'callback_depth < R1' mechanics in check_helper_call() must be marked precise. However, precision propagation for R1 was broken, when bpf_loop() call was verified on a second iteration. Consider the following verification trace: - main: bpf_loop(nr_loops, callback ...) - callback: BPF_EXIT - main: bpf_loop(nr_loops, callback ...) - ... While the first visit of the call to bpf_loop() propagated R1 precision as expected, the second call to mark_chain_precision() in the check_helper_call() set R1, but it was immediately reset when backtrack_insn() processed preceding BPF_EXIT in the loop deleted in this patch. Because of that, the second visit of the call to bpf_loop() injected checkpoint with R1 not marked as precise. Which could trick the verifier into accepting unsafe programs. See the next patch for an example of such program. Commit is structured in a way to minimize conflicts when 'bpf' would be eventually merged with 'bpf-next'. Fixes: ab5cfac139ab ("bpf: verify callbacks as if they are called unknown number of times") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20260831-bug-015-backtrack-cb-args-precise-v1-1-68a8e2a821e0@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2026-09-02ftrace: Synchronize the initialization of ftrace_opsSteven Rostedt1-2/+11
There's some internal state that ftrace_ops needs to have set, but since it can be declared outside of the ftrace.c code, it calls ftrace_ops_init() on the ops in every global function. The issue is that if two tasks call it on the same ops at the same time it is possible to have the initialization of one corrupt the initialization of the other call. Create a ops_mutex to use to synchronize every initialization of the ftrace_ops. The mutex is taken within checking the ftrace_ops flag that states it was initializied but the flag is checked again after the mutex has been taken. Checking first outside the mutex allows it to shortcut having to take the mutex. But then the check needs to be done again after the mute is taken in case of races. Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260902095501.6b59af20@gandalf.local.home Fixes: f04f24fb7e48d ("ftrace, kprobes: Fix a deadlock on ftrace_regex_lock") Reported-by: sashiko-bot@kernel.org Close: https://lore.kernel.org/all/20260829025528.49A831F000E9@smtp.kernel.org/ Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-09-02bpf: backtrack_insn(): Handle ld_{abs,ind} subprog exit edgeEduard Zingerman1-6/+19
Nicholas Carlini reported a bug in precision backtracking mechanism for BPF_LD | BPF_{IND,ABS} instructions. These instructions are modelled as two branches: - fallthrough; - implicit exit from current subprogram. The implicit exit case was not handled by the backtrack_insn() function. When backtracking such a path backtrack_insn() did not call bt_subprog_enter(), which meant that backtracking continued manipulating precision marks in a caller frame, while looking at instructions in a callee frame. This lead to segmentation faults during verification (see the selftest), or unsound state pruning. Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs") Reported-by: Nicholas Carlini <npc@anthropic.com> Suggested-by: Nicholas Carlini <npc@anthropic.com> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20260901-bug-016-backtrack-ld-abs-v1-1-59368f1be435@gmail.com
2026-09-02printk/nbcon: Change nbcon_irq_work to IRQ_WORK_LAZYJohn Ogness1-1/+1
Change the nbcon_irq_work to be IRQ_WORK_LAZY, thus not raising an IRQ upon irq_work queuing. The irq_work is then handled on the next kernel tick. This additional delay is acceptable because nbcon_irq_work is only responsible for non-emergency deferred printing, which is delayed anyway. This has the benefit of not needing to raise an IRQ for each printk() call. On a side note, the Tegra20 and Tegra30 platforms can hang if an irq_work IRQ is raised while entering cpuidle states. This problem was reproducible by calling printk() while entering cpuidle. So this change also provides a workaround for these platforms (as long as they are not running tickless). Link: https://lore.kernel.org/lkml/f3757a75-0ba1-4558-bf57-f19ab7e59a4c@nvidia.com Fixes: 76f258bf3f2a ("printk: nbcon: Introduce printer kthreads") Signed-off-by: John Ogness <john.ogness@linutronix.de> Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Reviewed-by: Petr Mladek <pmladek@suse.com> Tested-by: Jon Hunter <jonathanh@nvidia.com> Link: https://patch.msgid.link/20260901093245.344455-3-john.ogness@linutronix.de Signed-off-by: Petr Mladek <pmladek@suse.com>
2026-09-02printk/nbcon: Flush nbcon_irq_work in nbcon_free()John Ogness1-0/+2
Ensure any pending nbcon_irq_work is flushed before allowing the console to be recycled. Signed-off-by: John Ogness <john.ogness@linutronix.de> Reviewed-by: Petr Mladek <pmladek@suse.com> Link: https://patch.msgid.link/20260901093245.344455-2-john.ogness@linutronix.de Signed-off-by: Petr Mladek <pmladek@suse.com>
2026-09-02locking/lockdep: Invalidate stale class_cache entries for zapped classesEric Dumazet1-8/+42
syzbot reported a lockdep splat hitting DEBUG_LOCKS_WARN_ON(1) in hlock_class() due to an invalid class_idx: WARNING: kernel/locking/lockdep.c:238 at __lock_acquire+0x382/0x2cf0 kernel/locking/lockdep.c:5203 Workqueue: wg-crypt-wg0 wg_packet_tx_worker RIP: 0010:hlock_class kernel/locking/lockdep.c:238 [inline] RIP: 0010:check_wait_context kernel/locking/lockdep.c:4870 [inline] RIP: 0010:__lock_acquire+0x389/0x2cf0 kernel/locking/lockdep.c:5203 Call Trace: <IRQ> lock_acquire+0x106/0x350 kernel/locking/lockdep.c:5886 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:173 tcp_tsq_handler+0x29/0x200 net/ipv4/tcp_output.c:1291 tcp_tsq_workfn+0x384/0x410 net/ipv4/tcp_output.c:1325 ... When a lock class is zapped (e.g. during module unload or key unregistration), zap_class() clears the class's bit in lock_classes_in_use and removes it from the class hash table. However, existing lockdep_map instances embedded in data structures may still retain a pointer to the zapped class in their class_cache[] array. When __lock_acquire() subsequently runs on such a lock, it finds lock->class_cache[subclass] != NULL, skipping register_lock_class() and assigning hlock->class_idx to the index of the zapped class. When check_wait_context() or hlock_class() inspects the held_lock, it finds !test_bit(class_idx, lock_classes_in_use) and warns. Furthermore, if the zapped slot is subsequently re-allocated to an unrelated lock key, the stale class_cache entry would erroneously match the unrelated class (ABA issue). Add lock_class_cache_is_valid() to validate that the cached class is within lock_classes bounds, still allocated in lock_classes_in_use (using uninstrumented arch_test_bit() in __always_inline context so it is safe in noinstr contexts like match_held_lock()), and that class->key matches the expected subkey (taking lockdep_set_subclass() overrides into account). Also use READ_ONCE()/WRITE_ONCE() when accessing class_cache[]. If the entry is invalid or stale, fall back to register_lock_class() / look_up_lock_class(). Fixes: a0b0fd53e1e6 ("locking/lockdep: Free lock classes that are no longer in use") Closes: https://lore.kernel.org/netdev/6a8c66dc.4d75e56a.c9a88.0050.GAE@google.com/T/#u Reported-by: syzbot+2d770620059281e225a4@syzkaller.appspotmail.com Assisted-by: Gemini:gemini-3.1-pro Signed-off-by: Eric Dumazet <edumazet@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260824155129.676096-1-edumazet@google.com
2026-09-02perf: Fix use-after-free when perf mmap() revival races with the last munmap()Yilin Zhang1-10/+10
perf_mmap_close() drops rb->mmap_count *without* holding event->mmap_mutex (the refcount_dec_and_test() right before the refcount_dec_and_mutex_lock() of event->mmap_count). A concurrent perf_mmap_rb() can slot its entire "revival" path into that window (perf_mmap holds event->mmap_mutex for its whole duration, including rb_alloc): munmap side (perf_mmap_close) mmap side (perf_mmap_rb) ----------------------------------- -------------------------------- rb->mmap_count 1 -> 0 (no lock) (holds event->mmap_mutex) inc_not_zero(rb->mmap_count) fails ring_buffer_attach(event, NULL) rb_alloc() + attach new rb refcount_set(&event->mmap_count, 1) lock; event->mmap_count 1 -> 0 ring_buffer_attach(event, NULL) ring_buffer_put() -> frees the *new* rb The revival's refcount_set(&event->mmap_count, 1) is an invisible 1 -> 1 write: the close frees the just-revived buffer although the other process still has it mapped -- a page-level use-after-free allowing local privilege escalation to root by any unprivileged user (default kernel.perf_event_paranoid=2). Swap the order of the two counter updates: event->mmap_count is dropped first via refcount_dec_and_mutex_lock(), so its 1 -> 0 transition and the ring_buffer_attach() stay serialized with perf_mmap(). rb->mmap_count == 0 then implies every event using the buffer is detached already, so the result of the rb->mmap_count drop can gate the remaining teardown directly and detach_rest is no longer needed. An earlier fix for this race from Kyle Zeng and David Lee takes event->mmap_mutex around both counter updates [0]; here the not-last close stays lockless. Fixes: 59741451b49c ("perf: Identify the 0->1 transition for event::mmap_count") Reported-by: Kimi Security Team <bug-report@moonshot.ai> Suggested-by: Peter Zijlstra <peterz@infradead.org> Co-developed-by: Weiming Shi <shiweiming@moonshot.ai> Signed-off-by: Weiming Shi <shiweiming@moonshot.ai> Signed-off-by: Yilin Zhang <yilinzhang@moonshot.ai> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/linux-perf-users/20260804060931.711308-1-david.lee@trailofbits.com/ [0] Cc: <stable@vger.kernel.org> Cc: stable@vger.kernel.org # 6.18+ Link: https://patch.msgid.link/20260831162155.1437652-1-yilinzhang@moonshot.ai
2026-09-02perf/core: Skip empty AUX records with only format flagsLeo Yan1-2/+7
perf_aux_output_end() emits a PERF_RECORD_AUX when the recorded size is nonzero or when any flag other than PERF_AUX_FLAG_OVERWRITE is set. PMU format flags describe how an AUX payload is encoded. TRBE driver sets PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW for raw trace buffers, causing an AUX record to be emitted even when no trace data. This is noticeable when tracing a task with strace. Ptrace stops repeatedly end empty AUX transactions, producing many zero-sized PERF_RECORD_AUX records. For example: perf record -e cs_etm//u -m,128M -- strace ls perf script -D 2>&1 | awk '/PERF_RECORD_AUX offset/ { for (i = 1; i <= NF; i++) if ($i == "size:" && $(i + 1) == "0") count++ } END { print count }' 165 This recording contains 165 zero-sized AUX records which provide no useful information to userspace. Ignore PERF_AUX_FLAG_PMU_FORMAT_TYPE_MASK, together with PERF_AUX_FLAG_OVERWRITE, when deciding whether an empty AUX record is useful. Zero-sized records carrying TRUNCATED, PARTIAL or COLLISION are still emitted. Fixes: 547b60988e63 ("perf: aux: Add flags for the buffer format") Reported-by: Tamas Petz <tamas.petz@arm.com> Signed-off-by: Leo Yan <leo.yan@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260825-perf_core_fix_zero_aux_records-v1-1-23b95e8d5df3@arm.com
2026-09-02sched/fair: Avoid creating misfits during cache-aware balancingTim Chen1-5/+45
Cache-aware load balancing biases tasks toward their preferred LLC. On asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may contain CPUs that are too small to run the task. Pulling the task there turns it into a misfit, trading a cache-locality gain for a capacity loss that's more detrimental to performance. Guard both cache-aware migration entry points against this: - can_migrate_llc_task(): forbid the LLC migration when the task fits its source CPU but would not fit the destination CPU. - alb_break_llc(): veto the active balance under the same condition so the runnable task is not pushed onto a CPU that cannot accommodate it. Both checks are gated with checks for hybrid processors, so symmetric systems are unaffected. Tasks that already do not fit their source CPU are left to the existing LLC policy, since the move cannot make their fitness worse (this also preserves misfit up-migration to bigger CPUs). Additionally, if there are misfit tasks found in the load balancing classification phase, prioritize misfit task migrations over LLC load aggregation on asymmetric systems. A better fitting CPU will boost performance more than better cache locality. Reviewed-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> Tested-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> Reviewed-by: Chen Yu <yu.c.chen@intel.com> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com
2026-09-02sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()Wanwu Li1-1/+1
distribute_cfs_runtime() refreshes the rq clock and accounts elapsed runtime with update_curr() before redistributing bandwidth, but gates this on cfs_rq->curr. Since commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") cfs_rq->curr is only maintained on the root cfs_rq, so for the cgroup cfs_rqs it walks, the check never fires and the refresh is dead code. Use cfs_rq->h_curr, the per-level current entity, restoring the intended behaviour: only refresh when something is actually running at the throttled level, i.e. within the deferred throttle window. Without this, runtime consumed by a still-running task of the throttled hierarchy is not docked before redistribution; unthrottle_cfs_rq() catches up unconditionally since commit 28ad5427682b ("sched/fair: Call update_curr() before unthrottling the hierarchy"), so this is not a correctness hole today, but the refresh the check was written for is gone. Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Wanwu Li <liwanwu@kylinos.cn> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Aaron Lu <ziqianlu@bytedance.com> Tested-by: Aaron Lu <ziqianlu@bytedance.com> Link: https://patch.msgid.link/20260831101141.391382-3-liwanwu@kylinos.cn
2026-09-02sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq()Wanwu Li1-2/+2
After commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue"), cfs_rq->curr is only maintained on the root cfs_rq (set/cleared from set_next_task_fair()/put_prev_task_fair()), while cfs_rq->h_curr is the per-level current entity, set by set_next_entity() at every level of the hierarchy. For an intermediate cfs_rq (a cgroup), cfs_rq->curr is always NULL, but cfs_rq->h_curr is the group entity at that level. throttle_cfs_rq() reads cfs_rq->curr to decide whether there is a running entity at the throttled level, in which case it should request a full sched_cfs_bandwidth_slice() of runtime and arm the deferred throttle task_work via task_throttle_setup_work(). For intermediate cfs_rqs the check is always false, so bandwidth-controlled cgroups always get just 1ns of runtime and never arm the deferred throttle work; the running task then escapes throttling until the next pick arms the work instead, even though there is an on-rq entity at this level. Switch the read to cfs_rq->h_curr so intermediate bandwidth-controlled cgroups behave consistently with the root cfs_rq, matching the existing usage of cfs_rq->h_curr in update_curr() and check_enqueue_throttle(). Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Wanwu Li <liwanwu@kylinos.cn> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Aaron Lu <ziqianlu@bytedance.com> Tested-by: Aaron Lu <ziqianlu@bytedance.com> Link: https://patch.msgid.link/20260831101141.391382-2-liwanwu@kylinos.cn
2026-09-02sched/core: Skip rq->avg_idle update without a valid idle_stampShubhang Kaushik (Ampere)1-2/+8
Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out. The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes: rq_clock(rq) - rq->idle_stamp If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp. This can happen when sched_balance_newidle() returns before setting rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that case the rq can switch to the idle task with idle_stamp still zero and leave idle again when the pending wakeup is processed. Other paths can also switch to the idle task without setting rq->idle_stamp via newidle_balance(), for example find_proxy_task() or force-idling. Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval. Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Acked-by: John Stultz <jstultz@google.com> Link: https://patch.msgid.link/20260807-master-v3-1-c328354efed3@gentwo.org
2026-09-02sched/rt,dl: Skip migrate-disabled tasks when picking a push candidateSeiji Nishikawa2-4/+4
A migrate_disable()'d RT task cannot be moved to another CPU, but the scheduler still keeps such a task on that CPU's pushable list (rq->rt.pushable_tasks) and still marks the runqueue RT-overloaded (rq->rt.overloaded = 1). So the RT balancer keeps treating this CPU as having a task to move away, and keeps trying to move the task, but the push can never succeed. When the head is pinned, push_rt_task() does not give up either. It falls back to pushing rq->curr instead, using the per-CPU stopper, as added by commit a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing"). The CPU spends tens of milliseconds in this retry loop. The core is isolated for real-time work, but during the loop nearly half of its time is consumed by pushes that cannot succeed. An ftrace capture of the affected CPU, with sched_switch enabled and commit 94894c9c477e ("sched/rt: Skip currently executing CPU in rto_next_cpu()") applied, shows where the CPU time went. Two SCHED_FIFO tasks at equal priority shared the CPU, taskA migrate_disable()'d and queued, taskB as rq->curr. In one 89 ms window, taskB got only 52 ms of CPU. The other 37 ms went to the stopper thread. The scheduler kept trying to push taskA, the pinned head of the pushable list, fell back to pushing taskB instead, and woke the stopper 5204 times. Every one of those pushes failed and no task was moved. taskA stayed runnable and queued the whole time, and never ran. Pushing taskB fails on a re-check. find_lock_lowest_rq() drops the rq lock to take the target rq lock, then checks again with "task != pick_next_pushable_task(rq)". The task being pushed is taskB, but the pick returns taskA, the head of the pushable list. taskB is rq->curr, and set_next_task_rt() removes the running task from that list, so taskB can never be the head. The check expects a candidate taken from the pushable list, but the fallback pushes rq->curr, which is never on that list. So the check fails every time. .--> push-IPI arrives | | | v | pushable head = taskA -> pinned, cannot be pushed | | | v | so push taskB instead -> wake migration/N, a stop-class | | thread, so it preempts taskB | v | re-check compares taskB against the pushable head, | which is still taskA -> give up | | | v | nothing moved, taskA still queued, rq still overloaded | | '----------' repeats every ~17 us, 5204 times, for 89 ms The loop cannot stop itself. Every round leaves the runqueue exactly as it was, so the next push-IPI does the same thing. In the capture it ended only when taskB went to sleep on its own. taskA was then picked locally and left the pushable list. CPU time per task in the window, from sched_switch: taskB 51.95 ms real work migration/N 37.18 ms nothing moved taskA 0.00 ms queued the whole time, never picked idle 0.01 ms Counts over the same window: 7667 push-IPIs handled on this CPU 17481 pick_next_pushable_task() returned taskA, still pinned 5204 find_lock_lowest_rq() gave up on the re-check 1 push that actually completed 0 migrations of taskA The CPU times and the window length come from the standard sched_switch tracepoint. The counts needed tracepoints added inside the RT balancer for this investigation. The self-IPI path is closed by the rto_next_cpu() fix above, and that part works. But the runqueue is still marked overloaded, because the pinned task is still advertised as pushable. Other CPUs now send the push-IPIs during their own RT balancing, and the same loop runs again. Closing the self-IPI path did not stop a pinned task from triggering push balancing. A pinned task should never have been returned as a push candidate in the first place. A migrate_disable()'d task cannot be migrated, so it belongs in the same skip that was added for on_cpu tasks by commit e0ca8991b2de ("sched: Make class_schedulers avoid pushing current, and get rid of proxy_tag_curr()"). Add is_migration_disabled() to the skip condition in pick_next_pushable_task() and pick_next_pushable_dl_task(). With the skip in place, if the pinned task is the only extra runnable task the helpers return NULL, push_rt_task() and push_dl_task() give up early, and no stopper is woken. The pinned task then runs locally once curr yields. If a task that really can be migrated is queued behind the pinned head, it is now picked and pushed for real. This makes the fallback that pushes rq->curr unreachable when the pushable head is migrate-disabled. Nothing is lost, because that path was always stopped by the re-check described above. In the capture it ran 5204 times and moved nothing. Fixes: a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing") Signed-off-by: Seiji Nishikawa <snishika@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260830073746.2189355-1-snishika@redhat.com
2026-09-02sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callersZhan Xusheng1-2/+2
pick_task_fair() and yield_task_fair() call update_curr(&rq->cfs) to bring curr up to date before they look at the eevdf state. With cgroups that does not happen: update_curr() reads ->h_curr, which on the root cfs_rq is the top level group entity, and returns at the !entity_is_task() check before touching vruntime. Both then read ->curr, so the guard and the update disagree about which entity they mean. Counting how often ->h_curr and ->curr differ at pick_task_fair(), on one CPU for 10s with three busy tasks and one 200us-periodic task: all tasks in the root cgroup 43321 calls, 0 no-ops busy tasks in G0, periodic in G1 45211 calls, 45193 no-ops Whether that matters depends on what precedes the pick. Since commit 68e37487810a ("sched/fair: Fix flat hierarchy") the tick and enqueue/dequeue all update curr correctly, so on the normal reschedule path only the microseconds between those and the pick are missing, and I could not measure a latency difference there. Three paths have nothing before them on that rq though: - pick_task() on the sibling rqs of a core under core scheduling (kernel/sched/core.c), which updates that rq's clock first for exactly this reason - fair_server_pick_task() - yield_task_fair(), where the stale value feeds the entity_eligible() test that guards forfeiting the remaining vruntime There curr can be a full tick behind, as it was before that commit. No new behaviour for the entity being updated: without cgroups ->h_curr is already the task, so these two call sites already run the full update_curr() including update_deadline(), dl_server_update() and the resched_curr_lazy() at the end. This makes the cgroup case do the same. Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Link: https://patch.msgid.link/20260822105930.2352761-1-zhanxusheng1024@gmail.com
2026-09-01sched_ext: Don't deliver duplicate ops.cgroup_set_idle() for same valueTao Cui1-1/+2
ops.cgroup_set_idle() is documented to be invoked when a cgroup transitions between idle and non-idle states, and scx_group_set_weight() already skips value-preserving writes. scx_group_set_idle() delivers every write unconditionally, so rewriting an already-correct cpu.idle value feeds the BPF scheduler a transition callback each time, which toggle- or accounting-based schedulers miscount. Mirror the weight guard and only deliver on an actual change. Verified with a probe scheduler printing each callback: rewriting cpu.idle=1 twice on an already-idle cgroup delivered two callbacks before and none after. Fixes: 347ed2d566da ("sched/ext: Implement cgroup_set_idle() callback") Link: https://lore.kernel.org/r/b53c61a1-4d7d-4232-941f-d48b0563d4ed@linux.dev Signed-off-by: Tao Cui <cuitao@kylinos.cn> Reviewed-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
Copyright © 1996 – 2026 Jason A. Donenfeld. All Rights Reverse Engineered.