| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
|
|
There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.
Drop them for cleanup.
No functional changes intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev
|
|
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can
update it concurrently. KCSAN reports:
BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8
race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by
task 1817002 on cpu 40:
kick_pool_pick+0xf8/0x2d8
process_scheduled_works+0x2bc/0x888
worker_thread+0x394/0x548
kthread+0x1b8/0x1f0
ret_from_fork+0x10/0x20
value changed: 0x0000002b -> 0x0000002f
The race is harmless. wake_cpu is a best-effort placement hint: every
writer stores a valid CPU id and the wakeup path validates it through
select_task_rq(), so a stale value only affects which CPU the worker
wakes up on.
Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that
they are intentionally racy and to stop the compiler from reloading or
tearing them.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When testing a linux-next kernel with commit 59bd1d914bb5 ("memblock:
warn when freeing reserved memory before memory map is initialized"),
the following warning was hit when there was a "nohz_full" kernel boot
parameter.
Cannot free reserved memory because of deferred initialization of the memory map
WARNING: mm/memblock.c:904 at __free_reserved_area+0xde/0xf0, CPU#0: swapper/0/0
:
Call Trace:
<TASK>
memblock_phys_free+0xcb/0x100
housekeeping_init+0x14c/0x170
start_kernel+0x207/0x450
x86_64_start_reservations+0x24/0x30
x86_64_start_kernel+0xda/0xe0
common_startup_64+0x13e/0x141
</TASK>
IOW, we shouldn't free memblock allocated memory so early
in the boot process when memory map isn't fully initialized in
deferred_init_memmap().
Fix it by saving the housekeeping cpumask memblock memory to be
freed into a llist free list in housekeeping_init() and add a new
housekeeping_late_init() helper to defer the actual freeing of memblock
memory to when initcall's are being processed. The cpumask memblock
memory is treated as a llist_node with the size of a "long" type which
is also smallest cpumask size that can be allocated.
The non-atomic version of the llist APIs are used as there is no
contention.
This commit depends on the presence of commit 7c2eee9c1367 ("memblock:
don't touch memblock arrays when memblock_free() is called late")
to prevent a KASAN UAF bug report [1].
[1] https://lore.kernel.org/lkml/20260505051821.1107133-1-longman@redhat.com/
Fixes: 27c3a5967f05 ("sched/isolation: Convert housekeeping cpumasks to rcu pointers")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <pauld@redhat.com>
Link: https://patch.msgid.link/20260701195810.477326-1-longman@redhat.com
|
|
Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
performs an atomic_long_add_return() plus a CAS loop on the global
used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
All CPUs contend on the same cacheline, causing measurable throughput
degradation at scale.
Historically these counters were only compiled in under CONFIG_DEBUG_FS,
which means production kernels with debugfs paid the atomic overhead
unconditionally. Make the tracking boot-time opt-in instead so that it
is disabled by default with near-zero overhead via static_call, and can
be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.
Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled,
the "io_tlb_used" metric reports an approximate value rather than an
instantaneously exact one.
Suggested-by: Fan Du <fan.du@intel.com>
Signed-off-by: Jun Miao <jun.miao@intel.com>
Co-developed-by: Fan Du <fan.du@intel.com>
Signed-off-by: Fan Du <fan.du@intel.com>
Tested-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Signed-off-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Link: https://lore.kernel.org/r/20260812070459.637077-1-frankchen158@126.com
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
The commit 7619a0ee9340 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
bpf_trampoline_multi_detach() always returns 0 and the sole caller
ignores the return value. Change it to return void and drop the
WARN_ON_ONCE at the call site.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
When bpf_trampoline_update() fails before modify_fentry_multi()/
unregister_fentry_multi() is called, cur_image is unchanged
(cur_image == old_image) and ftrace still calls into it. Freeing
old_image in that case causes a UAF.
Only free old_image when it differs from cur_image.
Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
audit_del_rule() is used for both netlink deletion templates and internal
fsnotify autoremove. The former passes a parsed template which owns a
temporary tree reference; the latter passes the installed entry itself.
The unconditional audit_put_tree() at the end of audit_del_rule() assumes
the template case. For mixed AUDIT_DIR plus AUDIT_EXE rules, an fsnotify
autoremove event therefore drops the installed rule's live tree reference.
Repeating this across rules sharing the same tree can free the tree while
another rule still references it, and a later autoremove dereferences the
freed pathname while comparing rules.
Move the temporary-tree put to audit_rule_change(), the caller that owns
deletion templates. Keep it in the AUDIT_DEL_RULE cleanup so both
successful deletion and -ENOENT still release the parser-owned tree.
Cc: stable@kernel.org
Fixes: 34d99af52ad4 ("audit: implement audit by executable")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Ricardo Robaina <rrobaina@redhat.com>
Tested-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: dropped unnecessary comment for line length reasons]
Signed-off-by: Paul Moore <paul@paul-moore.com>
|
|
scx_bpf_cid_override() predates the cid-form arena transition and takes its
arrays as verifier-checked mem+size buffers, forcing scx_qmap to keep the
cpu_to_cid and shard_start arrays in writable bss while the rest of its
state lives in the arena. Unify on arena arguments before cid-form
schedulers start seeing real use.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Take the arrays as __arena arguments, with the counts passed in
entries. The counts now size the snapshot copies and are bounds-checked
before them.
scx_qmap moves the arrays into struct qmap_arena. As the arena is mmapped at
load, the loader populates them between load and attach instead of before
load.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling this kfunc load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init()
rebases them into the arena by hand.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the cmask arguments __arena so the kfuncs receive kernel
addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The
optional denied_out keeps its NULL not-provided signal via
__arena__nullable. The mandatory masks use plain __arena.
scx_qmap's call sites drop the (void *)(long) casts since the BPF-side
declarations type the cmask arguments __arena and take arena pointers
directly.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling these kfuncs load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.
The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
|
|
scx_bpf_cid_to_cpu(), scx_bpf_cpu_to_cid() and scx_bpf_cid_topo() live in
the scx_kfunc_ids_cid set, but scx_kfunc_context_filter() doesn't check
that set. The filter's first test treats any kfunc outside its known sets
as non-SCX and allows it, so these three kfuncs can be called from any
struct_ops program - e.g. a TCP congestion control program.
Add scx_kfunc_ids_cid to the filter's known sets, matching how in_any and
in_idle are handled.
Fixes: e9b55af47edf ("sched_ext: Add topological CPU IDs (cids)")
Assisted-by: Z.ai:glm-5.2
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.
Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but
every caller dereferences the result right away, so the WARN_ON_ONCE()
only moves the oops one frame up, as raised by Tejun.
Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE();
Fixes: b72fdc651056 ("workqueue: account nr_active by the backing pool")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation
path fills it in with plain loads and stores, which sparse flags:
kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@
Allocate the array as __rcu pointers and publish each pwq with
rcu_assign_pointer() once it is initialized and linked, the order
install_unbound_pwq() uses.
The warnings are not new: commit 79f23600bc7b ("workqueue: factor out
get_percpu_pool()") only turned the flagged assignment into an
initializer.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain
load, so sparse complains about the dropped __rcu:
kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@
A pwq is released with kfree_rcu() and the read is protected by the
surrounding preempt_disable(), which is what
commit fd5081f4ef33 ("workqueue: Remove redundant rcu_read_lock/unlock() in
workqueue_congested()") relied on when it dropped the rcu_read_lock()
here.
Use the rcu_dereference_sched() helper to make that explicit.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive:
c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d2175 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e44a ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da9587118d ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205c1 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d2175 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d2175 and 3dd52416e44a update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da9587118d instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205c1 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
It was an effort to enhance fscache as a kernel cache for lazy
pulling (at least according to previous Incremental FS discussion [1])
and EROFS over fscache was the in-tree user of this mode.
fscache has since evolved to be netfslib-oriented, serving network
filesystem inodes via the netfs library, but EROFS never acts as a
network filesystem and we need to cache golden filesystem images rather
than individual EROFS inodes.
Since EROFS over fscache is now removed, clean up netfs/fscache/
cachefiles upstream too.
[1] https://lore.kernel.org/r/CAOQ4uxi4dzxArY24YO=+kBCK2gGoq3Ptb8WkzCqSogPgU_R3dQ@mail.gmail.com
[dh] Fixed up comments on:
https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com
https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com
Signed-off-by: Gao Xiang <xiang@kernel.org>
Signed-off-by: David Howells <dhowells@redhat.com>
Link: https://patch.msgid.link/1046393.1786544127@warthog.procyon.org.uk
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-erofs@lists.ozlabs.org
cc: bpf@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
alloc_pid() checks PIDNS_ADDING only on the leaf pid namespace before
making a new struct pid visible in every ancestor namespace. That is
insufficient when an unborn descendant pid namespace outlives an
ancestor whose init task has already exited. The descendant can still be
initialized later through setns(), and the new pid is then published
into the dead ancestor as well.
Keep the existing ENOMEM behavior, but require PIDNS_ADDING to be set in
every namespace that will receive the new pid before publishing any of
them. This preserves the invariant that free_pid() never decrements
pid_allocated in a namespace whose child_reaper is no longer live.
Fixes: a3bdc23ba8ea ("pid_namespace: allow opening pid_for_children before init was created")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Commit 16a1d968358a ("mm/slab: remove mm/slab.c and slab_def.h")
removed the SLAB allocator, the only allocator that implemented cpuset
slab spreading. Commit 61a182ab61a6 ("cgroup/cpuset: Remove
cpuset_do_slab_mem_spread()") then removed the last task_spread_slab()
caller. Commit 3ab67a9ce82f ("cgroup/cpuset: Mark memory_spread_slab as
obsolete") marked the legacy control obsolete.
cpuset still updates PFA_SPREAD_SLAB when tasks attach to a legacy
cpuset and walks all tasks in a cpuset when memory_spread_slab changes.
Remove the unused task flag and its helpers, and make spread task
updates depend only on memory_spread_page.
Keep the memory_spread_slab control and CS_SPREAD_SLAB state so legacy
users retain the existing write, readback and inheritance behavior.
Update the comments and documentation to describe only page-cache
spreading as functional.
Assisted-by: LLM
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Core scheduling's pick_next_task() picks what to run on every SMT sibling of
the core in a single pass under the shared core-wide rq lock. The selection
state is consistent only while the lock is held continuously, so
->pick_task() originally could not release it. However, since 4c95380701f5
("sched/ext: Fold balance_scx() into pick_task_scx()"), sched_ext runs
dispatch from inside the pick and dispatching can drop the rq lock. To
support this, pick_next_task() has been updated to restart the whole
selection when a pick returns RETRY_TASK after releasing the lock.
When selections on the same core interleave through the dropped lock, they
corrupt each other's state: one clears the other's rq->core_pick leading to
a NULL deref, or invalidates its keep-the-previous-task decision leaving a
dequeued task running, which deadlocks the next wakeup and matches the
reported hard hangs. A cookied ping-pong load on an SMT machine makes the
interleavings frequent and kills the kernel within seconds.
Fix it by making the pick return RETRY_TASK whenever dispatch released the
rq lock, so that a selection only ever commits picks made under a
continuously held lock. The previous patch's rq->scx.lock_drop_seq counts
the releases. A dispatch that touched nothing never releases the lock and
its verdict, including "nothing to run", stands: retries are bounded, each
following a dispatch that actually did something, and an idle CPU does not
loop.
If another dispatch is already in flight on the rq, skip dispatching and
pick from what is already queued locally - the in-flight dispatch has
released the lock, so its own selection will retry and re-pick this rq,
while returning RETRY_TASK here would only spin on the lock that dispatch
needs to finish.
Balance callbacks must run in the context that queued them, so they can only
be queued on the CPU's own rq. When dispatching for another rq, run the
deferred work directly instead - that rq may consume all its picks through
the core-sched fast path and never queue the callback itself.
The put_prev_task_scx() warning about a runnable task being left behind
assumed that dispatch ran as part of the very pick that is switching away.
That now only holds on the non-core path, so gate it and drop the
cookie-match test, which is always true without core scheduling, from its
condition.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Reported-by: ElXreno <elxreno@gmail.com>
Link: https://github.com/sched-ext/scx/issues/3715
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Under core scheduling, pick_next_task() selects for all SMT siblings under
one continuous hold of the shared core-wide rq lock, and sched_ext's
dispatch can release that lock from inside the pick. In preparation for
making the core-sched pick detect the releases and retry, add
rq->scx.lock_drop_seq and bump it at every site that can release an rq lock
while a dispatch may be in flight. The counter is only maintained while core
scheduling is enabled. No functional changes.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Under core scheduling, dispatch runs from within the core-wide pick and can
target a sibling rq, so ops.dispatch() may execute on a CPU different from
the dispatched rq's. Several kfunc paths assumed the two always coincide:
- scx_dsq_move() decided whether an rq lock is held by testing this_rq()'s
rq flags and lock-danced accordingly. A dispatch for a sibling took the
unlocked-context branch and acquired the source rq lock on top of the
already held dispatched rq lock which could deadlock.
- scx_bpf_sub_dispatch() dispatched this_rq() with its stashed
sub_dispatch_prev, which is NULL when dispatching for a sibling.
- finish_dispatch(), scx_bpf_dsq_reenq() and scx_bpf_dsq_nr_queued()
resolved SCX_DSQ_LOCAL to this CPU's local DSQ rather than the dispatched
rq's. The latter two are callable from other rq-locked operations too,
where SCX_DSQ_LOCAL now likewise resolves to the op's rq. This changes
behavior also without core scheduling, e.g. for ops.enqueue() running a
remote wakeup on the waking CPU, and is intended: which CPU happens to
execute an operation is incidental, the op's rq is what it is operating
on, and the resolution now matches the insert side where SCX_DSQ_LOCAL
dispatches land on the task's rq.
Use the rq tracked by scx_locked_rq(), which is set to the dispatched rq
around ops invocations and NULL in unlocked contexts.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
SCX_RQ_BAL_KEEP tells the pick to keep running the previous task, a leftover
from when balancing and picking were separate operations. An rq-level flag
only works while dispatches and picks pair up one to one, which core
scheduling breaks: selections interleave through dispatch's lock drops and a
pick can consume a stale flag, keeping a task that has since been dequeued.
Fixing core scheduling support requires the decision to travel with the
dispatch that made it. Make scx_dispatch_sched() and balance_one() return an
explicit verdict instead and drop the flag's plumbing from the tools autogen
enum headers.
Also factor the pick-side invocation, its follow-up queueing and the
post-dispatch checks out of do_pick_task_scx() into dispatch_pick(). No
functional changes intended.
v2: Drop the SCX_RQ_BAL_KEEP plumbing from the tools autogen enum headers
as well (Andrea).
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Core scheduling's pick_next_task() operates on all sibling rqs under one
acquisition of the shared core-wide lock. A ->pick_task() that releases the
rq lock leaves every sibling __lock momentarily free, letting
__sched_core_flip(false) complete mid-selection and rebind rq_lockp() under
it. The selection resumes on the split locks, touching sibling state it no
longer protects, and __schedule() finally releases a lock that was never
taken while leaking the one that was.
Count in-flight core-wide selections in the leader's rq->core_pick_in_flight
and make __sched_core_flip() wait for the count to drain. The count only
changes under the shared lock, which the flip holds while sampling, so no
other ordering is needed. The wait can repeat while selections overlap, but
the flip backs off between samples and flips are rare cookie-lifetime
events.
sched_core_cpu_deactivate() moves the count to the new leader - a stale copy
left behind would bias it forever if that CPU later returns as its own
leader.
Fixes: 539f65125d20 ("sched: Add core wide task selection and scheduling")
Cc: stable@vger.kernel.org # v5.14+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
|
|
Core scheduling's pick_next_task() breaks when a ->pick_task()
implementation can release the rq lock. The selection state derived on entry
is only valid while the lock is held continuously. Once a pick can drop the
lock, an interleaving selection can invalidate all of it: the single-CPU
fast path can commit an uncookied pick although the core went cookied during
the release, and forceidle committed by the interleaving selection skews the
restarted pass's accounting.
Fix it by restarting the whole selection when a pick returns RETRY_TASK
after releasing the lock: a single restart point above the state derivation
replaces the per-loop restart labels, so a retry picks up state committed by
interleaving selections and accounts and resets forceidle like a fresh
selection would.
need_sync and fi_before latch across retries. Clock validity can't be
re-derived - there is no program-ordered way to tell whether the own and
core rq clocks are still updated after the lock was released, as other
lockers' pin cycles may or may not have invalidated them. When restarting,
clear core_clock_updated so that the sibling loop re-updates the core rq,
and update the own rq clock if invalidated.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
|
|
In the cleanup_srcu_struct() function, when iterating over per-cpu's
srcu_data, timer_delete_sync(&sdp->delay_work) is called to cancel the
delayed work before doing flush_work(&sdp->work).
However, suppose that timer_delete_sync() returns 1, which means that it
successfully deleted an pending timer before it had a chance to fire.
But this also means that the sdp->work will not be queued, so that the
subsequent flush_work(&sdp->work) will returns immediately without waiting
for anything. Taken together, all of this means that any recently queued
SRCU callbacks to not be invoked, which can result in memory leaks,
hangs, or worse.
Fix this by checking the return value of timer_delete_sync(), if it
returns 1, explicitly queue sdp->work so that the callbacks will be
invoked and the following flush_work() will correctly wait for all of
those callbacks to finish executing.
[ Zqiang: Apply feedback from Breno Leitao and kernel test robot. ]
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states
must not be equal, or the verifier can prune an unsafe path.
Compare the pointer type for STACK_ITER slots.
Fixes: dfab99df147b ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS")
Signed-off-by: Ning Ding <dingning04@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
When a function-like macro expands to an expression, that expression
doesn't need a semicolon after it. All uses have been verified to
have their own semicolons.
This was found using the following Coccinelle semantic patch:
@r@
identifier i : script:ocaml() { String.lowercase_ascii i = i };
expression e;
@@
*#define i(...) e;
Link: https://patch.msgid.link/20260801191002.1383835-6-Julia.Lawall@inria.fr
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
hrtimers use ktime_t in their implementation and API.
The timer list performs a lot of unnecessary conversion to nanoseconds
which make the code harder to read and are also wrong in case the values
ever become negative.
Remove the conversions.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-5-910cbd485390@linutronix.de
|
|
'%Ld' and '%Lu' are GNU extensions. While they do work for kernel code,
checkpatch complains about them all the time. Replace them with the
standard placeholders for 'long long' types, namely '%lld' and '%llu'.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-4-910cbd485390@linutronix.de
|
|
Document and verify that the hrtimer_cpu_base::lock is held at this point.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-3-910cbd485390@linutronix.de
|
|
Use an explicitly sized type to make the code a bit more consistent
with other fields of the datastructure and other sequence counters.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-2-910cbd485390@linutronix.de
|
|
The current name is not clear about its behavior.
Rename it.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-1-910cbd485390@linutronix.de
|
|
Re-arranging hrtimer_interrupt() switched the retry path to a local
counter and dropped the update of cpu_base->nr_retries, leaving the
field exported via /proc/timer_list stuck at zero.
Increment nr_retries only when another pass through the expiry loop
is started; the third attempt that falls through to hang handling is
still accounted by nr_hangs alone.
Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()")
Signed-off-by: Liang Hao <haohlliang@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260731150408.19554-1-haohlliang@gmail.com
|
|
The enable_data variable gets freed on most error paths in
event_enable_trigger_parse(). Use free() to free it and just before
returning normally, call retain_and_null_ptr(enable_data) just before a
successful exit to keep it from being freed. On success, the enable_data
is assigned to the trigger_data->private_data field.
Also add a comment to why event_trigger_free(trigger_data) is being called
before a successful exit.
Link: https://patch.msgid.link/20260807113558.0ff14e96@gandalf.local.home
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
WARN_ONCE() splats once per call site, so only the first offending event
registered is ever reported. The tree currently has six:
ice_{rx,tx}_dim_template, two hfi1 txq events, mtu3_ep and edma_log_io.
Whichever registers first hides the rest, and each has to be found again
on the next boot.
Add a pr_warn() next to the WARN_ONCE() so every offender is listed, the
same way test_event_printk() already pairs WARN_ON_ONCE() with pr_warn()
for unsafe %p* dereferences. The WARN_ONCE() stays so the condition still
fails tests and panics under panic_on_warn.
Link: https://patch.msgid.link/20260806215256.1680267-1-devnexen@gmail.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
On native sparc64, struct __kernel_old_timeval contains a four-byte hole
after tv_usec because tv_sec is 64-bit while __kernel_suseconds_t is 32-bit.
put_itimerval() fills only the named fields in a stack-allocated
__kernel_old_itimerval and copies the entire object to userspace, so
getitimer() can expose the two padding holes.
Zero-initialize the aggregate before assigning the fields so implicit
padding is deterministic before it crosses the user/kernel boundary.
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Codex:gpt-5
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260809190428.1523014-1-Jeremy.Jean@oss.cyber.gouv.fr
|
|
'select' does not work on config options in a 'choice', so currently the
'select VIRT_CPU_ACCOUNTING_GEN' for NO_HZ_FULL is dead, with the choice
option VIRT_CPU_ACCOUNTING_GEN only being enabled when NO_HZ_FULL=y because
the other choice members depend on NO_HZ_FULL=n.
Remove the dead select, and encode this relationship as a default of the
choice, instead.
This dead select was found by kconfirm, a static analysis tool for Kconfig.
Signed-off-by: Julian Braha <julianbraha@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
Link: https://patch.msgid.link/20260801160140.2391000-1-julianbraha@gmail.com
|
|
The last user was removed in commit a6d799608e6a ("ptp: Switch to
ktime_get_snapshot_id() for pre/post timestamps").
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260731-timekeeping-aux-must-check-v1-1-11ae93068497@linutronix.de
|
|
Writing 0 to kernel.ftrace_enabled has not reliably disabled ftrace
for years (FTRACE_OPS_FL_PERMANENT users already block it, and more
callers rely on ftrace always being on). Refuse the write instead of
leaving it in an inconsistent "disables some, not all" state: return
-EOPNOTSUPP and log a message. Reads and enabling (writing 1) are
unaffected.
Update the docs to note the deprecation up front.
Link: https://patch.msgid.link/20260806153000.4184871-2-andrey.grodzovsky@crowdstrike.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Pull bpf-next d114bb989367 ("Merge branch
'add-arena-argument-support-to-kfuncs-and-struct_ops'") to make the __arena
and __arena__nullable kfunc and struct_ops argument suffixes available. The
suffixed arguments will be used to convert sched_ext kfuncs and struct_ops
callbacks that currently pass arena pointers as scalars and rebase them by
hand.
|
|
apply_wqattrs_commit() updates node_nr_active->max unconditionally.
wq->node_nr_active[] is only allocated for unbound workqueues, so guard
the call before per-cpu workqueues start using this path.
No functional change: only unbound workqueues reach apply_wqattrs_*()
today.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
This allocates a pwq and binds it to the pool @attrs asks for.
Which pool that is becomes a property of the attrs (once per-cpu becomes
an affinity scope).
Remove the 'unbound" from the function name, given it will be bigger
than unbound.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
Leon Hwang
Breno Leitao
Waiman Long
chenhuguanshen
Xu Kuohai
Hui Zhu
Kumar Kartikeya Dwivedi
Jérémy Jean
Tejun Heo
fangqiurong
Daniel Borkmann
Gao Xiang
Guopeng Zhang
Zqiang
Ning Ding
Julia Lawall
Thomas Weißschuh (Schneider Electric)
Liang Hao
Steven Rostedt
David Carlier
Julian Braha
Andrey Grodzovsky