| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
dispatch_to_local_dsq() can run from scx_bpf_dsq_move_to_local() while
ops.dispatch() has recorded the current rq. Moving a task to a local DSQ
may switch to the source or destination rq before synchronously invoking
ops.dequeue() through the following path:
SCX_CALL_OP(dispatch, rq)
ops.dispatch()
scx_bpf_dsq_move_to_local()
scx_flush_dispatch_buf()
finish_dispatch()
dispatch_to_local_dsq()
scx_dispatch_enqueue()
local_dsq_post_enq()
call_task_dequeue()
SCX_CALL_OP_TASK(dequeue, locked_rq, ...)
The nested callback saves the recorded rq and restores it on return. If
the rq tracking does not follow the lock switch, update_locked_rq() can
trigger the following lockdep assertion while restoring an rq which is
no longer held:
WARNING: kernel/sched/sched.h:1641 at call_task_dequeue+0x160/0x170
Call Trace:
scx_dispatch_enqueue+0x2b0/0x460
dispatch_to_local_dsq+0x138/0x230
scx_flush_dispatch_buf+0x1af/0x220
scx_bpf_dsq_move_to_local___v2+0xe2/0x1c0
bpf__sched_ext_ops_dispatch+0x4b/0xa7
do_pick_task_scx+0x3b6/0x910
__pick_next_task+0x105/0x1f0
__schedule+0x3e7/0x1980
Introduce switch_rq_lock() to update the tracking state together with
each rq lock handoff. Use it in dispatch_to_local_dsq(),
move_remote_task_to_local_dsq() and the in-balance paths of
scx_dsq_move(), ensuring that scx_locked_rq() consistently refers to the
rq whose lock is actually held throughout the lock dance.
Fixes: 7fb39e4eb4c3 ("sched_ext: Save and restore scx_locked_rq across SCX_CALL_OP")
Cc: stable@vger.kernel.org # 7.1+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The signed gen_loader used to police its own metadata map from within
BPF: emit_signature_match() read the kernel-cached map->sha[] back
through hardcoded struct bpf_map offsets and compared it against a hash
that compute_sha_update_offsets() baked into the signed instructions,
after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[].
The kernel now verifies the metadata at BPF_PROG_LOAD time by folding
the frozen contents of the loader's exclusive fd_array maps into the
signature, so the loader no longer checks anything itself. Generated
loaders thus carry no verification logic of their own anymore: Nothing
in the signing chain depends on emitted loader bytecode doing the right
thing.
On the loading side, skel_internal.h now sets fd_array_cnt for a signed
load so the kernel scans fd_array for the exclusive metadata map -
still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD
round-trip to populate map->sha[] is gone. The struct bpf_map layout
BUILD_BUG_ON()s on the kernel side are removed as well: they only
pinned the ABI for the in-BPF read of map->sha[] that is no longer
needed. Same for the map->excl member. Note: gen_hash is retained; it
still marks a loader as signed so an untrusted host cannot re-dimension
maps or override initial values now covered by the signature.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A signed gen_loader program carries the programs, maps and relocations
it installs in a metadata array map. The loader instructions are covered
by the PKCS#7 signature, but the metadata map is not: Today the loader
compares the map contents from within BPF against a hash baked into its
(signed) instructions, using the kernel-cached map hash. The kernel
itself never actually attests that the metadata the loader installs is
the metadata that was signed.
This split is the core of the long-standing objection to the BPF signing
scheme from the LSM / integrity side: the integrity check of a light
skeleton only completes once the loader program runs, that is, after the
security_bpf_prog_load() hook, so at admission time an LSM observes a
program whose payload has not yet been verified. Auditing the chain
link is also not a purely cryptographic operation: whoever signs or
reviews an lskel has to disassemble the loader's preamble to convince
themselves that the embedded hash check is present and correct [0][1].
Two acceptable fixes were identified in those threads: Complete the
integrity check before the admission hook fires, or add a second hook
that collects the verification result after the loader ran [2]. Covering
both the loader and its maps directly with the PKCS#7 signature is what
Blaise Boscaccy's patchsets proposed in several forms. Let's implement
the former, without growing the UAPI, and in particular as a single
unified scheme where the signature spans the raw bytes rather than
derived hashes.
A signed loader binds its metadata map(s) through the existing fd_array,
and an exclusive map is already bound to a program digest (excl_prog_hash).
So when a signature is present, collect the exclusive maps from fd_array
and append their frozen contents to the instructions before verification:
The signature now covers insns || metadata_0 || metadata_1 || [...] in
the fd_array order, and verification completes in bpf_check(), once the
fd_array maps are resolved into used_maps, before the LSM admission hook
and the rest of verification. A program is either BPF_SIG_UNSIGNED or
BPF_SIG_VERIFIED, with nothing in between. While folding the fd_array
maps, a non-exclusive map bound to a signed program is rejected, so every
map folded into the signature is exclusive. A signed loader that fails
to cover its metadata thus does not load, and BPF_SIG_VERIFIED always
means the instructions and every exclusive map are authentic. The maps
must be frozen so the hashed bytes cannot change before the loader runs;
the map <-> program digest binding is enforced by the verifier for every
used map. Binding maps through fd_array_cnt makes the verifier resolve
and excl-check them (excl_prog_sha vs prog->digest) before it would
otherwise compute the digest, so compute prog->digest up front in
bpf_check(), over the unmodified instructions the signature covers, for
a load that folds metadata.
Unsigned programs are not affected by the signature path; for them the
LSM admission hook merely moves below fd_array resolution, with minimal
bounded work in between. Note, signed loaders generated by older libbpf/
bpftool versions need to be regenerated; some of the recent fixes we've
had on the signed loader side require the latter already to close gaps.
Finally, some remarks around the security_bpf_prog_load() placement
given there was discussion on whether a new hook is needed or the existing
security_bpf_prog() hook should be reused [3]: For a new hook it would
mean that just for loading a single BPF program it has to pass through
four layers of LSM hooks:
1) security_bpf (cmd=PROG_LOAD): for gating various bpf subcmds
2) security_bpf_prog_load: historical admission hook (CAP/token,
prog_type, attach point), pre-verification
3) security_bpf_prog_verify_signature: newly asked admission hook,
same role as 2), plus the BPF signature verdict
4) security_bpf_prog: gate handing the prog fd back to userspace,
verification done & signature verified
The use-cases of 2) and 3) conflate, thus BPF community prefers to just
keep a total of 3 LSM hooks (as-is today): 3) makes 2) incoherent given
they are the /same class/ of hook, that is, access-control admission on
the load and split only by _what_ they can see. Worse, with the split,
for a signed BPF program security_bpf_prog_load 2) admits a program whose
signature has not been checked, so a policy gating at 2) is structurally
unable to express "admit only verified" and every such policy is forced
onto 3) *anyway*. In other words, one doesn't get two complementary hooks,
but rather, one real admission hook aka 3) plus a now-degraded /legacy/
hook 2) that can't answer the question operators actually want to ask.
Reusing security_bpf_prog() 4) for admission is no alternative either:
it fires only after the entire verifier (and JIT) pipeline ran, so
denying a not-yet-verified program at that point burns exactly the
work a denial is supposed to avoid, and by then the program has an id
assigned and the kallsyms/perf/audit load events fired. Policies are
free to also consume the signature verdict at 4), but admission control
belongs into security_bpf_prog_load(). Hence the latter remains the only
admission hook, merely moved past signature verification; with moving
large allocations further down into the BPF verifier, there is now only
minimal work between the old and new location: The preparation work in
bpf_check() is reordered such that only the minimally necessary setup
happens up front: Allocating the env, initializing the verifier log and
resolving the fd_array that a signed BPF metadata map needs. The worst
case allocation up until security_bpf_prog_load() is ~90K which is the
env itself (~54K) plus the continuous fd_array cache (at most 32K). The
insn_aux_data array is moved into a later stage in the verification.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com [0]
Link: https://lore.kernel.org/lkml/ecf0521ed302db672672ebfbc670ecfba36a6e00.camel@HansenPartnership.com [1]
Link: https://lore.kernel.org/bpf/88703f00d5b7a779728451008626efa45e42db3d.camel@HansenPartnership.com [2]
Link: https://lore.kernel.org/bpf/DJOFY21DYUI4.19WKQ3NPZ4H5R@gmail.com [3]
Link: https://lore.kernel.org/bpf/20260708075343.358712-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The fd_array passed to BPF_PROG_LOAD carries the map and module BTF file
descriptors a program binds. The verifier reads it more than once during
a load: process_fd_array() walks it to bind the maps and BTFs, and
check_and_resolve_insns() and the kfunc BTF resolver later read it again
to resolve the program's BPF_PSEUDO_MAP_IDX* and module kfunc refs.
For signed BPF, we need these upfront in memory, thus resolve each fd to
its object once and cache it by fd_array index, then bind that cached
object for the rest of the load. env->fd_array becomes a small per-slot
{map, btf} cache rather than a bpfptr_t; every later reference is then
an in-bounds lookup of an already-resolved object, and an index outside
the cache is rejected instead of read from user memory:
- continuous (fd_array_cnt given): the caller declares the length and
every entry is resolved and bound up front (used also by the BPF
signed loader)
- sparse (no fd_array_cnt): left as the legacy path with no fd_array
cache; each reference reads its fd from the caller's fd_array and
resolves it on the spot. Deduplication in used_maps and the kfunc BTF
table keeps this correct, and only unsigned programs use this shape.
Split these into separate helpers to make it easier to follow.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Anton Protopopov <a.s.protopopov@gmail.com>
Link: https://lore.kernel.org/bpf/20260708075343.358712-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Remove unused arch_needs_cpu() hook. No architectures use it after
s390 removed its use case.
Suggested-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
|
|
Background:
==========
Currently lockdep does not print out the held locks of non-current
tasks that are running on some other CPU, due to the fact that
the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking
bugs is easier if we print this information too, because the
more locking information the merrier. In particular races are
bound to have multiple tasks running on different CPUs, and
the exclusion of their held locks information is unnecessarily
limiting.
So while it's still true that printing out their held locks
array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be
safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held,
which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API
with the constraint that it should only be called by drivers
or platform code if the task isn't actively running - we can
assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(),
while the task cannot go away, so the held-locks array itself is
safe (although potentially non-stable), AFAICS the worst-case race
can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another
CPU), but it should still be a valid class index bound by
MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use
should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);
But both are printed out as pointers, so no risk of dereference
of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks()
from printing garbage, as there's nothing that keeps a task from
becoming runnable a nanosecond after we've run the task_is_running()
check. In fact I'd argue that it's better to make this function
*more* racy, for the simple robustness reason that we absolutely
do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running
tasks too, as long as we print out the information as well
that a task is running, so that users are aware of any
racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into
a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a
dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also
be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very
useful information for races and for locking bugs in particular.
This basically extends the 'on CPU#%d' message we print for
running tasks to all tasks we print.
Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Tested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Boqun Feng <boqun@kernel.org>
Cc: Gary Guo <gary@garyguo.net>
Cc: Mark Brown <broonie@kernel.org>
Cc: Theodore Tso <tytso@mit.edu>
Cc: Miguel Ojeda <ojeda@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Waiman Long <longman@redhat.com>
Link: https://patch.msgid.link/akoeSIQGwqd9cZwd@gmail.com
|
|
Add a matching sysctl to go with CONFIG_SYSCALL_USER_DISPATCH.
kernel.syscall_user_dispatch (default 1 - allow) controls whether
userspace may arm syscall user dispatch (both via prctl and ptrace).
Disarming is always permitted - same semantics as comparable knobs.
Disabling while a task has armed syscall user dispatch does not
cause it to become inactive - instead it remains active until the
user attempts to disable/re-enable via prctl or ptrace. On the
next attempt to re-enable, the prctl/ptrace call fails gracefully.
The alternative would cause programs translating non-linux syscalls
to interpret those syscalls as linux syscalls, resulting in undefined
userland behavior.
Signed-off-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706140020.873735-3-gourry@gourry.net
|
|
Syscall User Dispatch is presently built under CONFIG_GENERIC_SYSCALL and
cannot be disabled independently.
Add CONFIG_SYSCALL_USER_DISPATCH to make it an optional feature.
Signed-off-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260706140020.873735-2-gourry@gourry.net
|
|
check_ptr_to_btf_access() lets program-type btf_struct_access callbacks
validate writes before the default BTF access path rejects non-read
accesses. That bypasses the read-only policy for untrusted BTF pointers
created by helpers such as bpf_rdonly_cast().
Reject non-read accesses through PTR_UNTRUSTED BTF pointers at the
common entry point, before the callback branch to handle all cases.
Fixes: 282de143ead9 ("bpf: Introduce allocated objects support")
Signed-off-by: Nicholas Dudar <main.kalliope@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The verifier loses precision when simulating stack spills for the
following register types:
- PTR_TO_TP_BUFFER
- PTR_TO_INSN
- CONST_PTR_TO_DYNPTR
These types are not allow-listed in the is_spillable_regtype(),
because of that check_stack_write_fixed_off() takes the branch
that marks the slots STACK_MISC.
There are no technical reasons for this limitation.
This commit replaces an explicit list of pointer types in
is_spillable_regtype() with explicit list of non-pointer types.
The function is renamed to is_pointer_regtype() for clarity.
Reported-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260707-missing-spillable-types-v1-1-44a92121dc41@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
timekeeping_apply_adjustment() modifies xtime_nsec to ensure monotonicity
when mult changes:
xtime_nsec -= offset
This ensures that the time reported to userspace does not jump when the
multiplier is adjusted from one tick to the next. However, the ntp_error
accumulator which tracks the difference between intended and actual
clock position was not being updated to reflect this additional
discrepancy.
An earlier attempt at this compensation existed as:
ntp_error -= (interval - offset) << ntp_error_shift
but was removed in commit c2cda2a5bda9 ("timekeeping/ntp: Don't align
NTP frequency adjustments to ticks") because it was a major source of
NTP error. That's because (interval - offset) was wrong: the subtraction
of "interval" prematurely accounted for the changed xtime_interval of
the next tick, which would be correctly accounted in the next
accumulation anyway — a double subtraction.
What is actually needed is just the "offset" part: ntp_error must be
told that xtime_nsec moved by "offset" without a corresponding change
in the intended position. For the normal ±1 mult dithering this is
negligible (the adjustments cancel over time), but for larger mult
changes — such as when an external reference clock sets a new
frequency — the one-time uncompensated offset is significant.
Fix by adjusting ntp_error by the correct amount:
ntp_error += offset << ntp_error_shift
This keeps ntp_error consistent with the actual xtime_nsec position
after the adjustment, and ensures the discrepancy is correctly smoothed
away over time and the clock returns to where it should have been.
Fixes: c2cda2a5bda9 ("timekeeping/ntp: Don't align NTP frequency adjustments to ticks")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Kiro:claude-opus-4.6-1m
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260621220051.1030462-3-dwmw2@infradead.org
|
|
The "old" time types use 32-bit seconds which are not y2038-safe.
Respect COMPAT_32BIT_TIME for functions using those types.
time(), stime() and gettimeofday() are disabled completely.
settimeofday() is kept as it is required to do the initial timewarping
after boot. However the 'tv' argument will be rejected.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Link: https://lore.kernel.org/lkml/e9487ebe-3730-438a-9c23-e45f75986ecc@app.fastmail.com/
Link: https://patch.msgid.link/20260702-vdso-compat_32bit_time-v3-1-db9f36d8d432@linutronix.de
|
|
The fault handler for the timens page does not have access to the target
task and therefore can not be invoked remotely.
Currently the handler relies on the fact that the vvar mapping is marked as
VM_IO and VM_PFNMAP for which the mm core always prevents remote access.
However the VM_IO and VM_PFNMAP flags are going to be removed.
Add an explicit check to prevent remote access to the mapping.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-4-6c93708ce723@linutronix.de
|
|
All architectures are now fully using the generic vDSO infrastructure.
They don't need these declarations anymore to implement the functions
in architecture-specific code.
Move them to the private header.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260707-timekeeping-header-cleanup-v1-2-e85ad96409a9@linutronix.de
|
|
This header uses some definitions from linux/hrtimer_bases.h.
Currently this header is included transitively, which will change.
Include the header explicitly.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-5-c50b19bda473@linutronix.de
|
|
This header uses some definitions from linux/hrtimer_bases.h.
Currently this header is included transitively, which will change.
Include the header explicitly.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-4-c50b19bda473@linutronix.de
|
|
The usage of the hrtimer base forces hrtimer.h to also expose the base
structure definitions.
Move the function to hrtimer.c to avoid this.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-3-c50b19bda473@linutronix.de
|
|
The usage of the hrtimer base introduces a dependency on the timer base
structure definitions from the widely-used hrtimer.h.
Move the helper to hrtimer_bases.h to trim this dependency.
Also adapt the two only callers to now include hrtimer_bases.h.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-2-c50b19bda473@linutronix.de
|
|
hrtimer_get_next_event() runs on every tick-stop decision via
get_next_timer_interrupt() -> cmp_next_hrtimer_event(). When high
resolution timers are active it must return KTIME_MAX -- the caller
documents and depends on this.
The function takes cpu_base->lock, checks hres_active, and returns
KTIME_MAX.
Taking the lock is not required because cpu_base->hres_active is only
written by the local CPU in hrtimer_switch_to_hres() from hard interrupt
context and in hrtimers_cpu_starting() during bring-up.
All callers of hrtimer_get_next_event() reach it from the tick-stop /
cpuidle paths with interrupts disabled on that CPU. No writer can therefore
race with the read, so an unlocked hres_active check is stable and the lock
can be skipped in this case.
On a 176-thread AMD EPYC 9D64 running a production workload, bucketing
callers of native_queued_spin_lock_slowpath(), the slowpath had 199 samples
in total, of which 54 are attributed to hrtimer_get_next_event(), i.e. this
accounts for ~27% of slowpath hits on this specific workload.
Move the hres_active check before the lock guard region to address this.
[ tglx: Massaged change log and comments ]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260707153800.542394-1-usama.arif@linux.dev
|
|
print_cpu() prints hrtimer_cpu_base and tick_sched state without
synchronizing with concurrent updates. The output is diagnostic only, so
use data_race(READ_ONCE()) for these scalar reads to document the
intentional races and avoid KCSAN reports.
Reported-by: syzbot+8f0e958900a14d08a51d@syzkaller.appspotmail.com
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260707125813.2182532-1-pengyu@kylinos.cn
Closes: https://syzkaller.appspot.com/bug?extid=8f0e958900a14d08a51d
|
|
The function validates tv_sec to be within [-KTIME_SEC_MAX, KTIME_SEC_MAX]
but never validates that tv_nsec is within the valid range of
[0, NSEC_PER_SEC-1] before using it in timespec64_add().
timespec64_add() expects both timespec64 structures to have normalized
values with tv_nsec in the range [0, 999999999]. If off->val.tv_nsec
contains invalid values (negative or >= NSEC_PER_SEC), it could lead to
incorrect calculations or unexpected behavior.
Add validation to ensure tv_nsec is within the valid range before
performing the addition.
Fixes: 04a8682a71be ("fs/proc: Introduce /proc/pid/timens_offsets")
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260704093429.89350-1-malayarout91@gmail.com
|
|
When the WARN_ON_ONCE(i >= tmigr_hierarchy_levels) assertion triggers,
the function returns -EINVAL without freeing the 'stack' memory allocated
via kzalloc_objs() at the beginning of the function.
Add kfree(stack) before returning to prevent the memory leak.
Fixes: 6c181b5667ee ("timers/migration: Convert "while" loops to use "for"")
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260704085533.87098-1-malayarout91@gmail.com
|
|
tk_aux_sysfs_init() creates one child kobject per auxiliary clock. If a
later child or sysfs group creation fails, the current error path only
puts the parent kobjects and leaves earlier children and groups behind.
Store the child kobjects during init and remove the successfully created
groups and kobjects on failure.
Fixes: 7b5ab04f035f ("timekeeping: Fix resource leak in tk_aux_sysfs_init() error paths")
Signed-off-by: Yuho Choi <dbgh9129@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260703165337.168445-1-dbgh9129@gmail.com
|
|
init_clocksource_sysfs() registers the clocksource subsystem before
registering the clocksource device. If device_register() fails, the
function returns the error while leaving the subsystem registered.
Unregister the clocksource subsystem on that failure path so the
successful subsystem registration is unwound before returning.
Fixes: d369a5d8fc70 ("clocksource: convert sysdev_class to a regular subsystem")
Signed-off-by: Yuho Choi <dbgh9129@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260702215733.84588-1-dbgh9129@gmail.com
|
|
The kstat_irqs field of struct irq_desc is used to store a per-cpu count
of interrupt events. It is initialized in init_desc(), along with all
the other fields in struct irq_desc that need explicit initialization,
and therefore it's always available (non-NULL) for any valid interrupt
descriptor (with a caveat - see below).
When CONFIG_SPARSE_IRQ is enabled, all interrupt descriptors are always
allocated dynamically via alloc_desc(), which calls init_desc(), so in
that case kstat_irqs is guaranteed to be non-NULL before a valid struct
irq_desc pointer is even returned.
By contrast, when CONFIG_SPARSE_IRQ is disabled, interrupt descriptors
are allocated statically in the irq_desc[] array, and kstat_irqs is
initialized implicitly to NULL. The per-cpu pointer is initialized only
later, for all descriptors, via start_kernel() -> early_irq_init() ->
init_desc().
The kstat_irqs field is used mostly for printing interrupt statistics
(i.e. reading /proc/interrupts), and that cannot happen until much
later, when user-space is fully initialized. So, there is no concern
with that use case.
The list below includes all functions where the NULL check is removed,
along with a list of all possible call chains and/or a brief explanation
of why it's safe to remove the NULL check in that case.
* irq_desc_kstat_cpu() [include/linux/irqdesc.h]
- All direct call sites use it for printing IRQ statistics.
- Indirect call site: per_cpu_count_show() - also used for printing
interruptstatistics.
* kstat_irqs_cpu() [kernel/irq/irqdesc.c]
- Called by sun3_int7() and sun3_int5() [arch/m68k/sun3/sun3ints.c]
These are interrupt handlers and cannot be called until their
corresponding interrupts are initialized in sun3_init_IRQ(). The
call chain leading to that is:
start_kernel() -> init_IRQ() [arch/m68k/kernel/ints.c] ->
mach_init_IRQ = sun3_init_IRQ()
The init_IRQ() call happens right *after* the early_irq_init() call,
which means the descriptors are already fully initialized by the
time the interrupt handlers are even registered.
- Called by show_interrupts() [arch/s390/kernel/irq.c] - used for
printing interrupt statistics.
* kstat_irqs() [kernel/irq/irqdesc.c]
The only possible call chain is via fs/proc/stat.c:
stat_open() -> show_stat() -> show_all_irqs() -> kstat_irqs_usr() ->
kstat_irqs()
It is used for printing interrupt statistics.
* kstat_snapshot_irqs()
The only possible call chain is via kernel/watchdog.c:
watchdog_timer_fn() -> is_softlockup() -> start_counting_irqs() ->
kstat_snapshot_irqs()
The watchdog timer cannot fire early, before early_irq_init().
* kstat_get_irq_since_snapshot()
The only possible call chain is via kernel/watchdog.c:
watchdog_timer_fn() -> report_cpu_status() -> print_irq_counts() ->
kstat_get_irq_since_snapshot()
The watchdog timer cannot fire early, before early_irq_init().
Signed-off-by: Radu Rendec <radu@rendec.net>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260705210951.2717741-2-radu@rendec.net
|
|
sock_map_update_common() and __sock_map_delete() hold stab->lock and call
sock_map_unref() -> sock_map_del_link(), which takes sk_callback_lock for
write. That gives the order stab->lock -> sk_callback_lock.
The reverse order comes from the SK_SKB stream parser.
sk_psock_strp_data_ready() holds sk_callback_lock for read, and after the
verdict tcp_bpf_strp_read_sock() acks the consumed data inline via
__tcp_cleanup_rbuf(). The ACK goes out egress, where a sched_cls program
deletes from the sockmap and takes stab->lock:
WARNING: possible circular locking dependency detected
------------------------------------------------------
syz.9.8824 is trying to acquire lock:
(&stab->lock){+.-.}-{3:3}, at: __sock_map_delete net/core/sock_map.c:421
but task is already holding lock:
(clock-AF_INET){++.-}-{3:3}, at: sk_psock_strp_data_ready net/core/skmsg.c:1173
-> #1 (clock-AF_INET){++.-}-{3:3}:
_raw_write_lock_bh
sock_map_del_link net/core/sock_map.c:167
sock_map_unref net/core/sock_map.c:184
sock_map_update_common net/core/sock_map.c:509
sock_map_update_elem_sys net/core/sock_map.c:588
map_update_elem kernel/bpf/syscall.c:1805
-> #0 (&stab->lock){+.-.}-{3:3}:
_raw_spin_lock_bh
__sock_map_delete net/core/sock_map.c:421
sock_map_delete_elem net/core/sock_map.c:452
bpf_prog_06044d24140080b6
tcx_run net/core/dev.c:4451
sch_handle_egress net/core/dev.c:4541
__dev_queue_xmit net/core/dev.c:4808
...
tcp_bpf_strp_read_sock net/ipv4/tcp_bpf.c:701
strp_data_ready net/strparser/strparser.c:402
sk_psock_strp_data_ready net/core/skmsg.c:1174
tcp_data_queue net/ipv4/tcp_input.c:5661
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
rlock(clock-AF_INET);
lock(&stab->lock);
lock(clock-AF_INET);
lock(&stab->lock);
*** DEADLOCK ***
A tc, xdp, socket_filter or flow_dissector program has no reason to
update or delete a sockmap, and redirect does not go through here. Drop
them from may_update_sockmap() so the verifier rejects it. It also
closes the matching sockhash inversion.
Suggested-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: John Fastabend <john.fastabend@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260630145410.3648099-2-rhkrqnwk98@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
parse_synth_field() builds a "__data_loc ..." type string before
assigning it to field->type. If the seq_buf check fails, the common
cleanup cannot free the temporary string. Free it before leaving.
Link: https://patch.msgid.link/20260603062533.1096320-2-pengyu@kylinos.cn
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
user_event_mm_dup() walks the parent mm's enabler list locklessly under
rcu_read_lock() during fork() (from copy_process()); it does not take
event_mutex:
rcu_read_lock();
list_for_each_entry_rcu(enabler, &old_mm->enablers, mm_enablers_link)
enabler->event = user_event_get(orig->event);
user_event_enabler_destroy() removes an enabler from that list with
list_del_rcu() and then, without waiting for a grace period, drops the
enabler's user_event reference with user_event_put() and frees the enabler
with kfree(). A reader that loaded the enabler before the list_del_rcu()
can still be walking it, which leads to two use-after-frees:
- kfree(enabler) frees the enabler while that reader dereferences
enabler->event.
- user_event_put() may drop the last reference to the user_event, which
is then freed (via delayed_destroy_user_event() on a work queue), while
the same reader does user_event_get(orig->event) on it.
Both are reachable by an unprivileged task that can open user_events_data:
one multithreaded process that registers an enabler and then concurrently
unregisters it and calls fork() triggers the race. KASAN reports a
slab-use-after-free in user_event_mm_dup() during clone(), with a
"refcount_t: addition on 0" warning when the user_event is freed.
The enabler use-after-free was found first; the user_event one was reported
by XIAO WU, and the earlier enabler-only fix did not address it.
Defer both the user_event_put() and the kfree(enabler) to a work item
queued with queue_rcu_work(), so they run only after an RCU grace period,
once all readers walking the enabler list have finished. The put must run
in process context because user_event_put() takes event_mutex on the last
reference, so a work queue is used rather than call_rcu(). The now-unlocked
put lets the locked argument of user_event_enabler_destroy() be removed;
all callers are updated.
Fixes: 7235759084a4 ("tracing/user_events: Use remote writes for event enablement")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260707165912.2560537-2-michael.bommarito@gmail.com
Reported-by: XIAO WU <xiaowu.417@qq.com>
Closes: https://lore.kernel.org/all/tencent_89647CE40DC452B891C65C94D1B271DE8E07@qq.com/
Suggested-by: Beau Belgrave <beaub@linux.microsoft.com>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
BTF struct walks relax the struct-size check for accesses through a
trailing flexible array. That is valid for ordinary BTF type walking, but
PTR_TO_BTF_ID | MEM_ALLOC values point to objects allocated with the static
BTF type size.
When walking a MEM_ALLOC object, reject the access before applying the
flexible-array relaxation if the access range extends past the struct size.
Apply the same policy to struct ID matching so kfunc and kptr type checks
do not walk past the allocated object bounds either.
Fixes: 958cf2e273f0 ("bpf: Introduce bpf_obj_new")
Fixes: 36d8bdf75a93 ("bpf: Add alloc/xchg/direct_access support for local percpu kptr")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/4b8c8a81102ba4b595011434c881194f264ddc59.1782807039.git.chenyy23@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Tracepoints require that RCU is watching. To prevent them from being used
in places that RCU is not watching, the trace_##event() macro always
calls rcu_is_watching() even when the event is not enabled and warns if
RCU is not watching. This is to make sure a warning is triggered even if
the tracepoint is never enabled (as it is only a bug when it is).
It was noticed that tracepoints could be hidden within
trace_#event#_enabled() calls, which are used to do extra work for the
tracepoint only if the tracepoint is enabled. But this also can hide the
fact that a tracepoint is placed in a location that can be called when RCU
is not watching.
Commit 9764e731ef6ab ("tracepoint: Add lockdep rcu_is_watching() check to
trace_##name##_enabled()") added a check to the trace_##event##_enabled()
macro to make sure RCU is watching when it is called to make sure not to
hide the bug of a tracepoint being called when RCU is not watching.
There is one case in the irq_disable tracepoint where it is within a
trace_irq_disable_enabled() block, but it checks if RCU is watching, and
if it isn't, it makes a call to ct_irq_enter() that makes RCU watch again.
But because trace_irq_disable_enabled() now checks if RCU is watching and
will trigger if it isn't. This is a false warning as the code within
the block handles this case.
Add a new internal macro __trace_##event##_enabled() that doesn't check if
RCU is watching, and convert the irq_enable/disable tracepoints over to
it.
Link: https://patch.msgid.link/20260701132744.6a7fc68b@robin
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Closes: https://lore.kernel.org/all/CAMuHMdXud_RpWag_hFqa2ByBGRxg6KnxGL1ObCWZrpTsk3TfAw@mail.gmail.com/
Fixes: 9764e731ef6ab ("tracepoint: Add lockdep rcu_is_watching() check to trace_##name##_enabled()")
Tested-by: Geert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
String event fields are not necessarily NUL-terminated, so the filter
predicate functions (filter_pred_string(), filter_pred_strloc() and
filter_pred_strrelloc()) pass the field length to the regex match
callbacks, and the length-aware matchers honour it.
regex_match_glob() was the exception: it ignored the length and called
glob_match(), which scans the string until it hits a NUL byte. Some
string fields are not NUL-terminated. One example is the dynamic char
array of the xfs_* namespace tracepoints, which is copied without a
trailing NUL. For such a field, glob matching reads past the end of
the event field, causing a KASAN slab-out-of-bounds read in
glob_match(), reached via regex_match_glob() and filter_match_preds()
from the xfs_lookup tracepoint.
Add a length-bounded glob_match_len() and use it from regex_match_glob()
so glob matching always stops at the field boundary. The matching loop
is factored into a shared helper so glob_match() keeps its behaviour.
Fixes: 60f1d5e3bac4 ("ftrace: Support full glob matching")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/da1aaf125fc3b63320b0c540fd6afa7c3d5b4f1a.1782836943.git.hhhuang@smu.edu.sg
Reported-by: Yuan Tan <yuantan098@gmail.com>
Reported-by: Yifan Wu <yifanwucs@gmail.com>
Reported-by: Juefei Pu <tomapufckgml@gmail.com>
Reported-by: Zhengchuan Liang <zcliangcn@gmail.com>
Reported-by: Xin Liu <bird@lzu.edu.cn>
Assisted-by: Codex:GPT-5.4
Signed-off-by: Huihui Huang <hhhuang@smu.edu.sg>
Signed-off-by: Ren Wei <n05ec@lzu.edu.cn>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
func_set_flag() dereferences tr->current_trace_flags before verifying
that the current tracer is actually the function tracer. When the active
tracer has been switched away from "function" (e.g., to "wakeup_rt"),
tr->current_trace_flags can be NULL, leading to a NULL pointer
dereference and kernel crash.
The call chain that triggers this is:
trace_options_write()
-> __set_tracer_option()
-> trace->set_flag() /* func_set_flag */
In func_set_flag(), the first operation is:
if (!!set == !!(tr->current_trace_flags->val & bit))
This dereferences tr->current_trace_flags unconditionally. The safety
check that guards against a non-function tracer:
if (tr->current_trace != &function_trace)
return 0;
is placed *after* the dereference, which is too late.
This was observed with the following crash dump:
BUG: unable to handle page fault at 0000000000000000
RIP: func_set_flag+0xd
Call Trace:
__set_tracer_option+0x27
trace_options_write+0x75
vfs_write+0x12a
ksys_write+0x66
do_syscall_64+0x5b
RIP: ffffffff914c973d RSP: ff67ec88b01dfdf0 RFLAGS: 00010202
RAX: 0000000000000000 RBX: ff3a826e80354580 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff93918080
The disassembly confirms the fault:
func_set_flag+0: mov 0x1f08(%rdi), %rax ; RAX = tr->current_trace_flags = NULL
func_set_flag+13: mov (%rax), %eax ; page fault: dereference NULL
At the time of the crash:
tr->current_trace_flags = 0x0 (NULL)
tr->current_trace = wakeup_rt_tracer (not function_trace)
The scenario is that a process opens a function tracer option file (such
as "func_stack_trace"), then the current tracer is switched to another
tracer (e.g., "wakeup_rt"), which sets current_trace_flags to NULL. When
the process subsequently writes to the option file, func_set_flag() is
invoked and crashes on the NULL dereference.
Fix this by moving the current_trace check before the
current_trace_flags dereference, so that func_set_flag() returns early
when the function tracer is not active.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260624061715.1445655-1-xiangzao@linux.alibaba.com
Fixes: 76680d0d2825 ("tracing: Have function tracer define options per instance")
Signed-off-by: Yuanhe Shu <xiangzao@linux.alibaba.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The tracepoint_printk symbol is not exported, so make it
static to remove the following sparse warning:
kernel/trace/trace.c:90:5: warning: symbol 'tracepoint_printk' was not declared. Should it be static?
Fixes: dd293df6395a2 ("tracing: Move trace sysctls into trace.c")
Link: https://patch.msgid.link/20260617105822.904164-1-ben.dooks@codethink.co.uk
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Commit 8928e4a3be34 ("ring-buffer: Show persistent buffer dropped events
in trace_pipe file") split the "commit" variable in
ring_buffer_read_page() into "commit" (raw) and "size" (masked page
size), but the inner copy loop's terminator was changed to compare rpos
against "event_size" instead of "size".
rpos is the cumulative read offset within the page; event_size is the
length of the single event just copied. The loop thus breaks after the
first event, so only one event is copied per call. This regresses the
per-event memcpy path (partial reads, the active commit page, and
mapped/remote buffers) used by splice/trace_pipe_raw and mmap consumers
into a one-event-at-a-time read.
Compare rpos against the page size as the original code did.
Link: https://patch.msgid.link/20260616175538.111628-1-devnexen@gmail.com
Fixes: 8928e4a3be34 ("ring-buffer: Show persistent buffer dropped events in trace_pipe file")
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A NULL old_name in a newly loaded livepatch's function entry causes a
NULL pointer dereference in strcmp():
klp_init_patch()
klp_add_nops()
klp_find_func()
strcmp(old_func->old_name, func->old_name)
Add klp_check_patch() at the beginning of klp_enable_patch() to reject
patches with NULL old_name before they reach this code path.
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/live-patching/20260529040130.95A9C1F00893@smtp.kernel.org/
Suggested-by: Petr Mladek <pmladek@suse.com>
Suggested-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260628114635.33572-1-laoar.shao@gmail.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
UAPI Changes:
- Add additional error components to xe drm_ras (Riana)
- Drop 'force_execlist' module parameter (Roper)
Cross-subsystem Changes:
- Perf events: Export perf_allow_{cpu,tracepoint} to be used by Xe (John)
Display Changes:
- Skip FORCE_WC and vm_bound check for external dma-bufs (Auld)
Driver Changes:
- Gate observation streams with perf_allow_cpu (John Hubbard)
- Documentation updates and fixes (Michal, Zhan)
- Remove unreleased NVL-S GuC (Daniele)
- Fix pcode init path (Michal)
- RTP fixes and improvements (Gustavo, Violet, Thomas, Roper)
- TLB invalidation fixes and improvements (Tilak)
- PXP detachment from HuC for newer platforms (Daniele)
- Multi-queue fix (Niranjana)
- Improve Kconfig.profile help (Rodrigo)
- Xe_drm_ras and hw_error updates and fixes (Raag, Riana)
- NVL-S updated PCI-IDs and W/a (Gustavo, Nitin)
- Fix dma_fence refcound (Wentao)
- Madvise: optimize invalidation path (Arvind)
- Fix a infinite gt-reset loop in the timeout recovery (Rodrigo)
- Fix wa_oob codegen recipe for external module builds (Thomas)
- Avoid global forcewake in cycle query path (Xin)
- Update TTM device benefical_order (Brost)
- Fix buffer overflow in guc capture (Tejas)
- Page-table fixes and improvements (Brian, Francois, Auld, Brost)
- Removing redundant check (Lu)
- General MCR and MMIO clean-up and improvements (Michal)
- Don't whitelist OA registers unconditionally (Ashutosh)
- SVM error return fix (Brost)
- Userptr fix and small related clean-ups (Shuicheng)
- Don't attempt to process FAST_REQ or EVENT relays on PF (Michal)
- Couple fdinfo improvements (Auld)
- Drop manual VF check on i2c (Raag)
- Probe info outside of xe_info_init functions (Gustavo)
Signed-off-by: Dave Airlie <airlied@redhat.com>
From: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/akf7xr96MI4Rd6Qj@intel.com
|
|
There are 2 possible scenarios where the cgroup_taskset structure
passed into the cgroup can_attach() and attach() methods can contain
task migration data with multiple source cpusets.
- A multithread application with threads in different cpusets is
fully migrated into a new cpuset.
- Disabling v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
The current cpuset_can_attach() and cpuset_attach() functions still
expect task migration is from one source cpuset to one destination
cpuset.
Fix that by tracking the set of source (old) cpusets in singly linked
lists. The list will be iterated when necessary to properly update
internal data.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both
the source and destination cpusets are decremented/incremented with
their values added to nr_deadline_tasks when the migration is successful.
The setting of the global attach_ctx.cpus_updated and
attach_ctx.mems_updated flags are also moved from cpuset_attach()
to cpuset_can_attach() as the correct source cpuset can no longer be
determined in cpuset_attach() and cpuset states will not be changed
between cpuset_attach() and cpuset_can_attach() with an earlier patch.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The cpuset_attach_task() was introduced in commit 42a11bf5c543
("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly")
to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like
moving a task from one cpuset into another one. That commits didn't
move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader
into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new
task is its own group leader. So it is still not equivalent to moving
task between cpusets in this case. Make CLONE_INTO_CGROUP behaves
more close to cpuset_attach() by moving the mpol_rebind_mm() and
cpuset_migrate_mm() calls inside cpuset_attach_task().
Also move the stack local cpus_updated, mems_updated and queue_task_work
flags into attach_ctx so that these flags can be accessed inside and
outside of cpuset_attach_task(). The cpuset_fork() function is updated
to set up these flags and do memory migration if necessary.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
There are two possible ways that migration of tasks from multiple source
cpusets to a target cpuset can happen. Either a multithread application
with threads in different cpusets is wholely migrated to a new cpuset
or disabling of v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that
really matters. So attach_ctx.old_cs should track the oldcs of the
thread leader. In the latter case, effective_mems of child cpusets
must always be a subset of the parent. So no real page migration
will not be necessary no matter which child cpuset is selected as
attach_ctx.old_cs.
IOW, attach_ctx.old_cs should be updated to match the latest task
group leader in cpuset_can_attach(), but fall back to that of the first
task if there is no group leader in the taskset.
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Expand the scope of cpuset_can_attach_check() by including the setting
of setsched flag inside cpuset_can_attach_check() with the new @oldcs
and @psetsched argument. As cpuset_can_attach_check() is also called
from cpuset_can_fork(), set the new arguments to NULL from that caller.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Extract the DL bandwidth allocation code in cpuset_attach() to a new
cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Put the task attach related cpuset_attach_old_cs and
cpuset_attach_nodemask_to static variables into the new attach_ctx
structure to improve readability and ease maintanence.
No functional change is expected.
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Commit e44193d39e8d ("cpuset: let hotplug propagation work wait for
task attaching") was introduced to let hotplug operation to wait
until the completion of task attach operation. However, it is still
possible that the states of the source or destination cpuset can
be changed between the cpuset_can_attach() call and the subsequent
cpuset_attach()/cpuset_cancel_attach() call.
As a result, data gathered during cpuset_can_attach() cannot be reliably
used in the subsequent cpuset_attach()/cpuset_cancel_attach()
call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and
cpuset_attach()/cpuset_cancel_attach() by making cpuset_write_resmask()
and cpuset_partition_write() wait for the completion of task attach
as well.
Ideally, an ongoing task attach operation should block any cpuset write
operation that can change its internal state until the operation is
completed. However, the attach_in_progress flag is currently per cpuset
and only the destination cpuset will have this flag set. The flag is not
set in the source cpuset where the tasks will be moved from. Even if we
extend the scope to include the source cpuset, it will not block cpuset
operation that changes the state of one of its ancestor cpuset which may
indirectly impact the state of the source or destination cpuset. It may
be too costly to set the flag for the whole subtree, it is far easier
to just make the flag global and block all the cpuset write operation
whenever a task attach operation is in progress.
Make that change by creating a new cpuset attach context (attach_ctx)
structure to hold the global in_progress flag and use it for blocking
cpuset write operation if a cpuset attach operation is in progress. Also
add a new wait_attach_done_lock() helper to do the waiting for an
ongoing attach operation and acquire the cpuset_mutex.
The comments about validate_change() are no longer valid as it won't
be called at all if an attach operation is in progress. So the comments
can be removed.
The per-cpuset attach_in_progress flag is also currently used in
partition_is_populated() and cpuset_is_populated() to determine if
an empty cpuset will have incoming task. This check will no longer be
needed as this function will not be called when there is a task attach
in progress. So the flag check is now removed.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Whenever memory node mask is changed, there are 4 places where the node
mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to
the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and
cpuset_attach(). However there are inconsistencies in what node masks
are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is
hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or
an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in
node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE].
However, node in node_states[N_ONLINE] may not have memory. So
node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should mostly be useful for v1
where mems_allowed is the same as effective_mems. With v2, the memory
nodes in effective_mems should be a subset of node_states[N_MEMORY]
except when a memory hot-unplug operation is in progress and a memory
node is removed from node_states[N_MEMORY] but not yet reflected in
the effective_mems's as cpuset_handle_hotplug() has not been called
from cpuset_track_online_nodes().
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems most of the time. For
v1, mpol_rebind_mm() uses mems_allowed which may differ from what
guarantee_online_mems() returns, but it conforms to what the cpuset v1
documentation says with respect to setting memory policy.
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The nr_deadline_tasks variable in the cpuset structure was introduced by
commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
in cpusets"). It is reported by sashiko [1] that nr_deadline_tasks
can currently be modified by inc_dl_tasks_cs() under rq->lock and
by cpuset_attach() under cpuset_mutex. So if both updates happen
simultaneously, the nr_deadline_tasks variable can be corrupted leading
to incorrect operations down the road.
Fix that by changing its type to atomic_t so that nr_deadline_tasks
are always atomically updated. This fix patch is a low hanging fruit.
It can handle some of the races between a concurrent sched_setscheduler()
and cpuset_can_attach()/cpuset_attach() calls, but not all of them like
the other issue raised by sashiko [2]. This will be handled hopefully
in a future follow up patch.
[1] https://sashiko.dev/#/patchset/20260626181923.133658-1-longman%40redhat.com
[2] https://sashiko.dev/#/patchset/20260630033344.352702-1-longman%40redhat.com
Fixes: 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task in cpusets")
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Creating a child cpuset where cpuset.mems is never set leads to a div/0
when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a
CPU hotplug event.
Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the
call to __nodes_fold()
The cpuset code passes (cs->mems_allowed) which is not guaranteed to have
nodes to the rebind routine. Use cs->effective_mems instead, which is
guaranteed to have a non-empty nodemask once we reach that code path.
Link: https://lore.kernel.org/all/CA+0ovCiEz6SP_sn3kN4Tb+_oC=eHMXy_Ffj=usV3wREdQrUtww@mail.gmail.com/
Fixes: ae1c802382f7 ("cpuset: apply cs->effective_{cpus,mems}")
Closes: https://lore.kernel.org/linux-mm/CA+0ovCgxbZkXa+OU8w3s84R3KNPNxxRfmsNR-udh+afQBbGNmw@mail.gmail.com/
Suggested-by: Gregory Price <gourry@gourry.net>
Suggested-by: Waiman Long <longman@redhat.com>
Acked-by: Waiman Long <longman@redhat.com>
Signed-off-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Ridong Chen <ridong.chen@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: "Michal Koutný" <mkoutny@suse.com>
Cc: <stable@vger.kernel.org>
[ david: add a comment, slightly rephrase description ]
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The automatic CPU-intensive work item detection reads the worker task's
se.sum_exec_runtime without a lock in wq_worker_running(),
wq_worker_tick() and process_one_work(). The scheduler updates that
field under the rq lock (from the tick via update_curr(), or cross-CPU
via task_sched_runtime()), raising:
BUG: KCSAN: data-race in wq_worker_running+0xa8/0xe8
race at unknown origin, with read to 0xffff0009a11d1df8 of 8 bytes by task 238535 on cpu 68:
wq_worker_running
schedule
schedule_preempt_disabled
__mutex_lock
mutex_lock_nested
cgroup_bpf_release
process_one_work
worker_thread
kthread
ret_from_fork
value changed: 0x0000000088482ba0 -> 0x00000000884893c0
The value only feeds a heuristic, so the race is benign-ish. Unlike
commit ecf5aad9a441 ("workqueue: annotate racy PWQ_STAT_CPU_TIME update
in wq_worker_tick()") that only needs data_race(), these are plain reads
whose result drives a subtraction and comparison, so use READ_ONCE() for
a single, non-torn load, which also silences KCSAN.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
In tracing_set_tracer(), the assignment 'ret = 0' following the
__tracing_resize_ring_buffer() error check is a dead store. After
this point, all subsequent code paths either return with a constant
value (-EINVAL, 0, -EBUSY) or reassign ret before reading it
(tracing_arm_snapshot_locked, tracer_init).
Remove the unnecessary assignment.
No functional change.
Link: https://patch.msgid.link/6a2a37c4.f0a9eb5a.2fc603.7724@mx.google.com
Signed-off-by: Wayen.Yan <win847@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
CONFIG_PM_WAKELOCKS_LIMIT is documented as the maximum number of
user-space wakeup sources, but the limit check is performed before
the counter is incremented and only rejects new wakeup sources when the
current number is greater than the limit. This allows one extra wakeup
source to be created.
Reject new wakeup sources once the counter has reached the limit.
Fixes: b86ff9820fd5 ("PM / Sleep: Add user space interface for manipulating wakeup sources, v3")
Signed-off-by: Haowen Tu <tuhaowen@uniontech.com>
[ rjw: Subject edits ]
Link: https://patch.msgid.link/20260624053839.2150567-1-tuhaowen@uniontech.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
|
|
This ensures that any RCU readers traversing the instance list
have finished, before releasing the reference on the tracer that
the instance points to.
Cc: stable@vger.kernel.org
Fixes: a6ed2aee54644 ("tracing: Switch to kvfree_rcu() API")
Link: https://patch.msgid.link/20260609045430.1589786-1-crwood@redhat.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Crystal Wood <crwood@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
Andrea Righi
Daniel Borkmann
Mete Durlu
Ingo Molnar
Gregory Price
Nicholas Dudar
Eduard Zingerman
David Woodhouse
Thomas Weißschuh
Usama Arif
Yu Peng
Malaya Kumar Rout
Yuho Choi
Radu Rendec
Sechang Lim
Michael Bommarito
Yiyang Chen
Steven Rostedt
Huihui Huang
Yuanhe Shu
Ben Dooks
David Carlier
Yafang Shao
Dave Airlie
Waiman Long
Farhad Alemi
Breno Leitao
Wayen.Yan
Haowen Tu
Crystal Wood