| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Cross-merge BPF and other fixes after downstream PR.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
Merge updates related to system sleep handling and runtime PM, a power
capping update, and power management tools updates for 7.3-rc1:
- Rename the PM core module parameter prefix to "pm" and allow the PM
transition (DPM) watchdog to be disabled by default (Tzung-Bi Shih)
- Fix off-by-one in wakelocks number limit check in the system sleep
sysfs interface (Haowen Tu)
- Remove kernel-doc markings from helper descriptions in the core
hibernation code (Adi Nata)
- Use %pe to print error pointer values in the hibernation core (Ronan
Marchal)
- Fix memory leak in snapshot_write_next() error path (Malaya Kumar
Rout)
- Delay allocating and linking the next swap_map_page in the
hibernation image saving code until another image page actually
needs to be recorded (Haesung Kim)
- Stop setting runtime_error on runtime resume callback failures to
allow drivers to recover from resume issues (Praveen Talari)
- Handle PMU registration failure during probe in the intel_rapl_tpmi
driver (Sumeet Pawnikar)
- Avoid optional imports in intel_pstate_tracer unless they are really
needed (Yousef Alhouseen)
- Add generic CPPC performance display to the cpupower utility, build
and call CPPC information on non-AMD processors, make cpupower print
kernel and hardware frequency information, and add libm to cpupower
for generic CPPC view (Jeremy Linton)
- Remove conditional return with no effect from cpupower (Sang-Heon
Jeon)
* pm-sleep:
PM: sleep: Allow disabling DPM watchdog by default
PM: sleep: Rename module parameters prefix to "pm"
PM: hibernate: swap: defer linking the next map page
PM: hibernate: Fix memory leak in snapshot_write_next() error path
PM: hibernate: Use %pe to print error pointer values
PM: hibernate: Remove kernel-doc markings from helper descriptions
PM: sleep: Fix off-by-one in wakelocks number limit check
* pm-runtime:
PM: runtime: Only set runtime_error on suspend callback failures
* pm-powercap:
powercap: intel_rapl_tpmi: Handle PMU registration failure during probe
* pm-tools:
cpupower: remove conditional return with no effect
tools/power: intel_pstate_tracer: avoid optional imports for help
cpupower: Add libm to cpupower for generic CPPC view
cpupower: Print kernel and hardware frequency information
cpupower: Build and call CPPC information on non-AMD processors
cpupower: Add generic CPPC performance display
|
|
Merge cpufreq updates for 7.3-rc1:
- Minor fixes and cleanups in assorted cpufreq drivers (Dan Carpenter,
Guru Das Srinagesh, Haoxiang Li, Karl Mehltretter, Sasha Finkelstein,
and Pan Chuang)
- Fix cpufreq table creation and bios_limits() callback in the Rust
bindings (Priya Bala Govindasamy)
- Add IPQ5210 support to qcom-nvmem driver (Varadarajan Narayanan)
- Adjust the .adjust_perf() cpufreq driver callback to allow the
maximum performance value to be passed to drivers and update the
intel_pstate driver to use it (Rafael Wysocki)
- Set policy->cur to the actual requested frequency in the intel_pstate
driver when the performance policy is used (Rafael Wysocki)
- Simplify HWP handling on Broadwell processors in intel_pstate (Rafael
Wysocki)
- Fix setting minimum P-state at init time in intel_pstate (Rafael
Wysocki)
- Consolidate frequency values computation in intel_pstate and clean up
code in that driver (Rafael Wysocki)
- Add missing kernel-doc desciptions for structure and union members in
the amd-pstate driver (David Vernet)
- Handle missing policy in dynamic EPP callbacks in the amd-pstate
driver (EDAMAMEX)
- Introduce EXPORT_SYMBOL_FOR_PSTATE_UT() to export amd-pstate driver
symbols to the amd-pstate-ut subdriver (K Prateek Nayak)
- Add dynamic EPP as an "energy_performance_preference" mode in
amd-pstate, remove the "amd_dynamic_epp" kernel command line option
and the "dynamic_epp" sysfs attribute, and update the dynamic_epp
documentation accordingly (K Prateek Nayak)
- Add unit tests for CPPC Performance Priority and the "dynamic" EPP
mode in the amd-pstate driver (K Prateek Nayak)
- Set min_limit_freq based on bios_min_perf in amd-pstate and remove
the defensive check for bios_min_perf from it (K Prateek Nayak)
- Fix EPP return type and handle errors in amd-pstate during
initialization, toggle auto_sel in active mode on shared memory
systems, and cache the firmware programmed EPP value (Marco Scardovi)
- Skip tests in amd-pstate-ut if the amd-pstate driver is not in active
use (Qianheng Peng)
- Replace sprintf() with sysfs_emit() in sysfs show in the cpufreq
schedutil governor and fix a self-contradictory comment in
sugov_iowait_apply() (Zhongqiu Han)
- Fix the usage example for the sampling_rate tunable of the ondemand
cpufreq governor in admin-guide (wangxiaodong)
* pm-cpufreq: (40 commits)
cpufreq: imx6q: fix out-of-bounds write when probed more than once
cpufreq: imx6q: fix devres accumulation across driver rebind
rust: cpufreq: Fix temporary write in Registration::bios_limit_callback
rust: cpufreq: Add CPUFREQ_TABLE_END as last table entry in TableBuilder::to_table
cpufreq: intel_pstate: Adjust policy->cur in active mode to policy
cpufreq/amd-pstate: Document missing kernel-doc members
cpufreq/amd-pstate-ut: Add unit test for CPPC Performance Priority
cpufreq/amd-pstate-ut: Add unit test for "dynamic" EPP mode
cpufreq/amd-pstate: Reduce the scope of exported symbols
Documentation/amd-pstate: Update dynamic_epp documentation with new behavior
cpufreq/amd-pstate: Remove "amd_dynamic_epp" cmdline and "dynamic_epp" sysfs
cpufreq/amd-pstate: Add dynamic EPP as an "energy_performance_preference" mode
cpufreq/amd-pstate: Extract platform profile to EPP conversion into a helper
cpufreq/amd-pstate: Remove the defensive check for bios_min_perf
cpufreq/amd-pstate: Set min_limit_freq based on bios_min_perf
cpufreq: apple-soc: Calculate frequency as a 64-bit value
kselftest: cpufreq: Backup and restore governor for sptests
selftests/cpufreq: Remove unnecessary sudo from quick_shuffle()
selftests/cpufreq: Remove unused local variables from switch_show_governor()
cpufreq/amd-pstate: handle missing policy in dynamic EPP callbacks
...
|
|
Commit 9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.
The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.
CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.
Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-6-bb500bf4afd4@linux.intel.com
|
|
sched_balance_find_src_rq() avoids selecting a runqueue with a single
running task as busiest if doing so results in migrating the task to a
CPU with less than ~5% of extra capacity. It also unintentionally
prevents migrations between CPUs of identical capacity.
When CONFIG_SCHED_CLUSTER is enabled, load should be balanced across
clusters of CPUs with the same capacity. Allowing migration between CPUs
of identical capacity is necessary to meet this goal.
Use get_actual_cpu_capacity() to reflect architectural capacity as well
as diminished capacity due to hardware or cpufreq pressure. Guard this
check with the sched_cluster_active static key so that systems without
cluster topology are unaffected.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-5-bb500bf4afd4@linux.intel.com
|
|
In domains with asymmetric capacity, identifying misfit load in a
scheduling group is not useful when the destination CPU cannot help (i.e.,
its capacity exceeds the group's maximum CPU capacity by less than ~5%). In
such cases, it also prevents load balance among clusters of equal capacity
when CONFIG_SCHED_CLUSTER is enabled. This happens because
update_sd_pick_busiest() skips candidate groups of type misfit_task if the
destination CPU has similar capacity.
Skipping misfit load accounting in this situation allows the group to be
classified as has_spare or fully_busy and lets load balancing proceed. Keep
marking scheduling groups as overloaded when misfit tasks are present. The
sg_overloaded flag propagates to the root domain and allows bigger CPUs in
it to help via newly idle balance.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-4-bb500bf4afd4@linux.intel.com
|
|
update_sd_pick_busiest() may incorrectly select a fully_busy group as the
busiest group when its per-CPU capacity exceeds that of the destination
CPU. This happens because the type of busiest group is initialized to
group_has_spare and allows the fully_busy group to win the type comparison.
update_sd_pick_busiest() should not choose a candidate scheduling group
with at most one runnable task if its per-CPU capacity is greater than that
of the destination CPU. Such a check already exists, but it is done too
late: after the type comparison, preventing a subsequent fully_busy group
of equal per-CPU capacity from being correctly selected.
Move this check to occur before comparing group types.
Fixes: 0b0695f2b34a ("sched/fair: Rework load_balance()")
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Christian Loehle <christian.loehle@arm.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-3-bb500bf4afd4@linux.intel.com
|
|
The argument sg_overloaded of update_sg_lb_stats() is only consumed when
balancing at the root domain. It only makes sense to update it in such a
case. Commit 3229adbe7875 ("sched/fair: Do not compute overloaded status
unnecessarily during lb") updated the logic accordingly but missed the case
in which the root domain has the SD_ASYM_CPUCAPACITY flag. Fix this.
Fixes: 3229adbe7875 ("sched/fair: Do not compute overloaded status unnecessarily during lb")
Reported-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-2-bb500bf4afd4@linux.intel.com
|
|
When picking a busiest CPU with only one running task, the function
sched_balance_find_src_rq() skips candidate CPUs if the destination CPU has
less than ~5% extra capacity. This condition only holds if all the SMT
siblings of a CPU are idle.
SMT siblings share the computing resources of a physical core and this
results in reduced capacity if more than one sibling is busy.
Skipping a CPU as described would prevent the load balancer from pulling
tasks from a scheduling group previously and correctly identified as
group_smt_balance (i.e., one with more than one task running).
Do not skip a candidate CPU of similar capacity if it has busy SMT
siblings.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-1-bb500bf4afd4@linux.intel.com
|
|
find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.
On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.
However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.
Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.
Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260804151324.918020-1-arighi@nvidia.com
|
|
perf_group_detach() handles leader and sibling detach differently. When the
group leader is detached, all siblings are promoted to singleton events and
their group_leader pointer is reset to themselves. When a sibling is
detached, it is removed from the leader's sibling_list, but its
group_leader pointer is left pointing at the old leader.
That is harmless when the sibling is being closed and freed immediately, as
in the DETACH_DEAD path. It is not safe when the sibling is detached but
kept alive, such as during CPU hotplug with DETACH_GROUP. In that case the
sibling is removed from the context, while its file descriptor can still
keep it alive.
A typical failing sequence is:
- A group contains leader L and sibling S.
- CPU hot-unplug detaches S with DETACH_GROUP, removing it from
L->sibling_list but leaving S->group_leader == L.
- L is later closed and freed.
- A PERF_IOC_FLAG_GROUP ioctl on S follows S->group_leader and
dereferences the freed leader.
This was reproduced by running the perf event fuzzer, CPU hotplug, and a
stress workload concurrently:
Unable to handle kernel paging request at virtual address 006b6b6b6b6b6cdb
CPU: 2 PID: 12489 Comm: perf_fuzzer 6.18.7 PREEMPT
pc : perf_ioctl+0x34c/0xc68
x20: ffffff89a3fa2c70 x8 : 6b6b6b6b6b6b6b6b
Code: 943c4a0e 340047a0 f9404a94 f9411e88 (f940b908)
Call trace:
perf_ioctl+0x34c/0xc68 (P)
__arm64_sys_ioctl+0xa0/0xf4
invoke_syscall+0x58/0xe4
el0_svc_common+0xa8/0xdc
do_el0_svc+0x1c/0x28
el0_svc+0x40/0xc0
el0t_64_sync_handler+0x68/0xdc
el0t_64_sync+0x1c4/0x1c8
The fault happened in perf_ioctl(), where perf_event_for_each() follows
the stale group_leader pointer and perf_event_for_each_child() then
dereferences the freed leader's context.
Fix the use-after-free by promoting the detached sibling to a singleton.
Also fix __event_disable() cgroup accounting and event state change.
Fixes: 8a49542c0554 ("perf_events: Fix races in group composition")
Assisted-by: PatchWise:gpt-5.5
Signed-off-by: Aditya Chillara <aditya.chillara@oss.qualcomm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260807-fix-group-leader-uaf-v3-1-b0c2310c9a0d@oss.qualcomm.com
|
|
perf_event_remove_on_exec() sets remove-on-exec events to the EXIT state
and detaches their group relationships. The event's file descriptor can
remain open, however, and perf_event_open() currently accepts that event
as a group leader because its early validation rejects only REVOKED and
DEAD events.
A new sibling can consequently be linked to the detached leader. When
the leader is closed, perf_group_detach() observes that its
PERF_ATTACH_GROUP bit is already clear and skips the new sibling. The
sibling then retains a group_leader pointer to the freed event.
Reject group leaders in the EXIT state. Perform the check while holding
the shared context mutex so that an exec in the target task cannot detach
the leader between validation and group attachment.
[peterz: make the earlier test fully consistent]
Fixes: 037a3c43edfb ("perf/core: Detach event groups during remove_on_exec")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260806205655.75722-1-kylebot@openai.com
|
|
queued_spin_unlock() gates its contended_release trace call behind a
static branch, so a NOP sits on the unlock path even while the
tracepoint is disabled. Removing that requires replacing the unlock
implementation only while contended_release is enabled, which needs a
callback when the tracepoint is toggled.
Convert contended_release to TRACE_EVENT_FN() and add weak no-op
arch_contended_release_trace_reg()/arch_contended_release_trace_unreg()
hooks.
The default hooks are empty, so this is a no-op until an architecture
overrides them.
No functional change intended.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/1c2fcccfb584c075c02890c484f22c76a1948bf1.1785778551.git.d@ilvokhin.com
|
|
Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf
lock contention --lock-owner" cannot attribute a contended spinlock to
its holder. The waiter-side contention_begin event records that a
spinlock is contended, but not by whom. Firing contended_release in the
holder's context at unlock is the only way to capture the holder of a
contended spinlock.
Combine the contention check, trace call and release in an out-of-line
queued_spin_release_traced() so the compiler need not preserve the lock
pointer in a callee-saved register across the call.
The check in queued_spin_unlock() is paid on every unlock, even while
the tracepoint is disabled: a static-branch NOP on x86_64, and a few
more instructions to manage a stack frame elsewhere. Gate it behind
CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody
pays for a tracepoint they do not use. Sleeping locks fire
contended_release regardless.
On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g.
defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call
unlock and are wired up separately.
All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option
enabled.
_raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but
disabled. The unlock is the single 'movb'. The only instruction added to
the executed path is the 2-byte static-branch NOP. The CALL to the
traced helper and the JMP back are emitted out of line and are reached
only once the static branch is patched on:
endbr64 ; 4 bytes
xchg %ax,%ax ; 2 static-branch NOP
; (added)
movb $0x0,(%rdi) ; 3 unlock (single store)
A: decl %gs:__preempt_count ; 7
je B ; 2
jmp __x86_return_thunk ; 5
call queued_spin_release_traced ; 5 out of line, reached
; only when the
; tracepoint is on
jmp A ; 2 (added)
B: call __SCT__preempt_schedule ; 5
jmp __x86_return_thunk ; 5
Baseline is the same stream without the NOP and the out-of-line
CALL/JMP: 31 bytes vs 40 (+9 bytes).
Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all
standard configs out-of-line unlock. Architectures with inlined unlock
(s390 (always), csky and loongarch (both when !PREEMPTION)) will see a
bigger increase in binary size.
On the same path (x86_64, PARAVIRT_SPINLOCKS=n) with the tracepoint
disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no
measurable difference between baseline and patched kernels in
throughput, latency, cycles, instructions, IPC, or L1 instruction-cache
misses (kernel and total): all deltas stay within run-to-run noise.
Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on
the executed path in addition to static-branch NOP. Binary size impact
on arm64, defconfig: +932 bytes (+0.00%).
The _raw_spin_unlock()-heavy nginx workload reflects the larger hot
path: L1 instruction-cache misses rise ~1.4% (kernel and total) and
instruction count ~0.4%, consistent with the per-unlock frame.
cpu_cycles, throughput and latency show no measurable change and are
within run-to-run noise.
Architectures with fully custom qspinlock implementations (e.g.
PowerPC) are not covered by this change.
[1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com
|
|
A task performing a custom private hash resize can remain blocked in
uninterruptible sleep indefinitely. The hung-task detector reports:
INFO: task futex-resizer:314 blocked for more than 10 seconds.
task:futex-resizer state:D stack:14824 pid:314 tgid:312 ppid:311
Call Trace:
__schedule+0x521/0xf30
schedule+0x22/0xa0
futex_hash_allocate+0x3db/0x490
__do_sys_prctl+0x6f5/0xbd0
do_syscall_64+0xf9/0x530
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Kernel panic - not syncing: hung_task: blocked tasks
futex_pivot_pending() allows the resize request to continue when
either no replacement hash is pending (hash_new == NULL) or the current
hash reference count has reached zero.
After the final-reference wake, another futex task can complete the
pivot between the two observations:
T1 T2
futex_hash_allocate()
wait_var_event(mm, ...)
futex_pivot_pending(mm)
hash_new != NULL
futex_hash()
futex_ref_get(old) -> false
futex_pivot_hash(mm)
hash_new = NULL
__futex_pivot_hash(mm, new)
rcu_assign_pointer(hash, new)
fph = rcu_dereference(hash) /* new */
futex_ref_is_dead(fph) -> false
schedule()
The pivot changes the state from hash_new != NULL with a dead current
hash to hash_new == NULL with a live current hash. Because
futex_pivot_pending() reads hash_new and hash without serialization,
the resize task can observe hash_new in the pre-pivot state and hash in
the post-pivot state, causing futex_pivot_pending() to return false even
though the pivot has completed. The task then goes to sleep after the
wakeup has already been consumed.
Serialize state reads in futex_pivot_pending() using futex_mm_phash::lock.
This guarantees that futex_pivot_pending() observes hash_new and hash
atomically, eliminating the race condition.
Fixes: bd54df5ea7ca ("futex: Allow to resize the private local hash")
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260804125530.3933754-1-yaokai34@huawei.com
|
|
Pull BPF fixes from Daniel Borkmann:
- Fix BPF verifier to preserve full pointer state for commuted
scalar += pointer arithmetic (Yiyang Chen, Eduard Zingerman)
- Fix a use-after-free of request sockets in the BPF TCP iterator
batching (Jose Fernandez)
- Fix a use-after-free of sk_redir in the BPF sockmap send verdict
path (Chengfeng Ye)
- Fix a netns reference imbalance in the BPF conntrack kfuncs
(Chengfeng Ye)
- Fix bpf_get_fsverity_digest() dynptr assumptions and silent
digest truncation (Eric Biggers)
- Fix bpf_tcp_{gen,check}_syncookie to check sk_state before
sk_protocol to make sure it is a full socket (Luxiao Xu)
- Fix rqspinlock to reset the tail when preserving the queue
on deadlock (Kumar Kartikeya Dwivedi)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
rqspinlock: Reset tail when preserving queue on deadlock
bpf: Check sk_state before sk_protocol in bpf_tcp_*_syncookie
fsverity: Fix silent truncation in bpf_get_fsverity_digest()
fsverity: Fix bpf_get_fsverity_digest() dynptr assumptions
bpf: tcp: Fix use-after-free in bpf_iter_tcp_established_batch()
bpf: Fix netns reference imbalance in conntrack kfuncs
bpf, sockmap: Fix sk_redir use-after-free in send verdict
selftests/bpf: Cover commuted pointer state propagation
bpf: Propagate untrusted pointer state in commuted arithmetic
bpf: Preserve pointer state for commuted arithmetic
bpf: Simplify sanitize_err() signature
|
|
A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier,
unlike a regular BPF_LDX, so the JIT emits a plain load with no exception
table entry and a fault panics the kernel instead of being handled.
Reject the source pointer types that a BPF_LDX would have had that fault
protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns
into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED,
PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY |
PTR_UNTRUSTED.
This is reachable e.g. by loading ->mm out of a trusted task_struct
yields an untrusted pointer to mm_struct, and it is NULL for a kernel
thread:
[...]
SEC("tp_btf/sched_switch")
int BPF_PROG(demo, bool preempt, struct task_struct *prev,
struct task_struct *next)
{
struct mm_struct *mm = next->mm; /* untrusted */
out_ldx = (__u64)mm->pgd; /* BPF_LDX */
out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */
return 0;
}
[...]
Both dereference the same pointer, but only the BPF_LDX is protected
(x86-64 JIT, jump targets shown prog-relative):
[...]
; out_ldx = (__u64)mm->pgd;
17: movq $-10485760, %r10
1e: movq %rsi, %r11
21: addq $184, %r11
28: subq %r10, %r11
2b: movabsq $140737498841088, %r10
35: cmpq %r10, %r11
38: ja 0x3e <-- kernel addr?
3a: xorl %edi, %edi <-- no: dst = 0, skip the load
3c: jmp 0x45
3e: movq 184(%rsi), %rdi <-- yes: load + extable entry
[...]
; load_acquire(&mm->pgd)
53: movq %rsi, %rdi
56: movq 184(%rdi), %rax <-- no check, no extable entry
[...]
Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as
bpf_insn_prepare_dump() rewrites it back to BPF_MEM.
A PTR_TRUSTED pointer is deliberately not on the list. Such a load is
not converted either, but it does not need to be, since the pointer is
guaranteed live, so load-acquire from it stays allowed.
The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release
error messages are unchanged; writes (RMW / store-release) to such
pointers are already rejected elsewhere, so only load-acquire needs this.
Fixes: 880442305a39 ("bpf: Introduce load-acquire and store-release instructions")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
These comments describing the page allocator are out of date:
- __alloc_pages() is no longer a public API and has no business being
described outside of mm/.
- The `wait` variable is gone.
It may be out of date for other reasons too but this patch is just
fixing the issues that stood out.
To fix it:
- Instead of referring to a specific function, instead to "the page
allocator"
- Completely drop out-of-date details of that function's internal
behaviour, since they were irrelevant anyway.
Link: https://lore.kernel.org/20260715-spin-trylock-followup-v3-2-fc4d246f705d@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Suggested-by: Zi Yan <ziy@nvidia.com>
Link: https://lore.kernel.org/all/DJP11T5V7BDW.2FZZZ8R6LOY4I@nvidia.com/
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Currently, the destruction of the waiter queue is suppressed for
rqspinlock in cases where a deadlock is detected. Deadlock checks happen
relatively frequently (on entry for AA, within 1ms for ABBA), and waiter
threads may not be involved in locking scenarios involving deadlocks.
Thus, it is useful to not flush the queue and let other waiters take a
stab at acquiring the lock after we detect a deadlock and exit.
However, we need to follow the same logic as what we did previously for
the waitq_timeout label: reset the tail, and if we cannot, signal the
next waiter appropriately. In case of deadlocks, this signal would just
mark the MCS node as unlocked, and in case of timeouts, it would signal
RES_TIMEOUT_VAL. The difference thus is in the value propagated, which
decides whether the queue remains active or gets flushed.
Not doing the tail reset, and waiting for the next waiter can lead to
cases where we are the final waiter, and thus no next waiter arrives,
leading to intermittent stalls in this path. Once the next waiter does
join, we will be unblocked. In the theoretical case when the next waiter
never joins, we risk stalling indefinitely.
This can only happen for ABBA deadlocks, since entry into the wait queue
is guarded with AA checks. A precise sequence of executions leading up
to this scenario can be:
CPU 0 holds lock A.
CPU 1 holds lock B.
CPU 2 attempts lock B, becomes the pending waiter for B.
CPU 0 attempts lock B. B has locked+pending bits set, thus CPU 0 queues.
CPU 1 attempts lock A.
CPU 0 detects an ABBA deadlock.
Once deadlock detection happens for CPU 0, it will sit waiting for the
next waiter in the queue to populate node->next, which will experience
delays until such a waiter arrives.
Fix this by adjusting the logic for the check for deadlocks preceding
the waitq_timeout label. It would make sense to consolidate code for
both cases and use 'ret' to distinguish the value being propagated, but
that is left as an exercise for a future refactoring task to avoid diff
noise in this patch.
Fixes: 7bd6e5ce5be6 ("rqspinlock: Disable queue destruction for deadlocks")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260802021759.1139457-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Add an optional reclaim callback to struct dmem_cgroup_region. When
dmem.max is set below the current usage of a cgroup pool, the new limit
is applied immediately (so that concurrent allocations are throttled
while reclaim is in progress) and then the driver is asked to evict
memory to bring usage back below the limit.
Reclaim is attempted up to a bounded number of times. No error is
returned to userspace if usage remains above the limit after reclaim,
and a pending signal will abort the reclaim loop early. This matches
the behavior of memory.max in the memory cgroup controller.
Also honor O_NONBLOCK so that if that flag is set during the
max value write, no reclaim is initiated. The idea is to avoid
charging the reclaim cost to the writer of the max value.
v2:
- Write max before reclaim is attempted (Maarten)
- Let signals abort the reclaim without error (Maarten)
- If a new max value is written with the O_NONBLOCK flag,
reclaim is not attempted (Maarten)
- Extract region from the pool parameter rather than
passing it explicitly to set_resource_xxx().
v3:
- Use an rw_semaphore (unregister_sem) to protect reclaim callbacks
against concurrent region unregistration: readers (reclaim) hold the
read side; dmem_cgroup_unregister_region() takes the write side to
drain in-flight callbacks before returning. (Sashiko-bot)
v5:
- Rebased on the introduction of struct dmem_cgroup_init.
- Use nonblock=true in reset_all_resource_limits() to avoid sleeping
inside rcu_read_lock() in dmemcs_offline(). (Sashiko-bot)
- Compare usage against the truncated limit value stored in cnt.max,
not the original u64. (Sashiko-bot)
- Use a DMEM_MAX_RECLAIM_RETRIES (16) retry budget instead of 5, matching
the memcg controller's MAX_RECLAIM_RETRIES. Only -ENOSPC (no progress)
counts against the retry budget; other errors terminate the loop
immediately.
v6:
- Fix dmem_cgroup_ops->reclaim docstring: -ENOSPC does not stop reclaim
immediately but is retried up to DMEM_MAX_RECLAIM_RETRIES times; only
other negative errors terminate the loop. (Sashiko-bot)
v7:
- Replace the per-region rw_semaphore with a static SRCU domain
(dmemcg_srcu). SRCU is a better fit than rwsem for this use: it
avoids the per-region lock overhead on every reclaim call, and
synchronize_srcu() at unregister time is a rare operation. (Maarten)
- Trim in-function comments to focus on what rather than how.
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Link: https://patch.msgid.link/20260725100036.2372-4-thomas.hellstrom@linux.intel.com
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
|
|
Replace the bare u64 size argument to dmem_cgroup_register_region() and
drmm_cgroup_register_region() with a const struct dmem_cgroup_init *
pointer. The struct currently carries only the size field, but using a
struct makes the API extensible: future callers can supply additional
initialization parameters without adding more positional arguments.
Update all in-tree callers (amdgpu, xe) to use a compound-literal
initializer.
v5:
- Commit introduced.
Assisted-by: GitHub_Copilot:claude-sonnet-4.6
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Link: https://patch.msgid.link/20260725100036.2372-3-thomas.hellstrom@linux.intel.com
Acked-by: Dave Airlie <airlied@redhat.com>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
|
|
Cross-merge networking fixes after downstream PR (net-7.2-rc7).
No conflicts, or adjacent changes.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
param_set_charp() stores charp parameters in allocated memory after slab is
available, and releases the previous value when the parameter is updated.
The previous value is released before the replacement allocation succeeds.
If kmalloc_parameter() fails, the setter returns -ENOMEM with the parameter
left as NULL.
Failing zswap's compressor update before zswap is initialized can later
trigger:
BUG: kernel NULL pointer dereference, address: 0000000000000000
RIP: 0010:strcmp+0x10/0x30
Call Trace:
zswap_setup+0x3b1/0x490
zswap_enabled_param_set+0x5b/0xa0
param_attr_store+0x93/0xe0
module_attr_store+0x1c/0x30
kernfs_fop_write_iter+0x116/0x1f0
Allocate and copy the replacement first, then replace the parameter value
only after allocation succeeds.
Fixes: e180a6b7759a ("param: fix charp parameters set via sysfs")
Cc: stable@vger.kernel.org
Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
In elf_validity_cache_sechdrs, section sizes and offsets are validated,
unless the section type is SHT_NULL or SHT_NOBITS.
Later, elf_validity_cache_secstrings and elf_validity_cache_index_str
access the section name table (.shstrtab) and symbol string table
(.strtab) headers without first ensuring that their types are
SHT_STRTAB. If a section type is SHT_NULL or SHT_NOBITS, sh_offset has
not been validated and may reference out-of-bounds memory when
dereferenced in elf_validity_cache_secstrings or
elf_validity_cache_strtab.
Validate that both string section headers are of type SHT_STRTAB before
caching them.
Cc: stable@vger.kernel.org
Signed-off-by: Thiébaud Weksteen <tweek@google.com>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Disabling preemption or local IRQs keeps the current CPU in an RCU
read-side critical section, but in_rcu_cs() does not account for either
state. The verifier therefore rejects safe kptr accesses and invalidates
pointers when another RCU source ends.
Include preemption-disabled and IRQ-disabled state in in_rcu_cs().
Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ
restore only after the final protection ends.
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260805233940.3966981-2-dingning04@gmail.com
[ kkd: Simplify was_in_rcu_cs on spin unlock and adjust the selftest. ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The kernel/module/dups.c file relies on the following definitions and
associated functions:
* module_param() -> linux/moduleparam.h,
* DEFINE_MUTEX() -> linux/mutex.h,
* LIST_HEAD(), list_for_each_entry(), ... -> linux/list.h,
* refcount_t, refcount_inc(), ... -> linux/refcount.h,
* MODULE_NAME_LEN -> linux/module.h,
* completion, complete_all(), ... -> linux/completion.h,
* delayed_work, work_struct, ... -> linux/workqueue.h,
* lockdep_assert_held() -> linux/lockdep.h,
* strcmp(), memcpy() -> linux/string.h,
* container_of() -> linux/container_of.h,
* DEFINE_FREE(), __free(), scoped_guard() -> linux/cleanup.h,
* kzalloc_obj(), kfree() -> linux/slab.h,
* pr_debug(), pr_warn() -> linux/printk.h,
* WARN() -> linux/bug.h,
* TASK_KILLABLE -> linux/sched.h,
* HZ -> linux/param.h.
Update the file's include list accordingly.
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Use strcmp() instead of strlen()+memcmp() to compare module names in
kmod_dup_request_lookup(), since all strings are NUL-terminated.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Use scope-based cleanup helpers for kmod_dup_mutex and kmod_req to shorten
the code and to clarify where the lock is taken in
kmod_dup_request_exists_wait().
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
The kmod dups code preallocates kmod_dup_req before taking kmod_dup_mutex
to avoid allocating memory while holding the lock. This provides little
benefit, since the allocation is fast and can safely be done under the
lock. On the other hand, it leads to unnecessary allocations when the
request turns out to be a duplicate and slightly complicates the code.
Allocate kmod_dup_req only when needed and introduce a helper function
alloc_kmod_req() to initialize the structure.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
The kmod dups code uses RCU to ensure that a kmod_dup_req instance is freed
only after it is no longer referenced. When releasing an instance, the
kmod_dup_request_delete() function removes the kmod_dup_req from the
dup_kmod_reqs list, waits via synchronize_rcu() and finally frees it.
However, this doesn't work correctly because parallel users referencing the
instance in kmod_dup_request_exists_wait() don't enter an RCU read-side
critical section. This can result in a use-after-free.
The kmod_dup_request_exists_wait() function may need to hold a valid
reference to a kmod_dup_req instance across a blocking wait until the
corresponding modprobe command completes. This makes it unsuitable for RCU.
Fix the issue by changing the lifecycle management of kmod_dup_req to use
reference counting.
Fixes: 8660484ed1cf ("module: add debugging auto-load duplicate module support")
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
When kmod_dup_request_announce() announces the completion of
a request_module() call to duplicate waiters, it queues a work item to
invoke kmod_dup_request_complete(), and only that function calls
complete_all().
This adds an arbitrary delay that is unnecessary and provides little
benefit. Call complete_all() directly from kmod_dup_request_announce()
instead.
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Historically, various parameter-handling code kept pointers into
module::args, most notably the charp support. However, in 2009,
commit e180a6b7759a ("param: fix charp parameters set via sysfs") changed
charp parameters to kstrdup() the input string as well. As a result,
module::args now mostly wastes memory.
The last users that still pointed into module::args have now been cleaned
up, so remove this data.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
module_total_size() returns unsigned int but uses a signed int
accumulator. While the result is numerically correct, the type
mismatch is misleading.
Change the accumulator to unsigned int to match the return type.
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
[ppavlu: correct the commit title]
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Both try_add_failed_module() and kmod_dup_request_exists_wait() use
memcpy() with strlen() to copy module names into fixed-size
char[MODULE_NAME_LEN] buffers. Neither performs a bounds check on the
copy. Current callers always pass names originating from
mod->name (itself char[MODULE_NAME_LEN]), so this is not exploitable
today. However both functions accept a plain const char * with no
documented length contract, making them latent buffer overflows if a
future caller passes a longer string.
Replace memcpy() with strscpy() in both sites, which bounds the copy
to MODULE_NAME_LEN and always NUL-terminates.
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
The comment wrongly references to /sys/modules/XYZ/parameters/ directory
instead of /sys/module/XYZ/parameters/. Fix it.
Signed-off-by: Zenghui Yu <zenghui.yu@linux.dev>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
The symbol lookup code assumes the queried address resides in either
MOD_TEXT or MOD_INIT_TEXT. This breaks for addresses in other module
memory regions (e.g. rodata or data), resulting in incorrect upper
bounds and wrong symbol size.
Select the module memory region the address belongs to instead of
hardcoding text sections. Also initialize the lower bound to the start
of that region, as searching from address 0 is unnecessary.
Cc: stable@vger.kernel.org
Signed-off-by: Stanislaw Gruszka <stf_xl@wp.pl>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
The verifier currently keeps a hard-coded list of kfuncs that may be
called while holding a bpf_spin_lock. With KF_SPINLOCK_SAFE available,
retaining this list creates two sources of truth and requires verifier
changes whenever another lock-safe kfunc is added.
Mark every kfunc currently accepted by kfunc_spin_allowed() with
KF_SPINLOCK_SAFE. This covers the graph, numeric iterator, resource
spin lock, arena, and stream kfuncs.
Remove the obsolete category checks and make kfunc_spin_allowed() rely
solely on the kfunc registration metadata. This preserves the behavior
of existing kfuncs while using the same mechanism for built-in and
module kfuncs.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-3-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Introduce the KF_SPINLOCK_SAFE kfunc metadata flag in BTF so kfuncs may
be explicitly marked as safe to call while holding bpf_spin_lock.
Allow kfuncs defined in kernel modules to be marked with KF_SPINLOCK_SAFE.
Example: BTF_ID_FLAGS(func, $kfunc_name, KF_SPINLOCK_SAFE)
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-2-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
DECLARE_EVENT_CLASS emitted __bpf_trace_btf_ids_<call> through
BTF_ID_LIST_GLOBAL, i.e. a global symbol named after the event class.
The class name is not unique across the kernel, so the symbol
multiply-defines whenever two translation units instantiate the same
class.
Switch to the file-local BTF_ID_LIST: the list is reached
only through the event_class_<call>.btf_ids pointer, initialised in the
same unit, so tracefs readers never reference the symbol by name and
resolve_btfids still fills the now-local .BTF_ids entries.
The handcrafted syscall classes are the one cross-unit consumer: give
them their own local BTF_ID_LIST rather than importing the generated
sys_{enter,exit} lists.
Link: https://patch.msgid.link/20260730-b4-fix_btf_tracefs-v2-1-6b66da8dc103@meta.com
Fixes: eadc0725ab8d3 ("tracing: Expose tracepoint BTF ids via tracefs")
Reported-by: Mark Brown <broonie@kernel.org>
Closes: https://lore.kernel.org/all/ff58b01c-3f5e-4d55-be82-609d2faaf12e@sirena.org.uk/
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
bloom_map_alloc() has two 32-bit-specific problems when the computed
bitmap reaches the U32_MAX fallback case.
First, BITS_TO_BYTES(U32_MAX) is evaluated with 32-bit arithmetic. The
addition performed by DIV_ROUND_UP wraps, so the map allocates only the
fixed-size bloom filter object while keeping bitset_mask == U32_MAX.
Subsequent updates can then write past the allocated object.
Second, fixing only the allocation size is not sufficient. The bloom hash
is a u32, but set_bit() takes a signed long bit number and x86 test_bit()
eventually feeds the index to variable_test_bit(long, ...). On 32-bit
kernels, hashes in [0x80000000, U32_MAX] therefore become negative bit
offsets. x86 bt/bts with a memory operand interpret those offsets relative
to the supplied base, so a map with bitset_mask == U32_MAX can read or
write before bloom->bitset even after allocating the full 512 MiB bitmap.
Keep the U32_MAX fallback, but split each hash into a word pointer and an
in-word bit number before calling test_bit() or set_bit(). The bitops
argument is then always in [0, BITS_PER_LONG - 1], while BIT_WORD(h) still
selects the intended word in the full bitmap.
Compute the bitset size from (u64)bitset_mask + 1 before passing the final
size to bpf_map_area_alloc(). This fixes the original under-allocation and
keeps the allocated storage consistent with the addressable bitset.
Exploitation note: local privilege escalation is possible on a 32-bit x86
kernel using the under-allocation bug from a binary with CAP_BPF.
Fixes: 9330986c0300 ("bpf: Add bloom filter map implementation")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260805060228.2703051-1-Jeremy.Jean@oss.cyber.gouv.fr
Assisted-by: Codex:gpt-5
|
|
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog
is allowed to attach to '__x64_'-alike prefix symbols.
It is because the verifier does not verify whether the symbol is a kernel
function or a bpf prog. That said, a sleepable tracing prog is allowed to
attach to a bpf prog target whose name has '__x64_'-alike prefix.
For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP
prog, and copies buffer from a user pointer with bpf_copy_from_user()
helper. After attaching the XDP prog to lo interface, the kernel BUG
could be triggered by 'ping -c 1 -W 1 127.0.0.1':
[ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324
Fix it by disallowing sleepable prog always when its target
btf is not a kernel's btf.
Fixes: 16d9c5660692 ("bpf: Always allow sleepable programs on syscalls")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/20260805150810.34907-2-leon.hwang@linux.dev
|
|
There's no need to modify the trace object bpf_get_stackid_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-13-jolsa@kernel.org
|
|
There's no need to modify the trace object bpf_get_stack_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-12-jolsa@kernel.org
|
|
get_perf_callchain() returns a per-CPU perf_callchain_entry buffer and
releases its recursion slot via put_callchain_entry() before returning,
so nothing keeps the entry reserved while __bpf_get_stack() consumes
it below.
A preemptible BPF program (e.g. a non-sleepable raw tracepoint program
on a PREEMPT kernel, which runs under migrate_disable() but not
preempt_disable()) can be scheduled out between obtaining the entry
and the copy. Another task scheduled on the same CPU then reuses the
same per-CPU buffer and overwrites trace->nr with a larger value.
copy_len is then computed from the inflated trace->nr and can exceed
the caller's buffer, causing an out-of-bounds write in the memcpy()
and in the build_id path.
The rcu_read_lock() taken here alone does not prevent this. It is
only taken on the may_fault path, and under CONFIG_PREEMPT_RCU it does
not disable preemption; it merely keeps perf's callchain buffer array
alive (freed via call_rcu()) and does nothing to stop another task
from reusing the entry.
Disable preemption around obtaining the callchain entry and copying
it into the caller's buffer, so the entry cannot be reused underneath
us and trace->nr stays bounded by max_depth. Build ID resolution may
fault and is therefore deferred until after preemption is re-enabled;
by then the instruction pointers have already been copied into buf,
so it operates only on that private copy. Note, preempt_disable() also
subsumes the buffer-lifetime guarantee the rcu_read_lock() provided,
since a preempt-disabled section is an RCU read-side critical section
for the callchain buffers' call_rcu() reclaim.
Fixes: c195651e565a ("bpf: add bpf_get_stack helper")
Reported-by: Tao Chen <chen.dylane@linux.dev>
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <borkmann@iogearbox.net>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260803210149.296496-11-jolsa@kernel.org
Closes: https://lore.kernel.org/bpf/20260206090653.1336687-1-chen.dylane@linux.dev/
[ changed Fixes: commit ]
|
|
Both bpf_get_task_stack and bpf_get_task_stack_sleepable helpers that
use __bpf_get_task_stack have buf defined as ARG_PTR_TO_UNINIT_MEM
argument and we should initialize the buf on every return path.
Adding missing buf memset for __bpf_get_task_stack fail paths. This
provides deterministic buffer contents, which is useful when the buffer
is used directly as a map key.
Fixes: 06ab134ce8ec ("bpf: Refcount task stack in bpf_get_task_stack")
Fixes: b992f01e6615 ("bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-10-jolsa@kernel.org
|
|
Now with the new callchain_* helper functions we can process trace_in
case directly in bpf_get_stack_pe function and remove it from
__bpf_get_stack which makes things easier for preemption fix in
following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-9-jolsa@kernel.org
|
|
The new callchain_finalize function calls the build-id retrieval
(if needed) and zeroes the buffer. This makes things easier for
preemption fix in following change.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-8-jolsa@kernel.org
|
|
The new callchain_store function stores trace entries buffer into
user supplied buffer. It covers both just-ip and buildid data.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-7-jolsa@kernel.org
|
|
The get_perf_callchain call needs disabled preemption plus we need
it disabled as long as we access its returned trace entries buffer.
Note the bpf_get_stackid_pe function is executed already with
preemption disabled.
Fixes: d5a3b1f69186 ("bpf: introduce BPF_MAP_TYPE_STACK_TRACE")
Reported-by: Tao Chen <chen.dylane@linux.dev>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260803210149.296496-6-jolsa@kernel.org
Closes: https://lore.kernel.org/bpf/20260206090653.1336687-2-chen.dylane@linux.dev/
|
|
Replacing __bpf_get_stackid calls with sequence of following functions:
stackid_fastpath
stackid_new_bucket
stackid_install
This makes code more structured and allows us to easily disable
preemption only in bpf_get_stackid in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-5-jolsa@kernel.org
|
Daniel Borkmann
Rafael J. Wysocki
Ricardo Neri
Andrea Righi
Aditya Chillara
Kyle Zeng
Dmitry Ilvokhin
Yao Kai
Linus Torvalds
Brendan Jackman
Kumar Kartikeya Dwivedi
Thomas Hellström
Jakub Kicinski
Jiacheng Yu
Thiébaud Weksteen
Ning Ding
Petr Pavlu
Naveen Kumar Chaudhary
Zenghui Yu
Stanislaw Gruszka
Kaitao Cheng
Mykyta Yatsenko
Jérémy Jean
Leon Hwang
Jiri Olsa
Daniel Borkmann