wireguard-linux/kernel/sched/fair.c, branch davem/net

sched/cache: Skip kernel threads for cache aware scheduling to rubustify the code

2026-09-22T08:50:49Z

Kernel thread should not be covered by cache aware scheduling as it borrows the statistics from the user space thread. Filter the kernel thread in account_mm_sched(). In theory a kernel thread does not have any valid cache group, so !grp should gate the kernel thread. Add the PF_KTHREAD check explicitly here for safety reasons, to guard against future modifications and to pair with task_tick_cache(). Fixes: df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing") Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Cc: # 7.2.x Link: https://patch.msgid.link/058f0c6ea7b991c177a17de347fa3157f25489a7.1790035273.git.tim.c.chen@linux.intel.com

sched/cache: Introduce task_struct->sched_cache_grp to fix UAF

2026-09-22T08:50:43Z

Add a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct. Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). This fixes the use-after-free when account_mm_sched() reaches the group through a task whose mm is being switched, as reported by Hyunwoo: https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/ Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp. Keep the fork/exec/exit reference management out of the generic mm paths: add sched_cache_fork(), sched_cache_fork_cleanup(), sched_cache_exec_mmap() and sched_cache_exit_mm() in kernel/sched/cache_sched.c (with empty stubs for !CONFIG_SCHED_CACHE), so fs/exec.c, kernel/fork.c and kernel/exit.c each call one helper instead of open-coding the refcounting under #ifdef. Also add sched_cache_group_get() and task_cache_group_get(). Fixes: df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing") Closes: https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/ Closes: https://lore.kernel.org/all/343a7e07-7fad-4979-9c9b-82ec038c293c@linux.dev/ Reported-by: Hyunwoo Kim Reported-by: Zenghui Yu (Huawei) Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Cc: #7.2.x Link: https://patch.msgid.link/ae7081dc54736bf115215f9867abb2711a7403fb.1790035273.git.tim.c.chen@linux.intel.com

sched/cache: Decouple sched_cache_group from mm to fix UAF

2026-09-22T08:50:35Z

Currently the sched cache grouping is by mm and the scheduling statistics sched_cache_stat lives in the mm structure. This ties the life cycle of scheduling stats with mm. In account_mm_sched(), the scheduling stats are accessed by task->mm->sc_stat. However, a task may be switching mm on one CPU when another CPU is running account_mm_sched(), and possibly accessing the old mm that was freed. This problem was found when running tests with KASAN by Hyunwoo: https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/ Instead of serializing the mm access by introducing extra acquisition of rq lock in the mm free path, extract sched_cache_stat from mm_struct, rename it as sched_cache_group and manage its life cycle apart from mm_struct with its own ref counting. This allows us in the next patch access sched_cache_group directly from task, and add a refcount on sched_cache_group when a task links to it. This prevents the use after free issue when accessing stale and released old mm and its sched cache stat a task switches to a new mm while account_mm_sched() is done elsewhere. The other benefit of this restructure is in the future, the grouping of tasks to a LLC would have the flexibility to be associated with a user defined grouping, or cgroup, cookie group, numa_group or others instead of just with a single mm address space. Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it. Fixes: df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing") Closes: https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/ Closes: https://lore.kernel.org/all/343a7e07-7fad-4979-9c9b-82ec038c293c@linux.dev/ Reported-by: Hyunwoo Kim Reported-by: Zenghui Yu (Huawei) Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Cc: #7.2.x Link: https://patch.msgid.link/91fd1e3266707c865bc9abecfb3e17bc676712df.1790035273.git.tim.c.chen@linux.intel.com

sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug

2026-09-22T08:50:29Z

Cache aware scheduling introduced the migrate_llc_task migration type to direct tasks toward their preferred LLC, but its semantics can be lost when passive load balance falls back to active load balance (ALB). This may allow ALB to select a candidate whose preferred LLC does not match the destination, moving it away from its preferred LLC. Example scenario: src_rq has two runnable tasks, p1 and p2. p1 prefers dst_rq (dst_llc), while p2 prefers src_rq (src_llc). In this case, migrate_llc_task is set because src_rq has at least one task, p1, that wants to migrate to dst_rq. In ALB, can_migrate_task() finds p2 and returns true for it, thus moving p2 out of its preferred LLC. Solution: The CPU stopper in ALB constructs a fresh lb_env that does not inherit migration_type from the passive load-balance pass. Two approaches are possible: (a) Add a new member to struct rq so ALB can inherit migrate_llc_task from the passive LB that triggered it. (b) Define a new flag LBF_ACTIVE_LB_LLC and select the stopper callback at kick time to preserve the migration semantics across the asynchronous boundary. We choose (b) because it avoids passing migration_type through the stopper, which would affect the meaning of migration_type for delayed-dequeue tasks. Fixes: e4c9a4cb244a ("sched/cache: Add migrate_llc_task migration type for cache-aware balancing") Suggested-by: Chen Yu Signed-off-by: Lu Wang Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Reviewed-by: Tim Chen Reviewed-by: Chen Yu Cc: # v7.2.x Link: https://patch.msgid.link/cb39f64a17fc2b76097264aaec74a2d6dfff4315.1790035273.git.tim.c.chen@linux.intel.com

sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix LLC mis-scheduling bug

2026-09-22T08:49:53Z

alb_break_llc() decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC: env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable But the two counters cover different sets. nr_pref_llc_running is updated in account_llc_enqueue()/account_llc_dequeue(), next to cfs_rq->nr_queued, so it follows queued tasks. h_nr_runnable is updated in set_delayed()/ clear_delayed() and drops delay-dequeued tasks. So under DELAY_DEQUEUE, a preferring task that goes to sleep stays counted in nr_pref_llc_running while h_nr_runnable falls. The equality then breaks, alb_break_llc() returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs, LBF_ACTIVE_LB skips the per-task test in can_migrate_task(). The runnable set is the one we want. Fix it on the counter side. A task should be counted in nr_pref_llc_running exactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once in task_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input: account_llc_enqueue(), account_llc_dequeue(), set_delayed() and clear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering. nr_llc_running and sd->llc_counts are not touched and stay on queued semantics. Fixes: 714059f79ff0 ("sched/cache: Handle moving single tasks to/from their preferred LLC") Closes: https://lore.kernel.org/lkml/20260827135000.735138-1-zhanxusheng@xiaomi.com/ Reported-by: Zhan Xusheng Suggested-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Reviewed-by: Kayra Cizmeci Cc: # v7.2.x Link: https://patch.msgid.link/06af61afedac32e6477f57feb4d658f6c411c3af.1790035273.git.tim.c.chen@linux.intel.com

sched: Account cgroup CPU time to the execution context

2026-09-10T08:22:52Z

Proxy execution separates the scheduling context from the execution context. Commit aa4f74dfd42b ("sched: Fix runtime accounting w/ split exec & sched contexts") made per-task and thread-group runtime accounting follow the task that actually executes, while cgroup CPU usage is charged to the donor. When the donor and execution task belong to different cgroups, this makes a task's execution time count against a different cgroup from the one the task belongs to. Cgroup CPU usage should follow the execution context, matching the per-task, thread-group, and cgroup user/system accounting. Keep scheduling state associated with the donor, but charge cgroup CPU usage to rq->curr. A reproducer with the donor and execution task in separate cgroups showed the execution task accumulating runtime while cgroup CPU usage was charged to the donor's cgroup. With this change, the execution task's cgroup accumulates the CPU usage instead. The same behavior was verified with an RT donor and with legacy cpuacct accounting. Fixes: aa4f74dfd42b ("sched: Fix runtime accounting w/ split exec & sched contexts") Suggested-by: Tejun Heo Signed-off-by: Hui Su Signed-off-by: Peter Zijlstra (Intel) Acked-by: Tejun Heo Acked-by: John Stultz Link: https://patch.msgid.link/20260904034707.268416-1-sh_def@163.com

sched/eevdf: Fix rb augmented with multi fields

2026-09-10T08:22:52Z

The eevdf rb tree maintains 3 augmented fields but only one is currently copied when balancing the tree. Add a more generic define that can be used when there are several augmented fields. In this case, we provide a function that takes care of copying all fields. Fixes: aef6987d8954 ("sched/eevdf: Propagate min_slice up the cgroup hierarchy") Signed-off-by: Vincent Guittot Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: K Prateek Nayak Tested-by: K Prateek Nayak Link: https://patch.msgid.link/20260909150522.858312-1-vincent.guittot@linaro.org

sched/eevdf: Fix augmented max_slice

2026-09-10T08:22:51Z

Similarly to se->min_slice, init se->max_slice with se->slice before enqueueing the entity so the augmented callback computes it correctly at parent level. Fixes: 6e3c0a4e1ad1 ("sched/fair: Fix lag clamp") Signed-off-by: Vincent Guittot Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: K Prateek Nayak Link: https://patch.msgid.link/20260907123855.1297976-1-vincent.guittot@linaro.org

sched/fair: Avoid creating misfits during cache-aware balancing

2026-09-02T07:17:50Z

Cache-aware load balancing biases tasks toward their preferred LLC. On asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may contain CPUs that are too small to run the task. Pulling the task there turns it into a misfit, trading a cache-locality gain for a capacity loss that's more detrimental to performance. Guard both cache-aware migration entry points against this: - can_migrate_llc_task(): forbid the LLC migration when the task fits its source CPU but would not fit the destination CPU. - alb_break_llc(): veto the active balance under the same condition so the runnable task is not pushed onto a CPU that cannot accommodate it. Both checks are gated with checks for hybrid processors, so symmetric systems are unaffected. Tasks that already do not fit their source CPU are left to the existing LLC policy, since the move cannot make their fitness worse (this also preserves misfit up-migration to bigger CPUs). Additionally, if there are misfit tasks found in the load balancing classification phase, prioritize misfit task migrations over LLC load aggregation on asymmetric systems. A better fitting CPU will boost performance more than better cache locality. Reviewed-by: Ricardo Neri Tested-by: Ricardo Neri Reviewed-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com

sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()

2026-09-02T07:17:50Z

distribute_cfs_runtime() refreshes the rq clock and accounts elapsed runtime with update_curr() before redistributing bandwidth, but gates this on cfs_rq->curr. Since commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") cfs_rq->curr is only maintained on the root cfs_rq, so for the cgroup cfs_rqs it walks, the check never fires and the refresh is dead code. Use cfs_rq->h_curr, the per-level current entity, restoring the intended behaviour: only refresh when something is actually running at the throttled level, i.e. within the deferred throttle window. Without this, runtime consumed by a still-running task of the throttled hierarchy is not docked before redistribution; unthrottle_cfs_rq() catches up unconditionally since commit 28ad5427682b ("sched/fair: Call update_curr() before unthrottling the hierarchy"), so this is not a correctness hole today, but the refresh the check was written for is gone. Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Wanwu Li Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Aaron Lu Tested-by: Aaron Lu Link: https://patch.msgid.link/20260831101141.391382-3-liwanwu@kylinos.cn