wireguard-linux/kernel/sched/ext/internal.h, branch stable

sched_ext: Pass the initial cmask to cid-form ops.enable()

2026-09-19T14:08:58Z

The cid-form API has an obvious hole. A task's cid mask is only visible through ops.set_cmask(), which fires on affinity changes and class switches but not when a task enters a scheduler through fork, sub-sched enable or re-home, and there is no p->cpus_ptr equivalent to fall back on. Schedulers work around it by seeding the mask in ops.init_task() from p->cpus_ptr cid by cid, which is subtly wrong: on sub-sched enable and re-home, an affinity change between init_task() and enable() is delivered to the sched the task is still on, and nothing corrects the new sched's copy afterwards. Fix it by adding struct scx_enable_args to cid-form ops.enable() carrying the task's cmask, built in the per-cpu scratch under the rq lock as the task enters the scheduler, and calling set_cmask() with the same mask right after enable(), ahead of set_weight(). A scheduler can then track affinity in set_cmask() alone, and scx_qmap drops its init_task() seed. set_cmask() no longer fires for a cid-form task before it is enabled, and the class-switch republish in switching_to_scx() is limited to the cpu form. This changes the cid-form ops.enable() signature, which is fine as the cid-form API is still considered unreleased. An args struct rather than a bare cmask argument leaves room for more initial state without another signature change, and the cmask travels as a plain arena address because BTF can't type arena struct members yet. v2: The cmask travels as a u64 arena address, cmask_arena_addr, instead of a kernel-typed pointer, with the typing limitation and the planned typed alias documented (Sashiko review). v3: The initial set_cmask() is delivered before set_weight() so that the mask is in place when weight-dependent state is derived (Andrea Righi). Selftest added. Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Don't run ops.dequeue() with a DSQ lock held

2026-09-17T17:20:04Z

ops.dequeue() is invoked with the source user DSQ's lock still held on the consume and move paths (scx_consume_dispatch_q(), move_task_between_dsqs()). A BPF scheduler which locks the source user DSQ from ops.dequeue() - e.g. by iterating it with bpf_iter_scx_dsq - self-deadlocks. ops.dequeue() can only call the "any" kfuncs and none of them can lock a builtin DSQ, so the global and bypass paths can't deadlock; however, all DSQ locks share one lockdep class, so iterating any user DSQ from ops.dequeue() on those paths trips the recursion check. Move the invocation after the DSQ unlock on all three paths. SCX_TASK_IN_CUSTODY is cleared under the lock serializing the transfer so that the callback is invoked exactly once. Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") Cc: stable@vger.kernel.org # v7.1+ Acked-by: Andrea Righi Signed-off-by: fangqiurong Signed-off-by: Tejun Heo

sched_ext: Wait for SCX_OPSS_DISPATCHING before reenqueueing a task

2026-09-15T22:34:44Z

ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") moved the final ops_state store in scx_dispatch_enqueue() after the DSQ unlock so that the custody update and ops.dequeue() precede it. A task can thus be found on a DSQ while still SCX_OPSS_DISPATCHING. The dequeue and core-sched pick paths wait for the state to clear in ops_dequeue() but the reenqueue paths don't. A reenqueue in that window runs ops.enqueue() and sets SCX_OPSS_QUEUED before the dispatch has completed. The dispatcher's final store then overwrites it with SCX_OPSS_NONE and finish_dispatch() drops every later dispatch of the task. Wait for SCX_OPSS_DISPATCHING to clear before dequeueing a task for reenqueue, the same way ops_dequeue() does. Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") Cc: stable@vger.kernel.org # v7.1+ Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Maintain an online cid mask in the scheduler arena

2026-09-15T16:57:20Z

Schedulers on the default cid mapping treat [0, nr_online_cids) as the online set and restart on hotplug. Schedulers that install their own mapping with scx_bpf_cid_override() have no way to learn which cids are online: the count no longer identifies members and the CPU-form cpumask is unusable from cid programs. This is an obvious hole in the cid API. Add scx_bpf_online_cmask(), a kernel-maintained cmask in the scheduler's arena, allocated alongside the per-CPU scratch masks and populated after the cid mapping is finalized and before ops.init(), for child schedulers too. The pointer stays valid through ops.exit() with no reference to take. It is the arena offset as a void pointer, the same form struct_ops arena arguments arrive in. The verifier types the void return as a scalar for the program's arena cast. The mask follows the SCX hotplug notifications: seeded from cpu_active_mask and updated before ops.cid_online/offline() runs, so it lags cpu_online_mask only inside a hotplug transition. Updates walk the scheduler list under the lock that also serializes unlinking. Reads are live, not atomic snapshots. Root initialization excludes hotplug. v2: Reworded the getter kerneldoc (Andrea Righi). Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: scx_qmap: Restore unused idle claims from ops.dispatch()

2026-09-15T16:57:19Z

scx_qmap tracks idle cids itself. pick_direct_dispatch_cid() claims a cid by clearing its bit and the task is inserted into that cid's local DSQ, which kicks the CPU. When the task does not arrive, for example because the insert fell back to the global DSQ after an affinity change, the CPU wakes, finds nothing and picks idle again. That is not an idle transition, so ops.update_idle() is not called and the cid stays marked busy until an unrelated task runs on it. Restore the claim from ops.dispatch(). The kick guarantees a dispatch on the kicked CPU, and when it finds nothing to run with a NULL @prev, the CPU is going back to idle. Document the pattern in ops.update_idle(), which reports only actual transitions. Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi Cc: Andrea Righi

sched_ext: Pass the initial cpu.idle state in scx_cgroup_init_args

2026-08-31T21:51:36Z

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later. Add the sched_idle state to scx_cgroup_init_args and fill it in all four places that build the args: scx_tg_online() for cgroups onlined under the scheduler, scx_cgroup_init() for cgroups that already exist when the scheduler is loaded, and the sub-scheduler handover paths scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(). Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows sched_idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle(). The sub-scheduler paths are compile-tested only. Fixes: 347ed2d566da ("sched/ext: Implement cgroup_set_idle() callback") Signed-off-by: Tao Cui Reviewed-by: Andrea Righi Signed-off-by: Tejun Heo

sched_ext: Fix several comment issues

2026-08-31T16:48:16Z

Fix several comment issues found during review: __setschduler_prio() -> __setscheduler_class() scx_iter_scx_dsq_new() -> bpf_iter_scx_dsq_new() scx_next_task_scx() -> set_next_task_scx() Signed-off-by: Wanwu Li Signed-off-by: Tejun Heo

sched_ext: Allow ops.cgroup_set_bandwidth() to be sleepable

2026-08-18T17:59:38Z

ops.cgroup_set_bandwidth() is delivered from scx_group_set_bandwidth(), which runs from the cpu.max cgroup interface write path (tg_set_bandwidth()) in process context. scx_group_set_bandwidth() holds percpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep. The call site is therefore sleepable, like ops.cgroup_init(). bpf_scx_check_member() rejects a sleepable program on any member not on its allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when a cgroup gains a cpu.max limit at runtime; it must instead pre-reserve memory for a callback that cannot allocate. Add cgroup_set_bandwidth() to the allow-list so the callback can allocate on demand, and document that it may block. A scheduler must decide at load time whether to mark the callback sleepable, but the allow-list entry is a verifier property with no symbol to probe. Add a compatibility marker whose presence in the kernel's BTF lets userspace detect this support: DEFINE_SCX_COMPAT_MARKER() emits an empty, callerless function, here scx_compat_marker_cgroup_set_bandwidth_may_sleep(). It is __used __retain so neither the compiler nor the linker (under CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) drops it. The markers share the scx_compat_marker_ prefix and are collected near the end of ext.c so more can be added as further capabilities appear. Signed-off-by: Changwoo Min Signed-off-by: Tejun Heo

sched_ext: Merge branch 'for-7.3-arena-args' into for-7.3

2026-08-17T21:20:34Z

Pull to receive the __arena argument conversion: 67f1f4a48c24 ("sched_ext: Pass kernel arena pointers to ops_cid callbacks") a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments") a05c5b5cb5cf ("sched_ext: Convert scx_bpf_cid_override() to __arena array arguments") along with the bpf-next branch carrying the __arena argument support they depend on. Conflict in kernel/sched/ext/ext.c between: c384ab8a0b13 ("sched_ext: Move the config-off sub-cap kfunc stubs into sub.c") and: a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments") which updated the stubs in their old ext.c location. Resolved by keeping ext.c without the stubs and applying the prototype conversion to the relocated stubs in sub.c. Signed-off-by: Tejun Heo

sched_ext: Make core-sched task ordering hierarchy-aware

2026-08-16T02:07:49Z

With sub-schedulers, tasks of different schedulers routinely share rqs and SMT siblings, but scx_prio_less() consults ops.core_sched_before() only when both tasks belong to the same scheduler. Every pair spanning two schedulers falls back to the default ordering, so no scheduler can express ordering across a scheduler boundary, including a root over its sub-schedulers' tasks. Order a pair spanning schedulers by the nearest common ancestor that implements ops.core_sched_before(): both tasks are in its subtree, making this the one op where a scheduler is called on tasks it delegated to its sub-schedulers and may not be scheduling anymore. Same-scheduler pairs keep using the owning scheduler's op so a parent never orders inside a subtree it delegated. The op is skipped when the deciding scheduler is bypassing on either task's CPU. Update scx_qmap to fall back to the kernel's default ordering when handed a delegated task it has no task_ctx for. Signed-off-by: Tejun Heo