linux-rng/kernel/sched/ext/types.h, branch master

sched_ext: Convert sub-cap kfuncs to __arena cmask arguments

2026-08-12T19:55:33Z

The sub-cap kfuncs take their cmask arguments as __ign pointers. The values cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init() rebases them into the arena by hand. BPF now translates between BPF and kernel arena addresses for __arena arguments. Tag the cmask arguments __arena so the kfuncs receive kernel addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The optional denied_out keeps its NULL not-provided signal via __arena__nullable. The mandatory masks use plain __arena. scx_qmap's call sites drop the (void *)(long) casts since the BPF-side declarations type the cmask arguments __arena and take arena pointers directly. The arena argument address translation is currently implemented only on x86-64. Schedulers calling these kfuncs load only there for now. Signed-off-by: Tejun Heo

sched_ext: Eject the top rescue consumer on overload

2026-08-03T21:01:36Z

When rescue demand on a cpu persistently exceeds the configured bandwidth, tasks age on that cpu's rescue DSQ until the stall watchdog fires. The watchdog blames the waiting task's owner, but the misbehaving party is whoever floods the queue, not whoever happens to time out. Track each sched's recent rescue consumption per cpu as a decaying average. Once the oldest waiter on a cpu's rescue DSQ has been queued past a threshold derived from the rescue knobs (4s at the defaults), the rescue timer ejects the sub with the highest recent consumption on that cpu with SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and nothing is ejected - the generic stall watchdog eventually blames the waiter's owner instead. Ejections on a cpu are spaced one threshold apart so the freed bandwidth can drain the backlog before another sub is judged. The overload check only wins the race against the stall watchdog when the watchdog timeout clears the threshold, and a single in-budget wait must not cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit and on knobs whose funding period exceeds half the threshold. v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check wraps 2^31 ticks after the last ejection and suppresses ejections. (sashiko AI) - Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage average in the victim pick. Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Add bandwidth-limited rescue execution for stranded tasks

2026-08-03T21:01:29Z

A local DSQ insert lacking the needed caps is diverted to the reject DSQ and bounced back through ops.enqueue() so the scheduler can re-decide. That recovery assumes the scheduler has somewhere legal to send the task. When it doesn't, e.g. when the task's affinity is restricted to cids delegated away, the task starves until the stall watchdog ejects the scheduler. An exiting task is worse - it skips ops.enqueue() and the rejection becomes a self-requeuing cycle that burns the CPU until the watchdog fires. Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the insert would be rejected for missing caps, the kernel takes over and runs the task on the target CPU without consulting the owning scheduler. The kernel sets the flag itself when enqueueing an exiting task. Rescue is a last-resort forward-progress backstop with a persistent disadvantage, not a way around cap enforcement. A per-CPU token bucket accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at a time in arrival order. Each is granted a slice of the rescue_quantum_us (default 5ms) quantum divided across the waiters, waits at the tail of the local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival once the slice is served. The schedulers keep their normal control over an admitted rescuee and may preempt or reslice it. Service is measured on CPU time actually received, so neither shortens the rescue. Prolonged denial escalates - the remaining slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee preempts the current task. Escalation is paced by the same bucket, and delivered service converges on the configured bandwidth no matter how aggressively the schedulers dispatch. Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making SCX_ENQ_RESCUE inserts reject as usual. v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form schedulers setting them still load on older kernels. (Andrea) Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Bound per-task reenqueues and eject the owning scheduler

2026-07-26T21:12:19Z

Unlike local reenqueues, cap rejections have no repeat limit. A malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps on, cycling the task through reject and reenqueue. This was assumed safe because a task that never runs trips the stall watchdog. However, the reenqueue irq_work re-arms itself and outranks the timer vector, blocking everything else on the CPU including stall detection and recovery, until the NMI hardlockup detector fires. Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT, which needs generalizing to cover all reenqueues. It also has an attribution problem. Counted per-cpu on root, it tears down the whole hierarchy even when a sub-scheduler caused the repeated reenqueues. Generalize by bounding every reenqueue with one per-task counter. reenq_cnt is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path every reenqueue producer passes through, and cleared in clr_task_runnable() when the task is picked to run and in scx_disable_task() when it leaves the scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to be picked up during sched exit. The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting repeat reenqueues from all sources. v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another reenqueue, not on every reenqueue. v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't carry over to the next owner across sched class switches, scheduler replacement or sub-scheduler rehoming (Andrea Righi). - Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst (Andrea Righi). Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Add scx_cmask_ref for validated arena cmask access

2026-07-14T08:18:42Z

kfuncs taking struct scx_cmask * from BPF arena memory have two problems. The pointer can be any value the BPF prog hands in, and the header (@base, @nr_cids, @alloc_words) can be mutated by the prog concurrently with kernel access. Add scx_cmask_ref, a validated handle. _init() normalizes the input pointer into the arena's kern_vm range via scx_arena_to_kaddr() and snapshots the header, rejecting a range outside the machine or a nr_cids whose words exceed the declared @alloc_words. Downstream sizing uses the snapshot, not the live header. _shard() reads slices while _or() and _copy() write back, all bounded by the snapshot. No callers yet. struct scx_cmask's bits[] carried __counted_by(alloc_words), so UBSAN_BOUNDS and FORTIFY_SOURCE bound accesses to the array. That bound is read from @alloc_words at the access. For an arena cmask @alloc_words is BPF-writable. A prog that sets it larger than the real allocation makes the check pass on a genuine overrun, so the annotation catches nothing, and it only runs under those debug configs. Drop it - _init() validates @alloc_words explicitly, and kernel-owned cmasks set it themselves. v2: Validate @alloc_words in _init(), drop __counted_by. (Andrea, sashiko AI) Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Add CID sharding

2026-07-14T08:18:42Z

Sub-sched operations need a scalable locking / work domain smaller than the whole cid space. Carve the cid space into topology-respecting shards: each shard is a contiguous cid range that stays within one LLC, and LLCs larger than the per-shard cap (default 24 cids, configurable via ops.cid_shard_size) split into enough shards to fit. A hard cap of SCX_CID_SHARD_MAX_CPUS prevents pathological sizes under custom configurations. No-topo cids pack into their own shards so every cid has a shard assignment. Also build scx_cid_shard_ranges[] for O(1) shard-to-cid-range lookup and scx_shard_node[] so callers can size or place work by NUMA without walking cids. Auto-built shards inherit their LLC's node. No-topo shards carry NUMA_NO_NODE. Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Make kernel/sched/ext/ sources self-contained for clangd

2026-06-22T20:41:26Z

The sources under kernel/sched/ext/ build as a single translation unit: build_policy.c includes the source files and headers. An LSP/clangd editor parses each as a standalone unit, sees no types, and reports a flood of errors. Give each header its dependencies and include guard, and have each source include the headers it uses. ext.c, arena.c and the ext headers now parse clean standalone. idle.c and cid.c still reference a few macros and helpers defined in ext.c. The next patch moves those to shared headers. Suggested-by: Peter Zijlstra Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi

sched_ext: Move sources under kernel/sched/ext/

2026-06-22T15:32:56Z

The sched_ext sources had grown to ten ext* files directly under kernel/sched/. Move them into a new kernel/sched/ext/ subdirectory and drop the now-redundant ext_ prefix. ext.c/h keep their names. kernel/sched/ext.{c,h} -> kernel/sched/ext/ext.{c,h} kernel/sched/ext_internal.h -> kernel/sched/ext/internal.h kernel/sched/ext_types.h -> kernel/sched/ext/types.h kernel/sched/ext_idle.{c,h} -> kernel/sched/ext/idle.{c,h} kernel/sched/ext_cid.{c,h} -> kernel/sched/ext/cid.{c,h} kernel/sched/ext_arena.{c,h} -> kernel/sched/ext/arena.{c,h} The include paths in build_policy.c and sched.h, the MAINTAINERS glob, and a few documentation and comment references are updated to match. No code or symbol changes. Suggested-by: Linus Torvalds Reviewed-by: Andrea Righi Signed-off-by: Tejun Heo