| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Pull to receive:
c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d2175 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e44a ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da9587118d ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205c1 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d2175 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d2175 and 3dd52416e44a update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da9587118d instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205c1 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The built-in idle masks are reset with all online CPUs marked idle
before sched_ext is enabled. Busy CPUs can therefore be incorrectly
advertised as idle until their next idle transition.
Initialize the masks empty so that the initial state is conservative.
When bypass is lifted, every CPU is rescheduled and idle-to-idle
re-picks populate the masks with CPUs that are actually idle. Later
idle transitions keep the masks up to date.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the
associated scheduler instances. Most accesses have since been converted to
resolve the sched from the program or task at hand. The remaining naked
sites divide into ones that semantically always want the root sched, which
this patch resolves, and one that is left to a later patch.
The resolved sites:
- The SCX_OPS_TID_TO_TASK validation and the ecaps sync kick already hold a
sched whose ancestors[] pins the root as entry 0 with plain pointers
stable for the sched's lifetime. Reach the root through the sched at hand.
- The dispatch entry, class switch, idle notification and fork init paths
only execute while the scheduler is live and scx_root never changes inside
the live window, so no update can race them. Add scx_root_protected_live()
which documents that invariant and resolves with a plain load.
- The hotplug path, including the ecaps reseeds, runs with the hotplug lock
held, which excludes the scx_root writers. Add scx_root_protected(), which
accepts either the hotplug lock or scx_enable_mutex.
- Is-root tests use a zero level instead of comparing against the global.
touch_core_sched_dispatch() stays naked, to be resolved by a later patch.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The flags passed to the pick-idle kfuncs are values from the
scx_pick_idle_cpu_flags enum, whose members are prefixed
SCX_PICK_IDLE_ (SCX_PICK_IDLE_CORE, SCX_PICK_IDLE_IN_NODE).
Three kernel-doc comments in idle.c erroneously used
%SCX_PICK_IDLE_CPU_* which does not correspond to any defined flag
name, while the adjacent scx_bpf_pick_idle_cpu_node() correctly
documents %SCX_PICK_IDLE_*.
Fix the three occurrences to use the correct SCX_PICK_IDLE_* prefix.
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
SCX's built-in idle CPU tracking is imperfect and can be out-of-sync
with the actual idle state of CPUs, especially immediately after
enabling SCX due to scx_idle_enable() marking all online CPUs idle.
scx_select_cpu_dfl() skips marking the selected CPU as busy if
the selected CPU is the waker CPU in the SCX_WAKE_SYNC case.
If the waker CPU was marked idle by SCX, it will still be marked idle
after CPU selection and potentially even after switching to the wakee.
In the allowed_cpus selftest, this can manifest as the test failing with
the following message in dmesg:
allowed_cpus.bpf.c:21: CPU 0 should be marked as busy
This patch explicitly marks the waker CPU as busy. With this patch,
the test failure no longer reproduces. There are still some pretty
unlikely races that could make the test fail (e.g. pick_task_idle()
marking the selected CPU idle between selection and validation), but
these can't be fixed easily.
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With CONFIG_EXT_SUB_SCHED=y but no sub-scheduler attached - the common case
- hot paths still pay for sub-sched bookkeeping. Gate it behind
__scx_has_subs, a static key counting live sub-schedulers, so that a
root-only system stops paying.
Most conversions are simple skip-if-no-sub tests. scx_idle_notify() is
special - it's a hierarchy walk, so give it a fast path which notifies the
root directly using the same tests as the walk. A pending
SCX_RQ_SUB_IDLE_RENOTIFY can be ignored as no sub can be owed one and the
caller clears the flag either way.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
__scx_update_idle() notified only the root scheduler. A sub-scheduler that
holds a cid needs that cid's idle state to place and kick on it.
Deliver ops.update_idle() to every scheduler that holds SCX_CAP_BASE on the
transitioning cid. The root holds every cap, so a real transition always
reaches it.
Real transitions are not enough on their own. A cid that is already idle
when a sub-sched gains baseline access produces no transition, so the new
holder would never learn it is idle. The ecaps sync arms a re-notify on the
gain, and the next idle pick delivers ops.update_idle() to just that sched,
leaving holders that already track the cpu untouched. A matching loss of
baseline access drops any pending re-notify.
Bypass suppresses ops.update_idle() too, so a cpu that goes idle during a
bypass window and stays idle yields no transition to re-deliver on
un-bypass. Arm the same re-notify for every sched leaving bypass. The acute
case is a child granted cids during its own ops.sub_attach(). The grant
lands while the child is bypassed and the notify walk skips it, so on
un-bypass it holds cids it never saw go idle. The root is owed the same and
is armed through a separate per-rq flag, which keeps this working when
sub-schedulers are compiled out.
v2: Gate the idle catch-up in pick_task_idle() to avoid a double ops.update_idle(). (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The sources under kernel/sched/ext/ build as a single translation unit:
build_policy.c includes the source files and headers. An LSP/clangd editor
parses each as a standalone unit, sees no types, and reports a flood of
errors.
Give each header its dependencies and include guard, and have each source
include the headers it uses.
ext.c, arena.c and the ext headers now parse clean standalone. idle.c and
cid.c still reference a few macros and helpers defined in ext.c. The next
patch moves those to shared headers.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The sched_ext sources had grown to ten ext* files directly under
kernel/sched/. Move them into a new kernel/sched/ext/ subdirectory and drop
the now-redundant ext_ prefix. ext.c/h keep their names.
kernel/sched/ext.{c,h} -> kernel/sched/ext/ext.{c,h}
kernel/sched/ext_internal.h -> kernel/sched/ext/internal.h
kernel/sched/ext_types.h -> kernel/sched/ext/types.h
kernel/sched/ext_idle.{c,h} -> kernel/sched/ext/idle.{c,h}
kernel/sched/ext_cid.{c,h} -> kernel/sched/ext/cid.{c,h}
kernel/sched/ext_arena.{c,h} -> kernel/sched/ext/arena.{c,h}
The include paths in build_policy.c and sched.h, the MAINTAINERS glob, and a
few documentation and comment references are updated to match. No code or
symbol changes.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
Tejun Heo
Andrea Righi
Liang Luo
Kuba Piecuch