diff options
| author |   Paul E. McKenney <paulmck@kernel.org> | 2020-08-06 15:12:50 -0700 |
|---|---|---|
| committer | Paul E. McKenney <paulmck@kernel.org> | 2020-08-24 18:40:25 -0700 |
| commit | 44bad5b3cca2d452d17ef82841b20b42a2cf11a0 (patch) | |
| tree | 511ed2b6da2d3d6996a64fbc3b37503c7f84cd6d /kernel/rcu | |
| parent | rcu: Always set .need_qs from __rcu_read_lock() for strict GPs (diff) | |
| download | linux-dev-44bad5b3cca2d452d17ef82841b20b42a2cf11a0.tar.xz linux-dev-44bad5b3cca2d452d17ef82841b20b42a2cf11a0.zip | |
rcu: Do full report for .need_qs for strict GPs
The rcu_preempt_deferred_qs_irqrestore() function is invoked at
the end of an RCU read-side critical section (for example, directly
from rcu_read_unlock()) and, if .need_qs is set, invokes rcu_qs() to
report the new quiescent state. This works, except that rcu_qs() only
updates per-CPU state, leaving reporting of the actual quiescent state
to a later call to rcu_report_qs_rdp(), for example from within a later
RCU_SOFTIRQ instance. Although this approach is exactly what you want if
you are more concerned about efficiency than about short grace periods,
in CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, short grace periods are
the name of the game.
This commit therefore makes rcu_preempt_deferred_qs_irqrestore() directly
invoke rcu_report_qs_rdp() in CONFIG_RCU_STRICT_GRACE_PERIOD=y, thus
shortening grace periods.
Historical note: To the best of my knowledge, causing rcu_read_unlock()
to directly report a quiescent state first appeared in Jim Houston's
and Joe Korty's JRCU. This is the second instance of a Linux-kernel RCU
feature being inspired by JRCU, the first being RCU callback offloading
(as in the RCU_NOCB_CPU Kconfig option).
Reported-by Jann Horn <jannh@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Diffstat (limited to 'kernel/rcu')
| -rw-r--r-- | kernel/rcu/tree_plugin.h | 8 |
1 files changed, 6 insertions, 2 deletions
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 668bbd2be807..dfdb9020f136 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -459,8 +459,12 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) return; } t->rcu_read_unlock_special.s = 0; - if (special.b.need_qs) - rcu_qs(); + if (special.b.need_qs) { + if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) + rcu_report_qs_rdp(rdp->cpu, rdp); + else + rcu_qs(); + } /* * Respond to a request by an expedited grace period for a |