diff options
| author |   Paul E. McKenney <paulmck@linux.ibm.com> | 2019-07-16 02:17:00 -0700 |
|---|---|---|
| committer | Paul E. McKenney <paulmck@linux.ibm.com> | 2019-08-13 14:38:24 -0700 |
| commit | f48fe4c586604c3a09938c6a6e9fd3356dfe8f3c (patch) | |
| tree | 011d5815d8de824c08a4d9aade25b9e01e6f78bc /kernel/rcu | |
| parent | rcu/nocb: Reduce __call_rcu_nocb_wake() leaf rcu_node ->lock contention (diff) | |
| download | wireguard-linux-f48fe4c586604c3a09938c6a6e9fd3356dfe8f3c.tar.xz wireguard-linux-f48fe4c586604c3a09938c6a6e9fd3356dfe8f3c.zip | |
rcu/nocb: Don't wake no-CBs GP kthread if timer posted under overload
When under overload conditions, __call_rcu_nocb_wake() will wake the
no-CBs GP kthread any time the no-CBs CB kthread is asleep or there
are no ready-to-invoke callbacks, but only after a timer delay. If the
no-CBs GP kthread has a ->nocb_bypass_timer pending, the deferred wakeup
from __call_rcu_nocb_wake() is redundant. This commit therefore makes
__call_rcu_nocb_wake() avoid posting the redundant deferred wakeup if
->nocb_bypass_timer is pending. This requires adding a bit of ordering
of timer actions.
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
Diffstat (limited to 'kernel/rcu')
| -rw-r--r-- | kernel/rcu/tree_plugin.h | 7 |
1 files changed, 5 insertions, 2 deletions
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index f56fb4e97a8e..2defc7fe74c3 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1909,8 +1909,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - if (rdp->nocb_cb_sleep || - !rcu_segcblist_ready_cbs(&rdp->cblist)) + smp_mb(); /* Enqueue before timer_pending(). */ + if ((rdp->nocb_cb_sleep || + !rcu_segcblist_ready_cbs(&rdp->cblist)) && + !timer_pending(&rdp->nocb_bypass_timer)) wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred")); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1929,6 +1931,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer")); rcu_nocb_lock_irqsave(rdp, flags); + smp_mb__after_spinlock(); /* Timer expire before wakeup. */ __call_rcu_nocb_wake(rdp, true, flags); } |