diff options
author | Paul E. McKenney <paulmck@linux.vnet.ibm.com> | 2013-09-03 09:52:20 -0700 |
---|---|---|
committer | Paul E. McKenney <paulmck@linux.vnet.ibm.com> | 2013-09-23 09:15:28 -0700 |
commit | 69c8d28c96445e28f081fcd987e34ea2afa65039 (patch) | |
tree | 8b7168675d2d11a4ddf82f74541d35fe94e6ba87 /kernel | |
parent | 289828e62de0334a0d01c0f65df91cd47d3a9e05 (diff) | |
download | op-kernel-dev-69c8d28c96445e28f081fcd987e34ea2afa65039.zip op-kernel-dev-69c8d28c96445e28f081fcd987e34ea2afa65039.tar.gz |
rcu: Micro-optimize rcu_cpu_has_callbacks()
The for_each_rcu_flavor() loop unconditionally scans all flavors, even
when the first flavor might have some non-lazy callbacks. Once the
loop has seen a non-lazy callback, further passes through the loop
cannot change the state. This is not a huge problem, given that there
can be at most three RCU flavors (RCU-bh, RCU-preempt, and RCU-sched),
but this code is on the path to idle, so speeding it up even a small
amount would have some benefit.
This commit therefore does two things:
1. Rearranges the order of the list of RCU flavors in order to
place the most active flavor first in the list. The most active
RCU flavor is RCU-preempt, or, if there is no RCU-preempt,
RCU-sched.
2. Reworks the for_each_rcu_flavor() to exit early when the first
non-lazy callback is seen, or, in the case where the caller
does not care about non-lazy callbacks (RCU_FAST_NO_HZ=n),
when the first callback is seen.
Reported-by: Chen Gang <gang.chen@asianux.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Diffstat (limited to 'kernel')
-rw-r--r-- | kernel/rcutree.c | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/kernel/rcutree.c b/kernel/rcutree.c index e6f2e8f..49464ade 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -2727,10 +2727,13 @@ static int rcu_cpu_has_callbacks(int cpu, bool *all_lazy) for_each_rcu_flavor(rsp) { rdp = per_cpu_ptr(rsp->rda, cpu); - if (rdp->qlen != rdp->qlen_lazy) + if (!rdp->nxtlist) + continue; + hc = true; + if (rdp->qlen != rdp->qlen_lazy || !all_lazy) { al = false; - if (rdp->nxtlist) - hc = true; + break; + } } if (all_lazy) *all_lazy = al; @@ -3297,8 +3300,8 @@ void __init rcu_init(void) rcu_bootup_announce(); rcu_init_geometry(); - rcu_init_one(&rcu_sched_state, &rcu_sched_data); rcu_init_one(&rcu_bh_state, &rcu_bh_data); + rcu_init_one(&rcu_sched_state, &rcu_sched_data); __rcu_init_preempt(); open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); |