diff options
author | Eric Dumazet <edumazet@google.com> | 2012-07-29 23:20:37 +0000 |
---|---|---|
committer | David S. Miller <davem@davemloft.net> | 2012-07-30 14:53:22 -0700 |
commit | 404e0a8b6a55d5e1cd138c6deb1bca9abdf75d8c (patch) | |
tree | 38e9748d38c415cc97b973fecb9279cd43f76393 /net/ipv4/route.c | |
parent | cca32e4bf999a34ac08d959f351f2b30bcd02460 (diff) | |
download | op-kernel-dev-404e0a8b6a55d5e1cd138c6deb1bca9abdf75d8c.zip op-kernel-dev-404e0a8b6a55d5e1cd138c6deb1bca9abdf75d8c.tar.gz |
net: ipv4: fix RCU races on dst refcounts
commit c6cffba4ffa2 (ipv4: Fix input route performance regression.)
added various fatal races with dst refcounts.
crashes happen on tcp workloads if routes are added/deleted at the same
time.
The dst_free() calls from free_fib_info_rcu() are clearly racy.
We need instead regular dst refcounting (dst_release()) and make
sure dst_release() is aware of RCU grace periods :
Add DST_RCU_FREE flag so that dst_release() respects an RCU grace period
before dst destruction for cached dst
Introduce a new inet_sk_rx_dst_set() helper, using atomic_inc_not_zero()
to make sure we dont increase a zero refcount (On a dst currently
waiting an rcu grace period before destruction)
rt_cache_route() must take a reference on the new cached route, and
release it if was not able to install it.
With this patch, my machines survive various benchmarks.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Diffstat (limited to 'net/ipv4/route.c')
-rw-r--r-- | net/ipv4/route.c | 16 |
1 files changed, 4 insertions, 12 deletions
diff --git a/net/ipv4/route.c b/net/ipv4/route.c index fc1a81c..d6eabcf 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -1199,11 +1199,6 @@ restart: fnhe->fnhe_stamp = jiffies; } -static inline void rt_free(struct rtable *rt) -{ - call_rcu_bh(&rt->dst.rcu_head, dst_rcu_free); -} - static void rt_cache_route(struct fib_nh *nh, struct rtable *rt) { struct rtable *orig, *prev, **p = &nh->nh_rth_output; @@ -1213,17 +1208,14 @@ static void rt_cache_route(struct fib_nh *nh, struct rtable *rt) orig = *p; + rt->dst.flags |= DST_RCU_FREE; + dst_hold(&rt->dst); prev = cmpxchg(p, orig, rt); if (prev == orig) { if (orig) - rt_free(orig); + dst_release(&orig->dst); } else { - /* Routes we intend to cache in the FIB nexthop have - * the DST_NOCACHE bit clear. However, if we are - * unsuccessful at storing this route into the cache - * we really need to set it. - */ - rt->dst.flags |= DST_NOCACHE; + dst_release(&rt->dst); } } |