The drbr(9) API appeared to be so unclear, that most drivers in

tree used it incorrectly, which lead to inaccurate overrated
if_obytes accounting. The drbr(9) used to update ifnet stats on
drbr_enqueue(), which is not accurate since enqueuing doesn't
imply successful processing by driver. Dequeuing neither mean
that. Most drivers also called drbr_stats_update() which did
accounting again, leading to doubled if_obytes statistics. And
in case of severe transmitting, when a packet could be several
times enqueued and dequeued it could have been accounted several
times.

o Thus, make drbr(9) API thinner. Now drbr(9) merely chooses between
  ALTQ queueing or buf_ring(9) queueing.
  - It doesn't touch the buf_ring stats any more.
  - It doesn't touch ifnet stats anymore.
  - drbr_stats_update() no longer exists.

o buf_ring(9) handles its stats itself:
  - It handles br_drops itself.
  - br_prod_bytes stats are dropped. Rationale: no one ever
    reads them but update of a common counter on every packet
    negatively affects performance due to excessive cache
    invalidation.
  - buf_ring_enqueue_bytes() reduced to buf_ring_enqueue(), since
    we no longer account bytes.

o Drivers handle their stats theirselves: if_obytes, if_omcasts.

o mlx4(4), igb(4), em(4), vxge(4), oce(4) and  ixv(4) no longer
  use drbr_stats_update(), and update ifnet stats theirselves.

o bxe(4) was the most correct driver, it didn't call
  drbr_stats_update(), thus it was the only driver accurate under
  moderate load. Now it also maintains stats itself.

o ixgbe(4) had already taken stats from hardware, so just
  - drop software stats updating.
  - take multicast packet count from hardware as well.

o mxge(4) just no longer needs NO_SLOW_STATS define.

o cxgb(4), cxgbe(4) need no change, since they obtain stats
  from hardware.

Reviewed by:	jfv, gnn

2 files changed, 3 insertions, 20 deletions

diff --git a/share/man/man9/buf_ring.9 b/share/man/man9/buf_ring.9
index ffade94..95b3a3d 100644
--- a/share/man/man9/buf_ring.9
+++ b/share/man/man9/buf_ring.9
@@ -25,7 +25,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd January 30, 2012
+.Dd September 27, 2012
 .Dt BUF_RING 9
 .Os
 .Sh NAME
@@ -33,7 +33,6 @@
 .Nm buf_ring_alloc ,
 .Nm buf_ring_free ,
 .Nm buf_ring_enqueue ,
-.Nm buf_ring_enqueue_bytes ,
 .Nm buf_ring_dequeue_mc ,
 .Nm buf_ring_dequeue_sc ,
 .Nm buf_ring_count ,
@@ -50,8 +49,6 @@
 .Fn buf_ring_free "struct buf_ring *br" "struct malloc_type *type"
 .Ft int
 .Fn buf_ring_enqueue "struct buf_ring *br" "void *buf"
-.Ft int
-.Fn buf_ring_enqueue_bytes "struct buf_ring *br" "void *buf" "int bytes"
 .Ft void *
 .Fn buf_ring_dequeue_mc "struct buf_ring *br"
 .Ft void *
@@ -91,12 +88,6 @@ The
 function is used to enqueue a buffer to a buf_ring.
 .Pp
 The
-.Fn buf_ring_enqueue_bytes
-function is used to enqueue a buffer to a buf_ring and increment the
-number of bytes enqueued by
-.Fa bytes .
-.Pp
-The
 .Fn buf_ring_dequeue_mc
 function is a multi-consumer safe way of dequeueing elements from a buf_ring.
 .Pp
@@ -134,9 +125,7 @@ otherwise.
 .Sh RETURN VALUES
 The
 .Fn buf_ring_enqueue
-and
-.Fn buf_ring_enqueue_bytes
-functions return
+function return
 .Er ENOBUFS
 if there are no available slots in the buf_ring.
 .Sh HISTORY
diff --git a/share/man/man9/drbr.9 b/share/man/man9/drbr.9
index 168ce10..3d9f881 100644
--- a/share/man/man9/drbr.9
+++ b/share/man/man9/drbr.9
@@ -25,7 +25,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd January 30, 2012
+.Dd September 27, 2012
 .Dt DRBR 9
 .Os
 .Sh NAME
@@ -37,7 +37,6 @@
 .Nm drbr_flush ,
 .Nm drbr_empty ,
 .Nm drbr_inuse ,
-.Nm drbr_stats_update ,
 .Nd network driver interface to buf_ring
 .Sh SYNOPSIS
 .In sys/param.h
@@ -57,8 +56,6 @@
 .Fn drbr_empty "struct ifnet *ifp" "struct buf_ring *br"
 .Ft int
 .Fn drbr_inuse "struct ifnet *ifp" "struct buf_ring *br"
-.Ft void
-.Fn drbr_stats_update "struct ifnet *ifp" "int len" "int mflags"
 .Sh DESCRIPTION
 The
 .Nm
@@ -123,9 +120,6 @@ there will not be more mbufs when
 is actually called.
 Provided the tx queue lock is held there will not be less.
 .Pp
-The
-.Fn drbr_stats_update
-function updates the number of bytes and the number of multicast packets sent.
 .Sh RETURN VALUES
 The
 .Fn drbr_enqueue