The drbr(9) API appeared to be so unclear, that most drivers in

tree used it incorrectly, which lead to inaccurate overrated
if_obytes accounting. The drbr(9) used to update ifnet stats on
drbr_enqueue(), which is not accurate since enqueuing doesn't
imply successful processing by driver. Dequeuing neither mean
that. Most drivers also called drbr_stats_update() which did
accounting again, leading to doubled if_obytes statistics. And
in case of severe transmitting, when a packet could be several
times enqueued and dequeued it could have been accounted several
times.

o Thus, make drbr(9) API thinner. Now drbr(9) merely chooses between
  ALTQ queueing or buf_ring(9) queueing.
  - It doesn't touch the buf_ring stats any more.
  - It doesn't touch ifnet stats anymore.
  - drbr_stats_update() no longer exists.

o buf_ring(9) handles its stats itself:
  - It handles br_drops itself.
  - br_prod_bytes stats are dropped. Rationale: no one ever
    reads them but update of a common counter on every packet
    negatively affects performance due to excessive cache
    invalidation.
  - buf_ring_enqueue_bytes() reduced to buf_ring_enqueue(), since
    we no longer account bytes.

o Drivers handle their stats theirselves: if_obytes, if_omcasts.

o mlx4(4), igb(4), em(4), vxge(4), oce(4) and  ixv(4) no longer
  use drbr_stats_update(), and update ifnet stats theirselves.

o bxe(4) was the most correct driver, it didn't call
  drbr_stats_update(), thus it was the only driver accurate under
  moderate load. Now it also maintains stats itself.

o ixgbe(4) had already taken stats from hardware, so just
  - drop software stats updating.
  - take multicast packet count from hardware as well.

o mxge(4) just no longer needs NO_SLOW_STATS define.

o cxgb(4), cxgbe(4) need no change, since they obtain stats
  from hardware.

Reviewed by:	jfv, gnn

1 files changed, 2 insertions, 10 deletions

diff --git a/sys/sys/buf_ring.h b/sys/sys/buf_ring.h
index 57e42e5..9f99fa2 100644
--- a/sys/sys/buf_ring.h
+++ b/sys/sys/buf_ring.h
@@ -48,7 +48,6 @@ struct buf_ring {
 	int              	br_prod_mask;
 	uint64_t		br_drops;
 	uint64_t		br_prod_bufs;
-	uint64_t		br_prod_bytes;
 	/*
 	 * Pad out to next L2 cache line
 	 */
@@ -74,7 +73,7 @@ struct buf_ring {
  *
  */
 static __inline int
-buf_ring_enqueue_bytes(struct buf_ring *br, void *buf, int nbytes)
+buf_ring_enqueue(struct buf_ring *br, void *buf)
 {
 	uint32_t prod_head, prod_next;
 	uint32_t cons_tail;
@@ -95,6 +94,7 @@ buf_ring_enqueue_bytes(struct buf_ring *br, void *buf, int nbytes)
 		prod_next = (prod_head + 1) & br->br_prod_mask;
 		
 		if (prod_next == cons_tail) {
+			br->br_drops++;
 			critical_exit();
 			return (ENOBUFS);
 		}
@@ -117,19 +117,11 @@ buf_ring_enqueue_bytes(struct buf_ring *br, void *buf, int nbytes)
 	while (br->br_prod_tail != prod_head)
 		cpu_spinwait();
 	br->br_prod_bufs++;
-	br->br_prod_bytes += nbytes;
 	br->br_prod_tail = prod_next;
 	critical_exit();
 	return (0);
 }
 
-static __inline int
-buf_ring_enqueue(struct buf_ring *br, void *buf)
-{
-
-	return (buf_ring_enqueue_bytes(br, buf, 0));
-}
-
 /*
  * multi-consumer safe dequeue 
  *