Do not use interrupt taskqueue on controllers with MSI/MSI-X

capability. One of reason using interrupt taskqueue in re(4) was
to reduce number of TX/RX interrupts under load because re(4)
controllers have no good TX/RX interrupt moderation mechanism.
Basic TX interrupt moderation is done by hardware for most
controllers but RX interrupt moderation through undocumented
register showed poor RX performance so it was disabled in r215025.
Using taskqueue to handle RX interrupt greatly reduced number of
interrupts but re(4) consumed all available CPU cycles to run the
taskqueue under high TX/RX network load.  This can happen even with
RTL810x fast ethernet controller and I believe this is not
acceptable for most systems.

To mitigate the issue, use one-shot timer register to moderate RX
interrupts. The timer register provides programmable one-shot timer
and can be used to suppress interrupt generation. The timer runs at
125MHZ on PCIe controllers so the minimum time allowed for the
timer is 8ns. Data sheet says the register is 32 bits but
experimentation shows only lower 13 bits are valid so maximum time
that can be programmed is 65.528us. This yields theoretical maximum
number of RX interrupts that could be generated per second is about
15260. Combined with TX completion interrupts re(4) shall generate
less than 20k interrupts. This number is still slightly high
compared to other intelligent ethernet controllers but system is
very responsive even under high network load.

Introduce sysctl variable dev.re.%d.int_rx_mod that controls amount
of time to delay RX interrupt processing in units of us. Value 0
completely disables RX interrupt moderation. To provide old
behavior for controllers that have MSI/MSI-X capability, introduce
a new tunable hw.re.intr_filter. If the tunable is set to non-zero
value, driver will use interrupt taskqueue. The default value of
the tunable is 0. This tunable has no effect on controllers that
has no MSI/MSI-X capability or if MSI/MSI-X is explicitly disabled
by administrator.

While I'm here cleanup interrupt setup/teardown since re(4) uses
single MSI/MSI-X message at this moment.

1 files changed, 10 insertions, 0 deletions

diff --git a/sys/pci/if_rlreg.h b/sys/pci/if_rlreg.h
index 0397d28..bb6a7ad 100644
--- a/sys/pci/if_rlreg.h
+++ b/sys/pci/if_rlreg.h
@@ -497,6 +497,14 @@
 
 #define	RL_EARLYTXTHRESH_CNT	0x003F	/* byte count times 8 */
 
+/* Timer interrupt register */
+#define	RL_TIMERINT_8169_VAL	0x00001FFF
+#define	RL_TIMER_MIN		0
+#define	RL_TIMER_MAX		65	/* 65.528us */
+#define	RL_TIMER_DEFAULT	RL_TIMER_MAX
+#define	RL_TIMER_PCIE_CLK	125	/* 125MHZ */
+#define	RL_USECS(x)		((x) * RL_TIMER_PCIE_CLK)
+
 /*
  * Gigabit PHY access register (8169 only)
  */
@@ -896,6 +904,8 @@ struct rl_softc {
 	struct task		rl_inttask;
 
 	int			rl_txstart;
+	int			rl_int_rx_act;
+	int			rl_int_rx_mod;
 	uint32_t		rl_flags;
 #define	RL_FLAG_MSI		0x0001
 #define	RL_FLAG_AUTOPAD		0x0002