MFC r319506, r319872, r321063, r321103, r321179, r321390, r321435,

r321582, r321671, r322014, r322034, r322055, r322123, r322167, r322425,
r322549, r322914, r322960, r322962, r322964, r322985, r322990, r323006,
r323026, r323041, r323069, r323078, r323343, r323514, r323520, r324296,
r324379, r324386, r324443, r324945, r325596, r325680, r325880,
r325883-r325884, r325961, r326026, r326042, r327062, r327093, r327332,
r327528, r328420, and r328423.

r319506:
cxgbe(4): Update the statistics for compound tx work requests once per
work request, not once per frame.

r319872:
cxgbe(4):  Do not request an FEC setting that the port does not support.

r321063:
cxgbe(4): Various link/media related improvements.

- Deal with changes to port_type, and not just port_mod when a
  transceiver is changed.  This fixes hot swapping of transceivers of
  different types (QSFP+ or QSA or QSFP28 in a QSFP28 port, SFP+ or
  SFP28 in a SFP28 port, etc.).

- Always refresh media information for ifconfig if the port is down.
  The firmware does not generate tranceiver-change interrupts unless at
  least one VI is enabled on the physical port.  Before this change
  ifconfig diplayed potentially stale information for ports that were
  administratively down.

- Always recalculate and reapply L1 config on a transceiver change.

- Display PAUSE settings in ifconfig.  The driver sysctls for this
  continue to work as well.

r321103:
cxgbe(4): New ioctls to flash bootrom and boot config to the card.

r321179:
cxgbe/t4_tom: Log more details about the newly ESTABLISHED tid to the
trace buffer.

r321390:
cxgbe(4): Install the firmware bundled with the driver to the card if it
doesn't seem to have one.  This lets the driver recover automatically
from incomplete firmware upgrades (panic, reboot, power loss, etc. in
the middle of an upgrade).

r321435:
cxgbe(4): Display some more TOE parameters related to retransmission
and keepalive in the sysctl MIB.  Provide tunables to change some of
these parameters.  These are supposed to be setup by the firmware so
these tunables are for experimentation only.

r321582:
cxgbe(4): Some updates to the common code.

- Updated register ranges.
- Helper routines for access to TP registers.
- Updated routine to read flash parameters.

r321671:
cxgbe/iw_cxgbe: Log the end point's history and flags to the trace
buffer just before it's freed.

r322014:
cxgbe(4): Initial import of the "collect" component of Chelsio unified
debug (cudbg) code, hooked up to the main driver via an ioctl.

The ioctl can be used to collect the chip's internal state in a
compressed dump file.  These dumps can be decoded with the "view"
component of cudbg.

r322034:
cxgbe(4): Always use the first and not the last virtual interface
associated with a port in begin_synchronized_op.

r322055:
cxgbe(4): Allow the TOE timer tunables to be set with microsecond
precision.  These timers are already displayed in microseconds in the
sysctl MIB.  Add variables to track these tunables while here.

r322123:
cxgbe(4): Avoid a NULL dereference that would occur during module unload
if there were problems earlier during attach.

r322167:
cxgbe(4): Add the T6 and T5 Unified Wire configuration files to the
kernel, just like for T4, when the driver is compiled into the kernel.

r322425:
cxgbe(4): Save the last reported link parameters and compare them with
the current state to determine whether to generate a link-state change
notification.  This fixes a bug introduced in r321063 that caused the
driver to sometimes skip these notifications.

r322549:
cxgbe/t4_tom: Use correct name for the ISS-valid bit in options2.

r322914:
cxgbe(4): Dump the mailbox contents in the same format as CH_DUMP_MBOX.

r322960:
cxgbe(4): Verify that the driver accesses the firmware mailbox in a
thread-safe manner.

r322962:
cxgbe(4): Remove write only variable from t4_port_init.

r322964:
cxgbe(4): vi_mac_funcs should include the base Ethernet function.  It is
already used in the driver as if it does.

r322985:
cxgbe(4): Maintain one ifmedia per physical port instead of one per
Virtual Interface (VI).  All autonomous VIs that share a port share the
same media.

r322990:
cxgbe(4): Do not access the mailbox without appropriate locks while
creating hardware VIs.

This fixes a bad race on systems with hw.cxgbe.num_vis > 1.

r323006:
cxgbe(4): Update T6/T5/T4 firmwares to 1.16.59.0.

r323026:
cxgbe(4): Zero out the memory allocated for the debug dump.
cudbg_collect seems to expect it this way.

r323041:
cxgbe(4): Add two new debug flags -- one to allow manual firmware
install after full initialization, and another to disable the TCB
cache (T6+).  The latter works as a tunable only.

Note that debug_flags are for debugging only and should not be set
normally.

r323069:
cxgbe/t4_tom: Add a knob to select the congestion control algorigthm
used by the TOE hardware for fully offloaded connections.  The knob
affects new connections only.

r323078:
cxgbe/t4_tom: There may not be a tid to update if the connection isn't
established.

r323343:
cxgbe(4): Fix a couple of problems in the sge_wrq data path.

- start_wrq_wr must not drain the wr_list if there are incomplete_wrs
  pending.  This can happen when a t4_wrq_tx runs between two
  start_wrq_wr.

- commit_wrq_wr must examine the cookie's pidx and ndesc with the
  queue's lock held.  Otherwise there is a bad race when incomplete WRs
  are being completed and commit_wrq_wr for the WR that is ahead in the
  queue updates the next incomplete WR's cookie's pidx/ndesc but the
  commit_wrq_wr for the second one is using stale values that it read
  without the lock.

r323514:
cxgbetool(8): mode must be specified when creating the dump file.

r323520:
cxgbe(4): Ignore capabilities that depend on TOE when the firmware
reports TOE is not available.

r324296:
cxgbe(4): Provide knobs to set the holdoff parameters of TOE rx queues
separately from NIC rx queues instead of using the same parameters for
both types of queues.

r324379:
cxgbetool(8): Do not create a large file devoid of useful content when
the dumpstate ioctl fails.  Make the file world-readable while here.

r324386:
cxgbe(4): Update T6, T5, and T4 firmwares to 1.16.63.0.

r324443:
cxgbetool(8): Do not close uninitialized fd on malloc failure.

r324945:
cxgbe(4): Read the MPS buffer group map from the firmware as it could be
different from hardware defaults.  The congestion channel map, which is
still fixed, needs to be tracked separately now.  Change the congestion
setting for TOE rx queues to match the drivers on other OSes while here.

r325596:
cxgbe(4): Do not request settings not supported by the port.

r325680:
cxgbe(4): Excluce mdi from the check against port capabilities.

r325880:
cxgbe(4):  Combine all _10g and _1g tunables and drop the suffix from
their names.  The finer-grained knobs weren't practically useful.

r325883:
cxgbe(4): Sanitize t4_num_vis during MOD_LOAD like all other t4_*
tunables.  Add num_vis to the intrs_and_queues structure as it affects
the number of interrupts requested and queues created.  In future
cfg_itype_and_nqueues might lower it incrementally instead of going
straight to 1 when enough interrupts aren't available.

r325884:
cxgbe(4): Remove rsrv_noflowq from intrs_and_queues structure as it does
not influence or get affected by the number of interrupts or queues.

r325961:
cxgbe(4): Add core Vdd to the sysctl MIB.

r326026:
cxgbe(4): Add a custom board to the device id list.

r326042:
cxgbe(4): Fix unsafe mailbox access in cudbg.

r327062:
cxgbe(4):  Read the MFG diags version from the VPD and make it available
in the sysctl MIB.

r327093:
cxgbe(4): Do not forward interrupts to queues with freelists.  This
leaves the firmware event queue (fwq) as the only queue that can take
interrupts for others.

This simplifies cfg_itype_and_nqueues and queue allocation in the driver
at the cost of a little (never?) used configuration.  It also allows
service_iq to be split into two specialized variants in the future.

r327332:
cxgbe(4): Reduce duplication by consolidating minor variations of the
same code into a single routine.

r327528:
cxgbe(4): Add a knob to enable/disable PCIe relaxed ordering.  Disable it by
default when running on Intel CPUs.

r328420:
cxgbe(4): Do not display harmless warning in non-debug builds.

r328423:
cxgbe(4): Accept old names of a couple of tunables.

Sponsored by:	Chelsio Communications

2 files changed, 30 insertions, 46 deletions

diff --git a/share/man/man4/cxgbe.4 b/share/man/man4/cxgbe.4
index 9ee9f84..1c66906 100644
--- a/share/man/man4/cxgbe.4
+++ b/share/man/man4/cxgbe.4
@@ -172,37 +172,22 @@ types.
 A negative value for such a tunable instructs the driver to create
 up to that many queues if there are enough CPU cores available.
 .Bl -tag -width indent
-.It Va hw.cxgbe.ntxq10g
-Number of tx queues used for a 10Gb or higher-speed port.
+.It Va hw.cxgbe.ntxq
+Number of NIC tx queues used for a port.
 The default is 16 or the number
 of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nrxq10g
-Number of rx queues used for a 10Gb or higher-speed port.
+.It Va hw.cxgbe.nrxq
+Number of NIC rx queues used for a port.
 The default is 8 or the number
 of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.ntxq1g
-Number of tx queues used for a 1Gb port.
-The default is 4 or the number
-of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nrxq1g
-Number of rx queues used for a 1Gb port.
-The default is 2 or the number
-of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nofldtxq10g
-Number of TOE tx queues used for a 10Gb or higher-speed port.
+.It Va hw.cxgbe.nofldtxq
+Number of TOE tx queues used for a port.
 The default is 8 or the
 number of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nofldrxq10g
-Number of TOE rx queues used for a 10Gb or higher-speed port.
-The default is 2 or the
-number of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nofldtxq1g
-Number of TOE tx queues used for a 1Gb port.
+.It Va hw.cxgbe.nofldrxq
+Number of TOE rx queues used for a port.
 The default is 2 or the
 number of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nofldrxq1g
-Number of TOE rx queues used for a 1Gb port.
-The default is 1.
 .It Va hw.cxgbe.num_vis
 Number of virtual interfaces (VIs) created for each port.
 Each virtual interface creates a separate network interface.
@@ -213,25 +198,27 @@ name from the table above.
 Additional virtual interfaces use a single pair of queues
 for rx and tx as well an additional pair of queues for TOE rx and tx.
 The default is 1.
-.It Va hw.cxgbe.holdoff_timer_idx_10G
-.It Va hw.cxgbe.holdoff_timer_idx_1G
+.It Va hw.cxgbe.holdoff_timer_idx
+.It Va hw.cxgbe.holdoff_timer_idx_ofld
 Timer index value used to delay interrupts.
 The holdoff timer list has the values 1, 5, 10, 50, 100, and 200
 by default (all values are in microseconds) and the index selects a
 value from this list.
+holdoff_timer_idx_ofld applies to queues used for TOE rx.
 The default value is 1 which means the timer value is 5us.
 Different interfaces can be assigned different values at any time via the
-dev.<port>.X.holdoff_tmr_idx sysctl.
-.It Va hw.cxgbe.holdoff_pktc_idx_10G
-.It Va hw.cxgbe.holdoff_pktc_idx_1G
+dev.<port>.X.holdoff_tmr_idx and dev.<port>.X.holdoff_tmr_idx_ofld sysctls.
+.It Va hw.cxgbe.holdoff_pktc_idx
+.It Va hw.cxgbe.holdoff_pktc_idx_ofld
 Packet-count index value used to delay interrupts.
 The packet-count list has the values 1, 8, 16, and 32 by default,
 and the index selects a value from this list.
+holdoff_pktc_idx_ofld applies to queues used for TOE rx.
 The default value is -1 which means packet counting is disabled and interrupts
 are generated based solely on the holdoff timer value.
 Different interfaces can be assigned different values via the
-dev.<port>.X.holdoff_pktc_idx sysctl.
-This sysctl works only when the interface has never been marked up (as done by
+dev.<port>.X.holdoff_pktc_idx and dev.<port>.X.holdoff_pktc_idx_ofld sysctls.
+These sysctls work only when the interface has never been marked up (as done by
 ifconfig up).
 .It Va hw.cxgbe.qsize_txq
 Number of entries in a transmit queue's descriptor ring.
@@ -256,6 +243,13 @@ Permitted interrupt types.
 Bit 0 represents INTx (line interrupts), bit 1 MSI, and bit 2 MSI-X.
 The default is 7 (all allowed).
 The driver selects the best possible type out of the allowed types.
+.It Va hw.cxgbe.pcie_relaxed_ordering
+PCIe Relaxed Ordering.
+-1 indicates the driver should determine whether to enable or disable PCIe RO.
+0 disables PCIe RO.
+1 enables PCIe RO.
+2 indicates the driver should not modify the PCIe RO setting.
+The default is -1.
 .It Va hw.cxgbe.fw_install
 0 prohibits the driver from installing a firmware on the card.
 1 allows the driver to install a new firmware if internal driver
diff --git a/share/man/man4/cxgbev.4 b/share/man/man4/cxgbev.4
index 9dc3421..b2fcac6 100644
--- a/share/man/man4/cxgbev.4
+++ b/share/man/man4/cxgbev.4
@@ -172,24 +172,15 @@ Tunables can be set at the
 prompt before booting the kernel or stored in
 .Xr loader.conf 5 .
 .Bl -tag -width indent
-.It Va hw.cxgbe.ntxq10g
-Number of tx queues used for a 10Gb or higher-speed port.
+.It Va hw.cxgbe.ntxq
+Number of tx queues used for a port.
 The default is 16 or the number
 of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nrxq10g
-Number of rx queues used for a 10Gb or higher-speed port.
+.It Va hw.cxgbe.nrxq
+Number of rx queues used for a port.
 The default is 8 or the number
 of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.ntxq1g
-Number of tx queues used for a 1Gb port.
-The default is 4 or the number
-of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.nrxq1g
-Number of rx queues used for a 1Gb port.
-The default is 2 or the number
-of CPU cores in the system, whichever is less.
-.It Va hw.cxgbe.holdoff_timer_idx_10G
-.It Va hw.cxgbe.holdoff_timer_idx_1G
+.It Va hw.cxgbe.holdoff_timer_idx
 Timer index value used to delay interrupts.
 The holdoff timer list has the values 1, 5, 10, 50, 100, and 200
 by default (all values are in microseconds) and the index selects a
@@ -197,8 +188,7 @@ value from this list.
 The default value is 1 which means the timer value is 5us.
 Different interfaces can be assigned different values at any time via the
 dev.<port>.X.holdoff_tmr_idx sysctl.
-.It Va hw.cxgbe.holdoff_pktc_idx_10G
-.It Va hw.cxgbe.holdoff_pktc_idx_1G
+.It Va hw.cxgbe.holdoff_pktc_idx
 Packet-count index value used to delay interrupts.
 The packet-count list has the values 1, 8, 16, and 32 by default,
 and the index selects a value from this list.