.\" Copyright (c) 2014 Qlogic Corporation. All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\"
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS'
.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
.\" THE POSSIBILITY OF SUCH DAMAGE.
.\"
.\" $FreeBSD$
.\"
.Dd April 29, 2012
.Dt BXE 4
.Os
.Sh NAME
.Nm bxe
.Nd QLogic NetXtreme II Ethernet 10Gb PCIe adapter driver
.Sh SYNOPSIS
To compile this driver into the kernel,
place the following lines in your
kernel configuration file:
.Bd -ragged -offset indent
.Cd "device bxe"
.Ed
.Pp
Alternatively, to load the driver as a module at boot time, place the
following line in
.Xr loader.conf 5 :
.Bd -literal -offset indent
if_bxe_load="YES"
.Ed
.Sh DESCRIPTION
The
.Nm
driver provides support for PCIe 10Gb Ethernet adapters based on the QLogic
NetXtreme II family of 10Gb chips.
The driver supports Jumbo Frames, VLAN
tagging, checksum offload (IPv4, TCP, UDP, IPv6-TCP, IPv6-UDP), MSI-X
interrupts, TCP Segmentation Offload (TSO), Large Receive Offload (LRO), and
Receive Side Scaling (RSS).
.Sh HARDWARE
The
.Nm
driver provides support for various NICs based on the QLogic NetXtreme II
family of 10Gb Ethernet controller chips, including the following:
.Pp
.Bl -bullet -compact
.It
QLogic NetXtreme II BCM57710 10Gb
.It
QLogic NetXtreme II BCM57711 10Gb
.It
QLogic NetXtreme II BCM57711E 10Gb
.It
QLogic NetXtreme II BCM57712 10Gb
.It
QLogic NetXtreme II BCM57712-MF 10Gb
.It
QLogic NetXtreme II BCM57800 10Gb
.It
QLogic NetXtreme II BCM57800-MF 10Gb
.It
QLogic NetXtreme II BCM57810 10Gb
.It
QLogic NetXtreme II BCM57810-MF 10Gb
.It
QLogic NetXtreme II BCM57840 10Gb / 20Gb
.It
QLogic NetXtreme II BCM57840-MF 10Gb
.El
.Sh CONFIGURATION
There a number of configuration parameters that can be set to tweak the
driver's behavior.
These parameters can be set via the
.Xr loader.conf 5
file to take affect during the next system boot.
The following parameters affect
ALL instances of the driver.
.Bl -tag -width indent
.It Va hw.bxe.debug
DEFAULT = 0
.br
Sets the default logging level of the driver.
See the Diagnostics and Debugging
section below for more details.
.It Va hw.bxe.interrupt_mode
DEFAULT = 2
.br
Sets the default interrupt mode: 0=IRQ, 1=MSI, 2=MSIX.
If set to MSIX and
allocation fails, the driver will roll back and attempt MSI allocation.
If MSI
allocation fails, the driver will roll back and attempt fixed level IRQ
allocation.
If IRQ allocation fails, then the driver load fails.
With MSI/MSIX,
the driver attempts to allocate a vector for each queue in addition to one more
for default processing.
.It Va hw.bxe.queue_count
DEFAULT = 4
.br
Sets the default number of fast path packet processing queues.
Note that one
MSI/MSIX interrupt vector is allocated per-queue.
.It Va hw.bxe.max_rx_bufs
DEFAULT = 0
.br
Sets the maximum number of receive buffers to allocate per-queue.
Zero(0) means
to allocate a receive buffer for every buffer descriptor.
By default this
equates to 4080 buffers per-queue which is the maximum value for this config
parameter.
.It Va hw.bxe.hc_rx_ticks
DEFAULT = 25
.br
Sets the number of ticks for host interrupt coalescing in the receive path.
.It Va hw.bxe.hc_tx_ticks
DEFAULT = 50
.br
Sets the number of ticks for host interrupt coalescing in the transmit path.
.It Va hw.bxe.rx_budget
DEFAULT = 0xffffffff
.br
Sets the maximum number of receive packets to process in an interrupt.
If the
budget is reached then the remaining/pending packets will be processed in a
scheduled taskqueue.
.It Va hw.bxe.max_aggregation_size
DEFAULT = 32768
.br
Sets the maximum LRO aggregration byte size.
The higher the value the more
packets the hardware will aggregate.
Maximum is 65K.
.It Va hw.bxe.mrrs
DEFAULT = -1
.br
Sets the PCI MRRS: -1=Auto, 0=128B, 1=256B, 2=512B, 3=1KB
.It Va hw.bxe.autogreeen
DEFAULT = 0
.br
Set AutoGrEEEN: 0=HW_DEFAULT, 1=FORCE_ON, 2=FORCE_OFF
.It Va hw.bxe.udp_rss
DEFAULT = 0
.br
Enable/Disable 4-tuple RSS for UDP: 0=DISABLED, 1=ENABLED
.El
.Pp
Special care must be taken when modifying the number of queues and receive
buffers.
FreeBSD imposes a limit on the maximum number of
.Xr mbuf 9
allocations.
If buffer allocations fail, the interface initialization will fail
and the interface will not be usable.
The driver does not make a best effort
for buffer allocations.
It is an all or nothing effort.
.Pp
You can tweak the
.Xr mbuf 9
allocation limit using
.Xr sysctl 8
and view the current usage with
.Xr netstat 1
as follows:
.Bd -literal -offset indent
# netstat -m
# sysctl kern.ipc.nmbclusters
# sysctl kern.ipc.nmbclusters=<#>
.Ed
.Pp
There are additional configuration parameters that can be set on a per-instance
basis to dynamically override the default configuration.
The '#' below must be
replaced with the driver instance / interface unit number:
.Bl -tag -width indent
.It Va dev.bxe.#.debug
DEFAULT = 0
.br
Sets the default logging level of the driver instance.
See
.Va hw.bxe.debug
above and
the Diagnostics and Debugging section below for more details.
.It Va dev.bxe.#.rx_budget
DEFAULT = 0xffffffff
.br
Sets the maximum number of receive packets to process in an interrupt for the
driver instance.
See
.Va hw.bxe.rx_budget
above for more details.
.El
.Pp
Additional items can be configured using
.Xr ifconfig 8 :
.Bl -tag -width indent
.It Va MTU - Maximum Transmission Unit
DEFAULT = 1500
.br
RANGE = 46-9184
.br
# ifconfig bxe# mtu <n>
.It Va Promiscuous Mode
DEFAULT = OFF
.br
# ifconfig bxe# [ promisc | -promisc ]
.It Va Rx/Tx Checksum Offload
DEFAULT = RX/TX CSUM ON
.br
Note that the Rx and Tx settings are not independent.
.br
# ifconfig bxe# [ rxcsum | -rxcsum | txcsum | -txcsum ]
.It Va TSO - TCP Segmentation Offload
DEFAULT = ON
.br
# ifconfig bxe# [ tso | -tso | tso6 | -tso6 ]
.It Va LRO - TCP Large Receive Offload
DEFAULT = ON
.br
# ifconfig bxe# [ lro | -lro ]
.El
.Sh DIAGNOSTICS AND DEBUGGING
There are many statistics exposed by
.Nm
via
.Xr sysctl 8 .
.Pp
To dump the default driver configuration:
.Bd -literal -offset indent
# sysctl -a | grep hw.bxe
.Ed
.Pp
To dump every instance's configuration and detailed statistics:
.Bd -literal -offset indent
# sysctl -a | grep dev.bxe
.Ed
.Pp
To dump information for a single instance (replace the '#' with the driver
instance / interface unit number):
.Bd -literal -offset indent
# sysctl -a | grep dev.bxe.#
.Ed
.Pp
To dump information for all the queues of a single instance:
.Bd -literal -offset indent
# sysctl -a | grep dev.bxe.#.queue
.Ed
.Pp
To dump information for a single queue of a single instance (replace the
additional '#' with the queue number):
.Bd -literal -offset indent
# sysctl -a | grep dev.bxe.#.queue.#
.Ed
.Pp
The
.Nm
driver has the ability to dump a ton of debug messages to the system
log.
The default level of logging can be set with the
.Va hw.bxe.debug
.Xr sysctl 8 .
Take care with this setting as it can result in too
many logs being dumped.
Since this parameter is the default one, it affects
every instance and will dramatically change the timing in the driver.
A better
alternative to aid in debugging is to dynamically change the debug level of a
specific instance with the
.Va dev.bxe.#.debug
.Xr sysctl 8 .
This allows
you to turn on/off logging of various debug groups on-the-fly.
.Pp
The different debug groups that can be toggled are:
.Bd -literal -offset indent
DBG_LOAD   0x00000001 /* load and unload    */
DBG_INTR   0x00000002 /* interrupt handling */
DBG_SP     0x00000004 /* slowpath handling  */
DBG_STATS  0x00000008 /* stats updates      */
DBG_TX     0x00000010 /* packet transmit    */
DBG_RX     0x00000020 /* packet receive     */
DBG_PHY    0x00000040 /* phy/link handling  */
DBG_IOCTL  0x00000080 /* ioctl handling     */
DBG_MBUF   0x00000100 /* dumping mbuf info  */
DBG_REGS   0x00000200 /* register access    */
DBG_LRO    0x00000400 /* lro processing     */
DBG_ASSERT 0x80000000 /* debug assert       */
DBG_ALL    0xFFFFFFFF /* flying monkeys     */
.Ed
.Pp
For example, to debug an issue in the receive path on bxe0:
.Bd -literal -offset indent
# sysctl dev.bxe.0.debug=0x22
.Ed
.Pp
When finished turn the logging back off:
.Bd -literal -offset indent
# sysctl dev.bxe.0.debug=0
.Ed
.Sh SUPPORT
For support questions please contact your QLogic approved reseller or
QLogic Technical Support at
.Pa http://support.qlogic.com ,
or by E-mail at
.Aq support@qlogic.com .
.Sh SEE ALSO
.Xr netstat 1 ,
.Xr altq 4 ,
.Xr arp 4 ,
.Xr netintro 4 ,
.Xr ng_ether 4 ,
.Xr vlan 4 ,
.Xr ifconfig 8
.Sh HISTORY
The
.Nm
device driver first appeared in
.Fx 9.0 .
.Sh AUTHORS
The
.Nm
driver was written by
.An Eric Davis Aq edavis@broadcom.com ,
.An David Christensen Aq davidch@broadcom.com ,
and
.An Gary Zambrano Aq zambrano@broadcom.com .