MFC 303522,303647,303860,303880,304168-304170,304479,304482,304485,305548,

305549:
Chelsio T4/T5 VF driver.

303522:
Various fixes to the t4/5nex character device.

- Remove null open/close methods.
- Don't set d_flags to 0 explicitly.
- Remove t5_cdevsw as the .d_name member isn't really used and doesn't
  warrant a separate cdevsw just for the name.
- Use ENOTTY as the error value for an unknown ioctl request.
- Use make_dev_s() to close race with setting si_drv1.

303647:
Store the offset of the KDOORBELL and GTS registers in the softc.

VF devices use a different register layout than PF devices.  Storing
the offset in a value in the softc allows code to be shared between the
PF and VF drivers.

303860:
Reserve an adapter flag IS_VF to mark VF devices vs PF devices.

303880:
Track the base absolute ID of ingress and egress queues.

Use this to map an absolute queue ID to a logical queue ID in interrupt
handlers.  For the regular cxgbe/cxl drivers this should be a no-op as
the base absolute ID should be zero.  VF devices have a non-zero base
absolute ID and require this change.  While here, export the absolute ID
of egress queues via a sysctl.

304168:
Make SGE parameter handling more VF-friendly.

Add fields to hold the SGE control register and free list buffer sizes to
the sge_params structure.  Populate these new fields in
t4_init_sge_params() for PF devices and change t4_read_chip_settings() to
pull these values out of the params structure instead of reading
registers directly.  This will permit t4_read_chip_settings() to be reused
for VF devices which cannot read SGE registers directly.

While here, move the call to t4_init_sge_params() to
get_params__post_init().  The VF driver will populate the SGE parameters
structure via a different method before calling t4_read_chip_settings().

304169:
Update mailbox writes to work with VF devices.

- Use alternate register locations for the data and control registers for
  VFs.
- Do a dummy read to force the writes to the  mailbox data registers to
  post before the write to the control register on VFs.
- Do not check the PCI-e firmware register for errors on VFs.

304170:
Add support for register dumps on VF devices.

- Add handling of VF register sets to t4_get_regs_len() and t4_get_regs().
- While here, use t4_get_regs_len() in the ioctl handler for regdump
  instead of inlining it.

304479:
Add structures for VF-specific adapter parameters.

While here, mark which parameters are PF-specific and which are
VF-specific.

304482:
Adjust t4_port_init() to work with VF devices.

Specifically, the FW_PORT_CMD may or may not work for a VF (the PF
driver can choose whether or not to permit access to this command),
so don't attempt to fetch port information on a VF if permission is
denied by the PF.

304485:
Reorder sysctls so that nodes shared with the VF driver are added first.

This permits a single early return for VF devices in the routines that
add sysctl nodes.

305548:
Don't break out of the m_advance() loop if len drops to zero.

If a packet contains the Ethernet header (14 bytes) in the first mbuf
and the payload (IP + UDP + data) in the second mbuf, then the attempt
to fetch the l3hdr will return a NULL pointer.  The first loop iteration
will drop len to zero and exit the loop without setting 'p'.  However,
the desired data is at the start of the second mbuf, so the correct
behavior is to loop around and let the conditional set 'p' to m_data of
the next mbuf (and leave offset as 0).

305549:
Chelsio T4/T5 VF driver.

The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters.  The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.

Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device.  It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.

t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.

t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.

VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages).  This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request.  In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices.  Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.

Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.

Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics.  In addition, TOE is not supported on VF devices, only for
the PF interfaces.

Sponsored by:	Chelsio Communications

21 files changed, 2365 insertions, 211 deletions

diff --git a/share/man/man4/Makefile b/share/man/man4/Makefile
index 10b73cf..74f5e70 100644
--- a/share/man/man4/Makefile
+++ b/share/man/man4/Makefile
@@ -107,6 +107,7 @@ MAN=	aac.4 \
 	cue.4 \
 	cxgb.4 \
 	cxgbe.4 \
+	cxgbev.4 \
 	cy.4 \
 	da.4 \
 	dc.4 \
@@ -585,6 +586,9 @@ MLINKS+=cxgb.4 if_cxgb.4
 MLINKS+=cxgbe.4 if_cxgbe.4 \
 	cxgbe.4 cxl.4 \
 	cxgbe.4 if_cxl.4
+MLINKS+=cxgbev.4 if_cxgbev.4 \
+	cxgbev.4 cxlv.4 \
+	cxgbev.4 if_cxlv.4
 MLINKS+=dc.4 if_dc.4
 MLINKS+=de.4 if_de.4
 MLINKS+=disc.4 if_disc.4
diff --git a/share/man/man4/cxgbe.4 b/share/man/man4/cxgbe.4
index 247e57a..c9d1de0 100644
--- a/share/man/man4/cxgbe.4
+++ b/share/man/man4/cxgbe.4
@@ -78,8 +78,7 @@ For more information on configuring this device, see
 .Sh HARDWARE
 The
 .Nm
-driver supports 40Gb, 10Gb and 1Gb Ethernet adapters based on the T5 ASIC
-(ports will be named cxl):
+driver supports 40Gb, 10Gb and 1Gb Ethernet adapters based on the T5 ASIC:
 .Pp
 .Bl -bullet -compact
 .It
@@ -319,6 +318,7 @@ email all the specific information related to the issue to
 .Xr altq 4 ,
 .Xr arp 4 ,
 .Xr cxgb 4 ,
+.Xr cxgbev 4 ,
 .Xr netintro 4 ,
 .Xr ng_ether 4 ,
 .Xr ifconfig 8
diff --git a/share/man/man4/cxgbev.4 b/share/man/man4/cxgbev.4
new file mode 100644
index 0000000..f4e3dc5
--- /dev/null
+++ b/share/man/man4/cxgbev.4
@@ -0,0 +1,290 @@
+.\" Copyright (c) 2011-2016, Chelsio Inc
+.\" All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions are met:
+.\"
+.\" 1. Redistributions of source code must retain the above copyright notice,
+.\"    this list of conditions and the following disclaimer.
+.\"
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" 3. Neither the name of the Chelsio Inc nor the names of its
+.\"    contributors may be used to endorse or promote products derived from
+.\"    this software without specific prior written permission.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+.\" POSSIBILITY OF SUCH DAMAGE.
+.\"
+.\" * Other names and brands may be claimed as the property of others.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd August 22, 2016
+.Dt CXGBEV 4
+.Os
+.Sh NAME
+.Nm cxgbev
+.Nd "Chelsio T4 and T5 based 40Gb, 10Gb, and 1Gb Ethernet VF driver"
+.Sh SYNOPSIS
+To compile this driver into the kernel,
+place the following lines in your
+kernel configuration file:
+.Bd -ragged -offset indent
+.Cd "device cxgbe"
+.Cd "device cxgbev"
+.Ed
+.Pp
+To load the driver as a
+module at boot time, place the following line in
+.Xr loader.conf 5 :
+.Bd -literal -offset indent
+if_cxgbev_load="YES"
+.Ed
+.Sh DESCRIPTION
+The
+.Nm
+driver provides support for Virtual Functions on PCI Express Ethernet adapters
+based on the Chelsio Terminator 4 and Terminator 5 ASICs (T4 and T5).
+The driver supports Jumbo Frames, Transmit/Receive checksum offload,
+TCP segmentation offload (TSO), Large Receive Offload (LRO), VLAN
+tag insertion/extraction, VLAN checksum offload, VLAN TSO, and
+Receive Side Steering (RSS).
+For further hardware information and questions related to hardware
+requirements, see
+.Pa http://www.chelsio.com/ .
+.Pp
+Note that ports of T5 VFs are named cxlv and attach to a t5vf parent device
+(in contrast to ports named cxgbev that attach to a t4vf parent for a T4 VF).
+Loader tunables with the hw.cxgbe prefix apply to both T4 and T5 VFs.
+The Physical Function driver for T4 and T5 adapters shares these tunables.
+The sysctl MIBs are at dev.t5vf and dev.cxlv for T5 cards and at dev.t4vf and
+dev.cxgbev for T4 cards.
+.Pp
+For more information on configuring this device, see
+.Xr ifconfig 8 .
+.Sh HARDWARE
+The
+.Nm
+driver supports Virtual Functions on 40Gb, 10Gb and 1Gb Ethernet adapters
+based on the T5 ASIC:
+.Pp
+.Bl -bullet -compact
+.It
+Chelsio T580-CR
+.It
+Chelsio T580-LP-CR
+.It
+Chelsio T580-LP-SO-CR
+.It
+Chelsio T560-CR
+.It
+Chelsio T540-CR
+.It
+Chelsio T540-LP-CR
+.It
+Chelsio T522-CR
+.It
+Chelsio T520-LL-CR
+.It
+Chelsio T520-CR
+.It
+Chelsio T520-SO
+.It
+Chelsio T520-BT
+.It
+Chelsio T504-BT
+.El
+.Pp
+The
+.Nm
+driver supports Virtual Functions on 10Gb and 1Gb Ethernet adapters based
+on the T4 ASIC:
+.Pp
+.Bl -bullet -compact
+.It
+Chelsio T420-CR
+.It
+Chelsio T422-CR
+.It
+Chelsio T440-CR
+.It
+Chelsio T420-BCH
+.It
+Chelsio T440-BCH
+.It
+Chelsio T440-CH
+.It
+Chelsio T420-SO
+.It
+Chelsio T420-CX
+.It
+Chelsio T420-BT
+.It
+Chelsio T404-BT
+.El
+.Sh LOADER TUNABLES
+Tunables can be set at the
+.Xr loader 8
+prompt before booting the kernel or stored in
+.Xr loader.conf 5 .
+.Bl -tag -width indent
+.It Va hw.cxgbe.ntxq10g
+The number of tx queues to use for a 10Gb or 40Gb port.
+The default is 16 or the number
+of CPU cores in the system, whichever is less.
+.It Va hw.cxgbe.nrxq10g
+The number of rx queues to use for a 10Gb or 40Gb port.
+The default is 8 or the number
+of CPU cores in the system, whichever is less.
+.It Va hw.cxgbe.ntxq1g
+The number of tx queues to use for a 1Gb port.
+The default is 4 or the number
+of CPU cores in the system, whichever is less.
+.It Va hw.cxgbe.nrxq1g
+The number of rx queues to use for a 1Gb port.
+The default is 2 or the number
+of CPU cores in the system, whichever is less.
+.It Va hw.cxgbe.holdoff_timer_idx_10G
+.It Va hw.cxgbe.holdoff_timer_idx_1G
+The timer index value to use to delay interrupts.
+The holdoff timer list has the values 1, 5, 10, 50, 100, and 200
+by default (all values are in microseconds) and the index selects a
+value from this list.
+The default value is 1 which means the timer value is 5us.
+Different interfaces can be assigned different values at any time via the
+dev.cxgbev.X.holdoff_tmr_idx or dev.cxlv.X.holdoff_tmr_idx sysctl.
+.It Va hw.cxgbe.holdoff_pktc_idx_10G
+.It Va hw.cxgbe.holdoff_pktc_idx_1G
+The packet-count index value to use to delay interrupts.
+The packet-count list has the values 1, 8, 16, and 32 by default
+and the index selects a value from this list.
+The default value is -1 which means packet counting is disabled and interrupts
+are generated based solely on the holdoff timer value.
+Different interfaces can be assigned different values via the
+dev.cxgbev.X.holdoff_pktc_idx or dev.cxlv.X.holdoff_pktc_idx sysctl.
+This sysctl works only when the interface has never been marked up (as done by
+ifconfig up).
+.It Va hw.cxgbe.qsize_txq
+The size, in number of entries, of the descriptor ring used for a tx
+queue.
+A buf_ring of the same size is also allocated for additional
+software queuing.
+See
+.Xr ifnet 9 .
+The default value is 1024.
+Different interfaces can be assigned different values via the
+dev.cxgbev.X.qsize_txq sysctl or dev.cxlv.X.qsize_txq sysctl.
+This sysctl works only when the interface has never been marked up (as done by
+ifconfig up).
+.It Va hw.cxgbe.qsize_rxq
+The size, in number of entries, of the descriptor ring used for an
+rx queue.
+The default value is 1024.
+Different interfaces can be assigned different values via the
+dev.cxgbev.X.qsize_rxq or dev.cxlv.X.qsize_rxq sysctl.
+This sysctl works only when the interface has never been marked up (as done by
+ifconfig up).
+.It Va hw.cxgbe.interrupt_types
+The interrupt types that the driver is allowed to use.
+Bit 0 represents INTx (line interrupts), bit 1 MSI, bit 2 MSI-X.
+The default is 7 (all allowed).
+The driver will select the best possible type out of the allowed types by
+itself.
+.It Va hw.cxgbe.fl_pktshift
+The number of bytes of padding inserted before the beginning of an Ethernet
+frame in the receive buffer.
+The default value of 2 ensures that the Ethernet payload (usually the IP header)
+is at a 4 byte aligned address.
+0-7 are all valid values.
+.It Va hw.cxgbe.fl_pad
+A non-zero value ensures that writes from the hardware to a receive buffer are
+padded up to the specified boundary.
+The default is -1 which lets the driver pick a pad boundary.
+0 disables trailer padding completely.
+.It Va hw.cxgbe.buffer_packing
+Allow the hardware to deliver multiple frames in the same receive buffer
+opportunistically.
+The default is -1 which lets the driver decide.
+0 or 1 explicitly disable or enable this feature.
+.It Va hw.cxgbe.allow_mbufs_in_cluster
+1 allows the driver to lay down one or more mbufs within the receive buffer
+opportunistically.
+This is the default.
+0 prohibits the driver from doing so.
+.It Va hw.cxgbe.largest_rx_cluster
+.It Va hw.cxgbe.safest_rx_cluster
+Sizes of rx clusters.
+Each of these must be set to one of the sizes available
+(usually 2048, 4096, 9216, and 16384) and largest_rx_cluster must be greater
+than or equal to safest_rx_cluster.
+The defaults are 16384 and 4096 respectively.
+The driver will never attempt to allocate a receive buffer larger than
+largest_rx_cluster and will fall back to allocating buffers of
+safest_rx_cluster size if an allocation larger than safest_rx_cluster fails.
+Note that largest_rx_cluster merely establishes a ceiling -- the driver is
+allowed to allocate buffers of smaller sizes.
+.El
+.Pp
+Certain settings and resources for Virtual Functions are dictated
+by the parent Physical Function driver.
+For example, the Physical Function driver limits the number of queues a
+Virtual Function is permitted to use.
+Some of these limits can be adjusted in the firmware configuration file
+used with the Physical Function driver.
+.Pp
+The PAUSE settings on the port of a Virtual Function are inherited from
+the settings of the same port on the Physical Function.
+Virtual Functions cannot modify the setting and track changes made to
+the associated port's setting by the Physical Function driver.
+.Pp
+Receive queues on a Virtual Function always drop packets in response to
+congestion
+.Po
+equivalent to setting
+.Va hw.cxgbe.cong_drop
+to 1
+.Pc .
+.Pp
+The VF driver currently depends on the PF driver.
+As a result, loading the VF driver will also load the PF driver as a
+dependency.
+.Sh SUPPORT
+For general information and support,
+go to the Chelsio support website at:
+.Pa http://www.chelsio.com/ .
+.Pp
+If an issue is identified with this driver with a supported adapter,
+email all the specific information related to the issue to
+.Aq Mt support@chelsio.com .
+.Sh SEE ALSO
+.Xr altq 4 ,
+.Xr arp 4 ,
+.Xr cxgbe 4 ,
+.Xr netintro 4 ,
+.Xr ng_ether 4 ,
+.Xr ifconfig 8
+.Sh HISTORY
+The
+.Nm
+device driver first appeared in
+.Fx 12.0 .
+.Sh AUTHORS
+.An -nosplit
+The
+.Nm
+driver was written by
+.An Navdeep Parhar Aq Mt np@FreeBSD.org
+and
+.An John Baldwin Aq Mt jhb@FreeBSD.org .
diff --git a/sys/arm/conf/NOTES b/sys/arm/conf/NOTES
index 85a4c16..e987477 100644
--- a/sys/arm/conf/NOTES
+++ b/sys/arm/conf/NOTES
@@ -83,6 +83,6 @@ nodevice	star_saver
 nodevice	warp_saver
 
 nodevice	cxgbe
-nodevice	pcii
+nodevice	cxgbev
 nodevice	snd_cmi
 nodevice	tnt4882
diff --git a/sys/conf/NOTES b/sys/conf/NOTES
index 30d6d65..58b71f6 100644
--- a/sys/conf/NOTES
+++ b/sys/conf/NOTES
@@ -1923,6 +1923,7 @@ device		xmphy		# XaQti XMAC II
 #	(and SMC COM90c66 in '56 compatibility mode) adapters.
 # cxgb: Chelsio T3 based 1GbE/10GbE PCIe Ethernet adapters.
 # cxgbe:Chelsio T4 and T5 based 1GbE/10GbE/40GbE PCIe Ethernet adapters.
+# cxgbev: Chelsio T4 and T5 based PCIe Virtual Functions.
 # dc:   Support for PCI fast ethernet adapters based on the DEC/Intel 21143
 #       and various workalikes including:
 #       the ADMtek AL981 Comet and AN985 Centaur, the ASIX Electronics
@@ -2104,6 +2105,7 @@ device		wb		# Winbond W89C840F
 device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')
 
 # PCI Ethernet NICs.
+device		cxgbev		# Chelsio T4 and T5 1GbE/10GbE/40GbE VF
 device		de		# DEC/Intel DC21x4x (``Tulip'')
 device		em		# Intel Pro/1000 Gigabit Ethernet
 device		igb		# Intel Pro/1000 PCIE Gigabit Ethernet
diff --git a/sys/conf/files b/sys/conf/files
index 52ad217..0e5d126 100644
--- a/sys/conf/files
+++ b/sys/conf/files
@@ -1156,8 +1156,12 @@ dev/cxgbe/t4_l2t.c		optional cxgbe pci \
 	compile-with "${NORMAL_C} -I$S/dev/cxgbe"
 dev/cxgbe/t4_tracer.c		optional cxgbe pci \
 	compile-with "${NORMAL_C} -I$S/dev/cxgbe"
+dev/cxgbe/t4_vf.c		optional cxgbev pci \
+	compile-with "${NORMAL_C} -I$S/dev/cxgbe"
 dev/cxgbe/common/t4_hw.c	optional cxgbe pci \
 	compile-with "${NORMAL_C} -I$S/dev/cxgbe"
+dev/cxgbe/common/t4vf_hw.c	optional cxgbev pci \
+	compile-with "${NORMAL_C} -I$S/dev/cxgbe"
 t4fw_cfg.c		optional cxgbe					\
 	compile-with	"${AWK} -f $S/tools/fw_stub.awk t4fw_cfg.fw:t4fw_cfg t4fw_cfg_uwire.fw:t4fw_cfg_uwire t4fw.fw:t4fw -mt4fw_cfg -c${.TARGET}" \
 	no-implicit-rule before-depend local				\
diff --git a/sys/dev/cxgbe/adapter.h b/sys/dev/cxgbe/adapter.h
index 7ba5553..451630f 100644
--- a/sys/dev/cxgbe/adapter.h
+++ b/sys/dev/cxgbe/adapter.h
@@ -189,6 +189,7 @@ enum {
 	ADAP_SYSCTL_CTX	= (1 << 4),
 	/* TOM_INIT_DONE= (1 << 5),	No longer used */
 	BUF_PACKING_OK	= (1 << 6),
+	IS_VF		= (1 << 7),
 
 	CXGBE_BUSY	= (1 << 9),
 
@@ -422,6 +423,7 @@ enum {DOORBELL_UDB, DOORBELL_WCWR, DOORBELL_UDBWC, DOORBELL_KDB};
 struct sge_eq {
 	unsigned int flags;	/* MUST be first */
 	unsigned int cntxt_id;	/* SGE context id for the eq */
+	unsigned int abs_id;	/* absolute SGE id for the eq */
 	struct mtx eq_lock;
 
 	struct tx_desc *desc;	/* KVA of descriptor ring */
@@ -730,8 +732,10 @@ struct sge {
 	struct sge_nm_txq *nm_txq;	/* netmap tx queues */
 	struct sge_nm_rxq *nm_rxq;	/* netmap rx queues */
 
-	uint16_t iq_start;
-	int eq_start;
+	uint16_t iq_start;	/* first cntxt_id */
+	uint16_t iq_base;	/* first abs_id */
+	int eq_start;		/* first cntxt_id */
+	int eq_base;		/* first abs_id */
 	struct sge_iq **iqmap;	/* iq->cntxt_id to iq mapping */
 	struct sge_eq **eqmap;	/* eq->cntxt_id to eq mapping */
 
@@ -774,6 +778,8 @@ struct adapter {
 		struct sge_rxq *rxq;
 		struct sge_nm_rxq *nm_rxq;
 	} __aligned(CACHE_LINE_SIZE) *irq;
+	int sge_gts_reg;
+	int sge_kdoorbell_reg;
 
 	bus_dma_tag_t dmat;	/* Parent DMA tag */
 
@@ -931,6 +937,9 @@ struct adapter {
 /* One for errors, one for firmware events */
 #define T4_EXTRA_INTR 2
 
+/* One for firmware events */
+#define T4VF_EXTRA_INTR 1
+
 static inline uint32_t
 t4_read_reg(struct adapter *sc, uint32_t reg)
 {
@@ -1065,13 +1074,34 @@ t4_use_ldst(struct adapter *sc)
 }
 
 /* t4_main.c */
+extern int t4_ntxq10g;
+extern int t4_nrxq10g;
+extern int t4_ntxq1g;
+extern int t4_nrxq1g;
+extern int t4_intr_types;
+extern int t4_tmr_idx_10g;
+extern int t4_pktc_idx_10g;
+extern int t4_tmr_idx_1g;
+extern int t4_pktc_idx_1g;
+extern unsigned int t4_qsize_rxq;
+extern unsigned int t4_qsize_txq;
+extern device_method_t cxgbe_methods[];
+
 int t4_os_find_pci_capability(struct adapter *, int);
 int t4_os_pci_save_state(struct adapter *);
 int t4_os_pci_restore_state(struct adapter *);
 void t4_os_portmod_changed(const struct adapter *, int);
 void t4_os_link_changed(struct adapter *, int, int, int);
 void t4_iterate(void (*)(struct adapter *, void *), void *);
+void t4_add_adapter(struct adapter *);
+int t4_detach_common(device_t);
 int t4_filter_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
+int t4_map_bars_0_and_4(struct adapter *);
+int t4_map_bar_2(struct adapter *);
+int t4_set_sched_class(struct adapter *, struct t4_sched_params *);
+int t4_set_sched_queue(struct adapter *, struct t4_sched_queue *);
+int t4_setup_intr_handlers(struct adapter *);
+void t4_sysctls(struct adapter *);
 int begin_synchronized_op(struct adapter *, struct vi_info *, int, char *);
 void doom_vi(struct adapter *, struct vi_info *);
 void end_synchronized_op(struct adapter *, int);
@@ -1112,7 +1142,7 @@ void t4_intr_err(void *);
 void t4_intr_evt(void *);
 void t4_wrq_tx_locked(struct adapter *, struct sge_wrq *, struct wrqe *);
 void t4_update_fl_bufsize(struct ifnet *);
-int parse_pkt(struct mbuf **);
+int parse_pkt(struct adapter *, struct mbuf **);
 void *start_wrq_wr(struct sge_wrq *, int, struct wrq_cookie *);
 void commit_wrq_wr(struct sge_wrq *, void *, struct wrq_cookie *);
 int tnl_cong(struct port_info *, int);
diff --git a/sys/dev/cxgbe/common/common.h b/sys/dev/cxgbe/common/common.h
index cc6edf3..8a738ed 100644
--- a/sys/dev/cxgbe/common/common.h
+++ b/sys/dev/cxgbe/common/common.h
@@ -215,6 +215,8 @@ struct sge_params {
 	int pad_boundary;
 	int pack_boundary;
 	int fl_pktshift;
+	u32 sge_control;
+	u32 sge_fl_buffer_size[SGE_FLBUF_SIZES];
 };
 
 struct tp_params {
@@ -278,12 +280,52 @@ struct chip_params {
 	u16 mps_tcam_size;
 };
 
+/* VF-only parameters. */
+
+/*
+ * Global Receive Side Scaling (RSS) parameters in host-native format.
+ */
+struct rss_params {
+	unsigned int mode;		/* RSS mode */
+	union {
+	    struct {
+		u_int synmapen:1;	/* SYN Map Enable */
+		u_int syn4tupenipv6:1;	/* enable hashing 4-tuple IPv6 SYNs */
+		u_int syn2tupenipv6:1;	/* enable hashing 2-tuple IPv6 SYNs */
+		u_int syn4tupenipv4:1;	/* enable hashing 4-tuple IPv4 SYNs */
+		u_int syn2tupenipv4:1;	/* enable hashing 2-tuple IPv4 SYNs */
+		u_int ofdmapen:1;	/* Offload Map Enable */
+		u_int tnlmapen:1;	/* Tunnel Map Enable */
+		u_int tnlalllookup:1;	/* Tunnel All Lookup */
+		u_int hashtoeplitz:1;	/* use Toeplitz hash */
+	    } basicvirtual;
+	} u;
+};
+
+/*
+ * Maximum resources provisioned for a PCI VF.
+ */
+struct vf_resources {
+	unsigned int nvi;		/* N virtual interfaces */
+	unsigned int neq;		/* N egress Qs */
+	unsigned int nethctrl;		/* N egress ETH or CTRL Qs */
+	unsigned int niqflint;		/* N ingress Qs/w free list(s) & intr */
+	unsigned int niq;		/* N ingress Qs */
+	unsigned int tc;		/* PCI-E traffic class */
+	unsigned int pmask;		/* port access rights mask */
+	unsigned int nexactf;		/* N exact MPS filters */
+	unsigned int r_caps;		/* read capabilities */
+	unsigned int wx_caps;		/* write/execute capabilities */
+};
+
 struct adapter_params {
 	struct sge_params sge;
-	struct tp_params  tp;
+	struct tp_params  tp;		/* PF-only */
 	struct vpd_params vpd;
 	struct pci_params pci;
-	struct devlog_params devlog;
+	struct devlog_params devlog;	/* PF-only */
+	struct rss_params rss;		/* VF-only */
+	struct vf_resources vfres;	/* VF-only */
 
 	unsigned int sf_size;             /* serial flash size in bytes */
 	unsigned int sf_nsec;             /* # of flash sectors */
@@ -509,6 +551,7 @@ int t4_get_fw_version(struct adapter *adapter, u32 *vers);
 int t4_get_tp_version(struct adapter *adapter, u32 *vers);
 int t4_get_exprom_version(struct adapter *adapter, u32 *vers);
 int t4_init_hw(struct adapter *adapter, u32 fw_params);
+const struct chip_params *t4_get_chip_params(int chipid);
 int t4_prep_adapter(struct adapter *adapter, u8 *buf);
 int t4_shutdown_adapter(struct adapter *adapter);
 int t4_init_devlog_params(struct adapter *adapter, int fw_attach);
@@ -716,4 +759,32 @@ int t4_config_watchdog(struct adapter *adapter, unsigned int mbox,
 int t4_get_devlog_level(struct adapter *adapter, unsigned int *level);
 int t4_set_devlog_level(struct adapter *adapter, unsigned int level);
 void t4_sge_decode_idma_state(struct adapter *adapter, int state);
+
+static inline int t4vf_query_params(struct adapter *adapter,
+				    unsigned int nparams, const u32 *params,
+				    u32 *vals)
+{
+	return t4_query_params(adapter, 0, 0, 0, nparams, params, vals);
+}
+
+static inline int t4vf_set_params(struct adapter *adapter,
+				  unsigned int nparams, const u32 *params,
+				  const u32 *vals)
+{
+	return t4_set_params(adapter, 0, 0, 0, nparams, params, vals);
+}
+
+static inline int t4vf_wr_mbox(struct adapter *adap, const void *cmd,
+			       int size, void *rpl)
+{
+	return t4_wr_mbox(adap, adap->mbox, cmd, size, rpl);
+}
+
+int t4vf_wait_dev_ready(struct adapter *adapter);
+int t4vf_fw_reset(struct adapter *adapter);
+int t4vf_get_sge_params(struct adapter *adapter);
+int t4vf_get_rss_glb_config(struct adapter *adapter);
+int t4vf_get_vfres(struct adapter *adapter);
+int t4vf_prep_adapter(struct adapter *adapter);
+
 #endif /* __CHELSIO_COMMON_H */
diff --git a/sys/dev/cxgbe/common/t4_hw.c b/sys/dev/cxgbe/common/t4_hw.c
index 580cdc5..35dd4e2 100644
--- a/sys/dev/cxgbe/common/t4_hw.c
+++ b/sys/dev/cxgbe/common/t4_hw.c
@@ -286,6 +286,14 @@ int t4_wr_mbox_meat_timeout(struct adapter *adap, int mbox, const void *cmd,
 	if ((size & 15) || size > MBOX_LEN)
 		return -EINVAL;
 
+	if (adap->flags & IS_VF) {
+		if (is_t6(adap))
+			data_reg = FW_T6VF_MBDATA_BASE_ADDR;
+		else
+			data_reg = FW_T4VF_MBDATA_BASE_ADDR;
+		ctl_reg = VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_CTRL);
+	}
+
 	/*
 	 * If we have a negative timeout, that implies that we can't sleep.
 	 */
@@ -340,6 +348,22 @@ int t4_wr_mbox_meat_timeout(struct adapter *adap, int mbox, const void *cmd,
 	for (i = 0; i < size; i += 8, p++)
 		t4_write_reg64(adap, data_reg + i, be64_to_cpu(*p));
 
+	if (adap->flags & IS_VF) {
+		/*
+		 * For the VFs, the Mailbox Data "registers" are
+		 * actually backed by T4's "MA" interface rather than
+		 * PL Registers (as is the case for the PFs).  Because
+		 * these are in different coherency domains, the write
+		 * to the VF's PL-register-backed Mailbox Control can
+		 * race in front of the writes to the MA-backed VF
+		 * Mailbox Data "registers".  So we need to do a
+		 * read-back on at least one byte of the VF Mailbox
+		 * Data registers before doing the write to the VF
+		 * Mailbox Control register.
+		 */
+		t4_read_reg(adap, data_reg);
+	}
+
 	CH_DUMP_MBOX(adap, mbox, data_reg);
 
 	t4_write_reg(adap, ctl_reg, F_MBMSGVALID | V_MBOWNER(X_MBOWNER_FW));
@@ -352,10 +376,13 @@ int t4_wr_mbox_meat_timeout(struct adapter *adap, int mbox, const void *cmd,
 	 * Loop waiting for the reply; bail out if we time out or the firmware
 	 * reports an error.
 	 */
-	for (i = 0;
-	     !((pcie_fw = t4_read_reg(adap, A_PCIE_FW)) & F_PCIE_FW_ERR) &&
-	     i < timeout;
-	     i += ms) {
+	pcie_fw = 0;
+	for (i = 0; i < timeout; i += ms) {
+		if (!(adap->flags & IS_VF)) {
+			pcie_fw = t4_read_reg(adap, A_PCIE_FW);
+			if (pcie_fw & F_PCIE_FW_ERR)
+				break;
+		}
 		if (sleep_ok) {
 			ms = delay[delay_idx];  /* last element may repeat */
 			if (delay_idx < ARRAY_SIZE(delay) - 1)
@@ -695,10 +722,14 @@ unsigned int t4_get_regs_len(struct adapter *adapter)
 
 	switch (chip_version) {
 	case CHELSIO_T4:
+		if (adapter->flags & IS_VF)
+			return FW_T4VF_REGMAP_SIZE;
 		return T4_REGMAP_SIZE;
 
 	case CHELSIO_T5:
 	case CHELSIO_T6:
+		if (adapter->flags & IS_VF)
+			return FW_T4VF_REGMAP_SIZE;
 		return T5_REGMAP_SIZE;
 	}
 
@@ -1177,6 +1208,18 @@ void t4_get_regs(struct adapter *adap, u8 *buf, size_t buf_size)
 		0x27e00, 0x27e04,
 	};
 
+	static const unsigned int t4vf_reg_ranges[] = {
+		VF_SGE_REG(A_SGE_VF_KDOORBELL), VF_SGE_REG(A_SGE_VF_GTS),
+		VF_MPS_REG(A_MPS_VF_CTL),
+		VF_MPS_REG(A_MPS_VF_STAT_RX_VF_ERR_FRAMES_H),
+		VF_PL_REG(A_PL_VF_WHOAMI), VF_PL_REG(A_PL_VF_WHOAMI),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_CTRL),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_STATUS),
+		FW_T4VF_MBDATA_BASE_ADDR,
+		FW_T4VF_MBDATA_BASE_ADDR +
+		((NUM_CIM_PF_MAILBOX_DATA_INSTANCES - 1) * 4),
+	};
+
 	static const unsigned int t5_reg_ranges[] = {
 		0x1008, 0x10c0,
 		0x10cc, 0x10f8,
@@ -1952,6 +1995,18 @@ void t4_get_regs(struct adapter *adap, u8 *buf, size_t buf_size)
 		0x51300, 0x51308,
 	};
 
+	static const unsigned int t5vf_reg_ranges[] = {
+		VF_SGE_REG(A_SGE_VF_KDOORBELL), VF_SGE_REG(A_SGE_VF_GTS),
+		VF_MPS_REG(A_MPS_VF_CTL),
+		VF_MPS_REG(A_MPS_VF_STAT_RX_VF_ERR_FRAMES_H),
+		VF_PL_REG(A_PL_VF_WHOAMI), VF_PL_REG(A_PL_VF_REVISION),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_CTRL),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_STATUS),
+		FW_T4VF_MBDATA_BASE_ADDR,
+		FW_T4VF_MBDATA_BASE_ADDR +
+		((NUM_CIM_PF_MAILBOX_DATA_INSTANCES - 1) * 4),
+	};
+
 	static const unsigned int t6_reg_ranges[] = {
 		0x1008, 0x101c,
 		0x1024, 0x10a8,
@@ -2529,6 +2584,18 @@ void t4_get_regs(struct adapter *adap, u8 *buf, size_t buf_size)
 		0x51300, 0x51324,
 	};
 
+	static const unsigned int t6vf_reg_ranges[] = {
+		VF_SGE_REG(A_SGE_VF_KDOORBELL), VF_SGE_REG(A_SGE_VF_GTS),
+		VF_MPS_REG(A_MPS_VF_CTL),
+		VF_MPS_REG(A_MPS_VF_STAT_RX_VF_ERR_FRAMES_H),
+		VF_PL_REG(A_PL_VF_WHOAMI), VF_PL_REG(A_PL_VF_REVISION),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_CTRL),
+		VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_STATUS),
+		FW_T6VF_MBDATA_BASE_ADDR,
+		FW_T6VF_MBDATA_BASE_ADDR +
+		((NUM_CIM_PF_MAILBOX_DATA_INSTANCES - 1) * 4),
+	};
+
 	u32 *buf_end = (u32 *)(buf + buf_size);
 	const unsigned int *reg_ranges;
 	int reg_ranges_size, range;
@@ -2540,18 +2607,33 @@ void t4_get_regs(struct adapter *adap, u8 *buf, size_t buf_size)
 	 */
 	switch (chip_version) {
 	case CHELSIO_T4:
-		reg_ranges = t4_reg_ranges;
-		reg_ranges_size = ARRAY_SIZE(t4_reg_ranges);
+		if (adap->flags & IS_VF) {
+			reg_ranges = t4vf_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t4vf_reg_ranges);
+		} else {
+			reg_ranges = t4_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t4_reg_ranges);
+		}
 		break;
 
 	case CHELSIO_T5:
-		reg_ranges = t5_reg_ranges;
-		reg_ranges_size = ARRAY_SIZE(t5_reg_ranges);
+		if (adap->flags & IS_VF) {
+			reg_ranges = t5vf_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t5vf_reg_ranges);
+		} else {
+			reg_ranges = t5_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t5_reg_ranges);
+		}
 		break;
 
 	case CHELSIO_T6:
-		reg_ranges = t6_reg_ranges;
-		reg_ranges_size = ARRAY_SIZE(t6_reg_ranges);
+		if (adap->flags & IS_VF) {
+			reg_ranges = t6vf_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t6vf_reg_ranges);
+		} else {
+			reg_ranges = t6_reg_ranges;
+			reg_ranges_size = ARRAY_SIZE(t6_reg_ranges);
+		}
 		break;
 
 	default:
@@ -7413,7 +7495,7 @@ static void set_pcie_completion_timeout(struct adapter *adapter,
 	}
 }
 
-static const struct chip_params *get_chip_params(int chipid)
+const struct chip_params *t4_get_chip_params(int chipid)
 {
 	static const struct chip_params chip_params[] = {
 		{
@@ -7492,7 +7574,7 @@ int t4_prep_adapter(struct adapter *adapter, u8 *buf)
 		}
 	}
 
-	adapter->chip_params = get_chip_params(chip_id(adapter));
+	adapter->chip_params = t4_get_chip_params(chip_id(adapter));
 	if (adapter->chip_params == NULL)
 		return -EINVAL;
 
@@ -7641,6 +7723,7 @@ int t4_init_sge_params(struct adapter *adapter)
 {
 	u32 r;
 	struct sge_params *sp = &adapter->params.sge;
+	unsigned i;
 
 	r = t4_read_reg(adapter, A_SGE_INGRESS_RX_THRESHOLD);
 	sp->counter_val[0] = G_THRESHOLD_0(r);
@@ -7683,6 +7766,7 @@ int t4_init_sge_params(struct adapter *adapter)
 	sp->page_shift = (r & M_HOSTPAGESIZEPF0) + 10;
 
 	r = t4_read_reg(adapter, A_SGE_CONTROL);
+	sp->sge_control = r;
 	sp->spg_len = r & F_EGRSTATUSPAGESIZE ? 128 : 64;
 	sp->fl_pktshift = G_PKTSHIFT(r);
 	sp->pad_boundary = 1 << (G_INGPADBOUNDARY(r) + 5);
@@ -7695,6 +7779,9 @@ int t4_init_sge_params(struct adapter *adapter)
 		else
 			sp->pack_boundary = 1 << (G_INGPACKBOUNDARY(r) + 5);
 	}
+	for (i = 0; i < SGE_FLBUF_SIZES; i++)
+		sp->sge_fl_buffer_size[i] = t4_read_reg(adapter,
+		    A_SGE_FL_BUFFER_SIZE0 + (4 * i));
 
 	return 0;
 }
@@ -7848,15 +7935,26 @@ int t4_port_init(struct adapter *adap, int mbox, int pf, int vf, int port_id)
 		} while ((adap->params.portvec & (1 << j)) == 0);
 	}
 
-	c.op_to_portid = htonl(V_FW_CMD_OP(FW_PORT_CMD) |
-			       F_FW_CMD_REQUEST | F_FW_CMD_READ |
-			       V_FW_PORT_CMD_PORTID(j));
-	c.action_to_len16 = htonl(
-		V_FW_PORT_CMD_ACTION(FW_PORT_ACTION_GET_PORT_INFO) |
-		FW_LEN16(c));
-	ret = t4_wr_mbox(adap, mbox, &c, sizeof(c), &c);
-	if (ret)
-		return ret;
+	if (!(adap->flags & IS_VF) ||
+	    adap->params.vfres.r_caps & FW_CMD_CAP_PORT) {
+		c.op_to_portid = htonl(V_FW_CMD_OP(FW_PORT_CMD) |
+				       F_FW_CMD_REQUEST | F_FW_CMD_READ |
+				       V_FW_PORT_CMD_PORTID(j));
+		c.action_to_len16 = htonl(
+			V_FW_PORT_CMD_ACTION(FW_PORT_ACTION_GET_PORT_INFO) |
+			FW_LEN16(c));
+		ret = t4_wr_mbox(adap, mbox, &c, sizeof(c), &c);
+		if (ret)
+			return ret;
+
+		ret = be32_to_cpu(c.u.info.lstatus_to_modtype);
+		p->mdio_addr = (ret & F_FW_PORT_CMD_MDIOCAP) ?
+			G_FW_PORT_CMD_MDIOADDR(ret) : -1;
+		p->port_type = G_FW_PORT_CMD_PTYPE(ret);
+		p->mod_type = G_FW_PORT_CMD_MODTYPE(ret);
+
+		init_link_config(&p->link_cfg, be16_to_cpu(c.u.info.pcap));
+	}
 
 	ret = t4_alloc_vi(adap, mbox, j, pf, vf, 1, addr, &rss_size);
 	if (ret < 0)
@@ -7869,14 +7967,6 @@ int t4_port_init(struct adapter *adap, int mbox, int pf, int vf, int port_id)
 	p->vi[0].rss_size = rss_size;
 	t4_os_set_hw_addr(adap, p->port_id, addr);
 
-	ret = be32_to_cpu(c.u.info.lstatus_to_modtype);
-	p->mdio_addr = (ret & F_FW_PORT_CMD_MDIOCAP) ?
-		G_FW_PORT_CMD_MDIOADDR(ret) : -1;
-	p->port_type = G_FW_PORT_CMD_PTYPE(ret);
-	p->mod_type = G_FW_PORT_CMD_MODTYPE(ret);
-
-	init_link_config(&p->link_cfg, be16_to_cpu(c.u.info.pcap));
-
 	param = V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_DEV) |
 	    V_FW_PARAMS_PARAM_X(FW_PARAMS_PARAM_DEV_RSSINFO) |
 	    V_FW_PARAMS_PARAM_YZ(p->vi[0].viid);
diff --git a/sys/dev/cxgbe/common/t4vf_hw.c b/sys/dev/cxgbe/common/t4vf_hw.c
new file mode 100644
index 0000000..adbeb0e
--- /dev/null
+++ b/sys/dev/cxgbe/common/t4vf_hw.c
@@ -0,0 +1,376 @@
+/*-
+ * Copyright (c) 2016 Chelsio Communications, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include "common.h"
+#include "t4_regs.h"
+
+#undef msleep
+#define msleep(x) do { \
+	if (cold) \
+		DELAY((x) * 1000); \
+	else \
+		pause("t4hw", (x) * hz / 1000); \
+} while (0)
+
+/*
+ * Wait for the device to become ready (signified by our "who am I" register
+ * returning a value other than all 1's).  Return an error if it doesn't
+ * become ready ...
+ */
+int t4vf_wait_dev_ready(struct adapter *adapter)
+{
+	const u32 whoami = VF_PL_REG(A_PL_VF_WHOAMI);
+	const u32 notready1 = 0xffffffff;
+	const u32 notready2 = 0xeeeeeeee;
+	u32 val;
+
+	val = t4_read_reg(adapter, whoami);
+	if (val != notready1 && val != notready2)
+		return 0;
+	msleep(500);
+	val = t4_read_reg(adapter, whoami);
+	if (val != notready1 && val != notready2)
+		return 0;
+	else
+		return -EIO;
+}
+
+
+/**
+ *      t4vf_fw_reset - issue a reset to FW
+ *      @adapter: the adapter
+ *
+ *	Issues a reset command to FW.  For a Physical Function this would
+ *	result in the Firmware reseting all of its state.  For a Virtual
+ *	Function this just resets the state associated with the VF.
+ */
+int t4vf_fw_reset(struct adapter *adapter)
+{
+	struct fw_reset_cmd cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.op_to_write = cpu_to_be32(V_FW_CMD_OP(FW_RESET_CMD) |
+				      F_FW_CMD_WRITE);
+	cmd.retval_len16 = cpu_to_be32(V_FW_CMD_LEN16(FW_LEN16(cmd)));
+	return t4vf_wr_mbox(adapter, &cmd, sizeof(cmd), NULL);
+}
+
+/**
+ *	t4vf_get_sge_params - retrieve adapter Scatter gather Engine parameters
+ *	@adapter: the adapter
+ *
+ *	Retrieves various core SGE parameters in the form of hardware SGE
+ *	register values.  The caller is responsible for decoding these as
+ *	needed.  The SGE parameters are stored in @adapter->params.sge.
+ */
+int t4vf_get_sge_params(struct adapter *adapter)
+{
+	struct sge_params *sp = &adapter->params.sge;
+	u32 params[7], vals[7];
+	u32 whoami;
+	unsigned int pf, s_hps;
+	int i, v;
+
+	params[0] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_CONTROL));
+	params[1] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_HOST_PAGE_SIZE));
+	params[2] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_TIMER_VALUE_0_AND_1));
+	params[3] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_TIMER_VALUE_2_AND_3));
+	params[4] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_TIMER_VALUE_4_AND_5));
+	params[5] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_CONM_CTRL));
+	params[6] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		     V_FW_PARAMS_PARAM_XYZ(A_SGE_INGRESS_RX_THRESHOLD));
+	v = t4vf_query_params(adapter, 7, params, vals);
+	if (v != FW_SUCCESS)
+		return v;
+
+	sp->sge_control = vals[0];
+	sp->counter_val[0] = G_THRESHOLD_0(vals[6]);
+	sp->counter_val[1] = G_THRESHOLD_1(vals[6]);
+	sp->counter_val[2] = G_THRESHOLD_2(vals[6]);
+	sp->counter_val[3] = G_THRESHOLD_3(vals[6]);
+	sp->timer_val[0] = core_ticks_to_us(adapter, G_TIMERVALUE0(vals[2]));
+	sp->timer_val[1] = core_ticks_to_us(adapter, G_TIMERVALUE1(vals[2]));
+	sp->timer_val[2] = core_ticks_to_us(adapter, G_TIMERVALUE2(vals[3]));
+	sp->timer_val[3] = core_ticks_to_us(adapter, G_TIMERVALUE3(vals[3]));
+	sp->timer_val[4] = core_ticks_to_us(adapter, G_TIMERVALUE4(vals[4]));
+	sp->timer_val[5] = core_ticks_to_us(adapter, G_TIMERVALUE5(vals[4]));
+
+	sp->fl_starve_threshold = G_EGRTHRESHOLD(vals[5]) * 2 + 1;
+	if (is_t4(adapter))
+		sp->fl_starve_threshold2 = sp->fl_starve_threshold;
+	else
+		sp->fl_starve_threshold2 = G_EGRTHRESHOLDPACKING(vals[5]) * 2 +
+		    1;
+
+	/*
+	 * We need the Queues/Page and Host Page Size for our VF.
+	 * This is based on the PF from which we're instantiated.
+	 */
+	whoami = t4_read_reg(adapter, VF_PL_REG(A_PL_VF_WHOAMI));
+	pf = G_SOURCEPF(whoami);
+
+	s_hps = (S_HOSTPAGESIZEPF0 +
+	    (S_HOSTPAGESIZEPF1 - S_HOSTPAGESIZEPF0) * pf);
+	sp->page_shift = ((vals[1] >> s_hps) & M_HOSTPAGESIZEPF0) + 10;
+
+	for (i = 0; i < SGE_FLBUF_SIZES; i++) {
+		params[0] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+		    V_FW_PARAMS_PARAM_XYZ(A_SGE_FL_BUFFER_SIZE0 + (4 * i)));
+		v = t4vf_query_params(adapter, 1, params, vals);
+		if (v != FW_SUCCESS)
+			return v;
+
+		sp->sge_fl_buffer_size[i] = vals[0];
+	}
+
+	/*
+	 * T4 uses a single control field to specify both the PCIe Padding and
+	 * Packing Boundary.  T5 introduced the ability to specify these
+	 * separately with the Padding Boundary in SGE_CONTROL and and Packing
+	 * Boundary in SGE_CONTROL2.  So for T5 and later we need to grab
+	 * SGE_CONTROL in order to determine how ingress packet data will be
+	 * laid out in Packed Buffer Mode.  Unfortunately, older versions of
+	 * the firmware won't let us retrieve SGE_CONTROL2 so if we get a
+	 * failure grabbing it we throw an error since we can't figure out the
+	 * right value.
+	 */
+	sp->spg_len = sp->sge_control & F_EGRSTATUSPAGESIZE ? 128 : 64;
+	sp->fl_pktshift = G_PKTSHIFT(sp->sge_control);
+	sp->pad_boundary = 1 << (G_INGPADBOUNDARY(sp->sge_control) + 5);
+	if (is_t4(adapter))
+		sp->pack_boundary = sp->pad_boundary;
+	else {
+		params[0] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+			     V_FW_PARAMS_PARAM_XYZ(A_SGE_CONTROL2));
+		v = t4vf_query_params(adapter, 1, params, vals);
+		if (v != FW_SUCCESS) {
+			CH_ERR(adapter, "Unable to get SGE Control2; "
+			       "probably old firmware.\n");
+			return v;
+		}
+		if (G_INGPACKBOUNDARY(vals[0]) == 0)
+			sp->pack_boundary = 16;
+		else
+			sp->pack_boundary = 1 << (G_INGPACKBOUNDARY(vals[0]) +
+			    5);
+	}
+
+	/*
+	 * For T5 and later we want to use the new BAR2 Doorbells.
+	 * Unfortunately, older firmware didn't allow the this register to be
+	 * read.
+	 */
+	if (!is_t4(adapter)) {
+		unsigned int s_qpp;
+
+		params[0] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+			     V_FW_PARAMS_PARAM_XYZ(A_SGE_EGRESS_QUEUES_PER_PAGE_VF));
+		params[1] = (V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_REG) |
+			     V_FW_PARAMS_PARAM_XYZ(A_SGE_INGRESS_QUEUES_PER_PAGE_VF));
+		v = t4vf_query_params(adapter, 2, params, vals);
+		if (v != FW_SUCCESS) {
+			CH_WARN(adapter, "Unable to get VF SGE Queues/Page; "
+				"probably old firmware.\n");
+			return v;
+		}
+
+		s_qpp = (S_QUEUESPERPAGEPF0 +
+			 (S_QUEUESPERPAGEPF1 - S_QUEUESPERPAGEPF0) * pf);
+		sp->eq_s_qpp = ((vals[0] >> s_qpp) & M_QUEUESPERPAGEPF0);
+		sp->iq_s_qpp = ((vals[1] >> s_qpp) & M_QUEUESPERPAGEPF0);
+	}
+
+	return 0;
+}
+
+/**
+ *	t4vf_get_rss_glb_config - retrieve adapter RSS Global Configuration
+ *	@adapter: the adapter
+ *
+ *	Retrieves global RSS mode and parameters with which we have to live
+ *	and stores them in the @adapter's RSS parameters.
+ */
+int t4vf_get_rss_glb_config(struct adapter *adapter)
+{
+	struct rss_params *rss = &adapter->params.rss;
+	struct fw_rss_glb_config_cmd cmd, rpl;
+	int v;
+
+	/*
+	 * Execute an RSS Global Configuration read command to retrieve
+	 * our RSS configuration.
+	 */
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.op_to_write = cpu_to_be32(V_FW_CMD_OP(FW_RSS_GLB_CONFIG_CMD) |
+				      F_FW_CMD_REQUEST |
+				      F_FW_CMD_READ);
+	cmd.retval_len16 = cpu_to_be32(FW_LEN16(cmd));
+	v = t4vf_wr_mbox(adapter, &cmd, sizeof(cmd), &rpl);
+	if (v != FW_SUCCESS)
+		return v;
+
+	/*
+	 * Transate the big-endian RSS Global Configuration into our
+	 * cpu-endian format based on the RSS mode.  We also do first level
+	 * filtering at this point to weed out modes which don't support
+	 * VF Drivers ...
+	 */
+	rss->mode = G_FW_RSS_GLB_CONFIG_CMD_MODE(
+			be32_to_cpu(rpl.u.manual.mode_pkd));
+	switch (rss->mode) {
+	case FW_RSS_GLB_CONFIG_CMD_MODE_BASICVIRTUAL: {
+		u32 word = be32_to_cpu(
+				rpl.u.basicvirtual.synmapen_to_hashtoeplitz);
+
+		rss->u.basicvirtual.synmapen =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_SYNMAPEN) != 0);
+		rss->u.basicvirtual.syn4tupenipv6 =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_SYN4TUPENIPV6) != 0);
+		rss->u.basicvirtual.syn2tupenipv6 =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_SYN2TUPENIPV6) != 0);
+		rss->u.basicvirtual.syn4tupenipv4 =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_SYN4TUPENIPV4) != 0);
+		rss->u.basicvirtual.syn2tupenipv4 =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_SYN2TUPENIPV4) != 0);
+
+		rss->u.basicvirtual.ofdmapen =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_OFDMAPEN) != 0);
+
+		rss->u.basicvirtual.tnlmapen =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_TNLMAPEN) != 0);
+		rss->u.basicvirtual.tnlalllookup =
+			((word  & F_FW_RSS_GLB_CONFIG_CMD_TNLALLLKP) != 0);
+
+		rss->u.basicvirtual.hashtoeplitz =
+			((word & F_FW_RSS_GLB_CONFIG_CMD_HASHTOEPLITZ) != 0);
+
+		/* we need at least Tunnel Map Enable to be set */
+		if (!rss->u.basicvirtual.tnlmapen)
+			return -EINVAL;
+		break;
+	}
+
+	default:
+		/* all unknown/unsupported RSS modes result in an error */
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ *	t4vf_get_vfres - retrieve VF resource limits
+ *	@adapter: the adapter
+ *
+ *	Retrieves configured resource limits and capabilities for a virtual
+ *	function.  The results are stored in @adapter->vfres.
+ */
+int t4vf_get_vfres(struct adapter *adapter)
+{
+	struct vf_resources *vfres = &adapter->params.vfres;
+	struct fw_pfvf_cmd cmd, rpl;
+	int v;
+	u32 word;
+
+	/*
+	 * Execute PFVF Read command to get VF resource limits; bail out early
+	 * with error on command failure.
+	 */
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.op_to_vfn = cpu_to_be32(V_FW_CMD_OP(FW_PFVF_CMD) |
+				    F_FW_CMD_REQUEST |
+				    F_FW_CMD_READ);
+	cmd.retval_len16 = cpu_to_be32(FW_LEN16(cmd));
+	v = t4vf_wr_mbox(adapter, &cmd, sizeof(cmd), &rpl);
+	if (v != FW_SUCCESS)
+		return v;
+
+	/*
+	 * Extract VF resource limits and return success.
+	 */
+	word = be32_to_cpu(rpl.niqflint_niq);
+	vfres->niqflint = G_FW_PFVF_CMD_NIQFLINT(word);
+	vfres->niq = G_FW_PFVF_CMD_NIQ(word);
+
+	word = be32_to_cpu(rpl.type_to_neq);
+	vfres->neq = G_FW_PFVF_CMD_NEQ(word);
+	vfres->pmask = G_FW_PFVF_CMD_PMASK(word);
+
+	word = be32_to_cpu(rpl.tc_to_nexactf);
+	vfres->tc = G_FW_PFVF_CMD_TC(word);
+	vfres->nvi = G_FW_PFVF_CMD_NVI(word);
+	vfres->nexactf = G_FW_PFVF_CMD_NEXACTF(word);
+
+	word = be32_to_cpu(rpl.r_caps_to_nethctrl);
+	vfres->r_caps = G_FW_PFVF_CMD_R_CAPS(word);
+	vfres->wx_caps = G_FW_PFVF_CMD_WX_CAPS(word);
+	vfres->nethctrl = G_FW_PFVF_CMD_NETHCTRL(word);
+
+	return 0;
+}
+
+/**
+ */
+int t4vf_prep_adapter(struct adapter *adapter)
+{
+	int err;
+
+	/*
+	 * Wait for the device to become ready before proceeding ...
+	 */
+	err = t4vf_wait_dev_ready(adapter);
+	if (err)
+		return err;
+
+	adapter->params.chipid = pci_get_device(adapter->dev) >> 12;
+	if (adapter->params.chipid >= 0xa) {
+		adapter->params.chipid -= (0xa - 0x4);
+		adapter->params.fpga = 1;
+	}
+	
+	/*
+	 * Default port and clock for debugging in case we can't reach
+	 * firmware.
+	 */
+	adapter->params.nports = 1;
+	adapter->params.vfres.pmask = 1;
+	adapter->params.vpd.cclk = 50000;
+
+	adapter->chip_params = t4_get_chip_params(chip_id(adapter));
+	if (adapter->chip_params == NULL)
+		return -EINVAL;
+
+	return 0;
+}
diff --git a/sys/dev/cxgbe/if_cxlv.c b/sys/dev/cxgbe/if_cxlv.c
new file mode 100644
index 0000000..5a169d0
--- /dev/null
+++ b/sys/dev/cxgbe/if_cxlv.c
@@ -0,0 +1,44 @@
+/*-
+ * Copyright (c) 2015 Chelsio Communications, Inc.
+ * All rights reserved.
+ * Written by: Navdeep Parhar <np@FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/param.h>
+#include <sys/kernel.h>
+#include <sys/module.h>
+
+static int
+mod_event(module_t mod, int cmd, void *arg)
+{
+
+	return (0);
+}
+static moduledata_t if_cxlv_mod = {"if_cxlv", mod_event};
+DECLARE_MODULE(if_cxlv, if_cxlv_mod, SI_SUB_EXEC, SI_ORDER_ANY);
+MODULE_VERSION(if_cxlv, 1);
+MODULE_DEPEND(if_cxlv, cxlv, 1, 1, 1);
diff --git a/sys/dev/cxgbe/iw_cxgbe/cq.c b/sys/dev/cxgbe/iw_cxgbe/cq.c
index e4df420..8c2feb5 100644
--- a/sys/dev/cxgbe/iw_cxgbe/cq.c
+++ b/sys/dev/cxgbe/iw_cxgbe/cq.c
@@ -172,7 +172,7 @@ create_cq(struct c4iw_rdev *rdev, struct t4_cq *cq,
 
 	cq->gen = 1;
 	cq->gts = (void *)((unsigned long)rman_get_virtual(sc->regs_res) +
-	    MYPF_REG(SGE_PF_GTS));
+	    sc->sge_gts_reg);
 	cq->rdev = rdev;
 
 	if (user) {
diff --git a/sys/dev/cxgbe/iw_cxgbe/qp.c b/sys/dev/cxgbe/iw_cxgbe/qp.c
index 3ee16b1..be8ffbe 100644
--- a/sys/dev/cxgbe/iw_cxgbe/qp.c
+++ b/sys/dev/cxgbe/iw_cxgbe/qp.c
@@ -181,9 +181,9 @@ static int create_qp(struct c4iw_rdev *rdev, struct t4_wq *wq,
 	pci_unmap_addr_set(&wq->rq, mapping, wq->rq.dma_addr);
 
 	wq->db = (void *)((unsigned long)rman_get_virtual(sc->regs_res) +
-	    MYPF_REG(SGE_PF_KDOORBELL));
+	    sc->sge_kdoorbell_reg);
 	wq->gts = (void *)((unsigned long)rman_get_virtual(rdev->adap->regs_res)
-			   + MYPF_REG(SGE_PF_GTS));
+			   + sc->sge_gts_reg);
 	if (user) {
 		wq->sq.udb = (u64)((char*)rman_get_virtual(rdev->adap->udbs_res) +
 						(wq->sq.qid << rdev->qpshift));
diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c
index a106441..2a69227 100644
--- a/sys/dev/cxgbe/t4_main.c
+++ b/sys/dev/cxgbe/t4_main.c
@@ -100,7 +100,7 @@ static driver_t t4_driver = {
 static int cxgbe_probe(device_t);
 static int cxgbe_attach(device_t);
 static int cxgbe_detach(device_t);
-static device_method_t cxgbe_methods[] = {
+device_method_t cxgbe_methods[] = {
 	DEVMETHOD(device_probe,		cxgbe_probe),
 	DEVMETHOD(device_attach,	cxgbe_attach),
 	DEVMETHOD(device_detach,	cxgbe_detach),
@@ -129,14 +129,9 @@ static driver_t vcxgbe_driver = {
 };
 
 static d_ioctl_t t4_ioctl;
-static d_open_t t4_open;
-static d_close_t t4_close;
 
 static struct cdevsw t4_cdevsw = {
        .d_version = D_VERSION,
-       .d_flags = 0,
-       .d_open = t4_open,
-       .d_close = t4_close,
        .d_ioctl = t4_ioctl,
        .d_name = "t4nex",
 };
@@ -171,15 +166,6 @@ static driver_t vcxl_driver = {
 	sizeof(struct vi_info)
 };
 
-static struct cdevsw t5_cdevsw = {
-       .d_version = D_VERSION,
-       .d_flags = 0,
-       .d_open = t4_open,
-       .d_close = t4_close,
-       .d_ioctl = t4_ioctl,
-       .d_name = "t5nex",
-};
-
 /* ifnet + media interface */
 static void cxgbe_init(void *);
 static int cxgbe_ioctl(struct ifnet *, unsigned long, caddr_t);
@@ -216,19 +202,19 @@ SLIST_HEAD(, uld_info) t4_uld_list;
  * Number of queues for tx and rx, 10G and 1G, NIC and offload.
  */
 #define NTXQ_10G 16
-static int t4_ntxq10g = -1;
+int t4_ntxq10g = -1;
 TUNABLE_INT("hw.cxgbe.ntxq10g", &t4_ntxq10g);
 
 #define NRXQ_10G 8
-static int t4_nrxq10g = -1;
+int t4_nrxq10g = -1;
 TUNABLE_INT("hw.cxgbe.nrxq10g", &t4_nrxq10g);
 
 #define NTXQ_1G 4
-static int t4_ntxq1g = -1;
+int t4_ntxq1g = -1;
 TUNABLE_INT("hw.cxgbe.ntxq1g", &t4_ntxq1g);
 
 #define NRXQ_1G 2
-static int t4_nrxq1g = -1;
+int t4_nrxq1g = -1;
 TUNABLE_INT("hw.cxgbe.nrxq1g", &t4_nrxq1g);
 
 #define NTXQ_VI 1
@@ -282,34 +268,34 @@ TUNABLE_INT("hw.cxgbe.nnmrxq_vi", &t4_nnmrxq_vi);
  * Holdoff parameters for 10G and 1G ports.
  */
 #define TMR_IDX_10G 1
-static int t4_tmr_idx_10g = TMR_IDX_10G;
+int t4_tmr_idx_10g = TMR_IDX_10G;
 TUNABLE_INT("hw.cxgbe.holdoff_timer_idx_10G", &t4_tmr_idx_10g);
 
 #define PKTC_IDX_10G (-1)
-static int t4_pktc_idx_10g = PKTC_IDX_10G;
+int t4_pktc_idx_10g = PKTC_IDX_10G;
 TUNABLE_INT("hw.cxgbe.holdoff_pktc_idx_10G", &t4_pktc_idx_10g);
 
 #define TMR_IDX_1G 1
-static int t4_tmr_idx_1g = TMR_IDX_1G;
+int t4_tmr_idx_1g = TMR_IDX_1G;
 TUNABLE_INT("hw.cxgbe.holdoff_timer_idx_1G", &t4_tmr_idx_1g);
 
 #define PKTC_IDX_1G (-1)
-static int t4_pktc_idx_1g = PKTC_IDX_1G;
+int t4_pktc_idx_1g = PKTC_IDX_1G;
 TUNABLE_INT("hw.cxgbe.holdoff_pktc_idx_1G", &t4_pktc_idx_1g);
 
 /*
  * Size (# of entries) of each tx and rx queue.
  */
-static unsigned int t4_qsize_txq = TX_EQ_QSIZE;
+unsigned int t4_qsize_txq = TX_EQ_QSIZE;
 TUNABLE_INT("hw.cxgbe.qsize_txq", &t4_qsize_txq);
 
-static unsigned int t4_qsize_rxq = RX_IQ_QSIZE;
+unsigned int t4_qsize_rxq = RX_IQ_QSIZE;
 TUNABLE_INT("hw.cxgbe.qsize_rxq", &t4_qsize_rxq);
 
 /*
  * Interrupt types allowed (bits 0, 1, 2 = INTx, MSI, MSI-X respectively).
  */
-static int t4_intr_types = INTR_MSIX | INTR_MSI | INTR_INTX;
+int t4_intr_types = INTR_MSIX | INTR_MSI | INTR_INTX;
 TUNABLE_INT("hw.cxgbe.interrupt_types", &t4_intr_types);
 
 /*
@@ -420,8 +406,6 @@ struct filter_entry {
         struct t4_filter_specification fs;
 };
 
-static int map_bars_0_and_4(struct adapter *);
-static int map_bar_2(struct adapter *);
 static void setup_memwin(struct adapter *);
 static void position_memwin(struct adapter *, int, uint32_t);
 static int rw_via_memwin(struct adapter *, int, uint32_t, uint32_t *, int, int);
@@ -446,7 +430,6 @@ static void t4_set_desc(struct adapter *);
 static void build_medialist(struct port_info *, struct ifmedia *);
 static int cxgbe_init_synchronized(struct vi_info *);
 static int cxgbe_uninit_synchronized(struct vi_info *);
-static int setup_intr_handlers(struct adapter *);
 static void quiesce_txq(struct adapter *, struct sge_txq *);
 static void quiesce_wrq(struct adapter *, struct sge_wrq *);
 static void quiesce_iq(struct adapter *, struct sge_iq *);
@@ -459,7 +442,6 @@ static void vi_refresh_stats(struct adapter *, struct vi_info *);
 static void cxgbe_refresh_stats(struct adapter *, struct port_info *);
 static void cxgbe_tick(void *);
 static void cxgbe_vlan_config(void *, struct ifnet *, uint16_t);
-static void t4_sysctls(struct adapter *);
 static void cxgbe_sysctls(struct port_info *);
 static int sysctl_int_array(SYSCTL_HANDLER_ARGS);
 static int sysctl_bitfield(SYSCTL_HANDLER_ARGS);
@@ -528,8 +510,6 @@ static int get_sge_context(struct adapter *, struct t4_sge_context *);
 static int load_fw(struct adapter *, struct t4_data *);
 static int read_card_mem(struct adapter *, int, struct t4_mem_range *);
 static int read_i2c(struct adapter *, struct t4_i2c_data *);
-static int set_sched_class(struct adapter *, struct t4_sched_params *);
-static int set_sched_queue(struct adapter *, struct t4_sched_queue *);
 #ifdef TCP_OFFLOAD
 static int toe_capability(struct vi_info *, int);
 #endif
@@ -672,6 +652,7 @@ t4_attach(device_t dev)
 {
 	struct adapter *sc;
 	int rc = 0, i, j, n10g, n1g, rqidx, tqidx;
+	struct make_dev_args mda;
 	struct intrs_and_queues iaq;
 	struct sge *s;
 	uint8_t *buf;
@@ -701,6 +682,8 @@ t4_attach(device_t dev)
 		sc->params.pci.mps = 128 << ((v & PCIEM_CTL_MAX_PAYLOAD) >> 5);
 	}
 
+	sc->sge_gts_reg = MYPF_REG(A_SGE_PF_GTS);
+	sc->sge_kdoorbell_reg = MYPF_REG(A_SGE_PF_KDOORBELL);
 	sc->traceq = -1;
 	mtx_init(&sc->ifp_lock, sc->ifp_lockname, 0, MTX_DEF);
 	snprintf(sc->ifp_lockname, sizeof(sc->ifp_lockname), "%s tracer",
@@ -709,9 +692,7 @@ t4_attach(device_t dev)
 	snprintf(sc->lockname, sizeof(sc->lockname), "%s",
 	    device_get_nameunit(dev));
 	mtx_init(&sc->sc_lock, sc->lockname, 0, MTX_DEF);
-	sx_xlock(&t4_list_lock);
-	SLIST_INSERT_HEAD(&t4_list, sc, link);
-	sx_xunlock(&t4_list_lock);
+	t4_add_adapter(sc);
 
 	mtx_init(&sc->sfl_lock, "starving freelists", 0, MTX_DEF);
 	TAILQ_INIT(&sc->sfl);
@@ -719,7 +700,7 @@ t4_attach(device_t dev)
 
 	mtx_init(&sc->reg_lock, "indirect register access", 0, MTX_DEF);
 
-	rc = map_bars_0_and_4(sc);
+	rc = t4_map_bars_0_and_4(sc);
 	if (rc != 0)
 		goto done; /* error message displayed already */
 
@@ -751,13 +732,16 @@ t4_attach(device_t dev)
 	setup_memwin(sc);
 	if (t4_init_devlog_params(sc, 0) == 0)
 		fixup_devlog_params(sc);
-	sc->cdev = make_dev(is_t4(sc) ? &t4_cdevsw : &t5_cdevsw,
-	    device_get_unit(dev), UID_ROOT, GID_WHEEL, 0600, "%s",
-	    device_get_nameunit(dev));
-	if (sc->cdev == NULL)
-		device_printf(dev, "failed to create nexus char device.\n");
-	else
-		sc->cdev->si_drv1 = sc;
+	make_dev_args_init(&mda);
+	mda.mda_devsw = &t4_cdevsw;
+	mda.mda_uid = UID_ROOT;
+	mda.mda_gid = GID_WHEEL;
+	mda.mda_mode = 0600;
+	mda.mda_si_drv1 = sc;
+	rc = make_dev_s(&mda, &sc->cdev, "%s", device_get_nameunit(dev));
+	if (rc != 0)
+		device_printf(dev, "failed to create nexus char device: %d.\n",
+		    rc);
 
 	/* Go no further if recovery mode has been requested. */
 	if (TUNABLE_INT_FETCH("hw.cxgbe.sos", &i) && i != 0) {
@@ -786,7 +770,7 @@ t4_attach(device_t dev)
 	if (rc != 0)
 		goto done; /* error message displayed already */
 
-	rc = map_bar_2(sc);
+	rc = t4_map_bar_2(sc);
 	if (rc != 0)
 		goto done; /* error message displayed already */
 
@@ -1040,7 +1024,7 @@ t4_attach(device_t dev)
 		}
 	}
 
-	rc = setup_intr_handlers(sc);
+	rc = t4_setup_intr_handlers(sc);
 	if (rc != 0) {
 		device_printf(dev,
 		    "failed to setup interrupt handlers: %d\n", rc);
@@ -1072,7 +1056,7 @@ done:
 	}
 
 	if (rc != 0)
-		t4_detach(dev);
+		t4_detach_common(dev);
 	else
 		t4_sysctls(sc);
 
@@ -1086,24 +1070,38 @@ static int
 t4_detach(device_t dev)
 {
 	struct adapter *sc;
+
+	sc = device_get_softc(dev);
+
+	return (t4_detach_common(dev));
+}
+
+int
+t4_detach_common(device_t dev)
+{
+	struct adapter *sc;
 	struct port_info *pi;
 	int i, rc;
 
 	sc = device_get_softc(dev);
 
-	if (sc->flags & FULL_INIT_DONE)
-		t4_intr_disable(sc);
+	if (sc->flags & FULL_INIT_DONE) {
+		if (!(sc->flags & IS_VF))
+			t4_intr_disable(sc);
+	}
 
 	if (sc->cdev) {
 		destroy_dev(sc->cdev);
 		sc->cdev = NULL;
 	}
 
-	rc = bus_generic_detach(dev);
-	if (rc) {
-		device_printf(dev,
-		    "failed to detach child devices: %d\n", rc);
-		return (rc);
+	if (device_is_attached(dev)) {
+		rc = bus_generic_detach(dev);
+		if (rc) {
+			device_printf(dev,
+			    "failed to detach child devices: %d\n", rc);
+			return (rc);
+		}
 	}
 
 	for (i = 0; i < sc->intr_count; i++)
@@ -1126,7 +1124,7 @@ t4_detach(device_t dev)
 	if (sc->flags & FULL_INIT_DONE)
 		adapter_full_uninit(sc);
 
-	if (sc->flags & FW_OK)
+	if ((sc->flags & (IS_VF | FW_OK)) == FW_OK)
 		t4_fw_bye(sc, sc->mbox);
 
 	if (sc->intr_type == INTR_MSI || sc->intr_type == INTR_MSIX)
@@ -1615,7 +1613,7 @@ cxgbe_transmit(struct ifnet *ifp, struct mbuf *m)
 		return (ENETDOWN);
 	}
 
-	rc = parse_pkt(&m);
+	rc = parse_pkt(sc, &m);
 	if (__predict_false(rc != 0)) {
 		MPASS(m == NULL);			/* was freed already */
 		atomic_add_int(&pi->tx_parse_error, 1);	/* rare, atomic is ok */
@@ -1789,8 +1787,16 @@ t4_fatal_err(struct adapter *sc)
 	    device_get_nameunit(sc->dev));
 }
 
-static int
-map_bars_0_and_4(struct adapter *sc)
+void
+t4_add_adapter(struct adapter *sc)
+{
+	sx_xlock(&t4_list_lock);
+	SLIST_INSERT_HEAD(&t4_list, sc, link);
+	sx_xunlock(&t4_list_lock);
+}
+
+int
+t4_map_bars_0_and_4(struct adapter *sc)
 {
 	sc->regs_rid = PCIR_BAR(0);
 	sc->regs_res = bus_alloc_resource_any(sc->dev, SYS_RES_MEMORY,
@@ -1815,8 +1821,8 @@ map_bars_0_and_4(struct adapter *sc)
 	return (0);
 }
 
-static int
-map_bar_2(struct adapter *sc)
+int
+t4_map_bar_2(struct adapter *sc)
 {
 
 	/*
@@ -3155,6 +3161,8 @@ get_params__post_init(struct adapter *sc)
 		sc->vres.iscsi.size = val[1] - val[0] + 1;
 	}
 
+	t4_init_sge_params(sc);
+
 	/*
 	 * We've got the params we wanted to query via the firmware.  Now grab
 	 * some others directly from the chip.
@@ -3596,7 +3604,7 @@ cxgbe_init_synchronized(struct vi_info *vi)
 	ifp->if_drv_flags |= IFF_DRV_RUNNING;
 	pi->up_vis++;
 
-	if (pi->nvi > 1)
+	if (pi->nvi > 1 || sc->flags & IS_VF)
 		callout_reset(&vi->tick, hz, vi_tick, vi);
 	else
 		callout_reset(&pi->tick, hz, cxgbe_tick, pi);
@@ -3648,10 +3656,10 @@ cxgbe_uninit_synchronized(struct vi_info *vi)
 	}
 
 	PORT_LOCK(pi);
-	if (pi->nvi == 1)
-		callout_stop(&pi->tick);
-	else
+	if (pi->nvi > 1 || sc->flags & IS_VF)
 		callout_stop(&vi->tick);
+	else
+		callout_stop(&pi->tick);
 	if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) {
 		PORT_UNLOCK(pi);
 		return (0);
@@ -3676,8 +3684,8 @@ cxgbe_uninit_synchronized(struct vi_info *vi)
  * It is ok for this function to fail midway and return right away.  t4_detach
  * will walk the entire sc->irq list and clean up whatever is valid.
  */
-static int
-setup_intr_handlers(struct adapter *sc)
+int
+t4_setup_intr_handlers(struct adapter *sc)
 {
 	int rc, rid, p, q, v;
 	char s[8];
@@ -3702,17 +3710,23 @@ setup_intr_handlers(struct adapter *sc)
 		return (t4_alloc_irq(sc, irq, rid, t4_intr_all, sc, "all"));
 
 	/* Multiple interrupts. */
-	KASSERT(sc->intr_count >= T4_EXTRA_INTR + sc->params.nports,
-	    ("%s: too few intr.", __func__));
+	if (sc->flags & IS_VF)
+		KASSERT(sc->intr_count >= T4VF_EXTRA_INTR + sc->params.nports,
+		    ("%s: too few intr.", __func__));
+	else
+		KASSERT(sc->intr_count >= T4_EXTRA_INTR + sc->params.nports,
+		    ("%s: too few intr.", __func__));
 
-	/* The first one is always error intr */
-	rc = t4_alloc_irq(sc, irq, rid, t4_intr_err, sc, "err");
-	if (rc != 0)
-		return (rc);
-	irq++;
-	rid++;
+	/* The first one is always error intr on PFs */
+	if (!(sc->flags & IS_VF)) {
+		rc = t4_alloc_irq(sc, irq, rid, t4_intr_err, sc, "err");
+		if (rc != 0)
+			return (rc);
+		irq++;
+		rid++;
+	}
 
-	/* The second one is always the firmware event queue */
+	/* The second one is always the firmware event queue (first on VFs) */
 	rc = t4_alloc_irq(sc, irq, rid, t4_intr_evt, &sge->fwq, "evt");
 	if (rc != 0)
 		return (rc);
@@ -3815,7 +3829,8 @@ adapter_full_init(struct adapter *sc)
 		    device_get_nameunit(sc->dev), i);
 	}
 
-	t4_intr_enable(sc);
+	if (!(sc->flags & IS_VF))
+		t4_intr_enable(sc);
 	sc->flags |= FULL_INIT_DONE;
 done:
 	if (rc != 0)
@@ -4064,7 +4079,7 @@ vi_full_uninit(struct vi_info *vi)
 		/* Need to quiesce queues.  */
 
 		/* XXX: Only for the first VI? */
-		if (IS_MAIN_VI(vi))
+		if (IS_MAIN_VI(vi) && !(sc->flags & IS_VF))
 			quiesce_wrq(sc, &sc->sge.ctrlq[pi->port_id]);
 
 		for_each_txq(vi, i, txq) {
@@ -4231,10 +4246,16 @@ read_vf_stat(struct adapter *sc, unsigned int viid, int reg)
 	u32 stats[2];
 
 	mtx_assert(&sc->reg_lock, MA_OWNED);
-	t4_write_reg(sc, A_PL_INDIR_CMD, V_PL_AUTOINC(1) |
-	    V_PL_VFID(G_FW_VIID_VIN(viid)) | V_PL_ADDR(VF_MPS_REG(reg)));
-	stats[0] = t4_read_reg(sc, A_PL_INDIR_DATA);
-	stats[1] = t4_read_reg(sc, A_PL_INDIR_DATA);
+	if (sc->flags & IS_VF) {
+		stats[0] = t4_read_reg(sc, VF_MPS_REG(reg));
+		stats[1] = t4_read_reg(sc, VF_MPS_REG(reg + 4));
+	} else {
+		t4_write_reg(sc, A_PL_INDIR_CMD, V_PL_AUTOINC(1) |
+		    V_PL_VFID(G_FW_VIID_VIN(viid)) |
+		    V_PL_ADDR(VF_MPS_REG(reg)));
+		stats[0] = t4_read_reg(sc, A_PL_INDIR_DATA);
+		stats[1] = t4_read_reg(sc, A_PL_INDIR_DATA);
+	}
 	return (((uint64_t)stats[1]) << 32 | stats[0]);
 }
 
@@ -4429,7 +4450,7 @@ static char *caps_decoder[] = {
 		    "\004PO_INITIATOR\005PO_TARGET",
 };
 
-static void
+void
 t4_sysctls(struct adapter *sc)
 {
 	struct sysctl_ctx_list *ctx;
@@ -4452,30 +4473,55 @@ t4_sysctls(struct adapter *sc)
 	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "nports", CTLFLAG_RD, NULL,
 	    sc->params.nports, "# of ports");
 
-	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "hw_revision", CTLFLAG_RD,
-	    NULL, chip_rev(sc), "chip hardware revision");
+	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "doorbells",
+	    CTLTYPE_STRING | CTLFLAG_RD, doorbells, sc->doorbells,
+	    sysctl_bitfield, "A", "available doorbells");
+
+	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "core_clock", CTLFLAG_RD, NULL,
+	    sc->params.vpd.cclk, "core clock frequency (in KHz)");
+
+	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "holdoff_timers",
+	    CTLTYPE_STRING | CTLFLAG_RD, sc->params.sge.timer_val,
+	    sizeof(sc->params.sge.timer_val), sysctl_int_array, "A",
+	    "interrupt holdoff timer values (us)");
+
+	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "holdoff_pkt_counts",
+	    CTLTYPE_STRING | CTLFLAG_RD, sc->params.sge.counter_val,
+	    sizeof(sc->params.sge.counter_val), sysctl_int_array, "A",
+	    "interrupt holdoff packet counter values");
+
+	t4_sge_sysctls(sc, ctx, children);
+
+	sc->lro_timeout = 100;
+	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "lro_timeout", CTLFLAG_RW,
+	    &sc->lro_timeout, 0, "lro inactive-flush timeout (in us)");
+
+	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "debug_flags", CTLFLAG_RW,
+	    &sc->debug_flags, 0, "flags to enable runtime debugging");
 
 	SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "tp_version",
 	    CTLFLAG_RD, sc->tp_version, 0, "TP microcode version");
 
+	SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "firmware_version",
+	    CTLFLAG_RD, sc->fw_version, 0, "firmware version");
+
+	if (sc->flags & IS_VF)
+		return;
+
+	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "hw_revision", CTLFLAG_RD,
+	    NULL, chip_rev(sc), "chip hardware revision");
+
 	if (sc->params.exprom_vers != 0) {
 		SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "exprom_version",
 		    CTLFLAG_RD, sc->exprom_version, 0, "expansion ROM version");
 	}
 
-	SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "firmware_version",
-	    CTLFLAG_RD, sc->fw_version, 0, "firmware version");
-
 	SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "cf",
 	    CTLFLAG_RD, sc->cfg_file, 0, "configuration file");
 
 	SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "cfcsum", CTLFLAG_RD, NULL,
 	    sc->cfcsum, "config file checksum");
 
-	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "doorbells",
-	    CTLTYPE_STRING | CTLFLAG_RD, doorbells, sc->doorbells,
-	    sysctl_bitfield, "A", "available doorbells");
-
 #define SYSCTL_CAP(name, n, text) \
 	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, #name, \
 	    CTLTYPE_STRING | CTLFLAG_RD, caps_decoder[n], sc->name, \
@@ -4492,19 +4538,6 @@ t4_sysctls(struct adapter *sc)
 	SYSCTL_CAP(fcoecaps, 8, "FCoE");
 #undef SYSCTL_CAP
 
-	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "core_clock", CTLFLAG_RD, NULL,
-	    sc->params.vpd.cclk, "core clock frequency (in KHz)");
-
-	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "holdoff_timers",
-	    CTLTYPE_STRING | CTLFLAG_RD, sc->params.sge.timer_val,
-	    sizeof(sc->params.sge.timer_val), sysctl_int_array, "A",
-	    "interrupt holdoff timer values (us)");
-
-	SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "holdoff_pkt_counts",
-	    CTLTYPE_STRING | CTLFLAG_RD, sc->params.sge.counter_val,
-	    sizeof(sc->params.sge.counter_val), sysctl_int_array, "A",
-	    "interrupt holdoff packet counter values");
-
 	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "nfilters", CTLFLAG_RD,
 	    NULL, sc->tids.nftids, "number of filters");
 
@@ -4512,15 +4545,6 @@ t4_sysctls(struct adapter *sc)
 	    CTLFLAG_RD, sc, 0, sysctl_temperature, "I",
 	    "chip temperature (in Celsius)");
 
-	t4_sge_sysctls(sc, ctx, children);
-
-	sc->lro_timeout = 100;
-	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "lro_timeout", CTLFLAG_RW,
-	    &sc->lro_timeout, 0, "lro inactive-flush timeout (in us)");
-
-	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "debug_flags", CTLFLAG_RW,
-	    &sc->debug_flags, 0, "flags to enable runtime debugging");
-
 #ifdef SBUF_DRAIN
 	/*
 	 * dev.t4nex.X.misc.  Marked CTLFLAG_SKIP to avoid information overload.
@@ -4892,6 +4916,9 @@ cxgbe_sysctls(struct port_info *pi)
 	SYSCTL_ADD_INT(ctx, children, OID_AUTO, "max_speed", CTLFLAG_RD, NULL,
 	    port_top_speed(pi), "max speed (in Gbps)");
 
+	if (sc->flags & IS_VF)
+		return;
+
 	/*
 	 * dev.(cxgbe|cxl).X.tc.
 	 */
@@ -8388,8 +8415,8 @@ set_sched_class_params(struct adapter *sc, struct t4_sched_class_params *p,
 	return (rc);
 }
 
-static int
-set_sched_class(struct adapter *sc, struct t4_sched_params *p)
+int
+t4_set_sched_class(struct adapter *sc, struct t4_sched_params *p)
 {
 
 	if (p->type != SCHED_CLASS_TYPE_PACKET)
@@ -8404,8 +8431,8 @@ set_sched_class(struct adapter *sc, struct t4_sched_params *p)
 	return (EINVAL);
 }
 
-static int
-set_sched_queue(struct adapter *sc, struct t4_sched_queue *p)
+int
+t4_set_sched_queue(struct adapter *sc, struct t4_sched_queue *p)
 {
 	struct port_info *pi = NULL;
 	struct vi_info *vi;
@@ -8585,18 +8612,6 @@ t4_iterate(void (*func)(struct adapter *, void *), void *arg)
 }
 
 static int
-t4_open(struct cdev *dev, int flags, int type, struct thread *td)
-{
-       return (0);
-}
-
-static int
-t4_close(struct cdev *dev, int flags, int type, struct thread *td)
-{
-       return (0);
-}
-
-static int
 t4_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag,
     struct thread *td)
 {
@@ -8641,7 +8656,7 @@ t4_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag,
 	}
 	case CHELSIO_T4_REGDUMP: {
 		struct t4_regdump *regs = (struct t4_regdump *)data;
-		int reglen = is_t4(sc) ? T4_REGDUMP_SIZE : T5_REGDUMP_SIZE;
+		int reglen = t4_get_regs_len(sc);
 		uint8_t *buf;
 
 		if (regs->len < reglen) {
@@ -8755,10 +8770,10 @@ t4_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag,
 		break;
 	}
 	case CHELSIO_T4_SCHED_CLASS:
-		rc = set_sched_class(sc, (struct t4_sched_params *)data);
+		rc = t4_set_sched_class(sc, (struct t4_sched_params *)data);
 		break;
 	case CHELSIO_T4_SCHED_QUEUE:
-		rc = set_sched_queue(sc, (struct t4_sched_queue *)data);
+		rc = t4_set_sched_queue(sc, (struct t4_sched_queue *)data);
 		break;
 	case CHELSIO_T4_GET_TRACER:
 		rc = t4_get_tracer(sc, (struct t4_tracer *)data);
@@ -8767,7 +8782,7 @@ t4_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag,
 		rc = t4_set_tracer(sc, (struct t4_tracer *)data);
 		break;
 	default:
-		rc = EINVAL;
+		rc = ENOTTY;
 	}
 
 	return (rc);
diff --git a/sys/dev/cxgbe/t4_netmap.c b/sys/dev/cxgbe/t4_netmap.c
index 9c0278d..6321579 100644
--- a/sys/dev/cxgbe/t4_netmap.c
+++ b/sys/dev/cxgbe/t4_netmap.c
@@ -204,7 +204,7 @@ alloc_nm_rxq_hwq(struct vi_info *vi, struct sge_nm_rxq *nm_rxq, int cong)
 		}
 	}
 
-	t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS),
+	t4_write_reg(sc, sc->sge_gts_reg,
 	    V_INGRESSQID(nm_rxq->iq_cntxt_id) |
 	    V_SEINTARM(V_QINTR_TIMER_IDX(holdoff_tmr_idx)));
 
@@ -364,7 +364,7 @@ cxgbe_netmap_on(struct adapter *sc, struct vi_info *vi, struct ifnet *ifp,
 		MPASS((j & 7) == 0);
 		j /= 8;	/* driver pidx to hardware pidx */
 		wmb();
-		t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+		t4_write_reg(sc, sc->sge_kdoorbell_reg,
 		    nm_rxq->fl_db_val | V_PIDX(j));
 
 		atomic_cmpset_int(&irq->nm_state, NM_OFF, NM_ON);
@@ -537,7 +537,7 @@ ring_nm_txq_db(struct adapter *sc, struct sge_nm_txq *nm_txq)
 		break;
 
 	case DOORBELL_KDB:
-		t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+		t4_write_reg(sc, sc->sge_kdoorbell_reg,
 		    V_QID(nm_txq->cntxt_id) | V_PIDX(n));
 		break;
 	}
@@ -820,7 +820,7 @@ cxgbe_netmap_rxsync(struct netmap_kring *kring, int flags)
 			}
 			if (++dbinc == 8 && n >= 32) {
 				wmb();
-				t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+				t4_write_reg(sc, sc->sge_kdoorbell_reg,
 				    nm_rxq->fl_db_val | V_PIDX(dbinc));
 				dbinc = 0;
 			}
@@ -829,7 +829,7 @@ cxgbe_netmap_rxsync(struct netmap_kring *kring, int flags)
 
 		if (dbinc > 0) {
 			wmb();
-			t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+			t4_write_reg(sc, sc->sge_kdoorbell_reg,
 			    nm_rxq->fl_db_val | V_PIDX(dbinc));
 		}
 	}
@@ -985,14 +985,14 @@ t4_nm_intr(void *arg)
 				fl_credits /= 8;
 				IDXINCR(nm_rxq->fl_pidx, fl_credits * 8,
 				    nm_rxq->fl_sidx);
-				t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+				t4_write_reg(sc, sc->sge_kdoorbell_reg,
 				    nm_rxq->fl_db_val | V_PIDX(fl_credits));
 				fl_credits = fl_cidx & 7;
 			} else if (!black_hole) {
 				netmap_rx_irq(ifp, nm_rxq->nid, &work);
 				MPASS(work != 0);
 			}
-			t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS),
+			t4_write_reg(sc, sc->sge_gts_reg,
 			    V_CIDXINC(n) | V_INGRESSQID(nm_rxq->iq_cntxt_id) |
 			    V_SEINTARM(V_QINTR_TIMER_IDX(X_TIMERREG_UPDATE_CIDX)));
 			n = 0;
@@ -1003,12 +1003,12 @@ t4_nm_intr(void *arg)
 	if (black_hole) {
 		fl_credits /= 8;
 		IDXINCR(nm_rxq->fl_pidx, fl_credits * 8, nm_rxq->fl_sidx);
-		t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+		t4_write_reg(sc, sc->sge_kdoorbell_reg,
 		    nm_rxq->fl_db_val | V_PIDX(fl_credits));
 	} else
 		netmap_rx_irq(ifp, nm_rxq->nid, &work);
 
-	t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS), V_CIDXINC(n) |
+	t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(n) |
 	    V_INGRESSQID((u32)nm_rxq->iq_cntxt_id) |
 	    V_SEINTARM(V_QINTR_TIMER_IDX(holdoff_tmr_idx)));
 }
diff --git a/sys/dev/cxgbe/t4_sge.c b/sys/dev/cxgbe/t4_sge.c
index 08bf9f1..b9e770e 100644
--- a/sys/dev/cxgbe/t4_sge.c
+++ b/sys/dev/cxgbe/t4_sge.c
@@ -52,6 +52,7 @@ __FBSDID("$FreeBSD$");
 #include <netinet/ip.h>
 #include <netinet/ip6.h>
 #include <netinet/tcp.h>
+#include <machine/in_cksum.h>
 #include <machine/md_var.h>
 #include <vm/vm.h>
 #include <vm/pmap.h>
@@ -222,10 +223,13 @@ static void add_fl_to_sfl(struct adapter *, struct sge_fl *);
 
 static inline void get_pkt_gl(struct mbuf *, struct sglist *);
 static inline u_int txpkt_len16(u_int, u_int);
+static inline u_int txpkt_vm_len16(u_int, u_int);
 static inline u_int txpkts0_len16(u_int);
 static inline u_int txpkts1_len16(void);
 static u_int write_txpkt_wr(struct sge_txq *, struct fw_eth_tx_pkt_wr *,
     struct mbuf *, u_int);
+static u_int write_txpkt_vm_wr(struct sge_txq *, struct fw_eth_tx_pkt_vm_wr *,
+    struct mbuf *, u_int);
 static int try_txpkts(struct mbuf *, struct mbuf *, struct txpkts *, u_int);
 static int add_to_txpkts(struct mbuf *, struct txpkts *, u_int);
 static u_int write_txpkts_wr(struct sge_txq *, struct fw_eth_tx_pkts_wr *,
@@ -624,11 +628,9 @@ t4_read_chip_settings(struct adapter *sc)
 	struct sw_zone_info *swz, *safe_swz;
 	struct hw_buf_info *hwb;
 
-	t4_init_sge_params(sc);
-
 	m = F_RXPKTCPLMODE;
 	v = F_RXPKTCPLMODE;
-	r = t4_read_reg(sc, A_SGE_CONTROL);
+	r = sc->params.sge.sge_control;
 	if ((r & m) != v) {
 		device_printf(sc->dev, "invalid SGE_CONTROL(0x%x)\n", r);
 		rc = EINVAL;
@@ -646,7 +648,7 @@ t4_read_chip_settings(struct adapter *sc)
 	/* Filter out unusable hw buffer sizes entirely (mark with -2). */
 	hwb = &s->hw_buf_info[0];
 	for (i = 0; i < nitems(s->hw_buf_info); i++, hwb++) {
-		r = t4_read_reg(sc, A_SGE_FL_BUFFER_SIZE0 + (4 * i));
+		r = sc->params.sge.sge_fl_buffer_size[i];
 		hwb->size = r;
 		hwb->zidx = hwsz_ok(sc, r) ? -1 : -2;
 		hwb->next = -1;
@@ -751,6 +753,9 @@ t4_read_chip_settings(struct adapter *sc)
 		}
 	}
 
+	if (sc->flags & IS_VF)
+		return (0);
+
 	v = V_HPZ0(0) | V_HPZ1(2) | V_HPZ2(4) | V_HPZ3(6);
 	r = t4_read_reg(sc, A_ULP_RX_TDDP_PSZ);
 	if (r != v) {
@@ -861,7 +866,8 @@ t4_setup_adapter_queues(struct adapter *sc)
 	 * Management queue.  This is just a control queue that uses the fwq as
 	 * its associated iq.
 	 */
-	rc = alloc_mgmtq(sc);
+	if (!(sc->flags & IS_VF))
+		rc = alloc_mgmtq(sc);
 
 	return (rc);
 }
@@ -1167,7 +1173,7 @@ t4_setup_vi_queues(struct vi_info *vi)
 	/*
 	 * Finally, the control queue.
 	 */
-	if (!IS_MAIN_VI(vi))
+	if (!IS_MAIN_VI(vi) || sc->flags & IS_VF)
 		goto done;
 	oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "ctrlq", CTLFLAG_RD,
 	    NULL, "ctrl queue");
@@ -1228,7 +1234,7 @@ t4_teardown_vi_queues(struct vi_info *vi)
 	 * (for egress updates, etc.).
 	 */
 
-	if (IS_MAIN_VI(vi))
+	if (IS_MAIN_VI(vi) && !(sc->flags & IS_VF))
 		free_wrq(sc, &sc->sge.ctrlq[pi->port_id]);
 
 	for_each_txq(vi, i, txq) {
@@ -1443,7 +1449,8 @@ service_iq(struct sge_iq *iq, int budget)
                                         break;
                                 }
 
-				q = sc->sge.iqmap[lq - sc->sge.iq_start];
+				q = sc->sge.iqmap[lq - sc->sge.iq_start -
+				    sc->sge.iq_base];
 				if (atomic_cmpset_int(&q->state, IQS_IDLE,
 				    IQS_BUSY)) {
 					if (service_iq(q, q->qsize / 16) == 0) {
@@ -1473,7 +1480,7 @@ service_iq(struct sge_iq *iq, int budget)
 				d = &iq->desc[0];
 			}
 			if (__predict_false(++ndescs == limit)) {
-				t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS),
+				t4_write_reg(sc, sc->sge_gts_reg,
 				    V_CIDXINC(ndescs) |
 				    V_INGRESSQID(iq->cntxt_id) |
 				    V_SEINTARM(V_QINTR_TIMER_IDX(X_TIMERREG_UPDATE_CIDX)));
@@ -1533,7 +1540,7 @@ process_iql:
 	}
 #endif
 
-	t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS), V_CIDXINC(ndescs) |
+	t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(ndescs) |
 	    V_INGRESSQID((u32)iq->cntxt_id) | V_SEINTARM(iq->intr_params));
 
 	if (iq->flags & IQ_HAS_FL) {
@@ -2074,7 +2081,7 @@ m_advance(struct mbuf **pm, int *poffset, int len)
 
 	MPASS(len > 0);
 
-	while (len) {
+	for (;;) {
 		if (offset + len < m->m_len) {
 			offset += len;
 			p = mtod(m, uintptr_t) + offset;
@@ -2145,7 +2152,7 @@ count_mbuf_nsegs(struct mbuf *m)
  * b) it may get defragged up if the gather list is too long for the hardware.
  */
 int
-parse_pkt(struct mbuf **mp)
+parse_pkt(struct adapter *sc, struct mbuf **mp)
 {
 	struct mbuf *m0 = *mp, *m;
 	int rc, nsegs, defragged = 0, offset;
@@ -2192,9 +2199,13 @@ restart:
 		goto restart;
 	}
 	set_mbuf_nsegs(m0, nsegs);
-	set_mbuf_len16(m0, txpkt_len16(nsegs, needs_tso(m0)));
+	if (sc->flags & IS_VF)
+		set_mbuf_len16(m0, txpkt_vm_len16(nsegs, needs_tso(m0)));
+	else
+		set_mbuf_len16(m0, txpkt_len16(nsegs, needs_tso(m0)));
 
-	if (!needs_tso(m0))
+	if (!needs_tso(m0) &&
+	    !(sc->flags & IS_VF && (needs_l3_csum(m0) || needs_l4_csum(m0))))
 		return (0);
 
 	m = m0;
@@ -2217,7 +2228,7 @@ restart:
 	{
 		struct ip6_hdr *ip6 = l3hdr;
 
-		MPASS(ip6->ip6_nxt == IPPROTO_TCP);
+		MPASS(!needs_tso(m0) || ip6->ip6_nxt == IPPROTO_TCP);
 
 		m0->m_pkthdr.l3hlen = sizeof(*ip6);
 		break;
@@ -2239,8 +2250,10 @@ restart:
 	}
 
 #if defined(INET) || defined(INET6)
-	tcp = m_advance(&m, &offset, m0->m_pkthdr.l3hlen);
-	m0->m_pkthdr.l4hlen = tcp->th_off * 4;
+	if (needs_tso(m0)) {
+		tcp = m_advance(&m, &offset, m0->m_pkthdr.l3hlen);
+		m0->m_pkthdr.l4hlen = tcp->th_off * 4;
+	}
 #endif
 	MPASS(m0 == *mp);
 	return (0);
@@ -2435,7 +2448,12 @@ eth_tx(struct mp_ring *r, u_int cidx, u_int pidx)
 			next_cidx = 0;
 
 		wr = (void *)&eq->desc[eq->pidx];
-		if (remaining > 1 &&
+		if (sc->flags & IS_VF) {
+			total++;
+			remaining--;
+			ETHER_BPF_MTAP(ifp, m0);
+			n = write_txpkt_vm_wr(txq, (void *)wr, m0, available);
+		} else if (remaining > 1 &&
 		    try_txpkts(m0, r->items[next_cidx], &txp, available) == 0) {
 
 			/* pkts at cidx, next_cidx should both be in txp. */
@@ -2766,7 +2784,7 @@ alloc_iq_fl(struct vi_info *vi, struct sge_iq *iq, struct sge_fl *fl,
 		FL_UNLOCK(fl);
 	}
 
-	if (is_t5(sc) && cong >= 0) {
+	if (is_t5(sc) && !(sc->flags & IS_VF) && cong >= 0) {
 		uint32_t param, val;
 
 		param = V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_DMAQ) |
@@ -2793,7 +2811,7 @@ alloc_iq_fl(struct vi_info *vi, struct sge_iq *iq, struct sge_fl *fl,
 
 	/* Enable IQ interrupts */
 	atomic_store_rel_int(&iq->state, IQS_IDLE);
-	t4_write_reg(sc, MYPF_REG(A_SGE_PF_GTS), V_SEINTARM(iq->intr_params) |
+	t4_write_reg(sc, sc->sge_gts_reg, V_SEINTARM(iq->intr_params) |
 	    V_INGRESSQID(iq->cntxt_id));
 
 	return (0);
@@ -2890,9 +2908,13 @@ alloc_fwq(struct adapter *sc)
 
 	init_iq(fwq, sc, 0, 0, FW_IQ_QSIZE);
 	fwq->flags |= IQ_INTR;	/* always */
-	intr_idx = sc->intr_count > 1 ? 1 : 0;
-	fwq->set_tcb_rpl = t4_filter_rpl;
-	fwq->l2t_write_rpl = do_l2t_write_rpl;
+	if (sc->flags & IS_VF)
+		intr_idx = 0;
+	else {
+		intr_idx = sc->intr_count > 1 ? 1 : 0;
+		fwq->set_tcb_rpl = t4_filter_rpl;
+		fwq->l2t_write_rpl = do_l2t_write_rpl;
+	}
 	rc = alloc_iq_fl(&sc->port[0]->vi[0], fwq, NULL, intr_idx, -1);
 	if (rc != 0) {
 		device_printf(sc->dev,
@@ -2972,6 +2994,7 @@ alloc_rxq(struct vi_info *vi, struct sge_rxq *rxq, int intr_idx, int idx,
     struct sysctl_oid *oid)
 {
 	int rc;
+	struct adapter *sc = vi->pi->adapter;
 	struct sysctl_oid_list *children;
 	char name[16];
 
@@ -2980,12 +3003,20 @@ alloc_rxq(struct vi_info *vi, struct sge_rxq *rxq, int intr_idx, int idx,
 	if (rc != 0)
 		return (rc);
 
+	if (idx == 0)
+		sc->sge.iq_base = rxq->iq.abs_id - rxq->iq.cntxt_id;
+	else
+		KASSERT(rxq->iq.cntxt_id + sc->sge.iq_base == rxq->iq.abs_id,
+		    ("iq_base mismatch"));
+	KASSERT(sc->sge.iq_base == 0 || sc->flags & IS_VF,
+	    ("PF with non-zero iq_base"));
+
 	/*
 	 * The freelist is just barely above the starvation threshold right now,
 	 * fill it up a bit more.
 	 */
 	FL_LOCK(&rxq->fl);
-	refill_fl(vi->pi->adapter, &rxq->fl, 128);
+	refill_fl(sc, &rxq->fl, 128);
 	FL_UNLOCK(&rxq->fl);
 
 #if defined(INET) || defined(INET6)
@@ -3317,6 +3348,7 @@ eth_eq_alloc(struct adapter *sc, struct vi_info *vi, struct sge_eq *eq)
 	eq->flags |= EQ_ALLOCATED;
 
 	eq->cntxt_id = G_FW_EQ_ETH_CMD_EQID(be32toh(c.eqid_pkd));
+	eq->abs_id = G_FW_EQ_ETH_CMD_PHYSEQID(be32toh(c.physeqid_pkd));
 	cntxt_id = eq->cntxt_id - sc->sge.eq_start;
 	if (cntxt_id >= sc->sge.neq)
 	    panic("%s: eq->cntxt_id (%d) more than the max (%d)", __func__,
@@ -3557,12 +3589,24 @@ alloc_txq(struct vi_info *vi, struct sge_txq *txq, int idx,
 
 	/* Can't fail after this point. */
 
+	if (idx == 0)
+		sc->sge.eq_base = eq->abs_id - eq->cntxt_id;
+	else
+		KASSERT(eq->cntxt_id + sc->sge.eq_base == eq->abs_id,
+		    ("eq_base mismatch"));
+	KASSERT(sc->sge.eq_base == 0 || sc->flags & IS_VF,
+	    ("PF with non-zero eq_base"));
+
 	TASK_INIT(&txq->tx_reclaim_task, 0, tx_reclaim, eq);
 	txq->ifp = vi->ifp;
 	txq->gl = sglist_alloc(TX_SGL_SEGS, M_WAITOK);
-	txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT) |
-	    V_TXPKT_INTF(pi->tx_chan) | V_TXPKT_VF_VLD(1) |
-	    V_TXPKT_VF(vi->viid));
+	if (sc->flags & IS_VF)
+		txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT_XT) |
+		    V_TXPKT_INTF(pi->tx_chan));
+	else
+		txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT) |
+		    V_TXPKT_INTF(pi->tx_chan) | V_TXPKT_VF_VLD(1) |
+		    V_TXPKT_VF(vi->viid));
 	txq->tc_idx = -1;
 	txq->sdesc = malloc(eq->sidx * sizeof(struct tx_sdesc), M_CXGBE,
 	    M_ZERO | M_WAITOK);
@@ -3572,6 +3616,8 @@ alloc_txq(struct vi_info *vi, struct sge_txq *txq, int idx,
 	    NULL, "tx queue");
 	children = SYSCTL_CHILDREN(oid);
 
+	SYSCTL_ADD_UINT(&vi->ctx, children, OID_AUTO, "abs_id", CTLFLAG_RD,
+	    &eq->abs_id, 0, "absolute id of the queue");
 	SYSCTL_ADD_UINT(&vi->ctx, children, OID_AUTO, "cntxt_id", CTLFLAG_RD,
 	    &eq->cntxt_id, 0, "SGE context id of the queue");
 	SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "cidx",
@@ -3676,7 +3722,7 @@ ring_fl_db(struct adapter *sc, struct sge_fl *fl)
 	if (fl->udb)
 		*fl->udb = htole32(v);
 	else
-		t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL), v);
+		t4_write_reg(sc, sc->sge_kdoorbell_reg, v);
 	IDXINCR(fl->dbidx, n, fl->sidx);
 }
 
@@ -3913,6 +3959,27 @@ txpkt_len16(u_int nsegs, u_int tso)
 }
 
 /*
+ * len16 for a txpkt_vm WR with a GL.  Includes the firmware work
+ * request header.
+ */
+static inline u_int
+txpkt_vm_len16(u_int nsegs, u_int tso)
+{
+	u_int n;
+
+	MPASS(nsegs > 0);
+
+	nsegs--; /* first segment is part of ulptx_sgl */
+	n = sizeof(struct fw_eth_tx_pkt_vm_wr) +
+	    sizeof(struct cpl_tx_pkt_core) +
+	    sizeof(struct ulptx_sgl) + 8 * ((3 * nsegs) / 2 + (nsegs & 1));
+	if (tso)
+		n += sizeof(struct cpl_tx_pkt_lso_core);
+
+	return (howmany(n, 16));
+}
+
+/*
  * len16 for a txpkts type 0 WR with a GL.  Does not include the firmware work
  * request header.
  */
@@ -3957,6 +4024,181 @@ imm_payload(u_int ndesc)
 }
 
 /*
+ * Write a VM txpkt WR for this packet to the hardware descriptors, update the
+ * software descriptor, and advance the pidx.  It is guaranteed that enough
+ * descriptors are available.
+ *
+ * The return value is the # of hardware descriptors used.
+ */
+static u_int
+write_txpkt_vm_wr(struct sge_txq *txq, struct fw_eth_tx_pkt_vm_wr *wr,
+    struct mbuf *m0, u_int available)
+{
+	struct sge_eq *eq = &txq->eq;
+	struct tx_sdesc *txsd;
+	struct cpl_tx_pkt_core *cpl;
+	uint32_t ctrl;	/* used in many unrelated places */
+	uint64_t ctrl1;
+	int csum_type, len16, ndesc, pktlen, nsegs;
+	caddr_t dst;
+
+	TXQ_LOCK_ASSERT_OWNED(txq);
+	M_ASSERTPKTHDR(m0);
+	MPASS(available > 0 && available < eq->sidx);
+
+	len16 = mbuf_len16(m0);
+	nsegs = mbuf_nsegs(m0);
+	pktlen = m0->m_pkthdr.len;
+	ctrl = sizeof(struct cpl_tx_pkt_core);
+	if (needs_tso(m0))
+		ctrl += sizeof(struct cpl_tx_pkt_lso_core);
+	ndesc = howmany(len16, EQ_ESIZE / 16);
+	MPASS(ndesc <= available);
+
+	/* Firmware work request header */
+	MPASS(wr == (void *)&eq->desc[eq->pidx]);
+	wr->op_immdlen = htobe32(V_FW_WR_OP(FW_ETH_TX_PKT_VM_WR) |
+	    V_FW_ETH_TX_PKT_WR_IMMDLEN(ctrl));
+
+	ctrl = V_FW_WR_LEN16(len16);
+	wr->equiq_to_len16 = htobe32(ctrl);
+	wr->r3[0] = 0;
+	wr->r3[1] = 0;
+	
+	/*
+	 * Copy over ethmacdst, ethmacsrc, ethtype, and vlantci.
+	 * vlantci is ignored unless the ethtype is 0x8100, so it's
+	 * simpler to always copy it rather than making it
+	 * conditional.  Also, it seems that we do not have to set
+	 * vlantci or fake the ethtype when doing VLAN tag insertion.
+	 */
+	m_copydata(m0, 0, sizeof(struct ether_header) + 2, wr->ethmacdst);
+
+	csum_type = -1;
+	if (needs_tso(m0)) {
+		struct cpl_tx_pkt_lso_core *lso = (void *)(wr + 1);
+
+		KASSERT(m0->m_pkthdr.l2hlen > 0 && m0->m_pkthdr.l3hlen > 0 &&
+		    m0->m_pkthdr.l4hlen > 0,
+		    ("%s: mbuf %p needs TSO but missing header lengths",
+			__func__, m0));
+
+		ctrl = V_LSO_OPCODE(CPL_TX_PKT_LSO) | F_LSO_FIRST_SLICE |
+		    F_LSO_LAST_SLICE | V_LSO_IPHDR_LEN(m0->m_pkthdr.l3hlen >> 2)
+		    | V_LSO_TCPHDR_LEN(m0->m_pkthdr.l4hlen >> 2);
+		if (m0->m_pkthdr.l2hlen == sizeof(struct ether_vlan_header))
+			ctrl |= V_LSO_ETHHDR_LEN(1);
+		if (m0->m_pkthdr.l3hlen == sizeof(struct ip6_hdr))
+			ctrl |= F_LSO_IPV6;
+
+		lso->lso_ctrl = htobe32(ctrl);
+		lso->ipid_ofst = htobe16(0);
+		lso->mss = htobe16(m0->m_pkthdr.tso_segsz);
+		lso->seqno_offset = htobe32(0);
+		lso->len = htobe32(pktlen);
+
+		if (m0->m_pkthdr.l3hlen == sizeof(struct ip6_hdr))
+			csum_type = TX_CSUM_TCPIP6;
+		else
+			csum_type = TX_CSUM_TCPIP;
+
+		cpl = (void *)(lso + 1);
+
+		txq->tso_wrs++;
+	} else {
+		if (m0->m_pkthdr.csum_flags & CSUM_IP_TCP)
+			csum_type = TX_CSUM_TCPIP;
+		else if (m0->m_pkthdr.csum_flags & CSUM_IP_UDP)
+			csum_type = TX_CSUM_UDPIP;
+		else if (m0->m_pkthdr.csum_flags & CSUM_IP6_TCP)
+			csum_type = TX_CSUM_TCPIP6;
+		else if (m0->m_pkthdr.csum_flags & CSUM_IP6_UDP)
+			csum_type = TX_CSUM_UDPIP6;
+#if defined(INET)
+		else if (m0->m_pkthdr.csum_flags & CSUM_IP) {
+			/*
+			 * XXX: The firmware appears to stomp on the
+			 * fragment/flags field of the IP header when
+			 * using TX_CSUM_IP.  Fall back to doing
+			 * software checksums.
+			 */
+			u_short *sump;
+			struct mbuf *m;
+			int offset;
+
+			m = m0;
+			offset = 0;
+			sump = m_advance(&m, &offset, m0->m_pkthdr.l2hlen +
+			    offsetof(struct ip, ip_sum));
+			*sump = in_cksum_skip(m0, m0->m_pkthdr.l2hlen +
+			    m0->m_pkthdr.l3hlen, m0->m_pkthdr.l2hlen);
+			m0->m_pkthdr.csum_flags &= ~CSUM_IP;
+		}
+#endif
+
+		cpl = (void *)(wr + 1);
+	}
+
+	/* Checksum offload */
+	ctrl1 = 0;
+	if (needs_l3_csum(m0) == 0)
+		ctrl1 |= F_TXPKT_IPCSUM_DIS;
+	if (csum_type >= 0) {
+		KASSERT(m0->m_pkthdr.l2hlen > 0 && m0->m_pkthdr.l3hlen > 0,
+	    ("%s: mbuf %p needs checksum offload but missing header lengths",
+			__func__, m0));
+
+		/* XXX: T6 */
+		ctrl1 |= V_TXPKT_ETHHDR_LEN(m0->m_pkthdr.l2hlen -
+		    ETHER_HDR_LEN);
+		ctrl1 |= V_TXPKT_IPHDR_LEN(m0->m_pkthdr.l3hlen);
+		ctrl1 |= V_TXPKT_CSUM_TYPE(csum_type);
+	} else
+		ctrl1 |= F_TXPKT_L4CSUM_DIS;
+	if (m0->m_pkthdr.csum_flags & (CSUM_IP | CSUM_TCP | CSUM_UDP |
+	    CSUM_UDP_IPV6 | CSUM_TCP_IPV6 | CSUM_TSO))
+		txq->txcsum++;	/* some hardware assistance provided */
+
+	/* VLAN tag insertion */
+	if (needs_vlan_insertion(m0)) {
+		ctrl1 |= F_TXPKT_VLAN_VLD |
+		    V_TXPKT_VLAN(m0->m_pkthdr.ether_vtag);
+		txq->vlan_insertion++;
+	}
+
+	/* CPL header */
+	cpl->ctrl0 = txq->cpl_ctrl0;
+	cpl->pack = 0;
+	cpl->len = htobe16(pktlen);
+	cpl->ctrl1 = htobe64(ctrl1);
+
+	/* SGL */
+	dst = (void *)(cpl + 1);
+
+	/*
+	 * A packet using TSO will use up an entire descriptor for the
+	 * firmware work request header, LSO CPL, and TX_PKT_XT CPL.
+	 * If this descriptor is the last descriptor in the ring, wrap
+	 * around to the front of the ring explicitly for the start of
+	 * the sgl.
+	 */
+	if (dst == (void *)&eq->desc[eq->sidx]) {
+		dst = (void *)&eq->desc[0];
+		write_gl_to_txd(txq, m0, &dst, 0);
+	} else
+		write_gl_to_txd(txq, m0, &dst, eq->sidx - ndesc < eq->pidx);
+	txq->sgl_wrs++;
+
+	txq->txpkt_wrs++;
+
+	txsd = &txq->sdesc[eq->pidx];
+	txsd->m = m0;
+	txsd->desc_used = ndesc;
+
+	return (ndesc);
+}
+
+/*
  * Write a txpkt WR for this packet to the hardware descriptors, update the
  * software descriptor, and advance the pidx.  It is guaranteed that enough
  * descriptors are available.
@@ -4409,7 +4651,7 @@ ring_eq_db(struct adapter *sc, struct sge_eq *eq, u_int n)
 		break;
 
 	case DOORBELL_KDB:
-		t4_write_reg(sc, MYPF_REG(A_SGE_PF_KDOORBELL),
+		t4_write_reg(sc, sc->sge_kdoorbell_reg,
 		    V_QID(eq->cntxt_id) | V_PIDX(n));
 		break;
 	}
@@ -4755,7 +4997,7 @@ handle_sge_egr_update(struct sge_iq *iq, const struct rss_header *rss,
 	KASSERT(m == NULL, ("%s: payload with opcode %02x", __func__,
 	    rss->opcode));
 
-	eq = s->eqmap[qid - s->eq_start];
+	eq = s->eqmap[qid - s->eq_start - s->eq_base];
 	(*h[eq->flags & EQ_TYPEMASK])(sc, eq);
 
 	return (0);
diff --git a/sys/dev/cxgbe/t4_vf.c b/sys/dev/cxgbe/t4_vf.c
new file mode 100644
index 0000000..54a46d1
--- /dev/null
+++ b/sys/dev/cxgbe/t4_vf.c
@@ -0,0 +1,951 @@
+/*-
+ * Copyright (c) 2016 Chelsio Communications, Inc.
+ * All rights reserved.
+ * Written by: John Baldwin <jhb@FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include "opt_inet.h"
+#include "opt_inet6.h"
+
+#include <sys/param.h>
+#include <sys/bus.h>
+#include <sys/conf.h>
+#include <sys/counter.h>
+#include <sys/kernel.h>
+#include <sys/module.h>
+#include <sys/priv.h>
+#include <dev/pci/pcivar.h>
+#if defined(__i386__) || defined(__amd64__)
+#include <vm/vm.h>
+#include <vm/pmap.h>
+#endif
+
+#include "common/common.h"
+#include "common/t4_regs.h"
+#include "t4_ioctl.h"
+#include "t4_mp_ring.h"
+
+/*
+ * Some notes:
+ *
+ * The Virtual Interfaces are connected to an internal switch on the chip
+ * which allows VIs attached to the same port to talk to each other even when
+ * the port link is down.  As a result, we might want to always report a
+ * VF's link as being "up".
+ *
+ * XXX: Add a TUNABLE and possible per-device sysctl for this?
+ */
+
+struct intrs_and_queues {
+	uint16_t intr_type;	/* MSI, or MSI-X */
+	uint16_t nirq;		/* Total # of vectors */
+	uint16_t intr_flags_10g;/* Interrupt flags for each 10G port */
+	uint16_t intr_flags_1g;	/* Interrupt flags for each 1G port */
+	uint16_t ntxq10g;	/* # of NIC txq's for each 10G port */
+	uint16_t nrxq10g;	/* # of NIC rxq's for each 10G port */
+	uint16_t ntxq1g;	/* # of NIC txq's for each 1G port */
+	uint16_t nrxq1g;	/* # of NIC rxq's for each 1G port */
+};
+
+struct {
+	uint16_t device;
+	char *desc;
+} t4vf_pciids[] = {
+	{0x4800, "Chelsio T440-dbg VF"},
+	{0x4801, "Chelsio T420-CR VF"},
+	{0x4802, "Chelsio T422-CR VF"},
+	{0x4803, "Chelsio T440-CR VF"},
+	{0x4804, "Chelsio T420-BCH VF"},
+	{0x4805, "Chelsio T440-BCH VF"},
+	{0x4806, "Chelsio T440-CH VF"},
+	{0x4807, "Chelsio T420-SO VF"},
+	{0x4808, "Chelsio T420-CX VF"},
+	{0x4809, "Chelsio T420-BT VF"},
+	{0x480a, "Chelsio T404-BT VF"},
+	{0x480e, "Chelsio T440-LP-CR VF"},
+}, t5vf_pciids[] = {
+	{0x5800, "Chelsio T580-dbg VF"},
+	{0x5801,  "Chelsio T520-CR VF"},	/* 2 x 10G */
+	{0x5802,  "Chelsio T522-CR VF"},	/* 2 x 10G, 2 X 1G */
+	{0x5803,  "Chelsio T540-CR VF"},	/* 4 x 10G */
+	{0x5807,  "Chelsio T520-SO VF"},	/* 2 x 10G, nomem */
+	{0x5809,  "Chelsio T520-BT VF"},	/* 2 x 10GBaseT */
+	{0x580a,  "Chelsio T504-BT VF"},	/* 4 x 1G */
+	{0x580d,  "Chelsio T580-CR VF"},	/* 2 x 40G */
+	{0x580e,  "Chelsio T540-LP-CR VF"},	/* 4 x 10G */
+	{0x5810,  "Chelsio T580-LP-CR VF"},	/* 2 x 40G */
+	{0x5811,  "Chelsio T520-LL-CR VF"},	/* 2 x 10G */
+	{0x5812,  "Chelsio T560-CR VF"},	/* 1 x 40G, 2 x 10G */
+	{0x5814,  "Chelsio T580-LP-SO-CR VF"},	/* 2 x 40G, nomem */
+	{0x5815,  "Chelsio T502-BT VF"},	/* 2 x 1G */
+#ifdef notyet
+	{0x5804,  "Chelsio T520-BCH VF"},
+	{0x5805,  "Chelsio T540-BCH VF"},
+	{0x5806,  "Chelsio T540-CH VF"},
+	{0x5808,  "Chelsio T520-CX VF"},
+	{0x580b,  "Chelsio B520-SR VF"},
+	{0x580c,  "Chelsio B504-BT VF"},
+	{0x580f,  "Chelsio Amsterdam VF"},
+	{0x5813,  "Chelsio T580-CHR VF"},
+#endif
+};
+
+static d_ioctl_t t4vf_ioctl;
+
+static struct cdevsw t4vf_cdevsw = {
+       .d_version = D_VERSION,
+       .d_ioctl = t4vf_ioctl,
+       .d_name = "t4vf",
+};
+
+static int
+t4vf_probe(device_t dev)
+{
+	uint16_t d; 
+	size_t i;
+
+	d = pci_get_device(dev);
+	for (i = 0; i < nitems(t4vf_pciids); i++) {
+		if (d == t4vf_pciids[i].device) {
+			device_set_desc(dev, t4vf_pciids[i].desc);
+			return (BUS_PROBE_DEFAULT);
+		}
+	}
+	return (ENXIO);
+}
+
+static int
+t5vf_probe(device_t dev)
+{
+	uint16_t d; 
+	size_t i;
+
+	d = pci_get_device(dev);
+	for (i = 0; i < nitems(t5vf_pciids); i++) {
+		if (d == t5vf_pciids[i].device) {
+			device_set_desc(dev, t5vf_pciids[i].desc);
+			return (BUS_PROBE_DEFAULT);
+		}
+	}
+	return (ENXIO);
+}
+
+#define FW_PARAM_DEV(param) \
+	(V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_DEV) | \
+	 V_FW_PARAMS_PARAM_X(FW_PARAMS_PARAM_DEV_##param))
+#define FW_PARAM_PFVF(param) \
+	(V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_PFVF) | \
+	 V_FW_PARAMS_PARAM_X(FW_PARAMS_PARAM_PFVF_##param))
+
+static int
+get_params__pre_init(struct adapter *sc)
+{
+	int rc;
+	uint32_t param[3], val[3];
+
+	param[0] = FW_PARAM_DEV(FWREV);
+	param[1] = FW_PARAM_DEV(TPREV);
+	param[2] = FW_PARAM_DEV(CCLK);
+	rc = -t4vf_query_params(sc, nitems(param), param, val);
+	if (rc != 0) {
+		device_printf(sc->dev,
+		    "failed to query parameters (pre_init): %d.\n", rc);
+		return (rc);
+	}
+
+	sc->params.fw_vers = val[0];
+	sc->params.tp_vers = val[1];
+	sc->params.vpd.cclk = val[2];
+
+	snprintf(sc->fw_version, sizeof(sc->fw_version), "%u.%u.%u.%u",
+	    G_FW_HDR_FW_VER_MAJOR(sc->params.fw_vers),
+	    G_FW_HDR_FW_VER_MINOR(sc->params.fw_vers),
+	    G_FW_HDR_FW_VER_MICRO(sc->params.fw_vers),
+	    G_FW_HDR_FW_VER_BUILD(sc->params.fw_vers));
+
+	snprintf(sc->tp_version, sizeof(sc->tp_version), "%u.%u.%u.%u",
+	    G_FW_HDR_FW_VER_MAJOR(sc->params.tp_vers),
+	    G_FW_HDR_FW_VER_MINOR(sc->params.tp_vers),
+	    G_FW_HDR_FW_VER_MICRO(sc->params.tp_vers),
+	    G_FW_HDR_FW_VER_BUILD(sc->params.tp_vers));
+
+	return (0);
+}
+
+static int
+get_params__post_init(struct adapter *sc)
+{
+	int rc;
+
+	rc = -t4vf_get_sge_params(sc);
+	if (rc != 0) {
+		device_printf(sc->dev,
+		    "unable to retrieve adapter SGE parameters: %d\n", rc);
+		return (rc);
+	}
+
+	rc = -t4vf_get_rss_glb_config(sc);
+	if (rc != 0) {
+		device_printf(sc->dev,
+		    "unable to retrieve adapter RSS parameters: %d\n", rc);
+		return (rc);
+	}
+	if (sc->params.rss.mode != FW_RSS_GLB_CONFIG_CMD_MODE_BASICVIRTUAL) {
+		device_printf(sc->dev,
+		    "unable to operate with global RSS mode %d\n",
+		    sc->params.rss.mode);
+		return (EINVAL);
+	}
+
+	rc = t4_read_chip_settings(sc);
+	if (rc != 0)
+		return (rc);
+
+	/*
+	 * Grab our Virtual Interface resource allocation, extract the
+	 * features that we're interested in and do a bit of sanity testing on
+	 * what we discover.
+	 */
+	rc = -t4vf_get_vfres(sc);
+	if (rc != 0) {
+		device_printf(sc->dev,
+		    "unable to get virtual interface resources: %d\n", rc);
+		return (rc);
+	}
+
+	/*
+	 * Check for various parameter sanity issues.
+	 */
+	if (sc->params.vfres.pmask == 0) {
+		device_printf(sc->dev, "no port access configured/usable!\n");
+		return (EINVAL);
+	}
+	if (sc->params.vfres.nvi == 0) {
+		device_printf(sc->dev,
+		    "no virtual interfaces configured/usable!\n");
+		return (EINVAL);
+	}
+	sc->params.portvec = sc->params.vfres.pmask;
+
+	return (0);
+}
+
+static int
+set_params__post_init(struct adapter *sc)
+{
+	uint32_t param, val;
+
+	/* ask for encapsulated CPLs */
+	param = FW_PARAM_PFVF(CPLFW4MSG_ENCAP);
+	val = 1;
+	(void)t4vf_set_params(sc, 1, &param, &val);
+
+	return (0);
+}
+
+#undef FW_PARAM_PFVF
+#undef FW_PARAM_DEV
+
+static int
+cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
+    struct intrs_and_queues *iaq)
+{
+	struct vf_resources *vfres;
+	int nrxq10g, nrxq1g, nrxq;
+	int ntxq10g, ntxq1g, ntxq;
+	int itype, iq_avail, navail, rc;
+
+	/*
+	 * Figure out the layout of queues across our VIs and ensure
+	 * we can allocate enough interrupts for our layout.
+	 */
+	vfres = &sc->params.vfres;
+	bzero(iaq, sizeof(*iaq));
+
+	for (itype = INTR_MSIX; itype != 0; itype >>= 1) {
+		if (itype == INTR_INTX)
+			continue;
+
+		if (itype == INTR_MSIX)
+			navail = pci_msix_count(sc->dev);
+		else
+			navail = pci_msi_count(sc->dev);
+
+		if (navail == 0)
+			continue;
+
+		iaq->intr_type = itype;
+		iaq->intr_flags_10g = 0;
+		iaq->intr_flags_1g = 0;
+
+		/*
+		 * XXX: The Linux driver reserves an Ingress Queue for
+		 * forwarded interrupts when using MSI (but not MSI-X).
+		 * It seems it just always asks for 2 interrupts and
+		 * forwards all rxqs to the forwarded interrupt.
+		 *
+		 * We must reserve one IRQ for the for the firmware
+		 * event queue.
+		 *
+		 * Every rxq requires an ingress queue with a free
+		 * list and interrupts and an egress queue.  Every txq
+		 * requires an ETH egress queue.
+		 */
+		iaq->nirq = T4VF_EXTRA_INTR;
+
+		/*
+		 * First, determine how many queues we can allocate.
+		 * Start by finding the upper bound on rxqs from the
+		 * limit on ingress queues.
+		 */
+		iq_avail = vfres->niqflint - iaq->nirq;
+		if (iq_avail < n10g + n1g) {
+			device_printf(sc->dev,
+			    "Not enough ingress queues (%d) for %d ports\n",
+			    vfres->niqflint, n10g + n1g);
+			return (ENXIO);
+		}
+
+		/*
+		 * Try to honor the cap on interrupts.  If there aren't
+		 * enough interrupts for at least one interrupt per
+		 * port, then don't bother, we will just forward all
+		 * interrupts to one interrupt in that case.
+		 */
+		if (iaq->nirq + n10g + n1g <= navail) {
+			if (iq_avail > navail - iaq->nirq)
+				iq_avail = navail - iaq->nirq;
+		}
+
+		nrxq10g = t4_nrxq10g;
+		nrxq1g = t4_nrxq1g;
+		nrxq = n10g * nrxq10g + n1g * nrxq1g;
+		if (nrxq > iq_avail && nrxq1g > 1) {
+			/* Too many ingress queues.  Try just 1 for 1G. */
+			nrxq1g = 1;
+			nrxq = n10g * nrxq10g + n1g * nrxq1g;
+		}
+		if (nrxq > iq_avail) {
+			/*
+			 * Still too many ingress queues.  Use what we
+			 * can for each 10G port.
+			 */
+			nrxq10g = (iq_avail - n1g) / n10g;
+			nrxq = n10g * nrxq10g + n1g * nrxq1g;
+		}
+		KASSERT(nrxq <= iq_avail, ("too many ingress queues"));
+
+		/*
+		 * Next, determine the upper bound on txqs from the limit
+		 * on ETH queues.
+		 */
+		if (vfres->nethctrl < n10g + n1g) {
+			device_printf(sc->dev,
+			    "Not enough ETH queues (%d) for %d ports\n",
+			    vfres->nethctrl, n10g + n1g);
+			return (ENXIO);
+		}
+
+		ntxq10g = t4_ntxq10g;
+		ntxq1g = t4_ntxq1g;
+		ntxq = n10g * ntxq10g + n1g * ntxq1g;
+		if (ntxq > vfres->nethctrl) {
+			/* Too many ETH queues.  Try just 1 for 1G. */
+			ntxq1g = 1;
+			ntxq = n10g * ntxq10g + n1g * ntxq1g;
+		}
+		if (ntxq > vfres->nethctrl) {
+			/*
+			 * Still too many ETH queues.  Use what we
+			 * can for each 10G port.
+			 */
+			ntxq10g = (vfres->nethctrl - n1g) / n10g;
+			ntxq = n10g * ntxq10g + n1g * ntxq1g;
+		}
+		KASSERT(ntxq <= vfres->nethctrl, ("too many ETH queues"));
+
+		/*
+		 * Finally, ensure we have enough egress queues.
+		 */
+		if (vfres->neq < (n10g + n1g) * 2) {
+			device_printf(sc->dev,
+			    "Not enough egress queues (%d) for %d ports\n",
+			    vfres->neq, n10g + n1g);
+			return (ENXIO);
+		}
+		if (nrxq + ntxq > vfres->neq) {
+			/* Just punt and use 1 for everything. */
+			nrxq1g = ntxq1g = nrxq10g = ntxq10g = 1;
+			nrxq = n10g * nrxq10g + n1g * nrxq1g;
+			ntxq = n10g * ntxq10g + n1g * ntxq1g;
+		}
+		KASSERT(nrxq <= iq_avail, ("too many ingress queues"));
+		KASSERT(ntxq <= vfres->nethctrl, ("too many ETH queues"));
+		KASSERT(nrxq + ntxq <= vfres->neq, ("too many egress queues"));
+
+		/*
+		 * Do we have enough interrupts?  For MSI the interrupts
+		 * have to be a power of 2 as well.
+		 */
+		iaq->nirq += nrxq;
+		iaq->ntxq10g = ntxq10g;
+		iaq->ntxq1g = ntxq1g;
+		iaq->nrxq10g = nrxq10g;
+		iaq->nrxq1g = nrxq1g;
+		if (iaq->nirq <= navail &&
+		    (itype != INTR_MSI || powerof2(iaq->nirq))) {
+			navail = iaq->nirq;
+			if (itype == INTR_MSIX)
+				rc = pci_alloc_msix(sc->dev, &navail);
+			else
+				rc = pci_alloc_msi(sc->dev, &navail);
+			if (rc != 0) {
+				device_printf(sc->dev,
+		    "failed to allocate vectors:%d, type=%d, req=%d, rcvd=%d\n",
+				    itype, rc, iaq->nirq, navail);
+				return (rc);
+			}
+			if (navail == iaq->nirq) {
+				iaq->intr_flags_10g = INTR_RXQ;
+				iaq->intr_flags_1g = INTR_RXQ;
+				return (0);
+			}
+			pci_release_msi(sc->dev);
+		}
+
+		/* Fall back to a single interrupt. */
+		iaq->nirq = 1;
+		navail = iaq->nirq;
+		if (itype == INTR_MSIX)
+			rc = pci_alloc_msix(sc->dev, &navail);
+		else
+			rc = pci_alloc_msi(sc->dev, &navail);
+		if (rc != 0)
+			device_printf(sc->dev,
+		    "failed to allocate vectors:%d, type=%d, req=%d, rcvd=%d\n",
+			    itype, rc, iaq->nirq, navail);
+		iaq->intr_flags_10g = 0;
+		iaq->intr_flags_1g = 0;
+		return (rc);
+	}
+
+	device_printf(sc->dev,
+	    "failed to find a usable interrupt type.  "
+	    "allowed=%d, msi-x=%d, msi=%d, intx=1", t4_intr_types,
+	    pci_msix_count(sc->dev), pci_msi_count(sc->dev));
+
+	return (ENXIO);
+}
+
+static int
+t4vf_attach(device_t dev)
+{
+	struct adapter *sc;
+	int rc = 0, i, j, n10g, n1g, rqidx, tqidx;
+	struct make_dev_args mda;
+	struct intrs_and_queues iaq;
+	struct sge *s;
+
+	sc = device_get_softc(dev);
+	sc->dev = dev;
+	pci_enable_busmaster(dev);
+	pci_set_max_read_req(dev, 4096);
+	sc->params.pci.mps = pci_get_max_payload(dev);
+
+	sc->flags |= IS_VF;
+
+	sc->sge_gts_reg = VF_SGE_REG(A_SGE_VF_GTS);
+	sc->sge_kdoorbell_reg = VF_SGE_REG(A_SGE_VF_KDOORBELL);
+	snprintf(sc->lockname, sizeof(sc->lockname), "%s",
+	    device_get_nameunit(dev));
+	mtx_init(&sc->sc_lock, sc->lockname, 0, MTX_DEF);
+	t4_add_adapter(sc);
+
+	mtx_init(&sc->sfl_lock, "starving freelists", 0, MTX_DEF);
+	TAILQ_INIT(&sc->sfl);
+	callout_init_mtx(&sc->sfl_callout, &sc->sfl_lock, 0);
+
+	mtx_init(&sc->reg_lock, "indirect register access", 0, MTX_DEF);
+
+	rc = t4_map_bars_0_and_4(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	rc = -t4vf_prep_adapter(sc);
+	if (rc != 0)
+		goto done;
+
+	/*
+	 * Leave the 'pf' and 'mbox' values as zero.  This ensures
+	 * that various firmware messages do not set the fields which
+	 * is the correct thing to do for a VF.
+	 */
+
+	memset(sc->chan_map, 0xff, sizeof(sc->chan_map));
+
+	make_dev_args_init(&mda);
+	mda.mda_devsw = &t4vf_cdevsw;
+	mda.mda_uid = UID_ROOT;
+	mda.mda_gid = GID_WHEEL;
+	mda.mda_mode = 0600;
+	mda.mda_si_drv1 = sc;
+	rc = make_dev_s(&mda, &sc->cdev, "%s", device_get_nameunit(dev));
+	if (rc != 0)
+		device_printf(dev, "failed to create nexus char device: %d.\n",
+		    rc);
+
+#if defined(__i386__)
+	if ((cpu_feature & CPUID_CX8) == 0) {
+		device_printf(dev, "64 bit atomics not available.\n");
+		rc = ENOTSUP;
+		goto done;
+	}
+#endif
+
+	/*
+	 * Some environments do not properly handle PCIE FLRs -- e.g. in Linux
+	 * 2.6.31 and later we can't call pci_reset_function() in order to
+	 * issue an FLR because of a self- deadlock on the device semaphore.
+	 * Meanwhile, the OS infrastructure doesn't issue FLRs in all the
+	 * cases where they're needed -- for instance, some versions of KVM
+	 * fail to reset "Assigned Devices" when the VM reboots.  Therefore we
+	 * use the firmware based reset in order to reset any per function
+	 * state.
+	 */
+	rc = -t4vf_fw_reset(sc);
+	if (rc != 0) {
+		device_printf(dev, "FW reset failed: %d\n", rc);
+		goto done;
+	}
+	sc->flags |= FW_OK;
+
+	/*
+	 * Grab basic operational parameters.  These will predominantly have
+	 * been set up by the Physical Function Driver or will be hard coded
+	 * into the adapter.  We just have to live with them ...  Note that
+	 * we _must_ get our VPD parameters before our SGE parameters because
+	 * we need to know the adapter's core clock from the VPD in order to
+	 * properly decode the SGE Timer Values.
+	 */
+	rc = get_params__pre_init(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+	rc = get_params__post_init(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	rc = set_params__post_init(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	rc = t4_map_bar_2(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	rc = t4_create_dma_tag(sc);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	/*
+	 * The number of "ports" which we support is equal to the number of
+	 * Virtual Interfaces with which we've been provisioned.
+	 */
+	sc->params.nports = imin(sc->params.vfres.nvi, MAX_NPORTS);
+
+	/*
+	 * We may have been provisioned with more VIs than the number of
+	 * ports we're allowed to access (our Port Access Rights Mask).
+	 * Just use a single VI for each port.
+	 */
+	sc->params.nports = imin(sc->params.nports,
+	    bitcount32(sc->params.vfres.pmask));
+
+#ifdef notyet
+	/*
+	 * XXX: The Linux VF driver will lower nports if it thinks there
+	 * are too few resources in vfres (niqflint, nethctrl, neq).
+	 */
+#endif
+
+	/*
+	 * First pass over all the ports - allocate VIs and initialize some
+	 * basic parameters like mac address, port type, etc.  We also figure
+	 * out whether a port is 10G or 1G and use that information when
+	 * calculating how many interrupts to attempt to allocate.
+	 */
+	n10g = n1g = 0;
+	for_each_port(sc, i) {
+		struct port_info *pi;
+
+		pi = malloc(sizeof(*pi), M_CXGBE, M_ZERO | M_WAITOK);
+		sc->port[i] = pi;
+
+		/* These must be set before t4_port_init */
+		pi->adapter = sc;
+		pi->port_id = i;
+		pi->nvi = 1;
+		pi->vi = malloc(sizeof(struct vi_info) * pi->nvi, M_CXGBE,
+		    M_ZERO | M_WAITOK);
+
+		/*
+		 * Allocate the "main" VI and initialize parameters
+		 * like mac addr.
+		 */
+		rc = -t4_port_init(sc, sc->mbox, sc->pf, 0, i);
+		if (rc != 0) {
+			device_printf(dev, "unable to initialize port %d: %d\n",
+			    i, rc);
+			free(pi->vi, M_CXGBE);
+			free(pi, M_CXGBE);
+			sc->port[i] = NULL;
+			goto done;
+		}
+
+		/* No t4_link_start. */
+
+		snprintf(pi->lockname, sizeof(pi->lockname), "%sp%d",
+		    device_get_nameunit(dev), i);
+		mtx_init(&pi->pi_lock, pi->lockname, 0, MTX_DEF);
+		sc->chan_map[pi->tx_chan] = i;
+
+		pi->tc = malloc(sizeof(struct tx_sched_class) *
+		    sc->chip_params->nsched_cls, M_CXGBE, M_ZERO | M_WAITOK);
+
+		if (is_10G_port(pi) || is_40G_port(pi)) {
+			n10g++;
+		} else {
+			n1g++;
+		}
+
+		pi->linkdnrc = -1;
+
+		pi->dev = device_add_child(dev, is_t4(sc) ? "cxgbev" : "cxlv",
+		    -1);
+		if (pi->dev == NULL) {
+			device_printf(dev,
+			    "failed to add device for port %d.\n", i);
+			rc = ENXIO;
+			goto done;
+		}
+		pi->vi[0].dev = pi->dev;
+		device_set_softc(pi->dev, pi);
+	}
+		
+	/*
+	 * Interrupt type, # of interrupts, # of rx/tx queues, etc.
+	 */
+	rc = cfg_itype_and_nqueues(sc, n10g, n1g, &iaq);
+	if (rc != 0)
+		goto done; /* error message displayed already */
+
+	sc->intr_type = iaq.intr_type;
+	sc->intr_count = iaq.nirq;
+
+	s = &sc->sge;
+	s->nrxq = n10g * iaq.nrxq10g + n1g * iaq.nrxq1g;
+	s->ntxq = n10g * iaq.ntxq10g + n1g * iaq.ntxq1g;
+	s->neq = s->ntxq + s->nrxq;	/* the free list in an rxq is an eq */
+	s->neq += sc->params.nports + 1;/* ctrl queues: 1 per port + 1 mgmt */
+	s->niq = s->nrxq + 1;		/* 1 extra for firmware event queue */
+
+	s->rxq = malloc(s->nrxq * sizeof(struct sge_rxq), M_CXGBE,
+	    M_ZERO | M_WAITOK);
+	s->txq = malloc(s->ntxq * sizeof(struct sge_txq), M_CXGBE,
+	    M_ZERO | M_WAITOK);
+	s->iqmap = malloc(s->niq * sizeof(struct sge_iq *), M_CXGBE,
+	    M_ZERO | M_WAITOK);
+	s->eqmap = malloc(s->neq * sizeof(struct sge_eq *), M_CXGBE,
+	    M_ZERO | M_WAITOK);
+
+	sc->irq = malloc(sc->intr_count * sizeof(struct irq), M_CXGBE,
+	    M_ZERO | M_WAITOK);
+
+	/*
+	 * Second pass over the ports.  This time we know the number of rx and
+	 * tx queues that each port should get.
+	 */
+	rqidx = tqidx = 0;
+	for_each_port(sc, i) {
+		struct port_info *pi = sc->port[i];
+		struct vi_info *vi;
+
+		if (pi == NULL)
+			continue;
+
+		for_each_vi(pi, j, vi) {
+			vi->pi = pi;
+			vi->qsize_rxq = t4_qsize_rxq;
+			vi->qsize_txq = t4_qsize_txq;
+
+			vi->first_rxq = rqidx;
+			vi->first_txq = tqidx;
+			if (is_10G_port(pi) || is_40G_port(pi)) {
+				vi->tmr_idx = t4_tmr_idx_10g;
+				vi->pktc_idx = t4_pktc_idx_10g;
+				vi->flags |= iaq.intr_flags_10g & INTR_RXQ;
+				vi->nrxq = j == 0 ? iaq.nrxq10g : 1;
+				vi->ntxq = j == 0 ? iaq.ntxq10g : 1;
+			} else {
+				vi->tmr_idx = t4_tmr_idx_1g;
+				vi->pktc_idx = t4_pktc_idx_1g;
+				vi->flags |= iaq.intr_flags_1g & INTR_RXQ;
+				vi->nrxq = j == 0 ? iaq.nrxq1g : 1;
+				vi->ntxq = j == 0 ? iaq.ntxq1g : 1;
+			}
+			rqidx += vi->nrxq;
+			tqidx += vi->ntxq;
+
+			vi->rsrv_noflowq = 0;
+		}
+	}
+
+	rc = t4_setup_intr_handlers(sc);
+	if (rc != 0) {
+		device_printf(dev,
+		    "failed to setup interrupt handlers: %d\n", rc);
+		goto done;
+	}
+
+	rc = bus_generic_attach(dev);
+	if (rc != 0) {
+		device_printf(dev,
+		    "failed to attach all child ports: %d\n", rc);
+		goto done;
+	}
+
+	device_printf(dev,
+	    "%d ports, %d %s interrupt%s, %d eq, %d iq\n",
+	    sc->params.nports, sc->intr_count, sc->intr_type == INTR_MSIX ?
+	    "MSI-X" : "MSI", sc->intr_count > 1 ? "s" : "", sc->sge.neq,
+	    sc->sge.niq);
+
+done:
+	if (rc != 0)
+		t4_detach_common(dev);
+	else
+		t4_sysctls(sc);
+
+	return (rc);
+}
+
+static void
+get_regs(struct adapter *sc, struct t4_regdump *regs, uint8_t *buf)
+{
+
+	/* 0x3f is used as the revision for VFs. */
+	regs->version = chip_id(sc) | (0x3f << 10);
+	t4_get_regs(sc, buf, regs->len);
+}
+
+static void
+t4_clr_vi_stats(struct adapter *sc)
+{
+	int reg;
+
+	for (reg = A_MPS_VF_STAT_TX_VF_BCAST_BYTES_L;
+	     reg <= A_MPS_VF_STAT_RX_VF_ERR_FRAMES_H; reg += 4)
+		t4_write_reg(sc, VF_MPS_REG(reg), 0);
+}
+
+static int
+t4vf_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag,
+    struct thread *td)
+{
+	int rc;
+	struct adapter *sc = dev->si_drv1;
+
+	rc = priv_check(td, PRIV_DRIVER);
+	if (rc != 0)
+		return (rc);
+
+	switch (cmd) {
+	case CHELSIO_T4_GETREG: {
+		struct t4_reg *edata = (struct t4_reg *)data;
+
+		if ((edata->addr & 0x3) != 0 || edata->addr >= sc->mmio_len)
+			return (EFAULT);
+
+		if (edata->size == 4)
+			edata->val = t4_read_reg(sc, edata->addr);
+		else if (edata->size == 8)
+			edata->val = t4_read_reg64(sc, edata->addr);
+		else
+			return (EINVAL);
+
+		break;
+	}
+	case CHELSIO_T4_SETREG: {
+		struct t4_reg *edata = (struct t4_reg *)data;
+
+		if ((edata->addr & 0x3) != 0 || edata->addr >= sc->mmio_len)
+			return (EFAULT);
+
+		if (edata->size == 4) {
+			if (edata->val & 0xffffffff00000000)
+				return (EINVAL);
+			t4_write_reg(sc, edata->addr, (uint32_t) edata->val);
+		} else if (edata->size == 8)
+			t4_write_reg64(sc, edata->addr, edata->val);
+		else
+			return (EINVAL);
+		break;
+	}
+	case CHELSIO_T4_REGDUMP: {
+		struct t4_regdump *regs = (struct t4_regdump *)data;
+		int reglen = t4_get_regs_len(sc);
+		uint8_t *buf;
+
+		if (regs->len < reglen) {
+			regs->len = reglen; /* hint to the caller */
+			return (ENOBUFS);
+		}
+
+		regs->len = reglen;
+		buf = malloc(reglen, M_CXGBE, M_WAITOK | M_ZERO);
+		get_regs(sc, regs, buf);
+		rc = copyout(buf, regs->data, reglen);
+		free(buf, M_CXGBE);
+		break;
+	}
+	case CHELSIO_T4_CLEAR_STATS: {
+		int i, v;
+		u_int port_id = *(uint32_t *)data;
+		struct port_info *pi;
+		struct vi_info *vi;
+
+		if (port_id >= sc->params.nports)
+			return (EINVAL);
+		pi = sc->port[port_id];
+
+		/* MAC stats */
+		pi->tx_parse_error = 0;
+		t4_clr_vi_stats(sc);
+
+		/*
+		 * Since this command accepts a port, clear stats for
+		 * all VIs on this port.
+		 */
+		for_each_vi(pi, v, vi) {
+			if (vi->flags & VI_INIT_DONE) {
+				struct sge_rxq *rxq;
+				struct sge_txq *txq;
+
+				for_each_rxq(vi, i, rxq) {
+#if defined(INET) || defined(INET6)
+					rxq->lro.lro_queued = 0;
+					rxq->lro.lro_flushed = 0;
+#endif
+					rxq->rxcsum = 0;
+					rxq->vlan_extraction = 0;
+				}
+
+				for_each_txq(vi, i, txq) {
+					txq->txcsum = 0;
+					txq->tso_wrs = 0;
+					txq->vlan_insertion = 0;
+					txq->imm_wrs = 0;
+					txq->sgl_wrs = 0;
+					txq->txpkt_wrs = 0;
+					txq->txpkts0_wrs = 0;
+					txq->txpkts1_wrs = 0;
+					txq->txpkts0_pkts = 0;
+					txq->txpkts1_pkts = 0;
+					mp_ring_reset_stats(txq->r);
+				}
+			}
+		}
+		break;
+	}
+	case CHELSIO_T4_SCHED_CLASS:
+		rc = t4_set_sched_class(sc, (struct t4_sched_params *)data);
+		break;
+	case CHELSIO_T4_SCHED_QUEUE:
+		rc = t4_set_sched_queue(sc, (struct t4_sched_queue *)data);
+		break;
+	default:
+		rc = ENOTTY;
+	}
+
+	return (rc);
+}
+
+static device_method_t t4vf_methods[] = {
+	DEVMETHOD(device_probe,		t4vf_probe),
+	DEVMETHOD(device_attach,	t4vf_attach),
+	DEVMETHOD(device_detach,	t4_detach_common),
+
+	DEVMETHOD_END
+};
+
+static driver_t t4vf_driver = {
+	"t4vf",
+	t4vf_methods,
+	sizeof(struct adapter)
+};
+
+static device_method_t t5vf_methods[] = {
+	DEVMETHOD(device_probe,		t5vf_probe),
+	DEVMETHOD(device_attach,	t4vf_attach),
+	DEVMETHOD(device_detach,	t4_detach_common),
+
+	DEVMETHOD_END
+};
+
+static driver_t t5vf_driver = {
+	"t5vf",
+	t5vf_methods,
+	sizeof(struct adapter)
+};
+
+static driver_t cxgbev_driver = {
+	"cxgbev",
+	cxgbe_methods,
+	sizeof(struct port_info)
+};
+
+static driver_t cxlv_driver = {
+	"cxlv",
+	cxgbe_methods,
+	sizeof(struct port_info)
+};
+
+static devclass_t t4vf_devclass, t5vf_devclass;
+static devclass_t cxgbev_devclass, cxlv_devclass;
+
+DRIVER_MODULE(t4vf, pci, t4vf_driver, t4vf_devclass, 0, 0);
+MODULE_VERSION(t4vf, 1);
+MODULE_DEPEND(t4vf, t4nex, 1, 1, 1);
+
+DRIVER_MODULE(t5vf, pci, t5vf_driver, t5vf_devclass, 0, 0);
+MODULE_VERSION(t5vf, 1);
+MODULE_DEPEND(t5vf, t5nex, 1, 1, 1);
+
+DRIVER_MODULE(cxgbev, t4vf, cxgbev_driver, cxgbev_devclass, 0, 0);
+MODULE_VERSION(cxgbev, 1);
+
+DRIVER_MODULE(cxlv, t5vf, cxlv_driver, cxlv_devclass, 0, 0);
+MODULE_VERSION(cxlv, 1);
diff --git a/sys/modules/cxgbe/Makefile b/sys/modules/cxgbe/Makefile
index 5c9ac56..30a40c7 100644
--- a/sys/modules/cxgbe/Makefile
+++ b/sys/modules/cxgbe/Makefile
@@ -4,6 +4,8 @@
 
 SUBDIR=	if_cxgbe
 SUBDIR+=	if_cxl
+SUBDIR+=	if_cxgbev
+SUBDIR+=	if_cxlv
 SUBDIR+=	t4_firmware
 SUBDIR+=	t5_firmware
 SUBDIR+=	${_tom}
diff --git a/sys/modules/cxgbe/if_cxgbev/Makefile b/sys/modules/cxgbe/if_cxgbev/Makefile
new file mode 100644
index 0000000..e6cebc5
--- /dev/null
+++ b/sys/modules/cxgbe/if_cxgbev/Makefile
@@ -0,0 +1,21 @@
+#
+# $FreeBSD$
+#
+
+CXGBE=	${.CURDIR}/../../../dev/cxgbe
+.PATH: ${CXGBE} ${CXGBE}/common
+
+KMOD=	if_cxgbev
+SRCS=	bus_if.h
+SRCS+=	device_if.h
+SRCS+=	opt_inet.h
+SRCS+=	opt_inet6.h
+SRCS+=	opt_ofed.h
+SRCS+=	opt_rss.h
+SRCS+=	pci_if.h
+SRCS+=	t4_vf.c
+SRCS+=	t4vf_hw.c
+
+CFLAGS+= -I${CXGBE}
+
+.include <bsd.kmod.mk>
diff --git a/sys/modules/cxgbe/if_cxlv/Makefile b/sys/modules/cxgbe/if_cxlv/Makefile
new file mode 100644
index 0000000..943ea58
--- /dev/null
+++ b/sys/modules/cxgbe/if_cxlv/Makefile
@@ -0,0 +1,11 @@
+#
+# $FreeBSD$
+#
+
+CXGBE=	${.CURDIR}/../../../dev/cxgbe
+.PATH: ${CXGBE}
+
+KMOD=	if_cxlv
+SRCS=	if_cxlv.c
+
+.include <bsd.kmod.mk>
diff --git a/sys/powerpc/conf/NOTES b/sys/powerpc/conf/NOTES
index ac0217c..3e2628d 100644
--- a/sys/powerpc/conf/NOTES
+++ b/sys/powerpc/conf/NOTES
@@ -58,6 +58,7 @@ device		windtunnel	# Apple G4 MDD fan controller
 
 nodevice	bktr
 nodevice	cxgbe		# XXX: builds on powerpc64 only.
+nodevice	cxgbev
 nodevice	fdc
 nodevice	ppc
 nodevice	splash