First cut at MI support for PCI Message Signalled Interrupts (MSI):

- Add 3 new functions to the pci_if interface along with suitable wrappers
  to provide the device driver visible API:
  - pci_alloc_msi(dev, int *count) backed by PCI_ALLOC_MSI().  '*count'
    here is an in and out parameter.  The driver stores the desired number
    of messages in '*count' before calling the function.  On success,
    '*count' holds the number of messages allocated to the device.  Also on
    success, the driver can access the messages as SYS_RES_IRQ resources
    starting at rid 1.  Note that the legacy INTx interrupt resource will
    not be available when using MSI.  Note that this function will allocate
    either MSI or MSI-X messages depending on the devices capabilities and
    the 'hw.pci.enable_msix' and 'hw.pci.enable_msi' tunables.  Also note
    that the driver should activate the memory resource that holds the
    MSI-X table and pending bit array (PBA) before calling this function
    if the device supports MSI-X.
  - pci_release_msi(dev) backed by PCI_RELEASE_MSI().  This function
    releases the messages allocated for this device.  All of the
    SYS_RES_IRQ resources need to be released for this function to succeed.
  - pci_msi_count(dev) backed by PCI_MSI_COUNT().  This function returns
    the maximum number of MSI or MSI-X messages supported by this device.
    MSI-X is preferred if present, but this function will honor the
    'hw.pci.enable_msix' and 'hw.pci.enable_msi' tunables.  This function
    should return the largest value that pci_alloc_msi() can return
    (assuming the MD code is able to allocate sufficient backing resources
    for all of the messages).
- Add default implementations for these 3 methods to the pci_driver generic
  PCI bus driver.  (The various other PCI bus drivers such as for ACPI and
  OFW will inherit these default implementations.)  This default
  implementation depends on 4 new pcib_if methods that bubble up through
  the PCI bridges to the MD code to allocate IRQ values and perform any
  needed MD setup code needed:
  - PCIB_ALLOC_MSI() attempts to allocate a group of MSI messages.
  - PCIB_RELEASE_MSI() releases a group of MSI messages.
  - PCIB_ALLOC_MSIX() attempts to allocate a single MSI-X message.
  - PCIB_RELEASE_MSIX() releases a single MSI-X message.
- Add default implementations for these 4 methods that just pass the
  request up to the parent bus's parent bridge driver and use the
  default implementation in the various MI PCI bridge drivers.
- Add MI functions for use by MD code when managing MSI and MSI-X
  interrupts:
  - pci_enable_msi(dev, address, data) programs the MSI capability address
    and data registers for a group of MSI messages
  - pci_enable_msix(dev, index, address, data) initializes a single MSI-X
    message in the MSI-X table
  - pci_mask_msix(dev, index) masks a single MSI-X message
  - pci_unmask_msix(dev, index) unmasks a single MSI-X message
  - pci_pending_msix(dev, index) returns true if the specified MSI-X
    message is currently pending
- Save the MSI capability address and data registers in the pci_cfgreg
  block in a PCI devices ivars and restore the values when a device is
  resumed.  Note that the MSI-X table is not currently restored during
  resume.
- Add constants for MSI-X register offsets and fields.
- Record interesting data about any MSI-X capability blocks we come
  across in the pci_cfgreg block in the ivars for PCI devices.

Tested on:	em (i386, MSI), bce (amd64/i386, MSI), mpt (amd64, MSI-X)
Reviewed by:	scottl, grehan, jfv
MFC after:	2 months

1 files changed, 46 insertions, 1 deletions

diff --git a/sys/dev/pci/pcivar.h b/sys/dev/pci/pcivar.h
index 1c026c9..cdb3fdf 100644
--- a/sys/dev/pci/pcivar.h
+++ b/sys/dev/pci/pcivar.h
@@ -77,8 +77,25 @@ struct pcicfg_vpd {
 /* Interesting values for PCI MSI */
 struct pcicfg_msi {
     uint16_t	msi_ctrl;	/* Message Control */
+    uint8_t	msi_location;	/* Offset of MSI capability registers. */
     uint8_t	msi_msgnum;	/* Number of messages */
-    uint16_t	msi_data;	/* Location of MSI data word */
+    int		msi_alloc;	/* Number of allocated messages. */
+    uint64_t	msi_addr;	/* Contents of address register. */
+    uint16_t	msi_data;	/* Contents of data register. */
+};
+
+/* Interesting values for PCI MSI */
+struct pcicfg_msix {
+    uint16_t	msix_ctrl;	/* Message Control */
+    uint8_t	msix_location;	/* Offset of MSI capability registers. */
+    uint16_t	msix_msgnum;	/* Number of messages */
+    int		msix_alloc;	/* Number of allocated messages. */
+    uint8_t	msix_table_bar;	/* BAR containing vector table. */
+    uint8_t	msix_pba_bar;	/* BAR containing PBA. */
+    uint32_t	msix_table_offset;
+    uint32_t	msix_pba_offset;
+    struct resource *msix_table_res;	/* Resource containing vector table. */
+    struct resource *msix_pba_res;	/* Resource containing PBA. */
 };
 
 /* config header information common to all header types */
@@ -120,6 +137,7 @@ typedef struct pcicfg {
     struct pcicfg_pp pp;	/* pci power management */
     struct pcicfg_vpd vpd;	/* pci vital product data */
     struct pcicfg_msi msi;	/* pci msi */
+    struct pcicfg_msix msix;	/* pci msi-x */
 } pcicfgregs;
 
 /* additional type 1 device config header information (PCI to PCI bridge) */
@@ -371,8 +389,35 @@ pci_find_extcap(device_t dev, int capability, int *capreg)
     return PCI_FIND_EXTCAP(device_get_parent(dev), dev, capability, capreg);
 }
 
+static __inline int
+pci_alloc_msi(device_t dev, int *count)
+{
+    return (PCI_ALLOC_MSI(device_get_parent(dev), dev, count));
+}
+
+static __inline int
+pci_release_msi(device_t dev)
+{
+    return (PCI_RELEASE_MSI(device_get_parent(dev), dev));
+}
+
+static __inline int
+pci_msi_count(device_t dev)
+{
+    return (PCI_MSI_COUNT(device_get_parent(dev), dev));
+}
+
 device_t pci_find_bsf(uint8_t, uint8_t, uint8_t);
 device_t pci_find_device(uint16_t, uint16_t);
+
+/* Used by MD code to program MSI and MSI-X registers. */
+void	pci_enable_msi(device_t dev, uint64_t address, uint16_t data);
+void	pci_enable_msix(device_t dev, u_int index, uint64_t address,
+    uint32_t data);
+void	pci_mask_msix(device_t dev, u_int index);
+int	pci_pending_msix(device_t dev, u_int index);
+void	pci_unmask_msix(device_t dev, u_int index);
+
 #endif	/* _SYS_BUS_H_ */
 
 /*