Posts in this series:

PCIe

PCIe like its predecessor PCI is a computer bus that passes transactions between devices. The vast majority of these transactions are called “Memory Write” and “Memory Read” transactions. These are, philosophically, the exact same memory transactions that the CPU uses to access RAM. Therefore, PCIe devices are very easily controlled using Memory Mapped I/O techniques like on microcontrollers.
Most processors that include PCIe also include a Memory Management Unit, so theres an extra layer of virtual -> physical translation happening.

In Microcontrollers, the peripherals (like SPI, I2C, UART, etc) are fixed functions - a given chip will have a specific number of each and they’ll be placed in the physical memory map at defined locations. However, PCIe is far more dynamic - PCIe devices are typically in a removable “card” form factor. A given computer won’t necessarily have the same PCIe devices as another. PCIe has a discovery mechanism built in that allows the OS to probe the devices and figure out what devices are present and what kind of device they are. I won’t go into detail about this because it’s quite involved, but suffice it to say that PCIe devices are identified by something called a VENDOR ID and a DEVICE ID (sometimes also a SUBVENDOR ID and SUBDEVICE ID).

Given that PCIe devices are dynamically discovered at runtime, considerations have to be made about how to place them into the system memory map. Most devices dedicate a very large portion of their system memory map for PCIe devices. The “bridge chips” that connect a single upstream port to multiple downstream ports are responsible for dividing the memory space they’re given. lspci can be used to query PCIe topology and configuration information.

lspci

lspci can be used to query information about PCIe devices.

The first place to start is lspci -tv. This will pretty-print a “topology” diagram, as well as the names of all the devices:

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GK106GLM [Quadro K2100M]
           |            \-00.1  NVIDIA Corporation GK106 HDMI Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
           +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
           +-1c.0-[02]--
           +-1c.2-[03-07]----00.0-[04-07]--+-00.0-[05]----00.0  Qualcomm Atheros AR9462 Wireless Network Adapter
           |                               +-02.0-[06]--
           |                               \-03.0-[07]--
           +-1c.3-[08-0b]--
           +-1c.4-[0c]--
           +-1c.6-[0d-14]--
           +-1c.7-[15]----00.0  O2 Micro, Inc. SD/MMC Card Reader Controller
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1
           +-1f.0  Intel Corporation QM87 Express LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
           \-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller

You can query information about a specific device using the device address: lspci -s 05:00.0 -v

05:00.0 Network controller: Qualcomm Atheros AR9462 Wireless Network Adapter (rev 01)
        Subsystem: Dell AR9462 Wireless Network Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 18
        Memory at f5400000 (64-bit, non-prefetchable) [size=512K]
        Expansion ROM at f5480000 [disabled] [size=64K]
        Capabilities: <access denied>
        Kernel driver in use: ath9k
        Kernel modules: ath9k

In this case, the device needs 512KB of MMIO space, and the kernel has assigned it to use physical addresses 0xF540_0000 through 0xF547_FFFF. To know how to use this space, you’d have to consult the documentation for the AR9462 device, or look at the Kernel driver. Either way, the space is likely to be a mix of control registers and RAM used for packet buffers.

MMIO

In that previous Wifi card example, if the CPU accesses a physical address in the range 0xF540_0000 through 0xF547_FFFF the PCIe root port will convert that into PCIe memory transactions which the device can handle. Some addresses might be mapped directly to RAM (like the system RAM attached to the CPU, but with more steps) or they might be control/status registers that cause the device to do something or report some status information. Either way, the processor just accesses memory like it was RAM.