In-class: PCI Enumeration

This assignment explores the 32-bit PCI bus, creating a utility in xv6 to list PCI devices and their parameters.

Submit your solutions before the beginning of the next lecture to the CS7680 submission web site.

Please collaborate with others on these exercises.

PCI bus overview

The PCI bus (see provides a standard architecture for accessing I/O devices on most modern computers, supporting discovery and configuration of attached devices as well as both memory-mapped and IO-mapped (e.g. via inb, outb instructions) access. In particular PCI provides a mechanism where each card or device specifies both its identity and the resources needed (memory-mapped or I/O space, interrupts) and the BIOS and/or OS can discover all devices in the system and allocate interrupts, physical memory space, and I/O space to each device.

PCI has three levels of structure - bus, device, and function. Buses are structured as a tree, with bus 0 as the root and additional buses connected by PCI bridges. An individual PCI card (or chip on the motherboard) that attaches to a PCI bus is a device; note that PCI bridges are themselves devices. Simple devices implement a single function (function 0); more complex devices (e.g. a multi-port ethernet card) may contain multiple functions, which operate (for the most part) like independent PCI devices.

The PCI bus itself is implemented by the motherboard chipset (actually now it's typically integrated onto the CPU itself), which handles tasks such as bus arbitration - ensuring that only one device can transmit on the bus at any time. In parallel PCI the data lines are shared between all devices on a bus, and arbitration is handled by separate bus request lines from each slot to the PCI controller, and corresponding bus grant lines from the controller back to each slot.

The PCI controller is also responsible for providing software access to the configuration space for each device, a 256-byte set of registers which identifies the device and allows configuration of device properties. This is performed via a separate signal to each card or device, so the CPU can detect and configure all devices on the bus without any prior knowledge of e.g. what address they are mapped at.

lspci and examples

The lspci command in Linux lists the devices connected to the PCI bus; here is an example from one of the machines in the first-floor computer lab:

pjd@prosoccer:~$ lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation Q67 Express Chipset Family LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 04)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
03:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
The configuration space stores 16-bit vendor and device IDs, rather than these descriptive names; lspci performs a table lookup to report these names. The actual numeric values on the same machine are as follows. (note that Intel is vendor id 0x8086)
pjd@prosoccer:~$ lspci -n
00:00.0 0600: 8086:0100 (rev 09)
00:01.0 0604: 8086:0101 (rev 09)
00:16.0 0780: 8086:1c3a (rev 04)
00:16.3 0700: 8086:1c3d (rev 04)
00:19.0 0200: 8086:1502 (rev 04)
00:1a.0 0c03: 8086:1c2d (rev 04)
00:1b.0 0403: 8086:1c20 (rev 04)
00:1c.0 0604: 8086:1c10 (rev b4)
00:1c.2 0604: 8086:1c14 (rev b4)
00:1d.0 0c03: 8086:1c26 (rev 04)
00:1e.0 0604: 8086:244e (rev a4)
00:1f.0 0601: 8086:1c4e (rev 04)
00:1f.2 0106: 8086:1c02 (rev 04)
00:1f.3 0c05: 8086:1c22 (rev 04)
01:00.0 0300: 1002:68f9
03:00.0 0c03: 1033:0194 (rev 03)
[note - tables and some descriptions adapted from]

Configuration space access

BIOS and OS code must be able to access these configuration registers in order to discover and configure PCI devices. On the PC platform this is done via two 32-bit IO ports, CONFIG_ADDRESS (0xCF8) and CONFIG_DATA (0xCFC). An address specifying bus, device, function, and a 4-byte-aligned byte offset is written to CONFIG_ADDRESS:
31 30 - 24 23 - 16 15 - 11 10 - 8 7 - 0
Enable Bit Reserved Bus Number Device Number Function Number Offset
and then a 32-bit value is read from or written to CONFIG_DATA. If no device is present, the result will be 0xFFFFFFFF. (note that 0xFFFF is an invalid vendor code, so the absence of a device can be detected by reading the first 32-bit word of the corresponding configuration space)

Reading configuration space on XV6

For this assignment you will add a user-space utility to XV6 to enumerate the PCI configuration space.

IO port access

First you'll need access to I/O instructions within a user process. The Intel architecture provides two mechanisms which allow access to in and out instructions while in user space:

The TSS bitmap hasn't been implemented in XV6, and we might not want to extend it far enough to cover ports 0xCF8 and 0xCFC anyway. (that would take over 400 bytes per process) Instead you'll implement a new system call, iopl, which takes a single argument in the range 0-3 and sets the I/O privilege level bits in EFLAGS to that value; your readpci utility can then call iopl(3) and will be able to access I/O ports directly.

Hint - where does EFLAGS come from when you return to user space? Do you actually have to modify the current EFLAGS register in the iopl system call?

Create a user program named readpci (add it to UPROGS in your makefile) and start out with the following code to test iopl:

#include "types.h"
#include "user.h"
#include "x86.h"

int main(int argc, char **argv)
    iopl(3);  // allow ring 3 to access IO ports
    unsigned char c = inb(0x3bc);
    printf(1, "%x\n", c);

If iopl was implemented correctly it should run, printing FF, while it will die with a fault otherwise.

PCI register access

To access CONFIG_ADDR and CONFIG_DATA you'll need 32-bit versions of the in/out instructions, which aren't provided in XV6:

unsigned int inl(int port)
        unsigned int data;
        __asm __volatile("inl %w1,%0" : "=a" (data) : "d" (port));
        return data;
void outl(int port, unsigned int data)
  __asm __volatile("outl %0,%w1" : : "a" (data), "d" (port));

Modify readpci to take 3 numeric arguments - bus, device, and function - and read and print the corresponding configuration space in hex. If you want the printout to look nice like this you'll have to modify printf.c to support formats like "%02x" :-)

cpu0: starting
init: starting sh
$ readpci 0 1 0
00: 86 80 00 70 03 00 00 02 00 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 F4 1A 00 11
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Configuration space structure

All PCI configuration spaces share a common header:
register bits 31-24 bits 23-16 bits 15-8 bits 7-0
00 Device ID Vendor ID
04 Status Command
08 Class code Subclass Prog IF Revision ID
0C BIST Header type Latency Timer Cache Line Size
The fields are as follows, ignoring ones that we don't need:

Note that all 16 and 32-bit values are in little-endian order. (not surprisingly, as it was originally developed by Intel) This means that you can access them directly on a PC without needing to translate byte order.

For a general-purpose device the remainder of the configuration space is structured as follows:

register bits 31-24 bits 23-16 bits 15-8 bits 7-0
10 Base address #0 (BAR0)
14 Base address #1 (BAR1)
18 Base address #2 (BAR2)
1C Base address #3 (BAR3)
20 Base address #4 (BAR4)
24 Base address #5 (BAR5)
28 Cardbus CIS Pointer
2C Subsystem ID Subsystem Vendor ID
30 Expansion ROM base address
34 Reserved Capabilities Pointer
38 Reserved
3C Max latency Min Grant Interrupt PIN Interrupt Line
The fields (other than base addresses) are:

We'll ignore all of these values for now, and talk about them during class.

PCI bridges

In a PCI bridge this portion of the configuration data has the following structure:

register bits 31-24 bits 23-16 bits 15-8 bits 7-0
10 Base address #0 (BAR0)
14 Base address #1 (BAR1)
18 Secondary Latency Timer Subordinate Bus Number Secondary Bus Number Primary Bus Number
1C Secondary Status I/O Limit I/O Base
20 Memory Limit Memory Base
24 Prefetchable Memory Limit Prefetchable Memory Base
28 Prefetchable Base Upper 32 Bits
2C Prefetchable Limit Upper 32 Bits
30 I/O Limit Upper 16 Bits I/O Base Upper 16 Bits
34 Reserved Capability Pointer
38 Expansion ROM base address
3C Bridge Control Interrupt PIN Interrupt Line

The main field of interest is the Secondary Bus Number, which identifies the "child" bus in the tree. For example, since the root PCI bus is always 0, if there is a bus 1 there must be a PCI bridge on bus 0 with secondary bus number 1.

You can recursively enumerate devices on the PCI bus by scanning bus 0, and whenever you detect a PCI bridge recursively scanning its secondary bus. In doing this keep in mind:

You may find the file pci-cfg.h useful, as it contains C structure definitions for PCI device and bridge configurations.

Modify readpci to recursively enumerate PCI buses. (e.g. if invoked as readpci -scan instead of e.g. readpci 0 1 0)

To test recursive enumeration you will need a newer version of QEMU - you can download the latest version for Ubuntu 12.04.3 (i.e. the virtual machine image most of you are using) as follows:

mkdir -p ~/bin
cd ~/bin
chmod +x qemu-system-i386

In your xv6 directory you should then be able to run QEMU as follows:

~/bin/qemu-system-i386 -nographic -hdb fs.img xv6.img -smp 1 -m 512 -bios /usr/share/qemu/bios.bin \
    -device i82801b11-bridge-device rtl8139,bus=pci.1 -device i82801b11-bridge -device ne2k_pci,bus=pci.2

This is the equivalent of running make qemu-nox - if you want the graphical window, eliminate the -nographic argument. It should print several warning messages and then start up, and if you enumerate the PCI bus correctly you'll see devices on buses 1 and 2.

Submit: Your working readpci.c file.


With the following code modification you can compile readpci.c for Linux with the command gcc -DHOST_VERSION readpci.c -o readpci; you will also need to change all your printf calls from printf(1, "..."); to cprintf("...");. You should then be able to run your readpci command on any Linux system where you have root access. (e.g. your class virtual machine image)

typedef unsigned char uchar;
#define cprintf printf
#include "types.h"
#include "user.h"
#include "x86.h"
#define cprintf(...) printf(1, __VA_ARGS__)
Last modified: Mon Mar 17 23:43:06 EDT 2014