In SGI Altix systems, the multimedia features have substantial local resources, so contention with multimedia for the use of main memory is lower. However, these systems also have multiple CPUs and multiple layers of address translation, and these factors can introduce latencies in PCI-X transactions.

It is important to understand that there is no guaranteed order of execution between separate PCI-X transactions in these systems . There can be multiple hardware layers between the CPU, memory, and the device. One or more data transactions can be “in flight” for durations that are significant. For example, suppose that a PCI-X bus-master device completes the last transfer of a DMA write of data to memory, and then executes a DMA write to update a status flag elsewhere in memory.

Under circumstances that are unusual but not impossible, the status in memory can be updated and acted upon by software, while the data transaction is still “in flight” and has not completely arrived in memory. The same can be true of a programmable I/O (PIO) read that polls the device. It can return “complete” status from the device while data sent by DMA has yet to reach memory.

Ordering is guaranteed when interrupts are used. An interrupt handler is not executed until all writes initiated by the interrupting device have completed.

When the Linux kernel scans the PCI-X buses on an SGI Altix system and finds an active device, it initializes the device configuration registers as follows:

Command register		The enabling bits for I/O access, memory access, and master are set to 1. Other bits, such as memory write, invalidate, and fast back-to-back are left at 0.
Cache line size		Set at 0x20 (32, 32-bit words, or 128 bytes).
Latency timer		Setting depends on bus speed and the device's Min_Gnt (minimum grant) register. If the device's Min_Gnt value is 0, the latency timer is set to 1 microsecond. Otherwise, it is set to (min_gnt_mult * Min_Gnt). Values for min_gnt_multdepend on bus speed, as follows: Bus Speed   min_gnt_mult Value 133 MHz   32 100 MHz   24 66 MHz   16 33 MHz   8
Base address registers		Each register that requests PCI memory or PCI I/O address space is programmed with a starting address. Note: This address is valid only for this PCI-X bus.

When attaching a device, the device driver can set any other configuration parameters.

---

Caution: If the driver changes the contents of a base address register, the results are unpredictable. Do not do this.

---

The following optional signal lines are not supported:

In SGI Altix systems, addresses are translated not once but at least twice and sometimes more often between the CPU and the device, or between the device and memory. Also, some of the logic for features, such as prefetching and byte-swapping, is controlled by the use of high-order address bits. There is no simple function on a physical memory address that yields a PCI-X bus address (nor vice-versa). The device driver must use the PIO addresses presented in the pci-dev structure. For more information, see Chapter 7, “PCI-X I/O and Memory Resources”.

It is also necessary for the device driver to call the relevant DMA mapping routines for DMA addresses (PCI-X bus addresses). For more information, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.

DMA Address Mapping

Any part of physical address space can be mapped into PCI-X bus address space for purposes of DMA access from a PCI-X bus-master device. The SGI Altix system architecture uses a 50-bit physical address, of which some bits designate a node board. The PCI-X adapter sets up a translation between an address in PCI-X memory space and a physical address, which can refer to a different node from the one to which the PCI-X bus is attached.

The device driver ensures correct mapping through the use of PCI DMA map routines.

If the PCI-X device supports only 32-bit addresses, DMA addresses can be established in 32-bit PCI-X space. When this is requested, extra mapping hardware is used to map a window of 32-bit space into the 50-bit memory space.

---

Caution: The number of mapping registers is limited, so it is possible that a request for DMA translation could fail.

---

Because of the possibility of the failure of a DMA translation request, it is preferable to use 64-bit DMA mapping when the device supports it. When the device supports 64-bit PCI-X bus addresses for DMA, the PCI-X adapter can use a simpler mapping method from a 64-bit address into the target 50-bit address, and there is no contention for mapping hardware. The device driver must request a 64-bit DMA map, and must program the device with 64-bit values. For further information on DMA mapping, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.

The PCI-X adapter maintains two priority groups, the real-time group and the low-priority group. Both groups are arbitrated in round-robin style. Devices in the real-time group always have priority for use of the bus. There is no kernel interface for changing the priority of a device.

Each PCI-X bus contains two unique interrupt signals. The INTA# and INTC# signals are wired together, and the INTB# and INTD# signals are wired together. A PCI-X device that uses two distinct signals must use INTA# and INTB#, or INTC# and INTD#. A device that needs more than two signals can use the additional signal lines, but such a device must also provide a register from which the device driver can learn the cause of the interrupt.

The PCI-X bus adapter chip that is used on all SGI Altix systems has eight input interrupts. PCI-X cards, however, can implement up to four different interrupts (A, B, C, and D), which might create a shared condition. Table 3-3, shows how interrupts can be shared on an SGI Altix system.

For example, if a card in slot 0 uses INTA# and a card in slot 4 uses INTB#, there will be a conflict. In this case, the interrupt service routines (ISRs) of both cards will be called when the bridge interrupt pin 0 changes to active. If you try to connect to all four interrupt lines from the card, you will create a shared condition. This is called interrupt overloading.

Because SGI Altix systems support two slots and line A and line C are wired together and line B and line D are wired together, devices on the same bus will never share the same interrupt line.