Programmable I/O (PIO) write operations on SGI Altix I/O subsystems can be cached in various components of the system, from the CPU to the PCI-X bridges. PIO write requests from the same CPU are guaranteed to be issued in program order. However, they are not synchronous. PIO write operations on this platform are posted. To guarantee that PIO write operations have completed, device drivers are required to push all prior PIO write operations out to the device by issuing a PIO read operation to the same controller after the last write operation before releasing a semaphore. This will prevent another CPU from acquiring the semaphore and having its PIO transactions complete before the previous holder of the semaphore.

PIO access and system memory access use different paths and hardware components on SGI Altix I/O subsystems. A get/ release operation on a memory-based lock can complete before a PIO write request.

You are strongly advised to program device drivers to flush all relevent PIO write operations with a PIO read operation to the same controller prior to releasing the relevent memory-based locks.

PIO write operation caching is a performance feature. Making each PIO write operation synchronous incurs unneccessary performance penalty. Other Linux based platforms also require the device driver to explicitly execute PIO write flushing for correct operation.

SGI Altix I/O subsystems provide support for posted direct memory access (DMA). With posted DMA capability, the host bridge can respond to the requester that the request is complete prior to actually transferring the data to target memory. This is a performance feature. DMA data is not guaranteed to arrive in memory “in-order”.

SGI Altix I/O subsystems use the interrupt mechanism to flush all posted DMA data to target memory. This is the only mechanism currently available to ensure that all posted DMAs are flushed into the target memory.

PCI-X bridge chipsets on SGI Altix systems do not automatically flush Posted DMA writes on any PIO reads. For information regarding software flushing of posted DMA write buffers, see “PIO Read Flushing Posted DMA Buffers” in Chapter 7.

On SGI Altix systems, direct memory access (DMA) data from controller cards to system memory is not guaranteed to arrive “in-order”. If a device driver is polling a memory location for completion status and the completion status is the last DMA operation by the controller card, it is not guaranteed that all the prior DMA data will arrive in memory before the DMA completion status word. If you are polling memory for completion status, you must use the equivalent mapping routines for this “Completion Status”. The equivalent mapping routine provides a DMA handle to flush all DMA data.

Using the appropriate mapping routine will ensure that all prior DMA data has arrived in memory before the “Completion Status” DMA data.

The Linux operating system provides two separate DMA mapping interfaces:

If your Altix system has devfs configured and activated, you need to ensure that your calls to these functions:

register_chrdev()
unregister_chrdev()

register_blkdev()
unregister_blkdev()

devfs_mk_dir()
devfs_register()
devfs_unregister()

All addresses on an Altix system are 64 bits long . Drivers have to ensure that any structures that are allocated to store any addresses must be 64 bits long.

---

Note: It is very important to note that if you want to translate a virtual memory address into a bus address (DMA for the card), using the following macros for translation WILL NOT work: bus_to_virt() virt_to_bus() An Example such as the following, will not work: /* This will NOT work ... */ dmabuf = kmalloc(size, GFP_KERNEL); writel(virt_to_bus(dmabuf), card's_dma_addr_reg); For more information, see “Direct Memory Access Addresses (DMA)”

---

An SGI Altix system does not have system physical memory smaller than or equal to 32 bits. To the device driver, system physical memory addresses are always 64 bits long.

The following macros will provide proper translation from physical-to-virtual or virtual-to-physical:

phys_to_virt()
virt_to_phys()

---

Note: A system physical address is not the same as a bus address. Therefore, system physical addresses cannot be used by the card for DMA, as is (see “Bus Addresses - PCI/PCI-X Buses”).

---

Bus addresses are addresses that allow the device to perform DMA operations from the card into system physical memory. An SGI Altix system supports either a 64-bit bus address or a 32-bit bus address. These bus addresses must be obtained from the various pci_map_XXX () routines. See the section on direct memory access ddresses (DMA). Legacy macros like virt_to_bus() and bus_to_virt() do not provide the correct translation.

---

Note: On an Altix system, there is no way to translate a bus address to virtual address. Moreover, there are no reasons why a driver needs to translate a bus Address to a virtual Address. Drivers are responsible to save the the corresponding virtual address to the mapped DMA address. For more information, see “Direct Memory Access Addresses (DMA)”.

---

The following legacy routines do almost no work on Intel Itanium 2 platforms:

Drivers must use the IO addresses provided in the pci_dev structure for the device.

An Example such as the following, will not work:

/* This will not work .. */
pci_read_config_dword(pci_dev, PCI_BASE_ADDRESS_0, &ioaddr);
cards_regs = ioremap(ioaddr, 0x1000);
writel(0x60002,(cards_regs + (PCI_INT_CFG/PltfMsk)));

Base address registers in the PCI Configuration Space of a card cannot be used, as is, by the device driver for PIO. Device Drivers have to use addresses initialized in the pci_dev structure allocated by the system for that device via this routine, as follows:

pci_resource_start(dev,bar)

Other resource routines of interest are, as follows:

pci_resource_end(dev,bar)
pci_resource_flags(dev,bar)
pci_resource_len(dev,bar)

On an SGI Altix system, these addresses are 64 bits long, regardless of whether they are PCI IO or memory resources. PCI IO resource addresses can then be used in the following macros:

inb/inw/inl/outb/outw/outl
insb/insw/insl/outsb/outsw/outsl

---

Note: Hardcoded legacy addresses for example, IO Port Number 0x360, used in IN/OUT macros will not work, for example, inb(0x360), and so on.

---

PCI memory resource addresses can then be used in the following macros:

readb/readw/readl/readq/writeb/writew/writel/writeq

Device drivers register their interrupt handling routines by calling the following code:

int request_irq(unsigned int irq,
         void (*handler)(int, void *, struct pt_regs *),
         unsigned long irqflags,
         const char * devname,
         void *dev_id)

Of particular interest here is irq integer, which traditionally is the interrupt line in the PCI configuration space. Device drivers should not be reading this value from the PCI configuration space to get the irq value for the request_irq(). Instead, device drivers should use the irq number as allocated in the pci_dev structure by the Linux operating system, as follows:

pci_dev->irq

An Example such as the following, will not work:

/* This will not work .. */
pci_read_config_byte(pci_dev, PCI_INTERRUPT_LINE, &irq);
request_irq(irq, ...);

DMA addresses (bus addresses) on Altix system are either 64 bits or 32 its, nothing in-between. Requests for DMA addresses between 33 and 63 bits are given 32 bits DMA addresses.

Device drivers cannot use legacy macros, such as the following:

bus_to_virt()
virt_to_bus()

Before calling any of the DMA mapping routines, a device driver should query the system for the DMA address size that the platform supports, using the following:

pci_dma_supported()

By default, the dma_mask is set by the Linux operating system to be 0xffffffff, which means 32 bits.

On an Altix system, there no calls to convert addresses from bus-to-virtual or virtual-to-bus. If the driver requires the corresponding virtual address of a bus address, it should save the virtual address.

Linux provides the following routines for mapping Virtual Address to DMA address:

pci_alloc_consistent()
pci_free_consistent()
pci_map_single()
pci_unmap_single()
pci_map_sg()
pci_unmap_sg()
pci_dma_sync_single()
pci_dma_sync_sg()

See linux/Documentation/DMA-mapping.txt for more details.

The pci_alloc_consistent() routine, by default, returns a 32 bit DMA address to the caller. On an Altix system, there is an exception. If your card is a PCIX card running in PCIX mode, only 64-bit DMA addresses are returned. For cards running in PCIX mode, please use the following: pci_set_consistent_dma_mask() to set the consistent mask bits to 0xffffffffffffff. Otherwise, your call to pci_alloc_consistent() will fail.

For performance reasons, PIO write calls are posted. That is, on return from a PIO write call for example, outb(X), an Altix system does not guarantee that the PIO has arrived and been received by the designated device. To ensure that a PIO write has actually been delivered and received by the designated device, device drivers are required to perform a PIO read to a safe register on the device, for example reading the vendor's identification, and so on:

outb(X);
outb(XX);
inb(safe register address);

---

Note: IO writes are delivered as soon as possible. In a ccNUMA architecture like used in an Altix system, if the system is very busy, a PIO write can be buffered by the IO chipsets.

---

The same rules apply to PIOs using the readb() family of macros.

For more information on synchronization issues regarding PIOs and memory references, see the Linux Device Drivers Guide.