SGI Altix systems support consistent DMA mappings and streaming DMA mappings. The following sections describe each of these types.

On SGI Altix systems, the mapped DMA address as seen and issued by the PCI-X device controller can be either 64 bits (dual address cycle (DAC)) or 32 bits (single address cycle (SAC)). This platform does not support drivers that cannot handle at least 32-bit DMA addresses. Devices that can handle DMA addresses from 33 bits to 63 bits are assigned 32-bit DMA addresses. Devices that are capable of handling 64-bit addresses are always assigned 64-bit DMA addresses.

On SGI Altix systems, the mapped DMA address as seen and issued by the PCI-X device controller can be either 64 bits (dual address cycle or DAC) or 32 bits (single address cycle or SAC). This platform supports only devices that are capable of using 32-bit or greater DMA addresses. Devices that can handle DMA addresses from 33 bits to 63 bits are assigned 32-bit direct- or page-mapped DMA addresses. Devices that are capable of handling 64-bit addresses are assigned 64-bit DMA addresses for 'streaming' type mappings (see pci_map_single and pci_map_sg ), and 32-bit DMA addresses for so-called 'consistent' mappings (see pci_alloc_consistent).

Direct-mapped addressing is per-bus. That is, when a device is on a bus by itself, the PCI-X bridge chip can be programmed to map 32-bit DMAs by the device into a particular 2-GB window of system memory. Page-mapped addressing operates on a per-address basis, which allows the driver to program the device to DMA into multiple 2-GB windows of system memory. These modes of addressing will be transparent to the driver, however, and are only described here for informational purposes. The following sections describe direct mapped DMA addresses and page mapped DMA addresses, respectively.

Format of 32-bit Direct Mapped DMA Addresses

Figure 9-3, shows the format of a PCI direct mapped register. Figure 9-4, shows the format of a direct mapped DMA address.

Figure 9-3. PCI Direct Mapped Register (one per PCI bridge)

Figure 9-4. 32-bit Direct Mapped Address As Returned by the System

A DMA address is a 32-bit direct mapped DMA address when bit 31 of the 32-bit DMA address is set. When a DMA address is direct mapped, the actual system physical address (PCI bus address) is formulated from both the direct map register and the 32-bit DMA mapped address.

When a direct map register is a 17-bit constant value and when a 32-bit mapped DMA address is a 31-bit variable value, a 32-bit direct mapped mechanism can map any 2-GB range of system physical memory space. Figure 9-5, shows the bits that comprise a 50-bit system address.

Figure 9-5. 50-bit System Memory Address

In Figure 9-5, the physical memory space is mapped as follows:

• Bits 30 - 0 from the 32-bit mapped DMA address become bits 30 - 0 of the system physical address.

• Bits 16 - 0 from the direct mapped register becomes bits 47 - 31 of the system physical address.

---

Note: There is only one direct mapped register per PCI-X bus.

---

Format of 32-bit DMA Page Mapped Addresses

The type is returned by pci_alloc_consistent . The type is also returned by pci_map_* when pci_dev.dma_mask is less than 64 bits. Figure 9-6, shows the format of a 32-bit DMA page mapped address.

Figure 9-6. 32-bit DMA Mapped Address

If bit 31 is set, it is a 32-bit direct mapped address. If bit 30 is set, it is a 32-bit page mapped address.

If the system page size is 4 KB, bits 11 to 0 are the page offset and the address translation entry (ATE) index is in bits 29 to 12. If the system page size is 16 KB, bits 13 to 0 are the page offset and the ATE index is in bits 29 to 14.

DMA mapped addresses 0x4000_0000 to 0x700_0000 are paged mapped addresses. This means that the address offset and the DMA attributes come from the targeted ATE register and direct map register.

DMA mapped addresses 0x8000_0000 to 0xffff_ffff are direct mapped adresses. This means that the address offset and the DMA attributes come from the device registers and the direct map register.

Format of a 64-bit DMA Mapped Adddress

The type is returned by pci_map_* when pci_dev.dma_mask is 64 bits (see pci_set_dma_mask). Figure 9-7, shows the format of a 64-bit DMA mapped address.

Figure 9-7. 64-bit DMA Mapped Address

In 64-bit DMA mapped addresses, bits 63 to 54 are set. These are DMA attributes. Bits 50 to 0 target the actual system memory location.

---

Caution: Devices asking for 64-bit DMA addresses without the capacity to generate dual address cycles will cause unknown hangs and data corruption.

---

Device drivers should follow these rules when using DMA resources on SGI Altix systems:

Example:

/* This driver can support up to 64-bit DMA Address. */
if (!pci_set_dma_mask(pdev, (u64) 0xffffffffffffffff)) {
                 // Yes, system can support 64-bit DMA addresses.
                 .. code ..
} else {
                 // Okay, can system support 32-bit DMA addresses?
                 err = pci_set_dma_mask(pdev, (u64) 0xffffffff);
                 if (err) {
                         // Need to abort as this driver cannot support
                         // DMA addresses smaller than 32 bits!
                         .. code ..
                 }
                 // Alright, we are running with 32-bit DMA addresses.
                 .. code ..
       }          

On SGI Altix systems, the platform can support either 64-bit or 32-bit DMA addresses. Cards that can support 32-bit to 63-bit DMA addresses are given 32-bit DMA addresses. Cards that can support 64-bit DMA addresses are always given 64-bit DMA addresses. Cards that can support only DMA addresses that have less than 32 bits are not supported on this platform. 32-bit DMA addresses can be used in dual address cycles as long as bits 63 to 32 are zero.

On SGI Altix systems, DMA memories are required to be mapped or unmapped via the following routines:

For consistent DMA mappings:

void * pci_alloc_consistent(struct pci_dev *hwdev, size_t size, dma_addr_t *dma_handle)
void pci_free_consistent(struct pci_dev *hwdev, size_t size, void *vaddr, dma_addr_t dma_handle)

For streaming DMA mappings:

dma_addr_t pci_map_single(struct pci_dev *hwdev, void *ptr,size_t size, int direction)

void pci_unmap_single(struct pci_dev *hwdev, dma_addr_t dma_addr, size_t size, int direction)

int pci_map_sg(struct pci_dev *hwdev, struct scatterlist *sg,int nents, int direction)
void pci_unmap_sg(struct pci_dev *hwdev, struct scatterlist *sg,int nents, int direction)

On SGI Altix platforms, all DMA operations prior to the controller card's issue of an interrupt on the PCI-X bus are guaranteed to be "completed and coherent" before the interrupt is forwarded to the CPU. In other words, the interrupt "pushes" any and all DMA cache data out into the system.

On SGI Altix systems, memory coherency between processor caches and memory buses is guaranteed, and use of the following routines is unnecessary:

void pci_dma_sync_single(struct pci_dev *hwdev,dma_addr_t dma_handle,size_t size, int direction)

void pci_dma_sync_sg(struct pci_dev *hwdev, struct scatterlist *sg, int nents, int direction)

For more details, see the documentation in the Linux source tree at the following location:

linux/documentation/DMA-mapping.txt