A node's global memory mapped register (MMR) space provides all processor nodes in the system with access to a node's MMRs (see Figure 2-6). Notice the position of the global MMR space in the physical address map shown in Figure 2-8. Following are the values of the bits for global MMR space:

Figure 2-6. Bit Values for Global MMR Space

---

Note: Programmable I/O addresses reside in this space (for example, SHub systems, registers set, PCI configuration space, PCI I/O and memory space, I/O brick registers, and so on).

---

When the address space (AS) bits are set to 10, the reference is to atomic memory operation (AMO) space. An AMO read operation (AMOR) or AMO write operation (AMOW) request is issued to the SHub that is identified by the number in the node ID (see Figure 2-7). Notice the position of the AMO space in the physical address map shown in Figure 2-8. The node offset bits specify a 36-bit offset within the SHub address space, as follows:

Figure 2-7. Bit Values for AMO Space

A number of fetch-and-op style AMOs are supported to optimize common synchronization primitives such as locks, tickets, and barriers. These AMOs operate on a read-modify-write basis. AMOs are defined only for word and doubleword data sizes and are performed using uncached loads and stores to the AMO address space. In addition, operations are allowed only on the first doubleword of each 64-byte block (half cache line) in memory. The AMO variable can be accessed either as one 64-bit AMO variable or as two 32-bit AMO variables.

In the AMO address space, bits 5:3 of the node offset (the three address bits above the doubleword offset) determine the type of AMO to perform.

The following AMO read operations are supported:

The following AMO write operations are supported:

When the AS bits are set to 11, the reference is to cacheable memory space. A memory request is issued to the SHub that is identified by the number in the node ID. The node offset bits specify a 36-bit offset within the SHub address space. UC, WB, and WC attributes are supported for cacheable memory space. Notice the position of the cacheable memory space in the physical address map shown in Figure 2-8. The 50-bit physical address has a 36-bit offset within the SHub address space, as follows:

---

Note: Direct memory access (DMA) addresses reside in cacheable memory space.

---

Figure 2-8 shows the SHub physical address map. On SHub, AMO space and global MMR space must be accessed uncached, and GET space must be accessed cached. Cacheable memory space can be accessed cached or uncached, subject to operating system constraints.

Figure 2-8. SHub Physical Address Map

---

Note: Linux drivers run in virtual mode (TLBs enabled for all addresses) all the time. Therefore, the address space they see depends not only on behavior of the SHub, but also on the TLB mapping conventions of the operating system.

---

Programmable I/O (PIO) is the term for a load or store instruction executed by the CPU that names an I/O device space as its operand. The CPU places a physical address on the system bus. The bus adapter repeats the read or write command on its bus, but not necessarily using the same address bits as the CPU put on the system bus.

One task of a bus adapter is to translate between the physical addresses used on the system bus and the addressing scheme used within the proprietary bus. The address placed on the target bus is not necessarily the same as the address generated by the CPU. The translation is done differently with different bus adapters and in different system models.

With the more sophisticated PCI and PCI-X buses, the translation is dynamic. Both of these buses support bus address spaces that are as large or larger than the physical address space of the system bus. It is impossible to hard-wire a translation of the entire bus address space. Furthermore, SGI Altix architecture provides multiple system buses. For more details, see “Address Spaces Supported” in Chapter 3.

The PCI/PCI-X resource addresses in the pci_dev structure are PIO mapped addresses that the device driver can use in their existing state.

To use a dynamic PIO address, a device driver can create a software object called a PIO map that represents that portion of bus address space that contains the device registers the driver uses. When the driver wants to use the PIO map, the kernel dynamically sets up a translation from an unused part of physical address space to the needed part of the bus address space. The driver extracts an address from the PIO map and uses it as the base for accessing the device registers. This is an extension that SGI provides.

A bus-master device on the PCI bus can be programmed to perform transfers to or from memory independently and asynchronously. A bus master is programmed using PIOs with a starting bus address and a length. The bus master generates a series of memory-read or memory-write operations to successive addresses. But what bus addresses should it use in order to store into the proper memory addresses?

The bus adapter translates the addresses used on the proprietary bus to corresponding addresses on the system bus. As shown in Figure 2-9, the operation of a DMA device is as follows:

The translation of bus virtual to physical addresses is done by the bus adapter and programmed by the kernel. A device driver requests the kernel to set up a dynamic mapping from a designated memory buffer to bus addresses. For more information, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.

Linux device drivers on SGI Altix systems must use the standard Linux pci_dma map routines. For more information, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.

The driver calls kernel functions to establish the range of memory addresses that the bus master device will need to access--typically the address of an I/O buffer. When the driver calls one of the pci_dma map routines, the kernel sets up the bus adapter hardware to translate between some range of bus addresses and the desired range of memory space. The driver uses PIO to program this bus address into the bus master device registers. SGI software supports 64- and 32-bit DMA addresses. For more information on 64- and 32-bit DMA map addresses, see Chapter 9, “PCI-X Direct Memory Access (DMA)”.