CXFS supports a client-only node running the Solaris operating system. This chapter contains the following sections:
This section contains the following information about CXFS on Solaris:
In addition to the items listed in “Requirements” in Chapter 1, using a Solaris node to support CXFS requires the following:
Solaris 10 May 08 (patch 120011-14)
Caution: All other releases of Solaris 10 are not supported and may cause system instabilities when used with CXFS.
The following supported Fibre Channel HBAs (you can use only one vendor for HBA, either LSI Logic or QLogic; you cannot mix HBA vendors):
LSI Logic models using the 01030600 firmware or newer:
LSI7102XP LSI7202XP LSI7402XP LSI7104XP LSI7204XP LSI7404XP QLogic models sold by Sun Microsystems and running with the driver supplied by Sun. (If you have a Qlogic HBA, your system will only access disks with GPT labels. For more information about GPT labels and CXFS, see the CXFS 5 Administration Guide for SGI InfiniteStorage.)
SG-XPCI1FC-QL2 (single-port 2 Gb) SG-XPCI2FC-QF2-Z (dual-port 2 Gb)
Note: CXFS does not automatically detect WWPNs for LSI HBAs or QLogic HBAs. See “I/O Fencing for Solaris”. Any system based on UltraSPARC III, IIIi, or IV with a spare 66-MHz (or faster) PCI slot for a Fibre Channel HBA and a spare 100-Mb/s (or faster) ethernet port for the CXFS private network. CXFS supports a Solaris node only on the SPARC platform. It is not supported on other hardware platforms.
For additional latest information, see the CXFS Solaris release notes.
The following commands are shipped as part of the CXFS Solaris package:
| /usr/cxfs_cluster/bin/cxfs_client |
| /usr/cxfs_cluster/bin/cxfs_info |
| /usr/cxfs_cluster/bin/cxfsdump |
| /usr/cxfs_cluster/bin/frametest |
| /usr/sbin/grioadmin |
| /usr/sbin/griomon |
| /usr/sbin/grioqos |
| /usr/cxfs_cluster/bin/xvm |
The cxfs_client and xvm commands are needed to include a client-only node in a CXFS cluster. The cxfs_info command reports the current status of this node in the CXFS cluster.
The pkgadd output lists all software added; see “Solaris Installation Procedure”.
For more information, see the man pages. For additional information about the GRIO commands, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and “GRIO on Solaris”.
The cxfs_client command creates a /var/log/cxfs_client log file. To rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file; see the cxfs_client man page for details.
For information about the log files created on server-capable administration nodes, see the CXFS 5 Administration Guide for SGI InfiniteStorage.
Solaris supports the CXFS mount scripts. See “CXFS Mount Scripts” in Chapter 1 and the CXFS 5 Administration Guide for SGI InfiniteStorage.
Although it is possible to mount a UFS or NFS filesystem on top of a Solaris CXFS filesystem, this is not recommended.
After a crash, attempts to reclaim locks and commit asynchronous writes to a CXFS filesystem from an NFS client may result in a stale file handle.
For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. See “maxphys System Tunable for Solaris”.
All disk devices attached to LSI Logic HBAs must be for use only by CXFS disks; do not attach non-disk devices to any Fibre Channel HBA that is configured for CXFS use. This restriction is required because all disk devices on these HBAs (configured for CXFS) make use of the whole disk volume, which must be conveyed to Solaris via modification in the HBA driver to the value returned by the READ_CAPACITY SCSI command.
CXFS does not automatically detect WWPNs for LSI HBAs. See “I/O Fencing for Solaris” for instructions to set up a fencing configuration.
The xvm command displays duplicate entries of physvols. The number of duplicate entries correspond to the devices for each LUN.
See also Appendix B, “Filesystem and Logical Unit Specifications”.
All CXFS files have UNIX mode bits (read, write, and execute) and optionally an access control list (ACL). For more information about POSIX ACLs, see the chmod and setfacl man pages.
If you restore a CXFS file that had an ACL containing only owner-ACL entries (that is, owner/group/other/mask) from a Solaris node, upon restoration one of the following will happen:
When using tar(1), cpio(1), and Legato Networker: The ACL will be lost because these tools behave "intelligently" by not calling acl to set an ACL if the file has only owner/group/other/mask entries. These tools will only set the file mode. However, this does not present a change in functionality because an access permissions check on the mode and the ACL containing only owner entries will give the same result.
When using other backup/restore utilities: A mask will be added to the ACL if the application calls aclfor every file.
A backup/restore utility that calls acl to set an ACL for every file will result in a file being restored with four ACL entries (that is, owner/group/other/mask), even though it may have originally had only three (that is, owner/group/other). This is due to a requirement in getfacl that it receive four ACL entries for the GETACL command to acl. (If fewer than four entries are returned, getfacl will report an error).
| Note: Normally, Solaris filesystem ACLs can have up to 1024 entries for a file and a directory can have 1024 entries as well as an additional 1024 entries for the default ACL. However, CXFS filesystems on Solaris nodes in a multiOS cluster must maintain compatibility with the metadata server. The CXFS filesystems on a Solaris node are limited to a maximum of 25 ACL entries for a file and a maximum total of 50 for a directory (that is, the directory ACL plus the default ACL). |
When using the ls command to display access permissions for a file with an ACL, the mode reported for a CXFS file follows Linux semantics instead of Solaris/UFS semantics.
On Solaris, a UFS file mode reports the group permission as the intersection of the GROUP and MASK entries in the ACL. If the GROUP entry is r-x and the MASK entry is rw-, the group permission will be reported as r--.
The CXFS model calls for reporting the ACL MASK for the group permission in the mode. Therefore, using the example above, the group permission will be reported as rw-. Although it appears that the group has write permission, it does not and an attempt to write to the file will be rejected. You can obtain the real (that is, effective) group permission by using the Solaris getfacl command.
For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. Do the following:
Make a backup copy of the /etc/system file.
Note: Exercise extreme caution in changing /etc/system and always make a backup copy. Change the value of maxphys to 0x800000 (hexadecimal) by adding the following to /etc/system :
set maxphys=0x800000
Reboot the Solaris node. This causes the change to take effect.
Verify that the new value for maxphys is in effect by running the following command:
solaris# echo "maxphys/X" | adb -k physmem 1f03f maxphys: maxphys: 800000
The QLogic driver is provided with Solaris 10.
This section discusses the following:
These procedures may be performed by you or by a qualified Sun service representative. You must be logged in as root to perform the steps listed in this section.
To install the LSI Logic HBA, perform the following steps. Additional details are provided in the Fibre Channel to PCI-X Host Adapters User's Guide.
Install the LSI Logic HBA into the Solaris system. See the chapter "Installing the Host Adapter" from the Fibre Channel to PCI-X Host Adapters User's Guide.
Bring the system back up.
Install the LSI Logic HBA driver software ( ITImpt, version 5.07.00 or later) according to the instructions in the driver's readme file.
Do the following:
Retrieve the driver package from the following LSI Logic website:
Install the driver package:
solaris# unzip itmpt-5.07.00.zip solaris# uncompress itmpt_install.tar.Z solaris# tar -xvf itmpt_install.tar solaris# cd install solaris# pkgadd -d .
Install the lsi utilities package:
solaris# uncompress lsiutils_v60.tar.Z solaris# tar -xvf lsiutils_v60.tar solaris# cd install solaris# pkgadd -d .
For each target/LUN pair to be used by the LSI Logic HBA, use the lsiprobe utility to add entries to /kernel/drv/ssd.conf.
For example, to add entries for targets 0 through 5 (inclusive), with each of those targets scanning LUNs 0, 2, 4, 5, and 6:
solaris# lsiprobe -a target 0-5 lun 0,2,4-6
Note: If you modify /kernel/drv/ssd.conf, you must reboot the system (as in step 5) in order for changes to take effect. solaris# init 6
After the system has rebooted, verify that the driver attached correctly to the HBA by following the steps “Verifying the HBA Installation”. Do not proceed until the verification succeeds.
After the system reboots, you should verify that the devices were correctly configured by running the Solaris format command. You should see a list of each device you selected.
solaris# format
Searching for disks...done
c2t200400A0B80C268Cd1: configured with capacity of 67.75GB
c2t200400A0B80C268Cd3: configured with capacity of 136.64GB
AVAILABLE DISK SELECTIONS:
0. c0t0d0 /pci@1c,600000/scsi@2/sd@0,0
1. c0t1d0 /pci@1c,600000/scsi@2/sd@1,0
2. c2t200400A0B80C268Cd1 /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1
3. c2t200400A0B80C268Cd3 /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,3
Specify disk (enter its number): |
In this example, disks 2 and 3 are being addressed by the QLogic driver, as indicated by the presence of SUNW,qlc@1 in the pathname.
You can also use the luxadm command to view the status of the HBA:
solaris# luxadm -e port
/devices/pci@1d,700000/SUNW,qlc@1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl NOT CONNECTED
solaris# luxadm probe
No Network Array enclosures found in /dev/es
Found Fibre Channel device(s):
Node WWN:200400a0b80c268b Device Type:Disk device
Logical Path:/dev/rdsk/c2t200400A0B80C268Cd1s2
Node WWN:200400a0b80c268b Device Type:Disk device
Logical Path:/dev/rdsk/c2t200400A0B80C268Cd3s2 |
The system log and console display may display warning messages similar to the following:
WARNING: /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,3 (ssd0):
Corrupt label; wrong magic number
WARNING: /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1 (ssd1):
Corrupt label; wrong magic number |
For QLogic HBA, these messages means that the disk has a bad label or a DVH label, which is not supported. (QLogic HBAs support only GPT labels.)
Similar messages for an LSI Logic HBA will appear on boot for LUNs that have DVH labels. When the XVM module is loaded or started, it installs hooks into the HBA driver and automatically translates the DVH labels into SUN labels (you should not try to relabel the disks with the format command); after XVM translates the labels, you will not see these error messages.
| Note: You can also use the lsiutil command to determine the number of LSI HBAs installed, the model numbers, and firmware versions. |
If you are having trouble with the verification steps, see “New Storage is Not Recognized on Solaris”.
Adding a new RAID can change the fabric name binding. To set up persistent binding, edit the /kernel/drv/itmpt.conf file and add entries of the following format, one for each target, where portWWN is the port world wide name of the RAID controller:
target-X-wwn="portWWN" |
For example:
target-0-wwn="200800a0b8184c8e" target-1-wwn="200900a0b8184c8d" target-2-wwn="200400a0b80c268c" target-3-wwn="200500a0b80c268c" |
In this example, target-0 will be bound to the device with the port WWN 200800a0b8184c8e and the resulting devices will be:
/pci@1d,700000/IntraServer,fc@1/ssd@0,* |
This section provides an overview of the steps that you or a qualified Sun service representative will perform on your Solaris nodes prior to installing the CXFS software. It contains the following sections:
The following procedure provides an overview of the steps required to add a private network to the Solaris system. A private network is required for use with CXFS. See “Use a Private Network” in Chapter 2.
You may skip some steps, depending upon the starting conditions at your site. For details about any of these steps, see the Solaris documentation.
If your system is already operational and on the network, skip to step 2.
If your Solaris system has never been set up, bring the system to single-user mode. For example, go to the PROM prompt and boot the Solaris node into single-user mode:
> boot -s
As a last resort, you can reach the PROM prompt by pressing the L1-A (or Stop-A) key sequence.
Edit the /etc/inet/ipnodes file so that it contains entries for each node in the cluster and its private interfaces.
The /etc/inet/ipnodes file has the following format, where primary_hostname can be the simple hostname or the fully qualified domain name:
IP_address primary_hostname aliases
You should be consistent when using fully qualified domain names in the /etc/inet/ipnodes file. If you use fully qualified domain names on a particular node, then all of the nodes in the cluster should use the fully qualified name of that node when defining the IP/hostname information for that node in their /etc/inet/ipnodes file.
The decision to use fully qualified domain names is usually a matter of how the clients (such as NFS) are going to resolve names for their client server programs, how their default resolution is done, and so on.
Even if you are using the domain name service (DNS) or the network information service (NIS), you must add every IP address and hostname for the nodes to /etc/inet/ipnodes on all nodes. For example:
190.0.2.1 server1.company.com server1 190.0.2.3 stocks 190.0.3.1 priv-server1 190.0.2.2 server2.company.com server2 190.0.2.4 bonds 190.0.3.2 priv-server2
You should then add all of these IP addresses to /etc/inet/ipnodes on the other nodes in the cluster.
For more information, see the hosts, named, and nis man pages.
Note: Exclusive use of NIS or DNS for IP address lookup for the nodes will reduce availability in situations where the NIS or DNS service becomes unreliable. For more information, see “Understand Hostname Resolution and Network Configuration Rules” in Chapter 2.
Edit the /etc/nsswitch.conf file so that local files are accessed before either NIS or DNS. That is, the ipnodes line in /etc/nsswitch.conf must list files first.
ipnodes: files nis dns
(The order of nis and dns is not significant to CXFS, but files must be first.)
Determine the name of the private interface by using the ifconfig command as follows:
solaris# ifconfig -a
If the second network does not appear, it may be that a network interface card must be installed in order to provide a second network, or it may be that the network is not yet initialized.
For example, on an Ultra Enterprise 250, the integrated Ethernet is hme0; this is the public network. The following ifconfig output shows that only the public interface exists:
solaris# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5
If the second network does not appear, do the following:
If you do not have the PCI card installed, install it. Refer to your PCI documentation for instructions.
If your card is already installed, skip to step b.
Use the output from the dmesg command to determine the interface name for the private network; look for the network interface that immediately follows the public network; you may wish to search for Found. For example:
solaris# dmesg Feb 6 09:38:36 ue250 last message repeated 42 times Feb 6 11:38:40 ue250 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0 Feb 6 11:38:40 ue250 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0 Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme0 : PCI IO 2.0 (Rev Id = c1) Found Feb 6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme0 is /pci@1f,4000/network@1,1 Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : PCI IO 2.0 (Rev Id = c1) Found Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : Local Ethernet address = 8:0:20:cc:43:48 Feb 6 11:38:41 ue250 pcipsy: [ID 370704 kern.info] PCI-device: SUNW,hme@1,1, hme1 Feb 6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme1 is /pci@1f,2000/SUNW,hme@1,1
The second network is hme1; this is the private network, and is displayed after hme0 in the dmesg output. In this example, hme1 is the value needed in step c and in step 5 below.
Initialize the private network's interface by using the ifconfig command as follows, where interface is the value determined in step b:
ifconfig interface plumb
For example:
solaris# ifconfig hme1 plumb
After performing the plumb, the hme1 interface will appear in the ifconfig output, although it will not contain the appropriate information (the correct information will be discovered after the system is rebooted later in step 8). For example, at this stage you would see the following:
solaris# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5 hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 0.0.0.0 netmask ff000000 broadcast 255.0.0.0 ether 8:0:20:d2:29:c5
Create a file named /etc/hostname.interface, where interface is the value determined in step 4. This file must contain the name of the private network. For example:
solaris# cat /etc/hostname.hme1 cxfssun3-priv
Note: In this scenario, /etc/hostname.hme0 must contain the same value as the /etc/nodename file. For example: solaris# cat /etc/hostname.hme0 cxfssun3 solaris# cat /etc/nodename cxfssun3
Edit the /etc/netmasks file to include the appropriate entries.
(Optional) Edit the /.rhosts file if you want to use remote access or if you want to use the connectivity diagnostics provided with CXFS. Ensure that the mode of the .rhosts file is set to 600 (read and write access for the owner only).
Make sure that the /.rhosts file on each Solaris node allows all of the nodes in the cluster to have access to each other. The connectivity tests execute a ping command from the local node to all nodes and from all nodes to the local node. To execute ping on a remote node, CXFS uses rsh as user root.
For example, suppose you have a cluster with three nodes: admin0, solaris1, and solaris2. The /.rhosts files could be as follows (the prompt denotes the node name):
admin0# cat /.rhosts solaris1 root solaris1-priv root solaris2 root solaris2-priv root solaris1# cat /.rhosts admin0 root admin0-priv root solaris2 root solaris2-priv root solaris2# cat /.rhosts admin0 root admin0-priv root solaris1 root solaris1-priv root
solaris# init 6
At this point, ifconfig will show the correct information for the private network.
For example:
ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5 hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 10.1.1.36 netmask ffffff00 broadcast 10.1.1.255 ether 8:0:20:d2:29:c5
For each private network on each Solaris node in the pool, verify access with the Solaris ping command. Enter the following, where nodeIPaddress is the IP address of the node:
solaris# /usr/sbin/ping -s -c 3 nodeIPaddress |
solaris# /usr/sbin/ping -s -c 3 128.162.2.91 PING 128.162.2.91: 56 data bytes 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=0. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=1. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=2. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=3. time=0. ms |
Also execute a ping on the public networks. If ping fails, follow these steps:
Verify that the network interface was configured up using ifconfig ; for example:
solaris# /usr/sbin/ifconfig eri0 eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.127 netmask ffffff00 broadcast 128.162.2.255 ether 0:3:ba:d:ad:77
In the first output line above, UP indicates that the interface was configured up.
Verify that the cables are correctly seated.
Repeat this procedure on each node.
The CXFS software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures:
Installing the CXFS client software for Solaris requires approximately 20 MB of space.
To install the required software on a Solaris node, SGI personnel will do the following:
Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS Solaris release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio
Verify that the node has been upgraded to Solaris 10 (also known as SunOS 5.10) according to the Solaris installation guide. Use the following command to display the currently installed system:
solaris# uname -r
This command should return a value of 5.10.
Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the package on the server will be as follows:
/usr/cluster/client-dist/CXFS_VERSION/solaris/10/noarch/SGIcxfs-sol10-vCXFS_VERSION.pkg
Install the package:
solaris# pkgadd -d SGIcxfs*.pkg
You must select the package to be installed and then confirm that you want allow the installation of the packages with superuser permission. For example:
solaris# pkgadd -d SGIcxfs*.pkg The following packages are available: 1 SGIcxfs SGI CXFS client software (sparc) 5.0.0.3 Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]: 1 Processing package instance <SGIcxfs> from </tmp/SGIcxfs-sol10-v5.0.0.3.pkg> SGI CXFS client software(sparc) 5.0.0.3 /* * Copyright (c) 2008 Silicon Graphics, Inc. ALL RIGHTS RESERVED ... */ Using </> as the package base directory. ## Processing package information. ## Processing system information. 3 package pathnames are already properly installed. ## Verifying disk space requirements. ## Checking for conflicts with packages already installed. ## Checking for setuid/setgid programs. This package contains scripts which will be executed with super-user permission during the process of installing this package. Do you want to continue with the installation of <SGIcxfs> [y,n,?] y Installing SGI CXFS client software as <SGIcxfs> ## Installing part 1 of 1. /etc/init.d/cxfs_cluster /etc/rc0.d/K28cxfs_cluster <symbolic link> /etc/rc1.d/K28cxfs_cluster <symbolic link> /etc/rc2.d/S77cxfs_cluster <symbolic link> /etc/rc3.d/S77cxfs_cluster <symbolic link> /etc/rcS.d/K28cxfs_cluster <symbolic link> /usr/cxfs_cluster/bin/cxfs_client /usr/cxfs_cluster/bin/cxfs_info /usr/cxfs_cluster/bin/cxfsdump /usr/cxfs_cluster/bin/frametest /usr/cxfs_cluster/bin/xvm /usr/kernel/drv/cell.conf /usr/kernel/drv/sparcv9/cell /usr/kernel/drv/sparcv9/xvm /usr/kernel/drv/xvm.conf /usr/kernel/fs/sparcv9/cxfs /usr/kernel/misc/sparcv9/hba_hook /usr/lib/sparcv9/libgrio.so /usr/sbin/clmount /usr/sbin/grioadmin /usr/sbin/griomon /usr/sbin/grioqos /usr/sbin/idbg /usr/share/man/man1/xvm.1 /usr/share/man/man1m/cxfs_client.1m /usr/share/man/man1m/cxfs_info.1m /usr/share/man/man1m/frametest.1m /var/cluster/cxfs_client-scripts/cxfs-post-mount /var/cluster/cxfs_client-scripts/cxfs-post-umount /var/cluster/cxfs_client-scripts/cxfs-pre-mount /var/cluster/cxfs_client-scripts/cxfs-pre-umount /var/cluster/cxfs_client-scripts/cxfs-reprobe [ verifying class <none> ] ## Executing postinstall script. Starting CXFS services... cxfs_client daemon started
To verify that the CXFS software has been installed properly, use the pkginfo command as follows:
pkginfo -l SGIcxfs |
For example, the following output indicates that the CXFS package installed properly:
solaris# pkginfo -l SGIcxfs
PKGINST: SGIcxfs
NAME: SGI CXFS client software
CATEGORY: system
ARCH: sparc
VERSION: 5.0.0.3
BASEDIR: /
VENDOR: Silicon Graphics Inc.
PSTAMP: cxfssun120080213222059
INSTDATE: Feb 14 2008 07:15
STATUS: completely installed
FILES: 38 installed pathnames
5 directories
22 executables
20179 blocks used (approx) |
I/O fencing is required on Solaris nodes in order to protect data integrity of the filesystems in the cluster.
The cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) for Solaris nodes that are connected to a switch that is configured in the cluster database. These HBAs are available for fencing.
However, if no WWPNs are detected, the following message will be logged to the /var/log/cxfs_client file:
cis_get_hbas no local HBAs found - falling back to /etc/fencing.conf |
If no WWPNs are detected, you can manually specify the WWPNs in the fencing file.
| Note: This method does not work if the WWPNs are partially discovered. |
The /etc/fencing.conf file enumerates the WWPN for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.
| Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric. |
You must update the /etc/fencing.conf file whenever the HBA configuration changes, including the replacement of an HBA.
For dual-ported HBAs, you must include the WWPNs of any ports that are used to access cluster disks. This may result in multiple WWPNs per HBA in the file; the numbers will probably differ by a single digit.
For example, if you determined that port 0 is the port connected to the switch, your fencing file should contain the following (comment lines begin with #):
# WWPN of the HBA installed on this system # 2000000173002c0b |
To configure fencing, see the CXFS 5 Administration Guide for SGI InfiniteStorage.
The /etc/init.d/cxfs_cluster script will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.
To start cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster start |
To stop cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster stop |
To stop and then start cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster restart |
This section contains the following:
| Note: Before upgrading CXFS software, ensure that no applications on the node are accessing files on a CXFS filesystem. |
To upgrade CXFS on a Solaris system, do the following:
Remove the current package:
solaris# pkgrm SGIcxfs The following package is currently installed: SGIcxfs SGI CXFS client software (sparc) releaselevel Do you want to remove this package? [y,n,?,q] y # Removing installed package instance <SGIcxfs> This package contains scripts which will be executed with super-user permission during the process of removing this package. Do you want to continue with the removal of this package [y,n,?,q] y # Verifying package dependencies ...
Reboot the Solaris system:
solaris# reboot
Obtain the CXFS update software from Supportfolio according to the directions in CXFS 5 Administration Guide for SGI InfiniteStorage
Follow the installation instructions to install the new package. See “Client Software Installation for Solaris”.
You can modify the behavior of the CXFS client daemon ( cxfs_client) by placing options in the /usr/cxfs_cluster/bin/cxfs_client.options file. The available options are documented in the cxfs_client man page.
| Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes. |
To see if cxfs_client is using the options in cxfs_client.options, enter the following:
solaris# ps -ef | grep cxfs |
On Solaris nodes, the cxfs-enumerate-wwns script enumerates the world wide names (WWNs) on the host that are known to CXFS. See “CXFS Mount Scripts” in Chapter 1.
The following script is run by cxfs_client when it reprobes the Fibre Channel controllers upon joining or rejoining membership:
/var/cluster/cxfs_client-scripts/cxfs-reprobe |
If you make changes to your storage configuration, you must rerun the HBA utilities to reprobe the storage. See “HBA Installation for Solaris”.
CXFS supports guaranteed-rate I/O (GRIO) version 2 on the Solaris platform if GRIO is enabled on the server-capable administration node. Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the node.
A Solaris node can mount a GRIO-managed filesystem and supports node-level reservations. A Solaris node will interoperate with the dynamic bandwidth allocator for all I/O outside of any reservation.
For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 for Linux Guide.
Following is an example of the /etc/failover2.conf file on Solaris using a QLogic HBA:
/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1 affinity=1 preferred /pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w200400a0b80c268c,1 affinity=1 /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200500a0b80c268c,1 affinity=2 /pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w200500a0b80c268c,1 affinity=2 |
In this case:
SUNW,qlc@1 is the first port on the PCI card
SUNW,qlc@1,1 is the second port on the PCI card
200400a0b80c268c is controller A on the TP9XXX
200500a0b80c268c is controller B on the TP9XXX
Following is an example using an LSI HBA:
<XVM physvol phys/cc_is4000-lun0>
pci@1f,2000/IntraServer,fc@1,1/ssd@0,0 <dev 130> affinity=1
pci@1f,2000/IntraServer,fc@1,1/ssd@2,0 <dev 146> affinity=0 preferred
<XVM physvol phys/cc_is4000-lun1>
pci@1f,2000/IntraServer,fc@1,1/ssd@0,1 <dev 738> affinity=1 preferred
pci@1f,2000/IntraServer,fc@1,1/ssd@2,1 <dev 930> affinity=0
<XVM physvol phys/cc_is4000-lun2>
pci@1f,2000/IntraServer,fc@1,1/ssd@0,2 <dev 746> affinity=1
pci@1f,2000/IntraServer,fc@1,1/ssd@2,2 <dev 938> affinity=0 preferred
<XVM physvol phys/cc_is4000-lun3>
pci@1f,2000/IntraServer,fc@1,1/ssd@0,3 <dev 754> affinity=1 preferred
pci@1f,2000/IntraServer,fc@1,1/ssd@2,3 <dev 946> affinity=0 |
For more information, see “XVM Failover and CXFS” in Chapter 1, the comments in the /etc/failover2.conf file, CXFS 5 Administration Guide for SGI InfiniteStorage, and the XVM Volume Manager Administrator's Guide.
This section contains the following:
For general troubleshooting information, see Chapter 10, “General Troubleshooting” and Appendix D, “Error Messages”.
Confirm that the cxfs_client is not running. The following command would list the cxfs_client process if it were running:
solaris# ps -ef | grep cxfs_client |
Check the cxfs_client log file for errors.
Restart cxfs_client as described in “Start/Stop cxfs_client for Linux” in Chapter 5 and watch the cxfs_client log file for errors.
If cxfs_info reports that cms is up but XVM or the filesystem is in another state, then one or more mounts is still in the process of mounting or has failed to mount.
The CXFS node might not mount filesystems for the following reasons:
The node may not be able to see all the LUNs. This is usually caused by misconfiguration of the HBA or the SAN fabric:
Can the HBA see all of the LUNs for the filesystems it is mounting?
Can the operating system kernel see all of the LUN devices?
The cxfs_client daemon may not be running. See “The cxfs_client Daemon is Not Started on Solaris ”.
The filesystem may have an unsupported mount option. Check the cxfs_client.log for mount option errors or any other errors that are reported when attempting to mount the filesystem.
The cluster membership (cms), XVM, or the filesystems may not be up on the node. Execute the /usr/cxfs_cluster/bin/cxfs_info command to determine the current state of cms, XVM, and the filesystems. If the node is not up for each of these, then check the /var/log/cxfs_client log to see what actions have failed.
Do the following:
If cms is not up, check the following:
Is the node is configured on the server-capable administration node with the correct hostname?
Has the node been added to the cluster and enabled? See “Verifying the Cluster Status” in Chapter 9.
If XVM is not up, check that the HBA is active and can see the LUNs.
If the filesystem is not up, check that one or more filesystems are configured to be mounted on this node and check the /var/log/cxfs_client file for mount errors.
If you have a problem with an HBA, verify that you enabled fabric mode. See “Recognizing Storage Changes for Solaris”.
The /var/log/cxfs_client log file may become quite large over a period of time if the verbosity level is increased. To manually rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file.
For information about the log files created on server-capable administration nodes, see the CXFS 5 Administration Guide for SGI InfiniteStorage.
To view the CXFS heartbeat value on Solaris, use the following:
# echo mtcp_hb_period/D | adb -k physmem 3df86 mtcp_hb_period: mtcp_hb_period: 600 |
Using the -k option to the adb(1) debugger causes it to attach to a live kernel. Echoing the command allows you to put it on a single line.
For example, to reset the value to 15 seconds, enter the following (the value is in Hz):
# echo mtcp_hb_period/W0t1500 | adb -kw physmem 3df86 mtcp_hb_period: 0x258 = 0x5dc |
Before reporting a problem to SGI, you should run the cxfsdump command:
solaris# cxfsdump |
This will collect the following information:
System information
CXFS registry settings
CXFS client logs
CXFS version information
Network settings
Event log
The cxfsdump -help command displays a help message.
Send the tar.gz file that is created in the /var/cluster/cxfsdump-data/date_time directory to SGI.
Also collect the following information:
Obtain information about the entire cluster by running the cxfsdump utility on a server-capable administration node. See the information in the CXFS 5 Administration Guide for SGI InfiniteStorage.
If there is a system panic, retain the system core file in /var/crash/hostname on a Solaris node.
A list of the Solaris patches that have been installed. Use the showrev command. The showrev command without options prints a summary and the -p option lists the revision information about patches.
A list of the loaded Solaris kernel modules and versions. Use the modinfo command.
Output from the LSI /usr/sbin/lsiutil command, which displays the number of LSI HBAs installed, the model numbers, and firmware versions.
If any of the above Solaris tools are not currently installed on your Solaris system, you should install them.