After you have added PVM function calls, a simple UNICOS PVM code can be linked as follows:

cc -o compute -lpvm3 compute.o

This command links the compute.o object code with the PVM library and creates an executable file named compute.

For IRIX systems, if you begin with the source file, you must specify the -I (include) option and the Application Binary Interface (ABI) of the application development library (N32 or 64 ABIs), as follows:

cc -I /usr/array/PVM/include -64 -o compute compute.c -lpvm3

For IRIX systems, if you begin with an object file, the code can be linked as follows:

cc -64 -o compute compute.o -lpvm3

If you have the optional IRIX mpt module loaded, use the following command:

cc -64 -o compute compute.c -lpvm3

After the code is linked, you can install the executable files on the Silicon Graphics systems you will be using. If you specified the ep option in the host file for a system, install the file in the specified directory. Otherwise, install it in the following directory:

$HOME/pvm3/bin/$PVM_ARCH

Each system in the PVM virtual machine must have a separate entry in the host file. Lines that begin with a hash symbol (#), possibly preceded by white space, are ignored.

If you do not want PVM to start a host immediately, but you might start it later by using the pvm_addhosts(3) function or the PVM console add command, you do not need to include the host in the host file. However, if you need to set any of the options described in Table 2-1, you should include the specified system in the host file, preceded by the ampersand (&) character.

A simple host file can be created automatically on UNICOS systems if NQE load balancing is available. Start the PVM daemon with the following command:

pvmd3 -h &

This command starts the PVM daemon in the background and tells it that automatic host file selection should be used. On UNICOS systems, the PVM daemon queries the load balancing server for available hosts and includes all available hosts in your virtual machine. Available hosts are determined by the PVM policy in NQE or by a policy specified in the PVM_POLICY environment variable. Hosts can be excluded based on many different resources. For more information on NQE policies, see NQE Administration. If a host file is also specified, PVM uses the options specified in the host file. A host specified in the host file will be included in the virtual machine only if that host is available, as determined by the NQE policy.

Example 2-1, is an example of a host file that contains the names of systems, which is the basic information necessary in a host file.

# my first host file
thud
fred
wilma
gust.cray.com
rain 

You should verify that no system is listed more than once, and that the system on which the master pvmd3(1) daemon will run (the master host) is included in the host file (see “Starting and Stopping the PVM Daemon”, for information on starting the pvmd3 daemon). Automatic host file selection always includes the host running the master pvmd3(1) daemon.

The $PVM_ROOT and $PVM_ARCH environment variables are set for you automatically when you load the mpt module to access the Message Passing Toolkit software. To customize your environment, you can specify the options listed in Table 2-1, after any system name in the host file.

A dollar sign ($) in an option introduces an environment variable name, for example, $PVM_ARCH. Each PVM daemon expands names from environment variables.

The simple host file in Example 2-1, works well if both of the following conditions are met:

To supply an alternative login name for the thud system, add the lo option to its host file entry, as follows:

thud lo=NAME 

To be queried for your password on a system named cyclone, add so=pw to its host file entry, as follows:

cyclone so=pw

To specify the path of the daemon executable file for a system named sun114, add the dx option, as follows:

sun114 dx=/usr/fred/pvm3/lib/Sun/pvmd3

---

Note: By default, the MPT version of pvmd3 is installed in $PVM_ROOT/lib/$PVM_ARCH/pvmd3 on Silicon Graphics systems, where $PVM_ROOT and $PVM_ARCH are set for you automatically when you load the mpt module.

---

The string specified in the previous example is passed to a shell so that variable expansion works. Following is another example that uses variable expansion:

sun114 dx=bin/$MYBIN/pvmd3

You can change the default value of any option for all hosts in a host file by specifying them on a line with an asterisk (*) in the host field, as in the following example:

thud.cs.utk.edu
gust.cray.com
sun114 dx=/tmp/pvmd3
* lo=afriend so=pw

The preceding example sets the default login name (on remote systems) to afriend and queries for a password on each system. Defaults set in this way are effective forward from the location at which they occur in the host file. They can be changed with another * line.

You can override the location of executable files by adding the ep option to your host file entries, as in the following example:

ep=$HOME/pvm3/bin

Unlike the dx option, which names the daemon file, the ep option names a directory.

Example 2-2 shows a more complex host file in which host names are followed by options.

# host file for testing on various platforms
# default to my executable
*                  dx=pvm/SUN4/pvmd3
fonebone
refuge
sigi.cs            dx=pvm/PMAX/pvmd3
# reset default for other systems
*                  dx=$PVM_ROOT/lib/$PVM_ARCH/pvmd3
# do not start this system, but define ep in case we add it later
& rain.cray.com  ep=$(HOME)/bin ip=rain-hippi
# borrowed accts, "guest", don't trust fonebone
*                  lo=guest  so=pw
sn666.jrandom.com  ep=$(HOME)/bin
cubie.misc.edu     ep=pvm/IPSC/pvmd3

Before you run a PVM executable file on an IRIX system, you must specify the architecture type by setting the PVM_ARCH environment variable. Four architecture types are supported for IRIX systems. With the software installed in the default locations, you must also set the PVM_ROOT environment variable to /usr/array/PVM and the PATH environment variable to $PVM_ROOT/lib/$PVM_ARCH. The following C shell example shows the setting of all three variables:

setenv PVM_ARCH SGIMP64
setenv PVM_ROOT /usr/array/PVM
setenv PATH ${PATH}:${PVM_ROOT}/lib/$PVM_ARCH

The architecture types shown in the following list are arranged in an approximate order of lowest to highest performance types:

After you have written a host file, you can start up the master pvmd3(1) daemon by passing it the host file as an argument. You must specify the appropriate path for pvmd3(1). On a Silicon Graphics system, for example, you can enter one of the following:

pvmd3 hostfile &orpvm [hostfile]

If you do not specify a host file when starting the PVM console, the PVM daemon found in the default location will be started on the local machine.

The ampersand (&) in the first line tells the operating system to run pvmd3(1) in the background, which is what you will normally want to do.

You should not run pvmd3(1) in the background if you have to enter passwords for any of the slave systems (that is, if you included the so=pw option for one or more systems). In this case, run pvmd3(1) in the foreground and then stop it (by pressing CONTROL-Z) and put it in the background (by entering bg at the prompt) after all systems have started up.

To shut off PVM, enter halt at a PVM console prompt. For detailed information on using console prompts, see “Using Console Commands”.

If the master pvmd3(1) daemon has trouble starting a slave pvmd3(1) daemon on a system, the error message written to the PVM log file from the master pvmd3(1) may indicate the problem.

When the pvmd3(1) daemon is running successfully, you can start your application. PVM provides the following methods of starting applications:

You can obtain help for the spawn command by typing help spawn at the console command line prompt.

Once the application has started, it displays standard output and standard error information for the initial task, but not for the other tasks in the application. PVM captures this output information and sends it to the master daemon. The daemon, in turn, prefaces each line with a PVM task identifier that identifies its source, and writes it to the PVM log file.

The log file can contain very useful information about the virtual machine and its tasks. By default, the log file contains output from the PVM daemon, including error messages and output from tasks. Optionally, the log file can contain debugging output from the daemon.

When PVM is run without NQS, the log file is located in /tmp. When PVM is run without NQE on UNICOS and UNICOS/mk systems, the log file is located in /tmp/pvml.uid, where uid is the user ID. The IRIX implementation allows overlapping PVM virtual machines. Therefore, more than one PVM daemon started by the same user can run on the same host. The log file is located in /tmp/pvml.uid.vmid, where uid is the user ID and vmid is the virtual machine ID. By default, is 0, but if the PVM_VMID (formerly PVMJID) environment variable is set, vmid will equal the numeric value of PVM_VMID.

When PVM is run using NQS on UNICOS and UNICOS/mk systems, the name of the log file is $TMPDIR/pvml.uid. $TMPDIR is a temporary directory created for the NQS job. You can examine this file at any time, but remember that each task buffers the output written to standard output independently (unless you flush the output after each write request), and so the sequence of output from two different tasks may vary.

Instead of having the data written to the PVM log file, you can request that output be sent as a PVM message to another task's output device. For more information, see the PvmOutputTid and PvmOutputCode options on the pvm_setopt(3) man page.

You can also redirect output by using options on the console spawn command (see Table 2-2) or by using the pvm_catchout(3) function. For more information about running your program in stand-alone mode, see “Running PVM in Stand-alone Mode on UNICOS Systems”, or Chapter 3, “UNICOS/mk Implementation”.

On UNICOS, UNICOS/mk, and IRIX systems, PVM applications can be run as part of an NQS job script. Each NQS job has its own PVM daemon; therefore, the PVM daemon must be started within the NQS job script. This is different from interactive use, in which one daemon is run per user per system. Any application run as part of the same NQS job script uses the same PVM daemon. Slave daemons that run on UNICOS and UNICOS/mk systems will also run one daemon per NQS job. IRIX users can achieve similar functionality as UNICOS and UNICOS/mk users by using the PVM_VMID environment variable. Using PVM_VMID allows more than one daemon to run per user per system.

In UNICOS and UNICOS/mk implementations, a special environment variable is checked for a batch (NQE) environment. If a batch environment exists, these implementations place the PVM log and daemon (pvmd socket) files in a special temporary directory. In IRIX implementations, a single user running multiple NQE jobs on a single host should set the PVM_VMID environment variable for each batch job.

PVM processes spawned by the daemon inherit the limits of the NQS job. This allows a user to run multiple NQS jobs that use PVM, each with limits of the NQS job being run. Previous versions of PVM used the same daemon for multiple NQS jobs.

The following example is an NQS job script to run the application foo:

module load mpt
pvmd3 hostfile &   # Start the daemon
sleep 60        # Wait for startup
foo             # Run application
pvm << EOF      # Start console to halt pvm
halt
EOF

Using the PVM console is an alternative to using the pvmd3(1) command to start the daemon and execute your application. The pvm(1) command starts the console, which can be started and stopped multiple times on any of the systems on which PVM is running.

pvm [hostfile]

% pvm
pvmd already running
pvm>

This section describes common problems encountered when using PVM and provides suggested solutions. There are several kinds of problems that can keep pvmd3(1) from building a virtual machine. The most common are permission problems.

If you do not specify the pw option for a particular system, your .rhosts file on that system must contain the name of the host from which you start the master pvmd. Otherwise, you will get a message like one of the following (although you may not get the entire message):

pvmd3@hostname: Permission denied

pvmd3@hostname: Login incorrect

To get the entire error message, enter the following command at a shell prompt:

rsh hostname daemon

On UNICOS and UNICOS/mk systems, you need to use remsh(1) rather than rsh(1) because the name rsh is used for a restricted shell, not for the remote shell command. The remsh command is not available on IRIX systems. daemon is the location of the PVM daemon (for example, /tmp/pvm/pvmd3 or $PVM_ROOT/lib/$PVM_ARCH/pvmd3).

Look at the output of the command and consult whichever of the following sections most closely applies.

pvmd3 Fails to Start on Remote System

If you use a shell (such as .kshrc) that does not automatically execute a startup script that sets $PVM_ROOT on added hosts, you can set the PVM_DPATH environment variable to the full or relative path of the pvmd startup script, or include the dx option in the host file to specify the path to the startup script. The pvmd startup script automatically sets $PVM_ROOT on the remote host.

The following command shows how to set the PVM_DPATH environment variable:

setenv PVM_DPATH $PVM_ROOT/lib/pvmd

The following command shows how to specify the pvmd startup script in the host file:

dx=/opt/ctl/mpt/mpt/pvm3/lib/pvmd

---

Note: The dx option in the host file overrides the PVM_DPATH environment variable, and $PVM_ROOT is not acknowledged for dx, so the dx path must be a full pathname.

---

local-host-name your-local-user-name

% who am i

if ( { tty -s } && $?prompt )  then
# example of printing something when you log in
 echo terminal type is $TERM
# example of setting terminal attributes
   stty erase '^?' kill '^u' intr '^c' echo endif

Several PVM functions are particularly useful when developing applications that involve UNICOS and UNICOS/mk systems. This section discusses some techniques that can help improve performance. As is true with programming in general, optimization with PVM involves trade-offs. Generally, the trade-off involves reducing generality in favor of better performance. Adding some of the optimizations discussed in the following sections will improve performance but will make the application harder to move to different PVM virtual machines or other systems.

---

Note: Information in this section is for UNICOS systems only.

---

Emulating a shared memory environment with the use of Cray multitasking software might provide unexpected program behavior. The goal is to preserve the original behavior as much possible. However, it is not efficient or productive to preserve completely the original PVM behavior in a multitasked environment. The intent is to document possible changes in behavior. For example, changes in behavior might occur with the use of signals; therefore, it is not recommended that signals be used with the shared memory version of PVM.

PvmDataDefault and PvmDataRaw packing are equivalent in the shared memory implementation of PVM. In stand-alone mode, data conversion is not necessary because the executable file never communicates outside of the UNICOS machine. Because the shared memory implementation of PVM running in standard mode does not handle data conversion, multitasked executable files currently can communicate only between UNICOS machines of the same architecture type. Communication to other architectures can still be achieved, however, through the master or nonmultitasked executable files that comprise the program.

The shared memory implementation of PVM supports the running of only 32 PVM tasks within a multitasking group. Running with more than the number of physical CPUs available on the UNICOS system will begin to degrade performance because PVM tasks must share CPU resources.

Table 2-5 does not contain some Fortran type names, such as INTEGER2, REAL4, and COMPLEX8. These map to INTEGER4, REAL8, and COMPLEX16, respectively. To avoid confusion and to ease porting to other systems, you should use these only with the greatest of care.

The Cray MPT implementation of PVM for UNICOS systems does not support the INTEGER8 specification. On UNICOS systems, you must specify INTEGER4 for UNICOS integers.

When PvmDataDefault packing is used, XDR converts data into a common format. XDR retains only 32 bits of precision for integer data; therefore, packing 64-bit integers results in a loss of the upper 32 bits of precision.

If you want all 64 bits of accuracy, use PvmDataRaw packing or specify untyped byte packing for PvmDataDefault.

The Fortran name INTEGER2 is implemented and maps into the same data characteristics as INTEGER4. Use of 16-bit data types is not recommended because the UNICOS/mk system does not support these data types.

On the UNICOS/mk system, PVM supports 32-bit Fortran data types. This support is implemented in PVM regardless of whether your Fortran compiler supports 32-bit data types. If you are not using such a compiler and specify one of these data types (INTEGER4, REAL4, or COMPLEX8), you will get incorrect results.

In the UNICOS and public domain versions of PVM, data conversion of integers is limited to 32 bits of accuracy. The UNICOS/mk version handles 64-bit integers in a manner that is compatible and interoperable with the network version. If you pack 64-bit integers into a PvmDataDefault block, only the low-order 32 bits of each value are packed. PVM checks the high-order 32 bits of each value; if they contain significant data, the pack call sends a PvmLostPrecision error. (This checking can be turned off; see PVM_CHECKING in Table 3-1.)

If you want 64 bits of accuracy, you can use PvmDataRaw packing or specify untyped byte packing for PvmDataDefault.

This section provides a table of environment variables you can set for IRIX systems only.

Table 2-9. Environment Variables on IRIX Systems

Variable	Description	Default
PVM_SHMEM_DIR	Directory location of the POSIX shared memory files.	/usr/tmp (Only valid for SGIMP64 and SGIMP64mips3 architecture types)
PVMBUFSIZE	Specifies the size of the shared memory buffer for each task and daemon.	1 Mbyte
PVM_VMID	Sets the virtual machine identification (VMID) number for the host. This environment variable allows a host to be included in more than one virtual machine by using one pvmd3 command per virtual machine per host. The virtual machine number is appended to the file name of the PVM log and daemon socket files, so that they appear as pvml.uid.vmid and pvmd.uid.vmid. The previous name of this variable is PVMJID. This name is supported in the MPT 1.3 release, but will not be supported in subsequent releases.	0
	Note: This environment variable prevents IRIX PVM from interoperating with any implementation other than Silicon Graphics IRIX PVM implementations.

This section provides a table of environment variables you can set for UNICOS, UNICOS/mk, and IRIX systems.

Table 2-10. Environment Variables on UNICOS, UNICOS/mk, and IRIX Systems

Variable	Description	Default
PVM_ROOT	Specifies the path where PVM libraries and system programs are installed. For PVM to function, this variable must be set on each PVM system.	Set automatically when you load the mpt module to access the Message Passing Toolkit software
PVM_EXPORT	Names the environment variables that a parent task exports to its children by using the pvm_spawn(3) function. Multiple names must be separated by a colon.	None
PVM_DEBUGGER	Specifies the debugger script to use when pvm_spawn(3) is called with PvmTaskDebug set.	$PVM_ROOT/lib/debugger
PVM_DPATH	Specifies the path of the pvmd3(1) command or the startup script. If you use a shell (such as .kshrc) that does not automatically execute a startup script that sets PVM_ROOT on added hosts, you can set PVM_DPATH to the full or relative path of the pvmd startup script, such as $PVM_ROOT/lib/pvmd. This startup script automatically sets PVM_ROOT.	$PVM_ROOT/lib/pvmd. You can override this setting by using the dx= loc option in the host file.
PVM_POLICY	Specifies the NQE policy used for load balancing. For more information on specifying policies, see NQE Administration. Note: Support for this environment variable is deferred on UNICOS/mk and IRIX systems.	PVM
NLB_SERVER	Specifies the location of the NQE load balancer. This host is known as the master server. Your system administrator might have this set automatically in the nqeinfo file. If NQE load balancing is enabled on your system, it is used automatically by PVM. To disable NQE load balancing for PVM applications, set the NLB_SERVER environment variable to 0. For more information, see the NQE User's Guide. Note: Support for this environment variable is deferred on UNICOS/mk and IRIX systems.	Value in the nqeinfo file
PVM_RSH	Specifies that an alternative remote shell command, such as krsh (a Kerberos version of rsh), can be selected. PVM_RSH can specify the full path or relative path to the alternative remote command.	IRIX: If using Array Services, /usr/sbin/arshell. If not using Array Services, /usr/bsd/rsh. UNICOS or UNICOS/mk: /usr/ucb/remsh.
PVM_SLAVE_STARTUP_TIMEOUT	Specifies the length of time that the master daemon will wait for a slave daemon to make contact after the slave daemon is started.	60 seconds