openpbs/doc/man8/pbsrun.8B

.\" Copyright (C) 1994-2018 Altair Engineering, Inc.
.\" For more information, contact Altair at www.altair.com.
.\"
.\" This file is part of the PBS Professional ("PBS Pro") software.
.\"
.\" Open Source License Information:
.\"
.\" PBS Pro is free software. You can redistribute it and/or modify it under the
.\" terms of the GNU Affero General Public License as published by the Free
.\" Software Foundation, either version 3 of the License, or (at your option) any
.\" later version.
.\"
.\" PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY
.\" WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
.\" FOR A PARTICULAR PURPOSE.
.\" See the GNU Affero General Public License for more details.
.\"
.\" You should have received a copy of the GNU Affero General Public License
.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
.\"
.\" Commercial License Information:
.\"
.\" For a copy of the commercial license terms and conditions,
.\" go to: (http://www.pbspro.com/UserArea/agreement.html)
.\" or contact the Altair Legal Department.
.\"
.\" Altair’s dual-license business model allows companies, individuals, and
.\" organizations to create proprietary derivative works of PBS Pro and
.\" distribute them - whether embedded or bundled with other software -
.\" under a commercial license agreement.
.\"
.\" Use of Altair’s trademarks, including but not limited to "PBS™",
.\" "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's
.\" trademark licensing policies.
.\"
.TH pbsrun 8B "13 October 2017" Local "PBS Professional"
.SH NAME
.B pbsrun
\- general-purpose wrapper script for mpirun

.SH SYNOPSIS
.B pbsrun

.B pbsrun
--version

.SH DESCRIPTION
.B pbsrun
is a wrapper script for any of several versions of mpirun.
This provides a user-transparent way for PBS to control jobs
which call mpirun in their jobscripts.
The
.B pbsrun_wrap
script instantiates 
.B pbsrun
so that the wrapper script for the specific version of mpirun
being used has the same name as that version of mpirun.

If the mpirun wrapper script is run inside a PBS job, it
translates any mpirun call of the form:
.br
    mpirun [<options>] <executable> [<args>]
.br
into
.br
    mpirun [<options>] pbs_attach [<special options to pbs_attach>] \\
.br
            <executable> [<args>]
.br
where 
.I special options to pbs_attach 
refers to any options needed by 
.B pbs_attach 
to do its job (e.g. -j $PBS_JOBID).

If the wrapper script is executed outside of PBS, a warning is issued
about "not running under PBS", but it proceeds as if the actual
program had been called in standalone fashion.

The 
.B pbsrun 
wrapper script is not meant to be executed directly; instead
it is instantiated by 
.B pbsrun_wrap.
It is copied to the target directory and renamed
"pbsrun.<mpirun version/flavor>" where 
.I mpirun version/flavor
is a string that identifies the mpirun
version being wrapped (e.g. ch_gm).

The 
.B pbsrun 
script, if executed inside a PBS job,
runs an 
.B initialization script, 
named $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init, then
parses mpirun-like arguments from the command line, sorting which
options and option values to retain, to ignore, or to transform,
before calling the actual mpirun script with a "pbs_attach" prefixed
to the executable.  The actual mpirun to call is found by tracing the
link pointed to by $PBS_EXEC/lib/MPI/pbsrun.<mpirun
version/flavor>.link.

For all of the wrapped MPIs, the maximum number of ranks that can be
launched is the number of entries in $PBS_NODEFILE.

The wrapped MPIs are:
.RS 5
MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh (
.B deprecated
as of 14.2.1)
.br
MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh (
.B deprecated
as of 14.2.1)
.br
MPICH-GM's mpirun (mpirun.mpd) with MPD (
.B deprecated
as of 14.2.1)
.br
MPICH-MX's mpirun (mpirun.mpd) with MPD (
.B deprecated
as of 14.2.1)
.br
MPICH2's mpirun
.br
Intel MPI's mpirun (
.B deprecated
as of 13.0)
.br
MVAPICH1's mpirun (
.B deprecated
as of 14.2.1)
.br
MVAPICH2's mpiexec
.br
IBM's poe
.RE

.SH OPTIONS
.IP "--version" 8
The 
.B pbsrun
command returns its PBS version information and exits.
This option can only be used alone.

.SH INITIALIZATION SCRIPT

The initialization script, called $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init,
where 
.I mpirun version/flavor 
reflects the mpirun flavor/version being wrapped,
can be modified by an administrator to customize against the local
flavor/version of mpirun being wrapped.

Inside this sourced init script, 8 variables are set:
.RS 5
options_to_retain="-optA -optB <val> -optC <val1> val2> ..."
.br
options_to_ignore="-optD -optE <n> -optF <val1> val2> ..."
.br
options_to_transform="-optG -optH <val> -optI <val1> val2> ..."
.br
options_to_fail="-optY -optZ ..."
.br
options_to_configfile="-optX <val> ..."
.br
options_with_another_form="-optW <val> ..."
.br
pbs_attach=pbs_attach
.br
options_to_pbs_attach="-J $PBS_JOBID"
.br
.RE

.B Initialization Script Options
.br
.IP "options_to_retain" 5
Space-separated list of options and values that
pbsrun.<mpirun version/flavor> passes on to the actual mpirun call. Options must
begin with "-" or "--", and option arguments must be specified by some
arbitrary name with left and right arrows, as in "<val1>".

.IP "options_to_ignore" 5
Space-separated list of options and values that pbsrun.<mpirun
version/flavor> does not pass on to the actual mpirun call. Options
must begin with "-" or "--", and option arguments must be specified by
arbitrary names with left and right arrows, as in "<n>".

.IP "options_to_transform" 5
Space-separated list of options and values
that 
.B pbsrun 
modifies before passing on to the actual mpirun call.

.IP "option_to_fail" 5
Space-separated list of options that will cause
.B pbsrun 
to exit upon encountering a match.

.IP "options_to_configfile" 5
Single option and value that refers to
the name of the "configfile" containing command line segments found in
certain versions of mpirun. 

.IP "options_with_another_form" 5
Space-separated list of options and values that can be found in
options_to_retain, options_to_ignore, or options_to_transform, whose
syntax has an alternate, unsupported form.

.IP "pbs_attach" 5
Path to 
.B pbs_attach, 
which is called before the
<executable> argument of mpirun.

.IP "options_to_pbs_attach" 5
Special options to pass to the
.B pbs_attach call. 
You may pass variable references (e.g. $PBS_JOBID) and
they are substituted  by 
.B pbsrun 
to actual values.
.LP
.RE

If 
.B pbsrun 
encounters any option not found in 
.I options_to_retain, options_to_ignore, 
and 
.I options_to_transform, 
then it is flagged
as an error.

.B Functions Created
.br
These functions are created inside the init script.  These can be
modified by the PBS administrator.
.RS 5

.IP "transform_action () {" 5
     # passed actual values of $options_to_transform
.br
     args=$*
.br
}

.IP "boot_action () {" 5
     mpirun_location=$1
.br
}

.IP "evaluate_options_action () {" 5
     # passed actual values of transformed options
.br
     args=$*
.br
}

.IP "configfile_cmdline_action () {" 5
     args=$*
.br
}
		
.IP "end_action () {" 5
     mpirun_location=$1
.br
}
.LP

.IP "transform_action()" 5
The pbsrun.<mpirun version/flavor> wrapper script invokes the function
.I transform_action() 
(called once on each matched item and value) with
actual options and values received matching one of the
"options_to_transform". The function returns a string to pass on
to the actual mpirun call.


.IP "boot_action()" 5
Performs any initialization tasks needed before running the actual
mpirun call. For instance, GM's MPD requires the MPD daemons to be
user-started first. This function is called by the pbsrun.<mpirun
version/flavor> script with the location of actual mpirun passed as
the first argument. Also, the pbsrun.<mpirun version/flavor> checks
for the exit value of this function to determine whether or not to
progress to the next step.


.IP "evaluate_options_action()" 5
Called with the actual options
and values that resulted after consulting 
.I options_to_retain, options_to_ignore, options_to_transform, 
and executing 
.I transform_action().
This provides one more chance for the script writer to evaluate all
the options and values in general, and make any necessary adjustments,
before passing them on to the actual mpirun call. For instance,
this function can specify what the default value is for a missing 
.I -np
option.


.IP "configfile_cmdline_action()" 5
Returns the actual options and values
to be put in before the option_to_configfile parameter. 

.IP "configfile_firstline_action()" 5
Returns the item that is 
put in the first line of the configuration file specified in the
.I option_to_configfile 
parameter.

.IP "end_action()" 5
Called by pbsrun.<mpirun version/flavor> at the end of execution.  It
undoes any action done by transform_action(), like cleanup of
temporary files. It is also called when pbsrun.<mpirun
version/flavor> is prematurely killed. This function is called
with the location of actual mpirun passed as first argument.
.RE

The actual mpirun program to call is the path pointed to by
    $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.link.


.B Modifying *.init scripts
.br
In order for administrators to modify *.init scripts without breaking
package verification in RPM, master copies of the initialization
scripts are named *.init.in.  
.B pbsrun_wrap 
instantiates the *.init.in files as *.init. For instance,
$PBS_EXEC/lib/MPI/pbsrun.mpich2.init.in is the master copy, and
.B pbsrun_wrap 
instantiates it as $PBS_EXEC/lib/MPI/pbsrun.mpich2.init.
.B pbsrun_unwrap 
takes care of removing the *.init files.


.SH MPIRUN VERSIONS/FLAVORS
-----------------------------------------------------------
.br
.B MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh: pbsrun.ch_gm
.br
-----------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.ch_gm <options> <executable> <arg1> 
.B <arg2> ... <argn>

.B Deprecated.
This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.ch_gm) with
rsh/ssh process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-GM
processes started by rsh/ssh so that PBS can perform accounting and 
have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
.B mpirun.ch_gm 
was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all 
.B mpirun.ch_gm 
options given are
passed on to the actual mpirun call with these exceptions:

.IP "-machinefile <file>" 5
The 
.I file 
argument contents are ignored and replaced by the contents of
the $PBS_NODEFILE.

.IP "-np" 5
If not specified, the number of entries found in the
$PBS_NODEFILE is used.

.IP "-pg" 5
The use of the 
.I -pg 
option, for having multiple executables on multiple
hosts, is allowed but it is up to user to make sure 
only PBS hosts are specified in the process group file; MPI processes
spawned are not guaranteed to be under the control of PBS.
.RE

WRAP/UNWRAP
.RS 5
To wrap MPICH-GM's mpirun script:
.RS 4
.B # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.ch_gm pbsrun.ch_gm
.RE
To unwrap MPICH-GM's mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.ch_gm
.RE
.RE
.RE

-----------------------------------------------------------
.br
.B MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh: pbsrun.ch_mx
.br
-----------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.ch_mx <options> <executable> <arg1> 
.B <arg2> ... <argn>

.B Deprecated.  
This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.ch_mx) with
rsh/ssh process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-MX
processes started by rsh/ssh so that PBS can perform accounting and 
has complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
mpirun.ch_mx was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all mpirun.ch_gm options given are
passed on to the actual mpirun call with some exceptions:

.IP "-machinefile <file>" 5
The 
.I file 
argument contents is ignored and replaced by the contents
of the $PBS_NODEFILE.

.IP "-np" 5
If not specified, the number of entries found in the
$PBS_NODEFILE is used.

.IP "-pg" 5
The use of the 
.I -pg 
option, for having multiple executables on multiple
hosts, is allowed but it is up to user to make sure 
only PBS hosts are specified in the process group file; MPI processes
spawned are not guaranteed to be under the control of PBS.
.RE

WRAP/UNWRAP
.RS 5
To wrap MPICH-MX's mpirun script:
.RS 4
.B # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.ch_mx pbsrun.ch_mx
.RE
To unwrap MPICH-MX's mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.ch_mx
.RE
.RE 
.RE

--------------------------------------------------------
.br
.B MPICH-GM's mpirun (mpirun.mpd) with MPD: pbsrun.gm_mpd
.br
--------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.gm_mpd <options> <executable> <arg1> 
.B <arg2> ... <argn>

.B Deprecated.  
This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.mpd) with
MPD process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-GM
processes started by the MPD daemons so that PBS can perform accounting 
have and complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
mpirun.ch_gm with MPD was used.
.RE

OPTIONS HANDLING
.RS 5 
If executed inside a PBS job script, all mpirun.ch_gm with MPD options given
are passed on to the actual mpirun call with these exceptions:

.IP "-m <file>" 5
The 
.I file 
argument contents are ignored and replaced by the contents of
the $PBS_NODEFILE.

.IP "-np" 5
If not specified, the number of entries found in the
$PBS_NODEFILE is used.

.IP "-pg" 5
The use of the 
.I -pg 
option, for having multiple executables on multiple
hosts, is allowed but it is up to user to make sure 
only PBS hosts are specified in the process group file; MPI processes
spawned are not guaranteed to be under the control of PBS.
.RE

STARTUP/SHUTDOWN
.RS 5
The script starts MPD daemons on each of the unique hosts listed in
$PBS_NODEFILE, using either rsh or ssh method based on value of environment
variable RSHCOMMAND.  The default is rsh.

The script also takes care of shutting down the MPD daemons at the end of
a run.
.RE

WRAP/UNWRAP
.RS 5
To wrap MPICH-GM's mpirun script with MPD:
.RS 4
.B # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.mpd pbsrun.gm_mpd
.RE
To unwrap MPICH-GM's mpirun script with MPD:
.RS 4
.B # pbsrun_unwrap pbsrun.gm_mpd
.RE
.RE
.RE

--------------------------------------------------------
.br
.B MPICH-MX's mpirun (mpirun.mpd) with MPD: pbsrun.mx_mpd
.br
--------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.mx_mpd <options> <executable> <arg1> 
.B <arg2> ... <argn>

.B Deprecated.  
This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.mpd) with
MPD process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-MX
processes started by the MPD daemons so that PBS can perform accounting 
and have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
mpirun.ch_mx with MPD was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all mpirun.ch_gm with MPD options given
are passed on to the actual mpirun call with these exceptions:

.IP "-m <file>" 5
The 
.I file
argument contents are ignored and replaced by the contents of
the $PBS_NODEFILE.

.IP "-np" 5
If not specified, the number of entries found in the $PBS_NODEFILE is used.

.IP "-pg" 5
The use of the 
.I -pg 
option, for having multiple executables on multiple
hosts, is allowed but it is up to user to make sure 
only PBS hosts are specified in the process group file; MPI processes
spawned are not guaranteed to be under the control of PBS.
.RE

STARTUP/SHUTDOWN
.RS 5
The script starts MPD daemons on each of the unique hosts listed in
$PBS_NODEFILE, using either rsh or ssh method, based on value of environment
variable RSHCOMMAND -- rsh is the default.

The script also takes care of shutting down the MPD daemons at the end of
a run.
.RE

WRAP/UNWRAP
.RS 5
To wrap MPICH-MX's mpirun script with MPD:
.RS 4
.B # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.mpd pbsrun.mx_mpd
.RE
To unwrap MPICH-MX's mpirun script with MPD:
.RS 4
.B # pbsrun_unwrap pbsrun.mx_mpd
.RE
.RE
.RE

------------------------------
.br
.B MPICH2's mpirun: pbsrun.mpich2
.br
------------------------------

SYNTAX
.RS 5

.B pbsrun.mpich2 [<global args>] [<local args>] 
.B <executable> 
.RS 14
.B [<args>]
.B [: [<local args>] <executable> [<args>]] 
.RE
.br
- or - 
.br
.B pbsrun.mpich2 -configfile <configfile>

where <configfile> contains command line segments as lines:
.RS 5
[local args] executable1 [args]
.br
[local args] executable2 [args]
.br
[local args] executable3 [args]
.RE

This is the PBS wrapper script to MPICH2's mpirun.

If executed inside a PBS job, this allows for PBS to track all MPICH2
processes so that PBS can perform accounting and have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
MPICH2's mpirun was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all MPICH2's mpirun options given
are passed on to the actual mpirun call with these exceptions:

.IP "-host and -ghost" 5
For specifying the execution host to run
on.  Not passed on to the actual mpirun call.

.IP "-machinefile <file>" 5
The 
.I file 
argument contents are ignored and replaced by the
contents of the $PBS_NODEFILE.

.IP "MPICH2's mpirun -localonly <x>" 5
For specifying the <x> number of
processes to run locally. Not supported. The user is advised
instead to use the equivalent arguments: 
.I "-np <x> -localonly".  
The reason
for this is that the 
.B pbsrun 
wrapper script cannot handle a variable number
of arguments to an option (e.g. "-localonly" has 1 argument and "-localonly <x>"
has 2 arguments).

.IP "-np" 5
If user did not specify a 
.I -np 
option, then no default value is provided
by the PBS wrapper scripts. It is up to the local mpirun to decide what
the reasonable default value should be, which is usually 1.
.RE 

STARTUP/SHUTDOWN
.RS 5
The script takes care of ensuring that the MPD daemons on each of the hosts
listed in the $PBS_NODEFILE are started. It also takes care of ensuring
that the MPD daemons have been shut down at the end of MPI job execution. 
.RE

WRAP/UNWRAP
.RS 5
To wrap MPICH2's mpirun script:
.RS 4
.B # pbsrun_wrap [MPICH2_BIN_PATH]/mpirun pbsrun.mpich2
.RE
To unwrap MPICH2's mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.mpich2
.RE
.RE
.RE

-----------------------------------
.br
.B Intel MPI's mpirun: pbsrun.intelmpi 
.br
-----------------------------------

Wrapping Intel MPI, and support for mpdboot, are 
.B deprecated.

SYNTAX
.RS 5

.B pbsrun.intelmpi  [<mpdboot options>] [<mpiexec options>]
.RS 16
.B <executable> [<prog args>]
.br 
.B [: [<mpiexec options>] <executable> [<prog args>]]
.RE
.br
- or - 
.br
.B pbsrun.intelmpi [<mpdboot options>] -f <configfile>

where 
.I mpdboot options 
are any options to pass to the mpdboot program,
which is automatically called by Intel MPI's mpirun to start MPDs, and
.I configfile 
contains command line segments as lines.

This is the PBS wrapper script to Intel MPI's mpirun.

If executed inside a PBS job, this allows for PBS to track all Intel MPI
processes so that PBS can perform accounting and have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
Intel MPI's mpirun was used.
.RE

Using 
.B pbsrun, pbsrun_wrap, 
and 
.B pbsrun_unwrap
with Intel MPI is 
.B deprecated 
as of 13.0.

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all of the options to the PBS 
interface to MPI's mpirun are passed to the actual 
mpirun call with these exceptions:

.IP "-host and -ghost" 5
For specifying the execution host to run
on.  Not passed on to the actual mpirun call.

.IP  "-machinefile <file>" 5
The 
.I file
argument contents are ignored and replaced by the
contents of the $PBS_NODEFILE.

.IP "mpdboot options --totalnum=* and --file=*" 5
Ignored and replaced by the number of unique entries in $PBS_NODEFILE
and name of $PBS_NODEFILE respectively.

.IP "arguments to mpdboot options --file=* and -f <mpd_hosts_file>" 5
Replaced by $PBS_NODEFILE.

.IP "-s" 5 
If 
.B pbsrun.intelmpi 
is called inside a PBS job, Intel MPIs mpirun 
.I -s 
argument
to mpdboot are not supported as this closely matches the mpirun option
.I "-s <spec>". 
The user can simply run a separate mpdboot 
.I -s 
before calling
mpirun.  A warning message is issued by 
.B pbsrun.intelmpi 
upon
encountering a 
.I -s 
option telling users of the supported form.

.IP "-np" 5
If the user does not specify a 
.I -np 
option, then no default value is provided
by the PBS wrap scripts. It is up to the local mpirun to decide what
the reasonable default value should be, which is usually 1.
.RE

STARTUP/SHUTDOWN
.RS 5
Intel MPI's mpirun itself takes care of starting/stopping the
MPD daemons. 
.B pbsrun.intelmpi 
always passes the arguments
.I -totalnum=<number of mpds to start> 
and 
.I -file=<mpd_hosts_file> 
to the actual
mpirun, taking its input from unique entries in $PBS_NODEFILE.
.RE

WRAP/UNWRAP
.RS 5
To wrap Intel MPI's mpirun script:
.RS 4
.B # pbsrun_wrap [INTEL_MPI_BIN_PATH]/mpirun pbsrun.intelmpi
.RE
To unwrap Intel MPI's mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.intelmpi
.RE
.RE
.RE


-----------------------------------------------------------
.br
.B MVAPICH1's mpirun: pbsrun.mvapich1
.br
-----------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.mvapich1 <mpirun options> <executable> <options>

.B Deprecated.  
Only one executable can be specified.
This is the PBS wrapper script to MVAPICH1's mpirun.  

If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH1 
ranks and track their resources, so that PBS can perform accounting and 
have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
.B mpirun
was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all 
.B mpirun
options given are
passed on to the actual mpirun call with these exceptions:

.IP "-map <list>" 5
The 
.I map
option is ignored.

.IP "-exclude <list>" 5
The 
.I exclude
option is ignored.

.IP "-machinefile <file>" 5
The 
.I machinefile
option is ignored.

.IP "-np" 5
If not specified, the number of entries found in the
$PBS_NODEFILE is used.

.RE


WRAP/UNWRAP
.RS 5
To wrap MVAPICH1's mpirun script:
.RS 4
.B # pbsrun_wrap  <path-to-actual-mpirun> pbsrun.mvapich1
.RE
To unwrap MVAPICH1's mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.mvapich1
.RE
.RE
.RE



-----------------------------------------------------------
.br
.B MVAPICH2's mpiexec: pbsrun.mvapich2
.br
-----------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.mvapich2 <mpiexec args> <executable> <executable's 
.RS 16
.B args> 
.B [: <mpiexec args> <executable> <executable's args>]
.RE
Multiple executables can be specified using the colon notation.
This is the PBS wrapper script to MVAPICH2's mpiexec, which have 
the same format.

If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH2
ranks and track their resources, so that PBS can perform accounting and 
have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
.B mpiexec
was used.
.RE

OPTIONS HANDLING
.RS 5
If executed inside a PBS job script, all 
.B mpiexec
options given are
passed on to the actual mpiexec call with these exceptions:

.IP "-host <host>" 5
The 
.I host
argument contents are ignored.

.IP "-machinefile <file>" 5
The 
.I file 
argument contents are ignored and replaced by the contents of
the $PBS_NODEFILE.

.RE

WRAP/UNWRAP
.RS 5
To wrap MVAPICH2's mpiexec script:
.RS 4
.B # pbsrun_wrap  <path-to-actual-mpiexec> pbsrun.mvapich2
.RE
To unwrap MVAPICH2's mpiexec script:
.RS 4
.B # pbsrun_unwrap pbsrun.mvapich2
.RE
.RE
.RE




-----------------------------------------------------------
.br
.B IBM's poe: pbsrun.poe
.br
-----------------------------------------------------------

SYNTAX
.RS 5

.B pbsrun.poe <options> <executable> <arg1> 
.B <arg2> ... <argn>

This is the PBS wrapper script to IBM's poe, allowing LAPI or MPI
programs to use InfiniBand or the HPS in US mode.

If executed inside a PBS job, this allows for PBS to track all resources and 
MPI ranks.
PBS can perform accounting and have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard
.B poe
was used.

The script will use the -euilib {ip | us} option and the MP_EUILIB environment
variable to indicate use of US mode, to maintain compatibility with standard poe.
.RE

OPTIONS AND ENVIRONMENT VARIABLES
.br
.RS 5
Users submitting jobs whose programs use poe can set environment
variables instead of using options to poe. The equivalent environment
variable is listed with its poe option. If executed inside a PBS job
script, all
.B pbsrun.poe
options and environment variables except the following are passed on 
to poe:

.IP "-devtype, MP_DEVTYPE" 5
If InfiniBand is not specified in either the option or the environment
variable, the InfiniBand interconnect is not used for the program.

.IP "-euidevice, MP_EUIDEVICE" 5
Ignored by PBS.

.IP "-euilib {ip|us}, MP_EUILIB"
If the command line option
.I -euilib
is set, it will take precedence over the 
.I MP_EUILIB
environment variable.
If set to "us", the program uses User Space mode.
If set to any other value, that value is passed to IBM poe.

.IP "-hostfile, -hfile, MP_HOSTFILE" 5
Ignored. If this is specified, PBS prints the following:
.br
.RS
.IP "" 3
"pbsrun.poe: Warning, -hostfile value replaced by PBS"
.LP
or
.IP "" 3
"pbsrun.poe: Warning -hfile value replaced by PBS"
.LP
.br
If this environment variable is set when a poe job is submitted,
PBS prints the following error message:
.br
.IP "" 3
"pbsrun.poe: Warning MP_HOSTFILE value replaced by PBS"
.LP
.RE

.IP "-instances, MP_INSTANCES" 5
The option and the environment variable are treated differently:
.RS 5
.IP "-instances" 5
If the option is set, PBS prints a warning:
.RS
.IP "" 3
"pbsrun.poe: Warning, -instances cmd line option removed by PBS"
.LP
.RE

.IP "MP_INSTANCES" 5
If the environment variable is set, PBS uses it to calculate
the number of network windows for the job.
The maximum value allowed can be requested by using
the string "max" for the environment variable.
If the environment variable is set to a value greater than
the maximum allowed value, it is replaced with the maximum
allowed value.
The default maximum value is 4.
.RE

.IP "-procs, MP_PROCS" 5
This option or environment variable should be set to the
total number of mpiprocs requested by the job when using
US mode.
.br
If neither this option nor the MP_PROCS environment
variable is set, PBS uses the number of entries in
$PBS_NODEFILE.
.br
If this option is set to N, and the job is submitted with a total
of M mpiprocs:
.RS
.IP "If N >=M:" 3
The value N is passed to IBM poe.
.IP "If N < M and US mode is not being used:" 3
The value N is passed to poe.
.br
.IP "If N < M and US mode is being used:" 3 
US mode is turned off and a warning is printed:
.RS
.IP "" 3
"pbsrun.poe: Warning, user mode disabled due to MP_PROCS setting"
.LP
.RE
.RE
.RE

WRAP/UNWRAP
.RS 5
To wrap IBM's poe:
.RS 4
.B # pbsrun_wrap <path to actual poe> pbsrun.poe
.RE
To unwrap the IBM poe mpirun script:
.RS 4
.B # pbsrun_unwrap pbsrun.poe
.RE
.RE
.RE



.SH REQUIREMENTS
The mpirun being wrapped
must be installed and working on all the nodes in the PBS cluster.

.SH ERRORS
If 
.B pbsrun 
encounters any option not found in 
.I options_to_retain, options_to_ignore, 
and 
.I options_to_transform, 
then it is flagged as an error.

.SH SEE ALSO
The 
.B PBS Professional Administrator's Guide
.LP
pbs_attach(8B), 
pbsrun_wrap(8B), 
pbsrun_unwrap(8B)