pbsrun(8B)

NAME
pbsrun - general-purpose wrapper script for mpirun

SYNOPSIS
pbsrun pbsrun --version

DESCRIPTION
pbsrun is a wrapper script for any of several versions of mpirun. This provides a user-transparent way for PBS to control jobs which call mpirun in their jobscripts. The pbsrun_wrap script instantiates pbsrun so that the wrapper script for the specific version of mpirun being used has the same name as that version of mpirun. If the mpirun wrapper script is run inside a PBS job, it translates any mpirun call of the form:
    mpirun [<options>] <executable> [<args>]
into
    mpirun [<options>] pbs_attach [<special options to pbs_attach>] \
            <executable> [<args>]
where special options to pbs_attach refers to any options needed by pbs_attach to do its job (e.g. -j $PBS_JOBID). If the wrapper script is executed outside of PBS, a warning is issued about "not running under PBS", but it proceeds as if the actual program had been called in standalone fashion. The pbsrun wrapper script is not meant to be executed directly; instead it is instantiated by pbsrun_wrap. It is copied to the target directory and renamed "pbsrun.<mpirun version/flavor>" where mpirun version/flavor is a string that identifies the mpirun version being wrapped (e.g. ch_gm). The pbsrun script, if executed inside a PBS job, runs an initialization script, named $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init, then parses mpirun-like arguments from the command line, sorting which options and option values to retain, to ignore, or to transform, before calling the actual mpirun script with a "pbs_attach" prefixed to the executable. The actual mpirun to call is found by tracing the link pointed to by $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.link. For all of the wrapped MPIs, the maximum number of ranks that can be launched is the number of entries in $PBS_NODEFILE. The wrapped MPIs are:
    MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh ( deprecated as of 14.2.1)
    MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh ( deprecated as of 14.2.1)
    MPICH-GM's mpirun (mpirun.mpd) with MPD ( deprecated as of 14.2.1)
    MPICH-MX's mpirun (mpirun.mpd) with MPD ( deprecated as of 14.2.1)
    MPICH2's mpirun
    Intel MPI's mpirun ( deprecated as of 13.0)
    MVAPICH1's mpirun ( deprecated as of 14.2.1)
    MVAPICH2's mpiexec
    IBM's poe

OPTIONS

--version
    The pbsrun command returns its PBS version information and exits. This option can only be used alone.

INITIALIZATION SCRIPT
The initialization script, called $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init, where mpirun version/flavor reflects the mpirun flavor/version being wrapped, can be modified by an administrator to customize against the local flavor/version of mpirun being wrapped. Inside this sourced init script, 8 variables are set:
    options_to_retain="-optA -optB <val> -optC <val1> val2> ..."
    options_to_ignore="-optD -optE <n> -optF <val1> val2> ..."
    options_to_transform="-optG -optH <val> -optI <val1> val2> ..."
    options_to_fail="-optY -optZ ..."
    options_to_configfile="-optX <val> ..."
    options_with_another_form="-optW <val> ..."
    pbs_attach=pbs_attach
    options_to_pbs_attach="-J $PBS_JOBID"
Initialization Script Options

options_to_retain
    Space-separated list of options and values that pbsrun.<mpirun version/flavor> passes on to the actual mpirun call. Options must begin with "-" or "--", and option arguments must be specified by some arbitrary name with left and right arrows, as in "<val1>".

options_to_ignore
    Space-separated list of options and values that pbsrun.<mpirun version/flavor> does not pass on to the actual mpirun call. Options must begin with "-" or "--", and option arguments must be specified by arbitrary names with left and right arrows, as in "<n>".

options_to_transform
    Space-separated list of options and values that pbsrun modifies before passing on to the actual mpirun call.

option_to_fail
    Space-separated list of options that will cause pbsrun to exit upon encountering a match.

options_to_configfile
    Single option and value that refers to the name of the "configfile" containing command line segments found in certain versions of mpirun.

options_with_another_form
    Space-separated list of options and values that can be found in options_to_retain, options_to_ignore, or options_to_transform, whose syntax has an alternate, unsupported form.

pbs_attach
    Path to pbs_attach, which is called before the <executable> argument of mpirun.

options_to_pbs_attach
    Special options to pass to the pbs_attach call. You may pass variable references (e.g. $PBS_JOBID) and they are substituted by pbsrun to actual values.

If pbsrun encounters any option not found in options_to_retain, options_to_ignore, and options_to_transform, then it is flagged as an error. Functions Created
These functions are created inside the init script. These can be modified by the PBS administrator.

    transform_action () {
             # passed actual values of $options_to_transform
             args=$*
        }

    boot_action () {
             mpirun_location=$1
        }

    evaluate_options_action () {
             # passed actual values of transformed options
             args=$*
        }

    configfile_cmdline_action () {
             args=$*
        }

    end_action () {
             mpirun_location=$1
        }

    transform_action()
        The pbsrun.<mpirun version/flavor> wrapper script invokes the function transform_action() (called once on each matched item and value) with actual options and values received matching one of the "options_to_transform". The function returns a string to pass on to the actual mpirun call.

    boot_action()
        Performs any initialization tasks needed before running the actual mpirun call. For instance, GM's MPD requires the MPD daemons to be user-started first. This function is called by the pbsrun.<mpirun version/flavor> script with the location of actual mpirun passed as the first argument. Also, the pbsrun.<mpirun version/flavor> checks for the exit value of this function to determine whether or not to progress to the next step.

    evaluate_options_action()
        Called with the actual options and values that resulted after consulting options_to_retain, options_to_ignore, options_to_transform, and executing transform_action(). This provides one more chance for the script writer to evaluate all the options and values in general, and make any necessary adjustments, before passing them on to the actual mpirun call. For instance, this function can specify what the default value is for a missing -np option.

    configfile_cmdline_action()
        Returns the actual options and values to be put in before the option_to_configfile parameter.

    configfile_firstline_action()
        Returns the item that is put in the first line of the configuration file specified in the option_to_configfile parameter.

    end_action()
        Called by pbsrun.<mpirun version/flavor> at the end of execution. It undoes any action done by transform_action(), like cleanup of temporary files. It is also called when pbsrun.<mpirun version/flavor> is prematurely killed. This function is called with the location of actual mpirun passed as first argument.
The actual mpirun program to call is the path pointed to by
    $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.link. Modifying *.init scripts
In order for administrators to modify *.init scripts without breaking package verification in RPM, master copies of the initialization scripts are named *.init.in. pbsrun_wrap instantiates the *.init.in files as *.init. For instance, $PBS_EXEC/lib/MPI/pbsrun.mpich2.init.in is the master copy, and pbsrun_wrap instantiates it as $PBS_EXEC/lib/MPI/pbsrun.mpich2.init. pbsrun_unwrap takes care of removing the *.init files.

MPIRUN VERSIONS/FLAVORS
-----------------------------------------------------------
MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh: pbsrun.ch_gm
----------------------------------------------------------- SYNTAX
    pbsrun.ch_gm <options> <executable> <arg1> <arg2> ... <argn> Deprecated. This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-GM processes started by rsh/ssh so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_gm was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpirun.ch_gm options given are passed on to the actual mpirun call with these exceptions:

    -machinefile <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.

    -np If not specified, the number of entries found in the $PBS_NODEFILE is used.

    -pg The use of the -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS.
WRAP/UNWRAP
    To wrap MPICH-GM's mpirun script:
        # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.ch_gm pbsrun.ch_gm
    To unwrap MPICH-GM's mpirun script:
        # pbsrun_unwrap pbsrun.ch_gm
-----------------------------------------------------------
MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh: pbsrun.ch_mx
----------------------------------------------------------- SYNTAX
    pbsrun.ch_mx <options> <executable> <arg1> <arg2> ... <argn> Deprecated. This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-MX processes started by rsh/ssh so that PBS can perform accounting and has complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_mx was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpirun.ch_gm options given are passed on to the actual mpirun call with some exceptions:

    -machinefile <file>
        The file argument contents is ignored and replaced by the contents of the $PBS_NODEFILE.

    -np If not specified, the number of entries found in the $PBS_NODEFILE is used.

    -pg The use of the -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS.
WRAP/UNWRAP
    To wrap MPICH-MX's mpirun script:
        # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.ch_mx pbsrun.ch_mx
    To unwrap MPICH-MX's mpirun script:
        # pbsrun_unwrap pbsrun.ch_mx
--------------------------------------------------------
MPICH-GM's mpirun (mpirun.mpd) with MPD: pbsrun.gm_mpd
-------------------------------------------------------- SYNTAX
    pbsrun.gm_mpd <options> <executable> <arg1> <arg2> ... <argn> Deprecated. This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.mpd) with MPD process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-GM processes started by the MPD daemons so that PBS can perform accounting have and complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_gm with MPD was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpirun.ch_gm with MPD options given are passed on to the actual mpirun call with these exceptions:

    -m <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.

    -np If not specified, the number of entries found in the $PBS_NODEFILE is used.

    -pg The use of the -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS.
STARTUP/SHUTDOWN
    The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either rsh or ssh method based on value of environment variable RSHCOMMAND. The default is rsh. The script also takes care of shutting down the MPD daemons at the end of a run.
WRAP/UNWRAP
    To wrap MPICH-GM's mpirun script with MPD:
        # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.mpd pbsrun.gm_mpd
    To unwrap MPICH-GM's mpirun script with MPD:
        # pbsrun_unwrap pbsrun.gm_mpd
--------------------------------------------------------
MPICH-MX's mpirun (mpirun.mpd) with MPD: pbsrun.mx_mpd
-------------------------------------------------------- SYNTAX
    pbsrun.mx_mpd <options> <executable> <arg1> <arg2> ... <argn> Deprecated. This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.mpd) with MPD process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-MX processes started by the MPD daemons so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_mx with MPD was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpirun.ch_gm with MPD options given are passed on to the actual mpirun call with these exceptions:

    -m <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.

    -np If not specified, the number of entries found in the $PBS_NODEFILE is used.

    -pg The use of the -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS.
STARTUP/SHUTDOWN
    The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either rsh or ssh method, based on value of environment variable RSHCOMMAND -- rsh is the default. The script also takes care of shutting down the MPD daemons at the end of a run.
WRAP/UNWRAP
    To wrap MPICH-MX's mpirun script with MPD:
        # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.mpd pbsrun.mx_mpd
    To unwrap MPICH-MX's mpirun script with MPD:
        # pbsrun_unwrap pbsrun.mx_mpd
------------------------------
MPICH2's mpirun: pbsrun.mpich2
------------------------------ SYNTAX
    pbsrun.mpich2 [<global args>] [<local args>] <executable>
        [<args>] [: [<local args>] <executable> [<args>]]
    - or -
    pbsrun.mpich2 -configfile <configfile> where <configfile> contains command line segments as lines:
        [local args] executable1 [args]
        [local args] executable2 [args]
        [local args] executable3 [args]
    This is the PBS wrapper script to MPICH2's mpirun. If executed inside a PBS job, this allows for PBS to track all MPICH2 processes so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard MPICH2's mpirun was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all MPICH2's mpirun options given are passed on to the actual mpirun call with these exceptions:

    -host and -ghost
        For specifying the execution host to run on. Not passed on to the actual mpirun call.

    -machinefile <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.

    MPICH2's mpirun -localonly <x>
        For specifying the <x> number of processes to run locally. Not supported. The user is advised instead to use the equivalent arguments: -np <x> -localonly . The reason for this is that the pbsrun wrapper script cannot handle a variable number of arguments to an option (e.g. "-localonly" has 1 argument and "-localonly <x>" has 2 arguments).

    -np If user did not specify a -np option, then no default value is provided by the PBS wrapper scripts. It is up to the local mpirun to decide what the reasonable default value should be, which is usually 1.
STARTUP/SHUTDOWN
    The script takes care of ensuring that the MPD daemons on each of the hosts listed in the $PBS_NODEFILE are started. It also takes care of ensuring that the MPD daemons have been shut down at the end of MPI job execution.
WRAP/UNWRAP
    To wrap MPICH2's mpirun script:
        # pbsrun_wrap [MPICH2_BIN_PATH]/mpirun pbsrun.mpich2
    To unwrap MPICH2's mpirun script:
        # pbsrun_unwrap pbsrun.mpich2
-----------------------------------
Intel MPI's mpirun: pbsrun.intelmpi
----------------------------------- Wrapping Intel MPI, and support for mpdboot, are deprecated. SYNTAX
    pbsrun.intelmpi [<mpdboot options>] [<mpiexec options>]
        <executable> [<prog args>]
        [: [<mpiexec options>] <executable> [<prog args>]]
    - or -
    pbsrun.intelmpi [<mpdboot options>] -f <configfile> where mpdboot options are any options to pass to the mpdboot program, which is automatically called by Intel MPI's mpirun to start MPDs, and configfile contains command line segments as lines. This is the PBS wrapper script to Intel MPI's mpirun. If executed inside a PBS job, this allows for PBS to track all Intel MPI processes so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard Intel MPI's mpirun was used.
Using pbsrun, pbsrun_wrap, and pbsrun_unwrap with Intel MPI is deprecated as of 13.0. OPTIONS HANDLING
    If executed inside a PBS job script, all of the options to the PBS interface to MPI's mpirun are passed to the actual mpirun call with these exceptions:

    -host and -ghost
        For specifying the execution host to run on. Not passed on to the actual mpirun call.

    -machinefile <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.

    mpdboot options --totalnum=* and --file=*
        Ignored and replaced by the number of unique entries in $PBS_NODEFILE and name of $PBS_NODEFILE respectively.

    arguments to mpdboot options --file=* and -f <mpd_hosts_file>
        Replaced by $PBS_NODEFILE.

    -s  If pbsrun.intelmpi is called inside a PBS job, Intel MPIs mpirun -s argument to mpdboot are not supported as this closely matches the mpirun option -s <spec> . The user can simply run a separate mpdboot -s before calling mpirun. A warning message is issued by pbsrun.intelmpi upon encountering a -s option telling users of the supported form.

    -np If the user does not specify a -np option, then no default value is provided by the PBS wrap scripts. It is up to the local mpirun to decide what the reasonable default value should be, which is usually 1.
STARTUP/SHUTDOWN
    Intel MPI's mpirun itself takes care of starting/stopping the MPD daemons. pbsrun.intelmpi always passes the arguments -totalnum=<number of mpds to start> and -file=<mpd_hosts_file> to the actual mpirun, taking its input from unique entries in $PBS_NODEFILE.
WRAP/UNWRAP
    To wrap Intel MPI's mpirun script:
        # pbsrun_wrap [INTEL_MPI_BIN_PATH]/mpirun pbsrun.intelmpi
    To unwrap Intel MPI's mpirun script:
        # pbsrun_unwrap pbsrun.intelmpi
-----------------------------------------------------------
MVAPICH1's mpirun: pbsrun.mvapich1
----------------------------------------------------------- SYNTAX
    pbsrun.mvapich1 <mpirun options> <executable> <options> Deprecated. Only one executable can be specified. This is the PBS wrapper script to MVAPICH1's mpirun. If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH1 ranks and track their resources, so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpirun options given are passed on to the actual mpirun call with these exceptions:

    -map <list>
        The map option is ignored.

    -exclude <list>
        The exclude option is ignored.

    -machinefile <file>
        The machinefile option is ignored.

    -np If not specified, the number of entries found in the $PBS_NODEFILE is used.
WRAP/UNWRAP
    To wrap MVAPICH1's mpirun script:
        # pbsrun_wrap <path-to-actual-mpirun> pbsrun.mvapich1
    To unwrap MVAPICH1's mpirun script:
        # pbsrun_unwrap pbsrun.mvapich1
-----------------------------------------------------------
MVAPICH2's mpiexec: pbsrun.mvapich2
----------------------------------------------------------- SYNTAX
    pbsrun.mvapich2 <mpiexec args> <executable> <executable's
        args> [: <mpiexec args> <executable> <executable's args>]
    Multiple executables can be specified using the colon notation. This is the PBS wrapper script to MVAPICH2's mpiexec, which have the same format. If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH2 ranks and track their resources, so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpiexec was used.
OPTIONS HANDLING
    If executed inside a PBS job script, all mpiexec options given are passed on to the actual mpiexec call with these exceptions:

    -host <host>
        The host argument contents are ignored.

    -machinefile <file>
        The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.
WRAP/UNWRAP
    To wrap MVAPICH2's mpiexec script:
        # pbsrun_wrap <path-to-actual-mpiexec> pbsrun.mvapich2
    To unwrap MVAPICH2's mpiexec script:
        # pbsrun_unwrap pbsrun.mvapich2
-----------------------------------------------------------
IBM's poe: pbsrun.poe
----------------------------------------------------------- SYNTAX
    pbsrun.poe <options> <executable> <arg1> <arg2> ... <argn> This is the PBS wrapper script to IBM's poe, allowing LAPI or MPI programs to use InfiniBand or the HPS in US mode. If executed inside a PBS job, this allows for PBS to track all resources and MPI ranks. PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard poe was used. The script will use the -euilib {ip | us} option and the MP_EUILIB environment variable to indicate use of US mode, to maintain compatibility with standard poe.
OPTIONS AND ENVIRONMENT VARIABLES
    Users submitting jobs whose programs use poe can set environment variables instead of using options to poe. The equivalent environment variable is listed with its poe option. If executed inside a PBS job script, all pbsrun.poe options and environment variables except the following are passed on to poe:

    -devtype, MP_DEVTYPE
        If InfiniBand is not specified in either the option or the environment variable, the InfiniBand interconnect is not used for the program.

    -euidevice, MP_EUIDEVICE
        Ignored by PBS.

    -euilib {ip|us}, MP_EUILIB
        If the command line option -euilib is set, it will take precedence over the MP_EUILIB environment variable. If set to "us", the program uses User Space mode. If set to any other value, that value is passed to IBM poe.

    -hostfile, -hfile, MP_HOSTFILE
        Ignored. If this is specified, PBS prints the following:

            "pbsrun.poe: Warning, -hostfile value replaced by PBS"

        or

            "pbsrun.poe: Warning -hfile value replaced by PBS"

        If this environment variable is set when a poe job is submitted, PBS prints the following error message:

            "pbsrun.poe: Warning MP_HOSTFILE value replaced by PBS"

    -instances, MP_INSTANCES
        The option and the environment variable are treated differently:

        -instances
            If the option is set, PBS prints a warning:

                "pbsrun.poe: Warning, -instances cmd line option removed by PBS"

        MP_INSTANCES
            If the environment variable is set, PBS uses it to calculate the number of network windows for the job. The maximum value allowed can be requested by using the string "max" for the environment variable. If the environment variable is set to a value greater than the maximum allowed value, it is replaced with the maximum allowed value. The default maximum value is 4.

    -procs, MP_PROCS
        This option or environment variable should be set to the total number of mpiprocs requested by the job when using US mode.
        If neither this option nor the MP_PROCS environment variable is set, PBS uses the number of entries in $PBS_NODEFILE.
        If this option is set to N, and the job is submitted with a total of M mpiprocs:

        If N >=M:
            The value N is passed to IBM poe.

        If N < M and US mode is not being used:
            The value N is passed to poe.

        If N < M and US mode is being used:
            US mode is turned off and a warning is printed:

                "pbsrun.poe: Warning, user mode disabled due to MP_PROCS setting"

WRAP/UNWRAP
    To wrap IBM's poe:
        # pbsrun_wrap <path to actual poe> pbsrun.poe
    To unwrap the IBM poe mpirun script:
        # pbsrun_unwrap pbsrun.poe

REQUIREMENTS
The mpirun being wrapped must be installed and working on all the nodes in the PBS cluster.

ERRORS
If pbsrun encounters any option not found in options_to_retain, options_to_ignore, and options_to_transform, then it is flagged as an error.

SEE ALSO
The PBS Professional Administrator's Guide

pbs_attach(8B), pbsrun_wrap(8B), pbsrun_unwrap(8B)