.\" Copyright (C) 1994-2018 Altair Engineering, Inc. .\" For more information, contact Altair at www.altair.com. .\" .\" This file is part of the PBS Professional ("PBS Pro") software. .\" .\" Open Source License Information: .\" .\" PBS Pro is free software. You can redistribute it and/or modify it under the .\" terms of the GNU Affero General Public License as published by the Free .\" Software Foundation, either version 3 of the License, or (at your option) any .\" later version. .\" .\" PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY .\" WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS .\" FOR A PARTICULAR PURPOSE. .\" See the GNU Affero General Public License for more details. .\" .\" You should have received a copy of the GNU Affero General Public License .\" along with this program. If not, see . .\" .\" Commercial License Information: .\" .\" For a copy of the commercial license terms and conditions, .\" go to: (http://www.pbspro.com/UserArea/agreement.html) .\" or contact the Altair Legal Department. .\" .\" Altair’s dual-license business model allows companies, individuals, and .\" organizations to create proprietary derivative works of PBS Pro and .\" distribute them - whether embedded or bundled with other software - .\" under a commercial license agreement. .\" .\" Use of Altair’s trademarks, including but not limited to "PBS™", .\" "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's .\" trademark licensing policies. .\" .TH pbsrun 8B "13 October 2017" Local "PBS Professional" .SH NAME .B pbsrun \- general-purpose wrapper script for mpirun .SH SYNOPSIS .B pbsrun .B pbsrun --version .SH DESCRIPTION .B pbsrun is a wrapper script for any of several versions of mpirun. This provides a user-transparent way for PBS to control jobs which call mpirun in their jobscripts. The .B pbsrun_wrap script instantiates .B pbsrun so that the wrapper script for the specific version of mpirun being used has the same name as that version of mpirun. If the mpirun wrapper script is run inside a PBS job, it translates any mpirun call of the form: .br mpirun [] [] .br into .br mpirun [] pbs_attach [] \\ .br [] .br where .I special options to pbs_attach refers to any options needed by .B pbs_attach to do its job (e.g. -j $PBS_JOBID). If the wrapper script is executed outside of PBS, a warning is issued about "not running under PBS", but it proceeds as if the actual program had been called in standalone fashion. The .B pbsrun wrapper script is not meant to be executed directly; instead it is instantiated by .B pbsrun_wrap. It is copied to the target directory and renamed "pbsrun." where .I mpirun version/flavor is a string that identifies the mpirun version being wrapped (e.g. ch_gm). The .B pbsrun script, if executed inside a PBS job, runs an .B initialization script, named $PBS_EXEC/lib/MPI/pbsrun..init, then parses mpirun-like arguments from the command line, sorting which options and option values to retain, to ignore, or to transform, before calling the actual mpirun script with a "pbs_attach" prefixed to the executable. The actual mpirun to call is found by tracing the link pointed to by $PBS_EXEC/lib/MPI/pbsrun..link. For all of the wrapped MPIs, the maximum number of ranks that can be launched is the number of entries in $PBS_NODEFILE. The wrapped MPIs are: .RS 5 MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh ( .B deprecated as of 14.2.1) .br MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh ( .B deprecated as of 14.2.1) .br MPICH-GM's mpirun (mpirun.mpd) with MPD ( .B deprecated as of 14.2.1) .br MPICH-MX's mpirun (mpirun.mpd) with MPD ( .B deprecated as of 14.2.1) .br MPICH2's mpirun .br Intel MPI's mpirun ( .B deprecated as of 13.0) .br MVAPICH1's mpirun ( .B deprecated as of 14.2.1) .br MVAPICH2's mpiexec .br IBM's poe .RE .SH OPTIONS .IP "--version" 8 The .B pbsrun command returns its PBS version information and exits. This option can only be used alone. .SH INITIALIZATION SCRIPT The initialization script, called $PBS_EXEC/lib/MPI/pbsrun..init, where .I mpirun version/flavor reflects the mpirun flavor/version being wrapped, can be modified by an administrator to customize against the local flavor/version of mpirun being wrapped. Inside this sourced init script, 8 variables are set: .RS 5 options_to_retain="-optA -optB -optC val2> ..." .br options_to_ignore="-optD -optE -optF val2> ..." .br options_to_transform="-optG -optH -optI val2> ..." .br options_to_fail="-optY -optZ ..." .br options_to_configfile="-optX ..." .br options_with_another_form="-optW ..." .br pbs_attach=pbs_attach .br options_to_pbs_attach="-J $PBS_JOBID" .br .RE .B Initialization Script Options .br .IP "options_to_retain" 5 Space-separated list of options and values that pbsrun. passes on to the actual mpirun call. Options must begin with "-" or "--", and option arguments must be specified by some arbitrary name with left and right arrows, as in "". .IP "options_to_ignore" 5 Space-separated list of options and values that pbsrun. does not pass on to the actual mpirun call. Options must begin with "-" or "--", and option arguments must be specified by arbitrary names with left and right arrows, as in "". .IP "options_to_transform" 5 Space-separated list of options and values that .B pbsrun modifies before passing on to the actual mpirun call. .IP "option_to_fail" 5 Space-separated list of options that will cause .B pbsrun to exit upon encountering a match. .IP "options_to_configfile" 5 Single option and value that refers to the name of the "configfile" containing command line segments found in certain versions of mpirun. .IP "options_with_another_form" 5 Space-separated list of options and values that can be found in options_to_retain, options_to_ignore, or options_to_transform, whose syntax has an alternate, unsupported form. .IP "pbs_attach" 5 Path to .B pbs_attach, which is called before the argument of mpirun. .IP "options_to_pbs_attach" 5 Special options to pass to the .B pbs_attach call. You may pass variable references (e.g. $PBS_JOBID) and they are substituted by .B pbsrun to actual values. .LP .RE If .B pbsrun encounters any option not found in .I options_to_retain, options_to_ignore, and .I options_to_transform, then it is flagged as an error. .B Functions Created .br These functions are created inside the init script. These can be modified by the PBS administrator. .RS 5 .IP "transform_action () {" 5 # passed actual values of $options_to_transform .br args=$* .br } .IP "boot_action () {" 5 mpirun_location=$1 .br } .IP "evaluate_options_action () {" 5 # passed actual values of transformed options .br args=$* .br } .IP "configfile_cmdline_action () {" 5 args=$* .br } .IP "end_action () {" 5 mpirun_location=$1 .br } .LP .IP "transform_action()" 5 The pbsrun. wrapper script invokes the function .I transform_action() (called once on each matched item and value) with actual options and values received matching one of the "options_to_transform". The function returns a string to pass on to the actual mpirun call. .IP "boot_action()" 5 Performs any initialization tasks needed before running the actual mpirun call. For instance, GM's MPD requires the MPD daemons to be user-started first. This function is called by the pbsrun. script with the location of actual mpirun passed as the first argument. Also, the pbsrun. checks for the exit value of this function to determine whether or not to progress to the next step. .IP "evaluate_options_action()" 5 Called with the actual options and values that resulted after consulting .I options_to_retain, options_to_ignore, options_to_transform, and executing .I transform_action(). This provides one more chance for the script writer to evaluate all the options and values in general, and make any necessary adjustments, before passing them on to the actual mpirun call. For instance, this function can specify what the default value is for a missing .I -np option. .IP "configfile_cmdline_action()" 5 Returns the actual options and values to be put in before the option_to_configfile parameter. .IP "configfile_firstline_action()" 5 Returns the item that is put in the first line of the configuration file specified in the .I option_to_configfile parameter. .IP "end_action()" 5 Called by pbsrun. at the end of execution. It undoes any action done by transform_action(), like cleanup of temporary files. It is also called when pbsrun. is prematurely killed. This function is called with the location of actual mpirun passed as first argument. .RE The actual mpirun program to call is the path pointed to by $PBS_EXEC/lib/MPI/pbsrun..link. .B Modifying *.init scripts .br In order for administrators to modify *.init scripts without breaking package verification in RPM, master copies of the initialization scripts are named *.init.in. .B pbsrun_wrap instantiates the *.init.in files as *.init. For instance, $PBS_EXEC/lib/MPI/pbsrun.mpich2.init.in is the master copy, and .B pbsrun_wrap instantiates it as $PBS_EXEC/lib/MPI/pbsrun.mpich2.init. .B pbsrun_unwrap takes care of removing the *.init files. .SH MPIRUN VERSIONS/FLAVORS ----------------------------------------------------------- .br .B MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh: pbsrun.ch_gm .br ----------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.ch_gm .B ... .B Deprecated. This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-GM processes started by rsh/ssh so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard .B mpirun.ch_gm was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all .B mpirun.ch_gm options given are passed on to the actual mpirun call with these exceptions: .IP "-machinefile " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .IP "-np" 5 If not specified, the number of entries found in the $PBS_NODEFILE is used. .IP "-pg" 5 The use of the .I -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. .RE WRAP/UNWRAP .RS 5 To wrap MPICH-GM's mpirun script: .RS 4 .B # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.ch_gm pbsrun.ch_gm .RE To unwrap MPICH-GM's mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.ch_gm .RE .RE .RE ----------------------------------------------------------- .br .B MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh: pbsrun.ch_mx .br ----------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.ch_mx .B ... .B Deprecated. This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-MX processes started by rsh/ssh so that PBS can perform accounting and has complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_mx was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all mpirun.ch_gm options given are passed on to the actual mpirun call with some exceptions: .IP "-machinefile " 5 The .I file argument contents is ignored and replaced by the contents of the $PBS_NODEFILE. .IP "-np" 5 If not specified, the number of entries found in the $PBS_NODEFILE is used. .IP "-pg" 5 The use of the .I -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. .RE WRAP/UNWRAP .RS 5 To wrap MPICH-MX's mpirun script: .RS 4 .B # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.ch_mx pbsrun.ch_mx .RE To unwrap MPICH-MX's mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.ch_mx .RE .RE .RE -------------------------------------------------------- .br .B MPICH-GM's mpirun (mpirun.mpd) with MPD: pbsrun.gm_mpd .br -------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.gm_mpd .B ... .B Deprecated. This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.mpd) with MPD process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-GM processes started by the MPD daemons so that PBS can perform accounting have and complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_gm with MPD was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all mpirun.ch_gm with MPD options given are passed on to the actual mpirun call with these exceptions: .IP "-m " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .IP "-np" 5 If not specified, the number of entries found in the $PBS_NODEFILE is used. .IP "-pg" 5 The use of the .I -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. .RE STARTUP/SHUTDOWN .RS 5 The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either rsh or ssh method based on value of environment variable RSHCOMMAND. The default is rsh. The script also takes care of shutting down the MPD daemons at the end of a run. .RE WRAP/UNWRAP .RS 5 To wrap MPICH-GM's mpirun script with MPD: .RS 4 .B # pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.mpd pbsrun.gm_mpd .RE To unwrap MPICH-GM's mpirun script with MPD: .RS 4 .B # pbsrun_unwrap pbsrun.gm_mpd .RE .RE .RE -------------------------------------------------------- .br .B MPICH-MX's mpirun (mpirun.mpd) with MPD: pbsrun.mx_mpd .br -------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.mx_mpd .B ... .B Deprecated. This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.mpd) with MPD process startup method. If executed inside a PBS job, this allows for PBS to track all MPICH-MX processes started by the MPD daemons so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_mx with MPD was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all mpirun.ch_gm with MPD options given are passed on to the actual mpirun call with these exceptions: .IP "-m " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .IP "-np" 5 If not specified, the number of entries found in the $PBS_NODEFILE is used. .IP "-pg" 5 The use of the .I -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. .RE STARTUP/SHUTDOWN .RS 5 The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either rsh or ssh method, based on value of environment variable RSHCOMMAND -- rsh is the default. The script also takes care of shutting down the MPD daemons at the end of a run. .RE WRAP/UNWRAP .RS 5 To wrap MPICH-MX's mpirun script with MPD: .RS 4 .B # pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.mpd pbsrun.mx_mpd .RE To unwrap MPICH-MX's mpirun script with MPD: .RS 4 .B # pbsrun_unwrap pbsrun.mx_mpd .RE .RE .RE ------------------------------ .br .B MPICH2's mpirun: pbsrun.mpich2 .br ------------------------------ SYNTAX .RS 5 .B pbsrun.mpich2 [] [] .B .RS 14 .B [] .B [: [] []] .RE .br - or - .br .B pbsrun.mpich2 -configfile where contains command line segments as lines: .RS 5 [local args] executable1 [args] .br [local args] executable2 [args] .br [local args] executable3 [args] .RE This is the PBS wrapper script to MPICH2's mpirun. If executed inside a PBS job, this allows for PBS to track all MPICH2 processes so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard MPICH2's mpirun was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all MPICH2's mpirun options given are passed on to the actual mpirun call with these exceptions: .IP "-host and -ghost" 5 For specifying the execution host to run on. Not passed on to the actual mpirun call. .IP "-machinefile " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .IP "MPICH2's mpirun -localonly " 5 For specifying the number of processes to run locally. Not supported. The user is advised instead to use the equivalent arguments: .I "-np -localonly". The reason for this is that the .B pbsrun wrapper script cannot handle a variable number of arguments to an option (e.g. "-localonly" has 1 argument and "-localonly " has 2 arguments). .IP "-np" 5 If user did not specify a .I -np option, then no default value is provided by the PBS wrapper scripts. It is up to the local mpirun to decide what the reasonable default value should be, which is usually 1. .RE STARTUP/SHUTDOWN .RS 5 The script takes care of ensuring that the MPD daemons on each of the hosts listed in the $PBS_NODEFILE are started. It also takes care of ensuring that the MPD daemons have been shut down at the end of MPI job execution. .RE WRAP/UNWRAP .RS 5 To wrap MPICH2's mpirun script: .RS 4 .B # pbsrun_wrap [MPICH2_BIN_PATH]/mpirun pbsrun.mpich2 .RE To unwrap MPICH2's mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.mpich2 .RE .RE .RE ----------------------------------- .br .B Intel MPI's mpirun: pbsrun.intelmpi .br ----------------------------------- Wrapping Intel MPI, and support for mpdboot, are .B deprecated. SYNTAX .RS 5 .B pbsrun.intelmpi [] [] .RS 16 .B [] .br .B [: [] []] .RE .br - or - .br .B pbsrun.intelmpi [] -f where .I mpdboot options are any options to pass to the mpdboot program, which is automatically called by Intel MPI's mpirun to start MPDs, and .I configfile contains command line segments as lines. This is the PBS wrapper script to Intel MPI's mpirun. If executed inside a PBS job, this allows for PBS to track all Intel MPI processes so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard Intel MPI's mpirun was used. .RE Using .B pbsrun, pbsrun_wrap, and .B pbsrun_unwrap with Intel MPI is .B deprecated as of 13.0. OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all of the options to the PBS interface to MPI's mpirun are passed to the actual mpirun call with these exceptions: .IP "-host and -ghost" 5 For specifying the execution host to run on. Not passed on to the actual mpirun call. .IP "-machinefile " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .IP "mpdboot options --totalnum=* and --file=*" 5 Ignored and replaced by the number of unique entries in $PBS_NODEFILE and name of $PBS_NODEFILE respectively. .IP "arguments to mpdboot options --file=* and -f " 5 Replaced by $PBS_NODEFILE. .IP "-s" 5 If .B pbsrun.intelmpi is called inside a PBS job, Intel MPIs mpirun .I -s argument to mpdboot are not supported as this closely matches the mpirun option .I "-s ". The user can simply run a separate mpdboot .I -s before calling mpirun. A warning message is issued by .B pbsrun.intelmpi upon encountering a .I -s option telling users of the supported form. .IP "-np" 5 If the user does not specify a .I -np option, then no default value is provided by the PBS wrap scripts. It is up to the local mpirun to decide what the reasonable default value should be, which is usually 1. .RE STARTUP/SHUTDOWN .RS 5 Intel MPI's mpirun itself takes care of starting/stopping the MPD daemons. .B pbsrun.intelmpi always passes the arguments .I -totalnum= and .I -file= to the actual mpirun, taking its input from unique entries in $PBS_NODEFILE. .RE WRAP/UNWRAP .RS 5 To wrap Intel MPI's mpirun script: .RS 4 .B # pbsrun_wrap [INTEL_MPI_BIN_PATH]/mpirun pbsrun.intelmpi .RE To unwrap Intel MPI's mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.intelmpi .RE .RE .RE ----------------------------------------------------------- .br .B MVAPICH1's mpirun: pbsrun.mvapich1 .br ----------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.mvapich1 .B Deprecated. Only one executable can be specified. This is the PBS wrapper script to MVAPICH1's mpirun. If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH1 ranks and track their resources, so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard .B mpirun was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all .B mpirun options given are passed on to the actual mpirun call with these exceptions: .IP "-map " 5 The .I map option is ignored. .IP "-exclude " 5 The .I exclude option is ignored. .IP "-machinefile " 5 The .I machinefile option is ignored. .IP "-np" 5 If not specified, the number of entries found in the $PBS_NODEFILE is used. .RE WRAP/UNWRAP .RS 5 To wrap MVAPICH1's mpirun script: .RS 4 .B # pbsrun_wrap pbsrun.mvapich1 .RE To unwrap MVAPICH1's mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.mvapich1 .RE .RE .RE ----------------------------------------------------------- .br .B MVAPICH2's mpiexec: pbsrun.mvapich2 .br ----------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.mvapich2 .B [: ] .RE Multiple executables can be specified using the colon notation. This is the PBS wrapper script to MVAPICH2's mpiexec, which have the same format. If executed inside a PBS job, this allows for PBS to be aware of all MVAPICH2 ranks and track their resources, so that PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard .B mpiexec was used. .RE OPTIONS HANDLING .RS 5 If executed inside a PBS job script, all .B mpiexec options given are passed on to the actual mpiexec call with these exceptions: .IP "-host " 5 The .I host argument contents are ignored. .IP "-machinefile " 5 The .I file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE. .RE WRAP/UNWRAP .RS 5 To wrap MVAPICH2's mpiexec script: .RS 4 .B # pbsrun_wrap pbsrun.mvapich2 .RE To unwrap MVAPICH2's mpiexec script: .RS 4 .B # pbsrun_unwrap pbsrun.mvapich2 .RE .RE .RE ----------------------------------------------------------- .br .B IBM's poe: pbsrun.poe .br ----------------------------------------------------------- SYNTAX .RS 5 .B pbsrun.poe .B ... This is the PBS wrapper script to IBM's poe, allowing LAPI or MPI programs to use InfiniBand or the HPS in US mode. If executed inside a PBS job, this allows for PBS to track all resources and MPI ranks. PBS can perform accounting and have complete job control. If executed outside of a PBS job, it behaves exactly as if standard .B poe was used. The script will use the -euilib {ip | us} option and the MP_EUILIB environment variable to indicate use of US mode, to maintain compatibility with standard poe. .RE OPTIONS AND ENVIRONMENT VARIABLES .br .RS 5 Users submitting jobs whose programs use poe can set environment variables instead of using options to poe. The equivalent environment variable is listed with its poe option. If executed inside a PBS job script, all .B pbsrun.poe options and environment variables except the following are passed on to poe: .IP "-devtype, MP_DEVTYPE" 5 If InfiniBand is not specified in either the option or the environment variable, the InfiniBand interconnect is not used for the program. .IP "-euidevice, MP_EUIDEVICE" 5 Ignored by PBS. .IP "-euilib {ip|us}, MP_EUILIB" If the command line option .I -euilib is set, it will take precedence over the .I MP_EUILIB environment variable. If set to "us", the program uses User Space mode. If set to any other value, that value is passed to IBM poe. .IP "-hostfile, -hfile, MP_HOSTFILE" 5 Ignored. If this is specified, PBS prints the following: .br .RS .IP "" 3 "pbsrun.poe: Warning, -hostfile value replaced by PBS" .LP or .IP "" 3 "pbsrun.poe: Warning -hfile value replaced by PBS" .LP .br If this environment variable is set when a poe job is submitted, PBS prints the following error message: .br .IP "" 3 "pbsrun.poe: Warning MP_HOSTFILE value replaced by PBS" .LP .RE .IP "-instances, MP_INSTANCES" 5 The option and the environment variable are treated differently: .RS 5 .IP "-instances" 5 If the option is set, PBS prints a warning: .RS .IP "" 3 "pbsrun.poe: Warning, -instances cmd line option removed by PBS" .LP .RE .IP "MP_INSTANCES" 5 If the environment variable is set, PBS uses it to calculate the number of network windows for the job. The maximum value allowed can be requested by using the string "max" for the environment variable. If the environment variable is set to a value greater than the maximum allowed value, it is replaced with the maximum allowed value. The default maximum value is 4. .RE .IP "-procs, MP_PROCS" 5 This option or environment variable should be set to the total number of mpiprocs requested by the job when using US mode. .br If neither this option nor the MP_PROCS environment variable is set, PBS uses the number of entries in $PBS_NODEFILE. .br If this option is set to N, and the job is submitted with a total of M mpiprocs: .RS .IP "If N >=M:" 3 The value N is passed to IBM poe. .IP "If N < M and US mode is not being used:" 3 The value N is passed to poe. .br .IP "If N < M and US mode is being used:" 3 US mode is turned off and a warning is printed: .RS .IP "" 3 "pbsrun.poe: Warning, user mode disabled due to MP_PROCS setting" .LP .RE .RE .RE WRAP/UNWRAP .RS 5 To wrap IBM's poe: .RS 4 .B # pbsrun_wrap pbsrun.poe .RE To unwrap the IBM poe mpirun script: .RS 4 .B # pbsrun_unwrap pbsrun.poe .RE .RE .RE .SH REQUIREMENTS The mpirun being wrapped must be installed and working on all the nodes in the PBS cluster. .SH ERRORS If .B pbsrun encounters any option not found in .I options_to_retain, options_to_ignore, and .I options_to_transform, then it is flagged as an error. .SH SEE ALSO The .B PBS Professional Administrator's Guide .LP pbs_attach(8B), pbsrun_wrap(8B), pbsrun_unwrap(8B)