pbs_mom(8B)

NAME
pbs_mom - run the PBS job monitoring and execution daemon

SYNOPSIS
pbs_mom [-a <alarm timeout>] [-C <checkpoint directory>]
    [-c <config file>] [-d <MoM home directory>] [-L <logfile>]
    [-M <MoM service port>] [-N] [-n <nice value>] [-p|-r]
    [-R <inter-MoM communication port>] [-S <server port>]
    [-s script_options]
pbs_mom --version

DESCRIPTION
The pbs_mom command starts the PBS job monitoring and execution daemon, called MoM. The standard MoM starts jobs on the execution host, monitors and reports resource usage, enforces resource usage limits, and notifies the server when the job is finished. The MoM also runs any prologue scripts before the job runs, and runs any epilogue scripts after the job runs. The MoM performs any communication with job tasks and with other MoMs. The MoM on the first vnode on which a job is running manages communication with the MoMs on the remaining vnodes on which the job runs. The MoM manages one or more vnodes. PBS may treat a host as a set of virtual nodes, in which case one MoM manages all of the host's vnodes. See the PBS Professional Administrator's Guide. Logging
The MoM's log file is in PBS_HOME/mom_logs. The MoM writes an error message in its log file when it encounters any error. If it cannot write to its log file, it writes to standard error. The MoM writes events to its log file. The MoM writes its PBS version and build information to the logfile whenever it starts up or the logfile is rolled to a new file. Required Permission
The executable for pbs_mom is in PBS_EXEC/sbin, and can be run only by root on Linux, and Admin on Windows. Cpusets
A cpusetted machine can have a "boot cpuset" defined by the administrator. A boot cpuset contains one or more CPUs and memory boards and is used to restrict the default placement of system processes, including login. If defined, the boot cpuset contains CPU 0. Run parallel jobs exclusively within a cpuset for repeatability of performance. HPE SGI states, "Using cpusets on an HPE SGI system improves cache locality and memory access times and can substantially improve an application's performance and runtime repeatability." The CPUSET_CPU_EXCLUSIVE flag prevents CPU 0 from being used by the MoM in the creation of job cpusets. This flag is set by default, so this is the default behavior. To find out which cpuset is assigned to a running job, use qstat -f to see the cpuset field in the job's altid attribute. HPE SGI Machine Running Supported Versions of Performance Software - Message Passing Interface
The cpusets created for jobs are marked cpu-exclusive. MoM does not use any CPU which was in use at startup. A PBS job can run across multiple machines that run supported versions of Performance Software - Message Passing Interface. PBS can run using HPE SGI's MPI (MPT) over InfiniBand. See the PBS Professional Administrator's Guide.

Effect on Jobs of Starting MoM
When MoM is started or restarted, her default behavior is to leave any running processes running, but to tell the PBS server to requeue the jobs she manages. MoM tracks the process ID of jobs across restarts. In order to have all jobs killed and requeued, use the r option when starting or restarting MoM. In order to leave any running processes running, and not to requeue any jobs, use the p option when starting or restarting MoM.

OPTIONS

-a <alarm timeout>
    Number of seconds before alarm timeout. Whenever a resource request is processed, an alarm is set for the given amount of time. If the request has not completed before alarm timeout, the OS generates an alarm signal and sends it to MoM. Default: 10 seconds. Format: integer.

-C <checkpoint directory>
    Specifies the path of the directory where MoM creates job-specific subdirectories used to hold each job's restart files. MoM passes this path to checkpoint and restart scripts. Overrides other checkpoint path specification methods. Any directory specified with the -C option must be owned, readable, writable, and executable by root only (rwx,---,---, or 0700), to protect the security of the checkpoint files. See the -d option. Format: string.
    Default: PBS_HOME/spool/checkpoint.

-c <config file>
    MoM will read this alternate default configuration file upon starting. If this is a relative file name it will be relative to PBS_HOME/mom_priv. If the specified file cannot be opened, pbs_mom will abort. See the -d option. MoM's normal operation, when the -c option is not given, is to attempt to open the default configuration file PBS_HOME/mom_priv/config. If this file is not present, pbs_mom will log the fact and continue.

-d <MoM home directory>
    Specifies the path of the directory to be used in place of PBS_HOME by pbs_mom. The default directory is given by $PBS_HOME. Format: string.

-L <logfile>
    Specifies an absolute path and filename for the log file. The default is a file named for the current date in PBS_HOME/mom_logs/. See the -d option. Format: string.

-M <MoM port>
    Specifies the number of the port on which MoM will listen for server requests and instructions. Overrides PBS_MOM_SERVICE_PORT setting in pbs.conf and environment variable. Default: 15002. Format: integer port number.

-n <nice value>
    Specifies the priority for the pbs_mom daemon. Format: integer.

-N  Specifies that when starting, MoM should not detach from the current session.

-p  Specifies that when starting, MoM should allow any running jobs to continue running, and not have them requeued. This option can be used for single-host jobs only; multi-host jobs cannot be preserved. Cannot be used with the -r option. MoM is not the parent of these jobs.

    HPE SGI systems running Performance Software - Message Passing Interface
        The cpuset-enabled pbs_mom will, if given the -p flag, use the existing CPU and memory allocations for the /PBSPro cpusets. The default behavior is to remove these cpusets. Should this fail, MoM will exit, asking to be restarted with the -p flag.

-r  Specifies that when starting, MoM should requeue any rerunnable jobs and kill any non-rerunnable jobs that she was tracking, and mark the jobs as terminated. Cannot be used with the -p option. MoM is not the parent of these jobs. It is not recommended to use the -r option after a reboot, because process IDs of new, legitimate tasks may match those MoM was previously tracking. If they match and MoM is started with the -r option, MoM will kill the new tasks.

-R <inter-MoM communication port>
    Specifies the number of the port on which MoM will listen for pings, resource information requests, communication from other MoMs, etc. Overrides PBS_MANAGER_SERVICE_PORT setting in pbs.conf and environment variable. Default: 15003. Format: integer port number.

-S <server port>
    Specifies the port number on which pbs_mom initially contacts the server. Default: 15001. Format: integer port number.

-s <script options>
    This option provides an interface that allows the administrator to add, delete, and display MoM's configuration files. See CONFIGURATION FILES. The script options are used this way:

    -s insert <scriptname> <inputfile>
        Reads inputfile and inserts its contents in a new site-defined pbs_mom configuration file with the filename scriptname. If a site-defined configuration file with the name scriptname already exists, the operation fails, a diagnostic is presented, and pbs_mom exits with a nonzero status. Scripts whose names begin with the prefix "PBS" are reserved. An attempt to add a script whose name begins with "PBS" will fail. pbs_mom will print a diagnostic message and exit with a nonzero status. Example: pbs_mom -s insert <scriptname> <inputfile>

    -s remove <scriptname>
        The configuration file named scriptname is removed if it exists. If the given name does not exist or if an attempt is made to remove a script with the reserved "PBS" prefix, the operation fails, a diagnostic is presented, and pbs_mom exits with a nonzero status. Example: pbs_mom -s remove <scriptname>

    -s show <scriptname>
        Causes the contents of the named script to be printed to standard output. If scriptname does not exist, the operation fails, a diagnostic is presented, and pbs_mom exits with a nonzero status. Example: pbs_mom -s show <scriptname>

    -s list
        Causes pbs_mom to list the set of PBS-prefixed and site-defined configuration files in the order in which they are executed. Example: pbs_mom -s list

    WINDOWS:
        Under Windows, the -N option must be used, so that pbs_mom will start up as a standalone program. For example: pbs_mom -N -s insert <scriptname> <inputfile> or pbs_mom -N -s list

--version
    The pbs_mom command returns its PBS version information and exits. This option can only be used alone.

CONFIGURATION FILES
MoM's configuration information can be contained in configuration files of three types: default, PBS-prefixed, and site-defined. The default configuration file is usually PBS_HOME/mom_priv/config. The "PBS" prefix is reserved for files created by PBS. Site-defined configuration files are those created by the site administrator. MoM reads the configuration files at startup and reinitialization. The files are processed in this order:
     The default configuration file
     PBS-prefixed configuration files
     Site-defined configuration files
The contents of a file read later override the contents of a file read earlier. For example, to change the cpuset flags, create a script "update_flags" containing only
    cpuset_create_flags <new flags>
then use the -s insert option:
    pbs_mom -s insert update_script update_flags
This adds the configuration file "update_script". Configuration files can be added, deleted and displayed using the -s option. MoM's configuration files can use either the syntax shown below under Default Syntax and Contents or the syntax for describing vnodes shown in Vnode Syntax. Location
The default configuration file is in PBS_HOME/mom_priv/. PBS places PBS-prefixed and site-defined configuration files in an area that is private to each installed instance of PBS. This area is relative to the default PBS_HOME. Note that the -d option changes where MoM looks for PBS_HOME. The -c option will change which default configuration file MoM reads. Site-defined configuration files can be moved from one installed instance of PBS to another. Do not move PBS-prefixed configuration files. To move a set of site-defined configuration files from one installed instance of PBS to another:

1   Use the -s list directive with the "source" instance of PBS to enumerate the site-defined files.

2   Use the -s show directive with each site-defined file of the "source" instance of PBS to save a copy of that file.

3   Use the -s insert directive with each file at the "target" instance of PBS to create a copy of each site-defined configuration file.

Vnode Configuration File Syntax and Contents
Configuration files with the following syntax describe vnodes and the resources available on them. They do not contain initialization values for MoM. See the PBS Professional Administrator's Guide for a definition of vnodes. PBS-prefixed configuration files use the following syntax. Other configuration files can use the following syntax. Any configuration file containing vnode-specific assignments must begin with this line:
    $configversion 2
The format a file containing vnode information is:
    <ID> : <ATTRNAME> = <ATTRVAL>
where

    <ID>
        sequence of characters not including a colon (":")

    <ATTRNAME>
        sequence of characters beginning with alphabetics or numerics, which can contain underscore ("_") and dash ("-")

    <ATTRVAL>
        sequence of characters not including an equal sign ("=")

    The colon and equal sign may be surrounded by spaces.
A vnode's ID is an identifier that will be unique across all vnodes known to a given pbs_server and will be be stable across reinitializations or invocations of pbs_mom. ID stability is of importance when a vnode's CPUs or memory might be expected to change over time and PBS is expected to adapt to such changes by resuming suspended jobs on the same vnodes to which they were originally assigned. Vnodes for which this is not a consideration may simply use IDs of the form "0", "1", etc. concatenated with some identifier that ensures uniqueness across the vnodes served by the pbs_server. A natural vnode does not correspond to any actual hardware. It is used to define any placement set information that is invariant for a given host, such as pnames. It is defined as follows:

    The name of the natural vnode is, by convention, the MoM contact name, which is usually the hostname. The MoM contact name is the vnode's MoM attribute. See the pbs_node_attributes(7B) man page.

    An attribute, "pnames", with value set to the list of resource names that define the placement sets' types for this machine.

    An attribute, "sharing" is set to the value "force_shared"

The natural vnode
is used to define any placement set information    that is invariant for
a given host (e.g. the placement set resource names themselves). The order of the pnames attribute follows placement set organization. If name X appears to the left of name Y in this attribute's value, an entity of type X may be assumed to be smaller (that is, be capable of containing fewer vnodes) than one of type Y. No such guarantee is made for specific instances of the types. For example, on an HPE SGI machine named "HostA", with two vnodes, a natural vnode, four processors and two cbricks, the description would look like this:
     HostA: pnames = cbrick
     HostA: sharing = force_shared
     HostA[001c02#0]: sharing = default_excl
     HostA[001c02#0]: resources_available.ncpus = 2
     HostA[001c02#0]: resources_available.cbrick = cbrick-0
     HostA[001c02#0]: resources_available.mem = 1968448kb
     HostA[001c04#0]: sharing = default_excl
     HostA[001c04#0]: resources_available.ncpus = 2
     HostA[001c04#0]: resources_available.cbrick = cbrick-1
     HostA[001c04#0]: resources_available.mem = 1961328kb
The natural vnode is described in the first two lines. The first vnode uses cbrick-0, and the second one uses cbrick-1. Default Syntax and Contents
Configuration files with this syntax list local resources and initialization values for MoM. Local resources are either static, listed by name and value, or externally-provided, listed by name and command path. See the -c option. Each configuration item is listed on a single line, with its parts separated by white space. Comments begin with a hashmark ("#"). The default configuration file must be secure. It must be owned by a user ID and group ID both less than 10 and must not be world-writable. Externally-provided Resources
Externally-provided resources use a shell escape to run a command. These resources are described with a name and value, where the first character of the value is an exclamation mark ("!"). The remainder of the value is the path and command to execute. Parameters in the command beginning with a percent sign ("%") can be replaced when the command is executed. For example, this line in a configuration file describes a resource named "escape":
    escape !echo 0xx %yyy

    If a query for the "escape" resource is sent with no parameter replacements, the command executed would be "echo 0xx %yyy". If one parameter replacement is sent, "escape[xxx=hi there]", the command executed would be "echo hi there %yyy". If two parameter replacements are sent, "escape[xxx=hi][yyy=there]", the command executed would be "echo hi there". If a parameter replacement is sent with no matching token in the command line, "escape[zzz=snafu]", an error is reported.

Windows Notes
If the argument to a MoM option is a pathname containing a space, enclose it in double quotes as in the following: hostn !"\Program Files\PBS Pro\exec\bin\hostn" host When you edit any PBS configuration file, make sure that you put a newline at the end of the file. The Notepad application does not automatically add a newline at the end of a file; you must explicitly add the newline. Replacing Actions

$action <default action> <timeout> <new action>
    Replaces the default action for an event with the site-specified new action. timeout is the time allowed for new action to run. See The PBS Professional Administrator's Guide. The default action can be one of:

    checkpoint
        Run new action in place of the periodic job checkpoint, after which the job continues to run.

    checkpoint_abort
        Run new action to checkpoint the job, after which the job must be terminated by the script.

    multinodebusy <timeout> requeue
        Used with cycle harvesting and multi-vnode jobs. Changes default behavior when a vnode becomes busy. Instead of allowing the job to run, the job is requeued. timeout is ignored. The only new action is requeue.

    restart
        Runs new action in place of restart.

    terminate
        Runs new action in place of SIGTERM or SIGKILL when MoM terminates a job.

Initialization Values
Initialization value directives have names beginning with a dollar sign ("$"). See The PBS Professional Administrator's Guide.

$alps_client <path>
    Cray only. MoM runs this command to get the ALPS inventory. Must be full path to command.
    Format: path to command
    Default: None

$alps_release_timeout <timeout>
    Cray only. Specifies the amount of time that PBS tries to release an ALPS reservation before giving up. After this amount of time has passed, PBS stops trying to release the ALPS reservation, the job exits, and the job's rsources are released. PBS sends a HUP to the MoM so that she re-reads the ALPS inventory to get the current available ALPS resources.
    We recommend that the value for this parameter be greater than the value for suspectbegin.
    Format: Seconds, specified as positive integer.
    Default: 600 (10 minutes)

$checkpoint_path <path>
    MoM passes this path to checkpoint and restart scripts. This path can be absolute or relative to PBS_HOME/mom_priv. Overrides default. Overridden by pbs_mom -C option and by PBS_CHECKPOINT_PATH environment variable.

$clienthost <hostname>
    hostname is added to the list of hosts which are allowed to connect to MoM as long as they are using a privileged port. For example, this allows the hosts "fred" and "wilma" to connect to MoM:
     "$clienthost fred"
     "$clienthost wilma"
    The following hostnames are added to $clienthost automatically: the server, the localhost, and if configured, the secondary server. The server sends each MoM a list of the hosts in the nodes file, and these are added internally to $clienthost. None of these hostnames need to be listed in the configuration file. Two hostnames are always allowed to connect to pbs_mom, "localhost" and the name returned to MoM by the system call gethostname(). These hostnames do not need to be listed in the configuration file. The hosts listed as "clienthosts" make up a "sisterhood" of machines. Any one of the sisterhood will accept connections from within the sisterhood. The sisterhood must all use the same port number.

$cpuset_error_action <action>
    When using a cpuset-enabled MoM, specifies the action taken when a cpuset creation error occurs. Can take one of the following values:

    continue
        The error is logged and the job is killed and requeued.

    offline
        The vnodes on this host for this job are marked offline, and the job is requeued.

    Format: String
    Allowable values: "continue", "offline"
    Default: "offline"

$cputmult <factor>
    This sets a factor used to adjust CPU time used by each job. This allows adjustment of time charged and limits enforced where jobs run on a system with different CPU performance. If MoM's system is faster than the reference system, set factor to a decimal value greater than 1.0. For example:
    $cputmult 1.5

    If MoM's system is slower, set factor to a value between 1.0 and 0.0. For example:
    $cputmult 0.75

$dce_refresh_delta <delta>
    Defines the number of seconds between successive refreshings of a job's DCE login context. For example:
    $dce_refresh_delta 18000

$enforce <limit>
    MoM will enforce the given limit. Some limits have associated values. Syntax:
    $enforce <variable name> <value>
    See The PBS Professional Administrator's Guide.

    $enforce mem
        MoM will enforce each job's memory limit.

    $enforce cpuaverage
        MoM will enforce ncpus when the average CPU usage over a job's lifetime usage is greater than the job's limit.

        $enforce average_trialperiod <seconds>
            Modifies cpuaverage. Minimum number of seconds of job walltime before enforcement begins. Default: 120. Integer.

        $enforce average_percent_over <percentage>
            Modifies cpuaverage. Gives percentage by which a job may exceed its ncpus limit. Default: 50. Integer.

        $enforce average_cpufactor <factor>
            Modifies cpuaverage. The ncpus limit is multiplied by factor to produce actual limit. Default: 1.025. Float.

    $enforce cpuburst
        MoM will enforce the ncpus limit when CPU burst usage exceeds the job's limit.

        $enforce delta_percent_over <percentage>
            Modifies cpuburst. Gives percentage over limit to be allowed. Default: 50. Integer.

        $enforce delta_cpufactor <factor>
            Modifies cpuburst. The ncpus limit is multiplied by factor to produce actual limit. Default: 1.5. Float.

        $enforce delta_weightup <factor>
            Modifies cpuburst. Weighting factor for smoothing burst usage when average is increasing. Default: 0.4. Float.

        $enforce delta_weightdown <factor>
            Modifies cpuburst. Weighting factor for smoothing burst usage when average is decreasing. Default: 0.4. Float.

$ideal_load <load>
    Defines the load below which the vnode is not considered to be busy. Used with the $max_load directive. No default. Float.

    Example:
        $ideal_load 1.8

    Use of $ideal_load adds a static resource to the vnode called "ideal_load", which is only internally visible.

$jobdir_root <stage directory root>
    Directory under which PBS creates job-specific staging and execution directories. PBS creates a job's staging and execution directory when the job's sandbox attribute is set to PRIVATE. If $jobdir_root is unset, it defaults to the job owner's home directory. In this case the user's home directory must exist. If stage_directory_root does not exist when MoM starts up, MoM will abort. If stage directory root does not exist when MoM tries to run a job, MoM will kill the job. Path must be owned by root, and permissions must be 1777. On Windows, this directory should have Full Control Permission for the local Administrators group.

    Example:
        $jobdir_root /scratch/foo

$kbd_idle <idle wait> <min use> <poll interval>
    Declares that the vnode will be used for batch jobs during periods when the keyboard and mouse are not in use. The vnode must be idle for a minimum of idle wait seconds before being considered available for batch jobs. No default. Integer. The vnode must be in use for a minimum of min use seconds before it becomes unavailable for batch jobs. Default: 10. Integer. Mom checks for activity every poll interval seconds. Default: 1. Integer.

    Example:
        $kbd_idle 1800 10 5

$logevent <mask>
    Sets the mask that determines which event types are logged by pbs_mom. To include all debug events, use 0xffffffff.
    Log events:
    Name       Hex Value  Message Category
    ---------------------------------------------------
    ERROR      0001       Internal errors
    SYSTEM     0002       System errors
    ADMIN      0004       Administrative events
    JOB        0008       Job-related events
    JOB_USAGE  0010       Job accounting info
    SECURITY   0020       Security violations
    SCHED      0040       Scheduler events
    DEBUG      0080       Common debug messages
    DEBUG2     0100       Uncommon debug messages
    RESV       0200       Reservation-related info
    DEBUG3     0400       Rare debug messages
    DEBUG4     0800       Limit-related messages

$max_check_poll <seconds>
    Maximum time between polling cycles, in seconds. Minimum recommended value: 30 seconds. See the PBS Professional Administrator's Guide for usage. The interval between each poll starts at $min_check_poll and increases with each cycle until it reaches $max_check_poll, after which it remains the same. The amount by which the cycle increases is 1/20 of the difference between $max_check_poll and $min_check_poll.
    Format: Integer
    Minimum value: 1 second
    Default value: 120 seconds

$max_load <load> [suspend]
    Defines the load above which the vnode is considered to be busy. Used with the $ideal_load directive. No new jobs are started on a busy vnode. The optional suspend directive tells PBS to suspend jobs running on the vnode if the load average exceeds the $ max_load number, regardless of the source of the load (PBS and/or logged-in users). Without this directive, PBS will not suspend jobs due to load. We recommend setting load to a value that is slightly higher than the number of CPUs, for example 0.25 + ncpus.
    Default: number of CPUs on machine
    Format: Float
    Example:
    $max_load 3.5

$min_check_poll <seconds>
    Minimum time between polling cycles, in seconds. Must be greater than zero and less than $max_check_poll. Minimum recommended value: 10 seconds.
    Format: Integer.
    Minimum value: 1 second
    Default value: 10 seconds

$prologalarm <timeout>
    Defines the maximum number of seconds the prologue and epilogue may run before timing out. Default: 30 seconds. Integer. Example:
    $prologalarm 30

$reject_root_scripts <True | False>
    When set to True, MoM won't aquire any new hook scripts, and MoM won't run job scripts that would execute as root or Admin. However, MoM will run previously-aquired hooks that run as root.
    Format: Boolean
    Default: False

$restart_background <True | False>
    Controls how MoM runs a restart script after checkpointing a job. When this option is set to True, MoM forks a child which runs the restart script. The child returns when all restarts for all the local tasks of the job are done. MoM does not block on the restart. When this option is set to False, MoM runs the restart script and waits for the result.
    Format: Boolean
    Default: False

$restart_transmogrify <True | False>
    Controls how MoM runs a restart script after checkpointing a job. When this option is set to True, MoM runs the restart script, replacing the session ID of the original task's top process with the session ID of the script. When this option is set to False, MoM runs the restart script and waits for the result. The restart script must restore the original session ID for all the processes of each task so that MoM can continue to track the job. When this option is set to False and the restart uses an external command, the configuration parameter restart_background is ignored and treated as if it were set to True, preventing MoM from blocking on the restart.
    Format: Boolean
    Default: False

$restrict_user <True | False>
    Controls whether users not submitting jobs have access to this machine. If value is True, restrictions are applied. See $restrict_user_exceptions and $restrict_user_maxsysid. Not supported on Windows.
    Format: Boolean
    Default: False

$restrict_user_exceptions <user_list>
    Comma-separated list of users who are exempt from access restrictions applied by $restrict_user. Leading spaces within each entry are allowed. Maximum of 10 names.

$restrict_user_maxsysid <value>
    Any user with a numeric user ID less than or equal to value is exempt from restrictions applied by $restrict_user. If $restrict_user is True and no value exists for $restrict_user_maxsysid, PBS looks in /etc/login.defs, if it exists, for the value. Otherwise the default is used. Integer. Default: 999

$restricted <hostname>
    The hostname is added to the list of hosts which are allowed to connect to MoM without being required to use a privileged port. Hostnames can be wildcarded. For example, to allow queries from any host from the domain "xyz.com":
    $restricted *.xyz.com

    Queries from the hosts in the restricted list are only allowed access to information internal to this host, such as load average, memory available, etc. They may not run shell commands.

$suspendsig <suspend signal> [resume signal]
    Alternate signal suspend signal is used to suspend jobs instead of SIGSTOP. Optional resume signal is used to resume jobs instead of SIGCONT.

$tmpdir <directory>
    Location where each job's scratch directory will be created. PBS creats a temporary directory for use by the job, not by PBS. PBS creates the directory before the the job is run and removes the directory and its contents when the job is finished. It is scratch space for use by the job. Permission must be 1777 on Linux, writable by Everyone on Windows. Example:
    $tmpdir /memfs

    Default on Linux: /var/tmp
    Default on Windows: value of the TMP environment variable

$usecp <hostname:source prefix> <destination prefix>
    MoM uses /bin/cp to deliver output files when the destination is a network mounted file system, or when the source and destination are both on the local host, or when the source prefix can be replaced with the destination prefix on hostname. Both source prefix and destination prefix are absolute pathnames of directories, not files. Overrides PBS_RCP and PBS_SCP. Use trailing slashes on both source and destination. For example:
    $usecp HostA:/users/work/myproj/ /sharedwork/proj_results/

$vnodedef_additive
    Specifies whether MoM considers a vnode that appeared previously either in the inventory or in a vnode definition file, but that does not appear now, to be in her list of vnodes.
    When $vnodedef_additive is True, MoM treats missing vnodes as if they are still present, and continues to report them as if they are present. This means that the server does not mark missing vnodes as stale.
    When $vnodedef_additive is False, MoM does not list missing vnodes, the server's information is brought up to date with the inventory and vnode definition files, and the server marks missing vnodes as stale.
    Visible in configuration file on Cray only.
    Format: Boolean
    Default for MoM on Cray login node: False

$wallmult <factor>
    Each job's walltime usage is multiplied by factor. For example:
    $wallmult 1.5

Cray-only Initialization Values

pbs_accounting_workload_mgmt <value>
    Controls whether CSA accounting is enabled. Name does not start with dollar sign. If set to "1", "on", or "true", CSA accounting is enabled. If set to "0", "off", or "false", accounting is disabled. Default: "true"; enabled.
HPE SGI-only Initialization Values

cpuset_create_flags <flags>
    Lists the flags for when MoM does a cpusetCreate(3) for each job. flags is an or-ed list of flags. The flags are:

    HPE SGI formerly known as ICE, with Performance Software - Message Passing Interface
        CPUSET_CPU_EXCLUSIVE
        0
        Default: 0

cpuset_destroy_delay <delay>
    MoM waits up to delay seconds before destroying a cpuset of a just-completed job, but not longer than necessary. This gives the operating system more time to clean up leftover processes after they have been killed. Example:
    cpuset_destroy_delay 10

    Default for HPE SGI machines: 0
    Format: Integer

memreserved <megabytes>
    Deprecated. The amount of per-vnode memory reserved for system overhead. This much memory is deducted from the value of resources_available.mem for each vnode managed by this MoM.
    Example:
    memreserved 16

    Default: 0MB.
Static Resources
Static resources local to the vnode are described one resource to a line, with a name and value separated by white space. For example, tape drives of different types could be specified by:
    tape3480    4
    tape3420    2
    tapedat     1
    tape8mm     1

FILES AND DIRECTORIES

$PBS_HOME/mom_priv
    Default directory for default configuration files.

$PBS_HOME/mom_priv/config
    MoM's default configuration file.

$PBS_HOME/mom_logs
    Default directory for log files written by MoM.

$PBS_HOME/mom_priv/prologue
    File containing administrative script to be run before job execution.

$PBS_HOME/mom_priv/epilogue
    File containing administrative script to be run after job execution.

SIGNAL HANDLING
pbs_mom handles the following signals:

SIGHUP
    The pbs_mom daemon rereads its configuration files, closes and reopens the log file, and reinitializes resource structures.

SIGALRM
    MoM writes a log file entry. See the -a alarm_timeout option.

SIGINT
    The pbs_mom daemon exits, leaving all running jobs still running. See the -p option.

SIGKILL
    This signal is not caught. The pbs_mom daemon exits immediately.

SIGTERM, SIGXCPU, SIGXFSZ, SIGCPULIM, SIGSHUTDN
    The pbs_mom daemon terminates all running children and exits.

SIGPIPE, SIGUSR1, SIGUSR2, SIGINFO
    These are ignored.

All other signals have their default behavior installed.

EXIT STATUS

Greater than zero
    If the pbs_mom daemon fails to start
    If the -s insert option is used with an existing scriptname
    If the administrator attempts to add a script whose name begins with "PBS"
    If the administrator attempts to use the -s remove option on a nonexistent configuration file, or on a configuration file whose name begins with "PBS"
    If the administrator attempts to use the -s show option on a nonexistent script

SEE ALSO
The PBS Professional Administrator's Guide, pbs_server(8B), pbs_sched(8B), qstat(1B)