pbs_sched(8B)

NAME
pbs_sched - run a PBS scheduler

SYNOPSIS
pbs_sched [-a <alarm>] [-c <clientsfile>] [-d <home dir>]
          [-I <scheduler name>] [-L <logfile>] [-n] [-N]
          [-p <output file>] [-R <port number>] [-S <port number>] pbs_sched --version

DESCRIPTION
pbs_sched is a PBS scheduling daemon. It schedules PBS jobs.

pbs_sched must be executed with root permission on Linux or Admin privilege on Windows.

OPTIONS

-a <alarm>
    Deprecated. Overwrites value of sched_cycle_length scheduler attribute.
    Time in seconds to wait for a scheduling cycle to finish.
    Format: Time, in seconds.

-c <clientsfile>
    Add clients to this scheduler's list of known clients. The clientsfile contains single-line entries of the form
    $clienthost <hostname>

    Each hostname is added to the list of hosts allowed to connect to this scheduler. If clientsfile cannot be opened, this scheduler aborts. Path can be absolute or relative. If relative, it is relative to PBS_HOME/sched_priv.

-d <home dir>
    The directory in which this scheduler will run.
    Default: PBS_HOME/sched_priv.

-I <scheduler name>
    Name of scheduler to start. Required when starting a multisched.

-L <logfile>
    The absolute path and filename of the log file. A scheduler writes its PBS version and build information to the logfile whenever it starts up or the logfile is rolled to a new file. See the -d option.
    Default: A scheduler opens a file named for the current date in the PBS_HOME/sched_logs directory.

-n  Tells this scheduler to not restart itself if it receives a sigsegv or a sigbus. A scheduler by default restarts itself if it receives either of these two signals more than five minutes after starting. A scheduler does not restart itself if it receives either one within five minutes of starting.

-N  Instructs this scheduler not to detach itself from the current session.

-p <output file>
    Any output which is written to standard out or standard error is written to output file. The pathname can be absolute or relative, in which case it is relative to PBS_HOME/sched_priv. See the -d option.
    Default: PBS_HOME/sched_priv/sched_out

-R <port number>
    The port for MOM to use. If this option is not given, the port number is taken from PBS_MANAGER_SERVICE_PORT, in pbs.conf.
    Default: 15003

-S <port number>
    The port for this scheduler to use. Required when starting a multisched. For the default scheduler, if this option is not specified the default port is taken from PBS_SCHEDULER_SERVICE_PORT, in pbs.conf.
    Default value for default scheduler: 15004
    Default value for multisched: none

--version
    The pbs_sched command returns its PBS version information and exits. This option can only be used alone.

CONFIGURATION FILE
The file PBS_HOME/sched_priv/sched_config contains configuration parameters for this scheduler. Each entry must be a single unbroken line.
Format: name: value [prime | non-prime | all | none]
where

    name
        Must not contain whitespace.

    value
        Must be double-quoted if it contains whitespace. value can be True | False | <number> | <string>. True and False are not case-sensitive.

    [prime | non-prime | all | none]
        Specifies when this setting applies: during primetime, non-primetime, all the time, or none of the time. A blank third field is equivalent to all which means that it applies to both primetime and non-primetime.
        Valid values: all , ALL , none , NONE , prime , PRIME , non_prime , NON_PRIME

Any line starting with a hashmark, "#", is a comment, and is ignored. Configuration Parameters

backfill
    Toggle that controls whether PBS uses backfilling. If this is set to True, this scheduler attempts to schedule smaller jobs around higher-priority jobs when using strict_ordering, as long as running the smaller jobs won't change the start time of the jobs they were scheduled around. A scheduler chooses jobs in the standard order, so other high-priority jobs will be considered first in the set to fit around the highest-priority job. When this parameter is True and help_starving_jobs is True, this scheduler backfills around starving jobs. Can be used with strict_ordering and help_starving_jobs.
    Format: Boolean.
    Default: True all

backfill_prime
    This Scheduler will not run jobs which would overlap the boundary between primetime and non-primetime. This assures that jobs restricted to running in either primetime or non-primetime can start as soon as the time boundary happens. See also prime_spill, prime_exempt_anytime_queues.
    Format: Boolean.
    Default: False all

by_queue
    If set to True, all jobs that can be run from the highest-priority queue are run, then any jobs that can be run from the next queue are run, and so on. If by_queue is set to False, all jobs are treated as if they are in one large queue. The by_queue parameter is overridden by the round_robin parameter when round_robin is set to True.
    Format: Boolean.
    Default: True all

cpus_per_ssinode
    Deprecated. Such configuration now occurs automatically.

dedicated_prefix
    Queue names with this prefix are treated as dedicated queues, meaning jobs in that queue are considered for execution only when the system is in dedicated time as specified in the configuration file PBS_HOME/sched_priv/dedicated_time.
    Format: String
    Default: ded

fair_share
    Enables the fairshare algorithm, and turns on usage collecting. Jobs will be selected based on a function of their recent usage and priority (shares). Not a prime option.
    Format: Boolean
    Default: False all

fairshare_decay_factor
    Decay multiplier for fairshare usage reduction. Each decay period, the usage is multiplied by this value.
    Valid values: between 0 and 1, not inclusive. Not a prime option.
    Format: Float
    Default: 0.5

fairshare_decay_time
    Time between fairshare usage decay operations. Not a prime option.
    Format: Duration
    Default: 24:00:00

fairshare_entity
    Specifies the entity for which fairshare usage data will be collected. Can be euser, egroup, Account_Name, queue, or egroup:euser. Not a prime option.
    Format: String
    Default: euser

fairshare_enforce_no_shares
    If this option is enabled, jobs whose entity has zero shares will never run. Requires fair_share parameter to be enabled.
    Format: Boolean
    Default: False

fairshare_usage_res
    Specifies the mathematical formula to use in fairshare calculations. Is composed of PBS resources as well as mathematical operations that are standard Python operators and/or those in the Python math module. When using a PBS resource, if resources_used.<resource name> exists, that value is used. Otherwise, the value is taken from Resource_List.<resource name>. Not a prime option.
    Format: String
    Default: cput

half_life
    Deprecated (as of 13.0). The half-life for fairshare usage; after the amount of time specified, the fairshare usage is halved. Requires that fair_share parameter be enabled. Not a prime option.
    Format: Duration
    Default: 24:00:00

help_starving_jobs
    Setting this option enables starving job support. Once jobs have waited for the amount of time given by max_starve they are considered starving. If a job is considered starving, no lower-priority jobs will run until the starving job can be run, unless backfilling is also specified. To use this option, the max_starve configuration parameter needs to be set as well. See also backfill, max_starve, and the server's eligible_time_enable attribute.
    At each scheduler iteration, PBS calculates estimated.start_time and estimated.exec_vnode for starving jobs being backfilled around.
    Format: Boolean
    Default: True all

job_sort_key
    Specifies how jobs should be sorted. job_sort_key can be used to sort either by resources or by special case sorting routines. Multiple job_sort_key entries can be used, one to a line, in which case the first entry will be the primary sort key, the second will be used to sort equivalent items from the first sort, etc. The HIGH option implies descending sorting, LOW implies ascending. See example for details. This attribute is overridden by the job_sort_formula attribute. If both are set, job_sort_key is ignored and an error message is printed.
    Syntax:
    job_sort_key: "<resource name> HIGH|LOW"
    job_sort_key: "fairshare_perc HIGH|LOW"
    job_sort_key: "job_priority HIGH|LOW"
    There are three special case sorting routines, which can be used instead of <resource name>:

        fairshare_perc HIGH
            Sort based on how much fairshare percentage the entity deserves, based on the values in the resource group file. If user A has more priority than user B, all of user A's jobs will always run first. Past history is not used. This should only be used if entity share (strict priority) sorting is needed. Incompatible with fair_share scheduling parameter being True.

        job_priority HIGH | LOW
            Sort jobs by the job priority attribute regardless of job owner.

        sort_priority HIGH|LOW
            Deprecated. See job_priority above.
    The following example shows how to sort jobs so that those with high CPU count come first:
        job_sort_key: "ncpus HIGH" all
    The following example shows how to sort jobs so that those with lower memory come first:
        job_sort_key: "mem LOW" prime
    Format: Quoted string
    Default: Not in force

key Deprecated. Use job_sort_key.

load_balancing
    When set to True, this scheduler takes into account the load average on vnodes as well as the resources listed in the resources line in sched_config. Load balancing can result in overloaded CPUs.
    Format: Boolean
    Default: False all

load_balancing_rr
    Deprecated. To duplicate this setting, enable load_balancing and set smp_cluster_dist to round_robin.

log_filter
    Defines which event types to keep out of this scheduler's logfile. The value should be set to the bitwise OR of the event classes which should be filtered. A value of 0 specifies maximum logging.
    Format: Integer
    Default: 3328

max_starve
    The amount of time before a job is considered starving. This variable is used only if help_starving_jobs is set.
    Upper limit: None
    Format: Duration
    Default: 24:00:00

mem_per_ssinode
    Deprecated. Such configuration now occurs automatically.

mom_resources
    This option is used to query the MoMs to set the value of resources_available.<resource name> where resource name is a site-defined resource. Each MoM is queried with the resource name and the return value is used to replace resources_available.<resource name> on that vnode. On a multi-vnoded machine with a natural vnode, all vnodes share anything set in mom_resources.
    Format: String

node_sort_key
    Defines sorting on resource or priority values on vnodes. Resource must be numerical, for example, long or float. Up to 20 node_sort_key entries can be used, in which case the first entry will be the primary sort key, the second will be used to sort equivalent items from the first sort, etc.
    Syntax:
    node_sort_key: <resource name>|sort_priority <HIGH | LOW>
    node_sort_key: <resource name> <HIGH | LOW> <total | assigned | unused>
    where

        total
            Use the resources_available value.

        assigned
            Use the resources_assigned value.

        unused
            Use the value given by resources_available - resources_assigned.

        sort_priority
            Sort vnodes by the value of the vnode priority attribute.
    When sorting on a resource, the default third field is "total".
    Format: String
    Default: node_sort_key: sort_priority HIGH all

nonprimetime_prefix
    Queue names which start with this prefix are treated as non-primetime queues. Jobs in these queues run only during non-primetime. Primetime and non-primetime are defined in the holidays file.
    Format: String
    Default: np_

peer_queue
    Defines the mapping of a pulling queue to a furnishing queue for peer scheduling. Maximum number is 50 peer queues per scheduler.
    Format: String
    Default: Unset

preemptive_sched
    Enables job preemption. See preempt_order.
    Format: String
    Default: True all

preempt_checkpoint
    Deprecated. Add "C" to preempt_order parameter.

preempt_fairshare
    Deprecated. Add "fairshare" to preempt_prio parameter.

preempt_order
    Defines the order of preemption methods which the scheduler uses on jobs. This order can change depending on the percentage of time remaining on the job. The ordering can be any combination of S, C, and R.

        S   Suspend

        C   Checkpoint

        R   Requeue
    The usage is an ordering (SCR) optionally followed by a percentage of time remaining and another ordering. Must be a quoted list(""). For example, PBS should first attempt to use suspension to preempt a job, and if that is unsuccessful, then requeue the job:
        preempt_order: SR
    For example, if the job has between 100% and 81% requested time remaining, first try to suspend the job, then try checkpoint, then requeue. If the job has between 80% and 51% requested time remaining, attempt suspend, then checkpoint. Between 50% and 0% time remaining, just attempt to suspend the job:
        preempt_order: SCR 80 SC 50 S
    Format: Quoted list
    Default: SCR

preempt_prio
    Specifies the ordering of priority for different preemption levels. Two or more job types may be combined at the same priority level with a + between them, using no whitespace. Comma-separated preemption levels are evaluated left to right, with higher priority to the left. The table below lists the six preemption levels. Any level not specified in the preempt_prio list is ignored.

        express_queue
            Jobs in express queues preempt other jobs. See also preempt_queue_prio. Does not require by_queue to be True.

        starving_jobs
            When a job becomes starving it can preempt other jobs.

        fairshare
            When the entity owning a job exceeds its fairshare limit.

        queue_softlimits
            Jobs which are over their queue soft limits

        server_softlimits
            Jobs which are over their server soft limits

        normal_jobs
            The preemption level into which a job falls if it does not fit into any other specified level.
    For example, starving jobs have the highest priority, then normal jobs, and jobs whose entities are over their fairshare limit are third highest:
        preempt_prio: starving_jobs, normal_jobs, fairshare
    For example,starving jobs whose entities are also over their fairshare limit are lower priority than normal jobs:
        preempt_prio: normal_jobs, starving_jobs+fairshare
    Format: Quoted list
    Default: "express_queue, normal_jobs"

preempt_queue_prio
    Specifies the minimum queue priority required for a queue to be classified as an express queue. Express queues do not require by_queue to be True.
    Format: Integer
    Default: 150

preempt_requeue
    Deprecated. Add an "R" to preempt_order parameter.

preempt_sort
    Specifies whether those jobs most eligible for preemption are sorted according to their start times.
    If set to min_time_since_start, first job preempted will be that with most recent start time.
    If not set, meaning that this parameter is commented out, preempted job will be that with longest running time.
    Must be commented out in order to be unset.
    Allowable values: min_time_since_start, or no setting.
    Format: String
    Default: min_time_since_start

preempt_starving
    Deprecated. Add "starving_jobs" to preempt_prio parameter.

preempt_suspend
    Deprecated. Add an "S" to preempt_order parameter.

primetime_prefix
    Queue names starting with this prefix are treated as primetime queues. Jobs in these queues run only during primetime. Primetime and non-primetime are defined in the holidays file.
    Format: String
    Default: p_

prime_exempt_anytime_queues
    Determines whether anytime queues are controlled by backfill_prime. If set to true, jobs in an anytime queue are not prevented from running across a primetime/nonprimetime or non-primetime/primetime boundary. If set to false, the jobs in an anytime queue may not cross this boundary, except for the amount specified by their prime_spill setting. See also backfill_prime, prime_spill.
    Format: Boolean
    Default: False

prime_spill
    Specifies the amount of time a job can spill over from non-primetime into primetime or from primetime into non-primetime. This option is only meaningful if backfill_prime is True. This option can be separately specified for primetime and non-primetime. See also backfill_prime, prime_exempt_anytime_queues.
    For example, non-primetime jobs can spill into primetime by 1 hour:
        prime_spill: 1:00:00 prime
    For example, jobs in either prime/non-prime can spill into the other by 1 hour:
        prime_spill: 1:00:00 all
    Format: Duration
    Default: 00:00:00

provision_policy
    Specifies how vnodes are selected for provisioning. Can be set by Manager only; readable by all. Can be set to one of the following:

    avoid_provision
        PBS first tries to satisfy the job's request from free vnodes that already have the requested AOE instantiated. PBS uses node_sort_key to sort these vnodes. If it cannot satisfy the job's request using vnodes that already have the requested AOE instantiated, PBS uses the server's node_sort_key to select the free vnodes that must be provisioned in order to run the job, choosing from any free vnodes, regardless of which AOE is instantiated on them. Of the selected vnodes, PBS provisions any that do not have the requested AOE instantiated on them.

    aggressive_provision
        PBS selects vnodes to be provisioned without considering which AOE is currently instantiated. PBS uses the server's node_sort_key to select the vnodes on which to run the job, choosing from any free vnodes, regardless of which AOE is instantiated on them. Of the selected vnodes, PBS provisions any that do not have the requested AOE instantiated on them.

    Format: string
    Default: aggressive_provision

resources
    Specifies those resources which are not to be over-allocated, or if Boolean, are to be honored, when scheduling jobs. Vnode-level Boolean resources are automatically enforced and do not need to be listed here. Limits are set by setting resources_available.<resource name> on vnodes, queues, and the server. A scheduler considers numeric (integer or float) items as consumable resources and ensures that no more are assigned than are available (e.g. ncpus or mem ). Any string resources are compared using string comparisons (e.g. arch ).
    If "host" is not added to the resources line, then when the user submits a job requesting a specific vnode in the following syntax:
        qsub -l select=host=vnodeName
    the job will run on any host.
    Format: String
    Default: ncpus, mem, arch, host, vnode, aoe

resource_unset_infinite
    Resources in this list are treated as infinite if they are unset. Cannot be set differently for primetime and non-primetime.
    Example: resource_unset_infinite: vmem, foo_licenses
    Format: Comma-delimited list of resources
    Default: Empty list

round_robin
    If set to True, this scheduler considers one job from the first queue, then one job from the second queue, and so on in a circular fashion. If sort_queues is set to True, the queues are ordered with the highest-priority queue first. Each scheduling cycle starts with the same highest-priority queue, which will therefore get preferential treatment. If round_robin is set to False, this scheduler considers jobs according to the setting of the by_queue attribute. When True, overrides the by_queue attribute.
    Format: Boolean
    Default: False all

server_dyn_res
    Directs this scheduler to replace the server's resources_available values with new values returned by a site-specific external program.
    Format: String
    Default: Unset

smp_cluster_dist
    Deprecated (12.2). Specifies how single-host jobs should be distributed to all hosts of the complex. Options:

        pack
            Keep putting jobs onto one host until it is full and then move on to the next.

        round_robin
            Put one job on each vnode in turn before cycling back to the first one.

        lowest_load
            Put the job on the lowest-loaded host.
    Format: String
    Default: pack all

sort_by
    Deprecated. Use job_sort_key.

sort_queues
    Deprecated. When set to True, queues are sorted so that the highest-priority queues are considered first. Queues are sorted by each queue's priority attribute. The queues are sorted in a descending fashion, that is, a queue with priority 6 comes before a queue with priority 3. When set to False, queues are not sorted. This is a prime option, which means it can be selectively applied to primetime or non-primetime. The sorted order of queues is not taken into consideration unless by_queue is set to True.
    Format: Boolean
    Default: True ALL

strict_fifo
    Deprecated. Use strict_ordering.

strict_ordering
    Specifies that jobs must be run in the order determined by whatever sorting parameters are being used. This means that a job cannot be skipped due to resources required not being available. If a job due to run next cannot run, no job will run, unless backfilling is used, in which case jobs can be backfilled around the job that is due to run next.
    Example line in PBS_HOME/sched_priv/sched_config:
    strict_ordering: true ALL
    Format: Boolean
    Default: False all

sync_time
    Deprecated. The amount of time between writing the fairshare usage data to disk. Requires fair_share to be enabled.
    Format: Duration
    Default: 1:00:00

unknown_shares
    The number of shares for the unknown group. These shares determine the portion of a resource to be allotted to that group via fairshare. Requires fair_share to be enabled.
    Format: Integer
    Default: The unknown group gets 0 shares

FORMATS

Boolean
    Allowable values (case insensitive): True|T|Y|1|False|F|N|0

Duration
    Period of time. Expressed either as
    integer seconds
    or
    [[hours:]minutes:]seconds[.milliseconds]
    Milliseconds are rounded to the nearest second.

Float
    Allowable values: [+-] 0-9 [[0-9] ...][.][[0-9] ...]

Long
    Long integer. Allowable values: 0-9 [[0-9] ...], and + and -

Size
    Number of bytes or words. The size of a word is 64 bits.
    Format: <integer>[<suffix>]
    where suffix can be

    b or  w
        bytes or words

    kb or kw
        Kilobytes or kilowords (2 to the 10th, or 1024)

    mb or mw
        Megabytes or megawords (2 to the 20th, or 1,048,576)

    gb or gw
        Gigabytes or gigawords (2 to the 30th, or 1,073,741,824)

    tb or tw
        Terabytes or terawords (2 to the 40th, or 1024 gigabytes)

    pb or pw
        Petabytes or petawords (2 to the 50th, or 1,048,576 gigabytes)

    Default: bytes

String
    (Resource value)
    Any character, including the space character.
    Only one of the two types of quote characters, " or ', may appear in any given value.
    Allowable values: [_a-zA-Z0-9][[-_a-zA-Z0-9 !"#$%' ()*+,-./:;<=>?@[\]^_{|}~] ...]
    String resource values are case-sensitive.

FILES
$PBS_HOME/sched_priv is the default directory for configuration files. $PBS_HOME/sched_priv/holidays is the holidays file.

SIGNAL HANDLING
All signals are ignored until the end of the cycle. Most signals are handled in the standard UNIX fashion.

SIGHUP
    This scheduler closes and reopens its log file and rereads its configuration file if one exists.

SIGALRM, SIGBUS, etc.
    Ignored until end of scheduling cycle. This scheduler quits.

SIGINT and SIGTERM
    This scheduler closes its log file and shuts down.

EXIT STATUS
Zero upon normal termination.

SEE ALSO
The PBS Professional Administrator's Guide, pbs_server(8B), pbs_mom(8B)