openpbs/doc/man8/pbs_snapshot.8B

.\" Copyright (C) 1994-2018 Altair Engineering, Inc.
.\" For more information, contact Altair at www.altair.com.
.\"
.\" This file is part of the PBS Professional ("PBS Pro") software.
.\"
.\" Open Source License Information:
.\"
.\" PBS Pro is free software. You can redistribute it and/or modify it under the
.\" terms of the GNU Affero General Public License as published by the Free
.\" Software Foundation, either version 3 of the License, or (at your option) any
.\" later version.
.\"
.\" PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY
.\" WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
.\" FOR A PARTICULAR PURPOSE.
.\" See the GNU Affero General Public License for more details.
.\"
.\" You should have received a copy of the GNU Affero General Public License
.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
.\"
.\" Commercial License Information:
.\"
.\" For a copy of the commercial license terms and conditions,
.\" go to: (http://www.pbspro.com/UserArea/agreement.html)
.\" or contact the Altair Legal Department.
.\"
.\" Altair’s dual-license business model allows companies, individuals, and
.\" organizations to create proprietary derivative works of PBS Pro and
.\" distribute them - whether embedded or bundled with other software -
.\" under a commercial license agreement.
.\"
.\" Use of Altair’s trademarks, including but not limited to "PBS™",
.\" "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's
.\" trademark licensing policies.
.\"
.TH pbs_snapshot 8B "26 April 2018" Local "PBS Professional"
.SH NAME
.B pbs_snapshot
- Linux only.  Capture PBS data to be used for diagnostics
.SH SYNOPSIS
.B pbs_snapshot 
-o <output directory path> [-H <server host>] 
.RS 12
[-l <log level>] [--accounting-logs=<number of days>] 
[--additional-hosts=<hostname list>] 
.br
[--daemon-logs=<number of days>] [--map=<file path>]
[--obfuscate] 
.RE
.br
.B pbs_snapshot
--version
.SH DESCRIPTION

You use 
.B pbs_snapshot 
to capture PBS data for diagnostics.  This tool is written in Python
and uses PTL libraries, including PBSSnapUtils, to extract the data.
You can optionally anonymize the PBS data.

.B Required Privilege
.br
To run 
.B pbs_snapshot
, you must be root on Linux or Admin on Windows.

.SH OPTIONS
.IP "-H <server host>" 5
Specifies server hostname.  By default, 
.B pbs_snapshot 
uses the value of the 
.I PBS_SERVER 
parameter in pbs.conf.  When you use this option, 
.B pbs_snapshot 
uses 
.I server host 
instead.

.IP "-l <log level>" 5
Specifies level at which 
.B pbs_snapshot
writes its log.  The log file is 
pbs_snapshot.log, in the output directory path specified using the 
.I -o <output directory path> 
option.

Valid values, from most comprehensive to least: DEBUG2, DEBUG,
INFOCLI2, INFOCLI, INFO, WARNING, ERROR, FATAL
.br
Default: INFOCLI2

.IP "--accounting-logs=<number of days>" 5
Specifies number of days of accounting logs to be collected; this
count includes the current day.
.br
Value of number of days must be >=0:
.br
   If number of days is 0, no logs are captured.
.br
   If number of days is 1, only the logs for the current day 
.br
   are captured.
.br
Default: 
.B pbs_snapshot
collects 30 days of accounting logs

.IP "--additional-hosts=<hostname list>" 5
Specifies that 
.B pbs_snapshot
should gather data from the specified list of non-server hosts.  
.B pbs_snapshot
always gathers data from the server host.  
.br
The command collects the following information from the specified hosts:
.br 
   MoM and comm logs, for the number of days of logs being 
.br
   captured, specified via the 
.I --daemon-logs=<number of days> 
.br
   option
.br
   The PBS_HOME/mom_priv directory
.br
   System information
.br
Format for 
.I hostname list 
is a comma-separated list of one or more hostnames: 
.br
.I <hostname>[, <hostname> ...]
.br
This option can greatly bloat the size of the snapshot, and cause 
.B pbs_snapshot
to take a long time copying what may be large amounts of data over the network.

.IP "--daemon-logs=<number of days>" 5
Specifies number of days of daemon logs to be collected; this count
includes the current day.  All daemon logs are captured on the server host,
and if you specify
.I --additional-hosts=<hostname list>, 
MoM logs are captured on those hosts as well.  
.br
Value of number of days must be >=0:
.br
   If number of days is 0, no logs are captured.
.br
   If number of days is 1, only the logs for the current day 
.br
   are captured.
.br
Default: 
.B pbs_snapshot
collects 5 days of daemon logs

.IP "--map=<file path>" 5
Specifies path for file containing obfuscation map, which is a
<key>:<value> pair-mapping of obfuscated data.  Path can be absolute
or relative to current working directory.
.br
Default: 
.B pbs_snapshot
writes its obfuscation map in a file called obfuscate.map in the
location specified via the
.I -o <output directory path> 
option.  
.br
Can only be used with the 
.I --obfuscate 
option.

.IP "--obfuscate" 5
Obfuscates (anonymizes) or deletes sensitive PBS data captured by 
.B pbs_snapshot.
.br
Obfuscates the following data: 
.RS 8
euser, egroup, project, Account_Name, operators, managers, group_list,
Mail_Users, User_List, server_host, acl_groups, acl_users,
acl_resv_groups, acl_resv_users, sched_host, acl_resv_hosts,
acl_hosts, Job_Owner, exec_host, Host, Mom, resources_available.host,
resources_available.vnode
.RE
.IP " " 5
Deletes the following data: 
.RS 8
Variable_List, Error_Path, Output_Path, mail_from, Mail_Points,
Job_Name, jobdir, Submit_arguments, Shell_Path_List
.RE

.IP "--version" 5
The 
.B pbs_snapshot
command returns its PBS version information and exits.
This option can only be used alone.

.SH Arguments to pbs_snapshot
.IP "-o <output directory path>" 5
Path to directory where 
.B pbs_snapshot
writes its output tarball.  Required.  Path can be absolute or
relative to current working directory.
.br
For example, if you specify 
.I -o /temp,
.B pbs_snapshot
writes "/temp/snapshot_<timestamp>.tgz".
.br
The output directory path must already exist. 

.SH Output
.B Output Location
.br
You must use the 
.I -o <output directory path> 
option to specify the directory where 
.B pbs_snapshot
writes its output.  The path can be absolute or relative to current
working directory.  The output directory must already exist.  As an
example, if you specify "-o /temp", 
.B pbs_snapshot
writes "/temp/snapshot_<timestamp>.tgz".

.B Output Contents
.br
The 
.B pbs_snapshot
command writes its output as a tarball.  The tarball contains the
following directory structure and files:
        
Directory  Directory
.br
or File\ \ \ \ Contents\ \ \ \ \ \ \ \ \ \ \ \ \ Description
.br
------------------------------------------------------------------------
.br
server/         
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_B.out\ \ \ \ \ \ \ \ \ \ Output of qstat -B
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_Bf.out         Output of qstat -Bf
.br
\ \ \ \ \ \ \ \ \ \ \ qmgr_ps.out          Output of qmgr print server
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_Q.out          Output of qstat -Q
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_Qf.out         Output of qstat -Qf
.br
\ \ \ \ \ \ \ \ \ \ \ qmgr_pr.out          Output of qmgr print resource
.br

server_priv/                    Copy of the PBS_HOME/server_priv 
.br
                                directory.   
.br
                                Core files are captured separately; 
.br
                                see 
.I core_file_bt/.  
.br

\ \ \ \ \ \ \ \ \ \ \ accounting/          Accounting logs from 
.br
                                PBS_HOME/server_priv/accounting/ 
.br
                                directory for the number of days 
.br
                                specified via
.I --accounting-logs 
option
.br

server_logs/                    Server logs from the 
.br
                                PBS_HOME/server_logs directory for the
.br
                                number of days specified 
.br
                                via
.I --daemon-logs 
option
.br

job/            
.br
\ \ \ \ \ \ \ \ \ \ \ qstat.out            Output of qstat
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_f.out          Output of qstat -f
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_t.out          Output of qstat -t
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_tf.out         Output of qstat -tf
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_x.out          Output of qstat -x
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_xf.out         Output of qstat -xf
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_ns.out         Output of qstat -ns
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_fx_F_dsv.out   Output of qstat -fx -F dsv
.br
\ \ \ \ \ \ \ \ \ \ \ qstat_f_F_dsv.out    Output of qstat -f -F dsv
node/           
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_va.out      Output of pbsnodes -va
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_a.out       Output of pbsnodes -a
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_avSj.out    Output of pbsnodes -avSj
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_aSj.out     Output of pbsnodes -aSj
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_avS.out     Output of pbsnodes -avS
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_aS.out      Output of pbsnodes -aS
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_aFdsv.out   Output of pbsnodes -aFdsv
.br
\ \ \ \ \ \ \ \ \ \ \ pbsnodes_avFdsv.out  Output of pbsnodes -avFdsv
.br
\ \ \ \ \ \ \ \ \ \ \ qmgr_pn_default.out  Output of qmgr print node @default
.br

mom_priv/                       Copy of the PBS_HOME/mom_priv 
.br
                                directory.
.br
                                Core files are captured separately; 
.br
                                see core_file_bt/.  
.br

mom_logs/                       MoM logs from the PBS_HOME/mom_logs 
.br
                                directory for the number of days 
.br
                                specified via
.I --daemon-logs 
option 
.br

comm_logs/                      Comm logs from the PBS_HOME/comm_logs 
.br
                                directory for the number of days 
.br
                                specified via 
.I --daemon-logs 
option  
.br

sched_priv/                     Copy of the PBS_HOME/sched_priv 
.br
                                directory, with all files.
.br
                                Core files are not captured; 
.br
                                see core_file_bt/. 
.br

sched_logs/                     Scheduler logs from the 
.br
                                PBS_HOME/sched_logs directory for 
.br
                                the number of days specified 
.br
                                via 
.I --daemon-logs 
option    
.br

sched_priv_<multisched name>/   Copy of the 
.br
                                PBS_HOME/sched_priv_<multisched_name>
.br
                                directory, with all files.
.br
                                Core files are not captured; 
.br
                                see core_file_bt/. 
.br

sched_logs_<multisched name>/   Scheduler logs from the 
.br
                                PBS_HOME/sched_logs_<multisched_name> 
.br
                                directory for the number
.br
                                of days specified
.br
                                via 
.I --daemon-logs 
option    
.br

reservation/            
.br
\ \ \ \ \ \ \ \ \ \ \ pbs_rstat_f.out      Output of pbs_rstat -f
.br

\ \ \ \ \ \ \ \ \ \ \ pbs_rstat.out        Output of pbs_rstat
.br

scheduler/              
.br
\ \ \ \ \ \ \ \ \ \ \ qmgr_lsched.out      Output of qmgr list sched
.br

hook/           
.br
\ \ \ \ \ \ \ \ \ \ \ qmgr_ph_default.out  Output of qmgr print hook @default
.br

\ \ \ \ \ \ \ \ \ \ \ qmgr_lpbshook.out    Output of qmgr list pbshook
.br

datastore/              
.br
\ \ \ \ \ \ \ \ \ \ \ pg_log/              Copy of the 
.br
                                PBS_HOME/datastore/pg_log directory 
.br
                                for the number of days specified 
.br
                                via 
.I --daemon-logs 
option
.br

core_file_bt/                   Stack backtrace from core files 
.br

\ \ \ \ \ \ \ \ \ \ \ sched_priv/          Files containing the output of thread 
.br
                                apply all backtrace full on all core 
.br
                                files captured from PBS_HOME/sched_priv
.br

\ \ \ \ \ \ \ \ \ \ \ sched_priv_          Files containing the output of thread 
.br
\ \ \ \ \ \ \ \ \ \ \ <multisched name>/   apply all backtrace full on all core 
.br
                                files captured from PBS_HOME/sched_priv
.br

\ \ \ \ \ \ \ \ \ \ \ server_priv/         Files containing the output of thread 
.br 
                                apply all backtrace full on all core 
.br
                                files captured from 
.br
                                PBS_HOME/server_priv
.br

\ \ \ \ \ \ \ \ \ \ \ mom_priv/            Files containing the output of thread 
.br
                                apply all backtrace full on all core 
.br
                                files captured from PBS_HOME/mom_priv
.br

\ \ \ \ \ \ \ \ \ \ \ misc/                Files containing the output of thread 
.br
                                apply all backtrace full on any other 
.br
                                core files found inside PBS_HOME
.br

system/         
.br
\ \ \ \ \ \ \ \ \ \ \ pbs_probe_v.out      Output of pbs_probe -v
.br

\ \ \ \ \ \ \ \ \ \ \ pbs_hostn_v.out      Output of pbs_hostn -v $(hostname)
.br

\ \ \ \ \ \ \ \ \ \ \ pbs_environment      Copy of PBS_HOME/pbs_environment file
.br

\ \ \ \ \ \ \ \ \ \ \ os_info              Information about the OS
.br

\ \ \ \ \ \ \ \ \ \ \ process_info         List of processes running on the system 
.br
                                when the snapshot was taken.  Output of
.br
                                ps -aux | grep [p]bs on Linux systems,
.br
                                or tasklist /v on Windows systems
.br

\ \ \ \ \ \ \ \ \ \ \ ps_leaf.out          Output of ps -leaf.  Linux only.
.br

\ \ \ \ \ \ \ \ \ \ \ lsof_pbs.out         Output of lsof | grep [p]bs.
.br
                                Linux only.
.br
\ \ \ \ \ \ \ \ \ \ \ etc_hosts            Copy of /etc/hosts file.  Linux only.
.br

\ \ \ \ \ \ \ \ \ \ \ etc_nsswitch_conf    Copy of /etc/nsswitch.conf file.
.br
                                Linux only.
.br

\ \ \ \ \ \ \ \ \ \ \ vmstat.out           Output of the command vmstat.  
.br
                                Linux only.
.br

\ \ \ \ \ \ \ \ \ \ \ df_h.out             Output of the command df -h.  
.br
                                Linux only.
.br

\ \ \ \ \ \ \ \ \ \ \ dmesg.out            Output of the dmesg command.  
.br
                                Linux only.
.br

pbs.conf                        Copy of the pbs.conf file on the 
.br
                                server host    
.br

ctime                           Contains the time in seconds since 
                                epoch when the snapshot was taken    
.br

pbs_snapshot.log                Log messages written by 
.B pbs_snapshot
    

.SH Examples
.IP "pbs_snapshot -o /tmp" 5
Writes a snapshot to /temp/snapshot_<timestamp>.tgz that includes 30
days of accounting logs and 5 days of daemon logs from the server
host.

.IP "pbs_snapshot --daemon-logs=1 --accounting-logs=1 -o /tmp --obfuscate --map=mapfile.txt" 5
Writes a snapshot to /temp/snapshot_<timestamp>.tgz that includes 1
day of accounting and daemon logs.  Obfuscates the data and stores the
data mapping in the map file named "mapfile.txt".