.\" Copyright (C) 1994-2018 Altair Engineering, Inc. .\" For more information, contact Altair at www.altair.com. .\" .\" This file is part of the PBS Professional ("PBS Pro") software. .\" .\" Open Source License Information: .\" .\" PBS Pro is free software. You can redistribute it and/or modify it under the .\" terms of the GNU Affero General Public License as published by the Free .\" Software Foundation, either version 3 of the License, or (at your option) any .\" later version. .\" .\" PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY .\" WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS .\" FOR A PARTICULAR PURPOSE. .\" See the GNU Affero General Public License for more details. .\" .\" You should have received a copy of the GNU Affero General Public License .\" along with this program. If not, see . .\" .\" Commercial License Information: .\" .\" For a copy of the commercial license terms and conditions, .\" go to: (http://www.pbspro.com/UserArea/agreement.html) .\" or contact the Altair Legal Department. .\" .\" Altair’s dual-license business model allows companies, individuals, and .\" organizations to create proprietary derivative works of PBS Pro and .\" distribute them - whether embedded or bundled with other software - .\" under a commercial license agreement. .\" .\" Use of Altair’s trademarks, including but not limited to "PBS™", .\" "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's .\" trademark licensing policies. .\" .TH pbs_snapshot 8B "26 April 2018" Local "PBS Professional" .SH NAME .B pbs_snapshot - Linux only. Capture PBS data to be used for diagnostics .SH SYNOPSIS .B pbs_snapshot -o [-H ] .RS 12 [-l ] [--accounting-logs=] [--additional-hosts=] .br [--daemon-logs=] [--map=] [--obfuscate] .RE .br .B pbs_snapshot --version .SH DESCRIPTION You use .B pbs_snapshot to capture PBS data for diagnostics. This tool is written in Python and uses PTL libraries, including PBSSnapUtils, to extract the data. You can optionally anonymize the PBS data. .B Required Privilege .br To run .B pbs_snapshot , you must be root on Linux or Admin on Windows. .SH OPTIONS .IP "-H " 5 Specifies server hostname. By default, .B pbs_snapshot uses the value of the .I PBS_SERVER parameter in pbs.conf. When you use this option, .B pbs_snapshot uses .I server host instead. .IP "-l " 5 Specifies level at which .B pbs_snapshot writes its log. The log file is pbs_snapshot.log, in the output directory path specified using the .I -o option. Valid values, from most comprehensive to least: DEBUG2, DEBUG, INFOCLI2, INFOCLI, INFO, WARNING, ERROR, FATAL .br Default: INFOCLI2 .IP "--accounting-logs=" 5 Specifies number of days of accounting logs to be collected; this count includes the current day. .br Value of number of days must be >=0: .br If number of days is 0, no logs are captured. .br If number of days is 1, only the logs for the current day .br are captured. .br Default: .B pbs_snapshot collects 30 days of accounting logs .IP "--additional-hosts=" 5 Specifies that .B pbs_snapshot should gather data from the specified list of non-server hosts. .B pbs_snapshot always gathers data from the server host. .br The command collects the following information from the specified hosts: .br MoM and comm logs, for the number of days of logs being .br captured, specified via the .I --daemon-logs= .br option .br The PBS_HOME/mom_priv directory .br System information .br Format for .I hostname list is a comma-separated list of one or more hostnames: .br .I [, ...] .br This option can greatly bloat the size of the snapshot, and cause .B pbs_snapshot to take a long time copying what may be large amounts of data over the network. .IP "--daemon-logs=" 5 Specifies number of days of daemon logs to be collected; this count includes the current day. All daemon logs are captured on the server host, and if you specify .I --additional-hosts=, MoM logs are captured on those hosts as well. .br Value of number of days must be >=0: .br If number of days is 0, no logs are captured. .br If number of days is 1, only the logs for the current day .br are captured. .br Default: .B pbs_snapshot collects 5 days of daemon logs .IP "--map=" 5 Specifies path for file containing obfuscation map, which is a : pair-mapping of obfuscated data. Path can be absolute or relative to current working directory. .br Default: .B pbs_snapshot writes its obfuscation map in a file called obfuscate.map in the location specified via the .I -o option. .br Can only be used with the .I --obfuscate option. .IP "--obfuscate" 5 Obfuscates (anonymizes) or deletes sensitive PBS data captured by .B pbs_snapshot. .br Obfuscates the following data: .RS 8 euser, egroup, project, Account_Name, operators, managers, group_list, Mail_Users, User_List, server_host, acl_groups, acl_users, acl_resv_groups, acl_resv_users, sched_host, acl_resv_hosts, acl_hosts, Job_Owner, exec_host, Host, Mom, resources_available.host, resources_available.vnode .RE .IP " " 5 Deletes the following data: .RS 8 Variable_List, Error_Path, Output_Path, mail_from, Mail_Points, Job_Name, jobdir, Submit_arguments, Shell_Path_List .RE .IP "--version" 5 The .B pbs_snapshot command returns its PBS version information and exits. This option can only be used alone. .SH Arguments to pbs_snapshot .IP "-o " 5 Path to directory where .B pbs_snapshot writes its output tarball. Required. Path can be absolute or relative to current working directory. .br For example, if you specify .I -o /temp, .B pbs_snapshot writes "/temp/snapshot_.tgz". .br The output directory path must already exist. .SH Output .B Output Location .br You must use the .I -o option to specify the directory where .B pbs_snapshot writes its output. The path can be absolute or relative to current working directory. The output directory must already exist. As an example, if you specify "-o /temp", .B pbs_snapshot writes "/temp/snapshot_.tgz". .B Output Contents .br The .B pbs_snapshot command writes its output as a tarball. The tarball contains the following directory structure and files: Directory Directory .br or File\ \ \ \ Contents\ \ \ \ \ \ \ \ \ \ \ \ \ Description .br ------------------------------------------------------------------------ .br server/ .br \ \ \ \ \ \ \ \ \ \ \ qstat_B.out\ \ \ \ \ \ \ \ \ \ Output of qstat -B .br \ \ \ \ \ \ \ \ \ \ \ qstat_Bf.out Output of qstat -Bf .br \ \ \ \ \ \ \ \ \ \ \ qmgr_ps.out Output of qmgr print server .br \ \ \ \ \ \ \ \ \ \ \ qstat_Q.out Output of qstat -Q .br \ \ \ \ \ \ \ \ \ \ \ qstat_Qf.out Output of qstat -Qf .br \ \ \ \ \ \ \ \ \ \ \ qmgr_pr.out Output of qmgr print resource .br server_priv/ Copy of the PBS_HOME/server_priv .br directory. .br Core files are captured separately; .br see .I core_file_bt/. .br \ \ \ \ \ \ \ \ \ \ \ accounting/ Accounting logs from .br PBS_HOME/server_priv/accounting/ .br directory for the number of days .br specified via .I --accounting-logs option .br server_logs/ Server logs from the .br PBS_HOME/server_logs directory for the .br number of days specified .br via .I --daemon-logs option .br job/ .br \ \ \ \ \ \ \ \ \ \ \ qstat.out Output of qstat .br \ \ \ \ \ \ \ \ \ \ \ qstat_f.out Output of qstat -f .br \ \ \ \ \ \ \ \ \ \ \ qstat_t.out Output of qstat -t .br \ \ \ \ \ \ \ \ \ \ \ qstat_tf.out Output of qstat -tf .br \ \ \ \ \ \ \ \ \ \ \ qstat_x.out Output of qstat -x .br \ \ \ \ \ \ \ \ \ \ \ qstat_xf.out Output of qstat -xf .br \ \ \ \ \ \ \ \ \ \ \ qstat_ns.out Output of qstat -ns .br \ \ \ \ \ \ \ \ \ \ \ qstat_fx_F_dsv.out Output of qstat -fx -F dsv .br \ \ \ \ \ \ \ \ \ \ \ qstat_f_F_dsv.out Output of qstat -f -F dsv node/ .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_va.out Output of pbsnodes -va .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_a.out Output of pbsnodes -a .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_avSj.out Output of pbsnodes -avSj .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_aSj.out Output of pbsnodes -aSj .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_avS.out Output of pbsnodes -avS .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_aS.out Output of pbsnodes -aS .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_aFdsv.out Output of pbsnodes -aFdsv .br \ \ \ \ \ \ \ \ \ \ \ pbsnodes_avFdsv.out Output of pbsnodes -avFdsv .br \ \ \ \ \ \ \ \ \ \ \ qmgr_pn_default.out Output of qmgr print node @default .br mom_priv/ Copy of the PBS_HOME/mom_priv .br directory. .br Core files are captured separately; .br see core_file_bt/. .br mom_logs/ MoM logs from the PBS_HOME/mom_logs .br directory for the number of days .br specified via .I --daemon-logs option .br comm_logs/ Comm logs from the PBS_HOME/comm_logs .br directory for the number of days .br specified via .I --daemon-logs option .br sched_priv/ Copy of the PBS_HOME/sched_priv .br directory, with all files. .br Core files are not captured; .br see core_file_bt/. .br sched_logs/ Scheduler logs from the .br PBS_HOME/sched_logs directory for .br the number of days specified .br via .I --daemon-logs option .br sched_priv_/ Copy of the .br PBS_HOME/sched_priv_ .br directory, with all files. .br Core files are not captured; .br see core_file_bt/. .br sched_logs_/ Scheduler logs from the .br PBS_HOME/sched_logs_ .br directory for the number .br of days specified .br via .I --daemon-logs option .br reservation/ .br \ \ \ \ \ \ \ \ \ \ \ pbs_rstat_f.out Output of pbs_rstat -f .br \ \ \ \ \ \ \ \ \ \ \ pbs_rstat.out Output of pbs_rstat .br scheduler/ .br \ \ \ \ \ \ \ \ \ \ \ qmgr_lsched.out Output of qmgr list sched .br hook/ .br \ \ \ \ \ \ \ \ \ \ \ qmgr_ph_default.out Output of qmgr print hook @default .br \ \ \ \ \ \ \ \ \ \ \ qmgr_lpbshook.out Output of qmgr list pbshook .br datastore/ .br \ \ \ \ \ \ \ \ \ \ \ pg_log/ Copy of the .br PBS_HOME/datastore/pg_log directory .br for the number of days specified .br via .I --daemon-logs option .br core_file_bt/ Stack backtrace from core files .br \ \ \ \ \ \ \ \ \ \ \ sched_priv/ Files containing the output of thread .br apply all backtrace full on all core .br files captured from PBS_HOME/sched_priv .br \ \ \ \ \ \ \ \ \ \ \ sched_priv_ Files containing the output of thread .br \ \ \ \ \ \ \ \ \ \ \ / apply all backtrace full on all core .br files captured from PBS_HOME/sched_priv .br \ \ \ \ \ \ \ \ \ \ \ server_priv/ Files containing the output of thread .br apply all backtrace full on all core .br files captured from .br PBS_HOME/server_priv .br \ \ \ \ \ \ \ \ \ \ \ mom_priv/ Files containing the output of thread .br apply all backtrace full on all core .br files captured from PBS_HOME/mom_priv .br \ \ \ \ \ \ \ \ \ \ \ misc/ Files containing the output of thread .br apply all backtrace full on any other .br core files found inside PBS_HOME .br system/ .br \ \ \ \ \ \ \ \ \ \ \ pbs_probe_v.out Output of pbs_probe -v .br \ \ \ \ \ \ \ \ \ \ \ pbs_hostn_v.out Output of pbs_hostn -v $(hostname) .br \ \ \ \ \ \ \ \ \ \ \ pbs_environment Copy of PBS_HOME/pbs_environment file .br \ \ \ \ \ \ \ \ \ \ \ os_info Information about the OS .br \ \ \ \ \ \ \ \ \ \ \ process_info List of processes running on the system .br when the snapshot was taken. Output of .br ps -aux | grep [p]bs on Linux systems, .br or tasklist /v on Windows systems .br \ \ \ \ \ \ \ \ \ \ \ ps_leaf.out Output of ps -leaf. Linux only. .br \ \ \ \ \ \ \ \ \ \ \ lsof_pbs.out Output of lsof | grep [p]bs. .br Linux only. .br \ \ \ \ \ \ \ \ \ \ \ etc_hosts Copy of /etc/hosts file. Linux only. .br \ \ \ \ \ \ \ \ \ \ \ etc_nsswitch_conf Copy of /etc/nsswitch.conf file. .br Linux only. .br \ \ \ \ \ \ \ \ \ \ \ vmstat.out Output of the command vmstat. .br Linux only. .br \ \ \ \ \ \ \ \ \ \ \ df_h.out Output of the command df -h. .br Linux only. .br \ \ \ \ \ \ \ \ \ \ \ dmesg.out Output of the dmesg command. .br Linux only. .br pbs.conf Copy of the pbs.conf file on the .br server host .br ctime Contains the time in seconds since epoch when the snapshot was taken .br pbs_snapshot.log Log messages written by .B pbs_snapshot .SH Examples .IP "pbs_snapshot -o /tmp" 5 Writes a snapshot to /temp/snapshot_.tgz that includes 30 days of accounting logs and 5 days of daemon logs from the server host. .IP "pbs_snapshot --daemon-logs=1 --accounting-logs=1 -o /tmp --obfuscate --map=mapfile.txt" 5 Writes a snapshot to /temp/snapshot_.tgz that includes 1 day of accounting and daemon logs. Obfuscates the data and stores the data mapping in the map file named "mapfile.txt".