RC created two simplified commands, O2squeue and O2sacct based on slurm squeue and sacct, that can be used to gather information about your active (pending or running) jobs and your past jobs.
This command is based on slurm command squeue and will return information about your pending and running jobs. For example:
The field STATE describes the states of your jobs and it will normally be either PENDING or RUNNING. When a job is pending NODELIST(REASON) describes the reason why the job is pending, most common reasons are:
BadConstraints: The job's constraints can not be satisfied.
Dependency: This job is waiting for a dependent job to complete.
InvalidQOS: The job's QOS is invalid.
JobHeldAdmin: The job is held by a system administrator.
JobHeldUser: The job is held by the user.
None: The job has not been evaluated yet by the scheduler
Priority: One or more higher priority jobs exist for this partition or advanced reservation.
QOSJobLimit: The job's QOS has reached its maximum job count.
QOSResourceLimit: The job's QOS has reached some resource limit.
QOSTimeLimit: The job's QOS has reached its time limit.
ReqNodeNotAvail: Some node specifically required by the job is not currently available.
Reservation: The job is waiting its advanced reservation to become available.
Resources: The job is waiting for resources to become available.
The field ELIGIBLE_TIME indicates the time when a job becomes eligible to be dispatched, this is usually the submit time unless there are reason why the job cannot be dispatched such as job dependencies or unavailable resources requested.
For running jobs START_TIME indicates the time when the job was dispatched. For pending jobs it indicates the expected start time. Note that expected start time is only calculated for the first few pending jobs of each user and it is in general an upper bound value.
TRES indicates the resources requested by the job
O2squeue can take as inputs the string R or PD to selectively list only running or pending jobs
This command is based on slurm command sacct and will return information about your past jobs. For example:
The field CPUTime indicates the total amount of CPU hours that were reserved by the job, calculated as the product of Elapsed time and number of cores. The field TotalCPU indicates the amount of CPU hours that were actually used by the job. For an efficient job those two values should be very close, If TotalCPU is significantly smaller (<1/2) than CPUTime your job is probably requesting more cores than it uses.
AllocTRES reports the total amount of resources (cpu, memory, etc.) allocated for the job.
MaxRSS reports the maximum amount of memory used by the job, if this value is significantly smaller than the allocated memory reported by AllocTRES you should reduce the memory requested by your job. (Note: for mpi jobs this is the max amount of memory used in each node)
O2sacct can take as arguments a jobid or a start-time, an end-time and a job state. Times can be used to define a search interval and must be in the format YYYY-MM-DD or YYYY-MM-DDThh-mm-ss, if a time range is not specify only recent jobs are searched. If a end-time is not specified end-time is set to now, if a job state is specified a search time window must be provided and the comand will return jobs that were in the specified state during the given time interval.
Possible job states are:
CA = job cancelled
CD = job completed
F = job failed
NF = job failed due to Node failure
TO = job timeout
R = job running
PD = job pending
O2sacct CD 2018-04-01
O2sacct 2018-04-01 R
O2sacct 2018-04-01 2018-04-10
O2sacct 2018-04-01 2018-04-10 R
If you are running a large batch of jobs, and a limited number of them require more memory and/or wall time than the average, it is almost always better to request resources for the average jobs. The few jobs that fail for timeout and/or exceeded memory can then be resubmitted with increased memory and/or wall time. Limiting the amount of requested memory and wall time can result in a significant reduction of the pending time and a much faster processing time for the overall workflow.