Skip to end of metadata
Go to start of metadata


About GPU Resources in O2

The first 2 GPU nodes are now available on O2, including: 4 Tesla M40 and 8 Tesla K80 GPUs. To list information about all the nodes with GPU resources you can use the command: 

login01:~$ sinfo --Format=nodehost,cpusstate,memory,gres|grep 'HOSTNAMES\|gpu'
HOSTNAMES           CPUS(A/I/O/T)       MEMORY              GRES
compute-g-16-176    10/10/0/20          257548              gpu:teslaM40:4
compute-g-16-177    1/23/0/24           257548              gpu:teslaK80:8

GPU Partition Limits


The amount of GPU resources that can be used at any time in the O2 cluster is measured in term of GPU hours / user, currently there is an active limit of 72 GPU hours for each user.


This means that at any time each user can allocate* at most 1 GPU card for 72 hours, 12 GPU cards for 6 hours or any intermediate combination, for example 6 GPU cards for 12 hours. 


The current limit will be increased as we migrate additional GPU nodes from the older cluster to O2. 


* as resources allow 

How to compile cuda programs

In most cases a cuda library and compiler module must be loaded in order to compile cuda programs. To see which cuda modules are available use the command module spider cuda, then use the command module load to load the desired version. Currently only the latest version of Cuda toolkit (V 9) is available 


login01:~ module spider cuda

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  cuda: cuda/9.0
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "cuda/9.0" module is available to load.

      gcc/6.2.0

    Help:
      For detailed instructions, go to:
      https://developer.nvidia.com/about-cuda

login01:~ module load gcc/6.2.0 cuda/9.0


How to submit a GPU job

To submit a GPU job in O2 you will need to use the partition gpu and add the flag --gres=gpu:1 to request a GPU resource. The example below shows how to start an interactive bash job requesting 1 CPU core and 1 GPU card:

login01:~ srun -n 1 --pty -t 1:00:00 -p gpu --gres=gpu:1 bash

srun: job 6900282 queued and waiting for resources
srun: job 6900282 has been allocated resources
compute-g-16-176:~


While this other example shows how to submit a batch job requesting 2 GPU cards and 4 CPU cores:

login01:sbatch gpujob.sh
Submitted batch job 6900310


where gpujob.sh contains


#-----------------------------------------------------------------------------------------
#!/bin/bash
#SBATCH -c 4
#SBATCH -t 6:00:00
#SBATCH -p gpu
#SBATCH --gres=gpu:2

module load gcc/6.2.0
module load cuda/9.0

./deviceQuery  #this is just an example 


#-----------------------------------------------------------------------------------------


It is also possible to request a specific type of GPU card by using the --gres flag. For example --gres=gpu:teslaM40:3 can be used to request 3 GPU Tesla M40 cards. Currently two GPU types are available: teslaM40 and teslaK80





  • No labels