ODU Research Computing Forum

Running a Python code on HPC through Jupyter

Data Analytics
,
#1

I have Python code that I would like to run on HPC. Can I run it through the Jupyter notebook? If so, how can I?

#2

Yes, … but may we recommend that you launch the script through the HPC job scheduler, called “SLURM”. Let’s call your script SCRIPT.py (include the full path if the script is not the same directory as where you want to run this). Here is a basic SLURM job script that may work for you:

#!/bin/bash
enable_lmod
module load container_env tensorflow-cpu/2.2.0
crun.tensorflow-cpu python3 SCRIPT.py

Important notes:

  • Replace SCRIPT.py with your actual script file name.
  • Your specific workload may need a GPU to run your calculation. In that case, please replace the words tensorflow-cpu with tensorflow-gpu everywhere and add the following two #SBATCH lines just after the first line of the script:
#SBATCH -p gpu
#SBATCH --gres gpu:1

Create this script using a text editor, save it to a file (say, name this JOB.sh). Then you will submit the script from the UNIX shell interface of the cluster by typing:

$ sbatch JOB.sh

(The “$” at the beginning of the line represents the shell prompt–do not type that.) If successful, there will be a message printed “Submitted batch job NNNNNN”, where NNNNNN is an integer called job ID or job number. The output file will be slurm-NNNNNN.out. If you desire a different output filename, consider adding this option:

#SBATCH --output=outputfile.txt

Here are a few documentations to help understand this process:

If you prefer a video introduction to SLURM, please take a look at this short video to understand what a job scheduler is: “Working with SLURM” (about 25 minutes). This is part of the Introduction to HPC workshop recording.

But I want to run my code in Jupyter, please…

Now, back to your original question: Yes, you can run the script from a notebook, but please read to the end. Once you opened a (blank) notebook, you will make this statement in a new cell:

%load SCRIPT.py

Press <Shift+Enter> once, the code will be loaded into that same cell. Press <Shift+Enter> once more, the code will execute. But please note that this mode of execution can be awkward, i.e. we have to open a Jupyter session, %load the script in order to run it, and even after that, the maximum time to run the script is under 24 hours. This mode is useful if you plan to interactively test or troubleshoot a script, but less useful when you have a bunch of calculations to do, or calculations that you expect to run for a long time.