Running Jupyter-notebook on GPU nodes

Objective: we will learn how to:

  • Run Jupyter notebook with custom conda environment
  • Use custom conda environment within a Notebook.

First we request some resources using SGE:

  • 1 GPU
  • 20 GB memory
  • 2h walltime
$ qlogin -q volta.q -l h_vmem=20G -l h_rt=2:00:00

A new session well be opened in a GPU node.

We load anaconda 

$ module load anaconda

We create a new environment with a specific version of python. 

$ conda create -y --prefix $WORK/meso1 python=3.7

We check for our conda environment 

$ conda env list
meso1
meso2
...

We use meso1 for example 

$ conda activate meso1
(meso1)$

By default Jupyter notebook use $HOME directory to store runtime data.
Since $HOME directory is read only in compute nodes. We need to tell Jupyter to use another directory.

Modify Jupyter data directory (we use $WORK) 

(meso1)$ export JUPYTER_DATA_DIR=$WORK/.jupyter

Set this conda environment (meso1) to Jupyter notebook, we need to install ipykernel 

(meso1)$ conda install -c anaconda ipykernel

and add meso1 to Jupyter kernel 

$ (meso1)$ python -m ipykernel install --user --name=meso1

And finally start Jupyter notebook 

 jupyter-notebook --no-browser --ip=0.0.0.0

Where command options are:

  • –no-browser do not start the browser automatically
  • –ip=0.0.0.0 the server will listen on all interfaces
  • The system will pick a random port number to use.

We should have an output like this: 

[deleted lines]
...
[I 15:24:14.851 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 15:24:14.851 NotebookApp] http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
[I 15:24:14.851 NotebookApp]  or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
[I 15:24:14.851 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:24:14.860 NotebookApp] 
 
    To access the notebook, open this file in a browser:
        file:///Work/Users/kmazouzi/.jupyter/runtime/nbserver-42070-open.html
    Or copy and paste one of these URLs:
        http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
     or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
(meso1)$

There are three important pieces of information, above, to remember:

  • The node used by SGE, here node3-70
  • The port picked by Jupyter-nootbook, here 8888
  • The token of the session, here token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23

We can't connect directly to the link above because node3-70 is located in private network. We have to use ssh port tunneling.

SSH tunnel

We need to establish a tunnel from mesoshared to this node (node3-70).

From a new terminal on mesoshared: 

$ ssh -g  -N -L 9999:localhost:8888 kmazouzi@node3-70

This command will map remote port 8888 from node3-70 to local port 9999 in mesoshared (localhost).

We can use another value than 9999 for local port

Finally we will modify the URL above to connect to our Jupyter session.

OLD URL proposed by Jupyter 

  http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23\\

Replace node3-70:8888 by mesoshared.univ-fcomte.fr:9999

The new URL to use, from your browser 

  http://mesoshared.univ-fcomte.fr:9999/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23

Open a browser on local machine and browse the URL above.
Notice that we can open our conda environment (meso1) from the Notebook