Running Jupyter notebook on computing nodes

Jupyter can be launched from a job submitted on the computing clusters.

Prerequisites

  • This tutorial assumes your computer is running Linux or MacOS with an SSH client and a web browser.

It is also possible to use Windows, but you will need to adapt the SSH tunnel to work with Putty or another SSH client.

Installing Jupyter

You can install Jupyter via the [Nix] system (/en/hpc/softenv/nix/):

[user@f-dahu ~]$ source /applis/site/nix.sh
[user@f-dahu ~]$ nix-env --switch-profile $NIX_USER_PROFILE_DIR/jupyter
[bzizou@f-dahu ~]$ cat > ~/python.nix <<EOF
with import <nixpkgs> {};
pkgs.python37.withPackages (ps: with ps; [ ipython jupyter ])
EOF
[user@f-dahu ~]$ nix-env -i -f ~/python.nix
installing ‘python3-3.7.3-env’

You can also install via Conda:

source /applis/environments/conda.sh
conda create -n jupyter conda-forge::jupyter conda::forge::ipython

Configuring Jupyter

Generate a default configuration file :

[user@f-dahu ~]$ jupyter notebook --generate-config
Writing default config to: /home/user/.jupyter/jupyter_notebook_config.py

Don’t start a browser when the notebook server launches:

[user@f-dahu ~]$ echo "c.NotebookApp.open_browser = False" >> ~/.jupyter/jupyter_notebook_config.py

Listen on all network interfaces:

[user@f-dahu ~]$ echo "c.NotebookApp.ip = '0.0.0.0'" >> ~/.jupyter/jupyter_notebook_config.py

Start a job to run the Jupyter notebook server

Once the environment has been set up, you can start a notebook server on a compute node.

Start an interactive job on a single core

[user@f-dahu ~]$ oarsub -I --project test -l /core=1,walltime=2:00:00
[ADMISSION RULE] Modify resource description with type constraints
OAR_JOB_ID=14181444
Interactive mode : waiting...
[...]
Starting...

Initialize X11 forwarding...
Connect to OAR job 14181444 via the node dahu38

Your terminal is now connected to the compute node. Please note that interactive jobs are time-limited (walltime) and will be killed after this time.

Start the notebook server

[user@dahu103 ~]$ source /applis/site/nix.sh
[user@dahu103 ~]$ jupyter notebook
[I 15:48:01.768 NotebookApp] Writing notebook server cookie secret to /home/user/.local/tmp/jupyter/notebook_cookie_secret
[I 15:48:09.697 NotebookApp] Serving notebooks from local directory: /home/user
[I 15:48:09.697 NotebookApp] The Jupyter Notebook is running at:
[I 15:48:09.697 NotebookApp] http://(dahu38 or 127.0.0.1):8888/?token=524bfbd752f764ef447509bde0a5d8f9169ab603ce33f966
[I 15:48:09.697 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:48:09.727 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/user/.local/tmp/jupyter/nbserver-276827-open.html
    Or copy and paste one of these URLs:
        http://dahu103:8888/?token=524bfbd752f764ef447509bde0a5d8f9169ab603ce33f966
     or http://127.0.0.1:8888/?token=524bfbd752f764ef447509bde0a5d8f9169ab603ce33f966

Note the url to connect to the server (here http://dahu103:8888/...), which identifies the compute node the server is running on (here dahu103) and the port it is listening on (here 8888).

Open a notebook via an ssh tunnel

Open the ssh tunnel

In a terminal on your local machine, open the ssh tunnel:

[user@workstation ~]$ ssh -fNL 8888:node_name:8888 cluster_name.ciment
  • Replace node_name with the name of compute node running the job (here dahu103)
  • Replace cluster_name with the name of the cluster running the job (here dahu)
  • You may need to replace .ciment by whatver suffix you defined in your transparent ssh configuration.
  • If the server is not listening on port 8888, replace 8888 with whichever port it is listening on

Open the notebook

Open your local machine’s web browser (the machine from which you opened the ssh tunnel) and navigate to the notebook server url that starts with http://127.0.0.1:

http://127.0.0.1:8888/?token=524bfbd752f764ef447509bde0a5d8f9169ab603ce33f966

You may need to replace 127.0.0.1 with localhost.

You should now be connected to the jupyter server running on the compute node.