This is an old revision of the document!
Jupyter Notebook Python Virtual Environment
The following steps will walk you through setting up an Anaconda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute note to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session running on a compute node. This setup will be done in the it_css workgroup directory. This will allow all users in the it_css workgroup to use the Jupyter Notebook virtual environment. If your workgroup has already created a Jupyter Notebook virtual environment and a corresponding VALET package, then proceed to using Jupyter Notebook Virtual Environment.
Before starting, make sure you set your workgroup and change to the directory where you would like to store your Jupyter Notebook virtual environment. In this example, a general directory anaconda-envs will be created for Anaconda virtual environments that can be shared with everyone in the workgroup it_css in the sw directory.
sw or sw/valet directory, please consult with your PI (stakeholder of the workgroup) on how to setup for your workgroup software installs Workgroup Directory on Caviness before proceeding with these instructions. On DARWIN, workgroup directories for sw and sw/valet are automatically created as part of the allocation.
[traine@login00 ~]$ workgroup -g it_css [(it_css:traine)@login00 ~]$ cd $WORKDIR/sw [(it_css:traine)@login00 it_css]$ mkdir anaconda-envs [(it_css:traine)@login00 it_css]$ chmod 02775 anaconda-envs
Create Jupyter Notebook Virtual Environment
As stated above, in this example we will be using Anaconda version 2024.02. This was the latest version available on the Caviness Cluster in August 2024.
(it_css:traine)@login00 ~$ vpkg_require anaconda/2024.02 Adding package `anaconda/2024.02` to your environment
After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it jupyter-notebook, but you are welcome to call it anything you would like. We will specify a version directory by date 20240801 for the installation of the August 2024 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. You will be asked to Proceed ([y]/n)? and you will want to choose y as this is advising you that prerequisite software is also going to be installed.
[(it_css:traine)@login00 ~]$ conda create --prefix $WORKDIR/sw/anaconda-envs/jupyter-notebook/20240801 -c jupyter python=3 Channels: - jupyter - conda-forge - nodefaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /work/it_css/sw/anaconda-envs/jupyter-notebook/20240801 added / updated specs: - python=3 The following packages will be downloaded: package | build ---------------------------|----------------- _libgcc_mutex-0.1 | conda_forge 3 KB conda-forge _openmp_mutex-4.5 | 2_gnu 23 KB conda-forge bzip2-1.0.8 | h4bc722e_7 247 KB conda-forge ca-certificates-2025.1.31 | hbcca054_0 154 KB conda-forge ld_impl_linux-64-2.43 | h712a8e2_2 654 KB conda-forge libexpat-2.6.4 | h5888daf_0 72 KB conda-forge libffi-3.4.2 | h7f98852_5 57 KB conda-forge libgcc-14.2.0 | h77fa898_1 829 KB conda-forge libgcc-ng-14.2.0 | h69a702a_1 53 KB conda-forge libgomp-14.2.0 | h77fa898_1 450 KB conda-forge liblzma-5.6.4 | hb9d3cd8_0 109 KB conda-forge libmpdec-4.0.0 | h4bc722e_0 88 KB conda-forge libsqlite-3.48.0 | hee588c1_1 858 KB conda-forge libuuid-2.38.1 | h0b41bf4_0 33 KB conda-forge libzlib-1.3.1 | hb9d3cd8_2 60 KB conda-forge ncurses-6.5 | h2d0b736_3 871 KB conda-forge openssl-3.4.0 | h7b32b05_1 2.8 MB conda-forge pip-25.0 | pyh145f28c_0 1.2 MB conda-forge python-3.13.1 |ha99a958_105_cp313 31.6 MB conda-forge python_abi-3.13 | 5_cp313 6 KB conda-forge readline-8.2 | h8228510_1 275 KB conda-forge tk-8.6.13 |noxft_h4845f30_101 3.2 MB conda-forge tzdata-2025a | h78e105d_0 120 KB conda-forge ------------------------------------------------------------ Total: 43.6 MB The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu bzip2 conda-forge/linux-64::bzip2-1.0.8-h4bc722e_7 ca-certificates conda-forge/linux-64::ca-certificates-2025.1.31-hbcca054_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.43-h712a8e2_2 libexpat conda-forge/linux-64::libexpat-2.6.4-h5888daf_0 libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5 libgcc conda-forge/linux-64::libgcc-14.2.0-h77fa898_1 libgcc-ng conda-forge/linux-64::libgcc-ng-14.2.0-h69a702a_1 libgomp conda-forge/linux-64::libgomp-14.2.0-h77fa898_1 liblzma conda-forge/linux-64::liblzma-5.6.4-hb9d3cd8_0 libmpdec conda-forge/linux-64::libmpdec-4.0.0-h4bc722e_0 libsqlite conda-forge/linux-64::libsqlite-3.48.0-hee588c1_1 libuuid conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0 libzlib conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2 ncurses conda-forge/linux-64::ncurses-6.5-h2d0b736_3 openssl conda-forge/linux-64::openssl-3.4.0-h7b32b05_1 pip conda-forge/noarch::pip-25.0-pyh145f28c_0 python conda-forge/linux-64::python-3.13.1-ha99a958_105_cp313 python_abi conda-forge/linux-64::python_abi-3.13-5_cp313 readline conda-forge/linux-64::readline-8.2-h8228510_1 tk conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101 tzdata conda-forge/noarch::tzdata-2025a-h78e105d_0 Proceed ([y]/n)? y Downloading and Extracting Packages ... ... Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use: # > conda activate /work/it_css/sw/anaconda-envs/jupyter-notebook/20240801 # # To deactivate an active environment, use: # > conda deactivate # $
VALET Package Definition
The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present:
[(it_css:traine)@login00 ~]$ ls -lad ${WORKDIR}/sw/valet [(it_css:traine)@login00 ~]$ echo ${WORKDIR}/sw/anaconda-envs/jupyter-notebook /work/it_css/sw/anaconda-envs/jupyter-notebook
Take note of the path echoed, then create a new file named ${WORKDIR}/sw/valet/jupyter-notebook.vpkg_yaml and add the following text to it:
jupyter-notebook: prefix: /work/it_css/sw/anaconda-envs/jupyter-notebook description: Jupyter notebook in Python flags: - no-standard-paths actions: - action: source script: sh: anaconda-activate.sh order: failure-first success: 0 versions: "20240801": description: environment built August 1, 2024 dependencies: - anaconda/2024.02
prefix show here /work/it_css/sw/anaconda-envs/jupyter-notebook will need to be changed based on your workgroup and directory names.
Using Jupyter Notebook Virtual Environment
The versions of the virtual environment declared in the VALET package are listed using the vpkg_versions command:
[(it_css:traine)@login00 ~]$ vpkg_versions jupyter-notebook Available versions in package (* = default version): [/work/it_css/sw/valet/jupyter-notebook.vpkg_yaml] jupyter-notebook Jupyter notebook in Python * 20240801 environment built August 1, 2024
Activating the virtual environment is accomplished using the vpkg_require command (in your login shell or inside job scripts):
[(it_css:traine)@login00 ~]$ vpkg_require jupyter-notebook/20240801 Adding dependency `anaconda/2024.02` to your environment Adding package `jupyter-notebook/20240801` to your environment (/work/it_css/sw/anaconda-envs/jupyter-notebook/20240801) [(it_css:traine)@login00 valet]$
Running Jupyter Notebook on the Caviness Cluster
Running Jupyter Notebook on Caviness, or any HPC cluster for that matter, takes some extra steps. You can simply install Jupyter Notebook on your personal laptop and start it up. On Caviness, you need to run the Jupyter Notebook on a compute node. The steps below will show you how to request an interactive compute node and use VALET to load the Jupyter Notebook virtual environment. After starting the virtual environment, we will run Jupyter Notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system.
Requesting an interactive job
In this example we will request an interactive job to connect us to a compute node with 2GB of memory for 1 hour on the it_css workgroup partition. Also it is important to pass the SLURM_EXPORT_ENV=NONE when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
[(it_css:traine)@login00 ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=it_css
Loading the Jupyter Notebook virtual environment
After the interactive job has been established, it is time to load the Jupyter Notebook virtual environment with VALET. Even though we did this earlier, it needs to be done again since we are now on a compute node.
[[traine@r00n50 1201]$ vpkg_require jupyter-notebook/20240801 Adding dependency `anaconda/2024.02` to your environment Adding package `jupyter-notebook/20240801` to your environment (/work/it_css/sw/anaconda_envs/jupyter-notebook/20240801) [traine@r00n50 1201]$
(/work/it_css/sw/anaconda_envs/jupyter-notebook/20240801) [traine@r00n50 1201]$
Starting Jupyter Notebook session
When starting the Jupyter Notebook session, specific options are passed, which are used to set up the tunnel connection.
[traine@r04n68 1201]$ jupyter notebook --no-browser --ip=$(hostname -s) ... ... ... [I 2024-08-01 15:16:27.747 LabApp] JupyterLab extension loaded from /opt/shared/anaconda/2024.02/lib/python3.11/site-packages/jupyterlab [I 2024-08-01 15:16:27.747 LabApp] JupyterLab application directory is /opt/shared/anaconda/2024.02/share/jupyter/lab [I 2024-08-01 15:16:27.748 LabApp] Extension Manager is 'pypi'. [I 2024-08-01 15:16:27.751 ServerApp] jupyterlab | extension was successfully loaded. [I 2024-08-01 15:16:27.756 ServerApp] notebook | extension was successfully loaded. [I 2024-08-01 15:16:27.757 ServerApp] panel.io.jupyter_server_extension | extension was successfully loaded. [I 2024-08-01 15:16:27.760 ServerApp] Serving notebooks from local directory: /home/3347 [I 2024-08-01 15:16:27.760 ServerApp] Jupyter Server 2.10.0 is running at: [I 2024-08-01 15:16:27.761 ServerApp] http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2 [I 2024-08-01 15:16:27.761 ServerApp] http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2 [I 2024-08-01 15:16:27.761 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 2024-08-01 15:16:27.765 ServerApp] To access the server, open this file in a browser: file:///home/1201/.local/share/jupyter/runtime/jpserver-20300-open.html Or copy and paste one of these URLs: http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2 http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
localhost
http://localhost:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
Setting up SSH tunnel to connect to Jupyter Notebook server
With the Jupyter Notebook server running on a compute node on Caviness, an SSH tunnel is needed to be able to make a connection and access the Jupyter Notebook server from a web browser on your local machine. This is done by opening a second SSH connection to Caviness. Follow the appropriate section below for Windows (PuTTY) or Terminal on a Linux/Mac laptop.
r00n50, please make sure to change that accordingly. This SSH tunnel connection will have to remain open while you are using Jupyter Notebook. If it is closed or internet connectivity is lost, then your connection to Jupyter Notebook will also be lost.
Windows (PuTTY)
Once you open the PuTTY, it will show the Session window. For the Host Name(or IP Address), you will need to enter the <user-name>@caviness.hpc.udel.edu.
In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then adding the tunnel settings based on the image below. The tunnel setting is found under the Category Connection → SSH → Tunnels
Add the Source port as 8888 and Destination as r00n50:8888 as shown in the image above, then click on Add. The Tunnel settings are now available to your session, so click Open to connect. You may need to enter the password for the Caviness/DARWIN account in the prompted window if you do not have an existing session. These settings are not saved. However, it is likely the necessary information to set up the tunnel the next time will change anyway. Remember, this SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
Linux/Mac
Open a new terminal session on your local machine. Set up a SSH Tunnel using the below ssh command.
$ ssh -L 8888:r00n50:8888 traine@caviness.hpc.udel.edu
......................................................................
Caviness cluster (caviness.hpc.udel.edu)
This computer system is maintained by University of Delaware
IT. Links to documentation and other online resources can be
found at:
http://docs.hpc.udel.edu/abstract/caviness/
For support, please contact consult@udel.edu
......................................................................
Last login: Thu Aug 1 12:28:19 2024
[traine@login00 ~]$
Using your local browser to access Jupyter Notebook server
If everything this far has been set up correctly the final step is as easy as opening a web browser of choice on your local machine and entering the correct URL. If you followed the directions exactly, you can now use the URL from above changing the compute node to localhost in your local browser to connect to Jupyter Notebook server running on the compute node on Caviness.
Example URL:
http://localhost:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2