The following steps will walk you through setting up a Anaconda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute note to run a Jupyter Notebook session on Caviness. Lastly it will explain how to set up ssh connections be able to connect to the Jupyter Notebook session running on a compute node. This setup will be done in the it_css
workgroup directory. This will allow all users in the it_css
workgroup to use the Jupyter Notebook virtual environment. If your workgroup has already created a Jupyter Notebook virtual enviornment and a corresponding VALET package, then proceed to using Jupyter Notebook Virtual Environment.
Before starting, make sure you set your workgroup and change to the directory where you would like to store your Jupyter Notebook virtual environment. In this example, a general directory anaconda-envs
will been created for Anaconda virtual environments that can be shared with everyone in the workgroup it_css
in the sw
directory.
sw
or sw/valet
directory, please consult with your PI (stakeholder of the workgroup) how to setup for your workgroup software installs Workgroup Directory on Caviness before proceeding with these instructions. On DARWIN, workgroup directories for sw
and sw/valet
are automatically created as part of the allocation.
[traine@login00 ~]$ workgroup -g it_css [(it_css:traine)@login00 ~]$ cd $WORKDIR/sw [(it_css:traine)@login00 it_css]$ mkdir anaconda-envs [(it_css:traine)@login00 it_css]$ chmod 02775 anaconda-envs
As stated above, in this example we will be using Anaconda version 5.2.0:python3
. This was the latest version available on the Caviness Cluster in June 2021.
(it_css:traine)@login00 ~$ vpkg_require anaconda/5.2.0:python3 Adding package `anaconda/5.2.0:python3` to your environment
After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it jupyter-notebook
, but you are welcome to call it anything you would like. Before creating it, we will want to see what "default" version of Python 3 is available on as part of Anaconda.
[(it_css:traine)@login00 ~]$ which python3 /opt/shared/anaconda/5.2.0-python3/bin/python3 [(it_css:traine)@login00 ~]$ python3 --version Python 3.6.5 :: Anaconda, Inc.
Now that we know the version of python3 loaded into our environment is 3.6.5
, we can run the commands to create virtual environment. We will also specify a version directory by date 20210601
for the installation of June 2021 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. You will be asked to Proceed ([y]/n)?
and you will want to chose y
as this is advising you that prerequisite software is also going to be installed.
[(it_css:traine)@login00 ~]$ conda create --prefix $WORKDIR/sw/anaconda-envs/jupyter-notebook/20210601 -c jupyter python=3.6.5 Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.5.11 latest version: 4.10.1 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /work/it_css/sw/anaconda-envs/jupyter-notebook/20210601 added / updated specs: - python=3.6.5 The following packages will be downloaded: package | build ---------------------------|----------------- _openmp_mutex-4.5 | 1_gnu 22 KB libstdcxx-ng-9.3.0 | hd4cf53a_17 4.0 MB certifi-2021.5.30 | py36h06a4308_0 141 KB pip-21.1.2 | py36h06a4308_0 2.1 MB libgomp-9.3.0 | h5101ec6_17 378 KB ca-certificates-2021.5.25 | h06a4308_1 118 KB libgcc-ng-9.3.0 | h5101ec6_17 7.8 MB ------------------------------------------------------------ Total: 14.6 MB The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-main _openmp_mutex: 4.5-1_gnu ca-certificates: 2021.5.25-h06a4308_1 certifi: 2021.5.30-py36h06a4308_0 libedit: 3.1.20210216-h27cfd23_1 libffi: 3.2.1-hf484d3e_1007 libgcc-ng: 9.3.0-h5101ec6_17 libgomp: 9.3.0-h5101ec6_17 libstdcxx-ng: 9.3.0-hd4cf53a_17 ncurses: 6.2-he6710b0_1 openssl: 1.0.2u-h7b6447c_0 pip: 21.1.2-py36h06a4308_0 python: 3.6.5-hc3d631a_2 readline: 7.0-h7b6447c_5 setuptools: 52.0.0-py36h06a4308_0 sqlite: 3.33.0-h62c20be_0 tk: 8.6.10-hbc83047_0 wheel: 0.36.2-pyhd3eb1b0_0 xz: 5.2.5-h7b6447c_0 zlib: 1.2.11-h7b6447c_3 Proceed ([y]/n)? y Downloading and Extracting Packages _openmp_mutex-4.5 | 22 KB | ####################################################################################################################################################################### | 100% libstdcxx-ng-9.3.0 | 4.0 MB | ####################################################################################################################################################################### | 100% certifi-2021.5.30 | 141 KB | ####################################################################################################################################################################### | 100% pip-21.1.2 | 2.1 MB | ####################################################################################################################################################################### | 100% libgomp-9.3.0 | 378 KB | ####################################################################################################################################################################### | 100% ca-certificates-2021 | 118 KB | ####################################################################################################################################################################### | 100% libgcc-ng-9.3.0 | 7.8 MB | ####################################################################################################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use: # > source activate /work/it_css/sw/anaconda-envs/jupyter-notebook/20210601 # # To deactivate an active environment, use: # > source deactivate # $
The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present:
[(it_css:traine)@login00 ~]$ ls -lad ${WORKDIR}/sw/valet [(it_css:traine)@login00 ~]$ echo ${WORKDIR}/sw/anaconda-envs/jupyter-notebook /work/it_css/sw/anaconda-envs/jupyter-notebook
Take note of the path echoed, then create a new file named ${WORKDIR}/sw/valet/jupyter-notebook.vpkg_yaml
and add the following text to it:
jupyter-notebook: prefix: /work/it_css/sw/anaconda-envs/jupyter-notebook description: Jupyter notebook in Python flags: - no-standard-paths actions: - action: source script: sh: anaconda-activate.sh order: failure-first success: 0 versions: "20210601": description: environment built June 1, 2021 dependencies: - anaconda/5.2.0:python3
prefix
show here /work/it_css/sw/anaconda-envs/jupyter-notebook
will need to be changed based on your workgroup and directory names.
The versions of the virtual environment declared in the VALET package are listed using the vpkg_versions
command:
[(it_css:traine)@login00 ~]$ vpkg_versions jupyter-notebook Available versions in package (* = default version): [/work/it_css/sw/valet/jupyter-notebook.vpkg_yaml] jupyter-notebook Jupyter notebook in Python * 20210601 environment built June 1, 2021
Activating the virtual environment is accomplished using the vpkg_require
command (in your login shell or inside job scripts):
[(it_css:traine)@login00 ~]$ vpkg_require jupyter-notebook/20210601 Adding dependency `anaconda/5.2.0:python3` to your environment Adding package `jupyter-notebook/20210601` to your environment (/work/it_css/sw/anaconda-envs/jupyter-notebook/20210601) [(it_css:traine)@login00 valet]$
Running Jupyter notebook on Caviness or any HPC cluster for that matter takes some extra steps. On your personal laptop you can simply install Jupyter notebook and start it up. On Caviness you need to run the Jupyter notebook on a compute node. The steps below will show you how to request a interactive compute node, and use VALET to load the Jupyter notebook virtual environment. After starting the virtual environment we will run Jupyter notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system.
In this example we will request an interactive compute node session with 2GB of memory for 1 hour on the it_css
workgroup partition. Also it is important to pass the SLURM_EXPORT_ENV=NONE
when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
[(it_css:traine)@login00 ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=it_css
After the interactive session has been established, it is time to load the Jupyter notebook virtual environment with VALET
. Even though we did this earlier, it needs to be done again since we are now on a compute node.
[[traine@r00n03 1201]$ vpkg_require jupyter-notebook/20210601 Adding dependency `anaconda/5.2.0:python3` to your environment Adding package `jupyter-notebook/20210601` to your environment (/work/it_css/sw/anaconda_envs/jupyter-notebook/20210601) [traine@r00n03 1201]$
(/work/it_css/sw/anaconda_envs/jupyter-notebook/20210601) [traine@r00n03 1201]$
When starting the Jupyter notebook session specific options will be passed which will be used for setting up the tunnel connection.
[traine@r04n68 1201]$ jupyter notebook --no-browser --ip=$(hostname -s) [I 09:00:22.644 NotebookApp] Writing notebook server cookie secret to /home/1201/.local/share/jupyter/runtime/notebook_cookie_secret [I 09:00:23.562 NotebookApp] JupyterLab beta preview extension loaded from /opt/shared/anaconda/5.2.0-python3/lib/python3.6/site-packages/jupyterlab [I 09:00:23.562 NotebookApp] JupyterLab application directory is /opt/shared/anaconda/5.2.0-python3/share/jupyter/lab [I 09:00:23.569 NotebookApp] Serving notebooks from local directory: /home/1201 [I 09:00:23.569 NotebookApp] 0 active kernels [I 09:00:23.569 NotebookApp] The Jupyter Notebook is running at: [I 09:00:23.569 NotebookApp] http://r04n68:8888/?token=689083c3b9de8f81c6955663c8d09ca8053b4672d8a5f83b [I 09:00:23.569 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 09:00:23.570 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://r04n68:8888/?token=689083c3b9de8f81c6955663c8d09ca8053b4672d8a5f83b&token=689083c3b9de8f81c6955663c8d09ca8053b4672d8a5f83b
localhost
http://localhost:8888/?token=689083c3b9de8f81c6955663c8d09ca8053b4672d8a5f83b
With the Jupyter Notebook server running on a compute node on Caviness, an SSH tunnel is needed to be able to make a connection and access the Jupyter Notebook server from a web browser on your local machine. This is done by opening a second SSH connection to Caviness. Follow the appropriate section below for Windows (PuTTY) or Terminal on a Linux/Mac laptop.
r04n68
, please make sure to change that accordingly. This SSH tunnel connection will have to remain open while you are using Jupyter Notebook. If it is closed or internet connectivity is lost, then your connection to Jupyter Notebook will also be lost.
Once you open the PuTTY, it will show the Session window. For the Host Name(or IP Address), you will need to enter the <user-name>@caviness.hpc.udel.edu
.
In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then add the tunnel settings based on the image below. The tunnel setting is found under the Category Connection → SSH → Tunnels
Add the Source port as 8888
and Destination as r04n68:8888
as shown in the image above, then click on Add. The Tunnel settings are now available to your session, so click Open to connect. You may need to enter the password for the Caviness/DARWIN account in the prompted window if you do not have an existing session. These settings are not saved, however it is likely the necessary information to setup the tunnel the next time will change anyway. Remember, this ssh tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
Open a new terminal session on your local machine. Set up a SSH Tunnel using the below ssh
command.
$ ssh -L 8888:r04n68:8888 traine@caviness.hpc.udel.edu ...................................................................... Caviness cluster (caviness.hpc.udel.edu) This computer system is maintained by University of Delaware IT. Links to documentation and other online resources can be found at: http://docs.hpc.udel.edu/abstract/caviness/ For support, please contact consult@udel.edu ...................................................................... Last login: Thu Aug 6 12:28:19 2021 [traine@login00 ~]$
If everything this far has been set up correctly the final step is as easy as opening a web browser of choice on your local machine and entering the correct URL. If you followed the directions exactly, you can now use the URL from above changing the compute node to localhost
in your local browser to connect to Jupyter Notebook server running on the compute node on Caviness.
Example URL:
http://localhost:8888/?token=c002318e20dc522585e82af06535dce9293be403636a62df