Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
technical:recipes:jupyter-notebook [2021-10-01 12:20] – [Jupyter Notebook Python Virtual Environment] anita | technical:recipes:jupyter-notebook [2024-01-19 09:23] (current) – [Setting up ssh tunnel to connect to Jupyter Notebook server] thuachen | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Jupyter Notebook Python Virtual Environment====== | ||
+ | |||
+ | The following steps will walk you through setting up a Anaconda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute note to run a Jupyter Notebook session on Caviness. Lastly it will explain how to set up ssh connections be able to connect to the Jupyter Notebook session running on a compute node. This setup will be done in the '' | ||
+ | |||
+ | <note tip> | ||
+ | |||
+ | Before starting, make sure you set your workgroup and change to the directory where you would like to store your Jupyter Notebook virtual environment. In this example, a general directory '' | ||
+ | |||
+ | <note warning> | ||
+ | |||
+ | <code bash> | ||
+ | [traine@login00 ~]$ workgroup -g it_css | ||
+ | [(it_css: | ||
+ | [(it_css: | ||
+ | [(it_css: | ||
+ | </ | ||
+ | |||
+ | ===== Create Jupyter Notebook Virtual Environment | ||
+ | As stated above, in this example we will be using Anaconda version '' | ||
+ | |||
+ | <code bash> | ||
+ | (it_css: | ||
+ | Adding package `anaconda/ | ||
+ | </ | ||
+ | |||
+ | After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it '' | ||
+ | |||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | / | ||
+ | [(it_css: | ||
+ | Python 3.6.5 :: Anaconda, Inc. | ||
+ | </ | ||
+ | |||
+ | Now that we know the version of python3 loaded into our environment is '' | ||
+ | |||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | Solving environment: | ||
+ | |||
+ | |||
+ | ==> WARNING: A newer version of conda exists. <== | ||
+ | current version: 4.5.11 | ||
+ | latest version: 4.10.1 | ||
+ | |||
+ | Please update conda by running | ||
+ | |||
+ | $ conda update -n base -c defaults conda | ||
+ | |||
+ | |||
+ | |||
+ | ## Package Plan ## | ||
+ | |||
+ | environment location: / | ||
+ | |||
+ | added / updated specs: | ||
+ | - python=3.6.5 | ||
+ | |||
+ | |||
+ | The following packages will be downloaded: | ||
+ | |||
+ | package | ||
+ | ---------------------------|----------------- | ||
+ | _openmp_mutex-4.5 | ||
+ | libstdcxx-ng-9.3.0 | ||
+ | certifi-2021.5.30 | ||
+ | pip-21.1.2 | ||
+ | libgomp-9.3.0 | ||
+ | ca-certificates-2021.5.25 | ||
+ | libgcc-ng-9.3.0 | ||
+ | ------------------------------------------------------------ | ||
+ | | ||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | _libgcc_mutex: | ||
+ | _openmp_mutex: | ||
+ | ca-certificates: | ||
+ | certifi: | ||
+ | libedit: | ||
+ | libffi: | ||
+ | libgcc-ng: | ||
+ | libgomp: | ||
+ | libstdcxx-ng: | ||
+ | ncurses: | ||
+ | openssl: | ||
+ | pip: | ||
+ | python: | ||
+ | readline: | ||
+ | setuptools: | ||
+ | sqlite: | ||
+ | tk: 8.6.10-hbc83047_0 | ||
+ | wheel: | ||
+ | xz: 5.2.5-h7b6447c_0 | ||
+ | zlib: 1.2.11-h7b6447c_3 | ||
+ | |||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | |||
+ | Downloading and Extracting Packages | ||
+ | _openmp_mutex-4.5 | ||
+ | libstdcxx-ng-9.3.0 | ||
+ | certifi-2021.5.30 | ||
+ | pip-21.1.2 | ||
+ | libgomp-9.3.0 | ||
+ | ca-certificates-2021 | 118 KB | ####################################################################################################################################################################### | ||
+ | libgcc-ng-9.3.0 | ||
+ | Preparing transaction: | ||
+ | Verifying transaction: | ||
+ | Executing transaction: | ||
+ | # | ||
+ | # To activate this environment, | ||
+ | # > source activate / | ||
+ | # | ||
+ | # To deactivate an active environment, | ||
+ | # > source deactivate | ||
+ | # | ||
+ | $ | ||
+ | </ | ||
+ | |||
+ | ==== VALET Package Definition ==== | ||
+ | |||
+ | The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present: | ||
+ | |||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | [(it_css: | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | Take note of the path echoed, then create a new file named '' | ||
+ | |||
+ | <code yaml> | ||
+ | jupyter-notebook: | ||
+ | prefix: / | ||
+ | description: | ||
+ | flags: | ||
+ | - no-standard-paths | ||
+ | actions: | ||
+ | - action: source | ||
+ | script: | ||
+ | sh: anaconda-activate.sh | ||
+ | order: failure-first | ||
+ | success: 0 | ||
+ | versions: | ||
+ | " | ||
+ | description: | ||
+ | dependencies: | ||
+ | - anaconda/ | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | |||
+ | ===== Using Jupyter Notebook Virtual Environment ===== | ||
+ | |||
+ | The versions of the virtual environment declared in the VALET package are listed using the '' | ||
+ | |||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | |||
+ | Available versions in package (* = default version): | ||
+ | |||
+ | [/ | ||
+ | jupyter-notebook | ||
+ | * 20210601 | ||
+ | </ | ||
+ | |||
+ | Activating the virtual environment is accomplished using the '' | ||
+ | |||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | Adding dependency `anaconda/ | ||
+ | Adding package `jupyter-notebook/ | ||
+ | (/ | ||
+ | </ | ||
+ | |||
+ | ===== Running Jupyter Notebook on the Caviness Cluster ===== | ||
+ | |||
+ | Running Jupyter notebook on Caviness or any HPC cluster for that matter takes some extra steps. On your personal laptop you can simply install Jupyter notebook and start it up. On Caviness you need to run the Jupyter notebook on a compute node. The steps below will show you how to request a interactive compute node, and use VALET to load the Jupyter notebook virtual environment. After starting the virtual environment we will run Jupyter notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system. | ||
+ | |||
+ | <note warning> | ||
+ | |||
+ | ==== Requesting an interactive compute node ==== | ||
+ | In this example we will request an interactive compute node session with 2GB of memory for 1 hour on the '' | ||
+ | <code bash> | ||
+ | [(it_css: | ||
+ | </ | ||
+ | |||
+ | ==== Loading the Jupyter notebook virtual environment ==== | ||
+ | After the interactive session has been established, | ||
+ | <code bash> | ||
+ | [[traine@r00n03 1201]$ vpkg_require jupyter-notebook/ | ||
+ | Adding dependency `anaconda/ | ||
+ | Adding package `jupyter-notebook/ | ||
+ | (/ | ||
+ | </ | ||
+ | |||
+ | <note tip>Once the Jupyter noteboook virtual environment had been loaded on the interactive compute node session, your prompt should look something like: | ||
+ | |||
+ | '' | ||
+ | |||
+ | </ | ||
+ | ==== Starting Jupyter notebook session ==== | ||
+ | When starting the Jupyter notebook session specific options will be passed which will be used for setting up the tunnel connection. | ||
+ | |||
+ | <code bash> | ||
+ | [traine@r04n68 1201]$ jupyter notebook --no-browser --ip=$(hostname -s) | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [I 09: | ||
+ | [C 09: | ||
+ | |||
+ | Copy/paste this URL into your browser when you connect for the first time, | ||
+ | to login with a token: | ||
+ | http:// | ||
+ | </ | ||
+ | |||
+ | <note warning> | ||
+ | The Jupyter Notebook server is running and there is no prompt. **Make sure you copy the line noted above and it will be used later by changing the compute node to '' | ||
+ | < | ||
+ | http:// | ||
+ | </ | ||
+ | |||
+ | </ | ||
+ | ==== Setting up SSH tunnel to connect to Jupyter Notebook server ==== | ||
+ | With the Jupyter Notebook server running on a compute node on Caviness, an SSH tunnel is needed to be able to make a connection and access the Jupyter Notebook server from a web browser on your local machine. This is done by opening a **second** SSH connection to Caviness. Follow the appropriate section below for Windows (PuTTY) or Terminal on a Linux/Mac laptop. | ||
+ | |||
+ | <note warning> | ||
+ | |||
+ | ==Windows (PuTTY)== | ||
+ | Once you open the PuTTY, it will show the //Session// window. For the //Host Name(or IP Address)//, you will need to enter the ''< | ||
+ | In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then add the tunnel settings based on the image below. The tunnel setting is found under the Category // | ||
+ | |||
+ | {{: | ||
+ | |||
+ | Add the **Source port** as '' | ||
+ | {{ : | ||
+ | ==Linux/ | ||
+ | Open a new terminal session on your local machine. Set up a SSH Tunnel using the below '' | ||
+ | < | ||
+ | |||
+ | $ ssh -L 8888: | ||
+ | ...................................................................... | ||
+ | |||
+ | |||
+ | Caviness cluster (caviness.hpc.udel.edu) | ||
+ | |||
+ | This computer system is maintained by University of Delaware | ||
+ | IT. Links to documentation and other online resources can be | ||
+ | found at: | ||
+ | |||
+ | http:// | ||
+ | |||
+ | For support, please contact consult@udel.edu | ||
+ | |||
+ | |||
+ | ...................................................................... | ||
+ | |||
+ | Last login: Thu Aug 6 12:28:19 2021 | ||
+ | [traine@login00 ~]$ | ||
+ | </ | ||
+ | ==== Using your local browser to access Jupyter Notebook server | ||
+ | If everything this far has been set up correctly the final step is as easy as opening a web browser of choice on your local machine and entering the correct URL. If you followed the directions exactly, you can now use the URL from above changing the compute node to '' | ||
+ | |||
+ | **Example URL:** | ||
+ | '' | ||
+ | |||
+ | <note important> | ||
+ | |||
+ | |||
+ | |||
+ | |||