software:juypter:caviness

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
software:juypter:caviness [2026-03-26 16:34] – Add VALET package definition and activation mbottosoftware:juypter:caviness [2026-03-27 15:33] (current) – Add output from ls command on workgroup VALET directory mbotto
Line 1: Line 1:
-====== Jupyter Notebook Python Virtual Environment on Caviness======+======Jupyter Notebook Python Virtual Environment on Caviness======
 <note warning> <note warning>
 This page is under construction. This page is under construction.
 </note> </note>
  
-The following steps will walk you through setting up a Conda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute node to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session running on a compute node.+The following steps will walk you through setting up a Conda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute node to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session from the browser on your local machine.
  
-Before starting, make sure you set your workgroup. This example will demonstrate how to create a shared directory in your workgroup storage where you can create Conda environments that are accessible to everyone in your workgroup.+Before starting, make sure you [[abstract:caviness:app_dev:compute_env#setting-workgroup|set your workgroup]] on Caviness. This example will demonstrate how to create a shared directory in your workgroup storage where you can create Conda environments that are accessible to everyone in your workgroup.
  
 <note warning> <note warning>
Line 27: Line 27:
 </note> </note>
  
-=====Create Jupyter Notebook Virtual Environment=====+=====Create Jupyter Notebook Virtual Environment with Conda=====
  
 This example will use the Miniforge VALET package to enable usage of the Conda package manager. If you are new to Conda, make sure to review the documentation for [[software:miniforge:caviness|Miniforge on Caviness]]. This example will use the Miniforge VALET package to enable usage of the Conda package manager. If you are new to Conda, make sure to review the documentation for [[software:miniforge:caviness|Miniforge on Caviness]].
Line 73: Line 73:
 # To activate this environment, use # To activate this environment, use
 # #
-#     $ conda activate /work/it_css/sw/miniforge-testing/jupyter-notebook/20260326+#     $ conda activate /work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326
 # #
 # To deactivate an active environment, use # To deactivate an active environment, use
Line 80: Line 80:
 </code> </code>
  
-=====VALET Package Definition=====+=====VALET Package Definition for the Virtual Environment=====
  
 The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present, and get the path to the parent directory of your Jupyter Notebook virtual environment: The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present, and get the path to the parent directory of your Jupyter Notebook virtual environment:
Line 86: Line 86:
 <code bash> <code bash>
 [(my_workgroup:user)@login00.caviness ~]$ ls -d ${WORKDIR_SW}/valet [(my_workgroup:user)@login00.caviness ~]$ ls -d ${WORKDIR_SW}/valet
 +/work/my_workgroup/sw/valet
 [(my_workgroup:user)@login00.caviness ~]$ echo ${WORKDIR_SW}/conda-envs/jupyter-notebook [(my_workgroup:user)@login00.caviness ~]$ echo ${WORKDIR_SW}/conda-envs/jupyter-notebook
 /work/my_workgroup/sw/conda-envs/jupyter-notebook /work/my_workgroup/sw/conda-envs/jupyter-notebook
Line 126: Line 127:
 Activating the virtual environment is accomplished using the ''vpkg_require'' command (in your login shell or inside job scripts). If you load the VALET package without having first loaded the ''miniforge'' package, ''miniforge'' will also be added as a dependency: Activating the virtual environment is accomplished using the ''vpkg_require'' command (in your login shell or inside job scripts). If you load the VALET package without having first loaded the ''miniforge'' package, ''miniforge'' will also be added as a dependency:
  
-<code>+<code bash>
 [(my_workgroup:user)@login00.caviness ~]$ vpkg_require jupyter-notebook/20260326 [(my_workgroup:user)@login00.caviness ~]$ vpkg_require jupyter-notebook/20260326
 Adding dependency `miniforge/25.11.0-1` to your environment Adding dependency `miniforge/25.11.0-1` to your environment
 Adding package `jupyter-notebook/20260326` to your environment Adding package `jupyter-notebook/20260326` to your environment
-(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness valet]$+(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness ~]$
 </code> </code>
  
 +You can deactivate the virtual environment and remove the VALET packages it depends on with:
 +<code bash>
 +(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness ~]$ vpkg_rollback
 +[(my_workgroup:user)@login00.caviness ~]$
 +</code>
 +
 +===== Running Jupyter Notebook on Caviness =====
 +
 +Running Jupyter Notebook on Caviness, or any HPC cluster for that matter, takes some extra steps. You can simply install Jupyter Notebook on your personal laptop and start it up. On Caviness, you need to run the Jupyter Notebook on a compute node. The steps below will show you how to request an interactive compute node and use VALET to load the Jupyter Notebook virtual environment. After starting the virtual environment, we will run Jupyter Notebook with arguments that will allow for the session to be accessed via an SSH tunnel connection on your local system. 
 +
 +<note warning>
 +On Caviness you're **required** to run Jupyter notebook on a **compute node**. If you run it on the login node, you could cause slowness or other issues with the login node, and IT might kill your Jupyter Notebook session without warning.
 +</note>
 +
 +====Requesting an Interactive Job====
 +
 +In this example we will request an interactive job to connect us to a compute node with 2GB of memory for 1 hour on the ''my_workgroup'' workgroup partition (replace ''my_workgroup'' with the name of your workgroup). Also it is important to pass the ''SLURM_EXPORT_ENV=NONE'' when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
 +<code bash>
 +[(my_workgroup:user)@login00.caviness ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=my_workgroup
 +</code>
 +
 +==== Loading the Jupyter Notebook virtual environment ====
 +After the interactive job has been established, it is time to load the Jupyter Notebook virtual environment with ''VALET''. Even though we did this earlier, it needs to be done again since we are now on a compute node. 
 +
 +<code bash>
 +[(my_workgroup:user)@r05n17 ~]$ vpkg_require jupyter-notebook/20260326
 +Adding dependency `miniforge/25.11.0-1` to your environment
 +Adding package `jupyter-notebook/20260326` to your environment
 +(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@r05n17 ~]$
 +</code>
 +
 +====Starting a Jupyter Notebook session====
 +
 +When starting the Jupyter Notebook session, specific options are passed which are used to set up the tunnel connection. You should see output similar to the following output:
 +
 +<code bash>
 +(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@r05n17 ~]$ jupyter notebook --no-browser --ip=$(hostname -s)
 +[I 2026-03-27 14:42:42.749 ServerApp] jupyter_lsp | extension was successfully linked.
 +[I 2026-03-27 14:42:42.756 ServerApp] jupyter_server_terminals | extension was successfully linked.
 +[I 2026-03-27 14:42:42.763 ServerApp] jupyterlab | extension was successfully linked.
 +[I 2026-03-27 14:42:42.769 ServerApp] notebook | extension was successfully linked.
 +[I 2026-03-27 14:42:43.761 ServerApp] notebook_shim | extension was successfully linked.
 +[I 2026-03-27 14:42:43.820 ServerApp] notebook_shim | extension was successfully loaded.
 +[I 2026-03-27 14:42:43.824 ServerApp] jupyter_lsp | extension was successfully loaded.
 +[I 2026-03-27 14:42:43.826 ServerApp] jupyter_server_terminals | extension was successfully loaded.
 +[I 2026-03-27 14:42:43.843 LabApp] JupyterLab extension loaded from /work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326/lib/python3.14/site-packages/jupyterlab
 +[I 2026-03-27 14:42:43.843 LabApp] JupyterLab application directory is /work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326/share/jupyter/lab
 +[I 2026-03-27 14:42:43.844 LabApp] Extension Manager is 'pypi'.
 +[I 2026-03-27 14:42:44.007 ServerApp] jupyterlab | extension was successfully loaded.
 +[I 2026-03-27 14:42:44.013 ServerApp] notebook | extension was successfully loaded.
 +[I 2026-03-27 14:42:44.016 ServerApp] Serving notebooks from local directory: /home/4296
 +[I 2026-03-27 14:42:44.017 ServerApp] Jupyter Server 2.17.0 is running at:
 +[I 2026-03-27 14:42:44.017 ServerApp] http://r05n17:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +[I 2026-03-27 14:42:44.017 ServerApp]     http://127.0.0.1:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +[I 2026-03-27 14:42:44.017 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
 +[C 2026-03-27 14:42:44.021 ServerApp]
 +
 +    To access the server, open this file in a browser:
 +        file:/home/4296/.local/share/jupyter/runtime/jpserver-361926-open.html
 +    Or copy and paste one of these URLs:
 +        http://r05n17:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +        http://127.0.0.1:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +[I 2026-03-27 14:42:44.079 ServerApp] Skipped non-installed server(s): basedpyright, bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyrefly, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
 +</code>
 +
 +<note warning>
 +At this point you will not be prompted for any further input, but the Jupyter Notebook server is running. Note the following URL in the output above:
 +<code bash>
 +http://r05n17:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +</code>
 +Before continuing, **make a note of the corresponding URL from your output**, since you will need it for a later step (the token and most likely the compute node name will be different for you).
 +</note>
 +
 +====Setting up SSH tunnel to connect to Jupyter Notebook server====
 +
 +With the Jupyter Notebook server running on a compute node on Caviness, an SSH tunnel is needed to be able to make a connection and access the Jupyter Notebook server from a web browser on your local machine. This is done by opening a **second** SSH connection to Caviness. Follow the appropriate section below for a Linux/Mac or Windows laptop.
 +
 +<note warning>
 +Your compute node name will likely be different than ''r05n17'', so please make sure to change that accordingly. This SSH tunnel connection will have to remain open while you are using Jupyter Notebook. If it is closed or internet connectivity is lost, then your connection to Jupyter Notebook will also be lost.
 +</note>
 +
 +==Linux/Mac==
 +Open a new terminal session on your local machine. Set up an SSH tunnel using the below ''ssh'' command. 
 +<code>
 +$ ssh -L 8888:r05n17:8888 user@caviness.hpc.udel.edu
 +</code>
 +
 +==Windows (Command Prompt or PowerShell)==
 +Open a new Command Prompt or PowerShell session on your local machine. Set up an SSH tunnel using the below ''ssh'' command. 
 +<code>
 +> ssh -L 8888:r05n17:8888 user@caviness.hpc.udel.edu
 +</code>
 +
 +==Windows (WSL)==
 +Open a Linux terminal via WSL on your local machine. Set up an SSH tunnel using the below ''ssh'' command. 
 +<code>
 +$ ssh -L 8888:r05n17:8888 user@caviness.hpc.udel.edu
 +</code>
 +
 +====Using your local browser to access Jupyter Notebook server====
 +
 +If everything so far has been set up correctly, the final step is as easy as opening a web browser of choice on your local machine and entering the correct URL. You should now use the URL that you previously made a note of from the output of the ''jupyter notebook'' command that you ran on the compute node. Change the compute node name in the URL to ''localhost'' and copy the URL into your browser. For instance, if your URL was:
 +
 +<code>
 +http://r05n17:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +</code>
 +
 +then in your browser, you would copy:
 +
 +<code>
 +http://localhost:8888/tree?token=0f2f120dc34903954fb4b62dcb92e2b0c5950b670c0559ef
 +</code>
 +
 +You should now see Jupyter running on your browser.
 +
 +Note that since this example ran the ''jupyter notebook'' command on the compute node from the home directory ''~'', when you open Jupyter on your browser you will see files in your home directory on Caviness. If you want to start from a different directory -- for example, a directory where you already have a Jupyter Notebook ''.ipynb'' file that you want to open -- make sure to navigate to that directory on Caviness before starting up the Jupyter Notebook server.
 +
 +<note important>
 +If you are not able to connect to the Jupyter Notebook session at this point, then you will need to review the prior steps and make sure that you have added and configured the SSH tunnel properly based on your compute node. Remember, the SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
 +</note>
  • software/juypter/caviness.1774557295.txt.gz
  • Last modified: 2026-03-26 16:34
  • by mbotto