Differences

This shows you the differences between two versions of the page.

--- technical:recipes:jupyter-notebook [2024-08-01 16:40] – [Jupyter Notebook Python Virtual Environment] thuachen
+++ technical:recipes:jupyter-notebook [2025-02-04 09:48] (current) – [VALET Package Definition] anita
@@ Line 24: / Line 24: @@
 </code>
-After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it ''jupyter-notebook'', but you are welcome to call it anything you would like. Before creating it, we will want to see what "default" version of Python 3 is available on as part of Anaconda.
+After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it ''jupyter-notebook'', but you are welcome to call it anything you would like. We will specify a version directory by date ''20240801'' for the installation of the August 2024 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. You will be asked to ''Proceed ([y]/n)?'' and you will want to choose ''y'' as this is advising you that prerequisite software is also going to be installed.
 <code bash>
-[(it_css:traine)@login00 ~]$ which python3
+[(it_css:traine)@login00 ~]$ conda create --prefix $WORKDIR/sw/anaconda-envs/jupyter-notebook/20240801 -c jupyter python=3 jupyter
-/opt/shared/anaconda/2024.02/bin/python3
-[(it_css:traine)@login00 ~]$ python3 --version
-Python 3.11.7
-</code>
-Now that we know the version of python3 loaded into our environment is ''3.11.7'', we can run the commands to create a virtual environment. We will also specify a version directory by date ''20240801'' for the installation of the August 2024 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. You will be asked to ''Proceed ([y]/n)?'' and you will want to choose ''y'' as this is advising you that prerequisite software is also going to be installed.
-<code bash>
-[(it_css:traine)@login00 ~]$ conda create --prefix $WORKDIR/sw/anaconda-envs/jupyter-notebook/20240801 -c jupyter python=3.11.7
 Channels:
  - jupyter
- - defaults
+ - conda-forge
+ - nodefaults
 Platform: linux-64
 Collecting package metadata (repodata.json): done
@@ Line 49: / Line 41: @@
   added / updated specs:
-    - python=3.11.7
+    - jupyter
+    - python=3
@@ Line 56: / Line 49: @@
     package                    |            build
     ---------------------------|-----------------
-    ca-certificates-2024.7.2   |       h06a4308_0         127 KB
+      :
-    pip-24.0                   |  py311h06a4308_0         3.3 MB
+    jupyter-1.1.1              |     pyhd8ed1ab_1           9 KB  conda-forge
-    python-3.11.7              |       h955ad1f_0        32.8 MB
+    jupyter-lsp-2.2.5          |     pyhd8ed1ab_1          54 KB  conda-forge
-    setuptools-69.5.1          |  py311h06a4308_0         1.3 MB
+    jupyter_client-8.6.3       |     pyhd8ed1ab_1         104 KB  conda-forge
-    wheel-0.43.0               |  py311h06a4308_0         146 KB
+    jupyter_console-6.6.3      |     pyhd8ed1ab_1          26 KB  conda-forge
+    jupyter_core-5.7.2         |     pyh31011fe_1          56 KB  conda-forge
+    jupyter_events-0.11.0      |     pyhd8ed1ab_0          22 KB  conda-forge
+    jupyter_server-2.15.0      |     pyhd8ed1ab_0         320 KB  conda-forge
+    jupyter_server_terminals-0.5.3|     pyhd8ed1ab_1          19 KB  conda-forge
+    jupyterlab-4.3.5           |     pyhd8ed1ab_0         7.3 MB  conda-forge
+    jupyterlab_pygments-0.3.0  |     pyhd8ed1ab_2          18 KB  conda-forge
+    jupyterlab_server-2.27.3   |     pyhd8ed1ab_1          48 KB  conda-forge
+    jupyterlab_widgets-3.0.13  |     pyhd8ed1ab_1         182 KB  conda-forge
+      :
+    yaml-0.2.5                 |       h7f98852_2          87 KB  conda-forge
+    zeromq-4.3.5               |       h3b0a872_7         328 KB  conda-forge
+    zipp-3.21.0                |     pyhd8ed1ab_1          21 KB  conda-forge
+    zstandard-0.23.0           |  py313h80202fe_1         414 KB  conda-forge
+    zstd-1.5.6                 |       ha6fb4c9_0         542 KB  conda-forge
     ------------------------------------------------------------
-                                           Total:        37.7 MB
+                                           Total:        87.4 MB
 The following NEW packages will be INSTALLED:
-  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
+    :
-  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
+  jupyter            conda-forge/noarch::jupyter-1.1.1-pyhd8ed1ab_1
-  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6
+  jupyter-lsp        conda-forge/noarch::jupyter-lsp-2.2.5-pyhd8ed1ab_1
-  ca-certificates    pkgs/main/linux-64::ca-certificates-2024.7.2-h06a4308_0
+  jupyter_client     conda-forge/noarch::jupyter_client-8.6.3-pyhd8ed1ab_1
-  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
+  jupyter_console    conda-forge/noarch::jupyter_console-6.6.3-pyhd8ed1ab_1
-  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1
+  jupyter_core       conda-forge/noarch::jupyter_core-5.7.2-pyh31011fe_1
-  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
+  jupyter_events     conda-forge/noarch::jupyter_events-0.11.0-pyhd8ed1ab_0
-  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
+  jupyter_server     conda-forge/noarch::jupyter_server-2.15.0-pyhd8ed1ab_0
-  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
+  jupyter_server_te~ conda-forge/noarch::jupyter_server_terminals-0.5.3-pyhd8ed1ab_1
-  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0
+  jupyterlab         conda-forge/noarch::jupyterlab-4.3.5-pyhd8ed1ab_0
-  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
+  jupyterlab_pygmen~ conda-forge/noarch::jupyterlab_pygments-0.3.0-pyhd8ed1ab_2
-  openssl            pkgs/main/linux-64::openssl-3.0.14-h5eee18b_0
+  jupyterlab_server  conda-forge/noarch::jupyterlab_server-2.27.3-pyhd8ed1ab_1
-  pip                pkgs/main/linux-64::pip-24.0-py311h06a4308_0
+  jupyterlab_widgets conda-forge/noarch::jupyterlab_widgets-3.0.13-pyhd8ed1ab_1
-  python             pkgs/main/linux-64::python-3.11.7-h955ad1f_0
+   :
-  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0
+  yaml               conda-forge/linux-64::yaml-0.2.5-h7f98852_2
-  setuptools         pkgs/main/linux-64::setuptools-69.5.1-py311h06a4308_0
+  zeromq             conda-forge/linux-64::zeromq-4.3.5-h3b0a872_7
-  sqlite             pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0
+  zipp               conda-forge/noarch::zipp-3.21.0-pyhd8ed1ab_1
-  tk                 pkgs/main/linux-64::tk-8.6.14-h39e8969_0
+  zstandard          conda-forge/linux-64::zstandard-0.23.0-py313h80202fe_1
-  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0
+  zstd               conda-forge/linux-64::zstd-1.5.6-ha6fb4c9_0
-  wheel              pkgs/main/linux-64::wheel-0.43.0-py311h06a4308_0
-  xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1
-  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1
 Proceed ([y]/n)? y
@@ Line 128: / Line 134: @@
         - action: source
           script:
-              sh: anaconda-activate.sh
+              sh: anaconda-activate-2024.sh
           order: failure-first
           success: 0
@@ Line 138: / Line 144: @@
 </code>
-<note>The ''prefix'' show here ''/work/it_css/sw/anaconda-envs/jupyter-notebook'' will need to be changed based on your workgroup and directory names.</note>
+<note>The ''prefix'' shown here ''/work/it_css/sw/anaconda-envs/jupyter-notebook'' will need to be changed based on your workgroup and directory names.</note>
 ===== Using Jupyter Notebook Virtual Environment =====
@@ Line 165: / Line 171: @@
 ===== Running Jupyter Notebook on the Caviness Cluster =====
-Running Jupyter notebook on Caviness or any HPC cluster for that matter takes some extra steps. On your personal laptop you can simply install Jupyter notebook and start it up. On Caviness you need to run the Jupyter notebook on a compute node. The steps below will show you how to request a interactive compute node, and use VALET to load the Jupyter notebook virtual environment. After starting the virtual environment we will run Jupyter notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system.
+Running Jupyter Notebook on Caviness, or any HPC cluster for that matter, takes some extra steps. You can simply install Jupyter Notebook on your personal laptop and start it up. On Caviness, you need to run the Jupyter Notebook on a compute node. The steps below will show you how to request an interactive compute node and use VALET to load the Jupyter Notebook virtual environment. After starting the virtual environment, we will run Jupyter Notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system.
-<note warning>On Caviness you're **required** to run Jupyter notebook on a **compute node**. If you run it on the login node you could cause slowness or other issues on the login node and IT might kill your Jupyter notebook session with out warning.</note>
+<note warning>On Caviness you're **required** to run Jupyter notebook on a **compute node**. If you run it on the login node, you could cause slowness or other issues with the login node, and IT might kill your Jupyter Notebook session without warning.</note>
-==== Requesting an interactive compute node ====
+==== Requesting an interactive job ====
-In this example we will request an interactive compute node session with 2GB of memory for 1 hour on the ''it_css'' workgroup partition. Also it is important to pass the ''SLURM_EXPORT_ENV=NONE'' when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
+In this example we will request an interactive job to connect us to a compute node with 2GB of memory for 1 hour on the ''it_css'' workgroup partition. Also it is important to pass the ''SLURM_EXPORT_ENV=NONE'' when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
 <code bash>
 [(it_css:traine)@login00 ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=it_css
 </code>
-==== Loading the Jupyter notebook virtual environment ====
+==== Loading the Jupyter Notebook virtual environment ====
-After the interactive session has been established, it is time to load the Jupyter notebook virtual environment with ''VALET''. Even though we did this earlier, it needs to be done again since we are now on a compute node.
+After the interactive job has been established, it is time to load the Jupyter Notebook virtual environment with ''VALET''. Even though we did this earlier, it needs to be done again since we are now on a compute node.
 <code bash>
 [[traine@r00n50 1201]$ vpkg_require jupyter-notebook/20240801
@@ Line 184: / Line 190: @@
 </code>
-<note tip>Once the Jupyter noteboook virtual environment had been loaded on the interactive compute node session, your prompt should look something like:
+<note tip>Once the Jupyter Notebook virtual environment has been loaded on the interactive compute node session, your prompt should look something like this:
 ''(/work/it_css/sw/anaconda_envs/jupyter-notebook/20240801) [traine@r00n50 1201]$''
 </note>
-==== Starting Jupyter notebook session ====
+==== Starting Jupyter Notebook session ====
-When starting the Jupyter notebook session specific options will be passed which will be used for setting up the tunnel connection.
+When starting the Jupyter Notebook session, specific options are passed, which are used to set up the tunnel connection.
 <code bash>
@@ Line 206: / Line 212: @@
 [I 2024-08-01 15:16:27.760 ServerApp] Jupyter Server 2.10.0 is running at:
 [I 2024-08-01 15:16:27.761 ServerApp] http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
-[I 2024-08-01 15:16:27.761 ServerApp]     http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+[I 2024-08-01 15:16:27.761 ServerApp] http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
 [I 2024-08-01 15:16:27.761 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
 [C 2024-08-01 15:16:27.765 ServerApp]
     To access the server, open this file in a browser:
-        file:///home/3347/.local/share/jupyter/runtime/jpserver-20300-open.html
+        file:///home/1201/.local/share/jupyter/runtime/jpserver-20300-open.html
     Or copy and paste one of these URLs:
         http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
@@ Line 232: / Line 238: @@
 ==Windows (PuTTY)==
 Once you open the PuTTY, it will show the //Session// window. For the //Host Name(or IP Address)//, you will need to enter the ''<user-name>@caviness.hpc.udel.edu''.
-In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then add the tunnel settings based on the image below. The tunnel setting is found under the Category //Connection -> SSH -> Tunnels//
+In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then adding the tunnel settings based on the image below. The tunnel setting is found under the Category //Connection -> SSH -> Tunnels//
 {{:technical:recipes:putty-1.png?400|}}
-Add the **Source port** as ''8888'' and **Destination** as ''r00n50:8888'' as shown in the image above, then click on **Add**. The Tunnel settings are now available to your session, so click **Open** to connect. You may need to enter the password for the Caviness/DARWIN account in the prompted window if you do not have an existing session. These settings are not saved, however it is likely the necessary information to setup the tunnel the next time will change anyway.  Remember, this ssh tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
+Add the **Source port** as ''8888'' and **Destination** as ''r00n50:8888'' as shown in the image above, then click on **Add**. The //Tunnel// settings are now available to your session, so click **Open** to connect. You may need to enter the password for the Caviness/DARWIN account in the prompted window if you do not have an existing session. These settings are not saved. However, it is likely the necessary information to set up the tunnel the next time will change anyway. Remember, this SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
 {{ :technical:recipes:jupyter-notebook-1.png?nolink&400 |}}
 ==Linux/Mac==
@@ Line 259: / Line 265: @@
 ......................................................................
-Last login: Thu Aug  6 12:28:19 2021
+Last login: Thu Aug  1 12:28:19 2024
 [traine@login00 ~]$
 </code>
@@ Line 268: / Line 274: @@
 ''http://localhost:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2''
-<note important>If you are not able to connect to the Jupyter Notebook session at this point, then you will need to review the prior steps and make sure that you had added and configured the ssh tunnel properly based on your compute node. Remember, the ssh tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.  </note>
+<note important>If you are not able to connect to the Jupyter Notebook session at this point, then you will need to review the prior steps and make sure that you have added and configured the SSH tunnel properly based on your compute node. Remember, the SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.  </note>