Jupyter Notebook Python Virtual Environment on Caviness

This is an old revision of the document!

This page is under construction.

The following steps will walk you through setting up a Conda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute node to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session running on a compute node.

Before starting, make sure you set your workgroup. This example will demonstrate how to create a shared directory in your workgroup storage where you can create Conda environments that are accessible to everyone in your workgroup.

If you do not have a sw or sw/valet directory in your workgroup storage directory WORKDIR, please consult with your PI (stakeholder of the workgroup) on how to setup for your workgroup software installs on Caviness before proceeding with these instructions (see Workgroup Software Installs on Caviness for more details).

Before creating the Jupyter Notebook environment, start by creating a directory in your workgroup storage for Conda environments. The --mode=2775 option ensures that other people in your workgroup will also be able to create Conda environments in this directory:

[user@login00.caviness ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login00.caviness ~]$ mkdir --mode=2775 ${WORKDIR_SW}/conda-envs

The environment variable WORKDIR_SW will only be set if the directory ${WORKDIR}/sw already existed before you set your workgroup with workgroup -g. You can check for existence of the directory ${WORKDIR}/sw before setting your workgroup by running:

[user@login00.caviness ~]$ ls -d ${WORKDIR}/sw

This example will use the Miniforge VALET package to enable usage of the Conda package manager. If you are new to Conda, make sure to review the documentation for Miniforge on Caviness.

Start by loading the Miniforge VALET package (this command loads the default version, but you can specify a different version as needed):

[(my_workgroup:user)@login00.caviness ~]$ vpkg_require miniforge
Adding package `miniforge/25.11.0-1` to your environment

After loading Miniforge, we can now use conda create to create the Jupyter Notebook virtual environment. In this example we will call it jupyter-notebook, but you are welcome to call it anything you would like. We will specify a version directory by date 20260326 for the installation of the March 2026 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. Note that the list of packages to be installed is omitted from the output below.

[(my_workgroup:user)@login00.caviness ~]$ conda create --prefix=${WORKDIR_SW}/conda-envs/jupyter-notebook/20260326 python=3 jupyter
Retrieving notices: done
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
  .
  .
  .
## Package Plan ##
 
  environment location: /work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326
 
  added / updated specs:
    - jupyter
    - python=3
  .
  .
  .
Proceed ([y]/n)? y
  .
  .
  .
Downloading and Extracting Packages:
 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /work/it_css/sw/miniforge-testing/jupyter-notebook/20260326
#
# To deactivate an active environment, use
#
#     $ conda deactivate

The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present, and get the path to the parent directory of your Jupyter Notebook virtual environment:

[(my_workgroup:user)@login00.caviness ~]$ ls -d ${WORKDIR_SW}/valet
[(my_workgroup:user)@login00.caviness ~]$ echo ${WORKDIR_SW}/conda-envs/jupyter-notebook
/work/my_workgroup/sw/conda-envs/jupyter-notebook

Take note of the path echoed, then create a new file in ${WORKDIR_SW}/valet named jupyter-notebook.vpkg_yaml and add the following text to it, making sure to replace my_workgroup with the name of your workgroup:

jupyter-notebook.vpkg_yaml

jupyter-notebook:
    prefix: /work/my_workgroup/sw/conda-envs/jupyter-notebook
    description: Jupyter Notebook virtual environments created with Conda
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: miniforge-activate.sh
          order: failure-first
          success: 0
    versions:
        "20260326":
            description: environment built March 26, 2026
            dependencies:
                - miniforge

The versions of the virtual environment declared in the VALET package are listed using the vpkg_versions command:

[(my_workgroup:user)@login00.caviness ~]$ vpkg_versions jupyter-notebook
 
Available versions in package (* = default version):
 
[/work/my_workgroup/sw/valet/jupyter-notebook.vpkg_yaml]
jupyter-notebook  Jupyter Notebook virtual environments created with Conda
* 20260326        environment built March 26, 2026

Activating the virtual environment is accomplished using the vpkg_require command (in your login shell or inside job scripts). If you load the VALET package without having first loaded the miniforge package, miniforge will also be added as a dependency:

[(my_workgroup:user)@login00.caviness ~]$ vpkg_require jupyter-notebook/20260326
Adding dependency `miniforge/25.11.0-1` to your environment
Adding package `jupyter-notebook/20260326` to your environment
(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness valet]$

Jupyter Notebook Python Virtual Environment on Caviness

Create Jupyter Notebook Virtual Environment

VALET Package Definition

Activating the Jupyter Notebook Virtual Environment with VALET

hpc documentation