hpc documentation

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Miniconda on Caviness ======
<note warning> This page is under construction.
</note>

Miniconda is available through the VALET packages on the Caviness cluster. To check the versions, use
<code>
$ vpkg_versions miniconda

Available versions in package (* = default version):

[/opt/shared/valet/2.1/etc/miniconda.vpkg_yaml]
miniconda  A free, miniature installation of Anaconda Distribution
* 25.1.1.2 Miniconda py39 25.1.1-2 for Linux (x86_64)
</code>
To load it, use
<code>
$ vpkg_require miniconda/25.1.1.2
Adding package `miniconda/25.1.1.2` to your environment
</code>

===== Installing Applications with Conda =====
<note tip>
Note: In the examples below, please modify paths, usernames, and environment names (e.g., ''/work/workgroup/sw/''..., ''myenv'') to match your own account and project setup. 
</note>
You can create a new Conda environment with a specific package and in a specific path. For example, to create an environment in a specific path, such as your ''/work/workgroup'' directory,
<code bash>
[(my_workgroup:user)@login01 ~]$ conda create -p /work/workgroup/sw/myenv <my_package>
</code>

<note important>
By default, Conda environments created by ''%%--%%name'' or ''-n'' are in your home directory. However, the limited home directory quotas (20 GB) can be filled up quickly. It is recommended to create environments by specifying path ''-p'' in your work directory instead of ''$HOME'' to avoid quota issues.
</note>
After creating the Conda environments, you can activate the local conda environment using,
<code bash>
[(my_workgroup:user)@login01 ~]$ conda activate /work/workgroup/sw/myenv
</code>
You can install the desired packages in the current environment, for example,
<code bash>
(/work/workgroup/sw/myenv)[(my_workgroup:user)@login01 ~]$ conda install <my_package>
</code>
To list packages in the currently active environment, use
<code bash>
(/work/workgroup/sw/myenv)[(my_workgroup:user)@login01 ~]$ conda list
# packages in environment at /work/workgroup/sw/myenv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.3.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
bzip2                     1.0.8                hda65f42_8    conda-forge
ca-certificates           2025.10.5            hbd8a1cb_0    conda-forge
certifi                   2025.10.5                pypi_0    pypi
</code>
To deactivate the environment, do:
<code bash>
(/work/workgroup/sw/myenv)[(my_workgroup:user)@login01 ~]$ conda deactivate
[(my_workgroup:user)@login01 ~]$
</code>
To remove an environment, run:
<code bash>
[(my_workgroup:user)@login01 ~]$ conda remove -p /work/workgroup/sw/myenv --all
</code>
To remove the unused packages and caches, do
<code bash>
[(my_workgroup:user)@login01 ~]$ conda clean --all
</code>

===== Migrating Environments =====

In the terminal window, to view the environments available to you, use
<code bash>
[(my_workgroup:user)@login01 ~]$ conda env list
# conda environments:
#
base                   /opt/shared/miniconda/25.1.1.2
                      */work/workgroup/sw/myenv
</code>

After activating the desired environment, export the current environment,
<code bash>
[(my_workgroup:user)@login01 ~]$ conda activate /work/workgroup/sw/myenv
(/work/workgroup/sw/myenv)[(my_workgroup:user)@login01 ~]$ conda export --file=myenv.yaml
name: /work/workgroup/sw/myenv
channels:
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=hda65f42_8
  - ca-certificates=2025.10.5=hbd8a1cb_0
  - cuda-version=11.8=h70ddcb2_3
  - cudatoolkit=11.8.0=h4ba93d1_13
  - cudnn=8.9.7.29=hbc23b4c_3 
...
...

prefix: /work/workgroup/sw/myenv  
</code>
Then you can duplicate the same environment with
<code bash>
(/work/workgroup/sw/myenv)[(my_workgroup:user)@login01 ~]$ conda deactivate
[(my_workgroup:user)@login01 ~]$ conda create -p /work/workgroup/sw/myenv_2 -f myenv.yml
</code>

===== Recipes ====

Examples documented as recipes to be used for specific installations, including using a ''workgroup'' directory as well as creating VALET packages for these environments and job scripts setup for batch runs, but can also perhaps help others in solving similar installation dilemmas.

  * [[technical:recipes:tensorflow-in-virtualenv|Building a TensorFlow Python Virtual Environment]]