Table of Contents

DeepMD Python Virtual Environment

This page documents the creation of a Python virtual environment (virtualenv) containing the DeepMD software on the Caviness HPC system1). It assumes that the user is adding the software to the workgroup storage.

Prepare Workgroup Directory

Prepare to add software in the standard sub-directories of the workgroup storage:

[user@login01 ~]$ workgroup -g my_workgroup
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/deepmd
[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet

These commands create any missing directories. All directories created will have group-write and -inherit permissions.

Create Virtualenv

The Python distribution will form the basis for the anaconda virtualenv, so add it to the environment:

[(my_workgroup:user)@login01 ~]$ vpkg_require anaconda/2024.02
Adding package `anaconda/2024.02` to your environment

The virtual environment is first populated with all packages that do not require OpenMM. Any packages requiring OpenMM must be installed after we build and install our local copy of OpenMM in the virtual environment. In this example, neither Numpy nor Scipy require OpenMM.

The two channel options are present to ensure only the default Anaconda channels are consulted – otherwise the command could still pick packages from the Intel channel, for example, which would still have the binary compatibility issues!

$ conda create --prefix=$WORKDIR/sw/deepmd/20240818 --override-channels --channel conda-forge python'=>3.11.9' numpy scipy
Solving environment: done
    :
Proceed ([y]/n)? y
    :
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#                                                                               
# To activate this environment, use                                             
#                                                                               
#     $ conda activate /work/it_css/sw/deepmd/20240818                          
#                                                                               
# To deactivate an active environment, use                                      
#                                                                               
#     $ conda deactivate 
#

Before building and installing DeepMD the environment needs to be activated:

$ conda activate $WORKDIR/sw/deepmd/20240818
(/work/it_css/sw/deepmd/20240818)$ 

Building DeepMD

With the new virtual environment activated, we can now build DeepMD.

(/work/it_css/sw/deepmd/20240818)$ conda create -n deepmd deepmd-kit lammps horovod -c conda-forge
Solving environment: done
    :
Proceed ([y]/n)? y
    :
Preparing transaction: done                                                     
Verifying transaction: done                                                     
    :
 
done

VALET Package Definition

The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Assuming the workgroup does not already have a DeepMD VALET package definition, the following text:

openmm:
    prefix: /work/my_workgroup/sw/deepmd
    description: deepmd within a conda virtualenv
    flags:
        - no-standard-paths
    actions:
        - action: source
          script:
              sh: anaconda-activate-2020.sh
          order: failure-first
          success: 0
    versions:
          "20240818":
              description: environment built August 18, 2024
              dependencies:
                  - anaconda/2024.02

would be added to ${WORKDIR}/sw/valet/deepmd.vpkg_yaml.

Using the Virtual Environment

The versions of the virtual environment declared in the VALET package are listed using the vpkg_versions command:

$ vpkg_versions deepmd
 
Available versions in package (* = default version):
 
[/work/it_css/sw/valet/deepmd.vpkg_yaml]
deepmd      deepmd within a conda virtualenv
* 20240818  environment built August 18, 2024

You can import deepmd, activating the virtual environment is accomplished using the vpkg_require command (in your login shell or inside job scripts):

$ vpkg_require deepmd/20240818
Adding dependency `anaconda/2024.02` to your environment
Adding package `deepmd/20240818` to your environment
(/work/it_css/sw/openmm/20240726)$ python
~/conda-envs/my-sci-app/20201102/bin/python3
Python 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:23:07) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openmm as mm
>>>
1)
The steps should also work on the DARWIN HPC system, though with different package versions.