Contributing Guide
Introduction
The Case class is the core of the CIME Case Control system. All interactions with a case are performed through this class. The variables used to create and manipulate a case are defined in XML files, and for each XML file, there is a corresponding Python class to interact with it.
XML files that are part of the CIME distribution and are intended to be read-only with respect to a case are typically named config_something.xml. The corresponding Python class is named Something and can be found in the file CIME.XML.something.py. These are referred to as the CIME config classes.
XML files that are part of a case and thus are read/write to a case are typically named env_whatever.xml. The corresponding Python modules are CIME.XML.env_whatever.py, and the classes are named EnvWhatever. These are referred to as the Case env classes.
The Case class includes an array of the Case env classes. In the configure function and its supporting functions, the case object creates and manipulates the Case env classes by reading and interpreting the CIME config classes.
Testing
CIME splits it’s testing into two categories unit and sys.
The unit category covers both doctests and unit tests. While the sys category covers regression tests. The tests are named accordingly e.g. unit tests are found as CIME/tests/test_unit*.
How to run the tests
There are two possible methods to run these tests.
Warning
scripts_regression_tests.py is deprecated and will be removed in the future.
pytest
CIME supports running tests using pytest. By using pytest coverage reports are automatically generated. pytest supports all the same arguments as scripts_regression_tests.py, see –help for details.
To get started install pytest and pytest-cov.
pip install pytest pytest-cov
# or
# conda install -c conda-forge pytest pytest-cov
Examples
Runing all the sys and unit tests.
pytest
Running only sys tests, sys can be replaced with unit to run only unit testing.
pytest CIME/tests/test_sys.*
Runnig a specific test case.
pytest CIME/tests/test_unit_case.py
A specific test can be ran with the followin.
pytest CIME/tests/test_unit_case.py::TestCaseSubmit::test_check_case
scripts_regression_tests.py
The scripts_regression_tests.py script is located under CIME/tests.
You can pass either the module name or the file path of a test.
Examples
Runing all the sys and unit tests.
python CIME/tests/scripts_regression_tests.py
Running only sys tests, sys can be replaced with unit to run only unit testing.
python CIME/tests/scripts_regression_tests.py CIME/tests/test_sys*
Runnig a specific test case.
python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case
A specific test can be ran with the followin.
python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case.TestCaseSubmit.test_check_case
Code Quality
To ensure code quality we require all code to be linted by pylint and formatted using black. We run a few other tools to check XML formatting, ending files with newlines and trailing white spaces.
To ensure consistency when running these checks we require the use of [pre-commit](https://pre-commit.com/).
Our GitHub actions will lint and check the format of each PR but will not automatically fix any issues. It’s up to the developer to resolve linting and formatting issues. We encourage installing pre-commit’s [Git hooks](#installing-git-hook-scripts) that will run these checks before code can be committed.
Installing pre-commit
pip install pre_commit
# or
# conda install -c conda-forge pre_commit
Running pre-commit
pre-commit run -a
Installing git hook scripts
If you install these scripts then pre-commit will automatically run on git commit.
pre-commit install
Docker container
GitHub actions runs all CIME’s tests in containers. The dockerfile can be found under the docker/ directory.
You can skip building the container and use the same container from the GitHub actions using the following commands. This will pull the latest [image](https://hub.docker.com/r/jasonb87/cime/tags), see the available [run modifiers](#running-the-container) to customize the container.
The current container supports the GNU compiler and OpenMPI library.
Running
The default environment is similar to the one used by GitHub Actions. It will clone CIME into /src/cime, set CIME_MODEL=cesm and run CESM’s checkout_externals. This will create a minimum base environment to run both unit and system tests.
The CIME_MODEL environment vairable will change the environment that is created.
Setting it to E3SM will clone E3SM into /src/E3SM, checkout the submodules and update the CIME repository using CIME_REPO and CIME_BRANCH.
Setting it to CESM will clone CESM into /src/CESM, run checkout_externals and update the CIME repository using CIME_REPO and CIME_BRANCH.
The container can further be modified using the environment variables defined below.
Note
It’s recommended when running the container to pass –hostname docker as it will match the custom machine defined in config_machines.xml. If this is omitted, –machine docker must be passed to CIME commands in order to use the correct machine definition.
Environment variables
Environment variables to modify the container environment.
Examples
docker run -it -e INIT=false cime:latest bash
docker run -it -e CIME_REPO=https://github.com/user/cime -e CIME_BRANCH=updates_xyz cime:latest bash
Persisting data
The config_machines.xml definition as been setup to provided persistance for inputdata, cases, archives and tools. The following paths can be mounted as volumes to provide persistance.
/storage/inputdata
/storage/cases
/storage/archives
/storage/tools
docker run -it -v ${PWD}/data-cache:/storage/inputdata cime:latest bash
It’s also possible to persist the source git repositories.
docker run -it -v ${PWD}/src:/src cime:latest bash
Local git respositories can be mounted as well.
Building
The container provides 3 targets.
base - Base image with no batch system.
slurm - Slurm batch system with configuration and single queue.
pbs - PBS batch system with configuration and single queue.
Customizing
When building the container some features can be customized. Multiple –build-arg arguments can be passed.
docker build -t ghcr.io/ESMCI/cime:latest --build-arg {name}={value} docker/
Argument |
Description |
Default |
|---|---|---|
MAMBAFORGE_VERSION |
Version of the condaforge/mambaforge image used as a base |
4.11.0-0 |
PNETCDF_VERSION |
Parallel NetCDF version to build |
1.12.1 |
LIBNETCDF_VERSION |
Version of libnetcdf, the default will install the latest |
4.8.1 |
NETCDF_FORTRAN_VERSION |
Version of netcdf-fortran, the default will install the latest |
4.5.4 |
ESMF_VERSION |
Version of ESMF, the default will install the latest |
8.2.0 |