Multi-instance

The CIME coupling infrastructure is capable of running multiple component instances (ensembles) under one model executable. There are two modes of ensemble capability, single driver in which all component instances are handled by a single driver/coupler component or multi-driver in which each instance includes a separate driver/coupler component. In the multi-driver mode the entire model is duplicated for each instance while in the single driver mode only active components need be duplicated. In most cases the multi-driver mode will give better performance and should be used.

The primary motivation for this development was to be able to run an ensemble Kalman-Filter for data assimilation and parameter estimation (UQ, for example). However, it also provides the ability to run a set of experiments within a single model executable where each instance can have a different namelist, and to have all the output go to one directory.

1. Create the case.

   ./scripts/create_newcase --case ~/MULTI_EXAMPLE --compset F2000_DEV --res f19_f19_mg17
   cd ~/MULTI_EXAMPLE
   

2. Assume this is the out-of-the-box pe-layout:

   Comp  NTASKS  NTHRDS  ROOTPE
   CPL :    144/     1;      0
   ATM :    144/     1;      0
   LND :    144/     1;      0
   ICE :    144/     1;      0
   OCN :    144/     1;      0
   ROF :    144/     1;      0
   GLC :    144/     1;      0
   WAV :    144/     1;      0
   ESP :      1/     1;      0
   

   Lets assume that the atm, lnd, rof, and glc components are activate in this compset. The ocn is a prescribed data component, cice is a mixed prescribed/active component (ice-coverage is prescribed), and wav and esp are stub components.

   In this example each each non-stub component will run two instances.

3. Using xmlchange the NINST can be modified for each of these components.

   ./xmlchange NINST_ATM=2
   ./xmlchange NINST_LND=2
   ./xmlchange NINST_ICE=2
   ./xmlchange NINST_ROF=2
   ./xmlchange NINST_GLC=2
   ./xmlchange NINST_OCN=2
   

   Each of these components will have each instance run on 72 MPI tasks (NTASKS/NINST) and all using the same driver/coupler component.

   Note

   If a separate driver/coupler component per instance is desired, the MULTI_DRIVER option can be used.

   ./xmlchange MULTI_DRIVER=TRUE
   

   This configuration will run each component instance on the original 144 tasks but will generate two copies of the model (in the same executable) for a total of 288 tasks.

4. Set up the case.

   ./case.setup
   

   A new user_nl_xxx_NNNN file is generated for each component instance when case.setup is called (where xxx is the component type and NNNN is the number of the component instance).

   For example if the ICE component is cice then the following files are created in $CASEROOT after calling ./xmlchange NINST_ICE=2 and ./case.setup:

   user_nl_cice_0001
   user_nl_cice_0002
   

   The namelist for each component instance can be modified by changing the corresponding user_nl_xxx_NNNN file.

   The components steam files can also be modified per instance. To change the DOCN stream txt file instance 0002, copy docn.streams.txt.prescribed_0002 to your $CASEROOT directory with the name user_docn.streams.txt.prescribed_0002 and modify it accordlingly.

   Note

   • These changes can be made when create_newcase is called using --ninst # where # is a positive integer, and --multi-driver to invoke the multi-driver mode.

   • Multiple component instances can differ ONLY in namelist settings; they ALL use the same model executable.

   • Calling case.setup with --clean DOES NOT remove the user_nl_xxx_NN (where xxx is the component name) files created by case.setup.

   • A special variable NINST_LAYOUT is provided for some experimental compsets, its value should be ‘concurrent’ for all but a few special cases and it cannot be used if MULTI_DRIVER=TRUE.

   • Using create_test these options can be invoked with testname modifiers _N# for the single driver mode and _C# for the multi-driver mode. These are mutually exclusive options.

   • In multi-driver mode you will always get 1 instance of each component for each driver/coupler, if you change a case using ./xmlchange MULTI_COUPLER=TRUE you will get a number of driver/couplers equal to the maximum NINST value over all components.