CIME package

Subpackages

Submodules

CIME.aprun module

Aprun is far too complex to handle purely through XML. We need python code to compute and assemble aprun commands.

CIME.aprun.get_aprun_cmd_for_case(case, run_exe, overrides=None)[source]

Given a case, construct and return the aprun command and optimized node count

CIME.bless_test_results module

CIME.bless_test_results.bless_history(test_name, case, baseline_name, baseline_root, report_only, force)[source]
CIME.bless_test_results.bless_namelists(test_name, report_only, force, baseline_name, baseline_root, new_test_root=None, new_test_id=None)[source]
CIME.bless_test_results.bless_test_results(baseline_name, baseline_root, test_root, compiler, test_id=None, namelists_only=False, hist_only=False, report_only=False, force=False, bless_tests=None, no_skip_pass=False, new_test_root=None, new_test_id=None)[source]

CIME.build module

functions for building CIME models

CIME.build.case_build(caseroot, case, sharedlib_only=False, model_only=False, buildlist=None, save_build_provenance=True, use_old=False, ninja=False, dry_run=False)[source]
CIME.build.clean(case, cleanlist=None, clean_all=False, clean_depends=None)[source]
CIME.build.get_standard_cmake_args(case, sharedpath, shared_lib=False)[source]
CIME.build.get_standard_makefile_args(case, shared_lib=False)[source]
CIME.build.post_build(case, logs, build_complete=False, save_build_provenance=True)[source]
CIME.build.uses_kokkos(case)[source]
CIME.build.xml_to_make_variable(case, varname, cmake=False)[source]

CIME.buildlib module

common utilities for buildlib

CIME.buildlib.build_cime_component_lib(case, compname, libroot, bldroot, use_old=True)[source]
CIME.buildlib.parse_input(argv)[source]
CIME.buildlib.run_gmake(case, compclass, compname, libroot, bldroot, libname='', user_cppdefs='')[source]

CIME.buildnml module

common implementation for building namelist commands

These are used by components/<model_type>/<component>/cime_config/buildnml

CIME.buildnml.build_xcpl_nml(case, caseroot, compname)[source]
CIME.buildnml.copy_inputs_to_rundir(caseroot, compname, confdir, rundir, inst_string)[source]
CIME.buildnml.create_namelist_infile(case, user_nl_file, namelist_infile, infile_text='')[source]
CIME.buildnml.parse_input(argv)[source]

CIME.code_checker module

Libraries for checking python code with pylint

CIME.code_checker.check_code(files, num_procs=10, interactive=False)[source]

Check all python files in the given directory

Returns True if all files had no problems

CIME.code_checker.get_all_checkable_files()[source]

CIME.compare_namelists module

CIME.compare_namelists.compare_namelist_files(gold_file, compare_file, case=None)[source]

Returns (is_match, comments)

CIME.compare_namelists.is_namelist_file(file_path)[source]

CIME.compare_test_results module

CIME.compare_test_results.append_status_cprnc_log(msg, logfile_name, test_dir)[source]
CIME.compare_test_results.compare_history(case, baseline_name, baseline_root, log_id)[source]
CIME.compare_test_results.compare_namelists(case, baseline_name, baseline_root, logfile_name)[source]
CIME.compare_test_results.compare_test_results(baseline_name, baseline_root, test_root, compiler, test_id=None, compare_tests=None, namelists_only=False, hist_only=False)[source]

Compares with baselines for all matching tests

Outputs results for each test to stdout (one line per test); possible status codes are: PASS, FAIL, SKIP. (A SKIP denotes a test that did not make it to the run phase or a test for which the run phase did not pass: we skip baseline comparisons in this case.)

In addition, creates files named compare.log.BASELINE_NAME.TIMESTAMP in each test directory, which contain more detailed output. Also creates *.cprnc.out.BASELINE_NAME.TIMESTAMP files in each run directory.

Returns True if all tests generated either PASS or SKIP results, False if there was at least one FAIL result.

CIME.cs_status module

Implementation of the cs.status script, which prints the status of all of the tests in one or more test suites

CIME.cs_status.cs_status(test_paths, summary=False, fails_only=False, count_fails_phase_list=None, expected_fails_filepath=None, out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)[source]

Print the test statuses of all tests in test_paths. The default is to print to stdout, but this can be overridden with the ‘out’ argument.

If summary is True, then only the overall status of each test is printed

If fails_only is True, then only test failures are printed (this includes PENDs as well as FAILs).

If count_fails_phase_list is provided, it should be a list of phases (from the phases given by test_status.ALL_PHASES). For each phase in this list: do not give line-by-line output; instead, just report the total number of tests that have not PASSed this phase (this includes PENDs and FAILs). (This is typically used with the fails_only option, but it can also be used without that option.)

If expected_fails_filepath is provided, it should be a string giving the full path to a file listing expected failures for this test suite. Expected failures are then labeled as such in the output.

CIME.cs_status_creator module

Creates a test suite-specific cs.status file from a template

CIME.cs_status_creator.create_cs_status(test_root, test_id, extra_args='', filename=None)[source]

Create a test suite-specific cs.status file from the template

Arguments: test_root (string): path to test root; the file will be put here. If

this directory doesn’t exist, it is created.

test_id (string): test id for this test suite. This can contain

shell wildcards if you want this one cs.status file to work across multiple test suites. However, be careful not to make this too general: for example, ending this with ‘*’ will pick up the *.ref1 directories for ERI and other tests, which is NOT what you want.

extra_args (string): extra arguments to the cs.status command

(If there are multiple arguments, these should be in a space-delimited string.)

filename (string): name of the generated cs.status file. If not

given, this will be built from the test_id.

CIME.date module

class CIME.date.date(year=1, month=1, day=1, hour=0, minute=0, second=0)[source]

Bases: object

Simple struct for holding dates and the time of day and performing comparisons

Difference in Hour, Minute, or Second >>> date(4, 5, 6, 9) == date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) != date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) < date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) <= date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) >= date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) > date(4, 5, 6, 8) True

>>> date(4, 5, 6, 4) == date(4, 5, 6, 8)
False
>>> date(4, 5, 6, 4) != date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) < date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) <= date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) >= date(4, 5, 6, 8)
False
>>> date(4, 5, 6, 4) > date(4, 5, 6, 8)
False

Difference in Day >>> date(4, 5, 8, 8) == date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) != date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) < date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) <= date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) >= date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) > date(4, 5, 6, 8) True

>>> date(4, 5, 5, 8) == date(4, 5, 6, 8)
False
>>> date(4, 5, 5, 8) != date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) < date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) <= date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) >= date(4, 5, 6, 8)
False
>>> date(4, 5, 5, 8) > date(4, 5, 6, 8)
False

Difference in Month >>> date(4, 6, 6, 8) == date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) != date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) < date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) <= date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) >= date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) > date(4, 5, 6, 8) True

>>> date(4, 4, 6, 8) == date(4, 5, 6, 8)
False
>>> date(4, 4, 6, 8) != date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) < date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) <= date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) >= date(4, 5, 6, 8)
False
>>> date(4, 4, 6, 8) > date(4, 5, 6, 8)
False

Difference in Year >>> date(5, 5, 6, 8) == date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) != date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) < date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) <= date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) >= date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) > date(4, 5, 6, 8) True

>>> date(3, 5, 6, 8) == date(4, 5, 6, 8)
False
>>> date(3, 5, 6, 8) != date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) < date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) <= date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) >= date(4, 5, 6, 8)
False
>>> date(3, 5, 6, 8) > date(4, 5, 6, 8)
False
day()[source]
static hms_to_second(hour, minute, second)[source]
hour()[source]
minute()[source]
month()[source]
second()[source]
second_of_day()[source]
static second_to_hms(second)[source]
year()[source]
CIME.date.get_file_date(filename)[source]

Returns the date associated with the filename as a date object representing the correct date Formats supported: “%Y-%m-%d_%h.%M.%s “%Y-%m-%d_%05s” “%Y-%m-%d-%05s” “%Y-%m-%d” “%Y-%m” “%Y.%m”

>>> get_file_date("./ne4np4_oQU240.cam.r.0001-01-06-00435.nc")
date(1, 1, 6, 0, 7, 15)
>>> get_file_date("./ne4np4_oQU240.cam.r.0010-1-06_00435.nc")
date(10, 1, 6, 0, 7, 15)
>>> get_file_date("./ne4np4_oQU240.cam.r.0010-10.nc")
date(10, 10, 1, 0, 0, 0)
>>> get_file_date("0064-3-8_10.20.30.nc")
date(64, 3, 8, 10, 20, 30)
>>> get_file_date("0140-3-5")
date(140, 3, 5, 0, 0, 0)
>>> get_file_date("0140-3")
date(140, 3, 1, 0, 0, 0)
>>> get_file_date("0140.3")
date(140, 3, 1, 0, 0, 0)

CIME.expected_fails module

Contains the definition of a class to hold information on expected failures for a single test

class CIME.expected_fails.ExpectedFails[source]

Bases: object

add_failure(phase, expected_status)[source]

Add an expected failure to the list

expected_fails_comment(phase, status)[source]

Returns a string giving the expected fails comment for this phase and status

CIME.get_timing module

Library for implementing getTiming tool which gets timing information from a run.

CIME.get_timing.get_timing(case, lid)[source]

CIME.hist_utils module

Functions for actions pertaining to history files.

CIME.hist_utils.compare_baseline(case, baseline_dir=None, outfile_suffix='')[source]

compare the current test output to a baseline result

case - The case containing the hist files to be compared against baselines baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config outfile_suffix - if non-blank, then the cprnc output file name ends with

this suffix (with a ‘.’ added before the given suffix). if None, no output file saved.

returns (SUCCESS, comments) SUCCESS means all hist files matched their corresponding baseline

CIME.hist_utils.compare_test(case, suffix1, suffix2, ignore_fieldlist_diffs=False)[source]

Compares two sets of component history files in the testcase directory

case - The case containing the hist files to compare suffix1 - The suffix that identifies the first batch of hist files suffix1 - The suffix that identifies the second batch of hist files ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their

field lists (i.e., all shared fields are bit-for-bit, but one case has some diagnostic fields that are missing from the other case), treat the two cases as identical.

returns (SUCCESS, comments)

CIME.hist_utils.copy_histfiles(case, suffix)[source]

Copy the most recent batch of hist files in a case, adding the given suffix.

This can allow you to temporarily “save” these files so they won’t be blown away if you re-run the case.

case - The case containing the files you want to save suffix - The string suffix you want to add to saved files, this can be used to find them later.

CIME.hist_utils.cprnc(model, file1, file2, case, rundir, multiinst_driver_compare=False, outfile_suffix='', ignore_fieldlist_diffs=False, cprnc_exe=None)[source]

Run cprnc to compare two individual nc files

file1 - the full or relative path of the first file file2 - the full or relative path of the second file case - the case containing the files rundir - the rundir for the case outfile_suffix - if non-blank, then the output file name ends with this

suffix (with a ‘.’ added before the given suffix). Use None to avoid permissions issues in the case dir.

ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their

field lists (i.e., all shared fields are bit-for-bit, but one case has some diagnostic fields that are missing from the other case), treat the two cases as identical.

returns (True if the files matched, log_name, comment)

where ‘comment’ is either an empty string or one of the module-level constants beginning with CPRNC_ (e.g., CPRNC_FIELDLISTS_DIFFER)

CIME.hist_utils.generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False)[source]
CIME.hist_utils.generate_teststatus(testdir, baseline_dir)[source]

CESM stores it’s TestStatus file in baselines. Do not let exceptions escape from this function.

CIME.hist_utils.get_ts_synopsis(comments)[source]

Reduce case diff comments down to a single line synopsis so that we can put something in the TestStatus file. It’s expected that the comments provided to this function came from compare_baseline, not compare_tests.

>>> get_ts_synopsis('')
''
>>> get_ts_synopsis('big error')
'big error'
>>> get_ts_synopsis('big error\n')
'big error'
>>> get_ts_synopsis('stuff\n    File foo had a different field list from bar with suffix baz\nPass\n')
'FIELDLIST field lists differ (otherwise bit-for-bit)'
>>> get_ts_synopsis('stuff\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'ERROR BFAIL some baseline files were missing'
>>> get_ts_synopsis('stuff\n    File foo had a different field list from bar with suffix baz\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'MULTIPLE ISSUES: field lists differ and some baseline files were missing'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\n    File foo had a different field list from bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('File foo had no compare counterpart in bar with suffix baz\n File foo had no original counterpart in bar with suffix baz\n')
'DIFF'
CIME.hist_utils.rename_all_hist_files(case, suffix)[source]

Renaming all hist files in a case, adding the given suffix.

case - The case containing the files you want to save suffix - The string suffix you want to add to saved files, this can be used to find them later.

CIME.locked_files module

CIME.locked_files.is_locked(filename, caseroot=None)[source]
CIME.locked_files.lock_file(filename, caseroot=None, newname=None)[source]
CIME.locked_files.unlock_file(filename, caseroot=None)[source]

CIME.namelist module

Module containing tools for dealing with Fortran namelists.

The public interface consists of the following functions: - character_literal_to_string - compress_literal_list - expand_literal_list - fortran_namelist_base_value - is_valid_fortran_name - is_valid_fortran_namelist_literal - literal_to_python_value - merge_literal_lists - parse - string_to_character_literal

In addition, the Namelist class represents a namelist held in memory.

For the moment, only a subset of namelist syntax is supported; specifically, we assume that only variables of intrinsic type are used, and indexing/co-indexing of arrays to set a portion of a variable is not supported. (However, null values and repeated values may be used to set or fill a variable as indexing would.)

We also always assume that a period (“.”) is the decimal separator, not a comma (“,”). We also assume that the file encoding is UTF-8 or some compatible format (e.g. ASCII).

Otherwise, most Fortran syntax rules implemented here are compatible with Fortran 2008 (which is largely the same as previous standards, and will be similar to Fortran 2015). The only exceptions should be cases where (a) we deliberately prohibit “troublesome” behavior that would be allowed by the standard, or (b) we rely on conventions shared by all major compilers.

One important convention is that newline characters can be used to denote the end of a record. This makes them equivalent to spaces at most locations in a Fortran namelist, except that newlines also end comments, and they are ignored entirely within strings.

While the treatment of comments in this module is standard, it may be somewhat surprising. Namelist comments are only allowed in two situations:

  1. As the only thing on a line (aside from optional indentation with spaces).

(2) Immediately after a “value separator” (the space, newline, comma, or slash after a value).

This implies that all lines except for the last are syntax errors, in this example:

` &group_name! This is not a valid comment because it's after the group name. foo ! Neither is this, because it's between a name and an equals sign. = 2 ! Nor this, because it comes between the value and the following comma. , bar = ! Nor this, because it's between an equals sign and a value. 2! Nor this, because it should be separated from the value by a comma or space. bazz = 3 ! Nor this, because it comes between the value and the following slash. /! This is fine, but technically it is outside the namelist, not a comment. `

However, the above would actually be valid if all the “comments” were removed. The Fortran standard is not clear about whether whitespace is allowed after inline comments and before subsequent non-whitespace text (!), but this module allows such whitespace, to preserve the sanity of both implementors and users.

The Fortran standard only applies to the interior of namelist groups, and not to text between one namelist group and the next. This module assumes that namelist groups are separated by (optional) whitespace and comments, and nothing else.

class CIME.namelist.Namelist(groups=None)[source]

Bases: object

Class representing a Fortran namelist.

Public methods: __init__ delete_variable get_group_names get_value get_variable_names get_variable_value merge_nl set_variable_value write

clean_groups()[source]
delete_variable(group_name, variable_name)[source]

Delete a variable from a specified group.

If the specified group or variable does not exist, this is a no-op.

>>> x = parse(text='&foo bar=1 /')
>>> x.delete_variable('FOO', 'BAR')
>>> x.delete_variable('foo', 'bazz')
>>> x.delete_variable('brack', 'bazz')
>>> x.get_variable_names('foo')
[]
>>> x.get_variable_names('brack')
[]
get_group_names()[source]

Return a list of all groups in the namelist.

>>> Namelist().get_group_names()
[]
>>> sorted(parse(text='&foo / &bar /').get_group_names())
['bar', 'foo']
get_group_variables(group_name)[source]
get_value(variable_name)[source]

Return the value of a uniquely-named variable.

This function is similar to get_variable_value, except that it does not require a group_name, and it requires that the variable_name be unique across all groups.

>>> parse(text='&foo bar=1 / &bazz bar=1 /').get_value('bar')  
Traceback (most recent call last):
...
CIMEError: ERROR: Namelist.get_value: Variable {} is present in multiple groups: ...
>>> parse(text='&foo bar=1 / &bazz /').get_value('Bar')
['1']
>>> parse(text='&foo bar(2)=1 / &bazz /').get_value('Bar(2)')
['1']
>>> parse(text='&foo / &bazz /').get_value('bar')
['']
get_variable_names(group_name)[source]

Return a list of all variables in the given namelist group.

If the specified group is not in the namelist, returns an empty list.

>>> Namelist().get_variable_names('foo')
[]
>>> x = parse(text='&foo bar=,bazz=true,bazz(2)=fred,bang=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bang', 'bar', 'bazz', 'bazz(2)']
>>> x = parse(text='&foo bar=,bazz=true,bang=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bang', 'bar', 'bazz']
>>> x = parse(text='&foo bar(::)=,bazz=false,bazz(2)=true,bazz(:2:)=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bar(::)', 'bazz', 'bazz(2)', 'bazz(:2:)']
get_variable_value(group_name, variable_name)[source]

Return the value of the specified variable.

This function always returns a non-empty list containing strings. If the specified group_name or variable_name is not present, [‘’] is returned.

>>> Namelist().get_variable_value('foo', 'bar')
['']
>>> parse(text='&foo bar=1,2 /').get_variable_value('foo', 'bazz')
['']
>>> parse(text='&foo bar=1,2 /').get_variable_value('foO', 'Bar')
['1', '2']
merge_nl(other, overwrite=False)[source]

Merge this namelist object with another.

Values in the invoking (self) Namelist will take precedence over values in the other Namelist, unless overwrite=True is passed in, in which case other values take precedence.

>>> x = parse(text='&foo bar=1 bazz=,2 brat=3/')
>>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /')
>>> y.get_value('bazz')
['1', '1', '1']
>>> x.merge_nl(y)
>>> sorted(x.get_group_names())
['foo', 'foo2']
>>> sorted(x.get_variable_names('foo'))
['baker', 'bar', 'bazz', 'brat']
>>> sorted(x.get_variable_names('foo2'))
['barter']
>>> x.get_value('bar')
['1']
>>> x.get_value('bazz')
['1', '2', '1']
>>> x.get_value('brat')
['3']
>>> x.get_value('baker')
['4']
>>> x.get_value('barter')
['5']
>>> x = parse(text='&foo bar=1 bazz=,2 brat=3/')
>>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /')
>>> x.merge_nl(y, overwrite=True)
>>> sorted(x.get_group_names())
['foo', 'foo2']
>>> sorted(x.get_variable_names('foo'))
['baker', 'bar', 'bazz', 'brat']
>>> sorted(x.get_variable_names('foo2'))
['barter']
>>> x.get_value('bar')
['2']
>>> x.get_value('bazz')
['1', '1', '1']
>>> x.get_value('brat')
['3']
>>> x.get_value('baker')
['4']
>>> x.get_value('barter')
['5']
set_variable_value(group_name, variable_name, value, var_size=1)[source]

Set the value of the specified variable.

>>> x = parse(text='&foo bar=1 /')
>>> x.get_variable_value('foo', 'bar')
['1']
>>> x.set_variable_value('foo', 'bar(2)', ['3'], var_size=4)
>>> x.get_variable_value('foo', 'bar')
['1', '3']
>>> x.set_variable_value('foo', 'bar(1)', ['2'])
>>> x.get_variable_value('foo', 'bar')
['2', '3']
>>> x.set_variable_value('foo', 'bar', ['1'])
>>> x.get_variable_value('foo', 'bar')
['1', '3']
>>> x.set_variable_value('foo', 'bazz', ['3'])
>>> x.set_variable_value('Brack', 'baR', ['4'])
>>> x.get_variable_value('foo', 'bazz')
['3']
>>> x.get_variable_value('brack', 'bar')
['4']
>>> x.set_variable_value('foo', 'red(2:6:2)', ['2', '4', '6'], var_size=12)
>>> x.get_variable_value('foo', 'red')
['', '2', '', '4', '', '6']
write(out_file, groups=None, append=False, format_='nml', sorted_groups=True)[source]

Write a the output data (normally fortran namelist) to the out_file

As with parse, the out_file argument can be either a file name, or a file object with a write method that accepts unicode. If specified, the groups argument specifies a subset of all groups to write out.

If out_file is a file name, and append=True is passed in, the namelist will be appended to the named file instead of overwriting it. The append option has no effect if a file object is passed in.

The format_ option can be either ‘nml’ (namelist) or ‘rc’, and specifies the file format. Formats other than ‘nml’ may not support all possible output values.

write_nuopc(out_file, groups=None, sorted_groups=True)[source]

Write a nuopc config file out_file

As with parse, the out_file argument can be either a file name, or a file object with a write method that accepts unicode. If specified, the groups argument specifies a subset of all groups to write out.

CIME.namelist.character_literal_to_string(literal)[source]

Convert a Fortran character literal to a Python string.

This function assumes (without checking) that literal is a valid literal.

>>> character_literal_to_string("'blah'")
'blah'
>>> character_literal_to_string('"blah"')
'blah'
>>> character_literal_to_string("'don''t'")
"don't"
>>> character_literal_to_string('"' + '""Hello!""' + '"')
'"Hello!"'
CIME.namelist.compress_literal_list(literals)[source]

Uses repetition syntax to shorten a literal list.

>>> compress_literal_list([])
[]
>>> compress_literal_list(['true'])
['true']
>>> compress_literal_list(['1', '2', 'f*', '3', '3', '3', '5'])
['1', '2', 'f*', '3', '3', '3', '5']
>>> compress_literal_list(['f*', 'f*'])
['f*', 'f*']
CIME.namelist.expand_literal_list(literals)[source]

Expands a list of literal values to get rid of repetition syntax.

>>> expand_literal_list([])
[]
>>> expand_literal_list(['true'])
['true']
>>> expand_literal_list(['1', '2', 'f*', '3*3', '5'])
['1', '2', 'f*', '3', '3', '3', '5']
>>> expand_literal_list(['2*f*'])
['f*', 'f*']
CIME.namelist.fortran_namelist_base_value(string)[source]

Strip off whitespace and repetition syntax from a namelist value.

>>> fortran_namelist_base_value("")
''
>>> fortran_namelist_base_value("f")
'f'
>>> fortran_namelist_base_value("6*")
''
>>> fortran_namelist_base_value("6*f")
'f'
>>> fortran_namelist_base_value(" \n6* \n")
''
>>> fortran_namelist_base_value("\n 6*f\n ")
'f'
CIME.namelist.get_fortran_name_only(full_var)[source]

remove array section if any and return only the variable name >>> get_fortran_name_only(‘foo’) ‘foo’ >>> get_fortran_name_only(‘foo(3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’ >>> get_fortran_name_only(‘foo(1::)’) ‘foo’ >>> get_fortran_name_only(‘foo(:+2:)’) ‘foo’ >>> get_fortran_name_only(‘foo(::-3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’

CIME.namelist.get_fortran_variable_indices(varname, varlen=1, allow_any_len=False)[source]

get indices from a fortran namelist variable as a triplet of minindex, maxindex and step

>>> get_fortran_variable_indices('foo(3)')
(3, 3, 1)
>>> get_fortran_variable_indices('foo(1:2:3)')
(1, 2, 3)
>>> get_fortran_variable_indices('foo(::)', varlen=4)
(1, 4, 1)
>>> get_fortran_variable_indices('foo(::2)', varlen=4)
(1, 4, 2)
>>> get_fortran_variable_indices('foo(::)', allow_any_len=True)
(1, -1, 1)
CIME.namelist.is_valid_fortran_name(string)[source]

Check that a variable name is allowed in Fortran.

The rules are: 1. The name must start with a letter. 2. All characters in a name must be alphanumeric (or underscores). 3. The maximum name length is 63 characters. 4. We only handle a single dimension !!!

>>> is_valid_fortran_name("")
False
>>> is_valid_fortran_name("a")
True
>>> is_valid_fortran_name("A")
True
>>> is_valid_fortran_name("A(4)")
True
>>> is_valid_fortran_name("A(::)")
True
>>> is_valid_fortran_name("A(1:2:3)")
True
>>> is_valid_fortran_name("A(1::)")
True
>>> is_valid_fortran_name("A(:-2:)")
True
>>> is_valid_fortran_name("A(1::+3)")
True
>>> is_valid_fortran_name("A(1,3)")
False
>>> is_valid_fortran_name("2")
False
>>> is_valid_fortran_name("_")
False
>>> is_valid_fortran_name("abc#123")
False
>>> is_valid_fortran_name("aLiBi_123")
True
>>> is_valid_fortran_name("A" * 64)
False
>>> is_valid_fortran_name("A" * 63)
True
CIME.namelist.is_valid_fortran_namelist_literal(type_, string)[source]

Determine whether a literal is valid in a Fortran namelist.

Note that kind parameters are not allowed in namelists, which simplifies this check a bit. Internal whitespace is allowed for complex and character literals only. BOZ literals and compiler extensions (e.g. backslash escapes) are not allowed.

Null values, however, are allowed for all types. This means that passing in a string containing nothing but spaces and newlines will always cause True to be returned. Repetition (e.g. 5*’a’) is also allowed, including repetition of null values.

Detailed rules and examples follow.

Integers: Must be a sequence of one or more digits, with an optional sign.

>>> is_valid_fortran_namelist_literal("integer", "")
True
>>> is_valid_fortran_namelist_literal("integer", " ")
True
>>> is_valid_fortran_namelist_literal("integer", "\n")
True
>>> is_valid_fortran_namelist_literal("integer", "5*")
True
>>> is_valid_fortran_namelist_literal("integer", "1")
True
>>> is_valid_fortran_namelist_literal("integer", "5*1")
True
>>> is_valid_fortran_namelist_literal("integer", " 5*1")
True
>>> is_valid_fortran_namelist_literal("integer", "5* 1")
False
>>> is_valid_fortran_namelist_literal("integer", "5 *1")
False
>>> is_valid_fortran_namelist_literal("integer", "a")
False
>>> is_valid_fortran_namelist_literal("integer", " 1")
True
>>> is_valid_fortran_namelist_literal("integer", "1 ")
True
>>> is_valid_fortran_namelist_literal("integer", "1 2")
False
>>> is_valid_fortran_namelist_literal("integer", "0123456789")
True
>>> is_valid_fortran_namelist_literal("integer", "+22")
True
>>> is_valid_fortran_namelist_literal("integer", "-26")
True
>>> is_valid_fortran_namelist_literal("integer", "2A")
False
>>> is_valid_fortran_namelist_literal("integer", "1_8")
False
>>> is_valid_fortran_namelist_literal("integer", "2.1")
False
>>> is_valid_fortran_namelist_literal("integer", "2e6")
False

Reals: - For fixed-point format, there is an optional sign, followed by an integer part, or a decimal point followed by a fractional part, or both. - Scientific notation is allowed, with an optional, case-insensitive “e” or “d” followed by an optionally-signed integer exponent. (Either the “e”/”d” or a sign must be present to separate the number from the exponent.) - The (case-insensitive) strings “inf”, “infinity”, and “nan” are allowed. NaN values can also contain additional information in parentheses, e.g. “NaN(x1234ABCD)”.

>>> is_valid_fortran_namelist_literal("real", "")
True
>>> is_valid_fortran_namelist_literal("real", "a")
False
>>> is_valid_fortran_namelist_literal("real", "1")
True
>>> is_valid_fortran_namelist_literal("real", " 1")
True
>>> is_valid_fortran_namelist_literal("real", "1 ")
True
>>> is_valid_fortran_namelist_literal("real", "1 2")
False
>>> is_valid_fortran_namelist_literal("real", "+1")
True
>>> is_valid_fortran_namelist_literal("real", "-1")
True
>>> is_valid_fortran_namelist_literal("real", "1.")
True
>>> is_valid_fortran_namelist_literal("real", "1.5")
True
>>> is_valid_fortran_namelist_literal("real", ".5")
True
>>> is_valid_fortran_namelist_literal("real", "+.5")
True
>>> is_valid_fortran_namelist_literal("real", ".")
False
>>> is_valid_fortran_namelist_literal("real", "+")
False
>>> is_valid_fortran_namelist_literal("real", "1e6")
True
>>> is_valid_fortran_namelist_literal("real", "1e-6")
True
>>> is_valid_fortran_namelist_literal("real", "1e+6")
True
>>> is_valid_fortran_namelist_literal("real", ".5e6")
True
>>> is_valid_fortran_namelist_literal("real", "1e")
False
>>> is_valid_fortran_namelist_literal("real", "1D6")
True
>>> is_valid_fortran_namelist_literal("real", "1q6")
False
>>> is_valid_fortran_namelist_literal("real", "1+6")
True
>>> is_valid_fortran_namelist_literal("real", "1.6.5")
False
>>> is_valid_fortran_namelist_literal("real", "1._8")
False
>>> is_valid_fortran_namelist_literal("real", "1,5")
False
>>> is_valid_fortran_namelist_literal("real", "inf")
True
>>> is_valid_fortran_namelist_literal("real", "INFINITY")
True
>>> is_valid_fortran_namelist_literal("real", "NaN")
True
>>> is_valid_fortran_namelist_literal("real", "nan(x56)")
True
>>> is_valid_fortran_namelist_literal("real", "nan())")
False

Complex numbers: - A pair of real numbers enclosed by parentheses, and separated by a comma. - Any number of spaces or newlines may be placed before or after each real.

>>> is_valid_fortran_namelist_literal("complex", "")
True
>>> is_valid_fortran_namelist_literal("complex", "()")
False
>>> is_valid_fortran_namelist_literal("complex", "(,)")
False
>>> is_valid_fortran_namelist_literal("complex", "( ,\n)")
False
>>> is_valid_fortran_namelist_literal("complex", "(a,2.)")
False
>>> is_valid_fortran_namelist_literal("complex", "(1.,b)")
False
>>> is_valid_fortran_namelist_literal("complex", "(1,2)")
True
>>> is_valid_fortran_namelist_literal("complex", "(-1.e+06,+2.d-5)")
True
>>> is_valid_fortran_namelist_literal("complex", "(inf,NaN)")
True
>>> is_valid_fortran_namelist_literal("complex", "(  1. ,  2. )")
True
>>> is_valid_fortran_namelist_literal("complex", "( \n \n 1. \n,\n 2.\n)")
True
>>> is_valid_fortran_namelist_literal("complex", " (1.,2.)")
True
>>> is_valid_fortran_namelist_literal("complex", "(1.,2.) ")
True

Character sequences (strings): - Must begin and end with the same delimiter character, either a single quote (apostrophe), or a double quote (quotation mark). - Whichever character is used as a delimiter must not appear in the string itself, unless it appears in doubled pairs (e.g. ‘’’’ or “’” are the two ways of representing a string containing a single apostrophe). - Note that newlines cannot be represented in a namelist character literal since they are interpreted as an “end of record”, but they are allowed as long as they don’t come between one of the aforementioned double pairs of characters.

>>> is_valid_fortran_namelist_literal("character", "")
True
>>> is_valid_fortran_namelist_literal("character", "''")
True
>>> is_valid_fortran_namelist_literal("character", " ''")
True
>>> is_valid_fortran_namelist_literal("character", "'\n'")
True
>>> is_valid_fortran_namelist_literal("character", "''\n''")
False
>>> is_valid_fortran_namelist_literal("character", "'''")
False
>>> is_valid_fortran_namelist_literal("character", "''''")
True
>>> is_valid_fortran_namelist_literal("character", "'''Cookie'''")
True
>>> is_valid_fortran_namelist_literal("character", "'''Cookie''")
False
>>> is_valid_fortran_namelist_literal("character", "'\"'")
True
>>> is_valid_fortran_namelist_literal("character", "'\"\"'")
True
>>> is_valid_fortran_namelist_literal("character", '""')
True
>>> is_valid_fortran_namelist_literal("character", '"" ')
True
>>> is_valid_fortran_namelist_literal("character", '"\n"')
True
>>> is_valid_fortran_namelist_literal("character", '""\n""')
False
>>> is_valid_fortran_namelist_literal("character", '""' + '"')
False
>>> is_valid_fortran_namelist_literal("character", '""' + '""')
True
>>> is_valid_fortran_namelist_literal("character", '"' + '""Cookie""' + '"')
True
>>> is_valid_fortran_namelist_literal("character", '""Cookie""' + '"')
False
>>> is_valid_fortran_namelist_literal("character", '"\'"')
True
>>> is_valid_fortran_namelist_literal("character", '"\'\'"')
True

Logicals: - Must contain a (case-insensitive) “t” or “f”. - This must be either the first nonblank character, or the second following a period. - The rest of the string is ignored, but cannot contain a comma, newline, equals sign, slash, or space (except that trailing spaces are allowed and ignored).

>>> is_valid_fortran_namelist_literal("logical", "")
True
>>> is_valid_fortran_namelist_literal("logical", "t")
True
>>> is_valid_fortran_namelist_literal("logical", "F")
True
>>> is_valid_fortran_namelist_literal("logical", ".T")
True
>>> is_valid_fortran_namelist_literal("logical", ".f")
True
>>> is_valid_fortran_namelist_literal("logical", " f")
True
>>> is_valid_fortran_namelist_literal("logical", " .t")
True
>>> is_valid_fortran_namelist_literal("logical", "at")
False
>>> is_valid_fortran_namelist_literal("logical", ".TRUE.")
True
>>> is_valid_fortran_namelist_literal("logical", ".false.")
True
>>> is_valid_fortran_namelist_literal("logical", ".TEXAS$")
True
>>> is_valid_fortran_namelist_literal("logical", ".f=")
False
>>> is_valid_fortran_namelist_literal("logical", ".f/1")
False
>>> is_valid_fortran_namelist_literal("logical", ".t\nted")
False
>>> is_valid_fortran_namelist_literal("logical", ".Fant astic")
False
>>> is_valid_fortran_namelist_literal("logical", ".t2 ")
True
CIME.namelist.literal_to_python_value(literal, type_=None)[source]

Convert a Fortran literal string to a Python value.

This function assumes that the input contains a single value, i.e. repetition syntax is not used. The type can be specified by passing a string as the type_ argument, or if this option is not provided, this function will attempt to autodetect the variable type.

Note that it is not possible to be certain whether a literal like “123” is intended to represent an integer or a floating-point value, however, nor can we be certain of the precision that will be used to hold this value in actual Fortran code. We also cannot use the optional information in a NaN float, so this will cause the function to throw an error if that information is present (e.g. a string like “NAN(1234)” will cause an error).

The Python type of the return value is as follows for different type_ arguments: “character” - str “complex” - complex “integer” - int “logical” - bool “real” - float

If a null value is input (i.e. the empty string), None will be returned.

>>> literal_to_python_value("'She''s a winner!'")
"She's a winner!"
>>> literal_to_python_value("1")
1
>>> literal_to_python_value("1.")
1.0
>>> literal_to_python_value(" (\n 1. , 2. )\n ")
(1+2j)
>>> literal_to_python_value(".true.")
True
>>> literal_to_python_value("Fortune")
False
>>> literal_to_python_value("bacon") 
Traceback (most recent call last):
...
CIMEError: ERROR: 'bacon' is not a valid literal for any Fortran type.
>>> literal_to_python_value("1", type_="real")
1.0
>>> literal_to_python_value("bacon", type_="logical") 
Traceback (most recent call last):
...
CIMEError: ERROR: 'bacon' is not a valid literal of type 'logical'.
>>> literal_to_python_value("1", type_="booga") 
Traceback (most recent call last):
...
CIMEError: ERROR: Invalid Fortran type for a namelist: 'booga'
>>> literal_to_python_value("2*1") 
Traceback (most recent call last):
...
CIMEError: ERROR: Cannot use repetition syntax in literal_to_python_value
>>> literal_to_python_value("")
>>> literal_to_python_value("-1.D+10")
-10000000000.0
>>> shouldRaise(ValueError, literal_to_python_value, "nan(1234)")
CIME.namelist.merge_literal_lists(default, overwrite)[source]

Merge two lists of literal value strings.

The overwrite values have higher precedence, so will overwrite the default values. However, if overwrite contains null values, or is shorter than default (and thus implicitly ends in null values), the elements of default will be used where overwrite is null.

>>> merge_literal_lists([], [])
[]
>>> merge_literal_lists(['true'], ['false'])
['false']
>>> merge_literal_lists([], ['false'])
['false']
>>> merge_literal_lists(['true'], [''])
['true']
>>> merge_literal_lists([], [''])
['']
>>> merge_literal_lists(['true'], [])
['true']
>>> merge_literal_lists(['true'], [])
['true']
>>> merge_literal_lists(['3*false', '3*true'], ['true', '4*', 'false'])
['true', 'false', 'false', 'true', 'true', 'false']
CIME.namelist.parse(in_file=None, text=None, groupless=False, convert_tab_to_space=True)[source]

Parse a Fortran namelist.

The in_file argument must be either a str or unicode object containing a file name, or a text I/O object with a read method that returns the text of the namelist.

Alternatively, the text argument can be provided, in which case it must be the text of the namelist itself.

The groupless argument changes namelist parsing in two ways:

  1. parse allows an alternate file format where no group names or slashes are present. In effect, the file is parsed as if an invisible, arbitrary group name was prepended, and an invisible slash was appended. However, if any group names actually are present, the file is parsed normally.

  2. The return value of this function is not a Namelist object. Instead a single, flattened dictionary of name-value pairs is returned.

The convert_tab_to_space option can be used to force all tabs in the file to be converted to spaces, and is on by default. Note that this will usually allow files that use tabs as whitespace to be parsed. However, the implementation of this option is crude; it converts all tabs in the file, including those in character literals. (Note that there are many characters that cannot be passed in via namelist in any standard way, including ‘

‘,

so it is already a bad idea to assume that the namelist will preserve whitespace in strings, aside from simple spaces.)

The return value, if groupless=False, is a Namelist object.

All names and values returned are ultimately unicode strings. E.g. a value of “6*2” is returned as that string; it is not converted to 6 copies of the Python integer 2. Null values are returned as the empty string (“”).

CIME.namelist.shouldRaise(eclass, method, *args, **kw)[source]

A helper function to make doctests py3 compatible http://python3porting.com/problems.html#running-doctests

CIME.namelist.string_to_character_literal(string)[source]

Convert a Python string to a Fortran character literal.

This function always uses double quotes (“) as the delimiter.

>>> string_to_character_literal('blah')
'"blah"'
>>> string_to_character_literal("'blah'")
'"\'blah\'"'
>>> string_to_character_literal('She said "Hi!".')
'"She said ""Hi!""."'

CIME.nmlgen module

Class for generating component namelists.

class CIME.nmlgen.NamelistGenerator(case, definition_files, files=None)[source]

Bases: object

Utility class for generating namelists for a given component.

add_default(name, value=None, ignore_abs_path=None)[source]

Add a value for the specified variable to the namelist.

If the specified variable is already defined in the object, the existing value is preserved. Otherwise, the value argument, if provided, will be used to set the value. If no such value is found, the defaults file will be consulted. If null values are present in any of the above, the result will be a merged array of values.

If no value for the variable is found via any of the above, this method will raise an exception.

add_nmlcontents(filename, group, append=True, format_='nmlcontents', sorted_groups=True)[source]

Write only contents of nml group

clean_streams()[source]
create_shr_strdata_nml()[source]

Set defaults for shr_strdata_nml variables other than the variable domainfile

create_stream_file_and_update_shr_strdata_nml(config, caseroot, stream, stream_path, data_list_path)[source]

Write the pseudo-XML file corresponding to a given stream.

Arguments: config - Used to look up namelist defaults. This is used in addition

to the config used to construct the namelist generator. The main reason to supply additional configuration options here is to specify stream-specific settings.

stream - Name of the stream. stream_path - Path to write the stream file to. data_list_path - Path of file to append input data information to.

get_default(name, config=None, allow_none=False)[source]

Get the value of a variable from the namelist definition file.

The config argument is passed through to the underlying NamelistDefaults.get_value call as the attribute argument.

The return value of this function is a list of values that were found in the defaults file. If there is no matching default, this function returns None if allow_none=True is passed, otherwise an error is raised.

Note that we perform some translation of the values, since there are a few differences between Fortran namelist literals and values in the defaults file: 1) In the defaults file, whitespace is ignored except within strings, so

the output of this function strips out most whitespace. (This implies that commas are the only way to separate array elements in the defaults file.)

  1. In the defaults file, quotes around character literals (strings) are optional, as long as the literal does not contain whitespace, commas, or (single or double) quotes. If a setting for a character variable does not seem to have quotes (and is not a null value), this function will add them.

  2. Default values may refer to variables in a case’s env_*.xml files. This function replaces references of the form $VAR or ${VAR} with the value of the variable VAR in an env file, if that variable exists. This behavior is suppressed within single-quoted strings (similar to parameter expansion in shell scripts).

get_group_variables(group_name)[source]
get_streams()[source]

Get a list of all streams used for the current data model mode.

get_value(name)[source]

Get the current value of a given namelist variable.

Note that the return value of this function is always a string or a list of strings. E.g. the scalar logical value .false. will be returned as “.false.”, while an array of two .false. values will be returned as [“.false.”, “.false.”]. Whether or not a value is scalar is determined by checking the array size in the namelist definition file.

Null values are converted to None, and repeated values are expanded, e.g. [‘2*3’] is converted to [‘3’, ‘3’, ‘3’].

For character variables, the value is converted to a Python string (e.g. quotation marks are removed).

All other literals are returned as the raw string values that will be written to the namelist.

init_defaults(infiles, config, skip_groups=None, skip_entry_loop=False)[source]

Return array of names of all definition nodes

new_instance()[source]

Clean the object just enough to introduce a new instance

static quote_string(string)[source]

Convert a string to a quoted Fortran literal.

Does nothing if the string appears to be quoted already.

set_abs_file_path(file_path)[source]

If file_path is relative, make it absolute using DIN_LOC_ROOT.

If an absolute path is input, it is returned unchanged.

set_value(name, value)[source]

Set the current value of a given namelist variable.

Usually, you should use add_default instead of this function.

The name argument is the name of the variable to set, and the value is a list of strings to use as settings. If the variable is scalar, the list is optional; i.e. a scalar logical can be set using either value=’.false.’ or value=[‘.false.’]. If the variable is of type character, and the input is missing quotes, quotes will be added automatically. If None is provided in place of a string, this will be translated to a null value.

Note that this function will overwrite the current value, which may hold a user-specified setting. Even if value is (or contains) a null value, the old setting for the variable will be thrown out completely.

update_shr_strdata_nml(config, stream, stream_path)[source]

Updates values for the shr_strdata_nml namelist group.

This should be done once per stream, and it shouldn’t usually be called directly, since create_stream_file calls this method itself.

write_modelio_file(filename)[source]

Write mct component modelio files

write_nuopc_config_file(filename, data_list_path=None)[source]

Write the nuopc config file

write_nuopc_modelio_file(filename)[source]

Write nuopc component modelio files

write_output_file(namelist_file, data_list_path=None, groups=None, sorted_groups=True)[source]

Write out the namelists and input data files.

The namelist_file and modelio_file are the locations to which the component and modelio namelists will be written, respectively. The data_list_path argument is the location of the *.input_data_list file, which will have the input data files added to it.

write_seq_maps(filename)[source]

Write mct out seq_maps.rc

CIME.provenance module

Library for saving build/run provenance.

CIME.provenance.get_test_success(baseline_root, src_root, test, testing=False)[source]

Returns (was prev run success, commit when test last passed, commit when test last transitioned from pass to fail)

Unknown history is expressed as None

CIME.provenance.save_build_provenance(case, lid=None)[source]
CIME.provenance.save_postrun_provenance(case, lid=None)[source]
CIME.provenance.save_prerun_provenance(case, lid=None)[source]
CIME.provenance.save_test_success(baseline_root, src_root, test, succeeded, force_commit_test=None)[source]

Update success data accordingly based on succeeded flag

CIME.provenance.save_test_time(baseline_root, test, time_seconds)[source]

CIME.simple_compare module

CIME.simple_compare.compare_files(gold_file, compare_file, case=None)[source]

Returns true if files are the same, comments are returned too: (success, comments)

CIME.simple_compare.compare_runconfigfiles(gold_file, compare_file, case=None)[source]

Returns true if files are the same, comments are returned too: (success, comments)

CIME.simple_compare.findDiff(d1, d2, path='', case=None)[source]

CIME.test_scheduler module

A library for scheduling/running through the phases of a set of system tests. Supports phase-level parallelism (can make progres on multiple system tests at once).

TestScheduler will handle the TestStatus for the 1-time setup phases. All other phases need to handle their own status because they can be run outside the context of TestScheduler.

class CIME.test_scheduler.TestScheduler(test_names, test_data=None, no_run=False, no_build=False, no_setup=False, no_batch=None, test_root=None, test_id=None, machine_name=None, compiler=None, baseline_root=None, baseline_cmp_name=None, baseline_gen_name=None, clean=False, namelists_only=False, project=None, parallel_jobs=None, walltime=None, proc_pool=None, use_existing=False, save_timing=False, queue=None, allow_baseline_overwrite=False, output_root=None, force_procs=None, force_threads=None, mpilib=None, input_dir=None, pesfile=None, mail_user=None, mail_type=None, allow_pnl=False, non_local=False, single_exe=False, workflow=None)[source]

Bases: object

get_testnames()[source]
run_tests(wait=False, wait_check_throughput=False, wait_check_memory=False, wait_ignore_namelists=False, wait_ignore_memleak=False)[source]

Main API for this class.

Return True if all tests passed.

CIME.test_status module

Contains the crucial TestStatus class which manages phase-state of a test case and ensure that this state is represented by the TestStatus file in the case.

TestStatus objects are only modifiable via the set_status method and this is only allowed if the object is being accessed within the context of a context manager. Example:

with TestStatus(test_dir=caseroot) as ts:

ts.set_status(RUN_PHASE, TEST_PASS_STATUS)

This file also contains all of the hardcoded phase information which includes the phase names, phase orders, potential phase states, and which phases are required (core phases).

Additional important design decisions: 1) In order to ensure that incomplete tests are always left in a PEND

state, updating a core phase to a PASS state will automatically set the next core state to PEND.

  1. If the user repeats a core state, that invalidates all subsequent state. For example, if a user rebuilds their case, then any of the post-run states like the RUN state are no longer valid.

class CIME.test_status.TestStatus(test_dir=None, test_name=None, no_io=False)[source]

Bases: object

flush()[source]
get_comment(phase)[source]
get_name()[source]
get_overall_test_status(wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, no_run=False)[source]

Given the current phases and statuses, produce a single results for this test. Preference is given to PEND since we don’t want to stop waiting for a test that hasn’t finished. Namelist diffs are given the lowest precedence.

>>> _test_helper2('PASS ERS.foo.A RUN')
'PASS'
>>> _test_helper2('PASS ERS.foo.A SHAREDLIB_BUILD\nPEND ERS.foo.A RUN')
'PEND'
>>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN')
'FAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN')
'PASS'
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP')
'PASS'
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP', check_throughput=True)
'FAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
'NLFAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
'PEND'
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A MEMCOMP')
'PASS'
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP', ignore_namelists=True)
'PASS'
>>> _test_helper2('PASS ERS.foo.A COMPARE_1\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
'FAIL'
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
'DIFF'
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
'FAIL'
>>> _test_helper2('PEND ERS.foo.A COMPARE_2\nFAIL ERS.foo.A RUN')
'FAIL'
>>> _test_helper2('PEND ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
'PEND'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD')
'PASS'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD', wait_for_run=True)
'PEND'
>>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD', wait_for_run=True)
'FAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', wait_for_run=True)
'PEND'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN', wait_for_run=True)
'FAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN', wait_for_run=True)
'PASS'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN\nPEND ERS.foo.A COMPARE')
'FAIL'
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', no_run=True)
'PASS'
>>> s = '''PASS ERS.foo.A CREATE_NEWCASE
... PASS ERS.foo.A XML
... PASS ERS.foo.A SETUP
... PASS ERS.foo.A SHAREDLIB_BUILD time=454
... PASS ERS.foo.A NLCOMP
... PASS ERS.foo.A MODEL_BUILD time=363
... PASS ERS.foo.A SUBMIT
... PASS ERS.foo.A RUN time=73
... PEND ERS.foo.A COMPARE_base_single_thread
... FAIL ERS.foo.A BASELINE master: DIFF
... PASS ERS.foo.A TPUTCOMP
... PASS ERS.foo.A MEMLEAK insuffiencient data for memleak test
... PASS ERS.foo.A SHORT_TERM_ARCHIVER
... '''
>>> _test_helper2(s, no_perm=True)
'PEND'
get_status(phase)[source]
increment_non_pass_counts(non_pass_counts)[source]

Increment counts of the number of times given phases did not pass

non_pass_counts is a dictionary whose keys are phases of interest and whose values are running counts of the number of non-passes. This method increments those counts based on results in the given TestStatus object.

phase_statuses_dump(prefix='', skip_passes=False, skip_phase_list=None, xfails=None)[source]
Args:

prefix: string printed at the start of each line skip_passes: if True, do not output lines that have a PASS status skip_phase_list: list of phases (from the phases given by

ALL_PHASES) for which we skip output

xfails: object of type ExpectedFails, giving expected failures for this test

set_status(phase, status, comments='')[source]

Update the status of this test by changing the status of given phase to the given status.

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "PASS")
...     ts.set_status(XML_PHASE, "PASS")
...     ts.set_status(SETUP_PHASE, "FAIL")
...     ts.set_status(SETUP_PHASE, "PASS")
...     ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL")
...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42')), ('COMPARE_base_rest', ('FAIL', '')), ('MODEL_BUILD', ('PEND', ''))])

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "PASS")
...     ts.set_status(XML_PHASE, "PASS")
...     ts.set_status(SETUP_PHASE, "FAIL")
...     ts.set_status(SETUP_PHASE, "PASS")
...     ts.set_status(BASELINE_PHASE, "PASS")
...     ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL")
...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
...     ts.set_status(SETUP_PHASE, "PASS")
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PEND', ''))])

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "FAIL")
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('FAIL', ''))])

CIME.test_utils module

Utility functions used in test_scheduler.py, and by other utilities that need to get test lists.

CIME.test_utils.get_test_status_files(test_root, compiler, test_id=None)[source]
CIME.test_utils.get_tests_from_xml(xml_machine=None, xml_category=None, xml_compiler=None, xml_testlist=None, machine=None, compiler=None, driver=None)[source]

Parse testlists for a list of tests

CIME.test_utils.test_to_string(test, category_field_width=0, test_field_width=0, show_options=False)[source]

Given a test dictionary, return a string representation suitable for printing

Args:
test (dict): dictionary for a single test - e.g., one element from the

list returned by get_tests_from_xml

category_field_width (int): minimum amount of space to use for printing the test category test_field_width (int): minimum amount of space to use for printing the test category show_options (bool): if True, print test options, too (note that the ‘comment’

option is always printed, if present)

Basic functionality: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel’

Printing comments: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks’

Printing other options, too: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’, ‘wallclock’: ‘0:20’, ‘memleak_tolerance’: 0.2}} >>> test_to_string(mytest, 10, show_options=True) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks # memleak_tolerance: 0.2 # wallclock: 0:20’

CIME.user_mod_support module

user_mod_support.py

CIME.user_mod_support.apply_user_mods(caseroot, user_mods_path, keepexe=None)[source]

Recursivlely apply user_mods to caseroot - this includes updating user_nl_xxx, updating SourceMods and creating case shell_commands and xmlchange_cmds files

First remove case shell_commands files if any already exist

If this function is called multiple times, settings from later calls will take precedence over earlier calls, if there are conflicts.

keepexe is an optional argument that is needed for cases where apply_user_mods is called from create_clone

CIME.user_mod_support.build_include_dirs_list(user_mods_path, include_dirs=None)[source]

If user_mods_path has a file “include_user_mods” read that file and add directories to the include_dirs, recursively check each of those directories for further directories. The file may also include comments deleneated with # in the first column

CIME.utils module

Common functions used by cime python scripts Warning: you cannot use CIME Classes in this module as it causes circular dependencies

exception CIME.utils.CIMEError[source]

Bases: SystemExit, Exception

class CIME.utils.EnvironmentContext(**kwargs)[source]

Bases: object

Context manager for environment variables Usage:

os.environ[‘MYVAR’] = ‘oldvalue’ with EnvironmentContex(MYVAR=’myvalue’, MYVAR2=’myvalue2’):

print os.getenv(‘MYVAR’) # Should print myvalue. print os.getenv(‘MYVAR2’) # Should print myvalue2.

print os.getenv(‘MYVAR’) # Should print oldvalue. print os.getenv(‘MYVAR2’) # Should print None.

CREDIT: https://github.com/sakurai-youhei/envcontext

class CIME.utils.IndentFormatter(indent, fmt=None, datefmt=None)[source]

Bases: logging.Formatter

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class CIME.utils.SharedArea(new_perms=2)[source]

Bases: object

Enable 0002 umask within this manager

class CIME.utils.Timeout(seconds, action=None)[source]

Bases: object

A context manager that implements a timeout. By default, it will raise exception, but a custon function call can be provided. Provided None as seconds makes this class a no-op

CIME.utils.add_mail_type_args(parser)[source]
CIME.utils.analyze_build_log(comp, log, compiler)[source]

Capture and report warning count, capture and report errors and undefined references.

CIME.utils.append_case_status(phase, status, msg=None, caseroot='.')[source]

Update CaseStatus file

CIME.utils.append_status(msg, sfile, caseroot='.')[source]

Append msg to sfile in caseroot

CIME.utils.append_testlog(msg, caseroot='.')[source]

Add to TestStatus.log file

CIME.utils.batch_jobid()[source]
CIME.utils.check_for_python(filepath, funcname=None)[source]
CIME.utils.check_minimum_python_version(major, minor)[source]

Check your python version.

>>> check_minimum_python_version(sys.version_info[0], sys.version_info[1])
>>>
CIME.utils.check_name(fullname, additional_chars=None, fullpath=False)[source]

check for unallowed characters in name, this routine only checks the final name and does not check if path exists or is writable

>>> check_name("test.id", additional_chars=".")
False
>>> check_name("case.name", fullpath=False)
True
>>> check_name("/some/file/path/case.name", fullpath=True)
True
>>> check_name("mycase+mods")
False
>>> check_name("mycase?mods")
False
>>> check_name("mycase*mods")
False
>>> check_name("/some/full/path/name/")
False
CIME.utils.compute_total_time(job_cost_map, proc_pool)[source]

Given a map: jobname -> (procs, est-time), return a total time estimate for a given processor pool size

>>> job_cost_map = {"A" : (4, 3000), "B" : (2, 1000), "C" : (8, 2000), "D" : (1, 800)}
>>> compute_total_time(job_cost_map, 8)
5160
>>> compute_total_time(job_cost_map, 12)
3180
>>> compute_total_time(job_cost_map, 16)
3060
CIME.utils.convert_to_babylonian_time(seconds)[source]

Convert time value to seconds to HH:MM:SS

>>> convert_to_babylonian_time(3661)
'01:01:01'
CIME.utils.convert_to_seconds(time_str)[source]

Convert time value in [[HH:]MM:]SS to seconds

We assume that XX:YY is likely to be HH:MM, not MM:SS

>>> convert_to_seconds("42")
42
>>> convert_to_seconds("01:01:01")
3661
>>> convert_to_seconds("01:01")
3660
CIME.utils.convert_to_string(value, type_str=None, vid='')[source]

Convert value back to string. vid is only for generating better error messages. >>> convert_to_string(6, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6’, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6.0’, type_str=”real”) == ‘6.0’ True >>> convert_to_string(6.01, type_str=”real”) == ‘6.01’ True

CIME.utils.convert_to_type(value, type_str, vid='')[source]

Convert value from string to another type. vid is only for generating better error messages.

CIME.utils.convert_to_unknown_type(value)[source]

Convert value to it’s real type by probing conversions.

CIME.utils.copyifnewer(src, dest)[source]

if dest does not exist or is older than src copy src to dest

CIME.utils.does_file_have_string(filepath, text)[source]

Does the text string appear in the filepath file

CIME.utils.expect(condition, error_msg, exc_type=<class 'CIME.utils.CIMEError'>, error_prefix='ERROR:')[source]

Similar to assert except doesn’t generate an ugly stacktrace. Useful for checking user error, not programming error.

>>> expect(True, "error1")
>>> expect(False, "error2") 
Traceback (most recent call last):
    ...
CIMEError: ERROR: error2
CIME.utils.filter_unicode(unistr)[source]

Sometimes unicode chars can cause problems

CIME.utils.find_files(rootdir, pattern)[source]

recursively find all files matching a pattern

CIME.utils.find_proc_id(proc_name=None, children_only=False, of_parent=None)[source]

Children implies recursive.

CIME.utils.find_system_test(testname, case)[source]

Find and import the test matching testname Look through the paths set in config_files.xml variable SYSTEM_TESTS_DIR for components used in this case to find a test matching testname. Add the path to that directory to sys.path if its not there and return the test object Fail if the test is not found in any of the paths.

CIME.utils.format_time(time_format, input_format, input_time)[source]

Converts the string input_time from input_format to time_format Valid format specifiers are “%H”, “%M”, and “%S” % signs must be followed by an H, M, or S and then a separator Separators can be any string without digits or a % sign Each specifier can occur more than once in the input_format, but only the first occurence will be used. An example of a valid format: “%H:%M:%S” Unlike strptime, this does support %H >= 24

>>> format_time("%H:%M:%S", "%H", "43")
'43:00:00'
>>> format_time("%H  %M", "%M,%S", "59,59")
'0  59'
>>> format_time("%H, %S", "%H:%M:%S", "2:43:9")
'2, 09'
CIME.utils.get_all_cime_models()[source]
CIME.utils.get_batch_script_for_job(job)[source]
CIME.utils.get_charge_account(machobj=None, project=None)[source]

Hierarchy for choosing CHARGE_ACCOUNT: 1. Environment variable CHARGE_ACCOUNT 2. File $HOME/.cime/config 3. config_machines.xml (if machobj provided) 4. default to same value as PROJECT

>>> import CIME
>>> import CIME.XML.machines
>>> machobj = CIME.XML.machines.Machines(machine="theta")
>>> project = get_project(machobj)
>>> charge_account = get_charge_account(machobj, project)
>>> project == charge_account
True
>>> os.environ["CHARGE_ACCOUNT"] = "ChargeAccount"
>>> get_charge_account(machobj, project)
'ChargeAccount'
>>> del os.environ["CHARGE_ACCOUNT"]
CIME.utils.get_cime_config()[source]
CIME.utils.get_cime_default_driver()[source]
CIME.utils.get_cime_location_within_e3sm()[source]

From within e3sm, return subdirectory where CIME lives.

CIME.utils.get_cime_root(case=None)[source]

Return the absolute path to the root of CIME that contains this script

>>> os.path.isdir(os.path.join(get_cime_root(), get_scripts_location_within_cime()))
True
CIME.utils.get_current_branch(repo=None)[source]

Return the name of the current branch for a repository

>>> if "GIT_BRANCH" in os.environ:
...     get_current_branch() is not None
... else:
...     os.environ["GIT_BRANCH"] = "foo"
...     get_current_branch() == "foo"
True
CIME.utils.get_current_commit(short=False, repo=None, tag=False)[source]

Return the sha1 of the current HEAD commit

>>> get_current_commit() is not None
True
CIME.utils.get_current_submodule_status(recursive=False, repo=None)[source]

Return the sha1s of the current currently checked out commit for each submodule, along with the submodule path and the output of git describe for the SHA-1.

>>> get_current_submodule_status() is not None
True
CIME.utils.get_e3sm_root()[source]

Return the absolute path to the root of E3SM that contains this script

CIME.utils.get_full_test_name(partial_test, caseopts=None, grid=None, compset=None, machine=None, compiler=None, testmod=None)[source]

Given a partial CIME test name, return in form TESTCASE.GRID.COMPSET.MACHINE_COMPILER[.TESTMODS] Use the additional args to fill out the name if needed

>>> get_full_test_name("ERS", grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS", caseopts=["D", "P16"], grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS_D_P16.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16.JGF.melvin_gnu.mods", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu.mods'
>>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu", testmod="mods/test")
'ERS.ne16_fe16.JGF.melvin_gnu.mods-test'
CIME.utils.get_htmlroot(machobj=None)[source]

Get location for test HTML output

Hierarchy for choosing CIME_HTML_ROOT: 0. Environment variable CIME_HTML_ROOT 1. File $HOME/.cime/config 2. config_machines.xml (if machobj provided)

CIME.utils.get_lids(case)[source]
CIME.utils.get_logging_options()[source]

Use to pass same logging options as was used for current executable to subprocesses.

CIME.utils.get_model()[source]

Get the currently configured model value The CIME_MODEL env variable may or may not be set

>>> os.environ["CIME_MODEL"] = "garbage"
>>> get_model() 
Traceback (most recent call last):
    ...
CIMEError: ERROR: model garbage not recognized
>>> del os.environ["CIME_MODEL"]
>>> set_model('rocky') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: model rocky not recognized
>>> set_model('e3sm')
>>> get_model()
'e3sm'
>>> reset_cime_config()
CIME.utils.get_model_config_location_within_cime(model=None)[source]
CIME.utils.get_model_config_root(model=None)[source]

Get absolute path to model config area”

>>> os.path.isdir(get_model_config_root())
True
CIME.utils.get_project(machobj=None)[source]

Hierarchy for choosing PROJECT: 0. Command line flag to create_newcase or create_test 1. Environment variable PROJECT 2 Environment variable ACCOUNT (this is for backward compatibility) 3. File $HOME/.cime/config (this is new) 4 File $HOME/.cesm_proj (this is for backward compatibility) 5 config_machines.xml (if machobj provided)

CIME.utils.get_python_libs_location_within_cime()[source]

From within CIME, return subdirectory of python libraries

CIME.utils.get_python_libs_root()[source]

Get absolute path to scripts

>>> os.path.isdir(get_python_libs_root())
True
CIME.utils.get_scripts_location_within_cime()[source]

From within CIME, return subdirectory where scripts live.

CIME.utils.get_scripts_root()[source]

Get absolute path to scripts

>>> os.path.isdir(get_scripts_root())
True
CIME.utils.get_time_in_seconds(timeval, unit)[source]

Convert a time from ‘unit’ to seconds

CIME.utils.get_timestamp(timestamp_format='%Y%m%d_%H%M%S', utc_time=False)[source]

Get a string representing the current UTC time in format: YYYYMMDD_HHMMSS

The format can be changed if needed.

CIME.utils.get_umask()[source]
CIME.utils.get_urlroot(machobj=None)[source]

Get URL to htmlroot

Hierarchy for choosing CIME_URL_ROOT: 0. Environment variable CIME_URL_ROOT 1. File $HOME/.cime/config 2. config_machines.xml (if machobj provided)

CIME.utils.gunzip_existing_file(filepath)[source]
CIME.utils.gzip_existing_file(filepath)[source]

Gzips an existing file, removes the unzipped version, returns path to zip file. Note the that the timestamp of the original file will be maintained in the zipped file.

>>> import tempfile
>>> fd, filename = tempfile.mkstemp(text=True)
>>> _ = os.write(fd, b"Hello World")
>>> os.close(fd)
>>> gzfile = gzip_existing_file(filename)
>>> gunzip_existing_file(gzfile) == b'Hello World'
True
>>> os.remove(gzfile)
CIME.utils.id_generator(size=6, chars='abcdefghijklmnopqrstuvwxyz0123456789')[source]
CIME.utils.indent_string(the_string, indent_level)[source]

Indents the given string by a given number of spaces

Args:

the_string: str indent_level: int

Returns a new string that is the same as the_string, except that each line is indented by ‘indent_level’ spaces.

In python3, this can be done with textwrap.indent.

CIME.utils.is_last_process_complete(filepath, expect_text, fail_text)[source]

Search the filepath in reverse order looking for expect_text before finding fail_text. This utility is used by archive_metadata.

CIME.utils.is_python_executable(filepath)[source]
CIME.utils.ls_sorted_by_mtime(path)[source]

return list of path sorted by timestamp oldest first

CIME.utils.match_any(item, re_list)[source]

Return true if item matches any regex in re_list

CIME.utils.model_log(model, arg_logger, msg, debug_others=True)[source]
CIME.utils.new_lid()[source]
CIME.utils.normalize_case_id(case_id)[source]

Given a case_id, return it in form TESTCASE.GRID.COMPSET.PLATFORM

>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.G.20151121')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod.G.20151121')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod'
CIME.utils.parse_args_and_handle_standard_logging_options(args, parser=None)[source]

Guide to logging in CIME.

logger.debug -> Verbose/detailed output, use for debugging, off by default. Goes to a .log file logger.info -> Goes to stdout (and log if –debug). Use for normal program output logger.warning -> Goes to stderr (and log if –debug). Use for minor problems logger.error -> Goes to stderr (and log if –debug)

CIME.utils.parse_test_name(test_name)[source]

Given a CIME test name TESTCASE[_CASEOPTS].GRID.COMPSET[.MACHINE_COMPILER[.TESTMODS]], return each component of the testname with machine and compiler split. Do not error if a partial testname is provided (TESTCASE or TESTCASE.GRID) instead parse and return the partial results.

>>> parse_test_name('ERS')
['ERS', None, None, None, None, None, None]
>>> parse_test_name('ERS.fe12_123')
['ERS', None, 'fe12_123', None, None, None, None]
>>> parse_test_name('ERS.fe12_123.JGF')
['ERS', None, 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('ERS_D.fe12_123.JGF')
['ERS', ['D'], 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('ERS_D_P1.fe12_123.JGF')
['ERS', ['D', 'P1'], 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('SMS_D_Ln9_Mmpi-serial.f19_g16_rx1.A')
['SMS', ['D', 'Ln9', 'Mmpi-serial'], 'f19_g16_rx1', 'A', None, None, None]
>>> parse_test_name('ERS.fe12_123.JGF.machine_compiler')
['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', None]
>>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods')
['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', 'test/mods']
>>> parse_test_name('SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Expected 4th item of 'SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods' ('A_XLND_SICE_SOCN_XROF_XGLC_SWAV') to be in form machine_compiler
>>> parse_test_name('SMS.f19_g16.2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Invalid compset name 2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV
CIME.utils.redirect_logger(new_target, logger_name)[source]
CIME.utils.redirect_stderr(new_target)[source]
CIME.utils.redirect_stdout(new_target)[source]
CIME.utils.redirect_stdout_stderr(new_target)[source]
CIME.utils.reset_cime_config()[source]

Useful to keep unit tests from interfering with each other

CIME.utils.resolve_mail_type_args(args)[source]
CIME.utils.run_and_log_case_status(func, phase, caseroot='.', custom_success_msg_functor=None)[source]
CIME.utils.run_bld_cmd_ensure_logging(cmd, arg_logger, from_dir=None, timeout=None)[source]
CIME.utils.run_cmd(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None)[source]

Wrapper around subprocess to make it much more convenient to run shell commands

>>> run_cmd('ls file_i_hope_doesnt_exist')[0] != 0
True
CIME.utils.run_cmd_no_fail(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None)[source]

Wrapper around subprocess to make it much more convenient to run shell commands. Expects command to work. Just returns output string.

>>> run_cmd_no_fail('echo foo') == 'foo'
True
>>> run_cmd_no_fail('echo THE ERROR >&2; false') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Command: 'echo THE ERROR >&2; false' failed with error ...

>>> run_cmd_no_fail('grep foo', input_str=b'foo') == 'foo'
True
>>> run_cmd_no_fail('echo THE ERROR >&2', combine_output=True) == 'THE ERROR'
True
CIME.utils.run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None, from_dir=None, timeout=None)[source]

This code will try to import and run each cmd as a subroutine if that fails it will run it as a program in a seperate shell

Raises exception on failure.

CIME.utils.safe_copy(src_path, tgt_path, preserve_meta=True)[source]

A flexbile and safe copy routine. Will try to copy file and metadata, but this can fail if the current user doesn’t own the tgt file. A fallback data-only copy is attempted in this case. Works even if overwriting a read-only file.

tgt_path can be a directory, src_path must be a file

most of the complexity here is handling the case where the tgt_path file already exists. This problem does not exist for the tree operations so we don’t need to wrap those.

preserve_meta toggles if file meta-data, like permissions, should be preserved. If you are copying baseline files, you should be within a SharedArea context manager and preserve_meta should be false so that the umask set up by SharedArea can take affect regardless of the permissions of the src files.

CIME.utils.safe_recursive_copy(src_dir, tgt_dir, file_map)[source]

Copies a set of files from one dir to another. Works even if overwriting a read-only file. Files can be relative paths and the relative path will be matched on the tgt side.

CIME.utils.set_logger_indent(indent)[source]
CIME.utils.set_model(model)[source]

Set the model to be used in this session

CIME.utils.setup_standard_logging_options(parser)[source]
CIME.utils.start_buffering_output()[source]

All stdout, stderr will be buffered after this is called. This is python’s default behavior.

CIME.utils.stop_buffering_output()[source]

All stdout, stderr will not be buffered after this is called.

CIME.utils.string_in_list(_string, _list)[source]

Case insensitive search for string in list returns the matching list value >>> string_in_list(“Brack”,[“bar”, “bracK”, “foo”]) ‘bracK’ >>> string_in_list(“foo”, [“FFO”, “FOO”, “foo2”, “foo3”]) ‘FOO’ >>> string_in_list(“foo”, [“FFO”, “foo2”, “foo3”])

CIME.utils.stringify_bool(val)[source]

Makes a symlink from link_name to target. Unlike the standard os.symlink, this will work even if link_name already exists (in which case link_name will be overwritten).

CIME.utils.touch(fname)[source]
CIME.utils.transform_vars(text, case=None, subgroup=None, overrides=None, default=None)[source]

Do the variable substitution for any variables that need transforms recursively.

>>> transform_vars("{{ cesm_stdout }}", default="cesm.stdout")
'cesm.stdout'
>>> member_store = lambda : None
>>> member_store.foo = "hi"
>>> transform_vars("I say {{ foo }}", overrides={"foo":"hi"})
'I say hi'
CIME.utils.verbatim_success_msg(return_val)[source]
CIME.utils.wait_for_unlocked(filepath)[source]

CIME.wait_for_tests module

CIME.wait_for_tests.create_cdash_build_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]
CIME.wait_for_tests.create_cdash_config_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]
CIME.wait_for_tests.create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]
CIME.wait_for_tests.create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_time, hostname, force_log_upload)[source]
CIME.wait_for_tests.create_cdash_xml(results, cdash_build_name, cdash_project, cdash_build_group, force_log_upload=False)[source]
CIME.wait_for_tests.create_cdash_xml_boiler(phase, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit)[source]
CIME.wait_for_tests.create_cdash_xml_fakes(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname)[source]
CIME.wait_for_tests.get_test_output(test_path)[source]
CIME.wait_for_tests.get_test_time(test_path)[source]
CIME.wait_for_tests.set_up_signal_handlers()[source]
CIME.wait_for_tests.signal_handler(*_)[source]
CIME.wait_for_tests.wait_for_test(test_path, results, wait, check_throughput, check_memory, ignore_namelists, ignore_memleak, no_run)[source]
CIME.wait_for_tests.wait_for_tests(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, cdash_build_name=None, cdash_project='E3SM', cdash_build_group='ACME_Latest', timeout=None, force_log_upload=False, no_run=False, update_success=False)[source]
CIME.wait_for_tests.wait_for_tests_impl(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, no_run=False)[source]

Module contents