CIME package¶
Subpackages¶
- CIME.BuildTools package
- CIME.Servers package
- CIME.SystemTests package
- Subpackages
- Submodules
- CIME.SystemTests.dae module
- CIME.SystemTests.eri module
- CIME.SystemTests.erio module
- CIME.SystemTests.erp module
- CIME.SystemTests.err module
- CIME.SystemTests.erri module
- CIME.SystemTests.ers module
- CIME.SystemTests.ers2 module
- CIME.SystemTests.ert module
- CIME.SystemTests.homme module
- CIME.SystemTests.icp module
- CIME.SystemTests.irt module
- CIME.SystemTests.ldsta module
- CIME.SystemTests.mcc module
- CIME.SystemTests.nck module
- CIME.SystemTests.ncr module
- CIME.SystemTests.nodefail module
- CIME.SystemTests.pea module
- CIME.SystemTests.pem module
- CIME.SystemTests.pet module
- CIME.SystemTests.pfs module
- CIME.SystemTests.pre module
- CIME.SystemTests.rep module
- CIME.SystemTests.restart_tests module
- CIME.SystemTests.seq module
- CIME.SystemTests.sms module
- CIME.SystemTests.system_tests_common module
- CIME.SystemTests.system_tests_compare_two module
- Module contents
- CIME.XML package
- Submodules
- CIME.XML.archive module
- CIME.XML.archive_base module
- CIME.XML.batch module
- CIME.XML.compilerblock module
- CIME.XML.compilers module
- CIME.XML.component module
- CIME.XML.compsets module
- CIME.XML.entry_id module
- CIME.XML.env_archive module
- CIME.XML.env_base module
- CIME.XML.env_batch module
- CIME.XML.env_build module
- CIME.XML.env_case module
- CIME.XML.env_mach_pes module
- CIME.XML.env_mach_specific module
- CIME.XML.env_run module
- CIME.XML.env_test module
- CIME.XML.env_workflow module
- CIME.XML.files module
- CIME.XML.generic_xml module
- CIME.XML.grids module
- CIME.XML.headers module
- CIME.XML.inputdata module
- CIME.XML.machines module
- CIME.XML.namelist_definition module
- CIME.XML.pes module
- CIME.XML.pio module
- CIME.XML.standard_module_setup module
- CIME.XML.stream module
- CIME.XML.test_reporter module
- CIME.XML.testlist module
- CIME.XML.tests module
- CIME.XML.testspec module
- CIME.XML.workflow module
- Module contents
- CIME.case package
- Submodules
- CIME.case.case module
- CIME.case.case_clone module
- CIME.case.case_cmpgen_namelists module
- CIME.case.case_run module
- CIME.case.case_setup module
- CIME.case.case_st_archive module
- CIME.case.case_submit module
- CIME.case.case_test module
- CIME.case.check_input_data module
- CIME.case.check_lockedfiles module
- CIME.case.preview_namelists module
- Module contents
- CIME.tests package
Submodules¶
CIME.aprun module¶
Aprun is far too complex to handle purely through XML. We need python code to compute and assemble aprun commands.
CIME.bless_test_results module¶
-
CIME.bless_test_results.
bless_history
(test_name, test_dir, baseline_name, baseline_root, report_only, force)[source]¶
CIME.build module¶
functions for building CIME models
CIME.buildlib module¶
common utilities for buildlib
CIME.buildnml module¶
common implementation for building namelist commands
These are used by components/<model_type>/<component>/cime_config/buildnml
CIME.code_checker module¶
Libraries for checking python code with pylint
CIME.compare_namelists module¶
CIME.compare_test_results module¶
-
CIME.compare_test_results.
compare_namelists
(case, baseline_name, baseline_root, logfile_name)[source]¶
-
CIME.compare_test_results.
compare_test_results
(baseline_name, baseline_root, test_root, compiler, test_id=None, compare_tests=None, namelists_only=False, hist_only=False)[source]¶ Compares with baselines for all matching tests
Outputs results for each test to stdout (one line per test); possible status codes are: PASS, FAIL, SKIP. (A SKIP denotes a test that did not make it to the run phase or a test for which the run phase did not pass: we skip baseline comparisons in this case.)
In addition, creates files named compare.log.BASELINE_NAME.TIMESTAMP in each test directory, which contain more detailed output. Also creates *.cprnc.out.BASELINE_NAME.TIMESTAMP files in each run directory.
Returns True if all tests generated either PASS or SKIP results, False if there was at least one FAIL result.
CIME.date module¶
-
class
CIME.date.
date
(year=1, month=1, day=1, hour=0, minute=0, second=0)[source]¶ Bases:
object
Simple struct for holding dates and the time of day and performing comparisons
Difference in Hour, Minute, or Second >>> date(4, 5, 6, 9) == date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) != date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) < date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) <= date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) >= date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) > date(4, 5, 6, 8) True
>>> date(4, 5, 6, 4) == date(4, 5, 6, 8) False >>> date(4, 5, 6, 4) != date(4, 5, 6, 8) True >>> date(4, 5, 6, 4) < date(4, 5, 6, 8) True >>> date(4, 5, 6, 4) <= date(4, 5, 6, 8) True >>> date(4, 5, 6, 4) >= date(4, 5, 6, 8) False >>> date(4, 5, 6, 4) > date(4, 5, 6, 8) False
Difference in Day >>> date(4, 5, 8, 8) == date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) != date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) < date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) <= date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) >= date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) > date(4, 5, 6, 8) True
>>> date(4, 5, 5, 8) == date(4, 5, 6, 8) False >>> date(4, 5, 5, 8) != date(4, 5, 6, 8) True >>> date(4, 5, 5, 8) < date(4, 5, 6, 8) True >>> date(4, 5, 5, 8) <= date(4, 5, 6, 8) True >>> date(4, 5, 5, 8) >= date(4, 5, 6, 8) False >>> date(4, 5, 5, 8) > date(4, 5, 6, 8) False
Difference in Month >>> date(4, 6, 6, 8) == date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) != date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) < date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) <= date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) >= date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) > date(4, 5, 6, 8) True
>>> date(4, 4, 6, 8) == date(4, 5, 6, 8) False >>> date(4, 4, 6, 8) != date(4, 5, 6, 8) True >>> date(4, 4, 6, 8) < date(4, 5, 6, 8) True >>> date(4, 4, 6, 8) <= date(4, 5, 6, 8) True >>> date(4, 4, 6, 8) >= date(4, 5, 6, 8) False >>> date(4, 4, 6, 8) > date(4, 5, 6, 8) False
Difference in Year >>> date(5, 5, 6, 8) == date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) != date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) < date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) <= date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) >= date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) > date(4, 5, 6, 8) True
>>> date(3, 5, 6, 8) == date(4, 5, 6, 8) False >>> date(3, 5, 6, 8) != date(4, 5, 6, 8) True >>> date(3, 5, 6, 8) < date(4, 5, 6, 8) True >>> date(3, 5, 6, 8) <= date(4, 5, 6, 8) True >>> date(3, 5, 6, 8) >= date(4, 5, 6, 8) False >>> date(3, 5, 6, 8) > date(4, 5, 6, 8) False
-
CIME.date.
get_file_date
(filename)[source]¶ Returns the date associated with the filename as a date object representing the correct date Formats supported: “%Y-%m-%d_%h.%M.%s “%Y-%m-%d_%05s” “%Y-%m-%d-%05s” “%Y-%m-%d” “%Y-%m” “%Y.%m”
>>> get_file_date("./ne4np4_oQU240.cam.r.0001-01-06-00435.nc") date(1, 1, 6, 0, 7, 15) >>> get_file_date("./ne4np4_oQU240.cam.r.0010-1-06_00435.nc") date(10, 1, 6, 0, 7, 15) >>> get_file_date("./ne4np4_oQU240.cam.r.0010-10.nc") date(10, 10, 1, 0, 0, 0) >>> get_file_date("0064-3-8_10.20.30.nc") date(64, 3, 8, 10, 20, 30) >>> get_file_date("0140-3-5") date(140, 3, 5, 0, 0, 0) >>> get_file_date("0140-3") date(140, 3, 1, 0, 0, 0) >>> get_file_date("0140.3") date(140, 3, 1, 0, 0, 0)
CIME.get_timing module¶
Library for implementing getTiming tool which gets timing information from a run.
CIME.hist_utils module¶
Functions for actions pertaining to history files.
-
CIME.hist_utils.
compare_baseline
(case, baseline_dir=None, outfile_suffix='')[source]¶ compare the current test output to a baseline result
case - The case containing the hist files to be compared against baselines baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config outfile_suffix - if non-blank, then the cprnc output file name ends with
this suffix (with a ‘.’ added before the given suffix). if None, no output file saved.
returns (SUCCESS, comments) SUCCESS means all hist files matched their corresponding baseline
-
CIME.hist_utils.
compare_test
(case, suffix1, suffix2)[source]¶ Compares two sets of component history files in the testcase directory
case - The case containing the hist files to compare suffix1 - The suffix that identifies the first batch of hist files suffix1 - The suffix that identifies the second batch of hist files
returns (SUCCESS, comments)
-
CIME.hist_utils.
copy
(case, suffix)[source]¶ Copy the most recent batch of hist files in a case, adding the given suffix.
This can allow you to temporarily “save” these files so they won’t be blown away if you re-run the case.
case - The case containing the files you want to save suffix - The string suffix you want to add to saved files, this can be used to find them later.
-
CIME.hist_utils.
cprnc
(model, file1, file2, case, rundir, multiinst_driver_compare=False, outfile_suffix='')[source]¶ Run cprnc to compare two individual nc files
file1 - the full or relative path of the first file file2 - the full or relative path of the second file case - the case containing the files rundir - the rundir for the case outfile_suffix - if non-blank, then the output file name ends with this
suffix (with a ‘.’ added before the given suffix). Use None to avoid permissions issues in the case dir.
returns (True if the files matched, log_name)
-
CIME.hist_utils.
generate_baseline
(case, baseline_dir=None, allow_baseline_overwrite=False)[source]¶ copy the current test output to baseline result
case - The case containing the hist files to be copied into baselines baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config allow_baseline_overwrite must be true to generate baselines to an existing directory.
returns (SUCCESS, comments)
-
CIME.hist_utils.
get_extension
(model, filepath)[source]¶ For a hist file for the given model, return what we call the “extension”
model - The component model filepath - The path of the hist file
>>> get_extension("cpl", "cpl.hi.nc") 'hi' >>> get_extension("cpl", "cpl.h.nc") 'h' >>> get_extension("cpl", "cpl.h1.nc") 'h1' >>> get_extension("cpl", "TESTRUNDIFF_Mmpi-serial.f19_g16_rx1.A.melvin_gnu.C.fake_testing_only_20160816_164150-20160816_164240.cpl.hi.0.nc.base") 'hi' >>> get_extension("cpl", "TESTRUNDIFF_Mmpi-serial.f19_g16_rx1.A.melvin_gnu.C.fake_testing_only_20160816_164150-20160816_164240.cpl.h.nc") 'h' >>> get_extension("clm","clm2_0002.h0.1850-01-06-00000.nc") '0002.h0' >>> get_extension("pop","PFS.f09_g16.B1850.cheyenne_intel.allactive-default.GC.c2_0_b1f2_int.pop.h.ecosys.nday1.0001-01-02.nc") 'h'
CIME.locked_files module¶
CIME.namelist module¶
Module containing tools for dealing with Fortran namelists.
The public interface consists of the following functions: - character_literal_to_string - compress_literal_list - expand_literal_list - fortran_namelist_base_value - is_valid_fortran_name - is_valid_fortran_namelist_literal - literal_to_python_value - merge_literal_lists - parse - string_to_character_literal
In addition, the Namelist class represents a namelist held in memory.
For the moment, only a subset of namelist syntax is supported; specifically, we assume that only variables of intrinsic type are used, and indexing/co-indexing of arrays to set a portion of a variable is not supported. (However, null values and repeated values may be used to set or fill a variable as indexing would.)
We also always assume that a period (“.”) is the decimal separator, not a comma (“,”). We also assume that the file encoding is UTF-8 or some compatible format (e.g. ASCII).
Otherwise, most Fortran syntax rules implemented here are compatible with Fortran 2008 (which is largely the same as previous standards, and will be similar to Fortran 2015). The only exceptions should be cases where (a) we deliberately prohibit “troublesome” behavior that would be allowed by the standard, or (b) we rely on conventions shared by all major compilers.
One important convention is that newline characters can be used to denote the end of a record. This makes them equivalent to spaces at most locations in a Fortran namelist, except that newlines also end comments, and they are ignored entirely within strings.
While the treatment of comments in this module is standard, it may be somewhat surprising. Namelist comments are only allowed in two situations:
As the only thing on a line (aside from optional indentation with spaces).
(2) Immediately after a “value separator” (the space, newline, comma, or slash after a value).
This implies that all lines except for the last are syntax errors, in this example:
`
&group_name! This is not a valid comment because it's after the group name.
foo ! Neither is this, because it's between a name and an equals sign.
= 2 ! Nor this, because it comes between the value and the following comma.
, bar = ! Nor this, because it's between an equals sign and a value.
2! Nor this, because it should be separated from the value by a comma or space.
bazz = 3 ! Nor this, because it comes between the value and the following slash.
/! This is fine, but technically it is outside the namelist, not a comment.
`
However, the above would actually be valid if all the “comments” were removed. The Fortran standard is not clear about whether whitespace is allowed after inline comments and before subsequent non-whitespace text (!), but this module allows such whitespace, to preserve the sanity of both implementors and users.
The Fortran standard only applies to the interior of namelist groups, and not to text between one namelist group and the next. This module assumes that namelist groups are separated by (optional) whitespace and comments, and nothing else.
-
class
CIME.namelist.
Namelist
(groups=None)[source]¶ Bases:
object
Class representing a Fortran namelist.
Public methods: __init__ delete_variable get_group_names get_value get_variable_names get_variable_value merge_nl set_variable_value write
-
delete_variable
(group_name, variable_name)[source]¶ Delete a variable from a specified group.
If the specified group or variable does not exist, this is a no-op.
>>> x = parse(text='&foo bar=1 /') >>> x.delete_variable('FOO', 'BAR') >>> x.delete_variable('foo', 'bazz') >>> x.delete_variable('brack', 'bazz') >>> x.get_variable_names('foo') [] >>> x.get_variable_names('brack') []
-
get_group_names
()[source]¶ Return a list of all groups in the namelist.
>>> Namelist().get_group_names() [] >>> sorted(parse(text='&foo / &bar /').get_group_names()) ['bar', 'foo']
-
get_value
(variable_name)[source]¶ Return the value of a uniquely-named variable.
This function is similar to get_variable_value, except that it does not require a group_name, and it requires that the variable_name be unique across all groups.
>>> parse(text='&foo bar=1 / &bazz bar=1 /').get_value('bar') Traceback (most recent call last): ... SystemExit: ERROR: Namelist.get_value: Variable {} is present in multiple groups: ... >>> parse(text='&foo bar=1 / &bazz /').get_value('Bar') ['1'] >>> parse(text='&foo bar(2)=1 / &bazz /').get_value('Bar(2)') ['1'] >>> parse(text='&foo / &bazz /').get_value('bar') ['']
-
get_variable_names
(group_name)[source]¶ Return a list of all variables in the given namelist group.
If the specified group is not in the namelist, returns an empty list.
>>> Namelist().get_variable_names('foo') [] >>> x = parse(text='&foo bar=,bazz=true,bazz(2)=fred,bang=6*""/') >>> sorted(x.get_variable_names('fOo')) ['bang', 'bar', 'bazz', 'bazz(2)'] >>> x = parse(text='&foo bar=,bazz=true,bang=6*""/') >>> sorted(x.get_variable_names('fOo')) ['bang', 'bar', 'bazz'] >>> x = parse(text='&foo bar(::)=,bazz=false,bazz(2)=true,bazz(:2:)=6*""/') >>> sorted(x.get_variable_names('fOo')) ['bar(::)', 'bazz', 'bazz(2)', 'bazz(:2:)']
-
get_variable_value
(group_name, variable_name)[source]¶ Return the value of the specified variable.
This function always returns a non-empty list containing strings. If the specified group_name or variable_name is not present, [‘’] is returned.
>>> Namelist().get_variable_value('foo', 'bar') [''] >>> parse(text='&foo bar=1,2 /').get_variable_value('foo', 'bazz') [''] >>> parse(text='&foo bar=1,2 /').get_variable_value('foO', 'Bar') ['1', '2']
-
merge_nl
(other, overwrite=False)[source]¶ Merge this namelist object with another.
Values in the invoking (self) Namelist will take precedence over values in the other Namelist, unless overwrite=True is passed in, in which case other values take precedence.
>>> x = parse(text='&foo bar=1 bazz=,2 brat=3/') >>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /') >>> y.get_value('bazz') ['1', '1', '1'] >>> x.merge_nl(y) >>> sorted(x.get_group_names()) ['foo', 'foo2'] >>> sorted(x.get_variable_names('foo')) ['baker', 'bar', 'bazz', 'brat'] >>> sorted(x.get_variable_names('foo2')) ['barter'] >>> x.get_value('bar') ['1'] >>> x.get_value('bazz') ['1', '2', '1'] >>> x.get_value('brat') ['3'] >>> x.get_value('baker') ['4'] >>> x.get_value('barter') ['5'] >>> x = parse(text='&foo bar=1 bazz=,2 brat=3/') >>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /') >>> x.merge_nl(y, overwrite=True) >>> sorted(x.get_group_names()) ['foo', 'foo2'] >>> sorted(x.get_variable_names('foo')) ['baker', 'bar', 'bazz', 'brat'] >>> sorted(x.get_variable_names('foo2')) ['barter'] >>> x.get_value('bar') ['2'] >>> x.get_value('bazz') ['1', '1', '1'] >>> x.get_value('brat') ['3'] >>> x.get_value('baker') ['4'] >>> x.get_value('barter') ['5']
-
set_variable_value
(group_name, variable_name, value, var_size=1)[source]¶ Set the value of the specified variable.
>>> x = parse(text='&foo bar=1 /') >>> x.get_variable_value('foo', 'bar') ['1'] >>> x.set_variable_value('foo', 'bar(2)', ['3'], var_size=4) >>> x.get_variable_value('foo', 'bar') ['1', '3'] >>> x.set_variable_value('foo', 'bar(1)', ['2']) >>> x.get_variable_value('foo', 'bar') ['2', '3'] >>> x.set_variable_value('foo', 'bar', ['1']) >>> x.get_variable_value('foo', 'bar') ['1', '3'] >>> x.set_variable_value('foo', 'bazz', ['3']) >>> x.set_variable_value('Brack', 'baR', ['4']) >>> x.get_variable_value('foo', 'bazz') ['3'] >>> x.get_variable_value('brack', 'bar') ['4'] >>> x.set_variable_value('foo', 'red(2:6:2)', ['2', '4', '6'], var_size=12) >>> x.get_variable_value('foo', 'red') ['', '2', '', '4', '', '6']
-
write
(out_file, groups=None, append=False, format_='nml', sorted_groups=True)[source]¶ Write a Fortran namelist to a file.
As with parse, the out_file argument can be either a file name, or a file object with a write method that accepts unicode. If specified, the groups argument specifies a subset of all groups to write out.
If out_file is a file name, and append=True is passed in, the namelist will be appended to the named file instead of overwriting it. The append option has no effect if a file object is passed in.
The format_ option can be either ‘nml’ (namelist) or ‘rc’, and specifies the file format. Formats other than ‘nml’ may not support all possible output values.
-
-
CIME.namelist.
character_literal_to_string
(literal)[source]¶ Convert a Fortran character literal to a Python string.
This function assumes (without checking) that literal is a valid literal.
>>> character_literal_to_string("'blah'") 'blah' >>> character_literal_to_string('"blah"') 'blah' >>> character_literal_to_string("'don''t'") "don't" >>> character_literal_to_string('"' + '""Hello!""' + '"') '"Hello!"'
-
CIME.namelist.
compress_literal_list
(literals)[source]¶ Uses repetition syntax to shorten a literal list.
>>> compress_literal_list([]) [] >>> compress_literal_list(['true']) ['true'] >>> compress_literal_list(['1', '2', 'f*', '3', '3', '3', '5']) ['1', '2', 'f*', '3', '3', '3', '5'] >>> compress_literal_list(['f*', 'f*']) ['f*', 'f*']
-
CIME.namelist.
expand_literal_list
(literals)[source]¶ Expands a list of literal values to get rid of repetition syntax.
>>> expand_literal_list([]) [] >>> expand_literal_list(['true']) ['true'] >>> expand_literal_list(['1', '2', 'f*', '3*3', '5']) ['1', '2', 'f*', '3', '3', '3', '5'] >>> expand_literal_list(['2*f*']) ['f*', 'f*']
-
CIME.namelist.
fortran_namelist_base_value
(string)[source]¶ Strip off whitespace and repetition syntax from a namelist value.
>>> fortran_namelist_base_value("") '' >>> fortran_namelist_base_value("f") 'f' >>> fortran_namelist_base_value("6*") '' >>> fortran_namelist_base_value("6*f") 'f' >>> fortran_namelist_base_value(" \n6* \n") '' >>> fortran_namelist_base_value("\n 6*f\n ") 'f'
-
CIME.namelist.
get_fortran_name_only
(full_var)[source]¶ remove array section if any and return only the variable name >>> get_fortran_name_only(‘foo’) ‘foo’ >>> get_fortran_name_only(‘foo(3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’ >>> get_fortran_name_only(‘foo(1::)’) ‘foo’ >>> get_fortran_name_only(‘foo(:+2:)’) ‘foo’ >>> get_fortran_name_only(‘foo(::-3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’
-
CIME.namelist.
get_fortran_variable_indices
(varname, varlen=1, allow_any_len=False)[source]¶ get indices from a fortran namelist variable as a triplet of minindex, maxindex and step
>>> get_fortran_variable_indices('foo(3)') (3, 3, 1) >>> get_fortran_variable_indices('foo(1:2:3)') (1, 2, 3) >>> get_fortran_variable_indices('foo(::)', varlen=4) (1, 4, 1) >>> get_fortran_variable_indices('foo(::2)', varlen=4) (1, 4, 2) >>> get_fortran_variable_indices('foo(::)', allow_any_len=True) (1, -1, 1)
-
CIME.namelist.
is_valid_fortran_name
(string)[source]¶ Check that a variable name is allowed in Fortran.
The rules are: 1. The name must start with a letter. 2. All characters in a name must be alphanumeric (or underscores). 3. The maximum name length is 63 characters. 4. We only handle a single dimension !!!
>>> is_valid_fortran_name("") False >>> is_valid_fortran_name("a") True >>> is_valid_fortran_name("A") True >>> is_valid_fortran_name("A(4)") True >>> is_valid_fortran_name("A(::)") True >>> is_valid_fortran_name("A(1:2:3)") True >>> is_valid_fortran_name("A(1::)") True >>> is_valid_fortran_name("A(:-2:)") True >>> is_valid_fortran_name("A(1::+3)") True >>> is_valid_fortran_name("A(1,3)") False >>> is_valid_fortran_name("2") False >>> is_valid_fortran_name("_") False >>> is_valid_fortran_name("abc#123") False >>> is_valid_fortran_name("aLiBi_123") True >>> is_valid_fortran_name("A" * 64) False >>> is_valid_fortran_name("A" * 63) True
-
CIME.namelist.
is_valid_fortran_namelist_literal
(type_, string)[source]¶ Determine whether a literal is valid in a Fortran namelist.
Note that kind parameters are not allowed in namelists, which simplifies this check a bit. Internal whitespace is allowed for complex and character literals only. BOZ literals and compiler extensions (e.g. backslash escapes) are not allowed.
Null values, however, are allowed for all types. This means that passing in a string containing nothing but spaces and newlines will always cause True to be returned. Repetition (e.g. 5*’a’) is also allowed, including repetition of null values.
Detailed rules and examples follow.
Integers: Must be a sequence of one or more digits, with an optional sign.
>>> is_valid_fortran_namelist_literal("integer", "") True >>> is_valid_fortran_namelist_literal("integer", " ") True >>> is_valid_fortran_namelist_literal("integer", "\n") True >>> is_valid_fortran_namelist_literal("integer", "5*") True >>> is_valid_fortran_namelist_literal("integer", "1") True >>> is_valid_fortran_namelist_literal("integer", "5*1") True >>> is_valid_fortran_namelist_literal("integer", " 5*1") True >>> is_valid_fortran_namelist_literal("integer", "5* 1") False >>> is_valid_fortran_namelist_literal("integer", "5 *1") False >>> is_valid_fortran_namelist_literal("integer", "a") False >>> is_valid_fortran_namelist_literal("integer", " 1") True >>> is_valid_fortran_namelist_literal("integer", "1 ") True >>> is_valid_fortran_namelist_literal("integer", "1 2") False >>> is_valid_fortran_namelist_literal("integer", "0123456789") True >>> is_valid_fortran_namelist_literal("integer", "+22") True >>> is_valid_fortran_namelist_literal("integer", "-26") True >>> is_valid_fortran_namelist_literal("integer", "2A") False >>> is_valid_fortran_namelist_literal("integer", "1_8") False >>> is_valid_fortran_namelist_literal("integer", "2.1") False >>> is_valid_fortran_namelist_literal("integer", "2e6") False
Reals: - For fixed-point format, there is an optional sign, followed by an integer part, or a decimal point followed by a fractional part, or both. - Scientific notation is allowed, with an optional, case-insensitive “e” or “d” followed by an optionally-signed integer exponent. (Either the “e”/”d” or a sign must be present to separate the number from the exponent.) - The (case-insensitive) strings “inf”, “infinity”, and “nan” are allowed. NaN values can also contain additional information in parentheses, e.g. “NaN(x1234ABCD)”.
>>> is_valid_fortran_namelist_literal("real", "") True >>> is_valid_fortran_namelist_literal("real", "a") False >>> is_valid_fortran_namelist_literal("real", "1") True >>> is_valid_fortran_namelist_literal("real", " 1") True >>> is_valid_fortran_namelist_literal("real", "1 ") True >>> is_valid_fortran_namelist_literal("real", "1 2") False >>> is_valid_fortran_namelist_literal("real", "+1") True >>> is_valid_fortran_namelist_literal("real", "-1") True >>> is_valid_fortran_namelist_literal("real", "1.") True >>> is_valid_fortran_namelist_literal("real", "1.5") True >>> is_valid_fortran_namelist_literal("real", ".5") True >>> is_valid_fortran_namelist_literal("real", "+.5") True >>> is_valid_fortran_namelist_literal("real", ".") False >>> is_valid_fortran_namelist_literal("real", "+") False >>> is_valid_fortran_namelist_literal("real", "1e6") True >>> is_valid_fortran_namelist_literal("real", "1e-6") True >>> is_valid_fortran_namelist_literal("real", "1e+6") True >>> is_valid_fortran_namelist_literal("real", ".5e6") True >>> is_valid_fortran_namelist_literal("real", "1e") False >>> is_valid_fortran_namelist_literal("real", "1D6") True >>> is_valid_fortran_namelist_literal("real", "1q6") False >>> is_valid_fortran_namelist_literal("real", "1+6") True >>> is_valid_fortran_namelist_literal("real", "1.6.5") False >>> is_valid_fortran_namelist_literal("real", "1._8") False >>> is_valid_fortran_namelist_literal("real", "1,5") False >>> is_valid_fortran_namelist_literal("real", "inf") True >>> is_valid_fortran_namelist_literal("real", "INFINITY") True >>> is_valid_fortran_namelist_literal("real", "NaN") True >>> is_valid_fortran_namelist_literal("real", "nan(x56)") True >>> is_valid_fortran_namelist_literal("real", "nan())") False
Complex numbers: - A pair of real numbers enclosed by parentheses, and separated by a comma. - Any number of spaces or newlines may be placed before or after each real.
>>> is_valid_fortran_namelist_literal("complex", "") True >>> is_valid_fortran_namelist_literal("complex", "()") False >>> is_valid_fortran_namelist_literal("complex", "(,)") False >>> is_valid_fortran_namelist_literal("complex", "( ,\n)") False >>> is_valid_fortran_namelist_literal("complex", "(a,2.)") False >>> is_valid_fortran_namelist_literal("complex", "(1.,b)") False >>> is_valid_fortran_namelist_literal("complex", "(1,2)") True >>> is_valid_fortran_namelist_literal("complex", "(-1.e+06,+2.d-5)") True >>> is_valid_fortran_namelist_literal("complex", "(inf,NaN)") True >>> is_valid_fortran_namelist_literal("complex", "( 1. , 2. )") True >>> is_valid_fortran_namelist_literal("complex", "( \n \n 1. \n,\n 2.\n)") True >>> is_valid_fortran_namelist_literal("complex", " (1.,2.)") True >>> is_valid_fortran_namelist_literal("complex", "(1.,2.) ") True
Character sequences (strings): - Must begin and end with the same delimiter character, either a single quote (apostrophe), or a double quote (quotation mark). - Whichever character is used as a delimiter must not appear in the string itself, unless it appears in doubled pairs (e.g. ‘’‘’ or “’” are the two ways of representing a string containing a single apostrophe). - Note that newlines cannot be represented in a namelist character literal since they are interpreted as an “end of record”, but they are allowed as long as they don’t come between one of the aforementioned double pairs of characters.
>>> is_valid_fortran_namelist_literal("character", "") True >>> is_valid_fortran_namelist_literal("character", "''") True >>> is_valid_fortran_namelist_literal("character", " ''") True >>> is_valid_fortran_namelist_literal("character", "'\n'") True >>> is_valid_fortran_namelist_literal("character", "''\n''") False >>> is_valid_fortran_namelist_literal("character", "'''") False >>> is_valid_fortran_namelist_literal("character", "''''") True >>> is_valid_fortran_namelist_literal("character", "'''Cookie'''") True >>> is_valid_fortran_namelist_literal("character", "'''Cookie''") False >>> is_valid_fortran_namelist_literal("character", "'\"'") True >>> is_valid_fortran_namelist_literal("character", "'\"\"'") True >>> is_valid_fortran_namelist_literal("character", '""') True >>> is_valid_fortran_namelist_literal("character", '"" ') True >>> is_valid_fortran_namelist_literal("character", '"\n"') True >>> is_valid_fortran_namelist_literal("character", '""\n""') False >>> is_valid_fortran_namelist_literal("character", '""' + '"') False >>> is_valid_fortran_namelist_literal("character", '""' + '""') True >>> is_valid_fortran_namelist_literal("character", '"' + '""Cookie""' + '"') True >>> is_valid_fortran_namelist_literal("character", '""Cookie""' + '"') False >>> is_valid_fortran_namelist_literal("character", '"\'"') True >>> is_valid_fortran_namelist_literal("character", '"\'\'"') True
Logicals: - Must contain a (case-insensitive) “t” or “f”. - This must be either the first nonblank character, or the second following a period. - The rest of the string is ignored, but cannot contain a comma, newline, equals sign, slash, or space (except that trailing spaces are allowed and ignored).
>>> is_valid_fortran_namelist_literal("logical", "") True >>> is_valid_fortran_namelist_literal("logical", "t") True >>> is_valid_fortran_namelist_literal("logical", "F") True >>> is_valid_fortran_namelist_literal("logical", ".T") True >>> is_valid_fortran_namelist_literal("logical", ".f") True >>> is_valid_fortran_namelist_literal("logical", " f") True >>> is_valid_fortran_namelist_literal("logical", " .t") True >>> is_valid_fortran_namelist_literal("logical", "at") False >>> is_valid_fortran_namelist_literal("logical", ".TRUE.") True >>> is_valid_fortran_namelist_literal("logical", ".false.") True >>> is_valid_fortran_namelist_literal("logical", ".TEXAS$") True >>> is_valid_fortran_namelist_literal("logical", ".f=") False >>> is_valid_fortran_namelist_literal("logical", ".f/1") False >>> is_valid_fortran_namelist_literal("logical", ".t\nted") False >>> is_valid_fortran_namelist_literal("logical", ".Fant astic") False >>> is_valid_fortran_namelist_literal("logical", ".t2 ") True
-
CIME.namelist.
literal_to_python_value
(literal, type_=None)[source]¶ Convert a Fortran literal string to a Python value.
This function assumes that the input contains a single value, i.e. repetition syntax is not used. The type can be specified by passing a string as the type_ argument, or if this option is not provided, this function will attempt to autodetect the variable type.
Note that it is not possible to be certain whether a literal like “123” is intended to represent an integer or a floating-point value, however, nor can we be certain of the precision that will be used to hold this value in actual Fortran code. We also cannot use the optional information in a NaN float, so this will cause the function to throw an error if that information is present (e.g. a string like “NAN(1234)” will cause an error).
The Python type of the return value is as follows for different type_ arguments: “character” - str “complex” - complex “integer” - int “logical” - bool “real” - float
If a null value is input (i.e. the empty string), None will be returned.
>>> literal_to_python_value("'She''s a winner!'") "She's a winner!" >>> literal_to_python_value("1") 1 >>> literal_to_python_value("1.") 1.0 >>> literal_to_python_value(" (\n 1. , 2. )\n ") (1+2j) >>> literal_to_python_value(".true.") True >>> literal_to_python_value("Fortune") False >>> literal_to_python_value("bacon") Traceback (most recent call last): ... SystemExit: ERROR: 'bacon' is not a valid literal for any Fortran type. >>> literal_to_python_value("1", type_="real") 1.0 >>> literal_to_python_value("bacon", type_="logical") Traceback (most recent call last): ... SystemExit: ERROR: 'bacon' is not a valid literal of type 'logical'. >>> literal_to_python_value("1", type_="booga") Traceback (most recent call last): ... SystemExit: ERROR: Invalid Fortran type for a namelist: 'booga' >>> literal_to_python_value("2*1") Traceback (most recent call last): ... SystemExit: ERROR: Cannot use repetition syntax in literal_to_python_value >>> literal_to_python_value("") >>> literal_to_python_value("-1.D+10") -10000000000.0 >>> shouldRaise(ValueError, literal_to_python_value, "nan(1234)")
-
CIME.namelist.
merge_literal_lists
(default, overwrite)[source]¶ Merge two lists of literal value strings.
The overwrite values have higher precedence, so will overwrite the default values. However, if overwrite contains null values, or is shorter than default (and thus implicitly ends in null values), the elements of default will be used where overwrite is null.
>>> merge_literal_lists([], []) [] >>> merge_literal_lists(['true'], ['false']) ['false'] >>> merge_literal_lists([], ['false']) ['false'] >>> merge_literal_lists(['true'], ['']) ['true'] >>> merge_literal_lists([], ['']) [''] >>> merge_literal_lists(['true'], []) ['true'] >>> merge_literal_lists(['true'], []) ['true'] >>> merge_literal_lists(['3*false', '3*true'], ['true', '4*', 'false']) ['true', 'false', 'false', 'true', 'true', 'false']
-
CIME.namelist.
parse
(in_file=None, text=None, groupless=False, convert_tab_to_space=True)[source]¶ Parse a Fortran namelist.
The in_file argument must be either a str or unicode object containing a file name, or a text I/O object with a read method that returns the text of the namelist.
Alternatively, the text argument can be provided, in which case it must be the text of the namelist itself.
The groupless argument changes namelist parsing in two ways:
parse allows an alternate file format where no group names or slashes are present. In effect, the file is parsed as if an invisible, arbitrary group name was prepended, and an invisible slash was appended. However, if any group names actually are present, the file is parsed normally.
The return value of this function is not a Namelist object. Instead a single, flattened dictionary of name-value pairs is returned.
The convert_tab_to_space option can be used to force all tabs in the file to be converted to spaces, and is on by default. Note that this will usually allow files that use tabs as whitespace to be parsed. However, the implementation of this option is crude; it converts all tabs in the file, including those in character literals. (Note that there are many characters that cannot be passed in via namelist in any standard way, including ‘
- ‘,
so it is already a bad idea to assume that the namelist will preserve whitespace in strings, aside from simple spaces.)
The return value, if groupless=False, is a Namelist object.
All names and values returned are ultimately unicode strings. E.g. a value of “6*2” is returned as that string; it is not converted to 6 copies of the Python integer 2. Null values are returned as the empty string (“”).
-
CIME.namelist.
shouldRaise
(eclass, method, *args, **kw)[source]¶ A helper function to make doctests py3 compatible http://python3porting.com/problems.html#running-doctests
-
CIME.namelist.
string_to_character_literal
(string)[source]¶ Convert a Python string to a Fortran character literal.
This function always uses double quotes (“) as the delimiter.
>>> string_to_character_literal('blah') '"blah"' >>> string_to_character_literal("'blah'") '"\'blah\'"' >>> string_to_character_literal('She said "Hi!".') '"She said ""Hi!""."'
CIME.nmlgen module¶
Class for generating component namelists.
-
class
CIME.nmlgen.
NamelistGenerator
(case, definition_files, files=None)[source]¶ Bases:
object
Utility class for generating namelists for a given component.
-
add_default
(name, value=None, ignore_abs_path=None)[source]¶ Add a value for the specified variable to the namelist.
If the specified variable is already defined in the object, the existing value is preserved. Otherwise, the value argument, if provided, will be used to set the value. If no such value is found, the defaults file will be consulted. If null values are present in any of the above, the result will be a merged array of values.
If no value for the variable is found via any of the above, this method will raise an exception.
-
add_nmlcontents
(filename, group, append=True, format_='nmlcontents', sorted_groups=True)[source]¶ Write only contents of nml group
-
create_shr_strdata_nml
()[source]¶ Set defaults for shr_strdata_nml variables other than the variable domainfile
-
create_stream_file_and_update_shr_strdata_nml
(config, caseroot, stream, stream_path, data_list_path)[source]¶ Write the pseudo-XML file corresponding to a given stream.
Arguments: config - Used to look up namelist defaults. This is used in addition
to the config used to construct the namelist generator. The main reason to supply additional configuration options here is to specify stream-specific settings.
stream - Name of the stream. stream_path - Path to write the stream file to. data_list_path - Path of file to append input data information to.
-
get_default
(name, config=None, allow_none=False)[source]¶ Get the value of a variable from the namelist definition file.
The config argument is passed through to the underlying NamelistDefaults.get_value call as the attribute argument.
The return value of this function is a list of values that were found in the defaults file. If there is no matching default, this function returns None if allow_none=True is passed, otherwise an error is raised.
Note that we perform some translation of the values, since there are a few differences between Fortran namelist literals and values in the defaults file: 1) In the defaults file, whitespace is ignored except within strings, so
the output of this function strips out most whitespace. (This implies that commas are the only way to separate array elements in the defaults file.)
In the defaults file, quotes around character literals (strings) are optional, as long as the literal does not contain whitespace, commas, or (single or double) quotes. If a setting for a character variable does not seem to have quotes (and is not a null value), this function will add them.
Default values may refer to variables in a case’s env_*.xml files. This function replaces references of the form $VAR or ${VAR} with the value of the variable VAR in an env file, if that variable exists. This behavior is suppressed within single-quoted strings (similar to parameter expansion in shell scripts).
-
get_value
(name)[source]¶ Get the current value of a given namelist variable.
Note that the return value of this function is always a string or a list of strings. E.g. the scalar logical value .false. will be returned as “.false.”, while an array of two .false. values will be returned as [“.false.”, “.false.”]. Whether or not a value is scalar is determined by checking the array size in the namelist definition file.
Null values are converted to None, and repeated values are expanded, e.g. [‘2*3’] is converted to [‘3’, ‘3’, ‘3’].
For character variables, the value is converted to a Python string (e.g. quotation marks are removed).
All other literals are returned as the raw string values that will be written to the namelist.
-
init_defaults
(infiles, config, skip_groups=None, skip_entry_loop=None)[source]¶ Return array of names of all definition nodes
-
static
quote_string
(string)[source]¶ Convert a string to a quoted Fortran literal.
Does nothing if the string appears to be quoted already.
-
set_abs_file_path
(file_path)[source]¶ If file_path is relative, make it absolute using DIN_LOC_ROOT.
If an absolute path is input, it is returned unchanged.
-
set_value
(name, value)[source]¶ Set the current value of a given namelist variable.
Usually, you should use add_default instead of this function.
The name argument is the name of the variable to set, and the value is a list of strings to use as settings. If the variable is scalar, the list is optional; i.e. a scalar logical can be set using either value=’.false.’ or value=[‘.false.’]. If the variable is of type character, and the input is missing quotes, quotes will be added automatically. If None is provided in place of a string, this will be translated to a null value.
Note that this function will overwrite the current value, which may hold a user-specified setting. Even if value is (or contains) a null value, the old setting for the variable will be thrown out completely.
-
update_shr_strdata_nml
(config, stream, stream_path)[source]¶ Updates values for the shr_strdata_nml namelist group.
This should be done once per stream, and it shouldn’t usually be called directly, since create_stream_file calls this method itself.
-
write_output_file
(namelist_file, data_list_path=None, groups=None, sorted_groups=True)[source]¶ Write out the namelists and input data files.
The namelist_file and modelio_file are the locations to which the component and modelio namelists will be written, respectively. The data_list_path argument is the location of the *.input_data_list file, which will have the input data files added to it.
-
CIME.provenance module¶
Library for saving build/run provenance.
CIME.simple_compare module¶
CIME.test_scheduler module¶
A library for scheduling/running through the phases of a set of system tests. Supports phase-level parallelism (can make progres on multiple system tests at once).
TestScheduler will handle the TestStatus for the 1-time setup phases. All other phases need to handle their own status because they can be run outside the context of TestScheduler.
-
class
CIME.test_scheduler.
TestScheduler
(test_names, test_data=None, no_run=False, no_build=False, no_setup=False, no_batch=None, test_root=None, test_id=None, machine_name=None, compiler=None, baseline_root=None, baseline_cmp_name=None, baseline_gen_name=None, clean=False, namelists_only=False, project=None, parallel_jobs=None, walltime=None, proc_pool=None, use_existing=False, save_timing=False, queue=None, allow_baseline_overwrite=False, output_root=None, force_procs=None, force_threads=None, mpilib=None, input_dir=None, pesfile=None, mail_user=None, mail_type=None)[source]¶ Bases:
object
CIME.test_status module¶
Contains the crucial TestStatus class which manages phase-state of a test case and ensure that this state is represented by the TestStatus file in the case.
TestStatus objects are only modifiable via the set_status method and this is only allowed if the object is being accessed within the context of a context manager. Example:
- with TestStatus(test_dir=caseroot) as ts:
ts.set_status(RUN_PHASE, TEST_PASS_STATUS)
This file also contains all of the hardcoded phase information which includes the phase names, phase orders, potential phase states, and which phases are required (core phases).
Additional important design decisions: 1) In order to ensure that incomplete tests are always left in a PEND
state, updating a core phase to a PASS state will automatically set the next core state to PEND.
If the user repeats a core state, that invalidates all subsequent state. For example, if a user rebuilds their case, then any of the post-run states like the RUN state are no longer valid.
-
class
CIME.test_status.
TestStatus
(test_dir=None, test_name=None, no_io=False)[source]¶ Bases:
object
-
get_overall_test_status
(wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False)[source]¶ Given the current phases and statuses, produce a single results for this test. Preference is given to PEND since we don’t want to stop waiting for a test that hasn’t finished. Namelist diffs are given the lowest precedence.
>>> _test_helper2('PASS ERS.foo.A RUN') 'PASS' >>> _test_helper2('PASS ERS.foo.A SHAREDLIB_BUILD\nPEND ERS.foo.A RUN') 'PEND' >>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN') 'PEND' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN') 'PASS' >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP') 'PASS' >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP', check_throughput=True) 'FAIL' >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP') 'NLFAIL' >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A MEMCOMP') 'PASS' >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP', ignore_namelists=True) 'PASS' >>> _test_helper2('PASS ERS.foo.A COMPARE_1\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN') 'FAIL' >>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN') 'DIFF' >>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN') 'FAIL' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD') 'PASS' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD', wait_for_run=True) 'PEND' >>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD', wait_for_run=True) 'FAIL' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', wait_for_run=True) 'PEND' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN', wait_for_run=True) 'FAIL' >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN', wait_for_run=True) 'PASS'
-
set_status
(phase, status, comments='')[source]¶ Update the status of this test by changing the status of given phase to the given status.
>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts: ... ts.set_status(CREATE_NEWCASE_PHASE, "PASS") ... ts.set_status(XML_PHASE, "PASS") ... ts.set_status(SETUP_PHASE, "FAIL") ... ts.set_status(SETUP_PHASE, "PASS") ... ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL") ... ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42') >>> ts._phase_statuses OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42')), ('COMPARE_base_rest', ('FAIL', '')), ('MODEL_BUILD', ('PEND', ''))])
>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts: ... ts.set_status(CREATE_NEWCASE_PHASE, "PASS") ... ts.set_status(XML_PHASE, "PASS") ... ts.set_status(SETUP_PHASE, "FAIL") ... ts.set_status(SETUP_PHASE, "PASS") ... ts.set_status(BASELINE_PHASE, "PASS") ... ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL") ... ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42') ... ts.set_status(SETUP_PHASE, "PASS") >>> ts._phase_statuses OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PEND', ''))])
>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts: ... ts.set_status(CREATE_NEWCASE_PHASE, "FAIL") >>> ts._phase_statuses OrderedDict([('CREATE_NEWCASE', ('FAIL', ''))])
-
CIME.test_utils module¶
Utility functions used in test_scheduler.py, and by other utilities that need to get test lists.
-
CIME.test_utils.
get_tests_from_xml
(xml_machine=None, xml_category=None, xml_compiler=None, xml_testlist=None, machine=None, compiler=None)[source]¶ Parse testlists for a list of tests
-
CIME.test_utils.
test_to_string
(test, category_field_width=0, test_field_width=0, show_options=False)[source]¶ Given a test dictionary, return a string representation suitable for printing
- Args:
- test (dict): dictionary for a single test - e.g., one element from the
list returned by get_tests_from_xml
category_field_width (int): minimum amount of space to use for printing the test category test_field_width (int): minimum amount of space to use for printing the test category show_options (bool): if True, print test options, too (note that the ‘comment’
option is always printed, if present)
Basic functionality: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel’
Printing comments: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks’
Printing other options, too: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’, ‘wallclock’: ‘0:20’, ‘memleak_tolerance’: 0.2}} >>> test_to_string(mytest, 10, show_options=True) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks # memleak_tolerance: 0.2 # wallclock: 0:20’
CIME.user_mod_support module¶
user_mod_support.py
-
CIME.user_mod_support.
apply_user_mods
(caseroot, user_mods_path, keepexe=None)[source]¶ Recursivlely apply user_mods to caseroot - this includes updating user_nl_xxx, updating SourceMods and creating case shell_commands and xmlchange_cmds files
First remove case shell_commands files if any already exist
If this function is called multiple times, settings from later calls will take precedence over earlier calls, if there are conflicts.
keepexe is an optional argument that is needed for cases where apply_user_mods is called from create_clone
-
CIME.user_mod_support.
build_include_dirs_list
(user_mods_path, include_dirs=None)[source]¶ If user_mods_path has a file “include_user_mods” read that file and add directories to the include_dirs, recursively check each of those directories for further directories. The file may also include comments deleneated with # in the first column
CIME.utils module¶
Common functions used by cime python scripts Warning: you cannot use CIME Classes in this module as it causes circular dependencies
-
class
CIME.utils.
EnvironmentContext
(**kwargs)[source]¶ Bases:
object
Context manager for environment variables Usage:
os.environ[‘MYVAR’] = ‘oldvalue’ with EnvironmentContex(MYVAR=’myvalue’, MYVAR2=’myvalue2’):
print os.getenv(‘MYVAR’) # Should print myvalue. print os.getenv(‘MYVAR2’) # Should print myvalue2.
print os.getenv(‘MYVAR’) # Should print oldvalue. print os.getenv(‘MYVAR2’) # Should print None.
-
class
CIME.utils.
IndentFormatter
(indent, fmt=None, datefmt=None)[source]¶ Bases:
logging.Formatter
-
format
(record)[source]¶ Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
-
Bases:
object
Enable 0002 umask within this manager
-
class
CIME.utils.
Timeout
(seconds, action=None)[source]¶ Bases:
object
A context manager that implements a timeout. By default, it will raise exception, but a custon function call can be provided. Provided None as seconds makes this class a no-op
-
CIME.utils.
analyze_build_log
(comp, log, compiler)[source]¶ Capture and report warning count, capture and report errors and undefined references.
-
CIME.utils.
append_case_status
(phase, status, msg=None, caseroot='.')[source]¶ Update CaseStatus file
-
CIME.utils.
check_minimum_python_version
(major, minor)[source]¶ Check your python version.
>>> check_minimum_python_version(sys.version_info[0], sys.version_info[1]) >>>
-
CIME.utils.
check_name
(fullname, additional_chars=None, fullpath=False)[source]¶ check for unallowed characters in name, this routine only checks the final name and does not check if path exists or is writable
>>> check_name("test.id", additional_chars=".") False >>> check_name("case.name", fullpath=False) True >>> check_name("/some/file/path/case.name", fullpath=True) True >>> check_name("mycase+mods") False >>> check_name("mycase?mods") False >>> check_name("mycase*mods") False >>> check_name("/some/full/path/name/") False
-
CIME.utils.
compute_total_time
(job_cost_map, proc_pool)[source]¶ Given a map: jobname -> (procs, est-time), return a total time estimate for a given processor pool size
>>> job_cost_map = {"A" : (4, 3000), "B" : (2, 1000), "C" : (8, 2000), "D" : (1, 800)} >>> compute_total_time(job_cost_map, 8) 5160 >>> compute_total_time(job_cost_map, 12) 3180 >>> compute_total_time(job_cost_map, 16) 3060
-
CIME.utils.
convert_to_babylonian_time
(seconds)[source]¶ Convert time value to seconds to HH:MM:SS
>>> convert_to_babylonian_time(3661) '01:01:01'
-
CIME.utils.
convert_to_seconds
(time_str)[source]¶ Convert time value in [[HH:]MM:]SS to seconds
>>> convert_to_seconds("42") 42 >>> convert_to_seconds("01:01:01") 3661
-
CIME.utils.
convert_to_string
(value, type_str=None, vid='')[source]¶ Convert value back to string. vid is only for generating better error messages. >>> convert_to_string(6, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6’, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6.0’, type_str=”real”) == ‘6.0’ True >>> convert_to_string(6.01, type_str=”real”) == ‘6.01’ True
-
CIME.utils.
convert_to_type
(value, type_str, vid='')[source]¶ Convert value from string to another type. vid is only for generating better error messages.
-
CIME.utils.
convert_to_unknown_type
(value)[source]¶ Convert value to it’s real type by probing conversions.
-
CIME.utils.
copy_umask
(src, dst)[source]¶ Preserves all file metadata except making sure new file obeys umask
-
CIME.utils.
copyifnewer
(src, dest)[source]¶ if dest does not exist or is older than src copy src to dest
-
CIME.utils.
does_file_have_string
(filepath, text)[source]¶ Does the text string appear in the filepath file
-
CIME.utils.
expect
(condition, error_msg, exc_type=<class 'SystemExit'>, error_prefix='ERROR:')[source]¶ Similar to assert except doesn’t generate an ugly stacktrace. Useful for checking user error, not programming error.
>>> expect(True, "error1") >>> expect(False, "error2") Traceback (most recent call last): ... SystemExit: ERROR: error2
-
CIME.utils.
find_proc_id
(proc_name=None, children_only=False, of_parent=None)[source]¶ Children implies recursive.
-
CIME.utils.
find_system_test
(testname, case)[source]¶ Find and import the test matching testname Look through the paths set in config_files.xml variable SYSTEM_TESTS_DIR for components used in this case to find a test matching testname. Add the path to that directory to sys.path if its not there and return the test object Fail if the test is not found in any of the paths.
-
CIME.utils.
format_time
(time_format, input_format, input_time)[source]¶ Converts the string input_time from input_format to time_format Valid format specifiers are “%H”, “%M”, and “%S” % signs must be followed by an H, M, or S and then a separator Separators can be any string without digits or a % sign Each specifier can occur more than once in the input_format, but only the first occurence will be used. An example of a valid format: “%H:%M:%S” Unlike strptime, this does support %H >= 24
>>> format_time("%H:%M:%S", "%H", "43") '43:00:00' >>> format_time("%H %M", "%M,%S", "59,59") '0 59' >>> format_time("%H, %S", "%H:%M:%S", "2:43:9") '2, 09'
-
CIME.utils.
get_charge_account
(machobj=None)[source]¶ Hierarchy for choosing CHARGE_ACCOUNT: 1. Environment variable CHARGE_ACCOUNT 2. File $HOME/.cime/config 3. config_machines.xml (if machobj provided) 4. default to same value as PROJECT
>>> import CIME >>> import CIME.XML.machines >>> machobj = CIME.XML.machines.Machines(machine="theta") >>> project = get_project(machobj) >>> charge_account = get_charge_account(machobj) >>> project == charge_account True >>> os.environ["CHARGE_ACCOUNT"] = "ChargeAccount" >>> get_charge_account(machobj) 'ChargeAccount' >>> del os.environ["CHARGE_ACCOUNT"]
-
CIME.utils.
get_cime_location_within_e3sm
()[source]¶ From within e3sm, return subdirectory where CIME lives.
-
CIME.utils.
get_cime_root
(case=None)[source]¶ Return the absolute path to the root of CIME that contains this script
>>> os.path.isdir(os.path.join(get_cime_root(), get_scripts_location_within_cime())) True
-
CIME.utils.
get_current_branch
(repo=None)[source]¶ Return the name of the current branch for a repository
>>> if "GIT_BRANCH" in os.environ: ... get_current_branch() is not None ... else: ... os.environ["GIT_BRANCH"] = "foo" ... get_current_branch() == "foo" True
-
CIME.utils.
get_current_commit
(short=False, repo=None, tag=False)[source]¶ Return the sha1 of the current HEAD commit
>>> get_current_commit() is not None True
-
CIME.utils.
get_e3sm_root
()[source]¶ Return the absolute path to the root of E3SM that contains this script
-
CIME.utils.
get_full_test_name
(partial_test, caseopts=None, grid=None, compset=None, machine=None, compiler=None, testmod=None)[source]¶ Given a partial CIME test name, return in form TESTCASE.GRID.COMPSET.MACHINE_COMPILER[.TESTMODS] Use the additional args to fill out the name if needed
>>> get_full_test_name("ERS", grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu") 'ERS.ne16_fe16.JGF.melvin_gnu' >>> get_full_test_name("ERS", caseopts=["D", "P16"], grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu") 'ERS_D_P16.ne16_fe16.JGF.melvin_gnu' >>> get_full_test_name("ERS.ne16_fe16", compset="JGF", machine="melvin", compiler="gnu") 'ERS.ne16_fe16.JGF.melvin_gnu' >>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu") 'ERS.ne16_fe16.JGF.melvin_gnu' >>> get_full_test_name("ERS.ne16_fe16.JGF.melvin_gnu.mods", machine="melvin", compiler="gnu") 'ERS.ne16_fe16.JGF.melvin_gnu.mods' >>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu", testmod="mods/test") 'ERS.ne16_fe16.JGF.melvin_gnu.mods-test'
-
CIME.utils.
get_logging_options
()[source]¶ Use to pass same logging options as was used for current executable to subprocesses.
-
CIME.utils.
get_model
()[source]¶ Get the currently configured model value The CIME_MODEL env variable may or may not be set
>>> os.environ["CIME_MODEL"] = "garbage" >>> del os.environ["CIME_MODEL"] >>> set_model('rocky') >>> get_model() 'rocky' >>> reset_cime_config()
-
CIME.utils.
get_model_config_root
(model=None)[source]¶ Get absolute path to model config area”
>>> os.path.isdir(get_model_config_root()) True
-
CIME.utils.
get_project
(machobj=None)[source]¶ Hierarchy for choosing PROJECT: 0. Command line flag to create_newcase or create_test 1. Environment variable PROJECT 2 Environment variable ACCOUNT (this is for backward compatibility) 3. File $HOME/.cime/config (this is new) 4 File $HOME/.cesm_proj (this is for backward compatibility) 5 config_machines.xml (if machobj provided)
-
CIME.utils.
get_python_libs_location_within_cime
()[source]¶ From within CIME, return subdirectory of python libraries
-
CIME.utils.
get_python_libs_root
()[source]¶ Get absolute path to scripts
>>> os.path.isdir(get_python_libs_root()) True
-
CIME.utils.
get_scripts_location_within_cime
()[source]¶ From within CIME, return subdirectory where scripts live.
-
CIME.utils.
get_scripts_root
()[source]¶ Get absolute path to scripts
>>> os.path.isdir(get_scripts_root()) True
-
CIME.utils.
get_timestamp
(timestamp_format='%Y%m%d_%H%M%S', utc_time=False)[source]¶ Get a string representing the current UTC time in format: YYYYMMDD_HHMMSS
The format can be changed if needed.
-
CIME.utils.
gzip_existing_file
(filepath)[source]¶ Gzips an existing file, removes the unzipped version, returns path to zip file. Note the that the timestamp of the original file will be maintained in the zipped file.
>>> import tempfile >>> fd, filename = tempfile.mkstemp(text=True) >>> _ = os.write(fd, b"Hello World") >>> os.close(fd) >>> gzfile = gzip_existing_file(filename) >>> gunzip_existing_file(gzfile) == b'Hello World' True >>> os.remove(gzfile)
-
CIME.utils.
indent_string
(the_string, indent_level)[source]¶ Indents the given string by a given number of spaces
- Args:
the_string: str indent_level: int
Returns a new string that is the same as the_string, except that each line is indented by ‘indent_level’ spaces.
In python3, this can be done with textwrap.indent.
-
CIME.utils.
is_last_process_complete
(filepath, expect_text, fail_text)[source]¶ Search the filepath in reverse order looking for expect_text before finding fail_text. This utility is used by archive_metadata.
-
CIME.utils.
normalize_case_id
(case_id)[source]¶ Given a case_id, return it in form TESTCASE.GRID.COMPSET.PLATFORM
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel') 'ERT.ne16_g37.B1850C5.sandiatoss3_intel' >>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod') 'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod' >>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.G.20151121') 'ERT.ne16_g37.B1850C5.sandiatoss3_intel' >>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod.G.20151121') 'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod'
-
CIME.utils.
parse_args_and_handle_standard_logging_options
(args, parser=None)[source]¶ Guide to logging in CIME.
logger.debug -> Verbose/detailed output, use for debugging, off by default. Goes to a .log file logger.info -> Goes to stdout (and log if –debug). Use for normal program output logger.warning -> Goes to stderr (and log if –debug). Use for minor problems logger.error -> Goes to stderr (and log if –debug)
-
CIME.utils.
parse_test_name
(test_name)[source]¶ Given a CIME test name TESTCASE[_CASEOPTS].GRID.COMPSET[.MACHINE_COMPILER[.TESTMODS]], return each component of the testname with machine and compiler split. Do not error if a partial testname is provided (TESTCASE or TESTCASE.GRID) instead parse and return the partial results.
>>> parse_test_name('ERS') ['ERS', None, None, None, None, None, None] >>> parse_test_name('ERS.fe12_123') ['ERS', None, 'fe12_123', None, None, None, None] >>> parse_test_name('ERS.fe12_123.JGF') ['ERS', None, 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('ERS_D.fe12_123.JGF') ['ERS', ['D'], 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('ERS_D_P1.fe12_123.JGF') ['ERS', ['D', 'P1'], 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('SMS_D_Ln9_Mmpi-serial.f19_g16_rx1.A') ['SMS', ['D', 'Ln9', 'Mmpi-serial'], 'f19_g16_rx1', 'A', None, None, None] >>> parse_test_name('ERS.fe12_123.JGF.machine_compiler') ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', None] >>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods') ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', 'test/mods'] >>> parse_test_name('SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') Traceback (most recent call last): ... SystemExit: ERROR: Expected 4th item of 'SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods' ('A_XLND_SICE_SOCN_XROF_XGLC_SWAV') to be in form machine_compiler >>> parse_test_name('SMS.f19_g16.2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') Traceback (most recent call last): ... SystemExit: ERROR: Invalid compset name 2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV
-
CIME.utils.
run_and_log_case_status
(func, phase, caseroot='.', custom_success_msg_functor=None)[source]¶
-
CIME.utils.
run_cmd
(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None)[source]¶ Wrapper around subprocess to make it much more convenient to run shell commands
>>> run_cmd('ls file_i_hope_doesnt_exist')[0] != 0 True
-
CIME.utils.
run_cmd_no_fail
(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None)[source]¶ Wrapper around subprocess to make it much more convenient to run shell commands. Expects command to work. Just returns output string.
>>> run_cmd_no_fail('echo foo') == 'foo' True >>> run_cmd_no_fail('echo THE ERROR >&2; false') Traceback (most recent call last): ... SystemExit: ERROR: Command: 'echo THE ERROR >&2; false' failed with error ...
>>> run_cmd_no_fail('grep foo', input_str=b'foo') == 'foo' True >>> run_cmd_no_fail('echo THE ERROR >&2', combine_output=True) == 'THE ERROR' True
-
CIME.utils.
run_sub_or_cmd
(cmd, cmdargs, subname, subargs, logfile=None, case=None, from_dir=None, timeout=None)[source]¶ This code will try to import and run each cmd as a subroutine if that fails it will run it as a program in a seperate shell
Raises exception on failure.
-
CIME.utils.
safe_copy
(src_path, tgt_path)[source]¶ A flexbile and safe copy routine. Will try to copy file and metadata, but this can fail if the current user doesn’t own the tgt file. A fallback data-only copy is attempted in this case. Works even if overwriting a read-only file.
tgt_path can be a directory, src_path must be a file
most of the complexity here is handling the case where the tgt_path file already exists. This problem does not exist for the tree operations so we don’t need to wrap those.
-
CIME.utils.
safe_recursive_copy
(src_dir, tgt_dir, file_map)[source]¶ Copies a set of files from one dir to another. Works even if overwriting a read-only file. Files can be relative paths and the relative path will be matched on the tgt side.
-
CIME.utils.
start_buffering_output
()[source]¶ All stdout, stderr will be buffered after this is called. This is python’s default behavior.
-
CIME.utils.
stop_buffering_output
()[source]¶ All stdout, stderr will not be buffered after this is called.
-
CIME.utils.
symlink_force
(target, link_name)[source]¶ Makes a symlink from link_name to target. Unlike the standard os.symlink, this will work even if link_name already exists (in which case link_name will be overwritten).
-
CIME.utils.
transform_vars
(text, case=None, subgroup=None, overrides=None, default=None)[source]¶ Do the variable substitution for any variables that need transforms recursively.
>>> transform_vars("{{ cesm_stdout }}", default="cesm.stdout") 'cesm.stdout' >>> member_store = lambda : None >>> member_store.foo = "hi" >>> transform_vars("I say {{ foo }}", overrides={"foo":"hi"}) 'I say hi'
CIME.wait_for_tests module¶
-
CIME.wait_for_tests.
create_cdash_test_xml
(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname)[source]¶
-
CIME.wait_for_tests.
create_cdash_upload_xml
(results, cdash_build_name, cdash_build_group, utc_time, hostname, force_log_upload)[source]¶
-
CIME.wait_for_tests.
create_cdash_xml
(results, cdash_build_name, cdash_project, cdash_build_group, force_log_upload=False)[source]¶
-
CIME.wait_for_tests.
wait_for_test
(test_path, results, wait, check_throughput, check_memory, ignore_namelists, ignore_memleak)[source]¶