CIME package

Subpackages

Submodules

CIME.aprun module

Aprun is far too complex to handle purely through XML. We need python code to compute and assemble aprun commands.

CIME.aprun.get_aprun_cmd_for_case(case, run_exe, overrides=None, extra_args=None)[source]: Given a case, construct and return the aprun command and optimized node count

CIME.bless_test_results module

CIME.bless_test_results.bless_history(test_name, case, baseline_name, baseline_root, report_only, force)[source]

CIME.bless_test_results.bless_namelists(test_name, report_only, force, pes_file, baseline_name, baseline_root, new_test_root=None, new_test_id=None)[source]

CIME.bless_test_results.bless_test_results(baseline_name, baseline_root, test_root, compiler, test_id=None, namelists_only=False, hist_only=False, report_only=False, force=False, pes_file=None, bless_tests=None, no_skip_pass=False, new_test_root=None, new_test_id=None, exclude=None, bless_tput=False, bless_mem=False, bless_perf=False, **_)[source]

CIME.bless_test_results.is_hist_bless_needed(test_name, ts, broken_blesses, overall_result, no_skip_pass, phase)[source]

CIME.build module

functions for building CIME models

class CIME.build.CmakeTmpBuildDir(macroloc=None, rootdir=None, tmpdir=None)[source]

Bases: object

Use to create a temporary cmake build dir for the purposes of querying Macros.

get_full_tmpdir()[source]

get_makefile_vars(case=None, comp=None, cmake_args=None)[source]

Run cmake and process output to a list of variable settings

case can be None if caller is providing their own cmake args

CIME.build.case_build(caseroot, case, sharedlib_only=False, model_only=False, buildlist=None, save_build_provenance=True, separate_builds=False, ninja=False, dry_run=False)[source]

CIME.build.clean(case, cleanlist=None, clean_all=False, clean_depends=None)[source]

CIME.build.generate_makefile_macro(case, caseroot)[source]: Generates a flat Makefile macro file based on the CMake cache system. This macro is only used by certain sharedlibs since components use CMake. Since indirection based on comp_name is allowed for sharedlibs, each sharedlib must generate their own macro.

CIME.build.get_standard_cmake_args(case, sharedpath)[source]

CIME.build.get_standard_makefile_args(case, shared_lib=False)[source]

CIME.build.post_build(case, logs, build_complete=False, save_build_provenance=True)[source]

CIME.build.uses_kokkos(case)[source]

CIME.build.xml_to_make_variable(case, varname, cmake=False)[source]

CIME.buildlib module

common utilities for buildlib

CIME.buildlib.build_cime_component_lib(case, compname, libroot, bldroot)[source]

CIME.buildlib.parse_input(argv)[source]

CIME.buildlib.run_gmake(case, compclass, compname, libroot, bldroot, libname='', user_cppdefs='')[source]

CIME.buildnml module

common implementation for building namelist commands

These are used by components/<model_type>/<component>/cime_config/buildnml

CIME.buildnml.build_xcpl_nml(case, caseroot, compname)[source]

CIME.buildnml.copy_inputs_to_rundir(caseroot, compname, confdir, rundir, inst_string)[source]

CIME.buildnml.create_namelist_infile(case, user_nl_file, namelist_infile, infile_text='')[source]

CIME.buildnml.parse_input(argv)[source]

CIME.code_checker module

Libraries for checking python code with pylint

CIME.code_checker.check_code(files, num_procs=10, interactive=False)[source]

Check all python files in the given directory

Returns True if all files had no problems

CIME.code_checker.get_all_checkable_files()[source]

CIME.compare_namelists module

CIME.compare_namelists.compare_namelist_files(gold_file, compare_file, case=None)[source]: Returns (is_match, comments)

CIME.compare_namelists.is_namelist_file(file_path)[source]

CIME.compare_test_results module

CIME.compare_test_results.append_status_cprnc_log(msg, logfile_name, test_dir)[source]

CIME.compare_test_results.compare_history(case, baseline_name, baseline_root, log_id)[source]

CIME.compare_test_results.compare_namelists(case, baseline_name, baseline_root, logfile_name)[source]

CIME.compare_test_results.compare_test_results(baseline_name, baseline_root, test_root, compiler, test_id=None, compare_tests=None, namelists_only=False, hist_only=False)[source]

Compares with baselines for all matching tests

Outputs results for each test to stdout (one line per test); possible status codes are: PASS, FAIL, SKIP. (A SKIP denotes a test that did not make it to the run phase or a test for which the run phase did not pass: we skip baseline comparisons in this case.)

In addition, creates files named compare.log.BASELINE_NAME.TIMESTAMP in each test directory, which contain more detailed output. Also creates *.cprnc.out.BASELINE_NAME.TIMESTAMP files in each run directory.

Returns True if all tests generated either PASS or SKIP results, False if there was at least one FAIL result.

CIME.config module

class CIME.config.Config[source]

Bases: ConfigBase

classmethod load_defaults()[source]

class CIME.config.ConfigBase[source]

Bases: object

classmethod instance()[source]

Access singleton.

Explicit way to access singleton, same as calling constructor.

classmethod load(customize_path)[source]

property loaded

print_method_rst()[source]

print_rst_table()[source]

print_variable_rst()[source]

CIME.config.print_rst_header(header, anchor=None, separator='"')[source]

CIME.config.print_rst_table(headers, *rows)[source]

CIME.cs_status module

Implementation of the cs.status script, which prints the status of all of the tests in one or more test suites

CIME.cs_status.cs_status(test_paths, summary=False, fails_only=False, count_fails_phase_list=None, check_throughput=False, check_memory=False, expected_fails_filepath=None, force_rebuild=False, out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Print the test statuses of all tests in test_paths. The default is to print to stdout, but this can be overridden with the ‘out’ argument.

If summary is True, then only the overall status of each test is printed

If fails_only is True, then only test failures are printed (this includes PENDs as well as FAILs).

If count_fails_phase_list is provided, it should be a list of phases (from the phases given by test_status.ALL_PHASES). For each phase in this list: do not give line-by-line output; instead, just report the total number of tests that have not PASSed this phase (this includes PENDs and FAILs). (This is typically used with the fails_only option, but it can also be used without that option.)

If expected_fails_filepath is provided, it should be a string giving the full path to a file listing expected failures for this test suite. Expected failures are then labeled as such in the output.

CIME.cs_status_creator module

Creates a test suite-specific cs.status file from a template

CIME.cs_status_creator.create_cs_status(test_root, test_id, extra_args='', filename=None)[source]

Create a test suite-specific cs.status file from the template

Arguments: test_root (string): path to test root; the file will be put here. If

this directory doesn’t exist, it is created.

test_id (string): test id for this test suite. This can contain: shell wildcards if you want this one cs.status file to work across multiple test suites. However, be careful not to make this too general: for example, ending this with ‘*’ will pick up the *.ref1 directories for ERI and other tests, which is NOT what you want.
extra_args (string): extra arguments to the cs.status command: (If there are multiple arguments, these should be in a space-delimited string.)
filename (string): name of the generated cs.status file. If not: given, this will be built from the test_id.

CIME.date module

class CIME.date.date(year=1, month=1, day=1, hour=0, minute=0, second=0)[source]

Bases: object

Simple struct for holding dates and the time of day and performing comparisons

Difference in Hour, Minute, or Second >>> date(4, 5, 6, 9) == date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) != date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) < date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) <= date(4, 5, 6, 8) False >>> date(4, 5, 6, 9) >= date(4, 5, 6, 8) True >>> date(4, 5, 6, 9) > date(4, 5, 6, 8) True

>>> date(4, 5, 6, 4) == date(4, 5, 6, 8)
False
>>> date(4, 5, 6, 4) != date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) < date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) <= date(4, 5, 6, 8)
True
>>> date(4, 5, 6, 4) >= date(4, 5, 6, 8)
False
>>> date(4, 5, 6, 4) > date(4, 5, 6, 8)
False

Difference in Day >>> date(4, 5, 8, 8) == date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) != date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) < date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) <= date(4, 5, 6, 8) False >>> date(4, 5, 8, 8) >= date(4, 5, 6, 8) True >>> date(4, 5, 8, 8) > date(4, 5, 6, 8) True

>>> date(4, 5, 5, 8) == date(4, 5, 6, 8)
False
>>> date(4, 5, 5, 8) != date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) < date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) <= date(4, 5, 6, 8)
True
>>> date(4, 5, 5, 8) >= date(4, 5, 6, 8)
False
>>> date(4, 5, 5, 8) > date(4, 5, 6, 8)
False

Difference in Month >>> date(4, 6, 6, 8) == date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) != date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) < date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) <= date(4, 5, 6, 8) False >>> date(4, 6, 6, 8) >= date(4, 5, 6, 8) True >>> date(4, 6, 6, 8) > date(4, 5, 6, 8) True

>>> date(4, 4, 6, 8) == date(4, 5, 6, 8)
False
>>> date(4, 4, 6, 8) != date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) < date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) <= date(4, 5, 6, 8)
True
>>> date(4, 4, 6, 8) >= date(4, 5, 6, 8)
False
>>> date(4, 4, 6, 8) > date(4, 5, 6, 8)
False

Difference in Year >>> date(5, 5, 6, 8) == date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) != date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) < date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) <= date(4, 5, 6, 8) False >>> date(5, 5, 6, 8) >= date(4, 5, 6, 8) True >>> date(5, 5, 6, 8) > date(4, 5, 6, 8) True

>>> date(3, 5, 6, 8) == date(4, 5, 6, 8)
False
>>> date(3, 5, 6, 8) != date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) < date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) <= date(4, 5, 6, 8)
True
>>> date(3, 5, 6, 8) >= date(4, 5, 6, 8)
False
>>> date(3, 5, 6, 8) > date(4, 5, 6, 8)
False

day()[source]

static hms_to_second(hour, minute, second)[source]

hour()[source]

minute()[source]

month()[source]

second()[source]

second_of_day()[source]

static second_to_hms(second)[source]

year()[source]

CIME.date.get_file_date(filename)[source]

Returns the date associated with the filename as a date object representing the correct date Formats supported: “%Y-%m-%d_%h.%M.%s “%Y-%m-%d_%05s” “%Y-%m-%d-%05s” “%Y-%m-%d” “%Y-%m” “%Y.%m”

>>> get_file_date("./ne4np4_oQU240.cam.r.0001-01-06-00435.nc")
date(1, 1, 6, 0, 7, 15)
>>> get_file_date("./ne4np4_oQU240.cam.r.0010-1-06_00435.nc")
date(10, 1, 6, 0, 7, 15)
>>> get_file_date("./ne4np4_oQU240.cam.r.0010-10.nc")
date(10, 10, 1, 0, 0, 0)
>>> get_file_date("0064-3-8_10.20.30.nc")
date(64, 3, 8, 10, 20, 30)
>>> get_file_date("0140-3-5")
date(140, 3, 5, 0, 0, 0)
>>> get_file_date("0140-3")
date(140, 3, 1, 0, 0, 0)
>>> get_file_date("0140.3")
date(140, 3, 1, 0, 0, 0)

CIME.expected_fails module

Contains the definition of a class to hold information on expected failures for a single test

class CIME.expected_fails.ExpectedFails[source]

Bases: object

add_failure(phase, expected_status)[source]: Add an expected failure to the list

expected_fails_comment(phase, status)[source]: Returns a string giving the expected fails comment for this phase and status

CIME.get_tests module

CIME.get_tests.get_build_groups(tests)[source]

Given a list of tests, return a list of lists, with each list representing a group of tests that can share executables.

>>> tests = ["SMS_P2.f19_g16_rx1.A.melvin_gnu", "SMS_P4.f19_g16_rx1.A.melvin_gnu", "SMS_P2.f19_g16_rx1.X.melvin_gnu", "SMS_P4.f19_g16_rx1.X.melvin_gnu", "TESTRUNSLOWPASS_P1.f19_g16_rx1.A.melvin_gnu", "TESTRUNSLOWPASS_P1.ne30_g16_rx1.A.melvin_gnu"]
>>> get_build_groups(tests)
[('SMS_P2.f19_g16_rx1.A.melvin_gnu', 'SMS_P4.f19_g16_rx1.A.melvin_gnu'), ('SMS_P2.f19_g16_rx1.X.melvin_gnu', 'SMS_P4.f19_g16_rx1.X.melvin_gnu'), ('TESTRUNSLOWPASS_P1.f19_g16_rx1.A.melvin_gnu',), ('TESTRUNSLOWPASS_P1.ne30_g16_rx1.A.melvin_gnu',)]

CIME.get_tests.get_full_test_names(testargs, machine, compiler)[source]

Return full test names in the form: TESTCASE.GRID.COMPSET.MACHINE_COMPILER.TESTMODS Testmods are optional

Testargs can be categories or test names and support the NOT symbol ‘^’

>>> get_full_test_names(["cime_tiny"], "melvin", "gnu")
['ERS.f19_g16_rx1.A.melvin_gnu', 'NCK.f19_g16_rx1.A.melvin_gnu']

>>> get_full_test_names(["cime_tiny", "PEA_P1_M.f45_g37_rx1.A"], "melvin", "gnu")
['ERS.f19_g16_rx1.A.melvin_gnu', 'NCK.f19_g16_rx1.A.melvin_gnu', 'PEA_P1_M.f45_g37_rx1.A.melvin_gnu']

>>> get_full_test_names(['ERS.f19_g16_rx1.A', 'NCK.f19_g16_rx1.A', 'PEA_P1_M.f45_g37_rx1.A'], "melvin", "gnu")
['ERS.f19_g16_rx1.A.melvin_gnu', 'NCK.f19_g16_rx1.A.melvin_gnu', 'PEA_P1_M.f45_g37_rx1.A.melvin_gnu']

>>> get_full_test_names(["cime_tiny", "^NCK.f19_g16_rx1.A"], "melvin", "gnu")
['ERS.f19_g16_rx1.A.melvin_gnu']

>>> get_full_test_names(["cime_test_multi_inherit"], "melvin", "gnu")
['TESTBUILDFAILEXC_P1.f19_g16_rx1.A.melvin_gnu', 'TESTBUILDFAIL_P1.f19_g16_rx1.A.melvin_gnu', 'TESTMEMLEAKFAIL_P1.f09_g16.X.melvin_gnu', 'TESTMEMLEAKPASS_P1.f09_g16.X.melvin_gnu', 'TESTRUNDIFF_P1.f19_g16_rx1.A.melvin_gnu', 'TESTRUNFAILEXC_P1.f19_g16_rx1.A.melvin_gnu', 'TESTRUNFAIL_P1.f19_g16_rx1.A.melvin_gnu', 'TESTRUNPASS_P1.f19_g16_rx1.A.melvin_gnu', 'TESTRUNPASS_P1.f45_g37_rx1.A.melvin_gnu', 'TESTRUNPASS_P1.ne30_g16_rx1.A.melvin_gnu', 'TESTRUNPASS_P2.ne30_g16_rx1.A.melvin_gnu', 'TESTRUNPASS_P4.f45_g37_rx1.A.melvin_gnu', 'TESTRUNSTARCFAIL_P1.f19_g16_rx1.A.melvin_gnu', 'TESTTESTDIFF_P1.f19_g16_rx1.A.melvin_gnu']

CIME.get_tests.get_recommended_test_time(test_full_name)[source]

>>> get_recommended_test_time("ERS.f19_g16_rx1.A.melvin_gnu")
'0:10:00'

>>> get_recommended_test_time("TESTRUNPASS_P69.f19_g16_rx1.A.melvin_gnu.testmod")
'0:13:00'

>>> get_recommended_test_time("PET_Ln20.ne30_ne30.FC5.sandiatoss3_intel.cam-outfrq9s")
>>>

CIME.get_tests.get_test_data(suite)[source]: For a given suite, returns (inherit, time, share, perf, tests)

CIME.get_tests.get_test_suite(suite, machine=None, compiler=None, skip_inherit=False, skip_tests=None)[source]: Return a list of FULL test names for a suite.

CIME.get_tests.get_test_suites()[source]

CIME.get_tests.infer_arch_from_tests(testargs)[source]

Return a tuple (machine, [compilers]) that can be inferred from the test args

>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A.melvin_gnu"])
('melvin', ['gnu'])
>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A"])
(None, [])
>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A", "NCK.f19_g16_rx1.A.melvin_gnu"])
('melvin', ['gnu'])
>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A.melvin_gnu", "NCK.f19_g16_rx1.A.melvin_gnu"])
('melvin', ['gnu'])
>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A.melvin_gnu9", "NCK.f19_g16_rx1.A.melvin_gnu"])
('melvin', ['gnu9', 'gnu'])
>>> infer_arch_from_tests(["NCK.f19_g16_rx1.A.melvin_gnu", "NCK.f19_g16_rx1.A.mappy_gnu"])
Traceback (most recent call last):
    ...
CIME.utils.CIMEError: ERROR: Must have consistent machine 'melvin' != 'mappy'

CIME.get_tests.is_perf_test(test)[source]

Is the provided test in a suite with perf=True?

>>> is_perf_test("SMS_P2.T42_T42.S.melvin_gnu")
True
>>> is_perf_test("SMS_P2.f19_g16_rx1.X.melvin_gnu")
False
>>> is_perf_test("PFS_P2.f19_g16_rx1.X.melvin_gnu")
True

CIME.get_tests.key_test_time(test_full_name)[source]

CIME.get_tests.suite_has_test(suite, test_full_name, skip_inherit=False)[source]

CIME.get_timing module

Library for implementing getTiming tool which gets timing information from a run.

CIME.get_timing.get_timing(case, lid)[source]

CIME.gitinterface module

class CIME.gitinterface.GitInterface(repo_path, logger, branch=None)[source]

Bases: object

config_get_value(section, name)[source]

config_set_value(section, name, value)[source]

git_operation(operation, *args, **kwargs)[source]

CIME.hist_utils module

Functions for actions pertaining to history files.

CIME.hist_utils.compare_baseline(case, baseline_dir=None, outfile_suffix='')[source]

compare the current test output to a baseline result

case - The case containing the hist files to be compared against baselines baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config outfile_suffix - if non-blank, then the cprnc output file name ends with

this suffix (with a ‘.’ added before the given suffix). if None, no output file saved.

returns (SUCCESS, comments) SUCCESS means all hist files matched their corresponding baseline

CIME.hist_utils.compare_test(case, suffix1, suffix2, ignore_fieldlist_diffs=False)[source]

Compares two sets of component history files in the testcase directory

case - The case containing the hist files to compare suffix1 - The suffix that identifies the first batch of hist files suffix1 - The suffix that identifies the second batch of hist files ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their

field lists (i.e., all shared fields are bit-for-bit, but one case has some diagnostic fields that are missing from the other case), treat the two cases as identical.

returns (SUCCESS, comments, num_compared)

CIME.hist_utils.copy_histfiles(case, suffix, match_suffix=None)[source]

Copy the most recent batch of hist files in a case, adding the given suffix.

This can allow you to temporarily “save” these files so they won’t be blown away if you re-run the case.

case - The case containing the files you want to save suffix - The string suffix you want to add to saved files, this can be used to find them later.

returns (comments, num_copied)

CIME.hist_utils.cprnc(model, file1, file2, case, rundir, multiinst_driver_compare=False, outfile_suffix='', ignore_fieldlist_diffs=False, cprnc_exe=None)[source]

Run cprnc to compare two individual nc files

file1 - the full or relative path of the first file file2 - the full or relative path of the second file case - the case containing the files rundir - the rundir for the case outfile_suffix - if non-blank, then the output file name ends with this

suffix (with a ‘.’ added before the given suffix). Use None to avoid permissions issues in the case dir.

ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their: field lists (i.e., all shared fields are bit-for-bit, but one case has some diagnostic fields that are missing from the other case), treat the two cases as identical.
returns (True if the files matched, log_name, comment): where ‘comment’ is either an empty string or one of the module-level constants beginning with CPRNC_ (e.g., CPRNC_FIELDLISTS_DIFFER)

CIME.hist_utils.generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False)[source]

CIME.hist_utils.generate_teststatus(testdir, baseline_dir)[source]: CESM stores it’s TestStatus file in baselines. Do not let exceptions escape from this function.

CIME.hist_utils.get_ts_synopsis(comments)[source]

Reduce case diff comments down to a single line synopsis so that we can put something in the TestStatus file. It’s expected that the comments provided to this function came from compare_baseline, not compare_tests.

>>> get_ts_synopsis('')
''
>>> get_ts_synopsis('\n')
''
>>> get_ts_synopsis('big error')
'big error'
>>> get_ts_synopsis('big error\n')
'big error'
>>> get_ts_synopsis('stuff\n    File foo had a different field list from bar with suffix baz\nPass\n')
'FIELDLIST field lists differ (otherwise bit-for-bit)'
>>> get_ts_synopsis('stuff\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'ERROR BFAIL some baseline files were missing'
>>> get_ts_synopsis('stuff\n    File foo had a different field list from bar with suffix baz\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'MULTIPLE ISSUES: field lists differ and some baseline files were missing'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\n    File foo had a different field list from bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
'DIFF'
>>> get_ts_synopsis('File foo had no compare counterpart in bar with suffix baz\n File foo had no original counterpart in bar with suffix baz\n')
'DIFF'
>>> get_ts_synopsis('file1=\nfile2=\nFailed to open file\n')
'ERROR CPRNC failed to open files'
>>> get_ts_synopsis('file1=\nfile2=\nSome other error\n')
'ERROR Could not interpret CPRNC output'
>>> get_ts_synopsis('file1=\nfile2=\n  diff_test: the two files seem to be IDENTICAL \n')
''

CIME.hist_utils.rename_all_hist_files(case, suffix)[source]

Renaming all hist files in a case, adding the given suffix.

case - The case containing the files you want to save suffix - The string suffix you want to add to saved files, this can be used to find them later.

CIME.jenkins_generic_job module

CIME.jenkins_generic_job.archive_old_test_data(machine, mach_comp, test_id_root, test_root, old_test_archive, avoid_test_id)[source]

CIME.jenkins_generic_job.cleanup_queue(test_root, test_id)[source]: Delete all jobs left in the queue

CIME.jenkins_generic_job.delete_old_test_data(mach_comp, test_id_root, scratch_root, test_root, run_area, build_area, archive_area, avoid_test_id)[source]

CIME.jenkins_generic_job.handle_old_test_data(machine, compiler, test_id_root, scratch_root, test_root, avoid_test_id)[source]

CIME.jenkins_generic_job.jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch, baseline_name, arg_cdash_build_name, cdash_project, arg_test_suite, cdash_build_group, baseline_compare, scratch_root, parallel_jobs, walltime, machine, compiler, real_baseline_name, baseline_root, update_success, check_throughput, check_memory, ignore_memleak, ignore_namelists, ignore_diffs, save_timing, pes_file, jenkins_id, queue)[source]: Return True if all tests passed

CIME.jenkins_generic_job.scan_for_test_ids(old_test_archive, mach_comp, test_id_root)[source]

CIME.locked_files module

CIME.locked_files.check_diff(case, filename, env_name, diff, quiet=False)[source]

CIME.locked_files.check_lockedfile(case, filebase, caseroot=None, quiet=False)[source]

CIME.locked_files.check_lockedfiles(case, skip=None, quiet=False, caseroot=None, whitelist=None)[source]

Check that all lockedfiles match what’s in case

If caseroot is not specified, it is set to the current working directory

CIME.locked_files.diff_lockedfile(case, caseroot, filename)[source]

CIME.locked_files.is_locked(filename, caseroot)[source]

CIME.locked_files.lock_file(filename, caseroot, newname=None)[source]

CIME.locked_files.unlock_file(filename, caseroot)[source]

CIME.namelist module

Module containing tools for dealing with Fortran namelists.

The public interface consists of the following functions: - character_literal_to_string - compress_literal_list - expand_literal_list - fortran_namelist_base_value - is_valid_fortran_name - is_valid_fortran_namelist_literal - literal_to_python_value - merge_literal_lists - parse - string_to_character_literal

In addition, the Namelist class represents a namelist held in memory.

For the moment, only a subset of namelist syntax is supported; specifically, we assume that only variables of intrinsic type are used, and indexing/co-indexing of arrays to set a portion of a variable is not supported. (However, null values and repeated values may be used to set or fill a variable as indexing would.)

We also always assume that a period (“.”) is the decimal separator, not a comma (“,”). We also assume that the file encoding is UTF-8 or some compatible format (e.g. ASCII).

Otherwise, most Fortran syntax rules implemented here are compatible with Fortran 2008 (which is largely the same as previous standards, and will be similar to Fortran 2015). The only exceptions should be cases where (a) we deliberately prohibit “troublesome” behavior that would be allowed by the standard, or (b) we rely on conventions shared by all major compilers.

One important convention is that newline characters can be used to denote the end of a record. This makes them equivalent to spaces at most locations in a Fortran namelist, except that newlines also end comments, and they are ignored entirely within strings.

While the treatment of comments in this module is standard, it may be somewhat surprising. Namelist comments are only allowed in two situations:

As the only thing on a line (aside from optional indentation with spaces).

(2) Immediately after a “value separator” (the space, newline, comma, or slash after a value).

This implies that all lines except for the last are syntax errors, in this example:

` &group_name! This is not a valid comment because it's after the group name. foo ! Neither is this, because it's between a name and an equals sign. = 2 ! Nor this, because it comes between the value and the following comma. , bar = ! Nor this, because it's between an equals sign and a value. 2! Nor this, because it should be separated from the value by a comma or space. bazz = 3 ! Nor this, because it comes between the value and the following slash. /! This is fine, but technically it is outside the namelist, not a comment. `

However, the above would actually be valid if all the “comments” were removed. The Fortran standard is not clear about whether whitespace is allowed after inline comments and before subsequent non-whitespace text (!), but this module allows such whitespace, to preserve the sanity of both implementors and users.

The Fortran standard only applies to the interior of namelist groups, and not to text between one namelist group and the next. This module assumes that namelist groups are separated by (optional) whitespace and comments, and nothing else.

class CIME.namelist.Namelist(groups=None)[source]

Bases: object

Class representing a Fortran namelist.

Public methods: __init__ delete_variable get_group_names get_value get_variable_names get_variable_value merge_nl set_variable_value write

clean_groups()[source]

delete_variable(group_name, variable_name)[source]

Delete a variable from a specified group.

If the specified group or variable does not exist, this is a no-op.

>>> x = parse(text='&foo bar=1 /')
>>> x.delete_variable('FOO', 'BAR')
>>> x.delete_variable('foo', 'bazz')
>>> x.delete_variable('brack', 'bazz')
>>> x.get_variable_names('foo')
[]
>>> x.get_variable_names('brack')
[]

get_group_names()[source]

Return a list of all groups in the namelist.

>>> Namelist().get_group_names()
[]
>>> sorted(parse(text='&foo / &bar /').get_group_names())
['bar', 'foo']

get_group_variables(group_name)[source]

get_value(variable_name)[source]

Return the value of a uniquely-named variable.

This function is similar to get_variable_value, except that it does not require a group_name, and it requires that the variable_name be unique across all groups.

>>> parse(text='&foo bar=1 / &bazz bar=1 /').get_value('bar')  
Traceback (most recent call last):
...
CIMEError: ERROR: Namelist.get_value: Variable {} is present in multiple groups: ...
>>> parse(text='&foo bar=1 / &bazz /').get_value('Bar')
['1']
>>> parse(text='&foo bar(2)=1 / &bazz /').get_value('Bar(2)')
['1']
>>> parse(text='&foo / &bazz /').get_value('bar')
['']

get_variable_names(group_name)[source]

Return a list of all variables in the given namelist group.

If the specified group is not in the namelist, returns an empty list.

>>> Namelist().get_variable_names('foo')
[]
>>> x = parse(text='&foo bar=,bazz=true,bazz(2)=fred,bang=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bang', 'bar', 'bazz', 'bazz(2)']
>>> x = parse(text='&foo bar=,bazz=true,bang=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bang', 'bar', 'bazz']
>>> x = parse(text='&foo bar(::)=,bazz=false,bazz(2)=true,bazz(:2:)=6*""/')
>>> sorted(x.get_variable_names('fOo'))
['bar(::)', 'bazz', 'bazz(2)', 'bazz(:2:)']

get_variable_value(group_name, variable_name)[source]

Return the value of the specified variable.

This function always returns a non-empty list containing strings. If the specified group_name or variable_name is not present, [‘’] is returned.

>>> Namelist().get_variable_value('foo', 'bar')
['']
>>> parse(text='&foo bar=1,2 /').get_variable_value('foo', 'bazz')
['']
>>> parse(text='&foo bar=1,2 /').get_variable_value('foO', 'Bar')
['1', '2']

merge_nl(other, overwrite=False)[source]

Merge this namelist object with another.

Values in the invoking (self) Namelist will take precedence over values in the other Namelist, unless overwrite=True is passed in, in which case other values take precedence.

>>> x = parse(text='&foo bar=1 bazz=,2 brat=3/')
>>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /')
>>> y.get_value('bazz')
['1', '1', '1']
>>> x.merge_nl(y)
>>> sorted(x.get_group_names())
['foo', 'foo2']
>>> sorted(x.get_variable_names('foo'))
['baker', 'bar', 'bazz', 'brat']
>>> sorted(x.get_variable_names('foo2'))
['barter']
>>> x.get_value('bar')
['1']
>>> x.get_value('bazz')
['1', '2', '1']
>>> x.get_value('brat')
['3']
>>> x.get_value('baker')
['4']
>>> x.get_value('barter')
['5']
>>> x = parse(text='&foo bar=1 bazz=,2 brat=3/')
>>> y = parse(text='&foo bar=2 bazz=3*1 baker=4 / &foo2 barter=5 /')
>>> x.merge_nl(y, overwrite=True)
>>> sorted(x.get_group_names())
['foo', 'foo2']
>>> sorted(x.get_variable_names('foo'))
['baker', 'bar', 'bazz', 'brat']
>>> sorted(x.get_variable_names('foo2'))
['barter']
>>> x.get_value('bar')
['2']
>>> x.get_value('bazz')
['1', '1', '1']
>>> x.get_value('brat')
['3']
>>> x.get_value('baker')
['4']
>>> x.get_value('barter')
['5']

set_variable_value(group_name, variable_name, value, var_size=1)[source]

Set the value of the specified variable.

>>> x = parse(text='&foo bar=1 /')
>>> x.get_variable_value('foo', 'bar')
['1']
>>> x.set_variable_value('foo', 'bar(2)', ['3'], var_size=4)
>>> x.get_variable_value('foo', 'bar')
['1', '3']
>>> x.set_variable_value('foo', 'bar(1)', ['2'])
>>> x.get_variable_value('foo', 'bar')
['2', '3']
>>> x.set_variable_value('foo', 'bar', ['1'])
>>> x.get_variable_value('foo', 'bar')
['1', '3']
>>> x.set_variable_value('foo', 'bazz', ['3'])
>>> x.set_variable_value('Brack', 'baR', ['4'])
>>> x.get_variable_value('foo', 'bazz')
['3']
>>> x.get_variable_value('brack', 'bar')
['4']
>>> x.set_variable_value('foo', 'red(2:6:2)', ['2', '4', '6'], var_size=12)
>>> x.get_variable_value('foo', 'red')
['', '2', '', '4', '', '6']

write(out_file, groups=None, append=False, format_='nml', sorted_groups=True)[source]

Write a the output data (normally fortran namelist) to the out_file

As with parse, the out_file argument can be either a file name, or a file object with a write method that accepts unicode. If specified, the groups argument specifies a subset of all groups to write out.

If out_file is a file name, and append=True is passed in, the namelist will be appended to the named file instead of overwriting it. The append option has no effect if a file object is passed in.

The format_ option can be either ‘nml’ (namelist) or ‘rc’, and specifies the file format. Formats other than ‘nml’ may not support all possible output values.

write_nuopc(out_file, groups=None, sorted_groups=True)[source]

Write a nuopc config file out_file

As with parse, the out_file argument can be either a file name, or a file object with a write method that accepts unicode. If specified, the groups argument specifies a subset of all groups to write out.

CIME.namelist.character_literal_to_string(literal)[source]

Convert a Fortran character literal to a Python string.

This function assumes (without checking) that literal is a valid literal.

>>> character_literal_to_string("'blah'")
'blah'
>>> character_literal_to_string('"blah"')
'blah'
>>> character_literal_to_string("'don''t'")
"don't"
>>> character_literal_to_string('"' + '""Hello!""' + '"')
'"Hello!"'

CIME.namelist.compress_literal_list(literals)[source]

Uses repetition syntax to shorten a literal list.

>>> compress_literal_list([])
[]
>>> compress_literal_list(['true'])
['true']
>>> compress_literal_list(['1', '2', 'f*', '3', '3', '3', '5'])
['1', '2', 'f*', '3', '3', '3', '5']
>>> compress_literal_list(['f*', 'f*'])
['f*', 'f*']

CIME.namelist.convert_bool(value)[source]

CIME.namelist.expand_literal_list(literals)[source]

Expands a list of literal values to get rid of repetition syntax.

>>> expand_literal_list([])
[]
>>> expand_literal_list(['true'])
['true']
>>> expand_literal_list(['1', '2', 'f*', '3*3', '5'])
['1', '2', 'f*', '3', '3', '3', '5']
>>> expand_literal_list(['2*f*'])
['f*', 'f*']

CIME.namelist.fortran_namelist_base_value(string)[source]

Strip off whitespace and repetition syntax from a namelist value.

>>> fortran_namelist_base_value("")
''
>>> fortran_namelist_base_value("f")
'f'
>>> fortran_namelist_base_value("6*")
''
>>> fortran_namelist_base_value("6*f")
'f'
>>> fortran_namelist_base_value(" \n6* \n")
''
>>> fortran_namelist_base_value("\n 6*f\n ")
'f'

CIME.namelist.get_fortran_name_only(full_var)[source]: remove array section if any and return only the variable name >>> get_fortran_name_only(‘foo’) ‘foo’ >>> get_fortran_name_only(‘foo(3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’ >>> get_fortran_name_only(‘foo(1::)’) ‘foo’ >>> get_fortran_name_only(‘foo(:+2:)’) ‘foo’ >>> get_fortran_name_only(‘foo(::-3)’) ‘foo’ >>> get_fortran_name_only(‘foo(::)’) ‘foo’

CIME.namelist.get_fortran_variable_indices(varname, varlen=1, allow_any_len=False)[source]

get indices from a fortran namelist variable as a triplet of minindex, maxindex and step

>>> get_fortran_variable_indices('foo(3)')
(3, 3, 1)
>>> get_fortran_variable_indices('foo(1:2:3)')
(1, 2, 3)
>>> get_fortran_variable_indices('foo(::)', varlen=4)
(1, 4, 1)
>>> get_fortran_variable_indices('foo(::2)', varlen=4)
(1, 4, 2)
>>> get_fortran_variable_indices('foo(::)', allow_any_len=True)
(1, -1, 1)

CIME.namelist.is_valid_fortran_name(string)[source]

Check that a variable name is allowed in Fortran.

The rules are: 1. The name must start with a letter. 2. All characters in a name must be alphanumeric (or underscores). 3. The maximum name length is 63 characters. 4. We only handle a single dimension !!!

>>> is_valid_fortran_name("")
False
>>> is_valid_fortran_name("a")
True
>>> is_valid_fortran_name("A")
True
>>> is_valid_fortran_name("A(4)")
True
>>> is_valid_fortran_name("A(::)")
True
>>> is_valid_fortran_name("A(1:2:3)")
True
>>> is_valid_fortran_name("A(1::)")
True
>>> is_valid_fortran_name("A(:-2:)")
True
>>> is_valid_fortran_name("A(1::+3)")
True
>>> is_valid_fortran_name("A(1,3)")
False
>>> is_valid_fortran_name("2")
False
>>> is_valid_fortran_name("_")
False
>>> is_valid_fortran_name("abc#123")
False
>>> is_valid_fortran_name("aLiBi_123")
True
>>> is_valid_fortran_name("A" * 64)
False
>>> is_valid_fortran_name("A" * 63)
True

CIME.namelist.is_valid_fortran_namelist_literal(type_, string)[source]

Determine whether a literal is valid in a Fortran namelist.

Note that kind parameters are not allowed in namelists, which simplifies this check a bit. Internal whitespace is allowed for complex and character literals only. BOZ literals and compiler extensions (e.g. backslash escapes) are not allowed.

Null values, however, are allowed for all types. This means that passing in a string containing nothing but spaces and newlines will always cause True to be returned. Repetition (e.g. 5*’a’) is also allowed, including repetition of null values.

Detailed rules and examples follow.

Integers: Must be a sequence of one or more digits, with an optional sign.

>>> is_valid_fortran_namelist_literal("integer", "")
True
>>> is_valid_fortran_namelist_literal("integer", " ")
True
>>> is_valid_fortran_namelist_literal("integer", "\n")
True
>>> is_valid_fortran_namelist_literal("integer", "5*")
True
>>> is_valid_fortran_namelist_literal("integer", "1")
True
>>> is_valid_fortran_namelist_literal("integer", "5*1")
True
>>> is_valid_fortran_namelist_literal("integer", " 5*1")
True
>>> is_valid_fortran_namelist_literal("integer", "5* 1")
False
>>> is_valid_fortran_namelist_literal("integer", "5 *1")
False
>>> is_valid_fortran_namelist_literal("integer", "a")
False
>>> is_valid_fortran_namelist_literal("integer", " 1")
True
>>> is_valid_fortran_namelist_literal("integer", "1 ")
True
>>> is_valid_fortran_namelist_literal("integer", "1 2")
False
>>> is_valid_fortran_namelist_literal("integer", "0123456789")
True
>>> is_valid_fortran_namelist_literal("integer", "+22")
True
>>> is_valid_fortran_namelist_literal("integer", "-26")
True
>>> is_valid_fortran_namelist_literal("integer", "2A")
False
>>> is_valid_fortran_namelist_literal("integer", "1_8")
False
>>> is_valid_fortran_namelist_literal("integer", "2.1")
False
>>> is_valid_fortran_namelist_literal("integer", "2e6")
False

Reals: - For fixed-point format, there is an optional sign, followed by an integer part, or a decimal point followed by a fractional part, or both. - Scientific notation is allowed, with an optional, case-insensitive “e” or “d” followed by an optionally-signed integer exponent. (Either the “e”/”d” or a sign must be present to separate the number from the exponent.) - The (case-insensitive) strings “inf”, “infinity”, and “nan” are allowed. NaN values can also contain additional information in parentheses, e.g. “NaN(x1234ABCD)”.

>>> is_valid_fortran_namelist_literal("real", "")
True
>>> is_valid_fortran_namelist_literal("real", "a")
False
>>> is_valid_fortran_namelist_literal("real", "1")
True
>>> is_valid_fortran_namelist_literal("real", " 1")
True
>>> is_valid_fortran_namelist_literal("real", "1 ")
True
>>> is_valid_fortran_namelist_literal("real", "1 2")
False
>>> is_valid_fortran_namelist_literal("real", "+1")
True
>>> is_valid_fortran_namelist_literal("real", "-1")
True
>>> is_valid_fortran_namelist_literal("real", "1.")
True
>>> is_valid_fortran_namelist_literal("real", "1.5")
True
>>> is_valid_fortran_namelist_literal("real", ".5")
True
>>> is_valid_fortran_namelist_literal("real", "+.5")
True
>>> is_valid_fortran_namelist_literal("real", ".")
False
>>> is_valid_fortran_namelist_literal("real", "+")
False
>>> is_valid_fortran_namelist_literal("real", "1e6")
True
>>> is_valid_fortran_namelist_literal("real", "1e-6")
True
>>> is_valid_fortran_namelist_literal("real", "1e+6")
True
>>> is_valid_fortran_namelist_literal("real", ".5e6")
True
>>> is_valid_fortran_namelist_literal("real", "1e")
False
>>> is_valid_fortran_namelist_literal("real", "1D6")
True
>>> is_valid_fortran_namelist_literal("real", "1q6")
False
>>> is_valid_fortran_namelist_literal("real", "1+6")
True
>>> is_valid_fortran_namelist_literal("real", "1.6.5")
False
>>> is_valid_fortran_namelist_literal("real", "1._8")
False
>>> is_valid_fortran_namelist_literal("real", "1,5")
False
>>> is_valid_fortran_namelist_literal("real", "inf")
True
>>> is_valid_fortran_namelist_literal("real", "INFINITY")
True
>>> is_valid_fortran_namelist_literal("real", "NaN")
True
>>> is_valid_fortran_namelist_literal("real", "nan(x56)")
True
>>> is_valid_fortran_namelist_literal("real", "nan())")
False

Complex numbers: - A pair of real numbers enclosed by parentheses, and separated by a comma. - Any number of spaces or newlines may be placed before or after each real.

>>> is_valid_fortran_namelist_literal("complex", "")
True
>>> is_valid_fortran_namelist_literal("complex", "()")
False
>>> is_valid_fortran_namelist_literal("complex", "(,)")
False
>>> is_valid_fortran_namelist_literal("complex", "( ,\n)")
False
>>> is_valid_fortran_namelist_literal("complex", "(a,2.)")
False
>>> is_valid_fortran_namelist_literal("complex", "(1.,b)")
False
>>> is_valid_fortran_namelist_literal("complex", "(1,2)")
True
>>> is_valid_fortran_namelist_literal("complex", "(-1.e+06,+2.d-5)")
True
>>> is_valid_fortran_namelist_literal("complex", "(inf,NaN)")
True
>>> is_valid_fortran_namelist_literal("complex", "(  1. ,  2. )")
True
>>> is_valid_fortran_namelist_literal("complex", "( \n \n 1. \n,\n 2.\n)")
True
>>> is_valid_fortran_namelist_literal("complex", " (1.,2.)")
True
>>> is_valid_fortran_namelist_literal("complex", "(1.,2.) ")
True

Character sequences (strings): - Must begin and end with the same delimiter character, either a single quote (apostrophe), or a double quote (quotation mark). - Whichever character is used as a delimiter must not appear in the string itself, unless it appears in doubled pairs (e.g. ‘’’’ or “’” are the two ways of representing a string containing a single apostrophe). - Note that newlines cannot be represented in a namelist character literal since they are interpreted as an “end of record”, but they are allowed as long as they don’t come between one of the aforementioned double pairs of characters.

>>> is_valid_fortran_namelist_literal("character", "")
True
>>> is_valid_fortran_namelist_literal("character", "''")
True
>>> is_valid_fortran_namelist_literal("character", " ''")
True
>>> is_valid_fortran_namelist_literal("character", "'\n'")
True
>>> is_valid_fortran_namelist_literal("character", "''\n''")
False
>>> is_valid_fortran_namelist_literal("character", "'''")
False
>>> is_valid_fortran_namelist_literal("character", "''''")
True
>>> is_valid_fortran_namelist_literal("character", "'''Cookie'''")
True
>>> is_valid_fortran_namelist_literal("character", "'''Cookie''")
False
>>> is_valid_fortran_namelist_literal("character", "'\"'")
True
>>> is_valid_fortran_namelist_literal("character", "'\"\"'")
True
>>> is_valid_fortran_namelist_literal("character", '""')
True
>>> is_valid_fortran_namelist_literal("character", '"" ')
True
>>> is_valid_fortran_namelist_literal("character", '"\n"')
True
>>> is_valid_fortran_namelist_literal("character", '""\n""')
False
>>> is_valid_fortran_namelist_literal("character", '""' + '"')
False
>>> is_valid_fortran_namelist_literal("character", '""' + '""')
True
>>> is_valid_fortran_namelist_literal("character", '"' + '""Cookie""' + '"')
True
>>> is_valid_fortran_namelist_literal("character", '""Cookie""' + '"')
False
>>> is_valid_fortran_namelist_literal("character", '"\'"')
True
>>> is_valid_fortran_namelist_literal("character", '"\'\'"')
True

Logicals: - Must contain a (case-insensitive) “t” or “f”. - This must be either the first nonblank character, or the second following a period. - The rest of the string is ignored, but cannot contain a comma, newline, equals sign, slash, or space (except that trailing spaces are allowed and ignored).

>>> is_valid_fortran_namelist_literal("logical", "")
True
>>> is_valid_fortran_namelist_literal("logical", "t")
True
>>> is_valid_fortran_namelist_literal("logical", "F")
True
>>> is_valid_fortran_namelist_literal("logical", ".T")
True
>>> is_valid_fortran_namelist_literal("logical", ".f")
True
>>> is_valid_fortran_namelist_literal("logical", " f")
True
>>> is_valid_fortran_namelist_literal("logical", " .t")
True
>>> is_valid_fortran_namelist_literal("logical", "at")
False
>>> is_valid_fortran_namelist_literal("logical", ".TRUE.")
True
>>> is_valid_fortran_namelist_literal("logical", ".false.")
True
>>> is_valid_fortran_namelist_literal("logical", ".TEXAS$")
True
>>> is_valid_fortran_namelist_literal("logical", ".f=")
False
>>> is_valid_fortran_namelist_literal("logical", ".f/1")
False
>>> is_valid_fortran_namelist_literal("logical", ".t\nted")
False
>>> is_valid_fortran_namelist_literal("logical", ".Fant astic")
False
>>> is_valid_fortran_namelist_literal("logical", ".t2 ")
True

CIME.namelist.literal_to_python_value(literal, type_=None)[source]

Convert a Fortran literal string to a Python value.

This function assumes that the input contains a single value, i.e. repetition syntax is not used. The type can be specified by passing a string as the type_ argument, or if this option is not provided, this function will attempt to autodetect the variable type.

Note that it is not possible to be certain whether a literal like “123” is intended to represent an integer or a floating-point value, however, nor can we be certain of the precision that will be used to hold this value in actual Fortran code. We also cannot use the optional information in a NaN float, so this will cause the function to throw an error if that information is present (e.g. a string like “NAN(1234)” will cause an error).

The Python type of the return value is as follows for different type_ arguments: “character” - str “complex” - complex “integer” - int “logical” - bool “real” - float

If a null value is input (i.e. the empty string), None will be returned.

>>> literal_to_python_value("'She''s a winner!'")
"She's a winner!"
>>> literal_to_python_value("1")
1
>>> literal_to_python_value("1.")
1.0
>>> literal_to_python_value(" (\n 1. , 2. )\n ")
(1+2j)
>>> literal_to_python_value(".true.")
True
>>> literal_to_python_value("Fortune")
False
>>> literal_to_python_value("bacon") 
Traceback (most recent call last):
...
CIMEError: ERROR: 'bacon' is not a valid literal for any Fortran type.
>>> literal_to_python_value("1", type_="real")
1.0
>>> literal_to_python_value("bacon", type_="logical") 
Traceback (most recent call last):
...
CIMEError: ERROR: 'bacon' is not a valid literal of type 'logical'.
>>> literal_to_python_value("1", type_="booga") 
Traceback (most recent call last):
...
CIMEError: ERROR: Invalid Fortran type for a namelist: 'booga'
>>> literal_to_python_value("2*1") 
Traceback (most recent call last):
...
CIMEError: ERROR: Cannot use repetition syntax in literal_to_python_value
>>> literal_to_python_value("")
>>> literal_to_python_value("-1.D+10")
-10000000000.0
>>> shouldRaise(ValueError, literal_to_python_value, "nan(1234)")

CIME.namelist.merge_literal_lists(default, overwrite)[source]

Merge two lists of literal value strings.

The overwrite values have higher precedence, so will overwrite the default values. However, if overwrite contains null values, or is shorter than default (and thus implicitly ends in null values), the elements of default will be used where overwrite is null.

>>> merge_literal_lists([], [])
[]
>>> merge_literal_lists(['true'], ['false'])
['false']
>>> merge_literal_lists([], ['false'])
['false']
>>> merge_literal_lists(['true'], [''])
['true']
>>> merge_literal_lists([], [''])
['']
>>> merge_literal_lists(['true'], [])
['true']
>>> merge_literal_lists(['true'], [])
['true']
>>> merge_literal_lists(['3*false', '3*true'], ['true', '4*', 'false'])
['true', 'false', 'false', 'true', 'true', 'false']

CIME.namelist.parse(in_file=None, text=None, groupless=False, convert_tab_to_space=True)[source]

Parse a Fortran namelist.

The in_file argument must be either a str or unicode object containing a file name, or a text I/O object with a read method that returns the text of the namelist.

Alternatively, the text argument can be provided, in which case it must be the text of the namelist itself.

The groupless argument changes namelist parsing in two ways:

parse allows an alternate file format where no group names or slashes are present. In effect, the file is parsed as if an invisible, arbitrary group name was prepended, and an invisible slash was appended. However, if any group names actually are present, the file is parsed normally.

The return value of this function is not a Namelist object. Instead a single, flattened dictionary of name-value pairs is returned.

The convert_tab_to_space option can be used to force all tabs in the file to be converted to spaces, and is on by default. Note that this will usually allow files that use tabs as whitespace to be parsed. However, the implementation of this option is crude; it converts all tabs in the file, including those in character literals. (Note that there are many characters that cannot be passed in via namelist in any standard way, including ‘

‘,

so it is already a bad idea to assume that the namelist will preserve whitespace in strings, aside from simple spaces.)

The return value, if groupless=False, is a Namelist object.

All names and values returned are ultimately unicode strings. E.g. a value of “6*2” is returned as that string; it is not converted to 6 copies of the Python integer 2. Null values are returned as the empty string (“”).

CIME.namelist.shouldRaise(eclass, method, *args, **kw)[source]: A helper function to make doctests py3 compatible http://python3porting.com/problems.html#running-doctests

CIME.namelist.string_to_character_literal(string)[source]

Convert a Python string to a Fortran character literal.

This function always uses double quotes (”) as the delimiter.

>>> string_to_character_literal('blah')
'"blah"'
>>> string_to_character_literal("'blah'")
'"\'blah\'"'
>>> string_to_character_literal('She said "Hi!".')
'"She said ""Hi!""."'

CIME.nmlgen module

Class for generating component namelists.

class CIME.nmlgen.NamelistGenerator(case, definition_files, files=None)[source]

Bases: object

Utility class for generating namelists for a given component.

add_default(name, value=None, ignore_abs_path=None)[source]

Add a value for the specified variable to the namelist.

If the specified variable is already defined in the object, the existing value is preserved. Otherwise, the value argument, if provided, will be used to set the value. If no such value is found, the defaults file will be consulted. If null values are present in any of the above, the result will be a merged array of values.

If no value for the variable is found via any of the above, this method will raise an exception.

add_defaults_for_group(group)[source]

Call add_default for namelist variables in the given group

This still skips variables that have attributes of skip_default_entry or per_stream_entry.

This must be called after init_defaults. It is often paired with use of skip_default_for_groups in the init_defaults call.

add_nmlcontents(filename, group, append=True, format_='nmlcontents', sorted_groups=True)[source]: Write only contents of nml group

clean_streams()[source]

confirm_group_is_empty(group_name, errmsg)[source]

Confirms that no values have been added to the given group

If any values HAVE been added to this group, aborts with the given error message.

This is often paired with use of skip_default_for_groups in the init_defaults call and add_defaults_for_group, as in:

if nmlgen.get_value(“enable_frac_overrides”) == “.true.”:
nmlgen.add_defaults_for_group(“glc_override_nml”)

else:
nmlgen.confirm_empty(“glc_override_nml”, “some message”)

Args: group_name: string - name of namelist group errmsg: string - error message to print if group is not empty

create_shr_strdata_nml()[source]: Set defaults for shr_strdata_nml variables other than the variable domainfile

create_stream_file_and_update_shr_strdata_nml(config, caseroot, stream, stream_path, data_list_path)[source]

Write the pseudo-XML file corresponding to a given stream.

Arguments: config - Used to look up namelist defaults. This is used in addition

to the config used to construct the namelist generator. The main reason to supply additional configuration options here is to specify stream-specific settings.

stream - Name of the stream. stream_path - Path to write the stream file to. data_list_path - Path of file to append input data information to.

get_default(name, config=None, allow_none=False)[source]

Get the value of a variable from the namelist definition file.

The config argument is passed through to the underlying NamelistDefaults.get_value call as the attribute argument.

The return value of this function is a list of values that were found in the defaults file. If there is no matching default, this function returns None if allow_none=True is passed, otherwise an error is raised.

Note that we perform some translation of the values, since there are a few differences between Fortran namelist literals and values in the defaults file: 1) In the defaults file, whitespace is ignored except within strings, so

the output of this function strips out most whitespace. (This implies that commas are the only way to separate array elements in the defaults file.)

In the defaults file, quotes around character literals (strings) are optional, as long as the literal does not contain whitespace, commas, or (single or double) quotes. If a setting for a character variable does not seem to have quotes (and is not a null value), this function will add them.
Default values may refer to variables in a case’s env_*.xml files. This function replaces references of the form $VAR or ${VAR} with the value of the variable VAR in an env file, if that variable exists. This behavior is suppressed within single-quoted strings (similar to parameter expansion in shell scripts).

get_group_variables(group_name)[source]

get_streams()[source]: Get a list of all streams used for the current data model mode.

get_value(name)[source]

Get the current value of a given namelist variable.

Note that the return value of this function is always a string or a list of strings. E.g. the scalar logical value .false. will be returned as “.false.”, while an array of two .false. values will be returned as [“.false.”, “.false.”]. Whether or not a value is scalar is determined by checking the array size in the namelist definition file.

Null values are converted to None, and repeated values are expanded, e.g. [‘2*3’] is converted to [‘3’, ‘3’, ‘3’].

For character variables, the value is converted to a Python string (e.g. quotation marks are removed).

All other literals are returned as the raw string values that will be written to the namelist.

init_defaults(infiles, config, skip_groups=None, skip_entry_loop=False, skip_default_for_groups=None, set_group_name=None)[source]

Return array of names of all definition nodes

infiles should be a list of file paths, each one giving namelist settings that take precedence over the default values. Often there will be only one file in this list. If there are multiple files, earlier files take precedence over later files.

If skip_default_for_groups is provided, it should be a list of namelist group names; the add_default call will not be done for any variables in these groups. This is often paired with later conditional calls to add_defaults_for_group.

new_instance()[source]: Clean the object just enough to introduce a new instance

static quote_string(string)[source]

Convert a string to a quoted Fortran literal.

Does nothing if the string appears to be quoted already.

rename_group(group, newgroup)[source]: Pass through to namelist definition

set_abs_file_path(file_path)[source]

If file_path is relative, make it absolute using DIN_LOC_ROOT.

If an absolute path is input, it is returned unchanged.

set_value(name, value)[source]

Set the current value of a given namelist variable.

Usually, you should use add_default instead of this function.

The name argument is the name of the variable to set, and the value is a list of strings to use as settings. If the variable is scalar, the list is optional; i.e. a scalar logical can be set using either value=’.false.’ or value=[‘.false.’]. If the variable is of type character, and the input is missing quotes, quotes will be added automatically. If None is provided in place of a string, this will be translated to a null value.

Note that this function will overwrite the current value, which may hold a user-specified setting. Even if value is (or contains) a null value, the old setting for the variable will be thrown out completely.

update_shr_strdata_nml(config, stream, stream_path)[source]

Updates values for the shr_strdata_nml namelist group.

This should be done once per stream, and it shouldn’t usually be called directly, since create_stream_file calls this method itself.

write_modelio_file(filename)[source]: Write mct component modelio files

write_nuopc_config_file(filename, data_list_path=None, sorted_groups=False)[source]: Write the nuopc config file

write_nuopc_modelio_file(filename)[source]: Write nuopc component modelio files

write_output_file(namelist_file, data_list_path=None, groups=None, sorted_groups=True)[source]

Write out the namelists and input data files.

The namelist_file and modelio_file are the locations to which the component and modelio namelists will be written, respectively. The data_list_path argument is the location of the *.input_data_list file, which will have the input data files added to it.

write_seq_maps(filename)[source]: Write mct out seq_maps.rc

CIME.provenance module

Library for saving build/run provenance.

CIME.provenance.get_recommended_test_time_based_on_past(baseline_root, test, raw=False)[source]

CIME.provenance.get_test_success(baseline_root, src_root, test, testing=False)[source]

Returns (was prev run success, commit when test last passed, commit when test last transitioned from pass to fail)

Unknown history is expressed as None

CIME.provenance.save_test_success(baseline_root, src_root, test, succeeded, force_commit_test=None)[source]: Update success data accordingly based on succeeded flag

CIME.provenance.save_test_time(baseline_root, test, time_seconds, commit)[source]

CIME.simple_compare module

CIME.simple_compare.compare_files(gold_file, compare_file, case=None)[source]: Returns true if files are the same, comments are returned too: (success, comments)

CIME.simple_compare.compare_runconfigfiles(gold_file, compare_file, case=None)[source]: Returns true if files are the same, comments are returned too: (success, comments)

CIME.simple_compare.findDiff(d1, d2, path='', case=None)[source]

CIME.status module

CIME.status.append_case_status(phase, status, msg=None, caseroot='.', gitinterface=None)[source]: Update CaseStatus file

CIME.status.append_status(msg, sfile, caseroot='.')[source]: Append msg to sfile in caseroot

CIME.status.append_testlog(msg, caseroot='.')[source]: Add to TestStatus.log file

CIME.status.run_and_log_case_status(func, phase, caseroot='.', custom_starting_msg_functor=None, custom_success_msg_functor=None, is_batch=False, gitinterface=None)[source]

CIME.test_scheduler module

A library for scheduling/running through the phases of a set of system tests. Supports phase-level parallelism (can make progres on multiple system tests at once).

TestScheduler will handle the TestStatus for the 1-time setup phases. All other phases need to handle their own status because they can be run outside the context of TestScheduler.

class CIME.test_scheduler.TestScheduler(test_names, test_data=None, no_run=False, no_build=False, no_setup=False, no_batch=None, test_root=None, test_id=None, machine_name=None, compiler=None, baseline_root=None, baseline_cmp_name=None, baseline_gen_name=None, clean=False, namelists_only=False, project=None, parallel_jobs=None, walltime=None, proc_pool=None, use_existing=False, save_timing=False, queue=None, allow_baseline_overwrite=False, output_root=None, force_procs=None, force_threads=None, mpilib=None, input_dir=None, pesfile=None, run_count=0, mail_user=None, mail_type=None, allow_pnl=False, non_local=False, single_exe=False, workflow=None, chksum=False, force_rebuild=False, driver=None)[source]

Bases: object

get_testnames()[source]

run_tests(wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_diffs=False, ignore_memleak=False)[source]

Main API for this class.

Return True if all tests passed.

CIME.test_status module

Contains the crucial TestStatus class which manages phase-state of a test case and ensure that this state is represented by the TestStatus file in the case.

TestStatus objects are only modifiable via the set_status method and this is only allowed if the object is being accessed within the context of a context manager. Example:

with TestStatus(test_dir=caseroot) as ts:
ts.set_status(RUN_PHASE, TEST_PASS_STATUS)

This file also contains all of the hardcoded phase information which includes the phase names, phase orders, potential phase states, and which phases are required (core phases).

Additional important design decisions: 1) In order to ensure that incomplete tests are always left in a PEND

state, updating a core phase to a PASS state will automatically set the next core state to PEND.

If the user repeats a core state, that invalidates all subsequent state. For example, if a user rebuilds their case, then any of the post-run states like the RUN state are no longer valid.

class CIME.test_status.TestStatus(test_dir=None, test_name=None, no_io=False)[source]

Bases: object

current_is(phase, status)[source]

flush()[source]

get_comment(phase)[source]

get_latest_phase()[source]

get_name()[source]

get_overall_test_status(wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_diffs=False, ignore_memleak=False, no_run=False)[source]

Given the current phases and statuses, produce a single results for this test. Preference is given to PEND since we don’t want to stop waiting for a test that hasn’t finished. Namelist diffs are given the lowest precedence.

>>> _test_helper2('PASS ERS.foo.A RUN')
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A SHAREDLIB_BUILD\nPEND ERS.foo.A RUN')
('PEND', 'RUN')
>>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN')
('FAIL', 'MODEL_BUILD')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN')
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP')
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP', check_throughput=True)
('DIFF', 'TPUTCOMP')
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A MEMCOMP', check_memory=True)
('DIFF', 'MEMCOMP')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
('NLFAIL', 'RUN')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
('PEND', 'RUN')
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A MEMCOMP')
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP', ignore_namelists=True)
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A COMPARE_1\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
('FAIL', 'COMPARE_2')
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
('DIFF', 'BASELINE')
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nPASS ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN', ignore_diffs=True)
('PASS', 'RUN')
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN', ignore_diffs=True)
('NLFAIL', 'RUN')
>>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
('FAIL', 'COMPARE_2')
>>> _test_helper2('PEND ERS.foo.A COMPARE_2\nFAIL ERS.foo.A RUN')
('FAIL', 'RUN')
>>> _test_helper2('PEND ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
('PEND', 'COMPARE_2')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD')
('PASS', 'MODEL_BUILD')
>>> _test_helper2('PEND ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN')
('PEND', 'MODEL_BUILD')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD', wait_for_run=True)
('PEND', 'RUN')
>>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD', wait_for_run=True)
('FAIL', 'MODEL_BUILD')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', wait_for_run=True)
('PEND', 'RUN')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN', wait_for_run=True)
('FAIL', 'RUN')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN', wait_for_run=True)
('PASS', 'RUN')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN\nPEND ERS.foo.A COMPARE')
('FAIL', 'RUN')
>>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', no_run=True)
('PASS', 'MODEL_BUILD')
>>> s = '''PASS ERS.foo.A CREATE_NEWCASE
... PASS ERS.foo.A XML
... PASS ERS.foo.A SETUP
... PASS ERS.foo.A SHAREDLIB_BUILD time=454
... PASS ERS.foo.A NLCOMP
... PASS ERS.foo.A MODEL_BUILD time=363
... PASS ERS.foo.A SUBMIT
... PASS ERS.foo.A RUN time=73
... PEND ERS.foo.A COMPARE_base_single_thread
... FAIL ERS.foo.A BASELINE master: DIFF
... PASS ERS.foo.A TPUTCOMP
... PASS ERS.foo.A MEMLEAK insuffiencient data for memleak test
... PASS ERS.foo.A SHORT_TERM_ARCHIVER
... '''
>>> _test_helper2(s, no_perm=True)
('PEND', 'COMPARE_base_single_thread')
>>> s = '''PASS ERS.foo.A CREATE_NEWCASE
... PASS ERS.foo.A XML
... PASS ERS.foo.A SETUP
... PEND ERS.foo.A SHAREDLIB_BUILD
... FAIL ERS.foo.A NLCOMP
... '''
>>> _test_helper2(s, no_run=True)
('NLFAIL', 'SETUP')
>>> _test_helper2(s, no_run=False)
('PEND', 'SHAREDLIB_BUILD')

get_status(phase)[source]

increment_non_pass_counts(non_pass_counts)[source]

Increment counts of the number of times given phases did not pass

non_pass_counts is a dictionary whose keys are phases of interest and whose values are running counts of the number of non-passes. This method increments those counts based on results in the given TestStatus object.

phase_statuses_dump(prefix='', skip_passes=False, skip_phase_list=None, xfails=None)[source]

Args:

prefix: string printed at the start of each line skip_passes: if True, do not output lines that have a PASS status skip_phase_list: list of phases (from the phases given by

ALL_PHASES) for which we skip output

xfails: object of type ExpectedFails, giving expected failures for this test

set_status(phase, status, comments='')[source]

Update the status of this test by changing the status of given phase to the given status.

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "PASS")
...     ts.set_status(XML_PHASE, "PASS")
...     ts.set_status(SETUP_PHASE, "FAIL")
...     ts.set_status(SETUP_PHASE, "PASS")
...     ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL")
...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42')), ('COMPARE_base_rest', ('FAIL', '')), ('MODEL_BUILD', ('PEND', ''))])

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "PASS")
...     ts.set_status(XML_PHASE, "PASS")
...     ts.set_status(SETUP_PHASE, "FAIL")
...     ts.set_status(SETUP_PHASE, "PASS")
...     ts.set_status(BASELINE_PHASE, "PASS")
...     ts.set_status("{}_base_rest".format(COMPARE_PHASE), "FAIL")
...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
...     ts.set_status(SETUP_PHASE, "PASS")
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PEND', ''))])

>>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
...     ts.set_status(CREATE_NEWCASE_PHASE, "FAIL")
>>> ts._phase_statuses
OrderedDict([('CREATE_NEWCASE', ('FAIL', ''))])

CIME.test_utils module

Utility functions used in test_scheduler.py, and by other utilities that need to get test lists.

CIME.test_utils.get_test_status_files(test_root, compiler, test_id=None)[source]

CIME.test_utils.get_tests_from_xml(xml_machine=None, xml_category=None, xml_compiler=None, xml_testlist=None, machine=None, compiler=None, driver=None)[source]: Parse testlists for a list of tests

CIME.test_utils.test_to_string(test, category_field_width=0, test_field_width=0, show_options=False)[source]

Given a test dictionary, return a string representation suitable for printing

Args:

test (dict): dictionary for a single test - e.g., one element from the: list returned by get_tests_from_xml

category_field_width (int): minimum amount of space to use for printing the test category test_field_width (int): minimum amount of space to use for printing the test category show_options (bool): if True, print test options, too (note that the ‘comment’

option is always printed, if present)

Basic functionality: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel’

Printing comments: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks’

Newlines in comments are converted to spaces >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘mynremarks’}} >>> test_to_string(mytest, 10) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks’

Printing other options, too: >>> mytest = {‘name’: ‘SMS.f19_g16.A.cheyenne_intel’, ‘category’: ‘prealpha’, ‘options’: {‘comment’: ‘my remarks’, ‘wallclock’: ‘0:20’, ‘memleak_tolerance’: 0.2}} >>> test_to_string(mytest, 10, show_options=True) ‘prealpha : SMS.f19_g16.A.cheyenne_intel # my remarks # memleak_tolerance: 0.2 # wallclock: 0:20’

CIME.user_mod_support module

user_mod_support.py

CIME.user_mod_support.apply_user_mods(caseroot, user_mods_path, keepexe=None)[source]

Recursivlely apply user_mods to caseroot - this includes updating user_nl_xxx, updating SourceMods and creating case shell_commands and xmlchange_cmds files

First remove case shell_commands files if any already exist

If this function is called multiple times, settings from later calls will take precedence over earlier calls, if there are conflicts.

keepexe is an optional argument that is needed for cases where apply_user_mods is called from create_clone

CIME.user_mod_support.build_include_dirs_list(user_mods_path, include_dirs=None)[source]: If user_mods_path has a file “include_user_mods” read that file and add directories to the include_dirs, recursively check each of those directories for further directories. The file may also include comments deleneated with # in the first column

CIME.utils module

Common functions used by cime python scripts Warning: you cannot use CIME Classes in this module as it causes circular dependencies

exception CIME.utils.CIMEError[source]: Bases: SystemExit, Exception

class CIME.utils.EnvironmentContext(**kwargs)[source]

Bases: object

Context manager for environment variables Usage:

os.environ[‘MYVAR’] = ‘oldvalue’ with EnvironmentContex(MYVAR=’myvalue’, MYVAR2=’myvalue2’):

print os.getenv(‘MYVAR’) # Should print myvalue. print os.getenv(‘MYVAR2’) # Should print myvalue2.

print os.getenv(‘MYVAR’) # Should print oldvalue. print os.getenv(‘MYVAR2’) # Should print None.

CREDIT: https://github.com/sakurai-youhei/envcontext

class CIME.utils.IndentFormatter(indent, fmt=None, datefmt=None)[source]

Bases: Formatter

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class CIME.utils.SharedArea(new_perms=2)[source]

Bases: object

Enable 0002 umask within this manager

class CIME.utils.Timeout(seconds, action=None)[source]

Bases: object

A context manager that implements a timeout. By default, it will raise exception, but a custon function call can be provided. Provided None as seconds makes this class a no-op

CIME.utils.add_flag_to_cmd(flag, val)[source]

Given a flag and value for a shell command, return a string

>>> add_flag_to_cmd("-f", "hi")
'-f hi'
>>> add_flag_to_cmd("--foo", 42)
'--foo 42'
>>> add_flag_to_cmd("--foo=", 42)
'--foo=42'
>>> add_flag_to_cmd("--foo:", 42)
'--foo:42'
>>> add_flag_to_cmd("--foo:", " hi ")
'--foo:hi'

CIME.utils.add_mail_type_args(parser)[source]

CIME.utils.analyze_build_log(comp, log, compiler)[source]: Capture and report warning count, capture and report errors and undefined references.

CIME.utils.batch_jobid(case=None)[source]

CIME.utils.check_name(fullname, additional_chars=None, fullpath=False)[source]

check for unallowed characters in name, this routine only checks the final name and does not check if path exists or is writable

>>> check_name("test.id", additional_chars=".")
False
>>> check_name("case.name", fullpath=False)
True
>>> check_name("/some/file/path/case.name", fullpath=True)
True
>>> check_name("mycase+mods")
False
>>> check_name("mycase?mods")
False
>>> check_name("mycase*mods")
False
>>> check_name("/some/full/path/name/")
False

CIME.utils.clear_folder(_dir)[source]

CIME.utils.compute_total_time(job_cost_map, proc_pool)[source]

Given a map: jobname -> (procs, est-time), return a total time estimate for a given processor pool size

>>> job_cost_map = {"A" : (4, 3000), "B" : (2, 1000), "C" : (8, 2000), "D" : (1, 800)}
>>> compute_total_time(job_cost_map, 8)
5160
>>> compute_total_time(job_cost_map, 12)
3180
>>> compute_total_time(job_cost_map, 16)
3060

CIME.utils.configure_logging(verbose, debug, silent, **_)[source]

CIME.utils.convert_to_babylonian_time(seconds)[source]

Convert time value to seconds to HH:MM:SS

>>> convert_to_babylonian_time(3661)
'01:01:01'
>>> convert_to_babylonian_time(360000)
'100:00:00'

CIME.utils.convert_to_seconds(time_str)[source]

Convert time value in [[HH:]MM:]SS to seconds

We assume that XX:YY is likely to be HH:MM, not MM:SS

>>> convert_to_seconds("42")
42
>>> convert_to_seconds("01:01:01")
3661
>>> convert_to_seconds("01:01")
3660

CIME.utils.convert_to_string(value, type_str=None, vid='')[source]: Convert value back to string. vid is only for generating better error messages. >>> convert_to_string(6, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6’, type_str=”integer”) == ‘6’ True >>> convert_to_string(‘6.0’, type_str=”real”) == ‘6.0’ True >>> convert_to_string(6.01, type_str=”real”) == ‘6.01’ True

CIME.utils.convert_to_type(value, type_str, vid='')[source]: Convert value from string to another type. vid is only for generating better error messages.

CIME.utils.convert_to_unknown_type(value)[source]: Convert value to it’s real type by probing conversions.

CIME.utils.copy_globs(globs_to_copy, output_directory, lid=None)[source]

Takes a list of globs and copies all files to output_directory.

Hiddens files become unhidden i.e. removing starting dot.

Output filename is derviced from the basename of the input path and can be appended with the lid.

CIME.utils.copy_local_macros_to_dir(destination, extra_machdir=None)[source]

Copy any local macros files to the path given by ‘destination’.

Local macros files are potentially found in: (1) extra_machdir/cmake_macros/.cmake (2) $HOME/.cime/.cmake

CIME.utils.copyifnewer(src, dest)[source]: if dest does not exist or is older than src copy src to dest

CIME.utils.deprecate_action(message)[source]

CIME.utils.does_file_have_string(filepath, text)[source]: Does the text string appear in the filepath file

CIME.utils.expect(condition, error_msg, exc_type=<class 'CIME.utils.CIMEError'>, error_prefix='ERROR:')[source]

Similar to assert except doesn’t generate an ugly stacktrace. Useful for checking user error, not programming error.

>>> expect(True, "error1")
>>> expect(False, "error2") 
Traceback (most recent call last):
    ...
CIMEError: ERROR: error2

CIME.utils.file_contains_python_function(filepath, funcname)[source]

Checks whether the given file contains a top-level definition of the function ‘funcname’

Returns a boolean value (True if the file contains this function definition, False otherwise)

CIME.utils.filter_unicode(unistr)[source]: Sometimes unicode chars can cause problems

CIME.utils.find_files(rootdir, pattern)[source]: recursively find all files matching a pattern

CIME.utils.find_proc_id(proc_name=None, children_only=False, of_parent=None)[source]: Children implies recursive.

CIME.utils.find_system_test(testname, case)[source]: Find and import the test matching testname Look through the paths set in config_files.xml variable SYSTEM_TESTS_DIR for components used in this case to find a test matching testname. Add the path to that directory to sys.path if its not there and return the test object Fail if the test is not found in any of the paths.

CIME.utils.fixup_sys_path(*additional_paths)[source]

CIME.utils.format_time(time_format, input_format, input_time)[source]

Converts the string input_time from input_format to time_format Valid format specifiers are “%H”, “%M”, and “%S” % signs must be followed by an H, M, or S and then a separator Separators can be any string without digits or a % sign Each specifier can occur more than once in the input_format, but only the first occurence will be used. An example of a valid format: “%H:%M:%S” Unlike strptime, this does support %H >= 24

>>> format_time("%H:%M:%S", "%H", "43")
'43:00:00'
>>> format_time("%H  %M", "%M,%S", "59,59")
'0  59'
>>> format_time("%H, %S", "%H:%M:%S", "2:43:9")
'2, 09'

CIME.utils.get_all_cime_models()[source]

CIME.utils.get_batch_script_for_job(job, hidden=None)[source]

CIME.utils.get_charge_account(machobj=None, project=None)[source]

Hierarchy for choosing CHARGE_ACCOUNT: 1. Environment variable CHARGE_ACCOUNT 2. File $HOME/.cime/config 3. config_machines.xml (if machobj provided) 4. default to same value as PROJECT

>>> import CIME
>>> import CIME.XML.machines
>>> machobj = CIME.XML.machines.Machines(machine="docker")
>>> project = get_project(machobj)
>>> charge_account = get_charge_account(machobj, project)
>>> project == charge_account
True
>>> os.environ["CHARGE_ACCOUNT"] = "ChargeAccount"
>>> get_charge_account(machobj, project)
'ChargeAccount'
>>> del os.environ["CHARGE_ACCOUNT"]

CIME.utils.get_cime_config()[source]

CIME.utils.get_cime_default_driver()[source]

CIME.utils.get_cime_root(case=None)[source]: Return the absolute path to the root of CIME that contains this script

CIME.utils.get_config_path()[source]

CIME.utils.get_current_branch(repo=None)[source]

Return the name of the current branch for a repository

>>> if "GIT_BRANCH" in os.environ:
...     get_current_branch() is not None
... else:
...     os.environ["GIT_BRANCH"] = "foo"
...     get_current_branch() == "foo"
True

CIME.utils.get_current_commit(short=False, repo=None, tag=False)[source]

Return the sha1 of the current HEAD commit

>>> get_current_commit() is not None
True

CIME.utils.get_current_submodule_status(recursive=False, repo=None)[source]

Return the sha1s of the current currently checked out commit for each submodule, along with the submodule path and the output of git describe for the SHA-1.

>>> get_current_submodule_status() is not None
True

CIME.utils.get_full_test_name(partial_test, caseopts=None, grid=None, compset=None, machine=None, compiler=None, testmods_list=None, testmods_string=None)[source]

Given a partial CIME test name, return in form TESTCASE.GRID.COMPSET.MACHINE_COMPILER[.TESTMODS] Use the additional args to fill out the name if needed

Testmods can be provided through one of two arguments, but not both: - testmods_list: a list of one or more testmods (as would be returned by

parse_test_name, for example)

testmods_string: a single string containing one or more testmods; if there is more than one, then they should be separated by a string of two hyphens (’–‘)

For both testmods_list and testmods_string, any slashes as path separators (‘/’) are replaced by hyphens (‘-‘).

>>> get_full_test_name("ERS", grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS", caseopts=["D", "P16"], grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS_D_P16.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu'
>>> get_full_test_name("ERS.ne16_fe16.JGF.melvin_gnu.mods", machine="melvin", compiler="gnu")
'ERS.ne16_fe16.JGF.melvin_gnu.mods'

testmods_list can be a single element: >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_list=[“mods/test”]) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test’

testmods_list can also have multiple elements, separated either by slashes or hyphens: >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_list=[“mods/test”, “mods2/test2/subdir2”, “mods3/test3/subdir3”]) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3’ >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_list=[“mods-test”, “mods2-test2-subdir2”, “mods3-test3-subdir3”]) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3’

The above testmods_list tests should also work with equivalent testmods_string arguments: >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_string=”mods/test”) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test’ >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_string=”mods/test–mods2/test2/subdir2–mods3/test3/subdir3”) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3’ >>> get_full_test_name(“ERS.ne16_fe16.JGF”, machine=”melvin”, compiler=”gnu”, testmods_string=”mods-test–mods2-test2-subdir2–mods3-test3-subdir3”) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3’

The following tests the consistency check between the test name and various optional arguments: >>> get_full_test_name(“ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3”, machine=”melvin”, compiler=”gnu”, testmods_list=[“mods/test”, “mods2/test2/subdir2”, “mods3/test3/subdir3”]) ‘ERS.ne16_fe16.JGF.melvin_gnu.mods-test–mods2-test2-subdir2–mods3-test3-subdir3’

CIME.utils.get_htmlroot(machobj=None)[source]

Get location for test HTML output

Hierarchy for choosing CIME_HTML_ROOT: 0. Environment variable CIME_HTML_ROOT 1. File $HOME/.cime/config 2. config_machines.xml (if machobj provided)

CIME.utils.get_lids(case)[source]

CIME.utils.get_logging_options()[source]: Use to pass same logging options as was used for current executable to subprocesses.

CIME.utils.get_model()[source]

Get the currently configured model value The CIME_MODEL env variable may or may not be set

>>> os.environ["CIME_MODEL"] = "garbage"
>>> get_model() 
Traceback (most recent call last):
    ...
CIMEError: ERROR: model garbage not recognized
>>> del os.environ["CIME_MODEL"]
>>> set_model('rocky') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: model rocky not recognized
>>> set_model('e3sm')
>>> get_model()
'e3sm'
>>> reset_cime_config()

CIME.utils.get_model_config_location_within_cime(model=None)[source]

CIME.utils.get_model_config_root(model=None)[source]

Get absolute path to model config area”

>>> os.environ["CIME_MODEL"] = "e3sm" # Set the test up don't depend on external resources
>>> os.path.isdir(get_model_config_root())
True

CIME.utils.get_project(machobj=None)[source]: Hierarchy for choosing PROJECT: 0. Command line flag to create_newcase or create_test 1. Environment variable PROJECT 2 Environment variable ACCOUNT (this is for backward compatibility) 3. File $HOME/.cime/config (this is new) 4 File $HOME/.cesm_proj (this is for backward compatibility) 5 config_machines.xml (if machobj provided)

CIME.utils.get_python_libs_location_within_cime()[source]: From within CIME, return subdirectory of python libraries

CIME.utils.get_schema_path()[source]

CIME.utils.get_scripts_root()[source]

Get absolute path to scripts

>>> os.path.isdir(get_scripts_root())
True

CIME.utils.get_src_root()[source]: Return the absolute path to the root of SRCROOT.

CIME.utils.get_template_path()[source]

CIME.utils.get_time_in_seconds(timeval, unit)[source]: Convert a time from ‘unit’ to seconds

CIME.utils.get_timestamp(timestamp_format='%Y%m%d_%H%M%S', utc_time=False)[source]

Get a string representing the current UTC time in format: YYYYMMDD_HHMMSS

The format can be changed if needed.

CIME.utils.get_tools_path()[source]

CIME.utils.get_umask()[source]

CIME.utils.get_urlroot(machobj=None)[source]

Get URL to htmlroot

Hierarchy for choosing CIME_URL_ROOT: 0. Environment variable CIME_URL_ROOT 1. File $HOME/.cime/config 2. config_machines.xml (if machobj provided)

CIME.utils.gunzip_existing_file(filepath)[source]

CIME.utils.gzip_existing_file(filepath)[source]

Gzips an existing file, removes the unzipped version, returns path to zip file. Note the that the timestamp of the original file will be maintained in the zipped file.

>>> import tempfile
>>> fd, filename = tempfile.mkstemp(text=True)
>>> _ = os.write(fd, b"Hello World")
>>> os.close(fd)
>>> gzfile = gzip_existing_file(filename)
>>> gunzip_existing_file(gzfile) == b'Hello World'
True
>>> os.remove(gzfile)

CIME.utils.id_generator(size=6, chars='abcdefghijklmnopqrstuvwxyz0123456789')[source]

CIME.utils.import_and_run_sub_or_cmd(cmd, cmdargs, subname, subargs, config_dir, compname, logfile=None, case=None, from_dir=None, timeout=None)[source]

CIME.utils.import_from_file(name, file_path)[source]

CIME.utils.indent_string(the_string, indent_level)[source]

Indents the given string by a given number of spaces

Args:: the_string: str indent_level: int

Returns a new string that is the same as the_string, except that each line is indented by ‘indent_level’ spaces.

In python3, this can be done with textwrap.indent.

CIME.utils.is_comp_standalone(case)[source]: Test if the case is a single component standalone such as FKESSLER

CIME.utils.is_last_process_complete(filepath, expect_text, fail_text)[source]: Search the filepath in reverse order looking for expect_text before finding fail_text. This utility is used by archive_metadata.

CIME.utils.is_python_executable(filepath)[source]

CIME.utils.ls_sorted_by_mtime(path)[source]: return list of path sorted by timestamp oldest first

CIME.utils.match_any(item, re_counts)[source]: Return true if item matches any regex in re_counts’ keys. Increments count if a match was found.

CIME.utils.model_log(model, arg_logger, msg, debug_others=True)[source]

CIME.utils.new_lid(case=None)[source]

CIME.utils.normalize_case_id(case_id)[source]

Given a case_id, return it in form TESTCASE.GRID.COMPSET.PLATFORM

>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.G.20151121')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel'
>>> normalize_case_id('ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod.G.20151121')
'ERT.ne16_g37.B1850C5.sandiatoss3_intel.test-mod'

CIME.utils.parse_args_and_handle_standard_logging_options(args, parser=None)[source]

Guide to logging in CIME.

logger.debug -> Verbose/detailed output, use for debugging, off by default. Goes to a .log file logger.info -> Goes to stdout (and log if –debug). Use for normal program output logger.warning -> Goes to stderr (and log if –debug). Use for minor problems logger.error -> Goes to stderr (and log if –debug)

CIME.utils.parse_test_name(test_name)[source]

Given a CIME test name TESTCASE[_CASEOPTS].GRID.COMPSET[.MACHINE_COMPILER[.TESTMODS]], return each component of the testname with machine and compiler split. Do not error if a partial testname is provided (TESTCASE or TESTCASE.GRID) instead parse and return the partial results.

TESTMODS use hyphens in a special way: - A single hyphen stands for a path separator (for example, ‘test-mods’ resolves to

the path ‘test/mods’)

A double hyphen separates multiple test mods (for example, ‘test-mods–other-dir-path’ indicates two test mods: ‘test/mods’ and ‘other/dir/path’)

If there are one or more TESTMODS, then the testmods component of the result will be a list, where each element of the list is one testmod, and hyphens have been replaced by slashes. (If there are no TESTMODS in this test, then the TESTMODS component of the result is None, as for other optional components.)

>>> parse_test_name('ERS')
['ERS', None, None, None, None, None, None]
>>> parse_test_name('ERS.fe12_123')
['ERS', None, 'fe12_123', None, None, None, None]
>>> parse_test_name('ERS.fe12_123.JGF')
['ERS', None, 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('ERS_D.fe12_123.JGF')
['ERS', ['D'], 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('ERS_D_P1.fe12_123.JGF')
['ERS', ['D', 'P1'], 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('ERS_D_G2.fe12_123.JGF')
['ERS', ['D', 'G2'], 'fe12_123', 'JGF', None, None, None]
>>> parse_test_name('SMS_D_Ln9_Mmpi-serial.f19_g16_rx1.A')
['SMS', ['D', 'Ln9', 'Mmpi-serial'], 'f19_g16_rx1', 'A', None, None, None]
>>> parse_test_name('ERS.fe12_123.JGF.machine_compiler')
['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', None]
>>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods')
['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', ['test/mods']]
>>> parse_test_name('ERS.fe12_123.JGF.*_compiler.test-mods')
['ERS', None, 'fe12_123', 'JGF', None, 'compiler', ['test/mods']]
>>> parse_test_name('ERS.fe12_123.JGF.machine_*.test-mods')
['ERS', None, 'fe12_123', 'JGF', 'machine', None, ['test/mods']]
>>> parse_test_name('ERS.fe12_123.JGF.*_*.test-mods')
['ERS', None, 'fe12_123', 'JGF', None, None, ['test/mods']]
>>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods--other-dir-path--and-one-more')
['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', ['test/mods', 'other/dir/path', 'and/one/more']]
>>> parse_test_name('SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Expected 4th item of 'SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods' ('A_XLND_SICE_SOCN_XROF_XGLC_SWAV') to be in form machine_compiler
>>> parse_test_name('SMS.f19_g16.2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Invalid compset name 2000_DATM%QI/A_XLND_SICE_SOCN_XROF_XGLC_SWAV

CIME.utils.redirect_logger(new_target, logger_name)[source]

CIME.utils.redirect_stderr(new_target)[source]

CIME.utils.redirect_stdout(new_target)[source]

CIME.utils.redirect_stdout_stderr(new_target)[source]

CIME.utils.reset_cime_config()[source]: Useful to keep unit tests from interfering with each other

CIME.utils.resolve_mail_type_args(args)[source]

CIME.utils.run_bld_cmd_ensure_logging(cmd, arg_logger, from_dir=None, timeout=None)[source]

CIME.utils.run_cmd(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None, executable=None, shell=True)[source]

Wrapper around subprocess to make it much more convenient to run shell commands

>>> run_cmd('ls file_i_hope_doesnt_exist')[0] != 0
True

CIME.utils.run_cmd_no_fail(cmd, input_str=None, from_dir=None, verbose=None, arg_stdout=<object object>, arg_stderr=<object object>, env=None, combine_output=False, timeout=None, executable=None)[source]

Wrapper around subprocess to make it much more convenient to run shell commands. Expects command to work. Just returns output string.

>>> run_cmd_no_fail('echo foo') == 'foo'
True
>>> run_cmd_no_fail('echo THE ERROR >&2; false') 
Traceback (most recent call last):
    ...
CIMEError: ERROR: Command: 'echo THE ERROR >&2; false' failed with error ...

>>> run_cmd_no_fail('grep foo', input_str=b'foo') == 'foo'
True
>>> run_cmd_no_fail('echo THE ERROR >&2', combine_output=True) == 'THE ERROR'
True

CIME.utils.run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None, from_dir=None, timeout=None)[source]

This code will try to import and run each cmd as a subroutine if that fails it will run it as a program in a seperate shell

Raises exception on failure.

CIME.utils.safe_copy(src_path, tgt_path, preserve_meta=True)[source]

A flexbile and safe copy routine. Will try to copy file and metadata, but this can fail if the current user doesn’t own the tgt file. A fallback data-only copy is attempted in this case. Works even if overwriting a read-only file.

tgt_path can be a directory, src_path must be a file

most of the complexity here is handling the case where the tgt_path file already exists. This problem does not exist for the tree operations so we don’t need to wrap those.

preserve_meta toggles if file meta-data, like permissions, should be preserved. If you are copying baseline files, you should be within a SharedArea context manager and preserve_meta should be false so that the umask set up by SharedArea can take affect regardless of the permissions of the src files.

CIME.utils.safe_recursive_copy(src_dir, tgt_dir, file_map)[source]: Copies a set of files from one dir to another. Works even if overwriting a read-only file. Files can be relative paths and the relative path will be matched on the tgt side.

CIME.utils.set_logger_indent(indent)[source]

CIME.utils.set_model(model)[source]: Set the model to be used in this session

CIME.utils.setup_standard_logging_options(parser)[source]

CIME.utils.start_buffering_output()[source]: All stdout, stderr will be buffered after this is called. This is python’s default behavior.

CIME.utils.stop_buffering_output()[source]: All stdout, stderr will not be buffered after this is called.

CIME.utils.string_in_list(_string, _list)[source]: Case insensitive search for string in list returns the matching list value >>> string_in_list(“Brack”,[“bar”, “bracK”, “foo”]) ‘bracK’ >>> string_in_list(“foo”, [“FFO”, “FOO”, “foo2”, “foo3”]) ‘FOO’ >>> string_in_list(“foo”, [“FFO”, “foo2”, “foo3”])

CIME.utils.stringify_bool(val)[source]

CIME.utils.symlink_force(target, link_name)[source]: Makes a symlink from link_name to target. Unlike the standard os.symlink, this will work even if link_name already exists (in which case link_name will be overwritten).

CIME.utils.touch(fname)[source]

CIME.utils.transform_vars(text, case=None, subgroup=None, overrides=None, default=None)[source]

Do the variable substitution for any variables that need transforms recursively.

>>> transform_vars("{{ cesm_stdout }}", default="cesm.stdout")
'cesm.stdout'
>>> member_store = lambda : None
>>> member_store.foo = "hi"
>>> transform_vars("I say {{ foo }}", overrides={"foo":"hi"})
'I say hi'

CIME.utils.verbatim_success_msg(return_val)[source]

CIME.utils.wait_for_unlocked(filepath)[source]

CIME.wait_for_tests module

CIME.wait_for_tests.create_cdash_build_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]

CIME.wait_for_tests.create_cdash_config_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]

CIME.wait_for_tests.create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)[source]

CIME.wait_for_tests.create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_time, hostname, force_log_upload)[source]

CIME.wait_for_tests.create_cdash_xml(results, cdash_build_name, cdash_project, cdash_build_group, force_log_upload=False)[source]

CIME.wait_for_tests.create_cdash_xml_boiler(phase, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit)[source]

CIME.wait_for_tests.create_cdash_xml_fakes(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname)[source]

CIME.wait_for_tests.get_nml_diff(test_path)[source]

CIME.wait_for_tests.get_test_output(test_path)[source]

CIME.wait_for_tests.get_test_phase(test_path, phase)[source]

CIME.wait_for_tests.get_test_time(test_path)[source]

CIME.wait_for_tests.set_up_signal_handlers()[source]

CIME.wait_for_tests.signal_handler(*_)[source]

CIME.wait_for_tests.wait_for_test(test_path, results, wait, check_throughput, check_memory, ignore_namelists, ignore_diffs, ignore_memleak, no_run)[source]

CIME.wait_for_tests.wait_for_tests(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_diffs=False, ignore_memleak=False, cdash_build_name=None, cdash_project='E3SM', cdash_build_group='ACME_Latest', timeout=None, force_log_upload=False, no_run=False, update_success=False, expect_test_complete=True)[source]

CIME.wait_for_tests.wait_for_tests_impl(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_diffs=False, ignore_memleak=False, no_run=False)[source]

CIME package

Subpackages

Submodules

CIME.aprun module

CIME.bless_test_results module

CIME.build module

CIME.buildlib module

CIME.buildnml module

CIME.code_checker module

CIME.compare_namelists module

CIME.compare_test_results module

CIME.config module

CIME.cs_status module

CIME.cs_status_creator module

CIME.date module

CIME.expected_fails module

CIME.get_tests module

CIME.get_timing module

CIME.gitinterface module

CIME.hist_utils module

CIME.jenkins_generic_job module

CIME.locked_files module

CIME.namelist module

CIME.nmlgen module

CIME.provenance module

CIME.simple_compare module

CIME.status module

CIME.test_scheduler module

CIME.test_status module

CIME.test_utils module

CIME.user_mod_support module

CIME.utils module

CIME.wait_for_tests module

Module contents