compare_test_results¶

compare_test_results is a script in CIMEROOT/scripts/Tools.
$ ./compare_test_results --help
usage: 
compare_test_results [-r <TESTROOT>] [-b <BRANCH> -c <COMPILER>] [-t <TESTID>] [<TEST> <TEST> ...] [--verbose]
OR
compare_test_results --help

[1mEXAMPLES:[0m
    [1;32m# From most recent run, compare all changes [0m
    > compare_test_results
    [1;32m# From most recent run, compare only changes for test foo and bar only [0m
    > compare_test_results foo bar
    [1;32m# For an old run where you know test-id, compare only changes for test foo and bar only [0m
    > compare_test_results foo bar -t mytestid
    [1;32m# From most recent run of jenkins, compare history changes for next [0m
    > compare_test_results -r /home/jenkins/acme/scratch/jenkins -b next
    [1;32m# For typical CESM workflow, where baselines are named with tags [0m
    > compare_test_results -t TESTID -b BASELINE_TAG

Analyze results from a test root area, comparing non-BFB changes. Purpose is,
instead of re-running tests in compare mode, which is very slow, allow for
very fast analysis of diffs. Outputs results for each test to stdout (one line
per test); possible status codes are: PASS, FAIL, SKIP. (A SKIP denotes a test
that did not make it to the run phase or a test for which the run phase did
not pass: we skip baseline comparisons in this case.) In addition, creates
files named compare.log.BASELINE_NAME.TIMESTAMP in each test directory, which
contain more detailed output. Also creates *.cprnc.out.BASELINE_NAME.TIMESTAMP
files in each run directory. Returns a 0 exit status if all tests are bit-for-
bit, and a non-zero exit status (TESTS_FAILED_ERR_CODE) if any tests differed
from the baseline. You may need to load modules for cprnc to work.

positional arguments:
  compare_tests         When comparing, limit the comparison to tests matching
                        these regex (default: None)

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print debug information (very verbose) to file /Users/
                        sacks/cime/scripts/Tools/compare_test_results.log
                        (default: False)
  -v, --verbose         Add additional context (time and file) to log messages
                        (default: False)
  -s, --silent          Print only warnings and error messages (default:
                        False)
  -n, --namelists-only  Only analyze namelists. (default: False)
  --hist-only           Only analyze history files. (default: False)
  -b BASELINE_NAME, --baseline-name BASELINE_NAME
                        Name of baselines to use. Default will use
                        BASELINE_NAME_CMP first if possible, otherwise branch
                        name. (default: None)
  --baseline-root BASELINE_ROOT
                        Root of baselines. Default will use BASELINE_ROOT from
                        the case. (default: None)
  -c COMPILER, --compiler COMPILER
                        Compiler of run you want to compare (default: gnu)
  -r TEST_ROOT, --test-root TEST_ROOT
                        Path to test results that are being compared (default:
                        /Users/sacks/projects/scratch)
  -t TEST_ID, --test-id TEST_ID
                        Limit processes to case dirs matching this test-id.
                        Can be useful if mutiple runs dumped into the same
                        dir. (default: None)