The Coupled Model Intercomparison Project (CMIP), now in its sixth phase (CMIP6) provides the climate and impact assessment community with a wide range of simulation output from Earth System Models (Eyring et al., 2016b). These simulations help us better understand past climate and also provide estimates for the future under different greenhouse gas emission scenarios. It is crucial to evaluate these simulations to better understand systemic biases in the models and the magnitude of uncertainty in future projections before using them to provide policy advice and in impact assessment studies.

There is also a need for a comprehensive framework to perform baseline aspects of model evaluation in a consistent and efficient manner. This will prevent waste of valuable time and effort by scientists individually re-implementing well established evaluation diagnostics instead of using readily available tools. A tool that implements basic metrics and diagnostics for model output evaluation, but also has the flexibility of allowing newly developed science-based evaluation measures is therefore the need of the hour. We present the Earth System Model Evaluation Tool (ESMValTool) as a tool compatible with current and future phases of CMIP, possessing many desirable features for comprehensive model evaluation (Eyring et al., 2016a).

Figure 1: Overview of ESMValTool’s workflow starting with user input (configuration and recipe files), data retrieval and ingestion, compliance checks and fixes, standardized preprocessing steps that are freely chosen by the user diagnostic run and final output.

ESMValTool is an open source community diagnostics and performance metric tool allowing comparison between multiple CMIP models, against different versions of these models, as well as against observations. The tool provides a versatile and highly optimized preprocessor core to perform standardized data operations such as re-gridding, vertical level selection, masking as well as temporal and spatial extraction capabilities. Standard recipes for scientific topics reproduce specific diagnostics and performance metrics considered important for ESM evaluation from the peer-reviewed literature (Lauer et al., 2017); integration of existing metrics and diagnostics from other packages is also possible due to the tool’s modular structure. The tool is currently at version 2.0 and the most recent release allows new analyses and recipes to be written in one of many open source languages such as Python, R, NCL or Julia. Figure 1 shows a high-level overview of ESMValTool.

Some key advantages of ESMValTool are:

1. Community shared and Open-source: In-built powerful preprocessor functionalities with diagnostics that can be reused or adapted from existing ones. An example is Figure 2, showing a standard plot diagnostic adapted from the IPCC’s Fifth Assessment Report to analyze average surface temperature and biases against reanalysis (ERA-Interim) data in the more recent CMIP6 models.
2. Scientific Validation and Reproducibility: Extensive documentation, log files for improved traceability and code sharing via GitHub enables the wider scientific community to reproduce and verify results from published diagnostics, as well as develop and share new diagnostics.
3. Scope and Flexibility: Wide scope with standard recipes covering different aspects of ESMs such as dynamics, clouds, radiation, aerosols and sea ice. The tool is also flexible, with diagnostics written in one of several open source languages (Python, NCL, R or Julia).
4. Central Installation: on the Centre for Environmental Data Analysis’s (CEDA’s) JASMIN infrastructure, with CMIP model data retrieval and a large selection of compliant observation datasets automatically facilitated.
5. Convenient Output Formats: Diagnostic outputs produced in a whole host of convenient formats such as plots (pdf, png etc.) and netcdf files.

Figure 2: Annual-mean near surface (2m) air temperature (^oC) for the period 1980 -2014. On the left is the multi-model ensemble mean constructed with one realization each for a selection of CMIP6 models available on CEDA-JASMIN and on the right is the multi-model-mean difference from the ECMWF reanalysis (ERA) – Interim data.

ESMValTool v2.0 continues to evolve with diagnostics ported from the previous release, relevant diagnostics ported from other assessment tools (see Figure 3 produced with the AutoAssess Stratosphere diagnostic, ported into ESMValTool v2.0) as well as new ones being added. Figure 4 shows the Atlantic Meridional Overturning Circulation (AMOC) at 26°N in some CMIP5 models. The AMOC is an important circulation feature, used to gauge the performance of models in representing ocean circulation (Cheng 2013).

The tool can be accessed from the ESMValTool GitHub page and a sample of CMIP results produced with ESMValTool is available from the German climate modelling center at DKRZ. ESMValTool is a powerful, optimised, documented, modular, and well-supported tool and we strongly encourage the climate modelling community to actively adopt it for their evaluation and analysis work.

Figure 3: Quasi-Biannual Oscillation (QBO): vertical cross-section versus time for zonal mean equatorial (5N to 5S) zonal wind speed. Comparison between UKESM1-00-LL and HadGEM3-GC31-LL; plots made using the Autoassess Stratosphere assessment area, which has been ported to ESMValTool and is available part of the diagnostics library of ESMValTool version 2.0.

Figure 4: Figure showing the Atlantic Meridional Overturning Circulation in a range of CMIP5 models

References

• Eyring et al. 2016a: Towards improved and more routine Earth System Model Evaluation in CMIP5. Earth System Dynamics, 7, 813-830, http://doi:10.5194/esd-7-813-2016
• Lauer et al. 2017: Benchmarking CMIP5 models with a subset of ESA CCI Phase 2 data using the ESMValTool, Remote Sensing Environment, 203, 9-39, http://doi:10.1016/j.rse.2017.01.007
• The ESMValTool Webpage: https://esmvaltool.org, http://doi.10.17874/ac8548f0315
• Eyring et al. 2016b: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geoscientific Model Development, 9, 1937-1958, http://doi.10.5194/gmd-9-1937-2016
• ESMValTool GitHub Page: https://github.com/ESMValGroup/ESMValTool
• German CMIP6 Project evaluation results with the ESMValTool: http://cmip-esmvaltool.dkrz.de/
• Cheng et al 2013, Atlantic Meridional Overturning Circulation (AMOC) in CMIP5 Models: RCP and Historical Simulations, Journal of Climate, 26 (18), p7178-7197, https://doi.org/10.1175/JCLI-D-12-00496.1