Colin Jones, Marc Stringer, Richard Hill, Colin Johnson, Mohit Dalvi and Jeremy Walton

The high resolution version of UKESM1 (UKESM1-hr-ESM) is at the forefront of international Earth system models, both with respect to model resolution and process realism. As a consequence UKESM1-hr-ESM is computationally expensive and, at present, can only be used for relatively short, decadal time scale simulations. For centennial simulations, such as planned in CMIP6, we would like to realize the benefits of both high model resolution and process realism. To do this we need to find a compromise that retains the main benefits of both while remaining computationally viable. The leading computational overhead in UKESM1-hr-ESM is the UKCA chemistry and aerosol scheme. Presently, UKCA is embedded within the Unified Model (UM) atmosphere of UKESM1-hr-ESM, running as part of a single UM executable and relying on a suite of UM parameterisations, as well as the UM advection scheme and ancillary files. One method being explored to reduce the overall cost of UKESM1-hr-ESM is to run UKCA interactively coupled to the UM atmosphere, but with chemistry and aerosol calculations and tracer advection performed at a lower resolution. Specifically, we are developing a system whereby the UM atmosphere is run at N216 resolution and UKCA is coupled to the UM every time step but run at N96 resolution. Each model runs on the same vertical grid. This configuration is referred to below as UKESM1-hyb.

Figure 1
Figure 1. Planned configuration of UKESM1-hyb.

To run UKCA independent of the UM requires frequent (time step level) two-way exchange of numerous 3D fields. Prognostic fields from the UM are required by UKCA, as well as a number of 3D physical parameterization fields (e.g. vertical turbulent fluxes and 3D precipitation fluxes). Conversely, UKCA trace gas and aerosol fields are required by the UM radiation and cloud parameterizations. After significant development the OASIS3-MCT coupler appears to be a technically viable option for the two-way exchange and interpolation of these 3D fields, with an acceptable computational overhead.

Initially, we aimed to run UKCA as a stand-alone model, driven by the required UM fields. This proved to be difficult, both because of the number and variety of fields to be exchanged and because UKCA relies on the UM STASH diagnostics and ancillary file system. As a result, it was decided the best approach was to run 2 separate UM jobs, termed Senior UM (S-UM at N216) and Junior UM (J-UM at N96). S-UM runs the full UM but does not have UKCA active, while J-UM runs both the UM and UKCA including tracer advection. Retention of UKCA within a formal UM model (J-UM at N96) solves the problem of UKCA requiring UM STASH and ancillary files, with both of these being automatically available. In this configuration S-UM simulates the evolution of the physical climate at N216 and every time step all 3D dynamical core fields and a number of other prognostics fields are passed to J-UM using the OASIS3-MCT coupler which performs the required interpolation from the S-UM to the J-UM grid. In this manner, every time step the S-UM prognostic fields overwrite the J-UM prognostic fields ensuring the evolving J-UM climate follows that of S-UM. After exchange of variables from S-UM to J-UM the UM parameterizations are also run in model J-UM providing the dynamical core fields on the N96 grid required for tracer advection as well as the single-time step updated N96 prognostic fields required by UKCA. Parameterized fields required by UKCA within J-UM are now available on the J-UM grid and the full UKCA model is run including tracer advection. At the end of every UKCA time step (1 hour) 3D trace gas and aerosol fields required by the UM cloud and radiation parameterizations are passed back from J-UM and interpolated onto the N216 S-UM grid, before being used. In this manner UKCA in J-UM is driven by a physical climate evolving on the N216 grid of S-UM, while the resulting N96 UKCA fields in turn impact the evolution of the N216 S-UM climate.

Figure 1 provides a schematic of the S-UM/J-UM model configuration. Once an atmosphere-only configuration of this system is performing acceptably, the NEMO+MEDUSA ocean model will be coupled to the S-UM model, leading to a coupled hybrid resolution ESM, UKESM1-hyb. Figure 2 provides a schematic representation of the variable exchange between S-UM and J-UM, with S-UM prognostic climate variables passed to J-UM every S-UM time step and J-UM UKCA fields passed back to S-UM every hour, in advance of the S-UM radiation call.

Figure 2

Figure 2. Schematic of the time step exchange of variables from Senior UM (S-UM) to Junior UM (J-UM).

Variables are passed every S-UM time step to JUM, while J-UM to S-UM UKCA fields are passed every hour prior to the SUM radiation time step.

Work is ongoing to fully develop this system, in particular to ensure the evolving physical climate fields in S-UM and J-UM are equivalent and that the UKCA fields once passed back to S-UM are sufficiently similar to those in J-UM. Once this hybrids resolution approach has been tested we will begin comparing it to full UM+UKCA simulations at both N216 and N96 to assess its scientific validity.

In the full N216 UM with UKCA active, ~80% of the computational time is spent in UKCA. Due to the significantly reduced resolution of J-UM, the combination of S-UM (UM only at N216) and J-UM (UM and UKCA at N96) runs at a fraction of the cost (order ~1/3) of a single UM+UKCA job at N216. Furthermore, S-UM and J-UM are separate executables allowing increased process concurrency and an increased opportunity for load balancing on parallel HPC systems. We aim to have a version of UKESM1-hyb technically functioning by the summer of 2016 and a scientifically functioning and evaluated version available by the second half of 2017. Parallel to this effort, a similar facility is being developed by the Paris NEMO system team for the advection of ocean biogeochemical tracers within NEMO, using degraded resolution dynamical fields. Once this system is available we plan to compare ocean-only and coupled simulations of UKESM1-hyb using full and degraded resolution advection applied to the MEDUSA biogeochemical tracers. As with UKCA this modification will significantly reduce the cost of NEMO-MEDUSA within UKESM1-hyb.

Download this article as a PDF (57KB)