An Earth system model contains numerous scientific components. Some, such as the core dynamics and moist parameterizations, are known to benefit from higher model resolution (e.g. Jung et al, 2012, Tao and Chern, 2017). Others, such as aerosols and chemistry, are computationally expensive and there is less evidence they benefit from increased model resolution.  In the UKESM hybrid model, there are two atmospheres: one at a higher resolution, for science which either benefits from high resolution or is computational cheap, which we call Senior (Snr); and one at a lower resolution which contains everything, and is referred to as Junior (Jnr). Where there is science overlap, Snr should take precedence over Jnr and we try to lock the temporal evolution of Jnr to follow that of Snr as much as possible, without greatly delaying either model. Any input required in Snr that is not directly simulated in Snr is provided to Snr from Jnr.

The science currently absent from Snr is the calculation of aerosol and chemistry processes, plus the transport and advection of the considerable number of associated 3D fields. In the first UKESM newsletter (https://ukesm.ac.uk/portfolio-item/reduced-resolution-chemistry-aerosols-ukesm1/), when we introduced the concept of running the aerosol and chemistry schemes at a reduced resolution, they slowed the model atmosphere by a factor of around five (i.e. 400% slower). Since then, the computational performance of the aerosol and chemistry has improved greatly, largely due to the introduction of OpenMP into both schemes, work carried out primarily by the UKESM team. Inclusion of aerosols and chemistry now only slows the model atmosphere by a factor of around three (i.e. 200% slower) – a significant improvement but still an important slow down.

Figure 1 shows a schematic of UKESM-hybrid, and shows the three main executables, a Snr Unified Model atmosphere (UM), a Jnr UM and an ocean. The latter consisting of the physical ocean (NEMO-ORCA) and ocean biogeochemistry (MEDUSA). There is an additional fourth executable for an ocean I/O server (XIOS), which is not shown in the figure. The aerosols and the chemistry in the UM are calculated by the UK Chemistry and Aerosol code (UKCA). Crossing out UKCA in the Snr image in figure 1 indicates UKCA is turned off in Snr. The arrows show the direction of coupling and indicate that coupling happens between all three models. The only exchange currently not performed is from Jnr to the ocean, since Snr is used to drive the ocean model. In the earlier newsletter article on UKESM-hybrid the atmosphere was not coupled to the ocean. We recently started working on this coupling as shown in figure 1. This is a new coupling and currently only runs for a few months before Jnr develops an instability and the model fails. We are presently working to understand and remedy this problem.

Thus far the hybrid configuration has consisted of a N216 Snr UM, a N96 Jnr UM and, when an ocean is included, an ORCA025 (0.25° resolution) ocean. This is likely to be the only N216-based UKESM configuration that will be computationally affordable for full (multi-century) CMIP experiments. We are currently also developing a hybrid model with a N96 Snr, a N48 Jnr and an ORCA1 (1°) ocean which should be a faster, but scientifically traceable, alternative to UKESM1 N96 ORCA1.

Figure 1: Schematic of UKESM1-hybrid. The Snr UM contains all the UKESM1 science except aerosols and chemistry. Input that Snr requires from UKCA are provided by the Jnr UM. The Jnr UM contains all the UKESM1 science, and is locked to Snr by passing dynamical core fields from Snr. The ocean, which consists of NEMO-ORCA and MEDUSA (biogeochemistry), passes its simulation results to both Snr and Jnr, while it only receives atmospheric forcing from Snr. All coupling is performed by OASIS3-MCT every model hour.

Coupling between all the models components is done through the OASIS3-MCT coupler on the hour, directly after most of the aerosol and chemistry science is calculated in Jnr. This means the 48 3D UKCA fields which Snr requires from Jnr are available for coupling at the earliest opportunity. These fields are re-gridded from the Jnr grid to the higher resolution Snr grid using the coupler. To lock the evolution of the physical atmosphere in Jnr to follow Snr, a number of dynamical core fields are passed from Snr to Jnr, with re-gridding also performed by the coupler. These fields overwrite the same native variables in Jnr. Determining which are the best dynamical core fields to use for the locking is still being evaluated. The green arrow in figure 1 shows locking being done using the horizontal velocity components and dry potential temperature. It is likely these are the minimum required. Additional fields to further constrain Jnr to follow Snr, such as surface and soil variables, are likely also necessary. More work is needed to identify the optimal set.

Performing this locking every timestep would improve how well Jnr is locked to Snr. However, the final timestep of every hour in Jnr is computationally long, as this is when the majority of the aerosol and chemistry science is carried out. The discrepancy in the relative duration of timesteps between those in Snr and those in Jnr means locking at every timestep causes Snr and Jnr to have to wait for one another, slowing the hybrid model. Hence, coupling from Snr to Jnr is only done once every hour. Ocean coupling in a standard UM-NEMO configuration already happens on the hour. The one special thing we do is to simultaneously send output from the ocean to both Snr at N216 and Jnr at N96, where the re-gridding is again performed by the coupler.

Until we can successfully couple the ocean for a long run, our evaluation of the hybrid model is limited to atmosphere only configurations. Jones et al. (2018) found that the hybrid model, with Snr at N216 and Jnr at N96 compared very well against a UM model with all components integrated at N216.  That assessment compared an atmosphere with full stratospheric chemistry configuration (known as GA+StratTrop) at both N216 and N96 resolutions with a hybrid configuration with Snr at N216 and Jnr at N96. Figure 2 is typical of the results. It shows annual mean TOA net shortwave solar radiation (SWD), with positive values indicating downward directed flux. The left column shows the absolute SWD values for N216 (top), the hybrid model (middle) and N96 (bottom). On the right is the bias in the annual mean SWD relative to CERES satellite observations (Loeb et al. 2009).  The absolute fields on the left column are very similar for all three models. However, looking at the biases against CERES observations, it is evident that the hybrid-simulated fluxes are more similar to those in the N216 model than the N96 model. This is particularly evident from the RMS difference between the models and CERES. For SWD the hybrid model has a RMS value of 8.03, marginally smaller than the RMS value of the N216 model, which is 8.21. These are both significantly smaller than the N96 model, which has a RMS difference of 10.0. This result is mirrored for individual seasons and most other variables considered.  It was surprising to find the hybrid model apparently outperforming the N216 model, if only slightly.

Figure 2. Absolute annual mean TOA net shortwave radiation (left column) and bias in TOA SWD against CERES observations (right column) for N216 (top row), hybrid (middle row) and N96 (bottom row).

Degrading the aerosol and chemistry resolution, as we do for the hybrid, is only acceptable if there is a large performance gain. In tables 1 and 2 the hybrid model is compared against the equivalent N216 resolution models. All runs were only for one month – hence the speeds are very approximate – and were done on the Met Office XCS which has 36 cores per node. They show the hybrid model giving between a 45-68% speed-up. We believe there is more science which can fairly easily be moved out of Snr which might increase this up to about 80%.

	On 60 nodes	Top speed on 2 threads
UKESM AMIP N216	1.1 model years/day	2.2 model years/day
UKESM-hybrid AMIP N216 N96	1.8 model years/day	3.7 model years/day
Speed-up of hybrid	64%	68%

Table 1. Approximate speeds of atmosphere only configurations comparing UKESM-AMIP N216 with UKESM-hybrid N216 N96. The hybrid model is listed with two resolutions where the first is the resolution of Snr and the second is the resolution of Jnr.

 

	On 60 nodes for atmosphere	Top speed on 2 threads
UKESM N216 ORCA025 without MEDUSA	1.1 model years/day	1.7 model years/day
UKESM-hybrid N216 N96 ORCA025 without MEDUSA	1.6 model years/day	2.8 model years/day
Speed-up of hybrid	45%	65%

Table 2.Comparing the approximate speeds of UKESM N216 ORCAO25 without the ocean biogeochemistry (MEDUSA) against UKESM-hybrid N216 N96 ORCA025 without MEDUSA. For the 60 node atmosphere jobs, UKESM N216 ORCA025 used a total of 85 nodes, whereas UKESM-hybrid N216 N96 ORCA025 used a total of 95 nodes. Note also that the 2 (OpenMP) threads for the top speed refer only to the atmosphere as NEMO can currently only be run on one OpenMP thread.

It should be noted in table 2 that the ocean biogeochemistry (MEDUSA) has been removed from the configuration. This is because MEDUSA slows the ocean by a factor of about three (200% slower), and the top speeds of any N216 ORCA025 configuration is bound to be limited by how fast the ocean can be run. Hence, increasing the speed of any N216 ORCA025 configuration requires increasing the speed of the ocean, and we have begun the process of applying a similar hybrid approach to the ocean biogeochemistry. This will not be a problem for UKESM-hybrid N96 N48 ORCA1, because our ORCA1 ocean is considerably faster than our N96 atmosphere.

References

• Jones, C., M. Stringer, R. Hill, C. Johnson, S. Rumbold, J. Walton, M. Dalvi and A. Sellar (2018). HCCP deliverable C1.3. A scientifically tuned version of the atmosphere-land configuration of UKESM1-hr with UKCA run interactively at degraded resolution analysed and compared to UKESM1-hr.
• Jung, T et al. 2012: High-Resolution Global Climate Simulations with the ECMWF Model in Project Athena: Experimental Design, Model Climate, and Seasonal Forecast Skill. Climate, 25, 3155–3172
• Loeb, N.G. et al. 2009: Toward Optimal Closure of the Earth’s Top-of-Atmosphere Radiation Budget. Climate, 22, 748–766
• Tao, W.‐ and J.‐D. Chern 2017: The impact of simulated mesoscale convective systems on global precipitation: A multiscale modeling study, J. Adv. Model. Earth Syst., 9, 790–809