ACME—scaling the heights of Earth system modeling

May 18, 2016

The Accelerated Climate Modeling for Energy (ACME) project, now in its second year, has already racked up numerous awards, most recently “best video” at Supercomputing 2015 in Austin, Texas. More important, the team, which consists of eight national laboratories, the National Center for Atmospheric Research, four academic institutions, and one private-sector company, is on schedule for release of Version 1 of the ACME model in June 2016.

The major activity this past year was completion of Version 1 of the model, based on the Community Earth System Model or CESM. The team has been running tests on the model since late last year. After the release in June, the team will start a series of science experiments, in the works for 2 years, that will run from July 2016 through July 2017.

The CCSI ACME team—which includes Peter Thornton, ACME council member and Land Model Task Team co-lead; Valentine Anantharaj, Workflow Task Team co-lead; and Patrick Worley, Performance Task Team co-lead—has been concentrating on land model development, atmospheric model development, and software engineering aspects of the project such as workflow and performance analysis.

Land model development

On the land model side, the most significant advance made this past year was integrating the phosphorus cycle. In the past, Thornton points out, CESM was unique in the community of global ESMs for including both a nitrogen cycle and a carbon cycle. No Earth system model to date, however, has included phosphorus. ACME Version 1 will include carbon, nitrogen, and phosphorus cycles, based on work by Xiaojuan Yang, a CCSI early career scientist. “This is world class new science that no other coupled model has, and it will have an important impact on predictions of future climate,” says Thornton. “We’re really excited about that.”

Another innovation the team has introduced is what’s called “reactive transport modeling capability.” This is a new, more sophisticated way to solve the coupled hydrology-temperature (freeze-thaw) dynamics and biogeochemistry reactions in the soil and between the soil and vegetation. Thornton says the approach more consistently integrates the mass, energy, and biological components of the model, including the phosphorus cycle work.

Uncertainty Quantification

Daniel M. Ricciuto from CCSI’s Integrative Ecosystem Science group (formerly known as the Terrestrial Ecosystem and Carbon Cycle Science group) works on the uncertainty quantification (UQ) component of the ACME project. Thornton says it’s a formal framework to estimate the uncertainty associated with some of the model parameters to optimize model predictions. That same framework is used to estimate parameter values for the model to optimize the metrics of model performance against other independent observations such as those from satellites. Tuning parameters in the model that are uncertain based on actual measurements will facilitate model optimization.

Atmospheric model development

“Snapshot of potential vorticity contours in the upper troposphere (200 mb level) from an ACME present-day prerelease simulation. Potential vorticity measures the fine-grained dynamical behavior of the large scale extra-tropical atmosphere and can be used to track the connection  between large scale flow and surface temperature extremes.”

Achievements by the ACME Atmospheric Model team have been no less impressive, but perhaps the most exciting have been in the area of high-resolution simulation. Most global models today don’t represent the impacts of climate change on water resources very well in mountainous regions or other regions with complex terrain. To address this, the team has developed a topography-based subgrid system based on high-resolution global elevation data from various sources. This results in a greater number of subgrid units over regions of complex terrain, such as mountains, leading to better representation of precipitation and surface water flow in such regions. For consistency, the same high-resolution subgrid system is also being used for the ACME Land Model.

Performance—to the exascale and, perhaps, beyond

Pat Worley is leading the performance evaluation effort, both for ORNL aspects of the project and for the project as a whole. Up to this point, that’s mainly been a matter of ensuring all the subcomponents of the system run as quickly as possible on Titan and other leadership computing resources and then ensuring that when they are coupled they continue to run quickly. The performance target they are aiming for is 5 simulation years per day for the highest resolution simulations.

One of the goals of the ACME project is to optimize model performance for future leadership computing architectures, and Thornton says that the Performance Task Team is already preparing for the next generation of computers, due in 2017 (the “mid-machines” before exascale such as ORNL’s Summit). As part of this preparation, the team submitted a successful proposal to the Oak Ridge Leadership Computing Facility’s Center for Accelerated Application Readiness or CAAR program. Through CAAR, the ACME team will gain early access to Summit’s hybrid CPU–GPU architecture and technical support for software development. And of course, Worley and the team are already looking at model development and needs for the exascale model.


Workflow is the final piece being worked on in CCSI. The goal is to have a robust system that domain scientists can use efficiently and effectively without a lot of tedious work up front. The workflow tool under construction is really a set of tools operating under a web-based user interface that will be in place when Version 1 is rolled out in June—available to the whole world. Not having a workflow tool such as the one being designed restricts the science that can be done and the use of the model to that very small population of computer programmers/modelers/software engineers who are also domain scientists. “So by looking at workflow, we really hope to open it up to a larger community,” Thornton says.


Communication and coordination are challenging elements in the success of any project, but as Thornton points out, communicating and coordinating across institutions is a huge challenge.

To make the communication-coordination task more manageable, the project team has taken advantage of online tools that facilitate collaboration and communication “At first there was skepticism about how effective they would be, but there has been across-the-board buy-in and recognition that this has helped us to achieve our goals.”

Thornton has high praise for Dave Bader (Lawrence Livermore National Laboratory), project lead (council chair) and principal investigator, for leading this effort. In addition to the investment in software, he says, Renata McCoy from Bader’s staff works full time to maintain the turnkey communication tools they have licensed (JIRA and Confluence, both from Atlassian). A lot of credit goes to her, he says, but also to the software tools and the team.

ACME is funded by the Earth System Modeling program within the US Department of Energy Office of Science, Biological and Environmental Research, and is led by Lawrence Livermore National Laboratory. For more information on the project, go to the project website at For more information on ORNL CCSI ACME work, contact Peter Thornton.

By VJ Ewing