IBM and NASA are building an AI foundation model for weather and climate

Foundation models are fast becoming indispensable tools for writing code, translating languages, and summarizing virtually any text, no matter how complex. Predicting the weather and climate could be next.

Fed enough raw data, foundation models can piece together the underlying structure of complex systems, whether that’s code, natural language, or molecules. From this general base of knowledge, a single foundation model can perform many tasks with additional training and task-specific examples. Their power lies in their ability, when prompted, to generate predictions or original content with limited instructions from humans.

The essential ingredient is data, which is something NASA has in abundance. To make NASA’s vast and growing data archive more accessible, IBM and NASA set out a year ago to build an open-source geospatial foundation model. Now available on Hugging Face, the model can help scientists estimate the extent of past floods and wildfires. IBM is also using the model to help map urban heat islands in the UAE and track reforestation in Kenya.

Encouraged by these results, IBM and NASA decided to branch out. A new foundation model aimed at making weather and climate applications faster, more accurate, and more accessible is now in the works. Other potential applications include helping climate experts infer high-resolution information from low-res data, identify conditions conducive to wildfires, and predict hurricanes, droughts, and other extreme events.

In September, IBM and NASA hosted a workshop to discuss their proposed roadmap for building a weather and climate foundation model. When finished, the model will be made open source and publicly available.

The renaissance in weather forecasting

Weather prediction has improved dramatically in recent decades. Today’s six-day forecast is as accurate as a five-day forecast 10 years ago. Hurricane tracks can be predicted more accurately three days out than they were 24 hours in advance 40 years ago.

This remarkable achievement is due to two things: decades of advances in atmosphere and ocean science and parallel progress in high-performance computing. Modern weather models base their predictions on massive computer simulations that take time and energy to run. That’s because they factor in both physics-based equations and weather observations, from winds and air pressure to temperature and precipitation.

But now, another revolution is underway. In the last year, a new way of forecasting the weather has emerged. The European Centre for Medium-Range Weather Forecasting (ECMWF) has started using several deep-learning models called AI emulators that generate forecasts based on historical weather patterns; the laws of physics are not explicitly encoded in AI emulators, but they can be inferred from the data. This simplicity means that a forecast can be dashed off on a desktop computer in minutes instead of the hours it can take an HPC system.

Google Deep Mind recently reported that its GraphCast emulator could provide a faster, more accurate 10-day forecast than current traditional models. In September, GraphCast accurately predicted that Hurricane Lee would make landfall in Nova Scotia nine days in advance — three days earlier than current models.

Technically, AI emulators are not foundation models. They were trained to perform one task, on one dataset, and were given explicit forecasting instructions. But they are the precursors to a general-purpose foundation model and hint at the benefits to come.

A multimodal foundation model for weather and climate

Foundation models have several advantages that come from their ability to process and analyze raw data of many types, allowing them to form a broad representation of the data that can be generalized to many scenarios. It’s an important capability in a field like climate, where conditions are constantly changing through time and space, and many downstream applications exist beyond forecasting.

Foundation models can take tens of thousands of GPU hours to train. But at inference time, they can be run in minutes to seconds. Currently, not many researchers have access to HPC computing resources to run traditional weather models. The emergence of pre-trained AI models means that the field of weather and climate modeling has effectively been democratized. This brings the potential for accelerated discovery.

Foundation models could also improve the accuracy of forecasting for other climate applications. Earth’s climate is changing rapidly and disrupting weather patterns globally. For logistics, businesses, and government agencies, the earlier that a pending disaster can be detected, the greater the chance that lives and money can be saved.

Inferring atmospheric dynamics from the data

To be foundational, a foundation model can’t be a one-trick pony. It should be capable of performing many tasks, and ideally, trained on many types of data. This is especially important in weather and climate prediction since many physical processes can often only be observed over certain timeframes and spatial scales. The cyclic El Niño weather pattern, for example, plays out over many months and across half the globe, while tornado initiation can take minutes and arise from processes at the sub-meter scale.

Sensors provide a continuous, highly localized record of changing temperatures, winds, and pressure. Satellite images, by contrast, capture environmental changes at longer intervals and lower resolution.

IBM and NASA's proposed foundation model will initially be trained on the MERRA-2 dataset, a combination of high-quality observations and estimates of past weather over the last 40 years. Observational data from fixed weather stations, floating weather balloons, and planet-orbiting satellites will be added later. IBM and NASA are currently experimenting with model architectures and techniques to integrate these varying time and spatial scales into one multimodal model.

Other challenges

Current AI models often miss extreme events. This tendency is a known problem for AI models which are trained to ignore outliers. Loss functions minimize the chance of making big mistakes, but consequently they can also miss extreme events. Methods to correct this tendency have been implemented in smaller models. IBM and NASA's challenge will be to extend this work to large foundation models.

Another issue is climate change itself. The past is not always a great predictor of the future, especially when the climate is warming as rapidly as it is today. A hurricane in 2024, for example, may have higher wind speeds than a hurricane in 1933. As a result, forecasters may not see it coming if their models are based exclusively on historical data. AI, however, makes it possible to continuously update models as circumstances evolve and new data becomes available.

What’s next

IBM and NASA’s goal is to create a multimodal AI foundation model for weather and climate prediction that can be adapted to many downstream tasks with relatively few GPUs. AI experts at IBM will work closely with climate scientists and other domain experts at NASA to test and validate the model on seven applications, including 10-14 day weather forecasts and things like dust storms and aviation turbulence.

Once trained, the model will be made openly available on Hugging Face, making weather and climate modeling much more accessible to the global research community. This work is part of a larger effort by IBM and NASA to develop foundation models that can answer some of the most pressing questions about our changing climate and environment.