Using Satellite Images, Scholars Develop a Model to Quantify Buildings’ Energy Use

In the face of climate change, reducing buildings’ energy consumption is critical, says Thomas Dougherty, a graduate student in civil and environmental engineering at Stanford University. By some estimates, buildings account for about 40% of the energy used in the United States. Reducing that usage in a significant way will require a comprehensive, holistic approach. 

Currently, researchers model individual buildings’ energy profiles without considering their surroundings. But that building-by-building approach doesn’t scale well. It also ignores the importance of context. 

“Buildings use energy in a holistic world that involves the interactions of not just the systems and people within the building, but also interactions with exterior systems like the streets and the trees outside,” Dougherty says.

For his PhD research, Dougherty decided to flip the building-by-building approach on its head. He would instead model buildings’ energy use from a distance – using computer vision analysis of a ubiquitous data type: satellite imagery. The resulting model, which he calls SCHMEAR, represents the first time researchers have modeled and quantified how much the context around a building contributes to its energy profile.

SCHMEAR provides a new tool to help city planners understand the array of solutions they might pursue as they begin redesigning cities for a low-carbon world, says Rishee Jain, assistant professor of civil and environmental engineering, an affiliated faculty member of the Stanford Institute for Human-Centered Artificial Intelligence, and co-author of the SCHMEAR paper. 

“This approach will be useful for developing large-scale urban interventions and helping urban planners decide what the texture of our cities should look like 50 years from now,” he says. And because satellite imagery is readily available for any city in the world, SCHMEAR enables this kind of analysis for cities from Terre Haute, Indiana, to Mumbai, India. “We should be able to scale this to any city anywhere,” Jain says.

Why SCHMEAR?

Until now, researchers who model urban buildings’ energy consumption used one of two approaches: They would either simulate buildings’ structural features with high granularity using detailed information about the structure and its systems, or they could build a data-driven model that relies only on basic building information such as the facility’s age and material. 

Although the simulation approach can yield useful information, it is not practical for widespread modeling at scale. And producing reliable models using the data-driven approach is challenging because different cities collect different kinds of data about their buildings. “If every city is capturing things slightly differently, it gets quite hard to create a consistent model,” Dougherty says.

Granular simulations of individual buildings also lack context. Researchers know that cities often contain “heat islands” that absorb heat and reflect it back on surrounding buildings and pedestrians. And big buildings can induce urban canyon effects and other physical phenomena that affect air flow and therefore impact buildings’ energy use. “The fact that a building is in a dynamic environment may change its energy use quite a bit,” Dougherty says.

SCHMEAR directly addresses the limitations of existing approaches. It relies on high-quality satellite photos, which are routinely available for buildings all around the world. This should allow Dougherty and his colleagues to conduct the same analysis, using the same model, for any city in the world, while still capturing the importance of buildings’ context. 

“We can curate satellite images for new cities that don’t have any data about their buildings, or the datasets aren’t in a common format or in a predictable location,” Dougherty says. “It really just opens doors for more analysis of a diverse array of cities’ energy profiles.”

A New Tool in the Urban Decarbonization Toolbox

The team created several models of New York City buildings’ energy use to see which would best predict the buildings’ known energy consumption profiles. One was a SCHMEAR model created using computer vision to analyze satellite photos. For comparison they also built a data-driven model that relied on a set of available data about each building, such as the number of floors, the age of the building, and its floor area.

The result: A SCHMEAR model based on a single close-up satellite photo provided as much useful information for predicting a building’s energy consumption as the data-driven model built from basic curated building data.

Understanding SCHMEAR’s Predictions

SCHMEAR’s computer vision analysis of satellite imagery relies on a convolutional neural network – a type of AI that is often considered a black box. This means it can be hard to determine what features of a building contribute to the model’s predictions. Nevertheless, if models of this type are going to assist cities in developing decarbonization plans, they will have to be interpretable, Jain says.

Dougherty has already taken a first stab at trying to extract explanations for SCHMEAR’s predictions. In particular, he compared the importance of the immediate region around the building to the region that’s farther from the building. Using a method called saliency mapping, he showed that in Manhattan, the region around a building is a much more significant predictor of energy use than it is in a place like Queens, where the buildings are more spaced out. “We essentially validated the idea that urban context is a significant topic that we should be thinking about – particularly in the densest parts of the city,” he says.

In addition, the team analyzed situations where the SCHMEAR system yielded poor predictions and compared that to the data-driven model’s descriptive statistics about each building. This approach showed that the SCHMEAR model was not as good as the data-driven model at taking building height into consideration. This strategy constitutes both a step toward improving the model and a means of understanding it.

Going forward, Dougherty is interested in detangling what is driving SCHMEAR’s electricity prediction by adding different kinds of data. While the SCHMEAR project used only high-resolution images, the next iteration might use satellite images made with different frequencies of light, or building outlines and radiuses. With data that is segregated by type, it’s easier to see how a particular feature contributes or doesn’t contribute to the prediction of energy consumption.

The Climate Change Opportunity

The need to decarbonize buildings and cities is becoming more urgent as climate change threats grow. Policymakers might consider a range of interventions – some that apply to individual buildings and others that apply to the city as a whole. For example, they might offer building owners free trees to reduce the urban heat island effect or they might consider adding a park to a neighborhood, changing the layout of streets, or altering the types of roofing materials permitted in new construction. Even reducing traffic in certain areas temporarily or permanently might help reduce buildings’ energy use.

 Some cities are already starting to take steps in this direction. In 2019, for example, New York City’s city council passed Local Law 97, which set limits for emissions from large buildings. As the New York Times recently reported, building owners are already struggling to understand how to comply.

To Jain, the need for planning represents an opportunity for modelers. “When you’re trying to reach these really ambitious goals, you’re going to need to think about the larger scale issues rather than small-scale, one-building-at-a-time options such as changing out the windows or adding insulation,” he says. Going forward, he says, you can start to imagine cities incorporating models like SCHMEAR into their policymaking and planning decisions. 

For example, Jain says, in San Francisco, where there are more than 100,000 buildings, city officials might want to provide subsidies for a limited number of onsite energy audits. 

“A tool like SCHMEAR can help them decide which buildings merit that kind of granular analysis,” Jain says. “It’s not about replacing the very detailed models. It’s about trying to understand where you want to use your resources.” 

Stanford HAI's mission is to advance AI research, education, policy, and practice to improve the human condition. Learn more.