Guidelines for evaluating energy efficiency policies and outcomes

22 Jun 2018

By Shruti Vaidyanathan, Senior Advisor for Research, ACEEE

Energy efficiency policies and programmes have been discussed as a key strategy for countries grappling with meeting ambitious greenhouse gas reduction targets set in place by the Paris Agreement. According to the International Energy Agency (IEA), energy efficiency will need to account for almost half of all the greenhouse gas emission reductions necessary through 2040 to limit the global increase in temperature to 2 degrees Celsius. Energy efficiency is an essential tool for addressing growing global energy consumption that also helps to reduce dependency on energy imports encourage development and job creation, and protect public health.

As efficiency becomes more integral to meeting nation-wide goals, countries will need to compare how they stack up to their peers as well as understand how they can further improve energy efficiency in their respective countries. Benchmarking efforts are the most effective way to create an overall picture of energy use and efficiency efforts. A number of organisations including the World Bank and the European Union have created tools to help evaluate national progress on energy efficiency (see box). Benchmarking energy use and efficiency policies can be challenging for a number of reasons, however, and settling on a single replicable methodology is not always an easy undertaking.

The American Council for an Energy-Efficient Economy (ACEEE) has a long history of tracking and evaluating progress on energy efficiency across multiple sectors at the state, local, and national levels. In 2012, we added an International Energy Efficiency Scorecard to our existing suite of ranking products in order to provide a birds-eye view of energy use and efficiency policies in major global economies. Additionally, the analysis showcases best practice policies and programmes that countries can pull from to create an energy efficiency road map of sorts. This year’s Scorecard evaluates 25 nations on 36 different metrics and produces some very interesting results.  Our experience with benchmarking international energy efficiency efforts and performance has highlighted a number of important factors to consider. We discuss a few of those lessons below.

Transparency is Important

There is not one single correct approach to international benchmarking. The chosen methodology will depend heavily on the audience you are addressing, the stated goals of the effort, and the available data. Additionally, some audiences will place more value on energy outcome metrics while others want to focus on how countries compare on specific policy actions.

For the International Scorecard, ACEEE settled on a methodology that combines policy and performance metrics (in a 59/41 point split) that we think have the most potential to reduce energy consumption and are palatable at the national level. Additionally, since our goal was to highlight current leaders, we award full points to at least one country for each metric in the report.

Given the subjective nature of the process, the key to benchmarking energy efficiency is to clearly describe the justification for and the limitations of a specific methodology. Readers should be able to understand what the chosen approach aims to measure and exactly why a country was awarded a particular score.

Energy use is impacted by factors other than efficiency

It is also difficult to find a methodology that adequately captures energy efficiency efforts and allows for comparison across a range of countries. The primary issue at play here is that energy use is impacted by factors besides energy efficiency. Physical factors such as geography, climate, elevation, and availability of natural resources affect the amount of energy a country uses across various sectors. Economic structure also governs energy use. Agriculture- and labour-based economies tend to have lower energy consumption than industrialised ones. Finally, demographic composition and population density also affect overall energy consumption, as do other social factors such as income levels and energy inequity.

It is incredibly difficult to control for these factors when evaluating trends in energy use and the impact of energy efficiency policies. The ODYSSEY-MURE database methodology is probably the most advanced in this regard, as it attempts to estimate efficiency outcomes by correcting for non-energy conditions. Nevertheless, it would be difficult to apply their methodology to data from non-European countries due to a lack of data. ACEEE’s International Energy Efficiency Scorecard adjusts for a few of the most important factors (i.e. impact of climate on energy use for space conditioning and impact of industry mix on industrial energy use) but otherwise we do not account for these factors in our evaluation. Since our primary goal is to provide a basic overview of energy use and policies across the 25 countries, we only made a few adjustments to raw data to enable basic comparisons. Once again, transparency about what is or isn’t being measured is key. 

You can only measure what you have data for

Tracking down energy data for certain countries is still very much a challenge for evaluators due to a lack of a data collection frameworks or protocols in many places.  Additionally, benchmarking requires data that is consistent across countries in order to rate and rank them fairly on their efficiency efforts. More often than not, this is also the case for data that may be used to correct for the impact of conditions extraneous to energy efficiency. As a result, metrics may have to be crafted to accommodate data available for use. The 2018 International Energy Efficiency Scorecard chooses metrics for each section of analysis based largely on information that we are able to find in centralised, freely accessible databases from international organisations. We supplement with on-the-ground data from country experts, which though incredibly valuable, can be inconsistent in terms of defining what is being measured.   

Energy efficiency has increasingly become an important tool in the race to curtail global greenhouse gas emissions. As countries strive to meet their ambitious GHG reduction goals, benchmarking efforts can help them understand the baseline level of energy efficiency as well as show them the best practice policies and programmes that can harness untapped efficiency potential.                                                                                                                                           

Shruti Vaidyanathan leads ACEEE's international work, and as the Senior Advisor for Research, she helps coordinate research efforts throughout the organisation. Additionally, she has 10 years’ experience in transportation efficiency issues, focused on improving mobility at the state and local levels and on evaluating the life-cycle emissions of vehicles.