Tuesday, December 8th, 2020
1:00 PM Pacific Time
Who is it for?
Utilities that are struggling with planning for exogenous impacts on energy consumption.
COVID-19 has upended the energy efficiency and demand flexibility industry’s assumptions about energy use and rendered traditional Measurement and Verification (M&V) all but useless. Longer-term trends of decarbonization, electrification, and increased renewable, intermittent generation and other exogenous factors create similar challenges to understanding and valuing flexibility interventions.
In this webinar, Recurve CEO Matt Golden and Director of Policy & Emerging Markets Carmen Best explain how the new GRIDmeter project successfully used privacy protected utility data to identify comparison groups in order to differentiate the way in which COVID impacted different kinds of commercial and residential energy consumers -- and how utilities can apply these insights immediately to understand the impacts of COVID and other exogenous factors on their own programs.
Alice Havenar-Daughton of MCE will be joining the discussion to talk about the value of this analysis to the evolution of their energy programs.
Six months ago, with support from the Department of Energy and help from MCE and other partners, Recurve initiated a working group to develop new open source Measurement and Verification (M&V) tools for an era in which energy consumption is defined by uncertainty.
These methods and open-source code was developed based upon funding from the U.S. Department of Energy, through the National Renewable Energy Laboratory. Recurve would also like to thank MCE for supporting this project by providing secure access to data.
Finally, Recurve thanks the members of the Comparison Groups Working Group, who devoted their time and effort to listening, reviewing, and providing feedback throughout the research and development of the methods and recommendations in this report.
Comparison groups can create uncertainty for multiple reasons:
Portfolio and comparison group sizing matter. There is a very predictable uncertainty reduction as group sizes increase.
what we now know
Program tracking with annual savings calculations and portfolios of 3,000+ heterogeneous buildings. Random sample of 3,000+ non-participant buildings from eligible population
Program tracking with annual savings calculation with portfolios of 1,000+ relatively homogeneous buildings. Random sample of 3,000+ non-participants from targeted population
Program tracking with marginal hourly savings calculations with smaller numbers of homogeneous buildings. Stratified sample of 3,000+ non participants from targeted population
Savings reconciliation after enrollment has closed and full treatment population is known, especially needed where program enrollments are substantially different than comparison pool. Stratified sample that maximizes sample size relative to error between treated and non-treated groups
●Would a comparison group that matches the baseline usage pattern of a treatment group start to exhibit bias over time that would negate some of the savings of the treatment group?
●We know that there might be self-selection issues that might need to be addressed during the selection of the comparison group
●But we don’t know what the variability of the bias might be and we don’t know whether or not it might get worse under conditions of large-scale energy changes, such as during COVID
●Simulate the selection of treatment and comparison groups for a program
●Target a subset of high peak load users, similar to what might done for a program that is focused on peak load reduction
●Select a variety of sizes of “treatment” and “comparison” groups from the larger targeted population that can be compared against each other
pure doppelganger vs two smallest groups vs two largest groups
Bias is inevitable in comparison groups due to factors beyond our ability to control for
2) Larger groups will reduce bias, but treatment group may still diverge from comparison group over time
3) Need for more systematic approach to determine optimal comparison and treatment group sizes
Stratification is about binning
●Equal distance or population?
●How many bins?
●How many parameters?
●In what order of priority?
●Should there be an order?
●How do you gauge success?
Stratified Sampling: What are the Constraints?
The only option in resampling is the elimination of meters.
Two main constraints:
1.The size of the comparison pool
2.The number of meters needed in the final sample
Standardizing Stratification: Dealing with Multiple Parameters
Multiple parameters can generate a more representative sample
●Complexity: Stratifying the first parameter and then moving on to the next will inherently change the first
○Solution = Simultaneous 2D or nD binning
●How should the individual parameters be prioritized and binned?
○Solution = An optimal scheme should be determined via algorithm.
●How do you gauge success?
Video of the August 28, 2020 Meeting: Comparison WG VIDEO - 2020-08-28
Slides from the August 28, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-08-28
Chat Record from the August 28, 2020 Meeting: Comparison WG Chat - 2020-08-28
Without a Comparison Group Why is COVID A Problem?
Residential Sector COVID Impacts:
Blue line is CalTRACK Hourly Counterfactual. Orange line is observed usage.
Without a comparison group to account for COVID, the increase in consumption wipes out program savings.
Chart Shows Analysis from March 19 - May 8; 7.2% Increase in consumption due to COVID
Diff-of-Diff: A (Slightly) Deeper Dive
The “Difference of Differences” Calculation
(Counterfactual_Treatment - Observed_Treatment) - (Counterfactual_Comparison - Observed_Comparison)
Video of the August 14, 2020 Meeting: Comparison WG VIDEO - 2020-08-14
Slides from the August 14, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-08-14
Chat Record from the August 14, 2020 Meeting: Comparison WG Chat - 2020-08-14
Stratifying a sample is done based on binning: Bins are defined based on the treatment group. The relative number of customers between treatment and comparison groups needs to match in every bin.
For most comparison pools, stratify on up to 3 parameters
Examples of normalized electric features
% Heating kWh
% Baseload kWh
% Summer Peak kWh
Video of the July 31, 2020 Meeting: Comparison WG VIDEO - 2020-07-31
Slides from the July 31, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-07-31
Chat Record from the July 31, 2020 Meeting: Comparison WG Chat - 2020-07-31
key research strategy
Create “Treatment” groups by selecting unique samples of customers
Stratified sampling to produce Comparison groups
Monitor divergence between “Treatment” and Comparison Groups (both pre- and post-COVID
These “Treatment” groups are not program participants (which is good)!
phase 1: stratified sampling and pre-covid testing
Goal: Develop and Demonstrate successful implementation of stratified sampling
Video of the July 17, 2020 Meeting: Comparison WG VIDEO - 2020-07-17
Slides from the July 17, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-07-17
Chat Record from the July 17, 2020 Meeting: Comparison WG Chat - 2020-07-17
why comparison groups?
proposed methodology to test
Video of the June 26, 2020 Meeting: Comparison WG VIDEO - 2020-06-26
Slides from the June 26, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-06-26
Chat Record from the June 26, 2020 Meeting: Comparison WG Chat - 2020-06-26
key concepts regarding metered savings
Video of the June 05, 2020 Meeting: Comparison WG VIDEO - 2020-06-05
Slides from the June 05, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-06-05
Chat Record from the June 05, 2020 Meeting: Comparison WG Chat - 2020-06-05