Institutional Roles and Goals for Retrospective Regulatory Analysis

Governments have developed significant programs of prospective ex ante impact assessment, seeking to foresee the environmental, economic, and other impacts of new policies and projects (OECD 2018; Wiener 2013), in order to take precautions against risks before they occur and to ensure that new measures will improve social well-being (Graham 2008; Revesz and Livermore 2008; Wiener 2018). Environmental Impact Assessment (EIA), enacted by the US in the National Environmental Policy Act (NEPA) in 1969 and now adopted around the world, calls on decision makers to stop and think ahead about potential future impacts before taking action (National Environmental Policy Act (NEPA) 1970; Craik 2008; Wiener and Ribeiro 2016a). Similarly, Regulatory Impact Assessment (RIA)—required by every US President since the 1970s, and increasingly adopted in other countries as well—has emphasized prospective ex ante assessment of the future impacts of proposed new regulations (OECD 2018; Wiener and Ribeiro 2016a).

In recent years, there has been increasing demand to supplement such ex ante analysis with ex post retrospective analysis of past regulation – to add hindsight to foresight. The actual impacts of a rule after it is put into operation almost certainly differ from the ex ante forecasts. The terms of the rule itself, and its implementation, may have changed after its ex ante RIA was initially conducted. Even if the rule has not changed, the social and economic conditions and technology may have changed, altering the impacts of the rule. The joint impacts of multiple rules may interact and accumulate in ways different from the sum of the forecasts of each rule on its own. And ex ante forecasts are inevitably imperfect because experts do not have perfect foresight and may make errors (Tetlock and Gardner 2015). Studies of ex ante RIAs indicate that they may be only roughly accurate, with errors of both overestimation and underestimation of actual costs, benefits, rates of technological change, rates of compliance, and related factors (Harrington, Morgenstern, and Nelson 2000; Office of Information and Regulatory Affairs 2005; Harrington 2006; Morgenstern 2015). Ex ante RIA occurs “when the least is known and any analysis must rest on many unverifiable and potentially controversial assumptions” (Greenstone 2009).

The accuracy of ex ante impact assessment can be improved through astute hindsight: learning from past experience via systematic retrospective analysis (Cropper, Fraas, and Morgenstern 2017; Tetlock and Gardner 2015; Gubler 2017). Environmental law itself is the product of social learning about the past adverse impacts of market failures and the need to address them through law. Regulatory systems can incorporate a learning process (Farber 1993; Pidot 2015; Bennear and Wiener 2019b; 2019a) to improve foresight via hindsight—they can couple prospective with retrospective analysis, and use the latter to improve the former (Cropper, Fraas, and Morgenstern 2017; Wiener and Ribeiro 2016b; Aldy 2014; Coglianese 2012).

In addition to the intellectual stimulus for retrospective analysis, there is also a strong political stimulus for such review. The politics of environmental policy in the United States (and many other countries) may simultaneously seek more protection and less regulation.[1] Past rules that over time become mismatches with current needs may foster frustration. This political pressure may lead to calls for “look back” reviews of the existing stock of regulation, especially to reduce costs.

One strategy for threading this political needle is to call for some version of “better” or “smarter” regulation, regulation that is more effective but lower cost, and affords more flexibility in compliance. Indeed, there have been just such calls by every US President, in both political parties, since the 1970s (see discussion in Section 3). For example, President Obama stated in his inaugural address in 2009 “The question we ask today is not whether government is too big or too small, but whether it works” (Obama 2009). Every US President since the 1970s has issued or maintained Executive Orders calling for economic analysis of regulation – including several calling for retrospective reviews of existing regulations (as detailed below). Congress has also enacted legislation to improve regulatory performance through, in part, regulatory review.

Despite these longstanding and bipartisan calls to ramp up retrospective analysis, it has been easier said than done. These efforts appear to have yielded limited results thus far (Cropper, Fraas, and Morgenstern 2017; Wiener and Ribeiro 2016b; Aldy 2014; Coglianese 2013; Lutter 2013; Dudley 2013; Sunstein 2014). The difficulty in successfully implementing ex post analysis presents a puzzle. Why have the US and Europe, among others, been able to implement ex ante analysis fairly widely, over several decades and across different political parties, whereas ex post analysis has been far less implemented despite similar decades-long calls across parties?

The answer to this puzzle is surely multi-dimensional—many different factors combine to hinder the widespread implementation of ex post analysis. But at least one of these dimensions concerns the institutional framework surrounding ex post analysis. The institutional framework for retrospective review in the US federal government has almost always focused on review of one rule at a time, conducted by the agency that issued or promulgated the rule. And such single-rule, single-agency review has typically focused primarily (often exclusively) on two critieria: (1) Is the rule still relevant or is it obsolete, and (2) what are the costs of the rule and how can those costs be reduced.

In this paper we argue that this institutional framework for retrospective review—one rule, by the promulgating agency, focused on relevance and cost—is only well-suited to meet the goals of retrospective review in a subset of cases. In particular, this narrow institutional framework cannot address several goals of retrospective review including: (a) analyzing cumulative impacts of multiple rules on an industry; (b) evaluating not only costs but also benefits, and ancillary impacts (unintended or unanticipated countrvailing harms and co-benefits of rules); (c) learning to improve the validity and accuracy of different methodologies for estimating regulatory impacts ex ante, including on the benefits side of the ledger; (d) learning about methodological and policy design options that span beyond the domain of a single regulatory agency; and more.

To move beyond this narrow or singular institutional approach, we suggest consideration of a broader set of institutional options for retrospective regulatory analysis and review, in order to match the broader goals of retrospective analysis and review with the insititutional framework for such analysis and review. The choice among institutions involves matching objectives and tasks with capabilities and incentives (Breyer 2009). As we recount below, calls for retrospective review of US federal regulation have typically asked the agency that issued the rule to review its costs, subject to oversight by the Office of Management and Budget (OMB) through its Office of Information and Regulatory Affairs (OIRA). That issuing agency may have the most data and expertise on its own rule, and the legal authority to revise the rule if warranted. But the issuing agency may also face inhibitions on retrospective analysis, including the opportunity cost of the time and staff diverted from other priorities, and the awkwardness of potentially critiquing the agency’s own past policies and analyses. In short, it may be that the slow progress of retrospective analysis to date derives in part from the choice of institutional framework. Perhaps more could be accomplished with a wider set of institutional options. And a broader set of institutional options could better match the broader set of goals for retrospective analysis and review.

The paper proceeds as follows. In section 2 we outline the different goals and tasks of retrospective regulatory analysis and review. Section 3 briefly reviews the history of retrospective regulatory review and demonstrates that past efforts have had a narrow institutional framework for such reviews. Section 4 articulates the different institutional options for retrospective analysis and review and how the different institutional options can be best matched to the different objectives and goals, such as a broader regulatory learning process. Section 5 offers our conclusions and recommendations.

[1] For example, the Gallup Poll Social Series, conducted in March or April from 2000-2006 and again from 2010-2018, asks respondents “Do you think the U.S. Government is doing too much, too little, or about the right amount in terms of protecting the environment.” In every year, the plurality of respondents said the government was doing “too little.” In 10 out of 15 years for which the survey was conducted, a strict majority of respondents answered “too little” to this question. The lowest percentage with this answer occurred in 2010 when 46% responded too little, while 15% said “too much” and 35% said “about the right amount.” At the same time, the Gallup Poll Social Series, conducted in September annually since 2001 (with two extra surveys in February and June of 2002), asks “In general, do you think there is too much, too little, or the right amount of regulation on business and industry.” In every year except for two (2002 and 2006), the plurality of respondents answered “too much,” with the percentage stating “too much” ranging from 34-50. (The survey was conducted three times in 2002, following the terrorist attacks of Sept. 11, 2001. In February 2002, only 28% of respondents felt there was “too much” regulation of business and industry, while 30% felt there was “too little” and 39% felt there was the right amount. The poll was conducted again in June and September of 2002 and the percentages reporting “too much” had risen to 32 and 35, respectively and too much was again the modal answer by September of 2002.) https://news.gallup.com/poll/1615/environment.aspx