Not a Sure Thing: Making Regulatory Choices under Uncertainty

The use of formal uncertainty assessment, especially Monte Carlo analysis, is becoming more common in many fields, including economics, finance, engineering, and certain scientific disciplines. In the 1990s, the U.S. Environmental Protection Agency (EPA) began investigating methods for conducting uncertainty analysis, issued a publication titled Guiding Principles for Monte Carlo Analysis, and conducted at least some form of uncertainty analysis in several congressionally mandated studies and regulatory impact analyses (RIAs). The incorporation and sophistication of uncertainty analyses in RIAs will continue to grow as EPA responds to Circular A-4, a document released in 2003 by the Office of Management and Budget that requires uncertainty analyses for RIAs that have annual costs and/or benefits in excess of $1 billion.

Introducing uncertainties into RIAs is an important activity for at least two reasons. The first is that public policy will be better served. If uncertainties are not considered, then the resulting policy choices may fail to meet internal (but often unstated) agency criteria -- or even the public's criteria -- for choosing the best policy amidst high uncertainty.

The second reason is more political in nature. Sole reliance on point estimates masks the underlying distribution of benefit and cost estimates, thereby giving a false sense of security to decisionmakers and the public. If the policy turns out to be a poor choice, then people may well look to the RIA to determine whether the agency had predicted the possibility of such an outcome. To avoid being blindsided, decisionmakers can build uncertainties into their ex ante analyses.

At the same time, better and more complete information does not necessarily lead to better policies. Complex information can confound rather than enlighten or can paralyze the decisionmaking process. Thus, any improvements in capturing uncertainty analytically must be matched by improvements in its communication -- not only to those who make regulatory decisions on the basis of such information but also to stakeholders, judges, the press, and the general public.

Accordingly, Resources for the Future (RFF) was asked to provide some guidance and recommendations to EPA about addressing the requirements of Circular A-4 regarding formal uncertainty analysis and improving its communication. In their report "Not a Sure Thing: Making Regulatory Choices under Uncertainty," the RFF team, led by Alan J. Krupnick and Richard D. Morgenstern, reviewed the uncertainty literature and EPA practice and conducted an in-depth case study on a hypothetical proposed rule to help test various ideas and new analytical directions. In addition, they went beyond Circular A-4 to address how to communicate such complex analyses to decisionmakers, presenting a simplified version of the case study to seven former presidential appointees to EPA and reporting their problems and successes in communicating their results to them.