Tag Archives: Report Card

How Complete Are Federal Agencies’ Regulatory Analyses?

A report released yesterday by the Government Accountability Office will likely get spun to imply that federal agencies are doing a pretty good job of assessing the benefits and costs of their proposed regulations. The subtitle of the report reads in part, “Agencies Included Key Elements of Cost-Benefit Analysis…” Unfortunately, agency analyses of regulations are less complete than this subtitle suggests.

The GAO report defined four major elements of regulatory analysis: discussion of the need for the regulatory action, analysis of alternatives, and assessment of the benefits and costs of the regulation. These crucial features have been required in executive orders on regulatory analysis and OMB guidance for decades. For the largest regulations with economic effects exceeding $100 million annually (“economically significant” regulations), GAO found that agencies always included a statement of the regulation’s purpose, discussed alternatives 81 percent of the time, always discussed benefits and costs, provided a monetized estimate of costs 97 percent of the time, and provided a monetized estimate of benefits 76 percent of the time.

A deeper dive into the report, however, reveals that GAO did not evaluate the quality of any of these aspects of agencies’ analysis. Page 4 of the report notes, “[O]ur analysis was not designed to evaluate the quality of the cost-benefit analysis in the rules. The presence of all key elements does not provide information regarding the quality of the analysis, nor does the absence of a key element necessarily imply a deficiency in a cost-benefit analysis.”

For example, GAO checked to see if the agency include a statement of the purpose of the regulation, but it apparently accepted a statement that the regulation is required by law as a sufficient statement of purpose (p. 22). Citing a statute is not the same thing as articulating a goal or identifying the root cause of the problem an agency seeks to solve.

Similarly, an agency can provide a monetary estimate of some benefits or costs without necessarily addressing all major benefits or costs the regulation is likely to create. GAO notes that it did not ascertain whether agencies addressed all relevant benefits or costs (p. 23).

For an assessment of the quality of agencies’ regulatory analysis, check out the Mercatus Center’s Regulatory Report Card. The Report Card evaluation method explicitly assesses the quality of the agency’s analysis, rather than just checking to see if the agency discussed the topics. For example, to assess how well the agency analyzed the problem it is trying to solve, the evaluators ask five questions:

1. Does the analysis identify a market failure or other systemic problem?

2. Does the analysis outline a coherent and testable theory that explains why the problem is systemic rather than anecdotal?

3. Does the analysis present credible empirical support for the theory?

4. Does the analysis adequately address the baseline — that is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future?

5. Does the analysis adequately assess uncertainty about the existence or size of the problem?

These questions are intended to ascertain whether the agency identified a real, significant problem and identified its likely cause. On a scoring scale ranging from 0 points (no relevant content) to 5 points (substantial analysis), economically significant regulations proposed between 2008 and 2012 scored an average of just 2.2 points for their analysis of the systemic problem. This score indicates that many regulations are accompanied by very little evidence-based analysis of the underlying problem the regulation is supposed to solve. Scores for assessment of alternatives, benefits, and costs are only slightly better, which suggests that these aspects of the analysis are often seriously incomplete.

These results are consistent with the findings of other scholars who have evaluated the quality of agency Regulatory Impact Analyses during the past several decades. (Check pp. 7-10 of this paper for citations.)

The Report Card results are also consistent with the findings in the GAO report. GAO assessed whether agencies are turning in their assigned homework; the Report Card assesses how well they did the work.

The GAO report contains a lot of useful information, and the authors are forthright about its limitations. GAO combed through 203 final regulations to figure out what parts of the analysis the agencies did and did not do — an impressive accomplishment by any measure!

I’m more concerned that some participants in the political debate over regulatory reform will claim that the report shows regulatory agencies are doing a great job of analysis, and no reforms to improve the quality of analysis are needed. The Regulatory Report Card results clearly demonstrate otherwise.

The Use of Science in Public Policy

For the budding social scientists out there who hope that their research will someday positively affect public policy, my colleague Jerry Ellig recently pointed out a 2012 publication from the National Research Council called “Using Science as Evidence in Public Policy.” (It takes a few clicks to download, but you can get it for free).

From the intro, the council’s goal was:

[T]o review the knowledge utilization and other relevant literature to assess what is known about how social science knowledge is used in policy making . . . [and] to develop a framework for further research that can improve the use of social science knowledge in policy making.

The authors conclude that, while “knowledge from all the sciences is relevant to policy choices,” it is difficult to explain exactly how that knowledge is used in the public policy sphere.  They go on to develop a framework for research on how science is used.  The entire report is interesting, especially if you care about using science as evidence in public policy, and doubly so if you are a Ph.D. student or recently minted Ph.D. I particularly liked the stark recognition of the fact that political actors will consider their own agendas (i.e., re-election) and values (i.e., the values most likely to help in a re-election bid) regardless of scientific evidence.  That’s not a hopeless statement, though – there’s still room for science to influence policy, but, as public choice scholars have pointed out for decades, the government is run by people who will, on average, rationally act in their own self-interest.  Here are another couple of lines to that point:

Holding to a sharp, a priori distinction between science and politics is nonsense if the goal is to develop an understanding of the use of science in public policy. Policy making, far from being a sphere in which science can be neatly separated from politics, is a sphere in which they necessarily come together… Our position is that the use of [scientific] evidence or adoption of that [evidence-based] policy cannot be studied without also considering politics and values.

One thing in particular stands out to anyone who has worked on the economic analysis of regulations.  The introduction to this report includes this summary of science’s role in policy:

Science has five tasks related to policy:

(1) identify problems, such as endangered species, obesity, unemployment, and vulnerability to natural disasters or terrorist acts;

(2) measure their magnitude and seriousness;

(3) review alternative policy interventions;

(4) systematically assess the likely consequences of particular policy actions—intended and unintended, desired and unwanted; and

(5) evaluate what, in fact, results from policy.

This sounds almost exactly like the process of performing an economic analysis of a regulation, at least when it’s done well (if you want to know well agencies actually perform regulatory analysis, read this, and for how well they actually use the analysis in decision-making,  read this).  Executive Order 12866, issued by President Bill Clinton in 1993, instructs federal executive agencies on the role of analysis in creating regulations, including each of the following instructions.  Below I’ve slightly rearranged some excerpts and slightly paraphrased other parts from Executive Order 12866, and I have added in the bold numbers to map these instructions back to summary of science’s role quoted above. (For the admin law wonks, I’ve noted the exact section and paragraph of the Executive Order that each element is contained in.):

(1) Each agency shall identify the problem that it intends to address (including, where applicable, the failures of private markets or public institutions that warrant new agency action). [Section 1(b)(1)]

(2) Each agency shall assess the significance of that problem. [Section 1(b)(1)]

(3) Each agency shall identify and assess available alternatives to direct regulation, including providing economic incentives to encourage the desired behavior, such as user fees or marketable permits, or providing information upon which choices can be made by the public. Each agency shall identify and assess alternative forms of regulation. [Section 1(b)(3) and Section 1(b)(8)]

(4) When an agency determines that a regulation is the best available method of achieving the regulatory objective, it shall design its regulations in the most cost-effective manner to achieve the regulatory objective. In doing so, each agency shall consider incentives for innovation, consistency, predictability, the costs of enforcement and compliance (to the government, regulated entities, and the public), flexibility, distributive impacts, and equity. [Section 1(b)(5)]

(5) Each agency shall periodically review its existing significant regulations to determine whether any such regulations should be modified or eliminated so as to make the agency’s regulatory program more effective in achieving the regulatory objectives, less burdensome, or in greater alignment with the President’s priorities and the principles set forth in this Executive order. [Section 5(a)]

OMB’s Circular A-4—the instruction guide for government economists tasked with analyzing regulatory impacts—similarly directs economists to include three basic elements in their regulatory analyses (again, the bold numbers are mine to help map these elements back to the summary of science’s role):

(1 & 2) a statement of the need for the proposed action,

(3) an examination of alternative approaches, and

(4) an evaluation of the benefits and costs—quantitative and qualitative—of the proposed action and the main alternatives identified by the analysis.

The statement of the need for proposed action is equivalent to the first (identifying problems) and second tasks (measuring their magnitude and seriousness) from NRC report.  The examination of alternative approaches and evaluation of the benefits and costs of the possible alternatives are equivalent to tasks 3 (review alternative policy interventions) and 4 (assess the likely consequences). 

It’s also noteworthy that the NRC points out the importance of measuring the magnitude and seriousness of problems.  A lot of public time and money gets spent trying to fix problems that are not widespread or systemic.  There may be better ways to use those resources.  Evaluating the seriousness of problems allows a prioritization of limited resources.

Finally, I want to point out how this parallels a project here at Mercatus.  Not coincidentally, the statement of science’s role in policy reads like the grading criteria of the Mercatus Regulatory Report Card, which are:

1. Systemic Problem: How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
2. Alternatives: How well does the analysis assess the effectiveness of alternative approaches?
3. Benefits (or other Outcomes): How well does the analysis identify the benefits or other desired outcomes and demonstrate that the regulation will achieve them?
4. Costs: How well does the analysis assess costs?
5. Use of Analysis: Does the proposed rule or the RIA present evidence that the agency used the Regulatory Impact Analysis in any decisions?
6. Cognizance of Net Benefits: Did the agency maximize net benefits or explain why it chose another alternative?

The big difference is that the Report Card contains elements that emphasize measuring whether the analysis is actually used – bringing us back to the original goal of the research council – to determine “how social science knowledge is used in policy making.”