Category Archives: Use of Science

How Complete Are Federal Agencies’ Regulatory Analyses?

A report released yesterday by the Government Accountability Office will likely get spun to imply that federal agencies are doing a pretty good job of assessing the benefits and costs of their proposed regulations. The subtitle of the report reads in part, “Agencies Included Key Elements of Cost-Benefit Analysis…” Unfortunately, agency analyses of regulations are less complete than this subtitle suggests.

The GAO report defined four major elements of regulatory analysis: discussion of the need for the regulatory action, analysis of alternatives, and assessment of the benefits and costs of the regulation. These crucial features have been required in executive orders on regulatory analysis and OMB guidance for decades. For the largest regulations with economic effects exceeding $100 million annually (“economically significant” regulations), GAO found that agencies always included a statement of the regulation’s purpose, discussed alternatives 81 percent of the time, always discussed benefits and costs, provided a monetized estimate of costs 97 percent of the time, and provided a monetized estimate of benefits 76 percent of the time.

A deeper dive into the report, however, reveals that GAO did not evaluate the quality of any of these aspects of agencies’ analysis. Page 4 of the report notes, “[O]ur analysis was not designed to evaluate the quality of the cost-benefit analysis in the rules. The presence of all key elements does not provide information regarding the quality of the analysis, nor does the absence of a key element necessarily imply a deficiency in a cost-benefit analysis.”

For example, GAO checked to see if the agency include a statement of the purpose of the regulation, but it apparently accepted a statement that the regulation is required by law as a sufficient statement of purpose (p. 22). Citing a statute is not the same thing as articulating a goal or identifying the root cause of the problem an agency seeks to solve.

Similarly, an agency can provide a monetary estimate of some benefits or costs without necessarily addressing all major benefits or costs the regulation is likely to create. GAO notes that it did not ascertain whether agencies addressed all relevant benefits or costs (p. 23).

For an assessment of the quality of agencies’ regulatory analysis, check out the Mercatus Center’s Regulatory Report Card. The Report Card evaluation method explicitly assesses the quality of the agency’s analysis, rather than just checking to see if the agency discussed the topics. For example, to assess how well the agency analyzed the problem it is trying to solve, the evaluators ask five questions:

1. Does the analysis identify a market failure or other systemic problem?

2. Does the analysis outline a coherent and testable theory that explains why the problem is systemic rather than anecdotal?

3. Does the analysis present credible empirical support for the theory?

4. Does the analysis adequately address the baseline — that is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future?

5. Does the analysis adequately assess uncertainty about the existence or size of the problem?

These questions are intended to ascertain whether the agency identified a real, significant problem and identified its likely cause. On a scoring scale ranging from 0 points (no relevant content) to 5 points (substantial analysis), economically significant regulations proposed between 2008 and 2012 scored an average of just 2.2 points for their analysis of the systemic problem. This score indicates that many regulations are accompanied by very little evidence-based analysis of the underlying problem the regulation is supposed to solve. Scores for assessment of alternatives, benefits, and costs are only slightly better, which suggests that these aspects of the analysis are often seriously incomplete.

These results are consistent with the findings of other scholars who have evaluated the quality of agency Regulatory Impact Analyses during the past several decades. (Check pp. 7-10 of this paper for citations.)

The Report Card results are also consistent with the findings in the GAO report. GAO assessed whether agencies are turning in their assigned homework; the Report Card assesses how well they did the work.

The GAO report contains a lot of useful information, and the authors are forthright about its limitations. GAO combed through 203 final regulations to figure out what parts of the analysis the agencies did and did not do — an impressive accomplishment by any measure!

I’m more concerned that some participants in the political debate over regulatory reform will claim that the report shows regulatory agencies are doing a great job of analysis, and no reforms to improve the quality of analysis are needed. The Regulatory Report Card results clearly demonstrate otherwise.

When Regulatory Agencies Ignore Comments from the Public

A few days ago, the Department of Energy (DOE) finalized a rule setting energy efficiency standards for metal halide lamp fixtures. Last October I wrote a public interest comment to the DOE to point out several problems with the agency’s preliminary economic analysis for the rule. As part of the Administrative Procedure Act, agencies are required to solicit, and respond to, comments from the public before finalizing regulations. Unfortunately, the DOE failed to even acknowledge many of the points I made in my submission.

As evidence, here are some of the main takeaways from my comment:

1)      The DOE claims consumers and businesses are acting in an irrational manner when purchasing metal halide lamp fixtures because they forgo modest long term energy savings in order to pay a low upfront price for lamp fixtures. Yet the agency offers no convincing evidence to support the theory that consumers act irrationally when purchasing metal halide lamp fixtures. At the same time, roughly 70% of the estimated benefits of the rule are the supposed benefits bestowed upon the public when products people would purchase otherwise are removed from the market.

2)      The DOE is currently adding together costs and benefits that occur in the future but that are discounted to present value using different discount rates. It makes no sense to add together costs and benefits calculated in this manner.

3)      The DOE is using a new value of the Social Cost of Carbon (SCC), a way to measure benefits from reducing carbon dioxide emissions, that may be of questionable validity since the analysts who arrived at the estimate ignored recent scientific evidence. Additionally, the DOE is using the new SCC in its analysis before the public has even had a chance to comment on the validity of the new number.

4)      In its analysis, the DOE is including benefits to foreign countries as a result of reduced carbon dioxide emissions, even while the costs of the metal halide lamp fixture regulation will be borne largely by Americans.

Regarding #1 above, the DOE provided no direct response to my comment in the preamble to its final rule. This even though #1 puts in doubt roughly 70% of the estimated benefits of the rule.

The DOE also failed to respond to #2 above, even though I cited as support a very recent and relevant paper on the subject that appeared in a reputable journal and was coauthored by Nobel laureate Kenneth Arrow.

Regarding #3 and #4, the DOE had this to say:

On November 26, 2013, the Office of Management and Budget (OMB) announced minor technical corrections to the 2013 SCC values and a new opportunity for public comment on the revised Technical Support Document underlying the SCC estimates. Comments regarding the underlying science and potential precedential effect of the SCC estimates resulting from the interagency process should be directed to that process. See 78 FR 70586. Additionally, several current rulemakings also use the 2013 SCC values and the public is welcome to comment on the values as applied in those rulemakings just as the public was welcome to comment on the use and application of the 2010 SCC values in the many rules that were published using those values in the past three years.

In other words, the DOE is committed to continuing to use a value of the SCC that may be flawed since the public has the opportunity to complain to the Office of Management and Budget. At the same time, the DOE tells us we can comment on other regulations that use the new SCC value, so that should reassure anyone whose comment the DOE ignored related to this regulation!

All of this is especially troubling since the DOE is required by statute to ensure its energy efficiency rules are “economically justifiable.” It is hard to argue this rule is economically justifiable when roughly 94% of the rule’s benefits are in doubt. This is the proportion of benefits justified on the basis of consumer irrationality and on the basis that Americans should be paying for benefits that will be captured by citizens in other countries. Without these benefits, the rule fails a benefit-cost test according to the DOE’s own estimates.

The requirement that agencies respond to public comments is designed to ensure a level of democratic accountability from regulators, who are tasked with serving the American public. A vast amount of power is vested in these agencies, who are largely insulated from Congressional oversight. As evidence, Congress has only used its Congressional Review Act authority to overrule major regulations once in its history. If agencies ignore the public, and face little oversight from Congress, what faith can we have that regulators will be held accountable for any harms that inevitably arise from poorly designed regulations?

A New Year’s Gift from the Department of Energy

On New Year’s Eve, the Department of Energy (DOE) announced it will be denying a petition brought to the agency by the Landmark Legal Foundation. The petition had requested the DOE reconsider a regulation related to energy efficiency standards for microwave ovens on the grounds that the Energy Department used a new, much higher, estimate of the social cost of carbon (SCC) in the final analysis of the regulation than had been used in the proposed version of the rule. The SCC is a number the Department uses to estimate benefits to society from reductions in greenhouse gas emissions. The public was denied the opportunity to comment on the higher estimate of the SCC since the new estimate was not used until after the time the public was allowed to comment on the regulation.

Here’s some of the DOE’s reasoning for denying the petition:

In the microwave oven rule, the SCC analysis did not affect DOE’s decision regarding the standards that were published in the Federal Register at either the proposed rule or final rule stage because the estimated benefits to consumers of the standard exceeded the costs of the standard, even without considering the SCC values. [emphasis added]

However, as I and others have stated before, these “benefits to consumers” are not benefits at all, and should be excluded from consideration when determining whether the DOE’s energy efficiency standards produce benefits in excess of costs. In a comment I wrote to the DOE as the agency considered this petition, I said:

The preponderance of the rule’s benefits, nearly 80 percent, are not related to reductions in carbon emissions, or even to any environmental effects at all. Instead, these benefits are based on the assumption that consumers behave in an irrational manner when purchasing microwave ovens and that the Department will be able to “fix” this behavior by issuing a regulation, thereby resulting in benefits to consumers. These “savings” should be excluded from the agency’s final analysis of benefits resulting from the regulation.

So the DOE is partly right. The new SCC really doesn’t make a difference in this particular case. However, this is because the regulation produces net costs to society with or without the higher estimate of the social cost of carbon. Thus, the rule can’t be justified on a cost-benefit basis even with the new social cost of carbon number the DOE uses. As I explained in my comment:

Given that the primary estimate of the total benefits resulting from this regulation is estimated at $294 million per year (2011$), and total costs are estimated at $66.4 million per year, subtracting the consumer “irrationality” benefits of $234 million produces net costs to society of $6.4 million per year (2011$).12 If DOE used a lower value of the SCC, like the estimate used in the proposed version of this regulation, that net cost figure would be even higher. The problem is further compounded if benefits to other countries are excluded from the estimates.

The DOE made no effort to respond to this particular critique in its response to the Landmark Legal Foundation petition. The agency does not view the questionable nature of its estimated benefits to consumers as within the scope of the issue it sought comment on. Perhaps this is so. However, there will be more such regulations in the future where this controversial technique is employed by the DOE. Indeed, at Mercatus we have already commented on such regulations. Additionally, the agency’s decision to slip this notice out on New Year’s Eve leads one to question the degree to which the agency is committed to transparent practices. As a result, an inefficient regulation will be implemented and Americans will be made worse off.

 

Energy Efficiency as Foreign Aid?

A recent suite of energy efficiency regulations issued by the Department of Energy (DOE) have been criticized due to the DOE’s claim that consumers and businesses are behaving irrationally when purchasing appliances and other energy using devises. The Department believes it is bestowing benefits on society by “correcting” these faulty decisions. Mercatus Center scholars have written about this extensively here, here, and here.

However, even if we set aside the Department’s claims of consumer and business “irrationality,” a separate rationale for these regulations is also very problematic. The vast majority of the environmental benefits of these rules stem from reductions in CO2 emissions due to lower emissions from power plants. However, in a 2010 report, the US government estimated only 7 to 23 percent of these benefits will be captured by Americans. The rest will go to people in other countries.

Here’s a recent example. In August, the DOE proposed a rule setting energy efficiency standards for metal halide lamp fixtures. In the agency’s analysis, it estimated total benefits from CO2 emission reductions at $1,532 million. Using the more optimistic estimate of the percentage of CO2 related benefits going to the US citizens (23%), Americans should capture about $450 million in environmental benefits from the rule (once we include benefits from reductions in NOx emissions as well). At the same time, the DOE estimates the rule will cost $1,294 million, much of which will be paid by American consumers and businesses. How can the DOE, which is tasked with serving the American public, support such a policy?

One might argue America is imposing costs on the rest of the world with its carbon emissions, and therefore should pay a type of tax to internalize this external cost we impose on others. However, the rest of the world is also imposing costs on us. In fact, US emissions are actually in decline, while global emissions are on the rise.

Even if we assume it is a sensible policy for Americans to compensate other countries for our carbon emissions, is paying for more expensive products like household appliances the best way to accomplish this goal? Given that no amount of carbon dioxide emission reductions in the US will do much of anything to reduce anticipated global warming, wouldn’t the rest of the world be better off with resources to adapt to climate change, instead of (at best) the warm feeling they might get from knowing Americans are buying more expensive microwave ovens? A more efficient policy would be a cash transfer to other countries, or the US could create a fund the purpose of which would be to help other countries adapt to climate change.

Energy efficiency regulations from the DOE are already difficult enough to justify. Knowing they are really just a roundabout form of foreign aid makes these rules look even less sensible.

The Use of Science in Public Policy

For the budding social scientists out there who hope that their research will someday positively affect public policy, my colleague Jerry Ellig recently pointed out a 2012 publication from the National Research Council called “Using Science as Evidence in Public Policy.” (It takes a few clicks to download, but you can get it for free).

From the intro, the council’s goal was:

[T]o review the knowledge utilization and other relevant literature to assess what is known about how social science knowledge is used in policy making . . . [and] to develop a framework for further research that can improve the use of social science knowledge in policy making.

The authors conclude that, while “knowledge from all the sciences is relevant to policy choices,” it is difficult to explain exactly how that knowledge is used in the public policy sphere.  They go on to develop a framework for research on how science is used.  The entire report is interesting, especially if you care about using science as evidence in public policy, and doubly so if you are a Ph.D. student or recently minted Ph.D. I particularly liked the stark recognition of the fact that political actors will consider their own agendas (i.e., re-election) and values (i.e., the values most likely to help in a re-election bid) regardless of scientific evidence.  That’s not a hopeless statement, though – there’s still room for science to influence policy, but, as public choice scholars have pointed out for decades, the government is run by people who will, on average, rationally act in their own self-interest.  Here are another couple of lines to that point:

Holding to a sharp, a priori distinction between science and politics is nonsense if the goal is to develop an understanding of the use of science in public policy. Policy making, far from being a sphere in which science can be neatly separated from politics, is a sphere in which they necessarily come together… Our position is that the use of [scientific] evidence or adoption of that [evidence-based] policy cannot be studied without also considering politics and values.

One thing in particular stands out to anyone who has worked on the economic analysis of regulations.  The introduction to this report includes this summary of science’s role in policy:

Science has five tasks related to policy:

(1) identify problems, such as endangered species, obesity, unemployment, and vulnerability to natural disasters or terrorist acts;

(2) measure their magnitude and seriousness;

(3) review alternative policy interventions;

(4) systematically assess the likely consequences of particular policy actions—intended and unintended, desired and unwanted; and

(5) evaluate what, in fact, results from policy.

This sounds almost exactly like the process of performing an economic analysis of a regulation, at least when it’s done well (if you want to know well agencies actually perform regulatory analysis, read this, and for how well they actually use the analysis in decision-making,  read this).  Executive Order 12866, issued by President Bill Clinton in 1993, instructs federal executive agencies on the role of analysis in creating regulations, including each of the following instructions.  Below I’ve slightly rearranged some excerpts and slightly paraphrased other parts from Executive Order 12866, and I have added in the bold numbers to map these instructions back to summary of science’s role quoted above. (For the admin law wonks, I’ve noted the exact section and paragraph of the Executive Order that each element is contained in.):

(1) Each agency shall identify the problem that it intends to address (including, where applicable, the failures of private markets or public institutions that warrant new agency action). [Section 1(b)(1)]

(2) Each agency shall assess the significance of that problem. [Section 1(b)(1)]

(3) Each agency shall identify and assess available alternatives to direct regulation, including providing economic incentives to encourage the desired behavior, such as user fees or marketable permits, or providing information upon which choices can be made by the public. Each agency shall identify and assess alternative forms of regulation. [Section 1(b)(3) and Section 1(b)(8)]

(4) When an agency determines that a regulation is the best available method of achieving the regulatory objective, it shall design its regulations in the most cost-effective manner to achieve the regulatory objective. In doing so, each agency shall consider incentives for innovation, consistency, predictability, the costs of enforcement and compliance (to the government, regulated entities, and the public), flexibility, distributive impacts, and equity. [Section 1(b)(5)]

(5) Each agency shall periodically review its existing significant regulations to determine whether any such regulations should be modified or eliminated so as to make the agency’s regulatory program more effective in achieving the regulatory objectives, less burdensome, or in greater alignment with the President’s priorities and the principles set forth in this Executive order. [Section 5(a)]

OMB’s Circular A-4—the instruction guide for government economists tasked with analyzing regulatory impacts—similarly directs economists to include three basic elements in their regulatory analyses (again, the bold numbers are mine to help map these elements back to the summary of science’s role):

(1 & 2) a statement of the need for the proposed action,

(3) an examination of alternative approaches, and

(4) an evaluation of the benefits and costs—quantitative and qualitative—of the proposed action and the main alternatives identified by the analysis.

The statement of the need for proposed action is equivalent to the first (identifying problems) and second tasks (measuring their magnitude and seriousness) from NRC report.  The examination of alternative approaches and evaluation of the benefits and costs of the possible alternatives are equivalent to tasks 3 (review alternative policy interventions) and 4 (assess the likely consequences). 

It’s also noteworthy that the NRC points out the importance of measuring the magnitude and seriousness of problems.  A lot of public time and money gets spent trying to fix problems that are not widespread or systemic.  There may be better ways to use those resources.  Evaluating the seriousness of problems allows a prioritization of limited resources.

Finally, I want to point out how this parallels a project here at Mercatus.  Not coincidentally, the statement of science’s role in policy reads like the grading criteria of the Mercatus Regulatory Report Card, which are:

1. Systemic Problem: How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
2. Alternatives: How well does the analysis assess the effectiveness of alternative approaches?
3. Benefits (or other Outcomes): How well does the analysis identify the benefits or other desired outcomes and demonstrate that the regulation will achieve them?
4. Costs: How well does the analysis assess costs?
5. Use of Analysis: Does the proposed rule or the RIA present evidence that the agency used the Regulatory Impact Analysis in any decisions?
6. Cognizance of Net Benefits: Did the agency maximize net benefits or explain why it chose another alternative?

The big difference is that the Report Card contains elements that emphasize measuring whether the analysis is actually used – bringing us back to the original goal of the research council – to determine “how social science knowledge is used in policy making.”