Tag Archives: Regulatory Report Card

How Complete Are Federal Agencies’ Regulatory Analyses?

A report released yesterday by the Government Accountability Office will likely get spun to imply that federal agencies are doing a pretty good job of assessing the benefits and costs of their proposed regulations. The subtitle of the report reads in part, “Agencies Included Key Elements of Cost-Benefit Analysis…” Unfortunately, agency analyses of regulations are less complete than this subtitle suggests.

The GAO report defined four major elements of regulatory analysis: discussion of the need for the regulatory action, analysis of alternatives, and assessment of the benefits and costs of the regulation. These crucial features have been required in executive orders on regulatory analysis and OMB guidance for decades. For the largest regulations with economic effects exceeding $100 million annually (“economically significant” regulations), GAO found that agencies always included a statement of the regulation’s purpose, discussed alternatives 81 percent of the time, always discussed benefits and costs, provided a monetized estimate of costs 97 percent of the time, and provided a monetized estimate of benefits 76 percent of the time.

A deeper dive into the report, however, reveals that GAO did not evaluate the quality of any of these aspects of agencies’ analysis. Page 4 of the report notes, “[O]ur analysis was not designed to evaluate the quality of the cost-benefit analysis in the rules. The presence of all key elements does not provide information regarding the quality of the analysis, nor does the absence of a key element necessarily imply a deficiency in a cost-benefit analysis.”

For example, GAO checked to see if the agency include a statement of the purpose of the regulation, but it apparently accepted a statement that the regulation is required by law as a sufficient statement of purpose (p. 22). Citing a statute is not the same thing as articulating a goal or identifying the root cause of the problem an agency seeks to solve.

Similarly, an agency can provide a monetary estimate of some benefits or costs without necessarily addressing all major benefits or costs the regulation is likely to create. GAO notes that it did not ascertain whether agencies addressed all relevant benefits or costs (p. 23).

For an assessment of the quality of agencies’ regulatory analysis, check out the Mercatus Center’s Regulatory Report Card. The Report Card evaluation method explicitly assesses the quality of the agency’s analysis, rather than just checking to see if the agency discussed the topics. For example, to assess how well the agency analyzed the problem it is trying to solve, the evaluators ask five questions:

1. Does the analysis identify a market failure or other systemic problem?

2. Does the analysis outline a coherent and testable theory that explains why the problem is systemic rather than anecdotal?

3. Does the analysis present credible empirical support for the theory?

4. Does the analysis adequately address the baseline — that is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future?

5. Does the analysis adequately assess uncertainty about the existence or size of the problem?

These questions are intended to ascertain whether the agency identified a real, significant problem and identified its likely cause. On a scoring scale ranging from 0 points (no relevant content) to 5 points (substantial analysis), economically significant regulations proposed between 2008 and 2012 scored an average of just 2.2 points for their analysis of the systemic problem. This score indicates that many regulations are accompanied by very little evidence-based analysis of the underlying problem the regulation is supposed to solve. Scores for assessment of alternatives, benefits, and costs are only slightly better, which suggests that these aspects of the analysis are often seriously incomplete.

These results are consistent with the findings of other scholars who have evaluated the quality of agency Regulatory Impact Analyses during the past several decades. (Check pp. 7-10 of this paper for citations.)

The Report Card results are also consistent with the findings in the GAO report. GAO assessed whether agencies are turning in their assigned homework; the Report Card assesses how well they did the work.

The GAO report contains a lot of useful information, and the authors are forthright about its limitations. GAO combed through 203 final regulations to figure out what parts of the analysis the agencies did and did not do — an impressive accomplishment by any measure!

I’m more concerned that some participants in the political debate over regulatory reform will claim that the report shows regulatory agencies are doing a great job of analysis, and no reforms to improve the quality of analysis are needed. The Regulatory Report Card results clearly demonstrate otherwise.

Why Regulations Fail

Last week, David Fahrenthold wrote a great article in the Washington Post, in which he described the sheer absurdity of a USDA regulation mandating a small town magician to develop a disaster evacuation plan for his rabbit (the rabbit was an indispensible part of trick that also involved a hat). The article provides a good example of the federal regulatory process’ flaws that can derail even the best-intentioned regulations. I list a few of these flaws below.

  1. Bad regulations often start with bad congressional statutes. The Animal Welfare Act of 1966, the statute authorizing the regulation, was meant to prevent medical labs from using lost pets for experiments. Over time, the statute expanded to include all warm-blooded animals (pet lizards apparently did not merit congressional protection) and to apply to zoos and circuses in addition to labs (pet stores, dog and cat shows, and several other venues for exhibiting animals were exempt).The statute’s spotty coverage resulted from political bargaining rather than the general public interest in animal welfare. The USDA rule makes the statute’s arbitrariness immediately apparent. Why would a disaster plan benefit circus animals but not the animals in pet stores or farms? (A colleague of mine jokingly suggested eating the rabbit as part of an evacuation plan, since rabbits raised for meat are exempt from the regulation’s requirements).
  2. Regulations face little oversight. When media reported on the regulation’s absurdity, the USDA Secretary Tom Vilsack ordered the regulation to be reviewed. It seems that even the agency’s head was caught off guard by the actions of his agency’s regulators. Beyond internal supervision, only a fraction of regulations face external oversight. Of over 2600 regulations issued in 2012, less than 200 were subject to the OMB review (data from GAO and OMB). Interestingly, the OMB did review the USDA rule but offered only minor revisions.
  3. Agencies often fail to examine the need for regulation. In typical Washington fashion, the agency decided to regulate in response to a crisis – Hurricane Katrina in this case. In fact, the USDA offered little more than Katrina’s example to justify the regulation. It offered little evidence that the lack of disaster evacuation plans was a widespread problem that required the federal government to step in. In this, the USDA is not alone. According to the Mercatus Center’s Regulatory Report Card, which evaluates agencies’ economic analysis, few agencies offer substantial evidence justifying the need for promulgated regulations.
  4. Agencies often fail to examine the regulation’s effectiveness. The USDA’s plan to save animals in case of a disaster was to require owners to draw up an evacuation plan. It offered little evidence that having a plan would in fact save the animals. For example, the magician’s evacuation plan called for shoving the rabbit into a plastic bag and getting out. In the USDA’s view, the magician would not have thought of doing the same had he not drawn up the evacuation plan beforehand.
  5. The public has little influence in the process. By law, agencies are required to ask the public for input on proposed regulations. Yet, small businesses and individual consumers rarely have time or resources to keep an eye on federal agencies. In general, organized interests dominate the commenting process. The article describes the magician’s surprise to learn that he was required to have a license and a disaster evacuation plan his rabbit, even though the regulation was in the works for a few years and was open for public comments for several months. Most small businesses, much like this magician, learn about regulations only after they have passed.
  6. Public comments are generally ignored. Most public comments that the USDA received argued against the rule. They pointed out that it would impose substantial costs on smaller businesses. The agency dismissed the comments with little justification. This case is not unique. Research indicates that agencies rarely make substantial changes to regulations in response to public comments.

Do “Indirect Effects” of Regulation Matter to Real People?

Congressional regulatory reformers recently caught criticism from advocacy groups for introducing legislation that would require federal regulatory agencies to analyze the “indirect effects” of proposed regulations. The only thing I’d criticize the reformers for is poor word choice.

The very term “indirect effects” suggests that they’re talking about something theoretical, inconsequential, and unimportant to the average citizen. But to economists, the indirect effects of a regulation are often the effects that touch the average citizen most directly.

Consider airport security, for example. The Department of Homeland Security (DHS) recently sought public comment on its decision to deploy Advanced Imaging Technology scanners instead of metal detectors at airports. The direct costs of this decision are the extra cost of the new machines, the electricity to run them, and the personnel to staff them – which airline passengers pay for via the taxes and fees on airline tickets. Those are pretty obvious costs, and DHS dutifully toted up these costs in its analysis of its proposed rule.

Less obvious but potentially more important are the other, indirect costs associated with airport security. Passengers who decline to walk through the new machines will receive additional pat-downs. This involves a cost in terms of time (which DHS acknowledges) and potentially diminished privacy and human dignity (which DHS does not discuss). The now-classic phrase “Don’t touch my junk” aptly summarizes one passenger’s reaction to an indirect effect of security regulation that touches passengers quite directly.

But that does not exhaust the list of significant, predictable, indirect effects associated with airport security regulation. The increased delays associated with enhanced, post-9/11 security measures prompted some travelers to substitute driving for flying on short trips. An article by Garrick Blalock, Vrinda Kadiyali, and Daniel H. Simon published in the November 2007 Journal of Law & Economics estimates that post-9/11 security measures cost the airline industry $1.1 billion in lost revenue in the fourth quarter of 2002. Driving is also riskier than flying. Blalock et. al. estimated that the security measures were associated with 129 additional highway deaths in the fourth quarter of 2002.

I’m all for making air travel as safe as possible, but I’d like to see it done smartly, with a minimum of hassle and a maximum of respect for the flying public who pays the bills. A full accounting of the indirect effects of airport security might just prompt policymakers to consider whether they are pursuing regulatory goals in the most sensible way possible.

Unfortunately, airport security is not an isolated example. Data from the Mercatus Center’s Regulatory Report Card reveal that for about 40 percent of the major regulations proposed by executive branch agencies between 2008 and 2012, the agencies failed to conduct any substantial analysis of costs that stem from the proposed regulation’s effects on prices or on human behavior – two classic types of indirect effects.

This won’t do. Telling federal agencies they do not need to understand the indirect effects of regulation is telling them they should proceed in willful ignorance of the effects of their decisions on real people. The reformers have a good idea here – even if it has a misleadingly boring name.