National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
Office of News and Public Information
National Academy of Engineering
Back | Home
News from the National Academies

Read Full Report

Date:  Jan. 31, 2008

Contacts:  Sara Frueh, Media Relations Officer

Luwam Yeibio, Media Relations Assistant

Office of News and Public Information

202-334-2138; e-mail <news@nas.edu>

 

for immediate release

 

Changes Needed in How Federal Government Evaluates

Efficiency of Research at EPA, Other Agencies

 

WASHINGTON -- The White House Office of Management and Budget evaluates research at the U.S. Environmental Protection Agency and other federal agencies using the Program Assessment Rating Tool (PART), a set of questions that asks agencies about many aspects of their programs, including whether they can measure and demonstrate annual improvements in efficiency.  Based on the answers, OMB rates research programs as effective, ineffective, or somewhere in between.  An "ineffective" rating can have serious adverse consequences for a program or agency.  After experiencing difficulty meeting OMB's requirements to demonstrate the efficiency of its research programs, EPA asked a National Research Council committee for guidance on how to measure efficiency.  

 

The committee's new report recommends four changes in how the federal government assesses the efficiency of research at EPA and other agencies.  First, the report emphasizes that assessing efficiency should be considered only one part of evaluating a program's quality, relevance, and effectiveness. 

 

Second, the report introduces a novel distinction between "investment efficiency" and "process efficiency," and recommends that these aspects be evaluated in different ways.  Assessments of investment efficiency should examine whether an agency's R&D portfolio, including the budget, is relevant, of high quality, matches the agency’s strategic plan, and is adjusted as new knowledge and priorities emerge.  These evaluations require panels of experts, which should comprise both scientists and other stakeholders.  In contrast, evaluations of process efficiency should focus on "inputs" (the people, funds, and facilities dedicated to research) and "outputs" (the services, grants, publications, monitoring, and new techniques produced by research), as well as their timelines.  Of nine measures currently used by R&D agencies, common examples are the number of grants awarded or publications produced annually, which could be assessed against appropriate benchmarks. 

 

Third, the committee clarified the roles of outputs and outcomes.  EPA's difficulties in answering PART's questions about efficiency have grown out of OMB's insistence that the agency find ways to measure the efficiency of its research based on outcomes, rather than outputs.  Measuring research efficiency based on what the committee describes as "ultimate outcomes" -- for example, whether a program eventually results in cleaner air or fewer deaths -- is neither achievable nor valid, because such outcomes occur far in the future and are highly dependent upon actions taken by many other people who may or may not use the research findings.  The committee's review of practices across  government R&D agencies revealed that no agency has found a way to demonstrate efficiency based on ultimate outcomes.  

 

The report does endorse evaluating efficiency based on "intermediate outcomes" -- for example, assessing whether a program has improved the body of knowledge for decision-making, or disseminated newly developed tools and models.  These assessments would be conducted most effectively using expert review panels and not just formulas.

 

Finally, the committee recommended that the efficiency of EPA's research programs be evaluated according to the same standards used at other agencies.  OMB has rejected some methods for measuring research efficiency when proposed by EPA, but accepted them when proposed by other agencies.  OMB should train and oversee its budget examiners to make sure they implement the PART questionnaire consistently and equitably across agencies.

 

The study was sponsored by the U.S. Environmental Protection Agency.  The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies.  They are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter.  The Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering.  A committee roster follows.

 

#       #       #

 

[ This news release and report are available at http://national-academies.org ]

 

 

NATIONAL RESEARCH COUNCIL

Division on Policy and Global Affairs

Committee on Science, Engineering, and Public Policy

and

Division on Earth and Life Studies

Board on Environmental Studies and Toxicology

 

Committee on Evaluating the Efficiency of Research and Development Programs

At the Environmental Protection Agency

 

Gilbert S. Omenn  (chair)
Professor of Internal Medicine, Human Genetics, and Public Health, and
Director
Center for Computational Medicine and Biology
University of Michigan Medical School
Ann Arbor 

George V. Alexeeff
Deputy Director
Office of Environmental Health Hazard Assessment
California Environmental Protection Agency
Oakland

Radford Byerly Jr.
Research Scientist
Center for Science and Technology Policy Research
University of Colorado
Boulder

Edwin H. Clark II
Senior Fellow
Earth Policy Institute
Arlington, Va.

Susan E. Cozzens
Professor
School of Public Policy
Georgia Institute of Technology
Atlanta

 
Linda J. Fisher
Vice President and Chief Sustainability Officer
E.I. du Pont de Nemours & Co.
Wilmington, Del.

Paul Gilman
Director
Oak Ridge Center for Advanced Studies
Oak Ridge National Laboratory
Oak Ridge, Tenn.

 

T.J. Glauthier
Head
TJG Energy Associates; and
Former President and CEO
Electricity Innovation Institute
Palo Alto, Calif.

Carol J. Henry

Independent Consultant

Bethesda, Md.; and

Vice President  
Industry Performance Programs
American Chemistry Council (retired)
Arlington, Va. 

Robert J. Huggett
Consultant, and
Professor Emeritus of Marine Science
College of William and Mary
Seaford, Va.

 

Sally Katzen
Visiting Professor of Law
George Mason University
Arlington, Va.

 

Terry F. Young
Independent Consultant; and
Senior Consulting Secretary
Environmental Defense Fund
Oakland, Calif.

 

RESEARCH COUNCIL STAFF

 

Richard Bissell

Study Director