Read Full Report

Date: Jan. 14, 2004
Contacts: Vanee Vines, Media Relations Officer
Heather McDonald, Media Relations Assistant
Office of News and Public Information
202-334-2138; e-mail <>


NSF Needs Improved Process to Rank Proposals for Large Research Facilities
And Manage the Projects Over Time

WASHINGTON – The National Science Foundation needs a clear process to lay out the criteria and rationale for the selection of large research-facility projects that will receive its financial support, ensuring that the agency evaluates proposals based on their potential returns to science, technology, and society, says a new National Academies report. In 1995, NSF created the Major Research Equipment and Facilities Construction (MREFC) account to support the construction of large research facilities, which enable scientists and engineers to reach previously unattainable scientific frontiers. In recent years, the number of plans to build such facilities has grown, and approved projects have become increasingly complex and expensive.

The committee that wrote the report offered NSF an approach that would strengthen the foundation's current processes for identifying, developing, prioritizing, and managing large research-facility projects funded through MREFC. Although they account for just under 4 percent of the foundation's total budget, these projects are highly visible because of their multimillion dollar budgets, their potential to shape the course of future research, and the economic benefits they bring to the regions where they are located. However, many researchers and federal policy-makers have expressed concerns about NSF's current method for deciding which projects would be submitted to Congress for funding.

"This is an opportunity for the National Science Foundation to further improve its work in the development of major research facilities by incorporating a more objective and transparent approach to screening and planning into its process, and by involving the research community more fully in the generation and ranking of ideas for these unique hubs of scientific innovation," said committee chair William F. Brinkman, professor of physics, Princeton University, Princeton, N.J., and retired vice president of research for Bell Labs, Murray Hill, N.J.

NSF should use the committee's proposed strategy as a guide to create and implement a road map for selecting facilities that may be built within the next 10 to 20 years, the report says. The National Science Board, which helps to oversee the foundation, should monitor this effort, approve the road map, and use it each year to rank projects proposed for funding. Additional input from the scientific community, including researchers who put forth ideas for specific projects, would enhance the process. Ultimately, the road map would inform NSF's annual budget submission to Congress for MREFC funding, and its overall budget requests to the White House. Because scientific opportunities and policy priorities sometimes change, a road map would be periodically revised and should be viewed as a dynamic document for the development of NSF's large research-facility program – and as such would not guarantee that projects selected would be funded.

Enhancing the Selection Process

Under the committee's proposal, overlapping categories of criteria in a three-stage ranking process should shape the selection of projects for the road map. High standards of scientific and technical quality should be at the core of all criteria, the report says. Also, NSF should ensure that budget details and other planning aspects of each proposal are thoroughly considered before it enters the new review system.

In the first stage of review and ranking, researchers in a given field or interdisciplinary area should determine which proposals have the greatest scientific potential within that domain. Reviewers also should assess the technological readiness of proposals, the scientific credentials of the projects' leading scholars, and the proposal teams' project-management capabilities, the report says.

The second stage of ranking should take place within NSF's seven major units, or "directorates." Each directorate oversees work in a set of related scientific fields. A proposal that survives the first stage of review would be sent to the senior leadership of the appropriate directorate, who should determine how well the proposal meets NSF's strategic goals – with guidance from the directorates' advisory committees, the report says. The directorates should consider the potential impact of projects on scientific advances within related fields, and whether proposals include opportunities to aid researchers from multiple disciplines or to facilitate interdisciplinary research. The potential for U.S. work-force development and for interagency or international collaboration also should be key criteria at this stage.

In the third stage of ranking, the National Science Board, working with NSF's director and its senior staff, should assess all proposals that withstood the directorates' review, using criteria that emphasize broad, national goals, the committee said. Reviewers should consider, for instance, whether proposals are in new or emerging fields of research that could transform science or engineering, and whether projects could help maintain U.S. leadership in critical areas. In this final stage, the board also should consider the impact of various projects on current national priorities and on the balance of research across fields in NSF's portfolio. NSB should then, in consultation with NSF's director and its senior staff, select and rank projects for funding.

On the whole, criteria could change as government-wide initiatives and unexpected events shift priorities, the report notes. Similarly, NSB could decide to place greater emphasis on certain issues at particular times. But key questions and issues should generally be identified before the ranking process begins. And a clear rationale should be provided when proposed large research-facility projects are ranked.

In addition, the road map should take into account the need for continued funding of existing facilities. It should rank newly approved projects that would begin development in a given year against others that would simultaneously require ongoing financial support, the report says. NSF could then develop annual budgets that include a five-year outlook, and supply the rationale for the decisions.

Greater Oversight of Projects Needed

Large research facilities are a vital component of NSF's science and technology portfolio, the committee emphasized. Some existing and planned facilities include more powerful versions of instruments that have been used for decades, such as telescopes. Other large facilities use new ways of gathering information, such as a project that measures gravity waves generated by cosmic events. Some primarily serve specific scientific disciplines; others, a cross-section. But overall, reforms are needed to ensure that funded projects are executed properly, on schedule, and within budget – and that facilities are well-managed over time.

Once projects receive funding, independent committees of engineering, construction, and scientific experts should conduct periodic reviews to gauge how well the original plans are being carried out. These committees should give their evaluations to the agency and the NSB, the report says.

After a research facility has been constructed, another independent committee of scientists and experts in managing such facilities should monitor operations as needed, the report adds. Within the next two years, a different independent committee should evaluate the operation and effectiveness of NSF's new office of the deputy of large-facility projects.

The White House Office of Science and Technology Policy should coordinate similar kinds of road maps across federal agencies and with other countries, the committee said. OSTP's active participation could help foster discussion of potential interagency and international collaborations at the earliest possible stages.

The study was sponsored by the National Science Foundation. The National Academies provide science, engineering, and medical advice to the federal government under a congressional charter granted to the National Academy of Sciences. A committee roster follows.
Copies of Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation are available from the National Academies Press; tel. 202-334-3313 or 1-800-624-6242 or on the Internet at Reporters may obtain a copy from the Office of News and Public Information (contacts listed above).
# # #

[ This news release and the report are available at ]

vv: b,d,f2,f5,h3,l,m2,n

Committee on Science, Engineering, and Public Policy
Division on Engineering and Physical Sciences
Board on Physics and Astronomy

Committee on Setting Priorities for Large Research-Facility Projects
Supported by the National Science Foundation

William Brinkman1 (chair)
Senior Research Physicist
Department of Physics
Princeton University, and
Vice President of Research
Bell Laboratories (retired)
Murray Hill, N.J.

David Auston1, 2
Kavli Foundation
Oxnard, Calif.

Persis Drell
Professor, and
Director of Research
Stanford Linear Accelerator Center
Menlo Park, Calif.

Alan Dressler1
Astronomer, and
Member of Scientific Staff
Observatories of the Carnegie Institution
Pasadena, Calif.

William Friend2
Executive Vice President
Bechtel Group Inc. (retired), and
University of California's President's Council on the National Laboratories
Washington, D.C.

Bruce Hevly
Associate Professor
Department of History
University of Washington

Wesley Huntress Jr.
Geophysical Laboratory
Carnegie Institution of Washington
Washington, D.C.

Christopher Llewellyn-Smith
Culham Division
The United Kingdom Atomic Energy Authority
Abingdon, United Kingdom

Linda Lee Magid
Department of Chemistry
University of Tennessee

Marc Pelaez
Rear Admiral, U.S. Navy (retired), and
Vice President
Newport News Shipbuilding (retired)
Newport News, Va.

Robert Rutford
Excellence in Education Foundation Chaired Professor in Geosciences
Geosciences DepartmentUniversity of Texas

Joseph Taylor Jr.1
Dean of Faculty, and
James S. McDonnell Distinguished University Professor of Physics
Princeton University
Princeton, N.J.

Michael Telson
Director of National Laboratory Affairs
University of California Office
of Federal Government Relations
Washington, D.C.

G. David Tilman1
Regents' Professor;
McKnight University Presidential Chair;
Department of Ecology, Evolution, and Behavior, and
Cedar Creek Natural History Area
University of Minnesota
Saint Paul


Deborah Stine
Study Director

Timothy Meyer
Staff Officer

1 Member, National Academy of Sciences
2 Member, National Academy of Engineering