Public Access Records Office
The National Academies
500 5th Street NW
Room KECK 219
Washington, DC 20001
Tel: (202) 334-3543
Email: paro@nas.edu
Project Information

Project Information


Developing Science, Technology, and Innovation Indicators for the Future


Project Scope:

An ad hoc panel, convened under the Committee on National Statistics (CNSTAT), in collaboration with the Board on Science, Technology, and Economic Policy (STEP), proposes to conduct a study of the status of the science, technology, and innovation (STI) indicators that are currently developed and published by the National Science Foundation's (NSF) National Center for Science and Engineering Statistics (NCSES).  The major focus of the panel will be to assess and provide recommendations regarding the need for revised, refocused, and newly developed indicators designed to better reflect fundamental and rapid changes that are reshaping global science, technology and innovation (STI) systems.  The study will address indicators development by NCSES in its role as a U.S. federal statistical agency charged with providing balanced, policy relevant but policy-neutral information to the President, federal executive agencies, the National Science Board, the Congress, and the public.  The study will involve assessing the utility of STI indicators currently used or under development in the United States and by other governments and international organizations. Based on these activities, the study panel will develop a priority ordering for refining, making more internationally comparable, or developing a set of new STI indicators on which NCSES should focus, along with a discussion of the rationale for the assigned priorities.  NCSES is particularly interested in the international scope of STI indicators and the need for developing new indicators that measure developments in innovative activities in the U.S. and abroad.   The forward-looking aspect of this study is paramount; NCSES requests the panel’s foresight on the types of data, metrics and indicators that will be particularly influential in evidentiary policy decision-making for years to come.  The panel will produce an interim report at the end of the first year of the study indicating its approach to reviewing the needs and priorities for STI indicators and a final report at the end of the study with conclusions and recommendations.

Status: Completed

PIN: DBASSE-CNSTAT-10-06

Project Duration (months): 24 month(s)

RSO: Fealing, Kaye



Geographic Focus:

Committee Membership

Committee Post Date: 03/03/2011

Robert E. Litan - (Co-Chair)
The Ewing Marion Kauffman Foundation

ROBERT E. LITAN (Co-chair) is the vice president for research and policy at the Kauffman Foundation where he oversees the foundation’s program for data collection and research on entrepreneurship and economic growth. He also serves as a senior fellow in economic studies at the Brookings Institution. During his career, he has authored or coauthored 22 books, edited another 15, and authored or coauthored over 200 articles in journals, magazines, and newspapers. Since the onset of the financial crisis, he has authored or coauthored a number of essays on financial reforms for the Brookings website. He served in several capacities in the federal government: as associate director of the Office of Management and Budget; deputy assistant attorney general, Antitrust Division, Department of Justice; and staff economist, Council of Economic Advisers. He has a B.S. degree in economics (summa cum laude) from the Wharton School of Finance at the University of Pennsylvania, a J.D. from Yale Law School, and M. Phil. and Ph.D. degrees in economics from Yale University. He is a member of the DBASSE Committee on Developing a Research Agenda for Social Science Evidence for Use.
Andrew W. Wyckoff - (Co-Chair)
Organization for Economic Cooperation and Development

ANDREW W. WYCKOFF (Co-chair) is director of the OECD’s Directorate for Science, Technology, and Industry (STI). Prior to this he was head of the Information, Computer, and Communications Policy (ICCP) division at the OECD, which supports the organization’s work on information society as well as consumer policy issues. Before heading ICCP, he was head of STI’s Economic Analysis and Statistics Division, which develops methodological guidelines, collects statistics, and undertakes empirical analysis in support of science, technology, and innovation policy analysis. His experience prior to the OECD includes being a program manager of the Information, Telecommunications, and Commerce program of the U.S. Congressional Office of Technology Assessment (OTA), an economist at the U.S. National Science Foundation (NSF), and a programmer at The Brookings Institution. He has a B.A. degree from the University of Vermont, and an MPP degree from the Kennedy School of Government at Harvard University.
Carter Bloch
University of Aarhus

CARTER BLOCH is a research director at the Danish Centre for Studies in Research and Research Policy at the University of Aarhus in Denmark. His research focuses on innovation measurement, innovation policy, knowledge spillovers, and the relationship between R&D, innovation, and economic performance. He has been involved in a number of international projects concerning the development of innovation indicators, micro-based data analysis, and the development of biotechnology statistics, and is currently heading an international project on the measurement of innovation in public sector organizations. He was also the main draft writer of the 2005 revision of the Eurostat/OECD Oslo Manual for innovation measurement. He has a B.A. degree in economics from the University of Wisconsin-Madison, and a Ph.D. degree in economics from the University of Aarhus, Denmark.
Nicholas Chrisman
Universite Laval

NICHOLAS CHRISMAN has been professor of geomatics sciences at the Université Laval (Québec City) since January 2005. His main assignment is to serve as scientific director of the GEOIDE Network, a Network of Centres of Excellence in the field of geomatics. His career has revolved around geographic information systems (GIS). In particular, his research has concentrated on time in GIS, data quality testing and the social and institutional aspects of GIS. From 1987-2004, he was professor of geography at the University of Washington, and served on the Mapping Sciences Panel of the NRC for a three-year term. From 1982-87 he was assistant professor in the Department of Landscape Architecture at the University of Wisconsin-Madison. In this position, he participated in a multidisciplinary team applying GIS to soil erosion planning. This project helped provide the impetus for the Wisconsin Land Records Committee that led to the current state-wide GIS program. From 1972-82 he was a programmer at the Harvard Lab for Computer Graphics. He participated in the design of prototype GIS software. For 30 years, his writing has tried to connect the technical details of GIS to larger issues of philosophy and culture. Recently, he has joined in an interdisciplinary group that studies the interactions of science, technology, and society. He graduated magna cum laude with a B.A. degree in geography from the University of Massachusetts–Amherst, and he has a Ph.D. degree in geography from the University of Bristol, England.
Carl J. Dahlman
Georgetown University

CARL J. DAHLMAN is the Henry R. Luce professor of international relations and information technology at Georgetown University’s Edmund A. Walsh School of Foreign Service. He previously had more than 25 years of service at the World Bank, where he served as senior advisor to the World Bank Institute. In this role, he managed the Knowledge for Development (K4D) program—an initiative providing training on the strategic use of knowledge for economic and social development to business leaders and policy makers in developing countries. Prior to developing the K4D program, he served as staff director of the 1998-1999 World Development Report, Knowledge for Development. He has also conducted extensive analytical work in major developing countries including Argentina, Brazil, Chile, Mexico, Russia, Turkey, India, Pakistan, China, Korea, Malaysia, Philippines, Thailand, and Vietnam. He is currently finishing a knowledge economy study on Mexico, working on a book on the challenge of the knowledge economy for education and training in China, and collaborating with research teams in Finland, Japan, and Korea to produce books on each country’s development strategies. He served as a member of the NRC Committee on Comparative Innovation Policy: Best Practice in National Technology Programs. He has a B.A. degree in international relations from Princeton University, and a Ph.D. degree in economics from Yale University.
Geoff Davis
Google, Inc.

GEOFF DAVIS is a senior researcher in the User Experience Group at Google. He earned his Ph.D. degree in applied mathematics from New York University’s Courant Institute in 1994 and has since employed his doctoral education in a variety of capacities. He has been an assistant professor in the Mathematics Department at Dartmouth College, the Texas Instruments visiting assistant professor in the Electrical and Computer Engineering Department at Rice University, a researcher in the Signal Processing Group at Microsoft Research, a developer at the San Francisco-based startup company 4charity, a visiting scholar at Sigma Xi: The Scientific Research Society, and a Werthheim fellow at the Labor and Worklife Program at Harvard Law School. Dr. Davis’s mathematical research centered around representations of information, with a particular focus on wavelets and related transforms. His work on image coding led to an NSF Mathematical Sciences Postdoctoral Fellowship and the 2000 IEEE Leon K. Kirchmayer Prize Paper Award. He has had a longstanding interest in science education and policy issues. In 1997 he created the PhDs.org website, which currently receives approximately 200,000 visitors each month. He later ran the National Graduate School Survey with NAPGS and the Sigma Xi Postdoc Survey. He is also a past member of the National Bureau of Economic Research's Science and Engineering Workforce Project.
Katharine G. Frase
International Business Machines Corporation

KATHARINE G. FRASE is vice president of Industry Solutions and Emerging Business at IBM Research. Prior to this she was vice president of Technical and Business Strategy at IBM Software Group (SWG). Her team is responsible for technical strategy, business strategy, business development, standards, competitive analysis, and the application of advanced technologies across SWG. Prior to this role, she was vice president of technology at IBM where she was responsible for technical resources, recognition, assessment, and strategy across IBM. In 2006, in recognition of her distinguished contributions to engineering, she was elected as a member of the National Academy of Engineering (NAE). Earlier IBM responsibilities included management of process development, design/modeling methodology and production for chip carrier assembly, and final test for IBM silicon products. Her research interests include mechanical properties/structural interactions in composites, high temperature superconductors, solid electrolytes (fast ionic conductors), ceramic powder synthetic methods, and ceramic packaging. She chaired an IBM/NAS workshop on lead solder reduction actions and is an ex officio member of the NRC’s Board on Assessment of National Institute of Standards and Technology Programs. She has an A.B. degree in chemistry from Bryn Mawr College, and a Ph.D. degree in materials science and engineering from the University of Pennsylvania.
Barbara M. Fraumeni
University of Southern Maine

BARBARA M. FRAUMENI is associate dean of research, chair of the Ph.D. program, and professor of public policy at the Muskie School of Public Service of the University of Southern Maine. She previously served as chief economist of the Bureau of Economic Analysis and was a research fellow of the Program on Technology and Economic Policy at the John F. Kennedy School of Government at Harvard University. Her areas of expertise and research interests include measurement issues and national income accounting; human and nonhuman capital, productivity, and economic growth; market and nonmarket accounts; investment in education and research and development; and measurement of highway capital stock and the real output of government by function. Currently, she serves on the National Research Council’s Panel on Measuring Higher Education Productivity. She has a B.A. degree from Wellesley College, and a Ph.D. degree from Boston College.
Richard B. Freeman
Harvard University

RICHARD B. FREEMAN is the Herbert Ascherman chair in economics at Harvard University. He is currently serving as faculty director of the Labor and Worklife Program at Harvard Law School. He also directs the National Bureau of Economic Research/Sloan Science Engineering Workforce Projects, and is senior research fellow in labor markets at the London School of Economics’ Centre for Economic Performance. He is a fellow of the American Academy of Arts and Science and is currently serving as a member of the AAAS Initiative for Science and Technology. He served on the study on Policy Implications of International Graduate Students and Postdoctoral Scholars in the United States. He also served on five panels of the National Academy of Sciences, including the Committee on National Needs for Biomedical and Behavioral Scientists. He received the Mincer Lifetime Achievement Prize from the Society of Labor Economics in 2006. In 2007 he was awarded the IZA Prize in Labor Economics. He has a B.A. degree from Dartmouth College, and a Ph.D. degree from Harvard University.
Frederick D. Gault
United Nations University

FREDERICK D. GAULT is a professorial fellow with the United Nations University–Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT). He is also a professor extraordinaire at the Tshwane University of Technology (TUT) in South Africa and a member of the TUT Institute for Economic Research on Innovation (IERI). He worked with the Organization for Economic Cooperation and Development (OECD) as a member of the Management Team coordinating the OECD Innovation Strategy delivered in May 2010. Prior to joining UNU MERIT, he held a visiting fellowship at Canada’s International Development Research Centre (IDRC) in Ottawa and, until April 2008, he was at Statistics Canada as the director of the division responsible for the development of statistics on all aspects of science, technology, and innovation. During this time he was chair of the OECD Working Party of National Experts on Science and Technology Indicators (NESTI) (2002 - 2008) and of the Working Party on Indicators for the Information Society (WPIIS) (1997 - 2002). Before joining Statistics Canada, he was senior lecturer in theoretical physics at the University of Durham in the U.K. He has a Ph.D. degree in theoretical physics and a B.Sc. degree in economics from the University of London. He served on the CNSTAT Panel to Review Research and Development Statistics at the National Science Foundation.
David Goldston
Natural Resources Defense Council

DAVID GOLDSTON is a visiting lecturer at the Harvard University Center for the Environment and director of government affairs at the National Resource Defense Council. Previously, he held a one-year appointment as a lecturer in the Science, Technology, and Environment Program at Princeton University’s Woodrow Wilson School of Public and International Affairs. He also writes the monthly column “Party of One” on Congress and science policy for the journal Nature. From 2001 through 2006, he was the chief of staff of the U.S. House of Representatives Committee on Science, which has jurisdiction over much of the federal research and development budget. He was also a key player in most environmental debates in the House from 1995, when he became legislative director to Representative Sherwood Boehlert of New York, until the end of 2006, when he retired from government service. He has a B.A. degree in American history from Cornell University, and has completed the course work for a Ph.D. degree in U.S. history at the University of Pennsylvania. He is a member of the Aeronautics and Space Engineering Board of the National Academies and a member of the Committee to Organize a Workshop on Measuring Economic and Other Returns on Federal Research Investments under the STEP Board. He served on the CNSTAT Panel on Modernizing the Infrastructure of the NSF Federal Funds for R&D Survey.
Michael Mandel
University of Pennsylvania, Wharton School of

MICHAEL MANDEL is a senior fellow at the Mack Center for Technological Innovation at the Wharton School at the University of Pennsylvania. Until 2009 he was chief economist at Business Week, where he won multiple awards for his stories on the U.S. and global economies. His particular areas of expertise include the link between innovation and growth, and the inability of the official economic statistics to properly measure today's economy. He is a senior fellow at both Wharton’s Mack Center for Technological Innovation and the Progressive Policy Institute in Washington, DC. He has written several books, including an introductory economics text, Economics: The Basics. He has a Ph.D. degree in economics from Harvard University.
John E. Rolph
University of Southern California

JOHN E. ROLPH is a professor emeritus of statistics at the Marshall School of Business of the University of Southern California, where he also holds appointments in the mathematics department and the law school. Previously, he spent 24 years as a statistician at the RAND Corporation, 12 of them as head of the statistical research and consulting group. His areas of expertise include statistics and public policy and empirical Bayes estimation. He served as a member of the National Research Council’s (NRC) Committee on National Statistics (CNSTAT) and as chair of the committee from 1998 to 2004; he has also served on the NRC Committee on Law and Justice. He has served on several NRC panels, on topics including ballistic imaging, statistical and operational test design in defense systems, methods for assessing discrimination, and decennial census methodology. He is an elected member of the International Statistical Institute, a fellow of the American Statistical Association, a fellow of the Institute of Mathematical Statistics, and a lifetime national associate of the National Academies. He is a past editor of CHANCE magazine and has served in many other editorial capacities. He has A.B. and Ph.D. degrees in statistics from the University of California, Berkeley.
Leland Wilkinson
SYSTAT Software, Inc.

LELAND WILKINSON is the executive vice president of SYSTAT Software Inc., adjunct professor of statistics at Northwestern University, and adjunct professor of computer science at the University of Illinois Chicago. He wrote the SYSTAT statistical package and founded SYSTAT Inc. in 1984. After the company grew to 50 employees, he sold SYSTAT to SPSS and worked there for 10 years on research and development of visualization systems. SPSS eventually sold SYSTAT to Cranes Software International and then he rejoined SYSTAT in 2008. He is a fellow of the American Statistical Association, an elected member of the International Statistical Institute, and a fellow of the American Association for the Advancement of Science. He has won Best Speaker Award at the National Computer Graphics Association and the Youden Prize for best expository paper in the statistics journal Technometrics. He has served on the Committee on Applied and Theoretical Statistics of the National Research Council and has been vice-chair of the board of the National Institute of Statistical Sciences (NISS). In addition to authoring journal articles, the original SYSTAT computer program and manuals, and patents in visualization and distributed analytic computing, he is the author (with Grant Blank and Chris Gruber) of Desktop Data Analysis with SYSTAT and The Grammar of Graphics. He has an A.B. degree from Harvard University, an S.T.B. degree from Harvard Divinity School, and a Ph.D. degree in psychology from Yale University.

Events



Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
This meeting is closed in its entirety.
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Andy Wyckoff
Mike Mandel
Richard Freeman


The following topics were discussed in the closed sessions:

Discussion of report chapters
Additional assignments and deadlines
Answering key questions


The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
September 04, 2012
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
This meeting is closed in its entirety
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carter Bloch
Nick Chrisman
Geoff Davis
Katharine Frase
Barbara Fraumeni
Richard Freeman
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
Andy Wyckoff

The following topics were discussed in the closed sessions:

International Comparability of STI Indicators
Core NCSES Activities and Related STI Indicators
Exploratory Activities
Prioritization of STI Indicators
Draft Sections of the Final Report
Committee Member Assignments Re: The Final Report
Finalize Recommendations in the Final Report

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
June 26, 2012
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
Wednesday, April 11, 2012

9:00 – 9:15 a.m.
Call to Order and Introductions
Panel Chairs: Bob Litan and Andrew Wyckoff

9:15 – 10:15
Measures of STEM Human Capital: Coverage and Content
Presenter: Emilda Rivers, NSF
Discussants: John Rolph and Nicholas Chrisman (panel members)

10:15 – 10:30
Break

10:30 – 11:30
NCSES’s Administrative Records Project Overview
Presenter: Chris Pece, NSF
Discussants: Fred Gault and Geoff Davis (panel members)

11:30 – 12:00 p.m.
International Sources of STI Indicators: Efficiency Considerations
Presenter: Esha Sinha, CNSTAT
Discussants: Carl Dahlman and Andrew Wyckoff (panel members)

12:00 – 12:30
What Do the Indicators Indicate?
Presenter: Leland Wilkinson (panel member)
Discussants: Bob Litan and David Goldston (panel members)

12:30 p.m.
Planned Adjournment of Open Session

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carter Bloch
Nick Chrisman
Geoff Davis
Katharine Frase
Barbara Fraumeni
Richard Freeman
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
Leland Wilkinson
Andy Wyckoff

The following topics were discussed in the closed sessions:

Wednesday, April 11, 2012
STEM Human Capital
DBASSE Report Review
Science and Technology

Thursday, April 12, 2012
Innovation
Policy Relevance & International Comparability
Prioritization & Recommendations
Other Items


The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
April 13, 2012
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
Tuesday, February 7, 2012. Room 201

11:00 – 11:20am Interim Report Discussion
Topics: Findings and recommendations in the interim report entitled: “Improving Measures of Science, Technology, and Innovation: Interim Report.” Presentation of the study’s focus, findings, and recommendations.Scope of final report.

Chair: David Goldston
Presenters: Bob Litan and Andy Wyckoff (panel chairs)

11:20 – 12:45 PM Science and Engineering Indicators: Updates and Future NCSES Activities
Topics: Highlights of SEI 2012 report and how it differs from 2010 version. Update on BRDIS data analysis. In-house and researcher access to NCSES datasets. Demonstration of Internet delivery of state indicators.

Chair: Bob Litan
Discussants: Nicholas Chrisman and Katharine Frase
Presenters: Rolf Lehming, John Jankowski, Nirmala Kannankutty, and Jeri Mulrow

12:45 – 1:45 Lunch (in room 201)

2:00 – 2:45 Payments and Receipts for R&D Services
Topics: Measuring trade in R&D Services; changes in U.S. R&D outsourcing; the location of the outsourced R&D; and specific countries that are major purchasers of U.S. R&D services.

Chair: Carter Bloch
Discussants: Fred Gault and Michael Mandel
Presenter: Sue Okubo, “Payments and Receipts for R&D Services: Outsourcing of U.S. R&D and Purchases of U.S. R&D services.”

2:45 – 3:00 Break

3:00 – 4:15 Conceptual and Methodological Frameworks for STI Indicators
Topics: Model of evidence-based metrics/indicators, especially service sector R&D; discussion of methodological foundations for developing statistics from data extracted from administrative records and web tools.

Chair: John Rolph
Discussants: Geoff Davis and Lee Wilkinson
Presenter: Adam Jaffe

4:15 – 5:00 Measuring Intellectual Property
Topic: New data developments at the U.S. Patent and Trademark Office and international governing bodies for patents, copyrights, and trademarks.

Chair: Andy Wyckoff
Discussants: Richard Freeman and Bob Litan
Presenter: Stuart Graham

5:00pm Adjournment
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carter Bloch
Nick Chrisman
Carl Dahlman
Geoff Davis
Katharine Frase
Barbara Fraumeni
Richard Freeman
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
Leland Wilkinson
Andy Wyckoff

The following topics were discussed in the closed sessions:

Tuesday, February 7, 2012
Interim and final report
Strategize for following day's meeting

Wednesday, February 8, 2012
Recap of previous day's meeting
Study's methodology
Final report chapters
Study's recommendations
Analysis of existing STI indicators
Planning future panel meetings

The following materials (written documents) were made available to the committee in the closed sessions:

none

Date of posting of Closed Session Summary:
February 09, 2012
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
Monday, September 26, 2011
Room 201

1:30 – 3:00pm Conceptual Framework for STI Indicators
Adam Jaffe, Brandeis University
Michelle Alexopoulos, University of Toronto

3:00 – 3:15 Break

3:15 – 5:00
National Center for Education Statistics, Tom Weko, Dan McGrath, Jessica Shedd, and Matt Soldner
U.S. Census Bureau, Erika McEntarfer
U.S. Bureau of Labor Statistics, Laurie Salmon

5:00pm Adjournment
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carter Bloch
Nicholas Chrisman
Carl Dahlman
Geoff Davis
Katharine Frase
Barbara Fraumeni
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
Leland Wilkinson
Andy Wyckoff

The following topics were discussed in the closed sessions:

Monday, September 26, 2011
8-9am Organization of day's events
9am-12:15pm Group discussions with federal agencies on state of agency data collection and distribution

Tuesday, September 27, 2011
9:00am-3:00pm Discussion on development of draft of interim report; future meeting dates

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
September 27, 2011
Publication(s) resulting from the event:

-


Location:

20 F Street NW
Conference Center
Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
DAY 1: Monday, July 11, 2011

8:00-9:00am REGISTRATION IN LOBBY (Breakfast available in main foyer)

9:00-9:10 WELCOME AND OVERVIEW
Chairs: Robert Litan (Kauffman Foundation) and Andrew Wyckoff (OECD)
• Connie Citro (National Academies/CNSTAT)
• Steve Merrill (National Academies/STEP)

9:10-10:35 SESSION I: NEXT-GENERATION STI STATISTICS—FRAMEWORKS AND DATA
Objective: Identify what the federal statistical system can produce now regarding science, technology and innovation trends. Specific measures of business and university inputs and outputs, and related outcomes and impacts are welcomed. What can be done now with existing surveys and administrative data? What do users want that requires new methods of gathering and disseminating data (types of data, linkages of agency surveys and periodicity)? What should NCSES produce to meet demand?
Chair: Bob Litan (Kauffman Foundation)
Discussant: John Rolph (USC)
Presenters:
• John Haltiwanger (U. Maryland)
• Matthieu Delescluse (European Commission)
• Alicia Robb (Kauffman Foundation)
• Stefano Bertuzzi (NIH)

10:35-10:45 BREAK (refreshments available in lobby)

10:45-12:45pm SESSION II: INTERNATIONAL STI INDICATORS—RECENT DEVELOPMENTS
Objective: Identify recent developments in measuring STI and what is currently planned for the future. Discussion should reveal what has been successfully and unsuccessfully measured. What are critical bottlenecks and perceived opportunities? Policy relevance of indicators is key. What global STI metrics and indicators should NCSES develop in the near and medium term (the next 5-10 years)?
Chair: Michael Mandel (U. Penn)
Discussant: Andrew Wyckoff (OECD)
Presenters:
• Jonathan Haskel (Imperial College Business School, UK)
• Brian MacAulay (NESTA, UK)
• Hugo Hollanders (UNU-MERIT, Netherlands)
• Shinichi Akaike (Institute of Innovation Research, Hitotsubashi University, Japan)
• Cheonsik Woo (Korean Development Institute)

12:45-1:45 LUNCH (in main foyer)

1:45-3:45 SESSION III: NEXT-GENERATION STI STATISTICS—FRONTIER METHODS
Objective: Identify frameworks and tools beyond survey instruments that yield measurements of research and commercialization productivity. Details on how a statistical agency can utilize these tools are key. Which tools are ripe for applications that NCSES should use to produce new STI indicators?
Chair: Geoff Davis (Google)
Discussant: Richard Freeman (Harvard U.)
Presenters:
• Erik Brynjolfsson (MIT)
• Lee Giles (Penn State U.)
• Carl Bergstrom (U. Washington)
• Richard Price (Academia.edu)

3:45-4:00 BREAK (refreshments available in lobby)

4:00-5:00 SESSION IV: ROUNDTABLE: INDUSTRY, ACADEMIC AND GOVERNMENT PERSPECTIVES
Objective: Identify what firms, universities and statistical agencies can be expected to contribute to data inputs for STI indicators. Determine new uses for STI indicators at firms, particularly multinationals. Establish what policymakers and university SPO/technology transfer managers need to know in their respective decision-making processes. Where will the indicators be used and why, and why have they not already been developed? What are the new data inputs and new statistical outputs that should be the laser focus for NCSES in the 5-10 years?
Chairs: Barbara Fraumeni (U. Southern Maine)
Discussants:
• Nick Donofrio (IBM)
• Richard Freeman (Harvard U.)
• David Goldston (Natural Resources Defense Council)

5:00-5:10 WRAP-UP
• Chairs: Robert Litan and Andrew Wyckoff
• Study Director: Kaye Husbands Fealing (National Academies/CNSTAT)

5:10-7:30 RECEPTION (in main foyer)

7:30pm ADJOURN


DAY 2: Tuesday, July 12, 2011

8:00-8:30am REGISTRATION IN LOBBY (Breakfast available in main foyer)

8:30-10:45 SESSION V: INTERNATIONAL STI INDICATORS—NEW REGIONS
Objective: Identify new STI indicators initiatives. Includes presentations on emerging economies’ measurement of STI diffusion and impacts. Discussion of service sector measures and measures of design activities. What indicators should NCSES develop to measure technological diffusion and design?
Chair: Carl Dahlman (Georgetown U.)
Discussant: Fred Gault (UNU-Merit)
Presenters:
• Howard Alper (Science, Technology and Innovation Council, Canada)
• Changlin Gao (Chinese Academy of Science and Technology for Development)
• Philippe Mawoko (NEPAD, South Africa)
• Gustavo Crespi (Inter American Development Bank; Uruguay)
• Jayanta Chatterjee (Indian Institute of Technology)

10:45-11:00 BREAK (refreshments available in lobby)

11:00-12:45pm SESSION VI: SUB-NATIONAL STI INDICATORS
Objective: Identify state and regional indicators of entrepreneurial activities and hot-spots of innovation. What indicators should NCSES develop to measure state and regional STI and diffusion activities?
Chair: Lee Wilkinson (SYSTAT)
Discussant: David Goldston (Natural Resources Defense Council)
Presenters:
• Andrew Reamer (George Washington U.)
• Robert Atkinson (Information Technology and Innovation Foundation)
• Maryann Feldman (North Carolina)
• David Winwood (APLU presenter)/ Robert Samors & Dana Bostrom (by telephone for Q&A)

12:45-12:50 Wrap-up
Chairs: Robert Litan (Kauffman Foundation) and Andrew Wyckoff (OECD)

12:50-2:00 LUNCH (in main foyer)

2:00pm ADJOURN
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carl Dahlman
Geoff Davis
Barbara Fraumeni
Richard Freeman
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
leland Wilkinson
Andy Wyckoff

The following topics were discussed in the closed sessions:

Structure of day's events
Future panel meetings
Summary of meeting
Next steps

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
July 13, 2011
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  202.334.3266

Agenda
Goals for the First Meeting:

This is the first open meeting of the panel to study the status of the science, technology, and innovation (STI) indicators that are currently developed and published by the National Science Foundation’s National Center for Science and Engineering Statistics (NCSES). The study addresses indicators development by NCSES in its role as a U.S. federal statistical agency charged with providing balanced, policy relevant but policy-neutral information to the President, federal executive agencies, the National Science Board, the Congress, and the public. The major focus is to assess and provide recommendations for revised, refocused, and newly developed indicators that better reflect fundamental and rapid changes reshaping the global STI systems.

The purpose of this meeting is threefold. First, we will establish the charge to the panel and hear from a variety of users of STI metrics and indicators what their perceived needs are in the near and long term. Second, the panel will determine the scope of the study, including the dimensions on which the panel will deliberate with the view to prioritized recommendations to NCSES when the study reaches conclusion. Finally, the panel will develop plans for follow-on activities, including its first workshop. Panel deliberations will take place in open and closed sessions as indicated on the agenda.

Thursday, April 14, Room 109
OPEN SESSION (open to public); 8:30am – 5:15pm

8:30am Welcome, Committee Introductions, Overview of Agenda
• Bob Litan and Andy Wyckoff, Co-Chairs
• Connie Citro, Director, Committee on National Statistics
• Kaye Husbands Fealing, Study Director

9:00 – 10:15 Data Users (Government)
Objective: Who are the key users, and what do they want?
• Presenters: Kei Koizumi, Assistant Director for Federal R&D, OSTP; Mark Doms, Chief Economist, Commerce Department; Phillip Singerman, Associate Director for Innovation and Industry, NIST; Gregory Tassey, Senior Economist, NIST
• Respondents: Bob Litan, Richard Freeman, David Goldston, Fred Gault

10:15 – 10:30 Break

10:30 – 11:30 Existing NCSES Indicators and Current Efforts
Objectives: What is the current state of NCSES indicators and what new indicator development is underway, broken down by time frame – short-, medium, and long-term? What are the main positive remarks and negative criticisms leveled at these indicators?
• Rolf Lehming, Program Director, NCSES; Lynda Carlson, Director, NCSES
• Q&A; panel member’s predetermined questions

11:30 – 12:30pm Private and Academic Sectors/Regional Clusters
Objective: What can businesses and academics do to more accurately report RD&E? What do they want from indicators now and in the future?
• Presenters: Katharine Frase (by phone) and Barbara Fraumeni
• Respondents: Bob Litan, Mike Mandel, Leland Wilkinson

12:30 – 1:30 Lunch

1:30 – 2:45 Drivers and Trends
Objective: What are the changes (existing, emerging, and expected) that are reshaping STI?
• Presenter: Michael Mandel
• Respondents: Andy Wyckoff, Bob Litan, Fred Gault, Nick Chrisman, Geoff Davis

2:45 – 3:30 Data Collection/Real-Time Data
Objective: Can we identify the creative experimentation underway (wherever) that can address user needs, answer criticisms, and position NSF indicators so that they reflect the changes that are underway and that we know are coming?
• Presenter: Richard Freeman
• Respondents: John Rolph, Nick Chrisman, Geoff Davis

3:30 – 3:45 Break

3:45 – 4:30 International Indicators
Objective: Why is it important to look at what is happening with indicators on a global level? What other indicators are being developed, and where, and how can they be utilized?
• Presenter: Fred Gault
• Respondents: Andy Wyckoff, Carter Bloch

4:30 – 5:15 Administrative Data and Policy Needs
Objective: How can NCSES use administrative data to improve existing indicators, or develop new ones?
• Presenter: David Goldston
• Respondents: John Rolph, Leland Wilkinson, Barbara Fraumeni

5:15pm Adjournment of Open Session

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Carter Bloch
Nick Chrisman
Geoff Davis
Barbara Fraumeni
Fred Gault
David Goldston
Bob Litan
Mike Mandel
John Rolph
Leland Wilkinson
Andy Wyckoff

The following topics were discussed in the closed sessions:

Friday, April 15

9:00am Call to Order
Bob Litan and Andy Wyckoff, Co-Chairs
Kaye Husbands Fealing, Study Director

9:00 – 9:45 Discussion of Potential Bias and Conflict of Interest
Connie Citro, Director, CNSTAT

9:45 – 12:00pm Reactions to Previous Day’s Discussion
Statement of Task; Is the scope broad enough? What are the big fixes?
Panel Membership. Is it desirable to add panel members in areas not sufficiently covered?
Discussion of 1st workshop goals, objectives, and focus areas. Presenters (Andrew Reamer on government users; Robert Samors on regions and university-industry connections; Tyler Cowen on invention and innovation, etc.). Workshop presenters chosen based on what NCSES needs and what panel considers priority ‘to do’ list.

12:00 – 1:00 Working Lunch to continue discussions
Developing Plans for Follow-On Activities
Commissioned papers (list of organizations and people)
Preliminary report to NCSES in December 2011 (format—short letter, brief report, etc.)

1:00 – 1:30 Continued Discussions on Follow-On Activities
Tentative date for fall meeting—early October 2011
Committee responsibilities and assignments
Research activities. Links to databases and reading materials (on portal). Reports, unpublished papers, conferences, etc. (Matt—NAS researcher)

1:30pm Adjournment


The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
April 15, 2011
Publication(s) resulting from the event:

-

Publications