Public Access Records Office
The National Academies
500 5th Street NW
Room KECK 219
Washington, DC 20001
Tel: (202) 334-3543
Email: paro@nas.edu
Project Information

Project Information


Reproducibility and Replicability in Science


Project Scope:

The National Academies of Sciences, Engineering, and Medicine will empanel an ad hoc committee of approximately 13 – 14 experts who will assess research and data reproducibility and replicability issues, with a focus on topics that cross disciplines. 

The committee will:

1.     provide definitions of "reproducibility" and "replication" accounting for the diversity of fields in science and engineering,

2.     assess what is known and, if necessary, identify areas that may need more information to ascertain the extent of the issues of replication and reproducibility in scientific and engineering research,

3.     consider if the lack of replication and reproducibility impacts the overall health of science and engineering as well as the public’s perception of these fields,

4.     review current activities to improve reproducibility and replication,

5.     examine (a) factors that may affect reproducibility or replication including incentives, roles and responsibilities within the scientific enterprise, methodology and experimental design, and intentional manipulation; (b) as well as studies of conditions or phenomena that are difficult to replicate or reproduce,

6.     consider a range of scientific methodologies as they explore research and data reproducibility and replicability issues, and

7.     draw conclusions and make recommendations for improving rigor and transparency in scientific and engineering research and will identify and highlight compelling examples of good practices.

While the committee may consider what can be learned from past and ongoing efforts to improve reproducibility and replication in biomedical and clinical research, the recommendations in the report will focus on research in the areas of science, engineering and learning that fall within the scope of the National Science Foundation.

In addressing the tasking above, the committee may consider the following questions:

·       Using definitions of “reproducibility” and “replicability” endorsed by the committee, explore what it means to successfully reproduce/replicate in different fields?  Which issues (e.g. perhaps pressures to publish, inadequate training) are common across all or most fields when there are failures to replicate results?

·       What is the extent of the absence of reproducibility and replication? Is there a framework that outlines the various reasons for lack of reproducibility and replication of a study?

·       What strategies have scientists employed other than replication/reproducing findings to gain confidence in scientific findings (e.g., in situations where replication/reproducing is not possible, such as studies of ephemeral phenomena), and what are the advantages/shortcoming of those approaches?

·       What cost-effective reforms could be applied?  Where would they be best applied?  What would their anticipated impact be?

Early in the process and throughout the study, scientific and engineering societies, communication experts, scientific tool developers, and other stakeholders will be engaged in the work of the committee as part of the data gathering process.  These same stakeholder groups will be tapped at the end of the study in the planned release event, to ensure a wide distribution of the report.

Status: Current

PIN: DBASSE-BBCSS-17-03

Project Duration (months): 18 month(s)

RSO: Heimberg, Jennifer

Topic(s):

Behavioral and Social Sciences
Biology and Life Sciences
Earth Sciences
Education
Engineering and Technology
Math, Chemistry, and Physics
Surveys and Statistics



Geographic Focus:
North America

Committee Membership

Committee Post Date: 11/22/2017

Harvey V. Fineberg - (Chair)
Harvey V. Fineberg (NAM) is president of the Gordon and Betty Moore Foundation. He previously served as the Presidential Chair of the University of California, San Francisco; president of the Institute of Medicine; provost of Harvard University; and dean of the Harvard School of Public Health. In the fields of health policy and medical decision-making, his past research has focused on the process of policy development and implementation, assessment of medical technology, evaluation and use of vaccines, and dissemination of medical innovations. Dr. Fineberg chairs the board of the Carnegie Endowment for International Peace and serves on the boards of the William and Flora Hewlett Foundation and the China Medical Board. He helped found and served as president of the Society for Medical Decision Making and also served as consultant to the World Health Organization. Dr. Fineberg is the recipient of several honorary degrees—the Frank A. Calderone Prize in Public Health, the Henry G. Friesen International Prize in Health Research and the Harvard Medal. Dr. Fineberg is a member of the National Academy of Medicine. He earned his M.D. degree from Harvard Medical School and his M.A. and Ph.D. degrees in public policy from Harvard University's Kennedy School of Government and Graduate School of Arts and Sciences, respectively.
David B. Allison
David Allison (NAM) is dean and provost professor school of public health at Indiana University, Bloomington. Previously he was associate dean for research and science in the School of Health Professions at the University of Alabama at Birmingham. Dr. Allison's research interests include obesity and nutrition, quantitative genetics, clinical trials, statistical and research methodology, and research rigor and integrity. He has authored over 500 scientific publications and edited five books. National Academy of Medicine of the National Academies He was elected as a Fellow of the American Statistical Association (ASA) in 2007, the American Psychological Association (APA) in 2008, the American Association for the Advancement of Science (AAAS) in 2009, the NY Academy of Medicine in 2014, the Gerontological Society of America in 2014, the Academy of Behavioral Medicine Research in 2017, and inducted into the Johns Hopkins Society of Scholars in 2013. He has won several awards, including the 2002 Lilly Scientific Achievement Award from The Obesity Society (TOS), the 2002 Andre Mayer Award from the International Association for the Study of Obesity (IASO), and the National Science Foundation Administered 2006 Presidential Award for Excellence in Science, Mathematics, and Engineering Mentoring (PAESMEM). In 2009, he was awarded the Centrum Award from the American Society of Nutrition (ASN) and the TOPS research achievement award from the Obesity Society. In 2013, he was awarded the Alabama Academy of Science's "Wright A. Gardner" award and the American Society of Nutrition's (ASN) "Dannon Institute Mentorship" award. He serves or has served on the editorial boards of numerous journals. He earned a Ph.D. in Clinical and School Psychology from Hofstra University.
Lorena A. Barba
Lorena A. Barba is associate professor of mechanical and aerospace engineering at George Washington University. Prior to joining GW, she was an assistant professor of mechanical engineering at Boston University and a lecturer/senior lecturer of applied mathematics at University of Bristol, UK. Dr. Barba leads a research group in computational science and fluid dynamics, often crossing disciplinary borders into applied mathematics and aspects of computer science. With a central interest in computational fluid dynamics, she extends her research program into other areas, driven by the motivation of using computational methods and high-performance computing in new fields. One of these is biomolecular physics, where she is developing computer methods for problems in protein electrostatics. Her team works using GPU accelerators and develops parallel algorithms for large-scale computing. Dr. Barba is an Amelia Earhart Fellow of the Zonta Foundation (1999), a recipient of the EPSRC First Grant program (UK, 2007), an NVIDIA Academic Partner award recipient (2011), and a recipient of the NSF Faculty Early CAREER award (2012). She was appointed CUDA Fellow by NVIDIA Corporation (2012) and is an internationally recognized leader in computational science and engineering. She received M.Sc. and Ph.D. degrees in aeronautics from the California Institute of Technology.
Dianne Chong
Dianne Chong (NAE) is a former Vice President of (Materials) Assembly, Factory & Support Technology, an organization within Boeing Research & Technology, the company’s central research and development unit. She began working as a Boeing employee in 1986. Chong’s team provides materials and processes engineering and manufacturing support for Boeing, including the company’s two major business units: Boeing Commercial Airplanes and Boeing Defense, Space & Security. Her team is responsible for providing non-destructive evaluation for programs such as the Boeing 787 Dreamliner. In addition, her organization researches and develops advanced assembly and integration concepts. Chong supports many professional societies and serves on several university boards and industry committees. She serves on the National Academy of Sciences’ Board on Global Science and Technology and is a commissioner to the Accreditation Board for Engineering and Technology (ABET). In 2010, she was presented with the Asian-American Executive of the Year Award by the Chinese Institute of Engineers, USA. An expert in metallurgical engineering, she holds Ph.D., Master’s and Bachelor’s degrees from the University of Illinois at Urbana-Champaign. She also holds of an executive Masters degree in Manufacturing Management from Washington University.
Juliana Freire
Juliana Freire is a Professor of Computer Science and Data Science at New York University. She also holds an appointment in the Courant Institute for Mathematical Science and is a faculty member at the NYU Center for Urban Science and Progress. She is the lead investigator and executive director of the NYU Moore-Sloan Data Science Environment, a partnership with UC Berkeley and University of Washington that aims to enhance data-driven discovery by supporting cross-disciplinary academic data scientists. She is elected chair of the ACM Special Interest Group on Management of Data (SIGMOD) and a council member of the Computing Research Association’s Computing Community Consortium (CCC). Her research interests are in large-scale data analysis and integration, visualization, provenance management, and web information discovery. She has made fundamental contributions to data management methods and tools that address problems introduced by emerging applications including urban analytics and computational reproducibility. Freire has published over 170 technical papers, several open-source systems, and is an inventor of 12 U.S. patents. She has co-authored 5 award-winning papers, including one that received the ACM SIGMOD Most Reproducible Paper Award. She is an ACM Fellow and a recipient of an NSF CAREER, two IBM Faculty awards, and a Google Faculty Research award. She has chaired or co-chaired workshops and conferences, and participated as a program committee member in over 70 events. Her research has been funded by the National Science Foundation, DARPA, Department of Energy, National Institutes of Health, Sloan Foundation, Gordon and Betty Moore Foundation, W. M. Keck Foundation, Google, Amazon, AT&T Research, Microsoft Research, Yahoo! and IBM. She has received M.Sc. and Ph.D. degrees in computer science from the State University of New York at Stony Brook.
Gerald Gabrielse
Gerald Gabrielse (NAS), the Board of Trustees Professor at Northwestern University, is one of the world's leading practitioners of fundamental, low energy physics and is a member of the National Academy of Sciences. He is relocating from Harvard to Northwestern to be the founding director of the Center for Fundamental Physics (CFP). An award-winning researcher and teacher, Dr. Gabrielse has chaired both the Harvard Physics Department and the Division of Atomic, Molecular and Optical Physics (DAMOP) of the American Physical Society (APS). He leads the international ATRAP Collaboration at CERN. The Gabrielse research group tested the most precise prediction of the Standard Model of Particle Physics by making the most precise measurement of a property of an elementary particle, tested the Standard Model's most fundamental symmetry to an exquisite precision, made one of the most stringent tests of Supersymmetry and other proposed improvements to the Standard Model, and started low energy antiproton and antihydrogen physics. His many awards and prizes include fellow of the American Physical Society (APS), the Davisson-Germer Prize of the APS, the Lilienfeld Prize of the APS, the Humboldt Research Award (Germany), and the Tomassoni Award (Italy). Harvard University awarded Professor Gabrielse both its George Ledlie Research Prize and its Levenson Teaching Prize. Hundreds of outside lectures include a Källén Lecture (Sweden), a Poincaré Lecture (France), a Faraday Lecture (Cambridge, U.K.), a Schrödinger lecture (Austria), a Zachariasen Lecture (University of Chicago), a Rosenthal Lecture (Yale) and the Hofstadter Lectures (Stanford). He is a member of the National Academy of Sciences. He has a B.S. from Calvin College and an M.S. and a Ph.D. in physics from the University of Chicago.
Constantine Gatsonis
Constantine Gatsonis the Henry Ledyard Goddard University professor of biostatistics, professor of biostatistics, chair of biostatistics, and director of the Center for Statistical Sciences at Brown University. He is a a leading authority on the evaluation of diagnostic and screening tests, and has made major contributions to the development of methods for medical technology assessment and health services and outcomes research. He is a world leader in methods for applying and synthesizing evidence on diagnostic tests in medicine and is currently developing methods for comparative effectiveness research in diagnosis and prediction. Dr Gatsonis currently chairs the Committee on Applied and Theoretical Statistics of the National Academies and is a member of the Committee on National Statistics and the Committee to Evaluate the Department of Veterans Affairs Mental Health Services. Previously, he co-chaired the Committee on the Needs of the Forensic Sciences Community and served on the Board of Mathematical Sciences and Applications and several committees of the National Academy of Medicine. He was the Founding Editor-in-Chief of Health Services and Outcomes Research Methods, and currently serves as Associate Editor of the Annals of Applied Statistics and Academic Radiology, and member of the editorial team of the Diagnostic Test Accuracy Reviews of the Cochrane Collaboration. Dr. Gatsonis was elected fellow of the American Statistical Association and the Association for Health Services Research. He received his Ph.D. in Mathematical Statistics from Cornell UnIversity.
Edward Hall
Edward Hall received his undergraduate degree from Reed College, where he majored in chemistry and philosophy. He earned his Ph.D. in philosophy from Princeton University in 1996; his dissertation focused on conceptual problems in the foundations of quantum mechanics, having to do with the quantum mechanical treatment of the measurement process, and of identical particles. After graduate school, he taught for 11 years in Massachusetts Institute of Technology’s Department of Linguistics and Philosophy, before moving to Harvard in 2005, where he is now the Norman E. Vuilleumier Professor of Philosophy and the Chair of the Philosophy Department. Professor Hall’s philosophical research focuses on the analysis, clarification, and logical interrelationships between a cluster of concepts of central importance across the sciences: causation, probability, laws of nature, counterfactual dependence, confirmation and disconfirmation, statistical inference, realism about unobservable structure, and the nature, formation, and justification of scientific consensus. He has a long-standing ‘semi-professional’ interest in the history of science, and in particular on the conceptual advances that underpinned the scientific revolution of the 17th century, culminating in Newton. Some of his research has focused on conceptual problems in the foundations of so-called “interventionist” approaches to causation and causal inference in statistics; on distinguishing concepts of causation that treat causation as a species of counterfactual dependence from those that treat it as a relation mediated by spatiotemporally continuous processes; on challenges for popular “Humean” accounts of laws of nature, that see such laws as nothing more than pervasive patterns in the physical phenomena; on clarifying the connection between rational degrees of confidence (or “subjective” probabilities) and the kinds of objective probabilities that figure in fundamentally stochastic physical theories; and on articulating basic presuppositions about the natural world that underwrite the possibility of any kind of scientific investigation of that world. Professor Hall’s interest in and approaches to these topics is driven by the conviction that the kind of conceptual clarity that careful philosophical investigation can yield itself constitutes a central and critical kind of scientific progress.
Thomas H. Jordan
Thomas Jordan (NAS) is a University Professor and the W. M. Keck Foundation Professor of Earth Sciences at the University of Southern California. His current research is focused on system-level models of earthquake processes, earthquake forecasting, continental structure and dynamics, and full-3D waveform tomography. As the director of the Southern California Earthquake Center between 2002-2017, Jordan coordinated an international research program in earthquake system science that involved over 1000 scientists at more than 70 universities and research organizations. In 2006, he established the international Collaboratory for the Study of Earthquake Predictability and, since 2006, has been the lead SCEC investigator on projects to create and improve a time-dependent Uniform California Earthquake Rupture Forecast. He has served as a member of the Council of the National Academy of Sciences (2006-2009) and the Governing Board of the National Research Council (2008-2011). Jordan received his Ph.D. from the California Institute of Technology in 1972. He was head of MIT’s Department of Earth, Atmospheric and Planetary Sciences from 1988 to 1998. In 2000, he moved from MIT to USC, and in 2004, he was appointed as a USC University Professor. He has been awarded the Macelwane, Lehmann, and Bowie Medals of the American Geophysical Union, the President's Medal and Woollard Award of the Geological Society of America, and the 2012 Award for Outstanding Contribution to Public Understanding of the Geosciences by the American Geosciences Institute.

Dietram A. Scheufele
Dietram A. Scheufele is the John E. Ross professor in science communication and Vilas Distinguished Achievement professor at the University of Wisconsin-Madison and in the Morgridge Institute for Research. He also holds an Honorary Professorship at the Dresden University of Technology in Germany. Formerly, Dr. Scheufele was on the faculty at Cornell University, a Shorenstein fellow at Harvard University, and a visiting scholar at the Annenberg Public Policy Center of the University of Pennsylvania. His consulting experience includes work for the Public Broadcasting System, the World Health Organization, and the World Bank. Dr. Scheufele’s research deals with the public and political interfaces of emerging science. He is a fellow of the American Association for the Advancement of Science and the Wisconsin Academy of Sciences, Arts & Letters, and a member of the German National Academy of Science and Engineering. He currently serves on the National Academies of Sciences, Engineering, and Medicine’s Division on Earth and Life Studies (DELS) Advisory Committee. He earned a Ph.D. in mass communications at the University of Wisconsin-Madison.
Victoria Stodden
Victoria Stodden is associate professor in the School of Information Sciences at the University of Illinois, Urbana-Champaign. Previously she was assistant professor of statistics at Columbia University where she taught courses in data science, reproducible research, and statistical theory and was affiliated with the Institute for Data Sciences and Engineering. Dr. Stodden is a leading figure in the area of reproducibility in computational science, exploring how we can better ensure the reliability and usefulness of scientific results in the face of increasingly sophisticated computational approaches to research. Her work addresses a wide range of topics, including standards of openness for data and code sharing, legal and policy barriers to disseminating reproducible research, robustness in replicated findings, cyberinfrastructure to enable reproducibility, and scientific publishing practices. She previously served as a co-chair for the NSF Advisory Committee for CyberInfrastructure as well as a former member of the NSF Directorate for Computer and Information Science and Engineering (CISE) Advisory Committee. She also served on the National Academies Committee on Responsible Science: Ensuring the Integrity of the Research Process. She co-edited two books released in 2014—Privacy, Big Data, and the Public Good: Frameworks for Engagement published by Cambridge University Press and Implementing Reproducible Research published by Taylor & Francis. She earned a PhD in statistics and a master of legal studies from Stanford University.
Timothy Wilson
Timothy D. Wilson received his Ph.D. in psychology from the University of Michigan in 1977. He was a faculty member at Duke University from 1977-1979. He joined the Department of Psychology at the University of Virginia in 1979, where he has been ever since. He is currently Sherrell J. Aston Professor of Psychology. He served as Chair of the department from 2001 – 2004. Dr. Wilson has published over 100 articles in scholarly journals and edited books, primarily on the topics of self-knowledge, unconscious processing, affective forecasting, and the applications of social psychology to addressing social problems. His research has been funded by the National Science Foundation, the National Institute of Mental Health, and the Russell Sage Foundation. He has served on numerous editorial boards, including the Board of Reviewing Editors at Science from 2010 – to the present. Dr. Wilson was elected to the American Academy of Arts and Sciences in 2009. In 2013 he received the Donald T. Campbell Award from the Society of Personality and Social Psychology, which recognizes “distinguished scholarly achievement and ongoing sustained excellence in research in social psychology.” In 2015 the Association for Psychological Science awarded Dr. Wilson the William James Fellow Award, to honor a “lifetime of significant intellectual contributions to the basic science of psychology.” In 2015 he received the Thomas Jefferson Award for Scholarship from the University of Virginia, the highest honor given to members of the university community.
Wendy Wood
Wendy Wood is Provost Professor of Psychology and Business at the University of Southern California and currently Distinguished Visiting Chair at INSEAD/Sorbonne. Her research addresses the ways that habits guide behavior—and why they are so difficult to break, as well as evolutionary models of gender differences. From 1982 until 2003, Wood was at Texas A&M University, where she was the Ella C. McFadden Professor of Liberal Arts, the Associate Vice President for Research, and Director of the Women’s Faculty Mentoring Program. In 2004, she moved to Duke University as the James B. Duke Professor of Psychology and Neuroscience and Professor of Marketing. At Duke, Dr. Wood served as Co-Director of the Social Science Research Institute. In 2009, Dr. Wood joined the University of Southern California, where she was Vice Dean of Social Sciences from 2012 to 2016. Dr. Wood is a fellow of numerous scientific societies and is ending a term as President of the Society for Personality and Social Psychology. In the past, she served as editor of the journals, Behavioral Science and Policy, Psychological Review, Journal of Personality and Social Psychology, Personality and Social Psychology Bulletin, and Personality and Social Psychology Review. Her research has been recognized through awards and funding from the National Science Foundation, the National Institute of Mental Health, the Rockefeller Foundation, the Templeton Foundation, and the Radcliffe Institute of Advanced Study.

Committee Membership Roster Comments

(A) The committee will assess research and data reproducibility and replicability issues, with a focus on topics that cross disciplines. The committee will provide definitions of "reproducibility" and "replication" accounting for the diversity of fields in science and engineering. They will assess what is known and, if necessary, identify areas that may need more information to ascertain the extent of the issues of replication and reproducibility in scientific and engineering research. The committee will review current activities to improve reproducibility and replication. The committee will examine factors that may affect reproducibility or replication including incentives, roles and responsibilities within the scientific enterprise, methodology and experimental design, and manipulation; as well as studies of conditions or phenomena that are difficult to replicate or reproduce. The committee will consider a range of scientific methodologies as they explore research and data reproducibility and replicability issues. In its report, the committee will draw conclusions and make recommendations for improving rigor and transparency in scientific and engineering research and will identify and highlight compelling examples of good practices.

(B) Source of Funding: National Science Foundation.

(C) The approximate starting date for the project is 10/01/2017.

(D) A Final Report will be issued at the end of the project in approximately 18 months.

(E) Project Duration: 18 Months.

(F) The committee composition has changed. David Donoho resigned on July 24, 2018 and is no longer a member.

Events


Event Type :  
Meeting

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Lesley Webb
Contact Email:  lwebb@nas.edu
Contact Phone:  (202) 334-1663

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Dianne Chong
Erin Forstag
Ed Hall
Tom Jordan
Lorena Barba
Constantine Gatsonis
Dietram Scheufele
Tim Wilson
Constantine Gatsonis
Wendy wood
Juliana Freire

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Publication(s) resulting from the event:

-

Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Diana Chong
David Allison
Michael Cohen
Juliana Freire
Harvey Fineberg
Dietram Scheufele
Tom Jordan
Timothy Wilson
Lorena Barba
Phone number
Erin Forstag
Gerald Gabrielse
Wendy Wood
Victoria Stodden

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 12, 2018
Publication(s) resulting from the event:

-

Event Type :  
-

Description :   

This meeting will be closed (a committee-only meeting) in its entirety.

Details will be posted after the meeting.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  -
Contact Email:  -
Contact Phone:  -

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Participant List:
Tom Jordan
Gatsonis
Lorena Barba
Scheufele
Simine
Tom Arrison
Ej Hall
Gabrielse
Erin Forstag
Victoria
TDW
Juliana Freire
Michael Cohen

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 04, 2018
Publication(s) resulting from the event:

-

Event Type :  
-

Description :   

This meeting is closed to the committee-only details will be posted after the meeting.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  -
Contact Email:  -
Contact Phone:  -

Supporting File(s)
-
Is it a Closed Session Event?
Yes

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 04, 2018
Publication(s) resulting from the event:

-


Location:

J. Erik Jonsson Woods Hole Center
314 Quissett Ave.
Woods Hole, Massachusetts
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

All members were present physically with the exception of Wendy Wood
who participated via teleconference.

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
August 03, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  202-334-1737

Agenda
Thursday, May 31, 2018

All Times Are Eastern Time Zone

8:30 AM Breakfast available in foyer outside of K100.

______________________________________________________________________________
9:00 – 10:30 AM
OPEN SESSION: Session Open to the Public
Remote connection available via Zoom

Public Sessions will be webcast and recorded via Zoom.
Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/298880874
9:00 AM Welcome and Call to Order
Harvey Fineberg
Committee chair
President of the Gordon and Betty Moore Foundation

9:15 Panel: International Perspectives on Reproducibility and Replication in Science and Engineering
Each panelist will have 15 minutes to address the set of questions below followed by 5 minutes of questions from the committee

• Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
• Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
• Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
• Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?
Global Young Academy (GYA):
Laura Fierce, Brookhaven National Laboratory and member of the GYA Executive Committee [in person]

and Koen Vermeir, French National Centre for Scientific Research (CNRS), former co-chair of the GYA’s Scientific Excellence and Open Science programs and member of the GYA Executive Committee [via Zoom]

Harry Xia, President, Alliance for Scientific Editing in China [in person]

Suman Chakraborty, Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, India [via Zoom]

10:30 End Public Session

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 1 and in the morning session of Day 2)

The following topics were discussed in the closed sessions:

• review of the open session’s briefings
• updates to the draft report and writing assignments
• the report’s main conclusions and recommendations
• plans for the committee’s writing meeting in July


The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee.

Date of posting of Closed Session Summary:
June 12, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Web-based meeting via Zoom
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  202-334-1737

Agenda
May 9, 2018
12:00 – 1:30 PM
OPEN SESSION: Session Open to the Public via Zoom

Public Sessions will be webcast and recorded via Zoom.
Join from PC, Mac, Linux, iOS or Android: Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/7315513143
All Times Are Eastern Time Zone

12:00 PM Welcome and Introductions
Harvey Fineberg
Committee chair
President of the Gordon and Betty Moore Foundation

12:10 Panel: Perspectives on Reproducibility and Replication of Results in Climate Science
Each panelist will have 15 minutes to address the set of questions below followed by 5 minutes of
questions form the committee

• How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
• Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
• Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.
Michael Evans, Associate Professor, University of Maryland
Gavin Schmidt, Director of the Goddard Institute for Space Studies, National Aeronautics and Space Administration
Rich Loft, Director of the Technology Development Division,
National Center for Atmospheric Research (NCAR)
Andrea Dutton, Assistant Professor, University of Florida

1:25 Wrap-up
Harvey Fineberg
Committee chair

1:30 End Public Session



Committee members in attendance via Zoom:
Harvey Fineberg
Juliana Freire
Lorena Barba
Ned Hall
Dianne Chong
Gerald Gabrielse
Simine Vazire
Timothy Wilson

Supporting File(s)
-
Is it a Closed Session Event?
No

Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  202-334-1737

Agenda
April 18, 2018

12:10 – 5:00 PM
OPEN SESSION: Session Open to the Public
Keck Center, Room 101
Public Sessions will be webcast and recorded via Zoom.
Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/817146422


12:10 PM Welcome and Introductions
Harvey Fineberg
Committee chair
President of the Gordon and Betty Moore Foundation

12:25 Perspectives on Scientific Progress and Irreproducibility
Richard Shiffrin, Indiana University

12:55 Panel 1: Reproducibility in the Physical and Earth Sciences
(30 minutes each, 20 minutes + 10 minutes for questions)
Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of
Chemical Engineering, The University of Texas at Austin
Peter Mohr, National Institute of Standards and Technology (NIST)

1:55 BREAK

2:10 Panel 2: Reproducibility in Industry and Industrial Engineering
(20 minutes each, 15 for briefing + 5 minutes for questions)
Carl Ascoli, Chief Science Officer, Rockland Immunochemicals
William Lyons, Director, Global Research & Development Strategy for the
Global Technology Organization, Boeing Research & Technology

2:50 Introduction to Economics and Reproducibility
Daniel L. Goroff
Vice President and Program Director
The Alfred P. Sloan Foundation

3:20 Panel 3: The Economics of Addressing Reproducibility Issues in Science
(4 panelists, each with 15 minutes + 5 minutes for questions from committee)
Heidi Williams, Massachusetts Institute of Technology
Myron P. Gutmann, University of Colorado in Boulder
Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University
Brent Goldfarb, University of Maryland

4:40 Public Comment

5:00 End Public Session

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

• the structure of the draft report and writing assignments
• the report’s main conclusions and recommendations
• review of the open session’s briefings


The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee in closed session.

Date of posting of Closed Session Summary:
April 24, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  202-334-1737

Agenda
THURSDAY, February 22, 2018

Public Sessions will be webcast and recorded via Zoom.
Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/278773591


12:00 – 4:45 PM
OPEN SESSION: Session Open to the Public
Keck Center, Room 101

12:00 Welcome and Introductions
Harvey Fineberg
Committee chair
President of the Gordon and Betty Moore Foundation

12:15 Perspectives on Reproducibility and Replication: American Economic Association
Margaret Levenstein
University of Michigan
Director, Inter-university Consortium for Political and Social Research (ICRPSR)

The speaker has been asked to focus on the following questions:
• Within economics, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results?
• Are there specific areas within economics that are more likely to have issues with reproducing scientific results?
• What reproducibility challenges does economics face with cross disciplinary research?

12:45 Panel 1: Overview of Extent of Reproducibility Issues in Scientific and Engineering Research
(15 minutes each + 15 minute total for Q&A)
John Ioannidis
C.F. Rehnborg chair in Disease Prevention, Co-Director, Meta-Research Innovation Center
Stanford University

Brian Nosek
Director for the Center for Open Science
Professor at the University of Virginia

Daniel Sarewitz
Co-Director, Consortium for Science, Policy & Outcomes, Professor of Science and Society
Arizona State University


THURSDAY, February 22, 2018 (continued)

The panelists have been asked to focus on the following session questions:
• How extensive is the lack of reproducibility in research results in science and engineering, in general?
• At what level does a lack of reproducibility become a problem for the wellbeing of science or engineering?
• Does the lack to reproduce scientific results impact the public perception of specific scientific fields and/or science and engineering in general?

13:45 Panel 2: Reproducibility Issues in Computational Sciences and Statistics (Part 1)
(15 minutes each + 15 minutes Q&A)

The panelists in Panel 2 (Part 1) have been asked to address the session questions with a focus on the management of computational code and data.
David Madigan
Executive Vice President and Dean of the Faculty of Arts and Sciences
Columbia University

Arjun Kumar Manrai
Harvard University
14:30 BREAK, 15 minutes

14:45 Panel 2: Reproducibility Issues in Computational Sciences and Statistics (Part 2)
(15 minutes each + 15 minutes Q&A)
The panelists in Panel 2 (Part 2) have been asked to address the session questions with a focus on the impact of the misuse of statistics in research.
Giovanni Parmigiani
Harvard University and Dana Farber Cancer Institute

Steven Goodman
Professor of Medicine and Epidemiology
Associate Dean for Clinical and Translational Research
Co-Director, Meta-research Innovation Center at Stanford (METRICS)
Stanford University

15:30 Panel 3: Reproducibility Issues in Economics and Social Science
(15 minutes each + 15 minutes Q&A)
The panelists in Panel 3 have been asked to address the session’s questions as they relate to economics, social sciences, and psychology.
Paul L. Joskow
Elizabeth and James Killian Professor of Economics, Emeritus
Department of Economics
Massachusetts Institute of Technology

Arthur “Skip” Lupia
Hal R Varian Collegiate Professor
University of Michigan

Joseph Simmons
University of Pennsylvania

16:30 Public Comments

16:45 End Public Session




FRIDAY, February 23, 2018

Public Sessions will be webcast and recorded via Zoom.
Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/275369777


9:00 – 10:45 AM
OPEN SESSION: Session Open to the Public
Keck Center, Room 101

09:00 Welcome and Introductions
Harvey Fineberg
Committee chair

09:15 Panel 5: Reproducibility Issues in Engineering
(15 minutes each + 10 minutes Q&A)
Gianluca Setti
Politecnico di Torino, Italy
IEEE editor
09:40 Reporting of Reproducibility Issues in Science
(15 minutes each + 15 minutes Q&A)

Christie Aschwanden
Lead science editor
FiveThirtyEight

Laura Helmuth
Science editor
Washington Post

10:25 Public Comments

10:45 End Public Session


Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis (via Zoom)
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

The committee:
reviewed its statement of task,
discussed possible topics for commissioned papers,
discussed initial report outlines, and
discussed possible topics for future meetings.


The following materials (written documents) were made available to the committee in the closed sessions:

No documents were made available to the committee in the closed sessions.

Date of posting of Closed Session Summary:
February 23, 2018
Publication(s) resulting from the event:

-


Location:

National Academy of Sciences Building
2101 Constitution Ave NW, Washington, DC 20418
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  202-334-1737

Agenda
FINAL AGENDA
All Times Are US Eastern Time Zone (UTC – 5:00)

Tuesday, December 12, 2017

Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/361396182

12:30 Welcome and Introductions
Mary Ellen O’Connell, Executive Director, Division on Behavioral, Social Science, and Education
Harvey Fineberg, Committee Chair and President of the Gordon and Betty Moore Foundation

1:00 The Scientific Enterprise
Edward (Ned) Hall, Chair, Department of Philosophy, Harvard University and committee member
Q&A: Harvey Fineberg, Moderator

1:45 National Science Foundation’s Interests and Goals for the Study
Dr. Joan Ferrini-Mundy, Chief Operating Officer, NSF (confirmed)
Q&A: Harvey Fineberg, Moderator

2:30 BREAK


Perspectives on Reproducibility and Replication: Scientific Societies, Part I
Panelists, primarily leaders from scientific societies and organizations, have been asked to focus on the following topics:
• Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results?
• Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
• What reproducibility challenges does your field of science face with cross disciplinary research?

Each panelist has 10 minutes for a presentation + 5 minutes for questions from the committee.

2:45 Behavioral and Social Sciences
William G. Jacoby, Professor, Department of Political Science, Michigan State University and Editor, American Journal of Political Science (confirmed)
Howard S. Kurtzman, Acting Executive Director, Science Directorate, American Psychological Association (confirmed)
Felice J. Levine, Executive Director and Ethics Officer, American Educational Research Association (confirmed)
Q&A: Harvey Fineberg, Moderator

3:30 Physical Sciences
Kate Kirby, CEO, American Physical Society (confirmed)
David Sholl, John F. Brock III School Chair, School of Chemical & Biomolecular Engineering, Georgia Institute of Technology (confirmed)
Q&A: Harvey Fineberg, Moderator

4:00 Statistics
Ron Wasserstein, executive director, American Statistical Association (confirmed)
Q&A: Harvey Fineberg, Moderator

4:15 Earth Sciences
Brooks Hanson, Senior Vice President Publications, American Geophysical Union (confirmed)
Q&A: Harvey Fineberg, Moderator

4:30 Engineering
Philip DiVietro, Managing Director of Publishing, American Society of Mechanical Engineers (ASME) (confirmed)
John Baillieul, Distinguished Professor of Engineering, Professors of Mechanical, of Electrical and Computer Engineering, and of Systems Engineering, Boston University (confirmed)
Q&A: Harvey Fineberg, Moderator

5:00 Public Comments

5:30 End Public Session


Wednesday, December 13, 2017

Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/882788432

8:30 am Welcome, Day One and Day Two overviews
Harvey Fineberg, chair

Perspectives on Reproducibility and Replication: Scientific Societies and Agencies, Part II
Panelists, primarily leaders from scientific societies and organizations, have been asked to focus on the following topics:
• Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results?
• Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
• What reproducibility challenges does your field of science face with cross disciplinary research?

Each panelist has 10 minutes for a presentation + 5 minutes for questions from the committee.


8:45 Life Sciences
Yvette Seger, Director of Science Policy, Federation of American Societies for Experimental Biology (FASEB) (confirmed)

9:00 Reproducibility of scientific research within the agencies:
Patricia Valdez, NIH Extramural Research Integrity Officer (confirmed)
Anne Plant, NIST Fellow in the Biosystems and Biomaterials Division (confirmed)
Q&A: Harvey Fineberg, Moderator

International Panelists have been asked to focus on the following topics:
• What is the level of awareness, interest, concern, and involvement in reproducibility and replicability (R&R) of research results within your national scientific societies?
• Are there specific areas of science that are more likely to have issues with reproducing scientific results?
• What R&R issues exist for cross disciplinary research?
Each panelist has 15 minutes for a presentation + 5 minutes for questions from the committee.

9:30 International Perspectives:
Eric-Jan Wagenmakers, professor of psychology, University of Amsterdam,
and Jean Phillipe de Jong, The Dutch Royal Society of Sciences (KNAW) (confirmed)
Q&A: Harvey Fineberg, Moderator

10:00 BREAK

10:25 Reporting of Reproducibility Issues in Science
Richard Harris, Science Correspondent, National Public Radio


Perspectives from Editors of Cross-disciplinary Journals
Questions for journal editors:
• Can journals assess levels of R&R across science?
• What R&R challenges does cross-disciplinary research pose that can be addressed by journals?
• Are cross-disciplinary papers handled differently from “pure” science papers in terms of peer review or publishing decisions?
• What R&R challenges does cross-disciplinary research pose that can be addressed by journals?


11:05 Veronique Kiermer, Executive Editor, PLOS (confirmed)
Q&A: Harvey Fineberg, Moderator

11:45 Public Comments

12:15 PM End Public Session
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele (via Zoom)

The following topics were discussed in the closed sessions:

The committee held its bias and conflict of interest discussion and reviewed its statement of task. Future meeting dates and plans for the study were also discussed.

The following materials (written documents) were made available to the committee in the closed sessions:

No written documents were provided to the committee.

Date of posting of Closed Session Summary:
December 15, 2017
Publication(s) resulting from the event:

-

Publications

  • Publications having no URL can be seen at the Public Access Records Office
Publications

No data present.