Public Access Records Office
The National Academies
500 5th Street NW
Room KECK 219
Washington, DC 20001
Tel: (202) 334-3543
Email: paro@nas.edu
Project Information

Project Information


Reproducibility and Replicability in Science


Project Scope:

The National Academies of Sciences, Engineering, and Medicine will empanel an ad hoc committee of approximately 13 – 14 experts who will assess research and data reproducibility and replicability issues, with a focus on topics that cross disciplines. 

The committee will:

1.     provide definitions of "reproducibility" and "replication" accounting for the diversity of fields in science and engineering,

2.     assess what is known and, if necessary, identify areas that may need more information to ascertain the extent of the issues of replication and reproducibility in scientific and engineering research,

3.     consider if the lack of replication and reproducibility impacts the overall health of science and engineering as well as the public’s perception of these fields,

4.     review current activities to improve reproducibility and replication,

5.     examine (a) factors that may affect reproducibility or replication including incentives, roles and responsibilities within the scientific enterprise, methodology and experimental design, and intentional manipulation; (b) as well as studies of conditions or phenomena that are difficult to replicate or reproduce,

6.     consider a range of scientific methodologies as they explore research and data reproducibility and replicability issues, and

7.     draw conclusions and make recommendations for improving rigor and transparency in scientific and engineering research and will identify and highlight compelling examples of good practices.

While the committee may consider what can be learned from past and ongoing efforts to improve reproducibility and replication in biomedical and clinical research, the recommendations in the report will focus on research in the areas of science, engineering and learning that fall within the scope of the National Science Foundation.

In addressing the tasking above, the committee may consider the following questions:

·       Using definitions of “reproducibility” and “replicability” endorsed by the committee, explore what it means to successfully reproduce/replicate in different fields?  Which issues (e.g. perhaps pressures to publish, inadequate training) are common across all or most fields when there are failures to replicate results?

·       What is the extent of the absence of reproducibility and replication? Is there a framework that outlines the various reasons for lack of reproducibility and replication of a study?

·       What strategies have scientists employed other than replication/reproducing findings to gain confidence in scientific findings (e.g., in situations where replication/reproducing is not possible, such as studies of ephemeral phenomena), and what are the advantages/shortcoming of those approaches?

·       What cost-effective reforms could be applied?  Where would they be best applied?  What would their anticipated impact be?

Early in the process and throughout the study, scientific and engineering societies, communication experts, scientific tool developers, and other stakeholders will be engaged in the work of the committee as part of the data gathering process.  These same stakeholder groups will be tapped at the end of the study in the planned release event, to ensure a wide distribution of the report.

Status: Current

PIN: DBASSE-BBCSS-17-03

Project Duration (months): 18 month(s)

RSO: Heimberg, Jennifer



Geographic Focus:
North America

Committee Membership

Committee Post Date: 11/22/2017

Harvey V. Fineberg - (Chair)
Harvey V. Fineberg (NAM) is president of the Gordon and Betty Moore Foundation. He previously served as the Presidential Chair of the University of California, San Francisco; president of the Institute of Medicine; provost of Harvard University; and dean of the Harvard School of Public Health. In the fields of health policy and medical decision-making, his past research has focused on the process of policy development and implementation, assessment of medical technology, evaluation and use of vaccines, and dissemination of medical innovations. Dr. Fineberg chairs the board of the Carnegie Endowment for International Peace and serves on the boards of the William and Flora Hewlett Foundation and the China Medical Board. He helped found and served as president of the Society for Medical Decision Making and also served as consultant to the World Health Organization. Dr. Fineberg is the recipient of several honorary degrees—the Frank A. Calderone Prize in Public Health, the Henry G. Friesen International Prize in Health Research and the Harvard Medal. Dr. Fineberg is a member of the National Academy of Medicine. He earned his M.D. degree from Harvard Medical School and his M.A. and Ph.D. degrees in public policy from Harvard University's Kennedy School of Government and Graduate School of Arts and Sciences, respectively.
David B. Allison
David Allison (NAM) is dean and provost professor school of public health at Indiana University, Bloomington. Previously he was associate dean for research and science in the School of Health Professions at the University of Alabama at Birmingham. Dr. Allison's research interests include obesity and nutrition, quantitative genetics, clinical trials, statistical and research methodology, and research rigor and integrity. He has authored over 500 scientific publications and edited five books. National Academy of Medicine of the National Academies He was elected as a Fellow of the American Statistical Association (ASA) in 2007, the American Psychological Association (APA) in 2008, the American Association for the Advancement of Science (AAAS) in 2009, the NY Academy of Medicine in 2014, the Gerontological Society of America in 2014, the Academy of Behavioral Medicine Research in 2017, and inducted into the Johns Hopkins Society of Scholars in 2013. He has won several awards, including the 2002 Lilly Scientific Achievement Award from The Obesity Society (TOS), the 2002 Andre Mayer Award from the International Association for the Study of Obesity (IASO), and the National Science Foundation Administered 2006 Presidential Award for Excellence in Science, Mathematics, and Engineering Mentoring (PAESMEM). In 2009, he was awarded the Centrum Award from the American Society of Nutrition (ASN) and the TOPS research achievement award from the Obesity Society. In 2013, he was awarded the Alabama Academy of Science's "Wright A. Gardner" award and the American Society of Nutrition's (ASN) "Dannon Institute Mentorship" award. He serves or has served on the editorial boards of numerous journals. He earned a Ph.D. in Clinical and School Psychology from Hofstra University.
Lorena A. Barba
Lorena A. Barba is associate professor of mechanical and aerospace engineering at George Washington University. Prior to joining GW, she was an assistant professor of mechanical engineering at Boston University and a lecturer/senior lecturer of applied mathematics at University of Bristol, UK. Dr. Barba leads a research group in computational science and fluid dynamics, often crossing disciplinary borders into applied mathematics and aspects of computer science. With a central interest in computational fluid dynamics, she extends her research program into other areas, driven by the motivation of using computational methods and high-performance computing in new fields. One of these is biomolecular physics, where she is developing computer methods for problems in protein electrostatics. Her team works using GPU accelerators and develops parallel algorithms for large-scale computing. Dr. Barba is an Amelia Earhart Fellow of the Zonta Foundation (1999), a recipient of the EPSRC First Grant program (UK, 2007), an NVIDIA Academic Partner award recipient (2011), and a recipient of the NSF Faculty Early CAREER award (2012). She was appointed CUDA Fellow by NVIDIA Corporation (2012) and is an internationally recognized leader in computational science and engineering. She received M.Sc. and Ph.D. degrees in aeronautics from the California Institute of Technology.
Dianne Chong
Dianne Chong (NAE) is a former Vice President of (Materials) Assembly, Factory & Support Technology, an organization within Boeing Research & Technology, the company’s central research and development unit. She began working as a Boeing employee in 1986. Chong’s team provides materials and processes engineering and manufacturing support for Boeing, including the company’s two major business units: Boeing Commercial Airplanes and Boeing Defense, Space & Security. Her team is responsible for providing non-destructive evaluation for programs such as the Boeing 787 Dreamliner. In addition, her organization researches and develops advanced assembly and integration concepts. Chong supports many professional societies and serves on several university boards and industry committees. She serves on the National Academy of Sciences’ Board on Global Science and Technology and is a commissioner to the Accreditation Board for Engineering and Technology (ABET). In 2010, she was presented with the Asian-American Executive of the Year Award by the Chinese Institute of Engineers, USA. An expert in metallurgical engineering, she holds Ph.D., Master’s and Bachelor’s degrees from the University of Illinois at Urbana-Champaign. She also holds of an executive Masters degree in Manufacturing Management from Washington University.
Juliana Freire
Juliana Freire is a Professor of Computer Science and Data Science at New York University. She also holds an appointment in the Courant Institute for Mathematical Science and is a faculty member at the NYU Center for Urban Science and Progress. She is the lead investigator and executive director of the NYU Moore-Sloan Data Science Environment, a partnership with UC Berkeley and University of Washington that aims to enhance data-driven discovery by supporting cross-disciplinary academic data scientists. She is elected chair of the ACM Special Interest Group on Management of Data (SIGMOD) and a council member of the Computing Research Association’s Computing Community Consortium (CCC). Her research interests are in large-scale data analysis and integration, visualization, provenance management, and web information discovery. She has made fundamental contributions to data management methods and tools that address problems introduced by emerging applications including urban analytics and computational reproducibility. Freire has published over 170 technical papers, several open-source systems, and is an inventor of 12 U.S. patents. She has co-authored 5 award-winning papers, including one that received the ACM SIGMOD Most Reproducible Paper Award. She is an ACM Fellow and a recipient of an NSF CAREER, two IBM Faculty awards, and a Google Faculty Research award. She has chaired or co-chaired workshops and conferences, and participated as a program committee member in over 70 events. Her research has been funded by the National Science Foundation, DARPA, Department of Energy, National Institutes of Health, Sloan Foundation, Gordon and Betty Moore Foundation, W. M. Keck Foundation, Google, Amazon, AT&T Research, Microsoft Research, Yahoo! and IBM. She has received M.Sc. and Ph.D. degrees in computer science from the State University of New York at Stony Brook.
Gerald Gabrielse
Gerald Gabrielse (NAS), the Board of Trustees Professor at Northwestern University, is one of the world's leading practitioners of fundamental, low energy physics and is a member of the National Academy of Sciences. He is relocating from Harvard to Northwestern to be the founding director of the Center for Fundamental Physics (CFP). An award-winning researcher and teacher, Dr. Gabrielse has chaired both the Harvard Physics Department and the Division of Atomic, Molecular and Optical Physics (DAMOP) of the American Physical Society (APS). He leads the international ATRAP Collaboration at CERN. The Gabrielse research group tested the most precise prediction of the Standard Model of Particle Physics by making the most precise measurement of a property of an elementary particle, tested the Standard Model's most fundamental symmetry to an exquisite precision, made one of the most stringent tests of Supersymmetry and other proposed improvements to the Standard Model, and started low energy antiproton and antihydrogen physics. His many awards and prizes include fellow of the American Physical Society (APS), the Davisson-Germer Prize of the APS, the Lilienfeld Prize of the APS, the Humboldt Research Award (Germany), and the Tomassoni Award (Italy). Harvard University awarded Professor Gabrielse both its George Ledlie Research Prize and its Levenson Teaching Prize. Hundreds of outside lectures include a Källén Lecture (Sweden), a Poincaré Lecture (France), a Faraday Lecture (Cambridge, U.K.), a Schrödinger lecture (Austria), a Zachariasen Lecture (University of Chicago), a Rosenthal Lecture (Yale) and the Hofstadter Lectures (Stanford). He is a member of the National Academy of Sciences. He has a B.S. from Calvin College and an M.S. and a Ph.D. in physics from the University of Chicago.
Constantine Gatsonis
Constantine Gatsonis the Henry Ledyard Goddard University professor of biostatistics, professor of biostatistics, chair of biostatistics, and director of the Center for Statistical Sciences at Brown University. He is a a leading authority on the evaluation of diagnostic and screening tests, and has made major contributions to the development of methods for medical technology assessment and health services and outcomes research. He is a world leader in methods for applying and synthesizing evidence on diagnostic tests in medicine and is currently developing methods for comparative effectiveness research in diagnosis and prediction. Dr Gatsonis currently chairs the Committee on Applied and Theoretical Statistics of the National Academies and is a member of the Committee on National Statistics and the Committee to Evaluate the Department of Veterans Affairs Mental Health Services. Previously, he co-chaired the Committee on the Needs of the Forensic Sciences Community and served on the Board of Mathematical Sciences and Applications and several committees of the National Academy of Medicine. He was the Founding Editor-in-Chief of Health Services and Outcomes Research Methods, and currently serves as Associate Editor of the Annals of Applied Statistics and Academic Radiology, and member of the editorial team of the Diagnostic Test Accuracy Reviews of the Cochrane Collaboration. Dr. Gatsonis was elected fellow of the American Statistical Association and the Association for Health Services Research. He received his Ph.D. in Mathematical Statistics from Cornell UnIversity.
Edward Hall
Edward Hall received his undergraduate degree from Reed College, where he majored in chemistry and philosophy. He earned his Ph.D. in philosophy from Princeton University in 1996; his dissertation focused on conceptual problems in the foundations of quantum mechanics, having to do with the quantum mechanical treatment of the measurement process, and of identical particles. After graduate school, he taught for 11 years in Massachusetts Institute of Technology’s Department of Linguistics and Philosophy, before moving to Harvard in 2005, where he is now the Norman E. Vuilleumier Professor of Philosophy and the Chair of the Philosophy Department. Professor Hall’s philosophical research focuses on the analysis, clarification, and logical interrelationships between a cluster of concepts of central importance across the sciences: causation, probability, laws of nature, counterfactual dependence, confirmation and disconfirmation, statistical inference, realism about unobservable structure, and the nature, formation, and justification of scientific consensus. He has a long-standing ‘semi-professional’ interest in the history of science, and in particular on the conceptual advances that underpinned the scientific revolution of the 17th century, culminating in Newton. Some of his research has focused on conceptual problems in the foundations of so-called “interventionist” approaches to causation and causal inference in statistics; on distinguishing concepts of causation that treat causation as a species of counterfactual dependence from those that treat it as a relation mediated by spatiotemporally continuous processes; on challenges for popular “Humean” accounts of laws of nature, that see such laws as nothing more than pervasive patterns in the physical phenomena; on clarifying the connection between rational degrees of confidence (or “subjective” probabilities) and the kinds of objective probabilities that figure in fundamentally stochastic physical theories; and on articulating basic presuppositions about the natural world that underwrite the possibility of any kind of scientific investigation of that world. Professor Hall’s interest in and approaches to these topics is driven by the conviction that the kind of conceptual clarity that careful philosophical investigation can yield itself constitutes a central and critical kind of scientific progress.
Thomas H. Jordan
Thomas Jordan (NAS) is a University Professor and the W. M. Keck Foundation Professor of Earth Sciences at the University of Southern California. His current research is focused on system-level models of earthquake processes, earthquake forecasting, continental structure and dynamics, and full-3D waveform tomography. As the director of the Southern California Earthquake Center between 2002-2017, Jordan coordinated an international research program in earthquake system science that involved over 1000 scientists at more than 70 universities and research organizations. In 2006, he established the international Collaboratory for the Study of Earthquake Predictability and, since 2006, has been the lead SCEC investigator on projects to create and improve a time-dependent Uniform California Earthquake Rupture Forecast. He has served as a member of the Council of the National Academy of Sciences (2006-2009) and the Governing Board of the National Research Council (2008-2011). Jordan received his Ph.D. from the California Institute of Technology in 1972. He was head of MIT’s Department of Earth, Atmospheric and Planetary Sciences from 1988 to 1998. In 2000, he moved from MIT to USC, and in 2004, he was appointed as a USC University Professor. He has been awarded the Macelwane, Lehmann, and Bowie Medals of the American Geophysical Union, the President's Medal and Woollard Award of the Geological Society of America, and the 2012 Award for Outstanding Contribution to Public Understanding of the Geosciences by the American Geosciences Institute.

Dietram A. Scheufele
Dietram A. Scheufele is the John E. Ross professor in science communication and Vilas Distinguished Achievement professor at the University of Wisconsin-Madison and in the Morgridge Institute for Research. He also holds an Honorary Professorship at the Dresden University of Technology in Germany. Formerly, Dr. Scheufele was on the faculty at Cornell University, a Shorenstein fellow at Harvard University, and a visiting scholar at the Annenberg Public Policy Center of the University of Pennsylvania. His consulting experience includes work for the Public Broadcasting System, the World Health Organization, and the World Bank. Dr. Scheufele’s research deals with the public and political interfaces of emerging science. He is a fellow of the American Association for the Advancement of Science and the Wisconsin Academy of Sciences, Arts & Letters, and a member of the German National Academy of Science and Engineering. He currently serves on the National Academies of Sciences, Engineering, and Medicine’s Division on Earth and Life Studies (DELS) Advisory Committee. He earned a Ph.D. in mass communications at the University of Wisconsin-Madison.
Victoria Stodden
Victoria Stodden is associate professor in the School of Information Sciences at the University of Illinois, Urbana-Champaign. Previously she was assistant professor of statistics at Columbia University where she taught courses in data science, reproducible research, and statistical theory and was affiliated with the Institute for Data Sciences and Engineering. Dr. Stodden is a leading figure in the area of reproducibility in computational science, exploring how we can better ensure the reliability and usefulness of scientific results in the face of increasingly sophisticated computational approaches to research. Her work addresses a wide range of topics, including standards of openness for data and code sharing, legal and policy barriers to disseminating reproducible research, robustness in replicated findings, cyberinfrastructure to enable reproducibility, and scientific publishing practices. She previously served as a co-chair for the NSF Advisory Committee for CyberInfrastructure as well as a former member of the NSF Directorate for Computer and Information Science and Engineering (CISE) Advisory Committee. She also served on the National Academies Committee on Responsible Science: Ensuring the Integrity of the Research Process. She co-edited two books released in 2014—Privacy, Big Data, and the Public Good: Frameworks for Engagement published by Cambridge University Press and Implementing Reproducible Research published by Taylor & Francis. She earned a PhD in statistics and a master of legal studies from Stanford University.
Timothy Wilson
Timothy D. Wilson received his Ph.D. in psychology from the University of Michigan in 1977. He was a faculty member at Duke University from 1977-1979. He joined the Department of Psychology at the University of Virginia in 1979, where he has been ever since. He is currently Sherrell J. Aston Professor of Psychology. He served as Chair of the department from 2001 – 2004. Dr. Wilson has published over 100 articles in scholarly journals and edited books, primarily on the topics of self-knowledge, unconscious processing, affective forecasting, and the applications of social psychology to addressing social problems. His research has been funded by the National Science Foundation, the National Institute of Mental Health, and the Russell Sage Foundation. He has served on numerous editorial boards, including the Board of Reviewing Editors at Science from 2010 – to the present. Dr. Wilson was elected to the American Academy of Arts and Sciences in 2009. In 2013 he received the Donald T. Campbell Award from the Society of Personality and Social Psychology, which recognizes “distinguished scholarly achievement and ongoing sustained excellence in research in social psychology.” In 2015 the Association for Psychological Science awarded Dr. Wilson the William James Fellow Award, to honor a “lifetime of significant intellectual contributions to the basic science of psychology.” In 2015 he received the Thomas Jefferson Award for Scholarship from the University of Virginia, the highest honor given to members of the university community.
Wendy Wood
Wendy Wood is Provost Professor of Psychology and Business at the University of Southern California and currently Distinguished Visiting Chair at INSEAD/Sorbonne. Her research addresses the ways that habits guide behavior—and why they are so difficult to break, as well as evolutionary models of gender differences. From 1982 until 2003, Wood was at Texas A&M University, where she was the Ella C. McFadden Professor of Liberal Arts, the Associate Vice President for Research, and Director of the Women’s Faculty Mentoring Program. In 2004, she moved to Duke University as the James B. Duke Professor of Psychology and Neuroscience and Professor of Marketing. At Duke, Dr. Wood served as Co-Director of the Social Science Research Institute. In 2009, Dr. Wood joined the University of Southern California, where she was Vice Dean of Social Sciences from 2012 to 2016. Dr. Wood is a fellow of numerous scientific societies and is ending a term as President of the Society for Personality and Social Psychology. In the past, she served as editor of the journals, Behavioral Science and Policy, Psychological Review, Journal of Personality and Social Psychology, Personality and Social Psychology Bulletin, and Personality and Social Psychology Review. Her research has been recognized through awards and funding from the National Science Foundation, the National Institute of Mental Health, the Rockefeller Foundation, the Templeton Foundation, and the Radcliffe Institute of Advanced Study.

Committee Membership Roster Comments

(A) The committee will assess research and data reproducibility and replicability issues, with a focus on topics that cross disciplines. The committee will provide definitions of "reproducibility" and "replication" accounting for the diversity of fields in science and engineering. They will assess what is known and, if necessary, identify areas that may need more information to ascertain the extent of the issues of replication and reproducibility in scientific and engineering research. The committee will review current activities to improve reproducibility and replication. The committee will examine factors that may affect reproducibility or replication including incentives, roles and responsibilities within the scientific enterprise, methodology and experimental design, and manipulation; as well as studies of conditions or phenomena that are difficult to replicate or reproduce. The committee will consider a range of scientific methodologies as they explore research and data reproducibility and replicability issues. In its report, the committee will draw conclusions and make recommendations for improving rigor and transparency in scientific and engineering research and will identify and highlight compelling examples of good practices.

(B) Source of Funding: National Science Foundation.

(C) The approximate starting date for the project is 10/01/2017.

(D) A Final Report will be issued at the end of the project in approximately 18 months.

(E) Project Duration: 18 Months.

(F) The committee composition has changed. David Donoho resigned on July 24, 2018 and is no longer a member.

(G) The committee composition has changed. Simine Vazire resigned on October 11, 2018 and is no longer a member.

Events


Event Type :  
-

Description :   

This symposium explored some of the main topics of the report and next steps toward improving research and how different audiences can gain confidence in scientific results. Four topis were explored through moderated discussions with panelists and the audience:

  • Actions and Responsibilities of Agencies and Scientific Leaders to Promote Confidence in Scientific Results
  • Replicability: What it Means to Replicate a Study
  • Reproducibility: Bridging the Gap between Funders and Providers of Needed Services for Computational Reproducibility
  • Moving Forward on Improving Science: Advice from Recent Academies Reports

Background Materials


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
Agenda

Supporting File(s)
-
Is it a Closed Session Event?
No

Publication(s) resulting from the event:

-

Event Type :  
-

Description :   

The webinar includes an overview of the study process and discussion of the report’s conclusions, recommendations, and key messages.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Supporting File(s)
-
Is it a Closed Session Event?
No

Publication(s) resulting from the event:

-

Event Type :  
Meeting

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Lesley Webb
Contact Email:  lwebb274@gmail.com
Contact Phone:  (202) 334-1663

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg (Chair)
David Allison
Dianne Chong
Dietram Scheufele
Juliana Freire
Lorena Barba
Thomas Jordan
Timothy Wilson
Victoria Stodden
Wendy Wood

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Publication(s) resulting from the event:

-

Event Type :  
Meeting

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Lesley Webb
Contact Email:  lwebb@nas.edu
Contact Phone:  (202) 334-1663

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg - Chair
Constantine Gatsonis
Dianne Chong
Dietram Scheufele
Ed Hall
Erin Forstag
Juliana Freire
Lorena Barba
Tim Wilson
Tom Jordan
Wendy Wood

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Publication(s) resulting from the event:

-

Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Diana Chong
David Allison
Michael Cohen
Juliana Freire
Harvey Fineberg
Dietram Scheufele
Tom Jordan
Timothy Wilson
Lorena Barba
Phone number
Erin Forstag
Gerald Gabrielse
Wendy Wood
Victoria Stodden

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 12, 2018
Publication(s) resulting from the event:

-

Event Type :  
-

Description :   

This meeting will be closed (a committee-only meeting) in its entirety.

Details will be posted after the meeting.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  -
Contact Email:  -
Contact Phone:  -

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Participant List:
Tom Jordan
Gatsonis
Lorena Barba
Scheufele
Simine
Tom Arrison
Ej Hall
Gabrielse
Erin Forstag
Victoria
TDW
Juliana Freire
Michael Cohen

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 04, 2018
Publication(s) resulting from the event:

-

Event Type :  
-

Description :   

This meeting is closed to the committee-only details will be posted after the meeting.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  -
Contact Email:  -
Contact Phone:  -

Supporting File(s)
-
Is it a Closed Session Event?
Yes

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
October 04, 2018
Publication(s) resulting from the event:

-


Location:

J. Erik Jonsson Woods Hole Center
314 Quissett Ave.
Woods Hole, Massachusetts
Event Type :  
-

Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

All members were present physically with the exception of Wendy Wood
who participated via teleconference.

The following topics were discussed in the closed sessions:

The committee reviewed the latest draft of its report and discussed its main findings and recommendations.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were provided to the committee in closed session.

Date of posting of Closed Session Summary:
August 03, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Description :   

Presentations on international perspectives on reproducibility and replication in science and engineering


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
1-Agenda

Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 1 and in the morning session of Day 2)

The following topics were discussed in the closed sessions:

• review of the open session’s briefings
• updates to the draft report and writing assignments
• the report’s main conclusions and recommendations
• plans for the committee’s writing meeting in July

The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee.

Date of posting of Closed Session Summary:
June 12, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Web-based meeting via Zoom
Event Type :  
-

Description :   

Presentation on perspectives on reproducibility and replication results in climate science


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
1-Agenda

Is it a Closed Session Event?
No

Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Description :   

The committee will hear about perspectives on scientific progress and irreproducibility, as well as perspectives on reproducibility from the physical and earth sciences, industry and industrial engineering, and economics.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
April 18, 2018 12:10 – 5:00 PM OPEN SESSION: Session Open to the Public Keck Center, Room 101 Public Sessions will be webcast and recorded via Zoom. Join from PC, Mac, Linux, iOS or Android: https://nasem.zoom.us/j/817146422 12:10 PM Welcome and Introductions Harvey Fineberg Committee chair President of the Gordon and Betty Moore Foundation 12:25 Perspectives on Scientific Progress and Irreproducibility Richard Shiffrin, Indiana University 12:55 Panel 1: Reproducibility in the Physical and Earth Sciences (30 minutes each, 20 minutes + 10 minutes for questions) Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of Chemical Engineering, The University of Texas at Austin Peter Mohr, National Institute of Standards and Technology (NIST) 1:55 BREAK 2:10 Panel 2: Reproducibility in Industry and Industrial Engineering (20 minutes each, 15 for briefing + 5 minutes for questions) Carl Ascoli, Chief Science Officer, Rockland Immunochemicals William Lyons, Director, Global Research & Development Strategy for the Global Technology Organization, Boeing Research & Technology 2:50 Introduction to Economics and Reproducibility Daniel L. Goroff Vice President and Program Director The Alfred P. Sloan Foundation 3:20 Panel 3: The Economics of Addressing Reproducibility Issues in Science (4 panelists, each with 15 minutes + 5 minutes for questions from committee) Heidi Williams, Massachusetts Institute of Technology Myron P. Gutmann, University of Colorado in Boulder Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University Brent Goldfarb, University of Maryland 4:40 Public Comment 5:00 End Public Session
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele
Thomas Jordan
David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

• the structure of the draft report and writing assignments
• the report’s main conclusions and recommendations
• review of the open session’s briefings

The following materials (written documents) were made available to the committee in the closed sessions:

No written materials were provided to the committee in closed session.

Date of posting of Closed Session Summary:
April 24, 2018
Publication(s) resulting from the event:

-


Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Description :   

Presentation on perspectives on reproducibility and replication from the American Economic Association


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
Agenda

Supporting File(s)
-
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg Juliana Freire Lorena Barba Edward Hall Dianne Chong Constantine Gatsonis (via Zoom) Gerald Gabrielse David Allison Victoria Stodden Simine Vazire Timothy Wilson Wendy Wood Dietram Scheufele Thomas Jordan David Donoho (via Zoom on Day 2)

The following topics were discussed in the closed sessions:

The committee: reviewed its statement of task, discussed possible topics for commissioned papers, discussed initial report outlines, and discussed possible topics for future meetings.

The following materials (written documents) were made available to the committee in the closed sessions:

No documents were made available to the committee in the closed sessions.

Date of posting of Closed Session Summary:
February 23, 2018
Publication(s) resulting from the event:

-


Location:

National Academy of Sciences Building
2101 Constitution Ave NW, Washington, DC 20418
Event Type :  
Meeting

Description :   

Perspectives on Reproducibility and Replication by Scientific Societies

Panelists, primarily leaders from scientific societies and organizations, will present the level of awareness, interests, concerns and involvement in reproducibility and replicability of research results within their field of science.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Garret Tyson
Contact Email:  gtyson@nas.edu
Contact Phone:  (202) 334-1737

Agenda
01-Agenda

Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Harvey Fineberg
Juliana Freire
Lorena Barba
Edward Hall
Dianne Chong
Constantine Gatsonis
Gerald Gabrielse
David Allison
Victoria Stodden
Simine Vazire
Timothy Wilson
Wendy Wood
Dietram Scheufele (via Zoom)

The following topics were discussed in the closed sessions:

The committee held its bias and conflict of interest discussion and reviewed its statement of task. Future meeting dates and plans for the study were also discussed.

The following materials (written documents) were made available to the committee in the closed sessions:

No written documents were provided to the committee.

Date of posting of Closed Session Summary:
December 15, 2017
Publication(s) resulting from the event:

-

Publications

  • Publications having no URL can be seen at the Public Access Records Office