Public Access Records Office
The National Academies
500 5th Street NW
Room KECK 219
Washington, DC 20001
Tel: (202) 334-3543
Email: paro@nas.edu
Project Information
Print    Feedback   Public Access Records Office (PARO)

Project Information


Opportunities for the National Assessment of Educational Progress in an Age of AI and Pervasive Computation: A Pragmatic Vision


Project Scope:

The National Academies of Sciences, Engineering, and Medicine will appoint an ad hoc panel to consider several innovations that could substantially reduce the cost structure of NAEP while maintaining its technical quality and value in informing the public about education progress. The panel will review the major cost components of NAEP and related assessment programs and consider the following possible changes to the NAEP program: 1) automatic item generation; 2) remote test administration; 3) computer adaptive testing; and 4) consolidation and elimination of substantive overlaps between NAEP assessments and between NAEP and other assessments, such as PISA, TIMSS, and PIRLS. The panel will also solicit and consider suggestions of other major changes that reflect modern methods of assessment and that could substantially reduce NAEP costs while largely preserving its technical quality and informative value. The panel will review relevant research and industry practice to draw conclusions about the likely effects of these potential changes on the cost, technical quality, and informative value of NAEP. 

The panel will produce a short and broadly accessible report that summarizes its findings and conclusions about these potential changes to NAEP and recommends potential assessment or programmatic changes and research needed for NAEP to explore innovations while balancing the competing objectives of cost reduction, technical quality and informative value.

Status: Current

PIN: DBASSE-CNSTAT-20-09

Project Duration (months): 18 month(s)

RSO: Harris-Kojetin, Brian
Division(s):

Division of Behavioral and Social Sciences and Education

Board(s)/Committee(s):

Committee on National Statistics

Topic(s):

Education
Surveys and Statistics

Parent Project(s): N/A

Child Project(s): N/A

Geographic Focus:

Committee Membership

Committee Post Date: 04/30/2021
Karen J. Mitchell - (Chair)
Karen J. Mitchell recently retired as senior director of the Medical College Admission Test (MCAT) at the Association of American Medical Colleges (AAMC). At AAMC, Dr. Mitchell oversaw the work of her colleagues on test development and scoring, test administration and reporting, test preparation services, testing research, and outreach and communication. She directed the redesign and 2015 launch of the current version of the MCAT exam and directed its continued administration during COVID-19. Previously, at the RAND Corporation and U.S. Army Research Institute for the Behavioral and Social Sciences, she directed research on the validity and use of large-scale accountability and selection tests. As a senior program officer at the National Research Council, Dr. Mitchell directed research on the role of licensure tests in improving teacher quality (2001), co-directed the congressionally mandated evaluation of the National Assessment of Educational Progress (1999), and worked on the evaluation of the Voluntary National Tests (1999). She has a Ph.D. in educational research methodology from Cornell University.
Isaac I. Bejar
Isaac I. Bejar retired from the Educational Testing Service in May 2020, where he held the position of principal research scientist. For most of his career, Dr. Bejar researched the application of technology and cognitive science to the scoring of constructed responses, adaptive testing, and automated item generation (AIG). Before his retirement, he received funding for several years from the Department of Defense to research AIG in the context of the Armed Services Vocational Aptitude Battery (ASVAB). Dr. Bejar has authored multiple journal articles, chapters, and books on these topics, and holds several patents related to human and automated scoring. He has a Ph.D. in psychology from the University of Minnesota.
Sean P. Buckley
Sean Patrick (Jack) Buckley is head of assessment and learning sciences at Roblox. Prior to this, he was president and chief scientist at Imbellus, a game-based assessment technology startup. Dr. Buckley was also senior vice president at the American Institutes for Research (AIR), where he led their research and evaluation area, and he still serves as institute fellow on several projects, including chairing the NAEP Validity Studies Panel. Before joining AIR, he helped lead the redesign of the Scholastic Aptitude Test (SAT) at the College Board, where he served as senior vice president of research. Before that, Dr. Buckley served as commissioner of the U.S. Department of Education’s National Center for Education Statistics (NCES), where he was responsible for the measurement of all aspects of U.S. education, including conducting the National Assessment of Educational Progress and coordinating participation in international assessments. He has a Ph.D. in political science from SUNY Stony Brook.
Brian Gong
Brian Gong is senior associate and co-founder of the Center for Assessment, a 501(c)3 organization that provides technical assistance around assessment and accountability primarily to states, districts, and related organizations, including CCSSO and USED. Through this technical advisory work, Dr. Gong has developed a wide-ranging knowledge of innovations in large-scale assessment—including CAT, online test administration, automated scoring, performance assessment, innovative item types—as well as the challenges of making such innovations operationally feasible and legally defensible, including accommodations and supports for the full range of students with disabilities. Gong has served on many committees, including the committee responsible for the most recent edition of the AERA/APA/NCME Standards for Educational and Psychological Testing (2014), the Validation Committee for the Common Core State Standards (2010), the NCES Initiative on the Future of NAEP (2012), and the NAEP Quality Assurance Technical Panel (2010-2018). He has a Ph.D. in education from Stanford.
Andrew D. Ho
Andrew Dean Ho is Charles William Eliot professor of education at the Harvard Graduate School of Education. He is a psychometrician whose research aims to improve the design, use, and interpretation of test scores in educational policy and practice. Dr. Ho is director of the National Council on Measurement in Education (NCME) and a trustee for the Carnegie Foundation for the Advancement of Teaching. He served two full-terms as a member of the National Assessment Governing Board from 2012 to 2020, where he chaired the Committee on Standards, Design, and Methodology. He has chaired the research committee for the Vice Provost for Advances in Learning (VPAL) at Harvard University, which governed research on "massive" open online courses (MOOCs). Dr. Ho was elected to the National Academy of Education in 2021. He has a Ph.D. in educational psychology from Stanford.
Stephen Lazer
Stephen Lazer retired as president and CEO of Questar Assessment in February 2022, a position he has held since 2017. He has worked in educational assessment since 1985. He served in various capacities at Educational Testing Service (ETS) between 1985 and 2017. Of pertinence to this project, he served as NAEP social studies coordinator (1991-1994), NAEP development director (1994-1996), NAEP executive director (1996-2004), and cognizant corporate officer for NAEP (as vice president of assessment development from 2004-2010 and as senior vice president for student and teacher assessment from 2011-2017). Mr. Lazer has written extensively on NAEP and helped serve as a resource to previous NAS studies of the program. In 2017, he left ETS to join Questar Assessment as president and CEO. He has a B.A. degree in political science and English from McGill University, and an M.A. degree in political science from Princeton.


Susan M. Lottridge
Susan Marie Lottridge is senior director of automated scoring at Cambium Assessment, Inc. (CAI). In this role, she leads CAI’s machine learning and scoring team on the research, development, and operation of CAI’s automated scoring software. This software includes automated essay scoring, short answer scoring, automated speech scoring, and an engine that detects disturbing content in student responses. Dr. Lottridge has worked in automated scoring for 12 years and has contributed to the design, research, and use of multiple automated scoring engines including equation scoring, essay scoring, short answer scoring, alert detection, and dialogue systems. She has a Ph.D. in assessment and measurement from James Madison University.
Richard M. Luecht
Richard M. Luecht is professor of educational research methodology at the University of North Carolina at Greensboro, where he teaches graduate courses in applied statistics and advanced measurement topics such as advanced item response theory modeling and estimation, computer-based testing systems and methods, and test score equating. In addition to being internationally recognized as a measurement expert, Dr. Luecht’s research and expertise includes the integration of statistical, computer science and cognitive science technologies in assessment, advanced psychometric modeling and estimation, and the application of industrial engineering design principles to the design and construction of test (i.e., the “assessment engineering” framework). He has designed numerous algorithms and software programs for automated test assembly and devised a comprehensive computerized adaptive multistage testing framework used by several large-scale testing programs. Dr. Luecht is also a technical consultant and advisor for many state departments of education and large-scale testing organizations. He has a Ph.D. in educational psychology, statistics, and measurement from the University of Wisconsin-Milwaukee.
Rochelle S. Michel
Rochelle S. Michel is senior proposal writer (state team lead) on the bids and proposals team at Curriculum Associates. She has held a number of roles in the areas of educational testing, assessment, educational research, and non-profit program management, including: executive director of admission programs at Educational Records Bureau; director of research within the Academic to Career Research Center at Educational Testing Service (ETS); research director at Curriculum Associates; and psychometric manager at ETS. Dr. Michel is also an active member of the American Psychological Association, National Council on Measurement in Education (NCME), and the Northeastern Educational Research Association. She currently serves on the editorial boards of the NCME Applications of Educational Measurement and Assessment Book Series and the NCME Educational Measurement: Issues and Practice journal. She is a past president of the Northeastern Educational Research Association (2018-2019). Dr. Michel has a Ph.D. in psychometrics from Fordham University.
Scott Norton
Scott Norton is deputy executive director of programs at the Council of Chief State School Officers (CCSSO). In this role, he oversees the programmatic work of the Council, including student expectations, student transitions, teacher workforce, and school leadership/school improvement. Dr. Norton first joined CCSSO in 2012 as the strategic initiative director for standards, assessment, and accountability. He previously worked as the assistant superintendent of the Office of Standards, Assessments, and Accountability at the Louisiana Department of Education, and he was a public school teacher in Louisiana. He previously served as a committee member for the National Academies report on the Evaluation of the Achievement Levels for Mathematics and Reading for the National Assessment of Educational Progress (2017), and he has been a member of the NAEP Validity Studies Panel since 2011. Dr. Norton has a Ph.D. in educational administration and supervision from Louisiana State University.
John Whitmer
John Whitmer is senior fellow in data science with the Federation of American Scientists. In this position, he collaborates with IES stakeholders to implement data science into operational programs, expand access and usability of IES data, and support the creation of an ongoing data science fellowship program. Prior to this, Dr. Whitmer led teams of data scientists, research scientists, and machine learning engineers in large educational assessment companies (ACTNext), edTech providers (Blackboard), and educational institutions (California State University and California Community Colleges). An educational researcher by training, he approaches these projects with a commitment to improving the lives of underserved and marginalized students through advanced computational methods. Dr. Whitmer has an Ed.D. in educational leadership from the University of California-Davis.

Disclosure of Conflicts of Interest

Stephen Lazer: The conflict of interest policy of the National Academies of Sciences, Engineering, and Medicine (http://www.nationalacademies.org/coi) prohibits the appointment of an individual to a committee authoring a Consensus Study Report if the individual has a conflict of interest that is relevant to the task to be performed. An exception to this prohibition is permitted if the National Academies determines that the conflict is unavoidable and the conflict is publicly disclosed. A determination of a conflict of interest for an individual is not an assessment of that individual's actual behavior or character or ability to act objectively despite the conflicting interest.

Under institutional policy, Stephen Lazer has a conflict of interest in relation to his service on the committee on Opportunities for NAEP in an Age of AI and Pervasive Computation: A Pragmatic Vision for 2030 and Beyond. This conflict exists because, as president and CEO of Questar Assessment, Mr. Lazer works for a company that is a wholly-owned, independently-operated subsidiary of ETS, the lead contractor for the NAEP program.

The National Academies has concluded that in order for the committee to accomplish the tasks for which it was established, it must include a committee member with current experience in large-scale assessment programs and extensive knowledge of the structural constraints of the NAEP program. As his bio makes clear, Mr. Lazer led key parts of the NAEP work at ETS for two decades, which gave him a detailed understanding of the constraints and trade-offs that are inherent to the program. At the same time, he has a decade of recent experience in other large-scale assessment programs in education that have addressed the practicality of a wide range of potential innovations and also provides him with the necessary breadth and independence which will be invaluable to assessing innovations for NAEP. Mr. Lazer’s combined knowledge of the structural constraints of NAEP and the range of innovations implemented in other large-scale assessment programs will be critical to understanding the potential value for the NAEP program of the innovations the study will consider.

The National Academies has determined that the experience and expertise of Mr. Lazer is needed for the committee to accomplish the task for which it has been established. The National Academies could not find another available individual with the equivalent expertise and breadth of experience who does not have a conflict of interest under institutional policy. Therefore, the National Academies has concluded that the conflict is unavoidable.

The National Academies believes that Mr. Lazer can serve effectively as a member of the committee, and the committee can produce an objective report, taking into account the composition of the committee, the work to be performed, and the procedures to be followed in completing the study.

Events


Event Type :  
Meeting

Description :   

This meeting is closed in its entirety.


Registration for Online Attendance :   
N/A

Registration for in Person Attendance :   
N/A

If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mannn
Contact Email:  amann@nas.edu
Contact Phone:  -
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Publication(s) resulting from the event:
-

Event Type :  
-

Description :   

This meeting is closed in its entirety.


Registration for Online Attendance :   
N/A

Registration for in Person Attendance :   
N/A

If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mannn
Contact Email:  amann@nas.edu
Contact Phone:  -
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Karen Mitchell
Isaac Bejar
Jack Buckley
Brian Gong
Andrew Ho
Stephen Lazer
Sue Lottridge
Ric Leucht
Rochelle Michel
Scott Norton
John Whitmer

The following topics were discussed in the closed sessions:

Report chapters
Future meetings

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
July 21, 2021
Publication(s) resulting from the event:
-

Event Type :  
-

Description :   

The main discussion will focus on clarifying panel questions to the sponsor.


Registration for Online Attendance :   
N/A

Registration for in Person Attendance :   
N/A

If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mannn
Contact Email:  amann@nas.edu
Contact Phone:  -
Supporting File(s)
-
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Karen Mitchell (Chair)
Isaac Bejar
Jack Buckley
Brian Gong
Stephen Lazer
Susan Lottridge
Richard Luecht
Rochelle Michel
John Whitmer

The following topics were discussed in the closed sessions:

Panel's next meeting
Summary of the day's discussions
Panel's report structure

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
June 08, 2021
Publication(s) resulting from the event:
-

Event Type :  
-

Description :   

This meeting is closed in its entirety.


Registration for Online Attendance :   
N/A

Registration for in Person Attendance :   
N/A

If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mannn
Contact Email:  amann@nas.edu
Contact Phone:  -
Supporting File(s)
-
Is it a Closed Session Event?
Yes

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Karen Mitchell (Chair)
Isaac Bejar
Jack Buckley
Brian Gong
Andrew Ho
Stephen Lazer
Sue Lottridge
Ric Luecht
Rochelle Michel
Scott Norton
John Whitmer

The following topics were discussed in the closed sessions:

Report sections
Future meetings

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
May 18, 2021
Publication(s) resulting from the event:
-

Event Type :  
-

Description :   

The panel will hear from the sponsor and interested parties on their vision for NAEP.


Registration for Online Attendance :   
N/A

Registration for in Person Attendance :   
N/A

If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mannn
Contact Email:  amann@nas.edu
Contact Phone:  -
Supporting File(s)
-
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Karen Mitchell (Chair)
Isaac Bejar
Jack Buckley
Brian Gong
Andrew Ho
Stephen Lazer
Sue Lottridge
Ric Luecht
Rochelle Michel
Scott Norton
John Whitmer

The following topics were discussed in the closed sessions:

Sponsor's presentation
Follow on meeting

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
May 13, 2021
Publication(s) resulting from the event:
-

Publications

  • Publications having no URL can be seen at the Public Access Records Office
A Pragmatic Future for NAEP: Containing Costs and Updating TechnologiesA Pragmatic Future for NAEP: Containing Costs and Updating Technologies