Public Access Records Office
The National Academies
500 5th Street NW
Room KECK 219
Washington, DC 20001
Tel: (202) 334-3543
Email: paro@nas.edu
Project Information

Project Information


Foreign Language Assessment for the U.S. Foreign Service Institute


Project Scope:

An ad hoc committee will evaluate the different approaches that (1) exist to assess foreign language proficiency and that (2) the State Department's Foreign Language Service Institute (FSI) could potentially use to assess language proficiency.  The committee will consider the key assessment approaches in the research literature that are appropriate for language testing, including, but not limited to, assessments that use task-based or performance-based approaches, adaptive online test administration, and portfolios. 

The committee will collect information that helps answer the following questions:

  • What assessment formats and approaches are feasible for language proficiency testing? What are the advantages and disadvantages of the various approaches?
  • How well do different assessment approaches measure reading and listening comprehension (interactive and non-interactive)?
  • How well do different assessment approaches measure speaking proficiency?
  • To what extent would different assessment approaches provide information to support the intended inferences about a candidate's language proficiency?
  • What are the psychometric characteristics (reliability, validity, classification accuracy) associated with different approaches? 
  • Are the different assessment approaches equally effective (fair and unbiased) for all groups that typically take the FSI assessments?
  • To what extent is unconscious bias a concern with different assessment strategies? Which assessment approaches minimize the effect of unconscious bias in foreign language proficiency testing?
  • Are the different assessment approaches equally practical and cost effective in a resource-limited government environment?

The committee will not recommend any specific assessment approach, but will describe the strengths and weaknesses of different assessment approaches, in light of the latest research and current scientific consensus. The committee will also take into account the practicality of various options in a resource-limited, government environment (in contrast to academic or private sector assessment applications). To the extent possible, the study should address the steps involved in conducting proficiency assessments to ultimately enable the State Department to determine the most appropriate method to utilize for the Foreign Service.

The assessment process currently used by FSI and the definition of language proficiency developed by the Interagency Language Roundtable provide the context for the study. However, the purpose of the consensus study is not to evaluate FSI's current assessment process. That process can serve as one possible benchmark for comparison when identifying the strengths and weaknesses of other assessment approaches. The focus of the study is also not to evaluate the current definition of language proficiency used by FSI, or its approach to language learning, but instead to identify the most effective means of assessing language proficiency as currently defined in the context of the U.S. Foreign Service.

Status: Current

PIN: DBASSE-DBASSE EO-18-01

Project Duration (months): 18 month(s)

RSO: Elliott, Stuart



Geographic Focus:

Committee Membership

Committee Post Date: 02/21/2019

Dorry M. Kenyon - (Chair)
Is senior fellow for assessment at the Center for Applied Linguistics (CAL) and acting director of CAL’s collaborative assessment activities with WIDA, a consortium of state departments of education dedicated to supporting English Language Learners in K-12 contexts. Dr. Kenyon also serves as chair of the five-member expert panel of the U.S. government’s Defense Language Testing and Assessment Project (DELTAP). He has considerable experience in all aspects of designing, developing, validating, and operationalizing second and foreign language assessments through many large projects at state, national, and international levels. He has served as a senior advisor on a variety of projects related to developing and researching assessments of the English language and foreign language skills of language learners spanning the ages of pre-school to adult. Prior to joining CAL, Dr. Kenyon taught German and English as a foreign/second language for seven years in the United States and abroad. He has a B.A. degree in German and economics from Bowdoin College, an M.T.S. in theology from Gordon-Conwell Theological Seminary, an M.A. degree in Teaching English as a Foreign Language from the American University in Cairo, and a Ph.D. in measurement, applied statistics, and evaluation from the University of Maryland, College Park.
David Dorsey
Is vice president and director of the Business Development Division at the Human Resources Research Organization (HumRRO). Prior to this, he was a senior data scientist within the U.S. Department of Defense. Dr. Dorsey’s work in government included establishing an approach for certification of critical skills and conducting cutting-edge work on foreign language acquisition. Before working in government, he was vice president at Personnel Decisions Research Institutes, a top human capital consulting firm. During his career, he has produced over 70 book chapters, articles, and presentations and his research has appeared in such journals as Industrial and Organizational Psychology, Consulting Psychology Journal: Practice and Research, and Human Factors. Dr. Dorsey is an elected fellow of the Society for Industrial and Organizational Psychology (Division 14 of the American Psychological Association). He has a B.A. degree in psychology, an M.A. degree in industrial/organizational psychology, and a Ph.D. in industrial/organizational psychology, all from the University of South Florida.
Lorena Llosa
Is associate professor of education in the Steinhardt School of Culture, Education, and Human Development at New York University. Her research focuses on second language teaching, learning, and assessment. Dr. Llosa’s studies have addressed standards-based classroom assessments of language proficiency, assessment of academic writing, placement testing of US-educated language minority students in community colleges, and the integration of language and content in instruction and assessment. Her research has appeared in such journals as Language Testing, Language Assessment Quarterly, Educational Measurement: Issues and Practice, Educational Assessment, Assessing Writing, Language Teaching Research, Language Learning, Reading and Writing Quarterly, Teachers College Record, and the American Educational Research Journal. Dr. Llosa has a B.A. degree in English and Spanish from Santa Clara University, an M.A. degree in teaching English as a second language from the University of California, Los Angeles, and a Ph.D. in applied linguistics with a specialization in language testing from the University of California, Los Angeles.
Robert J. Mislevy
Is Frederic M. Lord chair in measurement and statistics at Educational Testing Service (ETS). Dr. Mislevy’s research interests center on applying recent developments in technology, statistical methodology, and cognitive research to practical problems in educational assessment. He has recently published books on Bayesian Networks in Educational Assessment, Bayesian Psychometric Modeling, and Sociocognitive Foundations of Educational Measurement. He received the Lindquist Award for career contributions from the American Educational Research Association, the Samuel J. Messick Memorial Lecture Award from the International Language Testing Association, and a Career Contributions Award and four annual Awards for Technical Contributions from the National Council on Measurement in Education. Dr. Mislevy has served on several committees at the National Academies on issues concerning assessment, instruction, and cognitive psychology, was a member of the U.S. Defense Language Testing Advisory Panel, and was a primary author of the final report of the National Assessment Governing Board’s design feasibility team. He has B.S. and M.S degrees in mathematics from Northern Illinois University, and a Ph.D. in research methodology from the University of Chicago.
Lia Plakans
Is associate professor in foreign language and English as a second language education at the University of Iowa, and departmental executive officer for teaching and learning in the College of Education. Dr. Plakans’ research focuses on second language learning with particular emphasis on language assessment and literacy. Her research has explored assessments that integrate language skills, such as reading-into-writing tasks, to understand the underlying processes elicited as well as the nature and scoring of performances that require integration. Her research has appeared in such journals as TESOL Quarterly, Language Testing, Language Assessment Quarterly, and Journal of Second Language Writing. Dr. Plakans is co-author of Assessment Myths: Applying Second Language Research to Classroom Teaching and associate editor for Language Assessment Quarterly. She chairs the Committee of Examiners of the Test of English as a Foreign Language (TOEFL) at Educational Testing Service. She has a B.A. degree in anthropology and psychology from the University of Iowa, an M.A. degree in teaching English as a second language/applied linguistics from Iowa State University, and a Ph.D. in foreign languages/English as a second language education from the University of Iowa.
James Purpura
Is professor of linguistics and education at Teachers College, Columbia University. His research interests include the assessment of second and foreign language grammar and pragmatics; second and foreign language test validation; and learning-oriented and scenario-based assessment in classroom and large-scale standardized assessment contexts. Dr. Purpura has published articles in applied linguistics journals and has written chapters in handbooks, encyclopedias, companions, and other edited volumes. He currently serves on the U.S. Defense Language Testing Advisory Panel, and is an expert member of the European Association of Language Testing and Assessment (EALTA). He is co-editor of Language Assessment Quarterly, and co-editor of two book series devoted to language assessment. He was the first two-year President of the International Language Testing Association (ILTA). Also, he served twice on the Committee of Examiners for the Test of English as a Foreign Language (TOEFL) at Educational Testing Service, and has served as a language testing consultant for numerous foreign countries. Dr. Purpura has a B.A. degree in French language and literature from Marietta College, an M.A. degree in French linguistics from the University of Colorado, Boulder, and a Ph.D. in applied linguistics from the University of California, Los Angeles. Finally, he was a Fulbright Scholar at the University for Foreigners in Siena in 2016.
M. Elvis Wagner
Is associate professor of teaching English to speakers of other languages (TESOL) at Temple University. His research focuses on second language assessment, especially the assessment of second language listening ability. Dr. Wagner’s research has appeared in such journals as Language Assessment Quarterly, Language Testing, and Applied Linguistics. He recently co-authored the book Assessing L2 Listening: Moving towards Authenticity (2018). His article “Video Listening Tests: What are They Measuring?” in Language Assessment Quarterly won the International Language Testing Association’s “Best Article in Language Testing” award in 2008. Dr. Wagner’s research agenda is informed by his broad teaching experience in many different contexts. He has written and published widely on issues related to foreign and second language teaching methodology, as well as, teaching and testing of second language listening and oral communicative competence. He has a B.A. degree in English, Spanish, and history from Texas Christian University, an M.A. degree in English from the University of Nebraska, and an Ed.D. in applied linguistics from Teachers College, Columbia University.
Paula M. Winke
Is an associate professor of Second Language Studies (SLS) at Michigan State University (MSU), co-director of the SLS Eye-tracking Lab, and co-PI on a 5-year, Language Proficiency Flagship Initiative grant from the National Security Education Program. Prior to joining MSU, Dr. Winke was a German instructor at the University of Minnesota, and a test development manager at the Center for Applied Linguistics in Washington, DC. While in Washington, DC, she co-founded the East Coast Organization of Language Testers. She is a past president of the Midwest Association of Language Testers and recipient of the International TESOL Distinguished Researcher award (2012), and the CALICO Article of the Year award (2008). Dr. Winke was a Peace Corps volunteer to China (1998-2000), a Fulbright English language instructor in Hungary (2008), and will be a Fulbright scholar in Germany (2020). She teaches language assessment for classroom and research purposes, and individual differences in second language acquisition. Recent publications include Foreign Language Proficiency in Higher Education (2018; Springer). She is also editor of Language Testing. She has a B.A. degree in French and philosophy from the University of Wisconsin, Madison, an M.A. degree in linguistics from the University of Minnesota, and a Ph.D. in applied linguistics from Georgetown University.

Events



Location:

Keck Center
500 5th St NW, Washington, DC 20001
Event Type :  
-

Description :   

The purpose of this meeting is for NAS staff to discuss the draft report outline


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  (202) 334-3266

Supporting File(s)
-
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Publication(s) resulting from the event:

-


Location:

National Academy of Sciences Building
2101 Constitution Ave NW, Washington, DC 20418
Event Type :  
Meeting

Description :   

The purpose of this meeting is to hear from the sponsor in terms of what goals they have from the study. Staff will also discuss the draft report outline.


Registration for Online Attendance :   
NA

Registration for in Person Attendance :   
NA


If you would like to attend the sessions of this event that are open to the public or need more information please contact

Contact Name:  Anthony Mann
Contact Email:  amann@nas.edu
Contact Phone:  (202) 334-3266

Supporting File(s)
-
Is it a Closed Session Event?
Some sessions are open and some sessions are closed

Closed Session Summary Posted After the Event

The following committee members were present at the closed sessions of the event:

Dorry Kenyon (Chair)
David Dorsey
Lorena Llosa
Robert Mislevy
Lia Plakans
James Purpura
Elvis Wagner
Paula Winke

The following topics were discussed in the closed sessions:

Monday, March 18, 2019
Welcome and Introductions
National Academies Committee Work Processes and Resources
Committee Discussion of Statement of Task and FSI’s Goals
Committee Discussion of Draft Report Outline
- What needs to be changed?
- Which topics will be straightforward and which are likely to be difficult to address or contentious across the committee?

Tuesday, March 19, 2019
Committee Member Bias and Conflict of Interest Discussion
Committee Discussion of Possible Approaches to Difficult or Contentious Topics
Continued Discussion of Approaches to Difficult or Contentious Topics
Discussion of Next Steps
- Subgroup formation and plans to schedule calls
- Goal: Presentation of initial outline of key points and argument at Meeting 2
- Topics requiring additional resources outside the committee, involving outside presentations at Meetings 2 or 3, commissioned papers, or expert interviews
- Meeting 2 plans
- Reflections on Meeting 1

The following materials (written documents) were made available to the committee in the closed sessions:

None

Date of posting of Closed Session Summary:
March 20, 2019
Publication(s) resulting from the event:

-

Publications

  • Publications having no URL can be seen at the Public Access Records Office
Publications

No data present.