A lot of work goes into developing an assessment. At WIDA and the Center for Applied Linguistics (CAL), teams of language testing experts manage all stages of the test development process, from preparing new test content for field testing to measuring the validity and reliability of test content. (Read our recent article: Everything you’ve ever wanted to know: ACCESS test development)

One important component of the test development process is a cognitive lab. Cognitive labs are a research method that WIDA and CAL use to collect detailed information about students’ test-taking experiences. Cognitive labs involve observing student behavior and hearing directly from students and educators, and are particularly helpful when trying out a new kind of test question or a new way for students to engage with a test.

During cognitive labs, WIDA and CAL work with students at different grade levels and at varying levels of English language proficiency to understand the impact of all test changes on a diverse group of students, including those with Individualized Education Plans (IEPs).

Last fall, WIDA and CAL staff visited schools in Indiana, Virginia, Colorado and Tennessee where they spoke with students, teachers and some school leaders. The goal of these visits was to explore a potential change in how students respond to ACCESS for ELLs Online Speaking tasks — specifically, how students react to having two chances to respond to a task.

Mark Chapman, a researcher at WIDA, and Shauna Cannon, a language testing specialist at CAL, visited Pioneer Bilingual Elementary School (Escuela Bilingüe Pioneer) in the Boulder Valley School District in Colorado to conduct one round of these labs.

“We had a lovely time at Pioneer,” Chapman said. “We were warmly welcomed by Heather Riley and Sandy Sirepisios, who helped facilitate the activities with the school’s second and third graders.”

Seven educators and 17 students participated in the ACCESS for ELLs Online Speaking Cognitive Labs.

“There are different ways of conducting cognitive labs,” Chapman said. “For this cognitive lab, one person administered two different Speaking tasks to a student while the other person observed how the student engaged with and responded to the tasks.”

Chapman and Cannon looked for things like:

• whether the student was confused by having two chances to respond
• whether the student chose to say more in response to the second opportunity to speak, and if so, whether the student produced more original language compared with their first response

Each student’s teacher was also present and observed the student responding to the Speaking tasks. After administering the Speaking tasks, Cannon conducted a short interview with each student. Interviews were conducted in English and Spanish.

“We focused our interviews on why the students chose to respond once or twice to the task, their opinion about having two chances to respond, and how they thought this compared with the one opportunity they get now,” Chapman said.

Following the student interviews, Chapman and Cannon interviewed teachers to get their feedback on the new Speaking task design.

“We really wanted to hear from educators about how they thought their students had performed when they had two chances to respond,” Chapman said. “And we also wanted to know whether they thought this new design feature might help their students do better on the test.”

These cognitive labs were recorded. Now, WIDA and CAL researchers will review the data — including the observation notes and audio recordings — to learn more about the impact of introducing two chances to respond to Speaking tasks.

“I’m excited to learn from the data so that we can better understand whether this change will allow students to better demonstrate their language skills and have more of an opportunity to show what they can do,” Chapman said.

From Chapman, Cannon and everyone involved in the ACCESS for ELLs Online Speaking Cognitive Labs: “Thank you to the wonderful students and teachers who participated! Next time you see an opportunity to participate in a WIDA cognitive lab, think about signing up! We would love to work with you.”

Learn more about cognitive labs and the test development process on the Building a WIDA Assessment page.

About the Researcher

Mark Chapman is the senior innovation researcher at WIDA. His work focuses on researching innovative assessment practices to enhance language tests for young multilingual learners. Mark also conducts research into the validation of Speaking and Writing assessments for young multilingual learners. He publishes his work in journals and edited volumes and regularly presents at domestic and international conferences.