As artificial intelligence (AI) becomes more embedded in education research and classroom practice, understanding its limits and risks is vital to proper use. One key caveat for education researchers: A general-purpose AI model can perform well on standard evaluations (benchmark tests) but still fail when it encounters the messier, context-specific data education researchers and teachers work with every day. So, what’s the solution?

Education researchers should start by asking more than whether an AI model is simply “accurate,” according to WCER researcher Shamya Karumbaiah — they should test how it responds to the people, language and situations represented in their own research.

“The real question isn’t just whether a model is accurate,” said Karumbaiah, a UW–Madison assistant professor of educational psychology who studies human-centered AI for teaching and learning with the aim of augmenting human intelligence. “It’s whether it’s accurate for your students, in your context, for your task.”

At her recent WCER Community Learning Series workshop, Karumbaiah explained ways to systematically evaluate AI errors and biases using two recent studies from The Responsible AI for Learning (TRAIL) Lab she directs. The session blended hands‑on exploration of a TRAIL Lab-built AI evaluation tool with a discussion of study findings, giving participants a practical framework for evaluating errors, biases, and unexpected behaviors in language models used to analyze student writing and other kinds of qualitative data.

Key advice for education researchers using AI included:

• Define expectations clearly (for example, fairness for multilingual learners)
• Stress-test models systematically
• Compare AI judgments with human reasoning
• Document failures and patterns
• Choose models based on context, not hype

Why Standard AI Evaluation Falls Short in Education

Karumbaiah contrasted how AI models are typically evaluated in industry with how they are used in schools. Companies like Meta, Google and OpenAI test their models on benchmark datasets and generic tasks such as sentiment analysis or natural language inference. But these datasets rarely resemble educational tasks such as formative assessment and qualitative coding or use classroom language and other qualitative data, which can include:

• Academic language and disciplinary practices
• Varying linguistic abilities of children (e.g., spelling and grammar errors)
• Multilingual variations
• Partial or informal explanations
• Situated reasoning tied to specific classroom activities

Because of this, Karumbaiah emphasized that general-purpose evaluation is not enough.

“These models are trained and tested on data that look nothing like what students actually produce,” she said. “If we want to trust AI in education, we have to evaluate it in our context.”

Karumbaiah described the needed shift as moving from generalization to contextual evaluation, defined as a process that examines how a model behaves on the specific kinds of language and reasoning educators actually encounter.

Try to Make Your Model Fail

Participants then logged into TRAIL Lab’s AI Behavior Analysis Tool (AIBAT), which supports “stakeholder-driven” evaluation of language model failures, a process designed around the needs, values and concerns of the people impacted by the AI system. Participants created topics ranging from physics concepts to playful examples like “cheese” and then generated statements to test whether the model’s judgments aligned with their own.

This exercise illustrated a key principle: AI models often fail in surprising ways, even on simple statements. AIBAT helps users identify mismatches between human and model judgments and visualize where those mismatches cluster.

Karumbaiah described this as “AI behavior analysis,” a method for stress‑testing models by intentionally probing their boundaries.

“You’re all hackers today,” she joked to session attendees. “Your job is to push the model until it breaks and learn where it breaks from.”

To help participants understand how fragile language models can be, she encouraged them to deliberately introduce realistic language variations that often appear in context.

“Small changes in phrasing, spelling, or linguistic style can completely throw off a model,” said Karumbaiah, citing examples such as mixing English words with Spanish ones, using an acronym like “ht” for height or using nonstandard phrasing or informal grammar.

Participants explored behaviors related to fairness, robustness, and sensitivity, revealing how small changes can lead to inconsistent or biased model outputs.

Study 1: Prompt vs. Supervise

Karumbaiah next summarized findings from the first TRAIL Lab study, which compared two approaches to assessing students’ explanations of physics concepts from an inquiry-based learning unit. The two approaches involved either prompting general-purpose language models or training supervised classifiers using human-coded data.

The takeaway: Model choice involves trade-offs between accuracy, flexibility, resource availability, and adaptability to new tasks.

“The best-performing model isn’t always the best option,” Karumbaiah noted. “It depends on what you can realistically support.”

Specifically, the study found that fine-tuned supervised models often achieved higher accuracy, but required significant resources, such as human-coded data, computing power, and technical expertise. Prompted models were more flexible and easier to adapt, but sometimes less reliable, while traditional feature-engineered models — older approaches relying on keywords — performed surprisingly well in certain contexts.

Study 2: Beyond Generalization

The second study demonstrated why high accuracy alone is not enough to trust a model in real classrooms. Even models with strong overall performance can fail when student writing varies in realistic ways.

Using the behavior analysis tool, the TRAIL Lab stress‑tested models against variations expected in student writing such as spelling errors, acronyms, multilingual and translanguage variations, and nonstandard phrasing.

These stress tests revealed failures that standard evaluation methods missed, highlighting the need for contextual, stakeholder-driven evaluation rather than reliance on general benchmarks.

“You need to know how a model succeeds and where it fails,” said Karumbaiah.

A Framework for Responsible AI Use

Throughout the session, Karumbaiah emphasized that evaluating AI is not about achieving perfection; it’s about understanding model behavior well enough to use AI responsibly. Some questions that researchers and practitioners should ask themselves before adopting AI models for qualitative research or assessing student work include:

• Where does the AI model succeed and where does it fail?
• How easy or hard is it to locate these failures?
• Are these failures concentrated in certain contexts or student populations?
• What downstream harms will these failures lead to?
• How well should the model perform for me to use it reliably? To trust the automation?

Review the slides from Karumbaiah’s presentation to learn more.

For more support on working with AI in research, consider these resources:

MERIT AI team: Whether you’re exploring AI for teaching, research, administrative tasks or software development, this School of Education team can help you find answers and navigate university guidance. Send your questions to the MERIT AI team.

Campus hub for AI: Access generative AI tools that UW–⁠Madison has vetted and secured contracts for, including Gemini, Copilot and Zoom AI. The tools are free across the university and offer stronger data security and privacy protections than public services.

Researcher AI guide: From UW–Madison Libraries, this website offers guidance and resources for AI chatbots, large language models and other types of generative AI. Send questions about the guide to Dave Bloom or Heather Shimon.

About the Researcher

Shamya Karumbaiah is an assistant professor in the learning sciences area of the Department of Educational Psychology. Her academic work focuses on human-centered AI for teaching and learning, with a focus on equitable, responsible AI use in classrooms and identifying biases in automated learning systems.

A former computer scientist for more than a decade, she earned a doctorate in learning sciences from the University of Pennsylvania, where her dissertation empirically investigated sources of bias in AI-based learning systems. Before joining UW–Madison in 2023, she spent a year as a postdoctoral fellow at Carnegie Mellon University.

Her research has been nominated for best paper awards at premier educational AI conferences such as LAK, CSCL, AIED, EDM, ICQE, and ICCE. She has been recognized with the Undergrad Research Mentor Award (2026), Vilas Early Career Investigator Award (2025), SOLAR Early Career Research Award (2024), Nellie McKay Fellowship (2023), and MIT EECS Rising Stars (2021).

About The Responsible AI for Learning (TRAIL) Lab

Led by Karumbaiah, the TRAIL Lab is an interdisciplinary research group that studies the responsible use of AI in real-world classrooms. Members work with teachers, students, researchers, engineers, and the complex contexts in which learning happens. Visit the TRAIL Lab website for more information, including research, publications, tools, key personnel and resources. TRAIL Lab research has been funded by the Gates Foundation, Spencer Foundation, American Family, Institute for Diversity Sciences, and Wisconsin Alumni Research Foundation.