UW-Madison researcher’s instrument design fuels groundbreaking international study of teaching and learning

Featuring a Q&A with Courtney Bell

November 18, 2020   |   By Janet L. Kelly, WCER Communications

This week the Organisation for Economic Co-operation and Development (OECD) announced at its Paris headquarters the findings of an international education study unique in its scope and research methods. Launched to identify teaching practices used around the world, Global Teaching InSights: A Video Study of Teaching, also called the TALIS (Teaching and Learning International Survey) Video Study, looked directly into the classrooms of 700 teachers across eight countries and economies to capture on video how each taught the same mathematics topic to their students.

Essential to the study’s success are observation systems designed by Courtney Bell, a principal investigator of the study and a UW–Madison learning sciences professor who directs the School of Education’s Wisconsin Center for Education Research (WCER). In building these systems collaboratively with global teaching experts, Bell’s team created the first standardized observational instruments used to measure the teaching and learning of the same unit of instruction across multiple countries.

This innovative method of collecting data and evidence in video recordings spanned two lessons led by each of the 700 secondary mathematics instructors who participated in the study. Each teacher was recorded teaching quadratic equations in their own classrooms in Chile, China, Columbia, Germany, Japan, Mexico, Spain and the United Kingdom. The videos document the teachers’ practices for managing the classroom, providing social-emotional support and delivering quality instruction. The study links those practices to students’ growth in mathematics, personal interest in math and self-efficacy.

The OECD, which organized the Global Teaching InSights study and developed an associated interactive global video library platform, collected the study evidence to identify the complexities and nuances of teaching in classrooms around the world.

This is the first international study of teaching drawing upon so many different measures to create a comprehensive picture of teaching and learning. For instance, the study included observations of lessons, analyses of teaching materials, standardization of questions posed to teachers and students, and measurement of student outcomes.

The web-based Global Teaching InSights video library includes a collection of 28 classroom videos from the study, along with information on the participating countries. The website invites teachers anywhere to complete an online survey in about five minutes to see how their teaching compares with the teachers in the study.

In addition, through December 20, 2020, OECD encourages educators to upload two-minute videos to share their insights globally about innovation, collaboration and their future plans for teaching.

Findings from the study are based on five years of groundbreaking work by a research team that in addition to Bell, included Principal Investigators Eckhard Klieme, DIPF | Leibniz Institute for Research and Information in Education; V. Darleen Opfer, Rand Corporation; and Daniel McCaffrey, Educational Testing Service.

Questions & Answers with Courtney Bell

WCER Director and PI Courtney Bell has taken some time to reflect on the study. She answers questions and comments on her insights and extraordinary experiences as part of this innovative research team.

What were the goals of the study?

• Understand how teaching practices relate to students’ academic learning, as well as students’ interest and efficacy in the subject matter of mathematics;
• Document how teaching occurs in participating countries and economies; and
• Investigate how teaching practices relate to one another and to aspects of the schooling context.

Can you share more details about the significance of the Global Teaching InSights/TALIS Video Study?

Classrooms in many countries are private places. In the United States, it is still uncommon for teachers to see one another teach, so what teachers actually do to help all students learn is not clear. Internationally, we know little as a research community about what teaching actually looks like in other countries. While there are exemplars available, they often do not reflect “typical” teaching in a country.

This is the first study to capture videos of teaching practices in eight countries and/or economies so that we can “see” how teaching occurs in roughly 85 classrooms. In addition to this rich empirical description of teaching, the study demonstrates that it is possible for experts to collaborate across vastly different teaching, learning and cultural contexts to identify important aspects of teaching quality and co-develop standardized measures of teaching quality.

It shows that it is possible to train a group of experts and then have those experts train another group in their own countries. This “train-the-trainer model” and other such training approaches allow researchers to do larger-scale studies. The study shows we can do this kind of training across countries and with similar levels of reliability and validity.

This feat has never occurred globally and only rarely in national studies. The study also shows that different country and economy-based teams can successfully implement co-developed global research tools in their countries’ classrooms.

What countries and locations in each country participated?

The eight participating countries/economies reflect a diverse and rich array of teaching and learning traditions: Biobío, Metropolitana and Valparaíso in Chile; Colombia; England in the United Kingdom; Germany; Kumagaya, Shizuoka and Toda in Japan; Madrid in Spain; Mexico; and Shanghai in China.

In addition to the video recordings, what other evidence did the researchers examine from each classroom session?

We collected the teaching materials from the recorded lessons, as well as the teaching materials from an additional two lessons. The lessons and teaching materials were coded using standardized observation and artifact protocols. Before and after the mathematics unit, teachers and students completed questionnaires on their beliefs, practices and perspectives, and students took tests to measure their learning gains.

Are the observation systems you developed available for use in other studies and by other researchers?

Yes. Both systems now are in the public domain and researchers are free to use them. The observation tools were further adapted by the OECD for use by teachers. The TVS policy report and some of the technical report are now available at the OECD’s website for the study. There is also an initiative to make some of the participating country/economy lessons visible through the OECD’s Global Teaching InSights Video Platform.

How did the different countries’ teaching practices compare to each other, and to the United States?

There are so many fascinating similarities and differences across countries that we hope other researchers will explore further by making use of the study’s publicly available data. Our study revealed the same pattern of strength and weakness in teaching practices documented in recent United States‐based, large-scale observational studies.

Researchers frequently refer to three domains of teaching: classroom organization, social‐emotional support and instruction. In the U.S. studies, classroom organization and management practices were rated on the high end of the scale and rated higher than the other two domains of teaching. Instructional practices rated on the lower levels of the scale, and consistently rated lower than classroom management and social‐emotional teaching practices. These same findings were true in TVS classrooms around the globe.

Unfortunately, while the United States participated in the early development of the study’s instruments, it did not field‐test the study here.

What are some important findings?

In general, across participating country/economies teachers, students and external observers agreed teachers managed the classrooms well. Teachers had strategies for minimizing disruptions and carrying out common routines, such as handing out papers, checking homework together and calling on students. Across countries, the most common structure for lessons was the teacher teaching the whole class from the front of the room. In each country, students spent less than 25 percent of lesson segments in small groups or pairs.

Across countries, lessons had few negative interactions and students and teachers generally treated each other with mutual respect. Examples include taking turns to speak and using traditional markers of manners, such as “please” and “thank you.”

How about the actual teaching of subjects and interaction with students?

Instructional practices rated the lowest of the three groups of teaching practices, but the ratings were mixed. Teachers regularly elicited, assessed and responded to students’ thinking. However, feedback was brief and focused on the accuracy of the mathematics. The feedback did not focus often on why the mathematics was correct or incorrect.

Students had many opportunities to develop fluency, but had fewer opportunities to engage in cognitively demanding tasks. And while there were many lessons that used technology to support communication, teachers and students did not use the technology for cognitively engaging work.

In addition, students rarely had the opportunity to make connections between mathematics and real-world contexts. They also had few opportunities to explore patterns and generalizations in the mathematics or make connections between representations of the mathematics.

What surprised you in the results?

It surprised me that even in what can be considered a bounded topic in mathematics —quadratic equations —there was no common approach within or across the country/economies to teaching the topic. Variation was the norm. The sequence in which subtopics were ordered, the number of lessons in quadratic equations units, and the mathematical methods taught to students all varied within and across country/economies.

The study also found that after accounting for students’ pre-unit mathematics scores and their background characteristics, social-emotional support and classroom management practices predicted students’ personal interest and self-efficacy in mathematics at the end of the unit in half of the countries/economies. The strongest predictor of students’ mathematics learning during the unit was their pre-unit level of mathematics achievement.