Hand Matching vs. Propensity Score Matching: An Empirical Comparison of Results From a Quasi-Experiment

WCER Working Paper No. 2010-7

Natalie A. Tran, Alan B. Nathan, and Mitchell J. Nathan

May 2010, 10 pp.

ABSTRACT: This paper illustrates how propensity score matching (PSM) can provide a robust estimate of a program treatment effect on student learning outcomes. A quasi-experimental design was used to investigate the impact of enrollment in a pre-college engineering curriculum on student achievement on standardized state tests in science and mathematics, controlling for a small number of covariates (prior achievement score, gender, free/reduced-price lunch status). Models using either hand-matched or PSM controls produced comparable substantive results. The estimates for the impact of the pre-college engineering curriculum were statistically significant for math achievement but not for science. PSM was less laborious; generated a comparison group with more participants and less variability, thus increasing statistical power; and identified an additional significant predictor (gender).

Full Paper

keywords: Propensity Score Matching; Quasi-Experimental Design; Student Achievement; Pre-College Engineering Education; Career and Technical Education