CAPR published an RCT of Multiple Measures Assessments (MMA), a data analytic tool increasingly used by community colleges to place incoming students into college-level vs developmental (i.e., remedial) courses. Quick take: High-quality RCT finds little impact on student success at 4.5y mark. (Continued below.)

Program:

  • MMA uses predictive algorithms – based on high school GPA, placement test scores, time since HS graduation, etc – to place students into developmental (i.e., remedial) vs college-level courses. Colleges' traditional procedure is to use a standardized test to determine placement.

Study Design:

  • The study randomly assigned 12,796 incoming students at 7 State University of New York community colleges to placement using MMA (treatment) vs standardized test (control). Use of MMA changed the placement of 26% of students in math & 51% of students in English (vs their control group counterparts).

Findings:

  • Despite changing student placements, MMA had little impact on the primary study outcomes at the 4.5y follow-up: (i) % completing college-level math (40.5%T vs 39.3%C) & English (53%T vs 51%C); college credits earned (28T vs 27C); % graduating or transferring to 4y college (23.4%T v 23.7%C).

  • Of the above, only the effect on college-level English was statistically significant.

Comment:

  • Based on careful review, this was a well-conducted RCT (e.g., baseline balance, negligible attrition, pre-registered primary outcomes).

  • Unfortunately, the study abstract's summary of the results (shown above) doesn't mention the central finding of negligible impacts on the 4 prespecified primary outcomes. It instead paints a rosier picture based on exploratory findings (subgroups etc) that aren't reliable - only suggestive.

  • This key omission in the study abstract (which busy readers often rely upon to glean the main take-aways) could easily lead policy officials to consider and/or adopt MMA in the mistaken belief it has been shown effective.

We use cookies to improve your experience and to help us understand how you use our site. Please refer to our cookie notice and privacy statement for more information regarding cookies and other third-party tracking that may be enabled.

Intuit Mailchimp logo