Annenberg Institute posted a quasi-experimental study of a California program to train teachers in the science of reading. Quick take: Despite study claims of positive impacts (covered by NY Times & others), it found no discernible effect on 3rd grade reading achievement in its primary analysis.  Continued below.

Program:

  • The study evaluated California’s Early Literacy Support Block Grant, providing low-performing elementary schools with teacher training in the science of reading, new funding (~$1000/student), spending flexibility, expert assistance, & other supports to improve reading.

Study Design:

  • The study's primary, prespecified evaluation method (set out in the study registration) was to compare outcomes over 1y for (i) schools that were just above a threshold percentage of struggling readers (& thus received the program) to (ii) schools just below that threshold (which were ineligible for the program). This type of study is known as “regression discontinuity.”

Findings:

  • Using this method, the study found no statistically significant effect on the percent of 3rd graders scoring near-proficient or higher in reading after 1 school year. In fact, the impact table (above) shows a slight decrease of 4-7% points (though not statistically significant).

  • The study abstract unfortunately does not mention the disappointing results of the primary analysis, & instead presents the study's findings as unambiguously positive based on post-hoc (vs prespecified) analyses that are far less credible.  

  • They're less credible because, in post-hoc analyses, researchers have wide discretion over key study parameters - e.g., which schools comprise the comparison group, & which analysis methods to use - and can consciously or unconsciously select parameters that yield a desired result.

Comment:

  • Unfortunately, the study received uncritical coverage in the NY Times, Education Week, & elsewhere - coverage which largely repeats the study abstract's positive claims.

We use cookies to improve your experience and to help us understand how you use our site. Please refer to our cookie notice and privacy statement for more information regarding cookies and other third-party tracking that may be enabled.

Intuit Mailchimp logo