Prevention Science @SPRORG published an RCT of High School FLASH, a sex ed program, reporting that it reduces homophobic & transphobic beliefs. My quick take: Not a credible result (only suggestive), as this was a post-hoc analysis that can easily yield chance findings.

Program:

  • Per the study: “High School FLASH is a comprehensive sexual health education curriculum … for classroom settings, with the specific behavior change goals of preventing unintended pregnancy, preventing STDs, and preventing the perpetration of sexual violence.”

Study Design & Prior Findings:

  • The study randomly assigned 20 schools in the U.S. South and Midwest, with a student sample of 1,597 9th and 10th graders,  to (i) a treatment group that received FLASH vs (ii) a control group that received a knowledge-based sexual health curriculum.

  • A prior study report found no discernable impact of FLASH vs control on the study’s 2 pre-specified primary outcomes: the full-sample rates of vaginal sex, & vaginal sex without birth control, measured at 3 and 12 months after program completion.

New Findings:

  • The newly-published article reports statistically significant positive impacts on students' homophobic & transphobic beliefs at 3 & 12 months post-program.

  • Importantly, however, this was not a pre-specified primary or secondary outcome of the study, per the study registration.

    In other words, it’s a post-hoc outcome. Analyses of such outcomes are only suggestive under established scientific standards (FDA, IES), as they can easily yield chance (i.e., false) results.

  • The reason is that each statistical significance test has about a 1/20 chance of producing a false result, for a program whose true effect is zero. So a study like this that examines numerous outcomes & subgroups can easily turn up false findings.

Comment:

  • Claiming effectiveness based on post-hoc findings is a bit like placing a bet on a horse after watching the race. The findings are not without value but I think only suggestive.

We use cookies to improve your experience and to help us understand how you use our site. Please refer to our cookie notice and privacy statement for more information regarding cookies and other third-party tracking that may be enabled.

Intuit Mailchimp logo