Evidence-Based Policy Newsletter

The Coalition for Evidence-Based Policy publishes this e-newsletter for policymakers, researchers, and others who wish to keep abreast of rigorous evidence on “what works” in social policy. The newsletter shares our quick take on the latest findings — without spin, just the facts. We focus mainly on randomized controlled trials (RCTs) because, when feasible and well-conducted, they’re considered the strongest method of evaluating program effectiveness.

The Coalition is a nonprofit, nonpartisan organization that is unaffiliated with any social programs or interventions. Subscribe to our email newsletter here. Archived newsletter issues are shown below.

Year-End Newsletter (#16): Top Positive RCT Findings

December 20, 2023

This year-end issue of the Newsletter compiles the top U.S. RCT findings of sizable, sustained effects on important life outcomes that I’ve summarized over the past 15 months:

  • Nevada RESEA - a low-cost program aimed at reemploying Unemployment Insurance claimants (earnings gains of $13,000/person over 5 years & net savings to government). See full summary.

  • Accelerate, Complete, Engage (ACE) at John Jay College - providing comprehensive support to help students complete a bachelor’s degree (12%-point gain in bachelor’s completion at 4-year follow-up). See full summary.

  • ASAP Ohio - providing academic, personal, & financial support to low-income community college students (15%-point gain in college graduation & 11% gain in earnings at the 6-year follow-up). See full summary.

  • Saga Education's high-dosage math tutoring of low-income 9th & 10th graders in Chicago (large impact on math achievement sustained 1-2y after tutoring ends). See full summary.

  • Year Up - a job training program for low-income young adults that focuses on fast-growing economic sectors (long-term earnings gains of 30%, or $8000/year). See full summary.

  • Per Scholas - an employment/training program low-income adults, preparing them for well-paying jobs in Information Technology (earnings gain of 16%, or $5200/year, at the 5–7 year follow-up). See full summary.

  • Bottom Line - providing one-on-one advising to help low-income students get into & graduate from college (8%-point gain in bachelor's degree completion at the 6y follow-up). See full summary.

  • Project QUEST - occupational training of low-income adults for well-paying healthcare jobs (long-term earnings gains of 15–20%, or about $5,000/year). See full summary.

  • Denver Supportive Housing initiative - providing housing & supportive services to homeless people with over 7 prior arrests (roughly 30–40% reductions in jail & shelter use, emergency department visits, & arrests over 2–3 yrs). See full summary.

  • Big Brothers Big Sisters Community-Based Mentoring for at-risk youth (1/3 to 1/2 reduction in arrest rate and substance use over 18 months). See full summary.

  • Individual Placement & Support - a job counseling, search, & support program for adults with disabilities (earnings gain of 26%, or ~$2000 annually, over 2y). See full summary.

Evidence-Based Policy Newsletter #15

November 28, 2023

  1. AIR posted RCT results for Literacy First, providing high-dosage tutoring to struggling readers in 2nd grade. Quick take: Well-conducted RCT finds positive impact on end-of-year reading scores, but unfortunately no long-term (grade 5) impact on reading or science achievement. (See full summary.)

  2. MDRC published an RCT of NY City P-TECH 9-14 schools, aimed at preparing students for college & careers within six years. Quick take: Small or no impacts on high school graduation or college attainment at 4-7 yr follow-up (study claims notwithstanding) *but* future earnings impacts are plausible. (See full summary.)

  3. Impaq just posted long-term RCT results for Nevada's RESEA program, aimed at reemploying Unemployment Insurance (UI) claimants. Quick take: High-quality RCT finds that this low-cost program ($290/person) yields earnings gains of $13,000/person over 5 years & net savings to government. (See full summary.)

  4. JREE published a long-term evaluation of Reading Recovery (RR) - a tutoring program for 1st graders - that found adverse effects on grade 3-4 reading achievement, and received wide coverage (e.g., recent NY Times). Quick take: This is not a reliable finding, due to sample loss (attrition) of over 70% and other problems. (See full summary.)

  5. JAMA published 5-year RCT results for Million Hearts - a national HHS pay-for-performance model to prevent heart attacks & strokes in Medicare patients. Quick take: No discernible impact on the study’s primary health or cost outcomes (health findings are misreported as positive in JAMA). (See full summary.)

Evidence-Based Policy Newsletter #14

October 9, 2023

  1. An RCT of Fostering Healthy Futures mentoring program for teens in the child welfare system recently reported final results. Quick take: High-quality RCT finds a sizable 14% point decrease in juvenile justice involvement over ~2 years - an encouraging result though not definitive due to lack of statistical significance. (See full summary.)

  2. The Baby's First Years RCT, studying the impact of unconditional cash transfers to low-income US mothers with newborns, reported new results on 2 primary outcomes: child overall health & sleep over 3 years. Quick take: No impact on either outcome (but effects could emerge in future). (See full summary.)

  3. The US Department of Education's What Works Clearinghouse (WWC) published an evidence review of World of Words preschool program to improve children's vocabulary/language. Quick take: WWC's finding of "strong evidence" is far overstated; the results are preliminary & not yet reliable. (See full summary.)

  4. HHS/ACF reported RCT results for HPOG 2.0, a major federal demonstration program that trains low-income adults for skilled healthcare jobs. Quick take: High-quality RCT finds no program impact on participant earnings over 3-4 years, despite increasing completion of training (i.e., % earning any new credential) by 14% points. (See full summary.)

  5. Mathematica reported long-term RCT results for KIPP charter middle schools. Quick take: High-quality RCT finds a suggestive 4% point gain in college enrollment & no impact on college persistence. (Its reported large impact of attending *both* KIPP middle + high school is quasi-experimental & not yet reliable pending confirmation in future RCTs.) (See full summary.)

  6. NBER posted an RCT of Sit-D, a police training program teaching the value of developing multiple perspectives on a situation. Quick take: High-quality RCT finds suggestive evidence of a 12% reduction in police use of force over 1 yr (findings are moderately encouraging but not as strong as presented in study). (See full summary.)

  7. Science Magazine published an RCT at 22 US colleges, reporting that a brief "social belonging" writing exercise increased 1st-year college completion rates. Quick take: These claims are based on post-hoc (vs prespecified) analyses that aren't reliable under established scientific standards. (See full summary.)

Evidence-Based Policy Newsletter #13

September 6, 2023

  1. BMJ published an RCT in Switzerland of a smartphone app to reduce drinking among university students with unhealthy alcohol use. Quick take: High-quality RCT finds modest but meaningful 7% decrease in drinks/week at 1-year mark for a low-cost, scalable intervention. (See full summary.)

  2. PNAS published an RCT entitled "Unconditional Cash Transfers Reduce Homelessness." Quick take: The RCT found no discernible impact on any primary or secondary study outcome. The claimed effect on homelessness (an exploratory outcome) is unreliable due to sample loss >50%. (See full summary.)

  3. Pediatrics published an RCT of a program encouraging school children to drink more water (e.g., in lieu of sweetened drinks), to prevent overweight. Quick take: The study's claims of effectiveness notwithstanding, it found no impact on the prespecified primary outcomes: Body Mass Index (BMI) z-score & BMI percentile. (See full summary.)

  4. HUD published a large RCT of Integrated Wellness in Supportive Housing (IWISH), providing a Wellness Director & Nurse for older adults at HUD-assisted multifamily properties. Quick take: High-quality RCT finds no impact on any targeted outcome (e.g., transitions to long-term care, unplanned hospitalization) over 3yrs. (See full summary.)

  5. Science Magazine published an RCT of the Modeling Practices in Calculus (MPC) approach to teaching college calculus. Quick take: Sizable positive impact on end-of-semester test scores, but findings aren't definitive due to a key study limitation (faculty self-selected to teach MPC). (See full summary.)

Evidence-Based Policy Newsletter #12

August 8, 2023

  1. @EEPAjournal published long-term RCT findings for Future Forward — a program providing two years of literacy tutoring starting in K-2 in low-income Milwaukee schools. Quick take: High-quality RCT finds sustained, modest gain (1/3 grade level) on district reading test thru 7 year follow-up. (See full summary.)

  2. @IPRatNU posted long-term RCT results for the Graduation program in Ethiopia, providing very poor households a productive asset (e.g. livestock) & training in its use. Quick take: High-quality RCT finds a sizable 18% gain in household consumption at the 3-year mark, fading somewhat to 9% at 7 years. (See full summary.)

  3. @EconMobility published an RCT of English for Advancement (EfA), providing adult English-language learners with language instruction, career coaching, & job placement assistance. Quick take: High-quality RCT finds smallish (7%) earnings gains at 2-year mark, possibly dampened by pandemic. (See full summary.)

  4. @iza_bonn published an RCT of ReHire Colorado, a state program providing transitional, subsidized jobs to low-income unemployed or underemployed individuals. Quick take: High-quality RCT finds little impact on participant earnings or employment once the transitional jobs end. (See full summary.)

  5. US Dept of Labor (DOL) published RCT results for DOL's Ready to Work (RTW) Partnership Grants, providing employment services to long-term unemployed or underemployed workers. Quick take: High-quality RCTs of 4 RTW projects find no discernible positive earnings effects over 3-4 years. (See full summary.)

Evidence-Based Policy Newsletter #11

July 17, 2023

  1. Colleagues and I have monitored the RCT literature for many years for policy-important findings, but this study launched in the 80s only recently crossed our radar: A large smoking-cessation RCT found sizable impacts on sustained quit rates & mortality over 14.5 years. (See full summary.)

  2. Why a Control Group is Needed to Determine Whether a Social Program is Effective, in 4 simple charts. The example I’ll use is HHS’s Comprehensive Child Development Program (CCDP), which provided intensive case management services to low-income families with young children. (See full summary.)

  3. 21 of the RCTs I’ve tweeted on since last summer found disappointing effects (e.g., no discernible positive impacts on the primary outcomes). 7 of these RCTs (i.e., one-third) nevertheless portrayed the results as strongly positive in the study abstract. Here are specifics 🧵(See full summary.)

  4. US Department of Labor released RCT results for 2 DOL-funded "Pay for Success" projects - Roca (in MA) for justice-involved young men, & Center for Employment Opportunities (in NY) for adult parolees. Quick take: No impacts (yet) on recidivism or employment; the RCTs are ongoing. (See full summary.)

Evidence-Based Policy Newsletter #10

June 12, 2023

  1. @metisassociates and @CUNY released RCT results for Accelerate, Complete, Engage (ACE) at John Jay College, providing comprehensive support to help students complete a bachelor’s degree. Quick take: High-quality RCT finds large 12%-point gain in bachelor’s completion at 4-year mark. (See full summary.)

  2. @IESResearch published a large RCT of providing college-intending, disadvantaged adults with personalized texts on the college enrollment & financial aid process. Quick take: High-quality RCT finds no impact on rate of college enrollment or financial aid form completion. (See full summary.)

  3. @HUDgov published long-term (20-year) RCT results for Jobs Plus, providing public housing residents with employment services, along with financial incentives & support for work. My quick take: No impact on earnings & employment for full 6-site sample, & the reported positive impacts at 3 sites aren’t reliable. (See full summary.)

  4. @scienceadvances published an RCT of Working on Womanhood — a group counseling program for high school girls from disadvantaged backgrounds. Quick take: High-quality RCT finds no discernible effect on rates of clinical depression, anxiety, or PTSD (primary outcomes) at 1-year mark, but suggestive effects on a broader PTSD measure. (See full summary.)

  5. @AnnenbergInst posted RCT findings for Core Knowledge Charter Schools. Quick take: Notwithstanding study’s claim of large effects, no discernible effect on English Language Arts (ELA) achievement in 6th grade (primary outcome) — although suggestive effects in grades 4 & 5. (See full summary.)

Evidence-Based Policy Newsletter #9

April 24, 2023

  1. @TheLancet published an RCT of a low-cost program led by non-physician community healthcare providers to reduce hypertension in rural China. Quick take: Blockbuster results - a 1/3 reduction in cardiovascular events including death over 3 years, with major global health implications. (See full summary.)

  2. @JPAM_DC published an RCT examining the effect of NY City's Summer Youth Employment Program (SYEP) on youth crime. Quick take: Large, high-quality RCT finds negligible impact on youth crime over a 5-year follow-up (authors' claims notwithstanding). (See full summary.)

  3. Prevention Science @SPRORG published an RCT of High School FLASH, a sex ed program, reporting that it reduces homophobic & transphobic beliefs. My quick take: Not a reliable result (only suggestive), as this was a post-hoc analysis that can easily yield chance findings. (See full summary.)

  4. @MDRC_News reported RCT results for ASAP Ohio, which provides academic, personal, & financial support to low-income community college students. Quick take: High-quality RCT with blockbuster results - 15% point gain in college graduation & 11% gain in earnings at the 6 year mark. (See full summary.)

  5. American Economic Review published RCT findings for Saga Education's high-dosage math tutoring of low-income 9th & 10th graders in Chicago. Quick take: High-quality RCTs find sizable effect on math achievement that persists 1-2 years after tutoring ends. (See full summary.)

Evidence-Based Policy Newsletter #8

March 20, 2023

  1. @CMSGov reported RCT results for Million Hearts — a national HHS pay-for-performance model to prevent heart attacks & strokes in Medicare patients. Quick take: No impact on first-time heart attacks/strokes or Medicare spending over 3–4 yrs, but follow-up is ongoing. (See full summary.)

  2. A new Cochrane review of RCTs casts doubt on the effectiveness of masks in community settings for preventing COVID infection. But adherence to proper masking is often low in such settings. We don’t yet know if people who *diligently* mask are protected. Here’s how to find out. (See full summary.)

  3. @LEOatND reported RCT results for Padua, an intensive case management program for low-income adults in Texas, aimed at moving them permanently out of poverty. Quick take: High-quality RCT finds suggestive, but at best modest, effects on earnings, employment, etc over 2–5 yr follow-up. (See full summary.)

  4. @iza_bonn published an RCT evaluating generous unconditional cash transfers ($8K/year) to low-income households in Spain, to reduce poverty. Quick take: High-quality RCT finds sizable (9% point) adverse effect on employment at roughly the 2 year mark. (See full summary.)

  5. @MDRC_News published an RCT of @GrameenAmerica microfinance program in Union City NJ, providing low-income women with small loans & training to start or expand a business. Quick take: High-quality RCT finds modestly encouraging gains in economic well-being at 3 yr mark. (See full summary.)

Evidence-Based Policy Newsletter #7

February 13, 2023

  1. Chris Blattman et. al. report long-term RCT results for a program of cognitive-behavior therapy (CBT) + cash transfers for criminally-involved men in Liberia. Quick take: High-quality RCT finds sizable reduction in anti-social behavior (e.g., drug-selling, robbery) at 10-year follow-up. (See full summary.)

  2. The Vision for Baltimore RCT, evaluating a citywide program of free vision screening & eyeglasses in elementary & middle school, recently published results. Quick take: High-quality RCT finds no significant impact on English language arts (ELA) or math scores on the state test over 2 years. (See full summary.)

  3. Mathematica published RCT findings for SSI PROMISE, a $230M federal demonstration program to improve education & workforce outcomes of youth with disabilities. Quick take: High-quality RCTs find none of 6 programs in the demonstration improved education or earnings over 5 years. (See full summary.)

  4. NBER posted RCT results for READI Chicago, providing transitional jobs, cognitive behavioral therapy & other services to men at high risk of violence perpetration or victimization. Quick take: High-quality RCT finds no significant impact on overall serious violent crime over 20 months. (See full summary.)

  5. Family Relations published an RCT of Love Notes, a pregnancy prevention program for low-income youth ages 14–19. Quick take: RCT finds nearly 50% reduction in pregnancy rate over 12 months that is encouraging but not yet definitive (due to results’ sensitivity to study methods). (See full summary.)

Evidence-Based Policy Newsletter #6

January 10, 2023

  1. @JAMAInternalMed published an RCT examining whether remediation of abandoned housing in Philadelphia reduces surrounding crime. Quick take: Mostly well-conducted RCT finds reductions in gun-related crime that are moderately encouraging but not yet reliable — (might be due to chance). (See full summary.)

  2. @EEPAjournal published long-term RCT findings for Washington DC’s school voucher program — providing private school scholarships to low-income students. Quick take: High-quality RCT finds no effect on college enrollment rate within 2 years of scheduled high school graduation. (See full summary.)

  3. Having recently shared some of the most promising recent RCT findings, I now share one of the most disconcerting: The long-term results of the Tennessee voluntary pre-k study. Quick take: High-quality RCT finds adverse effects on student achievement & behavior in 6th-grade follow-up. (See full summary.)

  4. @MDRC_News recently published RCT findings for Detroit Promise Path (DPP), a program providing coaching & other supports to Detroit-area community college students. Quick take: High-quality RCT finds no discernible impact on college degree or certificate receipt at 4-yr mark.(See full summary.)

  5. A recent RCT evaluated CenteringPregnancy, a widely-used group prenatal care program that aims to improve racial equity in birth outcomes. Quick take: High-quality RCT finds no impact on rate of preterm birth or low birthweight for whole sample, or Black or Hispanic subgroups. (See full summary.)

Evidence-Based Policy Newsletter #5

November 29, 2022

  1. My tweets in coming days [items 2–5 below] will summarize some of the most promising RCT findings — in job training, education, and housing — over the past 18 months. I’ve already shared one: The blockbuster findings for @YearUp job training for low-income young adults.

  2. In March, @MDRC_News reported results of an RCT of @PerScholas — an employment/training program low-income adults, preparing them for well-paying jobs in Information Technology. Quick take: High-quality RCT finds sizable earnings gain (16%, or $5200/year) at the 5–7 year follow-up. (See full summary.)

  3. Another especially promising RCT finding of the past 18 months: Castleman & Barr’s study of @BottomLineOrg (1-on-1 advising to help low-income students get into & graduate from college). Quick take: High-quality RCT finds large (8% point) increase in 4-year college degree completion. (See full summary.)

  4. Another especially promising RCT finding of the past 18 months: @EconMobility’s study of @ProjectQUESTInc occupational training of low-income adults for well-paying healthcare jobs. Quick take: High-quality RCT finds sizable long-term earnings gains (15–20%, or about $5,000/year). (See full summary.)

  5. Another especially promising RCT of the past 18 months: @urbaninstitute’s study of the Denver Supportive Housing SIB initiative, providing housing & supportive services to homeless people with over 7 prior arrests. Quick take: High-quality RCT finds roughly 30–40% reductions in jail & shelter use, emergency department visits, & arrests over 2–3 yrs. (See full summary.)

  6. @OPRE_ACF published an excellent summary of results of 10 RCTs of job training programs for low-income adults, funded under HHS’s “PACE” & “HPOG 1.0” initiatives. Quick take: High-quality RCTs find no earnings impacts for 9 programs & large impacts for 1 program. (See full summary.)

Other Items:

  1. I’m pleased to share my recent op-ed in The Hill — “The Good and Bad News About Declining U.S. Poverty Rates.” It makes that case that, while we’re alleviating economic hardship, we’re not breaking the poverty cycle. (Link to op-ed).

Evidence-Based Policy Newsletter #4

November 3, 2022

  1. @NatureHumBehav just published an RCT in Germany of a program that teaches self-regulation skills in 1st grade (“Mental Contrasting with Implementation Intentions”). Quick take: The RCT had a key flaw, but found large effects on reading achievement — so findings are still somewhat promising. (See full summary.)

  2. Earlier this year, @OPRE_ACF @abtassociates reported long-term RCT results for the federal Health Profession Opportunity Grants program (HPOG 1.0), which trains low-income adults for healthcare jobs. Quick take: Large, high-quality RCT finds no earnings impacts over 6 years. (See full summary.)

  3. @bmj_latest published a large RCT in the UK evaluating the effects of offering pregnant smokers shopping vouchers up to $440 to engage with stop smoking services & stop smoking. Quick take: High-quality RCT finds the offer roughly doubles smoking abstinence in late pregnancy. (See full summary.)

  4. @OPRE_ACF @abtassociates published an RCT of Valley Initiative for Development and Advancement (VIDA)’s college-based occupational training for low-income students. Quick take: High-quality RCT finds sizable effect on degree/credential receipt, but it leads to no earnings gain over 7 years. (See full summary.)

  5. @abtassociates published an RCT of Professional Training Corps (PTC) — a lower-cost version of Year Up’s proven-effective job training program for low-income adults. Quick take: High-quality RCT finds PTC has no effect on earnings … yet (the PTC program is still being developed/refined). (See full summary.)

Evidence-Based Policy Newsletter #3

October 14, 2022

  1. Researchers David Dubois & Carla Herrera reported interim results of a national RCT of Big Brothers Big Sisters (BBBS) Community-Based Mentoring Program. Quick take: High-quality RCT finds sizable reductions in youth arrests and substance use over 18-month follow-up. (See full summary.)

  2. @TheLancetPH published the first RCT of a commercial group weight-loss program — Weight Watchers— with long-term (5-year) follow-up. Quick take: Pretty good RCT found minimal weight-loss impacts (2–4 lbs) at the 5-year mark that weren’t statistically significant. Sizable impacts at the 1-year mark had mostly faded. (See full summary.)

  3. A quick detour to a medical RCT, which affects me having newly turned 60. @NEJM just published the 1st large RCT of colonoscopy screening to prevent colorectal cancer. Quick take: High-quality RCT shows screening benefit, but much smaller than suggested by prior non-RCTs. (See full summary.)

  4. @PNASNews published an RCT of Procedural Justice training for police, to foster fair/respectful treatment of citizens. Quick take: High-quality RCT found no effect on community perceptions of police legitimacy (primary outcome) but suggestive effects on some other outcomes. (See full summary.)

Evidence-Based Policy Newsletter #2

September 27, 2022

  1. @JPAL published an RCT of three STEM summer programs for underrepresented high schoolers. Quick take: The main findings — including large impacts on college STEM degree receipt — are quasi-experimental, not RCT; the RCT impacts are mostly small & not statistically significant. (See full summary.)

  2. @OPRE_ACF @MDRC reported an RCT of Individual Placement & Support — a job counseling, search & support program for adults with disabilities. Quick take: High-quality RCT found an impressive 26% gain in two-year earnings, but no significant effect on time employed. (See full summary.)

  3. @nberpubs published an RCT of Bridge private schools in Kenya that’s gotten attention for the big impacts on student achievement. Quick take: Promising findings but not definitive due to key study limitations (especially in methods of testing achievement). (See full summary.)

  4. @OPRE_ACF @MDRC_News published an RCT of Paycheck Plus — Atlanta, evaluating a significant expansion of the Earned Income Tax Credit (EITC) for low-income childless workers. Quick take: High-quality RCT finds no meaningful effects on workers’ economic well-being over three years. (See full summary.)

Other Items:

  1. I’m pleased to share my recent Washington Post op-ed on the need for evidence-based policy reform.

Evidence-Based Policy Newsletter #1

September 9, 2022

  1. @IESResearch recently published results of an RCT evaluating a program that trained schools in “Multi-Tiered Systems of Support for Behavior” (MTSS-B) — an approach used by over 25,000 US schools. Quick take: a mostly well-conducted RCT, with mostly disappointing findings. (See full summary.)

  2. @nberpubs recently posted a report on the Baby’s First Years RCT, studying initial effects of providing an unconditional cash transfer to low-income US mothers with newborns. Quick take: High-quality RCT, interesting results but too early to tell if there’s any meaningful child or family benefit. (See full summary.)

  3. Another ongoing US RCT of unconditional cash transfers, in Stockton CA reported 12-month findings last year. Quick take: The report doesn’t provide key info needed to gauge study validity, & what info it does provide suggests major problems (e.g. severe differential attrition). (See full summary.)

  4. In May, @OPRE_ACF & @abtassociates reported new findings of a large RCT of @YearUp — a job training program for low-income young adults that focuses on fast-growing economic sectors. Quick take: High-quality RCT with blockbuster results: long-term earnings gains of 30% ($8000/year). (See full summary.)

  5. @JAMA_current recently published initial results of a large South Carolina RCT of @NFP_nursefamily partnership — a nurse home visiting program for first-time mothers starting in pregnancy. Quick take: High-quality RCT finds no impact on adverse birth outcomes (but study is ongoing with additional results to come). (See full summary.)

We use cookies to improve your experience and to help us understand how you use our site. Please refer to our cookie notice and privacy statement for more information regarding cookies and other third-party tracking that may be enabled.

Intuit Mailchimp logo