Reanalysis of Sanders et al. (2024): An Umbrella Review of the Benefits and Risks Associated with Youths' Interactions with Electronic Screens
Sanders et al. (2024) made the central claim that effects found in eight meta-analyses are “strong evidence” (P<0.001) for various health and educational outcomes in children associated with use of electronic screens (“screen time”). The eight effects highlighted as strong evidence ranged in size from r = –0.14 to 0.33. Although some of the primary studies were experimental and some observational, and the individual claims of strong evidence were causal, the design of the analysis – the umbrella review – does not pool data from meta-analyses. The authors converted effects in the screen time literature to a common measurement (r) and summarized the meta-analyses that survived exclusion. Therefore, the paper as a whole made a descriptive claim that this evidence exists in the literature and, in some cases, it is strong evidence. Sanders et al. (2024) excluded meta-analyses when they found significant publication bias using Egger’s and excess significance tests. The remaining effect sizes were not corrected for publication bias. This robustness replication calculated the same eight effects using a technique to correct for publication bias (PET-PEESE) and attempted to find coding, mathematical, data, and reporting errors.
This analysis found publication bias reduced the effect size in three of the eight meta-analyses to such a degree that these findings failed to replicate. It also found a pattern of results indicative of p-hacking in the screen time literature, and evidence that a more moderate interpretation of the data could have been presented in Sanders et al. had the authors chosen a different set of eight – or the full set – of high-certainty effects.