When Samples Shape Significance
Random sampling yields unbiased estimates, yet sampling variation alone can produce incorrect conclusions. Using a setting where repeated draws from the same data-generating process are observable, we replicate findings from 24 papers in top economics journals and re-estimate each result using 40 independent resamples. Across 3,640 resampled results, t-statistics shrink by 31%, frequently leading to lost significance. Proxies for researcher degrees of freedom and sampling noise predict reduced significance. Our exercise provides novel evidence quantifying the effect of sampling variability on empirical research and highlights how statistical evidence can vary even when holding research design fixed.
JEL-Klassifikation: C12, C18, C80