The 20% Statistician

Paul Simon’s says there are 50 ways to leave your lover, and a similar abundance characterizes the contemporary literature on how to analyse a replication study. The recent SCORE replication project in Nature makes this explicit by reporting “13 replication success metrics along with the number of papers to which each metric could be applied” (Tyner et al., 2026). The reason for this, as the auth…

With my collaborators, I am increasingly performing qualitative research. I find qualitative research projects a useful way to improve my understanding of behaviors that I want to explore with other methods in the future. For example, some years ago I performed qualitative interviews with researchers who believed their own research had no value whatsoever. Although I did not intend to publish the…

Note: This humorously intended sarcastic blog post directly mimics classifications of psychological disorders in the Diagnostic and Statistical Manual of the American Psychological Association, but Dogmatic Bayesianism is not in the DSM-5 as an actual psychological disorder - for now. Dogmatic Bayesianism Disorder Diagnostic Criteria F61.1 A. Individuals are convinced that they know what other pe…

In a recent post on Bluesky, where Richard Morey reflects on a paper he published with Clintin Davis-Stober that points out concerns with the p-curve method (Morey & Davis-Stober, 2025) , he writes: Also, I think people should stop using forensic meta-analytic techniques that have not been adequately vetted by experts in statistics. The p-curve papers have very little statistical detail, and were…

Update 30/09/2025: I have added a reply by Andrew Gelman below my original blog post. We recently posted a preprint criticizing the idea of Type S and M errors ( https://osf.io/2phzb_v1 ). From our abstract: “While these concepts have been proposed to be useful both when designing a study (prospective) and when evaluating results (retroactive), we argue that these statistics do not facilitate the…

Researchers increasingly use the Open Science Framework (OSF) to share files, such as data and code underlying scientific publications, or presentations and materials for scientific workshops. The OSF is an amazing service that has contributed immensely to a changed research culture where psychologists share data, code, and materials. We are very grateful it exists. But it is not always the most …

Are meta-scientists ignoring philosophy of science (PoS)? Are they re-inventing the wheel? A recent panel at the Metascience conference engaged with this question, and the first sentence of the abstract states “Critics argue that metascience merely reinvents the wheel of other academic fields.” It’s a topic I have been thinking about for a while, so I will share my thoughts on this question. In t…

& &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&&&&&&&&&&&&&&&&≠&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&& &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&≥&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& & &&&&&&&&&& &&&&&&&& It is increasingly common for researchers to preregister their studies ( Spitzer and Mueller 2023 ; Imai et al. 2025 ) . As preregistration is a new practice,…

Many of the facts in this blog post come from the biography ‘Neyman’ by Constance Reid . I highly recommend reading this book if you find this blog interesting. In recent years researchers have become increasingly interested in the relationship between eugenics and statistics, especially focusing on the lives of Francis Galton, Karl Pearson, and Ronald Fisher. Some have gone as far as to argue fo…

In the first partially in person scientific meeting I am attending after the COVID-19 pandemic, the Perspectives on Scientific Error conference in the Lorentz Center in Leiden, the organizers asked Eric-Jan Wagenmakers and myself to engage in a discussion about p-values and Bayes factors. We each gave 15 minute presentations to set up our arguments, centered around 3 questions: What is the goal o…

This is a post-publication peer review of " J oy and rigor in behavioral science ". A response by the corresponding author, Leslie John, is at the bottom of this post - make sure to read this as well. In a recent paper “Joy and rigor in behavioral science” https://doi.org/10.1016/j.obhdp.2021.03.002 Hanne Collins, Ashley Whillans, and Leslie John aim to examine the behavioral and subjective conse…

In 2009 I attended a European Social Cognition Network meeting in Poland. I only remember one talk from that meeting: A short presentation in a nearly empty room. The presenter was a young PhD student - Stephane Doyen. He discussed two studies where he tried to replicate a well-known finding in social cognition research related to elderly priming, which had shown that people walked more slowly af…

If you have educational material that you think will do a better job at preventing p-value misconceptions than the material in my MOOC, join the p-value misconception eradication challenge by proposing an improvement to my current material in a new A/B test in my MOOC. I launched a massive open online course “Improving your statistical inferences” in October 2016. So far around 47k students have …

On July 28, 2020, the first Dutch academic has been judged to have violated the code of conduct for research integrity for p -hacking and optional stopping with the aim of improving the chances of obtaining a statistically significant result. I think this is a noteworthy event that marks a turning point in the way the scientific research community interprets research practices that up to a decade…

This blog post is now included in the paper "Sample size justification" available at PsyArXiv. When you perform an experiment, you want it to provide an answer to your research question that is as informative as possible. However, since all scientists are faced with resource limitations, you need to balance the cost of collecting each additional datapoint against the increase in information that …

This is a guest blog by Tiago Lubiana , Ph.D. Candidate in Bioinformatics, University of São Paulo. Read also Part 1 , Part 2 , and Part 3 of The Red Team Challenge Two remarkable moments as a researcher are publishing your first first-author article and the first time a journal editor asks you to review a paper. Well, at least I imagine so. I haven’t experienced either yet. Yet,for some reason, …

By Daniel Lakens & Leo Tiokhin Also read Part 1 and Part 2 in this series on our Red Team Challenge . Six weeks ago, we launched the Red Team Challenge : a feasibility study to see whether it could be worthwhile to pay people to find errors in scientific research. In our project, we wanted to see to what extent a “Red Team” - people hired to criticize a scientific study with the goal to improve i…

by Nicholas A. Coles, Leo Tiokhin, Ruben Arslan, Patrick Forscher, Anne Scheel, & Daniël Lakens All else equal, scientists should trust studies and theories that have been more critically evaluated. The more that a scientific product has been exposed to processes designed to detect flaws, the more that researchers can trust the product (Lakens, 2019; Mayo, 2018). Yet, there are barriers to ad…

Based on our recent paper explaining power analysis for ANOVA designs, in this post I want provide a step-by-step mathematical overview of power analysis for interactions. These details often do not make it into tutorial papers because of word limitations, and few good free resources are available (for a paid resource worth your money, see Maxwell, Delaney, & Kelley, 2018 ). This post is a bit te…

When you perform multiple comparisons in a study, you need to control your alpha level for multiple comparisons. It is generally recommended to control for the family-wise error rate, but there is some confusion about what a 'family' is. As Bretz, Hothorn, & Westfall (2011) write in their excellent book “Multiple Comparisons Using R” on page 15: “The appropriate choice of null hypotheses being of…