The Danger of Small Samples: Why We Must Question P-Values and Statistical Significance

At the heart of many misleading studies lies the concept of the p-value. A p-value is a number between zero and one that indicates the probability of observing your data, or something more extreme, assuming that the null hypothesis is true. In simpler terms, if a drug actually has no effect, a p-value of 0.05 means there is only a five percent chance that you would see a result as strong as the one you got purely by random chance. Many fields have adopted a p-value below 0.05 as the threshold for calling a result “statistically significant.” This arbitrary cutoff has become a golden ticket to publication, but it is also a recipe for disaster, especially when sample sizes are small.

Consider a small study with only twenty participants. Because the sample is tiny, random fluctuations can easily produce large differences between groups. A handful of unusually high or low measurements can swing the average dramatically, generating a low p-value even when there is no real effect at all. This is the essence of what statisticians call low statistical power. A study with low power has little chance of detecting a true effect, but it also has a surprisingly high chance of producing a false positive when researchers test many hypotheses. And because small studies are cheap and quick to run, they are common in fields like psychology, nutrition, and early-phase medical research.

The problem becomes even more treacherous when researchers engage in what is known as p-hacking. P-hacking is the practice of analyzing data in multiple ways until a significant p-value appears. Researchers might exclude certain participants, add control variables, transform the data, or test subgroups until the magic 0.05 threshold is crossed. With a small sample, this kind of flexibility is especially dangerous. Because the data are noisy, there are many possible ways to slice them, and some of those slices will inevitably yield a low p-value by chance. The result is a published finding that is nothing more than a mirage, yet it can influence policy, spark media stories, and steer further research.

A classic example is the story of the so-called “power pose” research. A series of small studies claimed that holding expansive postures for two minutes could alter hormone levels and make people feel more powerful. These findings went viral, but later large-scale replication attempts failed to reproduce the effects. The original studies had small samples, and the researchers had likely tried multiple analyses before settling on the ones that worked. The lesson is not that the researchers were dishonest, but that small samples combined with flexible analysis strategies can produce compelling but false results.

To guard against this, readers of scientific claims should look for studies with adequate sample sizes. A simple rule of thumb is that the larger the sample, the more reliable the estimate. But sample size alone is not enough. Researchers should also pre-register their hypotheses and analysis plan before collecting data, so that p-hacking is impossible. And consumers of science should be skeptical when a study reports an astonishing effect from a handful of participants, especially if the effect size is huge and the confidence intervals are wide. Wide confidence intervals indicate that the true effect could be tiny, zero, or even in the opposite direction.

Beyond sample size, the replication crisis has taught us that single studies, no matter how statistically significant, should never be taken as definitive. Science advances through converging evidence. When multiple independent labs, using large samples and pre-registered designs, find the same result, confidence grows. But when a single small study makes a bold claim, the wise doubter asks: Has this been replicated? How many participants were involved? Could the result be a fluke of random chance?

The ultimate point is that doubt is not the enemy of science; it is its engine. Every significant p-value should be met with a question: How likely is this to be real? By understanding the fragility of small samples and the temptations of p-hacking, we can transform our doubt from a paralyzing force into a catalyst for sharper thinking. We do not have to reject all scientific claims, but we must demand evidence that is robust, transparent, and reproducible. Only then can we navigate the sea of headlines with confidence, knowing that the best science is the most doubted science.