What Is P-Hacking in Research? A Clear Guide

What is p-hacking in research?

P-hacking happens when researchers try different ways of analyzing data, pay attention to which choices produce a desirable p-value, and present the resulting analysis without making that search clear. A choice might involve which outcome to measure, which participants to include, which variables to adjust for, or when to stop collecting data. Each choice can be reasonable in some circumstances. The problem is letting the result guide the choice while reporting the analysis as though it had been settled in advance.

Imagine checking whether a new study routine helps students remember material. There may be several plausible quiz scores to examine and several plausible ways to compare students. Exploring those possibilities can be useful. But a reader needs to know whether the reported result was the original test or the one that looked strongest after many attempts. That distinction changes how surprising the result really is.

P-hacking does not require fabricated data, and a researcher may not recognize that their flexibility is affecting the evidence. It also does not mean every exploratory analysis is improper. The key question is whether the path from question to result is visible to the reader.

Why repeated searches change the odds

A p-value describes how unusual a result, or a more extreme one, would be under a specified statistical model if the null hypothesis were true. It is not the probability that the null hypothesis is true. Its interpretation also depends on the test and its assumptions. OpenStax explains that a hypothesis test can produce a Type I error: rejecting a true null hypothesis.

Testing many questions creates more opportunities for that kind of error. If a researcher reports only the smallest p-value, readers do not see the other chances the research process had to produce it. NIST’s guidance on multiple comparisons addresses this issue: conclusions drawn from a collection of tests need to account for the collection, not just an individual result.

This is why a small p-value discovered after a broad search is not equivalent to the same p-value from a single, prespecified test. The observed data have not changed. What has changed is the process that selected the result for attention.

A worked example with synthetic results

Every number in this example is synthetic and illustrative. Suppose a hypothetical research team considers 20 independent tests of a study routine. Assume that none of the tested effects is real, that each test produces a valid p-value under its null hypothesis, and that the team treats a p-value below 0.05 as noteworthy. For any one of these illustrative tests, the chance of crossing that threshold is 0.05 under the stated assumptions.

Across all 20 illustrative tests, the chance of seeing at least one p-value below 0.05 is 1 − (1 − 0.05)^20, or about 0.642. That is roughly a 64% chance of at least one apparently noteworthy result, even though the example assumes no real effects. This calculation relies on the illustrative assumption that the tests are independent; real analyses often share data or outcomes, so their results may be dependent.

Suppose the team finds a synthetic p-value of 0.03 and reports only that test. It clears the illustrative 0.05 threshold viewed alone, but the report hides the wider search. As one simple multiple-testing approach, a Bonferroni threshold for this illustrative set would be 0.05 ÷ 20 = 0.0025. The synthetic p-value of 0.03 would not clear that threshold. Bonferroni is not the answer to every research question; the example shows why the number of opportunities to find a result matters.

Where analysis flexibility appears

Some choices are easy to overlook. Researchers might compare several outcomes, divide participants into subgroups, try different adjustment variables, or remove observations under different rules. They might also inspect results while data collection is still underway and decide whether to continue. Such decisions can change the reported finding even when nobody alters a data point.

There are legitimate reasons to make each kind of choice. An observation may be invalid, or an unplanned subgroup may reveal a worthwhile question. The concern arises when choices are made in response to the results but described as if they were independent of them. A clearly labeled exploratory result can guide future work; it should not quietly take on the evidential status of a planned test.

Not every study uses a single p-value or a single fixed analysis. That makes transparency more important, not less. Readers should be able to tell which decisions came before the results were known, which came afterward, and how many relevant analyses were considered.

How researchers and readers can respond

Researchers can record their main questions, outcomes, exclusion rules, and analysis methods before examining the results. They can then report departures from that plan, describe exploratory analyses as exploratory, and disclose the range of relevant tests. When several hypotheses are tested, a suitable multiple-comparison method may help control erroneous conclusions. The appropriate method depends on the questions being asked and the structure of the tests.

A follow-up study offers a different check. If an exploratory finding is tested in new data with the question and method specified beforehand, it is less dependent on the original search that revealed it. Replication does not guarantee that a claim is correct, but it provides evidence that cannot be explained solely by selecting a favorable result from the first dataset.

As a reader, look beyond whether a result crosses a conventional threshold. Ask what was planned, what else was tested, and whether the reported analysis was selected after the results appeared. Those questions are useful even when a paper reports no p-values: selective reporting can affect other summaries of evidence too.

A practical checklist

Before treating a finding as strong evidence, check whether the main question and analysis were specified in advance. Look for a clear account of outcomes, subgroups, exclusions, and alternative analyses. If many tests were conducted, check whether the conclusions acknowledge that wider search and whether the method for handling multiple comparisons fits the research question.

Finally, distinguish a promising lead from a confirmed finding. An unexpected result can be valuable when it is reported honestly and tested again. The goal is not to prevent researchers from exploring data; it is to let readers see the exploration so they can judge the strength of the evidence.