A wet street does not make an umbrella a weather control
Suppose you count open umbrellas outside your window. On afternoons with more umbrellas, the pavement is usually wetter. The observation could be accurate and useful. It still would not justify confiscating umbrellas to keep the street dry.
Rain supplies a straightforward explanation for both observations. It wets the street and prompts people to open umbrellas. Your count detects something happening in the world; changing the count need not change the weather.
That distinction matters whenever a chart becomes a recommendation. “These things occur together” describes an association. “Do this to obtain that result” adds a causal claim. Before following the recommendation, we need evidence for the extra step.
One pattern, several possible explanations
Imagine a survey linking time spent practicing with higher test scores. Here are three possible accounts, offered as hypotheses rather than findings about any particular classroom:
| Possible explanation | What could be happening? |
|---|---|
| Practice helps performance | Extra practice improves what students can do. |
| Performance encourages practice | Students who already succeed choose to practice more. |
| Another factor affects both | A class offers extra practice alongside better instruction. |
These explanations can coexist. A precise estimate of the association does not, by itself, assign a share to each one. A common cause that interferes with that inference is called a confounder.
Judea Pearl’s account of causal inference makes the distinction explicit: observing a variable and intervening on it are different questions. Drawing possible causal connections helps expose what a study must distinguish. The arrows represent assumptions to defend, not facts manufactured by drawing them.
What is missing from a success story?
A person adopts a new routine, gets a good result, and credits the routine. That sequence is compatible with the routine working. It also leaves out the result they would have obtained without it.
A laboratory task shows how misleading repeated successes can be. In a 2021 study by Moreno-Fernández and colleagues, participants examined fictional patient records involving an invented medicine, Batatrim. The records were arranged so that recovery was equally probable with and without the medicine. Most participants in the first study nevertheless rated it as effective.
The result concerns judgments in a controlled fictional task. It is not evidence about a real treatment or a diagnosis of people who make this mistake. The paper also found an association between two judgment biases, but did not support the authors’ proposed mechanism connecting them. That distinction is an example of the care the topic requires: a finding and its explanation need separate support.
An unfair plant comparison cannot isolate plant food
Imagine feeding the plants on a bright windowsill while leaving unfed plants at the back of the room. A month later, the fed group is taller. Both food and light changed, so the result compares a package of conditions rather than isolating plant food.
Another hundred pots in that arrangement could estimate the combined difference more precisely. They would still leave its two components entangled. A larger sample adds information to the comparison you designed; it does not redesign it.
A useful redesign starts with similar plants and gives both groups comparable light. Randomly assign which plants receive food, instead of choosing the healthiest plants or the most convenient shelf. NIST’s description of completely randomized designs explains the central step: assign treatment levels to experimental units at random, then compare their responses.
Random assignment does not promise identical groups in every small experiment. It provides a selection procedure that does not favor plants because of their starting condition. Replication gives the comparison more information; consistent watering and measurement keep other avoidable differences from being introduced.
Even a persuasive result would concern the species, conditions, and treatment tested. It would not establish that every plant benefits in every setting.
A dramatic match may be the winner of an invisible contest
Imagine recording your commute time alongside dozens of unrelated daily details. After enough searching, one pair might produce a striking graph. Showing only that graph conceals all the less impressive comparisons.
NIST warns that repeatedly making comparisons does not preserve the significance level of a single comparison. A claim chosen after inspecting many results needs to account for that search. Specifying the question beforehand and using appropriate multiple-comparison methods address this problem.
Checking predictions on fresh data offers a different safeguard. Galit Shmueli’s paper on explanation and prediction describes why evaluating a model only on the observations used to build it can exaggerate predictive performance. Success on unseen observations is a stronger predictive test. It still does not establish that manipulating the predictor changes the outcome.
Useful predictions can point to the wrong intervention
Return to the window. Umbrellas may help you decide whether to expect wet pavement before you step outside. That is a legitimate use of the association.
Now imagine an umbrella shop offering a free demonstration on a dry afternoon. People outside open their new umbrellas to inspect them. The usual clue becomes less reliable because a different process is producing it.
This hypothetical change illustrates why a predictor needs to be evaluated in the setting where it will be used. It also exposes the danger of treating an indicator as a target. Making a dashboard number look better is useful only if the action changes the outcome you actually care about.
Causal evidence does not always require a laboratory
Some interventions cannot be randomly assigned. Researchers can sometimes use institutional rules or policy changes that create a useful comparison between groups. The Royal Swedish Academy of Sciences’ explanation of the 2021 economics prize describes how such natural experiments can support causal inference.
The comparison still needs a defensible reason for treating the groups as informative about each other. Participation choices and other differences matter. Some methods estimate an effect for people whose behavior the natural experiment changed, rather than for everyone. An observational study is neither automatically causal nor automatically useless; its design and assumptions determine what it can establish.
For an everyday claim, first spell out the proposed action. If users of a study app obtain better grades, is the recommendation to give that app to another student? Then ask who selected the users, what else differed, and what comparison separates those differences from the app’s contribution.
A plausible alternative explanation is a reason to investigate, not proof that the app fails. The goal is to find evidence that can distinguish the explanations. A correlation gives you somewhere to look. A causal conclusion needs a comparison that answers what changing something would do.
