The Average Person Doesn’t Understand Statistics
Get it? I’ll come back to this title — in the meantime, what follows are some common mistakes in statistics and probability, to show you what I’m talking about.
1: Correlation = Correlation
“Wow, Оставь, you’re right, this is so complicated. Thank you for explaining it to my dumb little brain.” Shut up, hypothetical smarmer, I don’t mean to condescend to you, I just want to clarify some terms.
People often say “correlation does not equal causation”, which means essentially that just because two things happen next to each other frequently doesn’t mean that one causes the other. But sometimes we take it too far, and act like correlation doesn’t mean anything at all. This is a reminder that when two things correlate, that means they are more likely to appear next to each other. If two events are highly correlated, and we see one event, it is more likely that we will see the other event as well. This isn’t because the first event causes the second.
2: Causation != Causation
That’s right, causation does not imply causation. Here’s an example:
You have a bike, and you know the chain rusts whenever it gets water on it. Water on the chain causes it to rust. You also know that your neighbor’s lawn sprinkler gets your bike wet: the neighbor’s sprinkler causes water to get on the chain. Therefore your neighbor’s sprinkler is causing your bike to rust. You go pound on his door and yell until he agrees to stop watering his yard. The next week, your chain has rusted again anyway, because it rained and water got on it regardless.
Technically, your neighbor’s sprinkler was “causing” your bike chain to rust. Pragmatically, when you made him stop, nothing else changed. Real world systems are rarely simple chains of events. They’re more like grass roots, a tangled mesh of interacting consequences. When you’re talking about causes, it’s best to stick as close to the root of the problem as possible.
Even in the realm of theory, causality is a tricky thing. Most statisticians try to study Effects of Causes (EoC), the results of doing one thing or another, so as to predict what might happen when we do those things in the future. But in fields such as law and policy, many problems are about the Causes of Effects (CoE), as in “what made this happen?” Statistics has little value to contribute to CoE problems without introducing some counterfactual reasoning — at most it can offer bounds on an answer, but those bounds themselves are subject to probabilistic uncertainty. If, for example, there is a court case concerning whether or not a certain manufactured drug caused heart attacks in some people who used it, it may not be possible to answer that question with the certainty required by law for a guilty verdict. (Even if the drug increased the risk, can you really say that they suffered heart attacks because of that drug and not for some other incidental reason? Of course, most drug companies try to settle out of court. I have not heard anything about the Servier trial I’m basing this example on since September 2019 — if you have updates let me know.)
3: Proportion != Probability
A demographic is not interchangeable with a member of that demographic, and neither are its features. This mistake is especially egregious around percentages, because we use them to mean two different things: proportion, and probability.
I have a bag with 80 red marbles and 20 white ones. I pick one out and hand it to you. You look at me and say: “There is an 80% chance this marble is red.” Only no there isn’t, it’s definitely red, look at it. You look at it and say: “So it is, so it is.”
The problem here is that the collection of marbles has a “red color” attribute which is a proportion — it can be 80% red, or 42.67% red, or 100% red. An individual marble has a “red color” attribute which is a binary: it can be yes or no. If I took out a marble and hid it and asked you to guess if it was red, you could say “I think there’s an 80% chance that it is red,” but that’s not actually a guess if it’s red or not, only an estimate of how many times you could get it right. In the end you will have to pick one to assume (gun to your head), and you will end up being either right or wrong — a binary.
You can see how this contrasts with the point about correlation above — you’re more likely to be right if you guess red, because you know that red marbles are more common than white ones. But that doesn’t mean your guess will be correct. On the contrary — you are almost guaranteed to get it wrong, after a few rounds. And the marbles themselves are never 80% anything. I get another bag, this one with 40 red marbles and 60 white ones. You point at the first bag and say: “That bag is twice as red as the first one.” You’re not just wrong — what you’re saying doesn’t even make sense. “You’re twice as likely to pull a red marble out of the first bag?” Closer to the truth, but very meaningful. “There are twice as many red marbles in the first bag.” Now you’ve got it.
Final note: so far I’ve just been talking about a binary attribute, but the problem becomes worse when you start talking numbers, not better. “The average male height is 5'9 so I’m guessing that’s how tall you are, Оставь.” No, I’m 6'0. “Ah, well, I was close.” You were still wrong.
4: Space != Subspace
Statistics about a group do not apply to a subsection of that group unless it is randomly chosen, and even then there’s not a guarantee. A Dutch doorknob manufacturer who makes 9.2% of its doorknobs left-handed for the 9.2% of people in the world who are lefties will find an angry mob of southpaws at its doors, because the left-handed population in the Netherlands is over 13 percent. The colored marble problem above becomes a lot different if you saw me dump the 80 red marbles in the bag first, then put the 20 white ones on top.
People will make this mistake even in situations where the subspace is explicitly chosen to match the attribute they’re talking about! “It’s incredible how Europeans are bilingual!” says the New Yorker who has only ever met immigrants. On the other side of the Atlantic, more like only half of people are able to carry on a conversation in more than one language, and less than a quarter are truly bilingual — at best, this claim is a coin flip. Unless you’re in Luxembourg, where about 70% of people are bilingual. Or the UK, with only an 8% bilingual population. Except in London, which has a 22%- etc, etc.
A basic rule of stats: match the population you gather data about to the population you have a question about. Are you trying to figure out behavior across the United States? Go to the Census Bureau. Looking to talk about college graduates, specifically? Throw away that census data and start over. Talking about a specific person? You know what needs done.
This torpedoes many polls, surveys, etc. Building a sub-population representative of the general population is all but impossible and most researchers take the opportunity to fudge it in one way or another. It’s a good idea to dig into the sample selection process if you can — figure out who the researchers actually studied, what results they got from them, and how they extrapolated those results to “people in general”. Most reports will have something to raise suspicion.
In Conclusion
Maybe these claims are unintuitive, or unhelpful, or situationally wrong. That’s the sad truth of statistics: it is more art than science, and at the core it is all language, and all this language can be twisted in one direction or another. But hopefully now at least you can understand the joke I made in the title — the average person doesn’t understand statistics. “There is no such thing as ‘the average person’ understanding or not,” you say. And you’re right. There’s the punchline. But at the same time, you understand what I mean, so maybe the joke doesn’t work after all? But you understood the punchline too, so it does work, and it’s funny, and it reminds you that more than half of people in the world don’t know how to use statistics or probability, and even half of researchers probably, and most of the claims you read built off stats are suspect.
