"This is what happens when 16 people are made to represent an entire population."
That's a line from a FiveThirtyEight article exploring a scientific study that got lots of media attention, despite some statistical shortcomings.
The original study explored the link between the use of e-cigarettes (which you may know as "vaping") and traditional cigarette smoking.
So what's the problem? In FiveThirtyEight's article, titled "Ignore The Headlines: We Don’t Know If E-Cigs Lead Kids To Real Cigs," author Christie Aschwanden explores a number of issues with the study, including:
- Drawing results from a small number of outcomes
- Extremely wide confidence intervals (the range in which you expect to see outcomes)
- Determining statistical significance based on these factors
- The importance of one study vs. existing research
The bottom line? We're not saying you should completely ignore the headlines - but definitely take them with a grain of salt.