PSA: Everything you know about the margin of error is wrong

February 27th, 2012 at 3:26 pm by under Nesi's Notes, On the Main Site

When our exclusive WPRI 12 poll drops at 6 p.m. tonight, you’ll see that the margin of error is plus or minus 4.38 percentage points for the entire poll and plus or minus 6.2 percentage points for the 1st District only.

Inevitably, some well-meaning folks will take our top-line results and then argue that anything in a range of, respectively, 8.76 points or 12.4 points could be true. If Cicilline is down 5 points, these folks may say the race is a “statistical tie.”

Those folks will be wrong, and here’s why, as explained by Kevin Drum back in 2008 (emphasis mine):

The idea of a “statistical tie” is based on the theory that (a) statistical results are credible only if they are at least 95% certain to be accurate, and (b) any lead less than the MOE is less than 95% certain.

There are two problems with this: first, 95% is not some kind of magic cutoff point, and second, the idea that the MOE represents 95% certainty is wrong anyway. A poll’s MOE does represent a 95% confidence interval for each individual’s percentage, but it doesn’t represent a 95% confidence for the difference between the two, and that’s what we’re really interested in.

In fact, what we’re really interested in is the probability that the difference is greater than zero — in other words, that one candidate is genuinely ahead of the other. But this probability isn’t a cutoff, it’s a continuum: the bigger the lead, the more likely that someone is ahead and that the result isn’t just a polling fluke. So instead of lazily reporting any result within the MOE as a “tie,” which is statistically wrong anyway, it would be more informative to just go ahead and tell us how probable it is that a candidate is really ahead.

Drum then posted this chart, which tells you how confident you can be about different sizes of poll leads based on the size of the lead and the margin of error. So for our poll, you’d want to look at the 4% MoE line for statewide questions, and assume somewhat lower probability than the bottom 5% MoE line for the 1st District questions:

As an example, take our final 1st District poll of 2010. That poll had a margin of error of about 6.2 points, which in this case is literally off the chart. But we can take an educated guess about what a 6% line would say.

In that poll, Cicilline led Loughlin 48% to 42%, giving Cicilline a six-point lead. So we’d look at the 6% percentage-lead box, then go down to the (imaginary) 6% margin-of-error line. Let’s say that box said 83%. Thus, we can say there was an 83% possibility Cicilline was ahead of Loughlin. And, lo and behold, he won by six points a week later.

Update: A smart reader who’s better at math than me writes in to offer another view:

All that I know about margin of error is that a 95% confidence interval means that 95% of the time the actual number lies between the two sides of the value given.

So saying that there is a statistical tie if the CIs of two poll numbers cross over one another may not be gospel; it does mean that the two numbers could actually be the same if not reversed, since there is no for-sure value within the bounds, 95% of the time. Of course, polling being an inexact science, the numbers given may be way off that other 5%, but in all likelihood, that isn’t the case.

I don’t know if you view this as contradicting what you had to say, just that it’s a different way of looking at it.

(chart: Kevin Drum/Washington Monthly)

Tags: , , ,

One Response to “PSA: Everything you know about the margin of error is wrong”

  1. Charlie Baker says:

    What are you going to confuse your readers with next, the fact that 95% is 2 standard deviation points from the mean?

    Oh the fun of statistics.

    Even better, Cicciline won by only 6 pts last time and that was before the roof fell in on Providence. Can’t wait to see what it turns out to be this November. Back to ambulance chasing and mobster defending for him.