Saturday, October 11, 2008

Bad Polling

Jim Geraghty over at NR's Campaign Spot blog calls out Newsweek for bad polling.

(poll found here - PDF Warning)

The internals simply don't make sense. Obama is supposed to be winning 18 to 34-year-olds by 10 percent, but at the same time winning 45 to 64-year-olds by 16 percent.


I get it that there are debates about how to weight some of these polls this year, particularly on Party ID, but when you end up with a whole bunch of anomalies like Geraghty points out you're looking at a bad poll. It's not just that it's wrong, it's that it's totally meaningless.

I don't even have to get past the methodology page of the poll to show that it is meaningless.

First up, the Universe:
N = 1.035 registered voters, screened from 1,212 adults, 18+
"Registered voters" is a yellow flag. Not quite a red flag like "adults", but not as good as "Likely voters".

This is the killer though:
Notes: Data are weighted so that sample demographics match Census Current Population Survey parameters for gender, age, education, race, region, and population density.

You do not under any circumstances weight a political poll to Census data. That is the dumbest thing I have ever seen. You can weight to Party ID based either on past turnout or to registered Party ID (or some educated guess in between), or gender, or maybe race, but when you start weighting the crap out of polls to the universe of the total population rather than the voting population you get an extremely flawed result.

Shame on you, Princeton Survey Research Associates International.

I don't take media polls too seriously unless they are done in conjunction with a reputable polling firm, preferably two - one from each party - like the Wall Street Journal does.


Samay said...

Polls are weird. I think it's more likely they ended up surveying more liberal older voters and more conservative younger voters. That's why there are so many weird polls showing McCain either up by 8 in Ohio or down by 5. The only real poll is Nov. 4, everything else is speculation.

Sockless Joe said...

Media polls are essentially a self-writing story. Whatever the result, voila - there's a story to go with it and space to be filled on the page. Unfortunately, it's financially irrelevant to the paper/magazine/TV-Network whether the poll is accurate.

The good pollsters (of which there are probably less than a dozen nationally, a few more with regional cred), are capable of making very accurate models of the electorate.

For all the complaints in 2004 about the election not coming in like people expected... that was bunk. The traditional phone pollsters had it pretty well nailed as long as you had a reasonable idea of which polls were to be ignored.

My understanding is that Zogby polls throughout the day, not just in the evening. That means Zogby is measuring a different set of people than every body else, so you have to discount his results.

I'll be putting up my final electoral map on election eve, and it will be largely based on public polling. I'm guessing I'll be really close.

Samay said...

Close? I doubt it. It's pretty much over. Even conservative sites like this come to that conclusion.

Sockless Joe said...

I meant I'll be close in my state by state prediction, not that the election will necessarily be close.