Presidential tracking polls are not created equal

Polling expert Nate Silver and his site FiveThirtyEight.com (photo: Neeta Linds and blakespot via Flickr.com)

With the daily announcement of multiple national tracking polls, voters may be getting more confused. On any given day there are now as many as eight national polls monitoring opinion about the McCain-Palin and Obama-Biden tickets, not to mention the weekly or biweekly national polls that use other formats. Plus the various state polls that come out.

And they frequently give disparate results. On Wednesday, the Associated Press released a poll showing Obama with a one-point lead nationally. On the very same day, ABC News and the Washington Post released a poll showing Obama with an 11-point lead nationally.

Which should be taken seriously and which should be dismissed?

East Lansing, Mich., native Nate Silver, the founder of polling analysis site fivethirtyeight.com, is the breakout star of the 2008 election because he’s introduced the fusion of rigorous analysis of polls with electoral math into the country’s consciousness. And in a new article he discusses some of the major polling companies that daily track the presidential election, their strengths and weaknesses, and how their results might be skewed by their methodology.

First, a note on what distinguishes a tracking poll from another kind of poll.

Most tracking polls are done in a “rolling” fashion, that is, spread out the sampling over a number of days, the total of which makes up their full sample. For example, the ABC News/Washington Post poll has a total sample of around 1,100 voters, but they actually poll about 350 of them each day. Today’s result comes from adding the previous three days together, then tomorrow’s result will come from dropping the result from three days ago and adding today’s sampling to the data. Other polling companies use three-, four- or even five-day rolling samples. This helps them get an accurate sense of how public opinion is trending over time.

Second, an explanation of how polling companies use “weighting” to try and get a sample that accurately reflects the overall population. Some forms of weighting are obvious: If the general population contains 55 percent women and 45 percent men, the pollsters want to make sure their sample has roughly those same proportions. Others are more difficult to get a handle on, like estimating how many voters are Democrats or Republicans, or how likely voters are in some demographic pie slice to actually go to the polls.

This is not always possible, of course, so pollsters use mathematical weighting to try and simulate the results they get as if they had a genuinely representative sample. Sometimes those weighting factors make the polls more accurate, but sometimes they don’t. Change the demographic weighting slightly, and it can have a relatively large effect on the accuracy of the poll.

Zogby Interactive

Zogby has a reputation for being one of the most accurate polling companies, based mostly on being one of the few pollsters to predict that Al Gore would win the popular vote in 2000.

But in the latest version of Silver’s pollster ratings — an analysis of which polling companies have the most accurate results in more than 170 electoral contests since 2000 — Zogby ranks near the middle among about 40 polling companies. Silver explains why:

Zogby’s telephone polls are not quite as bad as reputed, ranking just slightly below average — his Internet-based Zogby Interactive polls are another matter altogether. However, they seem to be getting worse rather than better. …

There is one very, very significant concern with Zogby, which is that he has a longstanding rule to set his party weightings based on the exit polls from the most recent election. In this case, that means 2004, when a roughly equal number of Democrats and Republicans turned out. However, according to essentially every available poll, Democrats now have somewhere between a 5-point and a 10-point advantage in party ID. …

Between 2000 and 2004, there was a shift in party ID toward the Republicans; as a result, Zogby’s numbers were 2-3 points high for Kerry across battleground states.

To the extent that Obama is leading in a Zogby poll, that means essentially that he’d have won the election given 2004 turnout dynamics. 2008 turnout dynamics are liable to be significantly more favorable to him.

Research 2000

Research 2000 is one of the less-prominent polling companies, but it is now doing a daily national tracking poll for the Daily Kos Web site. Today, Research 2000 via Daily Kos reports that Obama has a national 10-point lead, 51-41, over McCain. Silver says that Research 2000 has “an above-average track record” based on his pollster ratings, but he also notes that this is the first time it has ever done a national tracking poll.

Unlike Zogby, Research 2000 does weight its results based on current party ID trends; because more people in the current election cycle are identifying themselves as Democrats and more Democrats showed up to vote in the primaries than Republicans, the company gives more weight to Democratic voters in its sample. But there are other weighting factors in its polls that give Silver some concerns:

Racial demographics are aggressive — probably too aggressive — showing blacks making up 14 percent of the electorate and Hispanics another 13 percent. Turnout will be up among minorities this year, but probably not by quite that much. The +9 party ID split is arguably also aggressive, although within the broad range of what other polls have found this year.

Rasmussen Reports

He next discusses Rasmussen Reports, a pollster that ranks near the top of Silver’s historical ratings. That is partly due to the very large sample size it uses (3,000 voters using a three-day rolling sample) and partly due to the fact that it changes their weighting assumptions in response to current dynamics:

To the extent that any pollster should weight by party ID, I think Rasmussen is going about it the right way, setting targets based on a six-week rolling average of all interviews conducted.

In other words, if Rasmussen’s random calls over a six-week period show that 55 percent are identifying themselves as Democrats and 45 percent as Republicans, it will weight its sample accordingly. If that shifts over time, the six-week rolling sample allows it to respond that much more quickly than Zogby, which is using the same figures it used in 2004 to determine how the results are weighted.

Rasmussen’s latest presidential tracking poll, published today, says Obama is ahead by seven points, 52-45.

Diageo/Hotline

The Diageo/Hotline poll, associated with the National Journal, has the smallest sample size (about 800 voters with a three-day rolling sample) of all the national tracking polls and, according to Silver, has little track record to consider. Diageo apparently did no polling at all during the 2004 election and its results have bounced around quite a bit due to a lack of party ID weighting. Its latest results have Obama-Biden up by five.

Battleground

The Battleground poll, put out by the Tarrance Group in association with George Washington University, also has a relatively small sample size (800 to 1,000 voters with a four- or five-day rolling sample) and Silver says it has made some recent changes to make up for serious methodological problems that was making its results stand apart from the other polls:

They don’t conduct interviews on Fridays or Saturdays, which is a defensible decision, but means that the poll may be slower to react to new information early in the work week, especially when coupled with the relatively large (4-5 day) sample window.

As alluded to above, Battleground previously had significant problems with their sample, substantially undercounting young voters, which was likely causing the strong Republican lean. They have since revisited their turnout assumptions and altered their model, and have had more “normal” looking results since.

Gallup

Gallup, probably the most prominent and best-known polling company in the country, uses a large sample size (2,750 registered voters and 2,200 likely voters with a three-day rolling sample), but Silver has some concerns about its methodology, noting that, “Gallup’s results lag a bit behind its reputation, as it ranks in the lower half of our accuracy ratings.” He gives this reason for the problem:

The poll has been a bit “jumpy,” reacting more strongly than others (i.e., perhaps overreacting) to events in the news cycle. Still, the main point of concern is with the three different versions that they publish each day (registered voters, and then two separate likely voter models) — it’s hard to know which version they are really throwing their weight behind. Also, the differences between the registered voter numbers and the more conservative version of their likely voter numbers are sometimes implausibly large.

Today’s results for Gallup’s three polls show Obama-Biden with leads of seven (registered voters), six (likely voters, expanded) and four (likely voters, traditional).

IBD/TIPP

The Investor’s Business Daily/TIPP poll has a relatively small sample size (1,050 voters with a five-day rolling sample) and some inconsistent results over the years:

IBD/TIPP touts itself as the most accurate pollster based on its strong result in 2004, when they nailed the Bush-Kerry numbers within a couple of tenths of a point. One good result does not a pollster make, however, and in 2000, their performance was only average, missing the Bush-Gore margin by 2.5 points. Results were slightly erratic and counterintuitive earlier this year, when for instance they showed an 11-point Obama lead in mid-May when most other pollsters showed him struggling at that time.

Silver rates this poll as pretty good with a couple of conditions:

A 5-day tracking window is relatively long, and means that IBD/TIPP may be slower to react to new trends. Their D-R party ID gap has been narrower so far than that of most other pollsters (perhaps too narrow), though it does not appear as though they weight based on party ID as Rasmussen or Research 2000 does. Overall, the poll is probably fine given these caveats, but we have relatively little basis on which to evaluate it.

IBD/TIPP’s latest result, from Tuesday, has Obama-Biden up by six.

Washington Post/ABC News

The Washington Post/ABC News poll, mentioned above, just began in the last few days so it’s a bit early to evaluate its accuracy. But Silver notes that it has done fairly well in other polls recently:

Strong relative to that of other major news organizations, which generally do not do terrific polling, although just average compared to the field at large. ABC/Post had a fairly strong go of things when they conducted polling during this year’s primaries.

The ABC/Washington Post daily tracking poll for yesterday showed Obama-Biden with an 11-point lead, 54-43. It’s the largest margin in the poll’s short history, which the editors of the Washington Post chalk up to a bounce for Obama from Sunday’s endorsement from former Bush Secretary of State Colin Powell.

Conclusion

The overall conclusion Silver reaches in evaluating these daily tracking polls? That media polls like AP and ABC/Washington Post (which show the largest distance between one another — with Obama +1 in the former and +11 in the latter) are less accurate in general than those conducted by actual polling companies. And that in the world of polling companies, Rasmussen is good, Zogby is not:

In summation, none of these tracking polls are perfect, although Rasmussen — with its large sample size and high pollster rating — would probably be the one I’d want with me on a desert island. Conversely, the only one of the trackers that I consider obviously dubious is Zogby.

Incidentally, in a separate article, Silver notes that the AP poll mentioned at the top of this article that showed such a small national lead for McCain relative to all the other polls is one of several polls with a large discrepancy between its results for registered voters and its results for likely voters (four points difference):

Frankly, I find polls showing a 4-6 point gap between likely and registered voters to be utterly ridiculous.

Specifically, he discounts such a large discrepancy between registered and likely voters because multiple polls are showing an enormously high turnout in states with early voting, with most of it skewed toward Obama. That, he says, is why polls must be weighted to take into account the higher Democratic turnout this year rather than relying on numbers from past years.

All of these factors are worth considering when reading about the results of these tracking polls and trying to decide which one is likely to be the most accurate reflection of actual public opinion.