« You Know, I Think We're Finally Getting to 'Em | Main | Kerry Leads, Gains in Key Swing States After 3rd Debate »

Bad Gallup! No Biscuit!

Readers of the USA Today were treated to the following headline splashed in huge type across the front page today: "Poll: Bush leads by 8 points". The headline referred to a 52-44 lead that the new Gallup poll found among likely voters. The accompanying story pointed out this was quite a turnaround compared to the Gallup poll of one week earlier, which had Kerry ahead by a point among likely voters (LVs).

A 9 point swing. That's pretty impressive. Of course, if you read the story closely, it does mention that Bush was ahead by just 3 points (49-46) in their registered voter (RV) sample. And, as it turns out--though this isn't mentioned in the story--that's a shift of only 3 points from a week ago, when Kerry and Bush were dead-even in the RV sample.

Much less impressive. Well, which is more believable? I think this a good time to review the basic case against Gallup's LV data:

Sampling likely voters is a technique Gallup developed to measure voter sentiment on the eve of an election and predict the outcome, not to track voter sentiment weeks and months before the actual election. There is simply no evidence, and no good reason to believe, that it works well for the latter purpose. In fact, the evidence and compelling arguments are on the other side: that the registered voters are the more reliable guage of voter sentiment during the course of the campaign.

Here’s why. Gallup decides who likely voters are based on 7 questions about their interest in voting, attention to the campaign and knowledge about how to vote (e.g., where their polling place is located). The interested/attentive/knowledgeable voters are designated “likely” and the rest are thrown out of the sample. But as a campaign progresses, the level of interest among voters tends to change, particularly among those with partisan inclinations whose interest level will rise when their party seems to be mobilized and doing well and fall when it is not. Because of this, partisans of the mobilized party (lately, Republicans) tend to be screened into the likely voter sample and partisans of the demobilized party (lately, Democrats) tend to get screened out. But tomorrow, of course, the Democrats could surge, in which case their partisans may be the ones over-represented in likely voter samples.

That suggests the uncomfortable possibility that observed changes in the sentiments of “likely voters” represent not actual changes in voter sentiment, but rather changes in the composition of likely voter samples as political enthusiasm waxes and wanes among the different parties’ supporters. And that is exactly what political scientists Robert Erikson, Costas Panagopoulos, and Christopher Wlezien find in their analysis of Gallup's 2000 RV/LV data in their forthcoming paper, “Likely (and Unlikely) Voters and the Assessment of Campaign Dynamics” in Public Opinion Quarterly: “shifts in voter classification as likely or unlikely account for more observed change in the preferences of likely voters than do actual changes in voters’ candidate preferences.”

That means that, instead of giving you a better picture of voter sentiment and how it is changing than conventional registered voter data, likely voter data give you a worse one since true changes in voter sentiment are swamped by changes in who is classified as a likely voter.

So: focus on the RV data, ignore the LV data. Indeed, in my view, it's time for Gallup to drop reporting these data altogether because they are highly likely to give an inaccurate picture of the state of the race and, by doing so--especially given the high profile of Gallup's polls and how they tend to drive media coverage--unfairly pump up one side of the race and demoralize the other. That doesn't seem acceptable to me. At a minimum, Gallup and other polling organzations that use similar approaches to defining likely voters should lead with their RV data and provide the LV data as a supplement, not the other way around.

That would make a difference in how the race is covered. Based on the Gallup LV result plus a couple of other recent LV results with fairly solid Bush leads (50-44 among Newsweek LVs and 50-46 among Washington Post (WP) LVs), other media outlets had stories today and yesterday on how Bush was surging and even breaking the critical 50 percent barrier in voter support (see, for example, this story by Ronald Brownstein in the Los Angeles Times which prominently features the three 50+ LV results).

But here are the same three polls, with RV results (all 3-way to match the data released by Gallup):

Gallup: 49-46
Newsweek: 48-46
WP: 48-47

So these three polls would then all have Bush under 50 (average: 48 percent) with only a 1-3 point lead. A very different picture and much dicier for Bush.

And probably much more accurate. Consider these other recent data, all of which paint a very different political picture than that implied by the Gallup LV data and the gaudy USA Today headline:

1. The Gallup RV data, while preferable to their LV data, may themselves be flawed. Their RV sample has a 2 point edge for the Republicans in party ID. If that sample is re-weighted to conform to the 2000 exit poll party ID distribution, Kerry leads Bush by 2 points, 49-47.

2. The latest Democracy Corps survey, conducted October 14-16, the exact same dates as the Gallup survey, has Kerry up by 3, 50-47. (And see this post for evidence on the superior accuracy of the DCorps survey, relative to Gallup, in the 2000 election.)

3. The Zogby and Rasmussen tracking polls have both closed by 4 points in the last two days, eliminating Bush's leads in these polls. Indeed, as Jerome Armstrong points out over at MyDD, if you include leaners in their trial heats (as most reported national results do), Kerry is now slightly ahead in both polls: 47.2-46.6 in Zogby and 49.5-47.7 in Rasmussen.

Doesn't sound like an 8 point Bush lead to me. Or much of a lead at all, for that matter.

Comments

Could this be a function of voter enthusiasm. Bush seems to be energizing his base a lot better than Kerry... which is going to turn RVs into LVs in much higher numbers. While it's true that RV's are about the same in polling, a shift in LV is what's going to make the difference on election day.

Sounds like a crumbling incumbency.....I think the go negative GWB approach is starting to penetrate the regular guy image and show him as the mean SOB that he really is....

Ruy, with all the buzz about early voting, is there any exit polling being done on these voters like there is on election day? Secondly, what is the profile of these early voters? Are they more Democrat or Republican or younger versus older?

gallop 2000 10/19:

Bush 50%
Gore 40%

Thank you Ruy, for exposing this Gallup propaganda -- and that's the right word for it. It ain't science; it ain't math. How they can lower themselves like this is beyond me.

Looking better day by day for Kerry.
His victory will be the most gratifying political event in my life and my first vote was for JKF when I was a senior in college.

CBS Poll: Bush approval rating in low 40s. The guy is toast!!

Oct 18

CNN, MSNBC trying hard to say his buns with the Gallup polls.

OT question. I'm a new reader and I have a basic question. We know that it is a incontravertable fact that Gore and Bush got about equal numbers of votes in 2000. If a pollster asks the question "if you voted in 2000, who did you vote for" the polled group should contain equal numbers of Gore & Bush 2000 voters. If it doesn't the polled group is either unrepresentative or full of liars. Why don't pollsters automatically ask this questtion?

kalkat - While it sounds reasonable, ask yourself this: In 1972, Richard Nixon won by a large margin. In 1974, how many people would say they voted for him? Numbers may lie, but people lie more.

Kalkat:

I remember a recent poll in the last couple of months where they did ask that exact question, and the breakdown indeed was way out of wack with the results from 2000. Someone can help me here, but was the number not something like 50 plus for Bush and low to mid 40s for Gore. I heard a lot of bantering back and forth whether this is problematic, but my view on it is that the real issue with all polls is full disclosure and contexualization. I remember the pollster not providing crucial contextual information except for a big heading saying Bush was leading. In fact, I think this may have been Gallup? The reason why I think these poll are the way they are is that they are proprietary. They have a vested interest in their models. The truth is we have no idea where the election truly is except to say that is probably tied w/ Kerry having a slight momentum, but even that is guess work. That's the big secret of course about all of this- we would probably have a better chance of guessing the eventual outcome if we wrote each candidates name on a piece of paper with a guestimation of their likely vote, and by tossing them in the air, see which one fo them lands on top. GOTV.

As a political junkie and someone who actually took a statistics course in college, I am really bothered that LV results get reported as news in preference to RV results when only RV results can really be news since they are a sample of a population that really exists independent of the pollster while LV's are an artificial creation of the pollster and the results can only mean at most that people who meet the criteria set by the pollster would, if the election were held that day, vote in a certain way, subject to the margin of error. What is especially bizarre and annoying is the use of LV results by "pundits" to analyze the election with such questions as why is Kerry slipping, for which answers are then given such as his comment re Mary Cheney, rather than anybody asking if maybe the result of the LV sample is itself unreflective of voting reality.

Also, I think your comments about enthusiasm are especially relevant re battleground states. In polling which separately identifies a result for those states as a unit, Kerry always seems to do well even in polls where he does not do as well nationally. This could be because relative enthusiasm is greater for Kerry supporters in those states where there is a real contest going on.

I am looking forward to an election night on which I will not only be able to celebrate a Kerry victory but listen to all of these "pundits" explain what happened in the last two weeks that allowed Kerry to make up his 8-point Gallup deficit. Perhaps, I could help them. My recent reanlysis of poll results, through all the strange Kerry ups and down, using an expected electorate similar to 2000 exit poll party ID and allocating 2/3 of undecideds to Kerry have fairly consistently produced a 50-48 Kerry victory, give or take a point or so.

Because people give untrue responses. Many of them don't even remember how they voted, some say they voted yesterday for the guy they like today. If a President becomes unpopular enough, the majority will remember voting against him even if he won by a landslide.

I have been so confused by all the polls showing Bush ahead. My hopes for a Kerry win have risen and fallen by these polls. Frustrating! It is so hard to understand why Bush would have the lead with the mess in Iraq and an economy at home on the decline with rising deficits. This is the first time I have reviewed your site and data. Indeed, you expose the short comings of Gallup and others. Thanks for the detailed research and commentary. Hope is still alive

As a subscriber to USA Today I was amazed at the huge headline for Bush. The funny thing is I don't even think the editors and reporters believe the numbers, because most of the article was claiming how tight the race is. Eight point lead is tight??

I have called the political editor twice about polling coverage and how USA Today tends to publish polls favorable to Bush(with big headlines) and ignoring others such as Zogby, the Economist ,etc.

It will be interesting to see if they cover the latest Zogby and Ramussen polls which show a tie. or for that matter the TIME and Newsweek polls which show a close race.

One thing that is hard for me to understand (Ruy help please)
The ABC/W.P poll has bush's approval at 54, then CBS/N.Y. Times has it at 44%. What the hell is going on with such a big spread?

Please comment on how the early voters may affect the pollsters' polling. Is it likely that some early voters would vote differently if they waited to election day? Is the early voting going to substantially increase the number of voters because the polls will not be as crowded on election day? Who does early voting favor?

Hmm. Well I can only address the early voting in my state but would guess it would be a similar situation in others. In TN it seems to suggest a much higher turnout for the overall election, with as many as 25% of RVs expected to have voted early. The people showing up to vote early seem more heavily skewed toward the elderly by far than the Nov. 3rd turnout generally. I think those who vote early have firmly commited to one candidate and the leaners and undecideds will come out on Nov. 3rd. Overall early voting seems likely to increase the overall turnout considerably and that favors Kerry.

Mark Allan: It will be a very cold day in hell before USA Today publishes data from Zogby or the Economist.

Why? Because (1) USA Today has a proprietary interest itself in the Gallup survey (as does CNN). (2) The Zogby Poll is frequently conducted for Reuters. (3) USA Today has no interest in promoting the Economist (or Time or Newsweek either, for that matter).

It seems to me to be cockeyed ethically for a news reporting organization to be making news (in the form of a poll) and then publicizing the news it makes in an effort to increase profits by drawing readers and viewers. I don't like it, but then maybe I'm just behind the times and fail to appreciate the 21st century role of the Fourth Estate.

I'm glad to see this commentary--I emailed USA Today today and complained about their giant headline.

It was misleading. I find that is often the case with their website too.


I can't decide if they are just oh-so-proud of their poll, so they ignore all the other polls, or if their bias is showing big time. Whichever the case, I found the headline highly misleading and told them so. Email them!

Ruy or someone else: I want to go back to the issue raised by kalkat, a bit earlier, on voter's preference in the 2000 election.

In the most recent Democracy Corps and CBS/NYT poll there is a wide discrepancy between the 2000 results and the samples:

DC says its latest sample was 43 Gore/51 Bush and CBS/NYT says its sample consisted of 29 Gore/35 Bush/3 Other which is almost equivalent.

Is this undersampling of Gore voters (and the oversampling of previous Bush voters) significant in the present context ?

The worst thing about the gallup poll is the way it is being used on CNN. Yesterday afternoon, CNN had their bunson honeydew polling guy, I forget his name, talking about how that inspite of Kerrys performance in the debate mothers and parents, who are more likely to vote democrat, are most concerned about security and are going for Bush.


How anyone can file a report about actuall blocks of voters with such conviction in the face of such wildly swining polls is beyond me.


It is quite irresponsible. It crude but the bast way I can put it is - DON"T P!SS ON MY LEG AND TELL ME ITS RAINING!

Has anyone out there found a person who voted for Gore last time, who is voting for Bush this time? I would think that would be the true test of the CNN Bush lead.

I am a long time dedicated election observer and I have been following 2004 polls and the poll analysis closely. Traditionally LV has been a reliable #, more reliable in fact than RV, because not everyone votes. Gallup has also been right more than it has been wrong. Is it different this year because of expected turnout, new registrants, ABB?

If the polls are as close as Zogby, Rassmussen, Time, and Dem Corps show then Gallup should fall into line soon.

I am not a big conspiracy person but people do like to vote for the winner and unfortuantely perception is as important as facts (an 8 point Bush lead is not good for Kerry). Bloggers influence mainstream media everyday-keep the calls and emails up pointing out the inconsistancy of Gallup-use the other polls as proof.

I believe the tracking polls over Gallup LV but it is hard to sort out the passion from the facts. I would love to avoid the folly of only believing the polls that validate my hopes and dreams. Fat chance.

Zogby's prediction of a Kerry victory is meaningful because even if he is partisan his professional reputation is on the line. My prediction is that the US will eliminate or mitigate (like CO) the electoral college in the coming years.

Keep the poll analysis coming!

About the "who did you vote for last time" question.

I recall seeing research that people tend to say they voted for the winner in past elections, when they didn't. I also recall seeing research that people tend to say they voted in past elections, when they didn't. So you can't take the answer to the "who did you vote for last time" question and adjust your sample to match the actual outcome of the last election.

On the other hand, it seems pretty obvious that the answer to this question is going to be more stable over time than a "Republican-Democrat-Independent" question. So a superior alternative to adjusting your R-D-I breakdown to a rolling average of previous polls, would be to adjust the last election vote breakdown to a rolling average of previous polls. Both methods will provide the same large reduction in sampling error. I can see no reason of any kind for using the R-D-I question for this purpose rather than the last election question.

I would probably not be considered a likely by gallup.

Yet I've voted already: My absentee ballot went into the mail this AM. So, how do I fit into their system?

> Thank you Ruy, for exposing this Gallup
> propaganda -- and that's the right word for it. It ain't
> science; it ain't math. How they can lower themselves
> like this is beyond me.

> Looking better day by day for Kerry.


Well, at least it is not looking WORSE for Kerry; I am a bit surprised by the misleading "the sky is falling!" headlines during the last week. Compare any given poll this week vs. last week's numbers, and it seems "Shrub" is doing neither better nor worse than before the final debate.

BTW, the state polls don't look too bad. For example, Slate reported that Kerry was leading "Shrub" 274-264.

Sure -- it is disappointing that Kerry is still essentially tied in the polls. But the GOP incumbent has more reason to worry about that, with a 44% approval rating and "the country is on the wrong track" figure of 59%... "Shrub" is starting to run out of time since *he* is the one who needs to persuade undecided voters to go Republican. If it's still a tie on Nov 2, he will probably lose.


MARCU$

Maybe they're only including the votes that will be *counted*.

Ruy--great work as usual. Why can't the D's or somebody (ACT, etc.) blitz the tv talking heads with the straight skinny about these polls?

Tweety Bird on Hardball and the others must surely understand the technique abuse of Gallup, etc., but have been blathering the "Bush momentum" line since the beginning of Sept, in the face of you and others disputing the hell out of it.

Unfortunately, only a couple of tv reporters/anchors ever seem to understand the truth...Countdown has been pretty good about it, but most of the others suck.

The continuing mis-statements then depresses the democrats' turnout, etc.

TW

I just found this blog today, and I am so happy to see so many people discussing these polls! I am a PhD statistician, and I agree that one should completely ignore any poll based on "likely voters." It is well-known that stratifying a sample based on an unknown, predicted future response is invalid and will lead to a biased estimate, and that is exactly what the polls of "likely voters" is doing.

Dear Ruy Texeira:

In the 10/18/04 New Yorker article "The Pollster," on John Zogby, the author Larissa Macfarquhar makes a helpful critique of LV polls. She echoes your critique that LV polls give results according to the method used to determine likely voters, not necessarily according to changes in the electorate. However, she adds a point I haven't seen you make--although I may have missed a post. She argues that the sample in LV polls tends to be skewed Republican. LV polls identify likely voters by asking if a voter has voted regularly in past elections. But regular voting correlates most closely with high socioeconomic status, which correlates with voting Republican. What's more, because these polls estimate turnout of 50-55%, they throw out the 45-50% of those polled deemed least likely to vote, which might further skew the sample. LV polls would seems likely to have voter ID numbers that are highly Republican, simply because of the way they collect their sample. The article neatly amplifies your sharp-eyed critique of the Gallup LV polls that have receive so much attention--not least from Matthew Dowd.

Another LV beef: margins of error. It is the hight of irresponsability to put a margin of error on an LV poll that reflects only sampling error. It's simply wrong and they are lying LYING if they don't include and propagate error terms based on each of the criteria they introduce into the model to figure out who's likely. Do it right and the networks might start to shut up when they see the numbers with the +- 15 next to it.

Is there any evidence as to the accuracy of the method for determining likely voters? How many false positives and false negatives do these methods produce? I'm guessing no one knows, which makes the whole thing a bit suspect.

Are the questions used to determine if someone is an LV asked before or after the main question?

If after then the pollster can tell if the LV's really are significantly different than the others, and it would be interesting to have this information.