« Kerry Up 1 in NH See-Saw Race | Main | WAPO-ABC News Poll: Bush Ahead by 7 »

So, How Well Does That Gallup Likely Voter Model Really Work?

As I've repeatedly argued, Gallup's likely voter (LV) data would be inappropriate to use until right before the election even if they worked perfectly at that particular time. But it's interesting to note that their LV data lately have not been working terribly well, compared to RVs, even on election eve, which is supposed to be when theses data really shine. That kind of undercuts their pseudo-rationale for their cavalier use of these data.

After all, David Moore of Gallup has admitted: “We simply do not know, nor can we know, which model is better during the campaign itself.” But he defends the use of LV data months and months before the election because “if it is the most accurate model just before the election, it is probably the most accurate during the campaign as well.”

Oh really. Well, let's see just how accurate these data have been recently. Here are the last 4 presidential elections and the track records of the Gallup LVs and RVs in terms of predicting the final margin of the election:

2000: RVs, +1D; LVs, +2R; actual result, +.5D
1996: RVs, +16D; LVs, +11D; actual result, +8.3D
1992: RVs, +8D; LVs, +12D; actual result, +5.5D
1988: RVs, +8R; LVs, +12R; actual result, +7.7R

So:

1. In 2000, the RVs pretty much get it on the nose and correctly call the popular vote winner, which the LV data miss.

2. In 1996, the LVs are indeed substantially better--but in 1992, the RVs are substantially closer to the final outcome. And in 1988 the RVs are pretty much are dead-on while the LVs are more than 4 points off.

That's 3 out of the last 4 elections for the don't-get-no-respect RVs.

Not too impressive. And this track record justifies subjecting us to LV data from the very beginning of the campaign and acting like the RV data are somehow inferior? Puh-leeze.

As a public service, I reproduce here the fabulous questions with which the Gallup folks produce their fabulous LV samples:

1. SALIENCE: How much thought have you given to the upcoming election for president?— quite a lot, or only a little? (“Quite a lot” or “Some” as a volunteered response score one point)

2. KNOWLEDGE: Do you happen to know where people who live in your neighborhood go to vote? (“Yes” scores one point)

3. BEHAVIOR: Have you ever voted in your precinct or election district? (“Yes” scores one point)

4. BEHAVIOR: How often would you say you vote—always, nearly always, part of the time, or seldom? (“Always” or “Nearly always” scores one point}

5. INTENTION: Do you, yourself, plan to vote in the presidential election on November (*), or not? (“Yes” scores one point)

6. BEHAVIOR: In the [last] presidential election, did you vote for (*) or (*), or did things come up to keep you from voting?

7. INTENTION: I’d like you to rate your chances of voting in the upcoming election for president on a scale of 1 to 10. If “1” represents someone who definitely will not vote, and “10” represents someone who definitely will vote, where on this scale of 1 to 10 would you place yourself?

If a voter answers each of these questions the “right” way, they get a 7, miss one and you get a 6, and so on. In practice that typically means all of the 7s—given full weight—plus some proportion of those with lower scores (usually the 6s), who are weighted down so that the size of the likely voter sample matches the projected turnout for the year (apparently 55 percent this year). All other voters are discarded from the sample.

This seems like an awful lot of work for something that (a) doesn't work that well at election time; and (b) isn't even appropriate during most of the campaign.

But Frank Newport at Gallup insists this is a "scientific" approach to take to polling. Sounds more like dogma to me.

Comments

Interesting. I fail 3 and 4 because I'm new to the district, so as you characterize the process, it would seem probable that having scored only a 5, I would get screened out. Nevertheless, the chances that I will vote in November approach metaphysical certainty. Boo Gallup!

Back when Roy Rogers and Howdy Doody were on Saturday morning television, we had a game called "Mother, may I?" Sort of the same theme as SIMON SAYS.

One bad response, and you're out.

That looks like how the Gallup poll gets their LVs.

How LIKELY is the continued failure of their polls going to REGISTER with Gallup's credibility?

So I have a simple question. Is there anyone that voted for Gore in 2000 that is going to actually vote for Bush in 2004? Anyone?

Given the unprecidented motivation of the Dems to register new voters and GOTV efforts, this year is not like any other. All Models based on previous behavior is likely to be flawed. Add to that the cell phone undercount, and the question is, how far off will the polls be this year?

But the key will be turnout. We can't just go thru the motions, we have to do better than ever before. Fortunately, we have a lot more room to improve in our GOTV. The GOP doesn't have that much room to squeeze out bigger numbers. I really like our chances. And no President has ever deserved to lose more than this one. Teach him a lesson. He will be the pet goat of the GOP.

how come they don't just ask: "Do you plan to vote in this election?" (a) definitely, (b) probably (c) maybe or don't know (d) probably not and (e) definitely not. The (a)s and (b)s would then be your "likely" voters.
(No doubt, some polls do it that way) But I was wondering whether they felt their method was more accurate.
By the way, for some reason my last post didn't get posted; it was rather 'cynical' in suggesting that machine political agenda is a better predictor of election outcomes than the polls, but that is fair comment.

on that last point, my bad -- it did eventually get posted

So many people have asked the question of what Gore voters will now be Bush voters. I have never seen one attempt at answering the question empirically via a poll. has anyone ever seen any real work on this often asked, highly relevant question? It seems to me that the answer is not as clear as one might think----I think that, unfortunately, many Gore voters may become Bush voters just because of the strong Fear Factor which Bush has been so relentlessly promoting---it's his only message, for God's sake! The country's got soldiers in combat and has been physically attacked by scary terrorists-----that's a big change from 2000 to state the bleeding obvious. At the least I would posit some "older" or "elderly" voters who went for Gore are now going to go for Bush. Perhaps there are other "groups" like them. But who knows the numbers? Is this question ever to be answered scientifically?

Ruy-

Thanks for posting the exact list. What strikes me is that people who didn't vote last time and only reports voting "part of the time" will end up a 5. It's at the least plausible that this screens out a disproportionate number of Kerry voters. I suspect many who would generally (were they to vote) go Democratic just were not inspired by Gore in 2000 and stayed home. With all that's transpired in the last four years, I'd guess they would return. It would be very interesting to see how Gallup's LV would work out if they just used issues tied to the *current* election: questions 1, 5, and 7.

I'm not sure this is the proper thread, but I saw a stat I've been wondering about for a bit:

http://www.zogby.com/Soundbites/ReadClips.dbm?ID=9590

"Caller ID and telephone answering machines have also made it easier to screen pollsters' calls and harder to reach potential voters. A generation ago, 70 or 80 percent of people called randomly answered pollsters' questions. That number has dropped to 20 to 35 percent, say pollsters."

I would love to hear Ruy talk about the effect of dropping from a 70-80% response rate to 20-35%. It seems to me that it (a) has to generate more noise, which might help account for the wildly varying poll results to date; and (b) might be subject to bias toward one side or the other. Which party is more likely to be willing to answer? Surely there are data on this....

Hi Ruy,

Could you add to your to-do list a discussion of whether it makes sense for polls to push leaners in a population of ostensibly likely voters? The more I think about it the crazier it gets... if you need to be pushed to pick a candidate in what sense are you a LIKELY voter? (And it's a big difference sometimes, like Gallup 55-42=97% vs. Fox 45-43=88%) Maybe there's a good reason to include the indecisive as likely and then push them, but I'm not seein' it

My optimism may be dashed by polls in a couple of weeks, but Rasmussen says 36% say economy getting better, but 50% say worse and investor class split 43% better 44% worse. Not good numbers for an incumbent.
Zogby has consistently shown Bush approval ratings approx equal to his number in the horse race -- high 40s. OH and FL have high Dem voter registration per NYTimes. Iraq is a such a disaster that fact is bubbling through the filter of windsurfing and purple hearts. Young people are talking about a draft and may vote in record numbers.
We'll just have to watch and wait, but today it looks good for Monsieur Kerry.

Slight comment - the 2000 results are certainly within the range of potential sampling error.

I think the '88 and '92 results are the most telling as the likely voter numbers are well outside sampling error. I am certainly led to conclude significant model error.

Now, I suspect the good folks at Gallup realize this and are acting in good faith to adjust their models right? right?

It's a very conservative (small 'c') method, which obviously tends to screen out a lot of folks. So in a low turnout election, I bet they do fine (and as I recall, they nailed a bunch of 2002 results when no one else did). Any idea, Ruy, if they do better in off-year elections? I'll bet anything they do.

If what we're reading about voter interest is even halfway true, they're going to have a bad, bad year.

1) A lot, 1

2) No, 0

3) No, 0

4) Always, 1

5) Yes, 1

6) Voted, no respose on who voted for, 0 - ?

7) 10, 1

Total = 4-5 ? Gallop says I'm a non-voter and I have never missed a vote. That's from '76 onward. Ironically, my score wouldn't have changed in the intervening 30 years, because I have always moved for economic reasons. Wouldn't Gallop be more truthful if he just came out and said, "people who move for economic reasons don't vote" ?

So, because I wasn't old enough to vote in the last election, I get the wrong answer on 3, 4, and 6? I know a lot of people ignore the youth vote, but some of us plan to turn out.

Unless I'm reading the scoring description wrong, ALL newly registered voters could not be included in the LV's?

I find it fully baffling that they ask the "on a scale of 1-10 how likely are you to vote", and then throw out any 10s who are new to the neighborhood (thus failing 2 and 6).

Bizarre.

This should be as scientific as Gallup:
The Latest from talk show host and comedian Skip E. Lowe
Sylvester Stallone's mom, Jacqueline Stallone, has a dog that has predicted the outcome of the November elections: George Bush will win by 20 percent. Jacqueline put down two newspapers, one with Bush's picture and one with Kerry's. The dog peed on Bush.

Urinanalyis of election data?

I prefer sheep entrails myself.

I concur with those who have made this observation:

NAME ONE PERSON WHO VOTED FOR GORE WHO WILL CHANGE TO BUSH.

Zell Miller doesn't count.

Couple that with the huge gain Dems are making in registering voters, and we end up where I've been saying we will for months. Up 5 million in the presidential election.

We hear all about the right wanting to register 4 million evangelics who didn't vote last time. So what? So what if Alabama, Mississippi, Georgia, and South Carolina get more votes for Bush? They're already in his column. Unless a million of them have moved into Ohio or Wisconsin or Pennsylvania, they are not going to be a factor.

All this support Bush has in rightwing states does him no good. If you win a state by 65%, the overage gets you nothing at the Electoral College.

The lack of response to telephone polls has rendered them ineffective in many instances. They do not produce reasonable samples.

But the key will be turnout. We can't just go thru the motions, we have to do better than ever before.

I know firsthand that ACT-America Coming together has a Gigantic budget and is planning a huge get out the vote operation for election day, in addition to the Democratic parties usual get out the vote drive. And I get the sense among minorities and immigrants who can vote that they are more energized this election.

On the other hand, I do think that Bush is winning some gore voters over. Bush has used the tragedy of sept 11 to his advantage. I talked to some voters while doing door to door canvassing this summer who would say things like "now is'nt the time to change horses in midstream"

Newport was on NPR last week. He presented quite the rigorous defense of their methods, basing his argument on the premise that "this is how we've ALWAYS done it, so ipso facto it's the right way." Seriously. He placed great value on the fact that they've been asking the same questions since the 30s. "Dr. Gallup developed it during the time of FDR!!"

http://www.npr.org/templates/story/story.php?storyId=3930565

Well, I don't know about USA/Today Gallup RVs versus LVs, but the new poll released today shows GWB ahead among RVs 53-42, and LVs 52-44 - So, either way, Kerry is lagging and has much work to do to catch up.

Before I get attacked for posting USA Today/Gallup numbers, I hear the WP/ABC is coming out tonight with better numbers for Kerry, though I don't know for sure.

I'm sorry, but I think his attack on Iraq PM was a disaster, and trumped his arguably more forceful speeches from last week. The guy came here to thank the American people, and all he gets from Kerry is contempt and condescension. But don't take my word for it. Read the post below from moderate Dem blogger Ann Althouse. Kerry trashes Allawi at his own peril.

Ann Althouse
9/26/2004

Kerry's final mistake.

Here's what I'd like to read about the 2004 election: an analysis of what happened written ten years from now. Anything written today is part of the events themselves. Writing today, you're caught up in the event of the moment. I feel I've been remiss in not yet posting a condemnation of the Republican's ban-the-Bible letter. Writing today, you're influenced by a hope or fear of affecting the events. And you also don't know how things will turn out. You don't know who will win the election, whether some dramatic event will occur in October, and how the war in Iraq will play out.

I try to imagine how someone looking back on the election will analyze it. If Kerry loses, one question will be, what was the turning point? Did the Swift Boat ads set the campaign on a track that led to defeat, was it Kerry's own choice to make his Vietnam service the central argument that he should be President, or was it a mass delusion--Kerry is electable--that overtook voters back in the primaries? Another question will be: When could Kerry have done something to salvage his candidacy? And: What was the final, fatal mistake?

I'm writing today, so I have all the deficiencies of a person writing today, but I have a prediction of the answer to that last question. The final, fatal mistake was criticizing and contradicting Iraqi Prime Minister Ayad Allawi when he was visiting the U.S. Kerry is in a very difficult position needing to criticize Bush's handling of the war, because the criticism itself seems damaging to the war effort. Bringing Allawi to the U.S. and linking him to the Bush campaign message was a powerful political move by Bush, but it was not a checkmate. Yet it forced Kerry into a terrible blunder. The grisly takedown has begun:

BUSH: This brave man came to our country to talk about how he's risking his life for a free Iraq, which helps America. And Sen. Kerry held a press conference and questioned Mr. Allawi's credibility. You can't lead this country if your ally in Iraq feels like you question his credibility.

CHENEY: I must say I was appalled at the complete lack of respect Sen. Kerry showed for this man of courage. Ayad Allawi is our ally. He stands beside us in the war against terror. John Kerry is trying to tear him down and to trash all the good that has been accomplished, and his words are destructive."

And the Kerry campaign is now wasting a lot of breath pointing out that it is an election year and the President's conduct of the war must be open to criticism, but what can be said of the attack on Allawi? Kerry will never dig himself out of this one, I think. And any time he makes his old favorite argument that he is much better suited for interaction with our allies, his Allawi blunder will be thrown in his face.
POSTED BY ANN ALTHOUSE AT 8:54 AM

One wonders whether Ruy will abandon the RV side in the on-going LV/RV debate after seeing today's Gallup numbers.

Those numbers show Bush up 52-44 among LVs, but up 55-42 among RVs.

It's never too late to flip-flop!

Ruy -

You say that RV numbers are more accurate as compared to LV? Hate to mention this... but Gallup's latest RV numbers show Bush up 13... among likely voters... it's Bush by 8.

http://news.yahoo.com/news?tmpl=story&cid=615&u=/nm/20040927/pl_nm/campaign_poll_gallup_dc_3&printer=1

So let's use Registered Voters. The latest Gallup has President Bush up 13 among Registered Voters as compared with an 8 point lead a week ago among registered voters.

More interesting is that this is the first poll that I've seen that shows a better showing among registered voters for President Bush than among likely voters. Looking forward to seeing analysis of the internals on this poll

I'm glad that Ruy found the Gallup data for the past years. I remember posting earlier about how Gallup has not successfully called an election's popular vote since 1984, but I couldn't find the exact data. Thanks for doing what I admit I was too lazy to do.

Americans move, on average, every 7 years. Younger voters do so more frequently. So at least one in seven voters will have moved within the last year. Those voters, even if they have re-registered through motor-voter, may not know their polling place when called even if they do intend to vote in seven weeks. When the time comes, they'll find out. They may even get a sample ballot from their local elections officials which will tell them where to go, so its not as if lack of knowledge TODAY is going to pervent them from voting in November.

So Q1 is a poor screener, especially for the young.

Under 22? You'd have to answer Q 6 with a "No."

Q3: does that mean 'in any district in which you were then living' or merely 'in the district in which you currently reside'? Simple misunderstanding is going to trip up a lot of folks who DID vote last year but have moved, even if they have re-registered and intend to vote this year.

Q3: alternate misuerstanding: what if I absentee vote in state or district other than the one at which I am answering the phone, or go home to vote (snow birds, college students)? Again, simple misunderstanding means I'm not a likely voter? Please.

Q4: how often? Relative to what? If I vote every November, is that often? Maybe I'm very politically aware and I know that there are lots of special elections and primaries, but I just don't care about bond issues and county dog catchers, and so I only vote in the big ones every other November. I certainly wouldn't say I "always" or "nearly always" vote. In many places, odd-year spring local elections might get only 10% turnout - does "nearly always" mean that only those who have a rather masochistic streak are LVs?

What about my neighbor who says, "Yeah, I always vote - every November." Of course, he also skips the primaries because "those are just primaries - they don't really count." School bonds or voter iniatives on a May ballot? Special elections for Congressional seats and local judges? Tightly contested primaries where the winner gets to run unopposed in the November general? All are lost on him - but Gallup seems to think he's more likely to vote than I. Really?

That's four of seven questions that through simple misunderstandings or flawed assumptions will screen out lots of folks who, by any reasonable measure, really are likely voters this year. No wonder their model is so far out of whack - they're screening for LVs with a model tailored for a 30-50% turnout election, but this year will be far higher.

Gallup is out. In a 3-way, they have Bush up by 8% among LV, 9% among RV, so the LV advantage is gone, but, iirc, the RV's haven't really budged.

s

I note that you said you don't know where to vote in your neighborhood. When do you think you will know? Won't you become a "likely voter" then once you know.

I suspect that with the lag in replies, there's a backlog building up. But just in case it's been missed, the new ABC/Post poll is out with Bush up by 6% in the LV, 7% in RV. The LV is 3% closer than it was two weeks ago in the same poll. Haven't had time to check the RV. Suddenly the recent polls actually are pretty close to each other. At the RealClearPolitics site, there are 8 polls from 9/20 on. They range from Bush up 4 to Bush up 9. All are 3-way LV's.

The new Gallup numbers are in and they actually show a slightly closer race among likely voters than among registered voters. Is this a quirk or did they change the likely voter model?

Either way, this survey is still much more pro-Bush than most other reputable surveys, so it's still somewhat on the edge to me (meaning it's about as pro-Bush as a survey can be and still be taken somewhat seriously.)

"2. KNOWLEDGE: Do you happen to know where people who live in your neighborhood go to vote? (“Yes” scores one point)"

In all the places in Los Angeles County that I lived, the place to vote changed in nearly EVERY election. You do not know where to vote until you are sent your sample ballot by the county.

Not a good question.

Mike,

It's not just Gallup. WP/ABC is out tonight: RV - GWB 51, Kerry 44; LV - GWB 51, Kerry 45. And these results are consistent with AP, Battleground & CBS which came out over the weekend. When all the polls start to convey the same thing, a degree of certainty starts to take hold, and the trashing of Gallup becomes more like noise.

And you can't count ARG, Pew & Zogby from a week ago which do not take into account this past week's events. While Kerry made a few spirited speeches, he was arguably hurt last week by a number of events: His aide, Joe Lockhart, got caught up in the CBS forged documents mess, He trashed the courageous & thankful Iraqi PM, and he was hammered by a number of critical commercials. A week is an eternity in the 2004 campaign, and you can't look at ARG or Zogby from a week ago.

To be sure, President Bush is vulnerable to many events beyond his control, and Kerry still has time to turn things around. But time is getting awfully short for Kerry, and he has to change the dynamics quickly before a sense of finality takes hold.

ABC out and Bush leads by 6 points. But, the internals are worse.

First of all, Kerry has a 37/42 approval, which is lower than Dukakis in November, 1988. I think this is the key factor in this year's election. Democrats are in a rage over the president and think that's enough to win. However, even if many voters want to fire the president, they don't trust Kerry as a replacement. The best example: Handling on Iraq: Bush gets 47 approve/50 disapprove, and 60% think we've been bogged down, so certainly this is the president's weak spot. However, the voters trust the president 53 to 40 to do a better job on Iraq. By focusing on tearing down Bush without building up Kerry, the democrats failed to define the Senator and let the republicans do that job for him.

Senator Kerry's entire campaign has been based on the belief that anybody is better than Bush. That belief is widespread among the far left, but not the public at large. I think this hatefest cocoon spun by the likes of Michael Moore has done great damage Kerry's campaign.

The Senator has shifted in the last week to go on the offensive, but so far doesn't have much to show for it. I think Thursday night is probably his last chance. If he fails to win over voters as a reliable replacement, it won't matter if conditions worsen in Iraq, voters will still stick with the president.

On the other hand, he has a big opportunity. If he's credible, and the job's number on Friday is week, it's all tied up by Monday. Otherwise, the race is probably over.

Same with ABC/WaPo Bush +6 LV, +6 RV

nielej:
maybe some people - especially first-time voters - will only look up their polling place when they go out to vote on Election Day. The first poll to pick them out as ''likely voters" will be the exit poll.

As for recent polls, Scott Rasmussen's three-day rolling average has had Bush ahead by just 1-2% for 4 days now. That's just on the home page where he rounds to the nearest whole number; if you go to the premium page where he gives a decimal point, and then do the math, I think the average over the 4 days comes out to something like 1.2%.
Rasmussen makes no secret of being a Republican, but as far as I can tell he is quite professional in his polling.
FWIW.

Zogby, as quoted above, indicates that the percentage of people who answer pollsters questions has dropped from 70-80% a generation ago to 20-35%.

Ths is a remarkable statistic. If one more "right-leaning" voter per 25 is inclined to pick up a phone and answer a pollster's questions than "left-leaning" voters, the effect of the the error goes from a bit more than 2% a generation ago to over 6% today. In general, such bias errors would increase by a factor of 3, under fairly natural assumptions like this.

Seemingly very minor perturbations in the telephone etiquette adopted by different types of voters can have dramatic effects in the quite small pool of voters remaining who will actually pick up the phone and answer questions. One really is left with a rather abnormal group if, say, only 20% will answer questions.

And how can one ever expect to predict the norm from what looks like a pool of deviants?

Hey, here's another fun fact to think about. I believe that some or most of the national polls exclude both Alaska and Hawaii from their calling radius, essentially due to time zone differences. If that's the case, the results should be systematically skewed relative to final eventual vote totals.

In 2000, Hawaii went 205K to 138K for Gore, Alaska has about 167K-79K for Bush. Together, they're about 0.7% of the U.S. population. So say you take a sample of 1,000 Americans, which is supposed to be random and representative. In a complete sample, you would most likely have about 5 Hawaiian Democrats and 3 Alaskan Republicans. Okay, excluding them doesn't have much impact, but in an incredibly tight race, it does tend to "miss" a net of about 150,000 Kerry votes, maybe half a percentage point or so? (Just a thought, anyway.)

I've never missed a vote. 10 out of 10 is my likelihood of voting, but I only scored a 5 because my polling place keeps changing with every election. I guess I don't qualify to be a "Likely Voter".

Smooth Jazz writes:

> But time is getting awfully short for Kerry


Yeah, Smoothie... As you correctly point out, the election is all but over already. There is no need for you guys to work your *sses off anymore, handing out flyers to Evangelicals that those godless Dems will ban the bible and force men to marry other men if Kerry is elected:-).

Let's hope your hero the Warpresident becomes as smug and complacent as he was four years ago and decides to visit California before taking a day off a few days before the election. I also hope he hasn't done his homework before the debates, either.


MARCU$

To believe the Gallup polls, you have to believe that Bush is almost tied with Kerry in the states Kerry will win, and winning by 25 points in the states Bush will win. Yeah, right.

Few remember how badly Gallup did last time, and they won't remember this.

I was wondering if any of you could answer a few questions for me.

1. Has Gallup utilized the likely voter methodology (as Ruy described) throughout the 2004 Presidential campaign? If not, what methodology did Gallup use in the past?

2. How do the other major polling firms determine likely voters?

Thanks for your help.

It seems obvious to me that the entire concept of polling only "Likely Voters" is fundementally flawed - many voters which are not LV are still going to vote and as it happens, their votes will count as much as anyone else's.

If there is any systematic bias between LV and non-LV, counting the opinion of only LV while completely ignoring non-LV will give a biased result. One certainly expects there will be systematic bias, as many others have observed.

Better than counting LV as one vote in a poll and non-LV as zero votes in the poll, would be to try to build a model of the probability of voting based on a serious of questions and to weight the opinions of pollees by the probability of their voting.

In this election, I strongly suspect that many people who haven't cared enough to vote consistently in the past are going to be motivated, because Bush's "leadership" has been so polarizing. People are motivated. Therefore, determining non-LV based on past behavior may be especially inaccurate in this election - even if a model such as I propose were used. I have been doing some canvassing of supposed spotty voters and have been surprised by the intense enthusiasm of many to get out there and vote this time. Those who rely on polls of LV may be in for a surprise.