« Kerry, Bush Running Close in 3 PA Polls | Main | CBS News/New York Times Poll Has It Close to Even! »

Gallup Strikes Again!

Here are Bush's leads in the three national polls released before Gallup's current poll (no RV data available for DCorps and Harris; Pew and Harris matchups include Nader):

Democracy Corps, September 12-14 LVs: +1
Pew Research Center, September 11-14 RVs: tied
Harris Interactive: September 9-13 LVs: -1

Looks like a tie ball game, right? But according to the Gallup poll conducted September 13-15 and released today, Bush is up......13???

Let's just say I'm just a wee bit skeptical of this one. First, Gallup's poll only includes one day (the 15th) these three other polls do not, so it can't be Gallup's survey dates that explain the big Bush lead.

Second, this 13 point lead is an LV figure and, as I've repeatedly emphasized, Gallup's LV screening procedure produces completely untrustworthy measures of voter sentiment this far in advance of the election. Here is a summary of the case against Gallup's LV data:

Sampling likely voters is a technique Gallup developed to measure voter sentiment on the eve of an election and predict the outcome, not to track voter sentiment weeks and months before the actual election. There is simply no evidence, and no good reason to believe, that it works well for the latter purpose. In fact, the evidence and compelling arguments are on the other side: that the registered voters are the more reliable guage of voter sentiment during the course of the campaign.

Here’s why. Gallup decides who likely voters are based on 7 questions about their interest in voting, attention to the campaign and knowledge about how to vote (e.g., where their polling place is located). The interested/attentive/knowledgeable voters are designated “likely” and the rest are thrown out of the sample. But as a campaign progresses, the level of interest among voters tends to change, particularly among those with partisan inclinations whose interest level will rise when their party seems to be mobilized and doing well and fall when it is not. Because of this, partisans of the mobilized party (lately, Republicans) tend to be screened into the likely voter sample and partisans of the demobilized party (lately, Democrats) tend to get screened out. But tomorrow, of course, the Democrats could surge, in which case their partisans may be the ones over-represented in likely voter samples.

That suggests the uncomfortable possibility that observed changes in the sentiments of “likely voters” represent not actual changes in voter sentiment, but rather changes in the composition of likely voter samples as political enthusiasm waxes and wanes among the different parties’ supporters. And that is exactly what political scientists Robert Erikson, Costas Panagopoulos, and Christopher Wlezien find in their analysis of Gallup's 2000 RV/LV data in their forthcoming paper, “Likely (and Unlikely) Voters and the Assessment of Campaign Dynamics” in Public Opinion Quarterly: “shifts in voter classification as likely or unlikely account for more observed change in the preferences of likely voters than do actual changes in voters’ candidate preferences.”

That means that, instead of giving you a better picture of voter sentiment and how it is changing than conventional registered voter data, likely voter data give you a worse one since true changes in voter sentiment are swamped by changes in who is classified as a likely voter.

I think the case against the Gallup LV data looks rock solid. In my view, it's time for them to drop reporting these data because they are highly likely to give an inaccurate picture of the state of the race and, by doing so--especially given the high profile of Gallup's polls--unfairly pump up one side of the race and demoralize the other. That doesn't seem acceptable to me.

Of course they'll reply: well, our data work so well right before the election, they must be the best data to use all the time. But, for the reasons outlined above, that reasoning is completely specious. And then there's this: the LV data haven't been working so well lately even right before the actual election. In 3 of the last 4 presidential elections (including the last one), Gallup's final RV reading was actually closer to the final result than their final LV reading!

As I say, maybe it's time for a rethink down at Gallup HQ.

Throwing out the Gallup LV data, then, let's move on to their RV result: an 8 point Bush lead. Obviously pretty far off the results of the other contemporaneous polls summarized above, but....could be I suppose.

But then there's this: the Gallup internals show Kerry with a 7 point lead among independent RVs. Huh? Kerry's losing by 8 points overall, yet leading among indenpedents by 7. How is that possible? Only if there are substantially more Republicans than Democrats in the sample.

That suggests that reweighting the sample to reflect the 2000 exit poll distribution (39D/35R/26I) would give a different result. It does: the race then becomes dead-even, instead of an 8 point Bush lead. (Note: Steve Soto of The Left Coaster got Gallup to give him their party ID distributions for this poll and confirms a 5 point Republican party ID advantage in their RV sample.)

One final note: I mentioned the Pew Research Center poll had the race dead-even just like the reweighted Gallup data. And what was Pew's party ID distribution in their RV sample? You guessed it: a 4 point lead (37-33) for the Democrats, just like in the 2000 exits.

I think we've finally found out how to make these polls get along!

Comments

Actually, I read on MyDD that Soto was told that Kerry led by 11% among Independents, not 7%. Where did you get the 7% figure?

Opps, sorry, by 10%, 51-41.

Opps, sorry, by 10%, 51-41.

What do you think of the new CBS poll? http://www.cbsnews.com/stories/2004/09/17/opinion/polls/main644205.shtml

where Bush is up by 8-9?

It's all so very confusing

This site continues to be a joke, like Kaus said, in trying to analyze away bad poll results. Even people in the Kerry campaign have acknowledged he is about 4 to 6 points behind. The current Bush lead may be gone by election day, but right now Bush is ahead and you ought to just accept that reality.

Been a long time since I took stat and stat inference, but I recall that if one is measuring something that exists in the real world, then it should be measurable and presumably several honest measures should get essentially the same results.
Ergo, when Gallup comes up with these results so out of sync with others and the history of these things, then to paraphrase Keats, that is all ye need to know about Gallup: His polls have zero value as predictors or even honest measures of voter sentiment.
I'm convinced he's been bought and paid for.

It is important to address bogus polls, before they become conventional wisdom. And it is working. Even the mainstream media is looking askance at the Gallup poll.
Re the CBS poll, I see on their website that Bush got 87% Reps and Kerry got 83% Dems, so it is probably another case of oversampling Reps. Incredibly, they do not list the breakdown among Independents, which to me is the most important breakdown in any poll.
So keep the faith.

KC--

I hardly think that's fair. Ruy posted a number of negative state polls today, not to mention Gallup. And he's not trying to debunk a poll showing a 4-6 pt lead; he's attacking a poll showing a 12 pt lead. Given the four contemporaneous polls showing an even race, I don't think that's an unreasonable thing to do.

But if you don't like the site, why are you reading it? Given that the site is dedicated to "The Emerging Democratic Majority," I hardly think it sane to expect Ruy to throw his hands up and declare the race over (as Mickey would undoubtedly prefer).

KC --

There's nothing wrong with explaining away bad news if the explanation makes mathematical and statistical sense and it reconciles the bad news with other sources.

Indeed, it would seem incumbent upon you to provide some reconciliation of your 5 point number with the polls cited by Ruy. The only place I've seen the 5 point number lately is from "Republican sources" in wire reports.

Question--Are Pew, et al., weighting their samples by partisan affiliation? If they are--and Gallup isn't--that could explain the discrepancy--but to Gallup's advantage. This has been such a strange election season that wild swings in partisan identification may be no weirder than anything else that has happened.

The following url sheds light on how Gallup conducted their latest poll.
http://www.theleftcoaster.com/archives/002806.html
Friday :: Sep 17, 2004
Why You Should Ignore The Gallup Poll This Morning - And Maybe All Of Theirs

I wonder what Kerry's lead in the polls will be after the first debate? I'll bet it is at least 20 points. However I do worry because it is going to be in Florida.

What first debate? If Rove believes that Bush has a significant lead, there won't be any debates.

Albuquerque Rally--On Message and On Fire
enthusiastic crowd of 20,000 !

http://blog.johnkerry.com/

Kerry began by emphasizing how George Bush has made the wrong decisions, taken the country in the wrong direction, and how the Bush administration is the wrong leadership for this country. Over and over Kerry spoke about the wrong direction of America, wrong direction in Iraq, rising deficit, failed healthcare policies, and the point that hit home – loss of jobs!

The more he cited the facts of how Bush has been bad for this country as a whole, the more the crowd booed Bush’s failed leadership for this country. And when he brought the message home by citing the New Mexico statistics, the crowd was howling.

AWC and Nate:

Thanks for the polite responses. You make some valid points, but I still think the race is currently (for what that is worth) about a 4 to 6 point Bush lead.

I do have a couple questions. (1) for those who charge intentionnal bias in the polls, why would an unaffiated polling company shape results based on bias and what is the evidence that any company does that; and (2) the criticisms based on percentage of democrats and republicans in the sample seems like a logical argument, but I assume polling companies don't agree that samples should be structured to reflect party affiliation -- anyone know why not?

One thing that will come out of these large flucuations and preceived Party ID swings is a hard look at the method of sampling. The seat of the pants reason being proferred is "cell phones" but I suspect strongly that there are a multitude of subtle changes in the lifestyle of many demographic groups that are rendering the telephone poll unreliable. For instance I am on a modem and can never to polled.

As to the "wild swings in party ID," they only occur in statistically small samples or inferred from samples that do not ask the question. The large [aproximately 3000-5000 ] samples show nothing of the sort. Those show a shift after to Republicans after 9/11 and a shift back to historical indentification afterwards. A behavior that requires no elaborate rationale. Therefore, apply Occam's razor.

Am I the only one that read The Vulcans? In the book, the author describes Sec. Def. Rumsfeld's job in the Nixon campaign. He had a friendly relationship with one of the Gallups. Gallup would give him (and therefore the Nixon Administration) a heads up on the soon to be released polling data.

Check it out, I'm not a left wing conspiracy theorists. This is an author that bio'ed the National Security Team no in place. Could there be a similar relationship?

How else do youe explain estimating only 33% Democratic turn out in one of the most fevered elections in history?

CBS poll partisan breakdown. I don't know what to make of how they did the RV vs. LV numbers. They list the total numbers.

Total Respondents 1286
Unweighted RV total 1088
Weighted RV 1048


Unweighted Number
Rep 462 36%
Dem 394 30%
Ind. 431 33%


Weighted
Rep 426 33%
Dem 399 31%
Ind. 462 35%

Those are the percentages. However I don't know which set they used to derive what figures. They don't say.

Obviously there is a partisan tilt , but I don't know how much.

Thanks to Ruy, and the Leftcoaster for this one.

The Gallup poll still has me pissed, the more I think about it the more certain I am its dirty.

They over-selected Republicans and still only ended up with a 6 percent advantage for Bush.

That is the key thing to remember. Even with their huge bias of polling 40 percent GOP to 33 Dem, their precious poll only showed Bush up 6!!!!!
Think about that one for a second.

Gallup really ought to be investigated the more I think about this. A)They used highly questionably sampling B)Didn't provide the key party ID except upon request(if they are so certain its a great system to use and fair, you'd think it would be published with the breakdowns..things that make you say, hmmmm.) C) They compounded the problem by doing their own tricky Dick manipulations, again not initially disclosed, to obtain even more skewed results D)This one gives it away, their own past polls didn't reflect anything like 13 pts, their last poll showed a 7 pt spread, the week before 3. Even if they aren't paying attention to other organizations' polls, you'd think they would notice such a deviation in their own polling. Unless it was intentional as I'm starting to believe.

Thank you, Ruy...

Guys, we gotta figure out how to do something about how the Bushies and the media are spinning these garbage polls. It's going to demoralize kerry voters unless something is done about it.

I'd like to take a chainsaw to Donna Brazile. They asked her about the Gallup poll on Judy Woodruff today and she said, basically, "Oh, there's still some hope for Kerry." Still hope?? Why can't they get somebody to represent the Dems that will do more than just parrot a milder version of the Republican talking points? The Gallup poll is horsesh-t!

This is what I read in CBS's own article regarding its poll:

"In this poll, where the Republicans hold a significant lead in voter preference and more voters hold negative views about Democratic nominee John Kerry, when voters are asked about their partisan identification at the end of the questionnaire, more identify themselves as Republicans. 36 percent say they are Republican, 32 percent Democrats. The percentage that identifies themselves as Democrats in this poll is lower than it has been in CBS News Polls conducted earlier in the year."

Draw the obvious conclusions.

Don't sweat Gallup. The last time they got a presidential election right was 1984. In the four elections since then, they were wrong by 3 points or more in 3 out of 4 years. Their average error over those 4 elections was 4 points.

That means that, on average, they wrong by more than the margin of error.

Gallup is a joke. Ignore them.

"This site continues to be a joke, like Kaus said, in trying to analyze away bad poll results."

Anyone quoting Mickey Kaus approvingly is going to immediately lose credibility in my eyes and in the eyes of most Kerry supporters. Kaus, while claiming to be a Democrat, has been trashing Kerry since day one, just as he trashed every other Democratic candidate in the primaries.

I'm just going over the CBS poll, its complete garbage as well.

The pattern of these bogus polls is the same this week, sample many more republicans than democrats.

As Debra pointed out, 6 percent more Republicans were surveyed in the poll. Huh? Excuse me? This is the same strategy as the Gallup nonsense, albeit not quite as absurd(at least the CBS?NYT poll doesn't compound the error by sub sampling even more repubs!)

And Dumbo is correct btw, something has to be done. This isn't a court of law, we don't need proof that these are faked up. We had a spate of bogus polls out of the convention, shot down within a day or two, then we had a big surge in them last weekend or early this week(someone will have to check) then a few days and quite a few polls leading up to today showing Kerry back in the lead.

Now, two major polls come out on Friday, just in time for the weekend news cycle, and to make it fodder for the Sunday morning shows. And are these polls remotely fair? No, the CBS news poll, if you go to the last page of it has a disclaimer stating as much.

Drudge began the day headlining the bush "BLOWOUT" in the gallup poll, now its not even on his page. Tonight he's toplined a link to the CBS/NYT poll. Interesting.

I really recommend watching Drudge, he is the GOP's central conduit to mainstream crap. Recently it was Drudge that ran with Brent Bozell's ridiculous CyberCast News "debunking" of the CBS memo. Look in to Cyber Cast, it used to be Conservative News Network, and funded by Richard Mellon Scaith, remember him? FYI, he also funds Chris Ruddy's NewsMax...you remember Ruddy, he was the wonderful author of the Vince Foster Murder story, which ended up getting him let go from the NYPOST, which is impressive considering the Post is Murdoch's main outlet in the traditional press. Other interesting connections in the web o'crap is World Net Daily's Scaith funding, like Newsmax's, through something called the "Western Journalism Center," which Scaith donates heavily to.

Check out ConWebWatch for a good starting off point: http://conwebwatch.tripod.com/stories/primer.html

Weekend news cycle? Releasing something Friday is the best way to bury. People don't pay as much attention to the news that comes out on Friday as they do during the week.

Still have that problem..

THE USA TODAY/GALLUP/CNN spin machine..

The misinformation just keeps recycling and takes on a life of its own

I still don't understand why Gallup or, for god sake, CBS/NYT would be biased against Kerry in their polling. Why isn't it valid to assume these professional pollsters know what they are doing? With the margin or error, they don't claim they are dead on.

Dumbo is dead on about Donna Brazile. I'm sure she's a nice, thoughtful woman and everything... but she blows as a surrogate. Why she is let on these shows to talk for Kerry I have no idea. She seems half asleep like some narcoleptic and is so even tempered she comes off like Yoko Ono on ludes. For Godsake.

The more I've been considering, the more I come to the conclusions that Dems need to clean house on their surrogates. Again, they are nice people all and all, but we need compelling, entertaining, energetic firebrands. Carville is excellent at it, but come on, Carville is not a TV guy, I'm sorry. He looks like Hunter Thompson, in fact, thats my new conspiracy theory, have you seen HST and Carville at the same place and time?? Anyway, then there's Begala. Begala knows what he's doing but the man takes things too personally...he loses composure too easily(a symptom of being passionate).

There is a fine line, and its not easy to find, but the GOP has so many skilled surrogates. I'm convinced Hannity is some freakishs Ralph Reed, Christian Coalition, genetics experiment. Bill Bennet, Bill Kristol, Fred Barnes, Brit Hume, Joe Scarborough, Bill O'reilly, Michael Savage, Tucker Carlson, Tony Snow, Neil Cavuto, Charles Krauthammer, Ann Coulter, Larry Elder, Laura Ingraham, and the anti-christ Ken doll we all know as Shepard Smith.

As a group they are attractive, polished, well spoken, keep to the party line and are rotated like a stable of pitchers. There are just enough of them that they appear to be this giant group, and yet small enough so that they people can see them regularly enough on television the audience can identify with them. It's slick, sophisticated PR, and the Dems, STILL don't understand that this isn't about issues or policies, its about ACTING.

Does it mean anything that some aspects of the media are determining who will make up the Kerry cabinet? Does it matter that they are also working out who will be leaving washington en masse?

Maybe the press is learning to have little confidence in its own spin. Maybe... I dont know.

There was a good analysis of polls and their coverage by mainstream media on "Now" by Bill Moyers. Media would claim that election is over and this itself would affect the outcome of the election by discoraging people to vote for the "looser".
Probably, the best idea is to forget about polls altogether untill November 2nd, and spend your free time on talking to people, encourage them to vote.

I just pulled up the NYT story on the poll. At least Kerry's friends at the Times put a headline on it that helped Kerry as much as possible and the article, while it must make Kerry and his supporters cringe in spots, managed to report a very pro-Bush poll in terms that were generally negative towards Bush. I think if Kerry had achieved such a sustained surge in the polls, the Time would have been delivering high praise to him.

I assume the Times hates to do it, but this article probably will spread the pessimism/panic among democrats.

Has anyone seen my Bebo?

He has been lost in a sea of Democratic spin, liberal bias and total lack of common sense.

Bebo where are you?

Lola misses Bebo

Oh wait one more thing.........

4 more years! :)

You are right Natasha... Its pointless to get caught up in deciphering these polls. Unless you have a passion for such things, then its best to read the results and then go campaign some more.

I take no confidence in any polls no matter who is the leader. Just as polls showing bush with a commanding lead are obvviously wrong, any other poll showing Kerry with any kinda lead can also be wrong.

I think that best appraoch to this thing is to test the soil on which you walk, thereby listening to the people around you and guaging from there.

If however the bush lead in the polls will make folks mad and get out the vote for kerry, then I LOVE polls. In any event, I hope that people will gather the positive side from every poll.. so that if they show bush in the lead, then we will get out and get people mobilised for kerry.. and if the polls show kerry in the lead, then it would motivate us to get out and increase the lead... positive motion only.

PonyFan, you release bad news at the end of the day on Friday to give a weekend buffer. People have time to take it in, and come Monday its not news anymore(at least thats the hope.)

For a dubious polls it makes sense for the same reason. Bad polls are quickly knocked down and discredited. Releasing a bad poll on Friday gives 3 days of coverage to a poll that otherwise would get 1 day. You'll see these polls discussed, both Gallup and CBS/NYT on all the morning shows sunday. You'll see it on Howie Kurtz's weekend show. You'll see on Fox's Reliable sources. And of course you already got to see it today and tonight and will be news all of tomorrow,(granted not a big news day, but still its repeated on the half hour on HL News, the scroll never stops on Fox or CNN...

So I stand by the idea that releasing bad info without press conference or official WH response of Friday afternoon helps a nullify the bad news. Friday isn't some magic date, some things work better than others on the weekend. For a poll you don't have to cover a press conference, you don't have to contact officials not working etc....Polls are perfect for weekend news cycle.

And now KC. Could this be random? Sure. Why would it be smarter to assume it isn't?
1. the sampling method in the bogus polls were flawed in teh same manner, super selecting GOP
2. this is the GOP we're dealing with, I suggest you read The Hunting of the President and David Brock's Blinded by the Right and/or his Republican Noise Machine...you might also read up on Professor Robert Jensen's study of the Bush and media, also the Columbian Journalism Review, ConWebWatch, FAIR. You can start to get an idea of how they operate. Calling in a favor, paying people off, planting moles inside the polling agencies, legal intimidation...lots of ways the GOP campaigns....Oh and check out Kitty Kelly and what the GOP tried to keep her from publishing and now has been able to kill a Larry King interview.

Does that prove these polls are fakes? No. But again, this isn't a court of a law. We aren't criminally prosecuting someone, its smart to assume the polls are manipulated, now you know you'll have to be ready for the next crazy one pulled out the day before the election for example.

My meta-analysis of state polls (http://synapse.princeton.edu/~sam/pollcalc.html) suggests that the true Bush margin over Kerry in battleground states is currently about 1.8 points. This is far closer to the near-tied polls than to the Gallup polls.

My data set includes every pollster that releases state polls reported in the press (or in the case of Zogby and Rasmussen, to subscribers). This analysis provides independent confirmation that the Gallup-type polls are doing something fundamentally different from a large body of polling organizations.

Sam Wang
Princeton University

I recently did some polling field work for a political campaign. During the two hours I worked (6PM-8PM Monday evening), I got answering machines 60% of the time. The other person working at the time had a similar experience. I think the biggest problem with polls is not cell phones but answering machines and caller ID. How can you have a scientific poll when the subjects of the poll are screening calls. I think this is why you see such wild swings in polls like Gallup. Think about it. Who is going to want to pick up the phone right after a Republican convention? Maybe enthusiastic Republicans? Don't listen to the polls, at least until after the third debate.

I just sent a letter to the editor of USA Today via the internet expressing my doubts about the latest Gallup poll they seem to express as Gospel. I suggest anyone else who has doubts to do the same. Here is the link to the "Letter to the Editor" page:

http://www.usatoday.com/marketing/feedback/feedback-online.aspx?type=18

Be sure to include the party breakdown numbers in your e-letter so you can refute the poll with facts.

Professor Wang,

I appreciate someone of your academic stature weighing in on the matter. It would appear that some major polls may have some explaining to do. I think the questions presented by their results would at least warrant a restatement publically about their methodology. Although they are private organizations, during an election I believe they have duty to the public to at least explain themselves. As I said in an earlier post, if they truly have faith in the predictive value of their study, you think they would be stressing the merits of their procedure as opposed to the more traditional methods.

LV is always measuring as if this was the eve of the election.

The greater the distance between the poll date and the election date, the greater the difference between the population that pass the LV threshold in the poll and actual voters on election day.

Event's will change who is or isn't a LV. Good news for one candidate will elevate LVs for that candidate surpress LV responses for the other candidate. LVs are valid on the eve of the election is because people finally have to make up their minds.

Someone here expressed the view that pollsters' reputations for accuracy are on the line, so why wouldn't they have every incentive to be as accurate as possible?

One obvious response is that this far out from the election there is no objectively determinable "correct answer". At this point as a pollster you can be shown to be an outlier. But you can't be shown to be a liar. (Or at least it's difficult to do the latter.) Polls this far out are ungraded homework assignments. The final exam doesn't come until Election Day--that's the call that may affect a pollster's reputation.

It also seems obvious that whoever the key decisionmaker is with the consumer of polling services may be overly influenced by a pollster's name recognition or personal relationships developed with people who work for the polling firm, for example, as distinct from the firm's actual track record.

If you asked a lot of folks if they'd let Gallup do a poll for them they'd say yes in a heartbeat without correctly assessing how well Gallup has done on polling in the particular area they are interested in. Everyone has heard of Gallup. That is not the same thing as saying Gallup has historically been good at a particular kind of polling. Did USA Today choose Gallup as its pollster because of Gallup's superior performance in predicting the results of recent presidential elections?

And as we know, an assumption that purchasers of polling services are always interested primarily in "truth" (as in accurate measure of the underlying reality), as opposed to a preconceived desired result, is fallacious.

In short, the market works far from perfectly in rewarding better performing and punishing poorly performing pollsters. The points made above have been implicit in a number of other posts, but by generalizing them beyond the political campaign context we discuss here I think they can more easily be seen as the quite reasonable points they are rather than as exercises in self-delusion as a few here suggest.

Kc,

Suppose you had business in front of the FCC, (currently, all the networks do) and you've know that Mike Powell is not only a partisan hack, but one that doesn't mind being identified as one. Now you know Powell has displayed a vindictive streak in the past...would you please him on a throwaway poll or would you piss him off? Hey, were talking about billions of dollars here...no...no, American would stoop to dirty politics, would they?

When people don't know much about statistics, they say questionable things.

First, Gallup and CBS/NYT traditionally use random sampling. This is how the whole concept of sampling error is really supposed to work. In a population of, say, 230,000,000 adults, if 1,070 are randomly selected, there is a 95% probability that the whole population would answer the question the same way, within an interval of +/- 3%. This ONLY applies to a random sample. Adjusting the sample to fit a preconceived notion of what the sample should look like introduces systemic bias and makes the results less accurate, not more accurate.

Second, party identification is not a fixed characteristic like sex or age. It varies all the time, just like political opinions. There is no reason to believe it will not change over any four-year period. Zogby himself says that it changes from election to election. There is no reason to believe that voters in November will fall neatly into the 39% Dem - 35% Rep - 26% Ind mold. In fact, there is reason to think the opposite. According to the Pew Center's Survey of the 2004 Political Landscape, there was a net change of 5% in party identification in favor of the Republicans between 1997-2000 and 2001-2003. Things do change.

In conclusion, people who complain about random sampling being more biased than weighted samples are misinformed, as are people who don't understand that party identification changes, and that it has changed.

"I think the biggest problem with polls is not cell phones but answering machines and caller ID."

I think this is very likely true, but it fails to note the direct interaction between cell phone habits on the one hand and caller ID and answering machines on the other.

The vast majority of people with cell phones give out their mobile numbers only to people they know and generally are quite willing to speak to. The land line is, because of that, not something many people even bother to answer, particularly if the caller ID is not familiar. The more the meaningful calls go to the cell phones, the less incentive most people feel to answer their land line phones. The near ubiquitousness both of cell phones and caller ID, a quite new phenomenon, having reared its head dramatically in the last 4 years, has, I think, changed phone habits in way that is very hard to come to grips with as regards polling.

There's a revolution out there that pollsters won't even fully detect and appreciate until this very November.

LB, and bt,

Wow! Nicely put! Good crisp analysis.


4 More Years!

Lolas,

Have no fear, Bebo is here!

4 More Years!

Don't you right-wing cracks have anything better to do than show up on a Democratic site spewing crap?

A poll isn't valid if the sample doesn't represent the population. If you have concrete breakdown of party affiliation for 2004, please share them. Give me a website or something.

The notion that polling LVs tells us only about the composition of the LV contingent at any given moment is accurate, but incomplete. This is more or less all polling ever tells us.

This is because, as decades of research amply demonstrate, voters seldom change their minds once they choose a candidate to support. The question in any election concerns whom they choose when they make up their minds in the first place.

Early in campaigns, the more informed/interested/partisan voters already support their party's candidate(s)--even before these are nominated! The campaign--always, not just this year--is not about changing minds but rather about influencing the uninterested/non-partisan UN-likely voters. Because these people are largely uninformed and disinterested, they are much more open to campaign/media influence. And because they make their choices late and on a fairly insubstantial, aleatory basis, the latter stages of a close race have an outsize impact on the outcome.

The point concerning polls, then, is that they track changes in the composition of this ever-shifting group. Since campaigns do not persuade the vast majority of those who end up voting, changes in opinion surveys reflect the entry of new citizens into campaign consciousness. But these new entrants may not stay; they may be outnumbered by subsequent entrants; their opinions strongly reflect the latest news content; and they are far more likely to wind up not voting after all.

Of course, where polls DO make a huge difference is in "3rd person effects." That is, when voters (such as posters on this blog) imagine outsize effects of reports of "public opinion" on others, thereby closing the vicious circle that often results in reduced voter interest, turnout, etc.

None of which is to say that the Kerry campaign is doing as well as it should. It is seriously mismanaged and moribund, no matter what the polls say. The large difference between numbers of dissatisfied voters and those going for Kerry demonstrates that Kerry has utterly failed to tap the enormous discontent with the Bush administration. It should be the source of massive embarassment for the Democrats that they (we) have a candidate so disconnected from (political) reality that he has yet to win the support of millions openly disgusted with Bush.

First, Gallup and CBS/NYT traditionally use random sampling.

I think you're missing the point. What's troubling about these polls is that the odd results suggest that they're either using a non-random sampling or are adjusting the sample after the fact to fit a preconceived notion of party breakdown. Your statement that they are using a random sample just begs the question.

PonyFan is right. Gallup hasn't successfully called an election in 20 years. Zogby, on the other hand, have called every major race for the last 4 years and almost never been wrong.

Probably because Zogby actually takes samples that are representative of the actual population.

I mean I barely passed Stats and even I know that samples are no good unless they reflect broader population trends.

KC - One of the key points you need to realize to understand the posts that Ruy makes is that "bias" does not mean "intentional". When you're talking about people, bias is generally assumed to be intended; when you're talking about polls it can easily be unintentional. All Ruy has pointed out is that it's systematic -- it happens again and again.

The most likely cause for a bias in a poll is something in the methodology. It could be the poll itself - small details as the phrasing of questions and the order in which candidates are listed can have an effect. It may also have to do with the details of how they gather their sample, which may not be as scientifically random as it's supposed to be. Without a lot of detail it's impossible to tell from the outside what it might be. That's why Ruy is calling for a "rethink down at Gallup HQ"... they are generating demonstrably different results than the rest of the polling field, and should really be peer-reviewing their methods.

And remember -- any single polling organization is expected to generate unrepresentative results some of the time. Polls are supposed to be within the MoE 95% of the time, which means that 5% of the time (1 in 20 polls) they won't be. And recently, when compared to election results, political polling only gave results within the MoE 84% of the time:

http://www.mason-dixon.com/news_text.cfm?news_id=158

That's significantly worse: 84% would mean that about 1 in 6 polls are off by more than the MoE. No matter what your political views, that's food for thought.

Yes, you can explain the Gallup results away as shifting party ID. If you're a Republican and not afraid of false optimism, you may want to embrace that hypothesis since it would be fantastic news for the GOP. But one of Ruy's points is that no other research polls have yet shown a consistent and systematic shift in party ID, so there's not much evidence to support such a view right now.

Well I find Ruy's arguments interesting, and somewhat persuasive, I am not entirely convinced that Gallup, CBS, Time, et al are completely wrong. The main problem is that there is no underlying explanation for *why* their samples or methodology should be suddenly biased in favor of Republicans when there is no indication that methodology suddenly changed in late August.

That the distribution of party allegiance still favors of the Democrats seems to be more of a hunch or an assumption as opposed to something that has been empirically verified.

One can be *skeptical* that there has been a massive swing in party ID, in that such a swing is unprecedented, but a sudden shift in sampling bias toward the Republicans by several major polls at the same time would also be unprecedented.

Ack, I forgot about the "who did you vote for in 2000" questions, which show a clear pro-Republican polling bias. As party ID *is* potentially flexible, maybe the pollsters can start using the 2000 vote to weight the results.

What's fair is foul, what's foul fair.

Macbeth's Witches or Karl Rove??

Jimmy Breslin had an article a couple days ago about how the polls, whether it's Gallup, Pew, MSNBC, etc. are totally missing the boat. The fact is, millions of voters that are being registered don't show up in these polls. Here in Cleveland, OH, the board of elections is literally swamped with new voter registrations.

I don't care what any poll says, Bush is in trouble.... I can see it in the amount of pro-Kerry bumper stickers and yard signs.

That more people are registering is great news. Turnout almost has to be a lot higher this time than in recent years. To the extent that the polls underestimate turnout, they likely underestimate what the Dems will do.

Some people they phone are not registered but will register. That might yield some bias, though it's not a gimme that more Dems are registering that Republicans. Do we know the registration deadline in the various states?

I would like to believe this, but does it explain the worsening state polls? Are they oversampling Reps too?