« Just How Bad Is That CBS News/New York Times Poll for Bush? | Main | SurveyUSA Polls: Kerry has Edge in OH, NJ; Lags in MO, NV and VA »

Gallup Poll Racially-Biased

By this I don't mean that Gallup's pollsters are themselves racially-biased. Rather I mean that their likely voter (LV) samples--whose results Gallup continues to promote above all others--tend to be racially-biased because of the methodology Gallup employs to draw them.

Here's a basic sketch of how Gallup's methodology works:

Gallup asks each [RV] respondent seven LV screening questions, and gives each person an LV score of 0 to 7. [Assuming a turnout of 55 percent], the top 55% are classified as likely voters. In practice that typically means all of the "7"s--given full weight--plus some proportion of those with lower scores (usually the "6"s), who are weighted down so that the size of the likely voter sample matches the projected turnout for the year (apparently 55 percent this year). All other voters are discarded from the sample.

Note that the demographics of Gallup's LV sample are not adjusted in any way (as their overall samples are) and are simply allowed to fall where they may.

What this means is that if, say, minority voters are much less likely to answer the 7 questions "right", they will be correspondingly under-represented in the LV sample--perhaps severely under-represented.

That is exactly what turns out to be the case. According to data obtained by Steve Soto over at the Left Coaster, Gallup's latest LV sample--the one that showed Bush with an 8 point lead--has only 14.5 percent minority representation and only 7.5 percent black representation.

How plausible is this as a representation of the election day electorate? Not remotely plausible. In 1996, minority representation among voters was 17 percent; in 2000, 19.4 percent. In 2004, the minority proportion of voters should be more than this, because minorities are growing, not declining, as a percentage of the US population. So 14.5 percent for nonwhites as a prediction of the 2004 electorate is very, very unlikely. It would defy both recent history and powerful demographic trends.

As for 7.5 percent blacks. C'mon. Blacks were 10.1 percent in 1996 and 9.7 percent in 2000. And they're 12 percent of the voting age population. There's just no way in the world blacks will only be 7.5 percent of voters in 2004.

So, in effect, Gallup's likely voter approach is disenfranchising minorities in assessing American voters' inclinations on the coming election. That's wrong and Gallup should stop doing it.

And speaking of disenfranchisement, how about America's young people? This group is also full of voters who are relatively unlikely to answer the seven LV questions right and thus qualify for admission into the exalted realm of the Gallup LV sample.

Sure enough, Gallup informs us that young voters (18-29) only compose 11 percent of likely voters. Well, that would be quite a trick. In 1992, young voters were 21 percent of voters; in 1996, 17 percent of voters; and in 2000, 17 percent again. And we're supposed to believe that young voters are all of a sudden going to drop to 11 percent this year? Puh-leeze, this doesn't pass the laugh test.

As it happens, minorities--no big surprise--lean very heavily toward Kerry this year. But young voters are also Kerry's best age group this year. Systematically under-representing these groups in Gallup's LV samples will therefore have an obvious, and fairly substantial, effect on their results, tilting them in the direction of Bush and the Republicans.

That's not right. Gallup should know better. And we should all know better than to trust results that are based on effective disenfranchisement of large numbers of minority and young voters.

Comments

Ruy:

May I say that you are likely to find ALL the polls, exceptong Zogby perhaps, to have similar flaws. It is my view that pollsters are having a very difficult time structuring representative samples - more so than ever. It is my view that there is a very serious question as to the validity of most polls and that these flaws may lead to the largest error rate in polling history.

Gallup's numbers, then, would only be accurate if their sampling accurately represented the degrees to which minority and underaged voters were going to be disenfranchised this year. But please note that I would never suggest that there were any efforts being made at the grassroots to disenfranchise any particular group of voters.

I've been doing politics for a long time--and for as long as I can remember the adage ran that good weather favors high turnouts and high turnouts favor Democrats. All the fancy number crunchin' Donkey Rising is doing is certainly welcomed news, but seems mostly to validate the folk wisdom that if a lot of people show up, the Democrats will win.

A news story I read today suggest that Washington state is expecting an 84% voter turn out. I suppose that rates as "a lot of people."

Could it be that they are factoring in successful efforts at voter suppression activities?

Superb analysis.

Although this addictive site is becoming almost a full-time read already, I wonder if Ruy or pressed staff could occasionally take apart the kind of "poll analysis" the BBC TV news aired last night (Tuesday) with a Republican pollster spinning like crazy?

(BBC seems to me to usually come full circle over a period of days; but last night's segment could have been... a straight Gallup press release.)

Ruy,

I realize this isn't your area (and I appreciate this), but what do you think might be the REASON for Gallup skewing their sampling this way? I have always held Gallup in esteem, but this kind of sampling can serve no HONEST end. Is this like a cooked set of books? Does the Public get one set (this one), and the Dark Lords get a more accurate set?

How can Gallup stay in business if their polling is skewed so as to achieve predetermined results, as these appear to be?

However plausible Gallup may think its Likely Voter questions and formula might be apriori, the fact that it has demonstrably false implications such as the severe underrepresentation of minorities and young people would send any polling organization with aspirations to objectivity and science back to the drawing board.

I don't know whether to attribute Gallup's methodology to arrogance or partisanship. But science it is not. They should strive to rejoin the community of the reality-based.

I appreciate the attempt to keep the focus on what the polls are actually showing, but what I found disturbing in the Gallup poll was not the 8 point margin in LVs (though USA Today in particular did play this margin up in their headline Monday) but the 3 point margin among RVs. Ok, the internals do favor Kerry, some of the other polls are showing Kerry in better standing, he's running better in the swing states, if Zogby can be believed, and so forth. Still, what I'm seeing is the momentum that had been shifting in Kerry's direction after the first debate has largely stopped, even among RVs, which, as I understand it, is the most optimistic scenario for Kerry, unless there is a systematic sampling bias that is excluding Kerry RVs from the poll.

jwb

I think, the correct way to screen for likely voters is :

1. Ask the screening questions, but do not pre-determine the weights.

2. Instead, using the demographic profile information, find the weights to be assigned to screening questions so that the selection from the polled sample matches the best-known demographic profile ( i.e., last election extrapolated with any well-determined trends).

3. Once the scores to be given to each answer to the screening questions is known, compute the poll results.

To PM Summer: When you think about it who really has an interest in what, when it comes to these polls?

I am confident the Bush campaign sees enormous value in pro-Bush polls which they want to affect the dynamics of the race, as opposed to merely measuring accurately the current snapshot.

If you are a business doing, say, marketing research you have every reason to want the most accurate possible results, assuming you want to make the best possible business decisions.

If you are a political campaign, however, you employ your own private pollsters to tell you what is really going on, so as to be able to make the best decisions about where to deploy your resources and place the emphasis in your campaign strategy. When it comes to publicly reported polls, you may conclude that publicly reported polls showing your side doing much better than it really is doing are all to your advantage. You don't care that they present an accurate picture. In fact, you don't want them to give an accurate snapshot. You already believe you know what the real picture is from your private polls. The publicly reported poll is a tool of advocacy, not measurement, from your campaign's standpoint.

Look at it from the perspective of the newspaper reporting the poll. USA Today's Monday headline, citing the Gallup poll showing Bush up by 8 among LV's, was splashy, eye-catching. Translation: good for sales. No one--well, almost no one--will remember a week from now which newspaper reported which poll. And many of those who do remember will draw no negative conclusions about your newspaper if there is a huge swing from one poll to the next. They are as likely as not to scratch their heads and wonder what led to the "swing" as to doubt whether there really was a "swing" at all. Call it the allure of numbers if you will. With some, numbers have a surface aura of credibility, regardless of whether that is merited in the case in question. You know the saying--"statistics don't lie, but liars use statistics."

But don't pollsters have strong incentives to "get it right"? As I've suggested here awhile back, it is far from clear that anyone--the public or the media outlets that contract with them--really cares how well a pollster did during the course of the race, when it comes to deciding which polls to report on and emphasize, and how to report on them, in the future. In fact, it does not appear to me that either most of the public or the media outlets that hire pollsters and report their findings even really cares about how well the pollster predicted the *final* results in previous years. Think about it--do you ever recall a TV or newspaper piece reporting a poll result insert any reference whatsoever to the previous performance of that pollster? Is Gallup hurting for business as a result of their poor recent track record in recent presidential races? I am aware of one WSJ and one NYT article of late which did not report new poll results but which instead explored some of the methodological issues that arise as a result of these polls. But this is not the sort of stuff that likely draws a lot of attention from most ordinary citizens, many of whose eyes glaze over when it comes to such discussions.

An admittedly imperfect analogy is to sportswriters who offer predictions on how teams will do, either for a season or for particular playoff games, say. I have at least seen some football prognosticators post their performance results for the previous week or for the season. But I've seen nothing like that when it comes to including the track record of pollsters in the course of reporting on a particular new poll result.

When you come right down to it, none of the stakeholders--the public, political campaigns, publicly-reported pollsters, or the media outlets that hire and report on their efforts--has a consistent, strong stake in accurate *publicly reported* polls during a political campaign. And at least one side in this campaign may well see poll results which very possibly are highly inaccurate as a key peg of its campaign strategy.

It's remarkable--one might even say outrageous-- when you think about it, think about the relative stakes for a society involved in these kinds of disclosures. And when you consider that it is not at all unusual these days to see a report on a new poll result be the lead story in a major newspaper or media broadcast.

Jimbo:

Almost every poll- NYT/CBS, Zogby, Ramussen, ABC, NBC, Survey USA (to name just a few) and multiple state polls (not the least of which OH and PA show Kerry ahead, and FL shows a tie w/ Bush polling in many under 50 percent). So other than the WaPo and Gallup poll how are you coming to your conclusion. RVs by the way also, if I am not mistaken, weight for demographics. Moreover, Kerry, and this is something that is crucial, I don't think needs a momentum- Bush does. Why? Because Bush is the incumbent. But, just to let you know the internals I have been reading from both Ruy's site, RealClear Politics, Mystery Pollster and several other sites indicates independents are breaking for Kerry and he still has not obtain the full weight of his base which is also trending Kerry- although Bush has- there is an excellent article about base support over at Salon that was put out back in the Summer/Spring that you should read. Basically, it argues, as have several others, that Kerry has more room for growth than Bush does.

I disagree with frankly0. What other pollsters are doing by abitrarily weighting their results to 2000 exit surveys (why stop at 2000? why not weight them to 1992 or 1988, or why not weight them to the 2002 mid-terms?) they are introducing EVEN MORE bias and partisanship. Gallup is simply going out and reporting their raw findings. That is science--partyID weigthing is art.

Nielej: At the heart of science are replicability of findings, and transparency. That is why an emphasis is placed in the journals on disclosure of how the experiment was conducted--so others can (in theory, at least) seek to replicate the finding or disconfirm it. Pollsters who do not disclose their methodologies--for understandable proprietary reasons--prevent any possibility of replicability, and also prevent the sort of transparency that is essential for others to scrutinize the methodology used for possible bias or the use of assumptions which, even if they do not stem from conscious bias, cannot be adequately supported.

Gallup should be renamed Gwallup, GW-all-up.

This was a fantastic post. From my reading of the polling literature this critical perspective on the likely voter models is taking hold and becoming a new common sense. I wonder how to take the next step and develop a new and better LV model that could be lobbied for? What would its features be?

Ruy,

Why isn't Gallup defending itself? They have a long standing reputation to protect. Why aren't they slamming critics like you and Charles Cook? Is it because your criticism isn't landing on the front page of mainstream media?


This from today's Slate's "Election Scorecard"

Update 1:15 p.m. ET: New polls give Bush hope in New Hampshire but shore up New Mexico and Wisconsin for Kerry. The gradual isolation of Gallup in the latter two states makes us wonder whether to reexamine Gallup's numbers elsewhere.

That last sentence is gaining some resonance.

This is a two edged sword.

One the one hand, if the Republicans actually believe these biased polls, they are less likely to over-act (i.e., more than already) to suppress the vote, stymie the proper counting, etc.

On the other hand, Democrats believing these polls might not vote, thereby making the polls self-fulfilling pruphecies.

We must be sure to educate the Democrats, independents and other "unlikely" voters about this and encourage them by all possible means to vote anyway, while slipping around letting the Republicans know.

Of course, Karl Rove probably is already well aware of this. The desperation of the Republican leadership is obvious to those with eyes and ears, and even the media's shrill attachment to Bush-deregulation does not appear to be working any more.

Let's hope, pray, and work for the basic principle that Americans are NOT as stupid as we are led to believe, and that the other side WON'T be able to rig enough votes to steal this one, too.

Keep up the good work!

mhr

Roy,

I appreciate all the hard work you have done with respect to Gallop's bias.

My question is, do negative poll results (when they are this close) necessarily discourage voter participation for the candidate who is trailing? I always thought that the opposite was true. I was thinking that poll results this close to the election might actually work in Kerry's favor.

I'm not saying you shouldn't pursue Gallop's bias, I'm just wondering if anyone has done any research on whether the effect is necessarily harmful for Kerry.

Randy Farr

bruhrabbit, that last line is encouraging news:

"Basically, it argues, as have several others, that Kerry has more room for growth than Bush does."


I just wonder how different the final pre-election polls will be from the actual results. Last time around, we're many of them off by at least 3%? I would have thought they learned from their mistakes, but maybe not. Then again, maybe it doesn't really matter at all and we're paying too much attention to polls to early in the race.

Adrock:

I think you are making the wrong assumption: that they care about being accurate. Polling is a business. This iswhy their models are proprietary. They have a vested interest in defending their projections, not in making certain that their projections are accurate. This is why Gallup attacked Moveon's full page ads. Not merely b/c they thought their modeling was right (I can't know whether they do or do not know it is right). But, what I can assert with a relative high degree of certainty they have a vested financial interest in being perceived as right. This brings up one of the issues w/ private sector approaches- (I am a true business guy in that I believe that a lot of issues should be subject to the efficiencies of the private sector). But, there are some issues that are questions of the public interest or at least one should fully understand that there is (through full disclosure) there is a conflict of interest (transparency).

My question as I have been following your and Steve Soto's coverage of the Gallup bias is very much the same as Lester Mann's. Why hasn't Gallup been called on to defend this? Why haven't we had Gallup spokespeople being asked why they believe, for example, that black participation will be down several percentage points this time around?

We all, of course, know the answer in its simplest form -- that the SCLM simply doesn't do its job very well. But it's past time for Gallup to have to answer for its methods, and for USAToday and CNN to have to explain why they are not troubled about their association with this clearly slanted poll masquerading as a venerable, objective, shall we say "fair and balanced" source of information.

--stu

Gallup has a semi-defense on their blog, http://www.gallup.com/election2004/BLOG/. To summarize, they say that the 2000 exit poll data is untrustworthy and that 9/11 is going to make Republican turnout higher!

As one of the few political bloggers of color on the web, I have been asserting the fact that minorities are underrepresented in national polling, for some time now.

When blogging or posting on this subject matter. my reasoning is invariably met with silence.

It is no supprisae to me that Gallup dis counts minority voters. I know people all across this country and not one of them has ever had a friend who ever was polled or participated in a poll. Gallup like all the rest of these organizations know where to go to geth the results they want just like I would if I were to conduct a poll. Even in cities where the majority population is African American we seldom get polled and when we do it is for the wrong reason, nothing of a positive nature. African Americans need to ignore these polls and do wht we need to do, VOTE. Most polls are not biased they are usually from conservative organizations and Gallup is no different, control your own destiny do not allow gallup to do it.

Hi, I saw an interesting web log by the Iranian president who declares Kerry as the winner. It is worth seeing. I think he is the first person to call the winner.