Do the Exit Polls Indicate Voter Fraud?
There are two lines of analysis that are typically used to justify the claim that the 2004 election result was somehow stolen by the GOP. The first is various bits and pieces of "evidence"--the precincts in Cuyahoga County, Ohio with more votes than registered voters, the counties in Florida (Baker, Holmes) with huge Bush margins but big Democratic registration advantages, etc.--that supposedly indicate vote tampering. I find this evidence profoundly unconvincing and think Farhad Manjoo and others have it basically right: there's not a lot of there there. Vote tampering does not appear to have happened on the scale necessary to affect this election.
The second line of analysis invokes the now-infamous early releases of the NEP exit poll data, which showed Kerry with a 3 point national lead, solidly ahead in Ohio and also leading in Iowa, Nevada and New Mexico. The reasoning, laid out most clearly in a paper, "The Unexplained Exit Poll Discrepancy", by Steven Freeman of the University of Pennsylvania, is that exit polls are very accurate surveys and highly unlikely to produce the results referred to above by chance if the real world results truly were +3 Bush, etc. Therefore, the reasoning goes, our measurement of the real world (the actual vote counts) must be wrong and the original exit poll results right. Conclusion: there's something very funny going on with this election.
But there is a huge problem with this line of reasoning. The exit polls have always drawn samples that are off the real world results and have always had to be corrected (weighted) to eliminate bias, reflect new turnout patterns and, in the end, just flat-out conform to the election results. This year is no different (though it is possible that the magnitude of these corrections has been greater than normal).
Here is my understanding of how the exit poll samples are weighted, based on what I have been able to ferret out so far. (No doubt, I'm not getting it entirely right, but it's damnably difficult to track down good information about this--exit pollsters have never made much effort to publicly explain and document their methods.)
1. Samples are weighted to correct for oversampling of precincts (for example, exit polls have historically selected minority precincts in some states at higher rates than other precincts) and for non-response bias (exit poll interviewers try to keep track of refusers by sex, race and age).
2. Samples are weighted to correct for changing turnout patterns in the current election, since the sample design is based on past turnout behavior.
3. Samples are, in end, simply weighted to correspond to the actual election results. This is done by first weighting exit poll results in sample precincts to the true precinct results, as they are known, and then weighting the overall sample to the overall election result, once it is known.
At what point are these various weighting procedures performed? That's difficult to say because of the lack of public documentation of exit pollsters' methods. But it appears to be the case that weighting of flavors one and two takes place at least partially during the day (and continuously through the day), while the third flavor naturally has to wait until actual election results start to become available.
So where were we in this extremely complicated weighting process when those first +3 Kerry exit polls hit the CNN website? Who knows? (And exit pollsters have not exactly clarified the issue since).
But it's certainly clear that those data had not yet been weighted (or at least very much) to reflect the actual election outcome (again, part of standard exit poll procedure, not anything peculiar to this year). But how much had they been weighted to reflect the other factors (1. and 2.) mentioned above?
Possibly much of this weighting had already been done. If so, then the rest of the sample correction--that took their data from +3 Kerry to +3 Bush--was done by good old-fashioned weighting to the election outcome. Or perhaps it was some combination of additional weighting for factors 1. and 2. plus weighting to the election outcome.
Who knows? Again, exit pollsters don't seem to be particularly eager to share this information. Nor do they seem particularly eager to clarify how common it has historically been for exit poll samples at that time on election day to be that far off from the actual election result.
The issue of historical comparisons is an important one. Part of what has led to the brouhaha over this year's exit poll is people's lack of knowledge about how exit polls have been conducted in the past.
Consider this. The unweighted--completely unweighted--data from the last four presidential elections before this year are as follows:
1988: Dukakis, 50.3; Bush, 49.7
1992: Clinton, 46; Bush, 33.2
1996: Clinton, 52.2; Dole, 37.5
2000: Gore, 48.5; Bush, 46.2
President Dukakis? Obviously, the unweighted data have always been highly problematic and--interestingly--have always shown a strong Democratic bias. Now these unweighted data from past years do not, admittedly, correspond to where we were in the weighting process on election night this year when the +3 Kerry poll hit the 'net--those data had presumably already been weighted to some extent to correct for factors 1. and 2.--but it is still food for thought.
Of course, it's entirely possible that exit poll samples this year, controlling for similar points in the weighting process, were more "off" than in past years. I can't say at this point and I urge the NEP to make the appropriate historical comparisons available to answer the question. But, even if so, this is hardly evidence of skulduggery in the real world; much more likely it reflects the enormous--and perhaps increasing--difficulties of conducting surveys of this complexity in a rapidly changing country.
Of course, additional inaccuracy in the exit poll samples this year (if true) is not a development completely devoid of implications. It could mean that some of the specific results from the survey are less reliable than in the past. (I personally have my doubts about some of the numbers, like those for Hispanics.) But that's a far cry from assuming an election has been somehow stolen or tainted.
My advice: calm down and concentrate on what's really important--beating them next time.