Ranking the Polls

[Bradley in DC]

http://race42008.com/2007/12/27/repost-by-request-ranking-the-polls/

Repost By Request: Ranking the Polls
By request, here is a repost of my post from October 1st that ranked the polls in 2004. Note: I had left LA Times Bloomberg off this list because it was dead last, in 11th place.
Not to completely beat this subject to death, but I found this ranking of the accuracy of the polls in 2004 from DJ Drummond on Polipundit. I thought it gives a little overview of my arguments last night, and gives some thoughts on the other polling firms. Here is their rankings:
CRAP POLLS
#10 - Fox News Poll: Fox News made predictions in 6 states, all in Battleground States. The FNP made 3 calls right, and 3 wrong, and was off by an average of 8.33 points. None of their picks was the closest in a state, and one of their final polls was off by more than 10 points.

#9 - Quinnipiac University Poll: Quinnipiac made predictions in 5 states, and in 2 Battleground States. The QUP made 4 calls right, and 1 call wrong, and was off by an average of 7.00 points. None of their picks was the closest in a state, and one of their final polls was off by more than 10 points. In the Battleground States, the QUP made 1 call right, 1 call wrong, and was off by an average of 5.50 points.

#8 - American Research Group: ARG made predictions in all 50 states, and in all 14 Battleground States. ARG made 45 calls right, and 5 calls wrong, and was off by an average of 6.84 points. Two of their picks were the closest in a states (not counting those states where ARG was the only major poll to make a pick), and 7 of their final polls were off by more than 10 points. In the Battleground States, ARG made 9 calls right, and 5 calls wrong, and was off by an average of 5.50 points. 1 Battleground call by ARG was the closest of the major polls, and 1 was off by more than 10 points.

#7 - Strategic Vision: SV made predictions in 11 states, and in 7 Battleground States. SV got 7 calls right, and 4 wrong, and was off by an average of 6.27 points. None of their picks was the closest in a state, and one of their final polls was off by more than 10 points. In the Battleground States, SV got 4 calls right, and 3 wrong, and was off by an average of 5.43 points.

#6 - Zogby: Mr. “John Kerry Will Win” made predictions in 20 states, and in 13 Battleground States. Zogby got 16 calls right, and 4 wrong, and was off by an average of 6.10 points. Two of his final polls were the closest major poll, and another one of his final polls was off by more than 10 points. In the Battleground States, Zogby got 9 right, and 4 wrong, and was off by an average of 4.92 points.

MORE RELIABLE POLLS

#5 - Mason-Dixon: M-D made predictions in 24 states, and in 13 Battleground States. M-D got 23 calls right, and 1 wrong, and was off by an average of 5.75 points. None of their calls was the closest, but none of their polls was invalidated for being more than 10 points off. In the Battleground States, M-D got 12 right, 1 wrong, and was off by an average of 5.62 points.

#4 - CNN/USA Today/Gallup: CUG made predictions in 15 states, and in 12 Battleground States. CUG got 11 calls right and 4 wrong, and was off by an average of 5.33 points. Two of their final polls was the closest for that state (both in Battleground States), and none of their polls were invalidated for being more than 10 points off. In the Battleground States, CUG got 8 right and 4 wrong, and was off by an average of 5.33 points.

#3 - Research 2000: R2K made predictions in 13 states, and in 7 Battleground States. R2K got 12 calls right, and 1 wrong, and was off by an average of 5.15 points. One of their final polls was the closest for that state ( in a Battleground State), and none of their polls were invalidated for being more than 10 points off. In the Battleground States, R2K got 6 right and 1 wrong, and was off by an average of 4.57 points.

#2 - Rasmussen Reports: A close finish, but number two is Rasmussen Reports. RR made predictions in 33 states, and in 13 Battleground States. RR got all their calls correct, without a single miss, and they were off by an average of 5.82 points. What hurt them was their wide variance of accuracy in support. Three of their final polls were the closest for their state, but another 3 of their final polls were off by more than 10 points. In the Battleground States, RR got all 13 right, and they were off by an average of 4.15 points.

#1 - Survey USA: (drum roll, please) Survey USA. SUSA made predictions in 30 states, and in 9 Battleground States. SUSA got 29 right and 1 wrong, and was off by an average of 3.70 points. So, why does SUSA win with 29/30, and beat RR take second with 33/33? It comes down to hitting the bullseye. EIGHTEEN of Survey USA’s final polls were the closest for that state, almost twice as many as every other major poll PUT TOGETHER! Also, none of their polls were invalidated for being more than 10 points off. In the Battleground States, SUSA got 8 right and 1 wrong, and was off by an average of 3.44 points. Three of SUSA’s final polls in Battleground States were the closest for that state, again the best of any poll.

[Bradley in DC]

http://www.pollster.com/blogs/our_poll_of_pollsters_part_i_h.php

December 27, 2007
OUR POLL OF POLLSTERS - PART I: HOLIDAY INTERVIEWS
As we expected poll releases to be a little slower than usual this week, we decided to conduct one of our own. Last week, we invited about a hundred professional pollsters who work for the news media and political campaigns to participate in a brief survey about the public polls in Iowa and New Hampshire. We will be releasing results from our "Poll of Pollsters" over the next few days. Today I want to discuss the very last question we asked, which concerns the same topic I blogged on yesterday: The reliability of polls conducted this week, between Christmas and New Year's Day.
First, some details about the survey. We invited two categories of pollsters to participate, those that work for political campaigns and those that conduct surveys for news media outlets. We selected campaign pollsters that had been listed in a scorecard published by the National Journal's Hotline that had worked on behalf of candidates for Governor, Senate or U.S. Congress in 2006. The media pollsters included those who conduct the well-known national polls that we regularly track and the organizations that had released surveys in Iowa and New Hampshire.

We sent out email invitations to just over 100 individuals, and as of yesterday a total of 46 had completed the entire survey (21 media pollsters and 25 campaign pollsters; of the latter category 14 were Democrats and 11 Republicans). As with any other survey, we pledged to keep respondent identities confidential and report their answers only as aggregated data. The pollsters responded online, using an online survey form prepared by our sponsor/owner, Polimetrix.

We also want to be very clear on one point: This survey is not a "scientific" sampling of any population beyond the 46 pollsters that replied. Thus, there is no "margin of error." The results represent no more and no less than the opinions of 46 individuals that were willing to respond. Since we made it clear that we would ask about the public polls in Iowa and New Hampshire, I assume that many of the non-participants are simply less interested in the topic. As one prominent campaign pollster who emailed his regrets put it, "in this case I feel like an ill-informed respondent." So the 46 we heard from are probably among the most opinionated on this subject.
So let's start with the topic of polling between Christmas and the New Year. Inspired by the quotations from pollsters in this recent Associated Press article, we decided to ask pollsters choose between two statements about the challenge of polling this week. But rather than forcing them to agree entirely with one statement or another, we asked the pollsters to pick a point along a scale to indicate which statement they agreed with more. The scale they used is reproduced below. They could use their mouse to click on any point along the scale (you can click on it to see a full size version):
There has been a lot of talk about the difficulty of polling during final week or so before the Iowa caucuses that includes both Christmas and New Years Day. Some believe that polling in this period will be challenging but can be done reliably. Others believe that polls done in this period will be so unreliable that they should be ignored. What about you?
Please click or drag on the ruler below to indicate which statement you agree with more.

The pollsters' responses spread across the scale as shown in the chart below. Not one signaled total agreement with either statement by moving the slider all the way to the left or all the way to the right of the ruler. They divide almost evenly in terms of which statement they lean to, with 48% placing themselves closer to the notion that polls "can be done reliably" this week, and 50% placing themselves closer to the statement that polls this week are "so unreliable they should be ignored" (note: one pollster/respondent clicked "not sure," three skipped the question completely).

However, the skeptics hold stronger opinions than the believers. Compare the two bars at the far left of the chart above (representing 17% who have the most doubts about the reliability of Christmas week polling) to the two bars at the right (5% who have the last doubts). For purposes of tabulation, we have translated the hash-marks along the ruler into a 100-point-scale, where 1 is "should be ignored" and 100 is "can be done reliably." The mean score for the full sample is 48 and the median 45, indicating a modest skew toward "should be ignored" (for the truly wonky, each of the bars in the chart represents a decile, 1-10, 11-20, 21-30 and so on).
As the second chart (below) indicates, the media pollsters are more skeptical than the campaign pollsters. The average score for media pollsters is a 43 (with a median of 29). In other words, the media pollsters cluster near the "so unreliable they should be ignored" end of the scale, while the campaign pollsters divide more evenly, with an average score of 50 and a median of 48. [For those interested in more details, click here for a spreadsheet with the number of responses for each increment clicked on the 1-100 scale].

Think of it this way: Even professional pollsters are not sure what to make of the surveys done this week. While few advise ignoring these surveys altogether, most recommend treating the results with a lot more caution than usual. Their uncertainty probably reflects the overall lack of experience our profession has with surveys conducted during the latter half of December. Just about everyone sampled recommends treating the results from this week with a lot more caution than those available at other times, although few would advise that we ignore these results completely.
We will have more results from our "Poll of Pollsters" over the next few days.

[dircha]

Very informative on following up on the past performance of polls. Thanks.