Galileo Galilei
02-07-2008, 04:32 PM
The Secret Lives of Pollsters
By MARK BLUMENTHAL
Washington
POLITICAL pollsters have had a rocky 2008. In New Hampshire, after a final round of pre-election polls showed Barack Obama leading by an average of nine percentage points, Hillary Clinton defeated Mr. Obama by fewer than three percentage points. In South Carolina, the numbers were worse. Pre-election surveys there showed Mr. Obama leading Mrs. Clinton by an average of 12 points. He defeated her by 29 points.
In some of the primaries and caucuses held on Tuesday, things weren’t much better. The final polls seriously underestimated Mr. Obama’s performance in Alabama and Georgia, and Mike Huckabee’s vote in Georgia and Missouri. In California, two polls conducted over the same two-day period before the election yielded diametrically opposite results: one showed Mrs. Clinton leading by 10 percentage points, while the other reported Mr. Obama up by 13 points. In four other states — Illinois, Massachusetts and Connecticut for the Democrats and Alabama for the Republicans — polls showed large ranges of variation.
As the remaining states prepare to cast their decisive votes in this campaign, how are voters supposed to make sense of all the conflicting data?
Unfortunately, when the differences are as severe as they were in California, we can’t. Despite 22 years of experience as a Democratic pollster, I can only speculate about what might be going wrong.
Why? Because so many pollsters fail to disclose basic facts about their methods. Very few, for instance, describe how they determine likely voters. Did they select voters based on their self-reported history of voting, their knowledge of voting procedures, their professed intent to vote or interest in the campaign? Did they use actual voting history gleaned from official lists of registered voters?
Fewer still report the percentage of eligible adults that their samples of likely voters are supposed to represent. This is a crucial statistic, given the relatively low percentage of eligible adults who participate in party primaries. (In California, for example, turnout surged in 2008 but still amounted to about 30 percent of the state’s eligible adults.)
Incredibly, some organizations routinely report results without any indication of whether a live interviewer or a recorded voice asked the questions.
In California, two pollsters — including the one showing a huge, erroneous lead by Mr. Obama — failed to disclose the demographic characteristics of their samples. Only a handful of pollsters that conduct statewide surveys routinely provide this data, like the percentage of the sample that is male, or African-American, or under 30 or college-educated.
According to the network exit poll in South Carolina, African-Americans were 55 percent of South Carolina’s Democratic electorate, and 78 percent of black Democrats supported Mr. Obama. If pre-election polls in South Carolina sampled a smaller proportion of black voters, they would have underestimated Mr. Obama’s actual support. But only five of the nine pollsters who fielded surveys in South Carolina in January reported the racial makeup of their samples.
The ethical codes of organizations like the American Association for Public Opinion Research mandate the disclosure of these sorts of methodological details upon request. But when I asked all of the public pollsters surveying the Iowa caucuses last fall for their data, five pollsters refused to respond. Others provided answers that were incomplete or severely delayed.
Greater transparency might make some polls more accurate, by making pollsters less willing to cut corners that might otherwise go unnoticed. But the real benefit would be to journalists, political professionals and voters, who would be better equipped to analyze all the polls that confront us.
If pollsters disclosed more about how their polls were conducted, we would be in a better position to know which polls are likely to be right, and which ones can be safely ignored.
Mark Blumenthal is the editor and publisher of Pollster.com and a polling analyst for National Journal.
READ COMMENTS
http://www.nytimes.com/2008/02/07/opinion/07blumenthal.html?_r=1&oref=slogin
By MARK BLUMENTHAL
Washington
POLITICAL pollsters have had a rocky 2008. In New Hampshire, after a final round of pre-election polls showed Barack Obama leading by an average of nine percentage points, Hillary Clinton defeated Mr. Obama by fewer than three percentage points. In South Carolina, the numbers were worse. Pre-election surveys there showed Mr. Obama leading Mrs. Clinton by an average of 12 points. He defeated her by 29 points.
In some of the primaries and caucuses held on Tuesday, things weren’t much better. The final polls seriously underestimated Mr. Obama’s performance in Alabama and Georgia, and Mike Huckabee’s vote in Georgia and Missouri. In California, two polls conducted over the same two-day period before the election yielded diametrically opposite results: one showed Mrs. Clinton leading by 10 percentage points, while the other reported Mr. Obama up by 13 points. In four other states — Illinois, Massachusetts and Connecticut for the Democrats and Alabama for the Republicans — polls showed large ranges of variation.
As the remaining states prepare to cast their decisive votes in this campaign, how are voters supposed to make sense of all the conflicting data?
Unfortunately, when the differences are as severe as they were in California, we can’t. Despite 22 years of experience as a Democratic pollster, I can only speculate about what might be going wrong.
Why? Because so many pollsters fail to disclose basic facts about their methods. Very few, for instance, describe how they determine likely voters. Did they select voters based on their self-reported history of voting, their knowledge of voting procedures, their professed intent to vote or interest in the campaign? Did they use actual voting history gleaned from official lists of registered voters?
Fewer still report the percentage of eligible adults that their samples of likely voters are supposed to represent. This is a crucial statistic, given the relatively low percentage of eligible adults who participate in party primaries. (In California, for example, turnout surged in 2008 but still amounted to about 30 percent of the state’s eligible adults.)
Incredibly, some organizations routinely report results without any indication of whether a live interviewer or a recorded voice asked the questions.
In California, two pollsters — including the one showing a huge, erroneous lead by Mr. Obama — failed to disclose the demographic characteristics of their samples. Only a handful of pollsters that conduct statewide surveys routinely provide this data, like the percentage of the sample that is male, or African-American, or under 30 or college-educated.
According to the network exit poll in South Carolina, African-Americans were 55 percent of South Carolina’s Democratic electorate, and 78 percent of black Democrats supported Mr. Obama. If pre-election polls in South Carolina sampled a smaller proportion of black voters, they would have underestimated Mr. Obama’s actual support. But only five of the nine pollsters who fielded surveys in South Carolina in January reported the racial makeup of their samples.
The ethical codes of organizations like the American Association for Public Opinion Research mandate the disclosure of these sorts of methodological details upon request. But when I asked all of the public pollsters surveying the Iowa caucuses last fall for their data, five pollsters refused to respond. Others provided answers that were incomplete or severely delayed.
Greater transparency might make some polls more accurate, by making pollsters less willing to cut corners that might otherwise go unnoticed. But the real benefit would be to journalists, political professionals and voters, who would be better equipped to analyze all the polls that confront us.
If pollsters disclosed more about how their polls were conducted, we would be in a better position to know which polls are likely to be right, and which ones can be safely ignored.
Mark Blumenthal is the editor and publisher of Pollster.com and a polling analyst for National Journal.
READ COMMENTS
http://www.nytimes.com/2008/02/07/opinion/07blumenthal.html?_r=1&oref=slogin