Big data wars: How technology could tip the mid-term elections

http://www.networkworld.com/article/...ors_picks=true

After John Kerry lost a very winnable election in 2004, Democrats were worried that Republicans had gained an almost insurmountable lead in both technology and data analysis.

“Progressive technology infrastructure was born in 2004, when we got our teeth kicked in,” says Bryan Whitaker, COO of the NGP VAN, a privately held company that offers technology-based services to Democratic candidates.

“Back in 2004, we had no counter to the right’s consistent messaging machine. Fox News, talk radio, Drudge, etc. put out consistent, never-ending messages, and the left didn’t have a viable response to that,” he says. “As we investigated ways to catch up, one thing we realized we should focus on is figuring out how to build up better grassroots efforts. The most persuasive way to influence someone is through person-to-person interactions, but how do you do that effectively, especially in off-year elections?”

MORE ON NETWORK WORLD: Big data's biggest challenges
The answer ended up being technology. The Democratic Party started building databases with detailed voter information, started deploying data analytics tools, and quickly saw the possibilities of social media. Those advantages gave Democrats an edge in the 2012 election, where technology was widely credited with helping President Obama defeat Mitt Romney, particularly when Romney’s big-data-driven poll monitoring network, dubbed Project Orca, crashed and burned on election day.

The right starts to close the gap

After the debacle of 2012, the right has been playing catch-up. Its latest tech effort is Para Bellum Labs, which the Republican National Committee refers to as “a startup company housed in the RNC.” Other Republican-leaning or Republican-sponsored tools include VoterGravity, which leverages mapping software from Esri to create more accurate voter targeting and volunteer walk lists; Data Trust, the right’s Big Data tool; and i360, a rival Big Data platform sponsored by the Koch brothers.

Even with all of those efforts, the right is still behind in terms of technological know-how and savvy. And technologists on the right are often the first to admit this.

Ned Ryun, the CEO of VoterGravity, which bills itself as a center-right data-driven election tech platform, noted that culture is a big part of the problem. “The biggest challenge of the center-right is not talent or technology,” Ryun said. “Our biggest weakness is a culture where important things like data and analysis are not emphasized. As a guy who’s done grassroots campaigns in past and as a tech guy, as well, this worries me.”

Tools like VoterGravity should help close the gap – but only if the Democrats don’t pull too far ahead with other innovations. One of the goals of VoterGravity is to eliminate what Ryun refers to as data loss. In this case, data loss refers not to the kind of loss associated with a security breach, but to all of the information volunteers collect when they interact with voters – and then do nothing with.

“The way it typically works is you have people going door to door, taking down notes in the margins of their walk sheets, but then once they get back to the office, those sheets are just left on a desk for someone else to enter into the database,” Ryun said. The reality is that those notes are only rarely entered into databases, so all of that actionable voter intelligence is simply lost. With tools like VoterGravity, volunteers can enter this information on the fly into their smartphones and tablets.

Democrats have been capturing data like this for years now and are doing everything in their power to extend their tech lead.

The left leads in attracting tech talent

As Northeast Regional Press Secretary of Obama for America, Michael Czin had a front-row seat in 2012, seeing just how important technological innovations could be to an election. Today, Czin serves as the National Press Secretary of the Democratic National Committee, and he is one of the key movers behind the Democrat’s new technology platform, Project Ivy.

Project Ivy focuses on "four tools and strategies,” a voter file and data warehouse, analytics infrastructure, field and marketing tools, and “training and fostering a culture that cultivates further technological innovations.”

There’s that term again, “culture.” Fostering a culture of innovation may well be the most important advantage the left has in the tech wars. Czin argues that even if the right catches up in terms of technology, the technology itself will be “nearly useless unless there is a culture that values inclusion, expanding participation and the ability to use technology to apply those values to all levels of campaigns. Right now Republicans simply lack the technology and the culture to get the job done."

Whitaker of NGP VAN believes that culture has another advantage: the cultivation of talent.

“We [the left] are establishing a pipeline of talent that the right can’t match,” Whitaker says. “We attract technologists and data scientists from MIT, Harvard, Stanford, and all the top schools. We have progressive tech firms based in these cities, everywhere from Boston to D.C. to Oakland, California, and we know how to plug these people into campaigns where they can immediately start making a difference.”

+ ALSO ON NETWORK WORLD 11 big data certifications that will pay off +

Czin agrees with this assessment and also believes that the reason Democrats attract more tech-savvy campaign workers and volunteers is policy. Young tech-savvy people can be put off by the perception that right is anti-gay, anti-immigrant, anti-women, and pro-gun.

Another point Czin makes is how each side regards technology. “Republicans see technology as way to cover up deficiencies in policies or candidates,” he says. “Democrats look at technology much differently. It’s another tool, one of many, not an end in itself.”

What data is analyzed for political gain?
Political campaigns are a ideal place to put big data analytics into action. The data comes in all forms — there's public data on who is registered to vote, there's data on people who voted in recent elections, and the two parties have their own databases on who contributed money or time in the past. Then there's all of the additional data collected by campaign workers as they canvass door-to-door or set up information booths at public events. So, big data allows a campaign to create a database of voters who would be likely to support a particular candidate, who would be likely to hold signs or host campaign events or contribute money. Then campaigns can drop geolocation data on top of that to get an idea of where to send campaign workers in order to hit the most potential voters.

Same with election day efforts to give people rides to the polls or to call people who might not have voted yet. In fact, Mitt Romney's election day system used smartphone technology to allow campaign workers at the polling places to access a database of likely Romney voters at that specific polling place. Romney volunteers could then check off the names of people who voted and people who had not yet voted would be contacted and urged to vote.

On top of that, big data helps a campaign with fundraising. For example, big data analytics allows a campaign to send targeted fundraising emails and then to analyze the effectiveness of each mailing. Is an email or a fundraising letter from Michelle Obama or Joe Biden to a certain demographic more effective than one from President Obama? That's what big data helps a campaign determine.

Tech only takes you so far

Anyone who tracks technology trends understands the pitfall of creating technology for technology’s sake and not focusing on the problem that you’re trying to solve.

“There’s a dangerous perception on the right that if we build this great technology, we’ll win,” Ryun says. “People forget that it’s only a tool.

Another tech challenge for the right, according to Ryun, is demographics. Many Republican volunteers – and voters – are older. Some of them downright fear technology, and if the people you count on for get-out-the-vote efforts won’t or can’t use the technology, what good is it?

Conversely, Czin and others on the left are hoping that technology and more accurate, data-driven voter targeting can help Democrats overcome one of their own biggest weaknesses in midterm elections: low turnout. Democrat-leaning voters tend to turn out in droves for marquee Presidential elections, but they then stay home during the midterms, which has a ripple effect. As Republicans rack up victory after victory at the state and local levels, they can arguably cede the White House for years to come and still have the policy advantage on the ground.

“That’s a huge challenge for us, and we’re hoping we can study the data to learn how to change this dynamic,” Czin said. “We have historic information going back a decade or so in every state, and we’re using that data to figure out how to make informed and educated decisions about whom to spend time talking to, and what we need to do to get our supporters to the polls.”

“One thing we’re trying to do not just through technology, but institutionally, is change voting habits,” Czin adds. “If you voted in a Presidential year, you’re more likely to vote in the next election.”

That, of course, doesn’t mean you actually will vote in the next election, but you’re more likely to, and those are the people Democrats want to target.

A non-partisan platform could help “democratize democracy”

Even if the right is playing catch up in the tech arms race, it doesn’t mean they’ll lag behind indefinitely. Besides investing in their own data analytics, mapping, and targeting platforms, many candidates on the right are using the non-partisan platform from NationBuilder.

While originally developed as platform for political campaigns and non-profits, NationBuilder is now branching out to attract all sorts of organizations, from law firms to university alumni networks to restaurants. This could turn into a huge advantage as, say, marketing techniques coming out of the real estate sector are repurposed for political campaigns. The cross-pollination of tech innovation could lead to all sorts of techniques and tools that we can’t even imagine yet.

NationBuilder is already impacting elections. Two of the best examples come from overseas. During the Scottish election for independence, both supporters and opponents of independence used NationBuilder, and the turnout ended up breaking all U.K. records at 85 percent. (The Liberal Democrats, part of the coalition opposing independence, also used the VAN product from NGP VAN.)

The Scottish Independence vote was a unique election, but the huge turnout points to the most likely impact of tools like NationBuilder: it could well level the tech playing field.

Another, perhaps better example, is how NationBuilder helped a small group of grassroots organizers knock off an entrenched incumbent to elect an independent candidate, Cathy McGowan, to Australia’s Parliament.

The most interesting thing about the McGowan election is that McGowan didn’t drive it. The grassroots movement to replace incumbent Sophie Mirabella emerged first, and only after the movement had already gained momentum did it convince McGowan to run.

The McGowan election serves as a proof of concept for grassroots organizing, showing just how digital tools that simplify everything from fundraising to voter targeting to social media messaging can swing an election if leveraged to their fullest advantage.

Jeff Vance is a Santa Monica-based writer and the founder of Startup50, a site devoted to emerging tech startups. Follow him on Twitter @JWVance, connect with him on LinkedIn, or reach him by email at jeff@sandstormmedia.net.

-t

"By tracking people, the powers that be can make them vote the way they are 'supposed' to vote."

Quote:

---

Originally Posted by DamianTV

"By tracking people, the powers that be can make them vote the way they are 'supposed' to vote."

---

It's a bit more complex than that. There were balls dropped in 2008 and 2012 in the RP campaign, at least on my end, that wouldn't have been such a problem with better voter tech. Just think about ballot access. In some states you have to get the right number of voters on petitions that are eligible voters and that are in the right district. It's difficult and at times painful work. Better tech could have solved the problem as it would have made it easy to know if a voter was in the right district once you got his/her address. And the article is exactly right about data loss volunteers. When doing phone banking it's pretty each to capture data if your phone bank software is up to snuff. But there was nothing like that for door to door, or if there was we didn't have it. This isn't about making people vote your way as much as it is making sure you identify the people likely to vote your way and making sure those people vote. It was no good to have Ron Paul fill stadiums full of young people if those young people didn't show up on election day.

Quote:

---

Originally Posted by jmdrake

It's a bit more complex than that. There were balls dropped in 2008 and 2012 in the RP campaign, at least on my end, that wouldn't have been such a problem with better voter tech. Just think about ballot access. In some states you have to get the right number of voters on petitions that are eligible voters and that are in the right district. It's difficult and at times painful work. Better tech could have solved the problem as it would have made it easy to know if a voter was in the right district once you got his/her address. And the article is exactly right about data loss volunteers. When doing phone banking it's pretty each to capture data if your phone bank software is up to snuff. But there was nothing like that for door to door, or if there was we didn't have it. This isn't about making people vote your way as much as it is making sure you identify the people likely to vote your way and making sure those people vote. It was no good to have Ron Paul fill stadiums full of young people if those young people didn't show up on election day.

---

The deeper subtext in all of this, at least for me, is the fact that any of this is even necessary. Human beings demonstrate at the individual level microcosms of self-contradiction in so many dimensions that it leads me to wonder how it is that most of us get through our days at all. This does not appear to stem from any necessarily willful dishonesty by the Meaner. It is just that we are emotionally complex and that many of those which we carry within us are mutually conflicting and often violently so. This leads us to adopt irrational positions on almost any issue one cares to name: abortion, guns, speech, taxation, gay marriage, foreign policy... and so on.

Were we of a simpler construction, psychologically speaking, the problems we face would be far and away fewer and likely of much lower amplitude. Our greatest strengths as living beings also appear to serve as the potential basis for our ultimate undoing.

It is this intramurally warring complexity that leads us to our irrational behaviors. Imagine that even all of our desires are based in good intentions. We all know where those lead. Without the unbending will to push through the fears upon which our good intentions are based to those fundamental values that can be rationally demonstrated as universally applicable to all men, we are doomed to suffer the unintended and most often tragic and misery-laden consequences those sentiments carry when forced upon people in blanket fashion. It takes not just intelligence, but the guts to face the unpleasant truths to which rational thought often leads, to accept them, and to allow them their place in this world. Thus far, that is the place from which humanity has been fleeing for all its collective feet will carry it, under the false presumption that we are "evolving" away from the barbarities of the past, particularly with the aid of our current technologies.

From what I can see, nothing could be farther from truth because we have deluded ourselves into believing not only in the fundamental validity of that path, but we use those technologies to enforce the false system of beliefs in ways that insult and degrade all men in diametric opposition to the ostensive goal of honoring human dignity. Hence, the rise of the childishly masturbatory political philosophy of the communist and all of its branch versions including progressivism of all forms. The more proper label would be "regressivism" because it drags us centuries to the rear by elevating the will of the new king, AKA "the state" or "the people" or any of the other, similar code words for the tyrant-du-jour, above that of the Individual.

Authoritarian collectivism is naught other than the same old tyranny of the king and his men with new, "modern" faces painted upon it. It is Empire in a new dress.

Paging CPUd, Come in CPUd....

-n

Can big data reveal the mood of the electorate?
http://www.bbc.com/news/technology-32348412

It feels as if every day I get emails from companies with names like TheySay, TalkWalker, and emoSense telling me which party is winning the election based on social media buzz. There is a technical label for what they do: sentiment analysis.
But is it accurate, and what does it really tell us?
"Some of the commercial companies do it brilliantly, some do it terribly," says Carl Miller of the left-leaning think-tank Demos which has set up the Centre for the Analysis of Social Media to examine this booming business.
"It is a way of analysing hundreds of thousands of online conversations that we could never read ourselves but it should never be confused with an opinion poll."

While the nation was glued to its screens for the televised general election debates, Carl and his team at Demos monitored Twitter's "firehose" - the real-time feed of every tweet in the world.
During the clash between the seven main party leaders on 2 April, their algorithm identified 420,000 relevant tweets. They were classified as positive or negative - "cheers" or "boos".
David Cameron, Conservative: 32% cheers v 68% boos
Nigel Farage, UKIP: 40% cheers v 60% boos
Ed Miliband, Labour: 47% cheers v 53% boos
Nick Clegg, Liberal Democrat: 48% cheers v 52% boos
Natalie Bennett, Green: 64% cheers v 36% boos
Leanne Wood, Plaid Cymru: 66% cheers v 34% boos
Nicola Sturgeon, SNP: 83% cheers v 17% boos
The Demos model is based on technology developed by the Text Analytics Group at the University of Sussex.
"Computers are really good pattern recognition machines, and what you're trying to do is get the computer to connect the patterns in the tweets with the categories you are assigning tweets to," explains Dr Jeremy Reffin.

First, a human being chooses the hashtags that are likely to be most relevant.
Then the algorithm is taught how to classify each tweet, using technology called Natural Language Processing. It has to learn how to distinguish between an opinion and a statement of fact.
The computer throws up examples and asks whether it has made the right decision, a process known as assisted machine learning.

The system was honed using data from reality TV shows like X Factor, which are effectively elections that are held every week.
But some of the big challenges in this area became clear when doctoral student Simon Wibberley shows me a spreadsheet listing every tweet from the leaders debate.
One said: "Ad-break. Time for a kitten in a hat. #leadersdebate". But the algorithm classified this as a cheer.
There are other tweets that say one thing but that are classified as the opposite.
"It's slightly unfair to challenge it on a case-by-case basis," argues Mr Wibberley.
He claims the system can make errors on a tweet-by-tweet basis, but it tends to make the right decisions on a larger scale.
The team also has to employ a technique called network analysis to separate out clusters of journalists and political professionals who are tweeting each other.
Yet I cannot escape the feeling that the audience on Twitter is not as balanced as the sample for an opinion poll.
Then there is one particularly British issue.
"Sarcasm," says Dr Reffin. "At this stage computers have a real problem with sarcasm."
The number of Twitter accounts in the UK is dwarfed by the 35 million users of Facebook in Britain.
The social network has published details of the number of interactions - which include likes, comments and shares - for each political party between 1 January and 7 April.
UKIP: 9.7 million interactions
Conservatives: 8.2 million interactions
Labour: 6.6 million interactions
Liberal Democrats: 1.3 million interactions
SNP: 1.3 million interactions
But Facebook's politics specialist Elizabeth Linder warns about over-interpreting the data.

"I think it's difficult… because a lot of people are sharing content that they maybe don't agree with, or they're sharing content because they're saying 'I'm a little bit confused by all of this, what do you all think?'," she says.
"I think instead what we are seeing is the potential to reach people and that they care about politics on Facebook."
She adds that many users may comment publicly on a political party's page but limit their personal views to private conversations with family and friends so the rest of us cannot see them.
Facebook has been able to make some connections between users' likes - such as music and films - and their political views, though.
Like all big data, social scientists would ask whether those are direct relationships or just coincidences.
"It'll be quite some time before [big data] can stand shoulder to shoulder with the social sciences in terms of how rigorous it is," says Carl Miller of Demos.
As a political journalist, I will definitely soak up all this new information, but I will still be reading the polls. And spending too much time reading Twitter.

-t

Election 2016: The big data trail to our next president
Summary:Your next big privacy threat won't come from the NSA or the FBI. The upcoming presidential campaigns will know a staggering amount about you personally. It's a bit scary and more than a little creepy.

http://www.zdnet.com/article/you-bet...tial-politics/

Think your visit to the 2012 ballot box was secret? Think again. The odds are that President Obama's campaign team knows who you (you, specifically) voted for.
You better watch out,
You better not cry,
Better not pout,
I'm telling you why:
Big data is coming to town.

According to Wired, in 2012, "Obama's campaign began the election year confident it knew the name of every one of the 69,456,897 Americans whose votes had put him in the White House."

Let's be clear. This is not about the NSA or the FBI or the CIA. The president's election team wasn't able to push right through your apparent privacy using police state tactics.

Oh, no. They used something far more dangerous and intrusive: Math.

We are all aware that we leave a digital footprint everywhere we go, but the level of detail in that footprint can be breathtaking.

What you buy, what you read, what you share online, who you associate with, what your mood is, where you work, what you do, what your health situation is, where you've donated, what clothing styles you like, what car models you buy, your favorite Cola brand, your favorite phone brand -- all of that information is available to those with the budget to buy it and the algorithms to aggregate and sift through it.

This is where big data is changing the face of American election politics. For most of the 20th century, election politics was often about putting voters into demographic boxes and then reaching out to those boxes.

In other words, you were a soccer mom, an up-and-comer, a broke-and-angry, a comfortable middle-ager, etc. Each of those demographic and psychographic categories was considered enough to help a campaign target resources. All they had to do was count up who fit into what category, overlay that to a region, and they had their plan.

But what happens if you have 18 percent staunch conservatives living in the middle of a broadly liberal community? In the past, those outliers were ignored by retail politics, and the only way they could be reached was through national campaigns, debates, and ads.

In 2008 and 2012, that all began to change. Now, rather than targeting broad groups of voters, savvy campaigns are targeting you, the individual voter. Each individual over 18 years of age has a very complex and very lengthy dossier available to the campaigns willing to embrace data science.

Analysts can predict who you are going to vote for -- often before you, yourself, have made up your mind.

I spoke recently about this with Dave Wakeman of the Wakeman Consulting Group. Dave has some history in this business. He worked on the 2012 campaign, and wrote ads in support of President Obama for the AFL-CIO, AFSCME, and others. He also worked with MoveOn, the International Association of Fire Fighters, and the Maryland Fraternal Order of Police President.

According to Wakeman: "In 2012, the Obama team used data to target specific voters and make many more targeted conversations. One way that they tailored their messages through the use of data was in their fundraising appeals in email ... depending on your profile and the data they had associated with your email and profile, you would receive a different email with a different ask."
Of course, not all was perfect. Wakeman reported, "So if you had more than one email address, you might see asks for different amounts of money, different actions, and different messaging."

Even in the largest campaigns, there is clearly room for improvement.

As we move forward into 2016, big data will take on an even greater level of importance, and we're moving into a world where real-time big data becomes not only possible, but also economically practical.

This is because of the tremendous decline of RAM cost, which makes in-memory computing possible for large data sets. RAM is millions of times faster than hard drives, making analysis tasks that might take weeks to perform using spinning platters generate results within seconds.

But big data in presidential politics isn't about know whether you've been bad or good; it's about resource allocation. And this is where big data becomes game changing.

Think for a minute about the nuts and bolts of a presidential election. Where do you spend your ad dollars? What do you say? What communities do you target? Where do you set up campaign workers? Your headquarters? What doors do you tell volunteers to knock on? What do you tell them to say? What phones do you ring? What do you ask? Who is likely to donate? What is likely to sway big donors?

How much? Where? What? When?

This is where data analytics takes the front seat. The more rapidly you can move your resources, the more responsiveness your campaign can be to shifts in mood and counter-tactics by the competition.

Wakeman spoke about that in relation to the 2012 Romney campaign:

The Romney campaign never really embraced data in a way that allowed them to make smart decisions. A really good example is in polling. Somehow, the internal polling the Romney campaign was using to make decisions was counter to every other publicly available poll and any internal polls that the Democrats had. This made for decisions like Romney campaigning in Pennsylvania on the closing weekend of the campaign.
In other words, the Romney campaign allocated the wrong resources to the wrong state at the wrong time. And lost.

But there is more to consider than just resource allocation. There will be a lot of very sensitive data flowing through the campaigns. I discussed this concern with Joel S Winston, former deputy attorney general in New Jersey. Joel is also former director of Digital Media for a prominent DC political consulting firm, and now chairs the cybersecurity law practice at Winston Law Firm, LLC, in New York City.

He cautioned that, "The theme in 2016 will be privacy of voter data. Campaign databases will contain highly personal and sensitive data on hundreds of millions of American voters."

He warned that a data breach by a presidential campaign would pose significant risk.

Winston made some additional points worthy of concern:

Political campaigns and entities that collect, analyze, and act upon personal voter data have multiple legal responsibilities to protect the data and limit access. Modern campaigns have an enormous task to protect the big data they give to staff, vendors, and campaign volunteers.
He shares some of my concerns about security. "Every presidential campaign with an electoral strategy for victory must also have a rock-solid privacy plan to protect operational and big data files."

Finally, Winston made a simple statement that could be a real wake-up call to campaign operators everywhere: "An epic data breach can sink a modern campaign."

In looking toward 2016, Wakeman made a strong point: "Each side should be using data as a way to encourage meaningful interactions with their voters. The challenge for any campaign is going to come in the form of having access to reams of data and using it in a meaningful way. In campaigns, the challenge is truly finding ways to make the data actionable and not just theoretical."

That challenge applies not just to politics, but all data science. Having the data is one thing. Using it as a force for good is another thing entirely.

They see you when you're sleeping.
They know when you're awake.
They know if you've been bad or good,
So be good for goodness' sake!

PS I am already getting calls from the Hillary Clinton campaign, and it's barely 48 hours after her announcement. Here's a useful data point for anyone on her team: The more you call, the more likely I am to vote for anyone else.

By the way, I'm doing more updates on Twitter and Facebook than ever before. Be sure to follow me on Twitter at @DavidGewirtz and on Facebook at Facebook.com/DavidGewirtz.

-t

How President Obama’s campaign used big data to rally individual voters, Part 1.
http://www.technologyreview.com/feat...voters-part-1/

Too years after Barack Obama’s election as president, Democrats suffered their worst defeat in decades. The congressional majorities that had given Obama his legislative successes, reforming the health-insurance and financial markets, were swept away in the midterm elections; control of the House flipped and the Democrats’ lead in the Senate shrank to an ungovernably slim margin. Pundits struggled to explain the rise of the Tea Party. Voters’ disappointment with the Obama agenda was evident as independents broke right and Democrats stayed home. In 2010, the Democratic National Committee failed its first test of the Obama era: it had not kept the Obama coalition together.

But for Democrats, there was bleak consolation in all this: Dan Wagner had seen it coming. When Wagner was hired as the DNC’s targeting director, in January of 2009, he became responsible for collecting voter information and analyzing it to help the committee approach individual voters by direct mail and phone. But he appreciated that the raw material he was feeding into his statistical models amounted to a series of surveys on voters’ attitudes and preferences. He asked the DNC’s technology department to develop software that could turn that information into tables, and he called the result Survey Manager.

That fall, when a special election was held to fill an open congressional seat in upstate New York, Wagner successfully predicted the final margin within 150 votes—well before Election Day. Months later, pollsters projected that Martha Coakley was certain to win another special election, to fill the Massachusetts Senate seat left empty by the death of Ted Kennedy. But Wagner’s Survey Manager correctly predicted that the Republican Scott Brown was likely to prevail in the strongly Democratic state. “It’s one thing to be right when you’re going to win,” says Jeremy Bird, who served as national deputy director of Organizing for America, the Obama campaign in abeyance, housed at the DNC. “It’s another thing to be right when you’re going to lose.”

It is yet another thing to be right five months before you’re going to lose. As the 2010 midterms approached, Wagner built statistical models for selected Senate races and 74 congressional districts. Starting in June, he began predicting the elections’ outcomes, forecasting the margins of victory with what turned out to be improbable accuracy. But he hadn’t gotten there with traditional polls. He had counted votes one by one. His first clue that the party was in trouble came from thousands of individual survey calls matched to rich statistical profiles in the DNC’s databases. Core Democratic voters were telling the DNC’s callers that they were much less likely to vote than statistical probability suggested. Wagner could also calculate how much the Democrats’ mobilization programs would do to increase turnout among supporters, and in most races he knew it wouldn’t be enough to cover the gap revealing itself in Survey Manager’s tables.

His congressional predictions were off by an average of only 2.5 percent. “That was a proof point for a lot of people who don’t understand the math behind it but understand the value of what that math produces,” says Mitch Stewart, Organizing for America’s director. “Once that first special [election] happened, his word was the gold standard at the DNC.”

The significance of Wagner’s achievement went far beyond his ability to declare winners months before Election Day. His approach amounted to a decisive break with 20th-century tools for tracking public opinion, which revolved around quarantining small samples that could be treated as representative of the whole. Wagner had emerged from a cadre of analysts who thought of voters as individuals and worked to aggregate projections about their opinions and behavior until they revealed a composite picture of everyone. His techniques marked the fulfillment of a new way of thinking, a decade in the making, in which voters were no longer trapped in old political geographies or tethered to traditional demographic categories, such as age or gender, depending on which attributes pollsters asked about or how consumer marketers classified them for commercial purposes. Instead, the electorate could be seen as a collection of individual citizens who could each be measured and assessed on their own terms. Now it was up to a candidate who wanted to lead those people to build a campaign that would interact with them the same way.

After the voters returned Obama to office for a second term, his campaign became celebrated for its use of technology—much of it developed by an unusual team of coders and engineers—that redefined how individuals could use the Web, social media, and smartphones to participate in the political process. A mobile app allowed a canvasser to download and return walk sheets without ever entering a campaign office; a Web platform called Dashboard gamified volunteer activity by ranking the most active supporters; and “targeted sharing” protocols mined an Obama backer’s Facebook network in search of friends the campaign wanted to register, mobilize, or persuade.

But underneath all that were scores describing particular voters: a new political currency that predicted the behavior of individual humans. The campaign didn’t just know who you were; it knew exactly how it could turn you into the type of person it wanted you to be.

The Scores

Four years earlier, Dan Wagner had 
been working at a Chicago economic consultancy, using forecasting skills developed studying econometrics at the University of Chicago, when he fell for Barack Obama and decided he wanted to work on his home-state senator’s 2008 presidential campaign. Wagner, then 24, was soon in Des Moines, handling data entry for the state voter file that guided Obama to his crucial victory in the Iowa caucuses. He bounced from state to state through the long primary calendar, growing familiar with voter data and the ways of using statistical models to intelligently sort the electorate. For the general election, he was named lead targeter for the Great Lakes/Ohio River Valley region, the most intense battleground in the country.

After Obama’s victory, many of his top advisors decamped to Washington to make preparations for governing. Wagner was told to stay behind and serve on a post-election task force that would review a campaign that had looked, to the outside world, technically flawless.

In the 2008 presidential election, Obama’s targeters had assigned every voter in the country a pair of scores based on the probability that the individual would perform two distinct actions that mattered to the campaign: casting a ballot and supporting Obama. These scores were derived from an unprecedented volume of ongoing survey work. For each battleground state every week, the campaign’s call centers conducted 5,000 to 10,000 so-called short-form interviews that quickly gauged a voter’s preferences, and 1,000 interviews in a long-form version that was more like a traditional poll. To derive individual-level predictions, algorithms trawled for patterns between these opinions and the data points the campaign had assembled for every voter—as many as one thousand variables each, drawn from voter registration records, consumer data warehouses, and past campaign contacts.

This innovation was most valued in the field. There, an almost perfect cycle of microtargeting models directed volunteers to scripted conversations with specific voters at the door or over the phone. Each of those interactions produced data that streamed back into Obama’s servers to refine the models pointing volunteers toward the next door worth a knock. The efficiency and scale of that process put the Democrats well ahead when it came to profiling voters. John McCain’s campaign had, in most states, run its statistical model just once, assigning each voter to one of its microtargeting segments in the summer. McCain’s advisors were unable to recalculate the probability that those voters would support their candidate as the dynamics of the race changed. Obama’s scores, on the other hand, adjusted weekly, responding to new events like Sarah Palin’s vice-presidential nomination or the collapse of Lehman Brothers.

Within the campaign, however, the Obama data operations were understood to have shortcomings. As was typical in political information infrastructure, knowledge about people was stored separately from data about the campaign’s interactions with them, mostly because the databases built for those purposes had been developed by different consultants who had no interest in making their systems work together.

But the task force knew the next campaign wasn’t stuck with that situation. Obama would run his final race not as an insurgent against a party establishment, but as the establishment itself. For four years, the task force members knew, their team would control the Democratic Party’s apparatus. Their demands, not the offerings of consultants and vendors, would shape the marketplace. Their report recommended developing a “constituent relationship management system” that would allow staff across the campaign to look up individuals not just as voters or volunteers or donors or website users but as citizens in full. “We realized there was a problem with how our data and infrastructure interacted with the rest of the campaign, and we ought to be able to offer it to all parts of the campaign,” says Chris Wegrzyn, a database applications developer who served on the task force.

Wegrzyn became the DNC’s lead targeting developer and oversaw a series of costly acquisitions, all intended to free the party from the traditional dependence on outside vendors. The committee installed a Siemens Enterprise System phone-dialing unit that could put out 1.2 million calls a day to survey voters’ opinions. Later, party leaders signed off on a $280,000 license to use Vertica software from Hewlett-Packard that allowed their servers to access not only the party’s 180-million-person voter file but all the data about volunteers, donors, and those who had interacted with Obama online.

Many of those who went to Washington after the 2008 election in order to further the president’s political agenda returned to Chicago in the spring of 2011 to work on his reëlection. The chastening losses they had experienced in Washington separated them from those who had known only the ecstasies of 2008. “People who did ’08, but didn’t do ’10, and came back in ’11 or ’12—they had the hardest culture clash,” says Jeremy Bird, who became national field director on the reëlection campaign. But those who went to Washington and returned to Chicago developed a particular appreciation for Wagner’s methods of working with the electorate at an atomic level. It was a way of thinking that perfectly aligned with their *simple theory of what it would take to win the president reëlection: get everyone who had voted for him in 2008 to do it again. At the same time, they knew they would need to succeed at registering and mobilizing new voters, especially in some of the fastest-growing demographic categories, to make up for any 2008 voters who did defect.

Obama’s campaign began the election year confident it knew the name of every one of the 69,456,897 Americans whose votes had put him in the White House. They may have cast those votes by secret ballot, but Obama’s analysts could look at the Democrats’ vote totals in each precinct and identify the people most likely to have backed him. Pundits talked in the abstract about reassembling Obama’s 2008 coalition. But within the campaign, the goal was literal. They would reassemble the coalition, one by one, through personal contacts.

Has a Part 2&3 behind a paywall :(

-t