Analysis: Iowa caucus app is latest example of politicos building faulty tech with disastrous results

The Iowa caucus debacle is just the latest example of politicos building faulty technology - with serious political consequences.

Professional technologists shuddered at the apparent incompetence and hubris as details emerged Tuesday about the Iowa Democratic Party rushing a contract for its caucus results app with a little-known tech company founded by veterans of Hillary Clinton's campaign, pushing the app out faster than it could be responsibly built and rejecting opportunities for testing and security vetting.

The resulting implosion of the app from Shadow Inc. on the first vote of 2020 undermined faith in the electoral process just as Democratic Party leaders were trying to restore it.

"The most important lesson anyone should take away from this is that if you're going to use a new technology that you need to very rigorously test it and exercise it and plan for what your backup will be if it fails," Eric Rosenbach, a former top Pentagon official who leads the Defending Digital Democracy program at Harvard University, told me.

"That didn't happen here, which is disappointing," he said. "That's not something that's good for democracy."

The high-profile coding error, which produced inconsistencies in reported caucus results, was reminiscent of other times when the government or campaigns built digital tools - but the hard work of getting the tech right took a back seat to other priorities.

In 2013, health insurance seekers could barely use the Obama administration's online exchange Healthcare.gov when it was first released, dealing a serious blow to the former president's main domestic policy achievement. It prompted jabs from Republican critics that a government that couldn't build a website couldn't possibly revamp health care that some echoed this week.

It took two months and the intervention of a tech "tiger team" before the site was functional.

And on Election Day 2012, the Mitt Romney campaign outfitted thousands of volunteers with a custom app called Orca that was designed to share real-time analytics from polling places and get-out-the-vote efforts in swing states. It was supposed to mark a revolution in campaign technology and a Republican counterstrike to the 2008 Obama campaign's Project Houdini program, which helped deliver one of the most data-driven Election Day operations in history. Instead, the app failed to load for many volunteers and crashed repeatedly for others, leaving many volunteers with nothing to do.

The flops came as top leaders tried to meet political timelines leading to a major failure on the national stage. The Shadow app seems to have been built in an even hastier fashion.

The Iowa Democratic Party only opted to build an app for reporting caucus results late last year after the national party raised security concerns about reporting results by phone, the New York Times's Matthew Rosenberg, Nick Corasaniti, Sheera Frenkel and Nicole Perlroth reported. Shadow, which they reported won the contract over other multiple bidders, built the app in just two months, which is almost certainly far less time than major tech companies would take.

That rushed approach often spells disaster for tech projects, which need extensive testing to ensure all the major bugs are worked out, Steven VanRoekel, a former Microsoft executive who was the nation's chief information officer during the Healthcare.gov launch told me.

"It seems as if they tried to deploy something last minutes that hadn't been vetted, hadn't run through rigorous testing and that's when stuff breaks down," he said.

The Iowa party was also focused on dozens of other caucus priorities and seemed to be ignoring a lot of warning signs when it came to the app. That echoes the initial Healthcare.gov release when most of the people responsible for it in government had other important jobs and no one was laser focused on ensuring the technology worked, VanRoekel told me.

"If it's not a wake up everyday and do it project for someone, you shouldn't ship it," he said. "You shouldn't put it out there because it's just destined for failure."

On the company's side, Shadow was also hampered by a lack of coding expertise, an unfamiliarity with major tech projects and not enough time to get the app approved by Apple's app store, the Times reported. "Instead, the app had to be downloaded by bypassing a phone's security settings, a complicated process for anyone unfamiliar with the intricacies of mobile operating systems, and especially hard for many of the older, less tech-savvy caucus chairs in Iowa," they write.

The massive delay in the results on such a high-stakes night not only diminished public faith in Democrats' ability to run elections but created a situation ripe for disinformation, tainting the caucus as much as a foreign interference campaign might have.

"The danger of a snafu like this is that it undermines trust in the democratic process and the more often that happens the less likely Americans are to trust that the vote they put in is actually the vote that counts," said Rosenbach, whose group focuses on combating foreign interference in campaigns. "They're more likely to start to believe some of the fringe conspiracy theories that pop up on Twitter or on Facebook."

State election officials, who have spent months or years vetting voting systems and other technology before they're approved for use in general elections, described Iowa's situation as "cringeworthy." State election officials often don't play a role in technology used in caucuses, though they typically supply state-approved machines in states with primaries.

"Before you can make any technological change in elections, you have to have two things. You have to have confidence and you have to have competence," Ohio's Republican Secretary of State Frank LaRose told my Washington Post colleague Neena Satija. "What I saw was kind of a cringeworthy thing for any of us that run elections ... But they put themselves in this scenario by what sounds like a series of bad decisions that they had made."

There were plenty of warnings.

Congressional staff and computer science experts raised doubts for months about whether the app would work - but their questions were rebuffed by Democratic leaders, my Post colleagues Tony Romm, Neena and Drew Harwell reported.

The Department of Homeland Security also offered to test the app's security but was turned down, acting secretary Chad Wolf said on Fox and Friends. Iowa Democratic Party chair Troy Price later said he didn't know about the offer.

The Democratic National Committee's chief security officer, Bob Lord, even urged the state to drop the app but was ignored, sources told the Wall Street Journal's Dustin Volz, Tarini Parti, Alexa Corse and Robert McMillan.

For its part, the party cited security as one reason it sought to avoid the spotlight in the run-up to Iowa, and said the delay in testing until just weeks before the caucuses was intentional. "Iowa Democrats waited to introduce the software to avoid giving potential hackers time to penetrate it, the official said, adding that the party chose to keep the name of the vendor secret on the advice of national cybersecurity consultants," my Post colleagues Isaac Stanley-Becker and Michael Scherer report.

Still, by the day after the debacle, officials were also quick to denounce the app. Democratic officials in Nevada, who'd planned to use it for their own caucus Feb. 22, quickly reversed course. DNC chair Tom Perez declared the app "will not be used in Nevada or anywhere else during the primary election process," adding that "the technology vendor must provide absolute transparent accounting of what went wrong."

Price apologized for the delay when he released partial results yesterday afternoon but also stressed that none of the results were hacked or tampered with. "The underlying data, the raw data is secure," he said. "It was always secure. This was a coding error."

Shadow itself, meanwhile, acknowledged the coding error that prevented caucus leaders from sharing results and issued an apology on Twitter.

The company's chief executive Gerard Niemera, who previously built tech products for field organizers in Clinton's 2016 campaign, also offered a personal apology in a Bloomberg interview. "I'm really disappointed that some of our technology created an issue that made the caucus difficult," he said. "We feel really terrible about that."