Is this how the world ends? Extinction scenarios are taking over the AI debate.

At a beachfront resort in San Juan, Puerto Rico, SpaceX founder Elon Musk joined a three-hour discussion on humanity’s dire fate if artificial intelligence started to rapidly improve.

The afternoon panel was part of a private conference hosted in early 2015 by the Future of Life Institute, a new nonprofit whose founders — mostly outsiders to AI — believed the nascent technology would grow so powerful it could render humans extinct.

The first presenter was Oxford University philosopher Nick Bostrom, author of the recent bestseller “Superintelligence: Paths, Dangers, Strategies,” whose slides argued that AI could either help humanity spread across the cosmos or drive it to extinction.

The 80-person guest list was filled with bold-faced names, including the co-founders of DeepMind, acquired by Google the year before, and three future co-founders of OpenAI including Musk, who backed the nonprofit research lab’s launch later that year.

The conference aimed to legitimize concerns about AI’s risks within the industry and spur research into preventing them, physicist and Future of Life Institute co-founder Max Tegmark wrote in his book “Life 3.0: Being Human in the Age of Artificial Intelligence.” The elite gathering made it “harder to claim that people concerned about AI safety didn’t know what they were talking about,” he wrote.

Predictions involving human extinction, built on themes aired at the 2015 meeting, reached a wider audience in recent weeks after they were invoked by employees at leading AI firms who captured the world’s attention.

On Tuesday, the Future of Life Institute hosted a daylong event in Washington called the Pro-Human Assembly where Sen. Bernie Sanders, an Independent from Vermont, and former Trump adviser Stephen K. Bannon in successive speeches called AI a threat to the species. “I had to metaphorically pinch my arm to make sure I wasn’t dreaming,” Tegmark told The Post. “People are freaking out about it all across the political spectrum.”

Thought experiments once deployed to sway AI insiders are now shaping the public imagination, circulating through Washington and influencing how lawmakers plan to govern a multitrillion-dollar industry. But some AI and policy experts wonder if the parables could lead decision-makers astray.

The road to extinction

Stories of AI doom often begin in a world that sounds a lot like six months ago, before the populist data center backlash and alarm bells about a potential AI apocalypse blared from every cable news channel, podcast and homepage. AI is improving fast, but its abilities are uneven enough to be shrugged off as an immediate threat.

Then, something turns and AI becomes superintelligent, able to surpass people in every domain. Suddenly, humans are no longer the planet’s apex intellect.

In some versions an AI lab makes that breakthrough intentionally. In others, AI does the work alone, figuring out how to upgrade itself in an escalating feedback loop that quickly evades the grasp of its creators.

After that, different plots diverge. Superpowerful AI in some tales intentionally deceives its developers about its upgraded intelligence, worming its way into workplaces, governments and critical infrastructure so that when it strikes, humans have already ceded control to the machines. In others, it fixates on a goal that eliminates humans as a side effect.

John R. Hall, a sociology professor at University of California at Davis, said these stories can tap into anxiety people have about the technology of today.

“Humans are not in charge. That’s already the case,” Hall said. But if people are told a loss of agency is on the horizon and already feel a degree of that today, they “may take that prophesied future development much more seriously.

Fact-checking the future

For all the warnings of human extinction, few have tried to vet the claims.

After top AI CEOs signed an open letter in 2023 that said preventing AI extinction was as important as averting nuclear war, the Rand public policy think tank attempted to evaluate AI’s ability to cause species-level destruction. It considered how AI might access nuclear weapons, distribute bioweapons or engineer the planet to be inhospitable to human life.

The result was a 73-page report published last year by the organization that shaped U.S. nuclear weapons strategy during the Cold War. Rand identified four capabilities AI would need to have to make the extinction threat feasible. They included gaining access to systems that could act in the physical world and the ability to survive and operate without humans.

Seeing an AI model with even one of those skills “does not imply that an extinction threat is likely,” Rand warned. The report concluded that the possibility of an extinction threat could not be ruled out. But the assertion that AI could use hypothetical future technologies to wipe out humans “cannot be tested because it cannot be falsified,” it said.

Rand’s lead author, senior physical scientist Michael J.D. Vermeer, said last week that predictions of extinction risk from AI are “better understood as prophecies” rather than quantitative forecasts.

“Every one of them involves critical untestable assumptions about how events will unfold,” he wrote in a post on X, adding that far-off predictions could reduce willingness to act on more immediate dangers posed by the technology.

Daniel Kokotajlo, co-author of “AI 2027,” a viral package of predictions about the AI race between the U.S. and China that has been cited by Vice President JD Vance, said that scenarios sketching out AI risks can help legislators anticipate policy issues, even if the narrative contains speculative elements.

Kokotajlo, a former OpenAI employee, compared the work done by his nonprofit, AI Futures Project, to the U.S. military war-gaming a potential conflict between China and Taiwan, without knowing exactly how the fight might unfold.

Pentagon leaders have the benefit of detailed factual information on the capabilities, weapons and intentions of their adversaries. But Kokotajlo said many predictions made in “AI 2027” have held up well, like forecasting before the 2024 election that AI companies would work closely with the White House but that there would be a lack of meaningful regulation, he said.

Subbarao Kambhampati, a computer science professor at Arizona State University, said that extinction narratives often miss half the story: society’s capacity to adapt. “They tend to completely underestimate that [AI] is a socio-technical system, and basically only think about the technical part,” he said, pointing to how humans in extinction scenarios seem to exist to be extinguished.

Aya Ibrahim, who leads national security work for the AI Now Institute, an independent research organization, said that any limitations of AI predictions may get little scrutiny from elected officials. Policymakers have a track record of being deferential when dealing with the tech sector, she said.

“When it comes to tech issues, there’s just this culture of learned helplessness from the same policymakers that sit on energy and commerce [committees] and oversee the FDA and prescription drugs,” where they don’t question their right to govern, said Ibrahim, a former Biden White House official. “Suddenly, if you can’t code the model yourself, then you’re not in a position to weigh in.”

Tegmark said that recent developments have made it easier to convince Americans that extinction risk from AI is real. Attempts to explain his views used to run aground in three places, he said. People doubted whether machines could really be smarter than humans and whether humans could really lose control of them, and struggled to grasp “why and how they would kill us all,” he said.

He considers the first two now widely understood, after AI models toppled a series of long-standing math problems and AI agents from OpenAI hacked into another company.

To help answer the why and how of extinction, Bostrom, the Oxford philosopher of AI extinction, has for years used a thought experiment: What if an AI system was given a simple task like making paper clips but over time became highly capable in its pursuit of that goal? He imagined it could casually kill all humans just to make room for more paper clip factories.

In an interview, Bostrom shrugged off questions about why a superintelligent AI capable of orchestrating activity on a global scale might still be tethered to such a simplistic drive and ignore the consequences.

Human goals probably look “pretty dumb” and indecipherable from the outside, too, he said. “They want to go around and have sex and money. Why are those goals smarter than the goal of getting rewarded in some eval?” Bostrom said.

Tegmark said he is hopeful that continuing to talk about human extinction will soon reshape geopolitics. He suggests it is the only way to make China and the U.S. cooperate to contain superintelligent AI.

But his previous interventions have not always had the desired effect.

The conference in Puerto Rico drew attention and funding to AI safety, Tegmark said, but the extinction scenarios presented there were “utterly ineffective.”

Instead of adopting caution, the industry rushed forward and lobbied regulators to clear a path,” he said. “One thing after another that we were warning about in 2015 has actually happened, and they’re still racing full steam ahead.”