The AI Apocalypse

An artificial intelligence researcher made headlines across the world after announcing his resignation from Anthropic, claiming both his former company and their rival, OpenAI, are not acting responsibly with the rapid development of AI, sharing that many in the AI community genuinely believe AI will kill us all within the decade.

Jacob Coxon, who worked with OpenAI and Anthropic, created a media frenzy earlier this month after announcing his resignation from Anthropic in a social media post on X. His comments quickly went viral, with more than 173+ million views. 

In his post, Coxon emphasized that OpenAI and Anthropic “are racing straight to self-improving intelligence and gambling with our lives.” 

According to Coxon, many of his former colleagues and executives sugarcoat what they say to the press, and in reality they share the same fear. 

Yet how does Coxon believe that AI will actually kill us?

Coxon sat down with CBS correspondent Jo Ling Kent for an exclusive interview about his decision to resign. When Kent asked how he thinks AI will kill us all, he began describing a dystopian world that he thinks is very similar to a sci-fi movie, even drawing a comparison to The Terminator. 

Coxon says that advanced intelligence models would be smart enough to kill humans, warning that unregulated AI could copy its code across the internet, with the inability for humans to unplug or shut it off. 

He said, “It could convince or blackmail or persuade humans into buying more computing software to copy it.” Explaining how AI can take control of all software with the ability to regulate human life and deny requests, leading to full control. 

Evan Hubinger, a team lead at Anthropic, made a tweet agreeing with Coxon’s concerns about the rapid development of AI. 

Hubinger said, “Jacob is correct here- we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Coxon is calling for legislation to help pace the rapid growth of the AI industry. Coxon would like to see an agreement between labs like OpenAI and Anthropic with third-party safety evaluation before being released to the public. 

Coxon continues to explain how the risks are well understood at his former company, but they are locked in a “race”. Referring to an international artificial intelligence race against other countries, specifically China, with companies competing to release the next best AI model.

Coxon’s pleas are reflected across the industry. Many employees within the major AI companies, including Coxon, signed Pacing the Frontier, highlighting employees’ concerns as AI develops beyond human intelligence and the risk of not having the ability to understand and control future systems.  

According to Pacing the Frontier’s mission statement, “each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.”

They said, “We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development” 

More than 1,386 employees of frontier AI companies signed the document, emphasizing the importance of developing security measures and strengthening oversight on the AI industry. These signatures included John Schulman (Chief Scientist, Thinking Machines), Jakub Pacjoki (Chief Scientist, OpenAI), Jared Kaplan (Co-Founder and Chief Science Officer, Anthropic), Shengjia Zhao (Chief Scientist, Meta AI), and many more. 

On the Pacing the Frontier website, many wrote statements that reflect Coxon’s similar warnings and pleas regarding the need to regulate AI. 

Chief scientist and Meta AI Shengjia Zhao said, “AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a positive future, we need to develop AI in a way that is driven by responsibility and thoughtfulness.”

Leo Gao, member of Technical Staff for OpenAI, said, “The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race.”

The common consensus across these comments is that there must be regulation and coordination to ensure that AI will continue to be a positive tool for humanity, not one that will destroy us. 

After Coxon’s post, a blinding spotlight fell on Anthropic. Dario Amodei, CEO of Anthropic,  also sat with Jo Ling Kent to discuss the concerns about AI development. 

According to Amodei, the dilemma that plagues the AI community is how to pace and slow down when China will not. According to Amodei, working together internationally to pace the rate of AI will be extremely difficult and virtually impossible. 

Amodei said, “Because the incentives to pull ahead and the military advantage that you would get from that are so large, and honestly, I don’t know if it’s possible, but we should try”

A surprising moment in the interview is when Amodei agrees that it’s “strange” that this highly advanced technology is being built by private companies, not the government, and he is uncomfortable with it. 

Kent asks Amodei if he would be willing to give up his company’s technology to the government, and according to Amodei, he will give up the technology to the right combination of governments.

The Anthropic website openly talks about the rapid advancements of AI models and the company’s hopes of continuing conversations about AI safety and progress in the future.

The official Anthropic website wrote, “ In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.”