AI could pose extinction risk to humanity, experts raise alarm

The rapid development of artificial intelligence has opened up unprecedented technological possibilities while intensifying concerns about its potential risks to humanity.

The rapid development of artificial intelligence has opened up unprecedented technological possibilities while intensifying concerns about its potential risks to humanity.

Former Anthropic researcher Jacob Cookson has resigned from his position, alleging that leading AI companies including Anthropic and OpenAI are taking potentially dangerous risks in the race to develop increasingly powerful AI systems.

Cookson, who has worked as a researcher at both companies, said in a post on X that he resigned because he believed AI companies were “gambling with our lives.” He claimed that some people working in the field seriously believe advanced AI could potentially contribute to the extinction of humanity before the end of this decade.

He warned that the capabilities of future AI systems should not be underestimated. According to Cookson, highly advanced systems could eventually surpass human capabilities conduct cyberattacks, rapidly transform industries and potentially gain access to real-world power and resources.

His post has reportedly received more than 70 million views, bringing renewed attention to the long-running debate in Silicon Valley over whether increasingly advanced AI can be developed safely and kept under human control.

The issue of AI alignment is at the centre of the debate. Alignment refers to developing AI systems that reliably operate according to human values, goals and instructions.

OpenAI chief scientist Jakub Pachocki has also warned that no AI company has yet fully solved alignment and oversight challenges well enough to justify developing the most advanced systems at maximum speed for an extended period.

Another major concern is recursive self-improvement—the possibility that an AI system could eventually design and develop improved versions of itself without direct human intervention. While such capabilities do not currently exist in practical form, researchers have warned that they could make advanced AI systems significantly more difficult to control.

Anthropic AI alignment researcher Evan Hubinger has expressed even greater concern, saying he personally considers the probability of AI causing human extinction within the next decade to be above 10 percent. He acknowledged that Anthropic is working on the problem but said there is currently no complete solution for aligning super intelligent AI.

AI researchers often use the term “p(doom)” to describe estimated probabilities of catastrophic outcomes from advanced AI, including scenarios posing an extreme threat to human civilization.

Governments are also becoming increasingly involved. US lawmakers have introduced proposals aimed at establishing oversight frameworks for advanced AI systems and in some cases temporarily restricting the development of artificial superintelligence until safety regulations are established.

Meanwhile, growing criticism is also being directed at AI data centres because of their enormous computing and infrastructure requirements.

The debate ultimately centres on a difficult question: how can governments and technology companies continue developing AI’s benefits while ensuring that increasingly powerful systems do not create unacceptable risks to humanity?