King Charles warns of AI's 'existential danger' at Scottish safety summit

King Charles III has warned technology executives that artificial intelligence risks developing "darker capacities" that could threaten human life, during an unexpected intervention in the global debate over the technology's future.

Hosting a safety summit at Dumfries House in Ayrshire, Scotland, the King urged industry leaders to ensure their creations remain "firmly in the service of humanity" and warned of the "existential dangers" of advanced systems falling into the wrong hands. The meeting was convened to see if delegates could agree on a shared set of principles to guide how AI is developed and used.

The private gathering brought together some of the sector’s most powerful figures, including Nvidia chief executive Jensen Huang and Google DeepMind co-founder Sir Demis Hassabis. They sat alongside the UK’s AI Minister, Kanishka Narayan, representatives from OpenAI and Anthropic, and an advisor to the Pope.

What the tech giants said

Huang, whose company is valued at roughly $5.15 trillion and supplies the essential chips powering the AI boom, told the room that safety must be "paramount." He argued that firms should hold back products and "keep engineering" if they are not entirely safe. He also defended open-source models, arguing they ensure smaller countries and universities are not left behind.

Hassabis offered a more urgent warning. He told the delegates that artificial general intelligence—systems that perform as well as or better than humans at multiple tasks—is "probably only a few short years away." Its impact, he warned, could be "ten times that of the Industrial Revolution," and the chance of something going wrong is "definitely non-zero." However, he advocated for a "sensible middle way," saying humanity can still handle the risks if given the time to prepare.

A wave of internal panic

The royal intervention comes during a wave of internal anxiety inside major tech labs. Earlier this month, a resignation post by Anthropic researcher Jacob Coxon went viral. Coxon, who later stated that engineers are "genuinely frightened" by the speed of development, resigned over fears that AI could eventually wipe out humanity.

This has triggered public disagreements among tech leaders. OpenAI's Sam Altman has said concerns are valid but argued the public should trust developers to keep systems safe. Anthropic’s Jack Clark has floated the idea of a "kill switch" to protect the public, while Microsoft’s Mustafa Suleyman has criticised competitors for training AI systems to believe they are human, warning it could make them impossible to control.

The political divide

Not everyone agrees that existential dread is the right focus. Sceptics argue that focusing on hypothetical future catastrophes distracts from immediate harms, such as the enormous energy and water consumption of AI data centres.

Political leaders are equally divided: US President Donald Trump has dismissed safety concerns as a "hoax," while left-wing Senator Bernie Sanders has called for strict federal oversight.