How an AI hallucination nearly started a war between the US and China

Armed US personnel were preparing to board a Chinese vessel and military aircraft were already in the air this spring when officials made a startling discovery: the intelligence driving the operation was an AI hallucination.

The aborted operation, which occurred during the war with Iran, came after an intelligence report falsely claimed the Chinese ship was carrying components for a nuclear weapons programme. The crisis, first reported by CNN, was defused only at the last minute when officials took a closer look at how the report was made.

An analyst with the US Special Operations Command had queried an AI chatbot to synthesise open-source data with classified signals intelligence. The chatbot misidentified the vessel’s cargo manifest. The analyst then used the same tool to format the erroneous findings into an official-looking summary, which circulated rapidly up the military chain of command.

"It almost started a war," a source told CNN.

The near-miss highlights a growing worry inside and outside the military: while AI can accelerate decision-making, its errors can propagate faster than humans can verify them. Yet the Pentagon is pushing ahead. Under Defence Secretary Pete Hegseth, the US military is rapidly expanding its use of the technology to select targets, move military assets and manage logistics.

It is not just superpowers struggling to keep the technology under control. New research from the University of Cambridge shows that militant groups in West Africa are using commercial AI chatbots—including OpenAI's ChatGPT, Google's Gemini, and Elon Musk's Grok—to build bombs, plan attacks and solve battlefield problems.

Antonia Juelich, who interviewed 27 former Boko Haram and ISIL-affiliated fighters for the Cambridge study, found that the groups established dedicated AI units. They set up solar-powered rooms with projectors to train fighters on how to bypass safety guardrails.

Fighters used the chatbots as 24-hour technical advisers. Instead of searching for generic bomb-making instructions, they fed the chatbots lists of their limited local materials and asked how to make them more explosive. They also consulted AI on how to recover explosives from unexploded military bombs, and how to adapt commercial drones to drop munitions. In one combat incident, a commander uploaded live footage from a fighter’s chest camera to ChatGPT to get tactical advice during an active battle.

Other weapon designers have managed to go even further. Anthropic recently reported that a Yemen-based weapons engineering cell used its Claude AI model to write control software and run flight simulations for guided rockets. The group used the AI in place of human software engineers to integrate autopilot systems with flight computers. While a test launch of the guided rocket appeared to fail, the incident showed how close militant groups are to utilising the technology for complex engineering.

These discoveries come as tech executives and world leaders prepare to discuss the security implications of advanced AI. Next week, OpenAI chief executive Sam Altman is scheduled to brief the UN Security Council.

While some tech executives, such as Anthropic’s Dario Amodei, have urged a slowdown in the development of the most powerful models, the political appetite for regulation is low. The administration of US President Donald Trump has rejected a voluntary slowdown, arguing that American tech companies must remain free to innovate to maintain an edge over China.