AI Hallucinations Could Be The Downfall Of The Pentagon

The Pentagon is moving to revolutionize its wargaming capabilities with artificial intelligence, opening vulnerabilities to what is being dubbed “AI hallucinations.”

Code Metal, a Boston-based AI firm, received an $80 million contract to implement AI into WarMatrix, the Department of War’s wargaming program, Fortune reported on Friday. Experts warned the Daily Caller News Foundation that implementing AI into military technologies such as wargaming could make the U.S. military a victim of “AI hallucinations,” or falsely generated information.

“Hallucinations are associated with large language models [such as ChatGPT] and refer to outputs that seem plausible, or coherent, but are factually incorrect, inconsistent or outright non-sensical,” professor of political theory at Queen Mary University London Elke Schwarz told the DCNF. “Hallucinations are inevitable, they can perhaps be minimised but never be eradicated.”

As AI hallucinations are “inevitable,” the use of this technology in military applications could have serious consequences. Some analysts have speculated that AI could have been involved in the bombing of an Iranian girls’ school at the onset of the Iran War, which killed 168 children, mostly girls, according to Iranian officials cited by Reuters.

The strike was still under investigation as of March 11, but the U.S. does not deliberately target civilians, Defense Secretary Pete Hegseth told CBS News’ “60 Minutes” in an interview that aired March 8.

There are no updates to the investigation of the strike at this time, a defense official told the DCNF.

WarMatrix has been under development since at least 2025, according to Air Force records. WarMatrix incorporates military modeling and simulation infrastructure that predates the recent generative AI boom, with the Air Force now using artificial intelligence to accelerate and expand those existing capabilities.

Air Force Public Affairs did not respond to a request for comment. The Pentagon did not provide comment in time for publication.

“Robust testing and evaluation of AI systems is essential to ensure their effective use by the Pentagon. There is great potential to use these tools to improve wargaming and simulation, modernizing the tools to help enhance military planning,” Michael Horowitz, director of Perry World House and Richard Perry Professor at the University of Pennsylvania, told the DCNF. “Testing and continued checking for errors will be essential, like with the use of any tool, to ensure that outputs from AI systems improve, rather than detract from, military decision-making.”

The contracting process took less than a year, a pace Code Metal CEO Peter Morales said was unusually fast for the Pentagon, Fortune reported.

“I have never seen them move this fast, and that’s been really exciting,” Fortune reported, citing Morales.

Code Metal did not respond to a request for comment.

“The potential consequences for security and war are significant. In a time critical environment, there may not always be time to double check all outputs, especially when a system has been employed in order to accelerate the decision speed,” Schwarz told the DCNF. “Now imagine if there is a chance that an AI system has analysed a set of defence related documents and about 1% of the output is made up – that is the system will suggest something that does not exists (maybe the existence of a particular object or the likely intent of the enemy forces).”

In this case, the “particular object” may have been the misinterpretation of a girls’ school as a valid military target, resulting in the deaths of 168 children.

U.S. forces were “leveraging a variety of advanced AI tools” during Operation Epic Fury, U.S. Central Command commander Adm. Brad Cooper said in a March 11 video update posted on X.

“As long as the users of AI systems in wargaming and simulations environments are aware of the limitations of the systems, such as the potential for hallucination, and looking for those limits, they should be able to employ the tools effectively to capture the benefits,” Horowitz told the DCNF. “If users fall prey to automation bias and believe AI model outputs without question, even given the possibility for hallucination, that would be problematic and degrade the effectiveness of the tools.”

In wargaming or warfare, “problematic” hallucinations could directly result in the deaths of innocent civilians.

“Especially when comprehensive and relevant data for any conflicts is hard to come by this potential risk is amplified and you may find yourself in an environment where you rely on a system for information but can’t trust the system sufficiently to give you accurate info,” Schwarz told the DCNF. “Any AI system has the potential to present information that is a distortion of reality … Unless military planners are aware of how to assess the limitations with AI, there is a risk that assumptions and conclusions will become distorted and bad decisions are made.”