Artificial intelligence is supposed to help the U.S. military process information faster. Earlier this year, however, an AI-assisted intelligence report nearly pushed the United States toward a potentially explosive confrontation with China.

According to CNN reporting by senior national security reporter Zachary Cohen and correspondent Katie Bo Lillis, an intelligence report circulated throughout the U.S. military during the war with Iran. The report claimed that a Chinese vessel in the Middle East was carrying components connected to a nuclear weapons program.

The report was taken seriously enough that the U.S. military prepared to intercept the vessel. Armed personnel were preparing to board, while military aircraft were already airborne.

Then, just before the operation was set to commence, officials took a closer look at the intelligence.

As Cohen reported on CNN, officials discovered that the report had been generated with help from artificial intelligence and that a chatbot used by the intelligence analyst had misidentified what the Chinese ship was actually carrying. The exact cargo involved has not been publicly identified.

Cohen warned that a U.S. military operation against a Chinese vessel could have created a direct confrontation between the two countries and potentially spiraled into armed conflict.

That should be alarming.

The problem is not simply that an AI system made a mistake. AI systems can produce inaccurate information. The more serious issue is that inaccurate information apparently made its way into an intelligence report, was disseminated throughout the military and was acted upon before someone stopped to question its reliability.

According to the CNN reporting, the analyst had used a chatbot to examine intelligence concerning the ship’s manifest. The AI system combined information and reached an incorrect conclusion about the cargo. The analyst then used AI again to help package the findings into a standard intelligence report that was ultimately distributed to military officials.

That raises a fundamental question about the use of artificial intelligence in national security: Who is responsible for verifying the information before it becomes the basis for action?

CNN News Central also discussed the incident with Cal Newport, a professor of computer science at Georgetown University, and Josh Rogin, an intelligence reporter for The Washington Post. One of the two guests expressed concern that officials had misunderstood what these AI tools could and could not do.

There is a major difference between using AI to help organize information and allowing an AI-generated conclusion to become part of an intelligence assessment that could trigger military action. The faster technology allows information to move through the system, the more essential human verification becomes.

Cohen also reported that the episode comes as the Pentagon is pushing to expand the use of AI throughout the military, including in areas such as targeting. One source raised concerns about the lack of guidance for how having a human involved in this process would prevent serious mistakes such as civilian casualties or fratricide.

The Chinese vessel incident therefore raises questions that go beyond one bad intelligence report.

AI may be capable of helping analysts process enormous amounts of information. But processing information quickly is not the same thing as determining whether that information is true.

In this case, officials eventually caught the mistake. The operation did not proceed based on the false intelligence.

But imagine if the error had not been discovered until after armed personnel had boarded the vessel.

The consequences could have been dramatically different.

The lesson should not be that the military can never use artificial intelligence. It should be that the higher the consequences of a mistake, the more important independent verification becomes.

AI can help people get to an answer faster.

It cannot determine whether that answer is true simply because it produced it quickly.

As one source put it: “AI allows you to get to a bad idea faster.” In situations like this, a bad idea is not simply a bad idea. It can have catastrophic consequences.