Defense AI capabilities are moving faster than the design thinking for how operators should relate to them. Most of those trust relationships are being set by engineering defaults rather than by decision.
Every interface teaches the operator how much to trust the system behind it, whether or not anyone intended the lesson. In defense that lesson is still mostly unwritten.
TL;DR
AI systems will be increasingly influencing operational decisions across every phase of military workflows.
Over-trust and under-trust both increase risks of operational failure.
Explainability alone does not produce calibrated trust and may increase automation bias.
Operators often oscillate between blind acceptance and exhaustive skepticism.
Autonomous systems introduce new tensions between trust, oversight, and desire for control.
Commercial technology has already documented many of these human-AI trust failures.
Defense needs to treat trust calibration as a formal operational design requirement.
The undesigned relationship
An operator in the box, at a terminal, with an AI recommendation on screen for lethal action. The system has processed more data from more sources, faster than the operator could in real time. The recommendation is sitting there and the window is closing.
Whether the AI is right is one question. The one I'm interested in is whether the operator's trust in that recommendation, in that moment, against that threat, is calibrated correctly.
Too much trust and the operator defers when they should've intervened. Too little and they override a correct recommendation and throw away the advantage the system was bought to provide. Both are operational failures. Both carry consequences. And for all the momentum behind AI in defense, the question of how operators are supposed to relate to these systems once they're inside a real workflow still comes up less than the capability does.
Over the last several years on command and control platforms, autonomous systems, and other mission-critical interfaces, I've watched the AI conversation accelerate. Nearly every discussion now includes decision aids, cognitive load reduction, sensor integration, autonomous execution, machine-assisted analysis. That AI will be embedded in military workflows isn't speculative anymore. We're barreling toward it.
What's struck me is how late those conversations get to the warfighter's experience of trust. How should uncertainty be communicated? How should confidence be surfaced? When should an operator take the recommendation in front of them, and when should they push on it? What does meaningful oversight look like inside an interface, under pressure, inside a chain of command? Those get treated as downstream implementation details, things to sort out once the capability is demonstrated. I think they're the central user experience problem in modern defense systems.
Miscalibrated trust
The capability investment isn't the problem. AI-assisted systems can process sensor data faster than people, find patterns at scale, and move relevant information around more quickly than any operator working alone. In many cases the money is justified.
The problem is that the relationship between the operator and that capability is being shaped implicitly. Every interface communicates a trust model. If every recommendation looks identical, operators learn that every recommendation carries equal reliability. If override is cumbersome, operators learn not to question the system. If the system fails without explanation, distrust outlives the fix. These aren't philosophical concerns. They're interaction design decisions with operational consequences.
What I've seen in research environments is operators trusting systems to degrees that may or may not match what the system can do, and may or may not match their own training and expertise. That gap can open in either direction.
Overtrust is the familiar one. Aviation spent decades on automation bias, operators leaning on system recommendations and failing to step in when the system was wrong, and the human factors literature is clear that it shows up in experts as well as novices and doesn't train out easily. As AI gets deeper into defense workflows, I'd expect the same pattern.
During research sessions evaluating AI-assisted workflows, I've watched soldiers presented with interfaces that identify a threat, recommend an effector, and ask whether they want to engage. In some cases the response is almost immediate. Yes. Engage. The recommendation itself has become the authority signal.
These interactions are hard to interpret, because research environments are imperfect, imagined scenarios. The soldiers in them are highly trained, highly confident, and motivated to project competence. Many are used to adapting quickly to new systems and procedures. Getting beneath the surface layer of "Yeah, this works" is surprisingly hard, especially when the capability being evaluated is hypothetical or emerging.
Undertrust shows up just as often. Some operators don't trust the system at all. They want every input surfaced before they act. Every confidence level, every classification criterion, every piece of supporting metadata, every explanation of how the recommendation was generated. In a low-pressure environment that sounds responsible. Under the duress of real operations it becomes debilitating, and at some point the speed the system was supposed to provide collapses under the operator's need to validate everything by hand.
Control versus trust
There's a third pattern I've found more interesting than either of those, because it isn't really about trust.
Evaluating autonomous agent workflows, I've often watched operators manually define extremely detailed courses for systems built for high-level tasking. The concept behind those systems is simple. Assign agents, define objectives, set actions, and let the autonomy stack work out the details. Many operators instinctively micromanage every movement anyway. They don't distrust the autonomy, necessarily. They want the system to do exactly what they commanded. Their mental model is set to teleoperation when the system was built for command and observe.
That distinction matters because the interface that supports objective-setting may look nothing like the interface that supports high-touch manual control. Neither behavior is wrong, and both are understandable. Only one lines up with what the system was built to do.
The explainable AI conversation complicates this further. One of the more counterintuitive findings I've read in that research is that explanations themselves can increase automation bias. A clear, confident rationale works as an authority signal, and the more coherent it looks the more likely people are to defer, including when the recommendation is wrong.
DARPA's framing around explainable AI is useful here because it avoids a trap a lot of organizations still fall into. The goal was never complete trust. The goal is calibrated trust, and those aren't the same thing. Adding confidence scores to an interface doesn't design calibrated trust. More transparency doesn't automatically produce better judgment. In some cases it just makes operators more confident in a bad recommendation.
AI inside the OODA loop
Part of why this feels urgent is that AI won't sit adjacent to military decision-making. It's being embedded directly inside it.
The OODA loop has shaped military thinking about decision-making under pressure for decades, and historically systems supported each phase through information hierarchy, context, structured decision pathways, and transparent execution. AI is now entering every phase at once. Sensors filtering and prioritizing algorithmically. Analysis layers weighting incoming data through machine learning. Recommendation engines shaping decisions. All of it marching toward autonomous systems executing directly.
The loop hasn't changed. What the operator does inside each phase will.
At Observe, operators will need to know what's been filtered or prioritized by the system and what's still raw input. At Orient, they need to understand how machine-assisted analysis should sit next to their own expertise and the situational awareness the system can't see. At Decide, the interface has to avoid creating automation bias at exactly the phase where critical thinking matters most. This is where cognitive forcing functions earn their place. I've taken to calling them trigger guards. They're deliberate interface interventions that interrupt repetitive deference long enough to require engaged judgment before action, and there's research showing they reduce overreliance on wrong AI recommendations, at some cost to speed and to how much operators like using them.
At Act the challenge changes again. Once autonomous systems are maneuvering, sensing, coordinating, or executing on their own, the operator isn't deciding whether to follow a recommendation anymore. They're deciding whether they understand the system well enough to intervene if they have to. That's a different design problem from the one most current conversations acknowledge.
Commercial lessons with defense stakes
None of these patterns are theoretical. Commercial technology has spent years documenting them at scale. Users overtrust systems and make poor decisions. Users undertrust systems and abandon them. Users lose confidence after one unexplained failure and often never fully recover it. These are known patterns across consumer software, autonomous vehicles, medical technology, and aviation interfaces. Defense isn't facing a different human problem. The stakes are higher.
There's also an expectation gap already open. The next generation of warfighters arrives with a lifetime of commercial technology behind them. They've grown up with systems that personalize, that support fluid interaction, that fail gracefully, and that increasingly explain themselves. Military systems still feel fragmented, rigid, workflow-heavy, and old. I've heard Marines call legacy equipment "green gear," shorthand for technology generations behind everything else in their lives.
That mismatch matters because operators form trust relationships with systems fast. Early interactions set mental models that can persist for years, accurate or not.
Designing trust on purpose
The methods aren't new. Human factors engineering, usability testing, cognitive task analysis, iterative prototyping, contextual inquiry. Commercial technology, aviation, and medicine have all shown what it looks like to design for calibrated trust rather than assume it arrives with capability. Defense doesn't have to invent any of it. What it does have to do is treat trust calibration as a requirement, something that gets written down and tested, rather than something that emerges from whatever the engineering team defaulted to.
Because the design decisions are already being made. Every interface that ships makes them. The question is only who's making them, and whether that person is thinking about the operator at all.
The operator I started with wants to trust the system. They've got a window closing and a recommendation in front of them and no appetite for a seminar on automation bias. Whether that trust has been earned or merely assumed is being decided right now, often as a side effect of other decisions. I'd rather it be decided on purpose.
Further Reading
2026 Defense Industry Outlook — COTS Journal / IDC AI Spending Projection
User Experience Design for Defense Systems with AI — SpringerLink
Military UX and Product Design for Defense Systems — Visual Logic
UX/UI Design in Human-Machine Interfaces Within Defense — Merkur Design
From Swarms to Digital Twins: AI’s Future in Defense — Military Embedded Systems