Design in the Loop

"Human in the loop" is a governance phrase. It works in a briefing and disappears the moment technical specification and tactical product work begin. What's left behind are unresolved questions about the human-AI experience for the operator in the box.

The phrase says a person stays involved. It says nothing about what that person can perceive, understand, or change, and those are the things an interface has to be designed for.


TL;DR

  • "Human in the loop" isn't DoD policy and never was. Directive 3000.09 requires appropriate human judgment, which is a different standard and a harder one.

  • The oversight problem breaks into comprehension, monitoring, and intervention. Intervention gets attention, monitoring holds at current scale, and comprehension is where I keep seeing the gap.

  • Programs keep asking how one operator commands 2,000 agents. I've watched operators max out around six to eight, and the map view stops being useful long before 2,000.

  • Trust isn't a single dial. It moves by phase of the kill chain, and only two of its three components are things an interface can touch.

  • Three questions replace the phrase. Where autonomy operates in the OODA loop and at what level, what situational awareness the operator needs at that phase, and what the interface has to show to make that awareness possible.


Governance language

Every program briefing I've sat in describes the human as being in the loop. A pilot retains authority for lethal engagement. An air defender can intervene. An operator sets objectives and monitors behavior. The governance framing is consistent. The human is there.

Two months later I'm sitting with the engineering team actually building the system. Deciding what the interface shows. What the operator sees when the agent is making a decision. What happens when the system hits a scenario outside its parameters. In those conversations the phrase "human in the loop" doesn't come up. The phrase disappears because it doesn't mean anything specific enough to design to.

That gap matters. A phrase that shapes policy and satisfies oversight evaporates the moment a team has to decide what information belongs on the screen. The phrase isn't wrong, exactly. It's a starting point that gets mistaken for a specification. It says a human should stay involved in some meaningful way, and everything after that is undefined. What they perceive. What they understand. What they can influence. Who's accountable when the system acts.

DoD Directive 3000.09, the governing policy on autonomous weapons, doesn't require a human in the loop. Those words aren't in it. It requires "appropriate levels of human judgment over the use of force," and Michael Horowitz, who worked in the office that rewrote the directive, has been blunt that the two phrases mean different things. I used to read them as occupying the same space. I don't anymore. "Human in the loop" describes where a person sits. "Appropriate human judgment" describes what a person has to be able to do. That second thing is a design brief, even if it hasn't been written down as one.

The design questions underneath the language are comprehension, monitoring, and intervention. Does the operator understand what the system is doing and why, in terms they can act on and not just observe? Can they track behavior as conditions change? Can they step in before a bad decision turns into an operational or ethical consequence? Governance language doesn't answer any of those.

Intervention is usually where defense teams start. The kill switch. The ability to stop the system. Safety and control get serious attention. Monitoring stays manageable at current scales. Comprehension is where I consistently see the gap, and it's the problem that gets harder as systems get larger, faster, and more autonomous.

In OODA terms, if the autonomous system is observing and orienting faster than the human can follow, then the human's decision and action happen against incomplete understanding. The oversight gap becomes temporal before it becomes ethical. Policy can't close that gap. Interface design, information architecture, and interaction models decide whether supervision stays possible at machine speed.

The scaling problem

Most of the systems I work on right now are relatively small. Six to twelve agents. When I think about what an operator can realistically command and control under real mission pressure, inside a single domain, the number is probably closer to six to eight. Most of these systems are mobile, so the interaction model is still heavily map-driven. Agents, routes, tasks, telemetry, and status, displayed top-down.

DoD stakeholders consistently ask a very different question. Across programs and clients I hear some version of the same prompt. How does a single operator effectively command and control 2,000 autonomous agents?

I don't know where that number came from, but it shows up constantly, usually with a confidence that doesn't match anything I've seen demonstrated.

At 2,000 tracks the map view stops meaning anything. Density overwhelms comprehension. The display stops communicating because no human can process that much simultaneous activity in a way that supports a timely decision. The interaction model that works for eight agents doesn't scale to 2,000, and what replaces it is still an open question.

That uncertainty says something bigger than a missing interface pattern. At the scale many programs are imagining, a human isn't going to stay directly involved with every agent. The level of autonomy required goes up dramatically, and the oversight model itself changes with it. "Human in the loop" was written for a world where operators make decisions about discrete systems. It wasn't written for one where operators supervise thousands of agents executing their own plans at once. The governance framing persists. The cognitive model underneath it has changed.

Whether oversight, as currently imagined, stays cognitively achievable at that scale is the open question.

Automation versus mission autonomy

There's a distinction that gets blurred in most autonomy conversations. Most of the systems I encounter today are closer to automation than autonomy. They aren't generating novel plans or responding to evolving conditions with real agency. They're executing defined procedures. Sensor fusion, deterrent inventory checks, weapon pairing, route execution, rule-based responses. That's automation.

Mission autonomy is different. The system receives goals, constraints, and intent, and works out execution dynamically as conditions change. The distinction matters because the interface for a system executing instructions looks nothing like the interface for a system generating its own plans.

I think about the CCA program often in this context. Most current concepts still revolve around constrained teaming and supervisory control. The future vision programs describe is considerably more autonomous. Operators define intent, constraints, and objectives, and the systems determine execution. That's a different relationship between human and machine, and I don't think it's a more advanced version of today's interaction model.

Today the relationship is still fairly direct. The operator tasks the system, the system executes and reports back, and the cognitive model stays close to supervision. At scale it becomes something closer to orchestration. The operator defines objectives, priorities, constraints, and rules of engagement. Autonomous systems determine execution. The human monitors behavior, adjusts boundaries, reallocates intent, and intervenes when behavior diverges from expectation. They aren't executing actions through the system anymore. They're supervising distributed autonomous behavior across compressed timelines and incomplete information.

I've never liked "puppet mastering" for this. Puppets imply direct physical manipulation, which isn't the relationship emerging in autonomous systems and certainly isn't achievable at scale. Conducting is closer. A conductor doesn't play every instrument. The conductor sets coordination, timing, intent, and cohesion across systems playing at the same time.

That shift from direct control toward orchestration is real, and it needs a different interface model than the tasking-and-reporting systems most defense programs still rely on. The hard part is that these interfaces land in environments that are already cognitively saturated. A fighter pilot is already managing sensor fusion, tactical coordination, navigation, communications, electronic warfare, threat interpretation, and weapons under compressed time and extreme pressure. Autonomous wingman supervision doesn't arrive inside a clean workflow. It arrives inside one of the densest operational environments there is.

Trust is contextual

Trust calibration shifts across these workflows too, and not uniformly.

Last year I spent a lot of time in air defense environments, and one thing became clear quickly. Trust isn't a single dial operators turn up or down across the system. It changes with the phase of the workflow.

Trust is low during identification. That's the moment an operator decides whether a track is a legitimate threat. The consequences of getting it wrong are enormous, and in the interfaces I've worked with, system confidence rarely comes through clearly enough for an operator to act without going and looking for more.

Trust is considerably higher during deterrent selection and weapon pairing, where the logic is constrained and procedural. Inventory checks, targeting sequences, availability, rule-based pairing. Operators are far more comfortable deferring to automation there.

So the oversight relationship changes across the kill chain itself. "Human in the loop" applies the same governance framing to all of it. A designed oversight model would have to account for those differences explicitly.

Complexity transfers

When design isn't involved early in defining these systems, the burden moves somewhere else. Usually to training.

Training cycles get longer. Operators are asked to adapt themselves to interfaces that were never built around how they reason, prioritize, or decide under pressure. The assumption is often that autonomy reduces workload. What I've seen is that autonomy reduces execution workload while increasing supervisory workload, which is a different cognitive burden and a much harder one to train around. The complexity doesn't disappear when design is absent. It moves onto the operator and onto the organization preparing them, and operator resilience ends up covering for complexity that didn't have to be there.

There's an honest limitation inside the research itself that I think the industry should acknowledge more openly. Much of what defense teams observe happens during demonstrations rather than in real research conditions. Those aren't the same thing. Demonstrations reveal whether a system functions. They reveal much less about how operators reason under stress, uncertainty, fatigue, and mission pressure. The conditions that expose comprehension failures, trust breakdowns, and cognitive overload are often exactly the conditions design teams never get access to. That doesn't invalidate the work. It does mean the confidence placed in those observations should stay proportional to the environment they came from.

Explainability doesn't have a mature answer yet either. The most useful approaches I've seen are simple. Dashboards and interface layers that expose a handful of operationally meaningful data points about system behavior in real time. But explainability is highly contextual. What an air defender needs to understand about autonomous behavior differs from what a logistics operator or a SOF operator needs, and a universal explainability layer is probably the wrong objective. What seems more realistic is a principled framework for building context-specific explainability across domains, and that's still work to be done.

Designing oversight

The oversight model for autonomous systems is being written right now, in policy documents, acquisition strategies, and program plans, before the design work has tested whether those assumptions hold up cognitively in practice. By the time design teams are brought in, if they're brought in at all, the architecture is often fixed, and the interface becomes an expression of an oversight model rather than a chance to shape it.

The policy debate around autonomy will run for years. The design problem needs answers much sooner than that.

Everyone already agrees a human should stay involved. The question underneath is whether oversight stays cognitively achievable once autonomous systems operate at machine speed and organizational scale, and right now the policy assumes supervision scales along with autonomy. I don't know that it does. The interface is where that assumption is going to get tested, one operator at a time, and I'd rather we tested it on purpose.


Previous
Previous

More Than Panes of Glass

Next
Next

The Inverted Program