When to Hand Off

The question that determines everything

An autonomous agent encounters something unexpected. A function it's modifying has side effects it didn't anticipate. A test fails in a way the planner didn't predict. The codebase has a pattern it hasn't seen before.

What should it do?

If it proceeds, it might waste cycles producing work that needs to be thrown away — or worse, ship something broken. If it escalates, it interrupts a human who may not have the context to help efficiently. If it halts, it blocks the entire pipeline until someone notices.

The handoff decision — when an agent should stop working autonomously and transfer control to a human — is the single most consequential design choice in an agent system. Get it right and the agent is a force multiplier. Get it wrong and it's either a liability (proceeds too aggressively) or a bottleneck (escalates too often).

The spectrum: HITL to HOTL to autonomous

The industry is moving through three stages of human involvement.

Human-in-the-loop (HITL) means a human approves every significant action. The agent proposes; the human decides. This is safe but doesn't scale. If your agent produces 40 pull requests a day and each needs manual approval, you've replaced "writing code" with "reviewing AI code" — which may not be faster.

Human-on-the-loop (HOTL) means the agent operates autonomously by default and the human intervenes when something goes wrong. The agent monitors its own confidence and escalates when it drops below a threshold. The human watches a dashboard or receives notifications rather than approving each action.

Fully autonomous means the agent operates without human oversight for extended periods. The governance layer (Arbiter rules, cost caps, approval gates) provides the guardrails that a human would otherwise provide. The human reviews results periodically rather than monitoring in real time.

Most production systems today operate in the HOTL zone — and the design challenge is making the "on-the-loop" part work without overwhelming the human with false alarms or letting real problems slip through.

Confidence as the handoff signal

The bridge between agent autonomy and human oversight is a confidence score. When the agent knows it's likely right, it proceeds. When it knows it's uncertain, it escalates.

The problem is that confidence scores from language models are poorly calibrated. A model that says it's 90% confident is not necessarily right 90% of the time. Models are trained to be helpful and fluent, which creates a systematic bias toward expressing certainty even when the underlying generation is speculative.

Effective handoff requires calibrated confidence — scores that actually predict accuracy. The research on this is advancing: Holistic Trajectory Calibration (HTC) extracts 48 features from an agent's entire execution path to produce calibrated predictions. Multi-agent deliberation uses disagreement between agents as an uncertainty signal. Process-level monitoring watches for compounding errors across multi-step tasks.

None of these are plug-and-play. All of them are better than the default, which is no calibration — meaning the agent either escalates everything (HITL) or escalates nothing (autonomous with fingers crossed).

The multi-tier model

A tiered escalation framework maps confidence levels to different behaviors:

Tier 1: Autonomous (confidence > 90%). The agent commits code, merges PRs, publishes research briefs, and moves to the next task. No human involvement. The governance layer provides the guardrails — cost caps, approval gates for sensitive operations, structural verification of output.

Tier 2: Proceed with flag (confidence 70-90%). The agent completes the work but marks it for review. The PR is created but not auto-merged. The research brief is saved but tagged for editorial review. The human reviews it when convenient, not on the agent's timeline.

Tier 3: Escalate with context (confidence 50-70%). The agent stops, packages what it knows (the plan, what it tried, what went wrong, what it's uncertain about), and sends that package to a human. The human gets actionable context, not a raw error message.

Tier 4: Halt (confidence < 50%). The agent doesn't know enough to even suggest a direction. It stops, logs the state, and waits. This is the circuit breaker — the agent admitting it's out of its depth.

The exact thresholds vary by domain. Code generation on a well-understood codebase might use 85% as the autonomous threshold. Database migrations might require 95%. Research synthesis might be autonomous at 70% because the consequence of a mediocre brief is low.

Context preservation: the difference between a good handoff and a bad one

The worst handoff is one where the agent says "I'm stuck" and the human has to start from scratch. The best handoff is one where the agent says "here's what I tried, here's what worked, here's where I got stuck, here's what I think the options are."

Context preservation means:

Full execution history. What steps the agent took, in order, with results. Not just the error — the entire trajectory that led to it. This is where memory (Part 4 of the series) becomes critical for handoff quality.

The specific uncertainty. Not "something went wrong" but "the function signature changed and I'm unsure whether callers in module X need updating because I can't resolve the import path." Specificity lets the human focus instead of investigating from scratch.

Recommended options. Even when the agent isn't confident enough to proceed, it can usually suggest two or three paths. "I could update the callers based on the type signature, or I could skip this file and flag it for manual review, or I could revert to the previous approach." The human picks the path instead of inventing one.

Cost of delay. Is this blocking other work? How long has the agent been stuck? What's the cost of the human not responding quickly? This helps the human prioritize among multiple escalations.

In our system, the governance layer (Arbiter) handles the mechanics of escalation — which actions require approval, which can proceed, which should halt. But the quality of the handoff depends on how much context the agent preserves when it decides to escalate. Memory and perception give the agent the raw material. Governance triggers the handoff. The context package determines whether the human can actually help.

The patterns that work

After running autonomous agents across code generation, research, and documentation, several handoff patterns have proven reliable:

Escalate on novelty, not on difficulty. An agent that encounters unfamiliar code (a module it's never seen, a pattern that doesn't match anything in memory) should escalate faster than an agent that encounters a hard but familiar problem. Difficulty with familiar patterns suggests the task needs more compute. Novelty suggests the agent might be wrong about what it's looking at.

Escalate on structural mismatch. When the agent's plan says "modify 3 files" but structural analysis (Part 1 — perception) shows the change affects 12 files through transitive dependencies, the mismatch between plan and reality is an escalation signal. The agent's model of the task was wrong. Better to surface that than to proceed with a partial change.

Escalate on repeated failure. If the agent tries an approach, it fails, it tries a variation, it fails again — that's not persistence, it's drift. Our governance layer caps retry attempts. After N failures on the same step, the agent packages what it tried and escalates rather than continuing to generate variations.

Don't escalate on routine uncertainty. Not every low-confidence output needs human review. If the agent is 75% confident on a documentation PR and the worst case is an awkward sentence, let it ship. The human can fix it in the next review cycle. Reserve escalation for cases where the agent's uncertainty intersects with real consequence.

The anti-pattern: escalation as avoidance

The most common failure mode in escalation design is making escalation too easy. If the agent can escalate whenever it's unsure — with no cost or friction — it will escalate constantly. The human becomes the agent's crutch, and the autonomy promise evaporates.

Good escalation design includes friction:

Budget escalation. Each escalation "costs" something in the agent's resource allocation. If the agent escalates too often, the governance layer notices and adjusts — either by widening the agent's authority on low-consequence tasks or by flagging the agent's calibration as broken.

Require context. Don't let the agent escalate with just "I'm stuck." Require the execution history, the specific uncertainty, and at least one suggested path forward. This forces the agent to do the diagnostic work before punting.

Track escalation rates. If escalation rate is climbing over time, something is wrong — either the tasks are getting harder, the agent's capabilities degraded, or the confidence calibration drifted. The rate itself is a health metric for the system.

What this looks like in practice

Our research loop handles handoff through the Arbiter's escalation gate. When the local model's confidence on a research topic falls below 0.4, the system escalates to a cloud model (Claude Opus) for deep research. That's a machine-to-machine handoff — not every escalation goes to a human.

When the code generation agent hits a structural mismatch (plan says 3 files, reality says 12), it halts and logs the discrepancy. A human reviews the structural analysis and decides whether to expand the scope or split the task.

When the nanochat research loop generates a follow-up topic that doesn't match the focus areas, the Arbiter denies it. That's not escalation — it's governance. The distinction matters: governance prevents bad actions. Escalation transfers control when the agent can't determine the right action.

The handoff layer is not a single mechanism. It's the interplay between calibrated confidence, governance rules, memory context, and the structural understanding that tells the agent what it's actually looking at. When all four work together, the agent escalates rarely, usefully, and with enough context that the human can respond in minutes rather than hours.

---

Building a system that knows when to ask?

ResonanceWorks works with founders and small teams on agent architecture, governance, and handoff design. Talk to Consulting.

Want a system with governance built in?

Torque Engineering installs performance-tuned private AI with Arbiter escalation gates. Get Early Access.

Exploring human-machine culture?

Entrainment House publishes music, art, and cultural works shaped through human-machine coordination. Enter the House.