Autonomy demos sell. Autonomy incidents fire people. The difference is usually not model quality — it is whether irreversible actions required a human gate with a durable pause, a clear denial path, and an audit trail someone can export for a security review. Human-in-the-loop is not a philosophy slide. It is a product surface: who can approve, what they see, how long the agent waits, and what happens when the human says no or disappears for a day. If your agent can move money, mutate production data, or speak as the company, HITL belongs in the MVP — not in “phase two governance.”
Classify tools by blast radius
Start with a blunt inventory. Tools that only read are usually fine behind ordinary auth. Tools that write need a policy. Tools that are irreversible or externally visible need an approval gate by default: payments, refunds, deletes, bulk updates, outbound email or Slack, production deploys, and permission grants. Mark each tool with a needsApproval stance: never, sometimes, or always. “Sometimes” should be rule-based — amount thresholds, environment, customer tier — not a vibe from the last standup. Ambiguous classification is how teams ship agents that are “usually safe” until a Friday evening tool call proves otherwise.
Durable pause beats toast notifications
A modal that vanishes when the tab closes is not an approval system. Agent work must pause in durable state: workflow ID, proposed action payload, rationale, diffs, and expiration. Approvers should open a deep link hours later and still see the exact proposal the model intended to execute. Use the same patterns you would for payments: idempotency keys, signed payloads, and timeouts. If the human ignores the request, the agent should not silently proceed. Define expire behavior explicitly — cancel, escalate, or re-queue with a smaller scope — and test those paths like you test happy-path autonomy.
Denial handling is product design
Teams obsess over approve UX and forget deny. Denial should be first-class: reason codes, optional comments, and a next-step policy such as stop, revise the plan, or ask a different role. Agents that treat denial as an error and retry the same action are hostile software wearing a helpful tone. Also design partial approval: approve the email draft but not the attachment; approve a refund under a cap. Coarse yes/no forces rubber-stamping or perpetual blocks, both of which train humans to stop reading proposals carefully.
Who approves — and who must not
Map approvers to roles, not individuals. Support leads for customer-facing messages. Finance for money movement. Engineering for schema changes and destructive ops. Never let the same service account that proposes an action auto-approve it. Separation of duties is boring and load-bearing. For multi-tenant SaaS, approvals must be tenant-scoped. An admin in Tenant A must never see Tenant B proposals. Log actor, tenant, tool name, argument hash, decision, and latency for every gate so incident response is reconstruction, not folklore.
Ship the incident path with the agent
When something wrong still gets approved, you need rollback playbooks: soft deletes, message recall where possible, payment reverse paths, and customer communication templates. HITL reduces risk; it does not eliminate it. Pair gates with observability — which tools fire most, which approvals stall, which denials cluster by customer or by tool. Founders: put approval gates on the roadmap next to the first tool that can change the world outside the chat pane. Autonomy without a gate is a demo. Autonomy with durable human-in-the-loop is a product customers can trust with real work.
Proposal UX: make humans competent approvers
Approvers need a diff, not a novel. Show what will change, who is affected, the agent’s rationale in plain language, and links to source records. Highlight irreversible consequences in plain sight. If the proposal requires scrolling through raw JSON to understand risk, humans will click approve to make the notification go away. Mobile-friendly approval matters for on-call and executives. A deep link that opens a readable card with Approve / Deny / Need more info covers most real operations. Capture “need more info” as a structured loop back to the agent with a constrained question, not as a free-form chat that loses the original proposal.
Policy as code for when approval is required
Encode thresholds in config your security team can review: refund amounts, environments (prod vs staging), data classifications, and external vs internal audiences. Store policy version on every decision. When an incident happens, you must know which policy was live — not which engineer remembers the last Slack debate. Start strict and loosen with evidence. Teams that start loose and promise to tighten later rarely do. Use production metrics — approval latency, denial reasons, false-positive rate — to justify exceptions, not founder impatience before a launch date.
Training humans and agents together
HITL fails when humans are untrained. Short playbooks beat long wiki pages: when to deny, when to escalate, how to spot prompt-injection flavored proposals. Rotate examples from real (redacted) runs into onboarding for approver roles. On the agent side, train — via prompts and tools — to produce smaller, safer proposals. An agent that asks to “fix the CRM” is un-approvable. An agent that proposes updating three fields on one account with a clear reason is reviewable. Proposal quality is an eval target, not a soft skill.
