Human Approval Workflows for AI Infrastructure Agents

Human Approval Workflows for AI Infrastructure Agents

Your read-only agent correctly diagnosed the OOMKill, named the container, and quoted the exit code. Someone on the team asks the obvious next question: can it just bump the memory limit?

Adding patch to its ClusterRole is a one-line change. It is also the wrong change, because the thing standing between a hallucinated fix and a production outage would then be the system prompt — and the system prompt is a request, not a control. That argument is made in full in Give an AI Agent Read-Only Access to Kubernetes, and this post picks up where it ends: the answer is a human approval gate, and the interesting part is designing the gate, not wiring it.

The gate you build first is the one that fails

The instinct is to gate everything. Every tool call stops, a human says yes, the agent proceeds. It feels maximally safe and it decays fast.

An agent investigating a single incident makes ten to thirty tool calls. Most are reads. If each one raises a prompt, the human is answering thirty questions to get one fix, and after two incidents they stop reading and start clicking. Within a week the gate is a formality that produces an audit trail full of approvals nobody looked at.

That is worse than no gate at all, because the approvals are now evidence. When something breaks, the record says a named engineer approved it at 03:14. They didn't. They pattern-matched a dialog box while half asleep. A human who approves 200 requests a day is not a control; they are a latency tax with a signature attached.

The design constraint that follows: the gate should fire rarely enough that each firing is genuinely worth a person's attention. Everything else is downstream of that number.

Tier by reversibility, not by verb

The natural axis is the RBAC verb — get safe, patch risky, delete dangerous. It's the wrong axis, because verbs don't map to consequences. delete pod on a Deployment-owned pod is fully reversible; the ReplicaSet recreates it in seconds. patch on an HPA's minReplicas is not, in any way that matters, because by the time you notice, the traffic is gone.

Sort actions by what it takes to undo this instead:

• Auto-approve — reversible, self-correcting, bounded. Rollout restarts. Scale-ups. Deleting a controller-owned pod. The blast radius of being wrong is a brief capacity blip.
• Gate — destructive, capacity-reducing, or hard to reverse without a second decision. Scale-downs, resource-limit changes, config edits, anything touching a StatefulSet.
• Never — a short list the agent cannot do at all, with no approval path. Namespace deletion, PVC deletion, CRD deletion, RBAC mutations. Not because a human couldn't approve them, but because putting them behind a prompt invites someone to approve them at 3am. Keep this list to a handful of entries; if it grows, you're using it as a substitute for thinking about the middle tier.

Keep the whole policy in one file, as data:

1# policy.py — the only place that decides what needs a human.
2from typing import Literal
3
4Tier = Literal["auto", "gate", "never"]
5
6NEVER = {
7    ("delete_resource", "namespace"),
8    ("delete_resource", "persistentvolumeclaim"),
9    ("delete_resource", "customresourcedefinition"),
10    ("apply_manifest", "clusterrolebinding"),
11}
12
13
14def classify(tool: str, args: dict) -> tuple[Tier, str]:
15    """Return (tier, human-readable reason). Reason is shown to the approver."""
16    kind = args.get("kind", "").lower()
17
18    if (tool, kind) in NEVER:
19        return "never", f"{tool} on {kind} is not delegated to an agent"
20
21    if tool == "scale_deployment":
22        current, target = args["current_replicas"], args["replicas"]
23        if target == 0:
24            return "gate", "scaling to zero takes the service fully offline"
25        if target > current:
26            return "auto", "scale-up is reversible and bounded by ResourceQuota"
27        return "gate", f"scale-down {current} -> {target} removes capacity"
28
29    if tool == "restart_deployment":
30        return "auto", "rollout restart is reversible; pods return on their own"
31
32    if tool == "patch_resource_limits":
33        return "gate", "limit changes recreate every pod in the workload"
34
35    if tool == "delete_pod":
36        if args.get("owned_by_controller"):
37            return "auto", "controller-owned pod is recreated automatically"
38        return "gate", "unmanaged pod will not come back"
39
40    return "gate", "no rule matched — unknown actions require a human"

The default case is the important line. An agent that gains a new tool next month gets gated by default rather than silently auto-approved, and the reason string tells the approver exactly why they're being asked.

An approvable prompt shows the command, the blast radius, and the reasoning

"Agent wants to run a tool. Approve?" is not an approval request. It's a coin flip with extra steps.

Three things have to be in front of the human, and only one of them is obvious:

The exact command. Not the tool name and a JSON blob — the rendered kubectl invocation or the unified diff. The approver should be able to read it and, if they distrust the agent entirely, run it themselves.

The blast radius, computed at approval time. Not "this affects the checkout deployment" — that's restating the arguments. Query the cluster: ready replicas right now, whether a PodDisruptionBudget exists, whether it's behind an Ingress, what the HPA currently wants. One API read, and it's the difference between an approver guessing and knowing.

The agent's stated reasoning. The text blocks emitted alongside the tool call. "The OOMKill at 03:02 shows the container hitting its 512Mi limit while RSS peaked at 780Mi" is checkable. If it's vague, that itself is the signal to deny.

Gating happens client-side, in your loop, before the tool executes — the API returns a tool_use block and stops; nothing runs until you run it. The rest is your code:

1import hashlib, json, time
2from anthropic import Anthropic
3
4client = Anthropic()
5
6
7def run(symptom: str, approver, max_turns: int = 15) -> str:
8    messages = [{"role": "user", "content": symptom}]
9
10    for _ in range(max_turns):
11        response = client.messages.create(
12            model="claude-opus-5",
13            max_tokens=16000,
14            system=SYSTEM,
15            tools=TOOLS,
16            messages=messages,
17        )
18
19        # Check stop_reason before touching content: a refusal returns HTTP 200
20        # with an empty or partial content array.
21        if response.stop_reason == "refusal":
22            return "Request was declined by safety classifiers."
23
24        messages.append({"role": "assistant", "content": response.content})
25
26        if response.stop_reason != "tool_use":
27            return "".join(b.text for b in response.content if b.type == "text")
28
29        # The text emitted alongside the tool calls is the agent's stated
30        # reasoning. It goes in front of the approver.
31        reasoning = "".join(b.text for b in response.content if b.type == "text")
32
33        results = [
34            execute_gated(b, reasoning, approver)
35            for b in response.content
36            if b.type == "tool_use"
37        ]
38        messages.append({"role": "user", "content": results})
39
40    return "Hit the turn limit without reaching a conclusion."
41
42
43def execute_gated(block, reasoning: str, approver) -> dict:
44    def result(text: str, is_error: bool = False) -> dict:
45        return {
46            "type": "tool_result",
47            "tool_use_id": block.id,
48            "content": text,
49            "is_error": is_error,
50        }
51
52    tier, why = classify(block.name, block.input)
53
54    if tier == "never":
55        return result(
56            f"Blocked by policy: {why}. Do not retry. Propose an alternative "
57            "or hand off to a human operator."
58        )
59
60    if tier == "auto":
61        return result(DISPATCH[block.name](**block.input))
62
63    proposal = {
64        "tool": block.name,
65        "args": block.input,
66        "command": render_command(block.name, block.input),
67        "blast_radius": blast_radius(block.name, block.input),
68        "why_gated": why,
69        "agent_reasoning": reasoning[-2000:],
70        "expires_at": time.time() + 900,
71    }
72    # The approver approves this exact proposal, not "an action by the agent".
73    proposal["id"] = hashlib.sha256(
74        json.dumps(proposal, sort_keys=True).encode()
75    ).hexdigest()[:12]
76
77    decision = approver.request(proposal)          # blocks
78    audit.write(proposal=proposal, decision=decision)
79
80    if not decision.approved:
81        return result(
82            f"{decision.approver} denied this: {decision.reason}. "
83            "Do not re-propose the same action."
84        )
85    return result(DISPATCH[block.name](**block.input))

Two details earn their place. The proposal is hashed, so the approval binds to a specific set of arguments — approving "restart checkout" and having the agent restart checkout-worker is a class of bug worth designing out. And the denial comes back as a normal tool result with a reason, not an exception: the agent needs to know why so it can propose something else, and the explicit "do not re-propose" is what stops the retry loop that grinds an approver down until they say yes.

Cluster state moves between the check and the use

The approver looked at "6 ready replicas, scale to 3" and clicked approve. Ninety seconds later they finish reading Slack and it executes. In between, the HPA scaled to 4 and someone else drained a node. The approved decision — remove three replicas — is now a different decision.

This is time-of-check to time-of-use, and it's the failure mode most approval systems ship without. The fix is to carry the observed state into the mutation as a precondition and let the API server reject it. Kubernetes gives you this directly:

# RV and REPLICAS are the values captured when the proposal was rendered.
kubectl scale deployment/checkout \
  --namespace payments \
  --current-replicas="$REPLICAS" \
  --resource-version="$RV" \
  --replicas=3

--current-replicas and --resource-version are preconditions that kubectl validates before it sends anything: if either no longer matches, the scale is never attempted. For general updates the same idea moves server-side — set metadata.resourceVersion on the object you send, and the API server returns 409 Conflict on a stale version rather than silently overwriting whatever changed. Re-read, re-render the proposal, ask again.

Pair it with a short expiry. Fifteen minutes is generous; an approval sitting in a Slack thread overnight should expire rather than fire at 6am against a cluster nobody recognises.

Three ways to build the gate, honestly compared

Synchronous in-loop prompt. The agent blocks on stdin. Perfect for a human driving a terminal, and useless for anything autonomous — the loop stops dead if nobody's watching. Fine as a starting point, not a destination.

ChatOps approval. Post the proposal to Slack with approve/deny buttons; a callback resumes the loop. This is what most teams build, and it works: the approver is where they already are and it survives the agent running unattended. But you now own an interactive endpoint that mutates production — verifying request signatures, checking that the clicker is actually authorised (Slack tells you who clicked, not whether they should have), handling double-approvals, expiring stale proposals. That's a small security-relevant service, and it will be the least-reviewed code you own.

The agent opens a pull request. The strongest option, and the one that invents nothing. Instead of touching the cluster, the agent writes the change to your GitOps repo and opens a PR:

1#!/usr/bin/env bash
2set -euo pipefail
3BRANCH="agent/${PROPOSAL_ID}"
4
5git switch -c "$BRANCH"
6yq -i '.spec.template.spec.containers[0].resources.limits.memory = "1Gi"' \
7  clusters/prod/payments/checkout.yaml
8git commit -am "checkout: raise memory limit to 1Gi"
9git push -u origin "$BRANCH"
10
11gh pr create \
12  --base main --head "$BRANCH" \
13  --title "[agent] checkout: raise memory limit to 1Gi" \
14  --body-file proposal.md

Everything you already built now applies for free. CODEOWNERS decides who can approve. Branch protection stops a self-merge. CI runs your policy checks and manifest validation. Argo CD syncs on merge — provided you've set syncPolicy.automated, since manual is the default — so the cluster credential the agent holds can stay strictly read-only. See Argo CD in production for the sync-policy side of that. The audit trail is the commit, the review, and the deployment record, and none of it is code you have to keep secure yourself. If your deployments run through GitHub Actions instead, environment protection rules give you the same shape: up to six required reviewers — users or teams — and the job waits until one of them approves.

The honest limits: it only works for resources under GitOps, and it's slow. A PR is the right answer for "raise this memory limit" and the wrong answer at 03:00 with an active incident and a pod that needs deleting now. Most teams need the PR path for planned changes and a narrow ChatOps path for the incident case — and the incident path should be the one with the shortest never-list and the tightest expiry.

The audit record needs three facts

Not one. The proposal (exactly what was asked, including the blast radius as it looked then), the approver (a human identity — not the agent's service account, not a shared bot token), and the outcome (executed, denied, expired, or failed the precondition check).

Log denials as carefully as approvals. A rising denial rate on one tool is the clearest signal you have that the agent's judgment is drifting, and without it the only visible metric is "approvals went up," which looks like adoption. Rising approvals with rising denials is a healthy gate. Rising approvals with zero denials is a rubber stamp.

What this doesn't solve

It doesn't make the agent's judgment better. A gate catches bad actions a human recognises as bad. A plausible, well-reasoned, confidently wrong proposal sails through — that's what plausible means. If the agent misreads the evidence, the gate converts an autonomous mistake into a jointly-owned one.

It doesn't cover what the agent already saw. Everything in Give an AI Agent Read-Only Access to Kubernetes still applies: reads are egress, logs are an injection channel, and adding write tools doesn't change any of it. If the agent reaches the cluster through an MCP server, the policy check belongs in the client that owns the loop, not the server — the server can't see the conversation, so it can't render the reasoning that makes a proposal judgable.

It doesn't replace a recovery plan. An approved change is still a change, and approvals fail the same way automation does: correct action, wrong assumptions. Automation without recovery is dangerous is the longer argument; the short version is that every gated action needs a known undo, and you should have run it recently.

It doesn't scale by adding approvers. If the gate fires too often, the fix is moving actions into the auto tier with better preconditions — not a bigger on-call rota. Start narrow: build the read-only agent first, watch which fixes it proposes over a month of real incidents, and promote only the handful that were consistently right into gated tools. The runbook tells you which actions are routine enough to be candidates. Everything else stays a suggestion in a terminal, which is where most of it belongs.