When an agent does the wrong thing — contain, audit, postmortem

At a glance

Learning objectives

• Contain the damage before diagnosing — rollback first
• Reconstruct what the agent did from the audit trail
• Produce a blameless postmortem with dated, owned action items

What you need

• The scoped, logged execute path your agent writes through
• The repository the agent worked in
• [object Object]

Steps

1. Roll back first — a deploy broke production: correlate the signal with the change and roll back before diagnosing. (from runbook-bad-deployment )
2. Containment was designed in — the git server can commit and create branches in the scoped repository, but cannot push unless credentials are also present in the environment. (from mcp-git )
3. Reconstruct the change from the tools the agent had — git_status, git_diff_unstaged, git_diff_staged, git_log, git_commit. (from mcp-git )
4. Pull the approval record — any write-class tool against a production target was suspended for an approve-once token scoped to the single call, so the log names the human who approved it. (from hook-prod-write-confirm )
5. Walk the incident workflow in order — establish impact and start time, check for a correlated deploy, walk the matching runbook, capture the timeline for the postmortem. (from agent-incident-assistant )
6. Keep the assistant's own guardrails on during response — it suggests, never executes, production changes, and escalates to a human before any destructive step. (from agent-incident-assistant )
7. Close with the postmortem — timeline, impact, detection gap, contributing factors, and action items with owners and dates. (from skill-incident-postmortem-writer )

Check yourself

• You can list every tool call the agent made — read-only tools leave diffs and logs, writes leave approve-once records.
• The postmortem separates trigger from contributing factors, and every action item has an owner and a date.

Worked example

A deploy broke production; correlate the signal with the change and roll
back first.

Any write-class tool against a production target is suspended and surfaced
for a human "approve once" token scoped to the single call.

Timeline, impact, detection gap, contributing factors, and action items
with owners and dates.

Sourced from

• Bad deployment
• Git MCP Server
• Production Write Confirmation
• Incident Assistant
• Incident Postmortem Writer

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Roll back first — correlate the signal with the change before diagnosing.
• The git server cannot push unless credentials are also present — if the agent pushed, a credential was in scope.
• Mitigation runs under the same guardrails — suggest, never execute, production changes.

Production considerations

• Keep the approve-once token log with the postmortem — it is the audit record of who approved the write.
• Feed the detection gap back into the hooks — the postmortem's action items should name the gate that was missing.

Diagram

agent incident
   |
   |- rollback first --> correlate signal with change
   |
   |- contain --> no push credentials --> wrong commit stays local
   |
   |- audit --> git log/diff + approve-once token log
   |
   |- respond --> impact -> correlated deploy -> runbook -> timeline
   |
   |- close --> blameless postmortem -> owned, dated action items

Registry stats