Post-Mortem Culture: Learning from Failure

The 5 Whys Technique (Toyota Production System):

Start with the problem, ask "why" repeatedly until you find root cause.

Example: API returning 500 errors

1. Why? → Database connection timeout
   ↓
2. Why? → Connection pool exhausted
   ↓
3. Why? → Connections not being released
   ↓
4. Why? → Application bug doesn't close connections
   ↓
5. Why? → Code review missed connection handling
   ↓
ROOT CAUSE: Missing automated test for connection leaks

Action: Add connection leak detection to CI pipeline

Another Example: Deployment failed

1. Why? → Configuration file had syntax error
   ↓
2. Why? → Change was made manually, not reviewed
   ↓
3. Why? → No automated validation before deploy
   ↓
4. Why? → CI pipeline missing config validation step
   ↓
5. Why? → Team didn't prioritize adding validation
   ↓
ROOT CAUSE: No automated config validation

Action: Add `terraform validate` to CI pipeline

Characteristics of Good Action Items:

✓ Specific ("Add monitoring for X" not "Improve monitoring")
✓ Assignable (has an owner)
✓ Time-bound (has a deadline)
✓ Verifiable (can check if done)
✓ Prevention-focused (fixes root cause)

Bad Action Items → Good Action Items:

Bad: "Improve error handling"
Good: "Add retry logic with exponential backoff to payment service by Mar 15"

Bad: "Better documentation"
Good: "Add database connection troubleshooting to runbook by Mar 10"

Bad: "Fix monitoring"
Good: "Add alert for DB connection pool > 80% with owner @sre by Mar 12"

Bad: "Prevent this from happening again"
Good: "Add integration test for connection leak to CI pipeline by Mar 17"

Action Item Types:
┌─────────────────────────────────────────────────────────────────┐
│ Type          │ Example                                         │
├───────────────┼─────────────────────────────────────────────────┤
│ Monitoring    │ Add alert for high error rate                   │
│ Automation    │ Automate database backup verification          │
│ Process       │ Add config validation to PR checklist           │
│ Documentation │ Update runbook with troubleshooting steps      │
│ Testing       │ Add regression test for edge case               │
│ Training      │ Train team on new process                      │
└─────────────────────────────────────────────────────────────────┘

How to Run a Post-Mortem Meeting:

Before the Meeting (24-48 hours after incident):
┌─────────────────────────────────────────────────────────────────┐
│ 1. Collect timeline from incident channel                      │
│ 2. Gather metrics (duration, impact, etc.)                     │
│ 3. Send pre-read to attendees                                  │
│ 4. Remind everyone: this is BLAMELESS                          │
└─────────────────────────────────────────────────────────────────┘

During the Meeting (60-90 minutes):
┌─────────────────────────────────────────────────────────────────┐
│ 0:00 - 0:05  | Set the stage (blameless, agenda)              │
│ 0:05 - 0:20  | Review timeline (what happened, when)          │
│ 0:20 - 0:40  | Root cause analysis (5 Whys)                   │
│ 0:40 - 0:55  | Brainstorm action items                        │
│ 0:55 - 1:05  | Assign owners and due dates                    │
│ 1:05 - 1:10  | Close (thank participants)                     │
└─────────────────────────────────────────────────────────────────┘

Facilitator Tips:
┌─────────────────────────────────────────────────────────────────┐
│ • Neutral party (not incident commander)                       │
│ • If blame emerges, redirect to systems                        │
│ • Keep focused on learning, not fixing                         │
│ • Everyone speaks, no one dominates                            │
│ • Record decisions, not who said what                          │
└─────────────────────────────────────────────────────────────────┘

After the Meeting:
┌─────────────────────────────────────────────────────────────────┐
│ 1. Write post-mortem document                                  │
│ 2. Share with wider organization                               │
│ 3. Create tickets for action items                             │
│ 4. Schedule follow-up to check progress                        │
│ 5. Update runbooks                                             │
└─────────────────────────────────────────────────────────────────┘

Psychological Safety is the belief that you won't be punished for speaking up.

Why It Matters:
• Higher incident reporting (see problems earlier)
• Faster incident resolution (people ask for help)
• More innovation (people suggest improvements)
• Lower turnover

How to Build Psychological Safety:

1. Lead by Example
   ├── Managers admit their own mistakes publicly
   ├── "I don't know" is an acceptable answer
   └── Thank people for speaking up

2. Frame Work as Learning
   ├── "What can we learn from this?"
   ├── Not "Who made a mistake?"
   └── Celebrate learning, not just successes

3. Acknowledge Your Own Fallibility
   ├── "I might be missing something"
   ├── "What am I not seeing?"
   └── "Help me understand"

4. Invite Participation
   ├── "What are your thoughts?"
   ├── "Any concerns?"
   └── "What did I miss?"

5. Respond Productively
   ├── Not "Why didn't you..."
   ├── But "How can we prevent this next time?"
   └── Thank people for surfacing issues

Signs of Low Psychological Safety:
✗ People don't speak up in meetings
✗ Blame is common
✗ Failures are hidden
✗ People work in silos
✗ High turnover

Signs of High Psychological Safety:
✓ People ask questions freely
✓ Failures are shared as learning
✓ Cross-team collaboration
✓ People admit mistakes
✓ Low turnover

Measuring Post-Mortem Effectiveness:

┌─────────────────────────────────────────────────────────────────┐
│ Metric                    │ What It Tells You                   │
├───────────────────────────┼─────────────────────────────────────┤
│ Post-mortem completion    │ Are you learning from incidents?    │
│ rate (target: 100%)       │                                     │
├───────────────────────────┼─────────────────────────────────────┤
│ Time to post-mortem       │ How quickly do you learn?           │
│ (target: < 48 hours)      │                                     │
├───────────────────────────┼─────────────────────────────────────┤
│ Action item completion    │ Are you actually improving?         │
│ rate (target: > 80%)      │                                     │
├───────────────────────────┼─────────────────────────────────────┤
│ Repeat incidents          │ Are you fixing root causes?         │
│ (target: decreasing)      │                                     │
├───────────────────────────┼─────────────────────────────────────┤
│ MTTR (Mean Time To        │ Are post-mortems helping recovery?  │
│ Resolve) (target: down)   │                                     │
└───────────────────────────┴─────────────────────────────────────┘

Example Dashboard:
┌─────────────────────────────────────────────────────────────────┐
│ POST-MORTEM DASHBOARD - Q1 2025                                 │
├─────────────────────────────────────────────────────────────────┤
│ ✓ Post-mortems completed: 12/12 (100%)                         │
│ ✓ Avg time to post-mortem: 26 hours (target: 48)               │
│ ✓ Action items completed: 45/52 (86%)                          │
│ ⚠ Repeat incidents: 3 this quarter (up from 2 last quarter)   │
│ ✓ MTTR: 32 minutes (down 15% from last quarter)                │
│                                                                 │
│ Top Action Item Types:                                         │
│ • Monitoring improvements: 14                                  │
│ • Runbook updates: 10                                          │
│ • Code fixes: 8                                                │
│ • Automation: 6                                                │
│ • Process changes: 5                                           │
│ • Testing improvements: 4                                      │
│ • Documentation: 3                                             │
│ • Training: 2                                                  │
└─────────────────────────────────────────────────────────────────┘