Incident Management & On-Call Best Practices for DevOps and SRE Teams

Every production system fails eventually. When an outage strikes, the difference between a 10-minute recovery and a 3-hour crisis comes down to preparation. Incident management provides a repeatable process that reduces chaos and ensures the right people respond. In this lesson, you will learn the incident lifecycle, how to design on-call rotations that prevent burnout, build runbooks that reduce mean-time-to-resolution, and how to foster a blameless culture that drives continuous improvement.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the incident management lifecycle and its key stages
• Design and implement effective on-call rotations and escalation policies
• Create runbooks that reduce mean-time-to-resolution (MTTR)
• Classify incidents by severity and follow appropriate response protocols
• Conduct effective incident retrospectives that drive systemic improvements
• Build a blameless incident response culture in your organization
• Use tools like PagerDuty, Opsgenie, and Slack to automate incident response

2. Why This Matters

Every production system fails eventually. When an outage hits, the difference between a 10-minute recovery and a 3-hour crisis comes down to preparation. Incident management is not about preventing all failures (that is impossible) — it is about having a repeatable, practiced process that reduces chaos, ensures the right people are engaged, and captures learnings to make the system more resilient over time.

Think of incident management like a fire drill for your infrastructure. A building with fire drills, clearly marked exits, and trained marshals handles a real fire safely. A building without those things turns a small kitchen fire into a disaster. Your production environment is no different.

3. Core Concepts

3.1 The Incident Management Lifecycle

Incident management follows a well-defined lifecycle with five stages:

1. Detection — An alert fires from your monitoring system, or a user reports a problem
2. Triage — The on-call engineer assesses severity and decides what to do
3. Response — The incident response team executes the runbook to mitigate the issue
4. Resolution — The fix is applied and the system returns to normal operation
5. Follow-up — The team conducts a post-mortem to document root cause and preventive actions

3.2 Severity Levels

Classifying incidents by severity ensures the right resources are engaged at the right time. A standard severity taxonomy:

• SEV-1 (Critical) — Complete service outage affecting all users. Response: immediate, page the entire team
• SEV-2 (High) — Major feature degradation or partial outage affecting many users. Response: within 15 minutes
• SEV-3 (Medium) — Minor feature issue affecting a subset of users. Response: next business day
• SEV-4 (Low) — Cosmetic issues, non-critical bugs. Response: next sprint

3.3 On-Call Rotations

On-call rotations ensure 24/7 coverage without burning out any single engineer. Common rotation patterns:

• Primary/Secondary model — One engineer is primary responder, a second is backup if the primary does not acknowledge within the SLA window
• Follow-the-sun — Teams in different time zones handle daytime hours locally, with handoffs at shift boundaries
• Weekly rotations — Each engineer serves one week of primary on-call, then rotates off. Avoids the exhaustion of 24/7 on-call for a single person
• Escalation policies — If the primary does not respond within N minutes, the incident escalates to the secondary, then to the engineering manager, and finally to the director

3.4 Runbooks

A runbook (or playbook) is a step-by-step checklist for handling specific types of incidents. Good runbooks reduce MTTR by eliminating the need to figure out what to do during a crisis. Each runbook should include:

• Clear title and the symptoms it covers
• Severity classification guidance
• Step-by-step diagnostic commands to run
• Potential root causes and their specific fixes
• Service owner contact information
• A verification checklist to confirm the fix worked

4. Hands-On Practice: Building an Incident Response Runbook

4.1 Setting Up an On-Call Rotation Model

While a full PagerDuty setup requires a paid account, you can model the schedule structure locally to understand the concepts. Here is a Python snippet that models an on-call rotation for a team of 4 engineers:

import json
from datetime import datetime, timedelta

team = [
    "alice@example.com",
    "bob@example.com",
    "carol@example.com",
    "dave@example.com"
]

def generate_rotation(start_date, weeks):
    schedule = []
    for i in range(weeks):
        primary = team[i % len(team)]
        secondary = team[(i + 1) % len(team)]
        start = start_date + timedelta(weeks=i)
        end = start + timedelta(weeks=1)
        schedule.append({
            "week": i + 1,
            "primary": primary,
            "secondary": secondary,
            "start": start.strftime("%Y-%m-%d"),
            "end": end.strftime("%Y-%m-%d")
        })
    return schedule

rotations = generate_rotation(datetime(2026, 8, 1), 8)
for r in rotations:
    print(f"Week {r[chr(39)+chr(39)+chr(39)]week]}")

4.2 Creating a Database Slow-Query Runbook

Here is an example runbook for a common incident type — slow database queries causing high response latency. This runbook structure can be adapted for any recurring issue pattern:

# Runbook: Database Slow Queries / High Latency

## Symptoms
- API response times exceed 2 seconds
- Database CPU > 80%
- Application alerts: Query timeout

## Severity Assessment
- Single endpoint affected: SEV-3
- All endpoints affected: SEV-2
- Complete database unavailability: SEV-1

## Diagnosis Steps

1. SSH into the database server:
   ssh db-prod-01.example.com

2. Check active queries:
   SELECT pid, now() - pg_stat_activity.query_start AS duration,
          query, state
   FROM pg_stat_activity
   WHERE state != idle
   ORDER BY duration DESC
   LIMIT 20;

3. Terminate a stuck query if needed:
   SELECT pg_terminate_backend(pid);

4. Check slow query log:
   sudo tail -100 /var/log/postgresql/slow-query.log

5. Check connection pool saturation:
   SELECT count(*) FROM pg_stat_activity WHERE state = active;

## Resolution Actions

1. If a single slow query is blocking others:
   - Terminate the blocking query
   - Notify the developer who owns the query

2. If connection pool is full:
   - Increase max_connections in postgresql.conf
   - Restart: sudo systemctl restart postgresql

3. If disk I/O is saturated:
   - Check iostat -x 1
   - Consider scaling up or optimizing indexes

## Verification
- Run the same diagnostic queries again
- Confirm response times returned to normal
- Check application health endpoint

## Escalation
If unresolved in 15 minutes, escalate to the DBA team:
  #dba-oncall via Slack

4.3 Simulating an Incident Response Flow

Practice the incident response workflow with this simple Python script that simulates an alert and walks you through triage:

import json, time

class IncidentSimulator:
    def __init__(self):
        self.incidents = [
            {"id": "INC-001", "service": "api-gateway",
             "alert": "HTTP 503 errors rising > 5%",
             "severity": "SEV-2"},
            {"id": "INC-002", "service": "auth-service",
             "alert": "Login failures > 20% in last 5 min",
             "severity": "SEV-1"},
            {"id": "INC-003", "service": "worker-queue",
             "alert": "Queue depth > 10,000 messages",
             "severity": "SEV-3"},
        ]

    def start(self):
        inc = self.incidents.pop(0)
        print(f"ALERT: {inc[chr(39)+chr(39)+chr(39)]alert]}")        print(f"Service: {inc['service']} | Severity: {inc['severity']}")
        print()
        print("=== TRIAGE CHECKLIST ===")
        print("1. Acknowledge the alert")
        print("2. Determine impact (users affected)")
        print("3. Open the runbook for this service")
        print("4. Start incident channel in Slack")
        print("5. Begin diagnostic steps per runbook")
        return inc

sim = IncidentSimulator()
sim.start()

5. Common Errors & Solutions

5.1 Alert Fatigue

Error: Every minor metric fluctuation pages the on-call engineer. After weeks of false alarms, the team starts ignoring alerts entirely.

Solution: Review your alert rules quarterly. Every alert should trigger a specific, actionable response. If the response is "ignore it," the alert should be deleted or its threshold raised. Use alert deduplication and rate limiting to prevent alert storms.

5.2 The Hero Culture

Error: One engineer is always "the one who fixes things." They never go off-call, they burn out, and the team has no shared knowledge of the production systems.

Solution: Mandatory rotation with a minimum of two people per shift. No engineer serves primary on-call for more than one week at a time.

5.3 No Runbook

Error: Every incident requires the on-call engineer to reverse-engineer the system from scratch because there are no documented procedures.

Solution: After each incident, write or update the runbook. Store runbooks in version control with the same review process as code changes.

5.4 Blame Culture

Error: Post-mortems focus on "who made the mistake" instead of "what in the system allowed this mistake to cause an outage."

Solution: Adopt a blameless post-mortem culture. Frame every post-mortem around systemic improvements. The question is always: "How do we make the system more resilient so this failure cannot happen again?" — never "Who caused this?"

5.5 Forgetting to Verify the Fix

Error: The team applies a fix and marks the incident resolved without confirming the system is actually healthy. The same issue resurfaces minutes later.

Solution: Every runbook must include a verification section with specific health checks to run before declaring the incident resolved. Automate these checks in your monitoring system as post-fix acceptance tests.

6. Summary Checklist

• I understand the five stages of the incident management lifecycle
• I can classify incidents by severity level (SEV-1 through SEV-4)
• I can design a primary/secondary on-call rotation for my team
• I know the key components of an effective runbook
• I can avoid alert fatigue by tuning alert thresholds
• I understand the difference between blame culture and blameless culture
• I know to always verify a fix before closing an incident

7. Practice Exercise

Scenario: You are the SRE on-call for a SaaS platform. At 2:00 AM, you receive a SEV-1 alert: "Payment processing latency > 30 seconds" on the payment-svc service. The website is showing "Checkout Unavailable" to customers in the EU region.

Tasks:

1. Write a structured incident response — what are your first 5 actions in order?
2. Create a minimal runbook page for the payment-svc service with 3 diagnostic commands and 2 resolution options
3. Identify three questions you would ask during the post-mortem to find the systemic root cause
4. Design a 4-person on-call rotation with an escalation path

8. Next Steps

Now that you understand incident management and on-call best practices, the next logical topic in the DevOps & SRE series is SLIs, SLOs, and Error Budgets — the quantitative framework that SRE teams use to define reliability targets, measure service health objectively, and make data-driven decisions about when to prioritize new features vs. reliability improvements. This framework is the foundation of every mature SRE practice and directly informs how on-call teams set their alert thresholds and severity levels.