SRE is what happens when you treat operations as a software engineering problem. Google created SRE to run production at scale. The core principle: use software engineering to automate operations tasks. In this lesson, you'll learn SLOs, error budgets, toil reduction, and the practices that keep systems reliable.
1. Learning Objectives
By the end of this lesson, you will be able to:
- Understand the SRE role and principles
- Define SLIs (Service Level Indicators)
- Set SLOs (Service Level Objectives)
- Use error budgets to balance reliability and velocity
- Identify and reduce toil
- Implement SRE best practices
2. What is SRE?
Site Reliability Engineering (SRE) is what happens when you ask a software engineer to design an operations team. Google created SRE in 2003 to run production at planetary scale.
SRE vs DevOps:
- DevOps is a culture/philosophy of collaboration between Dev and Ops
- SRE is a job role and set of practices to implement DevOps principles
- SRE is "DevOps with a job description"
- Many companies use SRE practices without calling it SRE
3. Core SRE Concepts
SLIs, SLOs, and SLAs
╔═══════════════════════════════════════════════════════════════════╗
║ Service Level Terms ║
╠═══════════════════════════════════════════════════════════════════╣
║ ║
║ SLI (Service Level Indicator) ║
║ ──────────────────────────────────────────────────────────────── ║
║ A measurement of service behavior. ║
║ ║
║ Examples: ║
║ • Request latency (95th percentile) ║
║ • Error rate (percentage of 5xx responses) ║
║ • Availability (uptime percentage) ║
║ • Throughput (requests per second) ║
║ ║
║ SLO (Service Level Objective) ║
║ ──────────────────────────────────────────────────────────────── ║
║ A target value for a service level indicator. ║
║ ║
║ Examples: ║
║ • 99.9% availability ("three nines") ║
║ • 95% of requests < 500ms ║
║ • Error rate < 0.1% ║
║ ║
║ SLA (Service Level Agreement) ║
║ ──────────────────────────────────────────────────────────────── ║
║ A promise to customers about service performance. ║
║ Usually with consequences (refunds, credits) for missing. ║
║ ║
║ • Often looser than internal SLOs ║
║ • Example: 99.5% uptime guarantee ║
║ ║
╚═══════════════════════════════════════════════════════════════════╝SLI, SLO, SLA definitions
Choosing Good SLIs
How to Choose SLIs:
1. Focus on user-facing metrics (what matters to users)
2. Ask: "What would make users unhappy?"
3. Start simple, add complexity as needed
Common SLIs for Web Services:
┌─────────────────┬────────────────────────────────────────────┐
│ SLI │ Measurement │
├─────────────────┼────────────────────────────────────────────┤
│ Availability │ (successful requests / total requests) │
│ │ × 100% │
├─────────────────┼────────────────────────────────────────────┤
│ Latency │ histogram_quantile(0.99, │
│ │ http_request_duration_seconds) │
├─────────────────┼────────────────────────────────────────────┤
│ Error Rate │ (failed requests / total requests) │
│ │ × 100% │
├─────────────────┼────────────────────────────────────────────┤
│ Throughput │ rate(http_requests_total[1m]) │
├─────────────────┼────────────────────────────────────────────┤
│ Freshness │ time since last data update │
├─────────────────┼────────────────────────────────────────────┤
│ Correctness │ data validation success rate │
└─────────────────┴────────────────────────────────────────────┘
SLO Examples:
• Availability: 99.9% (three nines) = 8.76 hours downtime/year
• Availability: 99.99% (four nines) = 52.6 minutes downtime/year
• Availability: 99.999% (five nines) = 5.26 minutes downtime/year
• Latency: 95% of requests < 500ms
• Latency: 99% of requests < 1 second
• Error rate: < 0.1% of requests return 5xxChoosing and defining SLIs and SLOs
Error Budgets
Error Budget = 100% - SLO
If SLO is 99.9%, error budget is 0.1% (unreliable time allowed)
┌─────────────────────────────────────────────────────────────────┐
│ How Error Budgets Work │
├─────────────────────────────────────────────────────────────────┤
│ │
│ When error budget is available (budget > 0): │
│ ─────────────────────────────────────────────────────────────── │
│ • Can release new features │
│ • Can take calculated risks │
│ • Innovation velocity is prioritized over reliability │
│ │
│ When error budget is exhausted (budget ≤ 0): │
│ ─────────────────────────────────────────────────────────────── │
│ • Stop all feature releases │
│ • Focus only on reliability improvements │
│ • Fix bugs that are causing errors │
│ • Add monitoring and automation │
│ • Reduce toil │
│ │
└─────────────────────────────────────────────────────────────────┘
Example:
• SLO: 99.9% availability
• Error budget: 0.1% = 43.8 minutes per month
• If we have 30 minutes of downtime in a month
• Error budget remaining: 13.8 minutes
• Still safe to release new features
If we have 50 minutes of downtime in a month
• Error budget: -6.2 minutes (exhausted)
• STOP feature releases
• Only work on reliability
This aligns incentives: reliability is everyone's responsibilityError budgets for balancing reliability and velocity
Toil
Toil is manual, repetitive, automatable, tactical work that doesn't scale.
Characteristics of Toil:
✓ Manual (human does it)
✓ Repetitive (same tasks over and over)
✓ Automatable (can be coded)
✓ Tactical (not strategic)
✓ No enduring value
✓ Scales linearly with growth
Examples of Toil:
✗ Manually restarting failed services
✗ Manually approving deployments
✗ Copy-pasting configurations
✗ Manually creating cloud resources
✗ SSH into servers to check logs
✗ Manually rotating credentials
Non-Toil (Valuable Work):
✓ Writing automation to prevent toil
✓ Improving monitoring and alerting
✓ Capacity planning
✓ Architecture improvements
✓ On-call rotation improvements
✓ Writing documentation
SRE Toil Budget:
• SREs should spend no more than 50% of time on toil
• If toil exceeds 50%, either:
→ Hire more engineers
→ Automate the toil
→ Push back on operational work
• Remaining time on engineering projects (automation, reliability improvements)Understanding and reducing toil
4. SRE Practices
Key SRE Practices:
1. Error Budget Policies
─────────────────────
• Define clear SLOs
• Track error budget consumption
• Automatically block releases when budget exhausted
• Alert when budget is running low
2. Blameless Post-Mortems
────────────────────────
• After every incident, document
• Focus on systems, not people
• "How could we prevent this from happening again?"
• Action items with owners
3. Toil Reduction
───────────────
• Track time spent on toil
• Set toil budget (max 50%)
• Automate most common toil tasks
• Build self-service tools
4. Service Level Objectives (SLOs)
────────────────────────────────
• Define SLOs for every service
• Start with 1-2 key SLOs
• Make SLOs visible to everyone
• Review SLO attainment regularly
5. Capacity Planning
─────────────────
• Monitor resource usage trends
• Predict when you'll need more capacity
• Automate scaling where possible
• Plan for growth (storage, compute, network)
6. Change Management
─────────────────
• Automate deployments
• Progressive rollouts (canary)
• Fast rollback (seconds, not minutes)
• Feature flags
7. Emergency Response
───────────────────
• Clear on-call rotations
• Runbooks for common issues
• Alerting with proper severity
• Game days to practice
8. Data-Driven Decisions
─────────────────────
• Measure everything
• Dashboard with key metrics
• Regular reviews with data
• Use error budgets to decide: features vs reliabilitySRE key practices
5. SRE Role and Responsibilities
What SREs Do:
┌─────────────────────────────────────────────────────────────────┐
│ Software Engineering (50%) │
├─────────────────────────────────────────────────────────────────┤
│ • Write code to automate operations │
│ • Build self-service tools for developers │
│ • Improve monitoring and alerting │
│ • Implement auto-remediation │
│ • Chaos engineering │
│ • Capacity planning systems │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ Operations (50% max) │
├─────────────────────────────────────────────────────────────────┤
│ • On-call rotation │
│ • Incident response │
│ • Troubleshooting production issues │
│ • Deployment assistance │
│ • Performance analysis │
└─────────────────────────────────────────────────────────────────┘
SRE vs Traditional Ops:
┌─────────────────┬──────────────────┬─────────────────────────────┐
│ Aspect │ Traditional │ SRE │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ Work │ Reactive, toil │ Proactive, engineering │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ On-call │ All Ops staff │ Rotating, with comp │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ Change │ Change approvals │ Automated with canary │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ Incident │ Find culprits │ Find causes (blameless) │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ Metrics │ Uptime only │ SLIs, SLOs, error budgets │
├─────────────────┼──────────────────┼─────────────────────────────┤
│ Developer │ Throw over wall │ Shared responsibility │
│ relationship │ │ │
└─────────────────┴──────────────────┴─────────────────────────────┘SRE role and responsibilities
6. Implementing SLOs in Prometheus
# SLO recording rules in Prometheus
groups:
- name: slo
interval: 30s
rules:
# Availability SLI (success rate)
- record: slo:availability:ratio_rate5m
expr: |
sum(rate(http_requests_total{status!~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))
# Error budget burn rate
- record: slo:availability:error_budget_burn
expr: |
(1 - slo:availability:ratio_rate5m) / (1 - 0.999)
# Latency SLI (95th percentile)
- record: slo:latency:p95
expr: |
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)
# SLO alerting rules
- name: slo_alerts
rules:
- alert: SLOErrorBudgetBurnRate
expr: slo:availability:error_budget_burn > 3
for: 5m
labels:
severity: warning
team: sre
annotations:
summary: "Error budget burning too fast"
description: "Error budget burn rate is {{ $value }}x"
- alert: SLOMissed
expr: slo:availability:ratio_rate5m < 0.999
for: 10m
labels:
severity: critical
annotations:
summary: "SLO violation"
description: "Availability is {{ $value | humanizePercentage }}"Implementing SLOs with Prometheus
7. Summary Checklist
8. Next Steps
Next lesson: DevOps Lesson 3: Incident Management
Comments (0)
This is exactly what I needed! The initContainer approach solved our migration issues completely. Thanks for the detailed guide!
ReplyLeave a Comment