Monitoring & Observability: Building Complete System Visibility

Your application works fine on your laptop. But in production, with real users and real traffic, what's actually happening? Is it slow? Is it crashing? Is it about to run out of memory? Without monitoring and observability, you're flying blind. The three pillars—metrics, logs, and traces—give you complete visibility. In this comprehensive guide, you'll learn to instrument applications, collect telemetry data, and build dashboards that help you sleep at night.

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the three pillars of observability (Metrics, Logs, Traces)
Instrument applications with Prometheus metrics
Build Grafana dashboards for visualization
Centralize logs with Loki or ELK stack
Implement distributed tracing with Jaeger
Set up alerting for critical conditions
Design an observability strategy for your systems

2. Why Monitoring & Observability Matter

Real-world scenario without observability: Users report the site is slow. You have no metrics, no logs, no traces. You spend hours guessing: Is it the database? Is it the network? Is it the code? You're blind. Customers leave.

With observability: Your dashboard shows database query latency spiked at 3:15 PM. Your logs show a slow query. Your trace shows exactly which query and which endpoint. You fix it in 10 minutes.

The Three Pillars:

📊 Metrics: Numerical data over time (CPU usage, request rate, error count) - "What happened?"
📝 Logs: Structured events (error messages, audit trails) - "Why did it happen?"
🔗 Traces: Request flow across services - "Where did it happen?"

3. Core Concepts

Metrics with Prometheus

Installing Prometheus

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvf prometheus-2.48.0.linux-amd64.tar.gz
cd prometheus-2.48.0.linux-amd64

# Run Prometheus
./prometheus --config.file=prometheus.yml

# Using Docker
docker run -d \
  -p 9090:9090 \
  -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Access Prometheus UI
# http://localhost:9090

Installing and running Prometheus

# prometheus.yml
 global:
   scrape_interval: 15s
   evaluation_interval: 15s

 alerting:
   alertmanagers:
     - static_configs:
         - targets: ['alertmanager:9093']

 rule_files:
   - "alerts.yml"

 scrape_configs:
   - job_name: 'prometheus'
     static_configs:
       - targets: ['localhost:9090']

   - job_name: 'node'
     static_configs:
       - targets: ['node-exporter:9100']

   - job_name: 'application'
     scrape_interval: 10s
     static_configs:
       - targets: ['myapp:8000']
     metrics_path: '/metrics'

   - job_name: 'kubernetes-pods'
     kubernetes_sd_configs:
       - role: pod
     relabel_configs:
       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
         action: keep
         regex: true
       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
         action: replace
         target_label: __metrics_path__
         regex: (.+)
       - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
         action: replace
         regex: ([^:]+)(?::\d+)?;(\d+)
         replacement: $1:$2
         target_label: __address__

Prometheus configuration file

Instrumenting Applications with Prometheus Client

# app.py - Flask application with Prometheus metrics
from flask import Flask, request, jsonify
from prometheus_client import Counter, Histogram, Gauge, generate_latest, REGISTRY
import time
import random

app = Flask(__name__)

# Define metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'HTTP request latency', ['method', 'endpoint'])
ACTIVE_REQUESTS = Gauge('http_requests_active', 'Active HTTP requests')
ERROR_COUNT = Counter('app_errors_total', 'Total application errors', ['error_type'])

# Custom business metrics
ORDER_COUNT = Counter('orders_placed_total', 'Total orders placed', ['product_type'])
DB_QUERY_LATENCY = Histogram('db_query_duration_seconds', 'Database query latency', ['query_type'])
CACHE_HITS = Counter('cache_hits_total', 'Total cache hits')
CACHE_MISSES = Counter('cache_misses_total', 'Total cache misses')

@app.before_request
def before_request():
    request.start_time = time.time()
    ACTIVE_REQUESTS.inc()

@app.after_request
def after_request(response):
    latency = time.time() - request.start_time
    REQUEST_LATENCY.labels(method=request.method, endpoint=request.path).observe(latency)
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.path,
        status=response.status_code
    ).inc()
    ACTIVE_REQUESTS.dec()
    return response

@app.route('/')
def home():
    return jsonify({"message": "Hello, World!"})

@app.route('/orders', methods=['POST'])
def create_order():
    data = request.json
    product_type = data.get('product_type', 'unknown')
    
    # Simulate database query
    start = time.time()
    time.sleep(random.uniform(0.05, 0.2))  # Simulate DB work
    DB_QUERY_LATENCY.labels(query_type='insert_order').observe(time.time() - start)
    
    # Update business metrics
    ORDER_COUNT.labels(product_type=product_type).inc()
    
    # Simulate occasional errors
    if random.random() < 0.1:
        ERROR_COUNT.labels(error_type='database_timeout').inc()
        return jsonify({"error": "Database timeout"}), 500
    
    return jsonify({"status": "success", "order_id": random.randint(1, 1000)})

@app.route('/products/<int:product_id>')
def get_product(product_id):
    # Simulate cache check
    if random.random() < 0.7:
        CACHE_HITS.inc()
        return jsonify({"source": "cache", "product_id": product_id})
    else:
        CACHE_MISSES.inc()
        # Simulate database query
        start = time.time()
        time.sleep(0.1)
        DB_QUERY_LATENCY.labels(query_type='select_product').observe(time.time() - start)
        return jsonify({"source": "database", "product_id": product_id})

@app.route('/metrics')
def metrics():
    return generate_latest(REGISTRY), 200, {'Content-Type': 'text/plain'}

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

Flask application with Prometheus instrumentation

# Dockerfile with Prometheus metrics
docker build -t myapp .
docker run -d -p 8000:8000 myapp

# Test metrics endpoint
curl http://localhost:8000/metrics

# Sample output:
# http_requests_total{endpoint="/",method="GET",status="200"} 42.0
# http_request_duration_seconds_bucket{le="0.005",method="GET",endpoint="/"} 5.0
# http_request_duration_seconds_bucket{le="0.01",method="GET",endpoint="/"} 15.0
# http_requests_active 3.0
# orders_placed_total{product_type="electronics"} 127.0
# db_query_duration_seconds_count{query_type="insert_order"} 500.0
# db_query_duration_seconds_sum{query_type="insert_order"} 45.2
# cache_hits_total 1345.0
# cache_misses_total 456.0

Accessing Prometheus metrics endpoint

PromQL (Prometheus Query Language)

# Basic Queries
# Get current request rate
rate(http_requests_total[5m])

# Get 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Predicates
# Alert if error rate > 1%
(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.01

# Cache hit ratio
rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m]))

# Advanced Queries
# CPU usage per pod
topk(10, sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod))

# Memory usage per service
sum(container_memory_working_set_bytes{namespace="production"}) by (service)

# Request rate by endpoint with outliers
rate(http_requests_total[5m]) > (avg(rate(http_requests_total[5m])) + 3 * stddev(rate(http_requests_total[5m])))

# Alerting Rules (alerts.yml)
groups:
  - name: application_alerts
    rules:
      - alert: HighErrorRate
        expr: (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }}% for the last 5 minutes"
      
      - alert: SlowResponses
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow response times"
          description: "95th percentile latency is {{ $value }}s"
      
      - alert: HighMemoryUsage
        expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "{{ $labels.pod }} is using {{ $value | humanizePercentage }} of memory"

PromQL queries for metrics analysis

Grafana Dashboards

# Run Grafana with Docker
docker run -d -p 3000:3000 --name=grafana grafana/grafana

# Access Grafana
# http://localhost:3000
# Username: admin
# Password: admin

# Add Prometheus as data source:
# Configuration → Data Sources → Add data source → Prometheus
# URL: http://prometheus:9090
# Save & Test

# Import pre-built dashboards:
# Dashboard ID 1860 (Node Exporter Full)
# Dashboard ID 3662 (Prometheus Stats)
# Dashboard ID 11835 (Kubernetes Views)

Setting up Grafana

{
  "dashboard": {
    "title": "Application Performance Dashboard",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{endpoint}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
      },
      {
        "title": "Error Rate",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))",
            "format": "percent"
          }
        ],
        "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0}
      },
      {
        "title": "95th Percentile Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "{{endpoint}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
      },
      {
        "title": "Cache Hit Ratio",
        "type": "gauge",
        "targets": [
          {
            "expr": "rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m]))"
          }
        ],
        "gridPos": {"h": 4, "w": 6, "x": 12, "y": 8}
      }
    ]
  }
}

Grafana dashboard JSON configuration

Logging with Loki

# docker-compose-loki.yml
version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
  
  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - /var/lib/docker/containers:/var/lib/docker/containers
    command: -config.file=/etc/promtail/config.yml
  
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_INSTALL_PLUGINS=grafana-lokiexplorer-app

# Python logging with Loki
gunicorn myapp:app --log-level=debug --access-logfile=-

# Or use Python logging library
import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger()
handler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("User logged in", extra={"user_id": 123, "ip": "192.168.1.1"})
logger.error("Database connection failed", extra={"db": "postgres", "error": "timeout"})

Loki and Promtail setup for log aggregation

# LogQL Queries for Loki

# Basic search
{app="myapp"} |= "error"

# Regex search
{app="myapp"} |~ "(?i)timeout|failed"

# JSON parsing
{app="myapp"} | json | line_format "{{.user_id}} {{.message}}"

# Filter by time
{app="myapp", namespace="production"} |= "ERROR" | logfmt | duration > 1s

# Count errors over time
count_over_time({app="myapp"} |= "ERROR" [5m])

# Rate of errors
rate({app="myapp"} |= "ERROR" [5m])

# Top error messages
topk(10, count by(message) ({app="myapp"} |= "ERROR"))

# Pattern extraction
{app="myapp"} | pattern `<_> user <user> logged in from <ip>`

# Alert on log patterns
- alert: TooManyErrors
  expr: count_over_time({app="myapp"} |= "ERROR" [5m]) > 100
  annotations:
    summary: "Too many errors in logs"

LogQL queries for log analysis

Distributed Tracing with Jaeger

# Run Jaeger with Docker
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:latest

# Access Jaeger UI
# http://localhost:16686

Setting up Jaeger for distributed tracing

# Tracing with OpenTelemetry
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource

# Initialize tracer
resource = Resource.create({"service.name": "myapp"})
provider = TracerProvider(resource=resource)

jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
trace.set_tracer_provider(provider)

# Instrument Flask automatically
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

# Manual instrumentation
@app.route('/process/<int:order_id>')
def process_order(order_id):
    with tracer.start_as_current_span("process-order") as span:
        span.set_attribute("order.id", order_id)
        
        # Database call
        with tracer.start_as_current_span("db-query"):
            db_result = query_database(order_id)
        
        # External API call
        with tracer.start_as_current_span("external-api"):
            api_result = call_external_api(db_result)
        
        span.set_attribute("result.status", "success")
        return api_result

# Custom span with attributes
def process_payment(amount, user_id):
    with tracer.start_as_current_span("payment.process") as span:
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.user_id", user_id)
        span.set_attribute("payment.currency", "USD")
        
        # Record events within span
        span.add_event("payment.initiated")
        result = payment_gateway.charge(amount)
        span.add_event("payment.completed")
        
        if result.success:
            span.set_status(trace.StatusCode.OK)
        else:
            span.set_status(trace.StatusCode.ERROR, result.error)
        
        return result

OpenTelemetry instrumentation for distributed tracing

4. Complete Observability Stack with Docker Compose

# docker-compose-observability.yml
version: '3.8'

services:
  # Prometheus (Metrics)
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
  
  # Grafana (Visualization)
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    volumes:
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources
      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus
      - loki
  
  # Loki (Logs)
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki-data:/loki
  
  # Promtail (Log Collector)
  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - /var/lib/docker/containers:/var/lib/docker/containers
      - ./promtail/promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki
  
  # Jaeger (Tracing)
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    volumes:
      - jaeger-data:/data
  
  # Node Exporter (System Metrics)
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
  
  # Blackbox Exporter (Endpoint Monitoring)
  blackbox:
    image: prom/blackbox-exporter:latest
    ports:
      - "9115:9115"
    volumes:
      - ./blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml
  
  # cAdvisor (Container Metrics)
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  
  # Alertmanager (Alerting)
  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
  
  # Your Application
  myapp:
    build: .
    ports:
      - "8000:8000"
    environment:
      - JAEGER_AGENT_HOST=jaeger
      - JAEGER_AGENT_PORT=6831
    labels:
      - "prometheus.io/scrape=true"
      - "prometheus.io/port=8000"
      - "prometheus.io/path=/metrics"
    depends_on:
      - prometheus
      - jaeger

volumes:
  prometheus-data:
  grafana-data:
  loki-data:
  jaeger-data:

Complete observability stack with all three pillars

5. Alerting Best Practices

# alertmanager.yml
route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'pagerduty'
  routes:
    - match:
        severity: critical
      receiver: pagerduty
      continue: true
    - match:
        severity: warning
      receiver: slack_warnings

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: <PD_SERVICE_KEY>
        severity: critical
  
  - name: 'slack_warnings'
    slack_configs:
      - api_url: <SLACK_WEBHOOK>
        channel: '#alerts'
        title: '⚠️ Warning Alert'
        text: |
          *Alert:* {{ .CommonAnnotations.summary }}
          *Description:* {{ .CommonAnnotations.description }}
          *Severity:* {{ .CommonLabels.severity }}
          
  - name: 'email'
    email_configs:
      - to: 'oncall@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alerts'
        auth_password: '<PASSWORD>'

# PagerDuty configuration
# Better: Use Opsgenie, Slack, Teams, or custom webhooks

Alertmanager configuration for routing alerts

6. SLI/SLO/SLA Definitions

# Service Level Objectives (SLOs)
# SLI: Service Level Indicator (metric)
# SLO: Service Level Objective (target)
# SLA: Service Level Agreement (contract)

slo_definitions:
  availability:
    sli: "(total_requests - error_requests) / total_requests"
    target: 0.999  # 99.9% availability
    window: "30d"
    
  latency:
    sli: "histogram_quantile(0.95, http_request_duration_seconds)"
    target: 0.5  # 500ms
    window: "30d"
    
  throughput:
    sli: "rate(http_requests_total[1m])"
    target: 1000  # 1000 requests/second
    window: "5m"

# Error budget calculation
# error_budget = (1 - sli_value) / (1 - slo_target)
# If error budget < 0, you're violating SLO

# Prometheus recording rules for SLOs
- record: "slo:availability:ratio_rate5m"
  expr: |
    sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

- record: "slo:availability:error_budget"
  expr: |
    (1 - slo:availability:ratio_rate5m) / (1 - 0.999)

# Alert when error budget is almost exhausted
- alert: ErrorBudgetBurnRate
  expr: slo:availability:error_budget < 0.1
  annotations:
    summary: "Error budget is almost exhausted"

SLI/SLO definitions and error budgets

7. Common Errors & Solutions

8. Summary Checklist

9. Next Steps

Advanced observability topics to learn next:

eBPF: Zero-instrumentation observability with Cilium, Pixie
OpenTelemetry Collector: Unified telemetry processing
Service Mesh Observability: Istio telemetry and Kiali
ML for Observability: Anomaly detection, predictive scaling
Chaos Engineering: Testing observability with controlled failures

Practice resources:

Recommended books:

"Distributed Systems Observability" by Cindy Sridharan
"Monitoring Distributed Systems" by Google SRE Book
"The Art of Monitoring" by James Turnbull

Tags: monitoring, observability, prometheus, grafana, logging, metrics, devops

Gataya Med

DevOps Engineer & Backend Developer. Sharing insights on cloud, automation, and scalable systems.

gatayaMed View Profile @https://github.com/gatayaMed

Comments (0)

Sarah Chen January 29, 2025

This is exactly what I needed! The initContainer approach solved our migration issues completely. Thanks for the detailed guide!

Monitoring & Observability: Building Complete System Visibility

1. Learning Objectives

2. Why Monitoring & Observability Matter

3. Core Concepts

Metrics with Prometheus

Installing Prometheus

Instrumenting Applications with Prometheus Client

PromQL (Prometheus Query Language)

Grafana Dashboards

Logging with Loki

Distributed Tracing with Jaeger

4. Complete Observability Stack with Docker Compose

5. Alerting Best Practices

6. SLI/SLO/SLA Definitions

7. Common Errors & Solutions

8. Summary Checklist

9. Next Steps

Gataya Med

Comments (0)

Leave a Comment

Table of Contents

About the Author

Gataya Med

Related Posts

Monitoring & Observability: Building Complete System Visibility

1. Learning Objectives

2. Why Monitoring & Observability Matter

3. Core Concepts

Metrics with Prometheus

Installing Prometheus

Instrumenting Applications with Prometheus Client

PromQL (Prometheus Query Language)

Grafana Dashboards

Logging with Loki

Distributed Tracing with Jaeger

4. Complete Observability Stack with Docker Compose

5. Alerting Best Practices

6. SLI/SLO/SLA Definitions

7. Common Errors & Solutions

8. Summary Checklist

9. Next Steps

Gataya Med

Comments (0)

Leave a Comment

Table of Contents

About the Author

Gataya Med

Related Posts

Never miss a post