Distributed Tracing with OpenTelemetry: The Third Pillar of Observability

Metrics tell you what is broken. Logs tell you why. Traces tell you where. In this lesson, you will instrument a Python Flask application with OpenTelemetry, export traces to Jaeger, and learn to follow requests across microservice boundaries using W3C Trace Context propagation.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Explain what distributed tracing is and why it matters for microservices
• Instrument a Python application with OpenTelemetry SDK
• Export traces to Jaeger for visualization and analysis
• Propagate trace context across HTTP service boundaries
• Identify performance bottlenecks using trace spans and latency insights
• Integrate traces with metrics and logs for full observability

2. Why This Matters

Real-world scenario: Your e-commerce platform has five microservices: a web frontend, an auth service, a product catalog, a cart service, and a payment gateway. A customer reports that checkout takes over 10 seconds. Your Prometheus metrics show each service's CPU and memory are fine. Your Loki logs show no errors. Where is the delay? Without distributed tracing, you have no way to follow a single request across these services. A trace reveals the truth: the payment gateway is experiencing 3-second gRPC call latencies, but only for one specific credit card processor. Distributed tracing connects the dots that metrics and logs leave scattered.

3. Core Concepts

What is a Trace?

A trace represents the journey of a single request as it travels through a distributed system. It consists of:

• Trace ID — a unique identifier assigned to the root request that propagates across all services
• Spans — individual units of work within a trace, each with a start time, duration, and optional metadata
• Span Context — the parent-child relationship between spans, forming a tree structure

The Three Pillars of Observability

Distributed tracing completes the observability triad:

• Metrics (Prometheus) — what is happening? High-level rates, errors, and saturation
• Logs (Loki/ELK) — why is it happening? Detailed event records
• Traces (OpenTelemetry) — where is it happening? End-to-end request flow across services

OpenTelemetry Architecture

OpenTelemetry (OTel) is the industry-standard, vendor-neutral framework for telemetry data collection. Key components:

• OTel SDK — libraries in your application that create and manage spans
• OTel Collector — a standalone agent that receives, processes, and exports telemetry data
• Exporters — send trace data to backends like Jaeger, Zipkin, or Datadog
• Instrumentation Libraries — auto-instrumentation for frameworks like Flask, Django, gRPC, and HTTP clients

4. Hands-On Practice

Step 1: Setting Up OpenTelemetry with Python

Create a new directory and install the OpenTelemetry packages:

mkdir ~/otel-demo
cd ~/otel-demo
python3 -m venv venv
source venv/bin/activate

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp
pip install opentelemetry-instrumentation-flask
pip install opentelemetry-instrumentation-requests
pip install flask requests

Step 2: Instrument a Simple Flask Application

Create a Flask app that calls an external API. OpenTelemetry auto-instrumentation will wrap the Flask request handler and the outgoing HTTP call with spans:

# app.py
import os
import requests
from flask import Flask, jsonify

# Import OpenTelemetry
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

app = Flask(__name__)

# Set up tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure exporter (Jaeger via OTLP)
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4318/v1/traces"
)
span_processor = BatchSpanProcessor(exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Auto-instrument Flask and requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/")
def home():
    return jsonify({"message": "Hello, Observability!"})

@app.route("/fetch-posts")
def fetch_posts():
    # Create a custom span for business logic
    with tracer.start_as_current_span("fetch_posts") as span:
        span.set_attribute("endpoint", "/posts")
        response = requests.get("https://jsonplaceholder.typicode.com/posts/1")
        span.set_attribute("http.status_code", response.status_code)
        return jsonify(response.json())

if __name__ == "__main__":
    app.run(port=5000)

Step 3: Run Jaeger Backend with Docker

Jaeger is an open-source distributed tracing platform. Run it with its all-in-one image, which includes the UI, collector, and query service:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:1.57

This single command starts Jaeger with:

• Port 16686 — Jaeger UI (browser dashboard)
• Port 4318 — OTLP HTTP endpoint (where our app sends traces)
• Port 4317 — OTLP gRPC endpoint (alternative protocol)

Step 4: View Traces in Jaeger UI

Open http://localhost:16686 in your browser. In the Jaeger UI:

• Select service "app" from the dropdown
• Click "Find Traces"
• You will see one trace per request, with a waterfall diagram showing the root span (Flask request) and child spans (outgoing HTTP calls)
• Click on any span to see its duration, tags, and process metadata

Trace Waterfall View (conceptual):

  Service: app  |  Trace ID: abc123def456
  -----------------------------------------------------------------
  [0ms] GET /fetch-posts .................................. [450ms]
    ├─ [10ms] Flask dispatch_request ...................... [440ms]
    │    └─ [15ms] fetch_posts (custom span) .............. [425ms]
    │         └─ [20ms] HTTP GET jsonplaceholder ......... [400ms]
    │              └─ [400ms] external service ............. [0ms]
  -----------------------------------------------------------------
  Total: 450ms  |  External call: 400ms (89% of total)

Step 5: Multi-Service Tracing with Context Propagation

In real microservice architectures, traces must propagate across service boundaries. This is done via HTTP headers:

# service_a.py - Frontend service
import requests
from flask import Flask
from opentelemetry import propagate
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/process-order")
def process_order():
    # OpenTelemetry automatically injects trace headers into outgoing requests
    response = requests.get("http://service-b:5001/validate")
    return {"status": "processed", "backend": response.json()}

# service_b.py - Backend validation service
from flask import Flask, request
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
tracer = trace.get_tracer(__name__)

@app.route("/validate")
def validate():
    with tracer.start_as_current_span("validate_order") as span:
        span.set_attribute("order.valid", True)
        return {"valid": True, "user_id": 42}

The magic happens via W3C Trace Context headers. When Service A calls Service B, OpenTelemetry automatically injects these HTTP headers:

• traceparent — carries the trace ID, span ID, and trace flags
• tracestate — carries vendor-specific trace data

6. Common Errors & Solutions

7. Summary Checklist

• Installed OpenTelemetry SDK, Flask instrumentation, and OTLP exporter
• Instrumented a Flask app with auto-instrumentation
• Created a custom span with attributes for business logic
• Run Jaeger all-in-one with Docker
• Viewed traces in the Jaeger UI waterfall diagram
• Understood W3C Trace Context propagation headers
• Identified a latency bottleneck using the trace timeline
• Configured multi-service tracing across two Flask apps

8. Practice Exercise

Challenge: Build a Three-Service Trace Chain

Create three Flask services communicating in a chain:

1. Service A (port 5000): /search?q=query — calls Service B
2. Service B (port 5001): /search-proxy?q=query — calls Service C
3. Service C (port 5002): /results?q=query — returns a mock JSON response

Requirements:

• Each service must use FlaskInstrumentor and RequestsInstrumentor
• All three must export traces to the same Jaeger instance
• A single curl localhost:5000/search?q=otel should produce ONE trace with THREE spans (one per service)
• Add a custom attribute search.query to the Service A span with the query value

Bonus: Add a time.sleep(1) in Service C and confirm the extra delay appears in the Jaeger waterfall.

9. Next Steps

You have now completed the observability triad: metrics (Prometheus), logs (Loki/ELK), alerting, and distributed tracing. In the next lesson, we will explore Service Level Objectives (SLOs) and Error Budgets — how to define reliability targets using the data from all three observability pillars. You will learn to calculate SLIs, set SLO targets, and use error budgets to balance velocity with reliability.