Data is useless without visualization. Grafana transforms Prometheus metrics, Loki logs, and other data sources into beautiful dashboards that tell the story of your infrastructure. In this lesson, you'll learn to create dashboards that help you understand system behavior at a glance.

1. Learning Objectives

By the end of this lesson, you will be able to:

  • Install and configure Grafana
  • Connect data sources (Prometheus, Loki, CloudWatch)
  • Create dashboards with multiple panel types
  • Write dashboard variables for dynamic filtering
  • Configure alerts from dashboards
  • Share and export dashboards

2. Why This Matters

Real-world scenario: You have terabytes of metric data. Without visualization, finding patterns is impossible. Grafana dashboards let you see trends, anomalies, and relationships instantly.

3. Core Concepts

Grafana Installation

# Docker Compose with Grafana
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-clock-panel
    volumes:
      - grafana-data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    restart: unless-stopped

volumes:
  grafana-data:
EOF

# Run Grafana
docker-compose up -d

# Access at http://localhost:3000
# Login: admin / admin
Grafana installation with Docker

Data Sources Configuration

# provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
    jsonData:
      timeInterval: 15s
      queryTimeout: 30s
      httpMethod: POST

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: true
    jsonData:
      maxLines: 1000

  - name: CloudWatch
    type: cloudwatch
    access: proxy
    jsonData:
      authType: keys
      defaultRegion: us-east-1
    secureJsonData:
      accessKey: "${AWS_ACCESS_KEY_ID}"
      secretKey: "${AWS_SECRET_ACCESS_KEY}"

  - name: Elasticsearch
    type: elasticsearch
    access: proxy
    url: http://elasticsearch:9200
    database: "[logs-]YYYY.MM.DD"
    jsonData:
      timeField: "@timestamp"
      esVersion: 7
      interval: Daily
Data source provisioning

Dashboard Variables

# Dashboard variable configuration (JSON)

# Variable: environment
{
  "name": "environment",
  "type": "query",
  "query": "label_values(up, environment)",
  "datasource": "Prometheus",
  "refresh": 1,
  "includeAll": true,
  "multi": true
}

# Variable: instance
{
  "name": "instance",
  "type": "query",
  "query": "label_values(up{environment=~\"$environment\"}, instance)",
  "datasource": "Prometheus",
  "refresh": 1,
  "includeAll": true
}

# Variable: namespace
{
  "name": "namespace",
  "type": "query",
  "query": "label_values(kube_namespace_labels)",
  "datasource": "Prometheus",
  "refresh": 1
}

# Constant variable
{
  "name": "datacenter",
  "type": "constant",
  "value": "us-east-1",
  "hide": 2
}

# Custom variable
{
  "name": "interval",
  "type": "interval",
  "auto": true,
  "auto_step": 60,
  "values": ["1m", "5m", "15m", "1h"]
}

# Using variables in queries
# rate(http_requests_total{environment=~"$environment", instance=~"$instance"}[$interval])
Dashboard variables

Panel Types and Queries

# Time Series / Graph Panel
# Request rate over time
rate(http_requests_total{environment="production"}[5m])

# Query with legend formatting
rate(http_requests_total{method="GET", endpoint="/api/users"}[5m])
# Legend: {{method}} {{endpoint}}

# Stat Panel (single number)
# Total requests (last 24 hours)
sum(increase(http_requests_total[24h]))

# Gauge Panel
# Current CPU usage
avg(rate(node_cpu_seconds_total{mode="user"}[5m])) * 100

# Table Panel
# Top error endpoints
topk(10,
  sum by (endpoint) (rate(http_requests_total{status=~"5.."}[5m]))
)

# Heatmap Panel
# Request duration distribution
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)

# Pie Chart Panel
# Request distribution by status code
sum by (status) (increase(http_requests_total[1h]))

# Bar Gauge Panel
# Memory usage by instance
topk(10, max by (instance) (node_memory_MemTotal_bytes - node_memory_MemFree_bytes))

# Logs Panel (Loki)
# Show error logs
{app="myapp"} |= "error" | json | line_format "{{.message}}"

# Alert List Panel
# Show recent alerts
# Use built-in alert list panel

# Text Panel
# Markdown documentation
## System Overview
Last updated: {{ now | date }}
Environment: $environment
Grafana panel types and queries

Dashboard JSON Model

{
  "dashboard": {
    "title": "Application Performance Dashboard",
    "tags": ["application", "performance"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "targets": [
          {
            "expr": "sum by (endpoint) (rate(http_requests_total[$interval]))",
            "legendFormat": "{{endpoint}}",
            "refId": "A"
          }
        ],
        "xaxis": {"mode": "time"},
        "yaxes": [
          {"format": "reqps", "label": "Requests per second"},
          {"format": "short"}
        ],
        "tooltip": {"shared": true, "sort": 2}
      },
      {
        "title": "Error Rate",
        "type": "stat",
        "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))",
            "refId": "A"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percentunit",
            "thresholds": {
              "steps": [
                {"color": "green", "value": null},
                {"color": "yellow", "value": 0.01},
                {"color": "red", "value": 0.05}
              ]
            },
            "color": {"mode": "thresholds"}
          }
        }
      },
      {
        "title": "95th Percentile Latency",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[$interval])) by (le, endpoint))",
            "legendFormat": "{{endpoint}}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {"format": "s", "label": "Latency"}
        ]
      },
      {
        "title": "Cache Hit Ratio",
        "type": "gauge",
        "gridPos": {"h": 8, "w": 6, "x": 12, "y": 8},
        "targets": [
          {
            "expr": "app_cache_hit_ratio",
            "refId": "A"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percentunit",
            "min": 0,
            "max": 1,
            "thresholds": {
              "steps": [
                {"color": "red", "value": null},
                {"color": "yellow", "value": 0.7},
                {"color": "green", "value": 0.9}
              ]
            }
          }
        }
      },
      {
        "title": "Top Error Endpoints",
        "type": "table",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 16},
        "targets": [
          {
            "expr": "topk(5, sum by (endpoint) (increase(http_requests_total{status=~\"5..\"}[1h])))",
            "format": "table",
            "refId": "A"
          }
        ],
        "columns": [
          {"text": "Endpoint", "value": "endpoint"},
          {"text": "Errors", "value": "Value"}
        ]
      }
    ],
    "templating": {
      "list": [
        {
          "name": "interval",
          "type": "interval",
          "auto": true,
          "auto_step": 60,
          "query": "1m,5m,15m,30m,1h,6h,12h,24h",
          "current": {"value": "5m", "text": "5m"}
        },
        {
          "name": "environment",
          "type": "query",
          "query": "label_values(http_requests_total, environment)",
          "refresh": 1,
          "includeAll": true,
          "current": {"value": "production", "text": "production"}
        }
      ]
    },
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "refresh": "30s",
    "version": 1
  }
}
Complete dashboard JSON configuration

Alerts in Grafana

# Alert rule configuration (via UI or provisioning)

# provisioning/alerting/alert-rules.yml
apiVersion: 1
groups:
  - name: application_rules
    folder: Application
    orgId: 1
    rules:
      - title: High Error Rate Alert
        condition: A
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: |
                sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
              instant: true
            relativeTimeRange:
              from: 300
              to: 0
        for: 5m
        annotations:
          summary: High error rate detected
          description: Error rate is {{ $values.A.Value }}%
        labels:
          severity: critical
          team: backend
        isPaused: false

# Contact points
# provisioning/alerting/contact-points.yml
apiVersion: 1
contactPoints:
  - name: slack
    receivers:
      - uid: slack-webhook
        type: slack
        settings:
          url: https://hooks.slack.com/services/xxx/yyy/zzz
          title: '{{ .CommonLabels.alertname }}'
          text: |
            {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }}
              *Description:* {{ .Annotations.description }}
              *Severity:* {{ .Labels.severity }}
              *Time:* {{ .StartsAt }}
            {{ end }}

  - name: pagerduty
    receivers:
      - uid: pagerduty-integration
        type: pagerduty
        settings:
          integrationKey: YOUR_PD_KEY

  - name: email
    receivers:
      - uid: email-alerts
        type: email
        settings:
          addresses:
            - oncall@example.com

# Notification policies
# provisioning/alerting/notification-policies.yml
apiVersion: 1
policies:
  - context: org
    receiver: slack
    group_by: ['alertname', 'severity']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - match:
          severity: critical
        receiver: pagerduty
        group_wait: 0s
        continue: true
      - match:
          severity: warning
        receiver: slack
        continue: false
Grafana alerting configuration

4. Complete Project: Production Dashboard

# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: 'Default'
    orgId: 1
    folder: 'Production'
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /etc/grafana/dashboards

# Save dashboard JSON to /etc/grafana/dashboards/production.json
# Restart Grafana to load

docker-compose restart grafana

# Import pre-built dashboards
# Dashboard IDs from Grafana.com:
# 1860 - Node Exporter Full
# 3662 - Prometheus Stats
# 11835 - Kubernetes Views
# 9614 - Flask Application Monitoring
# 13770 - PostgreSQL Database
# 12884 - Redis Dashboard

grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-clock-panel
grafana-cli plugins install grafana-worldmap-panel
Dashboard provisioning

5. Common Errors & Solutions

6. Summary Checklist

7. Next Steps

Next lesson: Monitoring Lesson 11.4: Logging with Loki & ELK Stack

Tags: grafana, dashboards, visualization, monitoring, observability

Gataya Med

DevOps Engineer & Backend Developer. Sharing insights on cloud, automation, and scalable systems.

gatayaMed View Profile @https://github.com/gatayaMed

Comments (0)

Sarah Chen February 24, 2025

This is exactly what I needed! The initContainer approach solved our migration issues completely. Thanks for the detailed guide!

Reply

Leave a Comment