Grafana + Prometheus Monitoring Stack
Prometheus is a time-series database that scrapes metrics from HTTP endpoints. Grafana visualizes them. Alertmanager sends alerts. The combination has become the de facto open-source monitoring stack — every cloud-native project ships a /metrics endpoint by default.
How it works in one paragraph
Targets expose a plain text /metrics endpoint. Prometheus scrapes them every 15-30 seconds, stores the data with timestamps and labels, and lets you query it with PromQL. Grafana queries Prometheus for dashboards. Alertmanager receives alert events from Prometheus rules and routes them to email/Slack/PagerDuty.
Docker compose
services:
prometheus:
image: prom/prometheus:v2.54.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./alerts.yml:/etc/prometheus/alerts.yml:ro
- prom-data:/prometheus
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=30d
ports: ["9090:9090"]
node_exporter:
image: prom/node-exporter:v1.8.2
pid: host
volumes:
- /:/host:ro,rslave
command: --path.rootfs=/host
network_mode: host
grafana:
image: grafana/grafana:11.3.0
environment:
GF_SECURITY_ADMIN_PASSWORD: changeme
volumes:
- grafana-data:/var/lib/grafana
ports: ["3000:3000"]
alertmanager:
image: prom/alertmanager:v0.27.0
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
ports: ["9093:9093"]
volumes:
prom-data:
grafana-data:prometheus.yml
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files:
- alerts.yml
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: node
static_configs:
- targets: ['host.docker.internal:9100']
labels: { env: prod, host: web1 }
- job_name: app
metrics_path: /metrics
static_configs:
- targets: ['app1:8080', 'app2:8080']Alert rules that don't lie
groups:
- name: server
rules:
- alert: HighMemoryUsage
expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
for: 10m
labels: { severity: warning }
annotations:
summary: "Memory > 90% on {{ $labels.host }}"
- alert: DiskAlmostFull
expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1
for: 30m
labels: { severity: critical }
- alert: HighCPU
expr: 1 - rate(node_cpu_seconds_total{mode="idle"}[5m]) > 0.85
for: 15m
labels: { severity: warning }Three principles for alerts that don't burn out your on-call: a for: duration (avoid flaps), severity labels (not everything is a page), and human-readable annotations (the responder shouldn't have to derive context).
Dashboards
Skip building dashboards by hand. Import the community ones: Grafana → Dashboards → Import → enter ID. 1860 (Node Exporter Full) and 10000 (Linux Stats) cover most servers; specific stacks have their own (Postgres 9628, Redis 763, nginx 12708).
What to instrument in your own app
The four golden signals (Google SRE book): latency, traffic, errors, saturation. For an HTTP service: request rate, request duration histogram by status code, in-flight requests, queue depth. Every Prometheus client library makes this a few lines.