Observability — Metrics, Logs, Dashboards, Tracing

Endpoints

Các endpoint này mount ngoài /api/v1 để infra probes có thể gọi không cần JWT.

Metrics exposed

Implementation: apps/cms/src/routes/metrics.ts + middleware withMetrics().

Backend: prom-client. Process default metrics are collected only when the runtime exposes a working Node process.cpuUsage() implementation; Workers/Wrangler stubs are skipped safely.

Request path labels are normalized before recording to avoid high-cardinality labels from UUIDs and numeric IDs.

Tracing / Apache SkyWalking POC

Docker/Node runtime supports an optional tracing bootstrap. It is disabled by default and controlled by environment variables:

LUMIBASE_TRACING_ENABLED=false
LUMIBASE_TRACING_PROVIDER=skywalking
LUMIBASE_SERVICE_NAME=lumibase-cms
OTEL_EXPORTER_OTLP_ENDPOINT=http://skywalking-oap:11800
LUMIBASE_TRACING_SAMPLING_RATIO=1

Enable the SkyWalking stack in Docker by including the optional compose override:

docker compose \
  -f docker/docker-compose.yml \
  -f docker/docker-compose.skywalking.yml \
  up --build

Disable tracing by omitting docker/docker-compose.skywalking.yml or setting LUMIBASE_TRACING_ENABLED=false.

Notes:

• Tracing is Node/Docker-only for this POC. Cloudflare Workers should use Cloudflare-native observability until a separate Workers tracing design exists.
• OpenTelemetry SDK bootstrap must run before the Hono app import so auto-instrumentations can patch supported Node modules early.
• Changing tracing provider/endpoint should be treated as a process restart operation; hot reload is intentionally out of scope for this POC.
• Request spans use normalized paths and never copy Authorization, cookies, raw query strings, or request bodies into attributes.

Structured logging

Mọi request log JSON với fields:

{
  "timestamp": "2025-...",
  "level": "info",
  "requestId": "req_xyz",
  "method": "GET",
  "path": "/api/v1/items/posts",
  "status": 200,
  "duration": 45,
  "siteId": "s_abc",
  "userId": "u_123"
}

• Cloudflare: log đẩy qua Workers Logpush ra R2/S3/external.
• Docker: log stdout, được Promtail/Loki hoặc runtime log driver thu thập.

Stack monitoring

Cloudflare mode

Dùng built-in services:

• Workers Analytics Engine — emit custom metrics, query qua dashboard.
• Workers Logpush — log shipping.
• Cloudflare Trace cho APM.

Documentation only — không tự host gì cả.

Docker mode

Metrics/log dashboard stack chạy qua:

docker compose -f docker/docker-compose.yml -f docker/docker-compose.monitoring.yml up

Tracing POC stack chạy thêm file docker/docker-compose.skywalking.yml: Config: docker/prometheus/prometheus.yml, docker/grafana/provisioning/, docker/grafana/dashboards/lumibase.json, docker/grafana/dashboards/agent-harness.json.

Pre-provisioned Grafana dashboard

Dashboard LumiBase (auto-loaded) bao gồm:

• Request rate (req/s) over time, broken by status class.
• Latency percentiles p50, p95, p99.
• Error rate (%) and top error codes.
• Queue depth for each queue (search-index, media-thumbnails, webhook).
• Cache hit ratio (%).
• DB pool utilization (active vs idle).
• CPU / Memory of CMS container.

Dashboard LumiBase Agent Harness bao gồm run success/fail rate, budget stop reason, tool latency, approval latency, evaluation outcome, token/cost estimate và dead-letter enqueue rate. Run fail lặp lại được đẩy vào queue agent-dead-letter khi runtime queue adapter khả dụng.

Backup monitoring

pg-backup service runs pg_dump daily at 02:00 UTC, uploads to MinIO/S3 với retention 7 daily + 4 weekly. Failures send notification qua webhook (configurable).

Config: docker/prometheus/prometheus.yml, docker/grafana/provisioning/, docker/grafana/dashboards/lumibase.json.

Health check details

GET /health response:

{
  "status": "healthy",
  "services": {
    "database": "healthy",
    "cache": "healthy",
    "search": "healthy",
    "storage": "healthy",
    "queue": "healthy"
  }
}

GET /health always returns 200 so it is safe as a liveness/diagnostic endpoint. GET /health/ready returns 503 when any dependency probe is unhealthy, so orchestrators can use it for readiness.

Probe timeout: 750ms per dependency.

Alerting

Khuyến nghị alert rules trong Prometheus:

• Error rate > 5% trong 5 phút.
• p95 latency > 500ms trong 10 phút.
• Health readiness fail.
• Queue job failures rising continuously.
• DB query duration p95 above service SLO.

Xem apps/docs/content/guides/tooling-recommendations.md cho managed alternatives (Datadog, Sentry, New Relic).