Monitoring - MQUAD Platform

MQUAD uses a comprehensive observability stack based on OpenTelemetry, Prometheus, Grafana, Loki, and Jaeger.

Stack Overview

Tool	Port	Purpose
Prometheus	9090	Metrics collection and alerting
Grafana	3001	Metrics and log visualization
Loki	3100	Log aggregation
Jaeger	16686	Distributed tracing
OpenTelemetry Collector	4317	Trace/metric/log collection

The monitoring stack is defined in monitoring-stack/:

monitoring-stack/
├── grafana/
│   ├── dashboards/       # Pre-built dashboard JSON
│   └── provisioning/     # Datasource configs
├── loki/                 # Log storage config
├── otel-collector/       # OpenTelemetry config
├── prometheus/           # Scrape configs
└── promtail/             # Log shipping config

Health Check Endpoints

Every service exposes a health endpoint:

curl http://localhost:8000/api/v1/health/
# {"status": "healthy", "service": "backend", ...}

curl http://localhost:8001/api/v1/health
# {"status": "healthy", "service": "survey-service"}

Prometheus Metrics

Services expose metrics at /metrics (when Prometheus client is available):

# Scalar service metrics endpoint
@app.get("/metrics")
async def metrics():
    if PROMETHEUS_AVAILABLE:
        return generate_latest()
    return "Prometheus not available"

Default scrape configuration:

# prometheus/prometheus.yml
scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['traefik:8090']
  - job_name: 'backend'
    static_configs:
      - targets: ['backend:8000']

Grafana Dashboards

Pre-built dashboards are provisioned automatically via monitoring-stack/grafana/provisioning/:

Service Health: Uptime, response times, error rates per service
Database: Connection pool, query performance
Redis: Cache hit rates, memory usage
Celery: Queue lengths, task durations, failure rates

Distributed Tracing

OpenTelemetry is configured in each service:

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
FastAPIInstrumentor.instrument_app(app)