# Monitoring {#monitoring}


MQUAD uses a comprehensive observability stack based on OpenTelemetry, Prometheus, Grafana, Loki, and Jaeger.


## Stack Overview {#stack-overview}


| Tool | Port | Purpose |
|------|------|---------|
| **Prometheus** | 9090 | Metrics collection and alerting |
| **Grafana** | 3001 | Metrics and log visualization |
| **Loki** | 3100 | Log aggregation |
| **Jaeger** | 16686 | Distributed tracing |
| **OpenTelemetry Collector** | 4317 | Trace/metric/log collection |

The monitoring stack is defined in `monitoring-stack/`:

```
monitoring-stack/
├── grafana/
│   ├── dashboards/       # Pre-built dashboard JSON
│   └── provisioning/     # Datasource configs
├── loki/                 # Log storage config
├── otel-collector/       # OpenTelemetry config
├── prometheus/           # Scrape configs
└── promtail/             # Log shipping config
```


## Health Check Endpoints {#health-check-endpoints}


Every service exposes a health endpoint:

```bash
curl http://localhost:8000/api/v1/health/
# {"status": "healthy", "service": "backend", ...}

curl http://localhost:8001/api/v1/health
# {"status": "healthy", "service": "survey-service"}
```


## Prometheus Metrics {#prometheus-metrics}


Services expose metrics at `/metrics` (when Prometheus client is available):

```python
# Scalar service metrics endpoint
@app.get("/metrics")
async def metrics():
    if PROMETHEUS_AVAILABLE:
        return generate_latest()
    return "Prometheus not available"
```

Default scrape configuration:

```yaml
# prometheus/prometheus.yml
scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['traefik:8090']
  - job_name: 'backend'
    static_configs:
      - targets: ['backend:8000']
```


## Grafana Dashboards {#grafana-dashboards}


Pre-built dashboards are provisioned automatically via `monitoring-stack/grafana/provisioning/`:

- **Service Health**: Uptime, response times, error rates per service
- **Database**: Connection pool, query performance
- **Redis**: Cache hit rates, memory usage
- **Celery**: Queue lengths, task durations, failure rates


## Distributed Tracing {#distributed-tracing}


OpenTelemetry is configured in each service:

```python
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
FastAPIInstrumentor.instrument_app(app)
```