Observability & audit¶

Traces, metrics and structured logs — plus a hash-chained audit log.

Traces — OpenTelemetry¶

Every request opens a span. TracingMiddleware runs first, so trace_id exists before the request-ID logger emits anything.

Setting	Value
`OTEL_EXPORTER`	`otlp` · `stdout` · empty (off)
`OTEL_SAMPLE_RATIO`	Sampling ratio

Automatic instrumentation:

pgx — every database query becomes a span through otelpgx, so slow queries are visible in traces.
Redis — redisotel.
HTTP — chi middleware.

Metrics — Prometheus¶

MetricsMiddleware collects request counters and a latency histogram.

GET /metrics
Authorization: Bearer <OBSERVABILITY_TOKEN>   ← required in production

The ObservabilityGate middleware protects /metrics and /swagger/doc.json in production. If no token is configured these endpoints are closed, not left open.

Logs — Zap¶

Structured JSON logs. Every line carries request_id, trace_id and a category.

Category	What
`server`	Startup / shutdown, boot guards
`access`	One-line access log
`recovery`	Recovered panics

What is never logged

The user's AI question text, tokens, passwords, integration credentials. Knowledge search logs only the mode, hit count, best score and slugs — never the question itself.

Access log¶

AccessLogMiddleware emits one line per request: method, path, status, duration, client IP (resolved through TRUSTED_PROXIES) and the request ID.

Gateway telemetry¶

Proxied calls are recorded in gateway_request_logs:

Endpoint	What
`GET /v1/gateway/overview`	Request counts, error rate, latency
`GET /v1/gateway/logs`	Detailed log

UI: Admin → Gateway → Overview / Logs.

Audit log¶

A hash-chained, append-only log. Each row contains the hash of the previous one, so altering a row in the middle breaks the chain.

row N-1:  hash_{N-1} = H(payload_{N-1} ‖ hash_{N-2})
row N:    hash_N     = H(payload_N     ‖ hash_{N-1})

Endpoint	Permission	What
`GET /v1/audit/`	admin	Read the log (filtered)
`GET /v1/audit/verify`	admin	Verify chain integrity

UI: Admin → Audit.

What gets recorded: role and permission changes, user mutations, every super-admin action, client secret rotations, gateway configuration changes and signing operations.

What append-only means here

The app role holds no UPDATE / DELETE privileges on the audit table. The RLS policies grant read access to service and admin.

Security events¶

A separate stream recording suspicious activity:

Endpoint	Permission	What
`POST /v1/security/events`	authenticated	Record an event (can come from the client)
`GET /v1/security/events`	admin	Read

UI: Admin → Security. The security_events table is RLS-protected.

Health checks¶

Path	What
`/health`	Liveness — checks the database and Redis connections

In compose both api and web have healthchecks, and the deploy script waits for healthy.

platform-core also ships a standalone healthcheck binary:

go run github.com/gerege-systems/private-gerege-core/cmd/healthcheck

Graceful shutdown¶

On SIGTERM components close in this order: HTTP server → rate limiter → pgx pool → Redis → tracer. In-flight requests are allowed to finish.

Where to look for what¶

Question	Where
"Why is this request slow?"	Traces (span breakdown)
"How many 5xx did we serve?"	`/metrics` or Gateway → Overview
"Who changed this role?"	Admin → Audit
"Do the sign-in attempts look normal?"	Admin → Security
"Which app is hammering the gateway?"	Gateway → Logs