Health and observability

Monitor readiness, requests, provider calls, billing, workers, and audit evidence.

Health surfaces

Endpoint	Authentication	Meaning
`GET /healthz`	public	Process liveness
`GET /readyz`	public	Dependency-aware readiness
`GET /`	public	Human-readable deployment status
`GET /admin/system`	admin session	Detailed operator view

Public surfaces use functional dependency names such as database, search-index, storage, and extractor. They do not disclose product or vendor topology.

Critical dependency failure makes readiness fail. Non-critical optional dependencies can report degraded state while ordinary chat stays healthy. Requests that explicitly require an unavailable dependency still fail closed.

Request correlation

Every response includes:

X-Request-ID: <uuid>

Persist this value in client logs. The same identifier correlates:

HTTP transport and error;
provider call;
credit reservation and settlement;
audit event;
conversation persistence;
trace spans.

Successful OpenAI and Anthropic response IDs also embed the transport request identifier.

Structured logs

Set LOG_LEVEL to info in normal production. Use debug only for bounded diagnostic windows because it includes dependency wiring and higher-volume detail.

Centralize logs with at least:

timestamp and level;
request ID;
organization and user identifiers where present;
route and semantic outcome;
durable job or run ID;
connector ID without credential material.

Never log authorization headers, provider keys, OAuth tokens, prompts, tool arguments, or attachment data.

OpenTelemetry

Configure an OTLP collector:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
OTEL_EXPORTER_OTLP_PROTOCOL=http
OTEL_SERVICE_NAME=deeplinq-engine

Trace model, embedding, retrieval, sidecar, and durable-work latency. Sampling and retention belong to the deployment's observability platform.

Operational views

/admin overview: engine and dependency state;
/admin/usage: organization balances and month-to-date model usage;
GET /v1/admin/usage: machine-readable platform usage;
GET /v1/usage: tenant balance and breakdown;
ingestion run endpoints: durable document progress;
run endpoints: agent checkpoint and terminal status;
schedule events: authoritative delivery attempts.

Audit verification

Verify tamper-evident chains:

GET /v1/admin/audit/verify

Export evidence:

GET /v1/admin/audit/export
GET /v1/admin/audit/export?format=json
GET /v1/admin/audit/export?format=csv&org=<org-id>

The default export is NDJSON. CSV neutralizes spreadsheet-formula cells.

Treat a failed verification as an incident: stop destructive maintenance, preserve database and anchor state, record the response and request ID, and investigate before re-anchoring.

Alerting baseline

Alert on:

readiness failure or restart loops;
PostgreSQL, object, or vector dependency latency;
provider rejection or unreachable verification state;
sustained 402, 429, 502, or 503 rates;
durable job poison/abandonment;
agent approval timeouts and terminal failures;
webhook delivery exhaustion;
billing reconciliation drift;
audit verification failure.