Security and recovery
Back up authoritative state, rotate keys safely, control egress, and restore service.
Backup scope
Different stores have different authority:
| Store | Authority |
|---|---|
| PostgreSQL | Organizations, credentials metadata, grants, ledger, audit, jobs, schedules, application metadata |
| S3-compatible storage | Original uploads and derived durable objects |
| Weaviate | Rebuildable retrieval vectors, but required for immediate service restoration |
| FalkorDB | Semantic memory graphs |
| Redis | Admin-session validity only; loss requires re-login |
Back up PostgreSQL and object storage as the minimum truth set. Include vector
and graph stores when recovery time objectives do not allow full rebuilding.
Protect SECRETS_KEY and AUDIT_ANCHOR_KEY separately from data backups.
Restore order
- stop tenant traffic and workers;
- restore PostgreSQL;
- restore object storage;
- restore Weaviate and FalkorDB, or begin controlled rebuilds;
- supply the exact required key versions;
- start one engine replica and let migrations complete;
- verify
/readyz; - verify audit chains and reconcile billing;
- exercise a non-destructive tenant read and retrieval;
- resume workers, then traffic.
Do not declare recovery complete based only on process health. Verify non-empty tenant, ledger, audit-anchor, object, vector, and graph records through logical interfaces.
Rotate a tenant JWT key
- register a new public key with a new
kid; - start minting tokens with the new private key;
- wait beyond
JWT_MAX_TTL; - disable the old
kid; - verify old tokens fail and new tokens succeed.
Rotate an engine API key
Create the replacement first, update the consumer, verify traffic, then revoke the old key. Plaintext is never recoverable, so loss requires replacement.
Rotate SECRETS_KEY
Add the new highest-version key while retaining the old key:
2:<new-key>,1:<old-key>Restart, then re-seal stored credentials:
POST /v1/admin/secrets/resealReview the checked and resealed counts. Remove the old version only after every sealed column is confirmed migrated. Dropping it early makes remaining blobs unrecoverable.
Changing SECRETS_KEY also changes the derived admin-cookie signing key and
invalidates existing admin sessions.
Rotate AUDIT_ANCHOR_KEY
Verify chains with the old key immediately before rotation. Deploy the new key, then re-sign provably intact anchors:
POST /v1/admin/audit/reanchorThe operation refuses to sign a chain whose rows do not reproduce its exact count and head hash.
Egress policy
Allow only destinations used by enabled capabilities:
- configured model, embedding, guardrail, router, and reranker providers;
- approved crawl targets;
- registered webhook endpoints;
- Gmail, Google OAuth, Microsoft identity, and Microsoft Graph when enabled;
- registered external MCP server URLs;
- internal sidecars and observability collector.
The crawler and webhook clients include SSRF-safe dialing, but production network policy remains a deployment responsibility.
Vulnerability response
Maintain a documented intake, severity classification, remediation SLA, release-block policy, and time-bounded exception process. Scan the Go module, container image, sidecar dependencies, and JavaScript documentation/admin assets. A known reachable high-severity vulnerability should block release.
Recovery drills
Run disposable-stack restore and key-rotation drills regularly. A drill is complete only when audit verification, billing reconciliation, tenant isolation, source-object access, retrieval, and memory behavior are proven through the restored system.