Operations

Go-live checklist

The deployment, identity, data, observability, and recovery checks required before production traffic.

Use this as the release gate for a new environment. The linked pages contain the commands and detailed procedures.

Deployment

  • ENVIRONMENT=production is set.
  • PostgreSQL uses distinct migration, application, and system roles.
  • S3-compatible object storage, Weaviate, Redis, Docling, and required sidecars use durable production endpoints.
  • TLS terminates before the engine and PUBLIC_URL is the externally reachable HTTPS origin.
  • Every secret is injected by the deployment platform; none is in an image, repository, command line, or log.

Verify values and ranges against Configuration.

Identity and tenant onboarding

  • Keycloak and its database are backed up.
  • The platform realm contains operators only; each application has its own realm.
  • The engine's Keycloak service-account credential is configured and its rotation procedure is owned.
  • Every organization is bound, has a machine credential or human membership, has credits, and has at least one priced model grant.
  • A machine token completes one governed call and an end-user token completes one user-owned call.

Use Identity provider and Platform setup.

Health and observability

  • /healthz proves process liveness.
  • /readyz is green behind the production load balancer.
  • /v1/readiness is green for a representative organization.
  • Structured logs retain request_id and traces reach the configured OTLP collector.
  • Alerts cover readiness, authentication denials, billing reconciliation, failed ingestion, guardrail dependency failures, and exhausted capacity.

See Health and observability.

Security and recovery

  • The egress allowlist covers only the documented provider and sidecar destinations.
  • A PostgreSQL, object-store, vector-store, and Keycloak backup set has been taken together.
  • Restore has been rehearsed on disposable infrastructure.
  • SECRETS_KEY, AUDIT_ANCHOR_KEY, tenant machine credentials, and the Keycloak service-account credential each have an owner and rotation path.
  • Audit verification and billing reconciliation are green after the rehearsal.

See Security and recovery.

Release and rollback

  1. Record the engine image digest and database migration version.
  2. Back up before any schema or Keycloak version change.
  3. Deploy one environment or replica and run the readiness and representative tenant checks above.
  4. Promote traffic only after those checks pass.
  5. Roll back the image only when its database version is compatible. A destructive down migration or Keycloak schema rollback requires restoring the paired backup, not merely selecting an older image.

Enable Blinq roles only after old workers stop

Deploy the Blinq-capable engine and complete its migrations before enabling hiring in any consuming application. Built-in templates ship with the Engine and need no SaaS publication. Stop every older engine replica and worker first: an older worker does not enforce the saved workflow state. A mixed fleet is not a safe rollout for governed jobs.

Once governed hires, bound conversations or workflow runs exist, rolling back to a pre-Blinq engine is unsafe even though the schema additions are additive. Keep an engine that enforces Blinq governance, or deploy a forward fix. Do not use an older image merely because it can still connect to the database. See the Blinq contract.

On this page