Skip to content
Logo

Deploy and publish

A production deployment runs the app for clinicians, Celld for durable data and work retries, and Eve for AI drafts. The public edge serves only the app. Celld's peer and operator listener stays private. Eve has no object-store or database credentials.

The single-host template in deploy/celld-single-host/ is the operational reference. It starts the Cell before its dependents, keeps the edge closed until readiness passes, and uses an immutable release directory for upgrade and rollback.

Keep storage lanes separate

LaneHoldsWho can use it
Celld fleet bucketCell data and native Workflow stateCelld only
Application object bucketImmutable clinical source, document, export, and workspace bytesApp; narrowly scoped Cell cleanup and backup readers
ETL evidence storeLanceDB reference tablesRead-only app identity; separate ETL writer
Backup vaultEncrypted database chunks, sealed application objects, and receiptsBackup and short-lived restore operators
Prepared-file cacheRebuildable verified copies for app and EveApp writes; Eve reads through the oncobrain-prepared group

Use distinct buckets and credentials for every lane. The app and Eve never receive vault credentials. Eve never receives fleet or application-object credentials. Object retention belongs on the encrypted backup vault, not on fleet storage or application staging and reclaim prefixes that require deletion.

The app records an exact key, digest, and size intent in the Cell before an immutable object write. The clinical transaction makes the resulting manifest authoritative. Cron reclaims expired, unused staged keys through a closed Files SDK broker. Do not give browser, app, or Eve processes bucket listing or broad delete permissions.

Start and release

For a normal local stack:

bun run dev

It uses a disposable Celld project and local S3rver. It is a development convenience and does not qualify a hosted fleet or recovery path.

For a production node:

bun run celld:deploy
bun run celld:start

Before the first native Workflow publication to a fleet, use the one-time adoption preflight:

bun --no-env-file run ./scripts/deployment-authority-preflight.ts --adopt-native-workflow

It refuses a nonempty Cell namespace, including internal Workflow cells, before publishing the fixed native-effect-workflow binding. Do not rename that binding or introduce an alias after it has state.

The release wrapper closes admissions and lets active work finish before creating a fresh encrypted source-release backup. It then publishes the candidate Worker, switches the release, checks readiness and deployment authority, and completes the candidate's first backup. Only then does it reopen the edge.

A verified release drains work, seals the source backup, and activates and verifies the candidate before clearing its recovery guard. A failed drain can resume the source. Guarded failures keep services sealed; restore requires a verified backup. After the guard clears, edge or timer failures are repaired on the accepted candidate.

The operator-run single-host protocol. The point at which a release fails determines its recovery path. Explore release and recovery.

Failure pointRecovery
Drain fails before the guard is writtenKeep the unchanged source release and reopen Caddy if it was previously active.
Guarded cutover fails before a verified backup receipt existsKeep services sealed and investigate the backup failure. The wrapper has no supported resume or rollback command for this state. Do not remove the guard or attempt restore without a verified backup ID and digest.
Guarded cutover fails after a verified backup receipt existsPreserve the guard and receipt. Use the verified source-release wrapper to restore that backup on a separately credentialed, empty deployment.
Edge, public health, or timers fail after the guard clearsThe candidate remains authoritative and backed up. Repair the failed services on that candidate; this failure does not require a clean restore.

Never remove the recovery guard by hand or point an older Worker at a live Cell after schema activation. Celld v0.4.1 data must not be opened by the older v0.4.0 binary. Use the immutable-release section of deploy/celld-single-host/README.md for the exact build, rehearsal, deploy, and restore commands. The Azure demo below has a separate deployment flow.

Verify the actual provider

Run the authority preflight and writable Celld probe against the exact fleet provider:

bun run deployment:validate-authority
bun run celld:diagnose

The preflight runs from a short-lived operator environment that has the separate storage principals. It checks bucket and credential separation. celld diagnose --json conditionally writes and deletes a small object under its probe/ prefix. It is evidence of usable conditional writes, not merely connectivity.

Also run bun run --cwd apps/oncobrain celld:qualify:provider with the deployed Files SDK settings. It proves an application-namespace upload, byte-for-byte readback, and deletion. It does not prove backup retention, independent IAM, outage recovery, or fleet durability; those need their own review and exercises.

Celld's app and internal listeners default to loopback. A multi-node installation must set CELLD_LISTEN, CELLD_INTERNAL_LISTEN, CELLD_ADVERTISE, and ONCOBRAIN_CELLD_PRIVATE_NETWORK_OVERLAY=1 together behind an operator-managed private, mutually authenticated network. The launcher rejects CELLD_UNSAFE_PUBLIC_ADVERTISE. Set CELLD_TRUST_FORWARDED_HEADERS=1 only when the trusted edge supplies both forwarded host and protocol.

Operational guards

The repository pins the Celld v0.4.1 binary and checks its platform-specific SHA-256 before deploy or start. The launcher sets CELLD_OPERATION_DEADLINE_MS=180000 and CELLD_HANDLER_BUDGET_S=180 because an Eve effect may take up to 120 seconds, followed by bounded reconciliation and settlement. Raising both values is allowed; lowering either is refused.

A public-edge replacement is a separate change. Before cutover, prove that the Google Workspace callback is allowed only at its exact callback path and query delimiter, while sibling and prefix paths still follow normal managed rules. Test ordinary routes, TLS, the WAF, and a live sign-in callback before retiring the existing edge.

Required configuration

Put secrets in the respective service environment, never in Git. The complete named contract is in .env.example and deploy/celld-single-host/. These groups must remain separate:

Service or purposeRequired configuration group
App to CellONCOBRAIN_DEPLOYMENT_ORGANIZATION_ID, ONCOBRAIN_CELLD_PRIVATE_ORIGIN, ONCOBRAIN_CELLD_SERVICE_TOKEN, and ONCOBRAIN_EFFECT_BROKER_TOKEN
Public app and identityONCOBRAIN_SURFACE_APP_URL, BETTER_AUTH_URL, BETTER_AUTH_SECRET, and Google Workspace configuration
Native effect callbackONCOBRAIN_NATIVE_EFFECT_BROKER_URL
FleetCELLD_FLEET_BUCKET and CELLD_FLEET_* credentials
Application objectsONCOBRAIN_OBJECT_STORE_S3_BUCKET, prefix, region, and application credentials
ETL evidenceONCOBRAIN_INTERNAL_EVIDENCE_URL, dataset, and read-only storage identity
Backup and restoreDistinct backup-source reader, ONCOBRAIN_BACKUP_* vault credentials, encryption key, export token, and restore token
CleanupSeparate ONCOBRAIN_CLEANUP_* delete-only principal scoped to the application prefix

The single-host launcher rejects missing fixed identity, private tokens, app URL, Google SSO configuration, effect-broker URL, fleet bucket, or a cleartext fleet endpoint. Hosted deployments are SSO-only. The app and Eve launch environments remove all CELLD_FLEET_*, CELLD_BUCKET, and backup-vault values.

Static delivery

The app build uses Nitro's compressPublicAssets option to emit gzip and Brotli assets. The Azure demo adds a Front Door Rules Engine override only for /assets/, preserving the origin's immutable cache lifetime. HTML, authenticated server functions, source files, and API responses do not use this edge cache. bun run deploy checks the rule and route attachment; --apply reconciles them.

After deployment, verify a fingerprinted asset with Accept-Encoding: br, gzip: it should return a content encoding and become a Front Door cache hit on repeat requests. Also verify that authenticated case and source endpoints retain their private response policy.

Backup, recovery, and cache

Scheduled backup uses only the backup-export token. Restore uses a separate token from a short-lived operator environment. Backup writes encrypted database chunks and sealed application objects to the independent vault, then appends a non-PHI receipt. The freshness timer checks the newest receipt hourly; production monitoring must alert on its failure and on failed backup units. The included unit writes local journal events but cannot report a lost host.

The prepared-file cache is not a backup. The app prepares source bytes during intake and can recover exact stored identities after restart. Eve reads only manifest-authorized files. /readyz does not prove the cache is complete, so monitor prepared_files.* events before sending reference-dependent work to a replacement host. Missing files remain visible as unavailable instead of being fetched during AI admission.

Licensed release retention is also separate from the encrypted application-object backup. Backup and restore verify the release manifest and member availability; preparation and exact reads verify hashes. A missing licensed release is reported separately from database or object restore success.

The full procedures are in the repository: docs/research/celld-architecture-2026-08-29/BACKUP-AND-RESTORE.md, docs/research/celld-architecture-2026-08-29/RISKS-AND-GATES.md, and deploy/celld-single-host/README.md.

Planned fleet rotation

Celld v0.4 retains completed native Workflow cells. Use the planned fleet-rotation procedure in deploy/celld-single-host/README.md when storage growth or an upgrade calls for a fresh fleet. The procedure drains and seals the source, then restores its verified backup to a separate, empty deployment. Keep the source sealed while the destination is active; do not delete internal Workflow cells or reuse the old fleet namespace.

Azure demo

Use bun run deploy to inspect the established Azure demo. Use bun run deploy --apply to deploy one clean origin/main commit to the app, documentation, and website. It builds from a temporary archive of tracked files, publishes the complete private-source collection and builds the immutable image in parallel, then waits for both before changing any surface. The resulting image contains the app, Eve, Vocs docs, and the marketing site; the VM and the existing secretless Azure Container Apps receive the same digest.

scripts/config/hosted-demo.ts holds the fixed target settings. New source publication verifies uploaded files; repeat publication checks the exact committed manifests. Before every VM cutover, the installer verifies all source-file hashes, LanceDB search, exact-version reopening, candidate startup, and readiness. It can restore the prior VM image, environment, launcher, and service unit when that cutover fails; its temporary rollback files are removed after a successful installation. Private-source releases remain immutable, and the marked synthetic demo volume resets on service restart. Docs and site publication is verified separately, so the three surfaces do not have an atomic rollback promise. The repository guide at deploy/hosted-demo/README.md documents curator access and the disposable-data contract.

https://v2.oncobrain.ai is the pending canonical app origin. Activate it only after GoDaddy publishes Azure Front Door's current domain-validation TXT record, the domain routes to the established Front Door endpoint, and Azure provisions the certificate. Register the exact Google OAuth callback https://v2.oncobrain.ai/api/auth/callback/google before cutover, then verify that callback, WAF behavior, TLS, health, and readiness through the canonical host. Do not reuse an expired validation record or treat the Front Door endpoint as the canonical app URL.

Build and serve the docs

bun install --frozen-lockfile
bun run docs:build
bun run --cwd apps/docs preview

Set ONCOBRAIN_SURFACE_DOCS_URL to the final public documentation origin when building a release. Vocs serves human pages, /llms.txt, /llms-full.txt, and Markdown page views. The hosted-demo release builds those files into the same immutable image as the app and publishes that digest to the existing secretless documentation Container App. The public docs site must not use a fleet, application-object, or backup-vault bucket.

Code anchors

  • Celld launch guard: scripts/celld-start.ts
  • Fleet diagnostic: scripts/celld-diagnose.ts
  • Authority validation: scripts/validate-deployment-authority.ts
  • Runtime configuration: apps/oncobrain/src/config/runtime.ts
  • Cell gateway and readiness: apps/oncobrain/server/routes/
  • Single-host template: deploy/celld-single-host/
  • Backup and restore evidence: docs/research/celld-architecture-2026-08-29/