Deploy and publish
A production deployment runs the app for clinicians, Celld for durable data and work retries, and Eve for AI drafts. The public edge serves only the app. Celld's peer and operator listener stays private. Eve has no object-store or database credentials.
The single-host template in deploy/celld-single-host/ is the operational reference. It starts the Cell before its dependents, keeps the edge closed until readiness passes, and uses an immutable release directory for upgrade and rollback.
Keep storage lanes separate
| Lane | Holds | Who can use it |
|---|---|---|
| Celld fleet bucket | Cell data and native Workflow state | Celld only |
| Application object bucket | Immutable clinical source, document, export, and workspace bytes | App; narrowly scoped Cell cleanup and backup readers |
| ETL evidence store | LanceDB reference tables | Read-only app identity; separate ETL writer |
| Backup vault | Encrypted database chunks, sealed application objects, and receipts | Backup and short-lived restore operators |
| Prepared-file cache | Rebuildable verified copies for app and Eve | App writes; Eve reads through the oncobrain-prepared group |
Use distinct buckets and credentials for every lane. The app and Eve never receive vault credentials. Eve never receives fleet or application-object credentials. Object retention belongs on the encrypted backup vault, not on fleet storage or application staging and reclaim prefixes that require deletion.
The app records an exact key, digest, and size intent in the Cell before an immutable object write. The clinical transaction makes the resulting manifest authoritative. Cron reclaims expired, unused staged keys through a closed Files SDK broker. Do not give browser, app, or Eve processes bucket listing or broad delete permissions.
Start and release
For a normal local stack:
bun run devIt uses a disposable Celld project and local S3rver. It is a development convenience and does not qualify a hosted fleet or recovery path.
For a production node:
bun run celld:deploy
bun run celld:startBefore the first native Workflow publication to a fleet, use the one-time adoption preflight:
bun --no-env-file run ./scripts/deployment-authority-preflight.ts --adopt-native-workflowIt refuses a nonempty Cell namespace, including internal Workflow cells, before publishing the fixed native-effect-workflow binding. Do not rename that binding or introduce an alias after it has state.
The release wrapper closes admissions and lets active work finish before creating a fresh encrypted source-release backup. It then publishes the candidate Worker, switches the release, checks readiness and deployment authority, and completes the candidate's first backup. Only then does it reopen the edge.
The operator-run single-host protocol. The point at which a release fails determines its recovery path. Explore release and recovery.
| Failure point | Recovery |
|---|---|
| Drain fails before the guard is written | Keep the unchanged source release and reopen Caddy if it was previously active. |
| Guarded cutover fails before a verified backup receipt exists | Keep services sealed and investigate the backup failure. The wrapper has no supported resume or rollback command for this state. Do not remove the guard or attempt restore without a verified backup ID and digest. |
| Guarded cutover fails after a verified backup receipt exists | Preserve the guard and receipt. Use the verified source-release wrapper to restore that backup on a separately credentialed, empty deployment. |
| Edge, public health, or timers fail after the guard clears | The candidate remains authoritative and backed up. Repair the failed services on that candidate; this failure does not require a clean restore. |
Never remove the recovery guard by hand or point an older Worker at a live Cell after schema activation. Celld v0.4.1 data must not be opened by the older v0.4.0 binary. Use the immutable-release section of deploy/celld-single-host/README.md for the exact build, rehearsal, deploy, and restore commands. The Azure demo below has a separate deployment flow.
Verify the actual provider
Run the authority preflight and writable Celld probe against the exact fleet provider:
bun run deployment:validate-authority
bun run celld:diagnoseThe preflight runs from a short-lived operator environment that has the separate storage principals. It checks bucket and credential separation. celld diagnose --json conditionally writes and deletes a small object under its probe/ prefix. It is evidence of usable conditional writes, not merely connectivity.
Also run bun run --cwd apps/oncobrain celld:qualify:provider with the deployed Files SDK settings. It proves an application-namespace upload, byte-for-byte readback, and deletion. It does not prove backup retention, independent IAM, outage recovery, or fleet durability; those need their own review and exercises.
Celld's app and internal listeners default to loopback. A multi-node installation must set CELLD_LISTEN, CELLD_INTERNAL_LISTEN, CELLD_ADVERTISE, and ONCOBRAIN_CELLD_PRIVATE_NETWORK_OVERLAY=1 together behind an operator-managed private, mutually authenticated network. The launcher rejects CELLD_UNSAFE_PUBLIC_ADVERTISE. Set CELLD_TRUST_FORWARDED_HEADERS=1 only when the trusted edge supplies both forwarded host and protocol.
Operational guards
The repository pins the Celld v0.4.1 binary and checks its platform-specific SHA-256 before deploy or start. The launcher sets CELLD_OPERATION_DEADLINE_MS=180000 and CELLD_HANDLER_BUDGET_S=180 because an Eve effect may take up to 120 seconds, followed by bounded reconciliation and settlement. Raising both values is allowed; lowering either is refused.
A public-edge replacement is a separate change. Before cutover, prove that the Google Workspace callback is allowed only at its exact callback path and query delimiter, while sibling and prefix paths still follow normal managed rules. Test ordinary routes, TLS, the WAF, and a live sign-in callback before retiring the existing edge.
Required configuration
Put secrets in the respective service environment, never in Git. The complete named contract is in .env.example and deploy/celld-single-host/. These groups must remain separate:
| Service or purpose | Required configuration group |
|---|---|
| App to Cell | ONCOBRAIN_DEPLOYMENT_ORGANIZATION_ID, ONCOBRAIN_CELLD_PRIVATE_ORIGIN, ONCOBRAIN_CELLD_SERVICE_TOKEN, and ONCOBRAIN_EFFECT_BROKER_TOKEN |
| Public app and identity | ONCOBRAIN_SURFACE_APP_URL, BETTER_AUTH_URL, BETTER_AUTH_SECRET, and Google Workspace configuration |
| Native effect callback | ONCOBRAIN_NATIVE_EFFECT_BROKER_URL |
| Fleet | CELLD_FLEET_BUCKET and CELLD_FLEET_* credentials |
| Application objects | ONCOBRAIN_OBJECT_STORE_S3_BUCKET, prefix, region, and application credentials |
| ETL evidence | ONCOBRAIN_INTERNAL_EVIDENCE_URL, dataset, and read-only storage identity |
| Backup and restore | Distinct backup-source reader, ONCOBRAIN_BACKUP_* vault credentials, encryption key, export token, and restore token |
| Cleanup | Separate ONCOBRAIN_CLEANUP_* delete-only principal scoped to the application prefix |
The single-host launcher rejects missing fixed identity, private tokens, app URL, Google SSO configuration, effect-broker URL, fleet bucket, or a cleartext fleet endpoint. Hosted deployments are SSO-only. The app and Eve launch environments remove all CELLD_FLEET_*, CELLD_BUCKET, and backup-vault values.
Static delivery
The app build uses Nitro's compressPublicAssets option to emit gzip and Brotli assets. The Azure demo adds a Front Door Rules Engine override only for /assets/, preserving the origin's immutable cache lifetime. HTML, authenticated server functions, source files, and API responses do not use this edge cache. bun run deploy checks the rule and route attachment; --apply reconciles them.
After deployment, verify a fingerprinted asset with Accept-Encoding: br, gzip: it should return a content encoding and become a Front Door cache hit on repeat requests. Also verify that authenticated case and source endpoints retain their private response policy.
Backup, recovery, and cache
Scheduled backup uses only the backup-export token. Restore uses a separate token from a short-lived operator environment. Backup writes encrypted database chunks and sealed application objects to the independent vault, then appends a non-PHI receipt. The freshness timer checks the newest receipt hourly; production monitoring must alert on its failure and on failed backup units. The included unit writes local journal events but cannot report a lost host.
The prepared-file cache is not a backup. The app prepares source bytes during intake and can recover exact stored identities after restart. Eve reads only manifest-authorized files. /readyz does not prove the cache is complete, so monitor prepared_files.* events before sending reference-dependent work to a replacement host. Missing files remain visible as unavailable instead of being fetched during AI admission.
Licensed release retention is also separate from the encrypted application-object backup. Backup and restore verify the release manifest and member availability; preparation and exact reads verify hashes. A missing licensed release is reported separately from database or object restore success.
The full procedures are in the repository: docs/research/celld-architecture-2026-08-29/BACKUP-AND-RESTORE.md, docs/research/celld-architecture-2026-08-29/RISKS-AND-GATES.md, and deploy/celld-single-host/README.md.
Planned fleet rotation
Celld v0.4 retains completed native Workflow cells. Use the planned fleet-rotation procedure in deploy/celld-single-host/README.md when storage growth or an upgrade calls for a fresh fleet. The procedure drains and seals the source, then restores its verified backup to a separate, empty deployment. Keep the source sealed while the destination is active; do not delete internal Workflow cells or reuse the old fleet namespace.
Azure demo
Use bun run deploy to inspect the established Azure demo. Use bun run deploy --apply to deploy one clean origin/main commit to the app, documentation, and website. It builds from a temporary archive of tracked files, publishes the complete private-source collection and builds the immutable image in parallel, then waits for both before changing any surface. The resulting image contains the app, Eve, Vocs docs, and the marketing site; the VM and the existing secretless Azure Container Apps receive the same digest.
scripts/config/hosted-demo.ts holds the fixed target settings. New source publication verifies uploaded files; repeat publication checks the exact committed manifests. Before every VM cutover, the installer verifies all source-file hashes, LanceDB search, exact-version reopening, candidate startup, and readiness. It can restore the prior VM image, environment, launcher, and service unit when that cutover fails; its temporary rollback files are removed after a successful installation. Private-source releases remain immutable, and the marked synthetic demo volume resets on service restart. Docs and site publication is verified separately, so the three surfaces do not have an atomic rollback promise. The repository guide at deploy/hosted-demo/README.md documents curator access and the disposable-data contract.
https://v2.oncobrain.ai is the pending canonical app origin. Activate it only after GoDaddy publishes Azure Front Door's current domain-validation TXT record, the domain routes to the established Front Door endpoint, and Azure provisions the certificate. Register the exact Google OAuth callback https://v2.oncobrain.ai/api/auth/callback/google before cutover, then verify that callback, WAF behavior, TLS, health, and readiness through the canonical host. Do not reuse an expired validation record or treat the Front Door endpoint as the canonical app URL.
Build and serve the docs
bun install --frozen-lockfile
bun run docs:build
bun run --cwd apps/docs previewSet ONCOBRAIN_SURFACE_DOCS_URL to the final public documentation origin when building a release. Vocs serves human pages, /llms.txt, /llms-full.txt, and Markdown page views. The hosted-demo release builds those files into the same immutable image as the app and publishes that digest to the existing secretless documentation Container App. The public docs site must not use a fleet, application-object, or backup-vault bucket.
Code anchors
- Celld launch guard:
scripts/celld-start.ts - Fleet diagnostic:
scripts/celld-diagnose.ts - Authority validation:
scripts/validate-deployment-authority.ts - Runtime configuration:
apps/oncobrain/src/config/runtime.ts - Cell gateway and readiness:
apps/oncobrain/server/routes/ - Single-host template:
deploy/celld-single-host/ - Backup and restore evidence:
docs/research/celld-architecture-2026-08-29/