Data Cohorts — discovery record, decisions, and proposed design

Date: 2026-09-21, revised 2026-09-22 · Owner: Moe (data cohorts) · Sibling document: TRE design (owner Yahia) at https://cohorts-tre-design.pages.dev · Status: design only, no code.

0. Ownership split (2026-09-22)

The data cohorts product (catalogue, search, pricing, cart, checkout, orders, allocation, cohort files) and the TRE (the locked workspace the purchased data is mounted into) are two separate entities with no overlap. Moe owns cohorts; Yahia owns the TRE. They meet at one contract (§5.6): cohorts hands the TRE a provisioning request; the TRE hands back a workspace id and honours lock/unlock. Neither side reaches into the other's tables.

1. Context

bionl is building a cohort marketplace: researchers browse patient cohorts, configure a data package (patients × data types × access duration), pay, and receive the purchased data inside a Trusted Research Environment. A backend (data-cohorts-server, NestJS, built Feb–Apr 2026) and a frontend (data-cohorts-frontend, TanStack Start, Rami) exist as standalone repos and cover browse → price → cart. Nothing exists for checkout, purchase, file delivery, or the TRE hand-off. The cohorts work is to (a) complete browse → buy → hand-off end to end and (b) move both apps into the bionl nx monorepo the way sentinel and IMS were.

2. Current state (verified against code, not docs)

2.1 data-cohorts-server (branch patient-cohorts-poc, 91 commits, Feb 8 → Apr 2 2026)

• NestJS 11 + Fastify, Prisma 7 (prisma-client generator), Elasticsearch 8.13 (single index cohort-patients, one doc per patient), Redis session keyed by Firebase UID, Firebase Admin token verify (cookie auth_token first, then Bearer), global API-key guard.
• A cohort is not an entity. It is a cohort_id string on patient documents; cards are terms aggregations. Postgres stores cohortId as a bare string with no FK.
• Per-patient ES doc: subject_id, cohort_id, diseases[], geography, population, ancestry_ethnicity[], age, sex, clinical[], genomics[], imaging[], data_recency, data_longitude, study_type, survival_days, vital_status, bmi, medication_count, specimen_count, file_count (integer only), tissue_types[], preservation_methods[], data_source, provider. No file paths, no manifest, no data-type→object mapping anywhere.
• Modules: cohorts (browse/facets/detail+dashboard, ingest TCGA-API / MIMIC-CSV / Synthea-CSV, public unauth routes), session (filters + patientIds per cohort; SampleLockService picks a crypto-random patient subset), pricing (V1 deprecated; V2 per-patient-per-year with volume + duration discounts, all constants hardcoded in src/pricing/constants/pricing-tiers.constants.ts), cart (check-conflict/add/get/remove, per-User cart, multiple items per cohort), synthea-generation.
• Stubs / dead: PurchaseHistory model + enum only (no write path); no checkout endpoint; no payment SDK; Bull/mailer/GraphQL/AWS/JWT are template leftovers (no .hbs templates, mailer references undefined Handlebars, GraphQL exposes one helloWorld with playground on in prod, Bull Board unguarded).
• Auth reality: guard never touches the DB (docs claim a firebaseUid lookup that doesn't exist). The only user bridge is prisma.user.findFirst({ email }). Zero organization / workspace / role code.
• Data honesty: getPlaceholderGenomics() fabricates 1–3 genomics modalities per MIMIC patient from a hash; these feed facets, filters and pricing (WGS is the priciest line). Moot: MIMIC is disregarded (D4); the MIMIC ingest path is out of scope.
• Safety debt: console.log('decoded', decoded) logs full tokens (partially fixed in an uncommitted diff); listCohorts fires un-caught promises → unhandledRejection → process.exit(1); POST /cohorts/migrate-search (destructive reindex) open to any logged-in user; ES 9200 + Kibana 5601 bound to all interfaces on the dev VM; DATABASE_URL baked into the image from a GitHub variable; self-signed keys committed; session freshness = key-order-sensitive JSON.stringify equality.
• Tests: 3 unit specs (cart, discount, validation); e2e spec is dead scaffold; 1300-line ES query builder untested; no CI test job. Deploy: GCE VM via ssh + docker-compose (infrastructure-385413, us-east1), workflow_dispatch only.
• Docs folder is rich (journey, session, pricing/cart, dataset mappings, metadata spec, compliance) but stale on auth and silent on files, org, region, post-purchase.

2.2 data-cohorts-frontend (github.com/bionl/data-cohorts-frontend, Rami, deployed to cohorts.bionl.ai on Vercel)

• TanStack Start (SSR) + Nitro + TanStack Router/Query + shadcn/ui on Base UI + Tailwind v4 (not Mantine), zustand, hand-rolled SVG charts. Brand tokens match bionl.
• Real pages: / catalog (search, facets, sort, pagination), /cohort/$id (auth-gated tabs incl. configure & price via /pricing/v2), cart drawer. Presets and duration discounts are hardcoded client-side and can drift from the server.
• No checkout, no purchases page, no admin. "Request Access" button is hard-disabled.
• Auth handshake with lab: POST /api/auth/create-code verifies the Firebase ID token with jose, stores a 30-second code in Upstash, exchange → custom token → host-only httpOnly auth_token cookie; server functions call the backend with x-api-key + Bearer. requireAuthMiddleware enforcement is commented out; temporary @bionl.ai-only email gate; no sign-out UI; no Dockerfile/CI in repo.

2.3 bionl monorepo facts that change the plan

• Cart/cohort tables already exist in libs/prisma/schema.prisma and are migrated (2026-02-25 … 2026-04-15 cohorts_sync): Cart, CartItem, CohortPricingConfig, CohortAccessPricing, PurchaseHistory, CohortSessionArchive. Orphaned: no code references them. Cohort enums were copied into apps/sentinel/src/utils/enums/cohort-*.enum.ts, unused.
• Frame Payments is already integrated in sentinel (apps/sentinel/src/billing-provider/frame/*) behind a BillingProvider interface with createCharge / createSetupChargeIntent for one-off charges, webhook idempotency (ProviderWebhookEvent), Frame.js Card Element on the FE. sentinel Payment requires a subscriptionId, so a one-time cohort purchase needs its own order/payment shape.
• No Elasticsearch anywhere in the monorepo or infra/ terraform. New infra decision (deferred).
• bionl web has no cohorts pages; apps/web/src/components/cohorts-button.tsx opens the external app behind features.cohorts (profiles: default=true, dammam=false, lean=false) and isBionlian.
• IMS (apps/ims, own DB) has ProviderRegion, Bucket (per org/tier/region), Mount {workspaceId, bucketId, onlyDirPrefix, path, label, access read|readwrite, state}, read-only enforced server-side, shared platform bucket pattern (orgId='system'). Workspace has no region field; region lives on IMS buckets.
• bionl has RBAC tables (Role, RolePermission, UserRole per org), Organization flags isPartner/isGenomic/isMasterReviewer, credit.
• Template for a NestJS app: apps/sentinel (webpack build for decorator metadata, docker/Dockerfile.sentinel, Cloud Run, ENV_SECRET_<APP>, deploy.yml DEPLOYABLE list). Runbook to mirror: docs/ims-deployment.md.
• ClickUp "Cohorts" space, Engineering list (Abdullah Atia): epics 1 accounts, 2 catalog, 3 detail, 4 config+pricing, 5 cart+access requests, 6 secure exploration (6.1 = the TRE), 7 platform mgmt, 8 freshness + field registry (8.2). 2.1–2.3, 3.1–3.3, 4.1, 4.2, 5.1 are "be - in development"; rest backlog. Task "Extract Metadata from KHCC Data (100 patient)" assigned to Moe + Fatima.

3. Decision log (Moe; 2026-09-21, revised 2026-09-22)

4. Open decisions (cohorts side)

1. Top-up after expiry: if the first purchase has expired (TRE locked) and the org buys the same cohort again — new TRE, or unlock the old one with a new expiry? Proposal: unlock and set expiry from the new purchase (the "first purchase" rule applies only while the TRE is active).
2. File-level segmentation mechanism (D8/D9): per-purchase materialised prefix (server-side copy) vs object-level allowlist; drives bucket layout and what the TRE mounts. Proposal in §5.4.
3. Cohort entity ownership & publishing (D19): who marks a cohort published, provider, access level, "contact us" pricing.
4. Partner file handover: how a hospital delivers files + manifest into the regional cohort bucket (landing bucket + validation vs direct transfer). Relates to the KHCC 100-patient task.
5. ES hosting per region and reindex strategy (D18); KSA residency for patient-level metadata.
6. Pricing source of truth: DB-driven price book (CohortPricingConfig exists) vs constants; currency beyond USD.
7. Sample-lock semantics: one random set per cohort today vs D9's per-data-type samples; "exact match" (patients having ALL selected types) vs independent per-type samples.
8. RBAC permissions to define with PR #429: cohorts.purchase, cohorts.manage (admin), cohorts.review (future access requests).

5. Proposed target architecture (cohorts side)

5.1 Deployables

• apps/cohorts (NestJS + Fastify, Cloud Run, shared bionl DB, ES + Redis): catalogue/search, cohort entity + file manifest, org-owned cart, orders/checkout orchestration, per-purchase allocation, materialisation, expiry → lock, provisioning requests to the TRE.
• apps/cohorts-web (TanStack Start, Vercel): existing pages + checkout (Frame.js) + purchases page + "Open TRE".
• apps/sentinel: one-time charge via the existing BillingProvider (Frame) + webhook → notifies apps/cohorts (S2S) of payment success.
• apps/ims: platform cohort buckets per region (orgId='system', tier cohort); the TRE side consumes them via mounts.

5.2 Data model additions (bionl DB, owned by apps/cohorts)

• Cohort — id (slug = ES cohort_id), title, description, provider, region, accessLevel (open|restricted), status (draft|published|archived), bucketName, basePrefix, sensitivityTier, agreementRef, iaoContact, retentionUntil, lastSyncedAt, publishedAt.
• CohortFile (manifest) — cohortId, subjectId, dataType, objectKey, sizeBytes, checksum, contentType. Ingest populates it alongside ES; ES file_count becomes derived.
• Cart / CartItem — add organizationId; keep per-item snapshots.
• Order — organizationId, purchasedById, cohortId (single cohort per order, so the TRE mapping is 1:1), region, status (pending_payment|paid|provisioning|active|expired|failed), totalPrice, currency, provider (frame), providerRef, paidAt, expiresAt, purpose, intendedOutputs, mlPlan.
• OrderItem — orderId, dataType, sampleSize, selectedDataTypes, accessDurationMonths, pricingSnapshot, cohortSnapshot.
• OrderAllocation — orderItemId, dataType, subjectIds[] (random pick at purchase), fileKeys[] (from manifest), materializedPrefix.
• TreLink — organizationId, cohortId, treWorkspaceId (opaque id returned by the TRE), expiresAt (= first active order's expiry), state (active|locked); unique (organizationId, cohortId). Cohorts never reads the Workspace table for this.
• Replace PurchaseHistory with Order*.

5.3 Purchase flow

1. Checkout: apps/cohorts freezes the cart into Orders (one per cohort), asks sentinel for a Frame charge intent (org customer), returns clientSecret.
2. FE confirms with Frame.js; sentinel webhook charge_intent.succeeded → S2S orders/:id/paid on apps/cohorts (idempotent).
3. Allocation: per order item, per data type, sample N random subjects among those that have the type (crypto-random, once), resolve file keys from CohortFile, persist OrderAllocation.
4. Materialise: same-bucket server-side copy of allocated objects into purchases/<orderId>/<dataType>/<subjectId>/… (region-local, no egress). Alternative if copies are unacceptable: object-level allowlist consumed by the TRE mount layer (open decision 2).
5. Hand-off (§5.6): if no TreLink(org, cohort) → send provision with the first prefix and expiresAt; else send add-mount to the existing TRE (top-up; expiry unchanged).
6. Order → active; purchases page polls; "Open TRE" deep-links to the workspace id the TRE returned.
7. Expiry job: at TreLink.expiresAt → send lock; all orders under the link → expired. Data and materialised prefixes are kept per the retention policy (D10).

5.4 File segmentation (proposal for open decision 2)

Materialised per-order prefix, read-only mount by the TRE. Rationale: gcsfuse cannot express a random object set; a same-bucket rewrite is metadata-fast and keeps the cohort master untouched; revocation and proof of deletion are a prefix operation. Cost: duplicated bytes per purchase (bounded by sample size).

5.5 Region model

Cohort carries region; cohort bucket per region; an order inherits the cohort's region; the TRE inherits the order's region. Nothing in cohorts ever combines two cohorts, so cross-region mixing cannot arise on this side.

5.6 Contract with the TRE (the only coupling)

Cohorts → TRE (S2S, idempotent, region-aware):

• provision(orgId, cohortId, region, mount {bucket, prefix, label}, expiresAt, purchasedBy) → { treWorkspaceId }
• addMount(treWorkspaceId, mount {bucket, prefix, label}) → ok (top-up; expiry not touched)
• lock(treWorkspaceId, reason='expired') / unlock(treWorkspaceId, newExpiresAt) → ok TRE → cohorts: status(treWorkspaceId) for the purchases page; optional webhook on lock/unlock. Authorization on both sides = org owner + RBAC (D20). Everything about what a locked workspace is, the VM, mounts enforcement, SATRE controls, audit logging = TRE doc.

6. Debt to fix while landing (no product change)

Remove token logging; catch the fire-and-forget promises; guard migrate-search and ingest behind admin/S2S auth; drop GraphQL/Bull/mailer/AWS/JWT scaffolding; stop baking DATABASE_URL into images; lock ES/Kibana to the VPC; align FE hardcoded presets/discounts with server values (serve them from the API); fix the FE requireAuthMiddleware TODO; unit tests around allocation + pricing + ES query builder; docs refresh (auth section is wrong).

7. Phased plan (cohorts side; sizes rough)

• P0 Discovery — done.
• P1 Monorepo landing — apps/cohorts (sentinel pattern) + apps/cohorts-web (Vercel like web); delete dead scaffolding; §6 fixes. ~M.
• P2 Cohort entity + manifest + demo data — Cohort, CohortFile, ingest writes manifest, Synthea staged into a dev regional cohort bucket. ~M.
• P3 Org-owned cart + orders + Frame checkout — cart scoping, Order*, sentinel one-time charge + webhook → S2S. ~L.
• P4 Allocation + materialisation + TRE hand-off — random per-type sampling, per-order prefix, TreLink, provision/addMount/lock calls against the TRE contract (stub until Yahia's TRE lands), expiry job. ~L.
• P5 FE brief for Rami — contracts, purchases page + checkout + Open TRE spec.
• Dependency: P4's hand-off needs the TRE side's provision/addMount/lock endpoints; until then a stub that creates a plain workspace + IMS RO mount is enough for the dev demo.

8. Verification for D15

Dev: run nx serve cohorts + cohorts-web locally against dev ES/Redis; ingest a Synthea cohort with manifest; sign in via the lab handshake; configure 5 patients × 2 data types × 3 months; pay with a Frame sandbox card; confirm Order=active, OrderAllocation rows, materialised prefix objects, TreLink with the returned workspace id; buy 5 more of the same cohort → addMount sent, expiresAt unchanged; advance expiresAt, run the expiry job → lock sent and orders expired.

9. SATRE statements that land on the cohorts side

The TRE doc carries the full SATRE mapping; these statements are satisfied by cohorts-side features and are tracked here: 1.3.02 (purpose/DPIA capture at purchase), 1.4.01/1.4.02 (licence acceptance, funding = payment, expiry enforcement), 1.4.06 + 1.7.02 (public Safe-Projects register built from Order), 1.5.04 (Data Controller agreement per cohort: Cohort.accessLevel + licence), 3.1.03 (Cohort.sensitivityTier set by the provider), 3.1.04 + 3.1.20 (cohort ingress validation/approval, metadata disclosure check), 3.1.08 (Cohort as the record of data held), 3.1.13 (minimum data: only bought files materialised), 3.3.03 (intended outputs at purchase), 3.4.01–3.4.04 (catalogue, metadata model, query UI, synthetic previews), 4.4.01 (costs visible before purchase), 2.3.01 (TRE resources/costs shown on the purchase page — content from the TRE doc).

10. Next steps

1. CTO/Yahia: agree the §5.6 contract and D6/D10 semantics (top-up, lock).
2. Define the RBAC permissions (open decision 8) with PR #429.
3. Start P1 on feat/cohorts-monorepo when Moe gives the go.