Thnkrr: Swappable Suppliers, the Build Plan

9 October 2026. A step-by-step plan for an AI coding agent (Claude Opus 5.5 in Claude Code). It builds what PROVIDER-SWAP-RESEARCH-2026-10-09.md recommends. Nothing starts until the founder and CTO tick Gate 0.

Read the research file first. Read PRODUCTION-PLAN-2026-10-09.md second: this plan is designed to run interleaved with it, reuses its items where they overlap, and names every overlap so the two never fight.

0. Rules for the agent

1. Stop at every gate. Report in plain words: what was built, what was tested (paste the test output as it is, failures included), what is left. Never act on production.
2. Everything is code, in pull requests. Work in worktrees off dev/pilot, branch feat/provider-swap-<phase>, one PR per phase into dev/pilot. Never PR out of main. Never merge the demo/mock-ui branch. Never git stash bare; use a WIP commit.
3. Adapters, never forks. Every supplier sits behind the existing interface or a new one in src/lib/ or the owning module. The default implementation is always the one that works with no supplier at all (log it, send it to a person, do nothing).
4. Production refuses the wrong thing. Any setting that could send student data to an unapproved company fails at startup in production, as MODERATION_PROVIDER=anthropic already does. Tests prove the refusal.
5. Database changes only add. Next migration numbers follow whatever the production plan has used; check drizzle/ before numbering.
6. Test only what changed, then the whole suite before each PR. No Playwright runs unless a phase says so.
7. Docs. The decision record for this work is thnkrr-api/docs/29-swappable-suppliers.md (docs/28 belongs to the production plan). Index it in docs/00-overview.md. One grouped Sources section at the end; no inline links. Add decisions to docs/07-decision-log.md as D-80 onward, checking the last used number first.
8. Plain words in every runbook, log line and alarm: what is wrong for a person using Thnkrr first, the technical cause second.
9. Rules that stand: every AI model stays on Bedrock or an approved in-account cloud (D-70, Gate 0 decision P3); no student data to a model company's own API; everything in US East unless Gate 0 says otherwise.
10. The existing single-server deploy files stay in the repo as a working fallback.

1. Gate 0: decisions before any code

The founder decides P1 to P4 and P9; the CTO decides the rest. Recommended choice first.

Humans also do, before Phase 1 (no code):

• Open a Backblaze account (P2); create bucket thnkrr-offsite-backups with Object Lock (compliance mode) and a 400-day default retention; create an application key limited to that bucket. Store the key in the password manager.
• Open a Cloudflare account; add thnkrr.com as a zone without changing nameservers yet (it costs nothing and lets drills run). Enable R2.
• Open the spare email sender's account (P4); verify thnkrr.com (DKIM records); request the retention setting; send one test. Record the SMTP URL in the password manager.
• Confirm the registrar and current DNS host of thnkrr.com (docs/27 Phase A11).
• If P3 is Azure: open the Azure account, request the Foundry abuse-monitoring exemption, and do not use it until the written confirmation is filed.

Gate 0 ticked by: founder (P1 to P4, P9) and CTO (P5 to P8, P10 to P12, and the human steps).

2. Phase 1: the missing seams (agent, about 4 days)

All in thnkrr-api. Each item: what, where, done when.

1.1 Picture-check connector (the seam the docs describe and the code lacks)

• What: src/lib/image-screen.ts exporting ImageScreen { readonly providerId: string; screen(input: { bytes: Buffer; mime: string; surface: string }): Promise<ImageVerdict | null> } where ImageVerdict = { decision: "approve" | "reject" | "escalate"; labels: string[]; confidence: number; reason: string }. null means "no trustworthy verdict", which the caller treats as escalate, never approve (same contract as the text classifier). Adapters: image-screen-none.ts (returns null; the default) and a rekognition adapter behind IMAGE_SCREEN_PROVIDER=none|rekognition with REKOGNITION_REGION defaulting to us-east-1. The media pipeline calls it where a photo today goes straight to the human queue; an approve verdict still queues the item but marks it "AI-approved" so reviewers can prioritise; a reject or escalate keeps today's behaviour. In the pilot, people still see every image (docs/20 §10.2 "People" row); the connector narrows and labels, it never releases.
• Also: the Rekognition adapter must refuse to start unless IMAGE_SCREEN_OPTOUT_CONFIRMED= true is set, with a startup message naming the AWS Organizations AI-services opt-out (docs/20 §10.2 prerequisite). Face APIs are never called; a unit test asserts only DetectModerationLabels and DetectText are used.
• Where: src/lib/image-screen.ts, src/lib/image-screen-none.ts, src/lib/image-screen-rekognition.ts, src/modules/media/service.ts, src/config.ts, src/context.ts, .env.example. New dependency @aws-sdk/client-rekognition.
• Done when: unit tests: none-adapter escalates; a fake Rekognition response maps to each verdict; the opt-out guard refuses. Integration test: a photo with provider none lands in the human queue unchanged.

1.2 Generic AI connector with an allow-list

• What: src/modules/moderation/classifier-openai-compatible.ts: speaks the OpenAI Chat Completions API to AI_BASE_URL with AI_API_KEY and AI_MODEL_ID, same prompt, same verdict parsing as the Bedrock adapter (extract the shared parsing into verdict-parse.ts). MODERATION_PROVIDER gains the value openai-compatible. In production, AI_BASE_URL's host must appear in AI_ALLOWED_HOSTS (comma-separated; default: the Bedrock OpenAI-compatible endpoints for us-east-1), else the server refuses to start with a message naming D-70. The modelId stamped on verdicts becomes <provider>:<model> for every adapter, so a switch shows in the record.
• Keep-alive: keepalive.ts takes a make(modelId) for the active provider type, so the daily check works for any adapter, not only Bedrock. Backup model ids come from a provider-neutral MODERATION_KEEPALIVE_MODEL_IDS (keep the Bedrock name as an alias).
• Eval: scripts/moderation-eval.ts already calls createClassifier; add a --compare <providerA> <providerB> mode that runs the same set through two configurations and prints catch rate and wrong-block rate side by side. This is the gate before any switch (AI-CHECKS page, "A test set decides").
• Where: src/modules/moderation/{classifier-openai-compatible,verdict-parse,select, keepalive}.ts, src/config.ts, scripts/moderation-eval.ts, .env.example.
• Done when: unit tests with a fake HTTP server: verdict parsing, refusal on a non-allowed host in production, allowed in dev; keep-alive runs against the fake; the compare mode prints two columns. Existing Bedrock tests unchanged and green.

1.3 Delivery-network purge adapters

• What: CDN_PROVIDER=none|cloudfront|cloudflare|bunny. src/lib/cdn.ts keeps the interface; add cdn-cloudflare.ts (zone id + API token, "purge by URL", 30 per call, batch accordingly) and cdn-bunny.ts (pull-zone id + access key, per-URL purge). All adapters never throw, retry once, log a plain-words failure that names the cache-time backstop. CLOUDFRONT_MEDIA_DISTRIBUTION_ID still selects CloudFront when CDN_PROVIDER is unset.
• Where: src/lib/cdn*.ts, src/config.ts, .env.example.
• Done when: unit tests per adapter against a fake HTTP endpoint, including the 30-URL batching and the "never throws" contract; storage tests pass with each provider value.

1.4 Visitor address from any edge

• What: today the API trusts CloudFront's viewer-address header only when a shared secret arrives. Make the header names settings: EDGE_IP_HEADER (default CloudFront-Viewer-Address; Cloudflare uses CF-Connecting-IP; Bunny and most others use the first hop of X-Forwarded-For) and EDGE_SECRET_HEADER (default X-Thnkrr-Edge). Keep EDGE_SHARED_SECRET as the gate. Parse the CloudFront ip:port form and the plain form.
• Where: src/lib/client-ip.ts, src/config.ts, .env.example.
• Done when: unit tests cover CloudFront form, Cloudflare form, forged header without the secret, and the production plan's load-balancer hop (its item 1.4) if already merged.

1.5 Storage conformance test and the new stand-in

• What: scripts/storage-conformance.ts (pnpm storage:conformance): runs every method of the Storage interface against whatever S3_* settings are present (presign PUT then upload, head, get, copy to public, multipart create/part/list/complete/abort, remove with purge no-op) and prints a pass/fail table. Replace MinIO in docker-compose.dev.yml and the web repo's CI with the Gate 0 P5 choice; keep the S3_ENDPOINT path-style setting working for it. Note in docs/29 that R2 has no per-object lock and is for live media only.
• Where: scripts/storage-conformance.ts, docker-compose.dev.yml, README.md, thnkrr-web .github/workflows/ci.yml (the MinIO step), e2e/STANDARDS.md if it names MinIO.
• Done when: the conformance script passes against the new stand-in locally and in CI; the web E2E upload journeys pass against it; one documented run against R2's free tier passes (record the date in docs/29).

1.6 Mail sender check and spare

• What: scripts/mail-check.ts (pnpm mail:check <domain> [--smtp <url>]): resolves SPF, DKIM selectors (Amazon's three, Postmark's, ZeptoMail's), DMARC, and optionally sends one test message through the given SMTP URL to OPS_EMAIL. Add MAIL_PROVIDER_LABEL (free text, logged at startup and on the providers page, item 1.9) so logs say which sender is live.
• Done when: the script runs against thnkrr.com and reports; a unit test covers record parsing with fixtures.

1.7 Speech connector: the cloud adapter sits behind the seam

• What: if the production plan's D7 (Amazon Transcribe) is approved, implement it as src/lib/transcriber-transcribe.ts behind TRANSCRIBER_PROVIDER=whisper|transcribe|none, never as a replacement of transcriber.ts. Speaker identification is never requested (a unit test asserts ShowSpeakerLabels is absent). Whisper remains the fallback when the cloud call fails or times out.
• Done when: unit tests with a fake client; the existing video tests pass with whisper.

1.8 Secrets encrypted in the repository (P8)

• What: deploy/secrets/production.enc.env and staging.enc.env encrypted with SOPS and age; .sops.yaml with the age public keys of the CTO and the deploy host; scripts/secrets.sh {edit|decrypt|rotate-key}; deploy/README.md section on how the server decrypts at deploy time (age private key in /etc/thnkrr/age.key, chmod 600) and how Kamal reads it (Phase 2). No real secret is ever committed in plain text; a CI step greps for known secret names in plain files and fails if found.
• Done when: a test encrypts and decrypts a fixture; CI grep passes; deploy.sh can source the decrypted file.

1.9 The providers page and startup manifest

• What: src/lib/providers.ts builds one table at startup from config: category (hosting, database, storage, cdn, mail, ai, image-screen, transcriber, adult-check, errors, telemetry), provider id, endpoint host (never keys), and a reachability probe per category (database ping, bucket head, SMTP connect, AI keep-alive text, CDN auth check). Logged once at startup in plain words ("Files: Amazon S3 us-east-1, reachable"). Exposed at GET /health/providers for staff with the Safety Admin hat only, and as pnpm providers for the terminal. The same data feeds the public supplier list page the safety runbook requires (docs/22, "Supplier list"): the web gets a static JSON the API serves at GET /public/suppliers listing company names and purposes only.
• Done when: unit test builds the table from a config fixture; integration test checks the staff-only route refuses a student; the public route lists names only.

1.10 Off-site backups and the restore drill (P2)

• What: scripts/backup-offsite.sh: nightly pg_dump -F c, encrypted with age to the backup public key, uploaded with rclone to the B2 bucket under db/<date>/; then rclone sync of both media buckets to media/<bucket>/ (never deleting on the destination; evidence files included). scripts/restore-drill.sh: downloads the latest dump, decrypts, restores into a fresh Postgres (local Docker or a named host), runs pnpm db:migrate to prove the schema matches, counts rows in five key tables against the source, and prints a timed report. A systemd timer unit in deploy/ and the equivalent Kamal cron in Phase 2. OFFSITE_RCLONE_REMOTE and OFFSITE_BUCKET settings. Failure of the nightly job emails OPS_EMAIL.
• Done when: the drill runs end to end against the dev database into a throwaway container and prints the report; docs/29 records the first run's time.

Gate 1 ticked by: CTO, after CI is green on both repos, the conformance run against R2 and the restore drill report are in docs/29, and the agent has shown the providers page.

3. Phase 2: one packaged app, two ways to run it (agent, about 2 days)

2.1 Container image

• What: reuse the production plan's item 1.12 if merged (Dockerfile, non-root, ffmpeg, whisper.cpp with the base model). If not merged yet, build it here to that spec. Add a PROCESS_ROLE-aware entrypoint only if the production plan's 1.1 exists; otherwise the image runs the monolith as today. Push to GitHub Container Registry on every dev/pilot merge; also push to Amazon ECR when AWS credentials are present (both tags, same digest).
• Done when: the image builds for ARM and x86 in CI and answers /health.

2.2 Kamal as the any-host deploy path (P11)

• What: deploy/kamal/deploy.yml targeting a host given by KAMAL_HOST, with kamal-proxy for HTTPS (Let's Encrypt), health check on /health/ready (or /health until the production plan's 1.11 lands), .kamal/secrets reading the SOPS-decrypted file, a migrate hook that runs pnpm db:migrate before the switch, and the off-site backup cron as a Kamal accessory. One command deploys; kamal rollback is documented as the undo. The systemd and Caddy files stay untouched beside it.
• Where: deploy/kamal/, .kamal/, deploy/README.md ("Two ways to run the same image").
• Done when: a documented dry run deploys the image to a throwaway VM at a non-Amazon host (a $5 instance for one hour is acceptable; the CTO provides it or approves the spend), the smoke test from docs/27 §14 step 20 passes against it with the dev stand-ins for storage and mail, and the VM is destroyed. Time from "empty VM" to "healthy" is recorded.

2.3 Website publish script

• What: thnkrr-web/scripts/publish.sh <target> with targets s3-cloudfront (upload dist/, invalidate /index.html), cloudflare-pages (wrangler deploy with a _redirects file sending app routes to index.html and leaving /in/* and /school/* to the API), and static-dir (copy to a folder Caddy serves, for the single-host path). The CloudFront Function's logic and the _redirects rules are generated from one list of app routes so they cannot drift.
• Done when: a unit test generates both rule sets from the same list; static-dir works in the Kamal dry run; the S3 target is exercised against the dev stand-in.

Gate 2 ticked by: CTO, after reading the Kamal dry-run report and the publish script.

4. Phase 3: watching without lock-in (agent, about 2 days)

3.1 OpenTelemetry (P7)

• What: src/lib/telemetry.ts starts the Node OpenTelemetry SDK when OTEL_EXPORTER_ OTLP_ENDPOINT is set (off otherwise, so laptops and tests are unchanged): traces and metrics stable, logs via the pino bridge where the SDK supports it. Metric names follow the production plan's item 1.6 list (request count and latency per route group, 5xx, 503 busy, moderation queue age, AI latency and errors, job failures). A collector config in deploy/otel/ with two exporters ready: CloudWatch (via the AWS distro) and a generic OTLP backend (Grafana Cloud, SigNoz, Honeycomb); switching is one line in that file. If the production plan has already emitted CloudWatch embedded-metric-format lines, keep them for now and mark them "replaced by OpenTelemetry" in docs/28 and docs/29.
• Done when: a unit test asserts the SDK is inert without the endpoint; an integration run against a local collector (Docker) receives one trace and one metric.

3.2 Errors (P12)

• What: @sentry/node initialised only when SENTRY_DSN is set; the redaction list from the production plan's item 1.5 applied to error payloads too; web repo gets @sentry/react the same way behind VITE_SENTRY_DSN.
• Done when: a thrown test error appears in a local GlitchTip or Bugsink container.

3.3 Outside eyes

• What: documentation plus config, no app code: an uptime check from a non-Amazon service on https://thnkrr.com and https://api.thnkrr.com/health (UptimeRobot free or Better Stack free), a status page on the same service, and paging to a phone through the free PagerDuty or incident.io tier. The alarm text template is plain words. Recorded in docs/29 with account owners.

Gate 3 ticked by: CTO.

5. Phase 4: addresses and code hosting (agent, about 1 day)

4.1 DNS zone in the repo (P10)

• What: thnkrr-api/dns/dnsconfig.js (DNSControl) describing every record of thnkrr.com (apex, www, api, media, mail, the SES DKIM and MAIL FROM records, DMARC, the spare sender's DKIM, ACM validation). Two provider blocks, Route 53 and Cloudflare, with one of them active. pnpm dns:preview and pnpm dns:push wrap dnscontrol preview|push. CI runs preview on every PR touching dns/.
• Done when: preview against the live zone shows no unexpected diff; docs/29 records the registrar, DNS host, and the switch steps (lower TTL to 300 the day before; switch; keep old records a day).

4.2 Mirrors and registries

• What: a nightly GitHub Action in each repo that pushes --mirror to a second host (GitLab or Codeberg; the CTO creates the empty repos and a deploy token). Workflows reduced to: checkout, toolchain, pnpm ci:local (both repos already mirror CI this way; remove any step that is not in ci:local). Image push to GHCR and ECR from Phase 2.
• Done when: the mirror shows the latest commit; both CI workflows are three steps plus the E2E job.

Gate 4 ticked by: CTO.

6. Phase 5: the supplier register and the first drills (agent + CTO, about 3 days)

5.1 The register

• What: docs/29-swappable-suppliers.md gains a table with one row per supplier: category, company, what it holds, data location, retention terms, the connector and setting that selects it, the warm spare, the last drill date and result, the account owner. The public supplier page (item 1.9) is generated from the same rows minus internal columns.

5.2 Six drills, on staging or a throwaway copy, each timed and written down

Each drill gets a one-page runbook in thnkrr-api/runbooks/swap-<thing>.md written from what actually happened, in plain words, with the timing.

Gate 5 ticked by: CTO and founder, after reading the six reports. The drill calendar (P9) goes into the team's shared calendar with the account owner on each.

7. What this plan does not do

• It does not move anything off Amazon. It makes moving possible and practised.
• It does not change the product, the moderation rules, or any safety clock.
• It does not replace the production plan's infrastructure code; it keeps the app and data portable underneath it.
• It does not add any supplier that sees student data. The only new accounts hold encrypted backups (B2), DNS records (Cloudflare), or a spare email sender that is idle until needed.

8. Settings added or changed, in one place

9. Time and order

If only one thing can be done: Phase 1 item 1.10, the off-site backup and restore drill.