Going Live Properly: The Plan (9 October 2026, revised the same day)
What this is. The step-by-step plan that follows from PRODUCTION-RESEARCH-2026-10-09.md,
sized for a pilot: the one server and one database already decided in docs/27, about $66
a month at the start and about $103 at 1,000 students a year in (§2), plus the things that are nearly free and matter most: logs you can
search, alarms that reach a phone, deploys with no outage, backups you have proven you can
restore, and cookies that cannot leak across subdomains.
It has three parts:
- Part A: what you do by hand, every step spelled out, each with a check that proves it worked.
- Part B: what the agent builds, cleans up and updates, every item with its files and its "done when".
- Part C: the order, the gates, the runbooks and what is parked.
Status: the decisions in §1 were approved on 11 October 2026. Sections 3 to 7 explain the four that needed working out: videos, releases, retention and the night window, and how the plan grows. Nothing is built yet. Part A starts with the account steps (A-steps 1 to 14), and every later phase begins only after the gate before it is ticked by a named person.
1. Decisions first (Gate 0)
| # | Decision | Recommended | Alternative | Monthly effect |
|---|---|---|---|---|
| D1 | Region | US East (N. Virginia), as docs/27 decided | none | |
| D2 | Shape | One EC2 t4g.medium + RDS db.t4g.micro, as docs/27 decided | $44.56 | |
| D3 | One front door | Yes: API served at thnkrr.com/api through the existing CloudFront, host-only cookies | Keep api.thnkrr.com with the .thnkrr.com cookie | none |
| D4 | Logging and alarms | Yes, the right-sized set (B7, A-steps 22 to 25) | about $4 | |
| D5 | Releases with no outage | Decided: the new version runs beside the old one, passes automatic checks, then takes 1 request in 10 for 10 minutes before taking over; the old one stays warm for 30 minutes (§4, B9) | Restart with a brief outage | none |
| D6 | Firewall (WAF managed rules) | Switch on the week real students arrive | Never / from day one | $8 when on |
| D7 | Video | Decided: our own speech-to-text transcript and the captions get the writing check; a person watches every video; known child-abuse matching as D11; no AI judging the pictures (§3). Memory caps added | Frame check on every second of video | none / about $20 at 1,000 students |
| D8 | Retention numbers | Decided (§5), changeable on request: messages 2 years; closed accounts deleted in 30 days with a locked legal copy for 1 year; dormant accounts 3 years; removed content 1 year; logs 1 year; backups 12 months | Founder and lawyer set different numbers | slows storage growth |
| D9 | Monthly update window | Decided: Sunday 3:00 to 3:30 am US Eastern. The server is replaced by a freshly updated one rather than restarted, so there is no outage; the database's updates take under a minute, and its rare required fixes a few minutes, with a banner that works even while the server is away (§6) | A second server and load balancer (no window) | about 5 cents / about $45 |
| D10 | Budget alarms | Alarm at $80, ceiling $95 while the pilot is under about 300 students. Raise to $120 and $140 before 1,000 students (§2) | One fixed ceiling | none |
| D11 | Known child-abuse matching | Decide before deploying (by Gate 2). The code has a ready slot with nothing connected; the server won't start in production until the choice is written in its settings. Recommended: connect NCMEC's list and PhotoDNA for photos, Google CSAI Match and Project Arachnid for video, as each application is approved; until then none, where the person reviewing each upload is the only check | Deploy with none and connect later | none (all free) |
| D12 | Security monitoring | Decided: GuardDuty on from day one, base protection only (account activity, network and DNS records); its optional add-ons stay off. Findings go to the alarm inbox and phone | Off until a school asks | about $5 to $13 |
2. What it all costs
Everything this plan uses, priced in US East. "1,000 students" means 1,000 active students, each writing about 100 checked posts, comments and messages a month, uploading 3 photos, and 1 in 5 posting a 1.5-minute video (the same usage as docs/27 §12). All of it is paid from the AWS credits.
What we use from day one
| What | What it is | Start (about 20 students) | 1,000 students, first month | 1,000 students, a year in |
|---|---|---|---|---|
| Server | EC2 t4g.medium $24.53, its 30 GB disk $2.40, its fixed address $3.65 | $30.58 | $30.58 | $30.58 |
| Database | RDS db.t4g.micro $11.68, its 20 GB disk $2.30 | $13.98 | $13.98 | $13.98 |
| Files and delivery | Photos, videos and the website in S3; CloudFront delivery is free up to 1 TB a month | $0.10 | $1.15 | $13.80 |
| Logs, alarms, outside health check | CloudWatch above its free 5 GB, plus the Route 53 health check $1.50 | $2 | $4 | $4 |
| DNS | Route 53 for thnkrr.com: $0.50 for the domain plus lookups | $1 | $1 | $1 |
| AI writing check | Bedrock, about $0.155 per 1,000 checks | $0.31 | $15.50 | $15.50 |
| Photo check | Rekognition, $1 per 1,000 photos. Flags photos for the reviewer; never approves on its own. Needs the AI-services opt-out (A-step 11) first | $0.12 | $6 | $6 |
| Video transcripts | Whisper on our own server, so no extra charge | $0 | $0 | $0 |
| Known child-abuse matching (photos and video) | NCMEC's list, PhotoDNA, Google CSAI Match, Project Arachnid: free, once the founder's applications are approved (D11) | $0 | $0 | $0 |
| SES, $0.10 per 1,000 emails | $0.01 | $0.50 | $0.50 | |
| Firewall | WAF managed rules (D6: on the week real students arrive) | $8 | $8 | $8 |
| Security monitoring | GuardDuty (D12), base protection: $4 per million account events read, $1 per GB of network and DNS records. Watches for signs of a break-in and alarms; never blocks. Free for the first month, which also shows our real monthly estimate | $10 | $10 | $10 |
| Backups, CloudTrail, Budgets, Parameter Store, Session Manager, certificates, basic attack protection | Inside free allowances at this size | $0 | $0 | $0 |
| Total a month | about $66 | about $91 | about $103 |
The only line that grows on its own is files: storage adds about $1.15 a month for every month at 1,000 students. The retention numbers in §5 slow that: the full-quality originals, about half of it, go after 30 days. The monthly server swap (§6) adds about 5 cents a month.
Not used (the trigger to reconsider each is in C5): the frame-by-frame video check ($18 to $22 a month at 1,000 students) and Amazon Transcribe (Whisper does this job).
GuardDuty's line is an estimate between about $5 and $13: it depends on how busy the account is, which nobody knows until it runs. Its free first month shows the real figure; D10's alarms are set with room for the high end.
The credits. About $5,000 (docs/18). At $66 a month they last about 6 years; at $103, about 4 years.
Words used in sections 3 to 7
| Word | What it means here |
|---|---|
| Request | One question the app asks our server, such as "show me the feed". A page needs a few |
| Port | A numbered door on the server. Two versions of Thnkrr can run side by side on two doors |
| Table, row, field | The database keeps information in tables, like spreadsheets: a row is one item (one message), a field is one column (its text, its time) |
| Postgres, minor version | Postgres is our database program. A minor version is a small update to it (17.4 to 17.5), mostly fixes |
| Background work | Work nobody waits for on screen: the AI checks, emails, converting videos |
| The console | Amazon's website for running our account |
| DNS | The internet's address book: it turns thnkrr.com into the number of the machine to connect to |
| Security certificate | What lets a browser trust it is talking to the real Thnkrr over an encrypted connection |
origin.thnkrr.com | A behind-the-scenes name for our server. Only Amazon's delivery network (CloudFront) uses it; visitors never see it |
| Production | The real site that students use, as opposed to test copies |
| Standby database (Multi-AZ) | A second copy of the database, kept in step in another Amazon building, that takes over within about a minute |
| Containers (Fargate), templates | Containers package the app so Amazon can start and replace copies of it by itself. Templates are files that describe our whole Amazon setup, so it can be rebuilt by a command |
| NCMEC | The US national centre that receives reports of child sexual abuse material |
| Child Safety Lead | The staff member in charge of child-safety cases |
| A-step, B-item, C5 | A-steps are the hand steps in Part A; B-items (B9, B16) are entries in Part B, the build list; C5 is the parked list in Part C |
India's rules appear below even though the pilot is in the US. They apply once we have
Indian users, and we follow them now so nothing has to change when that happens
(DEADLINES-AND-DATA-2026-10-07.md explains each).
3. How videos are checked (D7)
Three things happen to every video, and no AI decides anything about the pictures.
- Our own server writes down what is said. Whisper (speech-to-text that runs on our server, so the audio never leaves it) turns the sound into text. That text, together with any captions the student typed, goes through the same writing check as a post. A spoken phone number or a spoken threat is caught the same way a written one is.
- A person watches every video before anyone else can see it. No video goes live on an AI's say-so, however clean its transcript is.
- Known child-abuse material is matched against shared lists once the founder's applications to those programmes (NCMEC's list, Microsoft's PhotoDNA, Google's CSAI Match; D11) are approved. This compares a fingerprint of the video with material NCMEC and others have already confirmed. It is matching, not an AI judging what it sees.
One detail to know. The photo check (an Amazon AI that looks for nudity and violence, switched on at launch once Amazon is told not to keep our pictures, A-step 11) also looks at one picture per video: the sheet of 30 small stills that our server makes for the reviewer. It costs about $1 per 1,000 videos and works exactly like the photo check: it never approves or rejects anything, and if it sees something, that video moves to the front of the reviewer's list. This is not the frame-by-frame check D7 rules out, which would look at every second of every video for about $90 per 1,000 videos. If you would rather it left videos alone, it is a one-line change.
4. Releasing a new version without an outage (D5)
The idea. We never stop the running version to start the new one. The new version starts next to it on the same server, proves itself first on automatic checks and then on a small share of real visitors, and only then takes over. The old version stays running, unused, for half an hour, so we can switch back in seconds. Big companies call the small-share step a "canary" (a few go first) and the two-copies idea "blue/green". We do both on one server, because one is all we need yet.
The steps. One command (deploy.sh) runs all of them and stops at the first problem.
| Step | What happens | If it goes wrong |
|---|---|---|
| 1. Prepare | The new version is built in its own folder. Database changes are applied. They only ever add things, so the running version doesn't notice | The script stops. Nothing has changed for anyone |
| 2. Start beside the old one | The new version starts on a spare port. Nobody is sent to it yet, and it does no background work (AI checks, emails, videos) | It is stopped. Nobody saw it |
| 3. Automatic checks | The script asks it directly: can it reach the database? Then a test account signs in and opens the feed, a profile, messages, alerts, search and the review list, plus a public profile signed out and the complaint and removal forms. Every answer must be right and arrive within a second | It is stopped. Nobody saw it |
| 4. Trial on real visitors | 1 request in 10 goes to the new version for 10 minutes. The script compares the two versions' errors and speed side by side | Everything goes back to the old one within seconds, with a message saying why, if the new one's failure rate is more than 1 in 100 above the old one's, or it takes over 1.5 times as long to answer |
| 5. Switch | All visitors and all background work move to the new version | |
| 6. Keep the old one warm | The old version keeps running, with nobody on it, for 30 minutes. One command (rollback.sh) brings it back in seconds | |
| 7. Tidy up | The old version stops. The last two versions stay on the server's disk, so going back later still takes only about a minute |
A release takes about 15 minutes, then the half hour with the old version kept warm. Release on a working morning when someone can watch. Never release on a Friday evening or just before the lunchtime rush.
The website (the part that runs in the browser) follows only once the server has fully switched. New files are added next to the old ones, which stay for 30 days, so a tab that is already open keeps working. The app checks for a newer version every 10 minutes and whenever the tab comes back into view. When it finds one, it says "A new version of Thnkrr is ready" and switches quietly the next time the student changes page, never in the middle of typing.
Three rules make this safe. Each is checked automatically before a release:
- Database changes only add things during a release. Removing or renaming something happens in a later release, once no running version uses it. For example, a table stops being used in one release and is deleted in the next. A check fails any release whose database change removes or renames something without marking it as that second half.
- The server keeps answering the previous website the same way. Fields are added, never removed or renamed in the same release. The type definitions the website and server already share make this a check, not a promise.
- Only one version does background work at a time. A job cut off half-way, such as a video being converted, simply starts again. Every job is already written to be safe to repeat.
During the 10-minute trial, the per-person limits (requests a minute, posts a minute) are counted by each version separately, so they are briefly up to twice as generous. That is harmless for 10 minutes. It gets fixed properly before a second server ever exists (§7, stage 3).
A snapshot first, only when needed. A snapshot is a full copy of the database at one moment. Today the release script asks the person releasing to take one before it changes the database. It will now ask only when a release removes something, because releases that only add things can always be undone by switching back.
5. How long we keep things (D8)
These are our numbers. They meet every legal deadline in DEADLINES-AND-DATA-2026-10-07.md
with room to spare, and each one is a single setting, so it can change whenever the founder
or the lawyer asks.
Four rules sit behind them:
- While an account is open, we keep what the person made, as LinkedIn does: the profile, posts, comments, connections, photos and videos.
- Where a law sets a minimum, we use the longest one that could apply to us, for everyone. One rule for all users is simpler to run and to explain. It is the same reason we use India's 2-hour removal deadline for everyone.
- Legal holds always win. Anything linked to a child-safety case, a police or court request, an open complaint or appeal, or a lawsuit is kept until that hold ends, whatever the table below says.
- Records staff look at stay in the database until they are deleted, so they always open on screen; nothing has to be fetched back from an archive. The only copies kept elsewhere are server logs and backups, which nobody browses.
| What | How long | Then | Why this number |
|---|---|---|---|
| Messages | 2 years from sending | Deleted | A problem reported late still has its record: grooming often comes to light months later. Longer than every legal minimum (1 year). Students still can't delete messages themselves; that safety rule stays. When an account closes, its messages follow the closed-account rule below, for both people in the conversation |
| Closed accounts | Hidden within 24 hours, deleted within 30 days | A locked copy of what the law needs (sign-up details, messages, the record of activity) is kept 1 year, seen only by the safety team or for a legal request, then deleted | LinkedIn deletes in 30 days. One year covers the 180 days in India's IT Rules, and the 1 year that India's data protection law requires from May 2027 |
| Accounts nobody signs into | 3 years without a sign-in | Emails 30 days and 7 days before; then closed as above | A student can come back after school. Three years covers a gap year and the start of college |
| Content we removed | 1 year after removal, locked | Deleted | India requires 180 days; the US child-abuse law requires 1 year for reported material. One number covers both |
| Child-safety evidence, and anything reported to NCMEC | At least 1 year after the report; longer if the police ask | Deleted only by the Child Safety Lead, once the hold ends | US law. Already built so that no reviewer can delete it |
| Moderation records (the AI's answer and the reviewer's decision) | 2 years | Deleted, unless linked to a hold | Long enough to see a pattern in someone's history and to answer an appeal. Each record also holds a copy of the words it checked; for approved items that copy is deleted after 90 days, since the post or message itself still has the words |
| Complaints, court and government orders, and how we handled each | 3 years after closing | Deleted | Proves we met each deadline. Three years is the usual time limit for bringing a claim, in both the US and India |
| "Who did what" (the record of staff actions) | 3 years | Deleted, unless linked to a hold | Same reason |
| A parent's permission (proof a guardian said yes) | While the account is open, plus 3 years | Deleted | We must be able to prove permission was given if anyone asks later |
| Sign-ups whose parent never answers | 30 days | Deleted | Already built |
| Alerts ("Maya commented on your post") | 90 days | Deleted | LinkedIn does the same |
| "Who viewed your profile" | 90 days | Deleted | Already built |
| Full-quality original uploads (locked away; nobody sees them) | 30 days after the safe versions are approved | Deleted, unless linked to a hold | Only the safe versions are ever shown, and keeping the originals doubles storage |
| Rejected uploads | 30 days | Deleted, unless the rejection opened a safety case | |
| Server logs | 90 days searchable, then a compressed copy until 1 year | Deleted | India's cyber-security agency (CERT-In) requires 180 days; India's data law asks for 1 year from May 2027. Logs never contain what people wrote |
| Expired sign-ins | 30 days after they end | Deleted | |
| Background job records | 30 days | Deleted | |
| Database backups | Daily ones 35 days; one a month for 12 months | Deleted automatically | Once something is deleted, the last backup holding it expires within 12 months. The privacy notice says so |
| A school leaves, or its contract ends | Students are offered a copy of their own work; then the school's student data is deleted within 60 days, confirmed in writing | The standard US school data agreement (NDPA) | |
| Anyone found to be under 13 | Deleted at once, except safety evidence | US children's privacy law (COPPA) |
Two things make deleting real.
- Backups can't be edited. So we keep a short list of what was deleted: account numbers only, no content. It is kept in file storage outside the database, so restoring an older copy of the database doesn't lose it. If a backup is ever restored, everything on the list is deleted again straight away, so nobody who was deleted comes back.
- Deleting a lot must not slow the site. The clean-up runs at night in small batches, so it never competes with students. Once a table passes about a million rows, it is split by month and an old month is removed in one step (docs/27 §16).
What it costs, or saves. These numbers slow the one cost in §2 that grows on its own: files. The full-quality originals are about half of that growth, and they go after 30 days.
6. The monthly night window (D9)
What needs updating every month. The server's operating system and programs receive security fixes all the time. Most install themselves every night without stopping anything. A few, mainly fixes to the core of the operating system, only take effect after a restart, so once a month the server needs to start fresh. The database receives its own updates from Amazon a few times a year.
When. Sunday, 3:00 to 3:30 am US Eastern: the quietest half hour for US schools, and the pilot schools are in the US. We check it against the first month of real sign-in times, and move it when schools in other time zones join.
The server: replaced, not restarted
Restarting our one server means 1 to 2 minutes with no Thnkrr. So instead we build a fresh, fully updated server, move the site to it, and keep the old one switched off for a day as a spare.
| Step | What happens | What students notice |
|---|---|---|
| 1 | A new server is made from the same recipe as the first, with every update already installed. The same version of Thnkrr goes on it, plus the one before (so going back still works), and the same automatic checks as a release (§4) run against it | Nothing |
| 2 | Background work (AI checks, emails, videos) moves to the new server. A video half-converted on the old one starts again on the new one | Nothing |
| 3 | The server's fixed internet address moves to the new server, which takes a few seconds | The few requests in flight in those seconds fail. The app repeats page loads by itself. Anything being sent shows "Couldn't send, try again", with the text still in the box |
| 4 | The old server is switched off, not deleted. If anything is wrong in the next 24 hours, the address moves back | Nothing |
| 5 | After 24 hours, the old server is deleted | Nothing |
This costs about 5 cents a month: two servers for an hour. And it gives us something for free: every month we prove we can rebuild the server from scratch, which is the disaster drill most small teams never get round to.
It works only if the server holds nothing that exists nowhere else. That is already true for
uploads (they go straight to file storage), secrets (Parameter Store) and logs (CloudWatch).
One thing needs work: the server's security certificate. The server gets its own name,
origin.thnkrr.com. It proves it owns that name through our DNS. So a new server can get its
certificate before it takes the address (B17). A video being converted at the moment of the
swap needs nothing: it simply starts again.
The database: Amazon's updates, on our schedule
| Kind of update | How often | What we do | What students notice |
|---|---|---|---|
| A new minor version of Postgres | A few times a year | Automatic upgrades stay off. In our window, Amazon makes an upgraded copy kept in step with the real one (its "blue/green" feature). We test the copy, then switch to it | Under a minute when saving pauses. What is already on screen stays |
| Amazon's own required fixes to the database machine | Rare, and announced weeks ahead in the console | Scheduled into our window, with the banner set the day before | A few minutes when Thnkrr can't save or load anything new. This is the one real outage left. A standby database removes most of it (§7, stage 2) |
When the server or the database is away for a few minutes
Even with all of the above, something can be away for a few minutes, planned or not. This is what students see then, and what each part needs:
| What | How it works | Where it stands |
|---|---|---|
| The website still opens | It is files on Amazon's delivery network (CloudFront), not on our server | Already true |
| A banner explains | A small status file sits on the delivery network, not on our server, so it works while the server is away. The app shows "Thnkrr is updating. Back in a few minutes. Nothing you've written is lost." as soon as the server stops answering. Staff can also set a planned message ahead of time | To build (B16) |
| Nobody gets signed out | Only a real "your sign-in has ended" answer signs anyone out. A server that doesn't answer signs nobody out | Already true (checked in the code) |
| Open pages keep their content | What a student has loaded stays on screen. The app retries quietly every 20 seconds and removes the banner when we're back | To build (B16) |
| Nothing typed is lost | A post, comment or message being written stays in its box. The Send button says "Thnkrr is updating, try again in a minute" | To build (B16) |
| A friendly page, not an error | When our server doesn't answer, the delivery network gives the app a short "updating" reply it recognises. Public profile pages show a plain "This page is being updated" page | To build (B16) |
| Urgent legal requests still get in | The complaint and intimate-image removal forms show an email address to use while we're away. The time the email arrives counts toward the deadline | To build (B16) |
| The server survives the database going away | Today it doesn't. We tested it: when the database restarts, our server program shuts down completely instead of waiting and reconnecting. The fix is one small change, with a test that restarts the database underneath a running server | Fix first (B16) |
| Alarms don't wake anyone for planned work | The window script pauses the "site down" alarm for its half hour and switches it back on at the end, whatever happens | To build (B17) |
Deadlines keep running through the window. A 2-hour legal clock doesn't stop for our update. That is why the window is half an hour at the quietest time, why the person doing it is the on-call person named in A-step 14, and why its last step is to open the review list and check that nothing urgent arrived.
What this adds up to. Releases and server updates: no outage, at most a few seconds in which a request is repeated. Database version updates: under a minute, a few times a year. Amazon's rare required database fixes: a few minutes, announced ahead. Unplanned failures are rarer. If the server's hardware fails, Amazon moves it to new hardware within minutes (the recovery alarm in A-step 21), or the swap script (B17) builds a new server in about 15 minutes. If the database machine fails, Amazon usually restarts it within minutes. At worst, we restore from backup, which we practise (A-step 35). A standby database (stage 2) cuts that case to about a minute.
7. How this plan grows
The plan above is sized for the pilot. Each step up has a trigger: a measurement that tells us it is time, not a date. Nothing is bought early. C5 lists the detailed triggers for every parked piece.
| Stage | When | What changes | Monthly effect |
|---|---|---|---|
| 1. Pilot launch | Now, up to about 300 students | Everything in this plan: one server and one database, releases with a trial (§4), the clean-up numbers (§5), the monthly server swap and the banner (§6) | about $66 to $75 |
| 2. More pilots | About 300 to 3,000 students, or the first paid school contract | A standby database; a small video server; a staging server; a night-and-weekend on-call rota (details below) | +$12 standby database, +$12 video server, +$5 to $10 staging |
| 3. 3,000 to 5,000 students using it each day | The server's processor is regularly above 70% at peak, or the load test (A-step 34) at the new size misses its marks | First the "before a second server" work from the load audit (PLUG-AND-PLAY-AUDIT-2026-10-10.md, Phase D): per-person limits shared between servers, and new messages and alerts pushed to the browser at once instead of each open tab asking every 30 seconds. Then a second server behind a load balancer. From here there is no window for the servers at all: releases and updates go one server at a time. The database moves up a size when its measurements say so. Big tables are split by month once past about a million rows | +$45 load balancer and second server, +$12 to $35 bigger database |
| 4. 10,000+ users, or schools outside the US | Indian schools join; or a second engineering team; or more servers than one person can look after by hand | Indian users' logs kept within India's reach, which may mean a setup in Mumbai. Containers (Fargate), with releases that roll themselves back. Every piece of AWS written down as templates, so a whole copy can be built in an hour. A read-only copy of the database for search | Priced at the time, with a table like §2 |
Stage 2 in more detail. Each piece has its own trigger:
- A standby database when we pass about 300 students or sign the first paid school. Database updates shrink to about a minute, and a broken database machine no longer takes us down.
- A small video server of its own once videos wait more than 15 minutes in school hours, or the site slows while they convert. A single setting, already built, moves video work there.
- A staging server, a copy of production for trying releases first, switched off when unused, from the first paid school or a second engineer.
- A night-and-weekend rota with phone calls. In stage 1 one named person is on call and gets texts. A rota of several people, with calls that move up the chain when nobody answers, is the only reliable way to meet a 2-hour deadline that starts at 2 am.
How releases, monthly updates and retention change along the way:
| Stage 1 | Stage 2 | Stage 3 | Stage 4 | |
|---|---|---|---|---|
| Releases | Two versions on one server, with the 1-in-10 trial | The same, tried on staging first | One server at a time behind the load balancer, with the same checks and trial | Containers; Amazon rolls back by itself when checks fail |
| Monthly updates | Server swapped with no outage; database a few minutes, a few times a year | Database about 1 minute | No window for servers; database about 1 minute | The same |
| Retention | The §5 numbers, with nightly clean-up | The same | Old months removed in one step once tables are split | The same, plus copies kept in India if the lawyer says so |
Part A: What you do by hand
You = the founder or CTO, in the AWS console, the domain's DNS, or a browser. Where a step says "run the script", the agent has written it (Part B) and you paste one command into Session Manager. Each step ends with Check: the thing you must see before moving on. Do them in order.
A1. The account (Phase 0, about half a day plus Amazon's SES wait)
| # | Step | Check |
|---|---|---|
| 1 | Sign in as root. Security credentials → turn on MFA with an authenticator app. Make sure root has no access keys. Account → Alternate contacts: fill billing, operations, security | Signing in as root asks for the 6-digit code. Access keys section says "No access keys" |
| 2 | IAM Identity Center → enable → create a user for yourself and one for the CTO, each in an Administrators permission set, each with MFA | You can sign in through the Identity Center portal, open the console, and root is no longer needed |
| 3 | Billing → Credits: read the expiry date and "applicable products". Write both into docs/28 (give them to the agent) | The two values are in docs/28 |
| 4 | Budgets: create two monthly cost budgets, $80 and $95 (D10), emailing OPS_EMAIL. Cost Anomaly Detection: create a monitor for all services with an alert to the same address | Both budgets show "OK"; the anomaly monitor shows "Active" |
| 5 | CloudTrail → create a trail for all regions, management events, into a new bucket thnkrr-cloudtrail-<account id>; leave the first copy free | After 15 minutes, the trail's bucket contains a folder with today's date |
| 6 | S3 → Block Public Access settings for this account → all four on. EC2 → Settings → EBS encryption → "Always encrypt new EBS volumes" on | Both pages show the setting enabled |
| 7 | Set the console region (top right) to US East (N. Virginia) and keep it there for every step below | The header says "N. Virginia" |
| 8 | SES → Account dashboard → Request production access: Transactional, website https://thnkrr.com, describe sign-up/consent/alert emails | An email from AWS says the request is received; within about a day, another says production access is granted. The dashboard no longer says "Sandbox" |
| 9 | SES → Identities → Create identity → Domain thnkrr.com, Easy DKIM 2048. Add the 3 CNAME records it shows at your DNS provider. Then Custom MAIL FROM mail.thnkrr.com (add its MX and TXT). Then add a DMARC TXT at _dmarc.thnkrr.com: v=DMARC1; p=none; rua=mailto:<OPS_EMAIL> | The identity shows Verified and DKIM Successful; MAIL FROM shows Verified; a DMARC lookup tool shows the record |
| 10 | Bedrock: run the data-retention command from docs/27 A8 in CloudShell (mode none), then delete the temporary API key. Leave model invocation logging off | The command returns success; Bedrock → API keys shows none |
| 11 | AWS Organizations → create (free) → Policies → AI services opt-out policy → opt out of all services → attach to the root. Optional, if you want the one-region rule enforced: an SCP denying actions outside us-east-1 using Amazon's example (the agent gives you the text) | The policy shows "Attached"; a test call to a service in Ohio (the agent provides a harmless one) is denied if the SCP is on |
| 12 | Service Quotas: write down the values for openai.gpt-oss-safeguard-20b requests and tokens per minute, and Rekognition DetectModerationLabels TPS | The numbers are in docs/28 |
| 13 | Domain: thnkrr.com is ours and fully under our control. Route 53 → Hosted zone thnkrr.com, then at the registrar change the nameservers to the four Route 53 gives you. Copy any records the domain already has (email, verification) into the hosted zone first | After the nameserver change (up to 48 h), dig NS thnkrr.com shows the Route 53 servers |
| 14 | Decide and write into docs/28: the on-call person, the phone number for alarm texts, the inbox for alarm emails (OPS_EMAIL) | Written in docs/28 |
| 15 | Tick the decisions in §1 with the CTO (D11 can wait until Gate 2) | Each row has a name and date |
Gate 0 ticked by: founder (decisions), CTO (steps 1 to 14).
A2. The AWS side (Phase 2, about one day; after the agent's Part B is merged)
| # | Step | Check |
|---|---|---|
| 16 | ACM (us-east-1) → request a public certificate for thnkrr.com, www.thnkrr.com, media.thnkrr.com, DNS validation → "Create records in Route 53" (one click) | Certificate status Issued |
| 17 | VPC → Security groups: thnkrr-api (inbound 443 and 80 from the CloudFront origin-facing managed prefix list only; nothing else), thnkrr-db (inbound 5432 from thnkrr-api only) | Each group's inbound tab shows exactly those rules; no 0.0.0.0/0 on thnkrr-api except none |
| 18 | RDS → Create database: PostgreSQL 17, db.t4g.micro, 20 GB gp3 with autoscaling to 100 GB, not publicly accessible, security group thnkrr-db, backups 35 days, deletion protection on, encryption on, Database Insights standard, automatic minor upgrade off, maintenance window Sunday 07:00 to 07:30 UTC (3:00 am US Eastern, §6), database and user thnkrr, master password managed in Secrets Manager: no (keep it simple; store it in Parameter Store in step 20). Parameter group: log_min_duration_statement = 1000, shared_preload_libraries includes pg_stat_statements | Status Available; the endpoint is visible; Backups tab shows retention 35 |
| 19 | IAM → Roles → Create role thnkrr-api-server for EC2 with the policy the agent wrote (deploy/aws/iam-api-server-policy.json: the two media buckets, the log-copy and evidence buckets (write only), the one Route 53 record for the origin certificate (B17), Bedrock InvokeModel on the listed models, CloudFront invalidation, Parameter Store read on /thnkrr/prod/*, CloudWatch agent, Session Manager) | The role exists with that policy and the two AWS-managed policies AmazonSSMManagedInstanceCore and CloudWatchAgentServerPolicy |
| 20 | Systems Manager → Parameter Store: create the secrets under /thnkrr/prod/ as SecureString (DATABASE_URL, COOKIE_SECRET, TOKEN_SECRET, DATA_ENCRYPTION_KEY, SMTP_URL, EDGE_SHARED_SECRET). Generate the random ones with the commands in deploy/README.md. Put the same values in the password manager | Six parameters listed; the password manager has all six |
| 21 | CloudFormation → Create stack → upload deploy/aws/server.yml (the server's recipe, B17: Ubuntu 24.04 ARM, t4g.medium, 30 GB gp3, role thnkrr-api-server, security group thnkrr-api, IMDSv2 required, no key pair), then launch the first server from its launch template. Then Elastic IPs → Allocate → Associate to it. Status checks → Actions → Create status check alarm with "Recover this instance" | Instance Running, 2/2 checks passed; the Elastic IP shows the instance; the recovery alarm exists |
| 22 | Connect → Session Manager. Paste the one command from deploy/README.md that runs scripts/server-setup.sh (installs Node, ffmpeg, whisper.cpp, Caddy, the CloudWatch agent, creates the thnkrr user, fetches secrets from Parameter Store into /etc/thnkrr/api.env, installs the systemd unit and Caddyfile) | The script ends with "SETUP OK" and prints the versions. systemctl status caddy amazon-cloudwatch-agent both active |
| 23 | Still in Session Manager: sudo /opt/thnkrr/api/scripts/deploy.sh origin/dev/pilot. Answer yes to the snapshot question only after taking one (RDS → Actions → Take snapshot; first deploy: the database is empty, so answer yes) | The script ends with "OK: <sha> is live and healthy". curl -s localhost:3001/health/ready returns "ok":true with the git version |
| 24 | CloudFormation → Create stack → upload deploy/aws/observability.yml (log groups with 90-day retention, metric filters, the 10 alarms, the SNS topic, the dashboard, the Route 53 health check). Parameters: OPS_EMAIL, on-call phone, health-check target. Then open the confirmation email SNS sends and click confirm | Stack CREATE_COMPLETE; SNS → Subscriptions shows the email Confirmed and the SMS Confirmed; CloudWatch → Alarms shows 10 alarms, all OK or Insufficient data; GuardDuty → Summary shows it enabled, and EventBridge shows the rule thnkrr-guardduty-findings |
| 25 | CloudFormation → Create stack → upload deploy/aws/storage.yml (web bucket; the two media buckets with their CORS and lifecycle rules from docs/27 §8; thnkrr-evidence with versioning and Object Lock compliance; thnkrr-logs for the monthly log copies, deleted after 1 year; status.json, maintenance.json and maintenance.html in the web bucket, B16) | Stack CREATE_COMPLETE; the evidence bucket's Properties tab shows Object Lock Enabled, Versioning Enabled |
| 26 | CloudFormation → Create stack → upload deploy/aws/edge.yml (the two CloudFront distributions: web with the /api/*, /in/*, /school/* behaviours to the server, CachingDisabled + AllViewerExceptHostHeader, the X-Thnkrr-Edge secret header, the SPA CloudFront Function, the response-headers policy; media with the 5-minute cache). Parameters: certificate ARN, Elastic IP hostname, the edge secret. WAF parameter off for now (D6) | Stack CREATE_COMPLETE; both distributions Deployed; the web one lists three API behaviours |
| 27 | CloudFormation → Create stack → upload deploy/aws/backup.yml (AWS Backup vault with Vault Lock, a plan taking a monthly RDS snapshot kept 12 months, §5) | Stack CREATE_COMPLETE; the vault shows Vault Lock: Locked (after its 3-day cooling period, Compliance) |
| 28 | Fill the remaining settings in /etc/thnkrr/api.env from .env.example's PROD lines (CLOUDFRONT_MEDIA_DISTRIBUTION_ID, bucket names, S3_PUBLIC_URL, WEB_INDEX_URL, OPS_EMAIL, MAIL_FROM, SUPPORT_EMAIL), then sudo systemctl restart thnkrr-api | journalctl -u thnkrr-api -n 20 shows the startup line with no [storage] unavailable |
| 29 | The web: in the thnkrr-web folder on your laptop run the one command in its README (VITE_API_URL=https://thnkrr.com/api pnpm build, then the upload and invalidation commands the agent wrote in scripts/publish-web.sh) | The script ends with "PUBLISHED <sha>" |
| 30 | DNS (Route 53): thnkrr.com and www → the web distribution (alias A/AAAA records), media.thnkrr.com → the media distribution (alias), origin.thnkrr.com → the Elastic IP (plain A record; only CloudFront may connect, A-step 17). Do not create api.thnkrr.com | https://thnkrr.com loads the sign-in page; https://thnkrr.com/api/health returns {"ok":true}; https://media.thnkrr.com/ returns a CloudFront error page, not a bucket listing |
| 31 | Test an alarm: CloudWatch → Alarms → thnkrr-site-down → Actions → Set alarm state to ALARM for a minute | The email and the text arrive within two minutes, each containing the runbook link; the alarm returns to OK on its own |
| 32 | Hand the agent the list of every click above with any deviation, so deploy/README.md matches reality | README updated and merged |
Gate 2 ticked by: CTO (steps 16 to 32).
A3. Prove it (Phase 3, about two days with the agent)
| # | Step | Check |
|---|---|---|
| 33 | Smoke test with staff accounts only, following deploy/runbooks/smoke-test.md (docs/27 B6 and B7: sign up, consent email, post text/photo/video, approve, take down, /in/ while signed out, intimate-image test request, suspend a test student, PDF on a test order, hide a profile, two reviewers on one item) | Every row in the runbook's table ticked; the removed photo's link is dead within 5 minutes |
| 34 | Load test: agree a 30-minute window; the agent runs pnpm load:k6 from a laptop against the live site before any student exists | The report meets the pass marks: 95th percentile under 500 ms for feed and profile, under 300 ms for typeahead, no 5xx, "busy" under 1% of searches, server and database CPU under 70%. If not: resize (D2 up one step) and re-run |
| 35 | Restore drill: RDS → Databases → thnkrr → Actions → Restore to point in time (yesterday 23:00) into thnkrr-drill, db.t4g.micro, same security group. Give the agent its endpoint; the agent runs the integration suite against it and times everything. Then delete thnkrr-drill (no final snapshot) | The suite passes against the restored copy; the time from click to green is written in docs/28; thnkrr-drill is gone |
| 36 | Also open one monthly log copy: S3 → thnkrr-logs → any object → Download | The file opens |
| 37 | Failure drills (agent drives, you watch the phone): a deploy that fails readiness; sudo systemctl kill thnkrr-api; stand-in AI switched off for 6 minutes; sudo reboot; the database rebooted from the RDS console; replace-server.sh while the agent's 20-a-second loop runs | Expected, in order: the old version keeps serving and the script reports; the service is back within 5 s and no alarm fires; "checking is busy" appears and the Moderation-stalled alarm texts you within 5 minutes; the site is back within 2 minutes and the Site-down alarm fires then clears; the banner shows, nobody is signed out, and the site recovers by itself when the database is back; failures only in the seconds the address moves |
| 38 | Security check: open https://observatory.mozilla.org for thnkrr.com; try https://<elastic ip>/api/health from your laptop | Grade A or better; the direct address times out (only CloudFront may reach the server) |
| 39 | Read the ten alarm runbooks and the incident runbook; put the on-call phone and the status page address in your phone | You can name, from memory, what to do when the Site-down text arrives |
Gate 3 ticked by: CTO (numbers) and founder (plain-words summary from the agent).
A4. Go live (Phase 4)
| # | Step | Check |
|---|---|---|
| 40 | In Session Manager: create the first staff accounts with the ops scripts; give hats with pnpm staff:role <email> <hat> (and a --backup for each). Never run pnpm db:seed | pnpm staff:role --list shows every hat with a primary and a backup |
| 41 | Vercel: archive the thnkrr-web project (keep thnkrr-demo if the founder uses the demo) | The old Vercel address no longer serves the app |
| 42 | For seven days, spend 15 minutes on the CloudWatch dashboard and the alarm history; note anything odd for the agent. On day 7, agree the final alarm thresholds | Thresholds recorded in docs/28 |
| 43 | The week real students arrive: if D6 says so, set the EnableWaf parameter in the edge stack to true and update the stack | The web ACL shows in CloudFront's distribution settings; a request with <script> in a query string is blocked |
| 44 | Then the docs/27 Phase C items: the real Bedrock measurements and the 50 → 150 → 300 ramp; the founder's names and registrations | As docs/27 |
Gate 4 ticked by: founder, before the first real school signs up.
A5. Ongoing
| When | What you do | Check |
|---|---|---|
| Monthly, Sunday 3:00 am US Eastern | Run scripts/replace-server.sh (§6) from CloudShell. Then open the review list | The script ends with "SWAPPED"; nothing urgent arrived during the window; the next run deletes the old server |
| When Amazon announces a database update (a few times a year) | Schedule it into the window. The day before, pnpm status set "Short update on Sunday 3:00 to 3:30 am. Saving may pause for a few minutes.". Follow database-upgrade.md, then pnpm status clear | The database's Maintenance tab shows nothing pending; the banner is gone |
| Quarterly | Restore drill (step 35), access review in Identity Center, pnpm outdated review with the agent, Postgres minor version check, alarm thresholds review | Each written in docs/28 with the date |
| Once, before the clean-up job starts (B11) | The agent runs it in dry-run against a copy of the pilot data; you read the counts; then it switches the retention_job flag on | Dry-run report read and approved |
| When the founder or lawyer changes a §5 number | Tell the agent; it changes the setting and runs a dry run first | Dry-run report read |
Part B: What the agent builds, cleans up and updates
Branch feat/production-ready in worktrees off dev/pilot in both repos; pull requests
into dev/pilot. Migrations numbered from 0035 (0031 to 0034 came with the load work), additive only. Each item has its
"done when"; the agent reports in plain words at each gate.
B1. Build: /api prefix and one front door
API_PREFIXsetting (production/api), applied insrc/server.tsto every route group except the public pages/in/*and/school/*, which stay at the root; CORS only whenAPI_PREFIXis unset (local development).- Web:
src/lib/api.tsreadsVITE_API_URL=https://thnkrr.com/api; e2e config parameterised. - Done when: integration tests pass with the prefix set and unset; the web e2e suite runs against the prefixed API.
B2. Build: cookies and sessions
- Cookie names
__Host-thnkrr_sessionand__Host-thnkrr_csrfin production (no Domain, Path/, Secure);COOKIE_DOMAINrefused when__Host-is on. - Anti-forgery token: a signed value bound to the session (OWASP signed double-submit), using
COOKIE_SECRET. - Migration:
sessions.last_seen_at; idle limit 14 days for members, 12 hours for staff hats; 30-day absolute limit kept; sign-out clears with matching attributes. - Done when:
test/integration/auth-flow.test.tscovers idle expiry (both kinds), a forged token, and an old-format cookie answered with a clean 401.
B3. Build: edge secret on every API route
- The
X-Thnkrr-Edgesecret is required on/api/*as well as/in/*and/school/*; without it the request is refused with a plain 403 (except/healthfor the local health check). - Done when: unit tests in
client-ipcover CloudFront-with-secret, forged header, and local health.
B4. Build: health endpoints
/healthunchanged./health/ready: database reachable, job queue connected, returns the git version andAPI_PREFIX.- Done when: tests for both;
deploy.shand the Route 53 health check target/health/ready.
B5. Build: structured logs and request ids
- Replace every
console.*undersrc/(16 files) with the Fastify logger, child-logged per module. - Request id from
X-Amz-Cf-Id(CloudFront) or generated; returned asX-Request-Id; on every log line; included in the "Something went wrong" message body. - Redaction: cookies, authorization,
set-cookie, email addresses, any field namedtoken,password,secret. - Request log fields: method, route pattern, status, duration, hashed user id, hashed session id, request id. Slow (> 1 s) at
warn, 5xx aterror. A startup line with the git version. LOG_FILEsetting; in production the app writes rotated JSON to/var/log/thnkrr/api.log(via systemdStandardOutput=append:), which the CloudWatch agent collects.- Metric lines in CloudWatch embedded-metric format: 5xx count, 503 "busy" count, Bedrock error and latency, moderation queue age, video job duration, search latency.
- Done when: a test asserts no
console.remains undersrc/(scripts excepted); a test asserts the masking of a cookie and an email; metric lines validate.
B6. Build: feature switches
- Migration:
feature_flagstable; read every 30 seconds;pnpm flag list|set|unset; each change audit-logged. - Flags:
video_uploads,mentor_messaging(env still wins if set),search_fuzzy_fallback,retention_job. The banner is not a flag: it lives instatus.json(B16), so it works while the server is away. - Done when: integration test flips
video_uploadsoff and sees the composer refuse politely within 30 seconds without a restart.
B7. Build: the server scripts and AWS templates (the files you upload or run in Part A)
scripts/server-setup.sh: everything in docs/27 §14 steps 14 anddeploy/README.md"Once" sections as one idempotent script, plus the CloudWatch agent install and its configdeploy/cloudwatch-agent.json(api log, Caddy access log, journal errors; memory and disk metrics), plus fetching/thnkrr/prod/*from Parameter Store into/etc/thnkrr/api.env(chmod 600). Ends with "SETUP OK".deploy/aws/iam-api-server-policy.json(least privilege, as listed in A-step 19).deploy/aws/observability.yml: log groups (90-day retention), metric filters, the 10 alarms from §C3 with runbook links in their descriptions, SNS topic with email and SMS subscriptions, the dashboard, the Route 53 health check on/health/ready, the GuardDuty detector and an EventBridge rule sending its medium and high findings to the SNS topic (D12).deploy/aws/storage.yml: the buckets (web, two media, evidence, logs) with their CORS, lifecycle, versioning and Object Lock settings; the three B16 files in the web bucket; the monthly CloudWatch log export (EventBridge Scheduler →CreateExportTask, or the agent's equivalent that needs no Lambda).deploy/aws/edge.yml: both CloudFront distributions, behaviours, policies, the SPA function, the response-headers policy (HSTS, CSP, nosniff, referrer policy), anEnableWafparameter that adds the web ACL with Amazon's Core rule set and Known bad inputs whentrue.deploy/aws/backup.yml: AWS Backup vault with Vault Lock and the monthly plan kept 12 months (§5).scripts/publish-web.sh(web repo): build, upload to the web bucket, invalidate/index.html, print "PUBLISHED <sha>".deploy/aws/scp-one-region.jsonand a harmless denied-call example for A-step 11.- Done when:
cfn-lintpasses on every template;server-setup.shruns clean twice on a fresh Ubuntu 24.04 ARM VM (Multipass or an EC2 spot box the CTO approves); each template's parameters are documented indeploy/README.md.
B8. Build: background work cannot starve the API
- Memory cap for the ffmpeg and whisper children (systemd slice
MemoryMaxor a per-spawn limit); keepnice; video jobs stay one at a time. - Done when: the video integration test's timing assertion on a concurrent API request holds while a 15-minute test video processes.
B9. Build: releases with no outage (§4)
scripts/deploy.shv2: build into/opt/thnkrr/releases/<sha>;db:migrate; startthnkrr-api@<spare port>(3001 ↔ 3002) with a new settingBACKGROUND_WORK=off(serve requests, register no workers); wait for/health/ready; runscripts/smoke.sh --target 127.0.0.1:<port>as a robot account (pnpm staff:smoke-account: a hidden test school, a view-only moderator hat, never shown in staff lists).- Trial: Caddy
lb_policy weighted_round_robin 9 1across both instances, thencaddy reload.scripts/compare-versions.shreads Caddy's access log per upstream for 10 minutes (5xx share, 95th-percentile time) and reverts the weights automatically on the §4 rule. - Switch: weights 0/1. Background work hands over through a Postgres advisory lock: the old instance releases it on a signal, the new one takes it, so only one ever holds it. The old instance stays up 30 minutes, then stops. Keep the previous two releases.
scripts/rollback.sh: swap the weights back (seconds, while the old instance is warm) or start the previous release on the spare port the same way (about a minute).scripts/check-migrations.tsin CI: fails onDROP,RENAME, column type changes orSET NOT NULLunless the migration carries a-- second-half:marker;deploy.shasks the snapshot question only for marked releases.- CI compares
packages/contractswith the last release tag and fails when a field is removed or renamed without the same marker. deploy/Caddyfileanddeploy/thnkrr-api@.serviceupdated; the Caddy site servesorigin.thnkrr.com(B17), notapi.thnkrr.com.- Web:
scripts/publish-web.shuploads new hashed files without deleting old ones, then removes files older than 30 days that the current and previous release don't use; writesversion.json. The app reads it every 10 minutes and on tab focus, shows "A new version of Thnkrr is ready", and reloads on the next page change. - Done when: on the test VM, a
curlloop at 20 requests a second across a deploy shows zero failures; a release with one route broken is stopped at step 3; a release failing 5% of requests is reverted at step 4 within a minute; a Playwright test sees the new-version note and the switch on the next page change. The sequence is indeploy/runbooks/deploy.md.
B10. Build: search check at pilot size
- Web: the typing box cancels the previous in-flight request on each keystroke (TanStack Query
signal). pnpm search:bench --profiles 10000 --users 3000: generates pilot-size data, replays the measured traffic mix, prints median and 95th percentile per query type and the rate at which "busy" first appears.- Done when: the numbers for a 1-processor, 1 GB Postgres are in docs/28 with the recommendation (stay on db.t4g.micro or move up one size).
B11. Build: the clean-up jobs (§5; switch off until the dry run is read)
- One setting per number in §5, defaults as there. One nightly job runs every rule in batches of 5,000 with a pause between, has a dry-run mode that prints the counts, and writes each run's counts to the audit log. Behind the
retention_jobflag (B6). - Every rule skips anything under a hold: a safety case, a legal order, an open complaint or appeal, an account action, or a new legal hold the Grievance Officer can place on a person or an item (a migration; the casework screens get the button).
- Delete my account (member settings) and staff closure: hidden within 24 hours; content gone at 30 days; the legally kept copy moves to a separate locked table readable only by the Safety Admin and Child Safety Lead, deleted after 1 year.
- Download my work (member settings): a zip of the profile, posts, photos and videos; needed before a school leaves.
- Moderation records: the text copy cleared 90 days after an approval; the record deleted after 2 years. Removed content kept 1 year (the code's 180-day constant becomes 365).
- Full-quality originals deleted 30 days after approval, in code rather than an S3 rule, because holds live in the database.
- Notifications after 90 days; expired sessions after 30; pg-boss
archiveCompletedAfterSeconds1 day anddeleteAfterDays30, set explicitly insrc/jobs/boss.ts. - Dormant accounts: the two warning emails, then closure.
- Deletion list: ids of deleted accounts and items, written to
thnkrr-logs/deletions/in S3 (outside the database, so a restore can't roll it back) and kept until the oldest backup has expired (12 months).pnpm restore:replay-deletionsis a step inrestore-drill.mdandrebuild-server.md. audit_logrows older than 3 years deleted, except rows linked to a hold.- Done when: integration tests prove a held item is never deleted by any rule; the restore drill replays the deletion list; a dry run on a copy of the pilot data is read by the CTO before the flag goes on.
B12. Build: load test, smoke test and drills
load/folder with k6 scripts for the mix in A-step 34 and a pass/fail report;pnpm load:k6 --target https://thnkrr.com.deploy/runbooks/smoke-test.mdwith the full table for A-step 33, andscripts/smoke.shfor the parts a script can check (/health/ready, a signed-out/in/link, a dead media link after takedown).- The failure-drill script for A-step 37 (now including the database reboot and the server swap under traffic) and the expected observations.
- Done when: k6 runs against the local prod-like compose and produces the report; the smoke script passes locally.
B13. Build: runbooks and the decision record
deploy/runbooks/: the ten alarm runbooks andsecurity.md(§C3),deploy.md,rollback.md,night-window.md,database-upgrade.md,retention.md,restore-drill.md,access.md,incident.md(paging, status page, blameless write-up template),rebuild-server.md,smoke-test.md.docs/28-production-ready.md: the decisions with names and dates, the benchmark and load-test numbers, the restore-drill time, the alarm thresholds, the credits facts; indexed indocs/00-overview.md; sources grouped at the end, no inline links.- Done when: every alarm's description in
observability.ymllinks to its runbook; docs/28 merged.
B14. Clean up
- Remove every
console.*undersrc/(B5). .env.example:COOKIE_DOMAINPROD line becomes "leave unset (host-only cookies)"; addAPI_PREFIX,LOG_FILE;EDGE_SHARED_SECRETnote covers all API routes;MENTOR_MESSAGING_ENABLEDdocumented as overridden by the flag.deploy/README.mdrewritten aroundserver-setup.sh, the templates anddeploy.shv2; the old hand-typed install commands removed.deploy/Caddyfile: remove theapi.thnkrr.comsite and access from anywhere but CloudFront.- Web: remove the
api.thnkrr.comassumption anywhere it appears (README, e2e config, env examples). - CI (
.github/workflows/ci.ymlin both repos): add the "no console" test,cfn-linton the templates, and the prefixed-API e2e run; keep everything else. - Remove the now-unused per-IP fallback comments that describe
trustProxyfor Caddy-only, replacing them with the CloudFront-only description.
B15. Update
- docs/27: a short note at the top of §7, §11 and §14 saying "replaced by docs/28 for: cookie domain, API address, deploy script, logging, alarms, templates"; nothing erased.
- docs/25 §6: add the pilot-size benchmark numbers and the "stay or resize" conclusion.
- docs/22 §1 and MODERATION-PLAN §6: add the smoke-test and drill references where they list launch checks.
FOUNDER-UPDATEstyle one-pager at the platform root after Gate 3, in plain words, with the numbers.
B16. Build: when the server or database is away (§6)
- First, the crash. Give the API's database pool an
errorlistener that logs and never exits (src/db/client.ts). Startup retries the database for up to 2 minutes instead of exiting. Check pg-boss reconnects the same way. Test: an integration test restarts a throwaway Postgres under a running server and sees requests answer "updating" (503) during the restart and succeed after it, with no process restart. status.jsonin the web bucket.pnpm status set "…" --from <time> --until <time>writes it, andpnpm status clearempties it. The app reads it on load, every 5 minutes, and at once when a request can't reach the server.- "Updating" mode in the web app when requests get no answer or the edge's maintenance reply: the banner, cached data kept on screen, a retry every 20 seconds, the composer keeping its text with the button explaining. A real 401 is still the only thing that signs anyone out.
- CloudFront custom error responses (in
edge.yml):/api/*502, 503 and 504 →/maintenance.json(status 503, 10-second cache);/in/*and/school/*→/maintenance.html. - The complaint and removal forms show the safety email address when the server can't be reached.
deploy/runbooks/night-window.mdsays the on-call person watches that inbox and enters each arrival in the Orders tab with its received time. - Done when: a Playwright test stops the API for 2 minutes and sees the banner within 30 seconds, nobody signed out, typed text kept, the banner gone after the API returns, and the legal forms showing the email address.
B17. Build: the monthly server swap (§6)
deploy/aws/server.yml: a launch template (latest Ubuntu 24.04 ARM image, t4g.medium, rolethnkrr-api-server, security groupthnkrr-api, IMDSv2 required, user data runningserver-setup.sh).scripts/replace-server.sh(run from CloudShell with the CTO's sign-in): launch from the template; wait for "SETUP OK"; deploy the running release; runsmoke.shagainst it; pause the Site-down alarm; hand over background work; move the Elastic IP; checkhttps://thnkrr.com/api/health/readythrough CloudFront; stop the old server (not terminate) with adelete-aftertag; switch the alarm back on, also on failure. Ends with "SWAPPED".--backmoves the address back. Each run first deletes stopped servers whosedelete-afterhas passed.- Caddy's certificate for
origin.thnkrr.comby DNS check through Route 53 (Caddy's Route 53 module). The server role may change only that name's validation record. The edge template's origin becomesorigin.thnkrr.com. server-setup.shturns on automatic security updates with automatic restart off.- RDS: automatic minor upgrade off; maintenance window Sunday 07:00 to 07:30 UTC (3:00 am US Eastern).
deploy/runbooks/database-upgrade.mdwalks through Amazon's blue/green upgrade. - Done when: on the test account,
replace-server.shunder the 20-requests-a-second loop shows failures only in the seconds the address moves;--backworks; the stopped server is deleted on the next run after 24 hours.
Gate 1 ticked by: CTO, after CI is green on both repos, B7's scripts and templates pass their checks on a fresh VM, and the benchmark numbers are in docs/28.
Part C: Order, alarms, parked items
C1. Order and time
| Phase | Who | About |
|---|---|---|
| 0. Account (A1) | founder, CTO | half a day + Amazon's SES wait |
| 1. Code, scripts, templates (Part B) | agent | 8 working days |
| 2. AWS side (A2) | CTO, with the agent's files | 1 day |
| 3. Prove it (A3) | agent, CTO | 2 days |
| 4. Go live (A4) | CTO, founder | half a day, then a week of watching |
| 5. After launch (A5, B11) | agent | clean-up job switched on after its dry run; the monthly swap; quarterly reviews |
Roughly three working weeks from Gate 0 to a production site that staff can test, at about $48 a month (the firewall's $8 starts the week real students arrive).
C2. Gates
| Gate | Ticked by | Means |
|---|---|---|
| 0 | founder, CTO | Decisions D1 to D10 and D12 (approved 11 October 2026); A-steps 1 to 14 |
| 1 | CTO | Part B merged; CI green; scripts and templates proven on a fresh VM; benchmark in docs/28 |
| 2 | CTO, founder | A-steps 16 to 32; a test alarm reached the phone; D11 decided and written in the server's settings (KNOWN_ABUSE_PROVIDER) |
| 3 | CTO, founder | A-steps 33 to 39 pass their checks |
| 4 | founder | Staff accounts, DNS, first week watched; before the first real school |
C3. Alarms (all within the 10 free) and their runbooks
| Alarm | Threshold to start | Runbook |
|---|---|---|
| Site down | Route 53 health check on /health/ready fails 3 times (90 s) | site-down.md |
| Errors | 5xx > 1% of requests for 5 min | errors.md |
| Moderation stalled | Oldest pending review > 5 min | moderation-stalled.md |
| Bedrock failing | Error rate > 5% or p95 > 10 s for 5 min | ai-check-failing.md |
| Server | CPU > 85% for 15 min, or memory > 90%, or disk > 80%, or status check failed | server.md |
| Database | CPU > 80% for 15 min, or free storage < 20%, or connections > 80% of max | database.md |
| Busy | 503 "busy" > 2% of requests for 5 min | busy.md |
| Backups | Automated backup or AWS Backup job failed | backups.md |
| SES bounce > 5% or complaint > 0.1% | email-reputation.md | |
| Cost | Budgets at $80 and $95, raised to $120 and $140 before 1,000 students (D10); anomaly | cost.md |
| Security (GuardDuty, D12; an EventBridge rule, not one of the 10 alarms) | Any medium or high finding | security.md |
C4. Settings that change from docs/27 §11
| Setting | docs/27 value | This plan |
|---|---|---|
COOKIE_DOMAIN | .thnkrr.com | unset (host-only __Host- cookies) |
API_PREFIX | (new) | /api |
EDGE_SHARED_SECRET | set on CloudFront for /in/*, /school/* | set for all three API behaviours |
LOG_FILE | (new) | /var/log/thnkrr/api.log |
BACKGROUND_WORK | (new) | on; deploy.sh starts a new version with off until it takes over (B9) |
| The server's name for CloudFront | api.thnkrr.com | origin.thnkrr.com, certificate by DNS check (B17) |
| The §5 retention numbers | (new) | one setting each, defaults as §5 (B11) |
| Secrets | /etc/thnkrr/api.env only | same file, filled from Parameter Store /thnkrr/prod/* by server-setup.sh; copy in the password manager |
| Everything else | as docs/27 | same |
C5. Parked, with the measurement that would justify each
| Piece | Monthly | Switch it on when |
|---|---|---|
| Bigger database (db.t4g.small / medium) | +$12 / +$35 | The pilot-size benchmark or load test misses its marks |
| Load balancer + a second server | about $45 | Server CPU regularly above 70% at peak (§7, stage 3). The monthly window doesn't need it: the server swap (§6) already avoids the outage |
| Live standby database (Multi-AZ) | +$12 to +$47 | About 300 students or the first paid school (§7, stage 2). Shrinks database updates from a few minutes to about one |
| Small video server of its own | about $12 | Videos wait more than 15 minutes in school hours, or the site slows while they convert. The switch is built (PROCESS_ROLE) |
| Staging server | $5 to $10, switched off when unused | The first paid school, or a second engineer |
| Containers on Fargate + infrastructure as code | about +$80 and a few days | A second environment or second server exists |
| WAF managed rules | $8 | D6: the week real students arrive, or the first attack pattern in the logs |
| Security Hub, Inspector | about $5 to $8 | A school or insurer asks for a security attestation |
| Secrets Manager | $2 | More than one server needs the same secrets |
| Valkey cache | from $6 | A second app process, or a measured hot spot |
| Amazon Transcribe (+ toxicity) | $14 per 1,000 videos | Reviewer time on video passes about an hour a day, or a spoken violation is missed |
| Automatic frame check (Rekognition) | $90 to $108 per 1,000 videos | Same trigger; the "our frames" shape is cheaper at volume |
| OpenSearch | $350 to $735 | Profiles in the millions, or relevance needs Postgres cannot meet |
| Monthly table partitioning | engineering time | A table passes about 1 million rows |
| Lambda, SQS, EventBridge | pennies | An event-driven need appears |
| X-Ray tracing | $5 per million traces | More than one service in the request path |