Moderation - The One-Pager
September 27, 2026
The plumbing is real and running: our own rules, the reviewer queue sorted worst first, message holds, the 2-hour image-removal flow, child-safety case files, complaint clocks, crisis lines, and every screen. What is NOT real yet is the AI brain: every text surface runs on a stand-in that fakes the model's verdict, because no vendor is chosen, no account exists, and no API key is in place. The table below says, per surface, exactly what protects students today, what the real system is by name, and what is missing to switch it on.
Every surface: real today vs what it still needs
| Surface | Protecting students today | The real system, by name | Missing to go live |
|---|---|---|---|
| Posts | Our own rules (contacts, links) + our reviewers; AI is a stand-in | OpenAI Moderation API + gpt-oss-safeguard on Amazon Bedrock + our reviewers | Your stack sign-off, OpenAI + AWS accounts and keys, the bake-off |
| Comments | Same rules + reviewers; AI is a stand-in | OpenAI Moderation API + gpt-oss-safeguard on Bedrock, shown the post for context | Same three: sign-off, keys, bake-off |
| Peer messages | Rules + reviewers; a held message freezes the chat; AI is a stand-in | OpenAI Moderation API + gpt-oss-safeguard on Bedrock, shown recent thread context | Same three: sign-off, keys, bake-off |
| Mentor messages | The feature is OFF (held) | All of the above + Thorn Safer Predict or Hive grooming detection over whole conversations | Your two emails, then a Thorn or Hive quote and contract; stays OFF until live |
| Student bios and goals | Rules + reviewers; old text shows until new clears; AI is a stand-in | OpenAI Moderation API + gpt-oss-safeguard on Bedrock | Same three: sign-off, keys, bake-off |
| Mentor bios | Rules + reviewers; AI is a stand-in | Same models; whether adults get a stricter bar is your open call | Same three, plus that call |
| Educator bio and school page | Rules + reviewers; AI is a stand-in | OpenAI Moderation API + gpt-oss-safeguard on Bedrock | Same three: sign-off, keys, bake-off |
| Short fields (names, titles) | Our own rules only - this row is FULLY REAL, no AI planned | The same rules + a banned-word list for display names | Nothing external; the word list is a small build |
| Avatars and banners | FULLY REAL: a person reviews every image; removed images are fingerprint-blocked from re-upload | Add Amazon Rekognition + a Bedrock vision check, and the free CSAM lists (PhotoDNA, IWF, Arachnid, NCMEC) | Your image-layer approval, the AWS no-training setting, four free applications |
| Post images | FULLY REAL as of September 26: posts carry up to 8 photos, and a person clears every photo post before anyone sees it - the AI can never publish one | The avatar/banner image stack (Rekognition + Bedrock vision + the CSAM lists), re-costed for feed volume | Your image-layer approval; nothing else - the build is done |
Bottom line: three rows are fully real today (short fields; avatars and banners; post images, built this week). Every other row is safe but simulated - unclear items still reach a human, so nothing harmful publishes, but the real AI catches more, cheaper, and faster. One row is off on purpose (mentor messages) until grooming detection exists.
Your September redline, ask by ask
Everything your redline asked for, and what became of it:
| You asked for | What became of it | What's left |
|---|---|---|
| A named reviewer on call, a response-time target, a cap on queue wait | The queue, the clocks and the alarms are built and running | The names (ask 3); lawyer sign-off on the deadlines |
| Grooming detection over whole conversations before mentors go live - hold the launch for it | Held, exactly as you said: mentor messaging is OFF in code | Your two emails to Thorn and Hive (ask 2), then quote and contract |
| A separate, signposted 48-hour intimate-image removal path (TAKE IT DOWN) | Built stricter than asked: a 2-hour clock, and removed images are fingerprint-blocked from re-upload | The public request page |
| A crisis line for India, shown whether or not we hold the message | Built: 988 for the US, Tele-MANAS (14416) for India, shown either way | Nothing |
| A legal process for reporting child-safety cases to the authorities, before the pilot | Case files, evidence holds, deadlines with alarms, and the written runbook are built | NCMEC registration (filing this month) and the filer's name (ask 3) |
| Test vendors before trusting; read every vendor's age terms | The bake-off is designed and waiting; Gemini is out on its under-18 terms, as you flagged | Your stack sign-off (ask 1) starts it |
| Look seriously at self-hosting the general model | Adopted: gpt-oss-safeguard, self-hosted, is the named stack - no third party receives students' writing | Keys and the bake-off |
| No training on students' writing, shortest possible retention | In every vendor plan; the AWS no-training setting is part of ask 4 | Flipping that setting |
| Consequences for repeat rule-breaking | Not built - unchanged from your list | Scoped after the pilot stack lands |
| Checking images, "assuming photos and banners ship" | They ship - and posts now carry photos too. A person reviews every image before anyone sees it | The image AI layer (ask 4) - details below |
Post photos went live - three calls that are yours to overturn
Photos in posts shipped on September 26, on the same pipeline as avatars and banners. Every promise this page makes for writing now covers photos too: a person clears each one, a held post's photos show only to their writer, a takedown deletes the photo files themselves so copied links die, a flagged post keeps its photos to fix and retry, and no AI sees any photo. Location data is stripped from every image.
Your redline never said how photos inside posts should work, so three calls were made. Overturning any of them is cheap this week and expensive after the pilot opens:
| Question | The call we made |
|---|---|
| What does the reviewer judge? | The whole post - photos and caption, one verdict |
| Is a photo's alt text "writing"? | Yes. Same checks as the body, contact-detail rules included |
| Which surfaces get photos? | Posts only. Comments and messages stay text-only |
The four asks - each one unblocks a column above
- Sign off the stack table. Unblocks vendor accounts, keys, and the bake-off for every text row.
- Send the two drafted emails to Thorn and Hive. Unblocks the grooming gate, which unblocks mentor messaging.
- Name four people: two reviewers, the NCMEC filer, the India Grievance Officer. Under Indian law the filer's name is personal legal exposure.
- Approve the image layer and flip the AWS no-training setting. Unblocks the image AI and the CSAM list applications.
Already moving without you
Submit the NCMEC registration this month. Publish the Grievance Officer page the moment ask 3 gives us the name. Lawyer approval of our self-set response deadlines. The golden scoring set we author together once ask 1 lands.
Three notes. Email verification is a trust marker, not a gate; whether to gate unverified accounts is an open call that is yours. Posts carry images now, exactly as LinkedIn: built and live behind human review - which also means photo review sits in the posting path from day one, so the reviewers you name in ask 3 are on the clock from the first student post. The pilot ships English-only; other languages wait for post-pilot.