Thnkrr: Text Moderation — Industry Practice and Per-Surface Build Flows
Research document, 2026-09-15. Not a decision document — anything adopted from here becomes a
D-entry inthnkrr-api/docs/07-decision-log.md.It builds on, and does not restate,
thnkrr-apidocs/04 §7 (the pipeline), docs/10 (messaging), docs/12 (AI), docs/13 (cross-cutting) and decisions D-9 … D-65. Code references were verified againstthnkrr-apimain@8636bdb andthnkrr-webmain@3b9823e. Nothing in either repository was changed.Source tags: [Primary] the organisation's own publication, statute or regulator text · [Reported] press, secondary or search-summary source · [Analysis] inference. Research is not legal advice.
Contents
- Summary · At a glance: what we use, for what, and where it runs
- What is already settled
- How the industry moderates text
- The canonical flow every moderated surface implements
- Per-surface flow cards
- How to build a moderated surface
- Verified gaps in the current build
- Build sequence and schema impact
- Decisions that need an owner
- Not verified
0. Summary
- The timing is already stricter than the industry. Holding every item until it clears is stricter than LinkedIn, Roblox, Discord, Instagram or Xbox — only classroom tools (Seesaw, Padlet) hold by default. Structural limits on adult contact and undeletable messages also match best practice and the US preservation duty.
- The gap is context, severity, and flow completeness — not timing. Every serious child-safety stack adds, on top of a per-item filter: a conversation- or account-level pass for grooming, severity tiers with graduated enforcement, and a fast human/NCMEC lane. The build judges each item alone and gates on the model's self-reported confidence, which the research does not support for grooming or obfuscated contact details.
- Per-surface flows have holes the pipeline diagram hides. Verified in
code: delayed human approvals deliver messages and comments without
re-checking eligibility (blocked, disconnected, mentorship ended); profile
text has no post-publication takedown (a
userreport's "Remove" is a no-op); self-harm holds show a generic "being checked" and the 988 line only appears after a human rejects; reviewers see no thread, parent post, or conversation kind; and nothing in the codebase ever setssuspendedorbanned. - The legal duty is wider than docs/04 records. Since the REPORT Act (May 2024), 18 U.S.C. §2258A covers online enticement of a minor — a text-only harm — not only CSAM. A held, never-delivered message can still be a reportable attempt.
- Build shape: replace today's scattered per-surface branches (appliers, delivery hooks, flag routing, report branches, copy) with one surface definition per surface (§5), so every surface declares the same eleven concerns and nothing falls through.
At a glance: what we use, for what, and where it runs
Built = in the code today. Proposed = recommended in this document. The AI model provider is not decided. The code has one swappable connection point for it, and §2.4 compares the options by what the industry actually uses.
The parts
| Job | What does it | Where it runs | What leaves our servers | Status |
|---|---|---|---|---|
| Decide who may post, comment or message | Our own permission rules (src/lib/authorize.ts) and per-feature checks | Our API server | Nothing | Built |
| Queue each new item for checking | pg-boss, a job queue that lives inside our Postgres database | Our API server and database | Nothing | Built |
| Undo disguises: invisible characters, look-alike letters, spaced-out digits, number words | Our code plus open-source text libraries (§2.4) | Our API server | Nothing | Proposed |
| Stage 1: catch contact details and a few fixed phrases | Our own pattern rules (src/modules/moderation/stage1.ts): emails, phone numbers, social handles, links, 4 self-harm phrases | Our API server — no AI, instant, free | Nothing | Built (rules); open-source libraries proposed |
| Stage 2: judge meaning — bullying, sexual content, grooming, hate, threats, scams and the rest | An AI language model that reads the text against our written policy (src/modules/moderation/policy.ts) and returns a verdict | The model provider's servers (hosted API), or our own GPU server if we run an open-weight model | The text, plus a coarse age band (13–15 / 16–17 / 18+). Proposed: a short thread excerpt with names replaced by roles. Never names, schools or account ids. | The connection point is built (classifier.ts). The only connector written so far calls Anthropic's Claude. Which model to use is undecided (§2.4). With no model configured, everything goes to a person. |
| Watch whole mentorship conversations for grooming patterns | The same AI model slot, or a specialist child-safety classifier (§2.4) | Provider's servers or our server | Thread excerpts with roles, not names | Proposed |
| Record every verdict (audit trail, appeals, test data) | content_reviews and audit_log tables | Our Postgres | Nothing | Built |
| Human review | Our moderation page in thnkrr-web and its API routes | Our servers; staff use a browser | Nothing | Built (basic queue); severity lanes and context panels proposed |
| Member reports | Our report sheet and report queue | Our servers | Nothing | Built |
| Tell authors and recipients | Our notifications module | Our servers | Nothing | Built |
| Measure accuracy | A labelled test set run through the same model connection | Our server → model provider | Synthetic or de-identified test items only | Proposed |
| Report child exploitation to the authorities | NCMEC CyberTipline (a US legal duty) | NCMEC | The report and its evidence, as the law requires | Proposed |
| Profile photos and banners (for completeness) | Human review of every upload (D-9) | Our servers | Nothing | Built |
Which checks run on each surface
| Surface | Stage 1 rules | AI model | Context the model sees | Conversation watch | Human lane |
|---|---|---|---|---|---|
| Mentorship message | Yes | Yes, stricter bar | Today: the message only. Proposed: last ~20 messages | Proposed | DM lane; safety lane for serious cases |
| Mentorship request note | Yes | Yes, stricter bar | Note (proposed: + topics) | — | DM lane |
| Peer message | Yes | Yes | Today: the message only. Proposed: last ~10 messages | — | DM lane |
| Comment | Yes | Yes | Today: the comment only. Proposed: the post + recent comments | — | Standard |
| Post | Yes (any link goes to a person) | Yes | The post and its link | — | Standard |
| Bios, career goals, experience, school description | Yes | Yes | The text and the author's role | — | Standard |
| Display name and short fields (major, organisation, hometown…) | Contact-details check only (proposed: + a word list) | No (D-45) | — | — | Through reports |
1. What is already settled (not reopened here)
| Settled | Where |
|---|---|
| Every free-text field a user writes and another user reads is screened before anyone else sees it; fail-closed | docs/04 §7, docs/12 §2 |
| Stage-1 deterministic rules, stage-2 AI classifier behind a swappable connection (the one connector written calls Claude; the provider is not decided), human queue, stale sweep | docs/04 §7, D-51, D-60 |
| Three outcomes (cleared / flagged / removed-after-publish) and the tombstone | docs/12 §3, D-58 |
| Held comments and messages invisible to their authors; pending posts visible to their author | docs/12 §4, D-19, D-20, D-27 |
| Flagged text is editable; published text is not | D-27 |
| The trigger is never named; category-level rule copy is the reading | docs/12 §5, D-59 |
| Stricter approve bar on adult channels, chosen per conversation kind | D-33, D-49 |
| Structured fields get a write-time contact-details gate, not the model | D-45 |
| Reports go to humans, never through the classifier | docs/12 §7, D-25, D-46, D-62 |
| Messages are never deletable | docs/10 §6, D-41 |
| Images are always human-reviewed in the pilot | D-9 |
| No mood/wellbeing inference, no generation, no ranking | docs/12 §8 |
| Pilot-first minimalism; safety systems are the exception | docs/04 §1 |
The pre-send "reconsider" nudge is the one industry-standard pattern this design deliberately forgoes: it inherently reveals the trigger (docs/12 §3–4). The evidence is summarised in §2.3 for the record, not as a recommendation.
2. How the industry moderates text
2.1 LinkedIn
LinkedIn publishes one shared pipeline for all content; the only surface with a dedicated published model is private messaging.
The shared pipeline [Primary, DSA Systemic Risk Assessment, Aug 2025]: three layers — (1) at creation, ML services "filter out certain policy-violating content within 300 milliseconds of creation, meaning the content is visible only to the author"; (2) content "likely to be violative but for which the algorithm is not sufficiently confident" goes to human review; (3) member reports. Thresholds are set by risk: "For higher-risk content, LinkedIn prioritizes recall with lower thresholds… For lower-risk content, LinkedIn prioritizes precision" [Primary, DSA transparency report H1 2026]. Model scores feed a Drools rules engine whose actions are challenges, warnings, human queue or restriction [Primary, CASAL].
| Surface | Method (published) |
|---|---|
| Posts | Per-category deep models run proactively "every few hours" plus reactive models on engagement velocity (viral spam, Apr 2023). Low quality is a distribution decision, not a moderation one. Queue prioritisation by an XGBoost violation probability auto-decides ~10% of queued items at "extremely high" precision (Nov 2023). [Primary] |
| Comments | Same pipeline. "Kindness Reminders" (English-only) before posting potentially harmful comments. Harassment is overwhelmingly human-decided: of 28,730 EU own-initiative cyber-harassment actions in H1 2026, 482 (~1.7%) were solely automated, versus ~92% for discrimination. [Primary] |
| Messages | A three-model cascade — sender behaviour → message content → sender–recipient interaction — each running only if the previous flags (Aug 2020). Action is recipient-side: message hidden behind a warning with view/report/mark-safe; a dedicated "trying to move the conversation off LinkedIn" warning (Help). [Primary] |
| Invitation notes | No published text classifier. Signals only: "I don't know this person" feeds sender restriction; invitation volume feeds behaviour models. [Primary] |
| Profile text | A CNN text classifier replaced a phrase blocklist that misfired on context ("escort" as in security escort) and only ever grew; they hand-labelled appropriate profiles containing trigger words to avoid learning the blocklist, and re-scored the whole member base (Jan 2020). [Primary] |
| LLMs | No published use of LLMs deciding member-content policy. GenAI features are filtered on input/output; outputs are checked in 100-token chunks and the whole response retracted on failure (SRA 2025). [Primary] |
| Enforcement | Moved from account restriction to feature restriction: violation → Statement of Reasons + in-product reminder at next compose → repeated: no posting, commenting or starting new threads (can still read, reply to DMs, delete own content) → permanent. CSAM/terrorism skip the ladder. Content appeals H1 2026: 12,135, ~25% reversed, median decision 2 h. [Primary, Help, SRA 2025] |
LinkedIn's headline automated precision (0.98) is computed as one minus the appeal-overturn share, so unappealed errors are invisible. [Analysis]
2.2 Platforms with large minor audiences
| Platform | Text method, as published |
|---|---|
| Roblox | Pre-delivery filter at 6.1B messages/day: rules → trained models → an LLM "for context-based reasoning when uncertainty exists" [Primary, Mar 2026]. A dedicated PII classifier reads a snippet of several chat lines with separate ASKING_FOR_PII / GIVING_PII labels: 98% recall, 94% F1 vs LlamaGuard-3-8B's 27.7% F1, P90 < 100 ms, open weights [Primary, Nov 2025]; v2 with conversation context moved F1 63.4 → 90.5 [Primary, Aug 2026]. Sentinel runs separately from the filter: one-minute snapshots, embeddings compared to ~13k confirmed child-endangerment examples, per-user risk skew over time, human analysts, ~1,200 NCMEC reports in H1 2025 [Primary, Aug 2025]. Age-banded chat since Jan 2026; under-13s strict. Profanity is now LLM-rewritten with everyone notified; PII is still blocked, never rewritten. An independent audit of 2M messages found grooming, sexualisation and PII still passing via obfuscation [Primary, May 2026]. |
| Discord | Teen DMs from new senders trigger a recipient-side safety alert (block/mute/tips; message still delivered). Unknown senders go to a Message Requests inbox by default (Feb 2026). AutoMod is keyword/regex + Discord-maintained lists; a blocked message notifies only its sender. "Account standing" instead of strikes; teens get more chances; child safety is zero-tolerance. [Primary, Oct 2023, Feb 2026] |
| Instagram (Teen Accounts) | Strictest Hidden Words on by default for teens' comments and DM requests. Comment warnings (~1M/day; ~50% edited or deleted). DMs are governed mainly by account signals: adults recently blocked or reported by a teen lose the message button on teen profiles; accounts whose name or bio suggests inappropriateness can't follow, DM or comment on teens; new chats show the other account's join date and country. [Primary, Sep 2024, Jul 2025, Oct 2025] |
| YouTube | Creator-chosen comment holds (None/Basic/Strict/Hold all); held comments invisible until approved. Comments disabled on most videos featuring minors (2019). Pre-post reminder with "post anyway"; 24 h timeouts for repeat abuse. [Primary/Reported] |
| Snapchat | Teens reachable only by mutual friends/contacts; warnings when chatting with someone "blocked or reported by others" or from an unusual region; friend requests silently not delivered when signals indicate scams. No disclosed text classifier for grooming. [Primary, Jun 2024] |
| Xbox | Recipient-side filter levels (child accounts default Friendly; hidden messages can't be revealed). Strikes weighted by severity, visible to the player, expire after 6 months; 2 → 1 day, 4 → 7 days, 8 → 1 year; <1% of players suspended in 2022. [Primary, Aug 2023] |
| Education tools | Seesaw: new student comments need teacher approval by default. Padlet Safety Net: AI screens every post/comment, flagged items blocked pending owner approval, admins can switch categories off (e.g. violence for a history class). Google Classroom: teacher mutes are silent to the student. [Primary] |
| Yubo (teen app until it raised its minimum age to 18, Nov 2025) | Pop-up before sharing a phone number or address; SafeBlur blurred harmful text in received DMs. [Primary/Reported] |
2.3 Cross-platform patterns
| Surface | Dominant industry method | Timing | Context used |
|---|---|---|---|
| Public post | Classifier + blocklist; AI hold in education tools | Mostly post-hoc; held in edu | Item + account history |
| Comment | Owner filters + classifier + pre-post prompt | Hidden or held | Item; commenter account signals |
| Peer DM | Real-time filter (Roblox) or recipient-side hide/blur | Pre-send or on delivery | Several lines (Roblox PII); per-user penalties |
| Adult↔minor DM | Prevented structurally first; where allowed, a background conversation/account-level detector → specialist humans → NCMEC | Near-real-time sweep | Conversation over time; reported-by-minor signal |
| First-contact text | Request inbox; silent non-delivery for risky senders | Routed or dropped | Sender signals (mutuals, reports, region, account age) |
| Profile fields | Blocklist → forced rename; bio as a risk signal | Refused or account restricted | Name/bio + history |
Industry-standard (≥3 platforms): structural limits on cold and adult contact before any text classification; request inboxes for first contact; recipient warnings driven by who the sender is; a fast per-item filter paired with a separate conversation/account-level grooming detector staffed by specialists; history-based graduated enforcement with visible standing.
The nudge evidence (recorded, not recommended): Twitter's RCT — 6% fewer offensive tweets (ICWSM 2022); Instagram — ~50% of warned comments edited/deleted; Nextdoor — 34% edited or withheld; OpenWeb/Perspective — secondary summaries report ~18% re-posting to test the filter. No published teen-specific effect size exists. Every platform that nudges does so only for incivility; PII, sexual content and grooming are blocked without explanation. [Primary/Reported]
2.4 Models, vendors, and what the industry actually uses
Checked 2026-09-15. "Adopted by" means a public statement that a platform uses the tool to moderate user content. Several vendor pages could only be read through search excerpts; those facts are tagged [Reported]. Prices are US$ per million input / output tokens unless stated.
What comparable platforms run
| Platform | Text moderation stack | Source |
|---|---|---|
| Roblox | In-house models (PII classifier, Sentinel) plus an LLM layer for uncertain cases | [Primary] |
| Discord | In-house AutoMod (keyword lists + ML filters), Osprey rules engine; tested gpt-oss-safeguard | [Primary] |
| Meta (Instagram, Facebook) | In-house LLMs trained on its own policies | [Primary] |
| In-house ML (§2.1) | [Primary] | |
| In-house harassment filter; Hive customer (2021) | [Primary] / [Reported] | |
| Twitch | In-house AutoMod | [Reported] |
| Xbox | Community Sift, a Microsoft-owned vendor (current status unverified) | [Reported] |
| Bluesky | Hive (images and video), Thorn Safer (CSAM), Osprey, Musubi PolicyAI (a vendor built on Gemini) | [Primary] / [Reported] |
| X | Thorn Safer Predict text classifier | [Reported] |
| Yubo | Hive | [Reported] |
| Khan Academy | OpenAI Moderation API for Khanmigo chats | [Primary] |
| Padlet | "Safety Net" AI moderation, reportedly OpenAI + Hive | [Reported] |
| Grindr | Spectrum Labs (now Alice); reportedly Musubi | [Reported] |
No public evidence was found for Duolingo, Quizlet, Brainly, Canva or Chegg.
What the evidence says
- There is no single industry-standard tool. The largest youth platforms build their own models, and those can't be bought.
- Platforms without large safety teams buy proven services: a general moderation classifier (OpenAI Moderation; Hive has the most named youth and social customers) and, for child exploitation, Thorn's Safer (80+ platforms — the most widely adopted tool for that harm).
- General AI models that follow a written policy are spreading, mostly through moderation vendors built on OpenAI (SafetyKit) or Google Gemini (Musubi). No youth or education platform has published which general model it calls directly.
- Open-weight guard models are emerging, not standard. gpt-oss-safeguard has the strongest backing (ROOST, tested by Discord) and no confirmed production user.
- No public adopter was found for moderating user content with Anthropic, Mistral, xAI or Cohere models. That measures adoption, not quality.
The building blocks, layer by layer
Layer 1 — deterministic rules, in our own code. Runs inside the Node API; nothing leaves our servers. No adoption data exists for libraries this small; these are the usual choices for each narrow job.
| Library | Job | Licence · latest release | Note |
|---|---|---|---|
| libphonenumber-js | Phone numbers | MIT · 1.13.13 (Sep 2026) | Finding numbers inside free text not verified |
| linkify-it / linkifyjs | Links; linkifyjs also finds emails and @mentions | MIT · 2026 | Active |
| obscenity | Profanity, including leetspeak and disguised spellings | MIT · 0.4.6 (Jan 2026) | English word set |
| decancer | Look-alike (confusable) characters | MIT · 3.3.3 (Jul 2025) | Unicode TR39 conformance not verified |
| words-to-numbers | "five five five" → 555 | MIT · 2018 | Unmaintained — expect to own this code |
Layer 2 — hosted moderation classifiers. Ready-trained per-category scoring, usually one message at a time.
| Service | Covers from our 10 categories | Doesn't cover | Price | Data terms | Adopted by |
|---|---|---|---|---|---|
| OpenAI Moderation | Harassment, hate, self-harm, sexual (incl. sexual/minors), violence, illicit | Grooming, contact details, spam, impersonation | Free | Abuse-monitoring logs up to 30 days; zero retention only with OpenAI's approval; no training on API data [Reported] | Khan Academy; built into ROOST Coop |
| Hive Text Moderation | Severity 0–3 for sexual, hate, violence, bullying, drugs, weapons, child safety, self-harm; yes/no for child exploitation, redirection off-platform, spam, phone numbers; patterns for email and address; Thorn grooming model (English/Spanish, conversations) | Impersonation | Reported $0.50 per 1,000 requests; enterprise via sales | Not verified | Reddit, Yubo, Padlet (reported) |
| Thorn Safer Predict | Child-exploitation labels: sextortion, child access, has-minor, CSAM, grooming signals | General harms | Sales only | Not verified | X; Safer is used by 80+ platforms |
| Mistral Moderation 2 | Sexual, hate, violence/threats, dangerous, criminal, self-harm, PII, jailbreaks; whole conversations | Grooming, spam, impersonation | Reported free | 30-day retention; the free tier trains on data unless opted out [Reported] | None found |
| Google Cloud moderateText | 16 attributes incl. toxic, insult, sexual, drugs, weapons | Grooming, contact details, spam | Per 100-character unit | Not verified | None found |
| Azure AI Content Safety | Hate, sexual, violence, self-harm, with severity levels | Grooming, contact details, spam, impersonation | Free tier, then per 1,000 records | Not verified | South Australia's EdChat (screens an AI chatbot, not member content) |
| Amazon Comprehend toxicity | 7 classes, English only | Grooming, contact details, drugs, spam | Per 100-character unit | Not verified | None found |
| Perspective API | Toxicity | — | — | — | Shuts down 2026-12-31 |
Layer 3 — general AI models that follow our written policy. This is what reads context: bullying aimed at a specific person, disguised contact exchange, grooming across a thread.
Hosted by the model's maker
| Provider | Models suited to classification | Price | Data terms | Minors and blocking | Moderation adopters |
|---|---|---|---|---|---|
| OpenAI | gpt-5-nano; GPT-5.6 Luna; flagship GPT-5.6 | $0.05/$0.40; $0.20/$1.20 [Reported] | No training by default; abuse logs up to 30 days; zero retention on approval; US/EU/UAE processing [Reported] | Not verified | SafetyKit (a moderation vendor) runs on GPT-5 |
| Google (Gemini API) | Gemini 2.5 / 3.1 Flash-Lite | $0.10/$0.40; $0.25/$1.50 [Primary] | Paid tier not used for training | Terms bar services "likely to be accessed by individuals under the age of 18" [Primary, 2026-03-23]; child-safety content always blocked, including in inputs | Musubi (a vendor) builds on Gemini; serves Bluesky |
| Anthropic | Claude Haiku 4.5; Sonnet 5; flagship Opus 5 | $1/$5; $2/$10; $5/$25 [Primary] | No training; 30-day retention; zero retention by agreement, but flagged content kept up to 2 years even then [Primary] | Minors allowed with extra safeguards under its guidelines [Reported] | None found |
| Mistral | Mistral Small 4 | $0.15/$0.60 [Reported] | Not verified | Not verified | None found |
| xAI | Grok 4.3 | $1.25/$2.50 [Primary] | Not verified | Not verified | None found |
Open-weight models — run on our own GPU, or on a host that keeps no data
| Model | Size · licence | Follows our written policy? | Languages | Where it can run | Adopters |
|---|---|---|---|---|---|
| gpt-oss-safeguard | 20B / 120B · Apache 2.0 | Yes, with its reasoning | Not stated | Groq $0.075/$0.30 (no retention by default); Amazon Bedrock $0.07/$0.20 (no retention for open models); own GPU with 16 GB | Tested by Discord; ROOST |
| Llama Guard 4 | 12B · Llama 4 licence | No — 14 fixed categories, no grooming, spam or impersonation | 8 | DeepInfra | Cloudflare, for AI prompts rather than member content |
| Qwen3Guard | 0.6B / 4B / 8B · Apache 2.0 | No — 9 fixed categories, including PII | 119 | Own GPU | None found |
| Granite Guardian 4.1 | 8B · Apache 2.0 | Built-in risks plus custom criteria | English | Own GPU | None found |
| Shieldstral 1.0 | 3B · Apache 2.0 | Yes — plain-language yes/no questions | 12 | Own GPU with 16 GB | None found (released Aug 2026) |
| Nemotron 3.5 Content Safety | 4B · OpenMDW + Gemma terms | Fixed categories plus a custom mode | 12 | Own GPU | None found |
| ShieldGemma | 2B / 9B / 27B · Gemma terms | 4 categories | English | Own GPU | None found |
| Roblox PII Classifier v2 | 0.6B · Apache 2.0 (one conflicting report) | Contact exchange only: asking for or giving personal info, moving off-platform; reads chat history | 189 | CPU or GPU; latency unpublished | Roblox |
Self-hosting means an inference server such as vLLM or SGLang (both Apache 2.0) on a rented GPU: roughly $358 a month for a RunPod L4 around the clock, or about €214 a month for a dedicated 24 GB Hetzner server [Reported]. At pilot volume (about 1.4 requests a minute) that GPU would sit more than 99% idle, while zero-retention hosts run the same open models for a few dollars a month. Hugging Face TGI is in maintenance mode.
Layer 4 — human review and rules infrastructure (open source).
| Tool | What it does | Licence · version | What it needs to run | Adopted by |
|---|---|---|---|---|
| ROOST Coop | Review queues, NCMEC reporting actions, strikes, appeals | Apache 2.0 · v1.0.1 (Jun 2026) | Redis, Postgres, ScyllaDB, ClickHouse | None found |
| ROOST Osprey | Rules engine and investigation UI | Apache 2.0 · v1.0.1 (May 2026) | Kafka, Druid, Postgres, MinIO | Discord, Bluesky, Matrix |
| Tech Coalition Lantern | Sharing child-safety signals between platforms | Membership after eligibility review | — | 38 members incl. Discord, Reddit, Roblox, Snap, Twitch, OpenAI |
Monthly model cost at pilot volume
Rough estimate for 2,000 items a day (about 36M input + 9M output tokens a month), before caching or batch discounts [Analysis]:
| Option | ≈ Per month |
|---|---|
| OpenAI Moderation | Free |
| gpt-oss-safeguard-20b on Amazon Bedrock or Groq | $4–11 |
| OpenAI gpt-5-nano | $5 |
| OpenAI GPT-5.6 Luna | $18 |
| Anthropic Claude Haiku 4.5 | $81 |
| Anthropic Claude Sonnet 5 | $160–210 |
| OpenAI GPT-5.6 Terra | $180 |
| Anthropic Claude Opus 5 (the connector written today) | ~$405 |
| A dedicated self-hosted GPU, around the clock | $250–360 |
Provider terms that change the design
- Google's Gemini API terms exclude services likely to be used by under-18s, and Vertex AI is reportedly 18+ as well [Reported]. Using Gemini would need an explicit arrangement with Google.
- Meta's open-weight Muse Glimmer 30B usage policy says it is not intended for under-18s.
- Retention: OpenAI up to 30 days; Anthropic 30 days, with flagged content kept up to 2 years even under zero retention; Mistral 30 days, and its free tier trains on data. Groq, Fireworks, DeepInfra and Bedrock keep no data for open models by default.
- Blocking: Gemini always blocks child-safety content, including in what we send it; Azure's content filters are on by default; guard models are built to take harmful text. For every other provider, refusal on harmful input is unverified — the test set has to include it.
- Every service added receives student text and becomes a named sub-processor in school data agreements (§2.6).
Notes on the connector written today (Anthropic Claude)
- Opus 5 can return a refusal instead of a verdict; the connector already sends that item to a person.
- Structured output doesn't enforce the 0–1 confidence range; the code re-checks it and fails closed.
- The ~900-token policy prompt clears Opus 5's 512-token caching minimum.
- At low reasoning effort Opus 5 may skip reasoning entirely.
- Haiku 4.5, the cost lever named in docs/04 §7, is listed for retirement no sooner than 2026-10-15.
Can an AI model be trusted as the classifier?
Self-reported confidence is not a calibrated gate.
- Verbalised LLM confidence is overconfident; sampling-consistency helps but no method dominates [Primary, ICLR 2024]; verbalised confidence was only a weak error predictor (AUROC 0.522) [Reported, same].
- Guard models are badly miscalibrated across 12 benchmarks; temperature scaling and contextual calibration help [Primary, ICLR 2025].
- Meaning-preserving rewording of a safety policy flipped LLM-judge verdicts, 18–43% of flips on unambiguous cases [Primary, May 2026].
- Industry tools score severity, not confidence: Hive returns 0–3 per class, Azure Content Safety returns severity levels, Qwen3Guard grades Safe / Controversial / Unsafe, and Anthropic's moderation guide recommends 0–3 risk levels.
Adversarial text and injection. Universal appended phrases push LLM judges to maximum scores [Primary, EMNLP 2024]; zero-width characters, Unicode tags, homoglyphs and emoji smuggling evaded six guardrails with up to 100% success [Reported, 2025]; one injected invisible character significantly degrades commercial toxicity classifiers [Primary, IEEE S&P 2022]. Microsoft's "spotlighting" (delimit / datamark / encode untrusted text) cuts indirect-injection success sharply [Reported, 2024]. Classic classifiers (Layer 2) don't follow instructions, so injected text can't talk them into a verdict — but they are exposed to character-level tricks.
General models vs specialised classifiers. OpenAI says classifiers trained on tens of thousands of labels "can still outperform" its policy-reasoning model on complex risks [Reported, Oct 2025]. Zero- and few-shot LLMs underperformed fine-tuned encoders on real non-English data [Primary, May 2025]. On Gen Alpha slang the best model tested (an older Claude) recognised 42.3% of safety issues [Reported, FAccT 2025]. No current model has a published evaluation on teen DMs — the only real evidence will be a Thnkrr-labelled test set. [Analysis]
Context helps most where harm is conversation-shaped. Annotators without discussion context found 2.7× fewer violent-speech cases [Primary, 2022]; Roblox's PII classifier went from F1 63 to 90 with context; LLMs over-find grooming in harmless chats and are "led by the prompt to find non-existent risks" [Primary, 2024]; a turn-level grooming model reached 0.475 precision vs 0.047 for prior work, flagging in the first ~20% of a conversation [Primary, 2025].
The industry-standard stack at Thnkrr's size [Analysis]
Built only from the adoption evidence above:
- Our own rules, using the standard libraries in Layer 1. Every platform has this layer.
- A bought classifier for broad harms — OpenAI Moderation (free, used in education) or Hive (also covers contact details, off-platform redirection and child safety, with the most named youth customers).
- A child-safety specialist for grooming and sextortion — Thorn Safer Predict, or Hive's Thorn-based grooming model. At pilot scale, layers 2 and 3 can be one vendor (Hive covers both).
- A general AI model following our written policy, for context and nuance. Pick it by running the test set through a shortlist. The adoption evidence points to OpenAI models (used through moderation vendors) or the open-weight gpt-oss-safeguard on a zero-retention host. Google's API terms exclude us. Anthropic has no public moderation adopters but is already connected, so it stays on the shortlist and is judged on results.
- People, working in our own review queue. Large platforms build their own; the open-source queues need far more infrastructure than our stack runs.
How the code supports it: the classifier connection (classifier.ts) is called once per
item and returns one verdict, so a combined connector that calls layers 2–4 and merges
their results fits behind it without changing the pipeline. Each service added is another
company receiving student text, another sub-processor, and another test-set run.
2.5 Techniques for the hard parts
Conversation-level grooming and sextortion. Grooming (O'Connell's stages: friendship → relationship → risk assessment → exclusivity → sexual) uses the vocabulary of real mentoring in its early stages, so per-message judgement misses it. Sextortion looks different: rapid, scripted, often posing as a peer, pushing off-platform; 38% of reports with chat logs contained "ruin your life" threats [Primary, Thorn/NCMEC 2024]. Early-detection research scores sliding windows and only alerts when k of the last n windows are positive ("skepticism") [Primary, ACL 2021]. Benchmark numbers are inflated by decoy data; LLMs over-find risk. Production systems use these detectors as high-recall candidate generators for human investigators, never as enforcers (Sentinel README). [Primary/Analysis]
Obfuscation. Normalise before matching: strip zero-width/bidi/Unicode-tag characters (their presence is itself a signal); NFKC + casefold; TR39 confusable skeletons are for comparison, not display, and NFKC must run first; map leetspeak; collapse single-character spacing ("s n a p"); convert number words to digits; also match over the sender's recent messages joined, to catch splitting. Roblox's ML PII filter found 25% more PII with 30% fewer false positives than its rules. Evasion is not always malicious: algospeak grows from moderation perceived as unfair. [Primary/Reported]
Severity and enforcement. Ofcom's codes require a prioritisation policy weighing reach, severity (including harm to children) and likelihood of illegality, plus recorded speed and accuracy targets. Strikes that expire (YouTube 90 days, Xbox 6 months) and education-first first offences (YouTube's warning-plus-training; Discord's teen leniency; Meta restricting only from the seventh violation, with ~80% of low-strike users not reoffending in 60 days) are the norm; child exploitation is zero-tolerance everywhere. Explanations of removals correlate with fewer future removals, and bot explanations work as well as human ones [Reported, CSCW 2019] — which supports the existing category-level copy.
False positives on teens. African American English up to 2× more likely labelled offensive [Primary, ACL 2019]; reclaimed slurs and in-group queer speech misclassified by LLMs; keyword self-harm flags on a youth forum produced many false positives while missing worrying messages without keywords [Reported, 2022].
Evaluation at small scale. Build a golden set; blind model labelling; disagreements rewrite the policy (OpenAI's loop). Binomial arithmetic: ±5% precision at 95% confidence needs ~385 flagged items per category (±10% needs ~96); zero misses in n audited approvals bounds the miss rate below ~3/n. Rare categories (grooming) need synthetic/red-team sets with recall reported on those. Pin model versions; re-run on every prompt change.
Latency. 0.1 s feels instant, 1 s keeps flow, 10 s is the attention limit (Nielsen). Field experiments with news sites found algorithm-supported pre-moderation reduced toxicity without lowering engagement [Reported, 2025].
2.6 Regulation and standards that shape the method
US — applies to the pilot now.
| Instrument | Requirement that shapes moderation |
|---|---|
| 18 U.S.C. §2258A as amended by the REPORT Act (in force since May 2024) [Primary, LII] | Report to the NCMEC CyberTipline "as soon as reasonably possible after obtaining actual knowledge" of apparent CSAM, child sex trafficking, or coercion/enticement of a minor (§2422(b), attempts included). No duty to scan. A report is a 1-year preservation request covering "commingled content" that provides context (the thread); secure, access-limited storage. Penalties up to $600k first failure for providers under 100M monthly users. NCMEC's guidelines list enticement indicators: moving to private/other apps, posing as a peer or love interest, gifts/money/fake modelling offers, threats or claims to hold images, a child sharing address/school/phone around a meet-up; aggravating factors include an offender in a position of trust. [Primary, NCMEC Oct 2024] |
| "Actual knowledge" | Undefined in the statute. A classifier score alone is weak; human confirmation clearly counts. Design: a fast human lane for enticement/trafficking, report on confirmation, never let queue backlog be the delay. [Analysis] |
| COPPA (amended rule; compliance from 2026-04-22) | With a 13 floor, applies only on actual knowledge of an under-13 user — a moderator reading "I'm 12" is knowledge and needs an account workflow. A vendor used only for integrity fits "support for internal operations" if it does not train on the data. [Reported/Analysis] |
| FTC §5 (Workado, 2025) | Accuracy claims about AI detection need "competent and reliable evidence" retained. Any safety claim to a school needs the eval data behind it. [Primary] |
| FERPA / SOPIPA / SDPC NDPA v2.2 / NY Ed Law 2-d | Vendor (and the AI model provider as a sub-processor) bound to use student data only for the service; sub-processor terms "no less stringent"; disposal within 60 days of request/termination; no AI-specific clause in the standard NDPA; its exceptions have no express carve-out for statutory CyberTipline reports. [Primary/Analysis] |
| California AB 1394 (if a pilot school is in CA) | Bars "knowingly facilitat[ing]… commercial sexual exploitation" ($1M–$4M per act); safe harbour needs design/feature audits at least twice a year. [Primary] |
| Texas SCOPE Act filtering duty | Enjoined (5th Cir., Jul 2026). Nothing required now. [Reported] |
| KOSA | Not law as of Sep 2026. [Reported] |
Expansion only.
| Jurisdiction | Requirement | Conflict with the settled design |
|---|---|---|
| UK — Ofcom Illegal Content Codes (in force Mar 2025), Protection of Children Codes (Jul 2025) | Moderation function, recorded speed + accuracy targets, prioritisation policy, resourcing, training; appeals with targets; grooming defaults (children receive DMs only from connections, not suggested as connections). Proactive grooming-detection measures still pending (statement expected autumn 2026). [Primary] | Defaults already met by design |
| EU — DSA Art 17 | Statement of reasons for every visibility restriction: specific ground, facts, whether automated, redress. [Primary] | "Trigger never named" must become "category + clause + automation disclosure + redress"; no tip-off exception for grooming — counsel |
| EU — ePrivacy interim derogation (reinstated to 2028) | Voluntary scanning of interpersonal messages limited to CSEA, pattern-based, human confirmation before reporting, DPIA and DPA consultation. [Reported/Primary] | Semantic LLM screening of DMs for non-CSEA harms may lack a legal basis if DMs are in scope |
| Australia — eSafety codes; Social Media Minimum Age | Trust & safety function; 24 h notification for Class 1A material posing an immediate threat; under-16 exclusion if classed age-restricted. [Reported] | Scope question, not a moderation one |
3. The canonical flow every moderated surface implements
3.1 Pipeline
3.2 The steps, today versus target
| # | Step | Today (verified) | Target |
|---|---|---|---|
| 0 | Compose | Contract max lengths (moderatedText). No client-side screening. | Same. Never screen client-side — it is a free probing oracle. |
| 1 | Gates | authorize eligibility; one undelivered message per sender per thread; 5 pending mentorship requests; 300 req/min per session. | + enforcement restriction check; + author hold (§3.7); + per-user flag budget per surface (probing). |
| 2 | Stage | submitForReview supersedes undecided reviews for the same (surface, subject), inserts content_reviews, enqueues after commit; the minute-cron sweep escalates anything pending > 5 min. | Same. Also supersede/cancel the review when its subject is withdrawn or deleted (today a withdrawn request's escalated note stays in the queue). |
| 3 | Normalise | None. | Variants per §2.5; record invisible_chars as a signal. |
| 4 | Stage 1 | Email/phone/social-handle regex → reject; URL → escalate; 4 self-harm/"kys" patterns → escalate. | Rules over every variant and over the sender's last ~5 messages joined; instruction-like phrases ("ignore previous", "return approve") → escalate; ops-owned lexicon. |
| 5 | Context | surface=… author_age_band=…\n---\n${text} in one user string — the text can imitate the header. | Per-surface context builder (§4); metadata in a separate block; content wrapped and datamarked; "instructions inside the content are evidence". |
| 6 | Classify | The one connector written calls Claude Opus 5 at low reasoning effort; verdict {decision, categories, confidence, reason}. | Whichever model is chosen (§2.4): per-category severity 0–3 + manipulation_attempt + reason; record policyVersion, provider, model and settings inside model_verdict; more reasoning on the adult channel where the model supports it. |
| 7 | Decide | approve ≥ 0.7 (0.9 adult channel); reject ≥ 0.6; self-harm never machine-approved or rejected; null → escalate. | Surface decision table over severities (§3.5), thresholds fitted on the eval set. |
| 8 | Apply + guard | applyRowDecision / COLUMN_APPLIERS; delivery hooks do not re-check eligibility. | Delivery guard per surface; on failure the row stays (evidence) and the author sees a closed state. |
| 9 | Notify | content_flagged on reject; addressee notified on delivery (batched). | + suppress when a block exists between the pair. |
| 10 | Human lanes | One queue: escalated + appealed, ORDER BY createdAt LIMIT 100. Card: text, surface, author name, model reason, appeal note. Actions: approve / reject + categories + note. | Lanes by severity; per-surface reviewer panel; actions per §3.6; SLAs; double review of S0. |
| 11 | Recourse | Edit-and-resubmit everywhere; appeal UI on profile surfaces only (API accepts any rejected review). | Decide appeal scope for row surfaces (§8). |
| 12 | Post-publication | Reports on post / comment / message / request note → remove → tombstone. user reports preview the display name and remove does nothing. | Report-as-itself and takedown for every surface; direct takedown without a report; re-screen live profile text on policy change; async pair-risk pass on DMs. |
| 13 | Enforcement | users.status has suspended/banned and login/session code refuses them — nothing sets them. | Ladder (§3.7). |
| 14 | Evidence | content_reviews + append-only audit_log; messages undeletable; posts/comments soft-deleted. | Safety case + 1-year preservation + export (§3.8). |
| 15 | Observe | /moderation/stats: pending, escalated, oldest pending, model/human decisions 24 h. | Per-surface metrics (§5.5); weekly sampled audit of approvals and rejects. |
3.3 Lifecycles
Row-shaped surfaces (post, comment, message, request note) keep their own
status; "held" is the row in pending_review with its review escalated.
Column-shaped surfaces (bios, career goals, experience description, school description) stage the candidate text in the review row; the previous approved value stays live until approval; clearing a field applies immediately.
3.4 Who sees what, by state
Row-shaped:
| State | Author | Addressee (DM counterpart · mentor · post author) | Other members | Moderator |
|---|---|---|---|---|
| Pending (model working) | Post: own post marked "visible to others in a moment" (D-20). Comment / message / note: status line, no text (D-19). | Nothing, no notification | Nothing | Not queued unless stale > 5 min |
| Held (escalated) | Identical to pending — deliberately indistinguishable | Nothing | Nothing | Queue |
| Published | Normal | Normal + notification | Per visibility rules | — |
| Flagged (rejected) | Category copy, text back in composer, edit and resubmit (+ clear, messages); content_flagged notification | Nothing | Nothing | Only if appealed |
| Discarded (messages) | Gone from composer; row kept | Nothing | Nothing | — |
| Cancelled (proposed) | "Couldn't be delivered — this conversation closed" | Nothing | Nothing | Row kept |
| Removed after publish | Tombstone, text stripped server-side | Tombstone | Tombstone | Report resolved |
Column-shaped:
| State | Author | Viewers | Moderator |
|---|---|---|---|
| Pending / held | "Your edit is being checked — this still shows the last approved version." | Previous approved value | Queue if held |
| Approved | New value | New value | — |
| Flagged | Edit form with category copy; appeal | Previous approved value | Queue if appealed |
| Removed after publish (proposed) | Field cleared with a neutral notice and a path to rewrite | Field empty | Report resolved |
3.5 Severity model and decision table
The classifier returns a 0–3 severity per category; code decides.
| Tier | Meaning | Examples | Outcome |
|---|---|---|---|
| S0 | Child exploitation, enticement, sextortion, credible threat of violence, imminent self-harm | Grooming indicators on the adult channel; requests for images; "I'm bringing a gun Monday"; a specific plan to self-harm | Hold → safety lane (≤ 15 min, waking hours) → author hold, evidence snapshot, NCMEC or emergency protocol on human confirmation |
| S1 | Slurs, sexual harassment, doxxing, contact exchange on the adult channel, targeted threats | Flag; counts toward a strike only when a human confirms | |
| S2 | Insults, mild sexual innuendo, contact exchange between peers | Flag; no strike on first offence | |
| S3 | Undirected profanity, heated but not targeted | Approve (per surface) |
Per-surface rules (examples; thresholds fitted on the eval set):
| Rule | Applies to |
|---|---|
| Approve only if every category is 0 and no manipulation flag | Adult channels (mentorship message, request note); adult-authored text read by minors — decision (§8) |
grooming_risk ≥ 1 → escalate, never auto-reject (humans must see it) | All DMs |
sexual ≥ 2 between minors → S0 (possible solicitation of images) | Peer DMs, comments |
violence threat against a school or person → S0 regardless of surface | All |
self_harm ≥ 1 → care lane (§3.9) | All |
pii_or_contact_exchange stage-1 hit → reject; model-only hit → escalate on adult channels, reject elsewhere | All |
3.6 Human review lanes, reviewer actions, SLAs
| Lane | Contents | Target (proposed, not a published norm) | Reviewer |
|---|---|---|---|
| Safety | S0 escalations; self_harm_concern and sexual/violence reports on DMs; author holds | ≤ 15 min in student waking hours; paging rota with a named backup | Safety lead (a narrower role than moderator); double review before any report or account action |
| DM | Held peer/mentorship messages and request notes | ≤ 1 h waking hours — the sender's thread is frozen by the one-undelivered rule | Moderator |
| Standard | Held posts, comments, profile text; appeals; other reports | ≤ 1 school day | Moderator |
Reviewer actions (today: approve, reject + categories + note):
| Action | Effect |
|---|---|
| Approve | Guard → publish/deliver/apply |
| Reject (category) | Flag with category copy; no strike |
| Reject + strike | Flag + enforcement record (§3.7) |
| Hold author | Every new item from the author escalates silently (§3.7) |
| Open safety case | Evidence snapshot, preservation clock, NCMEC workflow (§3.8) |
| Care response | Resources to the author (§3.9) |
| Needs second reviewer | Routes to safety lead |
| Record under-13 disclosure | Account workflow (COPPA) |
Reviewer hygiene: mask slurs and sexual text until click-to-reveal (reduces impact without hurting accuracy [Reported, 2020]); never show reviewer notes to authors.
3.7 Enforcement ladder and the author hold
Author hold is the build's cheapest safety primitive because it needs no new
UI: while a hold exists, processReview escalates every new item from that
author with reason author_hold. The author sees the ordinary "being checked"
state, so an S0 investigation is not tipped off, and nothing reaches anyone
until a human clears it. It also answers filter probing: N flags in a window on
one surface → temporary hold.
Ladder (education-first for minors; S0 skips it):
| Step | Trigger | Effect |
|---|---|---|
| Reminder | First human-confirmed S1, or repeated S2 | In-product policy reminder at next compose (category-level, never the phrase) |
| Feature restriction | Second confirmed S1 within 90 days | No posts, comments or new threads for 24 h–7 days; can read, reply in existing threads, edit profile, delete own content |
| Extended restriction | Continued | Up to 30 days; school-visible only if the school agreement says so (decision) |
| Suspension | S0 confirmed, or mentor contact-exchange confirmed (docs/04 §8) | users.status = suspended, sessions revoked; mentors terminated |
Strikes expire after ~90 days, are visible to their owner as standing (not a count), and a reversed decision removes its strike. Nothing is automatic from a model verdict alone.
3.8 Evidence, preservation, and the NCMEC lane
- S0 escalation → author hold + safety case (additive table) linking the review, subject, thread excerpt, both accounts, timestamps.
- Safety lead confirms or clears. On confirmation of apparent CSAM, trafficking or enticement: file a CyberTipline report (requires registering as a reporting provider), record the report id, start the 1-year preservation clock covering the thread context, restrict access to the case.
- Only after the report: enforcement (suspension, mentor termination). Held never-delivered messages are included — rejection does not discharge the duty.
- Account deletion (not yet built) must exempt anything under preservation.
- School data agreements need an explicit carve-out for statutory reports.
The procedure is designed with counsel before pilot launch (docs/04 §7 already requires this for CSAM; the scope is wider).
3.9 Self-harm care flow
Today: stage 1 or the model marks self_harm → escalate → the author sees the
generic "being checked" state. The 988 copy in REJECTION_MESSAGES.self_harm
("A person on our team is taking a look at this…") renders only on the
rejected state (thread.tsx, post.tsx, mentorship.tsx), i.e. after a human
rejects. In a DM, the one-undelivered rule also stops the student writing
anything else in that thread until a human acts. Reviewers have approve/reject
only.
Target flow (the options in step 2 and 4 are decisions, §8):
self_harm ≥ 1→ care lane with the safety-lane SLA.- At hold time, show support resources to the author for self-harm holds (the probing value of revealing this category is negligible; the cost of a student in crisis waiting for a human is not).
- Reviewer chooses: deliver/publish (peer support and disclosure are often healthy), hold with a care message, or the imminent-risk protocol.
- The imminent-risk protocol — who is contacted (school counsellor, guardian, emergency services) and when — is a counsel + school-agreement decision. It is an explicit expression by the student, not inference, so it does not cross docs/12 §8's mood-detection line, but it does surface a minor's disclosure to an adult and must be designed as its own decision.
self_harmnever produces a strike.
4. Per-surface flow cards
Each card lists only what differs from §3. "Built" and "Gap" are verified against the code references at the top of this document.
4.1 Mentorship message — message.body, conversation kind mentorship
| Concern | Spec |
|---|---|
| Who → who | Vetted mentor (18+) ↔ student (13–18), only after a student's request is accepted. The one adult↔minor channel. |
| Harms | Grooming, sextortion, off-platform contact in either direction (mentor-terminating, docs/04 §8), sexual content, self-harm disclosure. |
| Entry | messaging/service.ts sendMessage, editMessage, discardFlaggedMessage. |
| Gates | Built: party to a live mentorship; one undelivered message per sender. Add: author hold; restriction; first-N-messages scrutiny for a new pair. |
| Stage 1 | Add: variants + the sender's last 5 messages joined (split contact details). |
| Context sent | Last ~20 delivered messages as role tokens (S student 13–15/16–17, M mentor 18+), message index, days since the mentorship started. No names, schools or ids. Sending the counterpart's age band extends D-59 → counsel. |
| Decision | Approve only if all severities are 0; grooming_risk, sexual, pii ≥ 1 → escalate; S0 → safety lane; effort medium. |
| Delivery guard | Mentorship still active; neither account held, suspended or banned. Gap: onMessagePublished re-checks nothing, so a message a human approves after the mentorship ended publishes into the archived thread and notifies the counterpart. |
| Author | Built: "Your last message is being checked. You can write again once it's through."; flagged → category copy, edit or clear. Add: self-harm resources at hold (§3.9); closed state on cancel. |
| Counterpart | Built: nothing until delivered; batched notification; persistent monitored notice; tombstone. |
| Reviewer panel | Add: thread excerpt with names (staff only), the label mentorship, mentorship start date, both age bands, both parties' prior flags, open reports on the pair, pair risk score. Gap: the queue card shows none of this — a reviewer cannot tell a mentorship message from a peer message. |
| Post-publication | Built: report the message as itself → remove → tombstone. Add: pair risk pass — an async pg-boss job per mentorship keeps a rolling state of hard signals (off-platform, secrecy, meeting up, gifts/money, appearance, age probing), escalates when a signal persists across k of n windows or on any single hard signal, and routes only to the safety lane. |
| Enforcement | Mentor S0 confirmed → staff end the mentorship, suspend the account, NCMEC if enticement. Student contact-sharing → S1/S2 education-first. |
| Evidence | Built: undeletable (D-41), thread archived intact on end. Add: safety case; preservation. |
| Races | Mentorship ended while held → cancel. Account suspended while held → cancel. Held for hours → sender frozen (SLA). Delivered order: the thread sorts by createdAt and displays publishedAt, so a long-held message can land above the counterpart's later replies — sort delivered messages by publishedAt. |
| Metrics | p95 time-to-delivered; escalate rate; S0 count; pair-pass alerts and their confirmation rate. |
4.2 Mentorship request note — mentorship.request_note
| Concern | Spec |
|---|---|
| Who → who | Student → a vetted mentor they chose; 500 chars + topics. First contact with an adult. |
| Harms | A student oversharing contact details or personal information; inappropriate content to an adult; self-harm disclosure. Low grooming risk originates here (students initiate), but it is where an off-platform invitation would first appear. |
| Entry | mentorship/service.ts createRequest, editRequest, withdrawRequest. |
| Gates | Built: requestable mentor, one pending request per mentor, cap of 5 pending, 30-day expiry (including notes that never cleared). |
| Staging | Built: mentorship_requests.review_status, separate from the request lifecycle (D-32). |
| Context sent | Note + selected topics + student age band. |
| Decision | Built: adult bar. Add: severity table; stage-1 PII → reject. |
| Delivery guard | Built: onRequestNoteDelivered requires status = pending. Gap: withdrawRequest does not supersede the review, so a withdrawn request's escalated note stays in the human queue and approval still stamps deliveredAt on a withdrawn row. |
| Author | Built: "Your message is being checked. It goes to {mentor} once it's cleared."; flagged → category copy + edit; removed after a mentor's report → reads as a flag without the text (D-46). |
| Mentor | Built: sees nothing until delivered; accept/decline require a delivered note. |
| Reviewer panel | Add: topics, student age band, the student's other pending requests (spray pattern), prior flags. |
| Post-publication | Built: the mentor reports the note as itself → removal → back to the student as a flag. |
| Races | Withdraw while held (above); mentor stops accepting while held → deliver anyway (the request was valid at submit) — confirm as intended. |
4.3 Peer message — message.body, conversation kind peer
| Concern | Spec |
|---|---|
| Who → who | Student ↔ connected student (D-47); 1,000 chars. |
| Harms | Bullying, sexual content including peer requests for images, contact exchange (including split across messages), hate, self-harm. |
| Gates | Built: connected; one undelivered message per sender; block removes the connection and archives the thread (D-50). |
| Context sent | Last ~10 delivered messages as role tokens (A, B with age bands). |
| Decision | Peer bar (D-49). sexual ≥ 2 → S0 (solicitation between minors). |
| Delivery guard | Still connected; no block either way. Gap: same as 4.1 — a held message approved after a block is delivered into the archived thread and notifies the person who blocked. requestNotification does not consult blocks. |
| Reviewer panel | Thread excerpt, both age bands, reports and blocks between the pair, prior flags. |
| Post-publication | Built: report → remove → tombstone; block from the report sheet. |
| Enforcement | Ladder; S0 sextortion → safety case. |
4.4 Comment — comment.body
| Concern | Spec |
|---|---|
| Who → who | Connection of the post author → the post's audience; 600 chars; flat (docs/08). |
| Harms | Targeted bullying, pile-ons, harassment of the post author, hate, contact exchange. |
| Entry | feed/service.ts createComment, editComment (re-checks the connection gate). |
| Gates | Built: must be connected to the post author. Add: flag budget — comments are the surface D-20 names as the iteration target. |
| Context sent | Parent post body (the target) and the last ~5 published comments on it, with role tokens (POST_AUTHOR, COMMENTER, OTHER). Bullying severity depends on the target. |
| Delivery guard | Post still published and not deleted/removed; commenter still connected; no block. Gap: onCommentPublished checks only that the rows exist, so a late approval notifies the author of a deleted post, or a person who has since disconnected. |
| Author | Built: "Your comment is being checked. It'll appear here once it's cleared."; flagged → category copy + edit. |
| Post author | Built: notified only on publish; may delete comments on their own post, which tells nobody. Add: record author-deletions of others' comments as a weak signal (not a report). |
| Reviewer panel | Post text and author, the comment thread, the commenter's recent comments across posts (pile-on), prior flags. |
| Post-publication | Built: report → remove → tombstone (a tombstone does not count as a comment). |
4.5 Post — post.body (+ link)
| Concern | Spec |
|---|---|
| Who → who | Student → connections and same-school members per visibility; 1,200 chars + optional link. |
| Harms | Bullying naming classmates, hate, sexual, threats against a school, spam, contact exchange, self-harm. |
| Entry | feed/service.ts createPost, editPost (flagged only), deletePost (soft). |
| Stage 1 | Built: the link is concatenated into the snapshot, so every linked post routes to a human (D-22). |
| Context sent | Text + link domain. Nothing about named classmates (identifiers stay out, docs/12 §6); the reviewer sees names. |
| Decision | Peer bar. A threat of violence against a school or person → S0 with an emergency protocol (who at the school and which authorities, decided with counsel). |
| Author | Built: own pending post visible and marked; flagged → category copy + edit (D-27). |
| Reviewer panel | Post, link domain, author's recent posts, prior flags. |
| Post-publication | Built: report → remove → tombstone (D-58); no editing after publish. |
4.6 Student bio and career goals — profile.bio (200), profile.career_goals (300)
| Concern | Spec |
|---|---|
| Who → who | Student → profile viewers per visibility (indexing locked for minors). Long-lived and always on display. |
| Harms | Contact handles (the classic spot), sexual self-description, hate, self-harm. False-positive risk on career goals: "forensic pathologist", "join the Marines", "ROTC", "shooting guard", "pharmacist". |
| Entry | Onboarding (onboarding/service.ts) and PATCH /profiles/me (profiles/routes.ts). |
| Staging | Built: column-shaped; previous value stays live; clearing applies immediately. |
| Context sent | Surface + age band + a per-surface policy addendum (career aspirations are professional content). Gap: D-15 split the surface names so policy could differ, but MODERATION_POLICY has no per-surface guidance. |
| Author | Built: "Your edit is being checked — this still shows the last approved version."; flagged → edit form, category copy, appeal (the only surfaces with appeal UI); content_flagged routes to the profile editor. |
| Reviewer panel | Add: diff against the current approved value; prior flags. |
| Post-publication | Gap: no path. A user report previews only the display name, and resolveReport("remove") has no branch for user, so the report is marked actioned while the text stays. Add: report subject profile_text with a field; removal clears the field, marks the review removed, notifies the author neutrally. |
| Re-screen | Add: when the policy version changes, re-screen currently-live values (Messages API, not Batch, if ZDR is in place). |
| Client | useProfile polls every 2 s while anything is pending — including human-held items that may take hours. Back off after a few minutes. |
4.7 Mentor bio and experience description — mentor.bio (300), mentor.experience (300)
| Concern | Spec |
|---|---|
| Who → who | Adult mentor → every student browsing the cross-school directory. |
| Harms | An adult→minor contact channel hidden in prose ("DM me on…"), inappropriate self-presentation, paid tutoring solicitation, impersonation claims. |
| Entry | Mentor onboarding; updateMentorProfile. Experience title/organisation save immediately behind the D-45 gate; the description stages. |
| Decision | Today the peer bar (0.7): D-33/D-49 scoped the adult bar to channels. Adult-authored text read by minors is the D-45 concern in prose form → decision (§8). |
| Context sent | Surface, role mentor 18+, experience title and organisation. |
| Reviewer panel | Mentor verification status (domain match, D-4), other profile fields, prior flags. |
| Post-publication | Same gap as 4.6. |
| Enforcement | Confirmed contact solicitation by a mentor → termination (docs/04 §8), on human confirmation only. |
4.8 Educator bio and school description — educator.bio (300), school.description (600)
| Concern | Spec |
|---|---|
| Who → who | School staff → students and staff at that school. |
| Harms | Low. The live issue is a false positive: stage 1 rejects emails and phone numbers on every surface, so a school's official office number or address in its description is refused with the "keeping contact on Thnkrr keeps everyone safe" copy. Decide whether official school contact details belong in structured fields or an allowlist (§8). |
| Flag destination | Built: educator bio → profile editor; school description → school-page management. |
| Post-publication | Gap: school.description is not a report subject at all; educator bio has the 4.6 gap. |
4.9 Write-time gated fields
Display name (all roles, 50), major, minor, hometown, typed high school, experience title, organisation, typed college name, not-listed school name.
| Concern | Spec |
|---|---|
| Method | Built: assertNoContactDetails refuses the save synchronously (D-45); no staging, no model. |
| Gap | No normalisation; no hate or sexual screening — a slur as a display name renders on every post, comment, message and directory card. |
| Add | Normalised variants in the gate; a small ops-owned slur/sexual lexicon at the same refusal (consistent with D-45's "no model"). |
| Existing values | When the lexicon changes, flag existing matches and require a new value at next sign-in (Discord's forced-rename pattern); the display falls back to initials meanwhile — decision. |
| Reports | A user report's "Remove" should offer "require rename" instead of doing nothing. |
4.10 Future: connection request note
A documented known gap (docs/07: "needs its own policy pass, not a field"). Inputs for that pass: LinkedIn publishes no classifier for invitation notes; the industry handles first contact with request inboxes and sender signals; the shape (people not yet connected) is the highest-risk in the product. If built: row-shaped and held-invisible; guard = request still pending and no block; sender velocity caps (D-30); "I don't know this person" blocks (D-29); never deliver a note from a sender under hold or with a recent confirmed S1.
4.11 Not moderated by design
Report notes, appeal notes and reviewer notes are read only by staff, so they fall outside docs/12 §2's rule ("another human reads it" means another member). They stay out of the registry, are escaped on render, and are never shown to members.
5. How to build a moderated surface
5.1 What adding a surface touches today
A surface's behaviour is spread across: the contracts registry (moderatedText),
its call site (submitForReview), COLUMN_APPLIERS or ROW_SURFACES +
applyRowDecision, isAdultChannel, the post-approval if chain in
applyDecision, the notifyAuthorOfFlag switch, REJECTION_MESSAGES (in
contracts and duplicated in thnkrr-web/src/api/schemas.ts),
ReportSubjectType + assertReportable + subjectPreview + resolveReport,
getMyReviews' row-surface exclusion, the web composer and render states, and an
e2e spec. What fell through: the user report no-op, school.description not
reportable, delivery hooks without guards, and no per-surface policy text.
5.2 The surface definition
One object per surface, consumed by the pipeline; a contracts test asserts every registered surface has one. A design sketch, not an implementation:
interface ModeratedSurface {
name: SurfaceName; // the moderatedText registry key
shape: "row" | "column";
authorRoles: AccountRole[];
audience: "addressee" | "post_audience" | "profile_viewers" | "directory" | "school";
channel(ctx: Ctx, review: ReviewRow): Promise<"peer" | "adult">; // bar selection (D-49)
gates: Gate[]; // eligibility, caps, restriction, author hold, flag budget
buildContext(ctx: Ctx, review: ReviewRow): Promise<ClassifierContext>; // role tokens, no identifiers
policy: { addendum: string; decision: DecisionTable }; // per-surface prompt text + rules
apply: { approve(tx: Tx, r: ReviewRow): Promise<void>; reject(tx: Tx, r: ReviewRow): Promise<void> };
deliveryGuard(ctx: Ctx, r: ReviewRow): Promise<"deliver" | { cancel: string }>;
onDelivered?(ctx: Ctx, r: ReviewRow): Promise<void>; // notifications
flagDestination(ctx: Ctx, r: ReviewRow): Promise<NotificationTarget>;
copy: { held: string; cancelled?: string; removed: string }; // category copy stays shared
reviewerPanel(ctx: Ctx, r: ReviewRow): Promise<ReviewerContext>;
report: { subjectType: string; preview: Preview; remove: Remover } | null;
takedown: Remover; // moderator removal without a report
rescreenOnPolicyChange: boolean;
preservation: "undeletable" | "soft_delete" | "staged_history";
}
5.3 Web building blocks
- A moderated composer with the states idle · submitting · held · flagged · cancelled · restricted, reusing today's copy and D-19's "state, never content".
- A moderated content renderer: published · tombstone.
- Reviewer panels by audience type (thread excerpt; post + comment thread; profile diff), with click-to-reveal masking.
- Copy served from contracts only — remove the duplicate
REJECTION_MESSAGESinsrc/api/schemas.ts. - Polling backoff once an item has been pending for more than a few minutes.
5.4 Test matrix per surface
| Level | Cases |
|---|---|
| Unit | Decision table (severity × channel → outcome); stage-1 corpus over normalised variants; context builder emits no names or ids. |
| Integration | Every state × every viewer (author, addressee, third party same school, other school, moderator); notification never fires for held or flagged; races — block / disconnect / mentorship end / delete / withdraw during a hold, then a human approves → no delivery, no notification, row kept; edit supersedes; appeal; report → remove → tombstone; takedown without report; author hold escalates silently. |
| E2E | Flagged, held, published, tombstone, reviewer approve/reject, report remove — per surface, extending moderation-flow.spec.ts. |
| Eval | A labelled slice of the golden set per surface, including hard negatives and obfuscated and injection variants; gates every policy or model change. |
5.5 Metrics per surface
Volume · p50/p95 time-to-visible · auto-approve / flag / escalate rates · human decision time by lane · appeal overturn rate · weekly sampled missed-harm rate on approvals · sampled false-positive rate on flags · resubmit-after-flag rate · reports per 1,000 items · refusal rate from the model provider.
5.6 Changing the classifier safely
- Golden set — ~100 items per category and per high-risk surface, including reclaimed terms, counter-speech, students discussing their own mental health, career-goal false positives, obfuscation, injection, and some non-English.
- Policy version — a hash of the policy prompt, the surface addendum and the
output schema, stored in
model_verdictwith the model and effort (no migration). - Shadow mode — a candidate policy or model runs beside the live one for a
few days, its verdict stored in
model_verdict.shadow, no user impact; a disagreement report decides promotion. At pilot volume this costs almost nothing. - Gate — no promotion without per-category precision/recall at or above the live version on the golden set.
- Audit — weekly human review of a random 2–5% of approvals and a sample of flags per category.
- Evidence — keep these results; they are what any accuracy claim to a school rests on (FTC Workado).
5.7 Definition of done for a moderated surface
- Registered through
moderatedTextwith a surface definition (§5.2) - Gates, context builder (no identifiers), per-surface policy addendum
- Decision table and channel selection
- Delivery guard with a cancelled state
- Author states: pending · held · flagged · cancelled · restricted, with copy
- Addressee and third-party visibility per state
- Reviewer panel and lane
- Report-as-itself, removal, takedown, tombstone or cleared-field notice
- Enforcement hooks and author-hold behaviour
- Preservation mode
- Test matrix (§5.4) and golden-set slice
- Metrics wired
6. Verified gaps in the current build (ranked)
| # | Gap | Evidence | Surfaces |
|---|---|---|---|
| 1 | No conversation-level context; grooming, sextortion and split contact details are invisible | sendMessage submits only the new body; the adapter sends one item | 4.1, 4.3 |
| 2 | No NCMEC lane for enticement; queue unprioritised | getQueue orders by createdAt; docs/04 §7 scopes the duty to CSAM | All DMs, posts |
| 3 | Delivery without re-validation after a delayed approval | onMessagePublished, onCommentPublished check existence only; requestNotification ignores blocks | 4.1, 4.3, 4.4 |
| 4 | Self-harm holds show no resources; the DM sender is frozen | 988 copy renders only in rejected state; one-undelivered rule | All, sharpest in DMs |
| 5 | Confidence-threshold gate with no labelled data | MODERATION_*_CONFIDENCE defaults; no eval set | All |
| 6 | Injection and obfuscation unhardened | Header and content share one string; no normalisation in stage1.ts | All |
| 7 | Reviewers lack context and actions | QueueItem carries text, surface, author name, model reason; DecideInput is approve/reject | All |
| 8 | No post-publication path for profile text; user report "Remove" is a no-op; school.description unreportable | resolveReport branches; assertReportable list | 4.6–4.9 |
| 9 | No enforcement; suspended/banned are read but never written | identity/service.ts reads status; no writer | All |
| 10 | No direct takedown without a report | Removal functions are reachable only via resolveReport | All |
| 11 | Display names unscreened for hate/sexual content | D-45 gate is contact-details only | 4.9 |
| 12 | Policy prompt has no per-surface guidance | MODERATION_POLICY | 4.6–4.8 |
| 13 | Withdrawn request's review stays in the queue | withdrawRequest doesn't supersede | 4.2 |
| 14 | School official contact details refused | Stage 1 runs on school.description | 4.8 |
| 15 | Data terms of the model provider vs the school data agreement's 60-day disposal. For the connector written today (Anthropic): flagged content kept up to 2 years even under zero retention, and batch processing not covered by zero retention. Every candidate provider needs the same check (§2.4). | Provider terms; docs/04 §7 plans batch re-screening | All |
| 16 | Long-held messages can sort above later replies | Thread orders by createdAt, displays publishedAt | 4.1, 4.3 |
| 17 | Client polls every 2 s for as long as an item is held | useProfile refetchInterval | 4.6–4.8 |
7. Build sequence and schema impact
Every schema change below is additive; none is structural.
Before the first real student
| Step | Work | Schema |
|---|---|---|
| 1 | Normalisation + injection hardening (structured input, datamarking, manipulation flag) | None |
| 2 | Delivery guards + block-aware notifications + request-withdraw supersede | None |
| 3 | Golden set, severity verdict, decision tables, policy version, shadow mode | None (model_verdict jsonb) |
| 3a | Choose the AI stack (§2.4): run the golden set through the shortlist — a bought classifier (OpenAI Moderation or Hive), a child-safety specialist (Thorn or Hive), and a policy-following model (OpenAI, gpt-oss-safeguard on a zero-retention host, or the existing Claude connector); compare catches, wrong blocks, latency, cost and data terms | None — one connector per chosen service |
| 4 | Thread/parent context builders for messages and comments | None |
| 5 | Self-harm resources at hold time; care lane | None (copy + contract field) — pending §8 |
| 6 | Severity lanes and reviewer panels; reviewer actions | Additive: content_reviews.severity, content_reviews.lane |
| 7 | Author hold; safety cases; NCMEC registration and runbook; preservation | Additive: author_holds, safety_cases |
| 8 | Report-as-itself + takedown for profile text and school description | None (reports.subject_type is text; contract enum only) |
| 9 | Data agreements with whichever providers are chosen (no training, shortest retention); flagged-retention and statutory-report carve-outs in the school data agreement | None |
During the pilot
| Step | Work | Schema |
|---|---|---|
| 10 | Pair risk pass for mentorship threads (pg-boss job) | Additive: conversation_risk |
| 11 | Enforcement ladder in authorize | Additive: enforcement_actions |
| 12 | Display-name lexicon + forced rename | Additive column on profiles |
| 13 | Weekly audits, per-surface metrics | None |
| 14 | Add a child-safety specialist (Thorn Safer Predict or Hive) if the step 3a bake-off didn't include one | None |
Do not adopt for the pilot: ROOST Coop or Osprey as dependencies (they need Redis/ScyllaDB/ClickHouse or Kafka/Druid), a dedicated self-hosted GPU (idle at pilot volume), Google's Gemini API without an explicit under-18 arrangement, Perspective API (shutting down), Azure custom categories (retired), Hugging Face TGI (maintenance mode), engagement-velocity or ranking-based moderation, automatic closure of member reports.
8. Decisions that need an owner
| # | Decision | Options | Note |
|---|---|---|---|
| 1 | Self-harm resources at hold time | Show on self-harm holds · keep after human decision only · reviewer "care" action sends them | Recommended: show at hold + care action |
| 2 | Imminent-risk protocol | Who is contacted (counsellor, guardian, emergency services), when, by whom | Counsel + school agreement |
| 3 | Credible school-threat protocol | Contacts and timing | Counsel + school agreement |
| 4 | Adult bar for adult-authored profile text | Keep peer bar (D-33 scope) · adult bar for mentor.*, educator.bio, school.description | Extends D-33/D-49 |
| 5 | Appeals on row surfaces | Edit-and-resubmit only · appeal when the author believes the wording was legitimate | Rephrasing is the harm for misflagged identity or mental-health speech |
| 6 | Counterpart age band in DM context | Omit · include | Extends D-59 → counsel |
| 7 | Enforcement ladder for minors | Durations; whether a school ever sees restrictions | School agreement |
| 8 | Mentor restriction on S0 | Silent author hold only · visible suspension before NCMEC report | Tip-off risk |
| 9 | School official contact details | Structured fields · allowlist for verified school domain/phone · keep refusing | 4.8 |
| 10 | Existing display names after a lexicon change | Forced rename at next sign-in · reviewer-only | 4.9 |
| 11 | AI model provider, and whether to add a specialist grooming classifier | Per §2.4 — chosen by test-set results, data terms and industry adoption, not by what is already wired | Eval + quotes |
| 12 | NCMEC ownership | Named safety lead, backup, counsel sign-off | Before pilot |
9. Not verified
- Thorn Safer Predict and Hive pricing, eligibility, retention and languages.
- Adoption evidence comes from public statements, several read only as search excerpts; data terms for Hive, Thorn, Google Cloud, Azure, Amazon, Mistral and xAI; Vertex AI's minors terms; OpenAI Moderation's retention on its own page.
- For any hosted model: whether the provider's own safety filters refuse sexual, grooming or self-harm text sent for classification, and whether moderation traffic could trigger enforcement on our account — under fail-closed that would halt publishing. Ask each candidate provider.
- Discord safety-alert triggers; whether Meta or Snap classify DM text for grooming.
- The Community Sift shutdown (single competitor source).
- Exact status and text of the reinstated EU interim derogation; Ofcom's pending proactive-technology statement.
- Default data retention for each hosted provider must be read from its current terms at contract time (for Anthropic, its privacy center says 30 days; a third-party source says 7).
- The SLA targets in §3.6 are proposals, not published norms.
- Lantern eligibility for a pilot-stage company.