Thnkrr: Text Moderation — Industry Practice and Per-Surface Build Flows

Research document, 2026-09-15. Not a decision document — anything adopted from here becomes a D- entry in thnkrr-api/docs/07-decision-log.md.

It builds on, and does not restate, thnkrr-api docs/04 §7 (the pipeline), docs/10 (messaging), docs/12 (AI), docs/13 (cross-cutting) and decisions D-9 … D-65. Code references were verified against thnkrr-api main@8636bdb and thnkrr-web main@3b9823e. Nothing in either repository was changed.

Source tags: [Primary] the organisation's own publication, statute or regulator text · [Reported] press, secondary or search-summary source · [Analysis] inference. Research is not legal advice.

Contents

  1. Summary · At a glance: what we use, for what, and where it runs
  2. What is already settled
  3. How the industry moderates text
  4. The canonical flow every moderated surface implements
  5. Per-surface flow cards
  6. How to build a moderated surface
  7. Verified gaps in the current build
  8. Build sequence and schema impact
  9. Decisions that need an owner
  10. Not verified

0. Summary


At a glance: what we use, for what, and where it runs

Built = in the code today. Proposed = recommended in this document. The AI model provider is not decided. The code has one swappable connection point for it, and §2.4 compares the options by what the industry actually uses.

The parts

JobWhat does itWhere it runsWhat leaves our serversStatus
Decide who may post, comment or messageOur own permission rules (src/lib/authorize.ts) and per-feature checksOur API serverNothingBuilt
Queue each new item for checkingpg-boss, a job queue that lives inside our Postgres databaseOur API server and databaseNothingBuilt
Undo disguises: invisible characters, look-alike letters, spaced-out digits, number wordsOur code plus open-source text libraries (§2.4)Our API serverNothingProposed
Stage 1: catch contact details and a few fixed phrasesOur own pattern rules (src/modules/moderation/stage1.ts): emails, phone numbers, social handles, links, 4 self-harm phrasesOur API server — no AI, instant, freeNothingBuilt (rules); open-source libraries proposed
Stage 2: judge meaning — bullying, sexual content, grooming, hate, threats, scams and the restAn AI language model that reads the text against our written policy (src/modules/moderation/policy.ts) and returns a verdictThe model provider's servers (hosted API), or our own GPU server if we run an open-weight modelThe text, plus a coarse age band (13–15 / 16–17 / 18+). Proposed: a short thread excerpt with names replaced by roles. Never names, schools or account ids.The connection point is built (classifier.ts). The only connector written so far calls Anthropic's Claude. Which model to use is undecided (§2.4). With no model configured, everything goes to a person.
Watch whole mentorship conversations for grooming patternsThe same AI model slot, or a specialist child-safety classifier (§2.4)Provider's servers or our serverThread excerpts with roles, not namesProposed
Record every verdict (audit trail, appeals, test data)content_reviews and audit_log tablesOur PostgresNothingBuilt
Human reviewOur moderation page in thnkrr-web and its API routesOur servers; staff use a browserNothingBuilt (basic queue); severity lanes and context panels proposed
Member reportsOur report sheet and report queueOur serversNothingBuilt
Tell authors and recipientsOur notifications moduleOur serversNothingBuilt
Measure accuracyA labelled test set run through the same model connectionOur server → model providerSynthetic or de-identified test items onlyProposed
Report child exploitation to the authoritiesNCMEC CyberTipline (a US legal duty)NCMECThe report and its evidence, as the law requiresProposed
Profile photos and banners (for completeness)Human review of every upload (D-9)Our serversNothingBuilt

Which checks run on each surface

SurfaceStage 1 rulesAI modelContext the model seesConversation watchHuman lane
Mentorship messageYesYes, stricter barToday: the message only. Proposed: last ~20 messagesProposedDM lane; safety lane for serious cases
Mentorship request noteYesYes, stricter barNote (proposed: + topics)—DM lane
Peer messageYesYesToday: the message only. Proposed: last ~10 messages—DM lane
CommentYesYesToday: the comment only. Proposed: the post + recent comments—Standard
PostYes (any link goes to a person)YesThe post and its link—Standard
Bios, career goals, experience, school descriptionYesYesThe text and the author's role—Standard
Display name and short fields (major, organisation, hometown…)Contact-details check only (proposed: + a word list)No (D-45)——Through reports

1. What is already settled (not reopened here)

SettledWhere
Every free-text field a user writes and another user reads is screened before anyone else sees it; fail-closeddocs/04 §7, docs/12 §2
Stage-1 deterministic rules, stage-2 AI classifier behind a swappable connection (the one connector written calls Claude; the provider is not decided), human queue, stale sweepdocs/04 §7, D-51, D-60
Three outcomes (cleared / flagged / removed-after-publish) and the tombstonedocs/12 §3, D-58
Held comments and messages invisible to their authors; pending posts visible to their authordocs/12 §4, D-19, D-20, D-27
Flagged text is editable; published text is notD-27
The trigger is never named; category-level rule copy is the readingdocs/12 §5, D-59
Stricter approve bar on adult channels, chosen per conversation kindD-33, D-49
Structured fields get a write-time contact-details gate, not the modelD-45
Reports go to humans, never through the classifierdocs/12 §7, D-25, D-46, D-62
Messages are never deletabledocs/10 §6, D-41
Images are always human-reviewed in the pilotD-9
No mood/wellbeing inference, no generation, no rankingdocs/12 §8
Pilot-first minimalism; safety systems are the exceptiondocs/04 §1

The pre-send "reconsider" nudge is the one industry-standard pattern this design deliberately forgoes: it inherently reveals the trigger (docs/12 §3–4). The evidence is summarised in §2.3 for the record, not as a recommendation.


2. How the industry moderates text

2.1 LinkedIn

LinkedIn publishes one shared pipeline for all content; the only surface with a dedicated published model is private messaging.

The shared pipeline [Primary, DSA Systemic Risk Assessment, Aug 2025]: three layers — (1) at creation, ML services "filter out certain policy-violating content within 300 milliseconds of creation, meaning the content is visible only to the author"; (2) content "likely to be violative but for which the algorithm is not sufficiently confident" goes to human review; (3) member reports. Thresholds are set by risk: "For higher-risk content, LinkedIn prioritizes recall with lower thresholds… For lower-risk content, LinkedIn prioritizes precision" [Primary, DSA transparency report H1 2026]. Model scores feed a Drools rules engine whose actions are challenges, warnings, human queue or restriction [Primary, CASAL].

SurfaceMethod (published)
PostsPer-category deep models run proactively "every few hours" plus reactive models on engagement velocity (viral spam, Apr 2023). Low quality is a distribution decision, not a moderation one. Queue prioritisation by an XGBoost violation probability auto-decides ~10% of queued items at "extremely high" precision (Nov 2023). [Primary]
CommentsSame pipeline. "Kindness Reminders" (English-only) before posting potentially harmful comments. Harassment is overwhelmingly human-decided: of 28,730 EU own-initiative cyber-harassment actions in H1 2026, 482 (~1.7%) were solely automated, versus ~92% for discrimination. [Primary]
MessagesA three-model cascade — sender behaviour → message content → sender–recipient interaction — each running only if the previous flags (Aug 2020). Action is recipient-side: message hidden behind a warning with view/report/mark-safe; a dedicated "trying to move the conversation off LinkedIn" warning (Help). [Primary]
Invitation notesNo published text classifier. Signals only: "I don't know this person" feeds sender restriction; invitation volume feeds behaviour models. [Primary]
Profile textA CNN text classifier replaced a phrase blocklist that misfired on context ("escort" as in security escort) and only ever grew; they hand-labelled appropriate profiles containing trigger words to avoid learning the blocklist, and re-scored the whole member base (Jan 2020). [Primary]
LLMsNo published use of LLMs deciding member-content policy. GenAI features are filtered on input/output; outputs are checked in 100-token chunks and the whole response retracted on failure (SRA 2025). [Primary]
EnforcementMoved from account restriction to feature restriction: violation → Statement of Reasons + in-product reminder at next compose → repeated: no posting, commenting or starting new threads (can still read, reply to DMs, delete own content) → permanent. CSAM/terrorism skip the ladder. Content appeals H1 2026: 12,135, ~25% reversed, median decision 2 h. [Primary, Help, SRA 2025]

LinkedIn's headline automated precision (0.98) is computed as one minus the appeal-overturn share, so unappealed errors are invisible. [Analysis]

2.2 Platforms with large minor audiences

PlatformText method, as published
RobloxPre-delivery filter at 6.1B messages/day: rules → trained models → an LLM "for context-based reasoning when uncertainty exists" [Primary, Mar 2026]. A dedicated PII classifier reads a snippet of several chat lines with separate ASKING_FOR_PII / GIVING_PII labels: 98% recall, 94% F1 vs LlamaGuard-3-8B's 27.7% F1, P90 < 100 ms, open weights [Primary, Nov 2025]; v2 with conversation context moved F1 63.4 → 90.5 [Primary, Aug 2026]. Sentinel runs separately from the filter: one-minute snapshots, embeddings compared to ~13k confirmed child-endangerment examples, per-user risk skew over time, human analysts, ~1,200 NCMEC reports in H1 2025 [Primary, Aug 2025]. Age-banded chat since Jan 2026; under-13s strict. Profanity is now LLM-rewritten with everyone notified; PII is still blocked, never rewritten. An independent audit of 2M messages found grooming, sexualisation and PII still passing via obfuscation [Primary, May 2026].
DiscordTeen DMs from new senders trigger a recipient-side safety alert (block/mute/tips; message still delivered). Unknown senders go to a Message Requests inbox by default (Feb 2026). AutoMod is keyword/regex + Discord-maintained lists; a blocked message notifies only its sender. "Account standing" instead of strikes; teens get more chances; child safety is zero-tolerance. [Primary, Oct 2023, Feb 2026]
Instagram (Teen Accounts)Strictest Hidden Words on by default for teens' comments and DM requests. Comment warnings (~1M/day; ~50% edited or deleted). DMs are governed mainly by account signals: adults recently blocked or reported by a teen lose the message button on teen profiles; accounts whose name or bio suggests inappropriateness can't follow, DM or comment on teens; new chats show the other account's join date and country. [Primary, Sep 2024, Jul 2025, Oct 2025]
YouTubeCreator-chosen comment holds (None/Basic/Strict/Hold all); held comments invisible until approved. Comments disabled on most videos featuring minors (2019). Pre-post reminder with "post anyway"; 24 h timeouts for repeat abuse. [Primary/Reported]
SnapchatTeens reachable only by mutual friends/contacts; warnings when chatting with someone "blocked or reported by others" or from an unusual region; friend requests silently not delivered when signals indicate scams. No disclosed text classifier for grooming. [Primary, Jun 2024]
XboxRecipient-side filter levels (child accounts default Friendly; hidden messages can't be revealed). Strikes weighted by severity, visible to the player, expire after 6 months; 2 → 1 day, 4 → 7 days, 8 → 1 year; <1% of players suspended in 2022. [Primary, Aug 2023]
Education toolsSeesaw: new student comments need teacher approval by default. Padlet Safety Net: AI screens every post/comment, flagged items blocked pending owner approval, admins can switch categories off (e.g. violence for a history class). Google Classroom: teacher mutes are silent to the student. [Primary]
Yubo (teen app until it raised its minimum age to 18, Nov 2025)Pop-up before sharing a phone number or address; SafeBlur blurred harmful text in received DMs. [Primary/Reported]

2.3 Cross-platform patterns

SurfaceDominant industry methodTimingContext used
Public postClassifier + blocklist; AI hold in education toolsMostly post-hoc; held in eduItem + account history
CommentOwner filters + classifier + pre-post promptHidden or heldItem; commenter account signals
Peer DMReal-time filter (Roblox) or recipient-side hide/blurPre-send or on deliverySeveral lines (Roblox PII); per-user penalties
Adult↔minor DMPrevented structurally first; where allowed, a background conversation/account-level detector → specialist humans → NCMECNear-real-time sweepConversation over time; reported-by-minor signal
First-contact textRequest inbox; silent non-delivery for risky sendersRouted or droppedSender signals (mutuals, reports, region, account age)
Profile fieldsBlocklist → forced rename; bio as a risk signalRefused or account restrictedName/bio + history

Industry-standard (≥3 platforms): structural limits on cold and adult contact before any text classification; request inboxes for first contact; recipient warnings driven by who the sender is; a fast per-item filter paired with a separate conversation/account-level grooming detector staffed by specialists; history-based graduated enforcement with visible standing.

The nudge evidence (recorded, not recommended): Twitter's RCT — 6% fewer offensive tweets (ICWSM 2022); Instagram — ~50% of warned comments edited/deleted; Nextdoor — 34% edited or withheld; OpenWeb/Perspective — secondary summaries report ~18% re-posting to test the filter. No published teen-specific effect size exists. Every platform that nudges does so only for incivility; PII, sexual content and grooming are blocked without explanation. [Primary/Reported]

2.4 Models, vendors, and what the industry actually uses

Checked 2026-09-15. "Adopted by" means a public statement that a platform uses the tool to moderate user content. Several vendor pages could only be read through search excerpts; those facts are tagged [Reported]. Prices are US$ per million input / output tokens unless stated.

What comparable platforms run

PlatformText moderation stackSource
RobloxIn-house models (PII classifier, Sentinel) plus an LLM layer for uncertain cases[Primary]
DiscordIn-house AutoMod (keyword lists + ML filters), Osprey rules engine; tested gpt-oss-safeguard[Primary]
Meta (Instagram, Facebook)In-house LLMs trained on its own policies[Primary]
LinkedInIn-house ML (§2.1)[Primary]
RedditIn-house harassment filter; Hive customer (2021)[Primary] / [Reported]
TwitchIn-house AutoMod[Reported]
XboxCommunity Sift, a Microsoft-owned vendor (current status unverified)[Reported]
BlueskyHive (images and video), Thorn Safer (CSAM), Osprey, Musubi PolicyAI (a vendor built on Gemini)[Primary] / [Reported]
XThorn Safer Predict text classifier[Reported]
YuboHive[Reported]
Khan AcademyOpenAI Moderation API for Khanmigo chats[Primary]
Padlet"Safety Net" AI moderation, reportedly OpenAI + Hive[Reported]
GrindrSpectrum Labs (now Alice); reportedly Musubi[Reported]

No public evidence was found for Duolingo, Quizlet, Brainly, Canva or Chegg.

What the evidence says

The building blocks, layer by layer

Layer 1 — deterministic rules, in our own code. Runs inside the Node API; nothing leaves our servers. No adoption data exists for libraries this small; these are the usual choices for each narrow job.

LibraryJobLicence · latest releaseNote
libphonenumber-jsPhone numbersMIT · 1.13.13 (Sep 2026)Finding numbers inside free text not verified
linkify-it / linkifyjsLinks; linkifyjs also finds emails and @mentionsMIT · 2026Active
obscenityProfanity, including leetspeak and disguised spellingsMIT · 0.4.6 (Jan 2026)English word set
decancerLook-alike (confusable) charactersMIT · 3.3.3 (Jul 2025)Unicode TR39 conformance not verified
words-to-numbers"five five five" → 555MIT · 2018Unmaintained — expect to own this code

Layer 2 — hosted moderation classifiers. Ready-trained per-category scoring, usually one message at a time.

ServiceCovers from our 10 categoriesDoesn't coverPriceData termsAdopted by
OpenAI ModerationHarassment, hate, self-harm, sexual (incl. sexual/minors), violence, illicitGrooming, contact details, spam, impersonationFreeAbuse-monitoring logs up to 30 days; zero retention only with OpenAI's approval; no training on API data [Reported]Khan Academy; built into ROOST Coop
Hive Text ModerationSeverity 0–3 for sexual, hate, violence, bullying, drugs, weapons, child safety, self-harm; yes/no for child exploitation, redirection off-platform, spam, phone numbers; patterns for email and address; Thorn grooming model (English/Spanish, conversations)ImpersonationReported $0.50 per 1,000 requests; enterprise via salesNot verifiedReddit, Yubo, Padlet (reported)
Thorn Safer PredictChild-exploitation labels: sextortion, child access, has-minor, CSAM, grooming signalsGeneral harmsSales onlyNot verifiedX; Safer is used by 80+ platforms
Mistral Moderation 2Sexual, hate, violence/threats, dangerous, criminal, self-harm, PII, jailbreaks; whole conversationsGrooming, spam, impersonationReported free30-day retention; the free tier trains on data unless opted out [Reported]None found
Google Cloud moderateText16 attributes incl. toxic, insult, sexual, drugs, weaponsGrooming, contact details, spamPer 100-character unitNot verifiedNone found
Azure AI Content SafetyHate, sexual, violence, self-harm, with severity levelsGrooming, contact details, spam, impersonationFree tier, then per 1,000 recordsNot verifiedSouth Australia's EdChat (screens an AI chatbot, not member content)
Amazon Comprehend toxicity7 classes, English onlyGrooming, contact details, drugs, spamPer 100-character unitNot verifiedNone found
Perspective APIToxicity———Shuts down 2026-12-31

Layer 3 — general AI models that follow our written policy. This is what reads context: bullying aimed at a specific person, disguised contact exchange, grooming across a thread.

Hosted by the model's maker

ProviderModels suited to classificationPriceData termsMinors and blockingModeration adopters
OpenAIgpt-5-nano; GPT-5.6 Luna; flagship GPT-5.6$0.05/$0.40; $0.20/$1.20 [Reported]No training by default; abuse logs up to 30 days; zero retention on approval; US/EU/UAE processing [Reported]Not verifiedSafetyKit (a moderation vendor) runs on GPT-5
Google (Gemini API)Gemini 2.5 / 3.1 Flash-Lite$0.10/$0.40; $0.25/$1.50 [Primary]Paid tier not used for trainingTerms bar services "likely to be accessed by individuals under the age of 18" [Primary, 2026-03-23]; child-safety content always blocked, including in inputsMusubi (a vendor) builds on Gemini; serves Bluesky
AnthropicClaude Haiku 4.5; Sonnet 5; flagship Opus 5$1/$5; $2/$10; $5/$25 [Primary]No training; 30-day retention; zero retention by agreement, but flagged content kept up to 2 years even then [Primary]Minors allowed with extra safeguards under its guidelines [Reported]None found
MistralMistral Small 4$0.15/$0.60 [Reported]Not verifiedNot verifiedNone found
xAIGrok 4.3$1.25/$2.50 [Primary]Not verifiedNot verifiedNone found

Open-weight models — run on our own GPU, or on a host that keeps no data

ModelSize · licenceFollows our written policy?LanguagesWhere it can runAdopters
gpt-oss-safeguard20B / 120B · Apache 2.0Yes, with its reasoningNot statedGroq $0.075/$0.30 (no retention by default); Amazon Bedrock $0.07/$0.20 (no retention for open models); own GPU with 16 GBTested by Discord; ROOST
Llama Guard 412B · Llama 4 licenceNo — 14 fixed categories, no grooming, spam or impersonation8DeepInfraCloudflare, for AI prompts rather than member content
Qwen3Guard0.6B / 4B / 8B · Apache 2.0No — 9 fixed categories, including PII119Own GPUNone found
Granite Guardian 4.18B · Apache 2.0Built-in risks plus custom criteriaEnglishOwn GPUNone found
Shieldstral 1.03B · Apache 2.0Yes — plain-language yes/no questions12Own GPU with 16 GBNone found (released Aug 2026)
Nemotron 3.5 Content Safety4B · OpenMDW + Gemma termsFixed categories plus a custom mode12Own GPUNone found
ShieldGemma2B / 9B / 27B · Gemma terms4 categoriesEnglishOwn GPUNone found
Roblox PII Classifier v20.6B · Apache 2.0 (one conflicting report)Contact exchange only: asking for or giving personal info, moving off-platform; reads chat history189CPU or GPU; latency unpublishedRoblox

Self-hosting means an inference server such as vLLM or SGLang (both Apache 2.0) on a rented GPU: roughly $358 a month for a RunPod L4 around the clock, or about €214 a month for a dedicated 24 GB Hetzner server [Reported]. At pilot volume (about 1.4 requests a minute) that GPU would sit more than 99% idle, while zero-retention hosts run the same open models for a few dollars a month. Hugging Face TGI is in maintenance mode.

Layer 4 — human review and rules infrastructure (open source).

ToolWhat it doesLicence · versionWhat it needs to runAdopted by
ROOST CoopReview queues, NCMEC reporting actions, strikes, appealsApache 2.0 · v1.0.1 (Jun 2026)Redis, Postgres, ScyllaDB, ClickHouseNone found
ROOST OspreyRules engine and investigation UIApache 2.0 · v1.0.1 (May 2026)Kafka, Druid, Postgres, MinIODiscord, Bluesky, Matrix
Tech Coalition LanternSharing child-safety signals between platformsMembership after eligibility review—38 members incl. Discord, Reddit, Roblox, Snap, Twitch, OpenAI

Monthly model cost at pilot volume

Rough estimate for 2,000 items a day (about 36M input + 9M output tokens a month), before caching or batch discounts [Analysis]:

Option≈ Per month
OpenAI ModerationFree
gpt-oss-safeguard-20b on Amazon Bedrock or Groq$4–11
OpenAI gpt-5-nano$5
OpenAI GPT-5.6 Luna$18
Anthropic Claude Haiku 4.5$81
Anthropic Claude Sonnet 5$160–210
OpenAI GPT-5.6 Terra$180
Anthropic Claude Opus 5 (the connector written today)~$405
A dedicated self-hosted GPU, around the clock$250–360

Provider terms that change the design

Notes on the connector written today (Anthropic Claude)

Can an AI model be trusted as the classifier?

Self-reported confidence is not a calibrated gate.

Adversarial text and injection. Universal appended phrases push LLM judges to maximum scores [Primary, EMNLP 2024]; zero-width characters, Unicode tags, homoglyphs and emoji smuggling evaded six guardrails with up to 100% success [Reported, 2025]; one injected invisible character significantly degrades commercial toxicity classifiers [Primary, IEEE S&P 2022]. Microsoft's "spotlighting" (delimit / datamark / encode untrusted text) cuts indirect-injection success sharply [Reported, 2024]. Classic classifiers (Layer 2) don't follow instructions, so injected text can't talk them into a verdict — but they are exposed to character-level tricks.

General models vs specialised classifiers. OpenAI says classifiers trained on tens of thousands of labels "can still outperform" its policy-reasoning model on complex risks [Reported, Oct 2025]. Zero- and few-shot LLMs underperformed fine-tuned encoders on real non-English data [Primary, May 2025]. On Gen Alpha slang the best model tested (an older Claude) recognised 42.3% of safety issues [Reported, FAccT 2025]. No current model has a published evaluation on teen DMs — the only real evidence will be a Thnkrr-labelled test set. [Analysis]

Context helps most where harm is conversation-shaped. Annotators without discussion context found 2.7× fewer violent-speech cases [Primary, 2022]; Roblox's PII classifier went from F1 63 to 90 with context; LLMs over-find grooming in harmless chats and are "led by the prompt to find non-existent risks" [Primary, 2024]; a turn-level grooming model reached 0.475 precision vs 0.047 for prior work, flagging in the first ~20% of a conversation [Primary, 2025].

The industry-standard stack at Thnkrr's size [Analysis]

Built only from the adoption evidence above:

  1. Our own rules, using the standard libraries in Layer 1. Every platform has this layer.
  2. A bought classifier for broad harms — OpenAI Moderation (free, used in education) or Hive (also covers contact details, off-platform redirection and child safety, with the most named youth customers).
  3. A child-safety specialist for grooming and sextortion — Thorn Safer Predict, or Hive's Thorn-based grooming model. At pilot scale, layers 2 and 3 can be one vendor (Hive covers both).
  4. A general AI model following our written policy, for context and nuance. Pick it by running the test set through a shortlist. The adoption evidence points to OpenAI models (used through moderation vendors) or the open-weight gpt-oss-safeguard on a zero-retention host. Google's API terms exclude us. Anthropic has no public moderation adopters but is already connected, so it stays on the shortlist and is judged on results.
  5. People, working in our own review queue. Large platforms build their own; the open-source queues need far more infrastructure than our stack runs.

How the code supports it: the classifier connection (classifier.ts) is called once per item and returns one verdict, so a combined connector that calls layers 2–4 and merges their results fits behind it without changing the pipeline. Each service added is another company receiving student text, another sub-processor, and another test-set run.

2.5 Techniques for the hard parts

Conversation-level grooming and sextortion. Grooming (O'Connell's stages: friendship → relationship → risk assessment → exclusivity → sexual) uses the vocabulary of real mentoring in its early stages, so per-message judgement misses it. Sextortion looks different: rapid, scripted, often posing as a peer, pushing off-platform; 38% of reports with chat logs contained "ruin your life" threats [Primary, Thorn/NCMEC 2024]. Early-detection research scores sliding windows and only alerts when k of the last n windows are positive ("skepticism") [Primary, ACL 2021]. Benchmark numbers are inflated by decoy data; LLMs over-find risk. Production systems use these detectors as high-recall candidate generators for human investigators, never as enforcers (Sentinel README). [Primary/Analysis]

Obfuscation. Normalise before matching: strip zero-width/bidi/Unicode-tag characters (their presence is itself a signal); NFKC + casefold; TR39 confusable skeletons are for comparison, not display, and NFKC must run first; map leetspeak; collapse single-character spacing ("s n a p"); convert number words to digits; also match over the sender's recent messages joined, to catch splitting. Roblox's ML PII filter found 25% more PII with 30% fewer false positives than its rules. Evasion is not always malicious: algospeak grows from moderation perceived as unfair. [Primary/Reported]

Severity and enforcement. Ofcom's codes require a prioritisation policy weighing reach, severity (including harm to children) and likelihood of illegality, plus recorded speed and accuracy targets. Strikes that expire (YouTube 90 days, Xbox 6 months) and education-first first offences (YouTube's warning-plus-training; Discord's teen leniency; Meta restricting only from the seventh violation, with ~80% of low-strike users not reoffending in 60 days) are the norm; child exploitation is zero-tolerance everywhere. Explanations of removals correlate with fewer future removals, and bot explanations work as well as human ones [Reported, CSCW 2019] — which supports the existing category-level copy.

False positives on teens. African American English up to 2× more likely labelled offensive [Primary, ACL 2019]; reclaimed slurs and in-group queer speech misclassified by LLMs; keyword self-harm flags on a youth forum produced many false positives while missing worrying messages without keywords [Reported, 2022].

Evaluation at small scale. Build a golden set; blind model labelling; disagreements rewrite the policy (OpenAI's loop). Binomial arithmetic: ±5% precision at 95% confidence needs ~385 flagged items per category (±10% needs ~96); zero misses in n audited approvals bounds the miss rate below ~3/n. Rare categories (grooming) need synthetic/red-team sets with recall reported on those. Pin model versions; re-run on every prompt change.

Latency. 0.1 s feels instant, 1 s keeps flow, 10 s is the attention limit (Nielsen). Field experiments with news sites found algorithm-supported pre-moderation reduced toxicity without lowering engagement [Reported, 2025].

2.6 Regulation and standards that shape the method

US — applies to the pilot now.

InstrumentRequirement that shapes moderation
18 U.S.C. §2258A as amended by the REPORT Act (in force since May 2024) [Primary, LII]Report to the NCMEC CyberTipline "as soon as reasonably possible after obtaining actual knowledge" of apparent CSAM, child sex trafficking, or coercion/enticement of a minor (§2422(b), attempts included). No duty to scan. A report is a 1-year preservation request covering "commingled content" that provides context (the thread); secure, access-limited storage. Penalties up to $600k first failure for providers under 100M monthly users. NCMEC's guidelines list enticement indicators: moving to private/other apps, posing as a peer or love interest, gifts/money/fake modelling offers, threats or claims to hold images, a child sharing address/school/phone around a meet-up; aggravating factors include an offender in a position of trust. [Primary, NCMEC Oct 2024]
"Actual knowledge"Undefined in the statute. A classifier score alone is weak; human confirmation clearly counts. Design: a fast human lane for enticement/trafficking, report on confirmation, never let queue backlog be the delay. [Analysis]
COPPA (amended rule; compliance from 2026-04-22)With a 13 floor, applies only on actual knowledge of an under-13 user — a moderator reading "I'm 12" is knowledge and needs an account workflow. A vendor used only for integrity fits "support for internal operations" if it does not train on the data. [Reported/Analysis]
FTC §5 (Workado, 2025)Accuracy claims about AI detection need "competent and reliable evidence" retained. Any safety claim to a school needs the eval data behind it. [Primary]
FERPA / SOPIPA / SDPC NDPA v2.2 / NY Ed Law 2-dVendor (and the AI model provider as a sub-processor) bound to use student data only for the service; sub-processor terms "no less stringent"; disposal within 60 days of request/termination; no AI-specific clause in the standard NDPA; its exceptions have no express carve-out for statutory CyberTipline reports. [Primary/Analysis]
California AB 1394 (if a pilot school is in CA)Bars "knowingly facilitat[ing]… commercial sexual exploitation" ($1M–$4M per act); safe harbour needs design/feature audits at least twice a year. [Primary]
Texas SCOPE Act filtering dutyEnjoined (5th Cir., Jul 2026). Nothing required now. [Reported]
KOSANot law as of Sep 2026. [Reported]

Expansion only.

JurisdictionRequirementConflict with the settled design
UK — Ofcom Illegal Content Codes (in force Mar 2025), Protection of Children Codes (Jul 2025)Moderation function, recorded speed + accuracy targets, prioritisation policy, resourcing, training; appeals with targets; grooming defaults (children receive DMs only from connections, not suggested as connections). Proactive grooming-detection measures still pending (statement expected autumn 2026). [Primary]Defaults already met by design
EU — DSA Art 17Statement of reasons for every visibility restriction: specific ground, facts, whether automated, redress. [Primary]"Trigger never named" must become "category + clause + automation disclosure + redress"; no tip-off exception for grooming — counsel
EU — ePrivacy interim derogation (reinstated to 2028)Voluntary scanning of interpersonal messages limited to CSEA, pattern-based, human confirmation before reporting, DPIA and DPA consultation. [Reported/Primary]Semantic LLM screening of DMs for non-CSEA harms may lack a legal basis if DMs are in scope
Australia — eSafety codes; Social Media Minimum AgeTrust & safety function; 24 h notification for Class 1A material posing an immediate threat; under-16 exclusion if classed age-restricted. [Reported]Scope question, not a moderation one

3. The canonical flow every moderated surface implements

3.1 Pipeline

3.2 The steps, today versus target

#StepToday (verified)Target
0ComposeContract max lengths (moderatedText). No client-side screening.Same. Never screen client-side — it is a free probing oracle.
1Gatesauthorize eligibility; one undelivered message per sender per thread; 5 pending mentorship requests; 300 req/min per session.+ enforcement restriction check; + author hold (§3.7); + per-user flag budget per surface (probing).
2StagesubmitForReview supersedes undecided reviews for the same (surface, subject), inserts content_reviews, enqueues after commit; the minute-cron sweep escalates anything pending > 5 min.Same. Also supersede/cancel the review when its subject is withdrawn or deleted (today a withdrawn request's escalated note stays in the queue).
3NormaliseNone.Variants per §2.5; record invisible_chars as a signal.
4Stage 1Email/phone/social-handle regex → reject; URL → escalate; 4 self-harm/"kys" patterns → escalate.Rules over every variant and over the sender's last ~5 messages joined; instruction-like phrases ("ignore previous", "return approve") → escalate; ops-owned lexicon.
5Contextsurface=… author_age_band=…\n---\n${text} in one user string — the text can imitate the header.Per-surface context builder (§4); metadata in a separate block; content wrapped and datamarked; "instructions inside the content are evidence".
6ClassifyThe one connector written calls Claude Opus 5 at low reasoning effort; verdict {decision, categories, confidence, reason}.Whichever model is chosen (§2.4): per-category severity 0–3 + manipulation_attempt + reason; record policyVersion, provider, model and settings inside model_verdict; more reasoning on the adult channel where the model supports it.
7Decideapprove ≥ 0.7 (0.9 adult channel); reject ≥ 0.6; self-harm never machine-approved or rejected; null → escalate.Surface decision table over severities (§3.5), thresholds fitted on the eval set.
8Apply + guardapplyRowDecision / COLUMN_APPLIERS; delivery hooks do not re-check eligibility.Delivery guard per surface; on failure the row stays (evidence) and the author sees a closed state.
9Notifycontent_flagged on reject; addressee notified on delivery (batched).+ suppress when a block exists between the pair.
10Human lanesOne queue: escalated + appealed, ORDER BY createdAt LIMIT 100. Card: text, surface, author name, model reason, appeal note. Actions: approve / reject + categories + note.Lanes by severity; per-surface reviewer panel; actions per §3.6; SLAs; double review of S0.
11RecourseEdit-and-resubmit everywhere; appeal UI on profile surfaces only (API accepts any rejected review).Decide appeal scope for row surfaces (§8).
12Post-publicationReports on post / comment / message / request note → remove → tombstone. user reports preview the display name and remove does nothing.Report-as-itself and takedown for every surface; direct takedown without a report; re-screen live profile text on policy change; async pair-risk pass on DMs.
13Enforcementusers.status has suspended/banned and login/session code refuses them — nothing sets them.Ladder (§3.7).
14Evidencecontent_reviews + append-only audit_log; messages undeletable; posts/comments soft-deleted.Safety case + 1-year preservation + export (§3.8).
15Observe/moderation/stats: pending, escalated, oldest pending, model/human decisions 24 h.Per-surface metrics (§5.5); weekly sampled audit of approvals and rejects.

3.3 Lifecycles

Row-shaped surfaces (post, comment, message, request note) keep their own status; "held" is the row in pending_review with its review escalated.

Column-shaped surfaces (bios, career goals, experience description, school description) stage the candidate text in the review row; the previous approved value stays live until approval; clearing a field applies immediately.

3.4 Who sees what, by state

Row-shaped:

StateAuthorAddressee (DM counterpart · mentor · post author)Other membersModerator
Pending (model working)Post: own post marked "visible to others in a moment" (D-20). Comment / message / note: status line, no text (D-19).Nothing, no notificationNothingNot queued unless stale > 5 min
Held (escalated)Identical to pending — deliberately indistinguishableNothingNothingQueue
PublishedNormalNormal + notificationPer visibility rules—
Flagged (rejected)Category copy, text back in composer, edit and resubmit (+ clear, messages); content_flagged notificationNothingNothingOnly if appealed
Discarded (messages)Gone from composer; row keptNothingNothing—
Cancelled (proposed)"Couldn't be delivered — this conversation closed"NothingNothingRow kept
Removed after publishTombstone, text stripped server-sideTombstoneTombstoneReport resolved

Column-shaped:

StateAuthorViewersModerator
Pending / held"Your edit is being checked — this still shows the last approved version."Previous approved valueQueue if held
ApprovedNew valueNew value—
FlaggedEdit form with category copy; appealPrevious approved valueQueue if appealed
Removed after publish (proposed)Field cleared with a neutral notice and a path to rewriteField emptyReport resolved

3.5 Severity model and decision table

The classifier returns a 0–3 severity per category; code decides.

TierMeaningExamplesOutcome
S0Child exploitation, enticement, sextortion, credible threat of violence, imminent self-harmGrooming indicators on the adult channel; requests for images; "I'm bringing a gun Monday"; a specific plan to self-harmHold → safety lane (≤ 15 min, waking hours) → author hold, evidence snapshot, NCMEC or emergency protocol on human confirmation
S1Slurs, sexual harassment, doxxing, contact exchange on the adult channel, targeted threatsFlag; counts toward a strike only when a human confirms
S2Insults, mild sexual innuendo, contact exchange between peersFlag; no strike on first offence
S3Undirected profanity, heated but not targetedApprove (per surface)

Per-surface rules (examples; thresholds fitted on the eval set):

RuleApplies to
Approve only if every category is 0 and no manipulation flagAdult channels (mentorship message, request note); adult-authored text read by minors — decision (§8)
grooming_risk ≥ 1 → escalate, never auto-reject (humans must see it)All DMs
sexual ≥ 2 between minors → S0 (possible solicitation of images)Peer DMs, comments
violence threat against a school or person → S0 regardless of surfaceAll
self_harm ≥ 1 → care lane (§3.9)All
pii_or_contact_exchange stage-1 hit → reject; model-only hit → escalate on adult channels, reject elsewhereAll

3.6 Human review lanes, reviewer actions, SLAs

LaneContentsTarget (proposed, not a published norm)Reviewer
SafetyS0 escalations; self_harm_concern and sexual/violence reports on DMs; author holds≤ 15 min in student waking hours; paging rota with a named backupSafety lead (a narrower role than moderator); double review before any report or account action
DMHeld peer/mentorship messages and request notes≤ 1 h waking hours — the sender's thread is frozen by the one-undelivered ruleModerator
StandardHeld posts, comments, profile text; appeals; other reports≤ 1 school dayModerator

Reviewer actions (today: approve, reject + categories + note):

ActionEffect
ApproveGuard → publish/deliver/apply
Reject (category)Flag with category copy; no strike
Reject + strikeFlag + enforcement record (§3.7)
Hold authorEvery new item from the author escalates silently (§3.7)
Open safety caseEvidence snapshot, preservation clock, NCMEC workflow (§3.8)
Care responseResources to the author (§3.9)
Needs second reviewerRoutes to safety lead
Record under-13 disclosureAccount workflow (COPPA)

Reviewer hygiene: mask slurs and sexual text until click-to-reveal (reduces impact without hurting accuracy [Reported, 2020]); never show reviewer notes to authors.

3.7 Enforcement ladder and the author hold

Author hold is the build's cheapest safety primitive because it needs no new UI: while a hold exists, processReview escalates every new item from that author with reason author_hold. The author sees the ordinary "being checked" state, so an S0 investigation is not tipped off, and nothing reaches anyone until a human clears it. It also answers filter probing: N flags in a window on one surface → temporary hold.

Ladder (education-first for minors; S0 skips it):

StepTriggerEffect
ReminderFirst human-confirmed S1, or repeated S2In-product policy reminder at next compose (category-level, never the phrase)
Feature restrictionSecond confirmed S1 within 90 daysNo posts, comments or new threads for 24 h–7 days; can read, reply in existing threads, edit profile, delete own content
Extended restrictionContinuedUp to 30 days; school-visible only if the school agreement says so (decision)
SuspensionS0 confirmed, or mentor contact-exchange confirmed (docs/04 §8)users.status = suspended, sessions revoked; mentors terminated

Strikes expire after ~90 days, are visible to their owner as standing (not a count), and a reversed decision removes its strike. Nothing is automatic from a model verdict alone.

3.8 Evidence, preservation, and the NCMEC lane

  1. S0 escalation → author hold + safety case (additive table) linking the review, subject, thread excerpt, both accounts, timestamps.
  2. Safety lead confirms or clears. On confirmation of apparent CSAM, trafficking or enticement: file a CyberTipline report (requires registering as a reporting provider), record the report id, start the 1-year preservation clock covering the thread context, restrict access to the case.
  3. Only after the report: enforcement (suspension, mentor termination). Held never-delivered messages are included — rejection does not discharge the duty.
  4. Account deletion (not yet built) must exempt anything under preservation.
  5. School data agreements need an explicit carve-out for statutory reports.

The procedure is designed with counsel before pilot launch (docs/04 §7 already requires this for CSAM; the scope is wider).

3.9 Self-harm care flow

Today: stage 1 or the model marks self_harm → escalate → the author sees the generic "being checked" state. The 988 copy in REJECTION_MESSAGES.self_harm ("A person on our team is taking a look at this…") renders only on the rejected state (thread.tsx, post.tsx, mentorship.tsx), i.e. after a human rejects. In a DM, the one-undelivered rule also stops the student writing anything else in that thread until a human acts. Reviewers have approve/reject only.

Target flow (the options in step 2 and 4 are decisions, §8):

  1. self_harm ≥ 1 → care lane with the safety-lane SLA.
  2. At hold time, show support resources to the author for self-harm holds (the probing value of revealing this category is negligible; the cost of a student in crisis waiting for a human is not).
  3. Reviewer chooses: deliver/publish (peer support and disclosure are often healthy), hold with a care message, or the imminent-risk protocol.
  4. The imminent-risk protocol — who is contacted (school counsellor, guardian, emergency services) and when — is a counsel + school-agreement decision. It is an explicit expression by the student, not inference, so it does not cross docs/12 §8's mood-detection line, but it does surface a minor's disclosure to an adult and must be designed as its own decision.
  5. self_harm never produces a strike.

4. Per-surface flow cards

Each card lists only what differs from §3. "Built" and "Gap" are verified against the code references at the top of this document.

4.1 Mentorship message — message.body, conversation kind mentorship

ConcernSpec
Who → whoVetted mentor (18+) ↔ student (13–18), only after a student's request is accepted. The one adult↔minor channel.
HarmsGrooming, sextortion, off-platform contact in either direction (mentor-terminating, docs/04 §8), sexual content, self-harm disclosure.
Entrymessaging/service.ts sendMessage, editMessage, discardFlaggedMessage.
GatesBuilt: party to a live mentorship; one undelivered message per sender. Add: author hold; restriction; first-N-messages scrutiny for a new pair.
Stage 1Add: variants + the sender's last 5 messages joined (split contact details).
Context sentLast ~20 delivered messages as role tokens (S student 13–15/16–17, M mentor 18+), message index, days since the mentorship started. No names, schools or ids. Sending the counterpart's age band extends D-59 → counsel.
DecisionApprove only if all severities are 0; grooming_risk, sexual, pii ≥ 1 → escalate; S0 → safety lane; effort medium.
Delivery guardMentorship still active; neither account held, suspended or banned. Gap: onMessagePublished re-checks nothing, so a message a human approves after the mentorship ended publishes into the archived thread and notifies the counterpart.
AuthorBuilt: "Your last message is being checked. You can write again once it's through."; flagged → category copy, edit or clear. Add: self-harm resources at hold (§3.9); closed state on cancel.
CounterpartBuilt: nothing until delivered; batched notification; persistent monitored notice; tombstone.
Reviewer panelAdd: thread excerpt with names (staff only), the label mentorship, mentorship start date, both age bands, both parties' prior flags, open reports on the pair, pair risk score. Gap: the queue card shows none of this — a reviewer cannot tell a mentorship message from a peer message.
Post-publicationBuilt: report the message as itself → remove → tombstone. Add: pair risk pass — an async pg-boss job per mentorship keeps a rolling state of hard signals (off-platform, secrecy, meeting up, gifts/money, appearance, age probing), escalates when a signal persists across k of n windows or on any single hard signal, and routes only to the safety lane.
EnforcementMentor S0 confirmed → staff end the mentorship, suspend the account, NCMEC if enticement. Student contact-sharing → S1/S2 education-first.
EvidenceBuilt: undeletable (D-41), thread archived intact on end. Add: safety case; preservation.
RacesMentorship ended while held → cancel. Account suspended while held → cancel. Held for hours → sender frozen (SLA). Delivered order: the thread sorts by createdAt and displays publishedAt, so a long-held message can land above the counterpart's later replies — sort delivered messages by publishedAt.
Metricsp95 time-to-delivered; escalate rate; S0 count; pair-pass alerts and their confirmation rate.

4.2 Mentorship request note — mentorship.request_note

ConcernSpec
Who → whoStudent → a vetted mentor they chose; 500 chars + topics. First contact with an adult.
HarmsA student oversharing contact details or personal information; inappropriate content to an adult; self-harm disclosure. Low grooming risk originates here (students initiate), but it is where an off-platform invitation would first appear.
Entrymentorship/service.ts createRequest, editRequest, withdrawRequest.
GatesBuilt: requestable mentor, one pending request per mentor, cap of 5 pending, 30-day expiry (including notes that never cleared).
StagingBuilt: mentorship_requests.review_status, separate from the request lifecycle (D-32).
Context sentNote + selected topics + student age band.
DecisionBuilt: adult bar. Add: severity table; stage-1 PII → reject.
Delivery guardBuilt: onRequestNoteDelivered requires status = pending. Gap: withdrawRequest does not supersede the review, so a withdrawn request's escalated note stays in the human queue and approval still stamps deliveredAt on a withdrawn row.
AuthorBuilt: "Your message is being checked. It goes to {mentor} once it's cleared."; flagged → category copy + edit; removed after a mentor's report → reads as a flag without the text (D-46).
MentorBuilt: sees nothing until delivered; accept/decline require a delivered note.
Reviewer panelAdd: topics, student age band, the student's other pending requests (spray pattern), prior flags.
Post-publicationBuilt: the mentor reports the note as itself → removal → back to the student as a flag.
RacesWithdraw while held (above); mentor stops accepting while held → deliver anyway (the request was valid at submit) — confirm as intended.

4.3 Peer message — message.body, conversation kind peer

ConcernSpec
Who → whoStudent ↔ connected student (D-47); 1,000 chars.
HarmsBullying, sexual content including peer requests for images, contact exchange (including split across messages), hate, self-harm.
GatesBuilt: connected; one undelivered message per sender; block removes the connection and archives the thread (D-50).
Context sentLast ~10 delivered messages as role tokens (A, B with age bands).
DecisionPeer bar (D-49). sexual ≥ 2 → S0 (solicitation between minors).
Delivery guardStill connected; no block either way. Gap: same as 4.1 — a held message approved after a block is delivered into the archived thread and notifies the person who blocked. requestNotification does not consult blocks.
Reviewer panelThread excerpt, both age bands, reports and blocks between the pair, prior flags.
Post-publicationBuilt: report → remove → tombstone; block from the report sheet.
EnforcementLadder; S0 sextortion → safety case.

4.4 Comment — comment.body

ConcernSpec
Who → whoConnection of the post author → the post's audience; 600 chars; flat (docs/08).
HarmsTargeted bullying, pile-ons, harassment of the post author, hate, contact exchange.
Entryfeed/service.ts createComment, editComment (re-checks the connection gate).
GatesBuilt: must be connected to the post author. Add: flag budget — comments are the surface D-20 names as the iteration target.
Context sentParent post body (the target) and the last ~5 published comments on it, with role tokens (POST_AUTHOR, COMMENTER, OTHER). Bullying severity depends on the target.
Delivery guardPost still published and not deleted/removed; commenter still connected; no block. Gap: onCommentPublished checks only that the rows exist, so a late approval notifies the author of a deleted post, or a person who has since disconnected.
AuthorBuilt: "Your comment is being checked. It'll appear here once it's cleared."; flagged → category copy + edit.
Post authorBuilt: notified only on publish; may delete comments on their own post, which tells nobody. Add: record author-deletions of others' comments as a weak signal (not a report).
Reviewer panelPost text and author, the comment thread, the commenter's recent comments across posts (pile-on), prior flags.
Post-publicationBuilt: report → remove → tombstone (a tombstone does not count as a comment).

4.5 Post — post.body (+ link)

ConcernSpec
Who → whoStudent → connections and same-school members per visibility; 1,200 chars + optional link.
HarmsBullying naming classmates, hate, sexual, threats against a school, spam, contact exchange, self-harm.
Entryfeed/service.ts createPost, editPost (flagged only), deletePost (soft).
Stage 1Built: the link is concatenated into the snapshot, so every linked post routes to a human (D-22).
Context sentText + link domain. Nothing about named classmates (identifiers stay out, docs/12 §6); the reviewer sees names.
DecisionPeer bar. A threat of violence against a school or person → S0 with an emergency protocol (who at the school and which authorities, decided with counsel).
AuthorBuilt: own pending post visible and marked; flagged → category copy + edit (D-27).
Reviewer panelPost, link domain, author's recent posts, prior flags.
Post-publicationBuilt: report → remove → tombstone (D-58); no editing after publish.

4.6 Student bio and career goals — profile.bio (200), profile.career_goals (300)

ConcernSpec
Who → whoStudent → profile viewers per visibility (indexing locked for minors). Long-lived and always on display.
HarmsContact handles (the classic spot), sexual self-description, hate, self-harm. False-positive risk on career goals: "forensic pathologist", "join the Marines", "ROTC", "shooting guard", "pharmacist".
EntryOnboarding (onboarding/service.ts) and PATCH /profiles/me (profiles/routes.ts).
StagingBuilt: column-shaped; previous value stays live; clearing applies immediately.
Context sentSurface + age band + a per-surface policy addendum (career aspirations are professional content). Gap: D-15 split the surface names so policy could differ, but MODERATION_POLICY has no per-surface guidance.
AuthorBuilt: "Your edit is being checked — this still shows the last approved version."; flagged → edit form, category copy, appeal (the only surfaces with appeal UI); content_flagged routes to the profile editor.
Reviewer panelAdd: diff against the current approved value; prior flags.
Post-publicationGap: no path. A user report previews only the display name, and resolveReport("remove") has no branch for user, so the report is marked actioned while the text stays. Add: report subject profile_text with a field; removal clears the field, marks the review removed, notifies the author neutrally.
Re-screenAdd: when the policy version changes, re-screen currently-live values (Messages API, not Batch, if ZDR is in place).
ClientuseProfile polls every 2 s while anything is pending — including human-held items that may take hours. Back off after a few minutes.

4.7 Mentor bio and experience description — mentor.bio (300), mentor.experience (300)

ConcernSpec
Who → whoAdult mentor → every student browsing the cross-school directory.
HarmsAn adult→minor contact channel hidden in prose ("DM me on…"), inappropriate self-presentation, paid tutoring solicitation, impersonation claims.
EntryMentor onboarding; updateMentorProfile. Experience title/organisation save immediately behind the D-45 gate; the description stages.
DecisionToday the peer bar (0.7): D-33/D-49 scoped the adult bar to channels. Adult-authored text read by minors is the D-45 concern in prose form → decision (§8).
Context sentSurface, role mentor 18+, experience title and organisation.
Reviewer panelMentor verification status (domain match, D-4), other profile fields, prior flags.
Post-publicationSame gap as 4.6.
EnforcementConfirmed contact solicitation by a mentor → termination (docs/04 §8), on human confirmation only.

4.8 Educator bio and school description — educator.bio (300), school.description (600)

ConcernSpec
Who → whoSchool staff → students and staff at that school.
HarmsLow. The live issue is a false positive: stage 1 rejects emails and phone numbers on every surface, so a school's official office number or address in its description is refused with the "keeping contact on Thnkrr keeps everyone safe" copy. Decide whether official school contact details belong in structured fields or an allowlist (§8).
Flag destinationBuilt: educator bio → profile editor; school description → school-page management.
Post-publicationGap: school.description is not a report subject at all; educator bio has the 4.6 gap.

4.9 Write-time gated fields

Display name (all roles, 50), major, minor, hometown, typed high school, experience title, organisation, typed college name, not-listed school name.

ConcernSpec
MethodBuilt: assertNoContactDetails refuses the save synchronously (D-45); no staging, no model.
GapNo normalisation; no hate or sexual screening — a slur as a display name renders on every post, comment, message and directory card.
AddNormalised variants in the gate; a small ops-owned slur/sexual lexicon at the same refusal (consistent with D-45's "no model").
Existing valuesWhen the lexicon changes, flag existing matches and require a new value at next sign-in (Discord's forced-rename pattern); the display falls back to initials meanwhile — decision.
ReportsA user report's "Remove" should offer "require rename" instead of doing nothing.

4.10 Future: connection request note

A documented known gap (docs/07: "needs its own policy pass, not a field"). Inputs for that pass: LinkedIn publishes no classifier for invitation notes; the industry handles first contact with request inboxes and sender signals; the shape (people not yet connected) is the highest-risk in the product. If built: row-shaped and held-invisible; guard = request still pending and no block; sender velocity caps (D-30); "I don't know this person" blocks (D-29); never deliver a note from a sender under hold or with a recent confirmed S1.

4.11 Not moderated by design

Report notes, appeal notes and reviewer notes are read only by staff, so they fall outside docs/12 §2's rule ("another human reads it" means another member). They stay out of the registry, are escaped on render, and are never shown to members.


5. How to build a moderated surface

5.1 What adding a surface touches today

A surface's behaviour is spread across: the contracts registry (moderatedText), its call site (submitForReview), COLUMN_APPLIERS or ROW_SURFACES + applyRowDecision, isAdultChannel, the post-approval if chain in applyDecision, the notifyAuthorOfFlag switch, REJECTION_MESSAGES (in contracts and duplicated in thnkrr-web/src/api/schemas.ts), ReportSubjectType + assertReportable + subjectPreview + resolveReport, getMyReviews' row-surface exclusion, the web composer and render states, and an e2e spec. What fell through: the user report no-op, school.description not reportable, delivery hooks without guards, and no per-surface policy text.

5.2 The surface definition

One object per surface, consumed by the pipeline; a contracts test asserts every registered surface has one. A design sketch, not an implementation:

interface ModeratedSurface {
  name: SurfaceName;                        // the moderatedText registry key
  shape: "row" | "column";
  authorRoles: AccountRole[];
  audience: "addressee" | "post_audience" | "profile_viewers" | "directory" | "school";
  channel(ctx: Ctx, review: ReviewRow): Promise<"peer" | "adult">;    // bar selection (D-49)
  gates: Gate[];                            // eligibility, caps, restriction, author hold, flag budget
  buildContext(ctx: Ctx, review: ReviewRow): Promise<ClassifierContext>; // role tokens, no identifiers
  policy: { addendum: string; decision: DecisionTable };               // per-surface prompt text + rules
  apply: { approve(tx: Tx, r: ReviewRow): Promise<void>; reject(tx: Tx, r: ReviewRow): Promise<void> };
  deliveryGuard(ctx: Ctx, r: ReviewRow): Promise<"deliver" | { cancel: string }>;
  onDelivered?(ctx: Ctx, r: ReviewRow): Promise<void>;                 // notifications
  flagDestination(ctx: Ctx, r: ReviewRow): Promise<NotificationTarget>;
  copy: { held: string; cancelled?: string; removed: string };         // category copy stays shared
  reviewerPanel(ctx: Ctx, r: ReviewRow): Promise<ReviewerContext>;
  report: { subjectType: string; preview: Preview; remove: Remover } | null;
  takedown: Remover;                        // moderator removal without a report
  rescreenOnPolicyChange: boolean;
  preservation: "undeletable" | "soft_delete" | "staged_history";
}

5.3 Web building blocks

5.4 Test matrix per surface

LevelCases
UnitDecision table (severity × channel → outcome); stage-1 corpus over normalised variants; context builder emits no names or ids.
IntegrationEvery state × every viewer (author, addressee, third party same school, other school, moderator); notification never fires for held or flagged; races — block / disconnect / mentorship end / delete / withdraw during a hold, then a human approves → no delivery, no notification, row kept; edit supersedes; appeal; report → remove → tombstone; takedown without report; author hold escalates silently.
E2EFlagged, held, published, tombstone, reviewer approve/reject, report remove — per surface, extending moderation-flow.spec.ts.
EvalA labelled slice of the golden set per surface, including hard negatives and obfuscated and injection variants; gates every policy or model change.

5.5 Metrics per surface

Volume · p50/p95 time-to-visible · auto-approve / flag / escalate rates · human decision time by lane · appeal overturn rate · weekly sampled missed-harm rate on approvals · sampled false-positive rate on flags · resubmit-after-flag rate · reports per 1,000 items · refusal rate from the model provider.

5.6 Changing the classifier safely

  1. Golden set — ~100 items per category and per high-risk surface, including reclaimed terms, counter-speech, students discussing their own mental health, career-goal false positives, obfuscation, injection, and some non-English.
  2. Policy version — a hash of the policy prompt, the surface addendum and the output schema, stored in model_verdict with the model and effort (no migration).
  3. Shadow mode — a candidate policy or model runs beside the live one for a few days, its verdict stored in model_verdict.shadow, no user impact; a disagreement report decides promotion. At pilot volume this costs almost nothing.
  4. Gate — no promotion without per-category precision/recall at or above the live version on the golden set.
  5. Audit — weekly human review of a random 2–5% of approvals and a sample of flags per category.
  6. Evidence — keep these results; they are what any accuracy claim to a school rests on (FTC Workado).

5.7 Definition of done for a moderated surface


6. Verified gaps in the current build (ranked)

#GapEvidenceSurfaces
1No conversation-level context; grooming, sextortion and split contact details are invisiblesendMessage submits only the new body; the adapter sends one item4.1, 4.3
2No NCMEC lane for enticement; queue unprioritisedgetQueue orders by createdAt; docs/04 §7 scopes the duty to CSAMAll DMs, posts
3Delivery without re-validation after a delayed approvalonMessagePublished, onCommentPublished check existence only; requestNotification ignores blocks4.1, 4.3, 4.4
4Self-harm holds show no resources; the DM sender is frozen988 copy renders only in rejected state; one-undelivered ruleAll, sharpest in DMs
5Confidence-threshold gate with no labelled dataMODERATION_*_CONFIDENCE defaults; no eval setAll
6Injection and obfuscation unhardenedHeader and content share one string; no normalisation in stage1.tsAll
7Reviewers lack context and actionsQueueItem carries text, surface, author name, model reason; DecideInput is approve/rejectAll
8No post-publication path for profile text; user report "Remove" is a no-op; school.description unreportableresolveReport branches; assertReportable list4.6–4.9
9No enforcement; suspended/banned are read but never writtenidentity/service.ts reads status; no writerAll
10No direct takedown without a reportRemoval functions are reachable only via resolveReportAll
11Display names unscreened for hate/sexual contentD-45 gate is contact-details only4.9
12Policy prompt has no per-surface guidanceMODERATION_POLICY4.6–4.8
13Withdrawn request's review stays in the queuewithdrawRequest doesn't supersede4.2
14School official contact details refusedStage 1 runs on school.description4.8
15Data terms of the model provider vs the school data agreement's 60-day disposal. For the connector written today (Anthropic): flagged content kept up to 2 years even under zero retention, and batch processing not covered by zero retention. Every candidate provider needs the same check (§2.4).Provider terms; docs/04 §7 plans batch re-screeningAll
16Long-held messages can sort above later repliesThread orders by createdAt, displays publishedAt4.1, 4.3
17Client polls every 2 s for as long as an item is helduseProfile refetchInterval4.6–4.8

7. Build sequence and schema impact

Every schema change below is additive; none is structural.

Before the first real student

StepWorkSchema
1Normalisation + injection hardening (structured input, datamarking, manipulation flag)None
2Delivery guards + block-aware notifications + request-withdraw supersedeNone
3Golden set, severity verdict, decision tables, policy version, shadow modeNone (model_verdict jsonb)
3aChoose the AI stack (§2.4): run the golden set through the shortlist — a bought classifier (OpenAI Moderation or Hive), a child-safety specialist (Thorn or Hive), and a policy-following model (OpenAI, gpt-oss-safeguard on a zero-retention host, or the existing Claude connector); compare catches, wrong blocks, latency, cost and data termsNone — one connector per chosen service
4Thread/parent context builders for messages and commentsNone
5Self-harm resources at hold time; care laneNone (copy + contract field) — pending §8
6Severity lanes and reviewer panels; reviewer actionsAdditive: content_reviews.severity, content_reviews.lane
7Author hold; safety cases; NCMEC registration and runbook; preservationAdditive: author_holds, safety_cases
8Report-as-itself + takedown for profile text and school descriptionNone (reports.subject_type is text; contract enum only)
9Data agreements with whichever providers are chosen (no training, shortest retention); flagged-retention and statutory-report carve-outs in the school data agreementNone

During the pilot

StepWorkSchema
10Pair risk pass for mentorship threads (pg-boss job)Additive: conversation_risk
11Enforcement ladder in authorizeAdditive: enforcement_actions
12Display-name lexicon + forced renameAdditive column on profiles
13Weekly audits, per-surface metricsNone
14Add a child-safety specialist (Thorn Safer Predict or Hive) if the step 3a bake-off didn't include oneNone

Do not adopt for the pilot: ROOST Coop or Osprey as dependencies (they need Redis/ScyllaDB/ClickHouse or Kafka/Druid), a dedicated self-hosted GPU (idle at pilot volume), Google's Gemini API without an explicit under-18 arrangement, Perspective API (shutting down), Azure custom categories (retired), Hugging Face TGI (maintenance mode), engagement-velocity or ranking-based moderation, automatic closure of member reports.


8. Decisions that need an owner

#DecisionOptionsNote
1Self-harm resources at hold timeShow on self-harm holds · keep after human decision only · reviewer "care" action sends themRecommended: show at hold + care action
2Imminent-risk protocolWho is contacted (counsellor, guardian, emergency services), when, by whomCounsel + school agreement
3Credible school-threat protocolContacts and timingCounsel + school agreement
4Adult bar for adult-authored profile textKeep peer bar (D-33 scope) · adult bar for mentor.*, educator.bio, school.descriptionExtends D-33/D-49
5Appeals on row surfacesEdit-and-resubmit only · appeal when the author believes the wording was legitimateRephrasing is the harm for misflagged identity or mental-health speech
6Counterpart age band in DM contextOmit · includeExtends D-59 → counsel
7Enforcement ladder for minorsDurations; whether a school ever sees restrictionsSchool agreement
8Mentor restriction on S0Silent author hold only · visible suspension before NCMEC reportTip-off risk
9School official contact detailsStructured fields · allowlist for verified school domain/phone · keep refusing4.8
10Existing display names after a lexicon changeForced rename at next sign-in · reviewer-only4.9
11AI model provider, and whether to add a specialist grooming classifierPer §2.4 — chosen by test-set results, data terms and industry adoption, not by what is already wiredEval + quotes
12NCMEC ownershipNamed safety lead, backup, counsel sign-offBefore pilot

9. Not verified

●  Have your own?

Paste your markdown, get a link like this.

Write your own
Thnkrr: Text Moderation — Industry Practice and Per-Surface Build Flows · DocHouse