Transparency report
How the safety layer actually works, what it catches, what it does not, and how we tested it. Written to be checked, not to reassure.
Last updated August 2026.
How a message is checked
Every message sent to the AI companion is classified for risk by a separate model call, running in parallel with the reply being written, not after it. A short list of explicit phrases is also checked instantly, before that model call, so the most direct statements are caught even if the classifier is slow or unreachable — that list can only ever escalate a message, never downgrade one, so it is a floor underneath the model classifier, not a replacement for it. This runs on every message, in every tool (Companion, Relationship Coach, Conversation Simulator, and Journal reflections), with no exceptions.
If the classifier flags a message before the reply has finished streaming, whatever was drafted is discarded and a fresh reply is written under an explicit safety directive: name crisis resources for the person's own region in the same message, not a future one, and stay with them rather than just listing numbers and moving on. If the classifier cannot be reached at all, the system fails toward caution and treats the message as a concern rather than assuming it is safe.
Crisis resources are region-aware, not defaulted to one country. The model is told explicitly which country's numbers apply and instructed never to volunteer a number from elsewhere — a number someone cannot dial is worse than none. Where the region is unknown, the reply points to findahelpline.com, which lists free lines in over 130 countries.
What changed, and why it matters
The earlier approach: keyword matching alone
Health and Kind previously ran an older tool (since retired) that relied only on scanning messages for a fixed list of phrases like "suicide" or "kill myself." In testing against realistic ways people actually express crisis, keyword matching alone missed the large majority of cases, including phrasings that carry real risk but use no obvious keyword — "I don't want to be here anymore," "I've written the letters." A keyword list can only ever catch what it was told to look for in advance, which is why the current system uses one only as an instant first check, backed by the model classifier for everything it would miss.
The current approach: model-based classification
The classifier reads the actual meaning of a message rather than matching fixed phrases, so it can recognise crisis language it has never seen written that exact way before. It is not perfect — see limitations below — but it is a meaningfully different, and meaningfully better, approach than pattern matching.
Our own test results
We test the classifier against a small set of realistic crisis phrasings — not just the obvious ones, and deliberately including ambiguous, indirect language people actually use. This is a committed, runnable script (scripts/test-crisis-classifier.mjs), not just a claim — anyone can re-run it and get the same result. In our most recent run (4 August 2026), of 8 test phrasings covering direct statements, plans, and indirect language:
- 7 of 8 were classified as an acute crisis and triggered a full safety-directed reply.
- 1 of 8, a genuinely ambiguous phrase ("I'm done, I can't keep doing this"), was classified at a lower "concern" tier rather than full crisis — flagged, but not at the highest severity. We consider this a reasonable judgment call for a message that could plausibly mean several different things, not a miss.
- Ordinary difficult messages (stress, exhaustion, a hard day) were not misclassified as crisis in testing.
Limitations, stated plainly
- This is our own internal testing, run by the people who built the system, not an independent audit. Treat our numbers as a starting point for scrutiny, not proof.
- Eight test phrasings is a small set. It tells us the classifier handles the cases we thought to test, not that it catches everything a real crisis could sound like.
- The classifier can be wrong in both directions: it could miss a genuine crisis phrased in a way we have not tested, or flag an ordinary hard message as more serious than it is.
- If the classifier service itself is unreachable, the system defaults to caution rather than skipping the check, but a completely offline moment is still a real gap, however brief.
- None of this replaces a real crisis line or emergency service. The companion is a bridge to those resources in the moment, not a substitute for them.
Current system status
See what's operational right now