Insights · Start here
Similarity is not compliance: why AI similarity scores fail regulated communications
Why do teams reach for similarity scores?
The idea is intuitive. Turn the approved message into numbers that capture its meaning (an embedding), do the same for the new draft, and measure how close they are. Close means faithful; far means drift. Set a threshold, and you have an automatic compliance check.
We built the first version of our own checking on exactly this idea. Then we tested it.
What happened when we tested it?
We measured similarity against compliance several ways, on different sets of material:
| Test | What we found |
|---|---|
| Our own test sentences | The gap between compliant and violating restatements was 0.104 across seven sentences: too narrow to place any threshold in |
| A set of channel adaptations | Faithful rewrites scored 0.277 to 0.556 against the approved statement, below drifted versions. The checker preferred the drift |
| An earlier benchmark of 20,000 pairs | Area under the curve of 0.5099. A coin flip scores 0.5 |
| Real published assets from one approval | A cure claim ranked first (below) |
The last test is the one that settled it. We took four assets a large pharmaceutical company actually published around an approval, added two claims that must never pass, and ranked all six by similarity to the approved release:
| Similarity to the approved release | Asset |
|---|---|
| 0.763 | Test claim: "[product] cures [disease]" |
| 0.752 | Executive LinkedIn post (real, published) |
| 0.714 | Test claim: "works better than other [treatments]" |
| 0.679 | Patient advocacy story (real, published) |
| 0.621 | Consumer release (real, published) |
| 0.609 | Broadcast soundbite (real, published) |
The company and product are anonymized. The two bold rows are test claims we wrote; they were never published.
A cure claim outranked all four assets the company actually issued, and a claim of superiority over other treatments outranked three of them. Any threshold strict enough to catch the cure claim would have blocked every real, approved asset.
In a separate set of material, a version that deleted the restriction on who the product is indicated for scored 0.951, the highest of six versions tested.
Why does a good adaptation look "far" and a violation look "close"?
Because that is what good adaptation is. A strong patient version translates a clinical endpoint into everyday life: a change on a rating scale becomes what a person can manage on an ordinary day. The words change almost completely, so the similarity score falls, even though the meaning stays inside the evidence.
A violation does the opposite. "Cures [disease]" reuses the product, the disease and the approval frame almost word for word. It is on topic, so it scores high, while its meaning goes far beyond the evidence.
Similarity measures topical overlap. Both the faithful rewrite and the violation are on topic. One changes the words; the other changes the meaning. No threshold placed on a similarity score can tell those apart.
The sharpest form of the problem: deleting a qualifier moves a sentence closer to its source, not further. The most dangerous edit in regulated communications is the one a similarity check rewards.
What works instead?
Ask a different question. Not "does this resemble the approved message?" but "does every claim in this version trace to something that was approved?" That question is directional:
- Saying less is allowed. Dropping a statistic, describing an endpoint in everyday terms, simplifying a mechanism: that is what adapting for a channel means.
- Saying more is not. Any claim stronger, broader or newer than the approved set is unsupported, however natural it sounds.
- A missing required element is a prompt to check, not an automatic fault. The safety statement may sit elsewhere on the page.
Because the approved set is small and confirmed by people, this is a far easier question than "is this true?" asked against the whole internet, and it gives a sharper answer.
The verdict should also have three states, not two. Clear when nothing goes beyond the approved set; a read by a person when something may have been abstracted too far or a required element is missing; a block only when a version asserts something never approved, contradicts it, or reproduces a known forbidden phrasing word for word. A pass-or-fail gate on an uncertain signal either blocks good work or passes bad work. A three-state check clears the easy majority, puts the ambiguous minority in front of a person, and stops only the unambiguous.
Is similarity useless, then?
No. It is a good router. It tells you which of dozens of approved statements a paragraph is talking about, which is exactly the retrieval problem it was designed for, and it is a useful way to put the nearest approved claim in front of a reviewer. It just cannot tell you whether the paragraph is allowed. It ranks topic, and it should advise, never decide.
What should you ask of any AI checking tool?
- What does your check actually measure? If the answer is similarity to the approved text, ask how it handles a deleted qualifier.
- Can it tell "says less" from "says more"? Ask for a demonstration with a dropped statistic and with an added comparison.
- What does it block, and what does it pass to a person? A tool that blocks on a similarity threshold will either block faithful work or pass violations.
- Can you see why? A verdict a reviewer cannot trace to a specific claim and a specific approved statement is not one they can rely on.
How does CommsRx approach it?
CommsRx uses similarity only to find what a version is about and to show reviewers the nearest approved claim. The verdict comes from claim-by-claim authorization against the approved set, where a version may say less but never more, and from an exact match against each statement's forbidden versions. Only a word-for-word forbidden version, or a claim that goes beyond or contradicts the approved set, blocks. Everything else reaches the communications lead marked "needs a read."
Related questions
About this page
The measurements on this page are recorded in CommsRx's internal evaluation notes; the 20,000-pair benchmark comes from the founder's earlier work on the same method. The published assets are anonymized and the two test claims were written for the test. Drafted with AI assistance, then edited and checked by Abhi Basu, founder of CommsRx and Signal Fidelity Group. Last reviewed October 19, 2026. Corrections: help@signalfidelitygroup.com.