Claude Gives Itself a Safety Score After Each Reply—Users Ask First: Is This Just Grading Its Own Homework?

In an October 16 Safeguards blog post, Anthropic introduced a mechanism called self-assessment. After each Claude reply, the system appends a short instruction asking it to perform a safety evaluation of the entire conversation. The result is structured JSON containing whether it was flagged, the reason, harm category, sub-policy, whether it refused, and human review priority. Users do not see this evaluation, and it is not carried into subsequent conversations; it is used only for offline review.

Anthropic's motivation is straightforward: abuse of frontier models has already reached scale. In 2025, attackers with suspected nation-state backing used Claude Code to attempt intrusions into about 30 organizations; a September 2026 threat intelligence report said criminal groups and hacktivists are also running similar AI-driven attacks. It lists four monitoring challenges: harm is context-dependent, monitoring requires intelligence, it must generalize to unknown unknowns, and it must be scalable.

The selling point of self-assessment is efficiency. Once Claude has replied, the entire conversation is already in the KV cache; evaluation only needs to process one short instruction and generates about 10 extra tokens on average. Anthropic estimates that under the production API traffic distribution, self-assessment adds less than 2% inference compute and is not billed to customers.

In the effectiveness data, at a 1% false positive rate, Claude Fable 5's self-assessment recalled 78% of conversations that a stronger prompted classifier marked as potentially harmful, with 94% recall for high-priority cases. ROC AUC was 0.986.

Across four model sizes—XS, S, M, and L—trained only with SFT, recall at a 1% false positive rate rose from about 16% to about 96%, and AUC went from 0.862 to 0.995. Anthropic's conclusion is that self-assessment gets stronger as model capabilities increase.

The generalization experiment used leave-one-out. Removing fraud training lowered synthetic fraud recall from 92.8% to 90.0%; removing cyber training lowered synthetic cyber from 92.7% to 90.3%; on production traffic, fraud went from 83.3% to 82.1%, and cyber from 94.2% to 90.7%. Anthropic's interpretation is that training mainly teaches Claude how to report harm, while the sense of harm comes from the model's existing knowledge.

Post-deployment uses are more practical. Self-assessment flags are aggregated nightly by organization, and multiple high-priority flags go to human review. In August 2026, self-assessment surfaced more than 100,000 child sexual abuse and exploitation-related flags from a single API customer; the child safety team investigated and reported them to NCMEC. About one-third of the flags could not be cleanly mapped to the existing taxonomy; after clustering, they revealed illegal distillation, prompt injection, and Claude acting as an agent to perform actions the user had not authorized. More than half of the organizations surfaced by self-assessment had no flags from any other detection source in the previous month. In September, there was also a case of using Claude Code to build and run an automated romance scam; after human review, the account was banned.

But self-assessment is not strong as a blocking classifier. Across 24 offensive cybersecurity tasks, the undefended attack success rate was 79%; self-assessment blocking reduced it to 54%, and an independent prompted classifier reduced it to 25%. Tokens generated per successful attack: 163k undefended, 691k with self-assessment, and 2.1M with the prompted classifier. Offline monitoring caught at least once 21 of the 24 attack trajectories.

Deployment can also add a session policy. By adding an allowlist to the reflection instruction and testing vulnerability research transcripts, Fable 5 flagged 100% of non-allowlisted targets and 0% of allowlisted targets across a broad threshold range.

Anthropic itself lists limitations: it inherits teacher-model blind spots; coverage is limited; monitoring a model's own behavior has failure modes—the model may deceive, and it may be more lenient toward its own actions; self-assessment is not designed for standalone blocking, nor can it stop attackers specifically targeting it.

The comment section was not convinced. Some asked whether it uses extra usage; an Anthropic footnote says customers are not billed, but the concern did not disappear. Someone wrote 'Claude rating its own homework now'; someone asked how to handle the lies it tells; someone said this is the road to a social credit system. Developer Dávid Kasza's long reply was more specific: Blue access approval is strict, legitimate defensive security techniques are easily treated as cyberattacks, and even building user-mode anti-cheat for a game was repeatedly flagged; he worries about when he will be banned, while GPT can distinguish defensive work from attacks.

The contradiction here is not new: the more a safety system covers all traffic, the more it has to handle edge cases; the cheaper self-assessment is, the more likely it is to be pushed onto more traffic. Anthropic's defense-in-depth logic makes sense—self-assessment is only one layer of offline signal, and real-time blocking is left to other classifiers. Users care more about whether it can distinguish red-team exercises from attackers, and defensive vulnerability research from abuse. Whether the model can reflect on itself matters less. Until that distinction is solved, the efficiency advantage of self-assessment will only amplify both false positives and controversy.