Claude named authors of 56% of unsigned peer reviews in small preprint test

Editorial illustration: A gold-rimmed magnifying glass enlarges abstract marks on cream documents. A glowing turquoise network connects matching marks beneath the lens to a faceless profile card.

In brief

  • Claude Fable 5.2 named the writer of 56% of 398 unsigned reviews, a preprint reports
  • Six volunteers wrote the reports; the model saw each one alone, without web access
  • Accuracy reached 68.4% on standard reviews and 31.1% on shorter meta-reviews
  • Public reviews containing self-citations saw accuracy rise to 80.2%
  • The study is a preprint that hasn't been peer reviewed, and its sample was small

How the test worked

The researchers analyzed reports written between 2015 and 2026 for 26 double-blind machine learning and natural language processing conferences and workshops. They knew who wrote each one, but the reviewers' names had never been publicly linked to the reports (in 19 cases the text itself was already public without a name).

The results weren't even across report types. Claude named the writer on 68.4% of 266 standard reviews and 31.1% of 132 meta-reviews. The meta-reviews were shorter and came from just two of the six contributors, and the study didn't separate the effects of report length, type and authorship.

The method of counting matters too. The 56% figure counts each report equally, so contributors who supplied more reports had more pull on the result. With each reviewer given equal weight, Claude's average accuracy was 49.1%, according to the preprint.

Public reviews and self-citations

The team also tested 1,559 public reviews from 47 open-review journals. Claude correctly identified the reviewer in 21.4% of the 1,530 cases it answered. The paper reported 39.3% accuracy on a subset of 789 reviews that Pangram 4.1, an AI-text detector, had labeled human or mixed (that's the filtered rate the abstract rounds to 39%).

Accuracy was higher on public reviews whose writers cited their own work, according to the preprint. Claude was 80.2% accurate on 106 reviews with self-citations and 17.1% accurate on the 1,453 without them.

Even in that best-performing group, it still got some wrong.

What it doesn't show

Six volunteers is a small base. CryptoSlate wrote that estimating how often unsigned reports reveal their writers across confidential peer review would require a larger, more varied sample. It added that an editor or author given a name by Claude would still need independent evidence to establish who wrote an unsigned review.

The paper is a preprint and hasn't been peer reviewed. LeoDex News didn't review it independently. The researchers said they plan to add results from a substantially larger set of anonymous peer reviews in an update to the preprint.

Frequently asked questions

How did Claude identify the authors of anonymous peer reviews?

The model saw one report at a time, without web access, tools, a list of possible authors or other writing samples, according to the preprint reported by CryptoSlate. It named the writer of 56% of 398 unsigned reviews and meta-reviews written by six volunteers.

How did self-citations relate to Claude's accuracy in the study?

On public reviews, Claude was 80.2% accurate on 106 reviews containing self-citations and 17.1% accurate on 1,453 without them. It still made mistakes even in that best-performing group, the preprint reported.