How Often Do AI Detectors Flag Human Writing? We Published Our False-Positive Rate
We tested our detector on 715 passages and published the result. A 4.9% false-positive rate on human writing, 2.9% on non-native English, and why we capped false positives instead of chasing a higher catch rate.
Most AI detectors will tell you how much AI they catch. Almost none will tell you how often they are wrong about a real person. That second number is the one that matters, because a false positive is not a statistic. It is a student accused of cheating for work they wrote themselves.
So we are publishing ours.
The real harm is the false positive
When an AI detector flags human writing, someone can lose marks, face a misconduct hearing, or be told to rewrite work that was already their own. The cost of a false accusation is far higher than the cost of missing one piece of AI text. Yet the industry markets on catch rate, not on how rarely it gets an innocent person wrong.
There is a specific fairness problem underneath this. A 2023 Stanford study (Liang et al.) found that GPT detectors systematically flagged writing by non-native English speakers as AI-generated, because non-native and formal academic writing shares surface patterns with model output. The people most likely to be wrongly accused are the ones least equipped to contest it.
What we measured
We ran our detector across 715 passages: 391 AI-generated (from Claude, GPT and Gemini), 324 written by humans, of which 35 were written by non-native English speakers. We used five-fold stratified cross-validation, so every score is out-of-sample. Here is the shipped model.
Two honest things to sit with. First, a 4.9% false-positive rate means roughly one human passage in twenty gets flagged. That is not zero, and no one should treat a single flag as proof. Second, we catch about 55% of AI text, which means we miss close to half. Detection is genuinely hard, and any tool claiming near-certainty is overselling.
The number we are proudest of is the non-native one: 2.9%, lower than our overall human rate, not higher. On the cohort the Stanford research showed most at risk, we do not over-flag.
The trade we made on purpose
While tuning the model, we built a version that caught 62.4% of AI text, nearly eight points more than what we shipped. We did not ship it. That version flagged 14.3% of non-native English writers as AI, a fivefold jump in exactly the group least able to defend themselves.
We chose the model that catches less AI and protects non-native writers. Given the choice between catching more cheats and falsely accusing more real students, we think the answer is obvious, and we would rather be honest about making it than quietly optimise for a bigger headline number.
What this means if you are checking work
- Treat any result as a screening signal, never as proof. A 4.9% false-positive rate is low, but on a class of 200 essays it still means flags you must not act on blindly.
- Read the flagged passages, not just the score. The value is in seeing which sentences triggered the model, then judging them in context.
- Be especially careful with non-native English writers. Even a fair detector is a starting point for a conversation, not evidence for a hearing.
Methodology
The 391 AI passages span Claude (Haiku and Sonnet), GPT and Gemini, including passages post-processed to read as tired student writing. Human passages include formal academic prose and the non-native English cohort. Scores come from a stylometric logistic-regression model with a fingerprint booster, cross-validated on a fixed seed, with verdict bands published in our methodology notes. We report out-of-sample numbers only, because in-sample accuracy tells you nothing about how a detector behaves on writing it has never seen.
We will update these figures as the model changes. If you want to check a piece of writing against this detector, you can run three free scans a day at isitai.co.uk with no account.