We sampled 247 essays already graded by certified IELTS examiners and re-graded them with our Claude-based rubric pipeline. The setup, agreement, and the cases where we still disagree.
Method
Each essay was scored by one human rater on the four official IELTS descriptors (Task Response, Coherence & Cohesion, Lexical Resource, Grammatical Range & Accuracy). Bands ranged from 4.0 to 9.0. Our AI was given the same rubric verbatim, no fine-tuning, just careful prompting.
Agreement
| Within 0.0 bands | 61% |
|---|---|
| Within 0.5 bands | 89% |
| Within 1.0 band | 98% |
That 89% is the figure most candidates care about. For context, inter-rater agreement between two human IELTS examiners on the same essay is around 85% within 0.5 bands. We're slightly more consistent — but only because the AI doesn't get tired at essay #142.
Where we still disagree
The AI tends to be 0.5 bands stricter on Grammatical Range when the essay uses complex sentences with one or two errors. The human rater often gives credit for ambition; the AI penalises the errors. We've shipped this as a feature, not a bug — strictness here pushes candidates to fix the errors, which is what they came for.
What we changed
We added an explicit "intent vs. execution" line to the prompt: "Credit a candidate's clear attempt at complex grammar even when the execution has one error per attempt." Re-running the sample, agreement-within-0.5 jumped from 84% to 89%.
If you want to see your essay graded with the same pipeline, the 15-minute diagnostic includes one summary task that uses it.
Writes about scoring, calibration, and what actually moves a candidate's band. Meet the team →
