About the Cohen's Kappa Calculator
Cohen's Kappa Calculator measures how much two raters agree on a yes/no classification after subtracting out the agreement you'd expect from chance alone. It's the standard tool whenever two people independently code, diagnose, or classify the same set of items and you want one number describing how consistent their judgments really are.
How It Works
Enter the four counts from a 2x2 agreement table: both raters said yes, rater 1 said yes while rater 2 said no, rater 1 said no while rater 2 said yes, and both raters said no. The calculator works out each rater's own yes and no rates, uses those rates to estimate how often the two raters would agree purely by chance, and compares that expected agreement to how often they actually agreed.
Formula & Methodology
To work this by hand, first find each rater's own marginal rate of saying yes and no from the row and column totals of the 2x2 table, then multiply the matching yes-rates and no-rates together and add them to get the expected agreement under independence. The numerator, observed minus expected agreement, is the improvement over chance; dividing by 1 minus expected agreement rescales that improvement against the maximum possible improvement, which is what keeps kappa bounded at 1 for perfect agreement.
Examples
Medical Diagnosis Agreement
Out of 100 patient cases, two physicians both say 'yes' on 25, both say 'no' on 62, and disagree on the remaining 13. Observed agreement is 87%, expected chance agreement works out to about 56.8%, giving a kappa of roughly 0.70, in the 'Substantial' band.
Support Ticket Tagging
Out of 100 tickets, two reps both tag 'urgent' on 15, both tag 'not urgent' on 50, and disagree on 35. Despite 65% raw agreement, expected chance agreement is about 56%, giving a kappa near 0.20, only 'Fair' agreement once chance is factored out.
Advantages
- Corrects for chance agreement, which raw percent agreement completely ignores, especially important when one category is much rarer than the other.
- Reduces a full 2x2 agreement table down to a single, comparable number that can be tracked over time or across rater pairs.
- Classifies the result into a labeled band, from poor to almost perfect, so the number is easier to interpret without memorizing a scale.
Common Mistakes
- Reporting raw percent agreement instead of kappa when one outcome is rare, which can make agreement look artificially high.
- Ignoring that kappa depends on how common each category is in the data, so the same two raters can produce a different kappa on a dataset with a different yes/no split.
- Treating a moderate or fair kappa as 'good enough' without considering what's actually being rated; some applications, like medical diagnosis, call for a stricter bar than others.
Edge Cases to Watch For
- If expected agreement works out to exactly 100%, kappa is undefined (it would require dividing by zero), and the calculator returns an error instead of a number.
- Kappa can come out negative when the raters actually agree less often than chance alone would predict; the calculator labels this 'Poor (worse than chance)'.
- The calculator sorts the result into six labeled bands, from 'Slight' up through 'Almost perfect,' based purely on the numeric kappa value.
Common Use Cases
- Researchers checking whether two independent coders classified qualitative or survey data consistently before treating the coding as reliable.
- Clinicians or diagnosticians comparing how consistently two practitioners reach the same yes/no diagnostic call.
- Content moderation or quality assurance teams measuring whether two reviewers apply the same labeling criteria.