Cohen's kappa
Cohen's kappa coefficient (κ, lowercase Greek kappa) is a statistic that is used to measure interrater reliability (and also intrarater reliability) for qualitative (categorical) items.^{[1]} It is generally thought to be a more robust measure than simple percent agreement calculation, as κ takes into account the possibility of the agreement occurring by chance. There is controversy surrounding Cohen's kappa due to the difficulty in interpreting indices of agreement. Some researchers have suggested that it is conceptually simpler to evaluate disagreement between items.^{[2]}
History
The first mention of a kappalike statistic is attributed to Galton in 1892.^{[3]}^{[4]}
The seminal paper introducing kappa as a new technique was published by Jacob Cohen in the journal Educational and Psychological Measurement in 1960.^{[5]}
Definition
Cohen's kappa measures the agreement between two raters who each classify N items into C mutually exclusive categories. The definition of [math]\displaystyle{ \kappa }[/math] is
 [math]\displaystyle{ \kappa \equiv \frac{p_o  p_e}{1  p_e} = 1 \frac{1  p_o}{1  p_e}, }[/math]
where p_{o} is the relative observed agreement among raters, and p_{e} is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category. If the raters are in complete agreement then [math]\displaystyle{ \kappa=1 }[/math]. If there is no agreement among the raters other than what would be expected by chance (as given by p_{e}), [math]\displaystyle{ \kappa=0 }[/math]. It is possible for the statistic to be negative,^{[6]} which can occur by chance if there is no relationship between the ratings of the two raters, or it may reflect a real tendency of the raters to give differing ratings.
For k categories, N observations to categorize and [math]\displaystyle{ n_{ki} }[/math] the number of times rater i predicted category k:
 [math]\displaystyle{ p_e = \frac{1}{N^2} \sum_k n_{k1}n_{k2} }[/math]
This is derived from the following construction:
 [math]\displaystyle{ p_e = \sum_k \widehat{p_{k12}} \overset{\text{ind.}}{=} \sum_k \widehat{p_{k1}}\widehat{p_{k2}} = \sum_k \frac{n_{k1}}{N}\frac{n_{k2}}{N} = \frac{1}{N^2} \sum_k n_{k1}n_{k2} }[/math]
Where [math]\displaystyle{ \widehat{p_{k12}} }[/math] is the estimated probability that both rater 1 and rater 2 will classify the same item as k, while [math]\displaystyle{ \widehat{p_{k1}} }[/math] is the estimated probability that rater 1 will classify an item as k (and similarly for rater 2). The relation [math]\displaystyle{ \widehat{p_k} = \sum_k \widehat{p_{k1}}\widehat{p_{k2}} }[/math] is based on using the assumption that the rating of the two raters are independent. The term [math]\displaystyle{ \widehat{p_{k1}} }[/math] is estimated by using the number of items classified as k by rater 1 ([math]\displaystyle{ n_{k1} }[/math]) divided by the total items to classify ([math]\displaystyle{ N }[/math]): [math]\displaystyle{ \widehat{p_{k1}}= {n_{k1} \over N} }[/math] (and similarly for rater 2).
Binary classification confusion matrix
In the traditional 2 × 2 confusion matrix employed in machine learning and statistics to evaluate binary classifications, the Cohen's Kappa formula can be written as:^{[7]}
 [math]\displaystyle{ \kappa = \frac{2 \times (TP \times TN  FN \times FP)}{(TP + FP) \times (FP + TN) + (TP + FN) \times (FN + TN)} }[/math]
where TP are the true positives, FP are the false positives, TN are the true negatives, and FN are the false negatives. In this case, Cohen's Kappa is equivalent to the Heidke skill score known in Meteorology.^{[8]} The measure was first introduced by Myrick Haskell Doolittle in 1888.^{[9]}
Examples
Simple example
Suppose that you were analyzing data related to a group of 50 people applying for a grant. Each grant proposal was read by two readers and each reader either said "Yes" or "No" to the proposal. Suppose the disagreement count data were as follows, where A and B are readers, data on the main diagonal of the matrix (a and d) count the number of agreements and offdiagonal data (b and c) count the number of disagreements:
B A

Yes  No 

Yes  a  b 
No  c  d 
e.g.
B A

Yes  No 

Yes  20  5 
No  10  15 
The observed proportionate agreement is:
 [math]\displaystyle{ p_o = \frac{a+d}{a+b+c+d} = \frac{20+15}{50} = 0.7 }[/math]
To calculate p_{e} (the probability of random agreement) we note that:
 Reader A said "Yes" to 25 applicants and "No" to 25 applicants. Thus reader A said "Yes" 50% of the time.
 Reader B said "Yes" to 30 applicants and "No" to 20 applicants. Thus reader B said "Yes" 60% of the time.
So the expected probability that both would say yes at random is:
 [math]\displaystyle{ p_\text{Yes} = \frac{a+b}{a+b+c+d} \cdot \frac{a+c}{a+b+c+d} = 0.5 \times 0.6 = 0.3 }[/math]
Similarly:
 [math]\displaystyle{ p_\text{No} = \frac{c+d}{a+b+c+d} \cdot \frac{b+d}{a+b+c+d} = 0.5 \times 0.4 = 0.2 }[/math]
Overall random agreement probability is the probability that they agreed on either Yes or No, i.e.:
 [math]\displaystyle{ p_e = p_\text{Yes} + p_\text{No} = 0.3 + 0.2 = 0.5 }[/math]
So now applying our formula for Cohen's Kappa we get:
 [math]\displaystyle{ \kappa = \frac{p_o  p_e}{1  p_e} = \frac{0.70.5}{10.5} = 0.4 }[/math]
Same percentages but different numbers
A case sometimes considered to be a problem with Cohen's Kappa occurs when comparing the Kappa calculated for two pairs of raters with the two raters in each pair having the same percentage agreement but one pair give a similar number of ratings in each class while the other pair give a very different number of ratings in each class.^{[10]} (In the cases below, notice B has 70 yeses and 30 nos, in the first case, but those numbers are reversed in the second.) For instance, in the following two cases there is equal agreement between A and B (60 out of 100 in both cases) in terms of agreement in each class, so we would expect the relative values of Cohen's Kappa to reflect this. However, calculating Cohen's Kappa for each:
B A

Yes  No 

Yes  45  15 
No  25  15 
 [math]\displaystyle{ \kappa = \frac{0.600.54}{10.54} = 0.1304 }[/math]
B A

Yes  No 

Yes  25  35 
No  5  35 
 [math]\displaystyle{ \kappa = \frac{0.600.46}{10.46} = 0.2593 }[/math]
we find that it shows greater similarity between A and B in the second case, compared to the first. This is because while the percentage agreement is the same, the percentage agreement that would occur 'by chance' is significantly higher in the first case (0.54 compared to 0.46).
Properties
Hypothesis testing and confidence interval
Pvalue for kappa is rarely reported, probably because even relatively low values of kappa can nonetheless be significantly different from zero but not of sufficient magnitude to satisfy investigators.^{[11]}^{:66} Still, its standard error has been described^{[12]} and is computed by various computer programs.^{[13]}
Confidence intervals for Kappa may be constructed, for the expected Kappa values if we had infinite number of items checked, using the following formula:^{[1]}
 [math]\displaystyle{ CI: \kappa \pm Z_{1\alpha/2} SE_{\kappa} }[/math]
Where [math]\displaystyle{ Z_{1\alpha/2} = 1.960 }[/math] is the standard normal percentile when [math]\displaystyle{ \alpha = 5\% }[/math], and [math]\displaystyle{ SE_{\kappa} = \sqrt{{p_o (1 p_o)} \over { N (1 p_e) ^2 }} }[/math]
This is calculated by ignoring that p_{e} is estimated from the data, and by treating p_{o} as an estimated probability of a binomial distribution while using asymptotic normality (i.e.: assuming that the number of items is large and that p_{o} is not close to either 0 or 1). [math]\displaystyle{ SE_{\kappa} }[/math] (and the CI in general) may also be estimated using bootstrap methods.
Interpreting magnitude
If statistical significance is not a useful guide, what magnitude of kappa reflects adequate agreement? Guidelines would be helpful, but factors other than agreement can influence its magnitude, which makes interpretation of a given magnitude problematic. As Sim and Wright noted, two important factors are prevalence (are the codes equiprobable or do their probabilities vary) and bias (are the marginal probabilities for the two observers similar or different). Other things being equal, kappas are higher when codes are equiprobable. On the other hand, Kappas are higher when codes are distributed asymmetrically by the two observers. In contrast to probability variations, the effect of bias is greater when Kappa is small than when it is large.^{[14]}^{:261–262}
Another factor is the number of codes. As number of codes increases, kappas become higher. Based on a simulation study, Bakeman and colleagues concluded that for fallible observers, values for kappa were lower when codes were fewer. And, in agreement with Sim & Wrights's statement concerning prevalence, kappas were higher when codes were roughly equiprobable. Thus Bakeman et al. concluded that "no one value of kappa can be regarded as universally acceptable."^{[15]}^{:357} They also provide a computer program that lets users compute values for kappa specifying number of codes, their probability, and observer accuracy. For example, given equiprobable codes and observers who are 85% accurate, value of kappa are 0.49, 0.60, 0.66, and 0.69 when number of codes is 2, 3, 5, and 10, respectively.
Nonetheless, magnitude guidelines have appeared in the literature. Perhaps the first was Landis and Koch,^{[16]} who characterized values < 0 as indicating no agreement and 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect agreement. This set of guidelines is however by no means universally accepted; Landis and Koch supplied no evidence to support it, basing it instead on personal opinion. It has been noted that these guidelines may be more harmful than helpful.^{[17]} Fleiss's^{[18]}^{:218} equally arbitrary guidelines characterize kappas over 0.75 as excellent, 0.40 to 0.75 as fair to good, and below 0.40 as poor.
Kappa maximum
Kappa assumes its theoretical maximum value of 1 only when both observers distribute codes the same, that is, when corresponding row and column sums are identical. Anything less is less than perfect agreement. Still, the maximum value kappa could achieve given unequal distributions helps interpret the value of kappa actually obtained. The equation for κ maximum is:^{[19]}
 [math]\displaystyle{ \kappa_{\max} =\frac{P_{\max}  P_{\exp}}{1P_{\exp}} }[/math]
where [math]\displaystyle{ P_{\exp} = \sum_{i=1}^k P_{i+}P_{+i} }[/math], as usual, [math]\displaystyle{ P_{\max} = \sum_{i=1}^k \min(P_{i+},P_{+i}) }[/math],
k = number of codes, [math]\displaystyle{ P_{i+} }[/math] are the row probabilities, and [math]\displaystyle{ P_{+i} }[/math] are the column probabilities.
Limitations
Kappa is an index that considers observed agreement with respect to a baseline agreement. However, investigators must consider carefully whether Kappa's baseline agreement is relevant for the particular research question. Kappa's baseline is frequently described as the agreement due to chance, which is only partially correct. Kappa's baseline agreement is the agreement that would be expected due to random allocation, given the quantities specified by the marginal totals of square contingency table. Thus, κ = 0 when the observed allocation is apparently random, regardless of the quantity disagreement as constrained by the marginal totals. However, for many applications, investigators should be more interested in the quantity disagreement in the marginal totals than in the allocation disagreement as described by the additional information on the diagonal of the square contingency table. Thus for many applications, Kappa's baseline is more distracting than enlightening. Consider the following example:
Reference  

G  R  
Comparison  G  1  14 
R  0  1 
The disagreement proportion is 14/16 or 0.875. The disagreement is due to quantity because allocation is optimal. κ is 0.01.
Reference  

G  R  
Comparison  G  0  1 
R  1  14 
The disagreement proportion is 2/16 or 0.125. The disagreement is due to allocation because quantities are identical. Kappa is −0.07.
Here, reporting quantity and allocation disagreement is informative while Kappa obscures information. Furthermore, Kappa introduces some challenges in calculation and interpretation because Kappa is a ratio. It is possible for Kappa's ratio to return an undefined value due to zero in the denominator. Furthermore, a ratio does not reveal its numerator nor its denominator. It is more informative for researchers to report disagreement in two components, quantity and allocation. These two components describe the relationship between the categories more clearly than a single summary statistic. When predictive accuracy is the goal, researchers can more easily begin to think about ways to improve a prediction by using two components of quantity and allocation, rather than one ratio of Kappa.^{[2]}
Some researchers have expressed concern over κ's tendency to take the observed categories' frequencies as givens, which can make it unreliable for measuring agreement in situations such as the diagnosis of rare diseases. In these situations, κ tends to underestimate the agreement on the rare category.^{[20]} For this reason, κ is considered an overly conservative measure of agreement.^{[21]} Others^{[22]}^{[citation needed]} contest the assertion that kappa "takes into account" chance agreement. To do this effectively would require an explicit model of how chance affects rater decisions. The socalled chance adjustment of kappa statistics supposes that, when not completely certain, raters simply guess—a very unrealistic scenario. Moreover, some works^{[23]} have shown how kappa statistics can lead to a wrong conclusion for unbalanced data.
Related statistics
Scott's Pi
A similar statistic, called pi, was proposed by Scott (1955). Cohen's kappa and Scott's pi differ in terms of how p_{e} is calculated.
Fleiss' kappa
Note that Cohen's kappa measures agreement between two raters only. For a similar measure of agreement (Fleiss' kappa) used when there are more than two raters, see Fleiss (1971). The Fleiss kappa, however, is a multirater generalization of Scott's pi statistic, not Cohen's kappa. Kappa is also used to compare performance in machine learning, but the directional version known as Informedness or Youden's J statistic is argued to be more appropriate for supervised learning.^{[24]}
Weighted kappa
The weighted kappa allows disagreements to be weighted differently^{[25]} and is especially useful when codes are ordered.^{[11]}^{:66} Three matrices are involved, the matrix of observed scores, the matrix of expected scores based on chance agreement, and the weight matrix. Weight matrix cells located on the diagonal (upperleft to bottomright) represent agreement and thus contain zeros. Offdiagonal cells contain weights indicating the seriousness of that disagreement. Often, cells one off the diagonal are weighted 1, those two off 2, etc.
The equation for weighted κ is:
 [math]\displaystyle{ \kappa = 1  \frac{\sum_{i=1}^{k} \sum_{j=1}^{k} w_{ij} x_{ij}} {\sum_{i=1}^{k} \sum_{j=1}^{k} w_{ij} m_{ij}} }[/math]
where k=number of codes and [math]\displaystyle{ w_{ij} }[/math], [math]\displaystyle{ x_{ij} }[/math], and [math]\displaystyle{ m_{ij} }[/math] are elements in the weight, observed, and expected matrices, respectively. When diagonal cells contain weights of 0 and all offdiagonal cells weights of 1, this formula produces the same value of kappa as the calculation given above.
See also
Further reading
 Banerjee, M.; Capozzoli, Michelle; McSweeney, Laura; Sinha, Debajyoti (1999). "Beyond Kappa: A Review of Interrater Agreement Measures". The Canadian Journal of Statistics 27 (1): 3–23. doi:10.2307/3315487. https://digitalcommons.fairfield.edu/mathandcomputersciencefacultypubs/60.
 Chicco, D.; Warrens, M.J.; Jurman, G. (2021). "The Matthews correlation coefficient (MCC) is more informative than Cohen's Kappa and Brier score in binary classification assessment". IEEE Access 9: 78368–81. doi:10.1109/access.2021.3084050.
 Cohen, Jacob (1960). "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20 (1): 37–46. doi:10.1177/001316446002000104.
 Cohen, J. (1968). "Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit". Psychological Bulletin 70 (4): 213–220. doi:10.1037/h0026256. PMID 19673146.
 Fleiss, J.L.; Cohen, J. (1973). "The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability". Educational and Psychological Measurement 33 (3): 613–9. doi:10.1177/001316447303300309.
 Sim, J.; Wright, C.C. (2005). "The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements". Physical Therapy 85 (3): 257–268. doi:10.1093/ptj/85.3.257. PMID 15733050.
 Warrens, J. (2011). "Cohen's kappa is a weighted average". Statistical Methodology 8 (6): 473–484. doi:10.1016/j.stamet.2011.06.002.
External links
 Kappa, its meaning, problems, and several alternatives (Link is dead as of 20221216)
 Kappa Statistics: Pros and Cons
 Software implementations
 Windows program "ComKappa" for kappa, weighted kappa, and kappa maximum (Error "Access Denied (Error Code 1020)" as of 20221216)
References
 ↑ ^{1.0} ^{1.1} McHugh, Mary L. (2012). "Interrater reliability: The kappa statistic". Biochemia Medica 22 (3): 276–282. doi:10.11613/bm.2012.031. PMID 23092060.
 ↑ ^{2.0} ^{2.1} Pontius, Robert; Millones, Marco (2011). "Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment". International Journal of Remote Sensing 32 (15): 4407–4429. doi:10.1080/01431161.2011.552923. Bibcode: 2011IJRS...32.4407P. http://www.clarku.edu/~rpontius/.
 ↑ Galton, F. (1892) Finger Prints Macmillan, London.
 ↑ Smeeton, N.C. (1985). "Early History of the Kappa Statistic". Biometrics 41 (3): 795.
 ↑ Cohen, Jacob (1960). "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20 (1): 37–46. doi:10.1177/001316446002000104.
 ↑ Sim, Julius; Wright, Chris C. (2005). "The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements". Physical Therapy 85 (3): 257–268. doi:10.1093/ptj/85.3.257. ISSN 15386724. PMID 15733050.
 ↑ Chicco D.; Warrens M.J.; Jurman G. (June 2021). "The Matthews correlation coefficient (MCC) is more informative than Cohen's Kappa and Brier score in binary classification assessment". IEEE Access 9: 78368  78381. doi:10.1109/ACCESS.2021.3084050.
 ↑ Heidke, P. (19261201). "Berechnung Des Erfolges Und Der Güte Der Windstärkevorhersagen Im Sturmwarnungsdienst". Geografiska Annaler 8 (4): 301–349. doi:10.1080/20014422.1926.11881138. ISSN 20014422.
 ↑ Philosophical Society of Washington (Washington, D.C.) (1887). Bulletin of the Philosophical Society of Washington.. 10. Washington, D.C.: Published by the cooperation of the Smithsonian Institution. p. 83. https://www.biodiversitylibrary.org/page/55377146.
 ↑ Kilem Gwet (May 2002). "InterRater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity". Statistical Methods for InterRater Reliability Assessment 2: 1–10. http://agreestat.com/research_papers/inter_rater_reliability_dependency.pdf. Retrieved 20110202.
 ↑ ^{11.0} ^{11.1} Bakeman, R.; Gottman, J.M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge, UK: Cambridge University Press. ISBN 9780521275934.
 ↑ Fleiss, J.L.; Cohen, J.; Everitt, B.S. (1969). "Large sample standard errors of kappa and weighted kappa". Psychological Bulletin 72 (5): 323–327. doi:10.1037/h0028106.
 ↑ Robinson, B.F; Bakeman, R. (1998). "ComKappa: A Windows 95 program for calculating kappa and related statistics". Behavior Research Methods, Instruments, and Computers 30 (4): 731–732. doi:10.3758/BF03209495.
 ↑ Sim, J; Wright, C. C (2005). "The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements". Physical Therapy 85 (3): 257–268. doi:10.1093/ptj/85.3.257. PMID 15733050.
 ↑ Bakeman, R.; Quera, V.; McArthur, D.; Robinson, B. F. (1997). "Detecting sequential patterns and determining their reliability with fallible observers". Psychological Methods 2 (4): 357–370. doi:10.1037/1082989X.2.4.357.
 ↑ Landis, J.R.; Koch, G.G. (1977). "The measurement of observer agreement for categorical data". Biometrics 33 (1): 159–174. doi:10.2307/2529310. PMID 843571. https://repositorio.unal.edu.co/handle/unal/80119.
 ↑ Gwet, K. (2010). "Handbook of InterRater Reliability (Second Edition)" ISBN:9780970806222^{[page needed]}
 ↑ Fleiss, J.L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley. ISBN 9780471263708.
 ↑ Umesh, U. N.; Peterson, R.A.; Sauber M. H. (1989). "Interjudge agreement and the maximum value of kappa.". Educational and Psychological Measurement 49 (4): 835–850. doi:10.1177/001316448904900407.
 ↑ Viera, Anthony J.; Garrett, Joanne M. (2005). "Understanding interobserver agreement: the kappa statistic". Family Medicine 37 (5): 360–363. PMID 15883903.
 ↑ Strijbos, J.; Martens, R.; Prins, F.; Jochems, W. (2006). "Content analysis: What are they talking about?". Computers & Education 46: 29–48. doi:10.1016/j.compedu.2005.04.002.
 ↑ Uebersax, JS. (1987). "Diversity of decisionmaking models and the measurement of interrater agreement". Psychological Bulletin 101: 140–146. doi:10.1037/00332909.101.1.140. http://www.namic.org/Wiki/images/d/df/Kapp_and_decision_making_models.pdf. Retrieved 20101016.
 ↑ Delgado, Rosario; Tibau, XavierAndoni (20190926). "Why Cohen's Kappa should be avoided as performance measure in classification" (in en). PLOS ONE 14 (9): e0222916. doi:10.1371/journal.pone.0222916. ISSN 19326203. PMID 31557204. Bibcode: 2019PLoSO..1422916D.
 ↑ Powers, David M. W. (2012). "The Problem with Kappa". http://dl.dropbox.com/u/27743223/201209eacl2012Kappa.pdf. Retrieved 20120720.
 ↑ Cohen, J. (1968). "Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit". Psychological Bulletin 70 (4): 213–220. doi:10.1037/h0026256. PMID 19673146.
Original source: https://en.wikipedia.org/wiki/Cohen's kappa.
Read more 