Where Z 1 − α / 2 = 1.965 {displaystyle Z_{1-alpha /2}=1.965} is the default normal percentile if α = 5% {displaystyle alpha =5%} and S E κ = p o ( 1 − p o ) N ( 1 − p e ) 2 {displaystyle SE_{kappa }={sqrt {{p_{o}(1-p_{o})} over {N(1-p_{e})^{2}}}}} The prevalence of the code does not play a role with the increase in the code number. If the number of codes is equal to or greater than 6, the variability in prevalence does not matter and the standard deviation of kappa values obtained from observers with accuracies of 0.80, 0.85, 0.90 and 0.85 is less than 0.01. Factors that influence kappa values include observer accuracy and number of codes, as well as individual population prevalence and code observer bias. Kappa can only be equal to 1 if observers distribute the codes evenly. There is no value of kappa that can be considered universally acceptable; it depends on the accuracy of the observers and the number of codes. The calculation of Cohen`s kappa can be done according to the following formula: Cohen`s kappa measures the agreement between two evaluators, each of whom classifies N elements in mutually exclusive categories C. The definition of κ {textstyle kappa } is as follows: Although kappa is probably the most commonly used measure of agreement, it has been the subject of criticism. One such criticism is that kappa is a measure of exact agreement and treats approximate agreements in the same way as extreme disagreements. But with certain types of data, a “near miss” may be preferable to a “vast absence.” Although this is usually not the case if the categories that are noted are really nominal (as in our example of verbs vs. Nonverbia), the idea of a “near-accident” makes more sense for ordinal categories. Also note that for a number of observations, the more categories there are, the smaller the kappa. Even with our simple percentage match, we`ve seen that folding adjectives and nouns into a single category increases the “success rate.” Weighted kappas are a way around the “near miss” problem.

Essentially, the weighting system can distinguish between relatively proximal and relatively distal ordinal categories. Disagreements, when different evaluators choose categories that are not identical but proximal, contribute more positively to the degree of agreement than because of disagreements involving very different classifications by evaluators. Nevertheless, guidelines on magnitude have appeared in the literature. Perhaps the first Landis and Koch[13] who characterized the values < 0 as no match and 0–0.20 as weak, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect match. However, these guidelines are by no means generally accepted; Landis and Koch provided no evidence of this, but instead relied on personal opinions. It has been found that these guidelines can be harmful rather than useful. [14] Diligence[15]:218 Equally arbitrary guidelines characterize Kappas above 0.75 as excellent, 0.40 to 0.75 as just to good, and below 0.40 as bad. Again, this is just a fair level agreement. Note that although pathologists agree in 70% of cases, they are expected to have almost as high levels of agreement (62%) by chance only. Kappa Value Interpretation Landis & Koch (1977):<0 No Match0 — .20 Slight.21 — .40 Fair.41 — .60 Moderate.61 — .80 Substantial.81–1.0 Perfect, where po is the observed relative match between the evaluators (identical to the accuracy) and pe is the hypothetical probability of random match, using the observed data to calculate the probabilities of each observer who sees each category at random. If the evaluators completely agree, then κ = 1 {textstyle kappa =1}.

If there is no correspondence between the evaluators, except for what would be expected by chance (as given by pe), κ = 0 {textstyle kappa =0}. It is possible that the statistics are negative[6], which means that there is no effective agreement between the two evaluators or that the match is worse than random. Weighted kappa allows for different weighting of disagreements[21] and is particularly useful when ordering codes. [8]:66 Three matrices are involved, the matrix of observed scores, the matrix of expected scores based on random matching and the matrix of weights. The cells of the weight matrix on the diagonal (top left to bottom right) represent a match and therefore contain zeros. .