Original article
High agreement but low kappa: II. Resolving the paradoxes

https://doi.org/10.1016/0895-4356(90)90159-MGet rights and content

Abstract

An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers. Among the available other omnibus indexes, none offers a satisfactory solution for the paradoxes that occur with p0 and κ. The problem can be avoided only by using ppos and pneg as two separate indexes of proportionate agreement in the observers' positive and negative decisions. These two indexes, which are analogous to sensitivity and specificity for concordance in a diagnostic marker test, create the paradoxes formed when the chance correction in κ is calculated as a product of the increment in the two indexes and the increment in marginal totals. If only a single omnibus index is used to compared different performances in observer variability, the paradoxes of κ are desirable since they appropriately “penalize” inequalities in ppos and ppeg. For better understanding of results and for planning improvements in the observers' performance, however, the omnibus value of κ should always be accompanied by separate individual values of ppos and pneg.

References (13)

There are more references available in the full text version of this article.

Cited by (1407)

  • Sentiment analysis of the Spanish financial stability Report

    2024, International Review of Economics and Finance
View all citing articles on Scopus

Supported in part by a Grant from the Andrew W. Mellon Foundation and by Veterans Administration Funds.

Senior Research Psychologist and Biostatistician, Veterans Administration Medical Center, West Haven, Connecticut; and Senior Research Scientist, Department of Psychiatry and Child Study Center, Yale University School of Medicine, New Haven, Connecticut.

View full text