Cohen's Kappa
Cohen’s Kappa is pretty interesting - it’s a way to measure how much two independent classifiers agree, beyond what you’d expect by chance. It’s written out as:
\[\kappa = \frac{p_o - p_e}{1 - p_e}\]where:
- \(p_o\) is the observed agreement: the fraction of items both classifiers labeled the same.
- \(p_e\) is the expected agreement by chance: \(p_e = \sum_k p_{A,k} \cdot p_{B,k}\), where \(p_{A,k}\) and \(p_{B,k}\) are the fractions of items that A and B each assigned to category \(k\).
So, effectively, it measures how much the two classifiers agree after removing the agreement you’d expect by chance. A \(\kappa\) of 1 is perfect agreement, and 0 is no better than chance.
The algorithm is basically:
- Have the two parties, A and B, perform the classification.
- Figure out the expected agreement by chance. For each category \(k\), multiply the fraction of items A put in \(k\) by the fraction B put in \(k\), then sum across all categories to get \(p_e\).
- Calculate the observed agreement \(p_o\): the fraction of items where A and B chose the same category.
- Subtract the expected agreement from the observed agreement, and divide by the most agreement possible beyond chance: \(\kappa = (p_o - p_e) / (1 - p_e)\).