Confusion Matrix
Suppose we have an algorithm that recognizes cats in photos. Our data set consists of 1,000 photos. 250 show a cat, and the rest doesn’t. Our algorithm classifies 300 photos as a picture of a cat. Of these photos, 190 are correctly classified.
What is the likelihood that a photo classified by our algorithm as a “cat photo” really shows a cat?
Example 1
We can create a matrix that lists all possible results for that purpose.
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/03/Confusion_Matrix_01.jpg)
Here are some hints on how to fill the matrix.
- As we said, we have 250 pictures with a cat. 190 of them are correctly recognized by our algorithm. This leaves 60 photos that were incorrectly classified as “No Cat”.
- We also know that our algorithm found 300 photos were it thought they show a cat. Since only 190 were correct, 110 are no cat in reality.
- In total, we have 1,000 photos, which means that 1,000-250 = 750 photos don’t show a cat in reality.
- With this information, we can calculate the number of photos that our algorithm correctly classified as “No Cat”.
To answer our question, what is the likelihood that a photo classified as “Cat” by our algorithm shows a cat in reality, we calculate:
190/300 = 63.33%
This confusion matrix helps us to assess the quality of an algorithm or test.
The fields in the matrix give us the following information:
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/03/Confusion_Matrix_02.jpg)
Example 2
Let’s consider another example.
Suppose our algorithm classifies every photo as “Cat”. What would the confusion matrix look like?
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/03/Confusion_Matrix_03.jpg)
What is the likelihood of getting a cat photo if the test shows “cat”?
1,000/100 = 0.1
Example 3
Now, our test randomly displays “Cat” or “No Cat” (50:50). The confusion matrix would look like:
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/03/Confusion_Matrix_04.jpg)
What is the likelihood of getting a cat photo if the test shows “Cat”?
50/500 = 0.1
As you see, two different tests bring the same likelihood. Hence, in order to assess the quality of the algorithm (or test), we need to calculate some indicators:
Precision or Positive Prediction Value (PPV)
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/02/game_theory_precision_PPR.jpg)
Negative Prediction Value (NPV)
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/02/game_theory_neg_pred_NPV.jpg)
Sensitivity (SE) or true positive rate (TPR)
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/02/game_theory_Sensitivity.jpg)
Specificity (SP) or true negative rate (TNR)
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/02/game_theory_specifidity_SP.jpg)
KPI | Example 1 | Example 2 | Example 3 |
Precision or Positive Prediction Value (PPV) | 0.633 | 0.1 | 0.1 |
Negative Prediction Value (NPV) | 0.914 | DIV/0! | 0.9 |
Sensitivity (SE) | 0.76 | 1 | 0.5 |
Specificity (SP) | 0.85 | 0 | 0.5 |
The PPV (Positive Prediction Value) and NPV (Negative Prediction Value) can only be applied to other groups (or data sets) if the pre-test probability (prevalence) is the same in both groups.
To make up for this limitation, one can calculate the likelihood ratio:
![](https://blockchain-academy.hs-mittweida.de/wp-content/uploads/2022/02/game_theory_likelihood_ratio_01.jpg)
The positive likelihood ratio tells us how the likelihood of a positive result changes if the test result is positive (or the algorithm says “Cat”).
The negative likelihood ratio tells us how the likelihood of a positive result changes if the test result is negative (or the algorithm says “No cat”).