LDA classifier (and potentially others) are scale dependent
The LDA classifier (CAlgorithmClassifierLDA.cpp) has a wired tolerance on eigenvalues (1e-10) used to compute the pseudo-inverse of the global covariance matrix. This is not scale invariant so prone to fail. Here is a real world example. Consider the computed spatial filter spatial.xml. Given the range of the values, I had the silly idea of normalizing the filters to have norm 1. After all, these are just generalized eigenvalues, so the norm of the vector should not matter. Suddenly, after doing this a subsequent LDA changed drastically in performance from:
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cross-validation test accuracy is 82.8767% (sigma = 3.72634%)
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cls vs cls 1 2
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 1: 33.3 66.7 %, 198 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 2: 4.7 95.3 %, 789 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Training set accuracy is 90.5775% (optimistic)
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cls vs cls 1 2
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 1: 56.1 43.9 %, 198 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 2: 0.8 99.2 %, 789 examples
to:
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cross-validation test accuracy is 79.9392% (sigma = 0.600794%)
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cls vs cls 1 2
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 1: 0.0 100.0 %, 198 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 2: 0.0 100.0 %, 789 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Training set accuracy is 79.9392% (optimistic)
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Cls vs cls 1 2
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 1: 0.0 100.0 %, 198 examples
[ INF ] At time 331.844 sec <Box algorithm::(0x00005792, 0x000029eb) aka Classifier trainer> Target 2: 0.0 100.0 %, 789 examples
Experimenting with simpler normalizations showed that dividing the spatial filter values by a constant quantity (from 1e5 to 4e6) changed continuously the classification results from the first to the second one.
This is due to the absolute tolerance mentioned above because the eigenvalues are scaled with the spatial filter scale. Hence an absolute tolerance is wrong and fragile.