Performance Evaluation Criteria for High Dimensional Classification Problems

Abstract

In high dimensional small sample (HDSS) classification problems, the issue of relevant and irrelevant data, the curse of singularity, and dimensionality persist. The presence of irrelevant variables has generated different problems in the classification domain such as computational time, misclassification rate, and performance evaluation criteria. The covariance-dependent classification methods such as the Fisher linear classification method (FLCM) are redundant as such, the independent classification rule (ICR) was coined to solve these problems. Yet, the training and validation of the ICR learned model depends on the relevant and irrelevant data in the variables. To overcome these problems, we applied the principal component analysis (PCA) for dimension reduction on the FLCM (PCA- FLCM), the ICR method (PCA-ICR), F-weighted PCA called W-PCA, and the proposed benchmark extraction method (BEM) to tackle the above mentioned HDSS classification problems. For this study, we investigated the number and percentage of relevant variables selected, computational time, and the probability of correct classification (PCC). To evaluate the performance of these methods, we applied the performance evaluation criteria (PEC) to analyse the probability of correct classification for HDSS classification problems based on the axioms of the probability concept. The results revealed that the W-PCA procedure is very sensitive to select the most vital few variables (Minimum number of vital variables) followed by the BEM procedure. The W-PCA variants have the best computational time while the BEM has the overall best PCC for the data set investigated. The findings demonstrated that the BEM approach outperformed other methods in terms of probability of correct classification while the W-PCA has the best optimal variable search and selection capabilities than the other methods.