TWO-WAY METRIC LEARNING WITH MAJORITY AND MINORITY SUBSETS FOR CLASSIFICATION OF LARGE EXTREMELY IMBALANCED FACE DATASET

(Received: 16-Jul.-2021, Revised: 12-Sep.-2021 , Accepted: 27-Sep.-2021)

Authors Ashu Kaushik, Seba Susan,

Keywords #Face recognition #Metric learning #VGG-Face #Deep learning #Imbalanced learning #Extremely imbalanced dataset

Abstract This paper proposes a new learning methodology involving deep features and two-way metric learning for large, extremely imbalanced face datasets where the number of minority classes and the imbalance ratio are both very high. The problem arises because the faces of some celebrities, being more popular, are readily available in social media and the internet, while the faces of some relatively lesser-known personalities are fewer in number. Resampling being impractical in this scenario, we propose metric learning as the tool for mitigating the class- imbalance problem prior to the classification stage. To reduce the computational overhead associated with metric learning, we separately conduct weakly supervized metric learning with majority and minority class subsets, a process that we call two-way metric learning. Transformation matrices learnt from the majority and minority subsets are used to transform the entire input space twice. The test sample in the transformed space is assigned the class of its nearest neighbor in the training set of the twice-transformed input space. Deep features derived from the state-of-the-art pre-trained deep network VGG-Face form the input space and the aggregate cosine similarity measure is used to find the closest neighbor in the training set of the twice-transformed input space. Experiments on the benchmark LFW face database having 1680 classes of celebrity faces prove that the proposed methodology is more effective than existing methods for the classification of large, extremely imbalanced face datasets. The classification accuracies of the minority classes are especially found to be boosted which is a rare accomplishment among existing methods for imbalanced learning in deep frameworks.

References

[1] C. Huang, Y. Li, C. C. Loy and X. Tang, "Deep Imbalanced Learning for Face Recognition and Attribute Prediction," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 11, pp. 2781-2794, 2019.

[2] S. Susan and Ashu Kaushik, "Weakly Supervized Metric Learning with Majority Classes for Large Imbalanced Image Dataset," Proceedings of the 4th International Conference on Big Data and Internet of Things, pp. 16-19, DOI: 10.1145/3421537.3421549, 2020.

[3] H. He and E. A. Garcia, "Learning from Imbalanced Data," IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263-1284, 2009.

[4] S. Xuan, G. Liu, Z. Li, L. Zheng, S. Wang and C. Jiang, "Random Forest for Credit Card Fraud Detection," Proc. of the 15th IEEE International Conference on Networking, Sensing and Control (ICNSC), pp. 1-6, Zhuhai, China, 2018.

[5] F. Zhang, G. Liu, Z. Li, C. Yan and C. Jiang, "GMM-based Undersampling and Its Application for Credit Card Fraud Detection," Proc. of the IEEE International Joint Conference on Neural Networks (IJCNN), pp. 1-8, Budapest, Hungary, 2019.

[6] S. Susan and A. Kumar, "DST-ML-EkNN: Data Space Transformation with Metric Learning and Elite K-nearest Neighbor Cluster Formation for Classification of Imbalanced Datasets," Proc. of Advances in Artificial Intelligence and Data Engineering, Part of the Advances in Intelligent Systems and Computing Book Series (AISC), vol. 1133, pp. 319-328, Springer, Singapore, 2021.

[7] H. Zhu, G. Liu, M. Zhou, Y. Xie, A. Abusorrah and Q. Kang, "Optimizing Weighted Extreme Learning Machines for Imbalanced Classification and Application to Credit Card Fraud Detection," Neurocomputing, vol. 407, pp. 50-62, DOI: 10.1016/j.neucom.2020.04.078, 2020.

[8] Z. Li, M. Huang, G. Liu and C. Jiang, "A Hybrid Method with Dynamic Weighted Entropy for Handling the Problem of Class Imbalance with Overlap in Credit Card Fraud Detection," Expert Systems with Applications, vol. 175, pp. 114750, DOI: 10.1016/j.eswa.2021.114750, 2021.

[9] S. Wang and X. Yao, "Multiclass Imbalance Problems: Analysis and Potential Solutions," IEEE Trans. on Systems, Man and Cybernetics, Part B (Cybernetics), vol. 42, no. 4, pp. 1119-1130, 2012.

[10] T. Hasanin, T. M. Khoshgoftaar, J. L. Leevy and R. A. Bauder, "Severely Imbalanced Big Data Challenges: Investigating Data Sampling Approaches," J. of Big Data, vol. 6, no. 1, pp. 1-25, 2019.

[11] B. Kulis, "Metric Learning: A Survey," Foundations and Trends in Machine Learning, vol. 5, no. 4, pp. 287-364, 2012.

[12] A. Krizhevsky, I. Sutskever and G. E. Hinton, "Imagenet Classification with Deep Convolutional Neural Networks," Advances in Neural Information Processing Systems, vol. 25, pp. 1097-1105, 2012.

[13] A. Al-Shannaq and L. Elrefaei, "Age Estimation Using Specific Domain Transfer Learning," Jordanian Journal of Computers and Information Technology (JJCIT), vol. 6, no. 2, pp. 122-139, 2020.

[14] J. M. Johnson and T. M. Khoshgoftaar, "Survey on Deep Learning with Class Imbalance," Journal of Big Data, vol. 6, no. 1, pp. 1-54, 2019. [15] R.-C. Chen and C.-Y. Liao, "Deep Learning to Predict User Rating in Imbalance Classification Data Incorporating Ensemble Methods," Proc. of the IEEE International Conference on Applied System Invention (ICASI), pp. 200-203, Chiba, Japan, 2018.

[16] N. Wang, X. Zhao, Y. Jiang and Y. Gao, "Iterative Metric Learning for Imbalance Data Classification," Proc. of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18), pp. 2805-2811, [Online], available: https://www.ijcai.org/proceedings/2018/0389.pdf, 2018.

[17] L. Gautheron, A. Habrard, E. Morvant and M. Sebban, "Metric Learning from Imbalanced Data with Generalization Guarantees," Pattern Recognition Letters, vol. 133, pp. 298-304, 2020.

[18] S. Susan and A. Kumar, "Learning Data Space Transformation Matrix from Pruned Imbalanced Datasets for Nearest Neighbor Classification," Proc. of the IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 2831- 2838, Zhangjiajie, China, 2019.

[19] S. Barua, Md. M. Islam, X. Yao and K. Murase, "MWMOTE - Majority Weighted Minority Oversampling Technique for Imbalanced Data Set Learning," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 2, pp. 405-425, 2012.

[20] V. Ganganwar, "An Overview of Classification Algorithms for Imbalanced Datasets," International Journal of Emerging Technology and Advanced Engineering, vol. 2, no. 4, pp. 42-47, 2012.

[21] G. B. Huang, M. Mattar, T. Berg and E. Learned-Miller, "Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments," Technical Report in Workshop on Faces in'Real-Life'Images: Detection, Alignment and Recognition, [Online], Available: http://vis- www.cs.umass.edu/lfw/lfw.pdf, 2008.

[22] O. M. Parkhi, A. Vedaldi and A. Zisserman, "Deep Face Recognition," Proc. of the British Machine Vision Conference (BMVC), pp. 41.1-41.12, [Online], Available: https://www.robots.ox.ac.uk/~vgg/ publications/2015/Parkhi15/parkhi15.pdf, Sep. 2015.

[23] K. Q. Weinberger and L. K. Saul, "Distance Metric Learning for Large Margin Nearest Neighbor Classification," Journal of Machine Learning Research, vol. 10, no. 2, pp. 207-244, 2009.

[24] F. Schroff, D. Kalenichenko and J. Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815-823, Boston, MA, USA, 2015.

[25] Y. Taigman, M. Yang, M. A. Ranzato and L. Wolf, "DeepFace: Closing the Gap to Human-level Performance in Face Verification," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1701-1708, Columbus, OH, USA, 2014.

[26] Y. LeCun and Y. Bengio, "Convolutional Networks for Images, Speech and Time Series," The Handbook of Brain Theory and Neural Networks, vol. 3361, no. 10, pp. 1-14, 1995.

[27] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-scale Image Recognition," Proc. of ICLR 2015, arXiv preprint arXiv: 1409.1556, 2014.

[28] J. Goldberger, G. E. Hinton, S. T. Roweis and R. R. Salakhutdinov, "Neighborhood Components Analysis," Advances in Neural Information Processing Systems, pp. 513-520, [Online], Available: https://www.cs.toronto.edu/~hinton/absps/nca.pdf, 2005.

[29] K. Q. Weinberger and G. Tesauro, "Metric Learning for Kernel Regression," Artificial Intelligence and Statistics, pp. 612-619, [Online], Available: http://proceedings.mlr.press/v2/weinberger07a/Weinberger 07a.pdf, 2007.

[30] J. V. Davis, B. Kulis, P. Jain, S. Sra and I. S. Dhillon, "Information-theoretic Metric Learning," Proc. of the 24th Int. Conf. on Machine Learning, pp. 209-216, DOI: 10.1145/1273496.1273523, ACM, 2007.

[31] E. P. Xing, M. I. Jordan, S. J. Russell and A. Y. Ng, "Distance Metric Learning with Application to Clustering with Side-information," Proc. of the 15th International Conference on Neural Information Processing Systems (NIPS'02), pp. 521-528, 2003.

[32] G.-J. Qi, J. Tang, Z.-J. Zha, T.-S. Chua and H.-J. Zhang, "An Efficient Sparse Metric Learning in High- dimensional Space via I1-penalized Log-determinant Regularization," Proc. of the 26th Annual Int. Conf. on Machine Learning, pp. 841-848, DOI: 10.1145/1553374.1553482, ACM, 2009.

[33] H. S. Dadi and G. K. M. Pillutla, "Improved Face Recognition Rate Using HOG Features and SVM Classifier," IOSR Journal of Electronics and Communication Eng., vol. 11, no. 04, pp. 34-44, 2016.

[34] D. Chen, X. Cao, F. Wen and J. Sun, "Blessing of Dimensionality: High-dimensional Feature and Its Efficient Compression for Face Verification," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3025-3032, Portland, OR, USA, 2013.

[35] Y. C. Wong, L. J. Choi, R. S. Sarban Singh, H. Zhang and A. R. Syafeeza, "Deep Learning-based Racing Bib Number Detection and Recognition," Jordanian Journal of Computers and Information Technology (JJCIT), vol. 5, no. 3, pp. 181-194, 2019.

[36] A. Kaushik and S. Susan, "Metric Learning with Deep Features for Highly Imbalanced Face Dataset," Proc. of the International Conference on Innovative Computing and Communications, Part of the Advances in Intelligent Systems and Computing Book Series, vol. 1394, pp. 639-646, 2022.

[37] B. Knyazev, R. Shvetsov, N. Efremova and A. Kuharenko, "Leveraging Large Face Recognition Data for Emotion Classification," Proc. of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 692-696, Xi'an, China, 2018.

[38] S. Karahan, M. K. Yildirum, K. Kirtac, F. S. Rende, G. Butun and H. K. Ekenel, "How Image Degradations Affect Deep CNN-based Face Recognition?," Proc. of the International Conference of the Biometrics Special Interest Group (BIOSIG), pp. 1-5, Darmstadt, Germany, 2016.

[39] T. BDlWUXšDLWLV, P. Robinson and L.-P. Morency, "OpenFace: An Open Source Facial Behavior Analysis Toolkit," Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-10, Lake Placid, NY, USA, 2016.

,abstract={This paper proposes a new learning methodology involving deep features and two-way metric learning for large, extremely imbalanced face datasets where the number of minority classes and the imbalance ratio are both very high. The problem arises because the faces of some celebrities, being more popular, are readily available in social media and the internet, while the faces of some relatively lesser-known personalities are fewer in number. Resampling being impractical in this scenario, we propose metric learning as the tool for mitigating the class- imbalance problem prior to the classification stage. To reduce the computational overhead associated with metric learning, we separately conduct weakly supervized metric learning with majority and minority class subsets, a process that we call two-way metric learning. Transformation matrices learnt from the majority and minority subsets are used to transform the entire input space twice. The test sample in the transformed space is assigned the class of its nearest neighbor in the training set of the twice-transformed input space. Deep features derived from the state-of-the-art pre-trained deep network VGG-Face form the input space and the aggregate cosine similarity measure is used to find the closest neighbor in the training set of the twice-transformed input space. Experiments on the benchmark LFW face database having 1680 classes of celebrity faces prove that the proposed methodology is more effective than existing methods for the classification of large, extremely imbalanced face datasets. The classification accuracies of the minority classes are especially found to be boosted which is a rare accomplishment among existing methods for imbalanced learning in deep frameworks.},
keywords={Face recognition,Metric learning,VGG-Face,Deep learning,Imbalanced learning,Extremely imbalanced dataset},
ISSN={2413-9351},
month={December}}