Real-Time Facial Emotion Recognition: Insights and Comparative Analysis

2.1 FER-CNN

FER-CNN is a deep learning [1] method with two convolutional neural network layers and one fully connected layer. Max pooling, which pulls the maximum value from the pool and creates the output, has been used in both layers of the pool size (2.2). ReLU is the activation function employed in both top layers. In order to avoid overfitting and effectively train the model, the dropout function has also been used. The softmax function is used to obtain network outputs in the fully connected layer. The pictures are then categorized using labels like angry, happy, sad, disgusted, surprised, neutral, and fearful. There are 28.709 trained images in total, of which 7.215 are trained to detect the happy emotion, 3.995 to detect the angry emotion, 4.850 to detect the sad emotion, 4.097 to detect the fearful emotion, and 3.171 to detect the surprised emotion. Similar to this, 436 images are trained to detect the disgustful emotion, while 4.965 images are trained to detect the neutral emotion.

While training the model, we could achieve a training accuracy of around 97.63% for 200 epochs. Having this much higher accuracy while training the model is generally not considered good as there are high chances of overfitting, and the model will perform poorly on the test data. During the evaluation of the model, the validation accuracy achieved was 17%, i.e., the model is overfitting. To remove this, we used data augmentation and again evaluated the model on the testing data; this time, we could achieve a validation accuracy of around 71%.

2.2 AlexNet CNN

Here, the pre-trained model of Alexnet CNN has been modified a bit and used to find how well it works on the FER-2013 dataset. Alex Krizhevsky and his research team first put forth the pre-trained model of Alexnet CNN [6], which contained three fully connected layers and five convolutional layers. The AlexNet network's convolution layer's convolution Kernel size is a single value, and the generated feature map features aren't diverse, so there won't be enough feature extraction. The three flaws-a single size convolution kernel, an excessive number of parameters in the fully connected layer, and a change in data distribution-were fixed. In the conventional convolution layer, a single-sized convolution kernel [6] convolves the input data for many feature maps. Multichannel convolution technology is an extension of traditional convolution and consists of the addition of multiple convolution kernels of various sizes to a single convolution layer. The main issue was working with too many parameters, and to solve this problem, researchers came up with the idea of the GAP Layer which is the Global Average Pooling layer; this layer has been used in place of the fully connected layer. For the training purpose, the batch size chosen was 64, the number of steps in each epoch was 5000, and the image size for the facial expression was set to 48 x 48. 0.01 was the starting value for the learning rate. The network was made more efficient using Adam Optimizer. By substituting four different-sized convolution kernels (1*1, 3*3, 5*5, and 7*7) for the traditional single-sized convolution kernel, the proposed method introduces multichannel convolution technology. More features can be extracted, and the diversity of the convolution kernel is strengthened when multichannel convolution is compared to a single-sized convolution kernel. Additionally, batch normalization and global average pooling are included. The upgraded AlexNet network has thus greatly improved facial expression recognition accuracy.

2.3 HOG-ESRs

Studies in the past have demonstrated that the Histogram of Oriented Gradient (HOG) can be used to extract facial features and that the accuracy and robustness of the algorithm [7] can be improved by using ensemble methods.

The suggested approach that is used on the FER-2013 dataset is based on ESRs (ensembles with shared representations) and HOG features. Here, four branches of convolutional neural networks were constructed. A model with four convolutional branches and four convolutional branches is used for each network branch. The CNN model of each network branch is built on the original pixel data. Prior to the mixed feature set advancing to the full connection layer, a histogram of gradient features is added to the final convolution layer. Prior to sending the composite features into the first convolution layer network, also known as the single branch network of ESRs, the convolution layer in the new model of HOG-ESRs links the convolution layer's features to the HOG features. The primary element of the classification task used by the convolution layer is the original pixel data.

2.4 Affdex CNN

According to [1], the API for the Affdex SDK can recognize seven emotion metrics, twenty facial expression metrics, 13 emojis, and four appearance metrics.

More than 4 million facial videos and hundreds of thousands of facial frames made up the training set. This data, which depicts spontaneous, in-person facial expressions made under difficult circumstances like changing lighting, shifting head positions, and variations in facial features caused by race, age, gender, facial hair, and eyeglasses, represents more than 85 different nations. As can be seen, this tool for analyzing facial expressions is significantly more complex than earlier algorithms suggested in this paper. However, it was used to find out more about how closely businesses could use human-made emotion recognition techniques.

2.5 SVM of HOG

SVM is a powerful and widely used supervised learning algorithm with its own principles and techniques. It focuses on finding an optimal decision boundary that maximizes the margin between classes rather than isolating each class separately.

The training algorithm in SVM, known as the optimization algorithm, aims to find the hyperplane with the maximum margin. It does not explicitly decrease the distance between classes. Instead, it optimizes the decision boundary based on the support vectors, which are the data points closest to the decision boundary [8].

Support Vector Machines (SVM) with HOG feature descriptor is a popular approach for object detection and recognition tasks in computer vision.

HOG is a feature descriptor that captures the local shape and gradient information in an image. It divides the image into small cells, computes the gradient magnitude and orientation within each cell, and then constructs a histogram of the orientations within each block of cells. This histogram represents the distribution of edge orientations in that block, providing a concise representation of the local texture.

For each image in the dataset, the HOG feature descriptor is computed. This involves dividing the image into cells, calculating the gradient magnitude and orientation for each cell, and constructing histograms of orientations within each block.

Then, the HOG descriptors are converted into a feature vector representation. This typically involves concatenating the histograms from all the cells and blocks, resulting in a high-dimensional feature vector. Next, the SVM classifier is trained using the feature vectors obtained from the positive and negative examples. The SVM aims to find the optimal hyperplane that separates the positive and negative examples in the feature space.

SVM with HOG feature descriptor has been widely used in various applications such as pedestrian detection, face detection, and object recognition. It provides a robust and effective method for capturing local texture information and making accurate predictions.

3.1 FER-CNN

The test data from the FER2013 dataset [9], which contains 7178 images, is used to test the model. In general, OpenCV makes use of the device's camera to carry out real-time emotion detection. Using the haarcascade frontal-face XML file, the first task is to find a human face and then draw a bounding box around it. The rectangle is first formed around the detected face after the device's image has been preprocessed. The image is then cropped into a frame measuring 48 x 48 pixels, which is then made into a monochromatic version and fed into the model for feature extraction and emotion classification from the available 7 emotions. The matrix for the same is shown in Table 1 above.

From the above matrix, different metrics have been evaluated for each label as follows:

We can see from the above table that the emotion happy has an individual accuracy of 93.01%, and at the same time, the false negative rate is only 0.09, i.e., The FNR is very low, meaning that fewer images belong to the happy state and are classified as some other emotion. Alternatively, we can say that if an image belongs to the happy emotion state, then there is a high chance that it will be classified as happy. For the same emotion, the recall value is 0.91, meaning that most of the images are correctly classified.

Similarly, from the values calculated above, we can clearly see how our model works for identifying a particular emotion label. For the sad emotion, the individual accuracy is not good, and the FNR is also quite high for it, meaning that around 40% of images are falsely classified. Also, the FPR value is highest among the other emotions, meaning that around 8% of images don’t belong to the sad category but are classified as sad. Looking at the recall values, it seems that only the images belonging to the “happy” emotion are correctly classified, as the rest of the images belonging to other emotions have lower recall values.

If we want to rank the performances for different categories as per the F1 score, it is found that the classifier works well for surprise emotion, followed by happy, disgust, neutral, anger, sad and fear.

3.2 AlexNet CNN

The confusion matrix [10] is:

Out of all the algorithms analyzed in this paper, Alexnet CNN has the highest accuracy. It can be noticed that the individual accuracy of all the emotions is quite good, and at the same time, the FNR values are low, supporting the fact that this model really performs well.

Considering the happy emotion, the FPR and FNR values are lowest among all other emotions, and the specificity, recall, and precision values are highest among all other emotions, meaning that AlexNet’s performance for happy emotion images is exceptionally high.

It is noticed that the individual accuracy for neutral emotion is the lowest among all other emotions and has the highest value for FNR, meaning that out of all other emotions, AlexNet’s performance is the lowest for this particular emotion, but still, the performance of this model for neutral emotion is better than other models.

From the metrics calculated above, we can see that the FNR for neutral emotion is 0.17 (the highest), meaning that there are images that are neutral but are not classified as neutral; this is happening at a rate of 17%, i.e., out of 100 neutral images, 17 images are not classified as neutral. To rank the classifier on different categories, we’ll look at the F-1 scores calculated above; the classifier works well on happy emotions followed by surprise, sad, anger, fear, disgust and neutral.

3.3 HOG-ESRs

The confusion matrix is:

HOG-ESRs stand fourth in terms of accuracy when analyzed against other methods. From the metrics obtained above, it is clear that this algorithm has the highest individual accuracy for Disgust emotion, but the FNR is also 0.44, meaning that it wrongly classifies a significant number of images. One thing to understand is why the individual accuracy and the FNR are both high at the same time; the reason behind this is that the sum of true negatives and true positives is quite high, i.e., there is a high number of images that do not belong to the Disgust emotion, and they are correctly classified as Not Disgust meaning that these images belong to other emotion tags and thus are not classified as Disgust, due to this reason the individual accuracy is high. One more thing to notice is that for the "Fear" emotion, the FPR is 0.0029. The FNR is 0.48 (the highest), i.e., fewer images do not belong to the Fear emotion and are classified into the Fear emotion tag. More images belong to the Fear emotion but are not classified into the Fear emotion tag. Due to this, the recall value is only 0.511 (the lowest) in spite of having high individual accuracy and precision.

Here are the rankings based on F-1 scores as follows: happy comes first, followed by angry, neutral, sad, surprised, disgusted and fear.

3.4 SVM of HOG

The Confusion matrix is:

This model stands second in terms of overall accuracy; one thing to notice here is that this model has shown exceptionally high performance in determining the Fear emotion; unlike other models, the individual accuracy, precision, and recall all have comparatively good values that indicate better performance. The FPR and the FNR both values are low indicating that there are fewer images wrongly classified.

We can say that this is the best model for detecting fear emotions; the values of the above metrics show that this model works better for fear emotions than happy emotion images. Also, this model works greatly on Sad emotions as the values of the FNR and the FPR are both convincing, and it has good individual accuracy. The value of specificity for Angry and Sad emotions is the same, and the values of the FPR (1-specificity) are also the same, meaning that there is an equal ratio of the images that do not belong to these emotions and are not classified into this emotion (true negatives) to the total number of images having another emotion label (not angry or sad).

Rankings of the classifier are as follows:

First comes Fear, then sadness, followed by surprise, disgust, happiness, neutrality and anger.

3.5 Affdex CNN

Rankings for the classifier (f1-score) are as follows:

Happy comes first, then fear, followed by anger and sadness.