Brain Computer Interface for Emotion Recognition Based on EEG Signal

. This paper presents an emotion recognition system based on electroencephalography (EEG) signals. This system helps medical practitioners to analyse the mental health of an individual. Eight healthy volunteers/ subjects had participated in this experiment. A specific feeling is evoked using particular songs and videos that are collected to present before the subjects. Total 6 emotions namely neutral, happy, sad, disgust, fear and motivate are captured and analysed. Data is classified using eighteen statistical features. The sampling rate is 1200Hz. Signals are filtered using pre-processing techniques. Frequency, time and time-frequency domain features are extracted. An array of 10 classifiers is used including Decision Tree, Random Forest, Optimised Random Forest, Logistic regression, Support Vector Machine (SVM) Polynomial, SVM Sigmoid, SVM RBF, K-Nearest Neighbours, Gaussian NB, Gradient Boosting Classifier. Accuracy, recall, precision, and F1 score are employed as performance metrics. The accuracy obtained for SVM classifier was 79.34%.


Introduction
In our daily lives and at work, emotion plays an extremely essential role. As per an American Psychology Association survey [1] the stress rate was found to be 32%, which means that one out of every three subjects were stressed. It also helps in the treatment of patients, particularly those with expression problems. Moreover, clinicians are able to provide more appropriate medical care by knowing their true emotional condition. Text, voice, facial expressions, and gestures may all be used to detect emotions. Instrument data acquisition for various emotions was completed with the assistance of ADInsruments power lab [2].
Electroencephalogram (EEG) signals are used to determine emotions in this study. The electrical activity inside the brain's is collected by EEG. The EEG signals are collected from the electrical activity inside the brain which changes according to the feelings of a person and his emotions. The signals are then analysed and statistical parameters (features) are calculated, then these are passed to the classifier. Some readily available resources include the DREAMER, SEED, DEAP, MAHNOB-HCI, and DECAF datasets [3].
The paper includes five steps, such as procurement of data, pre-processing, feature extraction and classification, and testing accuracy. Medical tests including positron emission tomography (PET), computed tomography (CT-SCAN), Functional Magnetic Resonance Imaging (FMRI), magneto electroencephalography (MEG), and EEG are some of the modalities used to collect brain waves. The signals are contaminated and need to be filtered. A fusion of finite impulse response (FIR), adaptive filter, time-frequency analysis (Daubechies wavelet transform), and SVM classifier are utilized to identify discrete emotions [4] (happy, angry, sad, and weep) in a human by means of EEG input. The fundamental benefit of utilising EEG signals is that it recognises true emotions that arise directly from the mind, rather than outward aspects such as facial expressions or gestures.

Literature Review
The study of emotion recognition started with the recognition of facial expressions which was accomplished using traditional computer vision. The channel-specific structure [5] of EEG data was in-depth and developed an efficient feature extraction system is dependent upon flexible analytic wavelet transform (FAWT). EEG data were collected from 16 healthy people [6] utilising only three EEG channels, namely Fp1, Fp2, and a bipolar channel of F3 and F4 locations following to the 10-20 system, using a series of facial expression pictures projections. Seven different emotions [7][8] were classified based on EEG signals. It effectively detected seven emotions using SVM and Linear Discriminant Analysis (LDA), with an accuracy of 74.13% and 66.5%, respectively. These characteristics were translated into the corresponding emotions using a basic classifier called SVM. Another paper offers an experiment with Empirical Mode Decomposition (EMD) [9].
EMD has already been verified as a handy technique for signal analysis. The approach presented in this research uses SVM classifiers to categorise data using three characteristics retrieved from Intrinsic Mode Functions (IMFs): the first phase difference, initial time difference, and initial normalised energy. Five separate models were used to classify 10 different emotions based on valence, arousal, and dominance. Grounded on different emotions, the accuracy ranged from 50 to 70 %. An EEG and facial expression based multimodal emotion recognition system [10] was proposed. A Decision Tree was used for feature selection, and sparse representation was used to recognise face emotion. The classifier [11] Library for SVMs (LIBSVM) was employed. The accuracy was around 85.71%. The study puts forward a deep belief-conditional random field (DBN-CRF) architecture [12] to simulate multichannel EEG characteristics, and the authors grew a unique dynamical graph convolutional neural network (DGCNN). In the case of subject-dependent studies, DGCNN's average recognition accuracy can reach 90.40 %. EEG signals were crucial in the categorization [13][14] of emotions, especially for those with amyotrophic lateral sclerosis (ALS). The absence of data from EEG waves makes developing an emotion identification system difficult. The average valence accuracy was 70%, while the average arousal accuracy was 72.42 %.
Emotion recognition models [15] were implemented using neural networks. Design and implementation [16] of a wearable device using convolutional neural network (CNN) was introduced for emotion recognition. A SPARTAN-6 field programmable gate array (FPGA) was used for the implementation. The deep convolutional neural network (DECNN) [17] method was used to identify emotions in the brain. Average accuracies [18] of this method were 90.63 % and 92.58 %. A multi-channel EEG-based emotion recognition [19] was put forth using a deep forest method. The modifications done to the hyperparameters have little impact on the classification model, greatly increasing the simplicity of emotion recognition model. The characteristics of all channels were processed [20] into a 3D feature matrix based on the placements of electrode sensors.
The left and right regions [21] of the brain are known to communicate emotions in various ways. The BiHDM, or bi-hemispheric discrepancy model, helps to identify emotions. Research gaps observed deep neural network-based systems were mainly used for emotion recognition. The classification was done into two categories valence and arousal. Average accuracy observed was around 70%. Some readily available datasets were taken into use which led to the repetition of the same work.

Methodology
This paper presents an EEG based approach for classification of six emotions namely, i) Happy, ii) Sad, iii) Neutral, iv) Disgust, v) Fear, and vi) Motivate

Experimental setup
EEG is a test that uses small metal discs (electrodes) connected to the scalp to assess electrical activity in the brain. Emotions are brain activities which get ignited as a person feels stimulus.
Emotions recognition are carried out using EEG signals. A portable 32 channels EEG machine setup from Clarity is used for recording the EEG signals. Readings are recorded for 8 subjects out of which 4 are male and 4 are female. The readings include six activities namely neutral, happy, sad, motivated, disgust and fear. A specific feeling was evoked using videos that were collected and presented to the subjects, including blank black video (for neutral state), party songs (for happy state), sad songs (emotional state), motivational videos (for motivation state), disgust sounds (for disgust state) and jump scare video (for fear state).

3.2
Montage used for recording EEG signals.
Montages are logical, organised arrangements of electroencephalographic derivations or channels that are used to represent activity over the entire head and give lateralizing and localising data. Montage used is presented in Fig. 1.   Tri maps are a visual depiction of how the human brain works when performing any task. The tri maps showing the top, left and right activity of the brain for fear state and happy state are displayed in Fig. 2 and Fig. 3 distinctively.

3.3
Dataset description EEG signals are recorded for 8 subjects and categorised into 6 states. The data collected from 6 subjects is used to train the model and the data collected from 2 subjects is used to test the model resulting in an 80-20 split of data. The recorded data of different states namely neutral, happy, sad, motivated, disgust, fear is stored in separate files for all the subjects. There are a total of 5 columns in each dataset. The first column is just the indexing sequence and the last 4 columns consist of the data recorded from 4 EEG channels. The data is simply analysed depending on the voltage (in µv) readings captured. The voltage is greater when the brain is highly active and vice versa.

Feature extraction/ filtering
The raw EEG data contains the readings of an emotion at a particular time. This raw data cannot deliver any useful information. The raw data needs to be processed and followed by feature extraction to get useful information out of it. Discrete wavelet transform (DWT) is used for this feature extraction. DWT divides the lower frequency band into two halves to produce another high frequency and low frequency band to continue in the same manner. The formula for calculating DWT is presented in Eq. 1. , = ∫ ( )2 * * (2 − ) (1) where, dj,k = wavelet coefficients k = location j = level The frequency domain, time domain, and time-frequency domains are where features are extracted. The pre-processing includes filtering of the data. A Butterworth filter, also termed as a maximally flat magnitude filter is a signal processing filter having a frequency response that is as flat as possible within a passband. A 5th order of this filter, with a frequency range of 3Hz -150Hz is used. This frequency range is selected as the brain's activity shows results for states like happy, sad, neutral, motivated, disgust and fear. The sampling rate selected is 1200Hz.

Fig. 4. Brain Maps for Delta and Theta frequencies for different Emotion States
The brain maps for different emotions are shown in Fig.4. from which it is easy to determine the activity of the brain for a particular feeling. A band-stop filter (notch filter) with a high and low cut-off frequencies of 51Hz and 49Hz, and a sampling rate of 500Hz is used to remove 50Hz wave generated due to supply voltage frequency. The pre-processed EEG signals are used to extract features.  A spectrogram displays the signal amplitude at each frequency that is present in a given waveform across time. A spectrogram shows time on the y-axis and frequencies on the xaxis. The colours of a spectrogram can also be used to represent the signal strength; the brighter the hue, the stronger the signal. A spectrogram demonstrates how the signal's power is dispersed across all of its frequencies. Spectrograms that are plotted in Fig. 6. show the effect of emotion on EEG signals.

Discussions of features
A Time Domain Features (TDF) is calculated using either raw EEG data or a signal that has been time-domain decomposed. Collection of algebraic characteristics have commonly been utilized to distinguish between ictal (seizure) and normal patterns. Mean, median, variance, mode, kurtosis, and skewness (third moment expressing data asymmetry) are the major parameters, other statistics include the coefficient of variation (CV), defined as the ratio of the standard deviation (SD) to the sample mean and illustrates the data's dispersion in relation to the population mean. The standard deviation is calculated using Eq. 2. = ∑ (9) where, z = size of frame, y = sample points m = each value  As shown in Fig. 9, statistical features like RMS and AAC features can distinguish between various emotions.

Principal component analysis (PCA)
PCA is a method of unsupervised learning that is utilized to reduce dimensionality in machine learning. It is a statistical method that converts observations of correlated features into a pair of linearly uncorrelated data. These principal components are the recently transformed features. The EEG data extracted from raw data is of higher dimensions i.e. it has 129 features, and it needs to be reduced to some lower version. The variance must be calculated to obtain good dimensionally reduced data. The variance described by each of the major components is referred to as the variance of eigenvectors. The ratio of related eigenvalues to the sum of all eigenvectors can be used to calculate variance. These eigenvectors represent the principal components that contain most of the information (variance) in this case it is 98%, accordingly the number of principal components chosen are 41. Filtering of data

Results and discussion
This research introduced an EEG-based system for emotion recognition. EEG signals were recorded aiming 8 subjects using clarity's EEG signal recording machine. The raw data was collected for 6 different emotions (neutral, happy, sad, disgust, fear, motivation) and preprocessed to further pass it on to an array of classifiers. An accuracy of 79.34% was obtained from SVM Sigmoid (kernel) and SVM RBF (kernel) classifiers. The bar graph plotted in Fig. 11 shows the results for 3 binary classifications. A1 shows the accuracy for happy vs sad class, A2 shows the accuracy for motivate vs disgust and A3 shows the accuracy for neutral.