Channel Division Based Multiple Classifiers Fusion for Emotion Recognition Using EEG signals

. With the rapid development of computer technology, pervasive computing and wearable devices , EEG-based emotion recognition has gradually attracted much attention in affecting computing (AC) domain. In this paper, we propose an approach of emotion recognition using EEG signals based on the weighted fusion of multiple base classifiers. These base classifiers based on SVM are constructed using a channel division mechanism according to the neuropsychological theory that different brain areas are differ in processing intensity of emotional information. The outputs of channel base classifiers are integrated by a weighted fusion strategy which is based on the confidence estimation on each emotional label by each base classifier. The evaluation on the DEAP dataset shows that our proposed multiple classifiers fusion method outperforms individual channel base classifiers and the feature fusion method for EEG-based emotion recognition.


Introduction
Human emotion which synthesizes the feelings, thoughts and behaviours, plays a critical role in our daily life [1].With the rapid development of ubiquitous technology and computer technology, affective computing (AC) which is an interdiscipline related to cognitive science and artificial intelligence (AI) has gradually become a hot topic in the field of emotion research [2].Emotion recognition as a key issue of AC has gained increasing attentions and applications such as brain-computer interface (BCI), psychology and clinical assistant diagnosis of mental disorders, etc.There are various physiological signals (e.g., ECG, skin conductance, respiration, etc.) used to perform emotion assessment, in which EEG-based emotion recognition is widely used because EEG signal from the central nervous system has real-time diversity, non-disguise and non-invasive.
Since EEG not only indicates emotional states, but also reflects other cognitive activity of the brain, recently researchers tried to find the correlation between the emotional changes and EEG signals.Some of these studies belonging to neuropsychology have focused on the question of asymmetrical activation of the cerebral hemisphere [3][4].From these researches, a common conclusion that different brain areas are differ in the processing mode and intensity of emotional information can be obtained.In addition, more and more complex methods were proposed to improve the performance of EEG-based emotion recognition by presenting some novel feature extraction, feature selection and machine learning algorithms [5][6][7][8][9][10].Most of these studies utilized a simple feature fusion strategy that is to combine extracted features and EEG channels to form a single feature vector as the input of classification model.However, EEG signals have low signal-to-noise ratio (SNR) and often mixed with much noise when collected.The more challenge problem is that, unlike image or speech signals, EEG signals are temporal asymmetry and nonstationary because the human brain can be seen as a complicated nonlinear dynamic system [11].So these EEG-based emotion recognition methods using a simple feature integration or fusion are difficult to obtain stable recognition performance and even bring some negative effects.What's more, these methods cannot preserve the original domain information such as channels and frequency bands which are very important for understanding brain response.
Considering the problem mentioned above, this paper proposes an effective EEG-based emotion recognition approach with a novel decision fusion method based on multiple channels, namely multiple classifiers fusion (MCF).The rest of this paper is organized as follows.Section 2 briefly reviews the research background.Section 3 describes our approach in details.The detailed experimental settings, experimental results and analysis will be given in section 4. Finally, we give the remark and conclusion of this paper.

Emotional models
An emotional state refers to a psychological and physiological state in which emotions and behaviours are interrelated and appraised within a context [12].Many representations of emotional states, such as discrete scales [13] (e.g., anxiety, frustration, engagement and distress) and the arousal-valence space [14], have been used in emotion assessment studies.The bi-dimensionality theory of emotions proposes that nature of emotional experience, or at least its "affective core", is primarily determined by two main dimensions: arousal and valence.Arousal refers to the quantitative activation level ranging from calm to excited.Valence represents the quality of an emotion ranging from unpleasant to pleasant.Each discrete emotional state (e.g., happy, calm and sad) can be defined in the arousal-valence space as shown in Figure 1.   2 shows electrode placement for 32-channel EEG recording using the International 10-20 system.
The preprocessing operations of EEG signals include downsampled to128 Hz, bandpass filtered with cutoff frequency between 0.1 Hz and 45.0 Hz, EOG artifacts removal using blind source separation technique.All of the above preprocessing operations can be performed using EEGLAB toobox [16].The artifact removal procedure was applied to all EEG epochs for every subject under different emotion conditions since it is hard for participants to keep eyes fixed during watching entire music videos.According to the response of the subjects, only the experiment epochs when the target emotions were elicited were chosen for further analysis.Hence, we extracted the EEG segments corresponding to the duration of each music video after performing the preprocessing.Each channel of the EEG data was divided into the same-length epochs without overlapping using time windows.Finally, all features discussed below were further computed on each epoch of the EEG data.

Feature generation
Suitable representation of EEG data makes informative patterns for a classifier to recognize emotion.The main task of feature extraction is to derive the salient features which can map the EEG data into consequent emotional states.Recently, a wide range of features which are relevant for emotion recognition from EEG signals have been proposed.Because feature extraction was not the research emphasis of this study, the typical features of three different groups, namely time-statistical domain, frequency domain and nonlinear dynamic domain, were extracted to more comprehensively and accurately represent the EEG patterns related to emotional states.The specific features of each domain extracted on EEG data are shown in Table 1.In this paper, the features of EEG signals are extracted on five frequency bands that are delta (0.1-4Hz), theta (4-8Hz), alpha (8-13Hz), beta (13-30Hz) and gamma (30-45) by Wavelet Transform.Then, each epoch from one channel was mapped to three groups of features on the above five frequency bands.If all EEG features on 32 channels are analyzed in classification process, it will generate an enormous amount of computing cost, so the above typical feature values were computed merely on those channels related to emotion states.According to the analysis results in [7,15], 8 channels (namely, T7, CP5, P7, FC2, C4, T8, CP6, PO4) are selected to continue the following study.

Description of proposed method
As discussed before, many neurophysiological studies have proved that different brain regions have different intensity of processing emotion information.In addition, it was found that the performance of recognition changes with the varying channel positions in EEG-based emotion assessment researches.Traditional studies are to combine all the channels and features into one single feature vector using feature fusion method in order to represent the emotional patterns in EEG signals more adequately.However, it cannot improve the recognition performance stably because some bad-performed features bring negative influences.Furthermore, the simple feature concatenation always ignores the nature of emotion information processing in human brain and the non-linear structure of original EEG signals.To overcome these problems, this paper proposes an effective weighted fusion method of multiple classifiers constructed by channel division using Support Vector Machine (SVM).The framework of our proposed approach is shown in Figure 3. Firstly, channel base classifiers based on SVM are constructed on the basis of parts of channel training data, respectively.Secondly, the weight coefficients of each channel are learned based on the output of corresponding channel base classifier.Finally, a weighted fusion based multiple classifiers fusion model is constructed by integrating the outputs of channel base classifiers with the weight coefficients to estimate the overall confidence of one sample belonging to each emotional class.Following the principle of minimum error, the class with the maximum confidence is outputted as the final emotion recognition result.

Base classifier-SVM
We used SVM to construct base classifiers on each channel sample data and evaluate the performance of our proposed approach on EEG-based emotion recognition.SVM is a supervised learning algorithm based on statistical learning theory [17].The basic training principle of SVM is finding the optimal hyperplane where the expected classification error of test samples is minimized.It projects low dimension features into higher dimension using kernel functions which can create nonlinear decision boundaries and solve the inseparable cases.Different kernels used in our work to find the best performance of SVM in EEG-based emotion recognition are defined as follows: (1) where , and are the kernel parameters, d denotes the order of the polynomial kernel and T is the transpose operation.More information about SVM classifier can be found in [17].
SVM are known to have good generalization properties, to be insensitive to overtraining and to the curse-of-dimensionality.Finally, SVM has a few parameters (e.g., C, d) that need to be defined.These advantages are gained at the expense of a low speed of execution.SVM can deal with over-fitting problem by having misclassified instances on training data.Due to these advantages, SVM is more appropriate for emotion recognition because physiology data is noisy [18].

Weighted fusion of multiple base classifiers
Multiple classifiers fusion (MCF), sometimes called decision fusion, is constructed by integrating the classification results of several base classifiers according to some combination strategies.In this paper, each channel is processed independently by the corresponding classifier and their outputs are integrated to generate the final result.It is different from the existing studies with feature fusion methods which are straight forward considering the synchronous characteristics of the involved features by concatenating the different features extracted from signals to form a composite feature vector.However, the brain activity is a complicate nonlinear dynamic system.Thus, the feature fusion sometimes brings negative impact.Especially, their outputs miss the non-linear structures in the signal during the dimension reduction.The MCF supports to model the asynchronous characteristics of channels flexibly.It has an important advantage over feature fusion is that since each of the channel data is processed and classified independently in MCF, it is relatively easy to employ an optimal weighting scheme to adjust the contribution of each channel to the final decision according to their reliability of the feature.In this study, EEG-based emotion recognition using MCF mainly include the following steps.
Step (2) where represents the predicted emotional label of testing sample using the base classier built by the t-th channel training data and N is the number of testing samples.
is the true emotional label and the function is used to count the number of error-classification and total samples.
We can get the weight of classification model of each channel by formula (3) which is computed as below: (3) Step 4. Computing output probability For an input testing sample, the probability on each class is computed by the confidence estimation on each emotional label : Step 5. Obtaining the final results.
In the end, the emotional label with highest probability is the final decision output of this testing sample:

Experimental settings
Our proposed MCF method for EEG-based emotion recognition on two emotional dimensions(arousal and valence) is evaluated on the DEAP dataset.In this study, the participants' ratings scales during the emotion elicitation experiment are used as the ground truth.According to the unnormalized SAM-ratings values on a scale of 1-9, the performance of the proposed method is investigated in a two-class problem about arousal and valence.For the binary classification, we transformed ratings for arousal, valence to two categories, namely positive and negative.On the 9-point ratings scales, the division threshold was simple placed in the middle.
In this study, each dataset about different classification task was first divided into training set and testing set according to the same way as existing already, and its evaluating results are expressed by classification accuracy.As described earlier, there have been a lot of researches that addressed the EEG-based emotion recognition problem recently.However, it is not easy to compare directly the accuracies with the results in the literature, since they are differ in experimental paradigms and conditions which include the kinds of stimuli, the types of emotions, the processing methods of EEG data, etc.Nevertheless, most of them reported that the support vector machine (SVM) showed the best performance compared with other classifiers in the EEG-based emotion recognition problems.In this paper, SVM integrated in the LibSVM library [19] is applied to construct channel base classifiers using 10-fold cross validation scheme for evaluating our proposed approach.
In order to find a better performance of SVM, the performances of three different kernel functions, namely linear, polynomial and radial basis function (RBF), were compared in our study.Furthermore, among the SVM-based approaches, C-SVC showed the best performances according to Horling's study [20].Hence, we compared the results of the proposed method with those of C-SVC with three

Experimental results and discussion
Table 3 shows the average arousal and valence classification accuracies using C-SVC with different kernel functions after testing the full parameter settings for each emotion classification model based on SVM algorithm.It can be observed that the average classification accuracy of SVM with RBF kernel function outperforms SVM with linear kernel and polynomial kernel for building any type of classifier on arousal and valence dimension.This definitely proves the robustness of the RBF SVM over the linear SVM and polynomial SVM, so the SVM with RBF kernel was chosen as the basic classifier in the remaining paper.From the Figure 4, it is observed that the feature fusion achieves the performances higher than the lowest performances but lower than the highest performance by base channel classifiers.In other words, the feature decision method can not improve the classification performance obviously.Our proposed MCF method achieved better performance than the best channel base classifier C-FP1 and feature fusion method.It achieves the 7.37% accuracy improvement on valence from feature fusion and 3.77% accuracy improvement from the highest base channel classifier C-FP1.As for the arousal classification accuracy, the MCF outperforms feature fusion for 5.43% and C-FP1 for 2.29%.These results show that our proposed MCF method based channel division is effective to improve the classification performance of EEG-based emotion recognition.
In order to evaluate the effect of different EEG frequency bands on emotion classification, we use the sample data on each frequency band to train and test emotion classification models.Figure 5 presents the classification accuracies of our method and feature fusion (FF) method across different frequency bands of EEG signal on arousal and valence emotional dimension.
Figure 5. Classification accuracies of our method and feature decision method same conditions.In addition, it is found that the classification performance of beta and gamma bands is slightly better than those of delta, theta and alpha bands.This result shows that beta and gamma bands are the key bands in emotion recognition task.In other words, high frequency bands of EEG signals play a more important role in emotion activities than low frequency bands [22][23].

Conclusion
In this paper, we present an EEG-based emotion recognition approach by using multiple classifiers fusion (MCF) based on EEG channel division according to the neurophysiological theory that different brain areas have different processing intensity of emotion information.There are several important observations that can be drawn from our comparative experiments.Firstly, it can be observed that the classification performance of SVM with RBF kernel function outperforms SVM with linear kernel and polynomial kernel for building any type of classification model on arousal and valence emotional dimension.This definitely proves the robustness of the RBF SVM over the linear SVM and polynomial SVM for EEG-based emotion classification.Seconly, it is feasible to estimate emotion using single channel, especially those channels in the frontal lobe compared with other areas of scalp.Thirdly, gamma and beta frequency bands perform better than other frequency bands.It confirms that beta and gamma oscillation of brain activity are more related with emotional processing than other frequency oscillations.The last but the most important is that our proposed MCF method based on divisional channels outperforms all single channel base classifiers and the feature fusion method which combines all the channels and extracted features into one single feature vector.This indicates the effectiveness and reliability of multiple classifiers fusion in EEG-based emotion recognition.

Figure 3 .
Figure 3.The framework of EEG-based emotion recognition approach using multiple classifiers fusion.

Figure 4 .
Figure 4. Classification accuracies of our proposed method and other classification models

Table 1 .
The extracted features from different domains.

2. Computing classification error rate.
different kernel functions.The parameters of C-SVC were determined by 10-fold cross validation method.For all kernel functions, we estimated the classification accuracy using the same range of cost parameter :  ∈ {2 !! , 2 !! , … , 2 !, 2 !" }.The other parameter setting of different kernel is shown in the following table.

Table 2 .
Parameter setting for different kernel functions of C-SVC