Authors: Rodolfo Pavez, Jaime Diaz, Jeferson Arango-Lopez, Danay Ahumada, Carolina Mendez-Sandoval, Fernando Moreira
Categories: S.i.: Tam-Lhr, Autism spectrum disorders, Convolutional neural networks, Emotion recognition, Human–computer interaction
Source: Neural Computing & Applications
Authors: Rodolfo Pavez, Jaime Diaz, Jeferson Arango-Lopez, Danay Ahumada, Carolina Mendez-Sandoval, Fernando Moreira
Autism spectrum disorder (ASD) is a neurodevelopmental disorder defined as persistent difficulty in maturing the socialization process. Health professionals have used traditional methods in the therapies performed on patients with the aim of improving the expression of emotions by patients. However, they have not been sufficient to detect the different emotions expressed in the face of people according to different sensations. Therefore, different artificial intelligence techniques have been applied to improve the results obtained in these therapies. In this article, we propose the construction of an intelligent mirror to recognize five basic angry, scared, sad, happy and neutral. This mirror uses convolutional neural networks to analyze the images that are captured by a camera and compare it with the one that the patient should perform, thus supporting the therapies performed by health professionals in children with ASD. The proposal presents the platform and computer architecture, as well as the evaluation by specialists under the technology acceptance model.
Continuous technological development has become fundamental in different medical areas. One particular area is therapies for children with autism spectrum disorder (ASD), where facial emotion recognition interventions using technology are particularly promising [1].
The recognition of facial emotions is a component of social cognition [2] and is essential for effective communication and social interaction [3][4]. People with ASD, a neurodevelopmental disorder characterized by deficiencies in social communication and unusually restricted and repetitive behaviors [5], have difficulty in recognizing others' facial emotions, making social interaction hard [6]. Helping their emotion recognition skills through intervention tools could significantly improve these children's social interactions [7].
The main problem is that emotion recognition therapies are manual processes, which use tangible products with fixed settings and which today represent a danger due to the possible transmission of diseases. In addition, this traditional process sometimes does not generate motivation in patients; on the contrary, it makes them feel intimidated, as researchers of our team have expressed.
Technological resources can easily support this process [8, 9]. This option does not seek to eliminate the specialists from the therapy, but on the contrary, to facilitate their activities to achieve a concentration in other exercises that could be more critical.
This article details creating a smart mirror for emotion recognition (EMO-Mirror) based on convolutional neural networks (CNN). It is an extension of a previous study presented by its authors [10]. In this version, we will present the proposal's technological infrastructure, the results of the application, and validation by experts.
This paper is structured as follows. Section 2 presents the related concepts. Section 3 presents the related works and a comparison with our research. The materials and methods are presented in Sect. 4. The IT Platform and the architecture are presented in Sect. 5. Sect. 6 contains the main results of the research. Finally, the conclusions are presented in Sect. 7.
There are several terms that are a fundamental part of our research, we describe them as
Emotion recognition is highly relevant in a social environment. Children with ASD have deficiencies in initiating and responding to social or emotional interaction [5]. As a result, they have difficulty recognizing and understanding the emotions and mental states of others [11, 12].
Their difficulty with emotional recognition can generate unfavorable consequences over time, which can lead to psychiatric comorbidity, poor occupational performance, and problems in social relationships [13].
Consequently, professionals who work with children with ASD can use different techniques to assist in the training of emotion recognition [14, 15].
For our proposal, we consider the training processes that encourage imitation and recognition of emotions. The smart mirror has a set of images that explain different scenarios. This functional behavior is based on the proposal of Pictures of Facial Affect (POFA) by Ekman and Friesen [16–18]. This feature of the smart mirror solves the problem of static images (usually on physical cards), prone to being polluting agents, and which do not generate immediate feedback for the child.
Deep learning has emerged as a promising model for solving various problems, such as natural language processing, speech recognition, and visual recognition [19]. By definition, neural networks are a class of mathematical algorithms inspired by the brain's structure and natural functioning to classify information and make decisions. This approach works very well in the construction of prediction models using computer vision [20].
The concept of Smart healthcare is directly associated with the use of technologies in the medical area, where different actors are involved. Generally, this concept is implemented to support the prevention, diagnosis, monitor diseases and possible treatments [17]. Consequently, the smart healthcare concept uses technological advances to transform traditional methods into more efficient and personalized ones to improve results [17].
In the therapies carried out by health professionals to children with ASD, different activities are implemented so that children can imitate and recognize emotions. Professionals must show them the activity's meaning to better perform therapy [17, 18]. An alternative is the one we propose in this research, where a mirror is used to validate and show in their face the meaning of the emotion they are trying to imitate. Because a mirror is an essential element in people's daily routines, it is unlikely that children find it strange; therefore, they should feel comfortable developing activities.
For this reason and under the previously exposed context, we propose to develop a smart mirror for the support of facial emotion recognition therapies for children with ASD. We implemented a smart mirror according to the specifications presented in Fig. 1: (a) a 3 mm bidirectional mirror, (b) a Raspberry Pi 4 Model B of 4G ©, a refurbished LCD screen, and a (d) Raspberry Pi Camera Module v2.Fig. 1Physical components of the smart mirror [10]
Therefore, by joining the parts and pieces described above and together with the software that we detail in this research, it was possible to obtain concrete results associated with recognizing emotions based on computer vision. This proposal serves as an automated input for a professional who works in therapies with children with ASD to obtain real-time relevant information.
The use of technological tools in a therapeutic context with children with ASD generally turns out to be beneficial by increasing motivation, decreasing inappropriate behavior, and increasing their attention, which can be translated—on some occasions—into better learning compared to the traditional methods [21–25].
We first performed a bibliographic review related to our topic of interest to generate this proposal. Keywords (and their variations) were defined from the initial research autism spectrum disorders, ASD, autism, emotion recognition, expression recognition, emotion detection, human–computer interaction, computer-assisted technologies, adolescents, children. The search was carried out in the digital indexing engines Web of Science, Science Direct, SCOPUS and ACM. The methodology of this search can be found in the document by Pavez et al. [26]. Table 1 describes the details of the emotion recognition techniques, the technology used, and the article's contribution.Table 1The planning and control componentsRecognition techniqueAssociated technologiesMain findingsArtificial intelligence (convolutional neural networks)Desktop computer(Game) Fan et al. proposes that images used in the game should be in a familiar context, such as school, home, or park [24]Artificial intelligence (personal algorithm)Mobile devices(Game) Harrold et al. mention that children show a high level of motivation while playing. They had problems capturing emotion under specific scenarios related to lighting [21]Artificial intelligence (SVM and Logistic Regression Classifier)Portable motion camera (Google Glass) & Mobile device(IS) Voss et al., with his real-time facial recognition system, describes children as responding better to auditory and visual interactions [27]Artificial intelligence (SVM)Desktop computer(IS) Chu et al. propose a form of SVM-based facial emotion recognition with transition detection using a webcam [28]Artificial Intelligence (Machine Learning Algorithms: Bayes Network, Naïve Bayes, ANN, kNN, Random Forest, Decision Tree, SVM)Neurofeedback (Emotiv EPOC neuroheadset)(IS) Fan et al. uses electroencephalography (EEG) together with machine learning algorithms for the recognition of emotions in children. They conclude that using EEG it is possible to interpret the brain process associated with emotional expressions [29]Artificial Intelligence (Machine Learning Algorithms)Portable motion camera (Google Glass) & mobile devices(IS) Washington et al. mentioned that children tend to respond better to audio comments after the facial emotion recognition experiment than solely to visual feedback [30]Artificial Intelligence (DTW Classification Algorithm)Desktop computer(IS) Adams et al. concluded that based on a video containing facial expressions, the child tried to imitate what they were visualizing as it was happening. With this activity, the algorithm analyzes and compares both tables, providing feedback on the result [31]
In a previous research work [26], we have detailed a classification of studies related to emotion recognition in children with ASD. Figure 2 shows the distribution of the different types of technologies used according to the same results previously mentioned [26].Fig. 2Bubble diagram—Technologies, recognition technique and ages. Adapted from [26]
According to Fig. 2, it can be inferred that the use of "Desktop computers" is the most used device in manual emotion recognition, although other researchers use webcams for the same purpose. Other technologies that were identified are related to iOS or Android type mobile devices and RGB-D cameras, which are mainly represented by the Kinect V2 device. Also, wearable motion cameras, represented by the "Google Glass" and "Intel Realsense" devices, have been used in this regard. Less represented are neurofeedback and multimedia technologies.
Algorithms based on artificial intelligence stand out on the techniques used to support identifying emotions in children with ASD. This technique is used mainly in mobile devices, becoming the "Desktop Computers" the second most used option. Finally, concerning the general evolution of the research topic in recent years, starting from the year 2015, there is a constant upward trend.
About the technologies used, the advice given by the authors varies, depending on the Devices: The devices most used in research are desktop computers and mobile devices, mainly due to easy access and the massification of technology. Therefore, to reach a larger population, a recommendation would be to select this class of devices to continue creating new contributions in the area.Duration: The activity must have a defined duration depending on its complexity, but it should not exceed 10 min to avoid the boredom or monotony of the person performing the activity.Sound: The sounds should not be loud, preferably the use of calm and cheerful music.Graphics: We should consider using graphics with customizable environments to encourage the use of the activity according to the interests of the participants.Feedback: It can be visual or auditory, emphasizing the use of the latter as a narrative or aid element. Feedback through reading should be avoided.
For the smart mirror training process, we decided to evaluate 2 of the classic convolutional neural network (CNN) VGG 16 and ResNet50 [32, 33]. Regardless of the architectures or dataset used, the evaluation dataset was considered to inform the model's precision for each of the tests performed. Each model was trained from zero to 100 epochs on a Google Colab instance.
For both cases, when generating the model, a resize of 200 × 200 pixels is defined in the images. Stochastic gradient descent (SGD) is an optimization algorithm that is usually used on large data sets. By definition, the examples to be considered are drawn randomly from the sample and processes during the experiment [34]. For this reason, it was decided to use SGD with momentum at a learning rate of 0.01. Different optimizers were tested, including Adam, where SGD seemed to perform slightly better.
Under the same scheme, two additional activation layers of the RELU type were added with a dropout value of 0.5. In addition, to train the model with a more significant number of images and so the result obtained could be prepared to evaluate small variations at the entrance, the models were trained by increasing the data, mainly based on flipping the images horizontally to fulfill specific configuration parameters. The difference in the number of images takes approximately 2 to 4 h to train the model using the FER2013 dataset and about 10 min using the CK + dataset.
We worked with three psychologists specializing in ASD in the first instance regarding functional design. From these first meetings, it was possible to appreciate the traditional training and evaluation of emotions and how they could be improved through technology. From the list of functionalities obtained (described in the results section), those related to automatic recognition of emotions and feedback were implemented.
Subsequently, twelve health professionals who work directly with therapies for children with ASD used and validated the tool under the technology acceptance model (TAM) methodology [35]. There were also interventions in therapies, but they are not statistically significant to be analyzed in-depth and only served to validate functionalities.
Our smart mirror is made up of a mini-computer and a bidirectional mirror with an LCD screen attached. This screen shows relevant digital information without losing the main functionality of a traditional mirror (see Fig. 1): (a) a 3 mm bidirectional mirror, (b) a Raspberry Pi 4 Model B of 4G ©, a refurbished LCD screen, and a (d) Raspberry Pi Camera Module v2.
The proposed solution included a personalized welcome message for the child, reflected in the mirror, to obtain the child's attention. Subsequently, the health professional selected the emotion to work on in the session from the available angry, scared, happy, sad, and neutral (see Fig. 3). Once an emotion was selected, a set of associated images was displayed for the person to recognize the emotion projected by the mirror and later manage to imitate it to perfection. If this was not achieved, the mirror gave the possibility of moving to the next emotion or generating a pause (leaving this as a regular mirror), which gave the professional the opportunity of working with the face's image, reinforcing weak aspects. The smart mirror acts as a support in therapy, capturing and evaluating the child's emotion reflected in the mirror, generating training in emotional recognition and expression. Then, once the child's image was captured, only their facial area was extracted using the object detection method proposed by Viola-Jones [36]. Additionally, once the facial image of the child was obtained, it entered the classifier to decide what type of emotion it matched. This process is displayed step by step in Fig. 4.Fig. 3Smart Mirror. (1): Camera, (2) Main UI, (3) FrameFig. 4Description of the acquisition of the associated emotion from [10]
We are aware of the need to keep the information generated in each of the activities stored during our research. Considering that the smart mirror device's processing and storage capacity is relatively low, a technological platform has been implemented based on a set of REST-type web services that allow transparent interaction and centralization of data. However, there is still no graphical user interface for its administration. This will be one of the next steps we will carry out to provide specialists with patient progress visualization on a timeline.
Figure 5 shows the architecture considered for the implementation of the backend that allows the reception of requests by the Raspberry with the patient's information. According to the calls, GET and POST requests can be involved. Both are responsible for returning and storing data, respectively.Fig. 5Backend architecture diagram
This platform was designed in layers to allow future expansion that can be provided according to the users' needs, such as email notifications of the results to the patients' parents or a simple reminder of appointments.
Concurrency is an issue that can be important to consider in this type of platform, so NodeJs was considered a development technology for this backend. MongoDB was used as a database engine dynamic, expanding the data kept in this repository.
Regarding the results, we describe two sections as (i) From a software development perspective and (ii) an evaluation by specialists.
Due to the results obtained from the trained model and considering the data corresponding to the evaluation, the decision was made to use the ResNet50 architecture as a CK + dataset for the generation of the model included in the Raspberry Pi 4. In Table 2, we summarize the differences in the results. The differences appear not only due to the architecture but also due to the dataset used. In some cases, there is a difference exceeding 22% between different datasets but with the same architecture.Table 2Results using the evaluation data from [10]FER 2013CK + ResNet5072.3%89.9%VGG1670.9%93.3%
On the other hand, for both architectures, a maximum precision value was achieved when the training was close to 40 epochs. For example, the precision for the case of the VGG16 architecture and the FER2013 dataset tends to have very similar values between training and validation close to epoch number 40.
We used a confusion matrix to know the algorithm's performance in supervised learning (see Fig. 6). These results allowed us to compare the values obtained in each category using the evaluation data.Fig. 6Example of the confusion matrix using VGG16 with FER 2013 (left) and CK + (right)
On this occasion, we surveyed a group of twelve health professionals who work directly with therapies for children with ASD (see Table 3). All specialists are active members of the hospital and are part of the local treatment and rehabilitation center. They all have more than ten years of experience. These include seven psychologists, two occupational therapists, and three child psychiatrists.Table 3Demographic characteristics of research sample—IVariableClassification of variablesFrequencyPercentageGenderMale758.33%Female541.67%Age30–39 years old650.00%40–50 years old650.00%ProfessionPsychologist758.33%Occupational therapist216.67%Child and adolescent psychiatrist325.00%Knowledge about ASDHigh1191.67%Medium18.33%Low00.00%
Of the participants, seven are men, and six are women. All are between 30 and 50 years old, and according to their statements, 91.67% consider themselves to have high knowledge about ASD. Please note that this point was evaluated using a 3-point Likert scale and refers to a general knowledge of ASD.
Table 4 shows the instrument used to obtain information from the specialists on the following key external variables, perceived utility, perceived ease of use, and intent to use (see Fig. 7). This response to the technology acceptance model (TAM) was proposed by Davis in 1989. TAM [35] postulates that a person's most prominent beliefs about a system are mainly given by the perceived utility and the perceived ease of use, which can ultimately determine one’s attitude to using the system. Additionally, Davis hypothesized that attitude determines individual intention to adopt the system, which determines actual use of the system.Table 4Applied surveyIDQuestionsExternal variables1I have full knowledge on the subject of this research2I have conducted theoretical and/or experimental research related to this topic3I have experience from a professional activity related to this topic4I have done analysis of specialized literature and publications of national authors5I have done analysis of the specialized literature and publications of foreign authors6I have knowledge of the current state of the problem in my country and abroadPerceived utility7In general, I think that the EMO-MIRROR proposal is useful8I think this method would improve the accuracy of emotion recognition in people with ASD9In general, I think that this proposal provides an effective way to include emotions in people with ASD10The use of this technology would improve my work in evaluating emotions in people with ASDPerceived ease of use11In general, I think that the EMO-MIRROR proposal is easy to use12The EMO-MIRROR proposal is easy to understand13The instructions delivered on the EMO-MIRROR are clear and easy to understandUse intention14I will use this proposal if I have to assess emotions in people with ASD in the future15I find it easy to apply the EMO-MIRROR technology in my professional practice16I intend to use this technology in the futureEmotion recognition17I consider that the EMO-MIRROR manages to recognize the emotion of happiness18I consider that the EMO-MIRROR manages to recognize the emotion of fear19I consider that the EMO-MIRROR manages to recognize the emotion of anger20I consider that the EMO-MIRROR manages to recognize the emotion of sadness21I consider that the MIRROR manages to recognize the state of emotional neutralitySmart Mirror’ Feedback22I consider that the EMO-MIRROR provides the necessary feedback for the emotion of happiness23I consider that the EMO-MIRROR provides the necessary feedback for the emotion of fear24I consider that the EMO-MIRROR provides the necessary feedback for the emotion of anger25I consider that the EMO-MIRROR provides the necessary feedback for the emotion of sadness26I consider that the EMO-MIRROR provides the necessary feedback for emotional neutralityFig. 7TAM technology acceptance model
The exercise proposes an intentional, non-probabilistic sample calculation performed by two professionals using the Delphi method [37]. These professionals who made the instrument did not participate in the final evaluation.
Table 5 shows the questions Q2-Q6 that seek to refine the profile of the specialists. These were evaluated using a 3-point Likert scale. In summary, an average of 2.4 can be observed on "Innovation Experience" that relates to their theoretical and practical experience in the topic studied. A value of 2.4 is observed in the professional experience question, and finally, a general average of 2.8 on good practices related to ASD.Table 5Demographic characteristics of research sample—IIP1P2P3P4P5P6P7P8P9P10P11P12DimensionAVGSDQ2232232322322Innovation Experience2,400,47–Q3233132331322Professional Experience2,400,76–Q4331333313333Knowledge and best practices2,600,762,80Q53333333333333,000,00Q63323333233332,800,38
The rest of the questions (Q7-Q26) were evaluated using a 5-point Likert scale. Table 6 shows the questions related to categories of “Perceived Utility, Ease of Use, and Intent to Use.” Each of these questions contains 5-level responses. The lowest level is 1 with the option of "Totally disagree." The highest level is represented by 5 with the option "Totally agree." In Fig. 8, we describe the graph with the average score per question.Table 6Results from applied surveyP1P2P3P4P5P6P7P8P9P10P11P12AverageSDGeneralAverage Perceived UtilityQ75455554555554.830.394.35Q85345454454444.250.62Q95335553355444.170.94Q105245454454444.170.83Ease of UseQ115545555455554.830.394.83Q125545555455554.830.39Q135545555455554.830.39Use IntentionQ145245553455444.250.974.17Q154435544355444.170.72Q165235553355444.081.08Emotion RecognitionQ175355553555554.670.784.42Q185325553355444.081.08Q195345553455444.330.78Q205345553455444.330.78Q215355553555554.670.78Smart Mirror’ FeedbackQ225355454554444.420.674.38Q235335454354444.080.79Q245355453554444.330.78Q255355453554444.330.78Q265355554555554.750.62Fig. 8Average question score
Concerning the dimension of "Perceived Utility," the results show that the respondents mostly evaluated the Smart Mirror as useful (M = 4.35). Their responses were mostly uneven when asked whether this technology would improve their work in evaluating emotions in people with ASD (M = 4.17; SD = 0.83). Regarding the dimension of "Ease of Use," most respondents answered that the proposal is easy to use (M = 4.83). A difference was noted in their opinions regarding the "Use Intention," achieving a lower score for the question associated with whether they intend to use this technology in the future (M = 4.17). The latter can be explained by the different roles of the professionals who participated in evaluating the tool. For example, the sessions associated with psychologists are different from those interventions carried out by occupational therapy professionals.
According to the perception of the respondents regarding the categories of "Emotion Recognition" and "Smart Mirror's feedback," it is noted that the emotion with the lowest score was associated with fear (Q18) (M = 4.08; SD = 1.08). However, this result has a standard deviation of 1.08, probably related to the maturation stage of the recognition algorithm.
By contrast, the one that obtained the highest percentage of recognition was neutrality (Q21) (M = 4.75; SD = 0.62) followed by happiness (Q17) (M = 4.67; SD = 0.78). Related to the mirror's feedback, the emotion associated with fear obtained the lowest score (M = 4.08; SD = 0.79), while neutrality obtained the highest score (M = 4.75; SD = 0.62).
Some of the comments from health professionals regarding the smart "The proposal stands out in terms of innovation, which would be attractive to the patient, coming from "common" care. However, I would add raw emotions of amazement and displeasure to complete the first-order or raw emotions.”**"It would be good to include more emotions, such as surprise or disgust."
Some comments provide points to consider on using the "I would consider the shyness or embarrassment that a child with ASD can have when performing a task of this type. It should not be easy for everyone to place faces in the presence of another and, in turn, look in a mirror."**"The mediation of a trained therapist is important because the pure mirror is insufficient."
Other comments related to “This technology can advance to a phase of greater specialization using an emotion response scale.”“Different sounds must be added to complete the experience.”“A letter with the instructions.”**“Feedback could include where one failed to put the facial expression.”
Regarding the precautions that should be considered when using the mirror in children with ASD:*“I would consider the shyness or embarrassment that a child with ASD can have when performing a task of this type. It should not be easy for everyone to put faces in the presence of another and, in turn, look in a mirror.”*In a scenario where the person is learning the gesture for the emotions, it is essential that the expression be "natural" (not hypomimia or cartoonish).”
In this article, we presented a technological prototype in smart healthcare using artificial intelligence. This smart mirror supports therapies of children with ASD in recognizing emotions and the capacity for emotional expression. We analyzed other initiatives that try to solve the same problem. This previous study has given us a starting point for the design and construction of this proposal [26].
Regarding the (i) In general, there is no single formal technology for interacting with people with ASD, but there is a trend toward use in those devices that are easier to access, such as desktop computers and mobile devices. (ii) The primary recognition technique used is artificial intelligence. Along with the above, the most used classifier by some authors is the SVM [28, 38–40], reaching a recognition rate of 98.54% [28].
The smart mirror achieves the necessary interaction between the child and the recognition software. Together with an information system, the results obtained in each of the therapies are saved and can measure progress.
Regarding the classification model The different tests carried out to achieve the highest level of assertiveness among some of the most used architectures when working with images are evident, thus reaching 93.3% using the CK + dataset and the VGG16 architecture. Advances in technology and new techniques are expected to enable related research to continue to advance over time.
The professional ASD specialists evaluated the prototype to give their impression on the use of technologies in therapies and, more specifically, on the use of a smart mirror as part of their therapies for training emotions. The evaluation results showed a high level of utility and ease of use, which motivated professionals to use the prototype in their therapies.
The smart mirror, in addition to favoring the professional, unlike other tools such as the use of cards that are used today [6], those that have more significant manipulation by the health professional and the ASD user, exposing them to contact with COVID-19 contaminants.
In addition, because the training images can be modified, incorporating photos of emotional expressions of family members of children with ASD provides an even more individualized intervention. This generates a higher level of motivation as the patient is in an everyday environment with images of familiar people.
It is hoped that a mirror evaluation with children with autism spectrum disorder and without the condition (case–control study) will be performed in future work. Due to COVID-19 issues, it was not possible to carry out this intervention. In addition, we wish to incorporate narratives into the mirror that generate spontaneous emotions in children, complementing what exists associated with imitations of images.
In future work, we will use a user-centered approach to improve the quality of the interfaces and solve any errors [41, 42]. It should be noted that usability tests have not been carried out, prioritizing improving the emotion recognition algorithms. We hope to obtain results in a natural context with monitoring over time, using the different methods and techniques collected in the study and continue to test new technologies in the area [43, 44].