Authors: Syed Abu Huraira Hussain, Imran Raza, Syed Asad Hussain, Muhammad Hasan Jamal, Tauseef Gulrez, Ali Zia
Categories: Article, Brain-computer interface (BCI), Electroencephalogram (EEG), Mental state detection, Mobility solution for quadriplegic patients, Health care, Medical research, Mathematics and computing
Source: Scientific Reports
Authors: Syed Abu Huraira Hussain, Imran Raza, Syed Asad Hussain, Muhammad Hasan Jamal, Tauseef Gulrez, Ali Zia
Brain-computer interfaces (BCI) provide a mobility solution for patients with various disabilities. However, BCI systems require further research to enhance their performance while incorporating the physical and behavioral states of patients into the system. As the principal users of a BCI system, patients with disabilities are emotionally sensitive, so a BCI device that adaptively adjusts to the psychological effects of the patient could provide the foundation for refining BCI applications. This paper focuses on the collection and realization of human electroencephalogram (EEG) signals data, obtained as a response to different psychological effects of sound stimuli. Filtration and pre-processing of the data set are achieved using the frequency-based distribution of EEG signals. Different machine learning tools and techniques are evaluated and applied to abstracted powerbands of psychological signals. The experimental results show that the proposed system predicts mental states with an average accuracy of 74.26%. In addition, an automated BCI system is developed to control an electric wheelchair (EPW) while responding to the mental state of the user with a contingency mechanism. The results show that such a system could be designed to make BCI systems more reliable, safe, adaptable, and responsive to emotions for sensitive paralytic patients. The system also shows a satisfactory True Positive Rate (TPR) and False Positive Rate (FPR) with an average time of 8.4 s to generate the interpretable brain signal from the user.
The interface of actuating devices through brain signals acquired from the Electroencephalography (EEG) modality technique has gained a lot of attention among researchers in the last decade^1,2^. Common applications in this domain usually assist quadriplegic neurally damaged patients who are not able to move their body parts physically^3,4^. The primary goal of brain-computer interface (BCI) is to translate the intentions of the person from the brain signals and perform certain computing tasks, to control electric-powered wheelchairs (EPW), actuators, and other software or hardware devices. The most common techniques to acquire EEG signals can be classified into two invasive and non-invasive. The invasive techniques are those in which the electrodes are infiltrated into the scalp, and brain activity is analyzed. The signals acquired from the non-invasive techniques are quite handy as they are acquired from the scalp, and need a lot of pre-processing and classification to extract the signals of consideration before mapping the actual commands^5–7^. During acquisition, if the signals are amplified, noise is also amplified. Hence, the extraction of required signals or peaks of concern from the low-power signals is a challenge, and traditional signal processing techniques cannot be used for EEG signals^8^. Many systems have been proposed in the past that use BCI to assist physically disabled people. One of the major applications of BCI for quadriplegic patients is to provide ease of EPW control through brain signals. Patients suffering from Quadriplegia have no motor movements in all their limbs; though the level of muscle movement varies from the degree of disability or neural damage; such patients could be the potential users of BCI-controlled hardware systems to perform everyday tasks.
BCI-controlled hardware systems should be designed in such a way that they retain the sensitivity and safety measures of the patients, only then it could provide the assistance to perform everyday tasks for such extremely sensitive patients^9^. The accuracy of the classification techniques and transformation of brain signals into commands must be high as any miscalculation could lead to a disastrous situation and put the patient into a clinical emergency. Integrating additional assistive systems could greatly help in improving the safety of the system and managing complex tasks without compromising the distinction provided through the BCI^10^.
This paper presents a robust machine learning (ML) approach to analyze the EEG data using the One-vs-Rest logistics regression (OvR-LR) after abstracting the powerbands from each channel. The system predicts the mental state of the user and uses the trained model to provide the contingency mechanism for patients during their mobility tasks. Moreover, this research work presents an intelligent mental-state integrated EPW control through BCI to provide a novel mobility system for quadriplegic patients. Complete control over the hardware with sentiment-based monitoring will increase the sense of independence, boost the morale of the disabled person, and motivate the user in rehabilitation^11^. The classification of mental states will eventually trigger the contingency mechanism intelligently. The personalized solution will not only provide a secure and smooth drive but will also assist the patient intelligently in navigation and speed control, depending on the behaviour of the patient. The mental state monitoring will also provide the prevention of increased workload on the patient, as the additional workload could lead to fatigue and restlessness^12,13^. In the scenarios where the user’s mental state is at unrest, or he/she is neurally unstable, the system will automatically switch its navigation control to the joystick control, as the user might not be able to generate the commands from the brain anymore because of the stressful situation.
The main contribution of this research work is the design of a system that analyzes the mental state of the patient during vehicular movement (e.g. EPW). This system introduces a brain signal-based mobility mechanism adapted for patients with neural damage that results in quadriplegia, offering a novel approach to enhance their autonomy and mobility. The significance of this work lies in its potential to empower quadriplegic patients who suffer from paralysis in all four limbs and are heavily reliant on external assistance for daily activities, with a more independent and secure method of mobility. The research delineates a comprehensive system that not only monitors but also interprets the mental states of the patient to facilitate movement, thereby bridging a critical gap in assistive technologies for individuals with severe motor impairments. The contributions of this research at the module level are summarized as Innovative mental state classification mechanism: The paper introduces a new method for classifying mental states using Electroencephalogram (EEG) signals. This approach involves the collection of data through psychiatric and auditory-based experiments, followed by the analysis of sentiment-based mental states. This process includes the filtration and abstraction of power bands from the data of each channel. The mental state predictions are made using the One-vs-Rest Logistic Regression (OvR-LR) algorithm, showcasing a novel application of machine learning techniques in the analysis of brain signals.Enhanced security and adaptability in mobility systems: In consideration of the patient’s sensitivity, the system is engineered to offer a secure and adaptable brain-controlled mobility solution that surpasses traditional Brain-Computer Interface (BCI)-based EPWs. The system utilizes a wireless communication channel to acquire signals from the headset, eliminating the need for user interaction in converting brain signals into control commands. This design principle underscores the importance of creating a user-friendly and efficient interface for quadriplegic patients.Integration of speed control with mental states: To further the autonomy and morale of patients, the system incorporates a speed control mechanism that is responsive to the patient’s mental state. By continuously monitoring the patient’s mental condition, the system dynamically adjusts the speed of the EPW. When the patient exhibits feelings of safety and excitement, the EPW operates at a higher speed; conversely, it navigates at a reduced speed under less favourable mental states. This feature represents a significant advancement in personalized assistive technology, aligning the functionality of the EPW with the emotional and psychological well-being of the patient.The organization of the remainder of this paper begins with “Related work” Section, which conducts a literature review, offering a comparative analysis of existing systems and highlighting their limitations. “Proposed system” Section introduces the proposed system. The experimental design, along with the results and discussions, are comprehensively presented in “Mobility experimental design” and “Results and discussion” Sections, respectively. The paper culminates with “Conclusion” Section, where conclusions are drawn and the research is succinctly summarized.
To enhance the reliability and efficiency of EEG-controlled EPW, the Extended Brain-Computer Interface (EBCI) approach with the help of multiple sensors and EEG controls is adopted in^14^. However, there is much work to do to improve the reliability and autonomy of quadriplegic patients, not only by providing navigation but also through the speed control system and personalization of the system^15^. With a thorough literature review, previously implemented systems can be categorized into the following four different types based on their EEG acquisition techniques and the provision of navigation (i) visually evoked stimuli (P300 /SSVEP), (ii) motor imager (MI), (iii) extended/hybrid control systems, and (iv) expression based control.
Maksud et al. provided a mechanism that uses destination mapping, while the user moves around to generate a virtual map in which the user reaches a destination through a smart interface based on EEG signals^16^. The user must blink for some time on the desired location, which is shown on the visual interface, to execute the command. A phenomenon of steady-state visually evoked potential (SSVEP) for the navigation control of the EPW is used in^17^. In this design, the EPW is mounted with four LEDs flickering at 7 Hz, 9 Hz, 11 Hz, and 13 Hz frequencies, respectively. The EEG signals generated as a response to focusing on each flickering bulb are recorded and mapped for the control commands of the EPW. Chung-Kang et al. proposed a system based on the prototype of an EPW having a dual control system, i.e., real-time control through EEG signals acquired by EMOTIV and automated guide using GPS and Google Maps^18^. They have used the software interface to control the EPW for both real-time as well as automated guided navigation. In^19^, Qin et. al. proposed a system based on SSVEP for the navigation of EPW. The software interface consists of left, right, forward, stop, acceleration, and deceleration buttons modulating at frequencies of 6.45, 8.32, 10.61, 11.55, 13.51, and 14.25 Hz, respectively. Some researchers designed the navigation control system based on P300 having threshold-free switches^20^. In that system, the healthy users took a root mean of 4.25 s to execute the control command. Although the results are better, the researchers compared the classification accuracy only, without validating the adaptability and practicality of the system. A system based on the fusion of information of eyeball movement and SSVEP signals has been projected to achieve high accuracy and efficiency of EPW control commands^21^.
The system developed by Sethi and Aartika consists of data collection, input, interface, and output modules^22^. They proposed multiple data inputs for controlling the EPW and introduced the idea of controlling home appliances with EEG signals and Android mobile over the IoT. Chen et al. provided two integrated mechanisms for the facilitation of elderly and paralytic patients through EEG acquired from EMOTIV EPOC^23^. One is the navigation mechanism of EPW, and the other is the robotic arm or manipulator to pick up objects and materials suitable for patients having upper limb locked down syndrome (LDS). The laser assistance with the EEG signals has been introduced in another system for EPW navigation by controlling the laser pointer through EEG^24^. The system supports the user to set the direction of the laser pointer through the EEG signals generated by the brain. This is based on a certain threshold of the user’s attention level for every movement to navigate the EPW. By distinguishing between sustained and brief motor imagery of left/right hand. Wang et. al. proposed a system for controlling the EPW for quadriplegic patients^25^. The technical matrix proposed in^26^ is used for the performance evaluation of the system. The system is rated based on deviation from the optimal path and time to reach the destination.
A facial expression-based design to control the EPW for paralytic patients through EEG signals using only one channel has been proposed in^27^. The system analyses the EEG signals on intentional eye blinking and double blinking to map the control commands of left, right, and forward movement. A system is proposed by Md Fahim et al. that maps different facial expressions with the navigation controls of the EPW^28^. The EEG signals are recorded against facial expressions of eye blinks clench and eyebrow raise for the EPW control. Similarly, a hybrid approach using EEG and Electro-Oculogram (EOG) signals generated because of the eyeball movement has been proposed in^29^.
The extended system proposes a design mechanism of EPW control using collective data from EEG as well as the gyroscope mounted on the EPW^30^. In another system, Voznenko et al. proposed simultaneous control of the EPW by a patient as well as a specialist operator^31^. The patient can control the EPW through EEG, while the operator interfaces with the remote tracking through the application. In the study, Kim et al. identified the limitations in the current P300, SSVEP, and MI approach for EPW control^32^. According to him, the P300 and SSVEP approach standalone is not suitable for persons having a neural disability. The researchers designed a hybrid control system that involves both P300 and MI to achieve a higher number of navigation commands^33^. Prashant et al. mapped the intentional eye blinking of the users to control commands of an EPW^34^. The logical integration is done by using the intentional double blink coupled with a software interface. Kevin Warmerdam proposed a lightweight mechanism to extract meaningful information from a single-channel NeuroSky headset rather than analyzing and processing all channels of raw data^35^. For controlling different devices and applications the system provided a fusion of EEG and Electromyography (EMG) signals, which are recorded from muscle movements, for accurate feature extraction.
Mai et al.^36^ proposed a hybrid BCI combining SSVEP and EOG. The proposed hybrid BCI significantly outperformed the conventional SSVEP-based BCI and the results showed that the combinition of SSVEP and EOG signals in the proposed hybrid BCI significantly diminished the transition time in SSVEP-based BCI and achieve continuous, fluent control. Choi et al.^37^ proposed a brain-controlled mobility system to analyze real-time neural signals obtained from motor imagery. The proposed system contains shared control capabilities utilising EEG signals and surrounding environment information to improve navigating performance without precise control from the driver. Using a wheelchair with light detection and ranging (LiDAR) and inertial measurement unit (IMU) sensors, the participants in the conducted study drove wheelchair with and without proposed shared control approach using either proposed brain-controlled system or a keyboard in a physical environment. Results showed that proposed shared control approach was more feasible than the asynchronous BCI-based system.Table 1Comparison of previously proposed systems.ReferencesBCI techniqueBCI DeviceNo of control commandsSpeed controlContingency systemFeedback continuous learning^16^SSVEP/P300Custom made5✘✘✘^17^Neuro sky4✘✘✘^18^Ultra cortex4✘✘✘^19^Custom made6✘✘✘^20^NuAmps6✘✘✘^21^Emotiv epoc5✘✘✘^22^MIEmotiv epoc+4✘✘✘^23^Emotiv epoc5✘✘✔^24^Neuro eky4✘✘✘^25^Cognionics6✘✘✘^27^Facial expressionsTGAM Kit4✘✘✘^28^Emotiv epoc+4✘✘✘^29^Neuro eky13✔✘✘^30^Hybrid/extended BCIBrain wave4✘✘✘^31^Emotiv epoc4✘✘✘^32^Custom made6✘✘✘^33^Custom made11✔✘✘^34^Neuro sky5✘✘✘^35^Neuro sky7✘✘✘
Table 1 shows the comparison of existing systems, and it is evident that most of the systems do not have a mental state-based contingency mechanism or feedback system. As the principal users of the system are quadriplegic patients and the brain signals of every patient also vary, even the behavior of the same person may change over time, this changing nature of the mental activity will affect the input of BCI and can create difficulties in generating the exact mental commands. To maintain system functionality and reliability, a continuous mental state monitoring mechanism is needed. Table 1 shows that the systems with hybrid mechanisms can map more control commands with greater accuracy because of additional assistive hardware-based systems. Moreover, because of the computational limits of the brain signals, only those systems have integrated the speed control system which uses the hybrid approach or utilizes additional hardware. To ensure patient safety on the route, there should be a contingency plan depending on the condition and severity of the patient.
While P300 and SSVEP methods requiring a visual interface, can be effective for certain applications due to their higher accuracy, this study focuses on a system designed specifically for users with severe disabilities, such as quadriplegic patients. These individuals often experience limited mobility and disturbed viewing angles, making the use of visual interfaces impractical or less effective. As a result, comparing classification accuracy alone, without considering usability and adaptability for these patients, would not fully capture the benefits of the proposed system. The proposed system is designed to cater to users who cannot rely on visual stimuli by continuously monitoring and learning the user’s mental states. This allows the system to adapt over time and make personalized decisions based on the user’s unique cognitive patterns, rather than solely relying on visual inputs. This adaptability and personalized decision-making process, sets our system apart from visual-based interfaces like P300/SSVEP.Fig. 1Components flow of BCI based EPW control system.
The solution presented in this research work is personalized and patient-specific, with the capability to adapt according to the patient’s behavioral needs. The presented system, shown in Fig. 1, is adaptive and can be configured based on users’ EEG signals (representing the behavioral state) during personalized training sessions. Saline-based wet electrodes are used in EEG for mental state analysis and are interfaced with EPW control. For the prediction and classification of EEG signals, we test our training data on four different ML algorithms, i.e. Optimized K-nearest neighbours (KNN), Support Vector Machine (SVM), and Shallow and Wide Neural Networks (WNN). WNN is picked as a representative classifier based on its better classification rate both during the training and testing regime of the EEG signals. This is because, unlike deep neural networks that have higher number of hidden layers, WNN comprises less number of hidden layers with high number of neurons per layer which makes it practical for the data at hand. Moreover, to realize the brain-controlled mobility mechanism, the EPW, with a specially designed interfaceable controller, has been developed. In the EPW there is a specially designed compartment at the back of an EPW that inhabits the programmable controller and power bank. To achieve the goal of providing ease in navigation for paralytic patients, a system is designed that can be divided into three major interdependent (i) signal acquisition and classification, (ii) mental state data gathering and ML analysis, and (iii) EPW interface. The flow of the components is shown in Fig. 1.
Before the data collection process, written informed consent was obtained from all participants and the study protocol was approved by the ethical committee of COMSATS University. All experiments were performed in accordance with relevant guidelines and regulations. Informed consent was obtained from all participants for publication of identifying information/images in an online open-access publication.
The brain emits contrasting EEG signals for every activity, whether we think, see, sense, or perform any task. In the presented design, these signals are captured using a commercial-off-the-shelf device, EMOTIV EPOC+. The headset follows the international standards of the 10/20 system for the position of the electrodes on the scalp, as shown in Fig. 2. Here, letters F, C, T, P, and O stand for Frontal, Central, Temporal, Parietal, and Occipital portions of the brain/skull, respectively, with numeric suffixes for the reference of the channel^38^.Fig. 2Emotive EPOC+ electrodes scalp position.
EEG signals are time series signals that are complex as they are generated due to the simultaneous response of many activities. Desired signals or features can only be extracted by pre-processing and classification of the input signals. To map the navigation control, we are interested in Motor Imagery (MI), i.e., signals captured that are generated against the thoughts of motor movement. 14-channel EMOTIV Epoc+ is used to read the brain signals, which acquire the EEG signals at the rate of 256 samples per second, having values of each channel in micro-Volts (mV). The patient must wear the Epoc+ headset to wirelessly transmit the brain signals to the software on the PC. These acquired signals on the PC are then processed to translate them to control commands and transmit them to the integrated controller of the EPW. To improve the quality of the collected EEG signals, we implemented several preprocessing steps. First, we applied a bandpass filter with a range of 13–30 Hz to remove noise and artifacts outside the frequency band relevant to motor imagery tasks. Additionally, we used Independent Component Analysis (ICA) to effectively remove common artifacts such as eye blinks and muscle activity. Lastly, signal normalization was performed to ensure consistency across different sessions and patients. Activities recorded from the frontal lobe give information about the mental and emotional state that will be analyzed for stress detection, while the central region gives the data useful for the behaviour and movement of the muscles^39^. Brain signals from each electrode are recorded using a customized program written in Python 3.8 and the Emotiv native application software during the experiments. From the cortex API, the power band of alpha, beta, theta, and gamma waves are recorded and labelled as the data acquired in the relaxed mental state of the user. The API provides the power bands of each channel, but for this study, data from only the frontal, occipital, and central regions are considered as these regions are associated with the behaviour and mental state. The beta waves in the frequency range of 13–30 Hz are used for the learning phase, as they represent the behaviour and action-related activities. This pre-filtration of the data also increases the efficiency of the training algorithm in terms of processing.
The study was conducted at the Pakistan Society for the Rehabilitation of the Disabled (PSRD) (https://psrd.org.pk). Due to patient privacy restrictions imposed by PSRD, the data remains private. However, for the purposes of this experiment, data was collected from 64 patients (62.5% male and 37.5% female), with a mean age of 16 years and a standard deviation of 7.5 years. None of the participants had prior experience with brain-computer interfaces (BCI). The study spanned two years, from 2021 to 2023.
Generally, paralytic patients are sensitive, and a small environmental pressure or delay in the system can trigger nervousness and anxiety. Repetition of the same task can lead to fatigue^40,41^. Additionally, continuous driving of any vehicle or EPW can cause fatigue or mental pressure, which would affect performance while driving^42,43^. Moreover, environmental distractions and other disturbances cause mental stress that can eventually lessen the ability of the brain to work on its full potential^44,45^. To tackle such scenarios, a contingency mechanism (CM) is introduced in the system. The idea is to continuously observe mental states from the processed EEG signals of the patient, and whenever the system detects the stress level to go beyond a certain threshold, the system triggers the contingency plan. As mentioned earlier, for simplicity’s sake, the data is collected for three different but mentally dominant states of the EPW user, i.e. relaxed (positive), stressed (negative), and contented (neutral). The mental state signals are captured while providing two distinctive stimuli to the EPW user. A relaxing sound stimulus is provided to the user to collect EEG data for a relaxed or positive state, whereas a noise sound produces a stressed or negative EEG signal. A no-sound signal stimulus ensures the user’s neutral or contented state EEG signals. The neural activity during each phase of the experiment is significantly different, as shown in Fig. 3. These mental state images are captured by providing a stimulus to the user to segregate EEG data for the three different psychological states using the Emotiv BrainViz software version 1.1 (https://www.emotiv.com/collections/software/products/emotiv-brainviz) during the experimentation.Fig. 3Mental state images captured during stimulus provided to the user to segregate EEG data for three different psychological (a) normal state, (b) relaxed state, and (c) stressed state. Images are captured using the Emotiv BrainViz software version 1.1.
The brain signals received from the Emotiv headset are stored on a laptop through an application, which are then processed and classified against MI-based thoughts for each specific user. Once the signals are classified and stored for every control command, the application allows the user to interact with the EPW directly through brain signals. One module of the application maps the EEG response to the control commands and another module forwards those commands to the motors of the EPW. Table 2 shows the mapping of the EPW control commands and MI of the patients.Table 2Command mapping.Motor imagery Based FunctionsWheelchair commandsPushForwardRotate leftLeftRotate rightRightNeutralStopMental state (excited)Speed upMental state (relaxed)Speed down
For the real-time translation, the application uses the Cortex API which gives the real-time stream and translates the commands according to the model described below. The API provides the functions for establishing the connection and subscribing to the data from the headset.
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ U = { neutral, push, pull, rotateLeft, rotateRight}
undefined