Hybrid FCNN-LSTM Approach for Autism Spectrum Disorder Classification

1. Introduction

A complex Neuro Developmental Disorder (NDD) named Autism Spectrum Disorder (ASD) have the following symptoms, repetitive behaviours or limited interests, and ongoing challenges with social interaction and communication according to Eslami et al. [1]. Typically diagnosed in early childhood, ASD varies widely in its manifestation, ranging from mild to severe forms that can significantly impair daily functioning. Fig. 1 shows the rising prevalence of ASD, currently affecting about 1 in 54 children. The urgent need for accurate and ED was noted by Firouzi et al. [2]. For better results and prompt intervention, these accurate and ED are essential.

Subjective evaluation is a major component of traditional diagnostic techniques for ASD, including clinical observations and standardised tests like the Autism Diagnostic Observation Schedule (ADOS). Fig. 1 these methods can be time-consuming, and their accuracy is often influenced by the clinician’s expertise Plis et al. [2]. Furthermore, the heterogeneous nature of ASD, with its broad spectrum of symptoms, poses additional challenges for reliable diagnosis. Hossain et al. [3]. The use of data-driven methods, especially Machine Learning (ML) and Deep Learning (DL) techniques, has become more and more popular. The purpose of these ML and DL algorithms is to increase classification accuracy and improve the diagnostic procedure.

According to Wang et al. [4], DL has the capacity to automatically learn and extract spatial features from complicated datasets, so, DL models particularly Convolutional Neural Networks (CNN) have demonstrated significant potential in Medical Data Analysis (MDA) in the past few years, and it is demonstrated in Fig. 1. In order to find distinctive features associated with ASD, CNNs have been effectively used in research, especially in the processing of neuroimaging data, such as Functional Magnetic Resonance Imaging (FMRI), and behavioural assessments Sherkatghanad et al. [5]. For spatial FE, the CNN is effective, but it is not specifically designed for capturing temporal correlations. The progression and unpredictability of the ASD symptoms over time can be understood by CNN, so CNN is crucial.

According to Xu et al. [6], the Long Short-Term Memory (LSTM) networks executes well in the following 2 fields like long-term dependencies and sequential data modelling. The LSTM are widely employed in applications that using Time-Series (TS) data. Those applications are language processing and Speech Recognition (SR). The temporal data like developmental stages, and behavioral patterns over time can be effectively analyzed by the LSTM networks, so, this LSTM networks are effective in the context of ASD. Insights regarding the temporal dynamics of the disorder was also offered by this LSTM. Specifically, LSTM alone cannot be able to handle the spatial complexities in multimodal ASD, and it is stated by Xu et al. [7]. This study focusses on improving the field of ASD classification, so, novel Hybrid DL (HDL) model was suggested. This HDL model helps in overcoming the limitations of the current methods. Here, more accurate and reliable ASD diagnosis are the main aim of this study. For classifying the ASD, a Hybrid FCNN-LSTM method was suggested in this study, as it integrates the benefits of CNN and LSTM. The temporal sequence modelling abilities of LSTM and the spatial FE abilities of CNN was utilized by the hybrid model. Huang et al. [8] offer a comprehensive analysis of data related to ASD. This approach captures the multifaceted nature of ASD by using the UCI repository dataset. This dataset includes behavioural and physiological data [9]. This suggested Hybrid FCNN-LSTM model aims to enhance the classification accuracy by combining the spatial and temporal data. A more reliable diagnostic tool for ASD was also offered by this suggested model. The remainder of this study is arranges in the following way: Section 2 reviews the related work on ASD classification using DL methods. Section 3 describes the architecture and implementation of the Hybrid CNN-LSTM model. The experimental results are presented in Section 4, performance of the model in comparison to existing methods are presented. And finally, Section 5 discusses the implications of these outcomes, highlighting the ability of the Hybrid FCNN-LSTM approach for clinical applications in ASD diagnosis and future research directions.

2. Related Works

A technique for classifying ASD patients against control subjects utilising both fMRI and structural MRI data was presented by Rakić et al [10]. Both structural and functional processing pipelines leverage volumetric correspondences of Grey Matter (GM) volumes among cortical parcels and functional connectivity patterns between brain regions as features. The classification network combines supervised and unsupervised training of MultiLayer Perceptrons and Stacked AutoEncoders (SAE). 817 cases from the multisite international Autism Brain Imaging Data Exchange I (ABIDE I) dataset are subjected to quantitative analysis. Using an ensemble of classifiers, it achieved a classification accuracy of 85.06 ± 3.52% with 368 ASD patients and 449 control subjects.

A thorough evaluation of the earlier research conducted since 2011 was provided by Liu et al. [11]. In this research, the fMRI data of autistic individuals and Typical Controls (TCs) were analyzed using ML techniques.Feature construction from raw fMRI data, Feature Selection (FS) techniques, ML techniques, criteria for high classification accuracy, and crucial conclusions were all covered in the comprehensive process. Classification accuracies ranging from 48.3% to 97% were achieved by applying several ML techniques to fMRI data collected from various locations. This allowed for the identification of informative brain regions and networks. After careful examination, it was shown that research involving task-based fMRI data, a single dataset for a selection principle, efficient FS techniques, or sophisticated ML techniques typically had high classification accuracies. Particularly in the last four years, advanced DL and the multi-site ABIDE dataset have emerged as research topics. Advanced FS and ML techniques in conjunction with multi-site datasets or accessible task-based fMRI data may look like an effective diagnosis tool for ASD in the future.

High classification performance is achieved by using a hybrid lightweight Deep FE (DFE), as suggested by Baygin et al. [12]. A large EEG dataset containing signals from both normal controls and autistic patients was used to design and test the system. i) This work presents a novel signal to image conversion model. The EEG signal's features are extracted in this work using the One-Dimensional Local Binary Pattern (1D_LBP). The Short Time Fourier Transform (STFT) is then performed using the generated features as input. Spectrogram images are produced by STFT. ii) A combination of pre-trained MobileNetV2, Shuffle Net, and Squeeze Net models is used to extract the deep features of the output spectrogram images. The hybrid deep lightweight feature generator is the name of this technique. iii) For feature ranking and FS, a two-layered Relief method is employed. iv) For automated autism detection, a 10-fold Cross-Validation (CV) technique is used to construct deep classifiers, which are fed the most discriminative features.

An automatic identification of ASD using CNN and a brain imaging dataset was presented by Sherkatghanadet al. [5]. The most prevalent resting-state fMRI data from a multi-site dataset called the ABIDE. This ABIDE was used to identify ASD patients. Using patterns of functional connectivity, the suggested method successfully distinguished between people with ASD and control subjects. Using the ABIDE I dataset and the CC400 functional parcellation atlas of the brain, the experimental results show that the suggested model can accurately identify ASD with an accuracy of 70.22%. Additionally, the CNN model is computationally less demanding because it used fewer parameters than the State-Of-The-Art (SOTA) methods. The created model can be used to prescreen patients with ASD and is prepared for testing with further data.

Based only on the patient's brain activation patterns, Heinsfeld et al. [13] suggested DL algorithms to identify ASD patients from huge brain imaging datasets. Researchers looked into brain imaging data from ABIDE, a global multi-site database, for patients with ASD. Repeated behaviours and social deficiencies are symptoms of ASD, a brain-based disorder. One in 68 children in the United States suffers from ASD, according to new statistics from the Centres for Disease Control. In an effort to uncover the neurological patterns that arose from the classification, researchers looked into patterns of functional connectivity that can be used to objectively identify individuals with ASD from functional brain imaging data. By detecting ASD patients in the dataset with 70% accuracy compared to control patients, the results enhanced the SOTA. Current empirical evidence of anterior-posterior disruption in brain connectivity in ASD is supported by the patterns that arose from the classification. This classification demonstrates an ant correlation of brain function between anterior and posterior parts of the brain. According to the DL model, researchers describe the findings and pinpoint the parts of the brain that most helped distinguish ASD from normally developing controls.

An ASD detection hybrid model that operates on two distinct dataset types is reported by Sharma and Tanwar [14]. First, the Logistic Regression (LR) approach is used in behavioural datasets. The second dataset makes use of the CNN Classifier. It can determine if a person has autism by using facial recognition. Simulation results show that the proposed hybrid model, which uses a person's face and behavioural data, outperforms SOTA methods in terms of accuracy. The suggested hybrid model attained an accuracy of 82.76% for the CNN model and an average of 88% for the LR model.

In order to detect ASD in children, Han et al. [15] simultaneously analyse from the perspectives of internal neurophysiology and external behaviour. Researchers suggest a novel multimodal diagnosis paradigm that combines Electroencephalogram (EEG) and Eye-Tracking (ET) data. More specifically, researchers used the Stacked Denoising Auto Encoder (SDAE), a popular DL technique, to develop a two-step multimodal Feature Learning (FL) and Feature Fusion (FF) model. In the first step, two SDAE models are constructed especially for FL in the ET and EEG modality. Using concatenated learnt ET and EEG data, a third SDAE model is then developed in the second stage to perform multimodal fusion. The multimodal identification model can automatically identify correlations and complementarity between behaviour and neurophysiological modality in a latent Feature Space (FS). It generates more discriminative feature representations that are beneficial.

Using functional brain networks constructed from brain fMRI data, Yin et al. [16] develop DL approaches that can be utilised to diagnose ASD. Researchers test the efficacy of the suggested approaches using the entire ABIDE 1 dataset. Researchers first construct brain networks using brain fMRI data, and then they characterise raw features from these networks. Then use an AE to extract the advanced features from the raw data. Third, train a Deep Neural Network (DNN) with the improved features. The DNN obtains a receiving operating characteristic curve (AUC) of 79.7% and a classification accuracy of 76.2%.As a comparison, researchers also use the same advanced features to train other traditional ML methods.

In order to compare and choose the best classifiers, automate the diagnosis process, and identify the most crucial features for enabling detection,and it has been analyzed by Manoj and Praveen [17]. Through the use of LR, Decision Trees (DT), Naive Bayes (NB), Random Forests (RF), Support Vector Machines (SVM), Linear Discriminant Analysis (LDA), K-Nearest Neighbours (KNN), XGBoost classifiers, and Multi-Layer Perceptrons (MLP), researchers investigated whether it is possible to predict whether an individual is more likely to have ASD. The investigation examined ASD data for toddlers, children, teens, and adults, and the best classifier algorithms were found using metrics like R, P, and F-measures, among others. Here, the most effective and important features in the dataset can be identified by performing feature engineering. According to the researchers, best outcomes in the dataset can be generated by the LR and MLP models. The data from the adolescents can be analyzed effectively by the LR classifier.

To help medical professionals in DM, popularity of ML techniques, the development of new diagnostic tools based on these technologies, and the progress of brain imaging are effective and it was demonstrated [18]. In order to evaluate how effectively various ML models, including DNN. DNN could distinguish between patients with chronic pain and pain-free controls, researchers used resting-state fMRI data from 150 participants to compute functional brain connectivity. By utilizing the preprocessed data from the MSDL probabilistic atlas and Dynamic Temporal Warping (DTW) as a connection metric, CNN was trained, and it attains optimal outcomes. DL models outperformed less expensive, alternative models like SVM.

3. Hybrid FCNN-LSTM Approach for Autism Spectrum Disorder Classification

3.1 Proposed Methodology

Hybrid FCNN-LSTM model is introduced for ASD classification involves a two-stage approach. First, CNN is used to extract spatial features from behavioral or physiological data. The temporal dependencies in TS or sequential data are then modelled by inputting these features into LSTM networks. The CNN component captures spatial patterns, while the LSTM component learns temporal relationships [19]. The combined features are processed through Fully Connected (FC) layers for classification. Metrics like Acc and F1-score are used to assess the model's performance once it has been trained using suitable Loss Functions (LF) and optimizers. The implementation of the suggested work as follows in Fig. 2. This hybrid approach enhances ASD classification by integrating both spatial and temporal information. It will be show in Fig. 3.

3.1.1 Data pre-processing

The accuracy and reliability of the data impacts the models performance. Pre-processing is crucial in ML. Steps like Noise Reduction (NR), normalization, and Data Augmentation (DA) are included in this pre-processing. Because ASD data is frequently noisy, high-dimensional, and imbalanced, each step is specifically designed to address the particular risks it poses [20].

Noise reduction: in the dataset, an inaccurate or redundant data can be eliminated by this NR step. The models accuracy can be improved by this step. Sensors errors in brain imaging, variations in data collection methods, or irrelevant features in the dataset are the following reasons that causing the noise in ASD data. Here, the following techniques are commonly utilized, they are Gaussian filtering, wavelet denoising, and Principal Component Analysis (PCA). Then, the dimensionality can be reduced, most relevant features, all can be maintained by implementing this NR method. Focusing the most relevant features related to ASD was also ensured by this method.

Normalization: normalisation involves scaling the data to a predetermined range, typically [0, 1] or [-1, 1], to guarantee that each feature contributes equally to the model. Features with varying scales in the dataset are handled by this step, so it is crucial. Wide-ranging features are prevented from dominating the model's training. Aligning the intensity values over several images are facilitated by normalization. The Brain imaging data was used by the ASD classification models, and intensity values over various images was aligned by this normalization. Better FE was also contributed by this framework.

Data augmentation: when a dataset is small or unbalanced, DA techniques can be used to artificially increase its size and variability. Rotations, flipping, and cropping are only a few possible modifications included in this DA for image data. By allowing the model to learn from a greater range of data, this DA improves the model's generalisation. For enhancing the robustness of the model, DA has a major part in the context of ASD, because obtaining huge datasets is frequently difficult.

In ASD classification, the performance of Ml models can be improved by the effective pre-processing, so this effective pre-processing is crucial [21]. The models can be robust, accurate, and capable of handling the complexities in ASD data was ensured by the researchers via systematically applying NR, normalization, and DA.

3.1.2 FE using CNNs

The FE has played a major role in ML. From the raw data, a set of features can be employed for training the model [22]. Then, FE is done effectively via CNN, so this CNN is considered to be an effective method in the context of ASD classification. When working with image data such as brain images or other spatially structured data, this CNN are effective. This CNN are a type of DNN. Here, grid-like structure, that includes images and other data were processed by CNN. In CNN, the hierarchical features from the input data was automatically learnt and extracted by the Convolutional Layers (CL). When comparing this CNN with conventional ML, CNN have the ability to extracts features directly from the raw data, but the conventional ML demands manual FE. CNN are very efficient at tasks involving intricate patterns and huge datasets.

Convolutional layers

CL is the basic layer of CNN. Convolutional filters (kernels) are applied to the input data for detecting various features like edges, textures, or more intricate patterns. The input image or data matrix is processed by these filters. Each filter in the CL identifies a specific type of feature by computing a dot product between the filter and a small amount of the input data. The result is a Feature Map (FM) that shows the positions of particular features [23]. The low-level features like edges or blobs in brain data.

The low-level features like edges or blobs in the brain imaging data can be effectively detected by the initial CL layers in the ASD classification. More complex structures related to brain regions or patterns related to ASD was effectively captured by the deeper layers.

Activation functions

An Activation Functions (AF) (also known as a rectified linear unit, or ReLU) is used to add non-linearity to the model following the convolution operation. The network learns intricate patterns as a result. Without AF, the model would simply perform linear operations, which are insufficient for capturing the non-linear relationships in complex datasets like those used in ASD diagnosis.

Pooling layers

To down sample the FM, pooling layers are employed. By pooling layers, the most crucial information is retained while the dimensionality is decreased. The FM becomes invariant to slight translations or distortions in the input data due to this mechanism [24]. Max pooling is the most common type. The maximum value from each FM patch is chosen using max pooling. The FM's width and height are decreased by this process, which results in fewer parameters and calculations in the layers that follow. In the context of ASD, pooling layers help in focusing on the most prominent features in the neuroimaging data, such as the most activated regions in the brain scans, which might be critical for distinguishing between ASD and non-ASD cases.

Fully Connected layers

In ASD classification, the FC layers would integrate the features extracted by the CNN to classify whether a subject has ASD or not, based on the learnt patterns. The FM are flattened into a vector after several CL and pooling layers, and then they are passed through FC layers, which combine the extracted features to predict the output class.

Because CNNs are extremely effective at identifying spatial hierarchies in data, they are especially well-suited for classification tasks involving ASD.For instance, neuroimaging data used in ASD studies often involves complex spatial relationships between different brain regions [25]. CNNs can automatically learn to recognize these patterns, making them highly effective for diagnosing ASD based on subtle changes in brain structure or function that might not be apparent through manual Feature Extraction (FE) in Fig. 4.

In ASD classification, FE with CNNs is a potent method. Because, the data from the high-dimensional and complex features are automatically and effectively extracted by this effective method. CNNs are a vital tool for creating reliable and accurate models for diagnosing ASD because of this capability.

3.1.3 Sequence Modeling with LSTM networks

Sequence Modeling (SM) is an essential task in ML, particularly when the data has a temporal or sequential nature, meaning that the order of the data points matters. An LSTM network is a type of Recurrent Neural Network (RNN) designed specifically to represent and learn from data sequences. For identifying temporal relationships in TS data, such as EEG signals, speech patterns, or behavioural sequences, LSTM are crucial in the classification of ASD. For an accurate diagnosis, these detections are frequently essential [26]. Fig. 3 illustrates the LSTM network.

Long-term dependencies in sequential data can be learnt by specialised RNNs called LSTM networks. Unlike traditional RNNs, which can suffer from issues like vanishing and exploding gradients, LSTMs are designed with a more complex architecture that allows them to maintain and learn from long-term dependencies effectively in Fig. 5.

The essential component of an LSTM network is the LSTM cell. The input, forget, and output gates found in each cell regulate the information flow and determine which data should be retained or discarded as the sequence goes on. The extent to which of the new input should affect the cell state is controlled by the input gate. The information that should be deleted from the cell state is determined by the forget gate. The output gate uses the current cell state to determine the output. In ASD classification, an LSTM cell might process a sequence of behavioral data points, determining which aspects of past behavior are relevant for predicting future behavior, such as repetitive actions or speech patterns. Cell state acts as the memory of the LSTM, carrying relevant information across the sequence [27]. It is updated at each time step based on the inputs and the decisions made by the gates. The next LSTM cell in the sequence receives the hidden state, which is the output of the LSTM cell at each time step.

In a sequence of EEG signals used for ASD diagnosis, the cell state might store information about the overall brain activity trends, while the hidden state represents the current prediction or classification at each time step.

Sequence processing: in an LSTM network, data is processed sequentially, with the LSTM cells updating their states based on new inputs and previous states. In bidirectional LSTMs, the sequence is processed in both directions (forward and backward), capturing dependencies from past and future contexts. When analyzing a sequence of social interactions in a child suspected of having ASD, a bidirectional LSTM can understand the context by considering both the earlier and later parts of the sequence, improving the model's ability to detect subtle patterns indicative of ASD.

Temporal dependencies

LSTMs excel at capturing temporal dependencies, which are crucial in ASD data where the order of events or signals is important. For example, the progression of symptoms over time or the order in which specific behaviors occur can be vital for an accurate diagnosis. In speech pattern analysis, LSTMs can model the temporal dependencies between phonemes or words, helping to identify atypical speech patterns that are characteristic of ASD.

Because LSTMs can remember and use information over lengthy sequences, they are especially useful for tasks involving sequential or TS data. It is crucial for comprehending intricate patterns that change with time. In the context of ASD, where behaviors, brain activity, or other relevant signals are often sequentially structured, LSTMs provide a powerful tool for modeling these patterns.

3.1.4 Fuzzy logic for classification in hybrid models

Fuzzy Set Theory (FST) is the basis of FL, a type of multi-valued logic designed to handle approximate rather than exact reasoning. FL variables may have a truth value that varies from 0 to 1, signifying the degree of membership in a set, in contrast to standard binary sets (where variables must be either true or false) [28]. When data is unreliable, imprecise, or lacks clear boundaries, all of which are common features in medical diagnosis like ASD, this is very useful (Fig. 5).

ASD presents with a wide variety of symptoms that can overlap with other conditions, making diagnosis challenging. Traditional classifiers may struggle with the ambiguity inherent in such data, leading to lower accuracy. Fuzzy Logic (FL) helps by in below Fig. 6,

I. Handling uncertainty: instead of using predetermined thresholds, FL enables the model to make decisions based on degrees of truth. This reflects the real-world complexity of ASD symptoms, which may not always fit neatly into binary categories.

II. Enhanced interpretability: FL systems are more interpretable, as they provide a clear rationale for each decision, grounded in the degrees of membership to fuzzy sets. This is valuable in clinical settings where understanding the basis of a diagnosis is crucial.

III. Integration with other models: in a hybrid model like the CNN-LSTM-FL approach, FL can be used at the final stage to classify the outputs from CNN and LSTM networks. The CNN extracts spatial features, the LSTM captures temporal dependencies, and FL handles the uncertainty in these extracted features, leading to more robust and reliable classification.

FIS: the output is fed into a FIS following processing of the input data by the CNN and LSTM layers. A collection of Fuzzy Rules (FR) make up the FIS.

I. Fuzzification: the crisp inputs from the CNN-LSTM model are converted into fuzzy values (degrees of membership). For instance, a specific feature might partially belong to the "high" set with a membership value of 0.7 and to the "medium" set with a membership value of 0.3.

II. rule evaluation: to ascertain the degree of truth for each output class (such as "ASD" or "typical development"), the FR are applied to the fuzzified inputs.

III. Aggregation: the results from all the FR are combined to form a single fuzzy set for each output class. IV. Defuzzification: finally, the fuzzy output is converted back into a crisp value, which represents the model's final classification decision (e.g., a 0.85 likelihood of ASD).

Incorporating FL into a hybrid CNN-LSTM model enhances the model's ability to handle the imprecision and complexity of ASD data [29]. By managing uncertainty effectively, FL provides more accurate and interpretable classification results, making it a powerful tool in automated ASD diagnosis systems. The execution of the proposed work mentioned in the Fig. 7 matlab screens.

4. Discussion

Analysing and interpreting classifications takes time and requires a deep understanding of statistics. The models take a long time to finish, and expert analysis is needed to look at the data's classification and correlations.

5. Conclusion

CNN for spatial FE and LSTM networks for temporal analysis have been effectively incorporated in the Hybrid FCNN-LSTM model for ASD classification. The accuracy of the diagnosis will significantly increase as a result. Compared to conventional techniques, this strategy more thoroughly captures both spatial and temporal patterns in neuroimaging data, improving the model's capacity to more accurately diagnose ASD. Notwithstanding its advantages, the model requires a significant amount of processing power and lengthy training periods. Future research should concentrate on improving data preprocessing approaches, incorporating Multimodal data (MMD), such as genetic and behavioural information, to further enhance accuracy, and optimising model efficiency through transfer learning and compression techniques. In order to improve ED and personalised treatment plans for ASD and guarantee more applicability and efficacy in clinical settings, it will be essential to continue investigating novel NN architectures.

Figures