Decoding ChatGPT’s impact on student satisfaction and performance: a multimodal machine learning and explainable AI approach

1. INTRODUCTION

Artificial intelligence (AI) has changed education around the world at a fast pace. Generative AI tools such as ChatGPT are significant in teaching and learning^[1,2]. The sophisticated language models of ChatGPT enable it to offer personalized tutoring, automatic grading, research assistance, and support with academic writing^[3,4]. With the increased usage of AI tools in education, there is increased interest in how they influence student learning and academic performance^[5,6]. Recent studies have shown that AI tools make students more engaged, help them explain complex topics better, and provide rapid feedback, which makes learning easier and more individualized for each student^[7,8]. The ability to create smart, human-like conversations with chatbots such as ChatGPT can make the learning environment engaging and dynamic. It is an invaluable tool for most learning scenarios, including self-paced education and teamwork in learning^[9]. Increased practice using generative AI in education remains largely unsupported empirically, with the full impact of this technology on student academic outcomes and factors that explain variations in student satisfaction and engagement in traditional tasks with these tools.

Most research on generative AI tools in education has focused on their use and specific cases^[10]. It has not paid much attention to how these tools affect student satisfaction and performance. Past studies have examined how ChatGPT can automate regular academic tasks or provide knowledge when studying for an exam^[11,12]. These studies do not look into crucial areas, such as how students using these tools can impact their motivation, critical thinking, and satisfaction with learning^[13]. Moreover, unclear explanations of AI models applied in education make it hard to determine why and how something impacts performance. Without a clear answer, education policymakers and practitioners cannot draw helpful insights into making AI tools for various types of students. Wu et al.^[14] and Rehman^[15] stated that only simple statistics or ordinary ML models were used in studies to determine the impact of AI on academic performance of students. This limits how deeply such studies can probe for insights from predictions and to what extent the findings can be transported into different contexts. Studies on AI applications in education rarely use an explainable AI (XAI) approach to present explanations and what is discovered. This is required because it connects the technical sophistication of AI with how AI is implemented in practice within education^[16]. These limitations demand a more integrated approach that merges the best ML techniques with the power of XAI, thereby yielding rich insights into the dynamics between the applied AI tools, academic performance, and resultant student satisfaction^[17,18]. Our study fills these gaps by using advanced ML models coupled with XAI to investigate the effects of the ChatGPT on students' academic outcomes.

This research introduces a new way to use ChatGPT and its impact on students' academic performance with the help of machine learning (ML) and XAI. It does not just look at a basic test to see whether ChatGPT is effective; it also studies how this relates to learning satisfaction, academic results, and student engagement. This research uses ML and XAI techniques to explore the link between ChatGPT and academic performance. However, the ML models mainly act as "black boxes", which means they provide less information about why they made their predictions. We use XAI methods to understand the models' predictions and decision-making and seek essential factors. The combination of ML and XAI ensures our results are accurate and useful.

This study aims to determine how ChatGPT relates to student learning and performance in school. The research will: (1) Forecast changes in satisfaction and grades among students who use ChatGPT using ML models; (2) Deploy transparent AI techniques to identify the primary motivations among the most common reasons for students using ChatGPT; and (3) Provide valuable recommendations to teachers, policymakers, and platform developers on using generative AI effectively. This research contributes to the growing knowledge of AI in education. This study offers worthwhile concepts that would help develop better planning for improving the effectiveness of generative AI tools in schools to achieve these objectives.

2. LITERATURE REVIEW

AI has dramatically helped improve education. It aids the teacher in teaching and in determining how the students learn. AI tools help personalize learning experiences, simplify administrative work, and provide data-driven insights that improve teaching methods^[19,20]. Such new tools have helped meet the different needs of students, thereby allowing teachers to spend more time on teaching well and less on repetitive work. AI is used in learning platforms such as DreamBox and Knewton, which change educational content based on each student's strengths and weaknesses^[21]. These platforms make learning more engaging and improve the experience, especially for students who have trouble with traditional methods. The most significant application of AI in education is to make learning easier for everyone. It can translate speech into text, translate languages in real time, and provide content according to the needs of each learner^[22,23]. AI tools also help teachers know how students are doing in school, which allows them to step in when problems arise. Even with all these changes, issues still abound.

Generative AI models are becoming increasingly popular among education tools, such as the model used for ChatGPT. It is a conversational AI based on OpenAI's GPT-4 design^[24]. This model enables the generation of text similar to human writing and gives detailed explanations to help with various educational tasks, including automated grading, personalized tutoring, content generation, and assistance with academic writing^[25,26]. For example, students can use it to explain complicated topics, request feedback on their essays, or brainstorm project ideas^[26,27]. Similarly, instructors can use it to automatically make lesson plans, answer students' queries, and create exciting learning experiences^[28,29]. ChatGPT has greatly supported self-paced learning, wherein students can freely ask questions related to various subjects, developing confidence and facilitating a better understanding of them^[30]. Still, there are concerns about over-reliance on AI, loss of critical thinking skills, and the ethics of using AI-made content in assessments. According to ElSayary et al. (2024), research has proven that despite all these benefits, these tools should be used carefully to support classroom teaching instead of taking their place^[31].

ML is a part of AI and has helped many areas, including education. It started from fundamental computer theories in the middle of the 20th century and grew rapidly in the 1990s because of new algorithms that could sort and study large amounts of data^[32,33]. ML models are highly utilized in education for various purposes, including predicting student performance, learning habit analysis, and curriculum design enhancement^[32,34,35]. Among the best ML methods, Support Vector Machine (SVM), Random Forest (RF), and Extreme Gradient Boosting (XGBoost) can help find patterns in student data for education improvement^[36,37]. For example, ML models can predict student risk based on attendance data, grades, and behavior, so help may be pursued whenever required^[34,37]. Similarly, k-means techniques group students based on their learning preferences to create a personalized learning plan^[36]. Even though ML models are compelling, their "black-box" nature often deters people from using them in crucial sectors such as education, where clarity and responsibility are essential^[38-40]. XAI solves the problem by making the ML model easy to understand so that teachers and decision-makers know the reasons behind the predictions made by the model^[36,40]. Local Interpretable Model-Agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) make complex machine-learning models easily comprehensible. Methods are used to show how each feature affects predictions^[41,42]. XAI has been applied in education to reveal the primary reasons students drop out: their economic background, attendance, and grades^[41]. XAI helps teachers make decisions based on data, improving educational planning and student support^[43,44].

Introducing ML and XAI in education has opened new avenues for data-driven teaching and learning strategies. ML and XAI also play a significant role in developing intelligent tutoring systems (ITS). ML models have been used to predict students' exam performance, analyze student feedback, and automate administrative staff's work. With XAI, these predictions are still interpretable and actionable^[39,43,44]. Huang et al. (2024) examined factors influencing students' engagement with XGBoost and SHAP. According to them, time management and participation in group activities are some of the most important factors. This allows the teachers to devise specific plans for enhancing engagement^[45].

A clear example is using XAI to check the effectiveness of generative AI tools such as ChatGPT. The researchers use XAI techniques to identify the aspects, such as how often people engage with ChatGPT and the difficulty of questions that affect learning performance. This can lead to better tools for more tremendous educational success^[46]. Research studies indicate that ML and XAI can positively enhance academic results. Villegas-Ch et al. (2024) also highlighted how applying ML models for predicting students' performance in online courses may be related to engagement metrics as critical predictors^[47]. In addition, the SHAP explanation provided actionable insights for course designers^[48]. demonstrated the use of XGBoost and LIME in data analysis concerning social media addiction and how it impacts academic performance. It was found that too much screen time and a lack of self-control were the primary reasons. Lo and Kwan^[8] further discussed how ChatGPT can be used in teaching students. Using the empowerment and elements of XAI, the researchers established that ChatGPT's instant responses and compression of difficult topics further helped the students improve their understanding and confidence. However, it also calls for caution against over-reliance on this tool and demands a harmonious blend with the traditional teaching system^[49].

Combining AI, ML, and XAI in education will create significant opportunities for improving teaching and learning. For example, tools such as ChatGPT show what generative AI can do to make learning more personal and engaging. However, explainability, ethical issues, and dependency problems highlight the need for careful integration. The current research fills this gap by bridging technological innovation to practical applications in utilizing state-of-the-art ML models and XAI techniques that give much-needed insight into further enhancing AI tools in pursuit of educational achievement.

3. METHODOLOGY

Our study’s methodology starts with the first step, data preparation to deal with missing values due to imputation techniques and encoding categorical variables such as gender and institution type using one-hot encoding^[50]. This puts the data in a standard format and prepares it for analysis. Four dimensions were used to classify variables executed in feature engineering^[51]: Academic performance, satisfaction and perception, usage and interaction, and stress and motivation. These dimensions allowed for better interpretability of the models and their effectiveness. An Exploratory Data Analysis (EDA) was conducted through visualizations and correlation matrices to understand the patterns and relationships in the data. The dataset was split into a ratio of 80%-20% for training and testing to test the models sufficiently. Further, our novel study explored the utilization of ML and the XAI approach to predict the effects of ChatGPT on several academic outcomes: performance, satisfaction, motivation, and related variables. The experiment used three ML models: RF, XGBoost, and SVM on a custom-created dataset. Hyperparameter tuning was also performed to maximize predictive accuracy. The whole process of the training methodology ensured the most reliable and accurate results in predicting the outcome of the education process. XAI tools such as SHAP and LIME were also applied to investigate models, their usage, how a model arrives at its predictions or decisions, and to explain the working process. As demonstrated, the applied data analysis methods supported a more in-depth understanding of global and local features explaining predictions, with Q2_ChatGPT_Satisfaction, Q3_Performance_Impact, and Q4_Usage_Frequency variables being key predictors. The model's performance is evaluated using classical metrics, such as accuracy, precision, recall, F1-score, and the receiver operating characteristic curve and the area under this curve (AUC-ROC). Figure 1 depicts the overall workflow of the proposed approach.

Figure 1. Research flow diagram.

3.3. Exploratory data analysis

The exploratory data analysis (EDA) supports understanding the nature of our custom-made dataset's characteristics and relationships or patterns before applying advanced modeling techniques. The task of this study was to apply EDA to gain further insight into and analyze the responses made on the survey, detect problems such as outliers or skewed distributions, and finally better decide on data preprocessing and feature selection. Figure 2 plots the histograms showing distributions for every item of the survey. These histograms provide a clear view of the frequency of various responses across numeric variables, highlighting skewness or imbalance in the data. This visualization helped identify if any specific items were over- or underrepresented in the survey, which might affect the accuracy and generalizability of the models. It also showed whether the responses followed a normal distribution or had noticeable deviations.

Figure 2. Distribution of responses for each survey.

Moreover, Figure 3 shows feature box plots for the individual variables in the dataset, representing the dataset's central tendency, spread, and possible outliers in a graphical way. These box plots would prove helpful in identifying extreme values or variables with extensive interquartile ranges. We could see a need for features that may require further attention, either for normalization or to treat outliers, so as not to distort the model while in training.

Figure 3. Box plots for each feature.

Finally, Figure 4 is the correlation matrix heatmap, presented to display relationships between numeric features of our self-crafted dataset, namely ChatGPT Survey Data. The color intensities of the heatmap point toward the strength and direction of correlation to identify highly correlated variables. In this regard, finding multicollinearity degrading ML models' predictive ability was essential. This helped us to refine the feature selection process so that only the most relevant and independent variables were included in the model.

Figure 4. Correlation matrix heatmap.

4. MODEL INTERPRETATION AND EXPLAINABILITY

The explanation techniques of model interpretation were used to see how the models made their predictions, providing deep insight into how various features influence academic performance and user satisfaction while interacting with ChatGPT. The interpretability techniques used SHAP and LIME assume low feature redundancy for accurate attribution. We confirmed low pairwise correlations among features to meet this assumption, avoiding direct multicollinearity. The modest correlation between Q6_Problem_Solving and Q12_Learning_Motivation (r = 0.13) was further addressed by creating an interaction term. This strategy isolates their joint effect while minimizing overlap in their contributions to SHAP analysis, preserving the integrity of feature attributions and ensuring robust model explanations. Using the XGBoost and RF models, feature importance analysis was done to identify the main drivers of academic performance, satisfaction, and other outcomes in this dataset. Based on the feature importance plot, significant features are Q2_ChatGPT_Satisfaction and Q4_Usage_Frequency, which are the important predictors of academic performance. With this regard, this research further helped narrow the most crucial indicators that contribute to influencing outcomes among the different users to eventually assist in guidance for subsequent ChatGPT usage based on a solid academic purpose for its implementation in an educational situation. Some relevant XAI methods, including SHAP Values and LIME, were applied to add information. SHAP plots showed a global view of how each feature contributed to the predictions across all data points; for example, a strong impact from Q8_Confidence was often complemented by high scores in Q3_Performance_Impact, thus showing a positive feedback loop between confidence and performance. LIME provided instance-level insights into how specific features influenced individual predictions, such as how high values for Q6_Problem_Solving and Q12_Learning_Motivation significantly boosted predicted performance. Interpretation techniques ensured the transparency of the model, enabling actionable insights and practical recommendations for improving ChatGPT as an academic tool.

5. FINDINGS

Two sections of the study present the ML Model Analysis and the XAI Model Analysis results. Those discuss predictive performance and interpretability, discussing what factors inform academic performance or satisfaction with using ChatGPT.

5.1. ML model analysis

This work discusses the analysis of ML models such as XGBoost, RF, and SVM to predict academic performance and satisfaction. The summary of their performances is shown in Table 9 in terms of accuracy, precision, recall, F1-score, and AUC. Among these, XGBoost was the best at predicting academic performance as a regression task, with an R² value of 92%. This, in turn, means that, according to features such as ChatGPT use and satisfaction with ChatGPT/problem-solving skills, 92% of the variance in academic performance could be explained by a model such as XGBoost. The high R-squared value represents the strength and reliability of using XGBoost for modeling and good predictions.

Table 9

ML models performance results

Model	Target variable	Accuracy/R2	Precision	Recall	F1-score	AUC
XGBoost	Academic performance (regression)	0.92 (R2)	-	-	-	-
Random forest	Satisfaction (classification)	0.87 (accuracy)	0.85	0.88	0.86	0.90
SVM	Satisfaction (classification)	0.89 (accuracy)	0.86	0.89	0.87	0.92

Besides, the RF and SVM classifiers also had high values for the classification task of predicting student satisfaction, with the SVM slightly outperforming the RF at 89% and 87%, respectively. Also, the SVM model was more potent in other measures: precision was 0.86, the recall was 0.89, and the F1-score was 0.87. In contrast, the RF model achieved the following values for precision, recall, and F1-score: 0.85, 0.88, and 0.86, respectively. These results suggest that SVM is particularly adept at identifying students with high satisfaction levels, making it a suitable choice for classification tasks in this context.

Figure 5 compares performance metrics by various models, showing the superiority of each model in different aspects. The corresponding AUC scores also reflect the effectiveness of these models, with the highest achieved by SVM at 0.92 and standing at 0.90 for RF. It is interpretable that both models carry excellent discrimination capability, while SVM is leading a little ahead in class distinction.

Figure 5. Comparative performance of machine learning models.

These results emphasize the strengths of ML models in making predictions about outcomes related to the use of ChatGPT. The relatively strong performance of XGBoost in terms of the prediction of academic performance gives strong evidence of its applicability in education research. Similarly, robust performances by SVM and RF on classification tasks give their worth in making sense of student satisfaction. The performance evaluation of the RF and SVM classifiers shows excellent classification performance on binary outcome predictions, as will be informed by insights from confusion matrices. Confusion matrices for both models in Figure 6 have many true positives and true negatives but few false positives and false negatives, reflecting their high overall accuracy.

Figure 6. Confusion matrices for random forest and support vector machine (SVM) classifiers.

As shown in Figure 7, the RF classifier achieved an accuracy of 0.87, while the SVM classifier slightly outperformed it with an accuracy of 0.89.

Figure 7. ROC curves for random forest and support vector machine (SVM) classifiers.

The feature importance analysis of the classification performance of RF and XGBoost pointed to other significant predictors for academic performance and user satisfaction with ChatGPT. Among these, the factor Q2_ChatGPT_Satisfaction was vital in predicting better academic performance and improved satisfaction for overall users. This agrees with the previous literature, which has demonstrated the role of user satisfaction in improving learning outcomes in digital learning environments^[66,67]. Another strong predictor was Q4_Usage_Frequency, where high usage of ChatGPT was positively related to better academic performance. This would, therefore, mean that repeated interactions with AI tools create familiarity and engagement, enhancing learning efficiency. Previous research supports this by showing that consistent usage of tools strengthens cognitive and metacognitive skills and improves achievements^[68].

Q6_Problem_Solving also significantly contributed to academic performance. This underlines the capability of ChatGPT in complex problem-solving tasks, an essential component in pursuing educational success. In earlier studies, problem-solving using AI tools was said to enable critical thinking and deeper learning^[5,69]. Lastly, Q12_Learning_Motivation was another essential element in showing how intrinsic motivation plays a vital role in adequately using AI tools. Motivated learners tend to engage more deeply with learning tools, maximizing their potential benefits^[70].

While the analysis has shown that both RF and SVM classifiers are strong in their classification performance, SVM outperformed them in all scores of the evaluation metrics. The feature of importance analysis has shown that satisfaction, usage frequency, problem-solving, and learning motivation drive academic performance and user satisfaction. These results demonstrate the tremendous use value of ChatGPT and other AI tools in educational settings and give straightforward, practical suggestions on how best to use them to improve learning outcomes. Future research should develop an improved classification model that can help better understand how those identified features interact and contribute toward long-term academic success.

5.2. XAI model analysis

With the support of SHAP and LIME, central insight was gained into the XAI model decision-making processes, hence its interpretability^[41,42,71]. These techniques avoided typical ML model problems-"black-box", which explained how specific features influenced the models to make their prediction^[71]. In educational contexts, this transparency is essential for building trust among stakeholders, including educators and students, and ensuring ethical and informed AI usage.

The SHAP analysis for regression in academic performance is dominated by two features: Q2_ChatGPT_Satisfaction and Q4_Usage_Frequency. Increased levels of satisfaction with ChatGPT, captured in Q2, significantly raise the students' predicted academic performance as an essential factor from user experiences concerning this educational technology. Similarly, as measured by Q4, high usage of ChatGPT was strongly correlated with better performance. This shows how consistent engagement with the tools leads to academic success as discussed by Rehman et al.^[15]. As Figure 8 shows, the findings are supported by previous research, which evidences that user satisfaction and high use of learning tools improve performance since they encourage more profound learning and engagement^[72].

Figure 8. LIME explanation for academic performance prediction on XGBoost model.

In parallel, LIME analysis for Satisfaction Classification specified Q6_Problem_Solving and Q12_Learning_Motivation as important features determining student satisfaction levels. Further, the results indicated that a student who actively uses ChatGPT to solve a complicated problem would tend to express satisfaction with the tool's usefulness for critical thinking-enhanced problem-solving^[73]. Beyond this, captured in Q12, intrinsic motivation was crucial because students motivated to learn would likely place more value and satisfaction in using ChatGPT. That shows that cultivating inherent motivation will be a necessary prelude to deriving maximum benefit from this AI-driven learning tool, supported by the basic tenets of self-determination theory and other related studies on learner engagement^[74,75].

6. CONCLUSION AND FUTURE WORK

This research sought to explore and investigate the use of ML classifiers and XAI techniques to understand how AI applications, such as ChatGPT, affect academic and student performance and satisfaction better and ensure their effectiveness and explainability. In this paper, the predictive capabilities of ML models and the interpretability of XAI methods have been elaborated by considering key factors that determine academic success and user satisfaction through using RF and SVM in predictive modeling, as well as SHAP and LIME for explainability. The primary purpose of this study was to find and assess the predictors of performance and satisfaction with ChatGPT as a learning tool. This study concentrated on the following aspects of ChatGPT: satisfaction, usage frequency, problem-solving, and learning motivation. The RF and SVM models reached an accuracy of 87% and 89%, respectively, and possessed extreme discriminatory power, as evidenced by their AUC-ROC scores of 0.90 and 0.92. XGBoast delivered expected outcomes with an R2 score of 0.92. XAI analysis identified that satisfaction, usage frequency, and learning motivation were the most influential features. These results indicate the possibilities of AI in education. The ML models' strong performance highlights AI tools' ability to find the most influencing factors behind the student outcome, and XAI techniques ensure transparency, building trust among stakeholders. Practical implications are designing targeted interventions to enhance user experiences, building motivation, and engineering AI tools that meet the needs of individual learners. Despite its contributions, the study has limitations.

The analysis was based on a specific dataset and context, which may limit the generalizability of the findings. Additionally, reliance on self-reported data may introduce biases. Future research should focus on diverse datasets, longitudinal studies, and broader AI applications to further validate and extend these findings. Future research may incorporate comparisons with established educational prediction benchmarks to further contextualize model performance. This study demonstrates the transformative potential of integrating AI and XAI in education, providing actionable insights for educators and paving the way for ethical and effective adoption of AI technologies in learning environments.