Assessment of the LLM-based Chatbots on Student Engagement and Learning Outcomes in Afghanistan

Abstract

The integration of Generative AI (GenAI) technologies, such as ChatGPT, into online education is accelerating; however, their effectiveness in under-resourced contexts remains insufficiently studied. This paper investigates the impact of a Large Language Model (LLM)-based conversational agent on student engagement and learning outcomes in Afghanistan, where access to formal education—particularly for women—is severely restricted or banned. We conducted an experimental study involving 80 undergraduate computer science students (40 male, 40 female) in Afghanistan, randomly assigned to control and treatment groups. All participants attended identical 50-minute online lectures followed by 40-minute post-lecture discussions moderated by a human instructor, and completed a follow-up self-report questionnaire. The treatment group additionally engaged in AI-facilitated discussions using a GPT-4-based chatbot during post-lecture discussion. Analysis of discussion logs and post-intervention surveys revealed that the treatment group demonstrated significantly higher participation rates, with more posts and replies, during post-lecture discussion and reported greater confidence in their understanding of the course material. These findings highlight the potential of LLM-based chatbots to enhance interactive learning and foster educational inclusion, particularly for marginalized populations in low-resource environments.