Kholoud Khalil Aldous , Joni Salminen , Soon-gyo Jung , Jinan Y. Azem , Johanne Medina , Salar M. Khan , Amani Alabed , Bernard J. Jansen
International Conference on Foundation and Large Language Models (FLLM) (2025)
The evaluation of AI educational dialogue systems for middle-school students has been limited. This study employs a state-of-the-art AI chatbot that answers students’ questions exclusively based on educator-provided learning materials. Following an initial assessment by 10 middle-school teachers using the Chatbot Suitability Questionnaire, we conducted a mixed-method intervention user study involving 18 middle-school students to explore usability expectations, knowledge acquisition, and learning experience. Findings reveal that interacting with the AI chatbot enhanced self-reported knowledge acquisition, improved learning outcomes measured by test scores, and maintained student interest for future use. The chatbot achieved a usability score of 71.44% (± 16.28), attributed mainly to its high answer accuracy and effective interpretation of student input. Error management emerged as the most critical usability factor.