From Priming Modalities to Educational AI Chatbot Engagement: A Study of 67 Learners

Abstract

Trang Xuan , Joni Salminen , Ilkka Kaate , Farhan Ahmed , Danial Amin , Rajat Patil , Soon-Gyo Jung , Jinan Y. Azem , Bernard J. Jansen Learner interactions with AI have become crucial in education. A study with sixty-seven learners in a within-subject quasi-experimental study was conducted. Participants were randomly assigned to counterbalance sequences of visual and audio learning materials about wild species, completing a pre-survey, two learning sessions with an educational AI chatbot called Cipherbot and post session experience surveys. This study focused on learner interactions with Cipherbot, specifically (1) how material modalities influenced recall; (2) chatbot engagement through behavioral patterns; (3) material modalities’ influence on task completion; and (4) experiential interaction exploring how prior experiences moderated learning behaviors. Interactions were measured using surveys, audio-transcripts, and chatlogs. Results showed learners achieved higher knowledge recall with visual materials. Experienced AI-users interacted significantly less, whereas task-experienced learners interacted more after audio materials. The audio-butterfly sequence showed order effects, with learners spending more time producing more utterances. Chatbot-user guidelines with multiple modalities are recommended to enhance learning experience.