Examining the effects of generative AI chatbots on EFL learners’ listening and speaking skills and the perceived usefulness

The use of chatbots has attracted much attention because it can create interactive and immersive learning environments for EFL (English as a Foreign Language) learners. While learners benefit from interactions with chatbots, they indicate that chatbots still have limitations in understanding their intent and expressions to maintain natural conversations. To provide learners better experiences, our team created the CoolE Bot based on generative artificial intelligence (GenAI), which helps improve the bot's communication capabilities. This study investigated the effects of CoolE Bot on EFL learners' listening and speaking skills, as well as its perceived usefulness across two proficiency levels. It adopted a pre- and post-test design involving 42 students. They took listening and speaking tests, engaged with CoolE Bot for seven weeks, and completed a survey. Statistical analysis indicated that, while both high- and low-proficiency groups showed significant improvements in speaking skill, the improvements in listening skill were less pronounced and did not reach statistical significance. In terms of perceived usefulness, the low-proficiency group found CoolE Bot more beneficial than their high-proficiency peers across several aspects, including different language skills development and increased opportunities for speaking practice. These results suggest that CoolE Bot is a valuable tool for enhancing speaking skills and supporting listening development.