Insights from a multilevel meta-analysis on the effectiveness of dialogue-based CALL

Bibauw, Serge [UCL] Van den Noortgate, Wim [KU Leuven] François, Thomas [UCL] Desmet, Piet [KU Leuven]

Dialogue-based computer-assisted language learning (CALL) systems allow a learner to practice meaningfully a foreign language (L2) with an automated agent, whether through an oral (spoken dialogue systems) or written interface (chatbots) (Bibauw, François, & Desmet, 2015). While various dialogue-based CALL systems have been developed and tested with learners against different evaluation schemes, individual evaluations have provided limited information on their effectiveness on L2 development (Bibauw, François, & Desmet, 2019). In order to obtain a better comprehension of their effects on L2 proficiency development, we conducted a meta-analysis on all the experimental studies measuring an impact of such systems on language learning outcomes (40 publications). Effect sizes for each variable and group under observation were systematically computed from the available data (k=96). While most individual studies do not achieve significance in their results, often due to limitations in sample sizes and insufficiently sensitive outcome measurements, by combining all studies into a multilevel linear model, we observed a significant medium effect of dialogue-based CALL on general L2 proficiency development (d=.602). By integrating moderator variables into our statistical model, we are able to provide insights on the relative effectiveness of certain design characteristics (spoken vs. written interface, task-oriented vs. open-ended interaction, form-focused vs. meaning-focused design) on different learning outcomes (holistic writing/speaking proficiency, as well as specific complexity, accuracy and fluency measures…), and to model the effect of treatment duration and spacing on these outcomes, to better inform future system and research design. In comparison with Bibauw et al (2017), we updated our pool of studies, clarified our exclusion process, refined the effect sizes calculation methodology and the multilevel modelling, and have computed a new mixed-effects model to take into account variables of treatment duration and spacing effects.