AI-experiments in education: An AI-driven randomized controlled trial for higher education research

Abstract

This study presents a novel approach contributing to our understanding of the design, development, and implementation AI-based systems for conducting double-blind online randomized controlled trials (RCTs) for higher education research. The process of the entire interaction with the participants (n = 1193) and their allocation to test and control groups was executed seamlessly by our AI system, without human intervention. In this fully automated experiment, we systematically examined eight hypotheses. The AI-experiment strengthened five of these hypotheses, while not accepting three of the factors previously acknowledged in the literature as influential in students’ choices of universities. We showcased how AI can efficiently interview participants and collect their input, offering robust evidence through an RCT (Gold standard) to establish causal relationships between interventions and their outcomes. This approach may enable researchers and industry practitioners to collect data from large samples on which such experiments can be conducted with and by AI to produce statistically reproducible, reliable, and generalizable results in an efficient, rigorous and ethical way.