MetaMate: Understanding How Educational Researchers Experience AI-Assisted Data Extraction for Systematic Reviews

Abstract

Systematic reviews are essential for evidence synthesis in education, yet data extraction remains a bottleneck: labor-intensive and error-prone. Large language models offer automation potential, but questions remain about AI performance compared to human coders and how researchers experience these tools in practice. We present MetaMate, an open-access web-based tool for automated data extraction in educational systematic reviews. Our mixed-methods evaluation combines a quantitative validation study benchmarking MetaMate against trained human coders across 32 studies and 20 data elements with a qualitative user study involving six educational researchers using think-aloud protocols. MetaMate achieves precision (81-96%), recall (90-100%), and F1 scores (88-96%) comparable to or exceeding human coders, with strengths in mathematical reasoning and semantic comprehension. Qualitative findings reveal insights about trust calibration, verification behaviors, usability challenges, and human-AI collaboration. We contribute empirical evidence on LLM extraction capabilities and design implications for AI-assisted research tools balancing automation with human oversight. MetaMate is available at https://metamate.online.