Applying Generative Artificial Intelligence to Critiquing Science Assessments

Abstract

High-quality science assessments are multi-dimensional. They promote disciplinary practices, core ideas, cross-cutting concepts, and science sense-making. In this paper, we investigate the feasibility of using generative artificial intelligence (GenAI), specifically multimodal large language models (MLLMs), to annotate and provide improvement ideas for K-12 science assessments. The AI-generated annotations critique how the assessments align with the three dimensions of the Next Generation Science Standards (NGSS) and suggest ideas to elicit students’ science sense-making. We outline our process with various prompting strategies: few-shot and zero-shot learning (prompting with and without examples), chain of thought (eliciting the MLLM’s reasoning), and sampling strategies (outputting high or low level of randomness). Overall, the AI annotations can reason about the alignment between the assessments and NGSS dimensions and overlap with annotations from K-12 educators. Annotations generated with few-shot learning generally score higher overall and provide more details than zero-shot prompts. Further, interviews with science teachers reveal that the MLLM annotations can support teachers’ reflection on instructional practices and assessment revision. We discuss the application of MLLMs to develop three-dimensional science assessments.