Assessing AI Literacy at University Entry: A Mixed-Item Diagnostic Instrument

Abstract

As artificial intelligence (AI) tools increasingly shape academic, professional, and creative practices, higher education needs ways to assess students' AI literacy. This poster presents the development and exploratory evaluation of a mixed-item-format AI literacy questionnaire at university entry. The instrument was administered as a baseline during the first weeks of the semester to first-year students in design and information science programmes. It combines self-report rating-scale items across cognitive, behavioural, affective, and ethical dimensions. These are complemented by embedded performance-oriented indicators, including a selected-response knowledge item and open-response prompts assessing articulated AI use and ethical reasoning. The complete questionnaire is publicly available at https://doi.org/10.5281/zenodo.18759246. Self-report results indicated comparatively higher perceived competence in the behavioural and ethical dimensions, whereas perceived cognitive competence was lower. In contrast, open-responses often showed limited elaboration of concrete AI-use scenarios and ethical implications. In the cognitive dimension, comparisons between self-ratings and performance on the knowledge item revealed a misalignment between perceived and demonstrated knowledge at entry. These initial results support the use of the instrument as a baseline, multi-format diagnostic measure, to be iteratively refined within a design-based research cycle. We discuss implications for refining the questionnaire, and how it can inform the improvement of early-semester AI teaching formats in design and information science curricula.