Using a multi-order cognitive diagnosis model to assess scientific literacy

Abstract

In PISA 2015, scientific literacy is defined as “the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen”. There are four interdependent dimensions are specified in the scientific literacy assessment framework for PISA 2015: Competencies, Knowledge, Contexts, and Attitudes. Given that knowledge of scientific literacy contributes significantly to individuals’ personal, social, and professional lives, it is of vital importance to find an objectively and accurately assessment method for scientific literacy. However, only unidimensional IRT models were used in the analysis in PISA 2015. Which means that the analysis model does not match with such a multidimensional assessment framework. It is desired to develop a new analysis model. This study attempts to measure scientific literacy in cognitive diagnostic assessment for the first time. According to the scientific literacy assessment framework for PISA 2015, a third-order latent structure for scientific literacy is first pointed out. Specifically, the scientific literacy is treated as the third-order latent trait; Competencies, Knowledge, Contexts, and Attitudes are all treated as second-order latent traits; And nine subdomains, e.g., explain phenomena scientifically and content knowledge, were treated as first-order traits (or attributes). Unfortunately, however, there is still a lack of cognitive diagnosis models that can deal with such a third-order latent structure. To this end, a multi-order DINA (MO-DINA) model was developed in this study. The new model is an extension of the higher-order (HO-DINA) model, which is similar to the third-order IRT models. To illustrate the application and advantages of the MO-DINA model, a sub-data of PISA 2015 science assessment data were analyzed. Items were chosen from the S01 cluster, and participants were chosen from China. After data cleaning, 1076 participants with 18 items were retained. Three models were fitted to this sub-data and compared, the MO-DINA model, in which the third-order latent structure of scientific literacy was considered; the HO-DINA model, in which the scientific literacy was treated as a second-order latent trait and contacted with attributes directly; and the DINA model. All three models appear to provide a reasonably good fit to data according to the posterior predictive model checking. According to the –2LL, AIC, BIC, and DIC, the DINA model fits the data worst, and the MO-DINA model fits the data best, the results of MO-DINA model are used to make further interpretations. The results indicated that (1) the quality of 18 items are not good enough; (2) The correlations among second-order latent traits are high (0.8, approximately); (3) Knowledge has the greatest influence on scientific literacy, Contexts second, and Competencies least; (4) Explain phenomena scientifically, procedural knowledge, and local/national has the greatest influence on Competencies, Knowledge, and Contexts, respectively. In addition, a simulation study was conducted to evaluate the psychometric properties of the proposed model. The results showed that the proposed Bayesian MCMC estimation algorithm can provide accurate model parameter estimation. Overall, the proposed MO-DINA model works well in real data analysis and simulation study and meets the needs of assessment for PISA 2015 scientific literacy which included a third-order latent structure.