DISCOVERING INSIGHTS IN LEARNING ANALYTICS THROUGH A MIXED-METHODS FRAMEWORK: APPLICATION TO COMPUTER PROGRAMMING EDUCATION

Abstract

Aim/Purpose This article proposes a framework based on a sequential explanatory mixed-methods design in the learning analytics domain to enhance the models used to support the success of the learning process and the learner. The framework consists of three main phases: (1) quantitative data analysis; (2) qualitative data analysis; and (3) integration and discussion of results. Furthermore, we illus-trated the application of this framework by examining the relationships between learning process metrics and academic performance in the subject of Computer Programming coupled with content analysis of the responses to a students’ per-ception questionnaire of their learning experiences in this subject. Background There is a prevalence of quantitative research designs in learning analytics, which limits the understanding of students’ learning processes. This is due to the abundance and ease of collection of quantitative data in virtual environ-ments and learning management systems compared to qualitative data. Methodology This study uses a mixed-methods, non-experimental, research design. The quan-titative phase of the framework aims to analyze the data to identify behaviors, trends, and relationships between measures using correlation or regression anal-ysis. On the other hand, the qualitative phase of the framework focuses on con-ducting a content analysis of the qualitative data. This framework was applied to historical quantitative and qualitative data from students’ use of an automated feedback and evaluation platform for programming exercises in a programming course at the National University of Colombia during 2019 and 2020. The re-search question of this study is: How can mixed-methods research applied to learning analytics generate a better understanding of the relationships between the variables generated throughout the learning process and the academic per-formance of students in the subject of Computer Programming? Contribution The main contribution of this work is the proposal of a mixed-methods learn-ing analytics framework applicable to computer programming courses, which al-lows for complementing, corroborating, or refuting quantitatively evidenced re-sults with qualitative data and generating hypotheses about possible causes or explanations for student behavior. In addition, the results provide a better un-derstanding of the learning processes in the Computer Programming course at the National University of Colombia. Findings A framework based on sequential explanatory mixed-methods design in the field of learning analytics has been proposed to improve the models used to support the success of the learning process and the learner. The answer to the research question posed corresponds to that the mixed methods effectively complement quantitative and qualitative data. From the analysis of the data of the application of the framework, it appears that the qualitative data, represent-ing the perceptions of the students, generally supported and extended the quan-titative data. The consistency between the two phases allowed us to generate hy-potheses about the possible causes of student behavior and provide a better un-derstanding of the learning processes in the course. Recommendations for Practitioners We suggest implementing the proposed mixed-methods learning analytics framework in various educational contexts and populations. By doing so, practi-tioners can gather more diverse data and insights, which can lead to a better un-derstanding of learning processes in different settings and with different groups of learners. Recommendations for Researchers Researchers can use the proposed approach in their learning analytics projects, usually based exclusively on quantitative data analysis, to complement their re-sults, find explanations for their students’ behaviors, and understand learning processes in depth thanks to the information provided by the complementary analysis of qualitative data. Impact on Society The prevalence of exclusively quantitative research designs in learning analytics can limit our understanding of students’ learning processes. Instead, the mixed-methods approach we propose suggests a more comprehensive approach to learning analytics that includes qualitative data, which can provide deeper in-sight into students’ learning experiences and processes. Ultimately, this can lead to more effective interventions and improvements in teaching and learning practices. Future Research Potential lines of research to continue the work on mixed-method learning ana-lytics methodology include the following: first, implementing the framework on a different population sample, such as students from other universities or other knowledge areas; second, using techniques to correct unbalanced data sets in learning analytics studies; third, analyzing student interactions with the auto-mated grading platform and their academic activities in relation with their activ-ity grades; last, using the findings to design interventions that positively impact academic performance and evaluating the impact statistically through experimental study designs. In the context of introductory programming educa-tion, AI/large language models have the potential to revolutionize teaching by enhancing the learning experience, providing personalized support, and ena-bling more efficient assessment and feedback mechanisms. Future research in this area is to implement the proposed framework on data from an introductory programming course using these models.