Machine Learning Techniques to Evaluate Lesson Objectives

Abstract

The advancement of knowledge in medicine presents an important challenge when identifying gaps and deciding what content to include in a medical school curriculum and how to establish learning outcomes. Monitoring alignment between lesson objectives, the curriculum and achievement of intended outcomes can be difficult. A system that can automatically evaluate lesson objectives would be highly beneficial. We aim to assess the efficacy of using machine learning techniques to evaluate individual lesson objectives to a graduate entry allopathic medical school curriculum. The school’s curriculum objectives consist of 11 categories and 356 curriculum objectives sentences. We considered the first year courses with a total of 1888 lesson objectives. Using various word embeddings (TF-IDF, word2vec, fastText, BioBERT), we then use cosine similarity to map each lesson objective to the curriculum objectives. Cognitive levels of lesson objectives were compared against the school’s curriculum using Bloom’s Taxonomy verbs. After implementation, 319 lesson objectives from each approach were randomly sampled (sample size, 95% CL, 5% CI) to examine match with curriculum objectives and curriculum categories. BioBERT performed best with 46.71% and 80.56% match between lesson objectives and curriculum objectives, and lesson objectives and categories, respectively. Further validation by a domain expert shows 80% match (without order). Visualisation of the Bloom’s Taxonomy cognitive levels of lesson objectives and school’s curriculum objectives showed a good match. Machine learning can be used to evaluate lesson and curriculum and automatically mapping lesson objectives to the medical school curriculum and analysing cognitive levels of lesson objectives.