Assessing vocational competencies in civil engineering: Lessons from AHELO for future practice

Abstract

Background: There has been much interest in the notion of vocational competencies of late. The pressing question, however, is how to measure vocational competencies. In order to contribute to this 'how', the results of the Civil Engineering strand of the recent Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study are discussed. The focus of the assessment instrument was explicitly on " above content" areas of the domain, and the assessment framework provides an example of a construct which articulates certain domain-specific competence components. This paper draws specific attention the Constructed Response Tasks (CRTs) in the instrument and discusses the results of the study with a focus on the lessons to be learnt from AHELO for future practice. Methods: The assessment framework for the Civil Engineering strand was developed in line with the worldwide move to increase focus onto graduate competencies. The CRTs aimed to engage students with interesting, real-world problems. The tasks were geared towards finding out whether students could think like an engineer and display the non-technical competencies required of practicing engineers. Some items and the scoring rubrics are examined. Results: In the Civil Engineering strand, 9 countries participated comprising 92 universities and 6,078 students. Content and construct validity were arguably achieved, as was reasonable reliability and good inter-rater scorer reliability. Overall, the cohort of engineering students worldwide found the tasks too difficult. Surprisingly, there was a large proportion of " zero" scores for the CRTs. It seems that either many final year engineering students are lacking in many of the vocational competencies required of them as they enter the workforce, or the experts and test developers had unrealistic expectations of them. It is difficult to ascertain whether the items were truly too difficult or if there were motivational and/or test targeting issues at play, especially due to the low-stakes nature of the implementation. Conclusions: As the AHELO study shows, this type of undertaking is significantly complex. There is much room for improvement for future large-scale competency based assessments of this kind. In designing future work for measuring vocational competencies, lessons learnt from AHELO should be duly considered.