Finding a safe zone in the highlands: Exploring evaluator competencies in the world of AI

Abstract

Since the public launch of ChatGPT in November 2022, disciplines across the globe have grappled with questions about how emerging artificial intelligence will impact their fields. In this article I explore a set of foundational concepts in artificial intelligence (AI), then apply them to the field of evaluation broadly, and the American Evaluation Association's evaluator competencies more specifically. Given recent developments in narrow AI, I then explore two potential frameworks for considering which evaluation competencies are most likely to be impacted—and potentially replaced—by emerging AI tools. Building on Moravec's Landscape of Human Competencies and Lee's Risk of Replacement Matrix I create an exploratory Landscape of Evaluator Competencies and an Evaluation-Specific Risk of Replacement Matrix to help conceptualize which evaluator competencies may be more likely to contribute to long-term sustainability for the field. Overall, I argue that the interpersonal, and contextually-responsive aspects of evaluation work—in contrast to the more technical, program management, or methodological aspects of the field—may be the competencies least likely to be impacted or replaced by AI. As such, these may be the competencies we continue to emphasize, both in the day-to-day aspects of our operations, and in the training of new and emerging evaluators. This article is intended to be a starting point for discussions that continue throughout the remainder of this issue.