Trustworthy Artificial Intelligence in Psychometrics

Abstract

The availability of sensors, eye-trackers, smartwatches, Wi-Fi trackers, or other digital devices facilitates the collection of new types of data that can be used for measurement. The question is how to analyze them. Several psychometric models are available, but even though they have been applied successfully in many testing programs, they do have their limits with respect to the kind of data they can be applied to. Artificial intelligence (AI) offers many methods for dealing with these new and more complex datasets. They do have some limitations when it comes to reliable and valid measurement thought. The question arises how to apply them in the field of psychometrics. To answer this question, the field of psychometrics is introduced first. Besides, the benefits and disadvantages of artificial intelligence are illustrated in three examples. A promising development, when it comes to the application of AI in the field of psychometrics, is referred to as trustworthy AI (TAI), with principles related to fairness, explainability, and accountability. Based on the examples of the use of AI in social and health science and the lessons learned from the approaches to integrate new data types in existing psychometric models, a framework is defined with nine steps for the use of AI in psychometrics. For each of these steps, it is evaluated how TAI can be applied for reliable and valid measurement. The chapter concludes with the observation that straightforward application of AI in the field of Psychometrics might still be a step too far, but that the developments related to TAI go fast and offer new and exciting opportunities for the application of AI to psychometrics.