Weak Ground Truth Determination of Continuous Human-Rated Data

1Citations
Citations of this article
6Readers
Mendeley users who have this article in their library.

This article is free to access.

Abstract

The article presents a novel weak ground truth (WGT) determination procedure on continuous human-rated data. The notion of WGT is essential in cases where there is no direct empirical evidence for an observed construct and human annotations provide the most reliable means for determining the ground truth. The core idea behind the proposed procedure is to transform the ratings to reduce rater bias, maximize interrater agreement, and improve WGT. The procedure was evaluated on two behavioral datasets containing continuous annotations of several expressive dimensions. The results show that the procedure improves the size of WGT data by removing the disagreement originating from rater-specific distortions, such as rater mean and scaling bias. The entropy of residuals decreases after WGT optimization, meaning that more relevant information is retained. The average improvement of WGT data size is between 10.1 and 20.9 percentage points, depending on the respective dimension. However, in cases where the rater bias is small, the procedure does not substantially modify WGT. This indicates that the proposed optimization only removes rater biases derived from rater-specific distortions, while retaining and improving valid WGT. The proposed procedure is generalizable on any type and size of continuous or discrete numerical data where multiple raters are involved.

Cite

CITATION STYLE

APA

Košir, A., Strle, G., & Meža, M. (2021). Weak Ground Truth Determination of Continuous Human-Rated Data. IEEE Access, 9, 4594–4606. https://doi.org/10.1109/ACCESS.2020.3046293

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free