Crowd sourcing remote comparative lameness assessments for dairy cattle

Abstract

Lameness assessments are rarely conducted routinely on dairy farms and when completed typically underestimate lameness prevalence, hampering early diagnosis and treatment. A well-known feature of many perceptual tasks is that relative assessments are more accurate than absolute assessments, suggesting that creating methods that allow for the relative scoring of which cow is more lame will allow for reliable lameness assessments. Here we developed and tested a remote comparative lameness assessment method: we recruited nonexperienced crowd workers via an online platform and asked them to watch 2 videos side-by-side, each showing a cow walking, and to identify which cow was more lame and by how much (on a scale of −3 to 3). We created 11 tasks, each with 10 video pairs for comparison, and recruited 50 workers per task. All tasks were also completed by 5 experienced cattle lameness assessors. We evaluated data filtering and clustering methods based on worker responses and determined the agreement among workers, among experienced assessors, and between these groups. A moderate to high interobserver reliability was observed (intraclass correlation coefficient, ICC = 0.46 to 0.77) for crowd workers and agreement was high among the experienced assessors (ICC = 0.87). Average crowd-worker responses showed excellent agreement with the average of experienced assessor responses (ICC = 0.89 to 0.91), regardless of data processing method. To investigate if we could use fewer workers per task while still retaining high agreement with experienced assessors, we randomly subsampled 2 to 43 (1 less than the minimum number of workers retained per task after data cleaning) workers from each task. The agreement with experienced assessors increased substantially as we increased the number of workers from 2 to 10, but little increase was observed after 10 or more workers were used (ICC > 0.80). The proposed method provides a fast and cost-effective way to assess lameness in commercial herds. In addition, this method allows for large-scale data collection useful for training computer vision algorithms that could be used to automate lameness assessments on farm.