TaskLint: Automated Detection of Ambiguities in Task Instructions

Abstract

Clear instructions are a necessity for obtaining accurate results from crowd workers. Even small ambiguities can force workers to choose an interpretation arbitrarily, resulting in errors and inconsistency. Crisp instructions require significant time to design, test, and iterate. Recent approaches have engaged workers to detect and correct ambiguities. However, this process increases the time and money required to obtain accurate, consistent results. We present Task Lint, a system to automatically detect problems with task instructions. Lever aging a diverse set of existing NLP tools, Task Lint identifies words and sentences that might foretell worker confusion. This is analogous to static analysis tools for code (“linters”),which detect possible features in code that might indicate the presence of bugs. Our evaluation of Task Lint using task instruction screated by novices confirms the potential for static tools to improve task clarity and the accuracy of results, while also highlighting several challenges.