Crafting policy discussion prompts as a task for newcomers

Abstract

Inspired by policy deliberation methods and iterative writing in crowdsourcing, we developed and evaluated a task in which newcomers to an online policy discussion, before entering the discussion, generate prompts that encourage existing commenters to engage with each other. In an experiment with 453 Amazon Mechanical Turk (AMT) crowd workers, we found that newcomers can often craft acceptable prompts, especially when given guidance on prompt-writing and balanced opinions between the comments they synthesize. However, crafting these prompts had little effect on the quality of comments they posted to a simulated discussion forum following the prompt task, as measured by the reasoning and topic coherence of comments. Our results inform best practices and pose questions for the design of discussion systems, both in general and for online policy discussion in particular.