Rating scales for collective intelligence in innovation communities: Why quick and easy decision making does not get IT right

Abstract

The increasing popularity of open innovation approaches has lead to the rise of various innovation platforms on the Internet which might contain 10.000s user-generated ideas. However, a company's absorptive capacity is limited regarding such an amount of ideas so that there is a strong need for mechanism to identify the best ideas. Extending previous decision management research we focus on analyzing effective idea rating and selection mechanisms in online innovation communities and underlying explanations. Using a multi-method approach our research comprises a web-based rating experiment with 313 participants evaluating 24 ideas from a real-world innovation community, data from a survey measuring rating satisfaction of participants, and idea ratings from an independent expert jury. Our findings show that, despite its popular use in online innovation communities, simple rating mechanisms such as thumbs up/down rating or 5-star rating do not produce valid idea rankings and are significantly outperformed by the multi-attribute scale.