Improving the prediction of ranking data

Abstract

By using the same number of alternatives for every respondent, all ranking elicitation methods in the literature including full, partial, and best–worst rankings assume respondents know and are able to rank the same number of alternatives. A simple survey elicitation mechanism allowing for individual heterogeneity in the number of rankings for ranked-ordered data is proposed. Using the proposed ranking mechanism as a data augmentation tool yields higher prediction of ranking choices compared to conventional rankings and best–worst methods. The results provide robust evidence of differences in error variance scale and the structure of the underlying utility preferences across ranking stages, including best–worst rankings. The highest predictive power was achieved with the proposed ranking method using only the best ranked alternative. Including any additional rankings other than the best alternative reduces predictive power. Nevertheless, if more than one ranking is used to model preferences, then better predictions are achieved by using the top two best ranked alternatives as supposed to the exploded best–worst rankings. The results stand as a warning about equating ranking choices to true underlying utility preferences across different ranking elicitation stages or mechanisms without properly testing for symmetry and stability of preferences.