An Examination of Ranking Quality for Simulated Pairwise Judgments in relation to Performance of the Selected Consistency Measure

Abstract

An overview of current debates and contemporary research devoted to modeling decision making processes and their facilitation directs attention to techniques based on pairwise judgments. At the core of these techniques are various judgment consistency measures which, in a sense, control the prioritization process which leads to the establishment of decision makers' unknown preferences. If judgments expressed by decision makers were perfectly consistent (cardinally transitive), all available prioritization techniques would deliver the same solution. However, human judgments are consistently inconsistent, as it were; thus the preference estimation quality significantly varies. The scale of these variations depends, among others, on the chosen consistency measure of pairwise judgments. That is why it seems important to examine relations among various consistency measures and the preferences estimation quality. This research reveals that there are consistency measures whose performance may confuse decision makers with the quality of their ranking outcome. Thus, it introduces a measure which is directly related to the quality of the preferences estimation process. The main problem of the research is studied via Monte Carlo simulations executed in Wolfram Mathematica Software. The research results argue that although the performance of examined consistency measures deviates from the exemplary ones in relation to the estimation quality of decision makers preferences, solutions proposed in this paper can significantly improve that quality.