Multi-criteria group individual research output evaluation based on context-free grammar judgments with assessing attitude

Abstract

Individual research output (IRO) evaluation is a multi-criteria problem often conducted in groups. In practice, it is necessary to concurrently apply both bibliometric measures and peer review when evaluating the IRO. During the peer review process, different evaluators may use different linguistic terms because of individual differences in cognitive styles, and therefore, they may give ratings based on different assessing attitudes. Further, the weights between bibliometric measures and peer subjective judgments are difficult to determine. Motivated by these difficulties, this paper proposes a quantitative context-free grammar judgment description with an embedded assessing attitude. The proposed method quantitatively handles the assessing attitude and increases the flexibility of the linguistic information. Accordingly, this paper develops a multi-criteria group IRO evaluation method with context-free grammar judgments which concurrently considers bibliometric measures and peer review opinions. To overcome the weighting difficulties and achieve the maximum consensus, this paper proposes a distance-based method to determine the evaluators' weights and a weighted averaging operator to compute the criteria weights. After that, a TOPSIS-based aggregation method is applied to aggregate the objective and subjective ratings. A practical case study is then used to test the feasibility of the methodology. Finally, we discuss the effectiveness of the proposed method.