Analysis of the suitability of the Korean Federation of Science and Technology Societies journal evaluation tool

Abstract

The Korean Federation of Science and Technology Societies has provided funds to support scientific journals published in Korea. In 2012, it adopted an evaluation tool to assist journals in improving their style and format to meet international standards. This study analysed reliability and fitness of this tool. The 34 evaluation items were tested for goodness of fit, and item analysis was performed for 540 journals based on classical test theory, the Rasch model, and a two-parameter logistic model. The tool was found to be reliable, with a Cronbach’s alpha value of 0.94. Based on the goodness of fit test, the outfit mean square was over 2.0 (i.e. outlier) for the following four items: participation in editor training courses, turnover in editors in the most recent 10 years, publishing at least 40 articles annually, and online accessibility. Three items were found to be the most difficult to be fulfilled: participation in editor training courses, description of the length of time the journal has been publishing, and full-text XML. Above results indicated that this tool was acceptable for measuring the parameters assessed by the items. We concluded that this accreditation system was valid for evaluating a journal’s adherence to international publishing standards.