An intelligent testing system development based on the shingle algorithm for assessing humanities students' academic achievements

Abstract

Computer-based testing of humanities students has some inconveniences and difficulties, where the whole learning process is practically based on communicative methods. In this regard, one needs such a testing system, which would allow one to ask open-ended questions, and students would be able to enter detailed answers. Despite the popularity of using the shingle algorithm in determining plagiarism, few researchers have attempted to use it in assessing the academic achievements of students. In this regard, the aim of this study was to develop an intelligent testing system based on the shingle algorithm in assessing the academic achievements of humanities students. Taking into account that during testing humanities students will formulate answers of their own understanding, the developed system should be able to determine the degree of their identity to the correct answer. At the same time, answers with a high degree of correspondence to the answer stored in the dictionary should also be entered in the database as one of the variants of the correct answer. The shingle algorithm, stemming, and MD5 hashing algorithms were used to achieve this goal. The performance of the algorithm was evaluated in terms of degree of matching (S), completeness (P), F-measure and performance (t). The experiment involved 120 humanities students in 2–3 courses at the age of 18–20 years, including 80 girls and 40 boys. It was found that the effectiveness of the developed algorithm is achieved at the optimal time t = 77% and the degree of compliance of the final grade F = 77%. In this case, the final score of the F-measure fully reflects the result at the proportion of truthfulness equal to 0.5 and is directly proportional to the degree of compliance (S) and completeness (P) of use. It is found that a high value of the matching degree (S) is achieved with a smaller shingle length, while with a larger shingle length the matching degree decreases, thus, the probability of finding the same phrase in two documents increases. In addition, with smaller shingle lengths, the time spent calculating checksums is longer, and with larger shingle lengths, the time spent calculating checksums is shorter. Calculations showed that the optimal shingle algorithm efficiency was at the length of the shingle N = 5 of the average data processing time. The results of this study show that the developed algorithm can be included in pedagogical practice in order to objectively assess the learning achievements of humanities students, taking into account their communicative and cognitive abilities. In the future, the developed algorithm can also be used in other areas requiring text analysis, in particular for checking plagiarism.