Automated Flag Detection and Participant Performance Evaluation for Pwnable CTF

Abstract

The demand for cyber security awareness, education, evaluation of learning levels of students etc., has increased in the past few years. In order to meet this rising demand, several cyber security learning and training platforms have been developed. Capture the flag (CTF) platforms and cyber ranges have become primary tools that facilitate education, training and recruitment of cyber security personnel. These tools evaluate and rank the participants on the basis of challenges solved by them. A discrete evaluation mechanism focusing only on flags solved, fails to ensure that the effort and knowledge demonstrated by the participants while solving the challenge, are factored into the scoring system. Most of these tools do not even distinguish between participants actually solving the flags vs. those who might copy a captured flag without actually working on the problem. Further, in flag only scoring systems, participants feel discouraged as they fail to score without finding the flags – despite putting in enormous time and effort. In this paper, we present our novel approach to quantify participant’s learning, efforts, and any unethical practices. We award partial scores by automatically capturing their behavior while solving the CTF problems. We also provide an accurate ranking system with automated solved challenge detection which replaces the need for manual flag submission. In our system, participants get hybrid scores based on their efforts, and organizations get a better and an effective evaluation tool.