Hamcrest vs AssertJ: An Empirical Assessment of Tester Productivity

Abstract

Context. Extensive unit testing is worth its costs in terms of the higher quality of the final product and reduced development expenses, though it may consume more than fifty percent of the overall project budget. Thus, even a tiny percentage of saving can significantly decrease the costs. Since recently competing assertion libraries emerged, we need empirical evidence to gauge them in terms of developer productivity, allowing SQA Managers and Testers to select the best. Objective. The aim of this work is comparing two assertion frameworks having a different approach (matchers vs. fluent assertions) w.r.t. tester productivity. Method. We conducted a controlled experiment involving 41 Bachelor students. AssertJ is compared with Hamcrest, in a test development scenario with the Java language. We analysed the number of correct assertions developed in a tight time frame and used this measure as a proxy for tester productivity. Results. The results show that adopting AssertJ improves the overall tester’s productivity significantly during the development of assertions. Conclusions. Testers and SQA managers selecting assertion frameworks for their organizations should consider as first choice AssertJ, since our study shows that it increases the productivity of testers during development more than Hamcrest.