Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review Norsaremah Salleh, Emilia Mendes, and John C. Grundy, Member, IEEE Abstract���The objective of this paper is to present the current evidence relative to the effectiveness of pair programming (PP) as a pedagogical tool in higher education CS/SE courses. We performed a systematic literature review (SLR) of empirical studies that investigated factors affecting the effectiveness of PP for CS/SE students and studies that measured the effectiveness of PP for CS/SE students. Seventy-four papers were used in our synthesis of evidence, and 14 compatibility factors that can potentially affect PP���s effectiveness as a pedagogical tool were identified. Results showed that students��� skill level was the factor that affected PP���s effectiveness the most. The most common measure used to gauge PP���s effectiveness was time spent on programming. In addition, students��� satisfaction when using PP was overall higher than when working solo. Our meta-analyses showed that PP was effective in improving students��� grades on assignments. Finally, in the studies that used quality as a measure of effectiveness, the number of test cases succeeded, academic performance, and expert opinion were the quality measures mostly applied. The results of this SLR show two clear gaps in this research field: 1) a lack of studies focusing on pair compatibility factors aimed at making PP an effective pedagogical tool and 2) a lack of studies investigating PP for software design/modeling tasks in conjunction with programming tasks. Index Terms���Empirical studies, pair programming, systematic review. �� 1 INTRODUCTION PAIR programming (PP) involves two people sitting side by side, using only one computer and working collaboratively on the same design, algorithm, code, or test [4]. One is the ���driver,��� who is responsible for designing, typing the code, and having control over the shared resource (e.g., computer, mouse, and keyboard). The second is the ���navigator��� or ���observer,��� who has responsibility for observing how the driver works in order to detect errors and offer ideas in solving a problem. Throughout their work, pairs typically alternate their roles after a certain duration [70]. PP���s popularity has drawn the attention of many researchers, thus causing an increase in the number of studies conducted in both industrial as well as in educational contexts [1]. A survey of organizations from a software process improvement user group showed that 72 percent of the organizations from a variety of industries have implemented the PP practice [41]. Some studies have investigated PP���s usefulness and effectiveness as a Computer Science/Software Engineering (CS/SE) pedago- gical tool, e.g., [46], [47], [53], [18], [59], some with promising results. Early research on the use of PP as a pedagogical tool focused mainly on its ability to benefit students in terms of productivity and quality of work produced [69]. For example, evidence suggests that PP could enhance enjoy- ment [46], [47], [69], [73], increase students��� confidence level [5], [46], [28], reduce workload [9], improve course completion rate [46], [53], increase homework submission rate [28], improve exam performance [47], [49], [53], and facilitate working more efficiently on programming tasks [11], [67]. In 2000, Cockburn and Williams [12] investigated the cost and benefit of PP based on empirical evidence [71], [68], [55]. They concluded that with an increase of only 15 percent in the cost of development time, PP offers significant benefits such as improving design quality (fewer defects), team communication, and rapid solutions to problems, enhancing the learning process, and increasing enjoyment in learning. They suggest that PP is a promising approach to use as a pedagogical tool due to its capability of increasing learning capacity [12]. Dyba et al. [19] conducted a systematic literature review (SLR) investigating whether existing empirical evidence supports the claims that PP is more advantageous than solo programming. They reviewed 15 studies comparing solo and pairprogramming,andinvolvingbothstudentsandsoftware practitioners as subjects. The general aspects investigated wererelated to PP���seffectiveness, including ���duration��� (time spent to produce the system), ���effort��� (person-hours spent), and ���quality of the final product.��� Their meta-analysis IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 37, NO. 4, JULY/AUGUST 2011 509 . N. Salleh is with the Department of Computer Science, International Islamic University Malaysia, PO Box 10, 50728 Kuala Lumpur, Malaysia, and the Department of Computer Science, University of Auckland, Private Bag 92019, Auckland Mail Centre, Auckland 1142, New Zealand. E-mail: nsal017@ec.auckland.ac.nz, norsaremah@iium.edu.my. . E. Mendes is with the Department of Computer Science, University of Auckland, Private Bag 92019, Auckland Mail Centre, Auckland 1142, New Zealand. E-mail: emilia@cs.auckland.ac.nz. . J.C. Grundy is with the Faculty of Information and Communication Technologies, Swinburne University of Technology, PO Box 218, Hawthorn, Victoria 3122, Australia. E-mail: jgrundy@swin.edu.au. Manuscript received 20 Oct. 2008 revised 13 July 2009 accepted 30 Apr. 2010 published online 26 May 2010. Recommended for acceptance by P. Strooper. For information on obtaining reprints of this article, please send e-mail to: tse@computer.org, and reference IEEECS Log Number TSE-2008-10-0357. Digital Object Identifier no. 10.1109/TSE.2010.59. 0098-5589/11/$26.00 �� 2011 IEEE Published by the IEEE Computer Society