SOFTWARE Open Access pROC: an open-source package for R and S+ to analyze and compare ROC curves Xavier Robin1*, Natacha Turck1, Alexandre Hainard1, Natalia Tiberti1, Fr��d��rique Lisacek2, Jean-Charles Sanchez1 and Markus M��ller2* Abstract Background: Receiver operating characteristic (ROC) curves are useful tools to evaluate classifiers in biomedical and bioinformatics applications. However, conclusions are often reached through inconsistent use or insufficient statistical analysis. To support researchers in their ROC curves analysis we developed pROC, a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object- oriented and flexible interface. Results: With data previously imported into the R or S+ environment, the pROC package builds ROC curves and includes functions for computing confidence intervals, statistical tests for comparing total or partial area under the curve or the operating points of different classifiers, and methods for smoothing ROC curves. Intermediary and final results are visualised in user-friendly interfaces. A case study based on published clinical and biomarker data shows how to perform a typical ROC analysis with pROC. Conclusions: pROC is a package for R and S+ specifically dedicated to ROC analysis. It proposes multiple statistical tests to compare ROC curves, and in particular partial areas under the curve, allowing proper ROC interpretation. pROC is available in two versions: in the R programming language or with a graphical user interface in the S+ statistical software. It is accessible at http://expasy.org/tools/pROC/ under the GNU General Public License. It is also distributed through the CRAN and CSAN public repositories, facilitating its installation. Background A ROC plot displays the performance of a binary classi- fication method with continuous or discrete ordinal out- put. It shows the sensitivity (the proportion of correctly classified positive observations) and specificity (the pro- portion of correctly classified negative observations) as the output threshold is moved over the range of all pos- sible values. ROC curves do not depend on class prob- abilities, facilitating their interpretation and comparison across different data sets. Originally invented for the detection of radar signals, they were soon applied to psychology [1] and medical fields such as radiology [2]. They are now commonly used in medical decision mak- ing, bioinformatics [3], data mining and machine learning, evaluating biomarker performances or compar- ing scoring methods [2,4]. In the ROC context, the area under the curve (AUC) measures the performance of a classifier and is fre- quently applied for method comparison. A higher AUC means a better classification. However, comparison between AUCs is often performed without a proper sta- tistical analysis partially due to the lack of relevant, accessible and easy-to-use tools providing such tests. Small differences in AUCs can be significant if ROC curves are strongly correlated, and without statistical testing two AUCs can be incorrectly labelled as similar. In contrast a larger difference can be non significant in small samples, as shown by Hanczar et al. [5], who also provide an analytical expression for the variance of AUC���s as a function of the sample size. We recently identified this lack of proper statistical comparison as a potential cause for the poor acceptance of biomarkers as diagnostic tools in medical applications [6]. Evaluating a classifier by means of total AUC is not suitable when * Correspondence: Xavier.Robin@unige.ch markus.mueller@isb-sib.ch 1Biomedical Proteomics Research Group, Department of Structural Biology and Bioinformatics, Medical University Centre, Geneva, Switzerland 2Swiss Institute of Bioinformatics, Medical University Centre, Geneva, Switzerland Full list of author information is available at the end of the article Robin et al. BMC Bioinformatics 2011, 12:77 http://www.biomedcentral.com/1471-2105/12/77 �� 2011 Robin et al licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

the performance assessment only takes place in high specificity or high sensitivity regions [6]. To account for these cases, the partial AUC (pAUC) was introduced as a local comparative approach that focuses only on a portion of the ROC curve [7-9]. Software for ROC analysis already exists. A previous review [10] compared eight ROC programs and found that there is a need for a tool performing valid and stan- dardized statistical tests with good data import and plot functions. The R [11] and S+ (TIBCO Spotfire S+ 8.2, 2010, Palo Alto, CA) statistical environments provide an extensible framework upon which software can be built. No ROC tool is implemented in S+ yet while four R packages computing ROC curves are available: 1) ROCR [12] provides tools computing the perfor- mance of predictions by means of precision/recall plots, lift charts, cost curves as well as ROC plots and AUCs. Confidence intervals (CI) are supported for ROC analy- sis but the user must supply the bootstrapped curves. 2) The verification package [13] is not specifically aimed at ROC analysis nonetheless it can plot ROC curves, compute the AUC and smooth a ROC curve with the binomial model. A Wilcoxon test for a single ROC curve is also implemented, but no test comparing two ROC curves is included. 3) Bioconductor includes the ROC package [14] which can only compute the AUC and plot the ROC curve. 4) Pcvsuite [15] is an advanced package for ROC curves which features advanced functions such as cov- ariate adjustment and ROC regression. It was originally designed for Stata and ported to R. It is not available on the CRAN (comprehensive R archive network), but can be downloaded for Windows and MacOS from http:// labs.fhcrc.org/pepe/dabs/rocbasic.html. Table 1 summarizes the differences between these packages. Only pcvsuite enables the statistical comparison between two ROC curves. Pcvsuite, ROCR and ROC can compute AUC or pAUC, but the pAUC can only be defined as a portion of specificity. The pROC package was designed in order to facilitate ROC curve analysis and apply proper statistical tests for their comparison. It provides a consistent and user- friendly set of functions building and plotting a ROC curve, several methods smoothing the curve, computing the full or partial AUC over any range of specificity or sensitivity, as well as computing and visualizing various CIs. It includes tests for the statistical comparison of two ROC curves as well as their AUCs and pAUCs. The soft- ware comes with an extensive documentation and relies on the underlying R and S+ systems for data input and plots. Finally, a graphical user interface (GUI) was devel- oped for S+ for users unfamiliar with programming. Implementation AUC and pAUC In pROC, the ROC curves are empirical curves in the sensitivity and specificity space. AUCs are computed with trapezoids [4]. The method is extended for pAUCs by ignoring trapezoids outside the partial range and adding partial trapezoids with linear interpolation when necessary. The pAUC region can be defined either as a portion of specificity, as originally described by McClish [7], or as a portion of sensitivity, as proposed later by Jiang et al. [8]. Any section of the curve pAUC(t0, t1) can be analyzed, and not only portions anchored at 100% specificity or 100% sensitivity. Optionally, pAUC can be standardized with the formula by McClish [7]: 1 2 1 + pAUC - min max - min , (1) where min is the pAUC over the same region of the diagonal ROC curve, and max is the pAUC over the Table 1 Features of the R packages for ROC anaylsis Package name ROCR Verification ROC (Bioconductor) pcvsuite pROC Smoothing No Yes No Yes Yes Partial AUC Only SP1 No Only SP1 Only SP SP and SE Confidence intervals Partial2 Partial3 No Partial4 Yes Plotting Confidence Intervals Yes Yes No Yes Yes Statistical tests No AUC (one sample) No AUC, pAUC, SP AUC, pAUC, SP, SE, ROC Available on CRAN Yes Yes No, http://www.bioconductor. org/ No, http://labs.fhcrc.org/pepe/ dabs/ Yes 1Partial AUC only between 100% and a specified cutoff of specificity. 2Bootstrapped ROC curves must be computed by the user. 3Only threshold averaging. 4Only at a given specificity or inverse ROC. Robin et al. BMC Bioinformatics 2011, 12:77 http://www.biomedcentral.com/1471-2105/12/77 Page 2 of 8