sbv IMPROVER Diagnostic Signature Challenge

Abstract

Evaluating the performance of computational methods to analyze high throughput data are an integral component of model development and critical to progress in computational biology. In collaborative-competitions, model performance evaluation is crucial to determine the best performing submis- sion. Here we present the scoring methodology used to assess 54 submissions to the IMPROVER Diagnostic Signature Chal- lenge. Participants were tasked with classifying patients’ dis- ease phenotype based on gene expression data in four disease areas: Psoriasis, Chronic Obstructive Pulmonary Disease, Lung Cancer, and Multiple Sclerosis. We discuss the criteria underly- ing the choice of the three scoring metrics we chose to assess the performance of the submitted models. The statistical sig- nificance of the difference in performance between individual submissions and classification tasks varied according to these different metrics. Accordingly, we consider an aggregation of these three assessment methods and present the approaches considered for aggregating the ranking and ultimately deter- mining the final overall best performer.