REPR: Rule-Enhanced Penalized Regression

Abstract

This article describes a new rule-enhanced penalized regression procedure for the generalized regression problem of predicting scalar responses from observation vectors in the absence of a preferred functional form. It enhances standard L 1 -penalized regression by adding dynamically generated rules, that is, new 0-1 covariates, corresponding to multidimensional “box” sets. In contrast to prior approaches to this class of problems, we draw heavily on standard (but non-polynomial-time) mathematical programming techniques, enhanced by parallel computing. We identify and incorporate new rules using a form of classical column generation and solve the resulting pricing subproblem, which is NP-hard, either exactly by a specialized parallel branch-and-bound method or by a greedy heuristic based on Kadane’s algorithm. The resulting rule-enhanced regression method can be computation intensive when we solve the subproblems exactly, but our computational tests suggest that it outperforms prior methods at making accurate and stable predictions from relatively small data samples. Through selective use of our greedy heuristic, we can make our method’s run time generally competitive with some established methods, without sacrificing prediction performance. We call our method’s pricing subproblem rectangular maximum agreement.