RF Vector Modulator

Abstract

Probabilistic classifiers are developed by assuming generative mod-els which are product distributions over the original attribute space (as in naiveBayes) or more involved spaces (as in general Bayesian networks). While thisparadigm has been shown experimentally successful on real world applications,despite vastly simplified probabilistic assumptions, the question of why these ap-proaches work is still open.This paper resolves this question. We show that almost all joint distributions witha given set of marginals (i.e., all distributions that could have given rise to the clas-sifier learned) or, equivalently, almost all data sets that yield this set of marginals,are very close (in terms of distributional distance) to the product distribution onthe marginals; the number of these distributions goes down exponentially withtheir distance from the product distribution. Consequently, as we show, for almostall joint distributions with this set of marginals, the penalty incurred in using themarginal distribution rather than the true one is small. In addition to resolving thepuzzle surrounding the success of probabilistic classifiers our results contributeto understanding the tradeoffs in developing probabilistic classifiers and will helpin developing better classifiers.