Evaluating compositionality of sentence representation models

Abstract

We evaluate the compositionality of general-purpose sentence encoders by proposing two metrics to quantify compositional understanding capability of sentence encoders. We introduce a novel metric, Polarity Sensitivity Scoring (PSS), which utilizes sentiment perturbations as a proxy for measuring compositionality. We then compare results from PSS with those obtained via our proposed extension of a metric called Tree Reconstruction Error (TRE) (Andreas, 2019) where compositionality is evaluated by measuring how well a true representation-producing model can be approximated by a model that explicitly combines representations of its primitives.