Benchmarking Inference Algorithms for Probabilistic Relational Models

Abstract

In the absence of benchmark datasets for inference algorithms in probabilistic relational models, we propose an extendable benchmarking suite named ComPI that contains modules for automatic model generation, model translation, and inference benchmarking. The functionality of ComPI is demonstrated in a case study investigating both average runtimes and accuracy for multiple openly available algorithm implementations. Relatively frequent execution failures along with issues regarding, e.g., numerical representations of probabilities, show the need for more robust and efficient implementations for real-world applications.