Thompson Sampling for Real-Valued Combinatorial Pure Exploration of Multi-Armed Bandit

Abstract

We study the real-valued combinatorial pure exploration of the multi-armed bandit (R-CPE-MAB) problem. In R-CPE-MAB, a player is given d stochastic arms, and the reward of each arm s ∈ {1, . . ., d} follows an unknown distribution with mean µs. In each time step, a player pulls a single arm and observes its reward. The player’s goal is to identify the optimal action π∗ = arg max µπ from a finite-sized realπ∈A valued action set A ⊂ Rd with as few arm pulls as possible. Previous methods in the R-CPE-MAB require enumerating all of the feasible actions of the combinatorial optimization problem one is considering. In general, since the size of the action set grows exponentially large in d, this is almost practically impossible when d is large. We introduce an algorithm named the Generalized Thompson Sampling Explore (GenTS-Explore) algorithm, which is the first algorithm that can work even when the size of the action set is exponentially large in d. We also introduce a novel problem-dependent sample complexity lower bound of the R-CPE-MAB problem, and show that the GenTS-Explore algorithm achieves the optimal sample complexity up to a problem-dependent constant factor.