Continuous-time bandits

Abstract

For processes that are evolving and can be observed continuously, we would like to formulate a bandit problem in a manner analogous to our formulation for discrete time. First, we would like a strategy to indicate a process to select and observe at time t, for all t, depending on the history of observations prior to time t. Then we would define the worth of a strategy to be a weighted sum of the incremei,.s of the processes during the time they are being observed. Unfortunately, there are serious technical difficulties with various aspects of this plan. These difficulties are frequently ignored in the literature. While this may not be unreasonable when there are good discrete-time approximations, the technical problems must be understood and overcome before progress can be made on the general problem.