In this paper we derive convergence rates for Q-learning. We show an interesting relationship between the convergence rate and the learning rate used in the Q-learning. For a polynomial learning rate, one which is 1/tω at time t where ω ∈ (1/2, 1), we show that that the convergence rate is polynomial in 1/(1 − γ), where γ is the discount factor. In contrast we show that for a linear learning rate, one which is 1/t at time t, the convergence rate has an exponential dependence on 1/(1 − γ). In addition we show a simple example that proves that this exponential behavior is inherent for a linear learning rate.
CITATION STYLE
Even-Dar, E., & Mansour, Y. (2001). Learning rates for Q-learning. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 2111, pp. 589–604). Springer Verlag. https://doi.org/10.1007/3-540-44581-1_39
Mendeley helps you to discover research relevant for your work.