Genet: Automatic curriculum generation for learning adaptation in networking

Abstract

As deep reinforcement learning (RL) showcases its strengths in networking, its pitfalls are also coming to the public's attention. Training on a wide range of network environments leads to suboptimal performance, whereas training on a narrow distribution of environments results in poor generalization. This work presents Genet, a new training framework for learning better RL-based network adaptation algorithms. Genet is built on curriculum learning, which has proved effective against similar issues in other RL applications. At a high level, curriculum learning gradually feeds more "difficult"environments to the training rather than choosing them uniformly at random. However, applying curriculum learning in networking is nontrivial since the "difficulty"of a network environment is unknown. Our insight is to leverage traditional rule-based (non-RL) baselines: If the current RL model performs significantly worse in a network environment than the rule-based baselines, then further training it in this environment tends to bring substantial improvement. Genet automatically searches for such environments and iteratively promotes them to training. Three case studies-Adaptive video streaming, congestion control, and load balancing-demonstrate that Genet produces RL policies that outperform both regularly trained RL policies and traditional baselines.