Human-AI Collaborative Sub-Goal Optimization in Hierarchical Reinforcement Learning

Abstract

Hierarchical reinforcement learning often involves human expertise in defining multiple sub-goals to decompose complex objectives into relevant sub-tasks. However, manually specifying these sub-goals is labor-intensive, costly, and prone to introducing biases or misleading the agent. To overcome these challenges, we propose a collaborative human-AI algorithm that seamlessly integrates with hierarchical models to automatically update prior knowledge and optimize candidate sub-goals. Our algorithm can be easily incorporated into a wide range of goal-conditioned frameworks. We evaluate our approach in comparison with relevant baselines, we demonstrate the effectiveness of our algorithm in addressing and preventing negative inferences arising from confusing or conflicting sub-goals. Additionally, our algorithm shows robustness across different levels of human knowledge, accelerating convergence towards optimal sub-goal spaces and hierarchical policies.