An Open Framework for Nonverbal Communication in Human-Robot Interaction

Abstract

Nonverbal communication plays a vital role in human interaction. In the context of Human-Robot Interaction (HRI), social robots are designed primarily for verbal-based communication with humans, making nonverbal communication an open research area. We present a flexible, open framework designed to facilitate nonverbal interactions in HRI. Among its components is a P2P Browser-Based Computational Notebook, leveraged to code, run, and share reactive programs. Machine-learning models can be included for real-time recognition of gestures, poses, and moods, employing protocols such as MQTT. Another key component is a broker for distributing data among different physical devices like the robot, wearables, and environmental sensors. We demonstrate this framework’s utility through two interaction scenarios: (i) the first one employing proxemics and gaze direction to initiate an impromptu encounter, and (ii) a second one incorporating object recognition and a Large-Language Model to suggest meals to be cooked based on available ingredients. These scenarios illustrate how the framework’s components can be seamlessly integrated to address new scenarios, where robots need to infer nonverbal cues from users.