A Vision-based Object Detection and Localization System in 3D Environment for Assistive Robots’ Manipulation

Abstract

Robots are used today in many different fields and for many different tasks. However, they quickly reach their limits without a "sense of sight," especially when working directly with humans as assistive robots. Applications that require sophisticated intelligence must work under flexible conditions that are usually not feasible without vision systems. This study proposes a visionbased system for object detection and localization that can potentially be used for assistive robots. The goal is to facilitate Activities of Daily Living (ADL) tasks in an unstructured environment for individuals who use wheelchairs. For vision-based manipulation in an unstructured environment, interest object features and homography analysis are used to get the necessary information for controlling the robotic arm. SSD MobileNet V2, a pre-trained inference model using TensorRT optimized network along with a RealSense camera is used in this study to detect, recognize, and localize objects in the 3D environment. Experiments and results have shown that the system can be utilized to perform object grasping tasks robustly.