Video Processing from a Virtual Unmanned Aerial Vehicle: Comparing Two Approaches to Using OpenCV in Unity

Abstract

Virtual reality (VR) simulators enable the evaluation of engineering systems and robotic solutions in safe and realistic environments. To do so, VR simulators must run algorithms in real time to accurately recreate the expected behaviour of real-life processes. This work was aimed at determining a suitable configuration for processing images taken from a virtual unmanned aerial vehicle developed in Unity using OpenCV. To this end, it was focused on comparing two approaches to integrate video processing in order to avoid potential pitfalls such as delays and bottlenecks. The first approach used a dynamic link library (DLL) programmed in C++, and the second an external module programmed in Python. The native DLL ran internally on the same Unity thread, as opposed to the Python module that ran in parallel to the main process and communicated with Unity through the Message Queue Telemetry Transport (MQTT) protocol. Pre-transmission processing, data transmission and video processing were evaluated for a pair of typical image-processing tasks like colour and face detection. The analysis confirmed that running the Python module in parallel does not overload the main Unity thread and achieves better performance than the C++ plugin in real-time simulation.