A Design of Interface for Visual-Impaired People to Access Visual Information from Images Featuring Large Language Models and Visual Language Models

Abstract

We propose a design of interface for visual-impaired People to access visual information from images utilizing Large Language Models( LLMs), Visual Language Models (VLMs), and Segment-Anything. We use Semantic-Segment-Anything to generate the segmentation of semantic objects in images. The segmentation includes two parts: a term set describing the semantic object, and segmented mask which represents the shape of the semantic object. We provide two methods for the visual-impaired user to access the information of the semantic object and its peripheral information in image. In one method, the LLM summarize the term set to create an description. In the other method, the image with the object masked is provided to Visual Language Models which is prompted to respond with a description. In both methods, the mask can be accessed with dot display after processed for the visual-impaired people to access, and the description is prompted to the user in synthesized voice.