An architecture for transforming graphical interfaces

Abstract

While graphical user interfaces have gained much popularity in recent years, there are situations when the need to use existing applications in a nonvisuat modrdity is clear. Examples of such situations include the use of applications on hand-held devices with limited screen space (or even no screen space, as in the case of telephones), or users with visual impairments. We have developed an architecture capable of transforming the graphical interfaces of existing applications into powerful and intuitive nonvisual interfaces. Our system, called Mercator, provides new input and output techniques for working in the nonviswd domain. Navigation is accomplished by traveming a hierarchical tree representation of the interface structure. Output is primarily auditory, although other output modalities (such as tactile) can be used as well. The mouse, an inherently visually-oriented device, is replaced by keyboard and voice interaction. Our system is currently in its third major revision. We have gained insight into both the nonvisuat interfaces presented by our system and the architecture necessm to construct such interfaces. This architecture uses several novel techniques to efficiently and flexibly map graphical interfaces into new modalities.