The gMenu User Interface for Virtual Reality
Systems and Environments
Andrew Dunk and Adrian Haegee
Centre for Advanced Computing and Emerging Technologies
The University of Reading
a.dunk@reading.ac.uk
http://www.acet.reading.ac.uk
Abstract. Desktop computers are able to provide a user interface with
many features that allow the user to perform tasks such as execute ap-
plications load les and edit data. The gMenu system proposed in this
paper is a step closer to having these same facilities in virtual reality
systems. The gMenu can currently be used to perform a selection of
common tasks provided by a user interface, for example executing or
closing virtual reality applications or scenes. It is fully customisable and
can be used to create many dierent styles of menu by both program-
mers and users. It also has shown promising results bringing some of the
system based commands into the virtual environment, as well as keeping
the functionality and adaptions required by applications. The use cases
presented demonstrate a collection of these abilities.
Key words: 3D User Interfaces, Virtual Reality
1 Introduction
The development of quality 3D user interfaces and virtual environments can be
complicated and time consuming. Currently there are no recognised standards
on which to base the design and implementation of 3D user interfaces. Questions
regarding interaction techniques, such as selections, travelling and way nding
may have been answered already [1]. The area of environment and system control
however, has not been paid as much attention.
Every computer needs an operating system that allows the user to utilise
the computer hardware in an ecient way. These operating systems are com-
bined with user interfaces to enhance its usability and usefulness. This research
intends to be a step towards a three dimensional operating system for virtual
environments. It proposes combining a customisable menu system, currently be-
ing developed as part of this research, with a virtual reality frameworks such
as VieGen [2] which is currently being used for testing the gMenu system. This
combination will supply equivalents for the more common operations provided
by today's desktop operating systems. These will include simple le browsing,
application execution, system controls (such as audio settings), as well as being

2 Andrew Dunk and Adrian Haegee
able to perform basic object manipulations within an environment (including
scaling, rotations, and constraining movement to a specic axis).
This paper starts by introducing a background on existing user interfaces,
then continues by elaborating on the gMenu's current and proposed features.
Several use cases are described, and nally some conclusions are drawn and
future work is proposed.
2 Virtual Reality User Interfaces
In order to compile a standard for 3D user interfaces, studies need to be made
on existing 3D user interfaces which have been designed for specic applications,
and have gradually improved over time as new requirements and issues where
found. There has been signicant research into comparing and classifying these
user interfaces to gain a overall understanding of the requirements of a useful
and usable user interface. [1, 3]
2.1 Interaction Techniques
There are many interaction techniques available for 3D user interfaces that allow
users to navigate through environments, as well as select and manipulate objects.
Selection is an integral part of using a menu and there are many dierent
methods available for this; for example, collision detection is one of the simplest
methods of selection, the user would reach out and touch the object to select it.
This becomes dicult if the objects are placed at further distances. To overcome
this ray casting selection could be used; this method simulates a laser pointer
extending the selection to any distance.
Other selection methods include occlusion which works by concealing the
object to be selected. This method is relative to your eye and hand position. [4]
There are also selection techniques that use eye tracking only, enabling selections
to be made based on the users gaze rather than their hand position. [5]
2.2 System and Environment Control Menus
3D menu systems exist in many forms and designs. Types of menus include
2D menu systems that have been brought into 3D environments, 3D menus
positioned at specic locations relative to the environment, the users, or objects,
and menus that are specic to hardware devices used with the virtual reality
system.
There are many 2D graphical user interfaces available, one solution is to use
these systems and adapt them to the 3D environment. This gives advantages
to usability as systems like Windows Icons Menus and Pointers (WIMP) have
had many years of research and improvements applied to them. They are very
well recognised and would require a minimal amount of training for users to
condently use them.

The gMenu User Interface for Virtual Reality Systems and Environments 3
A variety of pop-up and pull-down style menus have been designed and stud-
ied in virtual environments [6], as well as implementations enabling the X win-
dow system to be brought into a virtual environment [7]. Menus have also been
dened by their position in the environment, the tool belt menu was displayed
around the users waist while in the environment and hand held menus such as
the ring menu is displayed by the users hand while being used.
A multitude of menus have been designed for use with particular hardware in
mind like the TULIP menu [8] which use pinch gloves. Menus can be projected
onto physical objects like tablets which in turn give a tactile feedback as well as
visuals when being used. [9]
Other menu systems include the Spin Menu [10], designed for quick access and
uses 3D icons and the Command and Control Cube [11] used on the holobench.
These two menus use an action or movement to select menu items rather than
the idea of pointing and selecting.
3 The gMenu System
This section is split into three parts. The rst describes some of the proposed
and current abilities of gMenu and how it can be constructed and congured.
The second describes the menu as a le browser and how it is able to create
icons for specic le types. The third section describes messaging between the
menus, objects within a scene, and the virtual environment in conjunction with
the VieGen framework.
3.1 A gMenu
Each new menu is constructed with a menu grid, which is made up of multiple
menu items. It is possible to create new menus within an existing menu producing
submenu functionality. Submenus automatically link back to their parent when
created.
Multiple gMenus can be created in a virtual environment, each gMenu could
be organised to give better clustering of the commands used within the virtual
environment.
The menus will be able to be created by programmers and customised by
users. Simple XML les will be able to be written by users to generate custom
menus and link there functionality to the system and environment. An XML
parser will be used to generate the user created menus and bring them into
the environment. The programmer will have a powerful library to congure all
aspects of a created gMenu
gMenu Items gMenu items are created and inserted into locations on a gMenu
grid. It is possible to develop new gMenu items by extending a base gMenu item
container. Currently there are three types of items that are available, these are
a textured button which is a cuboid object with textured surfaces, a 3D item

4 Andrew Dunk and Adrian Haegee
which is a detailed 3D model object rather than an
at texture, and a 3D text
string for displaying text within the menu.
Each item has some base properties including its position in the grid, its
visibility, if it is enabled, highlighting function, states, and messaging options
for communicating the states and commands throughout the system. When a
menu is being displayed within an environment, each of the items are frequently
checked for collisions with the users interface device. Items are able to send
messages describing their current state. for instance if the user is hovering over
the item, pressing a button on the users interface device over an item, or releasing
a button on the users interface device while over an item.
If a hover state is triggered then a highlighting function is applied to the
item. The default highlight translates the item forward towards the user by the
same depth as the item, giving the eect of the button being selected. However
the highlight functions can be replaced allowing the creator to develop their own
customised functionality.
The gMenu Grid The gMenu grid is designed to give the items created within
the menu a dened structure and keep the visual aspect of the menu clean. This
allows the creator to concentrate on the functionality of the menus rather than
their appearance.
The grid is never displayed and only exists as a function that generates a
position in 3D space for a given (x; y) item coordinate. To place an item into
the grid simply give the item an (x; y) coordinate, which is translated into the
position of the item in 3D space. This also allows new grid generation functions
to be written, meaning the menus physical shape and style can be customised.
A
at plane Fig.1(a) or a curved wall Fig.1(b) are possible examples of grids
that can be created for positioning each of the items. Items do not have to exist
on a single plain but they are dependant on the grid. Items can be placed at
any location within the grid allowing items to be clustered and arranged into
sections and dierent shapes Fig.2(b).
The size of the cells in the grid and their separation can be set when creating
the menu grid, this denotes the maximum size of the menu items and the distance
they are placed apart from one and another. The grid itself can be as large as
required. However considerations need to be made if creating large grids, at the
eciency of searching for an item within a large grid could possibly produce
poor results.
3.2 The gMenu File Browser
The gMenu le browser extends the functionality of the base gMenu system to
create an iconic display of a le system starting from a root directory.
When a gMenu le browser is created a recursion of the le system is per-
formed, each le and folder beneath the root path is read and an item is created
within the le browser grid. Submenus are created for each directory with all
appropriate les included as separate items. As each item is added to the menu a

The gMenu User Interface for Virtual Reality Systems and Environments 5
(a) Flat Plane Grid (b) Curved Wall Grid
Fig. 1. Examples of Possible Grid Positioning with Associated Coordinates
textured button is created and supplied a texture to represent the le or folder.
In addition a text item is created to identify each icon by displaying the le or
folder name.
Some le types are recognised by the le browser, these are given specic
icons. If supported image les are found within the le system the image will
be used as the buttons texture. Recognised 3D model les are either added to
the menu as a scaled version of the 3D model, or a textured button can be
displayed with an image created from the 3D model le. By creating images for
the 3D models the system load is reduced. Any model images created are saved
in the same directory as the 3D model to decrease load times on consecutive
runs. There is also support for basic ltering of le extension types, making it
possible to create le browsers of specic le types only. If a folder is selected
the corresponding submenu will replace the current menu, if an item is selected,
the items le name and location are returned.
Creating Button Textures If a button texture is needed for a 3D model a
set of steps are taken. Firstly a search is performed in the same directory as the
le for a previously created texture associated with the le, if one is found that
texture is simply returned as the les icon. If a texture le is not found then a
new one is created.
This is performed by loading the model into a predened scene that is never
rendered on the users display. The scene has some basic settings that can be
customised, such as the size of the viewport being used which determines the
end size if the icon, and the scenes background colour. Once the 3D model has
been loaded into the scene a camera is positioned to capture the entire model
and an image is rendered and saved in the same directory as the model so that
the texture will not need to be generated again, saving load time. Once the image
has been saved it is returned to be used in the menu item.

6 Andrew Dunk and Adrian Haegee
3.3 VieGen Specic Implementation
Within VieGen, the dierent entities that make up a virtual environment are
all derived from the base SceneObject class, which provides them with in-scene
attributes and behaviours. All SceneObjects have the ability to produce and
consume messages, which are used to provide communication and interaction
amongst these entities. Typical VieGen messages include those such as the user
selecting an object, or notication that one object has collided with another.
These messages provide the basis for interacting with the gMenu system. Focus
changing or selection events can be used to indicate changes in the gMenu Item
state, and the generation of new messages could result from these interactions.
While some items could be hard coded to provide specic actions, others could be
congurable. It the latter case, options could range from sending named system
events, through to the
exibility of sending XML fragments for direct object
control.
4 Applications
This section describes some simple applications using the gMenu and shows its
current functionality.
4.1 3D Model Viewer
To demonstrate the gMenus le browser features, a simple environment has been
created that allows users to brows through a le hierarchy and load selected 3D
models into the environment, position the models within the environment and
change their orientation. The environment itself is a small area with no visible
walls or
oor, it contains four lights to equally illuminate the area and has a
blue background colour.
The gMenu le browser can be opened in the environment by pressing a
button on the users wand. The menu appears a short distance in front of the
wand position and a ray casting technique is used to select items within the
menu. The le menu is ltered to only show supported 3D model les. Textures
have been generated for each of the models and predened textures are loaded
for directories and a back button of each subdirectory. These icons are a folder
and an arrow pointing to the left respectively.
By hovering over an item, text is displayed with the model's name; when the
item is selected the model le is loaded and attached to the users wand so that
it can be positioned in the environment. Any number of models and duplicates
can be loaded into the environment.
4.2 The Virtual Keyboard
This application shows the visibility of the gMenu system by creating a full
keyboard layout in front of the user, enabling text input while in a virtual envi-
ronment Fig.2(a).

The gMenu User Interface for Virtual Reality Systems and Environments 7
The user can type each letter by moving their hand to collide with the ap-
propriate button. It would be possible to \two nger type"using this gMenu
keyboard if two hand trackers were used within the environment.
The keyboard can be very useful for naming les, text messaging other users
within networked environments or debugging applications still under develop-
ment while still within the environment. This could be achieved by generating a
virtual terminal.
(a) gMenu Keyboard (b) Custom Item Layout
Fig. 2. Examples of the gMenu system in use
4.3 Environment Controller
This application combines a le browser with other control options to enable
users within a virtual environment to load, edit, and save scenes without having
to leave the virtual reality system. Scenes are loaded in the same way as the 3D
model viewer, but the scene objects can be selected bring up a control menu to
perform simple manipulations of the objects. The scene can then be saved and
closed.
Scenes that incorporate audio streams allow extra options in the control menu
to change the volume within the environment.
5 Conclusion and Future Work
The gMenu system is very easy to create as a programmer and customise as
a user. Because of these abilities it is very easy to simulate many types of 3D
menus that have already been developed and researched, and will also enable
quick prototype designs of new new menu ideas to be produced and tested in
the future.
There has been positive reactions towards usability from the initial tests and
applications created using the gMenu system. It shows potential to be used in
multiple application tasks with its ability to be highly customisable and styled.

8 Andrew Dunk and Adrian Haegee
Future user testing has been planned for the gMenu system comparing its use-
fulness and usability with other menu systems.
With a fully customisable system like gMenu and the ability to control both
a system and its environments brings us a step closer to some of the capabilities
that are possible with desktop operating systems.
There are plans to introduce a third dimension to the gMenu grid allowing
developers more freedom with the menu shapes and positions, making the op-
tion to generate menus resembling the Command and Control Cube easier for
example.
Customising and creating new menus live within the environment by adding
and removing items from menu grid to grid, or even placing menu items into the
environment on there own, are some ofthe features currently being developed.
There is also future plans to create more menu item types to be used in
gMenu such as toggle buttons, passive indicators, analogue output selections,
and dynamic items that can be created by the users.
References
1. Bowman, D.A., Kruij, E., LaViola, J.J., Poupyrev, I.: 3D User Interfaces: Theory
and Practice. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA,
USA (2004)
2. Haegee, A.: VieGen: An Accessible Toolset for the Conguration and Control of
Virtual Environments. PhD thesis, Centre for Advanced Computing and Emerging
Technologies (ACET), School of Systems Engineering, University of Reading (2008)
3. Dachselt, R., Hbner, A.: Three-dimensional menus: A survey and taxonomy. Com-
puters and Graphics 31(1) (2007) 53 { 65
4. Bowman, D.A.: Interaction Techniques for Common Tasks in Immersive Virtual
Environments. PhD thesis, Georgia Institute of Technology (1999)
5. Chan, C.N., Oe, S., Lin, C.S.: Active eye-tracking system by using quad ptz cam-
eras. Industrial Electronics Society, 2007. IECON 2007. 33rd Annual Conference
of the IEEE (Nov. 2007) 2389{2394
6. Jacoby, R.H., Ellis, S.R.: Using virtual menus in a virtual environment. In Alexan-
der, J.R., ed.: Society of Photo-Optical Instrumentation Engineers (SPIE) Confer-
ence Series. Volume 1668 of Society of Photo-Optical Instrumentation Engineers
(SPIE) Conference Series. (June 1992) 39{48
7. Coninx, K., Van Reeth, F., Flerackers, E.: A hybrid 2d/3d user interface for
immersive object modeling. Computer Graphics International, 1997. Proceedings
(Jun 1997) 47{55
8. Bowman, D.A., Wingrave, C.A.: Design and evaluation of menu systems for im-
mersive virtual environments. In: VR '01: Proceedings of the Virtual Reality 2001
Conference (VR'01), Washington, DC, USA, IEEE Computer Society (2001) 149
9. Wloka, M.M., Greeneld, E.: The virtual tricorder. Technical report, Department
of Computer Science, Brown University (1995)
10. Gerber, D., Bechmann, D.: The spin menu: a menu system for virtual environments.
Virtual Reality, 2005. Proceedings. VR 2005. IEEE (March 2005) 271{272
11. Grosjean, J., Burkhardt, J.M., Coquillart, S., Richard, P.: Evaluation of the com-
mand and control cube. Volume 0., Los Alamitos, CA, USA, IEEE Computer
Society (2002) 473