\documentclass{acmsiggraph}                     % final
%\documentclass[annualconference]{acmsiggraph}  % final (annual conference)
%\documentclass[review]{acmsiggraph}            % review
%\documentclass[widereview]{acmsiggraph}        % wide-spaced review
%\documentclass[preprint]{acmsiggraph}          % preprint

%% Uncomment one of the five lines above depending on where your paper is
%% in the conference process. ``review'' and ``widereview'' are for review
%% submission, ``preprint'' is for pre-publication, and ``final'' is for
%% the version to be printed. The ``final'' variant will accept the 
%% ``annualconference'' parameter, which changes the height of the space
%% left clear for the ACM copyright information.

%% The 'helvet' and 'times' packages define the typefaces used for
%% serif and sans serif type in this document. Computer Modern Roman 
%% is used for mathematics typesetting. The scale factor is set to .92
%% to bring the sans-serif type in line with the serif type.

\usepackage[scaled=.92]{helvet}
\usepackage{times}

%% The 'graphicx' package allows for the inclusion of EPS figures.

\usepackage{graphicx}

%% use this for zero \parindent and non-zero \parskip, intelligently.

\usepackage{parskip}

%% Optional: the 'caption' package provides a nicer-looking replacement
%% for the standard caption environment. With 'labelfont=bf,'textfont=it',
%% caption labels are bold and caption text is italic.

\usepackage[labelfont=bf,textfont=it]{caption}

%% If you are submitting a paper to the annual conference, please replace 
%% the value ``0'' below with the numeric value of your OnlineID. 
%% If you are not submitting this paper to the annual conference, 
%% you may safely leave it at ``0'' -- it will not be included in the output.

\onlineid{0}

%% Paper title.

\title{Designing a Hybrid User Interface: A Case Study on an Oil and Gas Application}

%% Author and Affiliation (single author).

%%\author{Roy G. Biv\thanks{e-mail: roy.g.biv@aol.com}\\Allied Widgets Research}

%% Author and Affiliation (multiple authors).

\author{Felipe Gomes de Carvalho \thanks{e-mail: kamel@tecgraf.puc-rio.br}\\PUC-Rio, Tecgraf%
\and Alberto Raposo\thanks{e-mail:abraposo@tecgraf.puc-rio.br}\\PUC-Rio, Department of Informatics%
\and Marcelo Gattass\thanks{e-mail:mgattass@tecgraf.puc-rio.br}\\PUC-Rio, Department of Informatics}

%% Keywords that describe your work.

\keywords{Hybrid User Interface, 3D Interaction, Interaction Devices, Transitional Interface}

%%%%%% START OF THE PAPER %%%%%%

\begin{document}

\teaser{
\includegraphics[bb=0 0 43.5mm 33mm]{teaser_fig02.png}
%\label{fig:teaser:01}
\includegraphics[bb=0 0 43.5mm 33mm]{teaser_fig03.png}
%\label{fig:teaser:02}
\includegraphics[bb=0 0 43.5mm 33mm]{teaser_fig01.png}
%\label{fig:teaser:03}
\includegraphics[bb=0 0 43.5mm 33mm]{vr_nav_anotation.png}
%\label{fig:teaser:04}
\caption{Features of HybridDesk: (a)(b) VR-Nav, (c) VR-Manip, and (d) Ray-casting technique for 3D Selection}
\label{fig:teaser:01}
}

%% The ``\maketitle'' command must be the first command after the
%% ``\begin{document}'' command. It prepares and prints the title block.

\maketitle

%% Abstract section.

\begin{abstract}

The post-WIMP (Windows, Icons, Menu and Pointer) user interfaces are creating new interaction modalities and the use of new input and output devices. Many of these new interfaces are not yet mature, and issues related with the clear definition of an application's context and technological requirements are still under investigation. The study of the relationship between the properties of interaction devices and their influence on the performance of 3D tasks (navigation, manipulation, and selection) is an important factor in the identification of adequate setups for carrying out these tasks. Evidences of this relationship are being described by new studies on interaction tasks. However, in a broader context, each task can be decomposed into subtasks whose technological demands can be a challenge, since they require multiple interaction environments as well as transitions between them. Therefore, this work proposes a hybrid technological setup to integrate the advantages of different functional environments. In order to achieve such goal, a semi-immersive environment composed of 3 functional environments was developed and transitions between these environments were exploited during a 3D annotation task in an oil and gas application. 

\end{abstract}

%% ACM Computing Review (CR) categories. 
%% See <http://www.acm.org/class/1998/> for details.
%% The ``\CRcat'' command takes four arguments.

%\begin{CRcatlist}
%  \CRcat{K.6.1}{Management of Computing and Information Systems}%
%{Project and People Management}{Life Cycle};
%  \CRcat{K.7.m}{The Computing Profession}{Miscellaneous}{Ethics}
%\end{CRcatlist}

%% The ``\keywordlist'' command prints out the keywords.
\keywordlist

\section{Introduction}

Most of the tasks performed in a conventional desktop computer are related to text editing, file organization, use of tables, and mathematical calculations, among others. However, other areas of application emerged, e.g. 2D image editing, animation, CAD (Computer Aided Design), interactive 3D visualization, entertainment and games. The inherent characteristics of these classes of applications have created a demand which was not fully met by the desktop metaphor as it was originally proposed to represent an office through WIMP interfaces. 

Applications such as CAD, 2D image editing and graphics applications involving two-dimensional data are still compatible with the desktop metaphor, because they remain similar to tasks performed by engineers and architects using their boards in their respective offices. However, the emergence of three-dimensional interactive applications showed the limitation of conventional 2D desktop devices (mouse, keyboard and monitor). Such conventional setup does not meet all the interaction needs related to the additional dimension in this new environment. The main evidence supporting this fact is the existence of a research line focused exclusively on 3D computer graphics which is as old as the very development of WIMP interfaces. The early the efforts were guided by the evolution of scientific visualization and flight simulator applications driving the development of a new set of interactive hardware, such as head mounted displays (HMD), data gloves and CAVES.

For a long time the 3DUIs (3D User Interfaces) were developed in parallel with WIMP interfaces, and their contributions were added gradually to desktop computers as demanded by the 3D applications. Evidences of that can be found in the use of devices such as goggles and gloves in 3D games and in scientific visualization applications. In the Wii game console, for example, there is a device called Wiimote that incorporates research results on the development of 3D tracking devices. Several 3D interactive techniques (3DIts) were also developed using mouse- and keyboard-oriented desktop, and the success of these efforts is evident in computer games.

This gradual convergence of interfaces from different natures (such as WIMP and 3DUIs) resulted in the emergence of other interfaces that share characteristics of both. The wide variety of hardware and software, and the different interaction techniques proposed are potential ``ingredients'' for a research field called Hybrid User Interfaces (HUI) ~\cite{Feiner91}. In this field, efforts are being joined in order to blend harmoniously both virtual  (applications, interaction techniques, graphics, etc.) and physical (the input and output) elements from one or more interfaces. In this way, the design of HUIs may result in new interfaces with hybrid resources which were inspired by other interfaces with the goal of integrating previously separate functions.

The way in which the hardware and software elements are integrated into an HUI can result in a kind of interface that connects others, e.g. a form of interface of transition or \textit{transitional interface} ~\cite{Billinghurst01,GrassetLB06}. Since this is an area of multidisciplinary research, it is very difficult to define clear rules to guide the  development of HUI, and research regarding the design of such interfaces is still insufficient. 

The purpose of this work is to match the devices and task properties in an attempt to guide the design and development of HUIs. For example, the concept of dimensional congruence ~\cite{Darken05} can guide the selection of devices to accomplish tasks, since this concept helps to identify the dimensional requirements of the tasks. In order to illustrate such concepts this paper presents the design and development of a workspace called \textit{HybridDesk}, featuring a mix of different devices to allow the use of three interactive environments for performing tasks with different hardware requirements. These requirements were established based on an investigation of the literature regarding the influence of the devices on 3D and 2D interaction tasks.

%%%%%

%%%%

\section{Related Work}

Interface systems that attempt to combine hardware and software components of different natures are sometimes called Hybrid User Interfaces (HUI), Mixed Reality Systems or, until recently, Hybrid Display Systems. They are characterized by the use of multiple elements, which can be multiple devices or multiple user interfaces. Throughout this text we are going to use the term HUI, first
proposed by Feiner~\cite{Feiner91}, referring to a heterogeneous environment, rich in interaction techniques, and with different kinds of devices used in a complementary way. 
   
A pioneer work in this area was the Office of the Future~\cite{Rascar98}, which combined several computer vision and computer graphics techniques to analyze surfaces of the real world and then add virtual information projected on them. 

Rekimoto~\cite{RekiSaitoh99} explored the HUI heterogeneity using several computers and displays (projections and notebooks) in the same work  environment. The EMMIE project~\cite{Butz99} developed a system designed for collaboration, similar to the work of Rekimoto, but including the use of Augmented Reality (AR) with a see-through HMD.

Nakashima~\cite{Nakashima05} presented a prototype of a collaborative work environment with 2D and 3D environments for graphics modeling tasks. The environment uses a display called IllusionHole for 3D interactions, while 2D interactions are supported by a projection. An HUI for the manipulation of medical data also using 2D and 3D interactions was developed  by Bornik~\cite{Bornik06}. In this HUI, a single 3D pointer is used as interaction tool and two visualization forms are available, one on a tablet and another on a projection.  
Benko~\cite{Benko05} created a hybrid environment composed of an LCD display placed vertically, a touchable display placed horizontally, and a see-through HMD for AR. This environment is used for the manipulation of archaeological objects.

Recently, Baumgartner~\cite{BaumgartnerED07} presented an HUI to explore the organization of a desktop, disposing documents spatially. The organization of documents is made through gestures, using a glove. The documents in the 3D space are visualized in an autostereoscopic display and edited using the keyboard and a pen on a tablet below the display.

Carvalho~\cite{Carvalho07} developed an HUI with the goal of 
exploring different 3D interaction tasks in a single work environment. VR was applied using an HMD for navigation tasks, and AR was used for manipulation tasks. WIMP interfaces were used for common desktop tasks, such as text editing.

In general, the works mentioned above sought to blend different technologies in order to execute certain tasks. However, there is the need for a criterion, or at least a reference to any methodology or concept driving this ``technological mixture''. Moreover, there seems to be no explicit concern about possible transitions among the integrated technologies.


\section{Devices Properties x 3D Interaction Tasks}
\label{sec:relatedworks:devices}

The majority of 3D Interaction Techniques (3DIts) for immersive environments has been developed using HMDs, mainly due to their lower cost compared to CAVEs, and to their capacity of supporting more immersion than conventional displays. Several studies on 3DIts have tried to identify their advantages and disadvantages in the execution of 3D tasks, and the use of HMDs prevails in these studies, which usually compare different techniques in the same 
display~\cite{Bowman97,BowmanJH99,BowmanKH98,PoupyrevI99,PoupyrevBI98}. 
These studies and others presented discussions and conclusions about the use of several 3DIts for different tasks and in different kinds of virtual environments (VEs). However, all of this knowledge is associated with characteristics of a single kind of display, namely HMD, which features limited FOV (Field of View), a 360-degree FOR (Field of Regard), a complete occlusion of the real world, the possibility of short distance interactions taking advantage of human proprioception (unconscious perception of movement and spatial orientation arising from stimuli within the body itself), ergonomic issues related to the weight of the device and the existence of cables, among others.

Nevertheless, research efforts trying to understand the impact of the displays'
features on the virtual experience are beginning to appear.  For instance, there are 
studies comparing the use of different displays to execute a single task, which can be performed by means of different 3DIts for each display. Another example are studies that analyze the impact of migrating a 3DIt from its primary display (i.e., the display for which it was initially developed) to another kind of display. These efforts are bringing to light other characteristics of the displays that benefit the execution of 3D tasks. Many of these studies analyze displays with very distinct characteristics, creating a kind of dichotomy, in order to reach conclusions that may serve as reference for the evaluation of other displays with similar characteristics. A common case is the comparison of HMDs and CAVEs, sometimes with the addition of FishTanks and conventional desktop monitors. 
    
Recent studies~\cite{SteedParker05,McMahanGGMB06,ReitingerSBB06,BowmanDRFV02,DemiralpKZL06,PrabhatFKWS08,Lessels04,Raymaekers05,BowmanBM07} 
about usability and performance involving different combinations of 3DIts, displays, input devices, and visual resources in applications have been presenting important partial conclusions about adequate conditions for the combined use of these elements. Such conditions are sometimes related to the characteristics of the VE, such as indoor or outdoor scenarios, densely or scarcely populated by objects, among others. Other conditions are related to visual attributes, such as FOV, FOR, stereoscopy, head-based rendering, resolution, level of detail, rendering quality (photorealism), and brightness. Features of the input devices (DOF, ergonomics, etc.) as well as characteristics of the workspace, i.e. the physical environment for the 
execution (physical space, illumination, etc), are being investigated. 

The relation between FOV and FOR represents a significant contribution of the studies mentioned above. Studies that evaluated displays with different kinds of FOVs suggest some relations between displays and 3D tasks. Traditional monitors in the form of FishTanks emphasize focal (or central) vision, with very little peripheral vision. Larger displays and with larger FOR provide a broader area for peripheral vision. The increase of peripheral vision supports more simultaneous visual information, and consequently a larger space for visual search.  Selection and navigation tasks are improved by visual stimuli coming from central and peripheral vision. Selection tasks are enhanced by a larger instantaneous visual search space, while navigation tasks are improved by the increased spatial awareness, which helps create mental maps, for instance. Besides these tasks, a larger viewing area also provides better support for tasks related to the analysis of spatial relations (topological, projective, or Euclidean) among objects. Manipulation tasks, on the other hand, require attention directed at a reduced viewing area - focal vision -, indicating a smaller need for peripheral vision. However, since attention is directed at a smaller area, other features become more evident, such as display resolution and sharpness.       

Another important issue raised is user preference for 6-DOF input devices for
manipulation tasks. Although such devices generally do not provide accurate
precision, they allow movements similar to those performed during the
manipulation of real objects, which facilitates quick visual inspections. Moreover, if more than one device of this kind is available, bimanual interactions are possible. 

Another aspect involving the relation between input devices and displays is the impression of manipulating the objects directly. For example, an attempt to adapt the WIM (World in Miniature) technique to a CAVE caused the impression of indirect manipulation, i.e., the objects were not within reach of the hands~\cite{BowmanBM07}. Using an HMD, the egocentric vision of the VE associated to proprioception help the users coordinate the visual feedback from the display and their hand movements. The size of the object being manipulated is also important. For instance, when the WIM virtual object was reduced in the CAVE adaptation, improvements were observed in agreement with previous studies~\cite{DemiralpKZL06}, showing that the manipulated objects must be smaller than the user's body. This fact indicates the need for further evaluation of Reachin displays, which allow direct access to the
virtual objects and provide a visualization similar to a FishTank. In  addition, they do not have the HMD ergonomic problems, such as weight and cables.         

Today it is still difficult to answer questions related to the display, input device, and 3DIt triple, which is the most favorable combination to execute 3D tasks. Nonetheless, maybe this triple is not enough to express the adequate configuration for a 3D task, since new relevant factors may be discovered. Previous studies show evidence that can support useful guidelines to identify potential combinations of displays and devices once an HUI designer knows the tasks' requirements (interaction techniques, interfaces paradigms, etc.) and the goals of the target application.

\section{HybridDesk - Design and Implementation}


\subsection{3D Annotation Task}
\label{sec:Annotation3d}

The design of the HUI proposed here was based on the requirements found for a 3D annotation task in an oil and gas application. These requirements helped identify the hardware set used to compose the interaction environments within HybridDesk. The 3D annotation task was considered important because it could be decomposed into different subtasks ranging from symbolic input information to spatial input used for 3D interaction techniques.

The 3D annotation task is a way to insert information into a VE, and often this information is related to a particular object. To handle the creation and management of symbolic information we have chosen to take advantage of past experiences of users with WIMP interfaces rather than creating a completely new environment. Basically, we want to use all the resources available in a WIMP interface to create files, and ``3D shortcuts'' (3D Icons) to these files are created inside the VE by the user. The visual feedback of the shortcuts has a shape of a 3D cube.

Annotations are very important in oil and gas applications. For instance, a virtual oil platform is composed by a huge number of different objects with many professionals working on it, and the need to leave different information (audio data, text data, presentations, movies, etc.) attached to some objects is important to convey some intentions or observations.

Regarding 3D interaction aspects related to the 3D annotation task, we have identified two subtasks:

\textbf{1) 3D Navigation:} A technique to enable a search for an object or an exploration of the entire VE. Models of oil platforms are extremely dense geometric scenarios, thus there are plenty of things to explore.

\textbf{2) 3D Selection and Manipulation:} Techniques to select and manipulate objects (through translations and rotations) to enable the insertion of 3D Icons. For manipulation tasks we have understood that it would be important to perform a detailed visual inspection of the mesh of a selected object in order to identify particular points to leave annotations. We therefore chose an isomorphic interaction technique\footnotemark[1] to handle virtual objects at short distance, like people do with real objects. 

\footnotetext[1]{\textit{Isomorphic Interaction Technique} suggests ``\textit{a strict, geometrical, one-to-one correspondence between hand motions in the physical and virtual worlds...}''~\cite{Bowman05}.}

As WIMP interface we chose Microsoft Windows XP, and concerning its associated hardware we maintained the conventional setup composed by monitor, keyboard and mouse.

Upon reviewing the requirements, an additional restriction was imposed to guide the choices of interaction techniques and the devices related to them. Such restriction refers to the physical work environment of the HUI: ``users must remain sitting during all stages of their work and have access to a desk where the keyboard and mouse are located''. This decision was influenced by the way people work on regular desktop computers. It was also considered important that such desk served as a resting aid during the execution of 3D interactions. We believe that allowing users to rest their elbows, for instance, would be beneficial during long working sessions using devices such as wands.


\subsection{Interaction Environments}

The setup introduced in the following subsection attempts to explore immersion displays and devices that could bring advantages for each subtask identified in the annotation task. Navigation tasks tend to benefit from displays with larger FOV and FOR. They also provide better support for searches during the execution of selection tasks, because the visual search space is larger. Local Manipulation tasks show evidence of being more prone to visual stimuli in a restricted visualization area, which suggests the need for a narrower FOV. However, as the focus of attention is more intense in this restricted area, visual cues are more noticeable. Thus, display features such as brightness, resolution, sharpness and photorealism become important.

\subsubsection{VR-Nav}

The semi-immersive interaction environment VR-Nav is composed by four projection screens used simultaneously to provide a broader FOV and a larger physical visualization area near the user (FOR). The display resulting from this set of screens was named MiniCave and was inspired by the idea of CAVE adapted to a desk (Figure ~\ref{fig:teaser:01}.a and ~\ref{fig:teaser:01}.b).

%\begin{figure}[hbt]
%\centering
%\includegraphics[bb=0 0 70mm 40mm]{wand_real_virtual.png}
%\caption{\textit{Wand} virtual acompanhando os movimentos da \textit{Wand} real.}
%\label{fig:ws_wand-real-virtual}
%\end{figure}

A Wand (a Wiimote tracked by an optical tracker) is used as input device, and a 3D representation for it was created in the VE. This representation is visually similar to the real device and receives the translations and rotations from the tracker device. Moreover, head tracking is provided through tracked glasses, and there is anaglyph stereo visualization.

%\begin{figure}[hbt]
%\centering
%\includegraphics[bb=0 0 70mm 40mm]{ws_vr_nav.png}
%\caption{\textit{VR Nav}.}
%\label{fig:ws_vr_nav_p2}
%\end{figure}

The physical space provided by a regular CAVE enables a wide variety of interaction techniques using physical movements from arms, hands and even legs, but the 3DIts that require physical locomotion are not well suited due to the reduced ``walking space'' inside the CAVE. These observations lead to a partial conclusion that a CAVE provides good immersion due to the great visual stimuli from the set of screens, and it is more suited to the use of 3DIts that require arm and hand movements. These advantages inspired us to take a CAVE layout and adapt it to the proposed hybrid environment in order to provide a partial immersion as well as to use some 3DIts from a CAVE. To adapt a CAVE to a desktop, we had to restrain the user's ability to remain standing during the interaction.

Two 3DIts for navigation were available. The first was based on the ``grabbing the air'' technique, i.e., the locomotion starts by dragging the wand in the air, and the visual feedback is the virtual world following this dragging movement. This movement requires a button in the wand to be pressed. The second technique uses the directional keys in the wand to determine the direction of movement based on the orientation of the wand. For instance, the ``Up'' key causes a forward movement in the direction of the current orientation of the wand, the ``Right'' key triggers movement in the positive perpendicular direction of the orientation of the wand, and the same idea applies to the ``Down'' and ``Left'' keys.

For selection tasks a ray-casting technique was implemented, and it is activated by pressing another button in the wand (Figure ~\ref{fig:teaser:01}.d). When the virtual ray is visible, and if it is pointing to an object, the selection of this object is completed by pressing the button again.

Since a wand is a 3D device because of the nature of the continuous data (3D positions and rotations) associated to it, we believe that we have achieved a dimensional congruence between the input device (wand) and the 3DIts (grabbing the air, directional keys, and ray-casting) used. All requirements for the 3DIts were fulfilled - for instance, grabbing the air requires 3D points to calculate a displacement vector along time, and navigation using directional keys and ray-casting require the 3D direction of the current orientation of the wand.


\subsubsection{VR-Manip}

This environment was created to perform a 3DIt for short distance manipulation (based on the requirements listed in Section ~\ref{sec:Annotation3d}). The inspiration for the technological setup came from the idea of a Reachin display which allows the manipulation of virtual objects within the arm's reachable area. Most Reachin displays use a mirror or a semi-transparent surface to show an illusion of a virtual content behind it. Such illusion is based on the reflection of an image from a CRT or a projection screen. Although a regular LCD is much thicker than a mirror or glass, we believe that this display provides reasonable access to the space behind it, which is the area for interaction with the virtual objects selected on the VR-Nav. We also enable head-tracking during this interaction.

The LCD can be placed along one axis in front of the MiniCave, and it must be either totally outside the MiniCave (during the use of VR-Nav) or in front of it (during the use of VR-Manip and WIMP - Figure ~\ref{fig:teaser:01}.c).

The orientation of the LCD did not follow the common orientations of Reachin displays (with inclination of around 45 degrees), because this layout would obstruct the view of the camera from the tracker device, affecting the tracking process of the wand. Normally the trackers used with Reachin displays are located behind the reflective surface, but in the case of the hybrid environment we could not put the tracker inside the MiniCave, thus it stayed on top of the physical structure.

The main reason to integrate something inspired by the Reachin display technology was our intention to avoid the use of an HMD (because of weight issues) while keeping the illusion of a near access to virtual objects somewhere in the physical environment (inside MiniCave, behind the LCD).

The same wand used in VR-Nav and its virtual representation were used here as input device and visual feedback, respectively. Although the view of the real hands and the real wand are obstructed when the user puts them behind the LCD, the movements of their virtual representations provide a good reference for spatial awareness.

%\begin{figure}[hbt]
%\centering
%\includegraphics[bb=0 0 75mm 45mm]{ws_vr_manip.png}
%\caption{\textit{VR Manip}.}
%\label{fig:ws_vr_manip_p2}
%\end{figure}


Based on the requirements for the manipulation tasks, we implemented an isomorphic interaction. When the user maintains a button pressed, the virtual object behind the LCD follows the wand's displacements. This feature provides an intuitive way to position the virtual object and inspect its details, thus benefitting from the properties of this display (good brightness, vivid color, contrast and higher resolution).

In the same way as in VR-Nav, we believe that a good match between the 3DIt and the input device was reached, since the wand provided the dimensional requirements for it.

When the user chooses a data file to attach to a virtual object as a 3D annotation, first it appears in VR-Manip on top of the virtual wand as a 3D Icon. The Icon remains attached to the wand, following its movements, until the user presses a button to release it in the 3D space. After the insertion, the Icon will appear connected to the object through a line from the center of the object to the center of the icon.


\subsubsection{WIMP}

To use the resources of the WIMP interface we created a desktop large enough to fit the screens of the MiniCave and the LCD (main screen). Mouse and keyboard were placed on a stand in front of the MiniCave with enough space for them and to be used for resting purposes (elbow, arms and hands) during interactions. 

The interactions with Windows XP are made in the conventional way. The only addition was the insertion of a context menu for any type of file. This menu has two items that start the transition to other environments with or without files, as explained in the following section.

\section{Transitions}

\begin{figure}[hbt]
\centering
\includegraphics[bb=0 15 75mm 60mm]{Transicao_devices.png}
\caption{Scheme of devices used in each environment and changes during the transitions.}
\label{fig:ws_trans_devices}
\end{figure}

In Figure ~\ref{fig:ws_trans_devices} there is a scheme of input devices and displays used in each interaction environment as well as the transitions between then. Care was taken in these transitions to maintain clear links between the environments, in an attempt to give the impression that all of them are part of a larger tool used to accomplish a task composed by subtasks.

We have tried to identify which would be the most impacting event during the transitions, and implemented mechanisms to make such events smoother. Also, we looked for interaction techniques that could be reused fully or partially in more than one environment, as this would decrease the cognitive overload after the transitions, when the user has to recognize the commands of the new environment.

There is an important event during the transition between VR-Nav and VR-Manip which is related to a display change. To handle it, we have implemented an animation mechanism as a visual sign to guide the user during the transition, showing what is happening and what to do next. This is an attempt to visually explain the transition from one display to another. For instance, when the user selects an object inside VR-Nav, an animation starts to move the virtual camera closer to the selected object. During this approximation, another animation (a virtual LCD moving from left to right and a label showing the name of the incoming environment - VR-Manip) appears, showing a sign to move the LCD display to the front of the MiniCave. The transition from VR-Manip back to VR-Nav shows an animation moving the camera back to the location where the object was selected before entering VR-Manip, and at the same time another animation shows a virtual LCD moving from right to left indicating the change of displays again (Figure ~\ref{fig:ws_transition_sample}).

Since VR-Nav and VR-Manip are 3D interaction environments and share the same input device (wand), we also chose interaction techniques with certain similarities in both environments. The technique for locomotion in VR-Nav and the one used in VR-Manip are almost identical, and the commands and spatial movements to move the camera in VR-Nav are the same commands to move the selected object behind the LCD in VR-Manip. 


\begin{figure}[hbt]
\centering
\includegraphics[bb=0 15 75mm 60mm]{transitionssample.png}
\caption{Visual signs used during the transitions between the VR-Nav and the VR-Manip environments.}
\label{fig:ws_transition_sample}
\end{figure}

The transitions between the 3D environments (VR-Nav and VR-Manip) and the WIMP interface could be considered abrupt because there is a change in paradigm, from 3D to 2D. The link we established during these transitions was based on interpretation and inferences. For instance, when the user chooses a file inside the file manager in the WIMP interface, then opens a context menu and activates the command to send the file to VR-Manip, VR-Manip reappears but with a 3D Icon. This visual difference in VR-Manip before and after using the WIMP interface (without the 3D Icon before and with the 3D Icon after) helps the user infer that this is the selected file in the previous environment or even a shortcut. 

When moving from VR-Nav to WIMP by clicking on a 3D Icon attached to an object in the VE, the WIMP interface appears with an open file manager and a selected file. This transition induces an interpretative link between the environments, i.e. something selected in VR-Nav followed by the appearance of something selected in WIMP can be interpreted as being the same object.

Preliminary user tests were conducted with 7 subjects (all had some experience with 3D computer games) in a task involving transitions along all interaction environments of HybridDesk. The task started in VR-Nav, and the subjects were asked to read the contents of an annotation icon near a specific object in the oil platform. When the subjects found the annotation, they had to go to the WIMP interface to read the data file. This file contained a text instructing to return to VR-Nav and create an annotation in a specific part of another object in the platform. To complete this new subtask the subjects had to find that object, go to VR-Manip to analyze it, and then go back to WIMP to create the annotation. After this, they had to return to VR-Manip and attach the annotation to the specified object.

All subjects completed the task, on average taking a time very close to the time taken by a VR expert to perform the whole task. Some problems were reported regarding ergonomic issues related to the LCD and the glasses. Nonetheless, all subjects approved the idea of using different interaction environments for different tasks, and also understood the transitions between these environments. 

\section{Conclusion}

The evolution of human-computer interaction led to the emergence of different forms of interfaces and interaction mechanisms which progressively are being converted into new interfaces. The need for support to tasks that require the use of different interfaces has become increasingly evident, with the emergence of fields of study such as hybrid user interfaces attempting to seamlessly integrate diverse elements of different interfaces.

The convergence of different interfaces (such as 2D and 3D) has been demanding significant research and development efforts, but this integration is still an open problem. For example, while 2D WIMP interfaces became consolidated with a standard technological setup composed by mouse, keyboard and monitor, 3D user interfaces still lack a somewhat standardized arrangement, and this leads to difficulties when choosing among a wide variety of input and output devices. This lack of reference makes it difficult to attempt new integrations, and any trials lack proper criteria. HybridDesk is an attempt to develop a HUI composed of a hybrid technological setup and transition mechanisms between interfaces in a specific context taking into account the demands of a 3D annotation task in an oil and gas application. 

Finally, we conclude that so far there is no theory that can guide a systematic design of future hybrid user interfaces or the treatment of transitions between interfaces. Although our user test results are still preliminary, this work shows that carrying out a detailed analysis of the requirements of a task and relying on new information on the properties of displays and devices constitute a significant initial step in the construction of a solid base for the design of hybrid user interfaces.


\section*{Acknowledgements}

Tecgraf is a laboratory mainly supported by Petrobras. Thanks also to the Brazilian National Research Council (CNPq) for the individual support for the authors.

\bibliographystyle{acmsiggraph}
%\nocite{*}
%\bibliography{template}
\bibliography{Tese}
\end{document}


