\documentclass[runningheads,a4paper]{llncs}

\usepackage{amssymb}
\setcounter{tocdepth}{3}
\usepackage{graphicx}

\usepackage{url}
\urldef{\mailsa}\path|{gcalpa, pribeiro, abraposo}@inf.puc-rio.br|    
 
\newcommand{\keywords}[1]{\par\addvspace\baselineskip
\noindent\keywordname\enspace\ignorespaces#1}

\begin{document}

\mainmatter  

\title{Multitouch Games Based on Collaboration Patterns to Encourage Communication and Social Interaction in People with Autism}

\titlerunning{Multitouch Games Based on Collaboration Patterns}

\author{Greis F. Mireya Silva \and Paula Ceccon Ribeiro \and Alberto Raposo}%
\authorrunning{Silva, Ribeiro \& Raposo}

\institute{Department of Informatics, Pontifical Catholic University of Rio de Janeiro,\\
Marqu\^es de S\~ao Vicente, 225, G\'avea, Rio de Janeiro, Brazil\\
\mailsa
\url{}}

\toctitle{Multitouch Games Based on Collaboration Patterns to Encourage Communication and Social Interaction in People with Autism}
\tocauthor{Greis F. Mireya Silva, Paula Ceccon Ribeiro \& Alberto Raposo}
\maketitle

\begin{abstract}
We present the design and evaluation of two multitouch collaborative games for people with autism. Both games were designed considering collaboration patterns to contribute in social skills of them. We conceived these collaboration patterns as interaction restrictive strategies with objects in a collaborative interface to encourage communication and interaction among users with autism. Both games were designed considering the requirements of a target group of users with severe autism and evaluated using research criteria about social interaction actions achieved by the users during the collaborative work in the games.
\keywords{Collaborative Games; Collaboration Patterns; Multitouch Interaction; Communication; Autism}
\end{abstract}

\section{Introduction}

People with autism are characterized by having an atypical development in their social interaction and communication skills, also presenting a high isolation degree and difficulties to establish social relations, to interact with others, impairment in verbal expressions, gestures and body postures. Besides, about 50\% of people diagnosed with autism have difficulties in developing any kind of functional language \cite{apa}.

Over the last few years, collaborative applications were developed to multitouch tangible interfaces in order to contribute to the therapeutic process for people with autism, aiming to reduce the difficulties that they usually have. Multitouch applications have been receiving attention in the field of autism therapies due to their characteristics to encourage communication and social interaction \cite{tse}. However, most of these applications are designed and evaluated just with users that are diagnosed with high-functioning autism (HFA), a mild autism, with a lesser degree in language and social interaction impairments \cite{klin}.

This fact shows the need to study particularities of people with severe autism, in order to establish collaborative strategies that encourage collaborative work and, therefore, that stimulate the generation of social interaction and communication. To achieve this goal, first we developed a set of strategies called ``collaboration patterns'', based on strategies that are used in studies with people with HFA. Our strategies were developed together with experts in autism. Besides, we consider specifics features of a set of people diagnosed with severe autism. Then, we developed two collaborative multitouch games to apply the collaboration patterns. After that, we evaluated these games in a target group of users with severe autism.

The next section presents the design process of the collaboration patterns and their application in the two collaborative multitouch games developed. Then, we present the evaluation and the results that were observed, which lead to discuss the conclusions of this study. Finally, we present some aspects to be studied in future works. 

%This fact shows the need to study particularities of people with severe autism, in order to establish collaborative strategies that encourage collaborative work and, therefore, that stimulate the generation of social interaction and communication. To achieve this goal, first we developed a set of strategies called ``collaboration patterns'', based on strategies that are used in studies with people with HFA. Our strategies were developed together with experts in autism. Besides, we consider specifics features of a set of people diagnosed with severe autism.
 
%The next section presents the design process of the collaboration patterns, the development of two collaborative multitouch games and the application of our patterns in these games. Then, we present the evaluation and results that were observed, which lead to discuss the conclusions of this study. Finally, we present some aspects to be studied in future works. 

\section{Collaboration Patterns}

We defined collaboration patterns as restrictive interaction strategies with objects in a collaborative interface to encourage the collaboration among people with autism. Initially, we adopted collaboration patterns  proposed in COSPATIAL project \cite{cos} and used in different applications developed for users with mild autism (HFA) \cite{bauminger}, \cite{battocchi}, \cite{giusti}. The good results found in social and collaborative behavior of children with HFA in those studies were an incentive for adopting them in the collaboration patterns' definition for people with severe autism.

Together with experts in autism, we studied advantages and disadvantages of these collaboration patterns applied to collaborative applications aimed at people with severe autism. We also chose a group of target users in order to know their specific requirements, such as needs, interests, skills and difficulties. Finally, we designed collaboration patterns to encourage communication and interaction among these users and developed two games applying these patterns.

\subsection{Design of the Collaboration Patterns}

The COSPATIAL project defines four collaboration patterns \cite{cos} that constrain the interaction of users to foster the collaboration among them:

\begin{itemize}
\item{\textbf{Choosing together:} to select an object, touch by some or all of the users interacting with the system is required (e.g., pressing a button together).}
\item{\textbf{Constraints on objects:} collaboration required due to explicit constraints on an object (e.g., an object is too heavy and the children cannot drag it on their own).}
\item{\textbf{Different role:} collaboration required because users have been assigned to play different roles (e.g., a driver who needs to focus on driving while a navigator checks a map).}
\item{\textbf{Ownership:} collaboration required because users have ownership of different objects which need to be negotiated.}
\end{itemize}

\textit{Choosing Together} and \textit{Ownership} patterns were not considered here because the high impairment degree of the users precludes them to conduct negotiations with others or to make coordination decisions to achieve a goal. Experts in autism suggested that these collaboration patterns could be used at a later time, when users would have acquired more interactive and collaborative skills.

On the other hand, \textit{Different Role} and \textit{Constraints on Objects} patterns were considered the most appropriate, capable of getting the interest of the users. Based on them we proposed four collaboration patterns.
Having in mind the difficulties of the target users, initially we used \textit{Different Role} assigning a minimum amount of roles for each user and a mediated collaboration to enable them to understand and adapt to play their roles in the game. The amount of roles is gradually increased to facilitate the adaptation of the users to the proposed activities. From \textit{Different Role} we designed two collaboration patterns: \textit{Passive Sharing} and \textit{Active Sharing}.

To further encourage the need of collaboration it is necessary to increase tasks that require a joint-performance. To foster synchronous collaboration, we obtained the third collaboration pattern called \textit{Active Sharing and Joint-Performance}, joining the \textit{Different Roles} and \textit{Constraints on Objects} patterns.

The last collaboration pattern has no interaction restrictions. It is called \textit{Unrestricted Interaction} and  allows collaborative actions among users in an unconstrained environment. Our four collaboration patterns are described below.

\begin{itemize}
\item \textbf{Passive sharing pattern:} this collaboration pattern assigns roles to users sharing resources in order to achieve a goal. The roles assigned to each user are just action and response from one user to another (Fig.~\ref{fig:pattern}.a).

\item \textbf{Active sharing pattern:} in addition to sharing resources, this pattern requires the exchange of information to achieve a goal. The collaboration is obtained through the sequence action---response---action of one user to another, where the response to a user will be given according to information received in action of the other (Fig.~\ref{fig:pattern}.b).

\item \textbf{Active sharing and joint-performance pattern:} this pattern uses the active sharing pattern and adds the necessity of help actions from the users. This help is achieved by the synchronous interaction of both users on their respective objects (Fig.~\ref{fig:pattern}.c).

\item \textbf{Unrestricted interaction pattern:} this pattern does not assign restricted roles to each user, neither restrictions on the game objects, allowing a free collaborative interaction in the game (Fig.~\ref{fig:pattern}.d).
\end{itemize}

\begin{figure}
\centering
\includegraphics[height=2 cm]{patterns}%eijkel2}
\caption{Collaboration patterns for applications aimed at people with autism. Restricted patterns (a, b, c) and unrestricted pattern (d).}
\label{fig:pattern}
\end{figure}

\subsection{Collaboration Patterns Applied to the Proposed Games}

We developed two games, called PAR (acronym in Portuguese to Ask, Help, Receive) and ComFiM (acronym in Portuguese for Picture Exchange Communication for Multitouch Devices), which are for multitouch tabletop and the for tablets, respectively. This section presents the patterns described before applied to these games.

PAR is a game that encourages the collaboration and social interaction between two users to get pieces of uniforms (shirt, shorts and sneakers) and to dress soccer players. PAR has three collaborative levels. Each level includes one of the three restricted interaction patterns (Fig.~\ref{fig:par}.a.b.c) and the unrestricted interaction pattern (Fig.~\ref{fig:par}.d). In each level, users have different roles according to their place around the multitouch tabletop. These levels are described below.

\begin{itemize}
\item \textbf{First level:} includes the \textit{Passive Sharing Pattern}. User 1 must take a piece of an uniform and put it in the box on the shelf. Then, the box will descend with the piece. User 2 should move the cart and take it to the shelf of the descending box to receive the piece sent by User 1 (Fig.~\ref{fig:par}.a). Likewise, User 1 must send the second piece of the uniform, from any shelf, while User 2 moves the cart to receive it. When the cart is filled with three pieces, User 2 should move the cart to the parking lot.

\item \textbf{Second level:} includes the \textit{Active Sharing Pattern}. User 1 must have information of three pieces of an uniform that should be sent. User 2 should ask for any piece by pressing a button (Fig.~\ref{fig:par}.b). User 1 should put the requested piece by the partner in the box; this piece can be in any of the three shelves. So, User 1 must locate the piece, take it and put it in the box. User 2 should move the cart to the shelf of the piece being sent to receive it.

\item \textbf{Third level:} includes the \textit{Active Sharing and Joint-Performance Pattern}. In addition to what was described in the previous levels, in this level the boxes are closed at the time that each piece is requested (Fig.~\ref{fig:par}.c). So, it is necessary that User 2 helps by pressing a button to open the boxes while User 1 takes and puts the piece requested in the box. When User 1 is taking a piece, a voice is heard stating: ``Help me by pressing the yellow button''. User 2 must press that button while User 1 puts a piece into the box.
\end{itemize}

The three levels include the \textit{Unrestricted Interaction Pattern}, applied after the cart is placed in the parking area, where both users can take the pieces of the cart to suit the player. Each user may take any piece and in any order to dress the soccer player (Fig.~\ref{fig:par}.d). At this moment a message will be displayed to the users informing them about the number of dressed soccer players and an option to continue to dress the next soccer player. User 2 must then take the cart from the parking lot and return to the lower part of the warehouse to continue asking and receiving uniform pieces.

\begin{figure}
\centering
\includegraphics[height=3.5cm]{par1}
\caption{Collaboration patterns in the three levels of PAR. a) Passive Sharing. b) Active Sharing. c) Active Sharing and Joint-Performance. d) Unrestricted Interaction.}
\label{fig:par}
\end{figure}

On the other hand, ComFiM is a game to encourage communication among people with autism by picture exchange in tablets. The game has as scenario a farm in which some tasks must be done. To do these tasks, the users have to communicate with each other, asking and sharing some tools represented as images. ComFiM has a virtual tutor guiding the users through the game (Fig.~\ref{fig:ComFiM}). ComFiM has three levels. In each level, users have roles that are set according to whom starts the communication. Besides, in the first level the user partner is the virtual tutor, i.e., this is an individual level. The second and third levels are played by two users. Here, the tutor is a mediator of the communication. The game works with two action concepts: ``Give me'' and ``I give''. The pattern used in all game levels was the \textit{Active Sharing Pattern}. This pattern was chosen because the game requires the user to know which item he needs to share, i.e., information exchange is necessary. Since this is not needed with the \textit{Passive Sharing Pattern}, this was not used. As the ComFiM doesn't require joint-performance actions, due to technology limitations, the \textit{Active Sharing and Joint-Performance Pattern} was not used either. The game levels are: 

\begin{figure} 
\centering
\includegraphics[width=11.5cm]{ComFiM}
\caption{\textit{Active Sharing Pattern} with two users in second and third level applied in ComFiM. }
\label{fig:ComFiM}
\end{figure}
\begin{itemize}
\item \textbf{First level:} the user have to communicate with the virtual tutor, asking for a tool or giving a tool to achieve the  goals. E.g., the virtual tutor says that the user need to get some apples on a tree. So, the user have to ask him for a ladder.

\item \textbf{Second level:} here, the tutor is a mediator of the communication between a pair of users. Users have to share tools to achieve the game goals. The roles vary according to each move. E.g., assuming that User 1 started the move, he should ask (``Give me'') User 2 for a tool to complete an specific task. So, User 2 should have to give (``I give'') this tool to User 1 in order to complete the task. Then, at the second move, User 2 should start the move and the roles will reverse (Fig.~\ref{fig:ComFiM}).

\item \textbf{Third level:} this level is similar to the previous one. But now, in a single move, each user has to play each possible role, helping each other to achieve a goal in common, working together. E.g., User 1 should ask User 2 for a tool to complete part of the current task. So, as in the previous case, User 2 should give it to User 1. Then, to complete the task, User 2, in the same move, should now ask for a tool to User 1, which should give it to User 2 (Fig.~\ref{fig:ComFiM}). So, the basic difference is that in this level a deeper communication degree is needed.  
\end{itemize}

\section{Evaluation}

The results shown that, in both cases, the use of the collaboration patterns presented in this paper allowed the games to achieve their goals. PAR and ComFiM were evaluated with people with severe autism; five youngsters (ages between 11 and 17 years old) and four children (ages between 5 and 10 years old), respectively. The evaluation indicated that both the interaction on the multitouch technology and the collaboration patterns allowed engaging users in an attractive experience and encouraging communication and social interaction.

In each level of PAR it was noted verbal and gestural interaction expressions among users, such as guide the partner to perform tasks in the game, physical contact, look at the partner, ask for help, answer, rectify, complain, smile, laugh, commemorate, reject, encourage, and thank (Fig. \ref{fig:parRes}). In the game progress users realized new kinds of interaction expressions and they engaged in social situations more extensively, i.e., with a high number of these interaction expressions.

\begin{figure} [!htb]
\centering
\includegraphics[width=11.6cm]{parResultado}
\caption{Verbal and gestural expressions performed by users in PAR.}
\label{fig:parRes}
\end{figure}

%\begin{figure} 
%\centering
%\includegraphics[width=7cm]{parResultado1}
%\caption{Social situations de acordo com o number of  interaction expressions. Type I ()Verbal and gestural interaction expressions obtained with each collaboration pattern in PAR game.}
%\label{fig:parRes1}
%\end{figure}

The results about ComFiM show that applying the \textit{Active Sharing Pattern} to the game allowed to provide a collaborative environment in which fifteen different communications intentions were noted, such as gestures, indications, smiles and even verbal speak. As Figure \ref{fig:ComFiMdata} shows, the communications intentions increased according to the level. Moreover, besides the impairments of the children, they achieve a higher communication degree interacting with real partners than with the virtual tutor. This show that this kind of game can be used to encourage communication among people with autism.

\begin{figure} [!htb]
\centering
\includegraphics[width=11.7cm]{ComFiMdata1}
\caption{Communications intentions (verbal and gestural) obtained with ComFiM.}
\label{fig:ComFiMdata}
\end{figure}

We highlight the motivation generated in the users to help and get support from the other user to achieve their goals during both games. More active users performed the tasks faster, and learned to respect the rules of the game, to help and motivate the partner to cooperate through interaction situations, such as orientation, physical contacts, encouragements, complaints and even using verbal expressions. More receptive users, with greater difficulty in the game, asked for help with several interactive expressions, and tried in any way to perform the required actions.

\section{Conclusions}

We emphasize the advantages of multitouch technology and the significance of the proposed collaboration patterns.
Multitouch interaction is very flexible to adapt to difficulties of the users; the fact of allowing the manipulation of objects with hands, without needing input devices, facilitates their interaction and adaptation to the interface. In addition, the multiuser feature is very favorable to contribute to the social aspect, which is important to encourage in users with severe autism. The results in this work indicate that applications in multitouch technology has a great potential to facilitate the development of social skills in people with autism. However, these benefits are linked with the quality of the applications developed. Moreover, some particular features of this kind of technology, mainly its price, make it less accessible to institutes specialized in autism and to the final users.

The collaboration patterns proposed in this work gradually encourage collaborative activities and allow the appearance of verbal and gestural interaction expressions among users with severe autism. This indicates the importance to design strategies according to specific characteristics of the target users.

The significant characteristics of the proposed collaboration patterns also allow us to suggest that they might be used in other games or collaborative applications aimed at fostering social interaction and collaboration among people with autism.

\section{Future Work}

We will work further to improve collaborative strategies for users with severe autism. During the evaluation process, we observed that the collaboration result in a ``foreign" process to these users. Therefore, we will analyze the patterns of collaboration proposed from an intercultural perspective of Human Computer Interaction. We will adopt the Cultural Viewpoint Metaphors (CVM) tool \cite{salgado} of Semiotic Engineering which supports the decision process on the aspects of intercultural experience in the design process.

In the future, we also plan to investigate the possibility of adapting existing collaboration models to the applications' design  for people with severe autism.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{thebibliography}{4}

\bibitem{apa} American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th Ed (2000).

\bibitem{klin} Klin, A. Autism and Asperger syndrome: an overview. Revista Brasileira de Psiquiatria, 28, s3-s11 (2006).

\bibitem{tse} Tse, E., Greenberg, S., Shen, C. and Forlines, C. Multimodal multiplayer tabletop gaming. In Proc. Workshop on Pervasive Games, (2006).

\bibitem {cos}	COSPATIAL project. Communication and social participation: collaborative technologies for interaction and learning, \url{http://cospatial.fbk.eu/sas}

\bibitem{bauminger} Bauminger, N., Goren-Bar, D., Gal, E., Weiss, P., Kupersmitt, J., Pianesi, F., Stock, O. and Zancanaro, M. Enhancing social communication in high-functioning children with autism through a co-located interface. In: 9th IEEE Workshop on Multimedia Signal Processing, pp. 18-21. (2007).

\bibitem{battocchi} Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Ben Sasson, A., Gal, E. and Weiss, P.L. Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). pp. 197 - 204. ACM, New York, NY, USA (2009).

\bibitem{giusti} Giusti, L., Zancanaro, M., Gal, E. and Weiss, P. Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder. In: Proceedings SIGCHI (CHI '11). pp. 3295-3304. ACM, New York, NY, USA (2011).

\bibitem{salgado} Salgado, L. C. C.; Leit\~{a}ƒo, C. F.; de Souza, C. S. A Journey Through Cultures: Metaphors for Guiding the Design of Cross-Cultural Interactive Systems. Heidelberg: Springer, 2012. 136 p.

\end{thebibliography}

\end{document}
