Blending speech and graphical user interfaces: an empirical study on multimodal mobile interaction

Abstract

Mobile phones have already outnumbered personal computers. Although until recently the majority of phones were used mainly as voice communication devices, the emergence of powerful application-centric mobile devices such as personal digital assistants (PDAs) and smart-phones, has created excitement for the future of mobile computing. Despite the recent explosion of advanced mobile applications such as web browsing and video consuming, constraints such as reduced display size and limited input interaction methods pose new challenges for interaction designers. The use of more than one interaction modalities has been proposed as a possible solution to overcome these limitations. Multimodal interfaces process two or more combined user input modalities such as speech, pen or touch, in a coordinated manner with multimedia system output and can potentially offer more rich, robust and adaptive interaction experience. This dissertation investigates multimodal interface design and evaluation with a focus on mobile interaction. One of the main aims is to showcase how to design information-filling multimodal systems that combine speech and graphical user interface (GUI) input (e.g. pen or touch). From the interaction design standpoint, the main focus is on identifying and exploiting the synergies resulting from the mixing of modalities in order to create robust and effective interfaces. The system designed and implemented, allows both unimodal and multimodal interaction and can be used across different platforms such as PCs, PDAs and mobiles such as the popular iPhone device. For the evaluation of multimodal interaction both established and novel metrics are employed. Two new metrics were devised that measure the relation of input modality preferences to unimodal efficiency and the synergies found in a multimodal system. The proposed metrics, relative modality efficiency and multimodal synergy, can provide valuable information to the interaction design process of multimodal systems. Furthermore affective evaluation incorporating biosignals such as skin conductance and brain waves (EEG) has provided a rich amount of data not previously available. Use of such physiological channels and their elaborated interpretation is a challenging but also a potentially rewarding direction towards emotional and cognitive assessment of multimodal interface design. Evaluation results show that multimodal systems can potentially outperform unimodal systems in terms of both performance and user satisfaction when designed to maximize the synergies between the modalities. Overall this research entails significant implications for designing efficient mobile interfaces.