Speech and graphical interaction in multimodal communication

Abstract

Graphical communications, such as dialogues using maps, drawings, or pictures, provide people with two independent modalities through which they can interact with each other. In such conversation, graphical interaction can be both sequential and parallel, affected by the activity-dependent constraints imposed by the task performed in the interaction (Umata, Shimojima, Katagiri, and Swoboda (2003)). In this paper, we compare the patterns of speech and graphical interaction in collaborative problem-solving tasks. The amount of speech overlap did not show a significant difference among four task conditions, although graphical overlaps did. This shows that both resource-dependent and activity-dependent constraints play a significant role in determining the interaction organization.