Title:
Traces of modal synergy: studying interactive musical sonification of images in general-audience use
Traces of modal synergy: studying interactive musical sonification of images in general-audience use
Author(s)
Rönnberg, Niklas
Lowgren, Jonas
Lowgren, Jonas
Advisor(s)
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
Photone is an interactive installation combining color images with musical sonification. The musical expression is generated based on the syntactic (as opposed to semantic) features of an image as it is explored by the userﾒs pointing device, intending to catalyze a holistic user experience we refer to as modal synergy where visual and auditory modalities multiply rather than add. We collected and analyzed two months' worth of data from visitorsﾒ interactions with Photone in a public exhibition at a science center. Our results show that a small proportion of visitors engaged in sustained interaction with Photone, as indicated by session times. Among the most deeply engaged visitors, a majority of the interaction was devoted to visually salient objects, i.e., semantic features of the images. However, the data also contains instances of interactive behavior that are best explained by exploration of the syntactic features of an image, and thus may suggest the emergence of modal synergy.
Sponsor
Date Issued
2019-06
Extent
Resource Type
Text
Resource Subtype
Proceedings
Rights Statement
Licensed under Creative Commons Attribution Non-Commercial 4.0 International License.