Perceptual disambiguation by audiovisual information association & congruence and its applications on brain-machine interface

Abstract

The user experience in Human – Computer Interaction (HCI) is constituted by bottom-up and top-down information processing. Due to the disparity between the bottom-up and top-down mechanisms, users often experience perceptual ambiguity which may disorient them in interaction. Herein, I investigated the effects of audiovisual associations on perceptual ambiguity in binocular rivalry and Stroop task and applied the mechanism on BMI applications in order to reduce the perceptual ambiguity in interfaces which provided beneficial effects on user experience and performance. First, I intended to understand the mechanism of how multimodal congruence can be formed at the visual stimulus level by utilizing binocular rivalry as an experimental tool to invoke perceptual ambiguity. The findings from this systematic approach revealed that the associative experience of dissimilar audiovisual information can create strong contextual cues that can reach the synergistic congruence and attain disambiguation of environmental cues. Next, in order to investigate the effects of disambiguation by repetitive associative experience on cognitive control, I carefully observed the degree of cognitive load in visual search task with different auditory congruence conditions which affected the response time of following Stroop task. The results showed that congruence, reverse congruence, and randomness of auditory cues by associative experience can affect the amount of cognitive load in visual search task which can extend its effect and significantly affect the performance in following Stroop task. Finally, the principle of associative experience is applied on Brain-Machine Interface (BMI) and showed that the same principle can deliver synergistic sensory information processing which can bring higher BMI control performance. The overall findings in current thesis suggests that the congruence formation by audiovisual association could reduce the perceptual ambiguity from the discrepancy between the bottom-up and top-down information processing and that this mechanism can be applied in BMI applications to enhance the user experience and performance.