The effects of association strength and cross-modal correspondence on the development of multimodal stimuli

Abstract

In addition to temporal and spatial contributions, multimodal binding is also influenced by association strength and the congruency between stimulus elements. A paradigm was established in which an audio–visual stimulus consisting of four attributes (two visual, two auditory) was presented, followed by questions regarding the specific nature of two of those attributes. We wanted to know how association strength and congruency would modulate the basic effect that responding to same-modality information (two visual or two auditory) would be easier than retrieving different-modality information (one visual and one auditory). In Experiment 1, association strengths were compared across three conditions: baseline, intramodal (100 % association within modalities, thereby benefiting same-modality retrieval), and intermodal (100 % association between modalities, thereby benefiting different-modality retrieval). Association strength was shown to damage responses to same-modality information during intermodal conditions. In Experiment 2, association strength was manipulated identically, but was combined with cross-modally corresponding stimuli (further benefiting different-modality retrieval). The locus of the effect was again on responses to same-modality information, damaging responding during intermodal conditions but helping responding during intramodal conditions. The potential contributions of association strength and cross-modal congruency in promoting learning between vision and audition are discussed in relation to a potential default within-modality binding mechanism.