Interference between gestures and words
This thesis explores the idea that a speaker's gestural and verbal behaviours are mutually influential in the comprehension process. A Stroop-type interference paradigm was adopted as a tool for investigating whether or not listeners process to-be-ignored gestural information and how this information influences the processing of spoken words. In Experiments 1-4, static pointing (deictic) gestures and corresponding spoken and written words showed symmetrical interference. Incongruent words slowed responses to gestures, and incongruent gestures slowed responses to words, compared with congruent arrangements. These findings support the idea that both pointing gestures and words are processed in comprehension. Furthermore, the results of Experiments 5-11 suggest that the mutual influence of the two dimensions is largely independent of specific stimulus-response compatibilities. Collectively, these findings are difficult to reconcile with models of Stroop interference which place the locus of the effect at response selection. Instead, they are more consistent with the position that the two sources of information interact at a semantic stage of processing. Arrows (Experiment 12) and spatially positioned dots (Experiment 13) also produced symmetrical interference effects when paired with spoken words, raising the possibility that it is the spatial nature of the pointing gestures which is important in influencing the comprehension of spoken words. In support of this suggestion, other non-spatial gestures such as emblems (Experiment 14), iconics (Experiment 15) and facial gestures (Experiment 16) did not interfere with responses to verbal material. However, symmetrical effects did return when subjects were asked to make affective judgements to either emotional words or schematic facial gestures (Experiment 17). The results are discussed with reference to research on the orienting of social attention, the stimulus-driven "capture" of attention, models of integration, and a processing framework which incorporates the notion of informational integration at "semantic" levels of processing.