Combining Information from Faces and Voices in Person Discrimination, Recognition, and Judgments of Learning

We often combine cues from our multisensory environment in ways that improve behavior. According to the principal of inverse effectiveness (PIE), weak responses to unisensory information (i.e., originating from one sensory modality) benefit most from the addition of concurrent information in a second modality (Meredith & Stein, 1986). The PIE has been widely extended to human perceptual responses (e.g., Collignon et al., 2008; Diederich & Colonius, 2004) but its role in other cognitive domains has not been examined. This dissertation examines perceptual distinctiveness (Experiment 1) and episodic recognition (Experiments 2 – 4) for unfamiliar faces and voices as we often encounter them in life—as multisensory pairs. In Experiment 1, separate face distinctiveness and voice distinctiveness were shown to contribute interactively to a combined face-voice space. Thus, although face distinctiveness contributed more to face-voice space, more important was the relation of faces and voices. In Experiments 2 – 4, pairing faces with voices at encoding did not enhance subsequent face-only recognition. However, pairing voices with faces at encoding enhanced voice-only recognition. Also, face-only recognition was superior to voice-only recognition under matching, unisensory encoding conditions (i.e., those without multisensory impairs). Thus, the PIE was extended to memory. Combined with the perceptual distinctiveness findings, it is surmised that unfamiliar faces and voices may be integrated via learned statistical properties concerning how face and voice features go together (i.e., crossmodal associations, Spence, 2011). Experiments 5 and 6 examined the effect of adding visual lip movement to spoken words in various levels of background noise on perception and judgments of learning (JOLs). Whereas visual articulations improved spoken word identification in perception more with higher noise levels (consistent with PIE and Ross et al., 2007), JOLs did not show a similar pattern. Thus, the PIE may not extend to some cognitive domains, perhaps because of overshadowing from more abstract, conceptual processes.