From Unisemiotic to Polysemiotic Narratives: Translating across semiotic systems
Author
Summary, in English
Human communication is both polysemiotic and multimodal; it is comprised of ensembles of representations from different semiotic systems in different sensory modalities. These semiotic systems, such as language, gestures, and pictures consist of signs and relations between signs, with system-specific affordances (Kendon, 2004; Zlatev, 2009; Sonesson, 2014). These systems all have their unique storytelling potentials, and how meaning is conveyed and what is communicated may differ across systems and modalities (Kress, 2010; Green, 2014). This thesis investigates the influence of the source semiotic system on the way the story content is transmitted from unisemiotic and unimodal narratives to polysemiotic and multimodal narratives. The source systems were language in the speech narrative mode (SNM) and a sequence of pictures in the picture narrative mode (PNM). An experimental method was used to see whether there are differences in the polysemiotic narratives – that is, in the retellings through speech and gestures – that could be connected to the semiotic system in which the narratives initially were perceived. The participants were 38 native Finnish speakers, and the experiment was carried out in Finnish. Due to the system-specific differences, it was hypothesised that pictorial iconicity would be reflected as a higher number of iconic gestures – especially first person enactments – and ideophones in PNM, and SNM would result in greater narrative coherence reflected, for example, in the more diverse use of connective devices and a higher number of plot elements.
Contrary to expected, more iconic gestures were found in the narratives translated from the SNM condition. However, in line with the hypotheses, first person enactments were more frequent in the narratives of the participants who had experienced the story through pictures. Also contrary to expected, the difference in the use of connective devices between the conditions was not remarkable. However, together with some additional differences between the groups that had not been anticipated, the results indicate that a story given in different semiotic systems may indeed lead to different polysemiotic narratives.
Contrary to expected, more iconic gestures were found in the narratives translated from the SNM condition. However, in line with the hypotheses, first person enactments were more frequent in the narratives of the participants who had experienced the story through pictures. Also contrary to expected, the difference in the use of connective devices between the conditions was not remarkable. However, together with some additional differences between the groups that had not been anticipated, the results indicate that a story given in different semiotic systems may indeed lead to different polysemiotic narratives.
Department/s
- Master's Programme: Language and Linguistics
Publishing year
2018
Language
English
Full text
- Available as PDF - 11 MB
- Download statistics
Document type
Student publication for Master's degree (two years)
Topic
- Languages and Literatures
Keywords
- bodily mimesis
- cognitive semiotics
- co-speech gestures
- embodied narratives
- experientiality
- iconicity
- ideophones
- intersemiotic translation
- language
- mimetic schema
- multimodality
- narratives
- phenomenology
- pictorial semiotics
- pictures
- polysemiotic communication
- semiotics
- semiotic systems
- sensory modalities
- signs
- sound symbolism
Supervisor
- Jordan Zlatev (Docent)