Avatar provides live signing on stage in unique project
With enormous eyes, a huge mouth and defined, prominent eyebrows, an avatar in the form of a ghost using sign language in real time takes the stage. During 2022, Riksteatern Crea – one of the world’s leading sign language theatres – and researchers at Lund University Humanities Lab trialled new innovative and creative solutions in order to make the technology work for this multi-dimensional theatre experience.
We are sitting in the Humanities Lab studio in Lund. On computer screens, research engineer Stefan Lindgren is showing us what it takes to bring an avatar to life. The technique is called motion capture, abbreviated to mocap, and it records and stores movements on a computer. Gaming and movies have long worked on making characters’ movements realistic. ABBA made their comeback using the technology, while the film “Avatar,” released a decade ago, features astonishing scenes.
Streaming in real time
The basic idea is to create a three-dimensional recording – small round markers are attached to different points on a person’s body. These markers are recorded by a large number of mocap cameras, which are placed around the person. When the person moves, the system’s computers record their movements in three dimensions. Special software transfers the movements from the person to the corresponding points on a computer-generated body. Until now, the Humanities Lab has never streamed the results in real time, instead they have worked with large volumes of data afterwards.
“We help many different kinds of researchers, such as archaeologists, linguists and behavioural scientists, and we have long experience of making recordings. But when Riksteatern Crea wanted to research whether signing avatars could be live-streamed, I thought blimey – that is a seriously exciting challenge,” Stefan Lindgren recalls.
Small details make the difference
Riksteatern Crea is the leading Swedish professional theatre for deaf people – actors and audience alike. Along with Humanities Lab’s mocap experts, Artistic Director Mindy Drapsa, the director and actors started testing and working in the mocap studio. One of the challenges is that signers express language through bodily gestures, while grammatical detail is conveyed by the face. How could we get at those tiny details in the pattern of movement and simultaneously stream in real time?
Placing markers on the face did not work, the nuances were difficult to capture. Eventually, the researchers found an app that uses a smart phone to film tiny facial movements. Along with gloves, which the researchers were able to calibrate better, as well as the filming from the mocap cameras, it was possible to use three systems to register all the movements at the same time and stream them to the virtual figure. The results were surprisingly good.
“Everything rests on the system being able to see the signing actor and how well the virtual character can interact. If a little marker falls off the actor, chaos ensues. That might mean that one leg will not move, which looks very strange. Taking the leap and daring to do this live, something that has never been done before, is brave,” Stefan Lundgren says.
Riksteatern Crea’s technological know-how also gives a lot of opportunities for the avatar to be projected onto any part of the stage, to fly in from the wings or appear behind the actor’s shoulder.
Globally unique projects
During the autumn, this unique concept has been tested on the audience for the play “Spöket på Canterville” (The Canterville Ghost), and received a positive response. In 2025, the theatre troupe will embark on a nationwide tour. The signing actor does not even need to travel – the technology can all be taken care of remotely. But how will the public be made to realise that it is live?
“The play might start with the ghost approaching the audience, talking to them in sign language and engaging in conversation. The avatar is not simply an add-on, it actually brings a lot to the scenography,” Stefan Lindgren explains.
After a long working process, Artistic Director Mindy Drapsa feels very positive about the development of Riksteatern Crea’s ambitions in this globally unique project.
“It will be incredibly exciting to see a signing avatar on the stage. The audience we tested it on were overwhelmed at experiencing an expressive avatar using sign language in real time. That is an experience that you cannot describe.”
Lund University Humanities Lab
The Humanities Lab is part of the university-wide research infrastructure, and is based at the Faculties of Humanities and Theology. The lab provides access to various kinds of measurement instruments, method knowledge of equipment, e-science tools and materials, and expertise in data processing. Lab activities are focused on research into communication, culture, cognition and learning. Many projects, however, are interdisciplinary, and are conducted in collaboration with social science, medicine, science, technology and e-science. Partners and collaborators are found locally at Lund University, as well as nationally and internationally.
Riksteatern Crea produces ground-breaking dramatic art in Swedish Sign Language. Since 1977, Riksteatern Crea has been part of the Swedish National Theatre.