Do you read me? (E)motion Legibility of Virtual Reality Character Representations
K. Brandstätter, B. J. Congdon and A. Steed, "Do you read me? (E)motion Legibility of Virtual Reality Character Representations," 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bellevue, WA, USA, 2024, pp. 299-308, doi: 10.1109/ISMAR62088.2024.00044.
We compared the body movements of five virtual reality (VR) avatar representations in a user study (N=53) to ascertain how well these representations could convey body motions associated with different emotions: one head-and-hands representation using only tracking data, one upper-body representation using inverse kinematics (IK), and three full-body representations using IK, motioncapture, and the state-of-the-art deep-learning model AGRoL. Participants’ emotion detection accuracies were similar for the IK and AGRoL representations, highest for the full-body motion-capture representation and lowest for the head-and-hands representation. Our findings suggest that from the perspective of emotion expressivity, connected upper-body parts that provide visual continuity improve clarity, and that current techniques for algorithmically animating the lower-body are ineffective. In particular, the deep-learning technique studied did not produce more expressive results, suggesting the need for training data specifically made for social VR applications.
Transcript 1,897 words
Welcome to our deep dive into virtual reality avatars and how well they convey emotion. Imagine you're in a social VR space and you can't see faces, just avatars. Could you still understand someone's feelings just from their avatar's movements? That's a fascinating question, and it's at the heart of the research we're diving into today. It is. The studies you've provided compare different types of VR avatars and how well they express six basic emotions.
Anger, disgust, fear, happiness, sadness, and surprise. Precisely. And as we'll see, the findings challenge some common assumptions. It turns out the most technologically advanced avatar isn't necessarily the most emotionally expressive. So before we get into the specifics, can you give us a rundown of the different types of avatars included in the research? Certainly. The researchers evaluated five distinct avatar types, each representing a different level of complexity and how it's animated and controlled.
Okay. I'm curious to hear more about them. Well, the first one is full body motion captured or FBMC for short. FBMC. Got it. This is the gold standard. It uses data captured from real human actors' movements, recording every nuance of their emotional performance. It's as close to a one-to-one representation as we can get with current technology. So FBMC sets the bar for realism. What about the others? What are we comparing it to?
Right. So moving down the line in terms of complexity, we have full body IK or FBIK. This type uses algorithms, specifically inverse kinematics, to calculate full body movement based solely on head and hand tracking data. So it's trying to extrapolate a whole body's worth of movement from just the head and hands. That's a tough task. It is indeed. Then we have full body age Ural or FBAG, which utilizes a deep learning model to predict full body poses also from limited tracking data.
Ah, so this is where AI comes into play. It's essentially trying to fill in the blanks, right? Exactly. It's predicting how the body should move based on the available input. Then we have upper body IK or UBI, similar to FBIK, but it focuses solely on animating the upper body. Okay. And finally? Finally, the simplest avatar type is head and hands or HH. As the name suggests, it only renders the head and hands. No lower body at all.
So we have a good range of avatar types, some very basic and some quite advanced. How did the researchers actually test how expressive these avatars were? They conducted a study with 53 participants. Participants viewed videos of these different avatars expressing the six basic emotions. Then they were asked to identify the emotion being conveyed and to rate how much they liked the avatar's movements and how difficult they found it to recognize the emotion.
So the researchers considered not only accuracy, but also how natural and convincing the avatar's movements were to the viewers. Precisely, they recognized that a technically accurate portrayal of emotion might not be perceived as authentic or engaging. Makes sense. So how did the different avatar types perform? Did the results match what you might expect? Well, some did and some didn't. As anticipated, the full body motion capture FBMC was the most accurate at conveying emotions with an accuracy rate of about 67%.
However, the deep learning model FBAG actually performed on par with the simpler IK avatars UBIK and FBIK. This suggests that despite its sophistication, AI isn't always the clear winner when it comes to emotional expressiveness. Hmm. So not exactly a straightforward correlation between technology and performance. And what about the simplest avatar head and hands? I imagine it struggled. It did, unfortunately. HH had the lowest accuracy. It seems that seeing more of the body, even if it's algorithmically generated, really aids in understanding emotions.
So a connected body, even if it's not perfectly realistic, seems crucial for transmitting emotional cues. Were there any other interesting patterns in the findings? Yes, several. The researchers found that across all the avatar types, happiness was actually the most difficult emotion to recognize. You know, that makes me wonder. Are we maybe becoming too accustomed to these exaggerated facial expressions we see all the time on social media and in emojis?
That's an intriguing thought. It's possible that we've come to expect a certain level of overt expression for happiness. Exactly. And maybe genuine happiness is more subtle and harder to convey without those facial cues. That's certainly consistent with what the study found. In contrast, anger and sadness were the most accurately recognized emotions across all the avatar types. So even without a frowning face or tears, we can still pick up on those emotions pretty clearly from body language alone.
Yes, it seems that the physical manifestations of those emotions, like clenched fists or slumped shoulders, translate more effectively across different forms of avatar representation. So the body can speak volumes even without the face. It can. But there were also some instances where emotions were easily confused. Participants often mistook happiness for anger. Oh, that's interesting. And they sometimes mistook surprise for anger as well.
I guess without facial expressions to provide context, the energy and sudden movements associated with those emotions could be misinterpreted. That's right. A quick upward motion of the hand, for example, could mean a lot of different things. It could be excitement or it could be frustration. It seems we need to learn to read those subtler cues of body language when facial expressions are missing.
Absolutely. Context is key. Were there any other standout findings? Did any particular avatar types excel at conveying certain emotions? Yes. One finding was particularly curious. The full-body IK avatar, FBIK, was surprisingly good at conveying fear. FBIK. The one that uses algorithms to calculate full-body movement. Yes, that's the one. Why would that be? The researchers speculated that it might be due to the somewhat jerky leg movements that the IK algorithms produce.
So a kind of happy accident. Technical limitation actually enhanced the emotional expression. It does seem that way. It reminds us that sometimes even imperfections can contribute to a sense of realism. Well, human movement isn't always perfectly smooth and graceful, is it? Exactly. It's often those little quirks and imperfections that make us human. Speaking of quirks, let's go back to A.G.Roll, the deep learning model.
You mentioned earlier that it had some trouble accurately representing certain movements. It did, yes. Despite being a very sophisticated AI age-year-old, struggled with leg movements and head direction. In what way? Well, it frequently produced very minimal leg movements, so the avatar ended up kind of sliding around, which looked rather unnatural. Like it was gliding on an invisible surface. Yes, a good way to put it.
And then with the head age-year-old, often tried to align the body's direction with the direction the head was facing. I can see why that would make sense for actions involving locomotion. You'd think so, but... But not so much for emotional expressions where the body and head might not always be in sync. Right. Think about it. We might look away in shame or lower our gaze in sadness, even while our body tenses with anger.
Our body language doesn't always neatly align with our gaze when we're experiencing strong emotions. It seems even with AI, the quality of the data it's trained on really matters. It's crucial. And in this case, the data used to train AGRaw was primarily focused on actions involving locomotion, which aren't as common in social VR interactions. Most of the action in those settings happens in the upper body.
So this begs the question, what do these findings mean for the future of VR? Are we stuck with avatars that are emotionally limited and artificial? I wouldn't say that at all. What this research highlights is that creating truly authentic and emotionally expressive avatars is a complex challenge. It requires a deep understanding of both technology and human behavior. It's not just about pushing the limits of what's technologically possible.
It's about understanding how we communicate and perceive emotions on a psychological level. Precisely. And as VR technology continues to advance, so too will our understanding of how to bring those subtle nuances of human emotion into the virtual world. I think that's a key point. As we refine our understanding of how to translate emotions into avatar movement, we open up new possibilities for human interaction.
We can connect in ways that go beyond physical limitations. So as we go deeper into this world of virtual embodiment, what else should we be thinking about? What other factors come into play? Well, we've talked about specific movements like hand gestures and head movements. But I think the real key to expressive avatars is how those movements are integrated. It's not just about having a bunch of well-animated gestures.
It's about making them work together in a way that feels natural and cohesive. So it's about the overall choreography, the way these movements flow together to tell a story, to convey an emotion authentically. Exactly. Think about the difference between a novice dancer who can do the steps but isn't very fluid or graceful. And then you have a seasoned performer who really embodies the music moving with intention and purpose.
That's the kind of artistry we should be aiming for in virtual embodiment. It's that intangible quality that takes movement beyond just mechanics and turns it into genuine expression. How do we get there with VR avatars? I think it requires a deep understanding of human kinetics. The science of how our bodies move in the real world. We have to translate that knowledge into the digital realm so that avatars move and respond in a way that feels authentic and believable.
It's a big challenge, but has so much potential. It's almost like we're teaching a machine to understand and speak the language of the human body. In a way, we are. And that's what makes this research so important. It gives us a starting point for exploring how we can create virtual experiences that are as emotionally rich and meaningful as our real-life interactions. It also shows how important collaboration is.
Researchers, developers, even artists, all working together to push the boundaries in this new frontier of human expression. It really speaks to human ingenuity. This drive to connect with each other no matter the physical distance. Well said. It looks like we've reached the end of our deep dive into the fascinating world of virtual reality avatars and emotions. We've covered a lot of ground, explored the challenges, and the incredible possibilities that lie ahead.
It has been an enlightening conversation. A reminder that human connection is still paramount, even in these virtual realms. And to our listeners, as you enter these digital spaces, remember to pay attention not only to the words being spoken, but also to the silent language of the body. Notice those subtle cues that reveal the emotions underneath. Because it's in those small gestures, those nuanced movements, that we find the true essence of human connection.
And on that note, we wrap up our deep dive. Until next time, keep exploring, keep questioning, and keep looking for those moments of genuine connection wherever they may be.