Detecting an Offset-Adjusted Similarity Score based on Duchenne Smiles
Detecting interpersonal synchrony in the wild through ubiquitous wearable sensing invites promising new social insights as well as the possibility of new interactions between humans-humans and humans-agents. We present the Offset-Adjusted SImilarity Score (OASIS), a real-time method of detecting similarity which we show working on visual detection of Duchenne smile between a pair of users. We conduct a user study survey (N = 27) to measure a user-based interoperability score on smile similarity and compare the user score with OASIS as well as the rolling window Pearson correlation and the Dynamic Time Warping (DTW) method. Ultimately, our results indicate that our algorithm has intrinsic qualities comparable to the user score and measures well to the statistical correlation methods. It takes the temporal offset between the input signals into account with the added benefit of being an algorithm which can be adapted to run in real-time will less computational intensity than traditional time series correlation methods.
Transcript 1,558 words
Hey, everyone, and welcome back for another deep dive. Thanks for having me. You ever notice how, like, a really good laugh with someone, you know, can make you feel, like, really connected, like you're on the same page, same wavelength? Yeah, totally. Well, get this. Researchers are finding ways to, like, actually measure that connection. Wow. And it all comes down to smiles. Interesting.
So today we're going to dive into this fascinating world of facial recognition and algorithms and a new study that's trying to decode those genuine, heartfelt smiles we all love. You know, like the Duchenne smile. Yes. Those Duchenne smiles. What's really cool about this is it really zeroes in on those Duchenne smiles. Right. The ones that engage not just, like, your mouth, but the muscles around your eyes, too.
And there's a whole body of research that links these smiles to, you know, genuine positive emotions. Oh, interesting. Yeah. And that's what makes them so intriguing to study. Okay, hold up. Before we get too far, I just want to know, what is the source material that we're looking at for this deep dive? So we are looking at a research paper that was presented at CHI 23.
Okay. Which is a huge conference, all about human-computer interaction. Okay, that makes sense. So how do they even go about measuring something, I guess, as subtle as a genuine smile? Did they, like, stick people in the lab and be like, all right, laugh? Well, they actually did something way more interesting than that. Okay. So they had people have these conversations over Zoom.
Right, yeah. And they were answering these questions that were specifically crafted to try to create a sense of closeness. Interesting. So think of it like a fast track of friendship, right? Oh, okay. So they record these interactions, and they use that data to train their algorithm. Oh, that's actually really smart. So it's more like observing smiles, I guess, in their natural habitat.
Right. Rather than forcing it out in some, like, sterile lab setting. Exactly. And that's what we call ecological validity. Mm-hmm. Because they wanted to capture those smiles that are coming from a genuine connection. Right. As opposed to just someone saying, cheese, hit a camera. Right, right. So to do this, they used this really clever software called OpenFace. And basically what it does is it tracks even the tiniest muscle movements in your face.
So, like, those little crinkles around your eyes. Oh, wow. Or the way your cheeks lift when you really smile. Okay. I have to ask, how do they even begin to measure something that, like, small? Yeah. Like, is there a microscope on these people's faces? Well, so picture this. They basically break down each smile into these, like, micro movements, almost like a secret code for every expression, right?
Wow. And so each movement has a name, like, a really fancy name. Okay. Like, they'll call it, like, AU6C for the eyes and AU12R for the cheeks. What? But essentially they're just labels for specific muscle actions. So they're mapping out, like, every twitch and twinge of a smile, like an emotional GPS, almost. That's a great way to put it. And you know what they do with that?
Yeah. They use those measurements to create what's called a time series. Okay. Which is basically a graph that shows how someone's smile changes over time, getting stronger or weaker. Okay. And then they feed these graphs into their algorithm that they designed. Okay. It's called Oasis. Oasis. That sounds kind of mysterious. So this algorithm, is it, like, a similarity score for smiles? Yeah.
Like, it tells you how much your smile matches the person you're talking to. Yeah. You're getting it. Yeah. So Oasis's, like, main job is to analyze those smile graphs and figure out how similar two people's smiles are during a conversation. Okay. But they didn't just stop there. Right. They wanted to know, like, is Oasis actually any good at this? Right. Can it match the way we humans perceive smile similarities?
Right. Right. Because I imagine comparing data that's changing over time, like, those smile graphs must get pretty complicated. Yeah. Yeah. It does. Yeah. Like, how do you even, how do you test that? Okay. So remember those Zoom conversations that they recorded? Yeah. So they actually had 27 people watch those videos. Oh. And press a button every time they felt like the two people talking were, you know, kind of vibing on the same smile wavelengths.
Interesting. Like, yep, those smiles totally match. Oh, okay. That's cool. Yeah. So they're basically crowdsourcing the, like, smile similarity judgment. Right. And using real people as the kind of benchmark for Oasis to measure up again. Exactly. They wanted to see if Oasis's analysis, like, lined up with how just, like, regular people perceive smile similarity. Right. But they didn't just compare it to human judgment.
Okay. They also wanted to see how it stacked up against other methods of analyzing, you know, those smile graphs, those time series that we were talking about. Right, right, right. Yeah. Because comparing, like, I imagine comparing data that's changing over time must get pretty complex. It does. So think about it this way. Like, imagine you're trying to compare two dance routines. Okay.
Right. And they might have the same moves. Right. But one dancer might be a little faster or a little slower. Right. It's still the same dance, just kind of shifted in time. Yeah. So some of the other methods they used are kind of like, you know, comparing those dances move for move. Okay. Whereas Oasis is more like saying, hey, these routines, they give off the same overall vibe.
Yeah. Even if the timing's a bit off. Ah, okay. So it's not just about perfectly matching every, like, uptick and downtick of a smile. It's about capturing kind of the essence of the similarity, even if the timing's a little off. Yeah, precisely. Okay. And so when they crunched the numbers, what they found was that Oasis was, like, the clear winner in the matching human perception game.
Yeah. Like, it was picking up on these, like, subtle moments where two people's smiles would, you know, sort of dip or rise together. Interesting. Yeah. Things of the other methods often missed. Wow. So Oasis is, like, the ultimate wingman for, like, reading those unspoken, like, smile-based cues we give each other. That's really cool. But I got to ask, like, why go to all this trouble to, like, measure smiles?
Right. What's the big deal? That's the million-dollar question, right? Yeah. It's one that kind of takes us beyond just this research paper. Okay. It's about recognizing that smiles aren't just, you know, some random facial twitch. Yeah. They're, like, a powerful tool that we use to connect and communicate with each other. Okay. I'm intrigued. Tell me more. Well, think about it this way.
Think about it this way. Like, what comes to mind? Spill the tea. Well, imagine, like, imagine a world where technology could not only hear our words. Right. But, like, it could understand the emotions behind them. Yeah. You know? Like, picture a virtual assistant. Okay. That could, like, sense when you're genuinely happy. Yeah. Maybe crack a joke to match your mood. Yeah. Or pick up on those moments when, like, a reassuring word would mean the world to you.
Okay. Now that you mention it, yeah, that would be a game changer. Right. Instead of those, like, robotic interactions, we could have tech that actually gets us, you know, like, responds to our vibes. Exactly. Yeah. And it's not just about, like, making our virtual assistants more personable. Right? Right. Like, think about the possibilities. Like, even in healthcare, imagine therapists using this technology.
Okay. To kind of get, like, real-time feedback. Right. On a patient's, you know, emotional engagement during, like, online sessions. Oh. Or picture, like, robots. Okay. Working alongside humans. Right. In, like, factories or hospitals. Yeah. Right. Yeah. That can sense a teammate's, you know, frustration. Right. And maybe offer assistance. Right. Before, like, things escalate. Wow. I hadn't even considered those angles. Yeah. It's like taking the guesswork out of our interactions.
Right. And, like, allowing for more empathy and understanding, even between humies and machines. Exactly. That's wild. That's the beauty of this research, right? Yeah. It just opens up, like, a whole new world of possibilities. Right. For designing technology that's more intuitive. Right. More responsive. Yeah. And ultimately, like, more human-centered. This has been, this deep dive has been mind-blowing. It's really cool stuff.
We started with a simple smile. I know. And look where we ended up. Robots reading emotions. Therapists getting a digital helping hand. Yeah. Yeah. And technology that actually gets us. It's amazing. Who knew those little crinkles around our eyes held so much potential. Right. It just goes to show, sometimes, like, the biggest breakthroughs come from understanding the smallest details. Thanks for joining us on this fascinating journey into the science of smiles.
It really makes you think twice about the power of a genuine grin, doesn't it? Until next time, keep those smiles shining bright.