Distinguished Paper Award: MoCaPose: Motion Capturing with Textile-integrated Capacitive Sensors in Loose-fitting Smart Garments
Today we deep dive into one publication that received a UbiComp 2024 distinguished paper awards.
We present MoCaPose, a novel wearable motion capturing (MoCap) approach to continuously track the wearer's upper body's dynamic poses through multi-channel capacitive sensing integrated in fashionable, loose-fitting jackets. Unlike conventional wearable IMU MoCap based on inverse dynamics, MoCaPose decouples the sensor position from the pose system. MoCaPose uses a deep regressor to continuously predict the 3D upper body joints coordinates from 16-channel textile capacitive sensors, unbound by specific applications. The concept is implemented through two prototyping iterations to first solve the technical challenges, then establish the textile integration through fashion-technology co-design towards a design-centric smart garment. A 38-hour dataset of synchronized video and capacitive data from 21 participants was recorded for validation. The motion tracking result was validated on multiple levels from statistics (R2 ~ 0.91) and motion tracking metrics (MP JPE ~ 86mm) to the usability in pose and motion recognition (0.9 F1 for 10-class classification with unsupervised class discovery). The design guidelines impose few technical constraints, allowing the wearable system to be design-centric and usecase-specific. Overall, MoCaPose demonstrates that textile-based capacitive sensing with its unique advantages, can be a promising alternative for wearable motion tracking and other relevant wearable motion recognition applications.
https://dl.acm.org/doi/10.1145/3580883
Transcript 1,614 words
ever feel like, uh, technology can be more of a hassle sometimes than it's worth. Like, don't get me wrong. I love that my fitness tracker keeps me accountable and all, but man, all that fiddling with it. Not exactly my idea of a good time. Totally. It's supposed to make life easier, not more complicated, right? Right. But okay, get this. What if instead of strapping on yet another gadget,
your clothes, your actual clothes could be the ones tracking your movements. Now that's the idea behind Mocha Pose. And let me tell you, it's pretty mind blowing. The deep dive today, we're looking at a research paper and it lays out how scientists and get this fashion designers teamed up to make motion capture like truly as easy as just putting on a jacket.
Okay. Now that's a fashion show I can get behind, but seriously, how can a jacket, even a super cool one actually capture movement? Well, think about it this way. How does your phone know when you're tapping on it? It just like senses my finger. Exactly. It senses changes in the electrical charge. So Mocha Pose, it uses that same principle, capacitive sensing, but with these special conductive fabric patches sewn right into the jacket.
Oh, okay. So no like sensors or wires, just the fabric itself acting as this giant touch screen kind of. You got it. And they actually created two prototypes. The researchers did first one, all about making sure this wild idea could actually work, you know, but then the second prototype, that's where they got really ambitious. They went for functionality for sure, but also style.
Okay. Now we're talking. So give me the fashion tech breakdown here. What did they change? So prototype number one, pretty basic, just off the shelf conductive fabric on like a standard bomber jacket, but prototype number two, they use this really cool breathable mesh and then incorporated the conductive patches using, get this, a special layering technique. Way more comfy, way more durable, basically ready for the real world, not just the lab. All right. I could see myself rocking that for
sure. Okay. So they've got people wearing these jackets moving around. What happens with all that data? That's where it gets really interesting because they collected a ton of it. We're talking 21 different people moving for over 38 hours. Wow. That's commitment. Right. All that to train the system, basically. Imagine feeding a computer 38 hours worth of movements, right? And they synced all of those movements with video recordings so they could use AI to start
finding the links between how the jacket moved and what the person was actually doing. So the AI is basically learning, okay, when the fabric stretches like this, it means the arm is doing that. And here's the real aha moment. Yeah. It's not just about how much the fabric stretches. It's about the whole picture. You know, Mocha Pose uses all that data to actually figure out the position of all your
joints in 3D space. Yeah. Your entire pose just from those fabric signals. Okay. So we've established this jacket. It's not just a fashion statement. It's like seriously high tech, but how accurate are we talking really? Can it tell the difference between me like reaching for another cup of coffee and, you know, busting out my victory dance? That's the thing. It's actually surprisingly good at that
to measure how accurate it is. They looked at how close the jacket's predictions were to the actual movements, right? Okay. So kind of like a bullseye closer to the center, the better. Exactly. And on average, it was only off by about 86 millimeters. So think about it. That's basically like the length of a credit card. Not bad. Not bad at all for a jacket that's also comfy enough to wear
all day long, right? Right. And they found something really interesting when they look specifically at horizontal movements. So, you know, like swinging your arms side to side. Okay. Which is, let's be real, where most the action is. Exactly. For those movements, the predictions were incredibly close to what was really happening. We're talking over 90% accuracy in some cases. Wow. Okay. So we know this jacket can
track your movements super cool, but what can you actually do with that information? Give me the real world applications here. Okay. Well, imagine this, someone who's recovering from an injury and they're using this during physical therapy. So instead of being hooked up to all that bulky equipment, they're just wearing the jacket, right? Way more low profile. Right. And therapists could be tracking
their progress, monitoring their form, even personalizing their therapy in real time, all thanks to the data coming from their clothes. Wow. Talk about a serious wardrobe upgrade. No more clunky sensors or anything. Gone. And it's not just for rehab either. Think about sports training, analyzing a tennis swing or a golf game with this level of detail, but without needing those expensive motion capture setups. It could be huge. Okay. Yeah. I'm seeing the potential here for sure. Athletes going for
gold in their comfy jackets. I love it. And it gets even cooler. Think gaming and VR. Imagine your avatar in the metaverse, but moving exactly like you are in real life, reacting to like your every gesture, all thanks to your jacket, no more clunky controllers or anything. Okay. You had me at comfy jacket, but now you're getting me really hyped about the metaverse. This is like stepping into the future.
And it gets even better because you know what they discovered? Mocha pose was actually more accurate than some of the current methods they're using when it comes to recognizing just like everyday activities. So walking, reaching, bending, just stuff we do all the time. And that opens up a ton of possibilities for how we can use this kind of tech to understand and interact with like the world around us. You know,
this is all like seriously impressive, but got to ask, there's always a, but right. What are the, what are the limitations we're looking at here? You're right. Totally. I mean, no tech is perfect, at least not yet. Right. And I think one of the big ones, the big challenges right now, it's depth perception. Think about it. Like if you try to judge how far away something is,
but you're only using one eye. Yeah. It's hard. Tricky, right. Our brains, they use two eyes to get that, that 3d view, you know, sense. And that's kind of what Mocha poses up against, right? The system as it is now, it's really great at tracking horizontal movement, vertical movement, but for depth, it still has to rely on like video analysis. Yeah. So they're using a single camera to kind of fill in those gaps and it
does, it does a pretty good job, but I think it's fair to say there's room for improvement there. Okay. So maybe it can't quite like replace my eyes and my depth perception just yet, but this is, this is where I start to geek out a little bit about the possibilities, you know, what if the jacket itself could sense depth or what about sensing other things? Like, okay,
what about temperature or even like, I'm just going to throw it out there, emotions. I mean, if my clothes are already reading my movements, why stop there? Now you're thinking like a true innovator. And honestly, I think the researchers behind Mocha pose, they are right there with you. Imagine like clothes that don't just sense, but they actually react to your environment too.
Like adjusting their properties based on the temperature or the humidity even, or, and this is wild, clothes that could actually sense if you're stressed out and then like subtly adjust, you know, to help you calm down. Okay. Clothes that can tell when I need to like take a chill pill. That is some next level self-care that I can definitely get behind. Right. And here's what I think is even
more exciting. Mocha pose, it's open source. That means they've made the design, the code, all of it available for anyone to just take and build upon. Oh, wow. So it's not like one company controls this whole thing. Anyone can just like experiment with it, come up with crazy new applications. Exactly. It's like, it's like giving the entire world the blueprints for this like fashion revolution,
right? Imagine artists, hobbyists, tech whizzes, everyone coming together to create clothes that can sense, that can react, maybe even think along with us. Right. So to wrap things up here, why should, why should any of this matter to like the average person? Because it's not like we're all going to be like motion capture actors. I mean, it's so much bigger than that. I think Mocha pose, it's not just about
capturing movement. It's really about fundamentally changing how we interact with, with technology, with the world around us. Right. It's a glimpse, a peek into a future where tech isn't just something we wear. It's something we kind of inhabit, I guess, something that understands us, that responds to our needs, but on this whole other level. And on that note, I want to hear from you. If your clothes could
sense your every move, react to your environment, what would you want them to do? Let us know. This has been the deep dive. Thanks for exploring with us.