A Conversation with Thad Starner on Mobile Sign Language Recognition
The Oscar best picture winning movie CODA has helped introduce Deaf culture to many in the hearing community. The capital "D" in Deaf is used when referring to the Deaf culture, whereas small "d" deaf refers to the medical condition. In the Deaf community, sign language is used to communicate, and sign has a rich history in film, the arts, and education. Learning about the Deaf culture in the United States and the importance of American Sign Language in that culture has been key to choosing projects that are useful and usable for the Deaf.
Transcript 2,588 words
Hello, I'm here with Thad Sterner. Hi, hello. Hi, how have you been? Yeah, so far so good. We are meeting at UbiComp and just in the conference marathon, I heard you, or I saw you giving the keynote at MobileHCI. Yes, I've had the MobileHCI and two workshops too. So I've been busy.
Yeah, so what was the topic of the MobileHCI? Well, I was looking at how to make mobile sign language recognition useful for the deaf community. So a lot of folks have been working on sign language recognition over the years, but they don't really have a good use case for it.
You know, these days, at least in the United States, if you're deaf, you can call up an interpreter on your smartphone, and they will interpret for you right then and there as a free service.
So, you know, and, you know, mobile sign language recognition is going to take a long time before it's good enough to actually do anything useful in that domain. So the question is, what are some good use cases for mobile sign language recognition? And I've been working on the space since, you know, 1994.
And back then we showed the first continuous sign language recognition system, a phrase level system. But it was dedicated to just me in a very controlled situation, wearing black clothes with a very specific camera setting. It was, you know, it showed that it could be done, but it was way too specific to be useful. Yeah, but it should be possible right now, no?
I mean, if you started it in 1994 during your master's. Well, the thing is that we don't really have much training data. For speech recognition, we now have millions and millions of hours of speech. But for sign recognition, we only have a couple thousand hours at most. And those are data sets that, you know, I've been recently releasing at Georgia Tech and at Google.
So before then it was like a couple hundred hours most.
And so given this resource-poor data set problem, you know, what sort of problems can you tackle? And for me it's been focusing on making smartphone games to help the hearing parents of deaf children learn sign language. Now it turns out that this is a crucial problem for the deaf community. Ninety-five percent of deaf kids are born to hearing parents, and most of those hearing parents never learn enough sign language to teach it to their children.
One of the good uses of sign language recognition that I've discovered working with the deaf community is trying to help hearing parents of deaf kids learn sign language. It turns out 95 percent of deaf kids are born to hearing parents, and most of those hearing parents never learn enough sign language to teach it to their children. And it's really critical for these deaf kids to learn sign language for the development, both socially and mentally.
What we see is that, you know, kids who are surrounded in sign, they'll grow up normally. Deaf kids are deaf parents. They grow up the same way as hearing kids of hearing parents. But not having access to English because you can't hear, or sign language because your parents aren't signing to you, because these kids do not, you know, get access to language at the same pace as hearing kids of hearing parents.
And that's really crucial for developing things like short-term memory. And so what we're trying to do now is bring sign language games to the hearing parents. And these parents really want to learn sign language. But for English speakers, American Sign Language is as hard to learn as Japanese. And at least in the state of Georgia where I'm in, it takes about an hour for every sign language class.
And so these parents want to learn, but they get overwhelmed with way too much time and effort to sink into it. So what we're doing is making games in sign language to help the parents learn. And we're using the sign language recognition on the smartphones to make those games. Our current game called PopSign AI. It's one of these bubble bursting games where you do three of a kind.
It's like you have an array of colored balloons at the top of the screen. And you have to shoot the right colored balloon to get three of a kind and burst them when you're trying to clear the screen.
And in the sign language recognition version of it, you aim at, like, you know, say the yellow balloons. And then you make the sign for yellow. And if you do the sign correctly, it'll shoot the yellow balloon and burst all the ones it touches.
Red for the red balloon, et cetera. And we do more vocabulary words. Like we'll do owl or elephant or dog or cat. And same sort of thing. It's just the different colored balloons with different labels on them. And you're trying to sign to match it. So do I get that right? So the person is holding the phone and then signs with one hand?
Yes. So you hold the phone with one hand and aim with your thumb of that first hand. And then you sign with your dominant hand. And when you let go of the touch screen, it then sends the video to the recognizer. It does the recognition. It figures out what color bubble you're trying to shoot. And then it shoots it. So what we've been showing is that we made a test game called Pops on AI Preview that's out there now.
You can get it on the App Store or the Play Store. And it works very well. People love it. And now expanding it to full 563 concepts. You teach any child in any language. We're doing it in American Sign Language first. But what we're hoping is that we can provide a toolkit called the Sign Language Recognition and Gesture Toolkit, which has the app for Android where you can actually collect signs from people who are deaf.
Collect enough examples of it. Then we have an annotation tool. You can sit there and make sure everybody signs the right thing. And then we have a simple LSTM model, which is good enough to do this type of recognition. And then we have all the game structure. And so what we're hoping is that people can take our American Sign Language game and translate it to Japanese Sign Language or Indian Sign Language or Arabic Sign Language or take your pick.
Okay. How many concepts do you need for that? So about 563 is what's called the MacArthur Bates Index. And those are the first concepts that you teach any child. We have released the video for 250 of them so far. And those are about, I think we have 50 signers doing 30 examples of each.
Each of those 250 signs. And now we're adding the rest of the 563 right now. And that will be up on sign-data.cc.architect.edu shortly. How much would you need to be fluent in a language, in a sign language? Well, sign languages often have a lot fewer words than spoken languages. That's because each sign has a lot of inflections. So, for example, if I put my thumbs up with my dominant hand and my hand underneath it, my left hand underneath it,
and I just pull it there, that's just a sign for help. But if I move it from me to you, this is me helping you. If I move it from you to me, it's you helping me. Oh, okay. So you need also the special information. Correct. And so it's all those inflections, all that movement that gets you all the speed and diversity of concept that you do in spoken language.
Now, what we found, though, for deaf kids, all you really need to get, before they get into, like, elementary school, is about 150 signs of survival signs. But what we're finding is that these parents are happy to learn more signs. They'll go to the, they'll keep on playing the game until they run out of vocabulary. And then we'll also find that these parents will sit there and practice, you know, go to bed or whatever they want to sign to little Johnny.
They'll sit there and practice until they're good enough, and then they'll go talk to their deaf child.
And so it's really, we really encourage that if we make it fun and convenient, that these parents will actually pick this up. And, you know, it will really make a huge difference in the development of these deaf children. Yeah, and also porting it to other languages. Yeah, and there's 150 different sign languages, that's a whole lot more than that. But there's about 150 sign languages that are sort of in the same type of sign language, that American sign language,
and French sign language, and British sign languages, and then the Asian sign languages. So that's what we're targeting first. And, you know, the hope is that we gather enough examples from all these different sign languages, it'll create a good embedding set, and then you just fine-tune it to a new language. And so, you know, while it may take us 50 to 100 signers to get American sign language,
by the time we take it to, say, you know, French sign language or German sign language, which is very similar to American sign language, you only may need, like, you know, 10 or 20 signers in order to get something that's good enough to make the new game. Just to understand you right, so you're talking about a large language model for signing? Well, it's not. It's isolated, right?
So it's individual signs. So it's, you know, what we know is that the human hand and body only has so many degrees of freedom. And so you can imagine that a deep learning system can have the, get the, like, embeddings from one or a couple of those sign languages, and then generalize to more sign languages, is the hope.
Yeah, it could work. Yeah, and I also have, you know, I have him or him or him or him or him or him, or him, or him, or him, or him, or him, or him, or him, or him, or him. Yeah. But it turns out that with enough data, the deep learning systems are getting some traction. And for this game, where we only have about five different color balloons in the screen at once,
so you've got to differentiate between five signs. So that's relatively easy to do with an LSTM, like a straight out-of-the-box LSTM. Yeah. So, yeah, and that's kind of the first practical use of the sign language recognition stuff. And then you start looking at things like dictionaries or being able to command the virtual assistants and that sort of stuff, or finger-spelling to do text entry on a phone.
For the deaf, they finger-spell whenever it's a proper noun, like introducing somebody's name or doing an address or URL or phone number or something like that. And so being able to do fast text entry with finger-spelling is useful, too.
Cool work. Thank you. Thanks. Go play with it, man. Yeah, definitely. Go get into it. Definitely. I already have PopSign on my smartphone as well. Oh, excellent. I started with it. The AI one, the sign language recognition one, or the original one that shows you the videos? So right now, just the original one. The original one? Yeah, I couldn't find it on iOS.
Is it available? It should be. We're recompiling it right now because there was a change in operating system. And so if you don't want to recompile for the new operating system, they'll drop you off. Oh, yeah. So we're trying to set up a monthly build for each major platform. Yeah, because I have it on Android, but I don't play so much with that.
But we're keeping two things up. The old PopSign game has all 563 signs in it already. It just shows you the sign. Sign, yeah. And then you fire at the right balloons. Now, the PopSign AI preview, that only has 35 signs in it right now. That's the one with a recognizer in it. That is, as far as I know, it's the first practical use of sign language recognition that's out there.
And it really does work.
Yeah, impressive. Can't wait for more updates on it. Yeah. More fun games hopefully coming out. The SLR GTK system we're putting together, the tool we're putting together, will hopefully help a lot of people make new games as well. So it's designed to make it easy. And hopefully we'll get all these game-making classes at universities involved. Because you can make some random throwaway game, or you can make a game that actually might help change somebody's life.
Yeah. And you might have fun playing a game and also learning a little design on the side. Yeah. Sound like it's great, man. You can talk at a distance. No, and it also just helps you staying active. I mean, learning a new language is always good. Well, and one of the really cool things is that it turns out that learning, teaching a hearing infant sign language allows them to communicate with you six months before they can speak.
It turns out the larynx is actually in the chest when you're born. It takes about 12 months to develop to the stage where you can actually go past just, you know, cooing sounds and be able to speak in the language normally. But a six-month-old can sign to you just fine. And so that makes it much easier, you know, instead of crying, they can actually, you know, ask for more food.
Or they can ask for milk or tell you they just pooped. Right? And what's really neat about this is they're actually, you know, learning how to sign. These kids actually start getting, acquiring short-term memory skills. And language skills. Yeah. That you wouldn't have otherwise. And so you can actually test these kids and they have an IQ boost up to, like, age five just from learning language soon.
So, in my opinion, you know, everybody should learn science and infant because it's pretty clear it actually helps you acquire linguistic skills in whatever language you end up speaking. Yeah. It's just another language. It's a language you can learn earlier that you can speak. And that forms a short-term memory earlier. It means you can learn everything faster. That's the hypothesis.
Given the series of experiments I've seen, that's the operating hypothesis about why these kids who are taught sign language get such a linguistic boost and such a maturity boost over the kids who never learn sign language. So, yeah. Yeah. So, thanks a lot. Sure. For joining. And, yeah. Looking forward to playing a little bit more and also learning American Sign Language and maybe also Japanese.
Catch you later. Catch you later. Good luck. Thanks. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.