Filtering on the Go: Effect of Filters on Gaze Pointing Accuracy During Physical Locomotion in Extended Reality

0:00Welcome to the Deep Dive. Today we are examining how to make eye tracking more accurate, especially when people are moving around. You provided a research paper on gaze filtering in extended reality, XR for short. We'll be diving into why this is such a challenge and what researchers are finding out about filtering gaze on the go. Yes, and as XR is becoming more prevalent, it's more

0:21likely that users will be moving around. So ensuring accurate gaze tracking in these dynamic environments is crucial. So let's unpack why gaze tracking in XR is more challenging than, say, on a desktop computer. Well, on a desktop, gaze is just a 2D point on the screen. But in XR, gaze is a 3D vector with both direction and origin. You can think of it as a laser pointer. Then on top

0:42of that, you have head movement. And I imagine spatial reference frames, SRFs, play a role in this complexity. Exactly. SRFs define the location of a target, something like a button or some virtual object relative to the user. Whether an object is anchored to the world or attached to the user's head, it occupies a specific SRF. And this can really impact how our eyes naturally track it,

1:02impacting accuracy. So even if a target appears stationary in my XR world, my eyes might be making adjustments to stay focused on it. That's right. And this is where filters come into play. Okay, so tell me more about why these filters are so important. All eye trackers, even the most advanced ones, have some inherent noise. This noise affects both accuracy and precision. Accuracy is how close the measured gaze is to the actual target. Precision is

1:29how stable that gaze point is. Online filters work in real time, and they smooth out this data to create a more stable and reliable gaze signal. So they're kind of cleaning up our eye movements. That's a good way to put it. The challenge is getting the balance right. Too much filtering, and you might get a lag in gaze response. Or you could miss those fast eye movements called saccades. Too little filtering,

1:52and well, then the noise makes it difficult to interact smoothly. Now, the researchers use the term target size. Can you explain what that is and why it matters? Target size is simply how large a target has to be for the system to reliably pick up the user's gaze. Bigger target size means less accuracy. I've certainly had my own experiences with imprecise gaze controls, constantly having to

2:13readjust my gaze to select something. Sounds like target size is a big part of that. Yes, it is. And this is where this research is particularly relevant. They found that something as simple as walking or jogging could make the target size noticeably worse in some cases. I suppose it makes sense, since our bodies naturally try to keep our vision stable as we move,

2:34and those movements can interfere with trying to point precisely with our eyes. That's it. It's an important observation, showing that filters designed for sitting still might not be enough. The research actually looks at five different filters, each with pros and cons. Let's discuss them. Starting with the weighted average, or WA filter. The WA filter is pretty simple, conceptually. It averages a number of past gaze points and gives more

2:57weight to the more recent ones. It's easy to implement, but it's limited by how big its averaging window is. So a good starting point, but maybe not the most sophisticated solution. What about this WASD filter? So the WASD filter, or weighted average with saccade detection, takes that WA filter and adds something important. It actually restarts the averaging process after each saccade. This prevents the filter from being influenced by the gaze data from before the eye jumped to a new spot.

3:25So it's like telling the filter, hey, ignore what happened before my eyes jumped to this new spot. What about this LPSDN filter? LPSDN stands for low-pass filter with N-backed saccade detection. It uses something called a low-pass filter, which smooths the data based on a cutoff frequency. This allows for more smoothing than just using a WA filter. But then it adds a saccade detection mechanism that looks further back in time and frames

3:50back, to be precise. That improved detection makes it better at handling the noise that comes from movement. So LPSDN is like the heavy-duty filter with a fancy detection system. How is that different from the LPWSD? The LPWSD, that's low-pass filter with window-based saccade detection, well, it also uses a low-pass filter. But the key difference is how it detects saccades. Instead of just looking at a single-passed gaze point,

4:14it looks at a window of them and averages them to figure out if there was likely a saccade. This can be really good at dealing with noise from things like head movements, but it can introduce a small delay in how quickly it responds. Ah, so it's that trade-off between noise reduction and responsiveness again. Ah. And lastly, we have the 1-year-old filter. What sets this one apart?

4:33The 1-year-old filter is adaptive, which means it changes how much smoothing it does based on how fast the input signal is changing. In this case, that means how fast your gaze is moving. It doesn't have an explicit saccade detection mechanism like the others, but because it's adaptive, it can respond dynamically to changes in how your gaze is behaving. So instead of having set rules, it's constantly adjusting on the fly.

4:54Exactly. And one thing that's important to point out, which the research emphasizes, is that there's no single best filter. Which filter works best depends on a few things. Well, you mentioned movement pace earlier. Right. As you move faster, you introduce more noise into the eye-pracking data. This means filtering becomes even more important. Imagine trying to keep your gaze fixed on a target

5:16while you're jogging versus standing still. Yeah. It sounds difficult. Do the SRFs we talked about earlier affect filter performance too? Definitely. Remember, SRFs define how a target is positioned relative to the user. Some SRFs are more prone to causing eye movements that mess with accurate pointing. For example, if a target is fixed to your head and you move your head, your eyes will move to stay

5:40focused on the target. These movements, though natural, add noise. So a filter that works well for a target anchored in the world might not be as effective for a target that's attached to my head. Exactly. Distance to the target also matters. The farther away a target is, the more impact even small head movements have. Kind of like pointing a laser pointer at something far away. Even a little shake in your hand can make

6:02the dot move all over. You know, the research also mentioned an unexpected finding related to coordinate systems. The researchers found that filtering works best when it's done in the same coordinate system as the target. Why is that unexpected? Well, a lot of the time, eye tracking data is filtered at the source, meaning in the eye tracker's own coordinate system. This research suggests that to get the best performance, you actually want to

6:27filter the data in the coordinate system of the target. So this means that for people developing XR experiences, relying on the eye tracker's built-in filtering might not be the most effective strategy. Instead, they might need to implement filtering in their own applications. That's right. By filtering the data in the target's coordinate system, you can create more accurate and responsive interactions. But we have to consider that a complex XR experience could have targets located in different SRFs, like we

6:55discussed before. You're right. The research suggests that it might be beneficial to dynamically switch filtering systems depending on what the user is focused on at that moment. For example, if I'm looking at a menu that's attached to my hand, the system would filter the gaze data using the coordinate system of my hand. Exactly. And if you shift your gaze to a world-anchored object, the system would switch to filtering in the

7:19world coordinate system. This adaptability makes sure the filtering process is always optimized for whatever target you're interacting with. So it's all about context-aware filtering. The system is constantly analyzing the situation and adjusting its approach to give you the most accurate and responsive gaze experience. That's a good way to sum it up. Now, in their tests, the researchers found that generally filters that included saccade detection worked best.

7:42Especially those that were good at dealing with that motion-induced noise, right? Right. The LPWSD and LPSDN filters, specifically with higher N values, were really good at balancing, smoothing out the data with being responsive to those quick eye movements. We've covered a lot of ground here. It's clear that gaze filtering in XR is, well, it's complex, but it's crucial if we want to create XR experiences that are intuitive and user-friendly.

8:09It really is. And there's more to uncover. We should probably dive deeper into the whole idea of the saccade threshold itself. The saccade threshold, as I understand it, is that sensitivity setting that controls the saccade detection. Exactly. And getting that setting right is important. Let's explore that further in part two of our deep dive. Welcome back. We're going to dig a bit deeper into these saccade thresholds,

8:33since they're so crucial for optimizing gaze filtering in XR. And as we discussed, getting that threshold right is this balancing act. It's all about how sensitive the filter is to those rapid eye movements. Precisely. If the threshold is set too low, the filter might mistake little wobbles in the gaze data for actual saccades. And that leads to unnecessary resets of the smoothing process. Essentially, you end up with overcorrection,

8:58which makes for a less smooth gaze experience. What about the opposite problem? What happens if you set the threshold too high? Well, if it's too high, then the filter might miss real saccades altogether. And the result is that the filtered gaze lags behind the user's actual eye movements. That can really affect how responsive the system feels. It sounds like the research offers some specific recommendations on how to strike this balance.

9:19It does. And it's based on the expected movement pace of the user. Which makes sense, because more vigorous movement means more noise in the gaze data. Can you walk me through those recommendations? Sure. For a one-frame look-back saccade detection mechanism, where the filter is comparing the current gaze point to the one right before it, the study suggests keeping the threshold between the 75th

9:42percentile of angular distances during fixations and the 25th percentile during actual saccades. So essentially finding that upper limit of those normal fixation jitters and the lower limit of saccade magnitudes. Yeah. Trying to filter out the noise without suppressing those genuine eye movements. Exactly. The goal is to find that sweet spot where you're effectively reducing noise without missing out on those real eye movements. Now, for a three-frame look-back, where the current gaze point is compared

10:10to the one three-frames earlier, the study recommends a slightly different approach. They suggest setting the threshold between the 95th percentile for fixations and the 5th percentile for saccades. So a higher threshold for the three-frame look-back. That's right. The idea is that by looking at a larger window of data, the filter becomes less sensitive to those momentary fluctuations. That way, it can better distinguish between noise and actual eye movements.

10:36This reminds me of those scenarios you mentioned earlier, walking versus jogging. Right. Let's look at those cases. If we're assuming the user is mostly going to be walking, the study suggests a saccade threshold no higher than 2.45 degrees for that three-frame look-back. And what about jogging? For jogging, they recommend dropping the threshold to 2.05 degrees. Since there's more noise involved with jogging, you need a slightly lower threshold to make sure you don't

11:02miss those rapid eye movements. The study emphasizes that using the higher end of these recommended ranges typically results in better accuracy. Any particular reason why that is? It comes back to the whole issue of noise. A higher threshold, within reason, of course, means the filter is less likely to react to those spurious jitters in the data, especially during movement. That means fewer restarts of the smoothing

11:25process, which lets the filter work more effectively. So the goal isn't just to detect saccades, but to also be smart about filtering out anything that might look like a saccade, but is really just noise. That's it. And the study even suggests that it might be possible to adjust the saccade threshold dynamically based on real-time measurements of how fast the user is moving. So instead of having a fixed

11:47threshold, the system could adapt to the user's current activity and fine-tune that sensitivity automatically. Yeah, it's an interesting area for future research, and it could lead to even more responsive and accurate gaze interactions in dynamic XR. So we've covered those five filters, and we've dug into the saccade threshold, even touched on the idea of dynamic adaptation. What other advice does this research offer for people who are building XR experiences?

12:12The study strongly recommends turning off any built-in filtering that the eye tracker itself provides. That seems a bit counterintuitive. Why would we want to turn off something that's supposed to help with accuracy? Well, remember our conversation earlier about that unexpected finding with the coordinate systems? Yeah, we found out that filtering works best when it's done in the same coordinate system as the target. Right. The eye tracker's built-in filtering is usually done in its own coordinate

12:36system, which is often different from the coordinate system of the virtual objects the user is interacting with. So it can actually work against you. So instead of just relying on the eye tracker's built-in filtering, developers should take control and implement their own filtering within their applications. That's a key takeaway. By filtering in the target's coordinate system, we can achieve more accurate and responsive interactions. But we also need to think about the fact that a complex XR experience

13:03could have targets located in different SRFs, right? Those SRFs we talked about earlier. You're absolutely right. That's where the research suggests that it might be beneficial to dynamically switch filtering systems based on what the user is looking at. So if I'm looking at a menu that's attached to my hand, the system would filter the gaze data using the coordinate system of my hand. Precisely. And if you then

13:24look at an object that's anchored to the world, the system would seamlessly switch over to filtering in the world coordinate system. This kind of adaptability ensures that the filtering is always optimized for the specific target you're interacting with. This dynamic approach to filtering, where the system is constantly analyzing and adjusting based on context, that seems to be a big takeaway here. It is. And while

13:48this research is focused specifically on gaze filtering in XR, I think it has broader implications. You're saying that these findings might be relevant to other technologies that use gaze tracking? Exactly. As gaze tracking is used in more and more fields, the need for effective filtering techniques is going to grow. You mentioned assistive technologies as an example. Think about someone who has motor impairments and uses

14:10their gaze to control a wheelchair or interact with a computer. In those cases, having precise and responsive gaze control isn't just a matter of convenience. It's essential for their independence and quality of life. And even small head movements could really throw off the accuracy in those situations. These filters we've talked about, especially those that incorporate saccade detection, could be really important to making sure

14:35those experiences are smooth and reliable. Exactly. And it goes beyond assistive technologies. Gaze tracking can be used to create more inclusive interfaces for people with all sorts of disabilities. Like someone with limited dexterity in their hands could use their gaze to browse the web or control their smart home devices. Exactly. And having effective filtering can make those interactions more intuitive and less

14:58tiring so they can use technology in a way that works best for them. And as we discussed earlier, gaze tracking is starting to show up in more mainstream consumer devices. We're already seeing laptops and smartphones with gaze tracking built in, and they can be used for things like monitoring attention or scrolling without using your hands. As the technology improves, I think we'll see even more innovative

15:19applications. Imagine using your gaze to control your TV, play video games, or interact with virtual assistants. The possibilities are really exciting. But to make gaze tracking truly successful in all these different areas, we need to make sure the underlying technology is robust, accurate, and responsive. And that's where this research on gaze filtering is so important. It gives developers a roadmap, regardless of what kind of

15:43application they're building. It reminds us that we need to consider the whole picture, how the user is moving, what the interface is like, and how sensitive those filtering algorithms are. It's about moving away from this idea that the user is always sitting still, and designing for a world where technology adapts to our gaze, no matter where we are or what we're doing. That's a vision worth striving for. A world where gaze interaction becomes intuitive and

16:08empowering, where it breaks down barriers and opens up new possibilities for how we interact with technology. Welcome back to our final segment on gaze filtering. We've delved into some pretty technical details, but let's take a step back and think about the bigger picture here. What are the wider implications of this research? As gaze tracking becomes more common in different areas, the need for those effective filtering techniques is only going to grow. We started by talking about how

16:34crucial gaze filtering is in XR, especially when users are on the move. But you mentioned that these findings could be applicable to a wider range of technologies as well. Yes, exactly. As gaze tracking gains traction across diverse fields, the demand for robust filtering methods becomes even more critical. You brought up assistive technologies earlier as an example. Imagine individuals with motor impairments using their gaze to control a wheelchair or interact with a computer. In those situations,

17:03precise and responsive gaze control is absolutely crucial for their independence and quality of life. It goes beyond just convenience. Even slight head movements could significantly impact the accuracy in those situations. Those filters we discussed, particularly the ones with SACA detection, could really make a difference in ensuring a smooth and reliable experience. Yes, precisely. And it extends beyond assistive technologies. Gaze tracking has the potential to create more

17:29inclusive interfaces for people with a variety of disabilities. For someone with limited dexterity in their hands, they could use their gaze to navigate a website or control smart home devices. Exactly. Effective filtering can make those interactions more intuitive and less tiring, allowing users to interact with technology in a way that suits them best. And as we touched on earlier, gaze tracking is starting to appear in more mainstream consumer devices as well. We're already seeing laptops and smartphones with

17:56gaze tracking capabilities used for things like attention monitoring or scrolling hands free. As this technology continues to advance, I anticipate even more innovative applications. Imagine a future where you can control your television, play games or even interact with a lot of the virtual assistants using just your gaze. The possibilities are really quite vast. But to fully realize the potential of gaze tracking across all these different domains, we have to ensure that the underlying technology is robust, accurate and

18:27responsive. And that's where this research on gaze filtering is so valuable. It provides a roadmap for developers no matter what specific application they're working on. It serves as a reminder that we need to consider the user's entire context, their movement patterns, the characteristics of the interface, and the sensitivity of those filtering algorithms. It's about moving beyond this assumption that the user is always stationary. And designing for a world where technology seamlessly

18:53adapts to our gaze, no matter where we are, or what we're doing. That's definitely a vision worth pursuing a world where gaze interaction is intuitive and empowering, where it breaks down barriers and opens up new possibilities for how we interact with technology. This exploration of gaze filtering has been really enlightening. It's amazing how a seemingly niche area of research can have such broad implications for the future of technology.

19:17I agree. As we've seen, it goes beyond just technical details. It's about understanding the intricacies of human eye movements, how noise and movement affect accuracy, and ultimately how we can use that knowledge to design more human-centered technologies. It was a pleasure. As we wrap up this deep dive, I'd like to leave our listeners with a question to ponder. We've talked about how technology can adapt to our gaze, but how might our understanding of gaze filtering inform the design of

19:44physical spaces and interactions in the real world? That's an interesting thought. Could we apply these principles to improve things like signage, wayfinding, or even the layout of environments to create experiences that are more intuitive and engaging? It's a challenge worth exploring, and it highlights the interconnectedness between the digital and physical realms. What we learn about gaze in one area can inform and enrich the other. Until our

20:09next deep dive, keep exploring, keep questioning, and keep delving deeper into the world of knowledge. Thanks for joining us on this enlightening journey. We hope our listeners will come away with a greater appreciation for the complexities of gaze tracking and the importance of effective filtering techniques.