GazeLLM: Multimodal LLMs incorporating Human Visual Attention
Processing high-resolution video with AI requires massive computational resources. GazeLLM offers an elegant solution inspired by human vision: use eye-tracking to focus only on what matters. By cropping first-person video to a small region around the user's gaze point, the system reduces pixel input to just one-tenth while achieving task comprehension equal to or better than full-resolution video. User evaluations across six real-world activities—cooking, bike repair, first aid, and sports—showed that gaze-focused video produces higher quality task descriptions than both full videos and center-cropped alternatives.
Jun Rekimoto. 2025. GazeLLM: Multimodal LLMs incorporating Human Visual Attention. In Proceedings of the Augmented Humans International Conference 2025 (AHs '25). Association for Computing Machinery, New York, NY, USA, 10 pages. https://doi.org/10.1145/3745900.3746075
Transcript 2,273 words
Welcome to the Deep Dive. Today we're examining a pivotal advance in how AI interprets human activity. Specifically, how it understands us through video. And we're going to analyze how mirroring human visual attention can fundamentally overhaul the efficiency and accuracy of multimodal large language models. You'll often hear them called MLLMs. That's right. Our focus is a system called Gazelle LM. You can think of it as a sort of efficiency hack for artificial intelligence.
It's designed to overcome the enormous computational hurdles that have plagued video processing. And it does this by focusing exclusively on what a human user is actually looking at. It's a powerful shortcut. And understanding this mechanism, well, it's key to unlocking the potential of wearable AI for skill transfer and for real world training. And that potential is enormous. MLLMs that integrate vision with, you know, time sequential data represent a major stride toward genuine artificial general intelligence.
But there's a huge barrier. There is. For you and for the AI itself, the challenge is handling the sheer volume of data involved in high resolution, long duration video from a first person perspective. It's just a flood of information. An unmanageable torrent. And the question for the system has always been, how do you sift through that fast enough to provide guidance in the moment?
That is the crux of it. Let's unpack that computational obstacle a bit. These MLLMs, they're built on transformer architectures. And they have a structural problem. As the input sequence gets longer, the memory requirements scale up dramatically, not just linearly. It's an exponential growth. To visualize this, you have to think about how videos even fed to the AI. It uses methods like vision transformers or VIT.
Breaking the image into little patches, turning each one into a data token. Precisely. So if your video resolution doubles, you now have four times the number of patches. If the video duration doubles. You double them again. The math just becomes overwhelmingly complex. It severely limits both the quality and the length of the video an MLM can process in a timely manner. And this is where the human visual system provides the perfect cheat sheet, for lack of a better term.
It does. We are inherently efficient because we simply do not process the entire world around us equally. We focus. We use strategic eye movements to focus selectively, what we call visual attention. The information coming in from your peripheral vision is, well, it's perceived at a much lower resolution. Or it's just used to detect motion. Right. Our brain uses this mechanism to strike a remarkable balance. It gives us the precision needed for complex tasks.
You know, tying a knot or performing surgery while maintaining efficiency. We filter the noise automatically. So the Gazellam concept is simply taking that human biology of selective vision and applying it directly to the AI's input. Instead of forcing the model to process every single pixel from the entire scene. It uses eye tracking data. It decomposes that video and only sends through the high resolution stream that corresponds to where the user's gaze is focused.
It's a profound shift in thinking. Instead of building a bigger, faster computer to handle more data, they just made the data itself smarter by focusing on relevance. And that achievement is probably best measured by the sheer efficiency gain. The Gazellam approach showed task comprehension that was equivalent to, or in some cases even superior to, processing the entire full resolution image. And here's the core data point for you to internalize.
They achieved this while reducing the input data volume by a huge margin. How much? The number of pixels fed into the MLLM was reduced to about one-tenth of the original. One-tenth. So 10.3% of the original, to be exact. 10% of the data yielded equal or better results. Okay, we need to pause on that figure. Are you saying that by deliberately ignoring 90% of the visual information, the resulting AI analysis was judged to be better by humans?
How does reducing the data enhance the outcome? I know it sounds counterintuitive, but it confirms a crucial insight about human learning. When a person is focused on a task, say, mixing ingredients or adjusting a tiny screw, their gaze is inherently directed toward the critical, actionable information. So the 90% that was discarded, that was just background noise. The wall, the clutter on the table.
Exactly. It was irrelevant. By giving the MLLM only that 10% of high-fidelity, high-relevance pixels, they eliminated the noise and forced the model to concentrate on the signal. The input quality skyrockets, even as the quantity plummets. That makes perfect sense. So AI didn't have a processing power problem so much as a filtering problem. And this efficiency has immediate hardware implications. We're not discussing research that needs bespoke server racks.
This can be deployed today. On what kind of devices? The system leverages commercial wearable devices, things already equipped with first-person cameras and eye trackers, such as the Pupil Labs, Nio-On, or Meta's area glasses. The same ones used in the Ego Exa40 dataset they studied. That's right. Because these devices capture the scene and the user's precise visual attention at the same time, Gazella moves straight out of the lab and into the field.
Real-time context-aware assistance becomes entirely feasible. Of course, to ensure these results were sound, the researchers needed robust validation. They did. They used the powerful Gemini 1.5 Pro MLLM for the evaluation. And they selected six diverse, real-world tasks from that Ego Exa4D dataset. It was a broad range of skills. Cooking, so preparing an omelette and sushi. The mechanical skills like bike repair, procedural activities like setting up for a PCR test.
And even training scenarios like CPR, a very wide sample. The first phase of validation relied on human judgment, which is paramount when you're assessing how useful an AI's description actually is. What did they have to do? Participants viewed the full original video and then they were asked to rate descriptions generated from three different AI inputs. The full video, a simple center-cropped video, and of course the gaze-cropped video.
And the outcome. It was definitive. Participants rated the descriptions from the gaze-video condition as receiving higher average scores than the others in five of the six tasks. So the qualitative evidence, just humans judging the usefulness, showed that the minimal gaze-focused data provided superior task descriptions. That's powerful. It is. The insight is that if the AI is trained to describe what the user focuses on, that description is inherently more useful for someone trying to learn that task.
The next step, then, was to confirm that strong human assessment with objective numerical measures. Indeed. They followed up by employing four established quantitative metrics for this kind of evaluation. Let's touch on those briefly, not to define every acronym, but to explain why they were used. They started with statistical measures like BLEU and RUGEL. These are standard tools. They confirm vocabulary and phrase overlap between the AI description and a reference one.
It's a first step to ensure the AI isn't, you know, just making things up. But simple word overlap isn't enough. No. So they added sentence-bert or es-bert. It moves beyond word matching to measure semantic similarity. So does it mean the same thing even if the phrasing is different? Exactly. And finally, they used an LLM-based evaluation with ChatGPT4O to score content coverage on a scale of 0 to 100.
And the consistency across these four methods provides compelling evidence that selectively targeting the human visual focus gives the MLLM the highest quality signal for interpreting these complex sequential actions. It's out. This level of validated efficiency and accuracy opens up immediate practical doors for, well, the learner. Let's look at some of the concrete applications of Gazelle, starting with what they call task verbalization.
This is the system creating the ultimate instructional manual. Imagine an MLLM observing someone prepare an omelet. Instead of a vague voiceover, Gazelle generates a precise step-by-step procedural description. Crack eggs into bowl, whisk for 30 seconds. And pinch of salt. And crucially, with corresponding video timestamps attached to each action. That ability to align focused action with descriptive language is a massive improvement. That clarity is invaluable.
But the application I find most compelling is the stop-and-ask guidance. This is where the real-time assistance potential cruelly shines. Picture a learner wearing the glasses, attempting a complicated recipe. They pause, they're not sure of the next step, and they just ask the system, what should I do now? And the system knows where they are in the process. It knows because it has been tracking their gaze in action.
It can accurately identify their progress by interpreting that small cropped gaze video. If the learner just set down a piece of ginger, the system doesn't scan the whole kitchen. It just looks at that small, high-resolution area of focus. Recognizes the ginger and provides the next contextual action. It could reply, the instructional video indicates the next step is to wash and peel the ginger.
And it could even reference the exact timestamp in the guiding video. That's context-aware coaching that feels personal and immediate. It is. And beyond simple text instructions, the system can enhance learning through procedure visualization. How does that work? When you're watching a complex demonstration, say a lab cask, with multiple things happening at once, Gazellum can analyze the sequence of focused actions and automatically generate a sophisticated data flow diagram.
A sort of high-level map of the procedure. That's a good way to put it. It illustrates branching decisions, tasks you can do in parallel, and it shows dependencies. This helps a learner understand the overall process flow, moving beyond just memorizing steps towards strategic execution. Looking ahead, even though Gazellum is highly effective as it is, the researchers have already mapped out paths for more sophistication.
The goal is always to make the AI's vision even more faithful to how the human eye actually functions. One of the most promising directions there is something they call dual-resolution encoding. Yes. Currently, Gazellum just crops a high-resolution rectangle. A dual-resolution system would use that gaze-centered stream alongside a heavily down-sampled full-view peripheral stream. Which is a powerful enhancement because it mirrors our own vision perfectly.
We see the world in high definition only in that small center, the fovea. But we maintain a low-resolution awareness for context and movement. The AI could use that peripheral stream to know, for instance, that a box of supplies is over there, without needing to dedicate massive computational power to it. And coupled with that, there's the idea of adaptive grids. For the vision transformer patches, yes. Instead of a uniform grid, the system introduces a deformed one.
The patches are denser, with more detail, right near the gaze point. And they get sparser toward the periphery. It achieves the same computational economy as cropping, but it retains that peripheral context more gracefully. Finally, we should acknowledge that gaze is a fantastic metric, but it's not the only relevant data point for skill transfer. Not at all. Future systems will need to incorporate more information.
Such as? Hand position and activity are crucial. If you're learning a physical task, the objects being manipulated by your hands are highly relevant, even if your gaze briefly shifts away. So combining hand tracking with gaze input. It should lead to an even more nuanced and accurate understanding of complex actions, pushing these systems toward truly flawless instruction. So, to summarize what we've learned here about Gazelle.
By integrating the economy of human visual attention, this system manages the data overload of first-person video. The fundamental lesson for you is that providing less data, so long as it's the most relevant, highest quality signal, can generate superior comprehension for MLLMs. It's an efficient methodology. And one that ensures the immediate, practical possibility of advanced skill transfer and contextual assistance through wearable tech.
The system's power stems from this selective, human-centric focus. But here's a final, provocative thought to take with you. Human gaze tends to anticipate the next point of action before the body even moves. How might AI harness that predictive quality in gaze data to offer guidance, or even execute the next step, before you even consciously form the intention to act fundamentally changing the definition of real-time support, from reactive instruction to preemptive assistance?
How might AI Allenya abuse systems are based on net inventive proof of external guidance? How might AI adds much doubt for mere 배?, ve people feel great about spending ground mental flowers in higher density of modernчасs? How might AI remote monitor health safety drive would control thoseMA players in higher pace for mental pocket with facing false IV information? How might AI IT characteristics would docended process or
At least then people use human aid tools that make sure they use dataigh As if all these decisions are related to normal devices in higher complexity of problems and projects are medical sites that mean by losing THened around music that should only videos as being digital teams are digital ett Neg�� within colomm lines of management that could account with greater Del Express Black DV spectators are natural signs that need all members that could drive through science of capital failure is often self-mem arkadaşlar into kept Apple lessons that means it can change your academic tension with effects for breath less consumer patterns that could slow leads from increase Please