The Frame When a Hand Touches an Object Is Harder Than Recognizing the Object

In egocentric video, a common approach to hand-object interaction is to classify a state: whether the hand has touched an object. This ACCV 2026 paper asks a different question: when does contact begin, and when does it end?

It splits a contact event into two temporal boundaries. touch: the moment the hand first establishes contact; release: the moment contact is released. The task is to pinpoint these two points in a video.

The difficulty is not semantic. The model already knows that the image contains a hand and a cup. The difficulty is the boundary itself. As the hand approaches the object, occlusion, motion blur, and head motion all pile up, and for the one or two frames where contact actually occurs, even human annotators have to look back and forth several times. Contact is a process, not a label that can be read off directly.

The paper is by Huy Anh Nguyen, Chang Dong, Feras Dayoub, and Minh Hoai, and is on arXiv as 2611.13486.

Methodologically, it presents a unified framework that jointly models the hand and the object during interaction. Specifically, on a hand-centered spatiotemporal representation, it overlays object-aware features learned directly from video, then uses a transition-aware event spotting formulation to treat contact transitions as event localization.

One engineering point worth noting: at inference time, it does not call an external object detector, nor does it rely on an off-the-shelf hand-object interaction detector. The whole pipeline is end to end. One fewer external dependency means one less layer of error propagation and one less segment of inference latency. If this is to become something deployable, this kind of trade-off is more practical than chasing benchmark numbers.

To support this task, the authors built the ContactTransition dataset, containing 4,021 egocentric videos, 8,498 touch annotations, and 8,077 release annotations, plus object-of-interest supervision.

In experiments, detection of both types of transitions consistently outperforms two-stage approaches and end-to-end event spotting baselines. The authors additionally verify one more thing: using these detected contact transitions as temporal anchors can help downstream egocentric hand-object reasoning.

The value of this idea is that it makes the temporal granularity one level finer. Knowing that contact exists in a video and knowing which frame contact begins on are completely different things for downstream tasks. Action understanding, robot demonstration learning, and egocentric manipulation assistance all need this starting point.

One caveat: at the scale of 4,000 videos, scene and object coverage is ultimately limited, and the stability of boundary detection in more cluttered everyday environments remains to be seen. This is also a common problem for fine-grained temporal tasks of this kind: establish the definition first, and leave generalization for later.

Paper link: https://arxiv.org/abs/2611.13486