Title:
Learning to Recognize Daily Actions using Gaze
Learning to Recognize Daily Actions using Gaze
Files
Author(s)
Fathi, Alireza
Li, Yin
Rehg, James M.
Li, Yin
Rehg, James M.
Advisor(s)
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
We present a probabilistic generative model for simultaneously recognizing daily actions and predicting gaze locations in videos
recorded from an egocentric camera. We focus on activities requiring
eye-hand coordination and model the spatio-temporal relationship between the gaze point, the scene objects, and the action label. Our model
captures the fact that the distribution of both visual features and object occurrences in the vicinity of the gaze point is correlated with the
verb-object pair describing the action. It explicitly incorporates known
properties of gaze behavior from the psychology literature, such as the
temporal delay between fixation and manipulation events. We present
an inference method that can predict the best sequence of gaze locations
and the associated action label from an input sequence of images. We
demonstrate improvements in action recognition rates and gaze prediction accuracy relative to state-of-the-art methods, on two new datasets
that contain egocentric videos of daily activities and gaze.
Sponsor
Date Issued
2012-10
Extent
Resource Type
Text
Resource Subtype
Article
Proceedings
Proceedings