The Video Mesh: A Data Structure for Image-based Three-dimensional Video Editing
Author(s)
Chen, Jiawen; Paris, Sylvain; Wang, Jue; Matusik, Wojciech; Cohen, Michael; Durand, Fredo; ... Show more Show less
DownloadDurand-The video mesh.pdf (9.585Mb)
OPEN_ACCESS_POLICY
Open Access Policy
Creative Commons Attribution-Noncommercial-Share Alike
Terms of use
Metadata
Show full item recordAbstract
This paper introduces the video mesh, a data structure for representing video as 2.5D “paper cutouts.” The video mesh allows interactive editing of moving objects and modeling of depth, which enables 3D effects and post-exposure camera control. The video mesh sparsely encodes optical flow as well as depth, and handles occlusion using local layering and alpha mattes. Motion is described by a sparse set of points tracked over time. Each point also stores a depth value. The video mesh is a triangulation over this point set and per-pixel information is obtained by interpolation. The user rotoscopes occluding contours and we introduce an algorithm to cut the video mesh along them. Object boundaries are refined with per-pixel alpha values. The video mesh is at its core a set of texture mapped triangles, we leverage graphics hardware to enable interactive editing and rendering of a variety of effects. We demonstrate the effectiveness of our representation with special effects such as 3D viewpoint changes, object insertion, depth-of-field manipulation, and 2D to 3D video conversion.
Date issued
2011-04Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer ScienceJournal
2011 IEEE International Conference on Computational Photography (ICCP)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Jiawen Chen et al. “The Video Mesh: A Data Structure for Image-based Three-dimensional Video Editing.” IEEE, 2011. 1–8. © Copyright 2011 IEEE
Version: Author's final manuscript
ISBN
78-1-61284-707-8