TY - JOUR
T1 - Watch to edit
T2 - Video retargeting using gaze
AU - Rachavarapu, Kranthi Kumar
AU - Kumar, Moneish
AU - Gandhi, Vineet
AU - Subramanian, Ramanathan
N1 - Funding Information:
This work was supported in part by Early Career Research Award, ECR/2017/001242, from Science and Engineering Research Board (SERB), Department of Science & Technology, Government of India and IIIT-H seed grant award. Special thanks to Claudia Stavisky, Auxane Dutronc and the cast and crew of ‘Death of a salesman’ and ‘Cat on a hot tin roof’.
Publisher Copyright:
© 2018 The Author(s) and 2018 The Eurographics Association and John Wiley & Sons Ltd.
Copyright:
Copyright 2019 Elsevier B.V., All rights reserved.
PY - 2018/5/22
Y1 - 2018/5/22
N2 - We present a novel approach to optimally retarget videos for varied displays with differing aspect ratios by preserving salient scene content discovered via eye tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. Our approach is (a) content agnostic as the same methodology is employed to re-edit a wide-angle video recording or a close-up movie sequence captured with a static or moving camera, and (b) independent of video length and can in principle re-edit an entire movie in one shot. Our algorithm consists of two steps. The first step employs gaze transition cues to detect time stamps where new cuts are to be introduced in the original video via dynamic programming. A subsequent step optimizes the cropping window path (to create pan and zoom effects), while accounting for the original and new cuts. The cropping window path is designed to include maximum gaze information, and is composed of piecewise constant, linear and parabolic segments. It is obtained via L(1) regularized convex optimization which ensures a smooth viewing experience. We test our approach on a wide variety of videos and demonstrate significant improvement over the state-of-the-art, both in terms of computational complexity and qualitative aspects. A study performed with 16 users confirms that our approach results in a superior viewing experience as compared to gaze driven re-editing [JSSH15] and letterboxing methods, especially for wide-angle static camera recordings.
AB - We present a novel approach to optimally retarget videos for varied displays with differing aspect ratios by preserving salient scene content discovered via eye tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. Our approach is (a) content agnostic as the same methodology is employed to re-edit a wide-angle video recording or a close-up movie sequence captured with a static or moving camera, and (b) independent of video length and can in principle re-edit an entire movie in one shot. Our algorithm consists of two steps. The first step employs gaze transition cues to detect time stamps where new cuts are to be introduced in the original video via dynamic programming. A subsequent step optimizes the cropping window path (to create pan and zoom effects), while accounting for the original and new cuts. The cropping window path is designed to include maximum gaze information, and is composed of piecewise constant, linear and parabolic segments. It is obtained via L(1) regularized convex optimization which ensures a smooth viewing experience. We test our approach on a wide variety of videos and demonstrate significant improvement over the state-of-the-art, both in terms of computational complexity and qualitative aspects. A study performed with 16 users confirms that our approach results in a superior viewing experience as compared to gaze driven re-editing [JSSH15] and letterboxing methods, especially for wide-angle static camera recordings.
KW - Computing methodologies → Scene understanding
KW - Convex optimization
KW - Image-based rendering
KW - Theory of computation → Dynamic programming
UR - http://www.scopus.com/inward/record.url?scp=85051538536&partnerID=8YFLogxK
U2 - 10.1111/cgf.13354
DO - 10.1111/cgf.13354
M3 - Article
AN - SCOPUS:85051538536
SN - 0167-7055
VL - 37
SP - 205
EP - 215
JO - Computer Graphics Forum
JF - Computer Graphics Forum
IS - 2
ER -