Adaptive Feature Abstraction for Translating Video to Text

Publication Date: 2/2/2018

Event: The Thirty-Second AAAI Conference on Artificial Intelligence

Reference: pp 1-13, 2018

Authors: Yunchen Pu, Duke University; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video features. However, the variable context-dependent semantics in the video may make it more appropriate to adaptively select features from the multiple CNN layers. We propose a new approach to generating adaptive spatiotemporal representations of videos for the captioning task. A novel attention mechanism is developed, which adaptively and sequentially focuses on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on three benchmark datasets: YouTube2Text, M-VAD and MSR-VTT. Along with visualizing the results and how the model works, these experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.

Publication Link: https://dl.acm.org/doi/10.5555/3504035.3504927