Leiden University Scholarly Publications

Wang, X.; Du, Y.; Verberne, S.; Verbeek, F.J. (2022)

Improving weakly supervised phrase grounding via visual representation contextualization with contrastive learning

Article / Letter to editor

Weakly supervised phrase grounding aims to map the phrases in an image caption to the objects appearing in the image under the supervision of image-caption correspondence. We observe that the current studies are insufficient to model the complicated interactions between the visual components (i.e., the visual regions) and between the visual and textual components (i.e., the phrases). Therefore, this paper presents a novel weakly supervised learning approach to phrase grounding in which we systematically model the visual contextualized representation with three modules: (1) object proposals pooling (OPP), (2) visual self-attention (VSA) and (3) visual-textual cross-modal attention (VTCA). OPP alleviates the suppression of the object proposals and benefits the visual representation in terms of trading off the richness of the visual components and the computational efficiency. VSA aims to capture the correlation among the object proposals and generate a representation of each proposal...Show moreWeakly supervised phrase grounding aims to map the phrases in an image caption to the objects appearing in the image under the supervision of image-caption correspondence. We observe that the current studies are insufficient to model the complicated interactions between the visual components (i.e., the visual regions) and between the visual and textual components (i.e., the phrases). Therefore, this paper presents a novel weakly supervised learning approach to phrase grounding in which we systematically model the visual contextualized representation with three modules: (1) object proposals pooling (OPP), (2) visual self-attention (VSA) and (3) visual-textual cross-modal attention (VTCA). OPP alleviates the suppression of the object proposals and benefits the visual representation in terms of trading off the richness of the visual components and the computational efficiency. VSA aims to capture the correlation among the object proposals and generate a representation of each proposal by incorporating the visual information of the others. To measure the cross-modal compatibility in terms of topics, we introduce the VTCA module to represent the visual topic corresponding to each textual component in a cross-modal common vector space. In the training process, we build a mixed contrastive loss function by considering both the cross-modal compatibility and the differences in the visual representations in the VSA module. Compared with the state-of-the-art methods, the proposed approach improves the performance by 3.88% points and 1.24% points on R@1, and by 2.23% points and 0.26% points on Pt_Acc, when trained on the MS COCO and Flickr30K Entities training sets, respectively. We have made our code available for follow-up research.Show less

All authors

Wang, X.; Du, Y.; Verberne, S.; Verbeek, F.J.

Editor(s)

Meng, X.; Wang, F.; Lu, C.; Huang Y.; Shekhar, S.; Xie, X.

Date

2022-11-02

Journal

Applied Intelligence; SIGSPATIAL '21: Proceedings of the 29th International Conference on Advances in Geographic Information Systems

Volume

53

Issue

11

Pages

14690 - 14702

ISBN (print)

978-1-4503-8664-7

DOI

doi:10.1007/s10489-022-04259-9

Conference

Conference

SIGSPATIAL '21: 29th International Conference on Advances in Geographic Information Systems

Date

2021-11-02 - 2021-11-05

Location

Beijing, China