Multimodal fusion using learned text concepts for image categorization

Qiang Zhu, Mei Chen Yeh, Kwang Ting Cheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

32 Citations (Scopus)

Abstract

Conventional image categorization techniques primarily rely on low-level visual cues. In this paper, we describe a multimodal fusion scheme which improves the image classification accuracy by incorporating the information derived from the embedded texts detected in the image under classification. Specific to each image category, a text concept is first learned from a set of labeled texts in images of the target category using Multiple Instance Learning [1]. For an image under classification which contains multiple detected text lines, we calculate a weighted Euclidian distance between each text line and the learned text concept of the target category. Subsequently, the minimum distance, along with lowlevel visual cues, are jointly used as the features for SVM-based classification. Experiments on a challenging image database demonstrate that the proposed fusion framework achieves a higher accuracy than the state-of-art methods for image classification.

Original languageEnglish
Title of host publicationProceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006
Pages211-220
Number of pages10
DOIs
Publication statusPublished - 2006 Dec 1
Externally publishedYes
Event14th Annual ACM International Conference on Multimedia, MM 2006 - Santa Barbara, CA, United States
Duration: 2006 Oct 232006 Oct 27

Other

Other14th Annual ACM International Conference on Multimedia, MM 2006
CountryUnited States
CitySanta Barbara, CA
Period06/10/2306/10/27

    Fingerprint

Keywords

  • Image annotation
  • Image categorization
  • Multimodal fusion
  • Multiple instance learning
  • Text detection

ASJC Scopus subject areas

  • Computer Science(all)
  • Media Technology

Cite this

Zhu, Q., Yeh, M. C., & Cheng, K. T. (2006). Multimodal fusion using learned text concepts for image categorization. In Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006 (pp. 211-220) https://doi.org/10.1145/1180639.1180698