Multimodal fusion using learned text concepts for image categorization

Qiang Zhu*, Mei Chen Yeh, Kwang Ting Cheng

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

33 Citations (Scopus)


Conventional image categorization techniques primarily rely on low-level visual cues. In this paper, we describe a multimodal fusion scheme which improves the image classification accuracy by incorporating the information derived from the embedded texts detected in the image under classification. Specific to each image category, a text concept is first learned from a set of labeled texts in images of the target category using Multiple Instance Learning [1]. For an image under classification which contains multiple detected text lines, we calculate a weighted Euclidian distance between each text line and the learned text concept of the target category. Subsequently, the minimum distance, along with lowlevel visual cues, are jointly used as the features for SVM-based classification. Experiments on a challenging image database demonstrate that the proposed fusion framework achieves a higher accuracy than the state-of-art methods for image classification.

Original languageEnglish
Title of host publicationProceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006
Number of pages10
Publication statusPublished - 2006 Dec 1
Externally publishedYes
Event14th Annual ACM International Conference on Multimedia, MM 2006 - Santa Barbara, CA, United States
Duration: 2006 Oct 232006 Oct 27


Other14th Annual ACM International Conference on Multimedia, MM 2006
Country/TerritoryUnited States
CitySanta Barbara, CA


  • Image annotation
  • Image categorization
  • Multimodal fusion
  • Multiple instance learning
  • Text detection

ASJC Scopus subject areas

  • Computer Science(all)
  • Media Technology


Dive into the research topics of 'Multimodal fusion using learned text concepts for image categorization'. Together they form a unique fingerprint.

Cite this