Content-Based Visual Search Learned from Social Media
Supervisor(s) and Committee member(s): Advisor(s): Arnold W.M. Smeulders (promotor), Marcel Worring (co-promotor), Cees G.M. Snoek (co-promotor), James Z. Wang (opponent), Maarten de Rijke (opponent), Guus Schreiber (opponent), Alan Hanjalic (opponent)
In a world with increasing amounts of digital pictures, content-based visual search is an important and scientifically challenging problem in ICT research. This thesis tackles the problem by learning from social media. The fundamental question addressed in this thesis is:
what is the value of socially tagged images for visual search?
To that end, we propose the neighbor voting algorithm (Chapter 2) and its multi-feature variant (Chapter 3) to verify whether what people spontaneously say about an image is factually in the pictorial content. The two algorithms are used to find high-quality positive examples for learning automated image taggers. To obtain negative training examples without manual verification, we go beyond the classical random sampling approach by introducing informative negative bootstrapping (Chapter 4). For answering complex visual searches, we introduce the notion of bi-concepts as a retrieval method for unlabeled images in which two concepts are co-occurring (Chapter 5). Finally, as users have their own associations with image semantics, we propose personalized image tagging by jointly exploiting personal tagging history and content-based analysis, optimized through Monte Carlo sampling (Chapter 6).
On the basis of the reported theories, algorithms, and experiments, this thesis has revealed the value of socially tagged images for content-based visual search, providing a basis for uncovering universal knowledge on images and semantics. With the methodologies established, this thesis opens up promising avenues for image search engines which provide access to the semantics of the visual content, but without the need of manual labeling.
Intelligent Systems Lab Amsterdam
The Intelligent Systems Lab Amsterdam ISLA at the University of Amsterdam performs fundamental, applied and spin-off research. We define intelligence as observing and learning; observing the world by video, still pictures, signals and text and abstracting knowledge or decisions to act from these observations.