SeeSaw: Interactive Ad-hoc Search Over Image Databases
Author(s)
Moll, Oscar; Favela, Manuel; Madden, Samuel; Gadepally, Vijay; Cafarella, Michael
Download3626754.pdf (1.673Mb)
Publisher with Creative Commons License
Publisher with Creative Commons License
Creative Commons Attribution
Terms of use
Metadata
Show full item recordAbstract
As image datasets become ubiquitous, the problem of ad-hoc searches over image data is increasingly important. Many high level data tasks in machine learning, such as constructing datasets for training and testing object detectors, imply finding ad-hoc objects or scenes within large image datasets as a key sub-problem. New foundational visual-semantic embeddings trained on massive web datasets such as CLIP can help users start searches on their own data, but we find there is a long tail of queries where these models fall short in practice. SeeSaw is a system for interactive ad-hoc searches on image datasets that integrates state-of-the-art embeddings like CLIP with user feedback in the form of box annotations to help users quickly locate images of interest in their data even in the long-tail of harder queries. One key challenge for SeeSaw is that many sensible approaches to incorporating feedback into future results, including state of the art active-learning algorithms, can worsen results compared to introducing no feedback at all, partly due to CLIP’s high average performance. Therefore, SeeSaw employs several algorithms to transform user feedback into consistent improvements over CLIP alone. We compare SeeSaw’s accuracy to both using CLIP alone as well as to a state-of-the-art active-learning baseline and find SeeSaw consistently helps improve results for users across four datasets and more than a thousand queries. SeeSaw increases Average Precision (AP) on search tasks by an average of .08 on a wide benchmark (from a base of .72), and by a .27 on a subset of harder queries where CLIP alone performs poorly.
Date issued
2023-12-12Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Lincoln LaboratoryJournal
Proceedings of the ACM on Management of Data
Publisher
ACM
Citation
Moll, Oscar, Favela, Manuel, Madden, Samuel, Gadepally, Vijay and Cafarella, Michael. 2023. "SeeSaw: Interactive Ad-hoc Search Over Image Databases." Proceedings of the ACM on Management of Data, 1 (4 (SIGMOD)).
Version: Final published version
ISSN
2836-6573
Collections
The following license files are associated with this item: