Targeted Trojan-Horse Attacks on Language-based Image Retrieval

When a retrieval system expands data, its database is at risk of being attacked. In this paper, we introduce the concept of targeted Trojan-horse (TTH) attacks for language-based image retrieval (LBIR), the first keyword-wise targeted attack against the database of the retrieval system. Specifically, given a specific keyword, TTH generates a QR-code patch that can be applied to a set of different images to gain the targeted Trojan-horse images, which closes to the target keyword in the common space of cross-modal matching of retrieval model. With Uploading the generated TTH images to the database, TTH images will rank high in a normal search, even though the images are completely irrelevant to the query. We evaluate the attacks on standard language-based image retrieval benchmarks (i.e. Flickr30k and MSCOCO) and compare the results retrieved with and without the Trojan-horse images.