Search engines and recommendation systems are built to efficiently display relevant information from those massive amounts of candidates. Typically a three-stage mechanism is employed in those systems: (i) a small collection of items are first retrieved by (e.g.,) approximate near neighbor search algorithms; (ii) then a collection of constraints are applied on the retrieved items; (iii) a fine-grained ranking neural network is employed to determine the final recommendation. We observe a major defect of the original three-stage pipeline: Although we only target to retrieve $k$ vectors in the final recommendation, we have to preset a sufficiently large $s$ ($s > k$) for each query, and ``hope'' the number of survived vectors after the filtering is not smaller than $k$. That is, at least $k$ vectors in the $s$ similar candidates satisfy the query constraints. In this paper, we investigate this constrained similarity search problem and attempt to merge the similarity search stage and the filtering stage into one single search operation. We introduce AIRSHIP, a system that integrates a user-defined function filtering into the similarity search framework. The proposed system does not need to build extra indices nor require prior knowledge of the query constraints. We propose three optimization strategies: (1) starting point selection, (2) multi-direction search, and (3) biased priority queue selection. Experimental evaluations on both synthetic and real data confirm the effectiveness of the proposed AIRSHIP algorithm. We focus on constrained graph-based approximate near neighbor (ANN) search in this study, in part because graph-based ANN is known to achieve excellent performance. We believe it is also possible to develop constrained hashing-based ANN or constrained quantization-based ANN.
翻译:建立搜索引擎和建议系统,以便有效地显示来自这些大批候选人的相关信息。通常,这些系统中使用一个三阶段机制:(一) 最初由(例如)近邻搜索算法获得少量的物品;(二) 然后对检索到的物品适用一系列限制;(三) 使用精细度级神经网络来确定最后建议。我们观察到最初的三阶段管道存在一个重大缺陷:虽然我们只打算在最后建议中检索到以美元为单位的矢量,但我们必须预先为每个查询预设一个足够大的费用($ > k$)的三阶段机制,以及“hope”在过滤后所幸存的矢量数量不小于美元;(二) 将精密的矢量收集到(a) 一个以美元为单位的量级数(a) 一个以美元为单位的量级(a) 一个系统,一个以用户定义的量值为单位的精度(a) 一个系统,一个精度(a) 一个精度的量级(a),一个精度(b) 一个精度的精度(b) 一个精度(bral)的精度(b)的精度(a)一个系统,这个系统需要先量级搜索的精度的精度的精度(W的精度) 的精度) 的精度搜索的精度(b) 的精度(b) 的精度搜索的精度) 的精度搜索的精度搜索的精度搜索) 的精度研究框架。