【速览】TIP 2021丨显著性引导的迭代非对称哈希快速行人再识别

2021 年 9 月 17 日 中国图象图形学学会CSIG

学会“成果速览”系列文章旨在将图像图形领域会议期刊重要成果进行传播，通过短篇文章让读者用母语快速了解相关学术动态，欢迎关注和投稿~

◆ ◆ ◆ ◆

显著性引导的迭代非对称哈希快速行人再识别

赵才荣

^{1}

, 涂远鹏

^{1}

, 赖志辉

^{2}

, 沈复民

^{3}

, 申恒涛

^{3}

，苗夺谦

^{1}

^{1}

同济大学，

^{2}

深圳大学，

^{3}

成都电子科技大学

TIP 2021

撰稿人：赵才荣，涂远鹏

通讯作者：赵才荣

推荐理事：林宙辰

原文标题：Salience-Guided Iterative Asymmetric Mutual Hashing for Fast Person Re-identification

原文链接： https://ieeexplore.ieee.org/document/9531552

原文代码链接: https://github.com/Vill-Lab/SIAMH

◆ ◆ ◆ ◆

摘要

随着数据集规模的增大，现有的实值行人再识别方法无法满足实时检索需求，而深度哈希已被广泛应用于大规模图像检索领域用于检索加速。现有的哈希行人再识别方法虽然在长编码情况可以取得较好的性能，但由于其未考虑编码间的信息冗余，这些方法在短编码情况表现欠佳。为了解决这一问题，本文提出了一种显著性引导的迭代非对称哈希方法（Salience-Guided Iterative Asymmetric Mutual Hashing，SIAMH），可以显式降低编码间的信息冗余，同时缓解互学习的缺点。在多个大规模行人数据集上，SIAMH不仅可以缩小实值方法与哈希方法在长编码情况下的精度差距，同时还可以显著提升哈希方法在短编码情况下的性能。

背景

现有的哈希行人再识别方法仅关注于最小化二值化过程中的量化误差而忽略了编码间存在的信息冗余，导致这些方法无法保证编码的紧致性，从而产生次优的结果。除此之外，现有的方法大多采用简单的网络结构以实现快速特征提取，然而这也在一定程度上限制了这些方法的性能。而互学习恰好可以同时实现这两个目的，但常规的互学习方法存在两个缺点：1）首先对两个模型施加相同的强硬显式约束将会对大模型的性能造成损害，2）其次这种策略无法充分利用两个模型的互补效应。

因此基于以上内容，我们提出了一种新颖的显著性引导的迭代非对称哈希方法（SIAMH）用于解决上述问题。具体而言，不同于已有的方法仅从单一方向进行优化，SIAMH的整个网络从两个方向优化最终的哈希编码：模型间信息冗余最小化、模型内量化误差最小化。而这两个方向分别对应SIAMH中的两个主要过程：显著性引导的迭代非对称互学习、自蒸馏的量化误差最小化。SIAMH网络的示意图如图1所示：

图 1 SIAMH网络流程图

自蒸馏的量化误差最小化

自蒸馏的量化误差最小化阶段旨在提升大模型中的编码的性能，主要包含两个模块：多样化划分模块和自蒸馏量化过程。前者具体结构如图2所示，主要由两个分支组成：全局分支、局部分支。全局分支包含均值池化与最大池化两个分支，而局部分支则包含水平与竖直两种划分分支。两个分支的结果最终输入到Divide-and-Encode模块，使得编码根据特征的不同部分生成，从而一定程度上降低大模型所得到编码间的信息冗余。

图 2 DPM网络结构

自蒸馏量化过程旨在降低量化过程中的信息损失，现有的方法大多直接施加L2约束降低实值特征与哈希编码的差异，然而由于二者的信息容纳能力的差异，直接施加如此强硬的约束会无可避免地损害实值特征的性能，因此我们选择最小化二者在预测分布上的差异。具体而言，实值特征的预测结果会作为哈希编码分类损失中的软标签以引导自蒸馏过程，具体公式如下所示：

$L_{kl}\left ( m_{1},m_{2} \right )=\sum_{i-1}^{N_{d}}m_{1}\left ( c_{i} \right )\mathrm{log}\left ( \frac{m_{1}\left ( c_{i} \right )}{m_{2}\left ( c_{i} \right )} \right )$

其中m1,m2分别为小模型与大模型的预测结果。

显著性引导的迭代非对称互训练

为了显式降低编码间的信息冗余同时保证编码的判别力，我们提出了显著性引导的迭代非对称训练策略，如图3所示。具体而言，主要包含两个关键模块：显著性引导的蒸馏分支和迭代非对称互学习训练策略。前者主要目的为使得学生模型可以充分过滤掉与最终检索无关的视觉信息并根据最为显著的区域生成哈希编码从而显式降低信息冗余。具体而言，来自教师模型的特征热力图将会以掩码的形式对学生模型的第一分支特征图进行过滤，随后该分支的分类结果将会作为软标签训练第二个分支的预测结果，其过程可总结为如下形式：

$F_{sal}=\mathrm{Re}\left ( \mathrm{In}\left ( L_{CAM} \right ) \right )\bigodot F$

而迭代非对称互学习训练策略则使得两个模型可以从彼此的输出中学习到正则化的暗知识，从而缓解互学习的缺点并且充分利用两个模型的互补效应。另外该策略还可以使得整个框架的性能不再受骨干模型参数量的限制，具有较好的灵活性。教师模型与学生模型分别使用来自对方不同层次的信息进行非对称互学习。具体而言，教师模型有良好的判别力，但由于其参数量过大，对于某些数据集噪声容易过拟合，而学生模型表现不佳，但其泛化能力较强。因此学生模型将会以教师模型的特征图、实值特征与分类结果作为监督信息进行训练，而教师模型则会以学生模型的最终编码预测结果作为软标签进行训练，二者充分利用彼此的优势弥补各自的缺点。另外，两个模块可以形成正向的循环效应，首先自蒸馏分支的权重图可以使得学生模型产生更好的分类结果，而这个分类结果将会给教师模型施加更为有效的正则化从而使得教师模型得到更好的特征图，进一步关注于行人的显著区域，提升最终哈希编码的性能。

图 3 迭代非对称训练策略

最终在特征提取阶段，SIAMH仅需要根据学生模型的轻量化分支生成哈希编码，因此框架中的教师模型可以替换为现有的大多数实值方法，同时并不会增加特征提取的时间消耗，具有较好的结构灵活性，算法细节如下所示：

实验结果

我们在行人数据集(MSMT-17, LaST)上对提出的算法进行了测试与验证。为了证明提出方法的有效性，我们不仅将我们的方法与哈希方法进行对比，同时我们也与实值方法进行了详细的对比，结果如表1-4所示。

实验结果验证了所提出方法的有效性，我们的方法相比于最为先进的DLBC以及CtF能取得更好的效果。与实值方法相比，我们的方法进一步缩小了哈希方法与实值方法在性能上的差距，同时可以显著提升现有实值方法的匹配效率。除此之外，在短编码情况下，相比于现有的哈希方法，SIAMH取得了更为显著的提升，这也进一步说明SIAMH可以显式降低编码间的信息冗余。

表 1 在三个大规模行人数据集上与实值方法的性能对比

表 2 在三个大规模行人数据集上与哈希方法的性能对比

表 3 在CUHK03数据集上与传统哈希方法的性能对比

表 4 在LAST行人数据集上的性能对比

参考文献

[1] X. Qian, Y. Fu, T. Xiang, W. Wang, J. Qiu, Y. Wu, Y. Jiang, and X. Xue, “Pose-normalized image generation for person re-identification,” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IX, ser. Lecture Notes in Computer Science, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11213, 2018, pp. 661–678.

[2] L. Zheng, Y. Yang, and A. G. Hauptmann, “Person re-identification: Past, present and future,” CoRR, vol. abs/1610.02984, 2016.

[3] H. Luo, Y. Gu, X. Liao, S. Lai, and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, 2019, pp. 1487–1495.

[4] M. M. Kalayeh, E. Basaran, M. Gokmen, M. E. Kamasak, and M. Shah, “Human semantic parsing for person re-identification,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018, pp. 1062–1071.

[5] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling (and A strong convolutional baseline),” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, ser. Lecture Notes in Computer Science, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11208, 2018, pp. 501–518.

[6] Y. Sun, Q. Xu, Y. Li, C. Zhang, Y. Li, S. Wang, and J. Sun, “Perceive where to focus: Learning visibility-aware part-level features for partial person re-identification,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019, pp. 393–402.

[7] T. Chen, S. Ding, J. Xie, Y. Yuan, W. Chen, Y. Yang, Z. Ren, and Z. Wang, “Abd-net: Attentive but diverse person re-identification,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019, pp. 8350–8360.

[8] Z. Zhang, C. Lan, W. Zeng, X. Jin, and Z. Chen, “Relation-aware global attention for person re-identification,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020, pp. 3183–3192.

[9] X. Chen, C. Fu, Y. Zhao, F. Zheng, J. Song, R. Ji, and Y. Yang, “Salience-guided cascaded suppression network for person reidentification,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020. IEEE, 2020, pp. 3297–3307.

[10] B. Bryan, Y. Gong, Y. Zhang, and C. Poellabauer, “Second-order nonlocal attention networks for person re-identification,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea(South), October 27 - November 2, 2019, 2019, pp. 3759–3768.

[11] K. Zhou, Y. Yang, A. Cavallaro, and T. Xiang, “Omni-scale feature learning for person re-identification,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019, pp. 3701–3711.

[12] Z. Zheng, X. Yang, Z. Yu, L. Zheng, Y. Yang, and J. Kautz, “Joint discriminative and generative learning for person re-identification,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019, pp. 2138–2147.

[13] R. Hou, B. Ma, H. Chang, X. Gu, S. Shan, and X. Chen, “Interaction and aggregation network for person re-identification,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019, pp. 9317–9326.

[14] R. Zhang, L. Lin, R. Zhang, W. Zuo, and L. Zhang, “Bit-scalable deep hashing with regularized similarity learning for image retrieval and person re-identification,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 4766–4779, 2015.

[15] F. Zhao, Y. Huang, L. Wang, and T. Tan, “Deep semantic ranking based hashing for multi-label image retrieval,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, 2015, pp. 1556–1564.

[16] Z. Cao, M. Long, J. Wang, and P. S. Yu, “Hashnet: Deep learning to hash by continuation,” in IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, 2017, pp. 5609–5618.

[17] Y. Cao, M. Long, B. Liu, and J. Wang, “Deep cauchy hashing for hamming space retrieval,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018, pp. 1229–1237.

[18] J. Chen, Y. Wang, J. Qin, L. Liu, and L. Shao, “Fast person reidentification via cross-camera semantic binary transformation,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017, pp. 5330–5339.

[19] F. Zhu, X. Kong, L. Zheng, H. Fu, and Q. Tian, “Part-based deep hashing for large-scale person re-identification,” IEEE Trans. Image Process., vol. 26, no. 10, pp. 4806–4817, 2017.

[20] Y. Zhao, S. Luo, Y. Yang, and M. Song, “Deepssh: Deep semantic structured hashing for explainable person re-identification,” in 2018 IEEE International Conference on Image Processing, ICIP 2018, Athens, Greece, October 7-10, 2018, 2018, pp. 1653–1657.

[21] Z. Liu, J. Qin, A. Li, Y. Wang, and L. V. Gool, “Adversarial binary coding for efficient person re-identification,” in IEEE International Conference on Multimedia and Expo, ICME 2019, Shanghai, China, J uly 8-12, 2019, 2019, pp. 700–705.

[22] D. Li, Y. Gong, D. Cheng, W. Shi, X. Tao, and X. Chang, “Consistency-preserving deep hashing for fast person re-identification,” Pattern Recognit., vol. 94, pp. 207–217, 2019.

[23] G. Wang, S. Gong, J. Cheng, and Z. Hou, “Faster person reidentification,” in Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII, ser. Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12353, 2020, pp. 275–292.

[24] J. Chen, J. Qin, Y. Yan, L. Huang, L. Liu, F. Zhu, and L. Shao, “Deep local binary coding for person re-identification by delving into the details,” in MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, C. W. Chen, R. Cucchiara, X. Hua, G. Qi, E. Ricci, Z. Zhang, and R. Zimmermann, Eds., 2020, pp. 3034–3043.

[25] W. Kang, W. Li, and Z. Zhou, “Column sampling based discrete supervised hashing,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, D. Schuurmans and M. P. Wellman, Eds., 2016, pp. 1230–1236.

[26] F. Shen, C. Shen, W. Liu, and H. T. Shen, “Supervised discrete hashing,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, 2015, pp. 37–45.

[27] W. Liu, J. Wang, R. Ji, Y. Jiang, and S. Chang, “Supervised hashing with kernels,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, 2012, pp. 2074–2081.

[28] Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin, “Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 12, pp. 2916–2929, 2013.

[29] M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni, “Locality-sensitive hashing scheme based on p-stable distributions,” in Proceedings of the 20th ACM Symposium on Computational Geometry, Brooklyn, New York, USA, June 8-11, 2004, J. Snoeyink and J. Boissonnat, Eds., 2004, pp. 253–262.