With the usage of appropriate inductive biases, Counterfactual Generative Networks (CGNs) can generate novel images from random combinations of shape, texture, and background manifolds. These images can be utilized to train an invariant classifier, avoiding the wide spread problem of deep architectures learning spurious correlations rather than meaningful ones. As a consequence, out-of-domain robustness is improved. However, the CGN architecture comprises multiple over parameterized networks, namely BigGAN and U2-Net. Training these networks requires appropriate background knowledge and extensive computation. Since one does not always have access to the precise training details, nor do they always possess the necessary knowledge of counterfactuals, our work addresses the following question: Can we use the knowledge embedded in pre-trained CGNs to train a lower-capacity model, assuming black-box access (i.e., only access to the pretrained CGN model) to the components of the architecture? In this direction, we propose a novel work named SKDCGN that attempts knowledge transfer using Knowledge Distillation (KD). In our proposed architecture, each independent mechanism (shape, texture, background) is represented by a student 'TinyGAN' that learns from the pretrained teacher 'BigGAN'. We demonstrate the efficacy of the proposed method using state-of-the-art datasets such as ImageNet, and MNIST by using KD and appropriate loss functions. Moreover, as an additional contribution, our paper conducts a thorough study on the composition mechanism of the CGNs, to gain a better understanding of how each mechanism influences the classification accuracy of an invariant classifier. Code available at: https://github.com/ambekarsameer96/SKDCGN
翻译:利用适当的感化偏差, 反事实生成网络( CGNs) 可以从形状、 纹理和背景元件的随机组合中生成新图像。 这些图像可以用来训练一个不易分解的分类器, 避免深层结构中广泛扩散的问题, 学习虚假的关联, 而不是有意义的关联。 结果, 外部强力得到了改善。 然而, 中央GN 架构由多个参数化网络, 即 BigGAN 和 U2- Net 组成。 培训这些网络需要适当的背景知识和广泛的计算。 由于人们并不总是能获得精确的培训细节, 也不总是拥有对反事实的必要精确的知识, 我们的工作可以解决以下问题: 我们能否利用预先训练的 CGNGs 中所含的知识来训练一个低能力模型, 假设黑箱访问( 仅访问经过预先训练的 CGNG 模型)? 在这个方向上, 我们提议了一个叫SKDCGNGNG 的新工作, 尝试利用知识蒸馏( KD) 来传授知识。 在我们提议的架构中, 每一个独立机制, 都使用一个正在学习的 CHR 工具, 工具, 来展示我们现有的 C.