Driving simulators play a large role in developing and testing new intelligent vehicle systems. The visual fidelity of the simulation is critical for building vision-based algorithms and conducting human driver experiments. Low visual fidelity breaks immersion for human-in-the-loop driving experiments. Conventional computer graphics pipelines use detailed 3D models, meshes, textures, and rendering engines to generate 2D images from 3D scenes. These processes are labor-intensive, and they do not generate photorealistic imagery. Here we introduce a hybrid generative neural graphics pipeline for improving the visual fidelity of driving simulations. Given a 3D scene, we partially render only important objects of interest, such as vehicles, and use generative adversarial processes to synthesize the background and the rest of the image. To this end, we propose a novel image formation strategy to form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photorealistic RGB images with a state-of-the-art Generative Adversarial Network (GAN) trained on real-world driving scenes. This replaces repetitiveness with randomly generated but photorealistic surfaces. Finally, the partially-rendered and GAN synthesized images are blended with a blending GAN. We show that the photorealism of images generated with the proposed method is more similar to real-world driving datasets such as Cityscapes and KITTI than conventional approaches. This comparison is made using semantic retention analysis and Frechet Inception Distance (FID) measurements.
翻译:驱动模拟器在开发和测试新型智能车辆系统方面起着很大的作用。 模拟的视觉真实性对于建立基于视觉的算法和进行人类驱动器实验至关重要。 低视觉真实性会断裂沉浸, 用于在环形人驾驶实验。 常规计算机图形管道使用详细的 3D 模型、 模具、 纹理和引擎来从 3D 场景生成 2D 图像。 这些过程是劳动密集型的, 它们不会生成光真化图像 。 在此我们推出一个混合的基因直观图像管道, 以改善驾驶模拟的视觉真实性。 在三维场景中, 我们只部分提供重要的对象, 如车辆等, 并且使用基因化的对称对抗程序来合成背景和图像的其余部分。 为此, 我们提出一个新的图像形成策略, 从 3D 场景中形成 2D 常规图像, 由简单的对象模型组成, 没有纹理。 这些精度图像随后被转换成光真性 RGB 图像, 并用一种最先进的直观性 Adversarial 图像网络( GAN ) 取代了这个模拟的图像, 和模拟模拟模拟模拟模拟的模拟模拟图像, 正在成一个模拟的图像。