VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight

State-of-art simulators primarily focus on providing full-stack simulation tools or state-only parallelizability. Due to the limitation of computing resources, they have to make trade-off among photo-realism and sampling efficiency. Yet, both factors are crucial for data-driven reinforcement learning tasks. Therefore, we introduce a both rapid-rendering and photo-realistic quadrotor simulator: VisFly. VisFly offers a user-friendly framework and interfaces for users to develop or utilize. It couples differentiable dynamics and habitat-sim rendering engines, reaching frame rate of up to 10000 frame per second in cluttered environments. The simulation is wrapped as a gym environment, facilitating convenient implementation of various baseline learning algorithms. It can directly import all the open-source scene datasets compatible with habitat-sim, which provides more fair benchmarks for comparing the intelligent policy. VisFly presents a general policy architecture for tasks, and the whole framework is verified by three regular quadrotor tasks with visual observation. We will make this tool available at \url{https://github.com/SJTU-ViSYS/VisFly}.