Programmable 3D snapshot microscopy with Fourier convolutional networks

3D snapshot microscopy enables fast volumetric imaging by capturing a 3D volume in a single 2D camera image, and has found a variety of biological applications such as whole brain imaging of fast neural activity in larval zebrafish. The optimal microscope design for this optical 3D-to-2D encoding is both sample- and task-dependent, with no general solution known. Highly programmable optical elements create new possibilities for sample-specific computational optimization of microscope parameters, e.g. tuning the collection of light for a given sample structure. We perform such optimization with deep learning, using a differentiable wave-optics simulation of light propagation through a programmable microscope and a neural network to reconstruct volumes from the microscope image. We introduce a class of global kernel Fourier convolutional neural networks which can efficiently decode information from multiple depths in the volume, globally encoded across a 3D snapshot image. We show that our proposed networks succeed in large field of view volume reconstruction and microscope parameter optimization where traditional networks fail. We also show that our networks outperform the state-of-the-art learned reconstruction algorithms for lensless computational photography.