PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

Recent researches in artificial intelligence have proposed versatile convolutional neural networks (CNN) with different structures and substantially improved the accuracy of various intelligent applications. Nevertheless, the CNN inference imposes heavy computation overhead on mobile devices, but uploading the large volume of raw data to the cloud causes significant network latency. Motivated by the spatial independence of convolution operation, we propose pipeline cooperation (PICO) framework to accelerate CNN inference using multiple diverse mobile devices in this paper. PICO divides the CNN and mobile devices into several stages and combines them into an inference pipeline. PICO faces three main challenges: (1) Parallelizing the convolution operation introduces redundant calculation. (2) The partition is greatly complicated since the structures of many CNNs are directed acyclic graphs (DAG). (3) The mobile devices own diverse computing resources. In response to these issues, a two-step optimization is proposed based on deep analysis. We first orchestrate the DAG into sequential pieces, then divides these pieces and devices into stages. The optimization goal is to minimize the redundant calculation during partition and maximize the throughput. In our experiment with $2 \sim 8$ RaspberryPi devices, the throughput can be improved by $1.8 \sim 6.8 \times$ under different CPU frequencies.