Neural Architecture Search based on Cartesian Genetic Programming Coding Method

Neural architecture search (NAS) is a hot topic in the field of automated machine learning (AutoML) and has begun to outperform human-designed architectures on many machine learning tasks. Motivated by the natural representation form of neural networks by the Cartesian genetic programming (CGP), we propose an evolutionary approach of NAS based on CGP, called CGPNAS, to solve sentence classification task. To evolve the architectures under the framework of CGP, the existing key operations are identified as the types of function nodes of CGP, and the evolutionary operations are designed based on Evolutionary Strategy (ES). The experimental results show that the searched architecture can reach the accuracy of human-designed architectures, such as Transformer. The transfer study proves that the searched architectures have the certain ability for dataset transfer. The ablation study identifies the Attention function as the single key function node. In addition, only through the linear transformations, the accuracy of the searched architectures is reduced by 4%, worthy of investigation in the future.