P4Testgen: An Extensible Test Oracle For P4

We present P4Testgen, a test oracle for the P4-16 language that supports automatic generation of packet tests for any P4-programmable device. Given a P4 program and sufficient time, P4Testgen generates tests that cover every reachable statement in the input program. Each generated test consists of an input packet, control-plane configuration, and output packet(s), and can be executed in software or on hardware. Unlike prior work, P4Testgen is open source and extensible, making it a general resource for the community. P4Testgen not only covers the full P4-16 language specification, it also supports modeling the semantics of an entire packet-processing pipeline, including target-specific behaviors-i.e., whole-program semantics. Handling aspects of packet processing that lie outside of the official specification is critical for supporting real-world targets (e.g., switches, NICs, end host stacks). In addition, P4Testgen uses taint tracking and concolic execution to model complex externs (e.g., checksums and hash functions) that have been omitted by other tools, and ensures the generated tests are correct and deterministic. We have instantiated P4Testgen to build test oracles for the V1model, eBPF, and the Tofino (TNA and T2NA) architectures; each of these extensions only required effort commensurate with the complexity of the target. We validated the tests generated by P4Testgen by running them across the entire P4C program test suite as well as the Tofino programs supplied with Intel's P4 Studio. In just a few months using the tool, we discovered and confirmed 25 bugs in the mature, production toolchains for BMv2 and Tofino, and are conducting ongoing investigations into further faults uncovered by P4Testgen.