P4Testgen: An Extensible Test Oracle For P4

We present P4Testgen, a test oracle for the P4-16 language that supports automatic test generation for any P4 target. Unlike prior tools, which were tailored to specific programs and targets, P4Testgen is designed to be general and extensible. It models the complete semantics of the entire packet-processing pipeline including the P4 language, architectures and externs, and target-specific extensions. Handling aspects of packet processing that lie outside of the official language specification is critical for supporting real-world targets. P4Testgen uses taint tracking and concolic execution to handle non-deterministic behaviors and complex externs (e.g., checksums and hash functions) and it provides path selection strategies that reduce the number of tests required to achieve full coverage. We have instantiated P4Testgen for the V1model, eBPF, and Tofino architectures. Each extension required effort commensurate with the complexity of the target. We validated the tests generated by P4Testgen by running them across the entire P4C test suite as well as the programs supplied with the Tofino P4 Studio. Using the tool, we have confirmed 28 bugs in the mature, production toolchains for BMv2 and Tofino.