Minimizing privilege for building HPC containers

HPC centers face increasing demand for software flexibility, and there is growing consensus that Linux containers are a promising solution. However, existing container build solutions require root privileges and cannot be built directly on HPC resources. This limitation is compounded as supercomputer diversity expands and HPC architectures become more dissimilar from commodity computing resources. Our evaluation of available options suggests this problem can best be solved with low-privilege containers. We detail Linux kernel features for varying container privilege and compare two open-source implementations, mostly-unprivileged rootless Podman and fully-unprivileged Charliecloud. Our analysis demonstrates that low-privilege container build on HPC resources works now and will continue to improve, giving normal users a better workflow to securely and correctly build containers. Minimizing privilege in this way can improve HPC user and developer productivity as well as reduce support workload for exascale applications.