An Evaluation of Open-source Tools for the Provision of Differential Privacy

The concept of differential privacy has widely penetrated academia and industry, with its formal guarantee on individual privacy that leads to compliances with privacy legislation, e.g., GDPR. However, there is a lack of understanding on tools capable of achieving differential privacy, and it is not clear what to expect from existing differential privacy tools when implementing privacy protection. Such an obstacle limits private applications' further prosperity. This paper reviews and evaluates the state-of-the-art open-source differential privacy tools of different domains using various estimating categories and privacy settings. Particularly, we look into the performances of three differential privacy tools for machine learning, two for statistical query, and four for synthetic data generation. We test all the tools on both continuous and categorical data and quantify their performance under different privacy budget and data size w.r.t. utility loss and system overhead. The accumulated evaluation results reveal several patterns that users can follow to optimally configure the tools, and provide preliminary guidelines on tool selection under different criteria. Finally, we openly release our evaluation coding repository, a framework that users can reuse to further evaluate the studied tools and beyond. We anticipate this work to provide a comprehensive insight into the performances of the existing dominant privacy tools, and a concrete reference for a potentially large developer community on private applications, thus narrowing the gap between conceptual differential privacy and private functionality development.