Fairness as a Service (FaaS): verifiable and privacy-preserving fairness auditing of machine learning systems

Abstract

Providing trust in machine learning (ML) systems and their fairness is a socio-technical challenge, and while the use of ML continues to rise, there is lack of adequate processes and governance practices to assure their fairness. In this paper, we propose FaaS, a novel privacy-preserving, end-to-end verifiable solution, that audits the algorithmic fairness of ML systems. FaaS offers several features, which are absent from previous designs. The FAAS protocol is model-agnostic and independent of specific fairness metrics and can be utilised as a service by multiple stakeholders. FAAS uses zero knowledge proofs to assure the well-formedness of the cryptograms and provenance in the steps of the protocol. We implement a proof of concept of the FaaS architecture and protocol using off-the-shelf hardware, software, and datasets and run experiments to demonstrate its practical feasibility and to analyse its performance and scalability. Our experiments confirm that our proposed protocol is scalable to large-scale auditing scenarios (e.g. over 1000 participants) and secure against various attack vectors.