Automating the Quantitative Analysis of Reproducibility for Build Artifacts derived from the Android Open Source Project

Abstract

This work proposes a modular automation toolchain to analyze current state and over-time changes of reproducibility of build artifacts derived from the Android Open Source Project (AOSP). While perfect bit-by-bit equality of binary artifacts would be a desirable goal to permit independent verification if binary build artifacts really are the result of building a specific state of source code, this form of reproducibility is often not (yet) achievable in practice. Certain complexities in the Android ecosystem make assessment of production firmware images particularly difficult. To overcome this, we introduce "accountable builds"as a form of reproducibility that allows for legitimate deviations from 100 percent bit-by-bit equality. Using our framework that builds AOSP in its native build system, automatically compares artifacts, and computes difference scores, we perform a detailed analysis of differences, identify typical accountable changes, and analyze current major issues leading to non-reproducibility and non-accountability. We find that pure AOSP itself builds mostly reproducible and that Project Treble helped through its separation of concerns. However, we also discover that Google's published firmware images deviate from the claimed codebase (partially due to side-effects of Project Mainline).