On the Use of Tests for Software Supply Chain Threats

Abstract

Development teams are increasingly investing in automating the updating of third-party libraries to limit the patch time of zero-day exploits such as the Equifax breach. GitHub bots such as Dependabot and Renovate build such functionality by leveraging existing test infrastructure in repositories to test and evaluate new library updates. However, two recent studies suggest that test suites in projects lack effectiveness and coverage to reliably find regressions in third-party libraries. Adequate test coverage and effectiveness are critical in discovering new vulnerabilities and weaknesses from third-party libraries. The recent Log4Shell incident exemplifies this, as projects will likely not have adequate tests for logging libraries. This position paper discusses the weaknesses and challenges of current testing practices and techniques from a supply chain security perspective. We highlight two key challenges that researchers and practitioners need to address: (1) the lack of resources and best practices for testing the uses of third-party libraries and (2) enhancing the reliability of automating library updates.