A Governance framework for data audit trail creation in large multi-disciplinary projects

Abstract

The creation of data audit trails, within large multi-disciplinary projects in the CSIRO Water for Healthy Country (WfHC) Research Flagship, have relied heavily on the development of appropriate data management tools for creating metadata and data audit trails, coupled with human and technological processes. However, having the tools and processes alone does not provide an effective data management system so we have established a Governance Framework which provides the holistic function of integrating the various teams, technologies, and tools in an organized and focused way. The Governance Framework creates an environment where there are clear objectives, roles and responsibilities for handling data management activities, protocols, procedures, and processes for delivering the components of data, metadata, and audit trails, as well as providing a means of assigning accountability and developing incentives that ensure that audit trails are completed with a high degree of confidence. While the Governance Framework forms a holistic function it does require the application of an efficient tool for cataloguing data and constructing data audit trails with provenance information. A metadata tool was originally developed in 2007 by CSIRO Land and Water through the WfHC Flagship. The main need for the metadata tool was for capturing a data audit trail for a basin-wide multi-disciplinary project known as the Murray Darling Basin Sustainable Yields (MDBSY) project. The tool was later modified to accommodate additional Sustainable Yields (SY) projects between 2008 and 2010 and proved a valuable asset as a data auditing platform for such high-profile projects. The modified tool still had some limitations and did not fully conform to a metadata standard. In 2011 the tool, known as the Regional Water Data Management System (RWDMS), was rebuilt to meet the ANZLIC metadata standard and with improvements in functionality. Based on a similar architecture to the original tool, the RWDMS has various new features such as metadata statistics, the ability to add new projects via the web interface, and import and export functionality. In addition, it now also provides the ability to generate audit trail lineage diagrams to depict the genesis of data throughout a project via parent-child relationships that are defined in a lineage field during metadata entry. The RWDMS is also able to export metadata in ANZLIC XML format, which can then be imported into the CSIRO Enterprise, Data Access Portal (DAP) for wider access. The tool includes a web-based user interface, a relational database for storing the metadata, and a metadata 'robot' which scans the data storage file system to detect new datasets placed there and then create blank records in the database, ready for manual metadata attribute population via the web interface. The RWDMS has been effectively applied to the creation of data audit trails. It relies on an underlying project data archive file structure with clearly defined directories and a naming convention which flags datasets so they can be detected by the metadata 'robot' and then catalogued by the user. The parent-child linkages are then defined for each dataset to create an audit trail which can be visualized within the RWDMS interface. The Governance Framework has created a vital operational environment for managing project data in the CSIRO, WfHC Flagship while the RWDMS provides a highly functional toolset for cataloguing data, for monitoring archive construction and data custodian input statistics, for reporting of project metadata statistics, for viewing audit trail diagrams, which depict project data genesis through parent-child linkages, and a process for publishing data products into the CSIRO Enterprise DAP, post-project, for wider data discovery and reuse. While there are still limitations in the capture of good quality provenance information, future developments in the capture of project workflows and automation of provenance capture will further enhance this system and provide further support to the development of a coherent data management culture.