Methodological Transparency and Big Data: A Critical Comparative Analysis of Institutionalization

Abstract

Big data is increasingly employed in predictive social analyses, yet there are many visible instances of unreliable models or failure, raising questions about methodological validity in data driven approaches. From meta-analysis of methodological institutionalization across three scholarly disciplines, there is evidence that traditional statistical quantitative methods, which are more institutionalized and consistent, are important to develop, structure, and institutionalize data scientific approaches for new and large n quantitative methods, indicating that data driven research approaches may be limited in reliability, validity, generalizability, and interpretability. Results also indicate that interdisciplinary collaborations describe methods in significantly greater detail on projects employing big data, with the effect that institutionalization makes data science approaches more transparent.