Validity Concerns in Research Using Organic Data

Abstract

With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.