A Hybrid Framework for Scalable Data Quality: Comparing PySpark and AI-Powered Validation in Microsoft Fabric

Abstract

The rapid growth of big data demands robust frameworks to ensure data quality in distributed platforms, as poor data quality undermines insights in domains like e-commerce and social media analytics. Traditional PySpark-based validation excels in syntactic checks (e.g., pattern violations) but struggles with semantic anomalies (e.g., sentiment drift, null detection). Large Language Models (LLMs) offer context-aware validation but face scalability and cost challenges. This study proposes an Adaptive Hybrid framework that integrates PySpark’s rule-based efficiency with AI-powered contextual validation in Microsoft Fabric’s Lakehouse environment to deliver high-performance data quality assessment. The framework uses a composite performance score (F = 0.4A + 0.4S − 0.2E) to quantify trade-offs in accuracy (A), scalability (S = 1/T), and development effort (E), ensuring reliable and efficient validation without runtime optimization. The weights reflect enterprise priorities for precision and throughput. Evaluated across four data quality scenarios null detection, pattern violations, semantic mismatches, and sentiment drift using real-world Twitter airline sentiment and Amazon product review datasets, the framework achieves high accuracy (∼0.9), scalability (1.898 s for 5,000 rows in syntactic tasks), and low effort (E ≈ 0.3). PySpark dominates syntactic validation (F1-score = 0.791 for pattern violations), while AI and Hybrid methods significantly improve semantic recall (0.674 for mismatches). Unlike prior syntactic-only or textclassification frameworks, this approach bridges structured data quality gaps, offering a scalable, intelligent solution for modern analytics platforms