Abstract :

Data Quality Is Critical To The Scalability And Effectiveness Of Large-scale Analytics Pipelines Deployed In Enterprise Environments. In This Situation, Inaccuracies, Incompleteness, Or Inconsistencies In The Data Cause A Propagation Error Through The Analytical Models. This Can Lead To Inaccurate Insights, Such As Operational Inefficiencies And Increased Company Risk. Traditional Data Quality Evaluation Approaches May Rely On Static Or Manual Tests, Which Are Challenging To Scale And Insufficiently Flexible To The Evolving Data Stream. As The Complexity And Volume Of Analytic Pipelines Increase, There Is A Need To Automate The Data-driven Mechanism To Ensure A Continual Assessment Of Data Quality. This Paper Describes An Entropy-driven Data Quality Approach Used In Large-scale Analytic Pipelines. This Methodology Leverages Information And Theoretical Entropy To Assist In Quantifying The Uncertainties, Variability, And Irregularities That Exist Within Streams. This Will Aid And Support The Goal Of Ensuring Quality In The Framework Enterprise On An Ongoing Basis. As A Result, Computing The Entropy-based Score Across The Feature Ensures That The Time Window And Data Source Are Approaching The Detection Of Abnormalities, Degradations, And Shifts In The Distribution Of Signals With Diminishing Data Quality. The Scores Are Aggregated Into A Consistent Quality Metric That Can Be Integrated Into The Workflow Metrics That Already Exist.

Published:

01-3-2025

Issue:

Vol. 25 No. 3 (2025)

Page Nos:

272 - 279

Section:

Articles

License:

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

How to Cite