A Survey of Data Quality Assessment Methods for Big Data (빅데이터를 위한 데이터 품질 평가 방법에 대한 연구)

Okjoo Choi (최옥주); Yukyong Kim (김유경)

doi:10.29056/jsav.2023.12.09

A Survey of Data Quality Assessment Methods for Big Data

Journal of Software Assessment and Valuation
Abbr : JSAV
2023, 19(4), pp.89-97
DOI : 10.29056/jsav.2023.12.09
Publisher : Korea Software Assessment and Valuation Society
Research Area : Engineering > Computer Science
Received : December 2, 2023
Accepted : December 20, 2023
Published : December 31, 2023

Okjoo Choi ¹, Yukyong Kim ²

¹강원대학교
²숙명여자대학교

Accredited

ABSTRACT

Data quality is very important in data-based information technologies such as big data analysis or machine learning. Even if high-performance data analysis algorithms or machine learning models are used, if the quality of the input data is not guaranteed, the results cannot be trusted. Therefore, in order to utilize and analyze big data, it is necessary to be able to extract high-quality data from massive and complex data. In this paper, we consider quality evaluation methods for big data that guarantee high quality. We examine international standardization trends for data life cycle and data quality factors for big data and define data quality characteristics that should be considered according to the big data life cycle. In addition, we compare and analyze existing major studies on data quality evaluation related to big data, and based on the result, we examine the elements necessary for big data quality evaluation. Based on the defined elements, we propose a big data quality evaluation process using goal-driven data quality metric. In the future, we expect that the proposed process will be used to develop evaluation indicators that can evaluate big data quality and to develop an integrated evaluation framework.

KEYWORDS

Big data, Data quality, Data quality assessment, Data quality metric, Data quality assessment framework

Citation status

* References for papers published after 2023 are currently being built.

[journal] Suwook Ha / 2020 / Data quality standardization for data analysis and machine learning / TTA Journal 192 : 22~27

[journal] Curtis Northcutt / 2021 / Confident Learning: Estimating Uncertainty in Dataset Labels / Journal of Artificial Intelligence Research / AI Access Foundation 70 : 1373~1411

[confproc] O. Reda / 2020 / Towards a Data Quality Assessment in Big Data / Proceedings of the International Conference on Intelligent Systems: Theories and Applications : 1~6

[journal] Diane M. Strong / 1997 / Data quality in context / Communications of the ACM / Association for Computing Machinery (ACM) 40(5) : 103~110

[web] / 2019 / ITU-T Y.3604, Big data – Requirements and conceptual model of metadata for data catalogue / https://www.itu.int/rec/T-REC-Y.3603-201912-1/en

[journal] Youdi Gong / 2023 / A survey on dataset quality in machine learning / Information and Software Technology / Elsevier BV 162 : 107268~

[other] / 2008 / ISO/IEC 25012, Software engineering –Software product quality requirements and evaluation (SQuaRE) - Data quality model

[web] / 2023 / ISO/IEC DIS 5259 / https://www.iso.org/standard/81088.html

[web] Wo Chang / 2022 / ISO/IEC JTC1/SC 42(AI)/WG2(Data) Data Qulaity for Analytics and Machine Learning (ML) / https://jtc1info.org/wp-content

[confproc] I. Taleb / 2018 / Big data quality: A survey / Proceeding of IEEE International congress on Big Data : 166~173

[journal] Li Ya / 2020 / RETRACTED: Studies on Data Quality Evaluation Index System for Internet Plus Government Services in Big Data Era / Journal of Physics: Conference Series / IOP Publishing 1584(1) : 012014~

[journal] Hong Chen / 2014 / A Review of Data Quality Assessment Methods for Public Health Information Systems / International Journal of Environmental Research and Public Health / MDPI AG 11(5) : 5170~5207

[journal] S. Chug / 2021 / Statistical learning to operationalize a domain agnostic data quality scoring

[journal] Li Cai / 2015 / The Challenges of Data Quality and Data Quality Assessment in the Big Data Era / Data Science Journal / Ubiquity Press, Ltd. 14(2)

[journal] Christian Bors / 2018 / Visual Interactive Creation, Customization, and Analysis of Data Quality Metrics / Journal of Data and Information Quality / Association for Computing Machinery (ACM) 10(1) : 1~26

[journal] Fakhitah Ridzuan / 2019 / A Review on Data Cleansing Methods for Big Data / Procedia Computer Science / Elsevier BV 161 : 731~738

[journal] Mohamad Al Jazaery / 2019 / Automated cleaning of identity label noise in a large face dataset with quality control / IET Biometrics / Institution of Engineering and Technology (IET) 9(1) : 25~30

KJCKorea
Journal Central

Journal of Software Assessment and Valuation 2023 KCI Impact Factor : 0.35

A Survey of Data Quality Assessment Methods for Big Data

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of Software Assessment and Valuation 2023 KCI Impact Factor : 0.35

A Survey of Data Quality Assessment Methods for Big Data

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (17) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.