Cleaning Noises from Time Series Data with Memory Effects (메모리 효과를 갖는 시계열 데이터에서 다수의 노이즈 클리닝에 대한 연구)

Jae-Han Cho (조재한); LeeSub Lee (이이섭)

doi:10.9708/jksci.2020.25.04.037

Cleaning Noises from Time Series Data with Memory Effects

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2020, 25(4), pp.37-45
DOI : 10.9708/jksci.2020.25.04.037
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : March 24, 2020
Accepted : April 13, 2020
Published : April 30, 2020

Jae-Han Cho ¹, LeeSub Lee ¹

¹금오공과대학교

Accredited

ABSTRACT

The development process of deep learning is an iterative task that requires a lot of manual work. Among the steps in the development process, pre-processing of learning data is a very costly task, and is a step that significantly affects the learning results. In the early days of AI's algorithm research, learning data in the form of public DB provided mainly by data scientists were used. The learning data collected in the real environment is mostly the operational data of the sensors and inevitably contains various noises. Accordingly, various data cleaning frameworks and methods for removing noises have been studied. In this paper, we proposed a method for detecting and removing noises from time-series data, such as sensor data, that can occur in the IoT environment. In this method, the linear regression method is used so that the system repeatedly finds noises and provides data that can replace them to clean the learning data. In order to verify the effectiveness of the proposed method, a simulation method was proposed, and a method of determining factors for obtaining optimal cleaning results was proposed.

KEYWORDS

Deep Learning, Time Series, Preprocessing, Noise, Data Cleaning

Citation status

* References for papers published after 2023 are currently being built.

[confproc] D. Xin / 2018 / Helix: Holistic optimization for accelerating iterative machine learning / Proceedings of the VLDB Endowment : 446~460

[journal] David Sousa Nunes / 2015 / A Survey on Human-in-the-Loop Applications Towards an Internet of All / IEEE Communications Surveys & Tutorials / Institute of Electrical and Electronics Engineers (IEEE) 17(2) : 944~965

[confproc] C. Xue / 2019 / Transferable AutoML by Model Sharing Over Grouped Datasets / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 9002~9011

[confproc] M. Terry / 2017 / The Data Linter: Lightweight, Automated Sanity Checking for ML Data Sets / Machine Learning Systems Workshop at NIPS

[journal] Yuji Roh / 2019 / A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective / IEEE Transactions on Knowledge and Data Engineering / Institute of Electrical and Electronics Engineers (IEEE) : 1~1

[other] Rekatsinas, Theodoros / 2017 / Holoclean: Holistic data repairs with probabilistic inference / arXiv preprint arXiv:1702.00820,

[confproc] S. Krishnan / 2016 / Activeclean : Interactive data cleaning for statistical modeling / Proceedings of the VLDB Endowment 9 : 948~959

[other] Krishnan, Sanjay / 2017 / Boostclean: Automated error detection and repair for machine learning / arXiv preprint arXiv:1711.01299

[confproc] K. H. Tae / 2019 / Data cleaning for accurate, fair, and robust models : A big data-AI integration approach / Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning : 1~4

[confproc] M. Dolatshah / 2018 / Cleaning crowdsourced labels using oracles for statistical classification / Proceedings of the VLDB Endowment 12 : 376~389

[book] Mahmoud Ghofrani / 2017 / Time Series and Renewable Energy Forecasting / intechopen

[journal] K. Bandara / 2020 / Forecasting across time series databases using recurrent neural networks on groups of similar series : A clustering approach / Expert Systems with Applications 140

[journal] Won-chang Lee / 2018 / Time Series Abnormal Data Detection for Smart Factory / International Journal of Control and Automation 11(1) : 91~98

[journal] J. Zaldivar / 2011 / Providing accident detection in vehicular networks through OBD-II devices and Android-based smartphones / IEEE Trans. on Local Computer Networks : 813~819

[journal] Smith / 2009 / On guiding the augmentation of an automated test suite via mutation analysis / Empirical software engineering : 341~369

[journal] L. Ma / 2018 / Deepmutation : Mutation testing of deep learning systems / IEEE Trans. on Software Reliability Engineering : 100~111

[confproc] C. Finn / 2016 / Guided cost learning : Deep inverse optimal control via policy optimization / International conference on machine learning : 49~58

[journal] F. P. Luus / 2015 / Multiview deep learning for land-use classification / IEEE Geoscience and Remote Sensing Letters : 2448~2452

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Cleaning Noises from Time Series Data with Memory Effects

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Cleaning Noises from Time Series Data with Memory Effects

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (2)

REFERENCES (18) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.