An Accurate Log Object Recognition Technique (정확한 로그 객체 인식 기술)

Jiho Ju (주지호); Byungchul Tak (탁병철)

doi:10.9708/jksci.2023.28.02.089

An Accurate Log Object Recognition Technique

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2023, 28(2), pp.89-97
DOI : 10.9708/jksci.2023.28.02.089
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : January 16, 2023
Accepted : February 6, 2023
Published : February 28, 2023

Jiho Ju ¹, Byungchul Tak ¹

¹경북대학교

Accredited

ABSTRACT

In this paper, we propose factors that make log analysis difficult and design technique for detecting various objects embedded in the logs which helps in the subsequent analysis. In today’s IT systems, logs have become a critical source data for many advanced AI analysis techniques. Although logs contain wealth of useful information, it is difficult to directly apply techniques since logs are semi-structured by nature. The factors that interfere with log analysis are various objects such as file path, identifiers, JSON documents, etc. We have designed a BERT-based object pattern recognition algorithm for these objects and performed object identification. Object pattern recognition algorithms are based on object definition, GROK pattern, and regular expression. We find that simple pattern matchings based on known patterns and regular expressions are ineffective. The results show significantly better accuracy than using only the patterns and regular expressions. In addition, in the case of the BERT model, the accuracy of classifying objects reached as high as 99%.

KEYWORDS

Log Analysis, Pattern Recognition, GROK, Text classification, BERT

Citation status

* References for papers published after 2023 are currently being built.

[confproc] Debnath, Biplob / 2018 / LogLens: A real-time log analysis system / 2018 IEEE 38th international conference on distributed computing systems (ICDCS) : 1052~1062

[confproc] Alspaugh, Sara / 2014 / Analyzing log analysis: An empirical study of user log mining / 28th Large Installation System Administration Conference (LISA14) : 62~77

[confproc] Kulkarni, Juily / 2020 / Analysis of system logs for pattern detection and anomaly prediction / Proceeding of International Conference on Computational Science and Applications / Springer : 427~436

[confproc] Chaudhari, Swati / 2019 / Real time logs and traffic monitoring, analysis and visualization setup for IT security enhancement / 5th International Conference on Next Generation Computing Technologies (NGCT-2019) 2020

[confproc] J. Zhu / 2019 / Tools and Benchmarks for Automated Log Parsing / 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) : 121~130

[confproc] Tak, Byungchul / 2021 / Lognroll : discovering accurate log templates by iterative filtering / Proceedings of the 22nd International Middleware Conference : 273~285

[confproc] S. J. Son / 2018 / Performance of ELK stack and commercial system in security log analysis / 2017 IEEE 13th Malaysia International Conference on Communications (MICC) : 187~190

[confproc] M. Bajer / 2017 / Building an IoT Data Hub with Elasticsearch, Logstash and Kibana / 2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW) : 63~68

[confproc] A. F. Rochim / 2020 / Design Log Management System of Computer Network Devices Infrastructures Based on ELK Stack / 2019 International Conference on Electrical Engineering and Computer Science (ICECOS) : 338~342

[confproc] Jin, Zhongjun / 2020 / Auto-transform : learning-to-transform by patterns / Proceedings of the VLDB Endowment 13(12) : 2368~2381

[confproc] A. Leadbetter / 2017 / Where big data meets linked data: Applying standard data models to environmental data streams / 2016 IEEE International Conference on Big Data (Big Data) : 2929~2937

[web] Diotalevi, Tommaso / 2021 / Collection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre / arXiv / arXiv:2106.02612

[web] Devlin, Jacob / 2018 / Bert: Pre-training of deep bidirectional transformers for language understanding / arXiv / arXiv:1810.04805

[confproc] Guo, Haixuan / 2021 / Logbert: Log anomaly detection via bert / 2021 international joint conference on neural networks (IJCNN) / IEEE

[web] Lee, Yukyung / 2021 / LAnoBERT: System log anomaly detection based on BERT masked language model / arXiv / arXiv:2111.09564

[journal] Song Chen / 2022 / BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model / Applied Artificial Intelligence / Informa UK Limited 36(1) : 2145642~

[confproc] T. Prakash / 2016 / Geo-identification of web users through logs using ELK stack / 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence) : 606~610

[confproc] P. He / 2017 / Drain: An Online Log Parsing Approach with Fixed Depth Tree / 2017 IEEE International Conference on Web Services (ICWS) : 33~40

[confproc] R. Vaarandi / 2003 / A data clustering algorithm for mining patterns from event logs / Proceedings of the 3rd IEEE Workshop on IP Operations & Management(IPOM 2003)(IEEE Cat. No. 03EX764) / IEEE : 119~126

[confproc] Makanju / 2009 / Clustering event logs using iterative partitioning / Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining : 1255~1264

[confproc] R. Vaarandi / 2015 / LogCluster-A data clustering and pattern mining algorithm for event logs / 2015 11th International Conference on Network and Service Management (CNSM) : 1~7

[confproc] Q. Fu / 2009 / Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis / 2009 Ninth IEEE International Conference on Data Mining : 149~158

[confproc] S. Messaoudi / 2020 / A Search-Based Approach for Accurate Identification of Log Message Formats / 2018 IEEE/ACM 26th International Conference on Program Comprehension(ICPC) : 167~16710

[confproc] M. Du / 2017 / Spell: Streaming Parsing of System Event Logs / 2016 IEEE 16th International Conference on Data Mining (ICDM) : 859~864

[web] Shima, Keiichi. / 2016 / Length matters: Clustering system log messages using length of words / arXiv / arXiv:1611.03213

[confproc] H. Hamooni / 2016 / LogMine: fast pattern recognition for log analytics / CIKM : 1573~1582

[confproc] Tang / 2011 / LogSig:Generating system events from raw textual logs / Proceedings of the 20th ACM international conference on Information and knowledge management : 785~794

[confproc] Mizutani, Masayoshi / 2013 / Incremental mining of system log format / 2013 IEEE International Conference on Services Computing : 7~

[journal] Zhen Ming Jiang / 2008 / An automated approach for abstracting execution logs to execution events / Journal of Software Maintenance and Evolution: Research and Practice / Wiley 20(4) : 249~267

[confproc] He, Pinjia / 2016 / An evaluation study on log parsing and its use in log mining / 2016 46th annual IEEE/IFIP international conference on dependable systems and networks (DSN)

[confproc] Du, Min / 2017 / Deeplog: Anomaly detection and diagnosis from system logs through deep learning / Proceedings of the 2017ACM SIGSAC conference on computer and communications security

[confproc] Xu, Wei / 2009 / Detecting large-scale system problems by mining console logs / Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles : 117~132

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

An Accurate Log Object Recognition Technique

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

An Accurate Log Object Recognition Technique

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (32) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.