KJCKorea
Journal Central

@article{ART002398197},
author={Byungchul Tak},
title={Accurate and Efficient Log Template Discovery Technique},
journal={Journal of The Korea Society of Computer and Information},
issn={1598-849X},
year={2018},
volume={23},
number={10},
pages={11-21},
doi={10.9708/jksci.2018.23.10.011}

TY - JOUR
AU - Byungchul Tak
TI - Accurate and Efficient Log Template Discovery Technique
JO - Journal of The Korea Society of Computer and Information
PY - 2018
VL - 23
IS - 10
PB - The Korean Society Of Computer And Information
SP - 11
EP - 21
SN - 1598-849X
AB - In this paper we propose a novel log template discovery algorithm which achieves high quality of discovered log templates through iterative log filtering technique. Log templates are the static string pattern of logs that are used to produce actual logs by inserting variable values during runtime. Identifying individual logs into their template category correctly enables us to conduct automated analysis using state-of-the-art machine learning techniques. Our technique looks at the group of logs column-wise and filters the logs that have the value of the highest proportion. We repeat this process per each column until we are left with highly homogeneous set of logs that most likely belong to the same log template category. Then, we determine which column is the static part and which is the variable part by vertically comparing all the logs in the group. This process repeats until we have discovered all the templates from given logs. Also, during this process we discover the custom patterns such as ID formats that are unique to the application. This information helps us quickly identify such strings in the logs as variable parts thereby further increasing the accuracy of the discovered log templates. Existing solutions suffer from log templates being too general or too specific because of the inability to detect custom patterns. Through extensive evaluations we have learned that our proposed method achieves 2 to 20 times better accuracy.
KW - Log template;Log analysis;Log parsing
DO - 10.9708/jksci.2018.23.10.011
ER -

Byungchul Tak. (2018). Accurate and Efficient Log Template Discovery Technique. Journal of The Korea Society of Computer and Information, 23(10), 11-21.

Byungchul Tak. 2018, "Accurate and Efficient Log Template Discovery Technique", Journal of The Korea Society of Computer and Information, vol.23, no.10 pp.11-21. Available from: doi:10.9708/jksci.2018.23.10.011

Byungchul Tak "Accurate and Efficient Log Template Discovery Technique" Journal of The Korea Society of Computer and Information 23.10 pp.11-21 (2018) : 11.

Byungchul Tak. Accurate and Efficient Log Template Discovery Technique. 2018; 23(10), 11-21. Available from: doi:10.9708/jksci.2018.23.10.011

Byungchul Tak. "Accurate and Efficient Log Template Discovery Technique" Journal of The Korea Society of Computer and Information 23, no.10 (2018) : 11-21.doi: 10.9708/jksci.2018.23.10.011

Byungchul Tak. Accurate and Efficient Log Template Discovery Technique. Journal of The Korea Society of Computer and Information, 23(10), 11-21. doi: 10.9708/jksci.2018.23.10.011

Byungchul Tak. Accurate and Efficient Log Template Discovery Technique. Journal of The Korea Society of Computer and Information. 2018; 23(10) 11-21. doi: 10.9708/jksci.2018.23.10.011

Byungchul Tak. Accurate and Efficient Log Template Discovery Technique. 2018; 23(10), 11-21. Available from: doi:10.9708/jksci.2018.23.10.011

Byungchul Tak. "Accurate and Efficient Log Template Discovery Technique" Journal of The Korea Society of Computer and Information 23, no.10 (2018) : 11-21.doi: 10.9708/jksci.2018.23.10.011