Out-Of-Domain Detection Using Hierarchical Dirichlet Process (계층적 디리클레 프로세스를 활용한미지원 도메인 검출)

Young-Seob Jeong (정영섭)

doi:10.9708/jksci.2018.23.01.017

Out-Of-Domain Detection Using Hierarchical Dirichlet Process

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2018, 23(1), pp.17-24
DOI : 10.9708/jksci.2018.23.01.017
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : November 13, 2017
Accepted : January 10, 2018
Published : January 31, 2018

Young-Seob Jeong ¹

¹순천향대학교

Accredited

ABSTRACT

With improvement of speech recognition and natural language processing, dialog systems are recently adapted to various service domains. It became possible to get desirable services by conversation through the dialog system, but it is still necessary to improve separate modules, such as domain detection, intention detection, named entity recognition, and out-of-domain detection, in order to achieve stable service offer. When it misclassifies an in-domain sentence of conversation as out-of-domain, it will result in poor customer satisfaction and finally lost business. As there have been relatively small number of studies related to the out-of-domain detection, in this paper, we introduce a new method using a hierarchical Dirichlet process and demonstrate the effectiveness of it by experimental results on Korean dataset.

KEYWORDS

Out-of-domain detection, Dialog system, Topic modeling, Hierarchical Dirichlet process

Citation status

* References for papers published after 2023 are currently being built.

[web] / Amazon Echo / https://www.amazon.com

[web] / SK Telecom Nugu / http://www.nugu.co.kr/main

[web] / Naver WAVE / https://clova.ai/ko

[web] / Naver AWAY IVI platform / http://www.naverlabs.com/newsroom/IVI.html

[web] / Hyundai-card Buddy / www.hyundaicard.com/

[web] / Naver search / https://www.naver.com

[web] / Google search / https://www.google.com

[confproc] Lidia Mangu. / 2000 / Lattice Compression in the Consensual Post-Processing Framework / Proceedings of the Third World Multiconference on Systemics, Cybernetics and Informatics : 246~252

[confproc] Ian R. Lane / 2004 / Out-Of-Domain Detection Based on Confidence Measures From Multiple Topic Classificati on / Proceedings of the International Conference on Acoustics, Speech, and Signal Processing : 757~760

[confproc] Seonghan Ryu. / 2014 / Exploiting Out-Of-Vocabulary Words For Out-Of-Domain Detection in Dialog Systems / Proceedings of the International Conference on Big Data and Smart Computing : 165~168

[other] Deirdre Hogan / 2013 / SMS Normalisation, Retrieval and Out-of-Domain Detection Approaches for SMS-Based FAQ Retrieval / Multilingual Information Access in South Asian Languages : 184~196

[confproc] Mikio Nakano / 2011 / A Two-Stage Domain Selection Framework for Extensible Multi-Domain Spoken Dialogue Systems / Proceedings of the 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue : 18~29

[confproc] Yoko Fujita / 2011 / Out-of-Task Utterance Detection Based on Bag-of-Words Using Automatic Speech Recognition Results / Proceedings of the third Annual Summit and Conference of Asia-Pacific Signal and Information Processing Association

[web] / Wordnet / https://wordnet.princeton.edu/

[confproc] Young-Seob Jeong / 2017 / Experimental Analysis for Out-Of-Domain Detection Using Features of Word Positions in Sentences / Proceedings of the Spring Conference of Korean Society for Internet Information

[book] Trevor Hastie / 2003 / The elements of statistical learning / Springer

[journal] Scott Deerwester / 1990 / Indexing by Latent Semantic Analysis / Journal of the American Society for Information Science (JASIS) 41(6) : 391~407

[confproc] Bernhard E. Boser / 1992 / A Training Algorithm For Optimal Margin Classifiers / Proceedings of the fifth Annual Workshop on Computational Learning Theory : 144~152

[book] David A. Freedman / 2009 / Statistical Models: Theory and Practice

[confproc] Ian R. Lane / 2004 / Example-Based Training of Dialogue Planning Incorporating User and Situation Models / Proceedings of the 8th International Conference on Spoken Language Processing

[journal] Adam L. Berger / 1996 / A Maximum Entropy Approach to Natural Language Processing / Computational Linguistics 22(1) : 39~71

[journal] Seonghan Ryu / 2017 / Neural Sentence Embedding Using Only In-Domain Sentences For Out-Of-Domain Sentence Detection in Dialog Systems / Pattern Recognition Letters 88 : 26~32

[journal] Yee Whye Teh / 2006 / Hierarchical Dirichlet Processes / Journal of the American Statistical Association 101(476) : 1566~1581

[journal] David M. Blei / 2003 / Latent Dirichlet Allocation / Journal of Machine Learning Research 3 : 993~1022

[journal] Machal Rosen-Zvi / 2010 / Learning Author-Topic Models / ACM Transactions on Information Systems 28(1) : 1~38

[confproc] Young-Seob Jeong. / 2012 / Sequential Entity Group Topic Model for Getting Topic Flows of Entity Groups Within One Document / Proceedings of the 16th Pacific-Asia Conference on Knowledge Discovery and Data Mining : 366~378

[confproc] Thomas L. Griffiths. / 2004 / Finding Scientific Topics / Proceedings of the National Academy of Sciences of the United States of America : 5228~5235

[journal] Leo Breiman / 2001 / Random Forests / Machine Learning 45(1) : 5~32

[confproc] Ian R. Lane. / 2005 / Incorporating Dialogue Context and Topic Clustering in Out-of-Domain Detection / Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing : 1045~1048

[book] John C. Platt / 1998 / Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines;Advances in Kernel Methods – Support Vector Learning / MIT Press

[journal] Sepp Hochreiter. / 1997 / Long Short-term Memory / Neural Computation 9(8) : 1735~1780

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Out-Of-Domain Detection Using Hierarchical Dirichlet Process

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Out-Of-Domain Detection Using Hierarchical Dirichlet Process

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (2)

REFERENCES (31) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.