An Open-Source AI-Based Multilingual Translation Lecture Script System for Global University Lectures (대학 강의를 위한 Open-Source AI 기반 다국어 번역 강의 자막 시스템)

DongHyeon Shin (신동현); Oh-Gyu Kwon (권오규); ManKi Min (민만기); Minseo Yoon (윤민서); DoHyun Oh (오도현); Jae-Woo Ryu (류재우); Young Deok Park (박영덕)

doi:10.9708/jksci.2025.30.12.101

An Open-Source AI-Based Multilingual Translation Lecture Script System for Global University Lectures

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2025, 30(12), pp.101~112
DOI : 10.9708/jksci.2025.30.12.101
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : October 30, 2025
Accepted : December 2, 2025
Published : December 31, 2025

DongHyeon Shin ¹, Oh-Gyu Kwon ², ManKi Min ¹, Minseo Yoon ¹, DoHyun Oh ¹, Jae-Woo Ryu ³, Young Deok Park ¹

¹영남대학교
²모비핀테크놀러지
³니어네트웍스

Accredited

ABSTRACT

We propose an open-source Artificial Intelligence (AI)-driven multilingual translation system that integrates real-time speech recognition and translation to resolve language barriers and misrecognition of technical terms in university lecture environments. The proposed system combines Whisper-based speech recognition, Bidirectional Encoder Representations from Transformers (BERT) and Levenshtein-distance based correction algorithm, and the DeltaLM multilingual translation model to automate the entire process from noise reduction and sentence refinement to context-aware translation and lecture script generation. In particular, a translation dataset containing domain-specific technical terms in the field of computer engineering is constructed and fine-tuned to improve translation precision in specialized domains. Experimental results show that the Word Error Rate (WER) decreases from 9.2 % to 4.7 %, achieving an improvement of approximately 51 %, while the average Bilingual Evaluation Understudy (BLEU) score increases from 56.3 to 60.2, corresponding to a 6.9 % performance gain. These results confirm that the proposed system achieves consistent translation quality improvements across all language pairs in academic lecture scenarios.

KEYWORDS

Educational technology, Fine-tuning, Open-Source AI, Speech Recognition, Translation

Journal Copyright Policy

No CCL information provided

Citation status

* References for papers published after 2025 are currently being built.

[report] Korea Education and Research Information Service / 2023 / University Distance Education Status and Satisfaction Survey (2022-2 Semester Report) / Korea Education and Research Information Service

[journal] J. Won / 2019 / How Foreign Students Perceive English-mediated Classes in Korea / Learner-Centered Curriculum and Instruction Research 19(21) : 377~406

[report] Ministry of Education / 2024 / Status of Foreign Students in Domestic Higher Education Institutions (1999-2024) / Korea Education Statistics Service

[journal] X. Li / 2023 / Towards deeper learning in EMI lectures: The role of cognitive load / J. Further Higher Educ. 48(2) : 205~221

[journal] S. Liao / 2020 / The impact of monolingual and bilingual subtitles on visual attention, cognitive load, and comprehension / J. Specialised Translation (33) : 70~98

[journal] E. N. Casey / 2025 / Descriptive analysis of interpreter service mode costs and usage in Northwestern Wisconsin pre-and peri-COVID-19 / BMC Health Serv. Res. 25(1) : 1~9

[journal] K. Kuhn / 2024 / Measuring the accuracy of automatic speech recognition solutions / ACM Trans. Accessible Comput. 16(4) : 1~23

[journal] N. Sethiya / 2025 / End-to-end speech-to-text translation: A survey / Comput. Speech Lang. 90 : 101751~

[confproc] A. Rafiei Oskooei / 2025 / Whisper, translate, speak, sync: Video translation for multilingual video conferencing using generative AI / Proc. Int. Conf. Comput. Sci. Its Appl. / Springer Nature : 217~234

[other] A. A. Attia / 2024 / CPT-boosted Wav2vec 2.0: Towards noise-robust speech recognition for classroom environments

[other] J. Wang / 2019 / Exploring lexicon-free modeling units for end-to-end Korean and Korean-English code-switching speech recognition / arXiv preprint arXiv:1910.11590

[confproc] E. Matusov / 2008 / Automatic sentence segmentation and punctuation prediction for spoken language translation / Proc. IEEE ICASSP : 353~356

[journal] Y. Ephraim / 2003 / Speech enhancement using a minimum mean-square error log-spectral amplitude estimator / IEEE Trans. Acoust., Speech, Signal Process. 33(2) : 443~445

[journal] Z.-Q. Wang / 2022 / STFT-domain neural speech enhancement with very low algorithmic latency / IEEE/ACM Trans. Audio, Speech, Lang. Process. 30 / https://zqwang7.github.io/publications/TASLP2022_STFTlowlat.pdf

[journal] R. J. Nasir / 2025 / A hybrid method for speech noise reduction using log-MMSE / Iraqi J. Sci. : 860~875

[other] I. López-Espejo / 2024 / On speech pre-emphasis as a simple and inexpensive method to boost speech enhancement / arXiv preprint

[confproc] E. Loweimi / 2013 / On the importance of pre-emphasis and window shape in phase-based speech recognition / Advances in Nonlinear Speech Process. / Springer 7911 : 160~167

[confproc] A. Radford / 2023 / Robust speech recognition via large-scale weak supervision / Proc. 40th Int. Conf. Mach. Learn. 202 : 28492~28518 / https://arxiv.org/abs/2212.04356

[other] A. Rao / 2023 / Transcribing educational videos using Whisper: A preliminary study on using AI for transcribing educational videos

[other] Anonymous / 2024 / Improving domain-specific ASR with LLMgenerated contextual descriptions

[other] M. Barański / 2025 / Investigation of Whisper ASR hallucinations induced by non-speech audio

[confproc] J. Devlin / 2019 / BERT: Pre-training of deep bidirectional transformers for language understanding / Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL-HLT) : 4171~4186

[journal] H. Yu / 2025 / Krongbert: Enhanced factorization based morphological approach for the korean pretrained language model / Information Processing & Management 62(3) : 104072~

[other] T. Kim / 2025 / Kogec: Korean grammatical error correction with pre-trained translation models

[journal] D. Min / 2024 / A Study on Improving the Accuracy of Korean Speech Recognition Texts Using KcBERT / Journal of KIISE 51(12) : 1115~1124

[other] S. Ma / 2021 / DeltaLM: Encoderdecoder pre-training for language generation and translation by augmenting pretrained multilingual encoders

[journal] V. I. Levenshtein / 1966 / Binary codes capable of correcting deletions, insertions, and reversals / Sov. Phys. Dokl. 10 : 707~710

[web] / Korean Single Speaker Speech Dataset / Kaggle / https://www.kaggle.com/datasets/bryanpark/korean-single-speaker-speech-dataset

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2025 KCI Impact Factor : 1.01

An Open-Source AI-Based Multilingual Translation Lecture Script System for Global University Lectures

ABSTRACT

KEYWORDS

Journal Copyright Policy

Citation status

* References for papers published after 2025 are currently being built.

Journal of The Korea Society of Computer and Information 2025 KCI Impact Factor : 1.01

An Open-Source AI-Based Multilingual Translation Lecture Script System for Global University Lectures

ABSTRACT

KEYWORDS

Journal Copyright Policy

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (28) * References for papers published after 2025 are currently being built.

Search PDF

Citation

* References for papers published after 2025 are currently being built.