본문 바로가기
  • Home
Basic Search

Music Sense: A Mobile Platform for Multimodal Music Perception for Individuals with Hearing Impairment

  • Journal of The Korea Society of Computer and Information
  • Abbr : JKSCI
  • 2026, 31(6), pp.175~184
  • Publisher : The Korean Society Of Computer And Information
  • Research Area : Engineering > Computer Science
  • Received : May 26, 2026
  • Accepted : June 16, 2026
  • Published : June 30, 2026

Sangyeop Kim 1,  Mirim Kim 1,  Jinwook Kim 2,  Jia Yoo 3,  Jeabum Lim 4,  Pilsang Jung 4,  Won Joo Lee 5

1숭실대학교
2한양대학교
3세종대학교
4서울대학교
5인하공업전문대학

Accredited

ABSTRACT

In this paper, we propose a Music Sense, a mobile platform–based application designed for individuals with hearing impairment and hard-of-hearing. The system adopts a client–server architecture in which computationally intensive AI-based audio analysis is performed on a cloud server, while the mobile client focuses on rendering synchronized visual and haptic feedback during playback. The server converts audio signals into time-series metadata through music source separation, pitch estimation, beat and onset detection, and phoneme-level lyric alignment. The generated metadata are normalized across multiple modalities and streamed in real time to mobile devices, where they are rendered as subtitles and visual representations of pitch and rhythm. Field evaluations conducted with children with hearing impairment demonstrate that the proposed application effectively supports rhythm reproduction, melody tracking, and the development of therapeutic music content. Furthermore, the proposed framework standardizes the conversion of music into reusable metadata and enables platform-level deployment via APIs and SDKs without modifying the original audio content, highlighting its scalability and practical applicability.

Statistics

  • 8 Viewed
  • 0 Downloaded

Tools

Issue List

Citation status

* References for papers published after 2024 are currently being built.

Basic Search Advanced Search