With an increasing number of international students in Korean universities, communication breakdowns among international students, domestic students, and faculty members have been an important issue. However, their language ability and difficulties have received little attention. Since English is a lingua franca, this study evaluated their English oral communication ability by designing a post-entry oral performance assessment (PEOPA), conducting a pilot test, and investigating the validity of PEOPA based on Toulmin’s (2003) argument structure. Firstly, a survey was conducted with 15 international students studying at a Korean university to ask about their English use and identify the target language use (TLU) domain. Among them, 11 took pilot PEOPA. A many-Facet Rasch analysis indicated that PEOPA tasks stably assessed the oral communication ability of participants and reliably separated them into different ability levels. Raters consistently assessed performances according to a scoring rubric reflecting intended construction, with each scoring category measuring distinct aspects of oral communication ability. Along with Rasch analysis, qualitative analysis of participants’ transcribed performances provided further backing for test validity. Ensured validity of the assessment allowed us to propose the implementation of PEOPA in universities in similar contexts. Finally, the implications of findings for the diagnosis phase of PEOPA are discussed.