https://jjcit.org/paper/292 10.5455/jjcit.71-1771168818 Hong-Nhu Nguyen,Phong-Cuong Ngo Keyhole channel,Rate-splitting multiple access (RSMA),Outage probability,Nakagami-m,Imperfect SIC,Diversity order,Non-orthogonal multiple access (NOMA) 5 1 20-Feb.-2026 16-Apr.-2026 30-Apr.-2026 This paper examines the theoretical performance of rate-splitting multiple access (RSMA) in keyhole fading channels. This rank-deficient environment is notorious for degrading the performance of traditional single-input single-output (SISO) systems. For a two-user downlink channel, exact and asymptotic outage probability results of RSMA with perfect and imperfect successive interference cancellation (pSIC and ipSIC) are derived. Closed- form solutions are derived by comparing the product of two independent Nakagami-m fading channels that model the keyhole effect. To further demonstrate RSMA's robustness in such an environment, we also examine the diversity order and unveil the effect of keyhole-induced rank deficiency on system reliability. Our results demonstrate that RSMA retains a performance edge over non-orthogonal multiple access (NOMA), especially with imperfect SIC or low SNR. Numerical results and Monte-Carlo simulations confirm the theoretical formulae and show that RSMA can combat the harmful effects of the keyhole channel better than other conventional schemes. The results confirm the promise of RSMA for future wireless systems operating in severe-fading environments.