ON THE EFFECT OF KEYHOLE CHANNEL IN RSMA NETWORKS: A THEORETICAL OUTAGE ANALYSIS

(Received: 20-Feb.-2026, Revised: 16-Apr.-2026 , Accepted: 30-Apr.-2026)

Authors Hong-Nhu Nguyen, Phong-Cuong Ngo,

Keywords #Keyhole channel #Rate-splitting multiple access (RSMA) #Outage probability #Nakagami-m #Imperfect SIC #Diversity order #Non-orthogonal multiple access (NOMA)

Abstract This paper examines the theoretical performance of rate-splitting multiple access (RSMA) in keyhole fading channels. This rank-deficient environment is notorious for degrading the performance of traditional single-input single-output (SISO) systems. For a two-user downlink channel, exact and asymptotic outage probability results of RSMA with perfect and imperfect successive interference cancellation (pSIC and ipSIC) are derived. Closed- form solutions are derived by comparing the product of two independent Nakagami-m fading channels that model the keyhole effect. To further demonstrate RSMA's robustness in such an environment, we also examine the diversity order and unveil the effect of keyhole-induced rank deficiency on system reliability. Our results demonstrate that RSMA retains a performance edge over non-orthogonal multiple access (NOMA), especially with imperfect SIC or low SNR. Numerical results and Monte-Carlo simulations confirm the theoretical formulae and show that RSMA can combat the harmful effects of the keyhole channel better than other conventional schemes. The results confirm the promise of RSMA for future wireless systems operating in severe-fading environments.

References

[1] W. Jaafar et al., "On the Downlink Performance of RSMA-based UAV Communications," IEEE Trans. on Vehicular Technology, vol. 69, no. 12, pp. 16258-16263, 2020.

[2] F. Xiao et al., "Outage Performance Analysis of RSMA-aided Semi-grant-free Transmission Systems," IEEE Open J. of the Communications Society, vol. 4, pp. 253-268, 2023.

[3] T.-H. Vu et al., "On Performance of Downlink THz-based Rate-splitting Multiple-access (RSMA): Is It Always Better Than NOMA?," IEEE Trans. on Vehicular Techn., vol. 73, no. 3, pp. 4435-4440, 2024.

[4] P. Almers, f. Tufvesson and A. F. Molisch, "Keyhole Effect in MIMO Wireless Channels: Measurements and Theory," IEEE Trans. on Wireless Comm., vol. 5, no. 12, pp. 3596-3604, 2006.

[5] H. Zhang et al., "Performance Analysis of MIMO-HARQ Assisted V2V Communications with Keyhole Effect," IEEE Trans. on Comm., vol. 70, no. 5, pp. 3034-3046, May 2022.

[6] X. Zang et al., "On the Secrecy Performance of MIMOME Keyhole Channels Aided with Artificial Noise," IEICE Trans. on Comm., vol. E108-B, no. 5, pp. 631-639, May 2025.

[7] A. Krishnamoorthy and R. Schober, "Downlink MIMO-RSMA with Successive Null-space Precoding," IEEE Trans. on Wireless Communications, vol. 21, no. 11, pp. 9170-9185, 2022.

[8] O. Dizdar et al., "RSMA for Overloaded MIMO Networks: Low Complexity Design for Max-min Fairness," IEEE Trans. on Wireless Comm., vol. 23, no. 6, pp. 6156-6173, Jun. 2024.

[9] Y. Zhang et al., "Enhancing Secrecy in Hardware Impaired Cell-free Massive MIMO by RSMA," IEEE Trans. on Wireless Comm., vol. 23, no. 12, pp. 18788-18805, Dec. 2024.

[10] Y. Xu, Y. Mao, O. Dizdar and B. Clerckx, "Rate-Splitting Multiple Access With Finite Block Length for Short-packet and Low-latency Downlink Communications," IEEE Trans. on Vehicular Technology, vol. 71, no. 11, pp. 12333-12337, Nov. 2022.

[11] M. Katwe, K. Singh, B. Clerckx and C.-P. Li, "Rate Splitting Multiple Access for Energy Efficient RIS-aided Multi-user Short-packet Communications," Proc. of IEEE Global Communications Conf. (GLOBECOM) Workshops, pp. 644-649, Rio de Janeiro, Brazil, 2022.

[12] S. K. Singh, K. Agrawal, K. Singh, B. Clerckx and C.-P. Li, "RSMA Enhanced RIS-FD-UAV Aided Short Packet Communications under Imperfect SIC," Proc. of IEEE Global Communications Conf. (GLOBECOM) Workshops, pp. 1549-1554, Rio de Janeiro, Brazil, 2022.

[13] T.-H. Vu et al., "Rate-splitting Multiple Access-assisted THz-based Short-packet Communications," IEEE Wireless Communications Letters, vol. 12, no. 12, pp. 2218-2222, Dec. 2023.

[14] H. Shin and J. H. Lee, "Performance Analysis of Space-time Block Codes over Keyhole Nakagamim Fading Channels," IEEE Transactions on Vehicular Technology, vol. 53, no. 2, pp. 351-362, 2004.

[15] N. H. Tran, H. H. Nguyen and T. Le-Ngoc, "Performance Analysis and Design Criteria of BICMID with Signal Space Diversity for Keyhole Nakagami-m Fading Channels," IEEE Trans. on Information Theory, vol. 55, no. 4, pp. 1592-1602, Apr. 2009.

[16] C. Zhong et al., "Ergodic Mutual Information Analysis for Multi Keyhole MIMO Channels," IEEE Trans. on Wireless Communications, vol. 10, no. 6, pp. 1754-1763, Jun. 2011.

[17] L. K. S. Jayasinghe, N. Rajatheva, P. Dharmawansa and M. Latva-Aho, "Non-coherent Amplify-and Forward MIMO Relaying with OSTBC Over Rayleigh-Rician Fading Channels," IEEE Trans. on Vehicular Technology, vol. 62, no. 4, pp. 1610-1622, May 2013.

[18] A. M. Magableh et al., "Performance of Non-orthogonal Multiple Access (NOMA) Systems over N-Nakagami-m Multipath Fading Channels for 5G and Beyond," IEEE Trans. on Vehicular Techn., vol. 71, no. 11, pp. 11615-11623, Nov. 2022.

[19] B. Ji et al., "Research on Secure Transmission Performance of Electric Vehicles under Nakagami-m Channel," IEEE Trans. on Intelligent Transportation Systems, vol. 22, no. 3, pp. 1881-1891, 2021.

[20] Z. Yang et al., "Optimization of Rate Allocation and Power Control for Rate Splitting Multiple Access (RSMA)," IEEE Transactions on Communications, vol. 69, no. 9, pp. 5988-6002, Sep. 2021.

[21] A. Krishnamoorthy and R. Schober, "Downlink Massive MU-MIMO with Successively-regularized Zero Forcing Precoding," IEEE Wireless Communications Letters, vol. 12, no. 1, pp. 114-118, 2023.

[22] X. Yue and Y. Liu, "Performance Analysis of Intelligent Reflecting Surface Assisted NOMA Networks," IEEE Trans. on Wireless Communications, vol. 21, no. 4, pp. 2623-2636, Apr. 2022.

[23] D.-T. Do et al., "UAV Relaying Enabled NOMA Network With Hybrid Duplexing and Multiple Antennas," IEEE Access, vol. 8, pp. 186993-187007, 2020.

[24] D.-T. Do et al., "Antenna Selection and Device Grouping for Spectrum-efficient UAV-Assisted IoT Systems," IEEE Internet of Things Journal, vol. 10, no. 9, pp. 8014-8030, May 2023.

[25] M. Elsayed et al., "Symbiotic Ambient Backscatter IoT Transmission Over NOMA-enabled Network," Proc. of IEEE Int. Conf. on Communi. (ICC), pp. 1549-1554, Seoul, S. Korea, 2022.

[26] I. S. Gradshteyn and I. M. Ryzhik, Table of Integrals, Series and Products, ISBN-10: 0-12-373637-4, Academic Press, 2014.

[27] T.-T. T. Dao, S. Q. Nguyen, H. N. Nguyen, P. X. Nguyen and Y.-H. Kim, "Performance Evaluation of Downlink Multiple Users NOMA-Enable UAV-aided Communication Systems Over Nakagami-m Fading Environments," IEEE Access, vol. 9, pp. 151641-151653, 2021.

[28] X. Yue, Y. Liu, S. Kang, A. Nallanathan and Z. Ding, "Exploiting Full/Half-duplex User Relaying in NOMA Systems," IEEE Trans. on Communications, vol. 66, no. 2, pp. 560-575, Feb. 2018.

[29] N. Jaiswal and N. Purohit, "Performance Analysis of NOMA-enabled Vehicular Communication Systems with Transmit Antenna Selection over Double Nakagami-m Fading," IEEE Trans. on Vehicular Technology, vol. 70, no. 12, pp. 12725-12741, Dec. 2021.

[30] Y. Cheng, K. H. Li, Y. Liu, K. C. Teh and H. V. Poor, "Downlink and Uplink Intelligent Reflecting Surface Aided Networks: NOMA and OMA," IEEE Trans. on Wireless Communications, vol. 20, no. 6, pp. 3988-4000, Jun. 2021.

[31] M. Abramowitz and I. A. Stegun, Handbook of Mathematical Functions with Formulas, Graphs and Mathematical Tables, vol. 55, US Government Printing Office, 1948.