End-to-End AI-Native Physical Layer for Robust 6G MIMO-OFDM: Adaptive Constellation Shaping and Attention-Based Detection

Fateh Bouguerra; Imed Benacer; Lamir Saidi

doi:10.48084/etasr.19842

Authors

Fateh Bouguerra Department of Electronics, University of Batna 2, Algeria
Imed Benacer Department of Telecommunication and Networking, Institute of Science and Applied Technology, University of Oum El Bouaghi, Algeria
Lamir Saidi Department of Electronics, University of Batna 2, Algeria

Volume: 16 | Issue: 4 | Pages: 37642-37649 | August 2026 | https://doi.org/10.48084/etasr.19842

Received: 7 May 2026 | Revised: 2 June 2026 and 8 June 2026 | Accepted: 11 June 2026 | Online: 13 June 2026

Corresponding author: Fateh Bouguerra

Abstract

Sixth-generation (6G) physical-layer designs require robustness against non-analytical channel distortions and hardware impairments that violate classical linear assumptions. This paper presents an AI-native framework for MIMO-OFDM systems that jointly optimizes adaptive constellation shaping and neural detection through end-to-end learning. The proposed model employs a differentiable channel layer incorporating Rayleigh fading, power amplifier nonlinearities, and phase noise, enabling gradient-based optimization of complex constellation coordinates under strict average power constraints. The receiver utilizes a Real-Valued Feedforward Neural Network with Spatial Attention (RFNN-SA) to dynamically weight fading streams and mitigate channel estimation errors. Extensive simulations demonstrate that the proposed model achieves a 3.2 dB SNR gain at BER=10⁻³ for 16-QAM, 3.8 dB for 64-QAM, and 4.1 dB at BER=10⁻² for 256-QAM over MMSE detection at equivalent operating points. Under realistic CSI uncertainty, performance degrades by only 22%, compared to 52% for classical baselines. With a 0.95 ms physical‑layer detection inference latency, the proposed architecture provides a computationally efficient and impairment-resilient foundation for practical 6G physical-layer deployments.

Keywords:

6G, AI-native physical layer, adaptive constellation shaping, MIMO-OFDM, neural detection, Rayleigh fading, end-to-end learning

References

[1] W. Saad, M. Bennis, and M. Chen, "A Vision of 6G Wireless Systems: Applications, Trends, Technologies, and Open Research Problems," IEEE Network, vol. 34, no. 3, pp. 134–142, Feb. 2020.

[2] F. Liu et al., "Integrated Sensing and Communications: Toward Dual-Functional Wireless Networks for 6G and Beyond," IEEE Journal on Selected Areas in Communications, vol. 40, no. 6, pp. 1728–1767, June 2022.

[3] E. G. Larsson, O. Edfors, F. Tufvesson, and T. L. Marzetta, "Massive MIMO for next generation wireless systems," IEEE Communications Magazine, vol. 52, no. 2, pp. 186–195, Feb. 2014.

[4] T. O’Shea and J. Hoydis, "An Introduction to Deep Learning for the Physical Layer," IEEE Transactions on Cognitive Communications and Networking, vol. 3, no. 4, pp. 563–575, Sept. 2017.

[5] S. Dörner, S. Cammerer, J. Hoydis, and S. ten Brink, "Deep Learning Based Communication Over the Air," IEEE Journal of Selected Topics in Signal Processing, vol. 12, no. 1, pp. 132–143, Oct. 2018.

[6] F. A. Aoudia and J. Hoydis, "End-to-End Learning of Communications Systems Without a Channel Model," in 2018 52nd Asilomar Conference on Signals, Systems, and Computers, Oct. 2018, pp. 298–303.

[7] S. Cammerer, F. A. Aoudia, S. Dörner, M. Stark, J. Hoydis, and S. ten Brink, "Trainable Communication Systems: Concepts and Prototype," IEEE Transactions on Communications, vol. 68, no. 9, pp. 5489–5503, Sept. 2020.

[8] B. N. Getu, A. Al-Ataby, and H. Attia, "Deep Learning-Based Signal Classification in Wireless Fading Channels," Engineering, Technology & Applied Science Research, vol. 15, no. 6, pp. 30296–30303, Dec. 2025.

[9] F. Bouguerra and L. Saidi, "An Eﬃcient ANN Interference Cancelation for High Order Modulation over Rayleigh Fading Channel," Journal of Telecommunications and Information Technology, vol. 4, no. 2018, pp. 75–80, Jan. 2019.

[10] A. Vaswani et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, pp. 5998–6008, 2017.

[11] J. Hoydis et al., "Sionna: An Open-Source Library for Next-Generation Physical Layer Research." arXiv, Mar. 20, 2023.

[12] Z. Jiang et al., "Recursive Flow: A Generative Framework for MIMO Channel Estimation." arXiv, Jan. 23, 2026.

[13] Z. Jiang et al., "One-Step Generative Channel Estimation via Average Velocity Field." arXiv, Jan. 23, 2026.

[14] N. Islam and S. Shin, "Deep Learning in Physical Layer: Review on Data Driven End-to-End Communication Systems and Their Enabling Semantic Applications," IEEE Open Journal of the Communications Society, vol. 5, pp. 4207–4240, 2024.

[15] A. Zaib, M. Masood, A. Ali, W. Xu, and T. Y. Al-Naffouri, "Distributed Channel Estimation and Pilot Contamination Analysis for Massive MIMO-OFDM Systems," IEEE Transactions on Communications, vol. 64, no. 11, pp. 4607–4621, Aug. 2016.

[16] "Study on channel model for frequencies from 0.5 to 100 GHz," 3GPP, Technical Report 38.901, 2017.

[17] C. Rapp, "Effects of HPA-nonlinearity on a 4-DPSK/OFDM-signal for a digital sound broadcasting signal," in ESA, Second European Conference on Satellite Communications (ECSC-2), Oct. 1991, vol. 332, pp. 179–184.

[18] B. Hassibi and H. Vikalo, "On the sphere-decoding algorithm I. Expected complexity," IEEE Transactions on Signal Processing, vol. 53, no. 8, pp. 2806–2818, Dec. 2005.

[19] G. Böcherer, F. Steiner, and P. Schulte, "Bandwidth Efficient and Rate-Matched Low-Density Parity-Check Coded Modulation," IEEE Transactions on Communications, vol. 63, no. 12, pp. 4651–4665, Sept. 2015.

[20] S. Zheng, S. Chen, and X. Yang, "DeepReceiver: A Deep Learning-Based Intelligent Receiver for Wireless Communications in the Physical Layer," IEEE Transactions on Cognitive Communications and Networking, vol. 7, no. 1, pp. 5–20, Mar. 2021.