ISSN: 3048-6807 An Open‑Access, Peer‑Reviewed International Research Journal
Open Access

Multilingual Sign Language Translation using Computer Vision and Generative AI

Author(s): Chichula Abhishek,Kotilingala Jeevitha,Attada Anvek, Gudla Sai Lahari, Dr Ajit Kumar Rout

Affiliation: Department of Information Technology, GMRIT Deemed to be University, Rajam, Andhra Pradesh, India ,Professor, Department of Information Technology, GMRIT Deemed to be University, Rajam, Andhra Pradesh, India Email: abhishekchichula3@gmail.com.

Page No: 41-55

Volume, Issue & Publishing Year: Volume 3 Issue 11,Nov-2026

Journal: International Journal of Advanced Engineering Application (IJAEA)

ISSN NO: 3048-6807

Abstract:
There is a great difficulty in communicating between Indian sign language users and nonusers. The paper introduces a multilingual sign language translation framework, which is based on computer vision, deep learning, and Generative AI to facilitate communication. The sign recognition module is responsible for capturing data from the video using OpenCV, detecting the hands in the image using MediaPipe Hands, extracting the spatial features using EfficientNet-B0 and learning the temporal features using Bidirectional Gated Recurrent Unit network. For the recognition model, the training and evaluation set consisted of INCLUDE which contains a total of 262 ISL sign classes. 32 frames were extracted for each input video sequence and 1,280 dimensional feature vectors were used. The performance obtained from the experimental evaluation for 619 samples was found to be 86.75% with 537 correct predictions. The framework also includes a user-controlled gloss buffer where in known sign labels are gathered and sent for sentence generation when the user has manually triggered translation. After the user manually triggers translation, Generative AI is used to convert the buffered sign labels into grammatically meaningful sentences. The generated sentences are then translated into English, Telugu, Hindi, and Tamil using the multilingual translation module and converted into speech using the Text-to-Speech module.
The following components for additional sentence generation, multilingual translation and speech are part of the framework, but have not been quantitatively assessed in this study. The results confirm the viability of the proposed architecture for the recognition of isolated signs using the ISL, and serve as a basis for developing the system towards multilingual sign language communication.

Keywords: Indian Sign Language, Isolated Sign Recognition, Computer Vision, EfficientNet-B0, Bidirectional GRU, Generative AI, Multilingual Translation, Text-to-Speech.

Reference:

  • [1] M. Geetha, N. Aloysius, D. A. Somasundaran, A. Raghunath, and P. Nedungadi, “Toward real-time recognition of continuous Indian Sign Language: A multi-modal approach using RGB and pose,” IEEE Access, vol. 13, pp. 60270–60283, 2025.
  • [2] Z. Wang, D. Li, R. Jiang, and M. Okumura, “Continuous sign language recognition with multi-scale spatial-temporal feature enhancement,” IEEE Access, vol. 13, pp. 5491–5506, 2025.
  • [3] M. Kumar, S. S. Visagan, T. Mahajan, A. Natarajan, and P. S. Sreeja, "Enhanced Sign Language Translation Between American Sign Language and Indian Sign Language Using LLMs," IEEE Access, vol. 13, pp. 156270–156284, 2025.
  • [4] K. A. Shams, M. R. Reaz, M. R. U. Rafi, S. Islam, M. S. Rahman, R. Rahman, M. T. Reza, M. Z. Parvez, S. Chakraborty, B. Pradhan, and A. Alamri, “MultiModal Ensemble Approach Leveraging Spatial, Skeletal, and Edge Features for Enhanced Bangla Sign Language Recognition,” IEEE Access, vol. 12, pp. 83638–83657, 2024, doi: 10.1109/ACCESS.2024.3410837.
  • [5] M. Y. Novopoltsev, A. Tulenkov, R. Murtazin, R. I. Akhidov, I. Zemtsova, E. E. Bojarskaja, D. N. Bondarenko, A. V. Savchenko, and I. Makarov, “From Data to Signs: A Foundation Model for Multilingual Sign Language Recognition,” IEEE Access, vol. 13, pp. 188170–188181, 2025, doi: 10.1109/ACCESS.2025.3627130.
  • [6] Y. Said, S. Boubaker, S. M. Altowaijri, A. A. Alsheikhy, and M. Atri, "Adaptive Transformer-Based Deep Learning Framework for Continuous Sign Language Recognition and Translation," Mathematics, vol. 13, no. 6, Art. no. 909, Mar. 2025
  • [7] Z. Liu, J. Wu, Z. Shen, X. Chen, Q. Wu, Z. Gui, L. Sen, and H. Shu, "Improving End-to-End Sign Language Translation With Adaptive Video Representation Enhanced Transformer," IEEE Transactions on Circuits and Systems for Video Technology, 2024.
  • [8] F. M. Najib, “Sign Language Interpretation Using Machine Learning and Artificial Intelligence,” Neural Computing and Applications, vol. 37, no. 2, pp. 841–857, 2025, doi: 10.1007/s00521-024-10395-9.
  • [9] X. Shen, L. Shen, S. Yuan, H. Du, H. Sun, and X. Yu, "Diverse Sign Language Translation," International Journal of Computer Vision, vol. 134, p. 329, 2026.
  • [10] F. M. Talaat, W. El-Shafai, N. F. Soliman, A. D. Algarni, F. E. Abd El-Samie, and A. I. Siam, "Real-time Arabic avatar for deaf-mute communication enabled by deep learning sign language translation," Computers and Electrical Engineering, vol. 119, p. 109475, 2024.
  • [11] S. K. Gadarla and S. B. Mane, “Multilingual Sign Language Recognition,” TechRxiv preprint, 2024.
  • [12] N. Shahin and L. Ismail, “From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation,” Artificial Intelligence Review, vol. 57, no. 10, Art. no. 271, 2024, doi: 10.1007/s10462-024-10895-z.
  • [13] N. Virag and J. Katona, “AI-powered real-time hand sign recognition: Enhancing communication for the hearing- and speech-impaired community,” Universal Access in the Information Society, vol. 25, p. 73, 2026.
  • [14] M. Alaftekin et al., “Real-time sign language recognition based on YOLO algorithm,” Neural Computing and Applications, vol. 36, pp. 7609–7624, 2024.
  • [15] I. M. M. Violet and R. Leena Sri, “A Comprehensive Framework on Recent Advances and Challenges in Sign Language Recognition Systems,” Discover Artificial Intelligence, vol. 5, p. 419, 2025.
  • [16] A. Sridhar, R. G. Ganesan, P. Kumar, and M. Khapra, “INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition,” in Proceedings of the 28th ACM International Conference on Multimedia (MM ’20), 2020, pp. 1366–1375, doi: 10.1145/3394171.3413528.