A MediaPipe and CNN-Based Framework for Real-Time Sign Language Recognition and Translation

Department of Computer Science, Vishwakarma Institute of Information Technology, Pune, Maharashtra, India. This email address is being protected from spambots. You need JavaScript enabled to view it. , This email address is being protected from spambots. You need JavaScript enabled to view it. , This email address is being protected from spambots. You need JavaScript enabled to view it. , This email address is being protected from spambots. You need JavaScript enabled to view it. , This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

This research is being aimed at the real-time sign language translation system developed in the absence of dedicated equipment like sensor gloves, under the sole observation of a standard camera and computer vision techniques. The project was divided into two major phases: Static Sign Recognition and Dynamic Sign Recognition. During the static phase, a fully original dataset comprising 120,000 images was created for 24 alphabets (A–Z excluding J and Z). Each image was then treated through two custom-designed filters: silhouette extraction and finger detection, carried out using MediaPipe for precise annotation and cropping on its own. The training of the static model happened with custom-designed CNN applied on the data to achieve accuracies of 94% (finger detection), 89% (silhouette), and 91% (combined). In the dynamic phase, signs with movement were captured using MediaPipe landmarks and registered as NumPy arrays (CSV and .NPY) files. Hand and facial keypoints were stored at every frame for temporal learning. This phase included one-hand, two-hand, and hand-with-face categories. The whole system achieved 15- 20 FPS real-time inference on CPU alone. While the static signs and single-handed dynamic signs were performing very well, the accuracy of complex two-handed and face-assisted gestures suffered due to scarcity and noise in the data. The results provide a very good baseline to improve upon for the future using deep temporal networks like LSTM or 3D CNNs.