Abstract :

Recognizing Hand Gestures Performed In Mid-air Enables An Intuitive And Contactless Mode Of Human–computer Interaction. This Project Presents AeroScribe, A Real-time Hand Gesture Recognition System That Combines MediaPipe-based Hand Tracking With A Custom Category-Aware MobileNetV2 Model To Identify A Comprehensive Set Of 81 Gestures, Including Numbers, Uppercase And Lowercase Letters, And Symbols. The System Operates Using A Standard RGB Camera And Detects 21 Hand Keypoints Without The Need For Markers, Which Are Utilized To Accurately Extract Hand Regions From Each Frame. These Regions Are Processed Through A Dual-branch Deep Learning Architecture Incorporating A Category-aware Mechanism That Initially Classifies Gestures Into Four High-level Groups Numbers, Uppercase, Lowercase, And Symbols Thereby Reducing Confusion Between Visually Similar Gestures Such As ‘0’ And ‘O’. Model Performance Is Further Enhanced Through A Tailored Training Strategy That Employs A Custom Loss Function With Category-based Penalties, Class-balanced Weighting, And A Two-phase Training Approach. Experimental Evaluation On A Diverse 81-class Dataset Demonstrates That AeroScribe Achieves An Accuracy Of 96.89% While Sustaining Real-time Performance At Over 30 Frames Per Second On Standard Hardware. Additionally, The System Provides Multimodal Feedback Through A Graphical User Interface And Text-to-speech Output, Making It A Reliable And Accessible Assistive Interaction Solution. Keywords—Air-writing Recognition, Hand Gesture Classification, MobileNetV2, MediaPipe, Transfer Learning, Assistive Technology, Computer Vision, Deep Learning.

Published:

01-4-2026

Issue:

Vol. 26 No. 4 (2026)

Page Nos:

56-65

Section:

Articles

License:

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

How to Cite