SignVision + Air-Pen

Communication barriers faced by people with hearing or speech impairments remain a major challenge in daily life. Most sign language communication systems are expensive, hardware-dependent, limited to small gesture sets, or not interactive enough for real-world usage. Traditional systems struggle with real-time responsiveness, accurate gesture recognition under varying lighting and background conditions, natural sentence formation, accessibility for multilingual users, and writing text without physical contact devices. Very few systems combine sign language recognition, speech generation, translation, and gesture-based writing into one unified platform.

Hand gestures can serve two powerful purposes simultaneously: communication through sign language and writing through motion in air. Instead of building only a static sign detector, the project was designed as a complete interactive communication system — detecting hand signs in real time, converting gestures into meaningful text, improving usability through phrase prediction and transcript stabilization, speaking the generated text aloud, translating communication into Hindi/English, and allowing users to write in air using natural finger motion. The project focuses not only on AI accuracy, but on making the interaction feel smooth, human, and accessible.

The system is a hybrid AI + rule-based intelligent vision system. Frontend: Next.js webcam overlay with real-time WebSocket communication. Backend: FastAPI handling gesture inference, transcript stabilization, and translation. The architecture enables low-latency streaming critical for accessibility applications.

MediaPipe Hands chosen over raw CNN detection for its robust landmark extraction across lighting conditions. Rule-based heuristics fused with CNN-based TensorFlow model using confidence thresholds — neither alone was sufficient. Temporal gating across multiple frames dramatically improved stability. Tesseract OCR with EasyOCR fallback for air-writing handles the variability of finger-drawn strokes better than a single engine. FastAPI + WebSockets chosen over REST for the latency requirements of a live accessibility system.

Deep practical learning in real-time computer vision, AI model integration, human-computer interaction, accessibility-focused product design, gesture recognition systems, and OCR pipelines for robust touchless writing. The project demonstrated that accessibility-focused AI requires equal investment in the UX layer as in the model layer — raw accuracy is insufficient without smooth interaction design.

A deployable web application with future Android support, including improved OCR reliability and smoother gesture-to-transcript stabilization for production-grade accessibility. Planned: real-time confidence visualization, user-trainable custom gestures, and multi-user session support.