Real-Time Two-Way ASL Translator on Raspberry Pi 5

Team: Liz Maria George, Naznin Amirul Haque, Ayush Kumar, Prayanshu Sharma
(M.Tech. MVLSI, ECE, IISc)
Code: GitHub Repository

1. Introduction

Communication between ASL users and non-signers is a persistent everyday barrier. This project builds a real-time, two-way ASL translator that runs entirely on a Raspberry Pi 5 — no cloud ML inference required.

Two operating modes:

The Sign-to-Speech pipeline recognises 19 ASL signs using MediaPipe landmark extraction + a CNN classifier (TFLite). Speech-to-Text uses Google Speech Recognition API.

2. System Architecture

High-Level Pipeline

┌──────────────────────────────────────────────────────────┐
│                  TWO-WAY ASL TRANSLATOR                  │
├──────────────────────────────────────────────────────────┤
│  SIGN-TO-SPEECH (Speaker Mode)                           │
│  Pi Camera ─► MediaPipe Holistic ─► Features ─► CNN     │
│  640×480        Pose+Hands (75 kp)   (30,504)  TFLite    │
│                                               ─► espeak-ng│
│                                                          │
│  SPEECH-TO-TEXT (Listener Mode)                          │
│  BT Mic ─► PyAudio ─► SpeechRecognition ─► Display      │
│  Sony XB12                 Google STT API    (Tkinter)   │
│                                                          │
│  SHARED: Tkinter GUI │ Threaded workers │ Mode toggle    │
└──────────────────────────────────────────────────────────┘

Figure 1: Top level system architecture

Sign-to-Speech: Step-by-Step

Picamera2          MediaPipe Holistic        Feature Eng.
640×480 @ 30fps ──► Pose (33 kp)      ──►  252-dim/frame
                    Left Hand (21 kp)        + velocity
                    Right Hand (21 kp)  ──►  504-dim/frame
                    (every 2nd frame)
                                        ──►  (30, 504) tensor
                                             ↓
                                         TFLite CNN
                                         conf > 85%?
                                             ↓
                                         espeak-ng TTS

Threading Architecture

Main Thread (Tkinter loop, 15 ms tick)
 ├── update_frame()  ← camera + MediaPipe + feature buffer + render
 ├── InferenceWorker ← TFLite on (30,504) via maxsize-1 queue
 ├── TTSWorker       ← espeak-ng subprocess, flushes stale requests
 └── STTWorker       ← PyAudio + Google STT (Listener mode only)

3. Hardware

4. Dataset

Data collected on the Pi itself (same camera as deployment) to eliminate domain gap.

Supported signs:

5. Feature Engineering

Raw landmarks → engineered features for robustness to hand size, distance, and orientation.

Per-Frame Feature Vector (252-dim)

+ Velocity features: np.diff of all 252 base features = 504 features/frame
Model input tensor: (30 frames × 504 features)

6. Model Training

Training on GPU workstation (train.py) — 3 automated stages:

Stage 1 — Neural Architecture Search (NAS)

Keras Tuner Hyperband searches over 1D-CNN vs LSTM, filters/units {32,64,96,128}, kernel sizes {3,5}, dropout {0.2–0.5}, LR {1e-3, 1e-4}. Up to 50 epochs per trial.

Stage 2 — 5-Fold Stratified Cross-Validation

Best NAS architecture trained for 100 epochs × 5 folds → mean val accuracy + std dev + mean best epoch (used as fixed epoch count for Stage 3).

Stage 3 — LR Scheduler Comparison

Best scheduler → final exported model.

Figure 2: Confusion matrix on held-out test set (19 classes)

7. Model Optimisation & Quantisation

• FP32: Standard TFLite conversion (TFLITE_BUILTINS + SELECT_TF_OPS)
• INT8: Post-training integer quantisation using 200 representative calibration samples; input/output remain float32 for compatibility
• Both models swappable at runtime via GUI radio buttons

8. Deployment Optimisations

9. Application Interface

Tkinter GUI designed for touchscreen:

• Live camera feed (640×480) with MediaPipe skeleton overlay
• Transcript box — 22pt bold text for detected sign / transcribed speech
• Mode toggle — Speaker ↔ Listener
• Model selector — FP32 / INT8 radio buttons (hot-swap)
• Real-time metrics — FPS + end-to-end inference latency (ms)
• Status bar — confidence score, hand tracking state, current mode

Figure 3: GUI in Speaker mode — live skeleton overlay + detected sign

Figure 4: GUI in Listener mode — transcribed speech displayed

10. Results

INT8 quantisation achieves 4× model compression and 3.6× faster inference with zero accuracy loss.

11. Challenges & Learnings

12. Future Work

• Vocabulary expansion: Fingerspelling (A–Z) + more conversational signs
• Fully offline STT: On-device distil-Whisper to replace Google Speech API
• Sentence construction: Sequential sign → grammatical English via lightweight LM
• Wearable form factor: Chest-/head-mounted camera on smaller SBC or custom PCB
• Continuous learning: Few-shot on-device fine-tuning for user-specific signs

13. Quick-Start Reproduction

# 1. Clone
git clone https://github.com/<your-username>/Realtime_ASL_RasPi.git && cd Realtime_ASL_RasPi

# 2. System deps (Pi OS Bookworm)
sudo apt update && sudo apt install -y \
    python3-pip python3-opencv espeak-ng portaudio19-dev \
    python3-tk libatlas-base-dev bluetooth bluez

# 3. Python deps
pip3 install mediapipe numpy opencv-python Pillow \
    SpeechRecognition tflite-runtime picamera2

# 4. Download dataset (optional – to retrain)
python3 fetch_dataset.py

# 5. Run the translator
cd Demo_RasPi && python3 pi_two_way_translator_V2.py

See README.md for full retraining pipeline (data_capture → preprocess → train).

14. Project Structure

Realtime_ASL_RasPi/
├── Demo_RasPi/
│   ├── pi_two_way_translator_V2.py   # Main app
│   ├── sign_model_fp32.tflite        # 628 KB
│   └── sign_model_qat.tflite         # 163 KB (INT8)
└── sign_to_speech/
    ├── 01_data_collection_raspi/
    │   ├── data_capture_raspi_V3.py  # On-device collection
    │   └── preprocess.py             # Feature extraction
    └── 02_training/
        └── train.py                  # NAS + K-Fold + LR search

15. References

• MediaPipe Holistic — Pose & hand landmarks (Google)
• TensorFlow Lite — On-device inference
• Keras Tuner — NAS / hyperparameter search
• espeak-ng — Open-source TTS
• SpeechRecognition — STT wrapper library
• Picamera2 — Pi camera Python API

AI Tools Disclosure: GitHub Copilot used for Tkinter boilerplate. Claude used for documentation. All architecture, feature engineering, and system design decisions were made by the team.

Team