IDD Edge AI Segmentation

Team: Debanshu Mallick, Chandan Rai, Tamaghna Mandal, Yuvaraj DC
Code: GitHub Repository

This is a more detail report documents the complete project flow for semantic segmentation and instance segmentation on edge hardware: dataset post-processing, model training, logit knowledge distillation, quantization/compilation for Hailo, and Raspberry Pi deployment. GPU-cluster setup details are intentionally omitted, but training logs were inspected to recover the reported losses and validation metrics. and contains all the detailed metrics

1. Project Scope

The project builds a real-time road-scene perception stack for the India Driving Dataset (IDD). The final deployed demo combines:

• A semantic segmentation model running as a Hailo HEF.
• A YOLOv8n-seg instance segmentation model running as a Hailo HEF.
• A Raspberry Pi PyQt/OpenCV application that overlays semantic classes and detected dynamic objects.
• Runtime switching between three semantic model variants.

2. End-to-End Pipeline

The full pipeline is:

1. Convert raw IDD polygon annotations into semantic masks, instance masks, color previews, or panoptic annotations.
2. Organize images and masks into train/validation/test splits for each label level.
3. Train strong semantic teachers on Label1ID, Label2ID, and Label3ID.
4. Train smaller semantic students using logit knowledge distillation.
5. Train YOLOv8n-seg on IDD instance classes using a dataset bridge.
6. Export selected PyTorch checkpoints to ONNX.
7. Parse ONNX to Hailo HAR, optimize/quantize with calibration images, and compile HEF files.
8. Deploy semantic and YOLO HEFs to Raspberry Pi with a PyQt/OpenCV/Hailo runtime.

3. IDD Dataset Post-Processing

The conversion logic is in idd_polygon_to_mask.ipynb. It reads the original IDD polygon annotations from gtFine and writes derived supervision targets.

The notebook exposes:

run_pipeline(
    datadir,
    out_basedir,
    encoding,
    do_semantic,
    do_instance,
    do_color,
    do_panoptic,
)

Supported encodings include:

Generated outputs can include:

Important conversion details:

• Unknown labels are skipped.
• Label names ending with group are normalized before lookup.
• Background/ignore pixels use ID 255 for semantic-style encodings.
• Instance counters are keyed by the actual encoded class value, which avoids collisions when using coarser label levels.
• Panoptic category IDs are generated from the selected encoding rather than being hardcoded to a single IDD level.

For semantic training, the project expects split folders like:

<IDDLx>/train/images
<IDDLx>/train/masks
<IDDLx>/val/images
<IDDLx>/val/masks
<IDDLx>/test/images
<IDDLx>/test/masks

All semantic logs report:

4. Label Spaces

The semantic projects train on three IDD label levels.

Label1ID

Label1ID is a 7-class coarse setup:

Label2ID

Label2ID is a 16-class medium-granularity setup and is the label space used for all deployed models:

Label3ID

Label3ID is a 26-class fine-granularity setup:

5. Semantic Training Setup

The semantic dataset loader is IDDSegDataset. It performs:

• ImageNet normalization.
• Random resized crop to 512 x 512 during training.
• Horizontal flip augmentation.
• Color jitter augmentation.
• Validation resize to 512 x 512.
• Ignore index 255.

Common training settings across all projects:

The mask encoding can run in automatic mode. If the mask already contains contiguous training IDs, it is used directly; otherwise raw IDD IDs are remapped into the target training label space.

6. Model Families

Original Teacher and Student

The original semantic project uses:

The Hailo-optimized LR-ASPP student uses deployment-friendly changes:

• Fixed interpolation sizes of 128 x 128 and 512 x 512.
• Sigmoid instead of hard-sigmoid style operations.
• Export mode returns a raw tensor instead of a dictionary.

ConvNeXt Teacher

The stronger teacher experiments use:

The ConvNeXt UPerNet project uses:

DeepLab Student Variants

Two additional students were trained against the ConvNeXt teacher:

7. Logit Knowledge Distillation

Logit KD combines supervised cross-entropy with masked KL divergence between teacher and student output distributions:

loss = CE(student, target) + alpha * KL(student_logits / T, teacher_logits / T)

where T = 4.0 is the distillation temperature and alpha = 1.0 is the distillation weight. This method is applied consistently across all label levels and all student architectures.

8. Semantic Training Results

The following values were recovered from the training logs. “Best mIoU” is the best validation mIoU recorded for that run.

Original MobileNetV4 Teacher/Student Project

Key observation: Logit KD provided modest improvement over the baseline for the original MobileNetV4-S LR-ASPP student. The teacher-student capacity gap remained the primary limiting factor.

ConvNeXt UPerNet Teacher to MobileNetV4-S LR-ASPP Student

Key observation: The ConvNeXt UPerNet teacher is materially stronger, but the compact LR-ASPP student remains capacity-limited and does not close much additional gap despite the stronger teacher signal.

ConvNeXt Teacher to MobileNetV4-L DeepLabV3+ Student

Key observation: MobileNetV4-L DeepLabV3+ closes much more of the teacher-student gap than the smaller LR-ASPP student, especially on Label1 and Label2. This is the highest-accuracy deployed student.

ConvNeXt Teacher to MobileNetV4-S DeepLabV3+ Student

Key observation: MobileNetV4-S DeepLabV3+ is a strong middle ground — it materially improves over the original LR-ASPP student while remaining much smaller than the MobileNetV4-L variant.

9. YOLOv8n Instance Segmentation

The dataset bridge script creates a YOLO-compatible dataset at:

datasets/idd_instance_seg

It hardlinks images and symlinks labels so Ultralytics can resolve images and segmentation labels from a local YOLO-style tree.

YOLO class mapping:

Training configuration:

YOLO dataset log summary:

The training log reports 381 of 417 pretrained weights transferred from the base YOLOv8n-seg checkpoint.

Best and final YOLO metrics from runs/idd_yolov8n_seg/results.csv:

10. Hailo Quantization and Compilation

The project exports selected PyTorch checkpoints to ONNX, parses them into Hailo HAR files, optimizes/quantizes with calibration images, and compiles HEF files for Hailo-8.

Common semantic Hailo flow:

1. Export PyTorch checkpoint to ONNX.
2. Parse ONNX to Hailo HAR.
3. Optimize HAR with a calibration set.
4. Compile optimized HAR to HEF for hailo8.
5. Copy HEFs into Final__Demo.

Common semantic model-script settings:

normalization mean = [123.675, 116.28, 103.53]
normalization std  = [58.395, 57.12, 57.375]
calibration size   = 64 images/tensors
calibration batch  = 1
target             = hailo8

The Hailo logs show that optimization level was reduced to level 0 because only 64 calibration entries were available and no GPU was available for higher-level optimization. QAT fine-tuning was skipped.

Semantic HEF Summary

Additional compile-log bottleneck FPS estimates:

YOLOv8n-Seg HEF Summary

The YOLO Hailo folder is yolov8n_seg_hailo.

Export settings:

YOLO model-script settings:

normalization([0, 0, 0], [255, 255, 255])
change_output_activation(conv45, sigmoid)
change_output_activation(conv61, sigmoid)
change_output_activation(conv74, sigmoid)

YOLO Hailo artifact summary:

YOLO compile log summary:

11. Raspberry Pi Deployment

The deployment application is:

Final__Demo/python_if_models_switch_pipelined.py

It uses:

• PyQt5 for the GUI.
• OpenCV for video/camera input and visualization.
• Hailo Platform APIs for loading HEFs and running inference.
• A thread-based inference loop to keep the UI responsive.

Deployed HEFs:

Runtime constants:

Semantic preprocessing:

1. Resize frame to 512 x 512.
2. Convert BGR to RGB.
3. Convert to float32.
4. Run semantic HEF.
5. Apply argmax over class logits.
6. Map class IDs to a color palette.

YOLO preprocessing and decoding:

1. Letterbox input to 512 x 512.
2. Convert BGR to RGB.
3. Run YOLO HEF.
4. Decode three detection heads:
   • stride 8: regression conv44, class conv45, mask conv46
   • stride 16: regression conv60, class conv61, mask conv62
   • stride 32: regression conv73, class conv74, mask conv75
   • prototype output: conv48
5. Decode boxes with DFL.
6. Apply confidence threshold and multiclass NMS.
7. Combine mask coefficients with prototype masks.
8. Undo letterbox scaling and overlay instance masks.

Application features:

• Load video files.
• Discover Raspberry Pi camera sources using libcamera/GStreamer and /dev/video*.
• Switch between semantic models at runtime.
• Enable or disable YOLO overlay.
• Adjust overlay alpha.
• Run a benchmark mode:

python python_if_models_switch_pipelined.py --benchmark <video> [frames]

The benchmark prints average milliseconds per frame and FPS for each semantic model.

12. Model Selection Notes

The logs support the following practical conclusions:

• ConvNeXt UPerNet is the best teacher family, reaching 81.92% Label1, 73.14% Label2, and 68.54% Label3 mIoU.
• MobileNetV4-L DeepLabV3+ gives the best distilled student accuracy, with 78.87% Label1 and 68.15% Label2 using logit KD.
• MobileNetV4-S DeepLabV3+ is a strong compact deployment option, reaching 63.99% Label2 and 56.33% Label3 with logit KD — a material improvement over the original LR-ASPP student.
• Logit KD is consistently effective across all student architectures and label levels.
• Hailo compilation succeeded for all selected semantic and YOLO models, but the calibration and optimization setup was conservative — only 64 calibration samples were used and QAT was skipped.

13. Known Limitations

• The MobileNetV4-L DeepLabV3+ Label2 run was only recorded to epoch 17 for the KD-D variant (not used); the logit KD run completed fully.
• The Hailo logs indicate optimization level 0 due to limited calibration data and no GPU-assisted optimization. More calibration images and full optimization/QAT could improve quantized accuracy.
• YOLO mask mAP is modest but remains useful as a dynamic-object overlay when combined with semantic segmentation.

The deployable system is a hybrid semantic-plus-instance perception pipeline: semantic segmentation supplies dense road-scene layout, YOLO supplies dynamic object instances, and the Raspberry Pi application fuses both outputs in real time through the Hailo accelerator.