Accepted to EMNLP 2026 (Main Conference) Assistive AI for Low-Vision On-Device VLM (< 450 MB)

Small yet Assistive: Spatially-Aware Post-Training for Low Vision

Rishabh Choudhary Shreyansh Raj Umesh Goyal Subh Kasyap Shrestha Kumar Sushovan Jena Komal Kumar Hisham Cholakkal Aditya Nigam

Bridging the spatial grounding and hazard awareness gap in compact vision-language models for blind and low-vision navigation — running offline at 39.3 tok/s on commodity smartphones.

Code Repository Hugging Face Model Architecture Results Qualitative Demos
442 MB
On-Device Footprint
Mixed-Precision IQ4_NL (Samsung A55)
39.3 tok/s
+106% vs Q4_K_M
Generation Throughput on Phone CPU
+101.5%
32.5% → 65.5%
OCR-Bench ANLS Gain
+44.2%
37.8% → 54.5%
TextVQA Accuracy Gain
100%
Complete Precision
Direction & Distance Coverage
Problem & Overview

Compact Multimodal Assistance for Low Vision

Current vision-language models generate vague descriptions inadequate for physical navigation. Smol-VL-BLV adapts a 500M VLM with teacher distillation and reinforcement learning to produce concise, hazard-aware, spatially-grounded descriptions.

Gemma-4 Distillation

Supervision from Gemma-4-31B over 9,646 video keyframes encoding professional Audio Description guidelines.

Online GRPO

Reinforcement learning targeting metric distances (+0.20), directional cues (+0.08), and hazard alerts (+0.40).

SFT-Patch Recovery

Lightweight 3-epoch patch recovering descriptive ambience while merging 3 LoRA adapters with zero overhead.

442MB On-Device

IQ4_NL non-linear quantization with Q5_K attention preservation for offline execution on Samsung Galaxy A55.

Pipeline & Architecture

End-to-End Post-Training Framework

LUV keyframes and a BLV text prompt are encoded through vision and text encoders, aligned with lightweight adapter layers, merged in a fusion block, and decoded via fine-tuned LoRA transformers.

Smol-VL-BLV Post-Training Pipeline Architecture Diagram
Figure 2: Architecture of the post-training pipeline showing LUV keyframe encoding (SigLIP), token concatenation fusion, and LoRA transformer decoding.
Distillation & Reinforcement Learning

Teacher Supervision & Composite BLV Reward

Figure 3 shows the coverage of BLV-relevant spatial attributes in teacher-generated captions (Charades & AVCaps), while the composite reward below guides online GRPO to optimize metric distances, directional anchors, and hazard detection.

Teacher Caption Coverage on BLV Metrics Pie Chart
Figure 3: Coverage of BLV-relevant attributes in teacher-generated captions. The captions consistently include core navigation cues such as directions, distances, and spatial layout, while covering social context, ambience, obstacles, and moving hazards.
+0.40 / -0.30

Hazard / CAUTION Alert

+0.40 for identifying hazards; -0.30 penalty if missing.

+0.08 / word

Directional Language

{left, right, center, ahead, behind} (max +0.24).

+0.20

Metric Distances

Awarded for explicit metric estimates (e.g. "1 meter ahead").

+0.16 / -0.10

Concise Structure

+0.16 for 2–4 sentences; -0.10 penalty if >6 sentences.

-0.20

Anti-Hedging Penalty

Penalizes subjective phrases ("appears to be", "seems like").

+0.10

Environment Context

Requires scene-type identification in the first sentence.

Mobile Deployment

Offline Execution on Commodity Android

Mixed-precision IQ4_NL quantization with protected Q5_K attention projections enables real-time CPU inference on a Samsung Galaxy A55.

End-to-End Mobile Deployment Pipeline Flowchart
Figure 5: End-to-end deployment pipeline from merged Hugging Face model → GGUF export → imatrix calibration → IQ4_NL mixed-precision on Samsung Galaxy A55.

Quantization Strategy

Standard 4-bit uniform compression degrades directional attention. We apply IQ4_NL (251 MB) guided by an importance matrix computed over 500 BLV captions, keeping self-attention projections (q_proj, k_proj, v_proj) at 5-bit (Q5_K).

Storage Breakdown: Language Model (251 MB) + Vision Projector (191 MB) = 442 MB total.
Samsung Galaxy A55
Exynos 1480 / 8GB RAM
$ llama-mtmd-cli -m blv_final_iq4nl_best.gguf \ --mmproj sft_patch_mmproj.gguf --image frame.jpg \ -p "Describe the scene for a blind person." \ -n 60 --temp 0.0 -t 4
Model Footprint
442 MB
Generation Speed
39.3 tok/s
RAM Load Time
291 ms
Total Latency / Frame
27.1 s (CPU)
Evaluation

Quantitative Benchmarks

Spatially grounded post-training delivers major gains on domain-specific navigation metrics, OCR, and general visual question answering.

Model Params OCR ANLS (%) VQA Accuracy (%) VQA ANLS (%)
SmolVLM2-256M base 256M 16.49% 1.08% 3.09%
SmolVLM2-500M base 500M 32.54% 37.75% 47.52%
SmolVLM2-2.2B base 2.2B 25.44% 27.51% 29.64%
LLaVA-1.5-7B 7B 28.78% 47.79% 62.96%
PaliGemma-3B 3B 69.76% 76.09% 87.36%
Qwen2-VL-2B 2B 86.10% 80.73% 89.24%
Qwen2.5-VL-3B 3B 88.56% 81.12% 90.39%
Qwen2-VL-7B 7B 87.23% 85.09% 92.89%
Smol-VL-BLV (Ours) 500M 65.54% (+101.5%) 54.47% (+44.2%) 65.97% (+38.8%)
Model Params Spatial Score (1–10) Social Score (1–10) OCR-Bench (%) TextVQA (%) GFLOPs
PaliGemma2-3B 3B 1.15 1.02 7.8 14.3 10,543
moondream2 1.8B 1.00 1.00 – – –
SmolVLM2-2.2B 2.2B 3.49 3.45 25.4 27.5 125,406
SmolVLM2-500M (Base) 0.5B 3.21 3.30 32.5 37.8 34,855
Smol-VL-BLV (Ours) 0.5B 3.83 (+19.3%) 3.79 (+14.8%) 65.5 54.5 34,855
Qwen2.5-VL-3B 3B (6× larger) 5.43 3.51 88.6 81.1 14,280
Hardware Environment Execution Backend Generation Speed Token Latency Total Latency
NVIDIA Tesla T4 GPU Cloud GPU 187.4 tok/s 5.34 ms 1.70 s
Apple Silicon Mac Metal GPU (IQ4_NL) ~132.6 tok/s ~7.54 ms 2.41 s
Samsung Galaxy A55 Termux (CPU-only) 39.3 tok/s 23.3 ms 27.1 s
Metric / Attribute Base (A) + SFT (B) + GRPO (C) Final Model (D) Gain (Δ)
BLEU-1 19.02 18.08 22.55 45.57 +26.55
BLEU-4 0.74 1.66 1.97 13.53 +12.79
ROUGE-L 10.74 17.66 17.93 30.49 +19.75
Direction Coverage 29% 51% 88% 100% +71%
Distance Coverage 45% 20% 22% 100% +55%
Judge Metric (1–10 Scale) Base (A) SFT v2 (B) GRPO (C) Final Model (D) Gain (Δ)
Multi-Context Framework (MCF) 1.88 2.26 2.39 2.94 +1.06
Navigational Assistance (NAF) 1.84 2.44 2.35 3.57 +1.73
NAF Objectivity 2.71 3.77 3.38 5.38 +2.67
MCF Spatial Orientation 1.21 1.71 1.85 3.23 +2.02
Qualitative Comparison

Navigation Scene Comparisons

Smol-VL-BLV produces concise, present-tense, spatially-grounded descriptions with distance estimates and directional anchors rather than generic captions.

Qualitative comparison of scene descriptions across indoor environments
Figure 4: Qualitative comparison across four indoor navigation scenes showing concise, spatially grounded descriptions with metric distances and hazard cues.
BLV Cues: Directional Cue Metric Distance Hazard / Obstacle Landmark / Entity
Ours (500M)

Smol-VL-BLV

Spatially Grounded & Actionable
Baseline (500M)

SmolVLM2-500M Base

Vague & Spatially Unaware
6× Larger (3B)

Qwen-3B Instruct

Generic Observation Style
Citation

BibTeX Reference

@inproceedings{choudhary2026small,
  title     = {Small yet Assistive: Spatially-Aware Post-Training for Low Vision},
  author    = {Choudhary, Rishabh and Raj, Shreyansh and Goyal, Umesh and Kasyap, Subh and Kumar, Shrestha and Jena, Sushovan and Kumar, Komal and Cholakkal, Hisham and Nigam, Aditya},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}