Edge-based Computer Vision & AR Nutritional HUD
Modern dietary tracking applications suffer from extreme user friction due to manual logging. The architectural challenge was to build an edge-capable computer vision system that could instantly analyze complex, mixed-food plates in real-time through a mobile camera stream, without causing device thermal throttling or relying entirely on high-latency cloud APIs.
Engineered a custom Flutter camera plugin utilizing native C++ bridges for zero-copy video frame extraction. Integrated an INT8 quantized TensorFlow Lite model directly optimized for Apple CoreML and Android NNAPI execution. For complex ambiguous ingredients, the system intelligently dispatches compressed vector embeddings to Gemini AI as a fallback verification layer, projecting nutritional vectors onto a custom AR Canvas HUD.
Native C++ camera pipelines bypass standard platform overhead to feed raw byte buffers directly to the NPU.
Local TFLite models handle 80% of standard classifications in <45ms; edge-case foods trigger asynchronous Gemini multimodal verification.
Custom Flutter rendering engine calculates bounding box matrices and stabilizes UI overlays over moving camera streams.