Real-time video classification with PaliGemma: architecture patterns for low-latency VLM inference
Pasquale Molinaro
In a previous article, we benchmarked three open-source Vision-Language Models on zero-shot object detection and arrived at an uncomfortable conclusion: even the fastest contender, Phi-3.5-vision-instruct, takes 4.45 seconds per frame on an NVIDIA L4. LLaVA-v1.6 sits at 8.13 seconds. For any application that needs to process a live video stream, these numbers are disqualifying. But the conclusion that VLMs are fundamentally incompatible with real-time workloads deserves more scrutiny. That 8-sec