August 23, 2026 · 5 min read
You probably don’t need a VLM on every frame
A practical look at when a small detector is the better tool—and where a VLM still earns its keep.
- computer-vision
- multimodal
- applied-ai
- systems
VLM demos are fun because you can point at an image and ask almost anything. That flexibility is also why I’d hesitate to put one in the hot path of a live camera system.
Say the job is to check an aisle for a forklift, ten times a second. The question isn’t changing. We don’t need a paragraph about the scene; we need a box, a score, and an answer before the next frame arrives. A small detector from the YOLO family is often a much better fit.
The demo below uses made-up but plausible numbers: 28ms for the detector and 840ms for the VLM. Those are not vendor benchmarks or numbers from Intuit. They’re there to make the system trade-off visible.
Assumed fixture: 28ms vs 840ms ≈ 30× faster specialized path, accuracy within ~2pp.
Run an operation above and every step lands here — scrub, replay, or slow it down.
Same camera frame, same user prompt. Run both paths to compare latency and what each model returns.
Run both paths and the difference is hard to miss. The detector finishes while the VLM is still working through image encoding and language generation.
Why the gap gets so large
A detector has one narrow job. It resizes the frame, runs one forward pass, decodes boxes, filters weak predictions, and applies non-maximum suppression. Its vocabulary and output shape are fixed.
A generative VLM does more. It encodes the image, combines it with the prompt, generates tokens, and then leaves you to turn those tokens back into something the rest of the system can use. That extra work is worthwhile when the prompt is genuinely open-ended. It’s wasteful when every request asks the same yes-or-no question.
This is the distinction I find useful:
| Question | Where I’d start |
|---|---|
| “What looks unusual in this image?” | VLM |
| “Is there a forklift in aisle 3?” | Detector |
| “Find objects I haven’t defined ahead of time” | VLM or open-vocabulary detector |
| “Run these six known checks on an edge device” | Detector |
The common mistake is taking the model that made the best demo and assuming it should also serve every frame in production.
The frame budget is unforgiving
At the demo’s assumed timings, serial throughput works out to:
detector: 1000 / 28 ≈ 35.7 frames/s
VLM: 1000 / 840 ≈ 1.2 frames/s
And inference never gets the whole budget. A 10 FPS stream gives you 100ms per frame for decoding, resizing, inference, tracking, rules, rendering, network time, and whatever jitter shows up that day.
An example budget might look like this:
| Stage | Budget |
|---|---|
| Decode and resize | 12ms |
| Inference | 45ms |
| NMS and tracking | 8ms |
| Event rules | 10ms |
| Breathing room | 25ms |
One warm, cherry-picked inference tells you very little. I’d measure the full pipeline at p50, p95, and p99, with batch size 1 and the actual resolution, hardware, precision, and runtime I plan to ship. Quantization through tools such as ONNX Runtime or TensorRT can move the numbers, sometimes a lot. It doesn’t remove the need to measure them.
A VLM can still help train the detector
Choosing a detector at runtime doesn’t mean throwing the larger model away. I’d use it earlier in the loop:
- Decide exactly which events matter, including the boring negative cases.
- Gather real video and split it by camera, site, and time.
- Add synthetic scenes with different lighting, poses, blur, and occlusion.
- Let a larger model propose boxes and labels.
- Have people review uncertain or high-impact examples.
- Train the smaller model on the real data, synthetic data, and hard negatives.
- Test it on sites and time periods it has never seen.
- Feed the misses back into the next dataset.
The larger model is a labeler, not an oracle. If its mistakes become training data without review, the smaller model simply learns those mistakes more efficiently.
Domain randomization and knowledge distillation help with different parts of this loop. The first broadens the simulated world; the second transfers behavior from a larger model to a smaller one. They work well together, but neither replaces a clean test set made from real cameras.
Boxes are not the product
It’s easy to spend weeks improving mAP and still ship a bad alerting system. A persistent false box can become hundreds of duplicate notifications. A detection that arrives ten seconds late may be technically correct and operationally useless.
Along with precision, recall, AP, and calibration, I’d track:
- time from the start of an event to the first alert
- false alerts per camera-hour
- duplicate alerts for one event
- misses by site, angle, lighting, and object size
- results after tracking and temporal smoothing
Those numbers describe what the user experiences. Frame-level scores only describe one part of it.
Where specialization breaks down
A detector is the wrong tool when users invent a new scenario every session or when the request depends on language like “the red forklift behind the spill.” It also won’t rescue a project with thin, unrepresentative training data.
A VLM has its own rough edges: variable latency, higher per-frame cost, network dependence, and output that can be harder to validate. If the result triggers something important, I like a cascade:
fast detector
├─ confident result → track or emit event
├─ uncertain result → ask a larger model to verify
└─ high-impact result → send to a person
That design isn’t free. Verification adds queues, timeouts, and another model that can fail in the same way. Test the whole chain, not just each model in isolation.
For a rough capacity check:
inferences/day = cameras × sampled_fps × 86,400
One hundred cameras sampled at 2 FPS produce 17.28 million inferences a day. At that scale, motion gates, tracking, sampling, and selective escalation often save more than another round of model surgery.
My rule of thumb is simple: use the VLM while the question is still changing. Once the question settles, see whether a smaller model can own the repetitive work.