Why official benchmarks aren't enough
MMMU and VQAv2 tell you how a model performs on tasks designed by academics. They tell you almost nothing about whether the model will work on your specific use case.
We spent three weeks running Gemini Ultra through tasks pulled from actual production deployments: invoice parsing, document understanding, medical image analysis, retail product identification, and visual QA over screenshots.
Where Gemini Vision genuinely excels
Document understanding is the standout capability. Given a complex multi-page document — contracts, financial statements, technical manuals — Gemini Ultra extracts structured information with remarkable accuracy. It handles rotated text, mixed layouts, and handwritten annotations better than any model we've tested.
Spatial reasoning over images has also improved substantially. The model can describe relationships between objects, estimate distances, and answer questions about layout with precision that GPT-4V struggles to match on our test set.
The failure modes that matter
Dense text in images remains surprisingly hard. When a document has more than ~500 words of small text, error rates climb. The model begins to hallucinate characters, transpose numbers, and miss content in the periphery of its attention.
More concerning for production use: inconsistency. Run the same image through the API three times and you'll get meaningfully different descriptions of the same scene. This variance is higher than we expected from a frontier model, and it's a real problem for any application where reliability matters.
Latency vs. accuracy tradeoffs
The Flash variant trades ~15% accuracy on our benchmark for a 4x speedup in time-to-first-token. For high-volume applications, this tradeoff is often worth it. For high-stakes decisions, Ultra is the clear choice.
Neither variant is cheap. Factor inference costs into any deployment decision before committing to multimodal features at scale.
Our recommendation
Gemini Vision is a legitimate choice for document understanding and spatial reasoning tasks. It's not a universal best-in-class multimodal model. Match the capability to your use case, test on your actual data, and plan for the inconsistency with appropriate guardrails.