Image captions that actually read the text in the image, not just describe the scene.
Standard image captioning systems describe the visual scene but ignore text visible within the image: charts, signs, slides, product labels. This creates a major gap for real-world applications: a model that describes 'a whiteboard' instead of reading the equations on it, or 'a sign' instead of reading what it says. OmniBridge bridges this gap by fine-tuning BLIP-2 on the TextCaps dataset using LoRA, a parameter-efficient technique that adapts a large pretrained model without retraining all parameters.
BLIP-2 (Bootstrapping Language-Image Pretraining) combines a frozen vision encoder, a learned Q-Former bridge, and a frozen language model to generate captions from images.
Low-Rank Adaptation (LoRA) inserts small trainable matrices into BLIP-2's attention layers. This allows domain adaptation on TextCaps with minimal compute: only 0.1% of parameters are trained.
TextCaps provides image-caption pairs where captions specifically incorporate OCR text visible in the image. The model learns to ground language in both visual content and readable text.
At inference, OmniBridge produces captions that seamlessly integrate scene description with in-image text, far more useful for accessibility, document indexing, and multimodal search.
Significantly improved caption quality on text-heavy images vs base BLIP-2
LoRA fine-tuning achieved results with a fraction of the compute of full fine-tuning
Model correctly reads and incorporates visible text (signs, labels, slides) into generated captions
TextCaps CIDEr score improved substantially over the base BLIP-2 baseline