Recent advances in multimodal training have significantly improved the integration of image understanding and generation within a unified model. This study investigates how vision-language models (VLMs) handle image-understanding tasks, focusing on how visual information is processed and transferred to the textual domain. We compare text-output VLMs, models that generate only text, and non-native multimodal VLMs, adapted from pre-trained large language models, with native multimodal VLMs, models trained from scratch on multimodal data to generate both text and images, highlighting key differences in information flow. We find that in native multimodal VLMs, image and text embeddings are more separated within the residual stream. Moreover, VLMs differ in how visual information reaches text: text-output VLMs and non-native multimodal VLMs exhibit a distributed communication pattern, where information is exchanged through multiple image tokens, whereas models trained natively for joint image and text generation tend to rely on a single post-image token that acts as a narrow gate for visual information. We show that ablating this single token significantly deteriorates image-understanding performance, whereas targeted, token-level interventions reliably steer image semantics and downstream text with fine-grained control. Finally, we introduce a fine-tuning strategy that eliminates the narrow gate by redistributing visual-to-text transfer across many image tokens.
@misc{serra2024narrowgatelocalizedimagetext,
title={The Narrow Gate: Localized Image-Text Communication in Vision-Language Models},
author={Alessandro Serra and Francesco Ortu and Emanuele Panizon and Lucrezia Valeriani and Lorenzo Basile and Alessio Ansuini and Diego Doimo and Alberto Cazzaniga},
year={2024},
eprint={2412.06646},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.06646},
}