Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

Xuhui Zhan, Tyler Derr
Vanderbilt University
Paper Code
Comparison of the LLaVA and Inverse-LLaVA information flows on an MME benchmark example.

High-level overview of the proposed Inverse-LLaVA framework on an MME benchmark example. (a) LLaVA projects visual features into discrete text space via an explicit projection (red), requiring alignment pre-training, and produces the wrong answer ("Yes"). (b) Inverse-LLaVA maps text embeddings into continuous vision space for fusion, eliminating alignment pre-training, and yields the correct answer ("No"). Blue indicates vision flow, green indicates text flow, red denotes explicit projection, purple represents LLM components, and orange indicates LLM output.

Abstract

Traditional multimodal learning approaches rely on alignment pre-training to bridge vision and language modalities, typically by projecting visual features into discrete text token spaces using large-scale image–text data. We revisit this design choice and propose Inverse-LLaVA, a multimodal architecture that inverts the conventional mapping direction by projecting text embeddings into continuous visual representation space and performing fusion within intermediate transformer layers. This representation-first design enables effective multimodal reasoning without relying on an explicit alignment pretraining stage and significantly reduces dependence on large alignment datasets. Across nine multimodal benchmarks, Inverse-LLaVA demonstrates strong learning efficiency under reduced supervision, achieving substantial gains on reasoning-intensive tasks while exhibiting selective performance drops on perception tasks that depend on explicit visual–text grounding. Our analysis indicates that these trade-offs primarily reflect differences in supervision regime rather than architectural limitations. Together, these results show that alignment pretraining is not strictly required for effective multimodal reasoning and highlight the importance of preserving continuous modality representations, opening a new direction for multimodal architecture design that decouples representation structure from supervision regime for more flexible and efficient multimodal systems.

Architecture

Architecture and training comparison between LLaVA and Inverse-LLaVA.

Architecture comparison between LLaVA and Inverse-LLaVA. LLaVA employs a two-stage training approach with alignment pre-training followed by instruction fine-tuning, where vision and text tokens are concatenated before being fed to the LLM. In contrast, Inverse-LLaVA uses single-stage training with text-guided visual fusion in intermediate layers, where vision information is integrated through learnable text-to-vision projections and combined with original hidden states via residual connections. Fire icons denote trainable components, while snowflake icons denote frozen components.

Results

Visual Information Preservation Through Inverse Mapping

Table 1 shows that the inverse mapping strategy achieves competitive performance across nine vision–language benchmarks while fundamentally rethinking cross-modal interaction. Rather than compressing visual features to conform to the discrete structure of text tokens, our method projects textual embeddings into the visual feature space. This design preserves spatial structure and fine-grained visual information that may otherwise be lost during discretization. The results reveal advantages: Inverse-LLaVA outperforms LLaVA-1.5 on MM-VET (31.2 vs 31.1), VizWiz (50.95 vs 50.0), and ScienceQA-IMG (67.84 vs 66.80). However, we observe consistent degradation on tasks requiring precise visual–text alignment, including TextVQA (52.02 vs 58.2) and GQA (58.46 vs 62.0). This performance pattern suggests our approach excels at reasoning tasks while struggling with direct visual–linguistic correspondence.

Table 1: Performance comparison of vision-language models using Vicuna-7B as the backbone LLM across multiple vision-language benchmarks. We evaluate models on MM-VET; VizWiz; SQAI: ScienceQA-IMG; MMB: MMBench; MMBCN: MMBench-CN; MMEp: MME-Perception; VQAV2: VQA-v2; VQAT: TextVQA; and GQA. Bold indicates the best performance and underlined indicates the second-best performance in each benchmark. "-" denotes unreported results or experiments not conducted due to computational constraints.
Model MM-VET VizWiz SQA-IMG MMB MMB-CN MME-p VQA-V2 VQA-T GQA
LLaVA-1.5 31.1 50.0 66.80 64.3 58.3 1510.7 78.5 58.2 62.0
InstructBLIP 26.2 34.5 60.5 36.0 23.7 - - 50.1 49.2
InternVL-Chat - 52.5 - - - 1521.1 79.3 57.0 62.9
EVE-7B 25.6 41.8 63.0 49.5 - 1217.3 75.4 51.9 60.8
Inverse-LLaVA (Ours) 31.2 50.95 67.84 54.55 41.84 1293.15 74.76 52.02 58.46
Inverse-LLaVA-HD (Ours) - - - - - 1335.67 - - 59.33

Table 2: Training data usage comparison. #Samples indicates the number of samples used in the alignment pre-training and instruction fine-tuning stages.
Model Alignment #Samples Finetune #Samples
LLaVA-1.5 558K 665K
InstructBLIP 129M 1.2M
InternVL-Chat 4.98B 665K
EVE-7B 33M 665K
Inverse-LLaVA (Ours) 0 665K

MME cognitive, overall, and perception task results for LLaVA-1.5 and Inverse-LLaVA variants.

MME benchmark analysis comparing LLaVA-1.5-7B-LoRA, Inverse-LLaVA, and Inverse-LLaVA-HD across cognitive and perception tasks in the MME benchmark. (A) Cognitive task performance, where Inverse-LLaVA outperforms the baseline in numerical calculation (+69%) and text translation (+125%). (B) Overall performance comparison. (C) Perception task evaluation, showing strong performance of Inverse-LLaVA variants on Existence and Count tasks, with Inverse-LLaVA-HD achieving perfect Existence scores. Performance drops in Celebrity recognition (-50%) and OCR (-21%) account for most of the overall perception gap. Overall, inverse training preserves cognitive performance while exhibiting task-specific effects on perception.

BibTeX


@article{zhan2025inverse,
  title={Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping},
  author={Zhan, Xuhui and Derr, Tyler},
  journal={arXiv preprint arXiv:2508.12466},
  year={2025}
}

Acknowledgement

This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.