COMPACT

We propose COMPACT (COMPositional Atomic-to-complex Visual Capability Tuning), a data recipe that explicitly controls for the compositional complexity of the training examples. COMPACT data allows MLLMs to train on combinations of atomic capabilities to learn complex capabilities more efficiently. Across complex multi-capability benchmarks, COMPACT outperforms the LLaVA-665K VIT while using less than 10% of its data budget.

Training on COMPACT data, whose samples require up to three atomic capabilities, generalizes well to tasks that have even higher capability requirements. For example, COMPACT achieves a substantial 83.3% improvement on MMStar and 94.0% improvement on MM-Vet compared to the full-scale VIT on particularly complex questions that require four or more atomic capabilities. COMPACT offers a scalable, data-efficient, visual compositional tuning recipe to improve on complex visual-language tasks.

COMPACT scales capabilities of MLLMs from atomic (k = 1) to composite (k > 1) complexity levels. Our approach generates multi-capability training data by prompting vision-language models to create questions that integrate k = (1, 2, 3) atomic visual capabilities.

COMPACT Data Generation Pipeline: (Left): We sample atomic capabilities (k = 1) such as color, object recognition, and spatial relationship. (Center): We generate questions (k = 1, 2, 3) that integrate all the sampled capabilities. (Right): We verify the quality of generated conversations and combine them with instruction tuning data to maintain instruction following capability.

With 32K samples of our compositional tuning data and 5% of the LLaVA-665K VIT data (only 10% of the size of the full VIT dataset), COMPACT matches the performance of full-scale VIT (100.18% relative score) and demonstrates exceptional generalization to complex tasks.

Baseline Comparisons: COMPACT (65K) outperforms the random subset of the VIT data (65K), gradient-based approach selected subset of the VIT data (65K), and even the full VIT data (665K) on diverse multimodal benchmarks. The best and second best results for each benchmark are shown in bold and underlined, respectively. COMPACT integrates atomic capabilities into tasks of higher compositional complexity, enabling models to generalize and handle complex tasks without explicit decomposition.

Compositional Generalization to Higher Complexities: We compare the performance of COMPACT (65K) and LLaVA-665K VIT (665K) at each compositional complexity (k) level. COMPACT exceeds the LLaVA-665K baseline at k = (3,4,5) tasks while using significantly less training data.

Performance Across Compositional Tuning Data Scales: We fix the VIT subset (5% of LLaVA-665K) and scale the compositional tuning data in COMPACT from 2K to 32K. For comparison, we remove the compositional tuning data and add more VIT data (2K-32K) instead to prepare VIT only baselines with equal data budgets. With much fewer data, COMPACT (solid lines) consistently outperforms the VIT only baselines (dashed lines). The performance gap is pronounced for complex reasoning benchmarks such as MM-Vet and MMStar, where the 8K COMPACT model often exceeds the VIT only baseline at 32K. This demonstrates the data efficiency of COMPACT, requiring substantially less data than LLaVA-665K VIT to achieve comparable or better results.

We conduct a series of ablation studies to investigate key design considerations in COMPACT.

A. Compositional Complexity Distribution: In order to show that the performance improvement of COMPACT mainly comes from the balanced distribution of compositional complexity in the compositional tuning data, we analyze the impact on the performance when its compositional complexity is unbalanced. We generate a 16K-sample compositional tuning data whose distribution of k resembles LLaVA-665K VIT data. The performance of unbalanced COMPACT (96.28%) is close to random baseline. However, the balanced COMPACT (original) performance jumps to 98.83%, suggesting that most of the performance gain in COMPACT comes from the fair represenation of higher k samples in the dataset.

D. Compositional Complexity Range: To isolate the effect of the range of compositional complexities while controlling for data quality, we generate three sets of 16K-sample compositional tuning data, each it k=1, k=1,2 and k=1,2,3. For fair comparison, we maintain consistent sample counts and use identical set of images in all three settings. The model trained on k=1,2,3 outperforms other two settings. Although the model trained on k=1 data can solve tasks with lower compositional complexity, it does not generalize to higher k tasks.

We perform a quantitative analysis comparing COMPACT with LLaVA-665K VIT. COMPACT's compositional tuning data has a more balanced distribution of capabilities compared to LLaVA-665K VIT. In additional, the bar plot below shows the distribution of compositional complexity in a random subset of LLaVA-665K VIT data. Unlike COMPACT, where the compositional complexity is designed to be balanced, the compositional complexity in LLaVA-665K VIT data shows a complexity cliff, characterized by a lack of higher k samples.