Meta AI Introduces Chameleon: A New Family of Early-Fusion Token-based Foundation Models that Set a New Bar for Multimodal Machine Learning

Meta researchers present Chameleon, a mixed-modal foundation model that facilitates generating and reasoning with interleaved textual and image sequences, enabling comprehensive multimodal document modeling. Unlike traditional models, Chameleon employs a unified architecture, treating both modalities equally by tokenizing images akin to text. This approach, termed early fusion, allows seamless reasoning across modalities but poses optimization challenges. To address these, the researchers propose architectural enhancements and training techniques. By adapting transformer architecture and finetuning strategies.

Researchers developed a novel image tokenizer, …

architecture document enabling family foundation foundation model fusion image images machine machine learning machinelearningnews meta meta ai meta researchers mixed modal modeling multimodal reasoning researchers set textual token unified architecture