[D] - Can multimodal models tell images apart from text? Like if a text token and an image token are close vectors, will the model be able to "tell" if it is reading or seeing?

I ran into this doing some work with multimodal models. It seemed like they couldn't tell which part of the information was from the text vs the image portions of an input.

Is there any research on this?

image images information machinelearning multimodal multimodal models part ran reading text the information token vectors will work