HuggingFaceM4

/

Idefics3-8B-Llama3

like 213

Yes it's totally possible (it was mostly trained this way during the SFT)

Could you give example?

It's exactly like the given snippet

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },       
]

but you can remove the extra {"type": "image"} where you don't want to put an image