Fine-tuning Multimodal Embedding Models
This is the 4th article in a larger series on multimodal AI. In the previous post, we discussed multimodal RAG systems, which can retrieve and synthesize information from different data modalities (e.g. text, images, audio). There, we saw how we could implement such a system using CLIP. One issue with this approach, however, is that vector search results from a general-purpose embedding model (like CLIP) may perform poorly in domain-specific use cases. In this article, I'll discuss how we can mitigate these issues via fine-tuning multimodal embedding models.
Multimodal embeddings represent multiple data modalities in the same vector space such that similar concepts are co-located. A visual example of this is shown below, where semantically similar items (e.g. a picture of a dog and its corresponding caption) are close, while dissimilar items (e.g. a picture of a cat and a caption describing a dog) are far apart.
A popular multimodal embedding model is CLIP, which was trained on a massive corpus of image-caption pairs using contrastive learning. The key insight from CLIP was that such a model unlocks 0-shot abilities such as image classification, search, and captioning [1].
One limitation here is that CLIP's 0-shot abilities may not transfer well to domains involving specialized information e.g. architectural drawings, medical imaging, and technical jargon. In such cases, we can improve CLIP's performance through fine-tuning.
Fine-tuning CLIP
Fine-tuning involves adapting a model to a particular use case through additional training. This is powerful because it enables us to build on top of existing state-of-the-art models to develop powerful specialized models with relatively small data.
We can do this with CLIP through the following key steps.
- Collect text-image training pairs
- Pre-process training data
- Define Evals
- Fine-tune the model
- Evaluate the model
I will discuss each of these steps in the context of a concrete example. If you are curious about what this looks like for text embedding (i.e. text-text pairs), I did that in a previous blog post.
Example: Fine-tuning CLIP on YouTube Titles and Thumbnails
Here, I will fine-tune CLIP on titles and thumbnails from my YouTube channel. At the end of this, we will have a model that can take title-thumbnail pairs and return a similarity score. This can be used for practical applications such as matching title ideas to an existing thumbnail or performing search over a thumbnail library.
The example code, dataset, and fine-tuned model are freely available on GitHub and the Hugging Face Hub, respectively. You can use this code and data to train your own models. If you end up publishing any work using this dataset, please cite the original source
