ColPali Query Generator using Qwen2.5-VL

ColPali is a very exciting new approach to multimodal document retrieval which aims to replace existing document retrievers which often rely on an OCR step with an end-to-end multimodal approach.

To train or fine-tune a ColPali model, we need a dataset of image-text pairs which represent the document images and the relevant text queries which those documents should match. To make the ColPali models work even better we might want a dataset of query/image document pairs related to our domain or task.

One way in which we might go about generating such a dataset is to use a VLM to generate synthetic queries for us. This space uses the Qwen/Qwen2.5-VL-7B-Instruct VLM model to generate queries for a document, based on an input document image.

Note there is a lot of scope for improving to prompts and the quality of the generated queries! If you have any suggestions for improvements please open a Discussion!

This blog post gives an overview of how you can use this kind of approach to generate a full dataset for fine-tuning ColPali models.

If you want to convert a PDF(s) to a dataset of page images you can try out the PDFs to Page Images Converter Space.