A developer built a semantic search tool for a photo library over the weekend. The library holds 42,000 images spread across 2,800 galleries. After finishing the tool, the search results turned out better than expected.

There was a critical pitfall in the process. When asking an AI model to generate embeddings directly from the images themselves, the performance was terrible — the model simply couldn’t understand what was in the pictures. For context, embeddings can be thought of as converting images or text into sequences of numbers; the closer the numbers are to each other, the more similar the underlying content is. They switched to an alternative approach: first use gpt-4o-mini to write a text caption for each image, then generate embeddings from these captions. The performance improved immediately.

In terms of cost, caption generation and embedding generation combined cost a total of $30. The embedding model used was text-embedding-3-large, with 3072 dimensions.


What makes this case interesting is that image search ultimately bypassed the images themselves. When generating image embeddings directly, the model captures visual features rather than semantic content. Generating captions first explicitly translates what is in the image into text, turning the search into a matching process between text embeddings.

Next time you run into underperforming vision models, try having the model write out what it sees first.