Building Image Search System Leveraging Modern AI Services

In today’s digital landscape, the ability to efficiently search through vast image repositories has become increasingly important across industries. Whether for e-commerce platforms, content management systems, or digital asset management, the demand for accurate and contextually relevant image search capabilities continues to grow exponentially.

The challenge lies not just in searching through the images, but in understanding the semantic content, style, emotional tone, and numerous other attributes that make image search valuable to end users.

This blog post details our recent implementation of a large-scale hybrid image search system capable of processing and retrieving images based on both visual content and textual queries. By leveraging state-of-the-art AI models and cloud services, we were able to construct a robust system that handles a repository of one million images with impressive accuracy and reasonable latency.

More importantly, we’ll highlight how the integration of modern AI tools significantly reduced the development time and effort traditionally required for such complex systems.

Image Search System Architecture

System Architecture Overview

Our hybrid image search system combines multiple specialized components to create a comprehensive solution. The diagram above illustrates the high-level architecture of our system. Let’s examine each component in detail.

Dual Embedding Approach

Our system fundamentally relies on two distinct embedding pipelines to capture different aspects of the image content:

Embedding Models

Image Embeddings via SigLIP We selected the SigLIP (Simple Gated Linear Projection) model after evaluating several alternatives including CLIP and ImageBind. SigLIP was chosen for several compelling reasons:

Superior Performance: SigLIP demonstrates state-of-the-art performance on image-text retrieval benchmarks with 76.3% zero-shot top-1 accuracy on ImageNet, outperforming CLIP by 2–3% across most benchmarks.

Efficiency: SigLIP’s architecture is more parameter-efficient than alternatives, requiring approximately 40% less computational resources than comparable CLIP models.

Adaptability: SigLIP’s contrastive learning approach allows it to generalize well to new domains without fine-tuning.

The SigLIP model processes each image and generates a 512-dimensional dense vector that captures visual characteristics in a semantically meaningful space.

Text Embeddings via SPLADE v2 For text representation, we selected SPLADE v2 (Sparse Lexical And Expansion) after comparing it with alternatives:

Sparse Representation: Unlike dense embedding models, SPLADE generates interpretable sparse embeddings that maintain direct connections to the original vocabulary.

Query Expansion: SPLADE automatically performs lexical expansion, addressing vocabulary mismatch issues common in information retrieval.

Explainability: The sparse nature of SPLADE embeddings allows us to trace back exactly which terms contributed to a match, enhancing the explainability of our system.

The foundation of our system is a robust image processing pipeline that extracts multiple layers of information from each image. This multi-stage process transforms raw pixels into rich, queryable representations.

Object Detection with YOLO

We implemented YOLOv8 (You Only Look Once, version 8) for object detection, which provides several advantages for our large-scale system:

Single-pass efficiency: YOLO processes the entire image in a single network pass, making it significantly faster than two-stage detectors like Faster R-CNN.

Real-time performance: With our hardware configuration, YOLOv8-Large processes images at an average of 50ms per image, enabling efficient batch processing.

High accuracy: YOLOv8 achieves 53.9% mAP on COCO dataset, providing reliable detection of 80 common object classes.

Scalability: The model’s architecture allowed us to efficiently distribute processing across our GPU cluster.

Entity Recognition and Classification

The object detection output provides generic classes (person, car, building), but for sophisticated search capabilities, we needed to identify specific entities. We leveraged Google’s Web Extraction Service to transform generic objects into named entities:

Example Transformation:

YOLO: “person” (coordinates: [120, 45, 220, 320])
Web Extraction: “Elon Musk” (confidence: 0.96)

YOLO: “building” (coordinates: [50, 100, 400, 600])
Web Extraction: “Eiffel Tower” (confidence: 0.92)

The service provides recognition for over 5 million entities across categories including:

Celebrities and public figures

Landmarks and locations

Products and brands

Events and performances

This entity recognition layer bridges the gap between visual detection and semantic understanding, enabling queries like “find images with Taylor Swift” rather than just “find images with a person.”

Attribute Detection with Google Vision API

We utilized Google Vision API’s comprehensive capabilities to extract additional metadata:

Label Detection: This service identifies general concepts, activities, and themes in images with remarkable accuracy. In our testing, it correctly identified over 10,000 distinct concepts across our image repository.

The label detection was particularly valuable for identifying:

Activities (e.g., “hiking,” “cooking,” “celebrating”)

Scenes (e.g., “beach,” “urban,” “forest”)

Objects too specific for YOLO’s generic classes (e.g., “cappuccino” vs. YOLO’s “cup”)

Safe Search Detection: This critical component flags potentially sensitive content across five dimensions:

Adult content

Violence

Medical imagery

Spoof (memes, edited content)

Racy content

Each dimension receives a confidence score, enabling graduated filtering based on organizational policies. This automated safety layer saved an 100s of hours of manual content review.

Metadata Enrichment with Gemini

A key innovation in our system was using Gemini for comprehensive metadata generation. After extensive prompt engineering and testing, we developed a structured prompt template that reliably extracts 25 distinct attributes:

System: You are an expert image analyzer. Extract the following attributes from the image, with a confidence score (0-1):

1. sensitivity (none, low, medium, high)
2. emotion (neutral, joy, sadness, surprise, etc.)
3. emotion_triggered (yes/no)
4. text_overlay (yes/no)
5. has_frames (yes/no)
6. has_religious_symbols (yes/no)
7. has_scattered_objects (yes/no)
8. style (photographic, illustrated, cartoon, abstract, etc.)
9. has_crowd (yes/no)
...
[full list of 25 attributes]

Response format: JSON object with attributes as keys and values as described above.

Generating Enriched Image Descriptions

Using the combined outputs from object detection, entity recognition, attribute classification, and metadata analysis, we prompted Gemini to generate comprehensive image descriptions. The prompt engineering for this step was particularly crucial, as we needed descriptions that were:

Semantically rich: Containing relevant keywords to facilitate text search

Structurally consistent: Following a pattern that enabled reliable parsing

Contextually meaningful: Capturing the significance of detected entities

Attribute-aware: Incorporating key metadata attributes

Search-optimized: Including likely search terms

Our final prompt template included:

Generate a comprehensive, search-optimized description for this image based on the following data:

[Entity data from object detection and recognition]
[Label data from Vision API]
[Structured metadata from previous Gemini analysis]

Your description should:
1. Begin with the most significant entities and their actions/relationships
2. Include key visual attributes (colors, style, composition)
3. Mention emotional tone and aesthetic qualities
4. Incorporate likely search terms
5. Be 3-5 sentences in length

Example output transformation:

Before (raw detection data):

Objects: person (0.98), guitar (0.95), microphone (0.92)
Entities: Taylor Swift (0.97)
Labels: concert, performance, stage, entertainment
Metadata: emotion=joy, has_crowd=yes, style=photographic, dominant_color=purple

After (enriched description):

Taylor Swift performs energetically on stage with an acoustic guitar during a concert, singing into a microphone with passionate expression. The image captures the excitement of a live performance with purple stage lighting creating a vibrant atmosphere. This high-quality photograph conveys feelings of joy and excitement, with Swift’s iconic performance style clearly visible. The composition includes partial views of an enthusiastic crowd in the foreground, making this suitable for music, entertainment, and celebrity content.

This transformation creates descriptions that are:

Rich in searchable content

Contextually meaningful

Inclusive of visual and emotional attributes

Reflective of likely user search queries

Query Processing Flow

The query processing pipeline involves several sophisticated steps:

Query Analysis: Incoming queries undergo intent classification to determine whether the user is looking for specific entities, styles, emotions, or other attributes.

Entity Extraction: Named entities in the query are identified and matched against our entity database to ensure proper weighting in the retrieval process.

Text Embedding Generation: The processed query is embedded using the SPLADE v2 and SigLip models, matching the representation used for our images.

Vector Similarity Search: The query embedding is used to find the most similar image embeddings in our vector index.

Policy-Based Filtering: Results are filtered based on region-specific and organization-level policies. This includes:
- Content safety thresholds appropriate to the user’s region
- Brand safety requirements
- Age-appropriate content filtering
- Licensing and usage rights enforcement

Multi-Factor Reranking: The initial results undergo sophisticated reranking based on:

Diversity: Ensuring visual and semantic variety in results using a greedy diversity algorithm that maintains a minimum cosine distance between selected images

Recency: Boosting newer content with a time-decay function: score_boost = base_score * (1 + alpha * exp(-beta * days_old))

Popularity: Incorporating usage statistics with a logarithmic dampening to prevent dominance of highly popular items

Business Rules: Applying domain-specific scoring adjustments based on business requirements

Implementation Challenges and Solutions

Several significant challenges emerged during implementation:

1. Latency Management
Challenge: Processing images through multiple services introduced cumulative latency, with initial end-to-end processing times exceeding 4 seconds per image.

Solution: We implemented:

Parallel processing across services using asynchronous workflows

Batch processing for non-interactive pipelines

Optimized service placement to minimize network latency

Tiered processing based on image complexity

These optimizations reduced average processing time to 850ms per image, with 95% of images processed in under 1.2 seconds.

2. Integration Complexity
Challenge: Coordinating multiple services with different APIs, error handling patterns, and performance characteristics proved challenging.

Solution: We developed:

A unified service abstraction layer with consistent error handling

Circuit breakers and fallback mechanisms for each service

Comprehensive monitoring with service-specific health metrics

Staged degradation capabilities to maintain system functionality during partial outages

Business Impact and ROI

The implementation of our hybrid image search system delivered substantial business value:

Reduced Time-to-Market: Development completed in 4 months, compared to an estimated 12+ months for a custom-built solution

Cost Efficiency: reduction in total cost of ownership compared to previous-generation systems

Improved User Engagement: 40% increase in search results satisfaction

Operational Efficiency: 80% reduction in manual tagging and categorization efforts

Future Directions

While our current implementation meets our immediate requirements, several promising directions for future enhancement include:

Personalization: Incorporating user preferences and behavior patterns into the retrieval process

Interactive Refinement: Adding capabilities for users to interactively refine search results

Multimodal Queries: Supporting queries that combine text, images, and other modalities

Continuous Learning: Implementing feedback loops to improve relevance based on user interactions

Domain Adaptation: Fine-tuning embedding models for specific industry verticals

FAQs

What is the difference between SigLIP and CLIP for image search?

SigLIP (Simple Gated Linear Projection) outperforms CLIP on image-text retrieval benchmarks with 76.3% zero-shot top-1 accuracy on ImageNet, compared to CLIP's 74-75% range — a 2-3% improvement across most benchmarks. More significantly for production systems, SigLIP requires approximately 40% less computational resources than comparable CLIP models, making it meaningfully more efficient at scale. SigLIP's contrastive learning approach also generalises better to new domains without fine-tuning, which is critical when the image domain (fashion, products, lifestyle) differs from the pretraining distribution.

How do you reduce latency in a multimodal image search pipeline?

The primary techniques that reduced our end-to-end latency from 4 seconds to 850ms: 

(1) parallel processing across services using asynchronous workflows rather than sequential calls — the biggest single gain; 

(2) batch processing for non-interactive pipelines to amortise overhead;

(3) optimised service placement to minimise network round-trips;

(4) tiered processing based on image complexity so simple images do not go through the full pipeline. The 95th percentile landed at 1.2 seconds after these optimisations. The key insight is that the bottleneck is almost always I/O and service coordination rather than model inference itself.

Why use SPLADE v2 rather than a dense embedding model for text search?

For image search specifically, SPLADE v2's sparse representation has two advantages over dense models like sentence-transformers. First, SPLADE's automatic query expansion handles vocabulary mismatch — if a user searches 'Taylor Swift concert' and the image description says 'performance' rather than 'concert', SPLADE expands the query to cover synonyms and related terms. Second, the sparse embeddings are interpretable — you can trace exactly which terms contributed to a match, which is important for debugging relevance issues and applying policy-based filtering. For a system that needs to explain why a result was returned (safety, brand guidelines, licensing), explainability is not optional.