Vector Databases: Architecture, Indexing Strategies, Query Pipelines, and Trade-offs

Vector databases power semantic search, recommendation engines, and retrieval-augmented generation by storing and retrieving high-dimensional embeddings with lightning-fast approximate nearest neighbor (ANN) search. Unlike traditional relational stores, they optimize for similarity search across vector spaces using distance metrics like cosine, dot product, and Euclidean. What makes them special? A purpose-built architecture for vector ingestion, training and maintaining indexes, hybrid filtering over metadata, and low-latency query execution. This guide breaks down how vector databases are built, the strengths and weaknesses of common indexing strategies (HNSW, IVF, PQ, DiskANN), and the practical trade-offs between latency, recall, cost, and freshness. Whether you’re scaling semantic search or productionizing RAG, you’ll learn how to choose the right design, tune it, and avoid common pitfalls.

Core Architecture of Vector Databases

At the heart of a vector database is a separation of concerns: the data plane serves queries at low latency, while the control plane orchestrates index building, training, compaction, and cluster management. The data plane handles ingestion of embeddings and metadata, admission into a write-ahead log (WAL), and durable object storage or SSD-backed segments. The control plane triggers background tasks: training centroids for IVF, building HNSW graphs, or quantizing with PQ/OPQ. This division allows online writes alongside offline optimization without blocking queries.

Storage usually combines a columnar metadata store with a vector segment store. Metadata fields support filters (e.g., product category, language) through inverted indexes or bitmap structures, while vectors live in memory-mapped files or compressed blocks to balance RAM pressure and throughput. Many systems maintain a mutable in-memory tier for fresh writes and a compacted immutable tier for query-heavy workloads, periodically merging data to reduce fragmentation and update index topology.

Because distance metrics matter, architectures often normalize vectors at write time for cosine similarity or maintain per-collection settings for dot product vs L2 distance. Systems may support hybrid retrieval by pairing vector indexes with lexical indexes (BM25), enabling fusion strategies. The result is a pipeline that cleanly aligns ingestion, indexing, storage, and query execution with predictable performance.

ANN Indexing Strategies: IVF, HNSW, PQ and Beyond

No single index wins everywhere; each technique makes an explicit latency–recall–memory trade-off. HNSW (Hierarchical Navigable Small World) builds a multi-layer graph enabling fast greedy search with excellent recall at low latency, at the expense of higher memory usage and slower build times. IVF (Inverted File) clusters vectors into coarse centroids; queries probe a subset of lists, making performance tunable via “nprobe” while keeping memory more moderate. PQ/OPQ (Product Quantization / Optimized PQ) compress vectors to compact codes, slashing memory and boosting cache efficiency with some recall loss.

On large datasets that exceed RAM, disk-centric indexes like DiskANN or graph-on-SSD leverage cache-aware layouts and prefetching. They trade warm-up and cold-read characteristics against peak recall but enable billion-scale retrieval on commodity hardware. For GPU acceleration, libraries like FAISS or cuVS use SIMD/GPU kernels and mixed-precision arithmetic, dramatically improving throughput for batch queries and re-ranking stages.

How do you choose? Consider your constraints:

• HNSW: Best for low-latency, high-recall, hot data in RAM. Memory-heavy, slower builds.
• IVF-Flat: Tunable recall/latency; good balance for varied workloads; simpler operations.
• IVF-PQ/OPQ: Memory-efficient at massive scale; small recall tax; excellent for cost control.
• DiskANN / Graph-on-SSD: Enables billion-scale with limited RAM; mind cold start and SSD IOPS.

Training quality (centroids, codebooks) and hyperparameters (M, efConstruction, nlist, nprobe, code size) materially affect recall, so treat indexing as a learnable model, not a fixed setting.

Query Execution Pipeline and Hybrid Retrieval

A robust vector search request flows through distinct stages: request parsing, pre-filtering, candidate generation (ANN), candidate refinement, and response shaping. Pre-filters narrow the search space using metadata indexes (e.g., price range, locale), improving both relevance and cost. The ANN stage returns top-k candidates with approximate scores; a refinement pass can perform exact distance computations on the short list (re-scoring from raw vectors), recovering recall lost to approximation while keeping latency low.

Real workloads require post-filters (permissions, safety, deduplication) and re-ranking with learned models. For RAG, many teams use cross-encoders or MLR re-rankers on the ANN shortlist, or apply MMR (Maximal Marginal Relevance) to increase diversity and reduce redundancy. Robust APIs support attribute boosting, tie-breaking, and pagination without redoing full searches. Careful result caching for frequent queries can help, but remember: vector queries are less cache-friendly than keyword queries due to high dimensionality and personalization.

Hybrid retrieval blends lexical signals (BM25) with semantic vectors. Fusion strategies like RRF (Reciprocal Rank Fusion) or weighted score blending produce superior relevance on head and tail queries. Some databases support payload-aware indexing that co-locates vectors and metadata or uses filter-aware quantization to avoid scanning irrelevant partitions. The key is minimizing wasted distance computations while preserving recall and governance constraints.

Distributed Scaling, Consistency, and Durability

At scale, vector databases shard data across nodes. Common approaches include hash-based sharding on IDs, partitioning by semantic clusters, or a two-level scheme combining both. Routing queries to shards can be random, centroid-aware, or based on learned partitioners to reduce cross-shard fan-out. Replication provides availability and read throughput; leaders handle writes with Raft or similar consensus, while followers serve reads for locality. Cross-AZ/region replication increases resilience but requires careful handling of clock skew and index rebuild windows.

Durability hinges on WAL+snapshot mechanics. Writes land in the WAL, then in-memory segments; periodic snapshots capture a consistent state for fast recovery. Background compaction merges segments, removes tombstones, and retrains or merges index structures. Because ANN indexes aren’t inherently transactional, systems use versioned views: queries run on a consistent snapshot while rebuilds occur in the background, then atomically swap pointers to new indexes.

Updates and deletes deserve special attention. Upserts may create transient duplicates unless you enforce per-ID uniqueness during merge. Deletes can be marked via bitmaps and applied during compaction to avoid expensive in-place graph edits. For strict SLAs, stage index changes behind feature flags, monitor tail latency (p95/p99), and roll back if recall or error rates drift.

Operational Trade-offs and Best Practices

Every production system faces the recall–latency–cost triangle. Need sub-50 ms latency at high recall? Expect higher memory footprint (HNSW or IVF with large nprobe) or GPU inference for re-ranking. Need to cut cost per query? Embrace PQ/OPQ, reduce dimensionality, and tighten filters. To maintain relevance, track Recall@k, NDCG, MRR, and latency percentiles; sample queries for offline evaluation and run A/B tests in shadow mode before flipping traffic.

Embeddings are not static. Model upgrades, domain drift, and multilingual traffic change the distribution. Plan for dual-write migration: index new embeddings in parallel, compare online metrics, then backfill and deprecate the old space. Normalize vectors (L2) if using cosine similarity; regularly audit dimensionality—sometimes 384 dims with strong models outperform bloated 1536-dim embeddings at lower cost and better cache behavior.

Hardware and tuning matter:

• Use SIMD/GPU acceleration for batch distance ops and re-ranking.
• Prefer SSDs with high random IOPS for disk-backed indexes; align block sizes with index prefetch.
• Tune HNSW (M, efSearch) or IVF (nlist, nprobe) per collection; avoid one-size-fits-all settings.
• Compress wisely: OPQ often recovers recall vs vanilla PQ at the same code size.

Finally, add guardrails: rate limits, timeouts, circuit breakers, and backpressure to protect the ANN layer under load.

FAQ

Which distance metric should I use: cosine, dot product, or Euclidean?

Cosine similarity is common for normalized embeddings and language tasks; it’s scale-invariant. Dot product works well when magnitude carries semantic weight (e.g., certain recommendation models). Euclidean (L2) fits dense, isotropic spaces. Many systems normalize vectors and use cosine for stability; choose the metric aligned with how your embedding model was trained.

How many vectors per shard is “too many”?

It depends on index type and hardware. As a rule of thumb, keep per-shard HNSW graphs under a few hundred million vectors in RAM and ensure at least several gigabytes of headroom for graph navigation. For IVF/PQ, billions per shard are feasible with SSDs and careful cache sizing. Monitor p95 latency as you scale; split shards before compaction falls behind.

Do I need re-ranking after ANN?

Often yes. ANN approximates neighbors; a lightweight exact-distance re-score on the top 100–1000 candidates typically improves quality. For high-stakes relevance (RAG, legal, medical), add a cross-encoder or domain-specific re-ranker and layer in MMR for diversity.

How do I handle deletes and GDPR/CCPA requests?

Use soft deletes (bitmaps) for immediate effect and run expedited compactions to physically remove vectors. Maintain deletion logs tied to user IDs, and verify purges through periodic scans or cryptographic tombstone markers. For strict requirements, avoid long-lived snapshots that retain deleted payloads.

Conclusion

Vector databases bring semantic search to production by pairing purpose-built storage with fast ANN indexing and a sophisticated query pipeline. The right architecture separates online serving from background optimization, blends metadata filters with vector retrieval, and scales via sharding and replication without sacrificing durability. Index choice—HNSW, IVF, PQ, DiskANN—defines your recall, latency, and cost envelope; tuning and re-ranking close the quality gap. Operate with discipline: monitor relevance metrics, plan embedding migrations, and optimize hardware paths. With clear goals and measured trade-offs, you can deliver trustworthy, low-latency similarity search that scales from prototypes to billion-vector workloads.