Vector Database

What Is Vector Database?

A vector database is a datastore built to hold embeddings, the numeric vectors that represent the meaning of text, images, or code, and to answer similarity queries over them fast. Given a query vector, it returns the stored items closest in meaning, which makes it the retrieval backbone of RAG systems, semantic search, and agent memory.

Key Takeaways

• The core operation is approximate nearest neighbor search: finding the most similar vectors among millions in milliseconds by trading a small amount of recall for enormous speed.
• Vectors alone are not enough. Production retrieval depends on the metadata stored beside them, for permission filtering, tenant isolation, freshness, and hybrid keyword scoring.
• The category spans dedicated engines and extensions to existing databases, most prominently pgvector inside Postgres, and the extension route wins by default at small and medium scale.
• A vector database returns similar items, not correct answers. Retrieval quality is governed upstream by chunking and the embedding model, not by the database.

How It Works

An embedding model turns each piece of content into a vector embedding, a list of hundreds or thousands of floating-point numbers positioned so that similar meanings land near each other. The database stores these vectors with an ID, the source text or a pointer to it, and structured metadata. A query arrives as another vector, and the database returns the k nearest stored vectors ranked by cosine similarity or dot product.

Comparing the query against every vector, a brute-force scan, is exact but scales linearly and becomes too slow past a few hundred thousand entries. Vector databases therefore build approximate nearest neighbor indexes. The dominant structure is HNSW, a layered graph navigated greedily from coarse to fine; IVF-style indexes that partition the space into clusters are the other common family, and quantization techniques shrink vectors to cut memory. These indexes answer in milliseconds at the cost of occasionally missing a true neighbor, with recall tunable against latency. The scale ceilings were established years before the current AI wave: Meta's Faiss library demonstrated GPU similarity search 8.5x faster than the prior state of the art in 2017, building a high-accuracy k-NN graph over 1 billion vectors in under 12 hours on four GPUs [1]. Microsoft's DiskANN then showed at NeurIPS 2019 that a billion-point index can live on a single workstation with 64GB of RAM and an SSD while serving over 5,000 queries per second at under 3ms mean latency with 95%+ recall [2].

The features that separate a production system from a demo mostly live outside the index. Filtered search applies metadata predicates, such as tenant ID or document permissions, without gutting recall, which is genuinely hard and handled well only by mature engines. Hybrid search fuses vector scores with keyword scoring so exact identifiers still match. Then come the ordinary database virtues: upserts and deletes that keep the index consistent as documents change, namespaces for isolation, and backups. In agent stacks the same machinery serves double duty, powering both document retrieval and long-term agent memory, where past interactions are embedded and recalled by relevance.

Example

A legal-tech startup builds contract search across 2 million clause-level chunks for 400 client firms. Each chunk is embedded and stored with metadata: firm ID, contract date, clause type, governing law. When a lawyer at one firm searches "termination for convenience with less than 30 days notice," the query is embedded and the database runs nearest-neighbor search filtered to that firm's documents only, returning 20 candidate clauses in under 50 milliseconds, which a reranker then narrows to five. The filter is doing legal-grade tenant isolation, so the team verified that filtering happens inside the index traversal rather than after it, both for recall and to guarantee another firm's clauses can never appear. They started on pgvector, and only migrated the retrieval path to a dedicated engine when the corpus and filter complexity outgrew it.

What People Get Wrong

The recurring mistake is starting an AI project by procuring a dedicated vector database, as if it were the hard part. For corpora under a few million vectors, pgvector or the vector type in an existing search engine is usually sufficient, keeps embeddings transactionally consistent next to the source data, and adds zero new operational surface. Meanwhile the actual determinants of retrieval quality, chunking strategy and embedding model choice, get a fraction of the attention. Pick the boring storage option first, spend the saved effort on the pipeline, and upgrade the database when scale or filtering demands force it.

FAQ

Do I need a vector database for RAG? You need vector storage and similarity search; whether that is a dedicated product is a scale question. Retrieval-augmented generation over thousands or even a few million chunks runs fine on pgvector or an existing search stack. Dedicated engines earn their place with very large corpora, heavy filtered-search requirements, or strict latency budgets at high query volume.

How is a vector database different from a regular database? A regular database retrieves rows matching exact conditions. A vector database ranks items by geometric similarity to a query vector, a fuzzy, meaning-based operation that requires specialized indexes. In practice the line has blurred: Postgres, OpenSearch, Redis, and most major databases now bolt on vector search, so "vector database" increasingly names a capability rather than a separate product category.

What actually gets stored in one? Three things per item: the embedding vector, metadata for filtering, and the payload, either the original chunk text or a reference to it. The embeddings are produced outside the database by an embedding model, and if you ever change that model, every vector must be regenerated, since vectors from different models are not comparable.

Sources

1. Johnson, Douze & Jégou, Meta AI. "Faiss billion-scale GPU similarity search benchmarks." https://arxiv.org/abs/1702.08734. Accessed August 2026.
2. Subramanya et al., Microsoft Research. "DiskANN billion-point search on a single workstation." https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node. Accessed August 2026.