How Vector Databases Power AI Search (And What Experts Do)

You type deploy fails when env var is empty into a docs search box. The page that actually fixes your problem never uses the word "deploy" once — it says "missing key at runtime." Keyword search shrugs. Vector search finds it anyway.

That gap between what you typed and what the answer says is the whole reason this category exists. But the interesting part isn't the magic trick. It's that most teams ship a mediocre version of it and never find out.

What a vector database actually does

An embedding model turns text into a list of numbers — a vector — where similar meanings sit close together in space. OpenAI's text-embedding-3-small gives you 1,536 floats per chunk. That's the input.

A vector database does four jobs on top of that: store the vectors, store metadata alongside them, build an approximate nearest-neighbor (ANN) index so lookups don't scan everything, and keep all of it fresh as documents change. Then it answers one question, very fast: give me the k closest vectors to this query, filtered by these conditions.

That last clause is where products separate. A library like FAISS can find neighbors beautifully. It won't handle multi-tenant filtering, live updates, or a backup story. Those are database problems, not math problems.

Quick sanity check on scale: one million chunks at 1,536 dimensions × 4 bytes per float is roughly 6 GB of raw vectors before any index overhead. That number drives most architecture decisions below.

How vector search works under the hood

Almost nobody searches exhaustively — comparing your query to a million vectors is too slow. So indexes cheat, carefully.

HNSW builds a multi-layer graph, like a skip list in graph form. Top layers are sparse highways, bottom layers are dense neighborhood streets. The search walks down, greedy at each level. Two knobs matter: M (edges per node, default 16 in most implementations) and ef_search (how wide the candidate list is at query time). Crank ef_search up and recall improves while latency climbs. There's no free lunch here, just a dial.

IVF clusters vectors with k-means and only searches the nearest few buckets. Fast to build, and you can trade accuracy by probing more lists.

Quantization shrinks memory. Product quantization compresses subvectors; binary quantization goes to 1 bit per dimension — roughly a 32× reduction versus float32. You lose some recall, then recover most of it with a re-scoring pass over the original vectors.

Here's the part that bites people: every one of these is approximate. "Approximate" means you can silently lose the right answer. Expert teams measure that loss instead of assuming it away.

Semantic search vs keyword search: stop picking sides

I've watched teams rip out BM25 and replace it with embeddings, then act surprised when searching for ERR_CONNECTION_REFUSED returns three vaguely related blog posts.

Embeddings are bad at exact strings. Error codes, SKUs, product names, function signatures, legal citations — these need literal matching. Meaning-based retrieval is bad at them by design.

So the teams who get good results run both and fuse the rankings, usually with Reciprocal Rank Fusion (RRF, k≈60 is the common default). Elasticsearch, OpenSearch, Weaviate, Qdrant, and Vespa all ship hybrid scoring now. If your vendor makes you choose, that's a smell.

What experts actually do differently

The model choice is the least interesting decision on this list (and everyone obsesses over it). The rest is where quality lives.

They chunk with structure, not character counts

Fixed 512-token windows slice sentences in half and orphan table headers. Better: split on headings and paragraphs, land in the 200–800 token range with modest overlap, and prepend the document title or section path to each chunk so a fragment still knows where it came from. Some teams store a small chunk for the index but return the parent section to the LLM — "small-to-big" retrieval. Cheap, and it fixes a lot of half-answer problems.

They filter before searching, not after

This one causes more silent failures than anything else. If you retrieve 100 neighbors and then drop the 90 belonging to another tenant, you've got 10 results and terrible recall. Real filters — tenant_id, permissions, date ranges, doc type — need to run inside the index traversal. Post-filtering looks fine in a demo with 40 documents and falls apart at 4 million.

They re-rank

Two-stage retrieval is standard practice now. Pull 50–100 candidates with the fast vector index, then score that short list with a cross-encoder (Cohere Rerank, the BGE reranker family, or a hosted equivalent) and keep the top 5. It's the single biggest quality jump per hour of engineering effort I've seen, and it costs milliseconds.

They build an eval set before they tune anything

Fifty to two hundred real queries with the correct document labeled. Then you measure recall@k and nDCG every time you touch chunking, the embedding model, or the index parameters. Without that, "the results feel better" is the whole methodology — and it's not a methodology.

They treat the embedding model as a schema migration

Change text-embedding-3-small for something newer and every vector in the store is invalid. There's no ALTER TABLE for this. You re-embed the corpus, version the index, and cut over. Teams that plan for it do it in an afternoon. Teams that don't discover it during an incident.

They quantize and shard only after measuring

Compression is a memory-bill decision, not a default. Measure recall at your target k with and without it. Same for sharding — a single well-tuned node holds a lot more than people assume.

A practical shortlist

Option	Pick it when	Watch out for
Postgres + pgvector	You already run Postgres and have under ~10M vectors	Index builds eat RAM; keep it off your hottest primary
Qdrant	You need strong payload filtering and self-hosting	You're now operating another service
Weaviate	You want hybrid search and modules built in	Opinionated schema, some lock-in
Milvus	Serious scale, multiple index types	Operational weight is real
Pinecone	You'd rather not run infrastructure at all	Per-unit costs add up at high query volume
OpenSearch / Elasticsearch	Search is already your thing	Vector tuning is a newer muscle for the team
Chroma, LanceDB	Prototypes, notebooks, local dev	Don't assume the dev story scales to prod

My default recommendation: start on Postgres with pgvector. It's open source, it lives next to your relational data, and it's genuinely good now. Move to a dedicated engine when filtering complexity or index size forces your hand — and by then you'll know exactly which feature you're paying for.

One warning I'll repeat: don't let vector search become your permission system. If access control isn't enforced as a hard filter inside the query, "the model won't surface it" is not a security control.

Key Takeaways

• Vector databases store embeddings plus metadata and answer "nearest k with these filters" fast — filtering placement matters more than the model.
• HNSW's ef_search, IVF's probe count, and quantization are all recall-versus-latency dials. Measure recall, don't assume it.
• Hybrid search with RRF beats pure semantic search whenever exact strings matter (error codes, names, IDs).
• Two-stage retrieval — ANN for candidates, cross-encoder for the shortlist — delivers the best quality-per-effort.
• Build 50–200 labeled queries first. Tuning without an eval set is guesswork.
• Plan for re-embedding as a migration, because you will change models.

Frequently Asked Questions

Do I need a vector database for a small RAG project in 2026? Probably not a dedicated one. pgvector on a Postgres instance you already pay for handles hundreds of thousands of chunks comfortably, and you keep joins, backups, and transactions. Reach for a specialized engine when filtering gets complex or your index outgrows a single node's RAM.

What's the difference between a vector database and a vector index? The index (HNSW, IVF, DiskANN) is the data structure that makes similarity search fast. The database wraps that index with persistence, metadata filtering, updates and deletes, replication, and backups. FAISS gives you an index; Qdrant or Milvus gives you the whole system.

Is semantic search better than keyword search? Neither wins outright. Semantic search handles paraphrases, synonyms, and questions phrased differently from the source. Keyword search wins on exact tokens — codes, names, acronyms. Run both and fuse the results; that's what most production search stacks do now.

How much does a vector database cost? Self-hosted options like pgvector, Qdrant, and Milvus cost you infrastructure and someone's attention. Managed services price by stored vectors and query volume, so small prototypes are cheap and high-traffic systems get expensive fast. Model the re-embedding cost too — it's usually the surprise line item.