Vector Databases Explained: When You Need Them and When You Don't

The Problem With How Vector Databases Are Discussed

Every AI-in-production article seems to include a vector database these days. They've become a sort of shorthand for "serious AI implementation" — the thing you include in your architecture diagram to signal that you're doing real AI work, not just calling an API.

The reality is more nuanced. Vector databases are genuinely important for specific use cases. They're overkill — or just wrong — for others. And because the hype has outrun the explanation, a lot of developers and architects are either building vector search infrastructure they don't need or missing places where they'd benefit from it.

Let me explain what vector databases actually are, what they're actually for, and how to decide whether you need one.

What a Vector Database Is (Plain English)

A vector database stores numerical representations of data — called embeddings or vectors — and makes it fast to find items that are similar to a query based on mathematical distance between their vectors.

Here's what that means in practice. When you embed a piece of text (using a model like OpenAI's embedding models or Anthropic's Claude), you convert it into a list of numbers — typically 768 to 3072 numbers — that capture the semantic meaning of the text in a geometric space. Texts with similar meanings end up with vectors that are mathematically close to each other.

A vector database stores these vectors alongside the original content and provides efficient similarity search — "find the 10 stored items most similar to this query." It does this at a speed that would be impossible if you computed similarity between the query and every stored item sequentially.

The key word is "similar" in the semantic sense — conceptually related — rather than "matching" in the keyword sense. This is the fundamental difference from traditional search. A keyword search for "automobile" won't return documents about "cars" unless they contain the word "automobile." A semantic vector search will.

The Use Cases Where Vector Databases Earn Their Place

RAG (Retrieval-Augmented Generation)

This is the primary use case driving vector database adoption. RAG is the pattern where, instead of relying purely on the language model's parametric knowledge (what it learned during training), you retrieve relevant documents from your knowledge base and include them in the model's context.

The flow: user asks a question, you embed the question, you query your vector database for the most semantically similar document chunks, you include those chunks in the model's prompt, the model answers the question grounded in the retrieved content.

Vector databases are the right tool for this because semantic similarity is exactly what you need. You want documents that are conceptually related to the question, not just documents that contain the same words. A question about "budget forecasting" should retrieve documents about "financial projections" and "revenue planning" even if they don't use the words "budget forecasting."

If you're building a RAG system over a non-trivial document corpus (more than a few hundred documents), you need a vector database or similar vector search infrastructure.

Semantic Search Applications

Any application where users search for content by meaning rather than keywords benefits from vector search. Customer support knowledge bases where users describe their problem in their own words. Internal documentation search where the terminology in the documents doesn't always match how people ask questions. Product search where "comfortable running shoes for wide feet" should return relevant results regardless of how the product is described.

Vector search dramatically improves search quality in these scenarios over keyword approaches, and the improvement is user-visible — users find what they're looking for more often.

Recommendation Systems Based on Content Similarity

If you're recommending content, products, or items based on similarity to something the user has engaged with, vector similarity is a natural fit. Embed the items, store the embeddings, query for items similar to the user's history.

This is different from collaborative filtering (recommending based on similar users' behavior), but complementary to it. Content-based vector recommendation works well for cold-start scenarios where you don't have behavioral data on the user.

Deduplication and Near-Duplicate Detection

Finding documents, records, or content items that are similar but not identical is a natural vector database use case. Duplicate customer records with slightly different name spellings. Near-duplicate product listings. Plagiarism detection. Code clone detection.

Traditional exact-match deduplication misses these. Vector similarity finds them.

When You Don't Need a Vector Database

Small-Scale or Fixed-Size Content Collections

If your application needs semantic search over a fixed collection of a few hundred documents, you don't need a dedicated vector database. You can embed the documents at startup, store the embeddings in memory or in a regular database, and compute similarity at query time. The computation cost is negligible at this scale.

Reaching for Pinecone or Weaviate for 200 FAQ entries is over-engineering. A simple in-memory similarity search is fine.

When Keyword Search Actually Meets the Need

If users are searching for specific technical terms, product codes, exact names, or other precision-required queries, keyword search (with good indexing) often works better than semantic search. Semantic search trades off precision for recall — it finds more potentially relevant things at the cost of sometimes finding things that are semantically adjacent but not actually what the user wanted.

PostgreSQL with good full-text search, or Elasticsearch for scale, solves keyword search problems well. Don't add a vector database when what you need is a good keyword index.

When You Haven't Modeled Your Data Yet

I see teams spin up vector databases before they've thought through their data model, chunking strategy, and embedding approach. This is backwards. The vector database is infrastructure for a retrieval system. Design the retrieval system first — what are you searching, what chunking strategy makes sense, what metadata do you need to filter by — then choose the infrastructure.

Choosing Between Vector Database Options

The market has settled around a few main categories:

Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus): Purpose-built for vector search, excellent performance at scale, rich filtering capabilities. Right choice when vector search is a primary workload and you're operating at significant scale.

PostgreSQL with pgvector: Adds vector operations to PostgreSQL. If you're already on PostgreSQL, this is often the right starting point. Simpler infrastructure, good performance for moderate scale, SQL query interface. My default choice for new projects that need vector search at reasonable scale.

Embedded options (SQLite with vector extensions, local FAISS): For applications that run locally or need self-contained vector search without network dependencies. Good for AI-native desktop tools or edge deployments.

Managed cloud services (AWS OpenSearch with k-NN, Azure Cognitive Search, Google Vertex AI Search): Appropriate when you're deeply embedded in a cloud provider's ecosystem and want managed vector search integrated with other services.

My recommendation for most enterprise projects starting out: begin with PostgreSQL + pgvector. It gives you vector search with the operational simplicity of a database you're already managing. Migrate to a dedicated vector database when you have evidence that pgvector's performance is a bottleneck at your scale — which, for most enterprise applications, is a long way off.

The Architecture Decision

Vector databases are a real tool that solves real problems. They're not magic, and they're not required just because you're doing AI work.

The decision framework I use: Am I building semantic search or RAG over a non-trivial corpus? If yes, vector search infrastructure is appropriate — start with pgvector, scale to a dedicated solution if needed. Is this a content similarity or recommendation use case? Same answer. Otherwise, examine whether the problem is actually better solved by keyword search, exact matching, or structured queries before introducing the complexity of vector infrastructure.

When the use case is right, vector databases unlock capabilities that aren't achievable with traditional data infrastructure. When the use case doesn't require them, they add operational complexity and cost without proportional benefit.

If you're evaluating whether your application needs vector search and want a clear assessment of the trade-offs, schedule a conversation at Calendly. I'm happy to help you make the right architectural call for your specific situation.