Chapter 5: What Is a Vector Database, and Why?

Time: 10 minutes. Cost: $0 with Ollama, a fraction of a cent with OpenAI.

Imagine a librarian who shelves books not alphabetically by title, but by what they're actually about, so that two books on similar topics end up right next to each other on the shelf, even if their titles have nothing in common. Hand her a brand-new book, and she doesn't scan every title in the building. She already knows roughly where it belongs, and walks straight there.

A vector database does exactly this with embeddings. It's a place to store the vectors you learned about in Chapter 4, built specifically so that finding "what's closest in meaning to this new thing" is fast, even when you're searching millions of them.

A bit of history: the algorithm that made fast nearest-neighbor search practical at real scale, called FAISS, came out of Facebook AI Research in 2017. Dedicated vector database products built around that idea, like Pinecone, Weaviate, and Chroma, followed a few years later, mostly 2019 to 2022, once developers building search and RAG-style tools created real demand for this as its own category of software.

Why Chapter 4's approach doesn't scale

In the Chapter 4 lab, you compared six sentences against each other with a plain Python loop: embed everything, then check cosine similarity between every pair by hand. That works fine for six sentences.

Now imagine a million documents instead of six. Every time you want to find the closest match to a new sentence, that approach means comparing it against all one million stored embeddings, one at a time, from scratch. That's slow, and it gets slower as your collection grows. A vector database solves this with smarter indexing, so it can find close matches without checking every single record.

What a vector database actually stores and does

At its core, a vector database holds records that look like this:

Field	Example
Embedding	[0.12, -0.44, 0.08, ...]
Original text	"my dog won't stop barking"
Metadata (optional)	{"topic": "pets"}

You can add as many of these records as you want. Then, when you have a new piece of text, you embed it the same way, hand that new vector to the database, and ask for the top-k nearest matches, meaning "give me the k records whose embeddings are closest to this one." The database returns them fast, using an index built for exactly this kind of search, instead of comparing against every record one by one.

"Closest" has to mean something specific, and that's where the distance metric from Chapter 4 comes back in. Most vector databases default to Euclidean distance rather than cosine similarity, so when you create a collection, you tell it which one to use. The lab below explicitly asks for cosine, to match the similarity scores you already saw in Chapter 4.

That optional metadata is also worth knowing about: most vector databases let you filter by it alongside the similarity search, for example "only search records where topic is pets." You won't need this yet, but it's a useful thing to know exists. Intermediate covers it in more depth.

Hands-on lab: store and query vectors locally

In this lab you'll store around ten sentences in a local vector database, then send in one new sentence and watch the database instantly return the closest matches, without ever comparing it to every sentence by hand.

Full instructions: labs/foundations/05-vector-db-basics

Here's what you should see:

Adding 10 sentences to the vector database...

Query: "my cat keeps meowing at 3am"

Top 3 closest matches:
1. (0.63) my dog won't stop barking
2. (0.52) our puppy barks at everything
3. (0.46) I'm making pasta for dinner

💡 One thing to know before you run it: just like Chapter 4, this lab needs an embedding model, and Anthropic doesn't offer one. If your .env still has PROVIDER=anthropic, switch it to ollama or openai for this lab.

Checkpoint

What problem does a vector database solve that a plain Python loop doesn't?

Speed at scale. A loop comparing a new vector against every stored vector one by one works fine for a handful of items, but gets slow as the collection grows. A vector database uses an index built for nearest-neighbor search, so it can find close matches quickly even across millions of records.

What gets stored in a vector database record?

At minimum, an embedding and the original text it came from. Often also optional metadata, extra fields like a category or a source, that you can filter on alongside the similarity search.

What does "top-k nearest neighbors" mean?

Given a new vector, return the k stored vectors that are closest to it in meaning (k being however many results you asked for, like the top 3).

Check Your Knowledge

Click to start the quiz

1. According to the chapter's history note, what came first: FAISS (the fast nearest-neighbor search algorithm), or dedicated vector database products like Pinecone, Weaviate, and Chroma?

Vector database products came first, then FAISS was built to speed them upFAISS came out of Facebook AI Research in 2017; dedicated vector database products followed a few years later, mostly 2019 to 2022They were all released in the same yearChroma predates FAISS by over a decade

2. The chapter mentions optional metadata you can filter on alongside a similarity search, like "only search records where topic is pets." How deep does this chapter actually go into that feature?

It builds a full working example of metadata filtering in the labIt only mentions that metadata filtering exists, and says Intermediate covers it in more depthIt says metadata filtering is impossible in vector databasesIt requires a separate paid product to use at all

3. In the lab's real run, a cat-related query returns the two dog sentences as its top two matches (0.89 and 0.84), well ahead of the third match at 0.21. What does that gap actually show?

That the vector database made an error and should be re-runThat the two dog sentences are much more related in meaning to the cat query than the third sentence is, exactly the kind of gap similarity scores are meant to surfaceThat a score of 0.21 means the third sentence is written in a different languageThat scores below 0.5 are automatically discarded by the database

4. What distance metric does Chroma default to, and why does that matter for this course?

Chroma defaults to cosine similarity, so no extra setup is ever neededChroma does not support choosing a distance metric at allChroma always uses dot product and it cannot be changedChroma defaults to Euclidean distance, so you need to explicitly ask for cosine similarity when creating a collection to match the scores used since Chapter 4

What's next

You can now store text as searchable vectors and instantly pull back the closest matches. Chapter 6 puts that to work: instead of just returning matching text, you'll hand what you found to an LLM so it can answer a question using it. That combination has a name: RAG.