When (Not) to Use Vector DB

. They remedy an actual drawback, and in lots of instances, they’re the precise alternative for RAG methods. However right here’s the factor: simply since you’re utilizing embeddings doesn’t imply you want a vector database.

We’ve seen a rising development the place each RAG implementation begins by plugging in a vector DB. Which may make sense for large-scale, persistent data bases, however it’s not all the time essentially the most environment friendly path, particularly when your use case is extra dynamic or time-sensitive.

At Planck, we make the most of embeddings to reinforce LLM-based methods. Nonetheless, in one among our real-world functions, we opted to keep away from a vector database and as a substitute used a easy key-value retailer, which turned out to be a significantly better match.

Earlier than I dive into that, let’s discover a easy, generalized model of our state of affairs to clarify why.

Foo Instance

Let’s think about a easy RAG-style system. A person uploads a couple of textual content recordsdata, perhaps some stories or assembly notes. We break up these recordsdata into chunks, generate embeddings for every chunk, and use these embeddings to reply questions. The person asks a handful of questions over the following couple of minutes, then leaves. At that time, each the recordsdata and their embeddings are ineffective and could be safely discarded.

In different phrases, the info is ephemeral, the person will ask solely a few questions, and we wish to reply them as quick as doable.

Now pause for a second and ask your self:

The place ought to I retailer these embeddings?

Most individuals’s intuition is: “I’ve embeddings, so I want a vector database”, however pause for a second and take into consideration what’s really taking place behind that abstraction. If you ship embeddings to a vector DB, it doesn’t simply “retailer” them. It builds an index that quickens similarity searches. That indexing work is the place loads of the magic comes from, and in addition the place loads of the price lives.

In a long-lived, large-scale data base, this trade-off makes good sense: you pay an indexing price as soon as (or incrementally as knowledge adjustments), after which unfold that price over thousands and thousands of queries. In our Foo instance, that’s not what’s taking place. We’re doing the alternative: consistently including small, one-off batches of embeddings, answering a tiny variety of queries per batch, after which throwing every part away.

So the actual query is just not “ought to I take advantage of a vector database?” however “is the indexing work price it?” To reply that, we are able to have a look at a easy benchmark.

Benchmarking: No-Index Retrieval vs. Listed Retrieval

This part is extra technical. We’ll have a look at Python code and clarify the underlying algorithms. If the precise implementation particulars aren’t related to you, be happy to skip forward to the Outcomes part.

We wish to examine two methods:

1. No indexing in any respect, simply retains embeddings in reminiscence and scans them instantly.
2. A vector database, the place we pay an indexing price upfront to make every question quicker.

First, think about the “no vector DB” strategy. When a question is available in, we compute similarities between the question embedding and all saved embeddings, then choose the top-k. That’s simply Ok-Nearest Neighbors with none index.

import numpy as np

def run_knn(embeddings: np.ndarray, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
    sims = embeddings @ query_embedding
    return sims.argsort()[-top_k:][::-1]

The code makes use of the dot product as a proxy for cosine similarity (assuming normalized vectors) and types the scores to search out the perfect matches. It actually simply scans all vectors and picks the closest ones.

Now, let’s have a look at what a vector DB usually does. Underneath the hood, most vector databases depend on an approximate nearest neighbor (ANN) index. ANN strategies commerce a little bit of accuracy for a big enhance in search velocity, and probably the most extensively used algorithms for that is HNSW. We’ll use the hnswlib library to simulate the index conduct.

import numpy as np
import hnswlib

def create_hnsw_index(embeddings: np.ndarray, num_dims: int) -> hnswlib.Index:
    index = hnswlib.Index(house='cosine', dim=num_dims)
    index.init_index(max_elements=embeddings.form[0])
    index.add_items(embeddings)
    return index

def query_hnsw(index: hnswlib.Index, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
    labels, distances = index.knn_query(query_embedding, ok=top_k)
    return labels[0]

To see the place the trade-off lands, we are able to generate some random embeddings, normalize them, and measure how lengthy every step takes:

import time
import numpy as np
import hnswlib
from tqdm import tqdm

def run_benchmark(num_embeddings: int, num_dims: int, top_k: int, num_iterations: int) -> None:
    print(f"Benchmarking with {num_embeddings} embeddings of dimension {num_dims}, retrieving top-{top_k} nearest neighbors.")

    knn_times: listing[float] = []
    index_times: listing[float] = []
    hnsw_query_times: listing[float] = []

    for _ in tqdm(vary(num_iterations), desc="Operating benchmark"):
        embeddings = np.random.rand(num_embeddings, num_dims).astype('float32')
        embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
        query_embedding = np.random.rand(num_dims).astype('float32')
        query_embedding = query_embedding / np.linalg.norm(query_embedding)

        start_time = time.time()
        run_knn(embeddings, query_embedding, top_k)
        knn_times.append((time.time() - start_time) * 1e3)

        start_time = time.time()
        vector_db_index = create_hnsw_index(embeddings, num_dims)
        index_times.append((time.time() - start_time) * 1e3)

        start_time = time.time()
        query_hnsw(vector_db_index, query_embedding, top_k)
        hnsw_query_times.append((time.time() - start_time) * 1e3)

    print(f"BENCHMARK RESULTS (averaged over {num_iterations} iterations)")
    print(f"[Naive KNN] Common search time with out indexing: {np.imply(knn_times):.2f} ms")
    print(f"[HNSW Index] Common index development time: {np.imply(index_times):.2f} ms")
    print(f"[HNSW Index] Common question time with indexing: {np.imply(hnsw_query_times):.2f} ms")

run_benchmark(num_embeddings=50000, num_dims=1536, top_k=5, num_iterations=20)

Outcomes

On this instance, we use 50,000 embeddings with 1,536 dimensions (matching OpenAI’s text-embedding-3-small) and retrieve the top-5 neighbors. The precise outcomes will differ with completely different configs, however the sample we care about is identical.

I encourage you to run the benchmark with your individual numbers, it’s one of the best ways to see how the trade-offs play out in your particular use case.

On common, the naive KNN search takes 24.54 milliseconds per question. Constructing the HNSW index for a similar embeddings takes round 277 seconds. As soon as the index is constructed, every question takes about 0.47 milliseconds.

From this, we are able to estimate the break-even level. The distinction between naive KNN and listed queries is 24.07 ms per question. That suggests you want 11,510 queries earlier than the time saved on every question compensates for the time spent constructing the index.

Moreover, even with completely different values for the variety of embeddings and top-k, the break-even level stays within the 1000’s of queries and stays inside a reasonably slim vary. You don’t get a state of affairs the place indexing begins to repay after just some dozen queries.

Now examine that to the Foo instance. A person uploads a small set of recordsdata and asks a couple of questions, not 1000’s. The system by no means reaches the purpose the place the index pays off. As an alternative, the indexing step merely delays the second when the system can reply the primary query and provides operational complexity.

For this kind of short-lived, per-user context, the easy in-memory KNN strategy is just not solely simpler to implement and function, however additionally it is quicker end-to-end.

If in-memory storage is just not an possibility, both as a result of the system is distributed or as a result of we have to protect the person’s state for a couple of minutes, we are able to use a key-value retailer like Redis. We will retailer a novel identifier for the person’s request as the important thing and retailer all of the embeddings as the worth.

This offers us a light-weight, low-complexity answer that’s well-suited to our use case of short-lived, low-query contexts.

Actual-World Instance: Why We Selected a Key-Worth Retailer

At Planck, we reply insurance-related questions on companies. A typical request begins with a enterprise identify and tackle, after which we retrieve real-time knowledge about that particular enterprise, together with its on-line presence, registrations, and different public data. This knowledge turns into our context, and we use LLMs and algorithms to reply questions based mostly on it.

The necessary bit is that each time we get a request, we generate a recent context. We’re not reusing current knowledge, it’s fetched on demand and stays related for a couple of minutes at most.

In case you assume again to the sooner benchmark, this sample ought to already be triggering your “this isn’t a vector DB use case” sensor.

Each time we obtain a request, we generate recent embeddings for short-lived knowledge that we’ll seemingly question just a few hundred occasions. Indexing these embeddings in a vector DB provides pointless latency. In distinction, with Redis, we are able to instantly retailer the embeddings and run a fast similarity search within the software code with nearly no indexing delay.

That’s why we selected Redis as a substitute of a vector database. Whereas vector DBs are wonderful at dealing with massive volumes of embeddings and supporting quick nearest-neighbor queries, they introduce indexing overhead, and in our case, that overhead is just not price it.

In Conclusion

If you could retailer thousands and thousands of embeddings and assist high-query workloads throughout a shared corpus, a vector DB could be a greater match. And sure, there are positively use instances on the market that really want and profit from a vector DB.

However simply since you’re utilizing embeddings or constructing a RAG system doesn’t imply you need to default to a vector DB.

Every database expertise has its strengths and trade-offs. The only option begins with a deep understanding of your knowledge and use case, slightly than mindlessly following the development.

So, the following time you could select a database, pause for a second and ask: am I selecting the best one based mostly on goal trade-offs, or am I simply going with the trendiest, shiniest alternative?