HNSW at Scale: Why Your RAG System Will get Worse because the Vector Database Grows

a contemporary vector database—Neo4j, Milvus, Weaviate, Qdrant, Pinecone—there’s a very excessive probability that Hierarchical Navigable Small World (HNSW) is already powering your retrieval layer. It’s fairly seemingly you didn’t select it whereas constructing the database, nor did you tune it and even know it’s there. And but, HNSW is quietly deciding what your LLM sees as reality. It determines which doc chunks are fed into your RAG pipeline, which recollections your agent remembers, and finally, whether or not the mannequin solutions accurately—or hallucinates confidently.

As your vector database grows, retrieval high quality degrades progressively:

• No exceptions are raised
• No errors are logged
• Latency typically seems completely effective

However the context high quality deteriorates, and your RAG system turns into much less dependable over time—although the embedding mannequin and distance metric stay unchanged.

On this article, I exhibit—utilizing managed experiments and actual information—how HNSW impacts retrieval high quality as database measurement grows, why this degradation is worse than flat search, and what you possibly can realistically do about it in manufacturing RAG techniques.

Particularly, I’ll:

• Construct a sensible, reproducible use case to measure the impact of HNSW on RAG retrieval high quality utilizing Recall@ok.
• Present that, for fastened HNSW settings, recall degrades sooner than flat search because the corpus grows.
• Talk about sensible tuning methods for balancing recall and latency past merely growing ef_search of HNSW.

What’s HNSW?

HNSW is a graph-based algorithm for Approximate Nearest Neighbor (ANN) search. It organizes information into a number of layers of related neighbors and makes use of this graph construction to hurry up search.

Every vector is related to a restricted variety of neighbors in every layer. Throughout a search, it performs a grasping search via these layers, and the variety of neighbors checked at every layer is fixed (managed by M and ef_search), which makes the search course of logarithmic with respect to the variety of vectors. In comparison with flat search, the place time complexity is O(N), HNSW search has a time complexity of O(log N), which suggests the time required for a search grows very slowly (logarithmically) as in comparison with linear search. We’ll see this in the results of our use case.

Parameters of HNSW index

1. Construct time parameters: M and ef_construction. Might be set earlier than constructing the database solely.

M defines the utmost variety of connections (neighbors) that every vector (node) can have in every layer of the graph. The next M means extra connections, making the graph denser and doubtlessly growing recall however at the price of extra reminiscence and slower indexing.

Ef_construction controls the measurement of the candidate set used in the course of the development of the graph. Basically, it governs how completely the graph is constructed throughout indexing. The next worth for ef_construction means the graph is constructed extra completely, with extra candidates being thought of earlier than making every connection, which results in a greater high quality graph and higher recall at the price of elevated reminiscence and slower indexing.

For a basic function RAG software, typical values of M are inside a variety of 12 and 48 and ef_construction between 64 and 200.

2. Question time parameter: ef_search

This defines the variety of candidate nodes (or vectors) to discover in the course of the question course of (i.e., in the course of the seek for nearest neighbors). It controls how thorough the search course of is by figuring out what number of candidates are evaluated earlier than the search result’s returned. The next worth for ef_search means the search will discover extra candidates, main to raised recall however doubtlessly slower queries.

What’s Recall@ok?

Recall@ok is a key metric for measuring the accuracy of vector search and RAG techniques. It measures the power of the retriever to search out the related chunks for a person question throughout the high ok outcomes. It’s vital as a result of If the retriever misses the chunks containing the data required to reply the query (low recall), the LLM can not presumably generate an correct reply within the response synthesis step, no matter how highly effective it’s.

[ text{Recall}@k = frac{text{relevant items retrieved in top } k}{text{total number of relevant items in the corpus}} ]

In follow, this can be a troublesome metric to measure as a result of the denominator (floor reality paperwork) is just not simply identified for a real-life manufacturing system. What we’ll do as an alternative, is design a use case the place the bottom reality (eg; vector index) is exclusive and identified, and Recall@ok will measure the typical variety of occasions it’s retrieved in top-k outcomes, over numerous pattern queries.

As an example, Recall@5 will measure the typical variety of occasions the bottom reality index appeared in top-5 retrievals over 500 queries.

For a RAG, the suitable vary of Recall@5 is 70-90% and Recall@10 is 80-95%, and we’ll see that our use case adheres to those ranges for the Flat index.

Use Case

To check HNSW, we want a vector database with sufficiently massive variety of vectors (> 100,000). There doesn’t appear to be such a big public dataset obtainable consisting of doc chunks and related question(ies) for which the actual chunk can be thought of as floor reality. And even when it had been there, pure language may be ambiguous, so it’s troublesome to confidently say which all chunks within the corpus might be thought of as related for a question (the denominator in Recall@ok components). Growing such a curated dataset would require discovering numerous paperwork, chunking and embedding them, then growing queries for the chunks. That may be a useful resource intensive course of.

As an alternative, lets re-imagine our RAG drawback as “given a brief caption (question), we wish to retrieve essentially the most related photographs from the dataset”.

For this method, I utilized the publicly obtainable LAION-Aesthetics dataset. To entry, you will have to be logged in to Hugging Face, and comply with the phrases talked about. Particulars concerning the dataset is offered on the LAOIN web site right here. It accommodates an enormous variety of rows containing URLs to pictures together with a textual content caption. They seem like the next:

I downloaded a subset of rows and generated 200,000 CLIP embeddings of the pictures to construct the vector database. The textual content captions of the pictures may be conveniently used as queries for RAG. And every caption has just one picture vector as the bottom reality so the denominator of Recall@ok is strictly identified for all queries. Additionally, the CLIP embeddings of the picture and its caption are by no means a precise match, so there’s sufficient “fuzziness” in retrievals just like a purely doc RAG, the place a textual content question is used to retrieve related doc chunks utilizing a distance metric. This will likely be evident after we see the chart of Recall@ok within the subsequent sections.

Measuring Recall@ok for Flat vs HNSW

We undertake the next method:

1. Embeddings of 200k photographs are saved as .npy file.
2. From the laion dataset, 500 captions(queries) are randomly chosen and embedded utilizing CLIP. The chosen question indices additionally type the bottom reality as they correspond to the distinctive picture for the question.
3. The database is in-built increments of fifty,000 vectors, so 4 iterations of measurement 50k, 100k, 150k and 200k vectors. Each flat and HNSW indexes are constructed. HNSW is constructed utilizing M=16 and ef_construction=100.
4. Recall@ok is calculated for ok = 1, 5, 10, 15 and 20 primarily based upon if the bottom reality indices are included in top-k outcomes.
5. First, the Recall@ok values are calculated for every of the question vectors and averaged over the variety of samples (500).
6. Then, common Recall@ok values are calculated for HNSW ef_search values of 10, 20, 40, 80 and 160.
7. Lastly, 5 charts are drawn, one for every of the Recall@ok values. Every chart depicts the evolution of Recall@ok as database measurement grows for Flat index and completely different ef_search values of HNSW.

import pandas as pd
import numpy as np
import faiss
import torch
import open_clip
import os
import random
import matplotlib.pyplot as plt

def evaluate_subset(measurement, embeddings_all, df_all, query_vectors_all, eval_indices_all, ef_search_values):
    # Subset embeddings
    embeddings = embeddings_all[:size]
    dimension = embeddings.form[1]
    
    # Construct Indices in-memory for this subset measurement
    index_flat = faiss.IndexFlatL2(dimension)
    index_flat.add(embeddings)
    
    index_hnsw = faiss.IndexHNSWFlat(dimension, 16)
    index_hnsw.hnsw.efConstruction = 100
    index_hnsw.add(embeddings)

    num_samples = len(eval_indices_all)
    outcomes = []

    ks = [1, 5, 10, 15, 20]

    # Consider Flat
    flat_recalls = {ok: 0 for ok in ks}
    for i, qv in enumerate(query_vectors_all):
        _, I = index_flat.search(qv, max(ks))
        goal = eval_indices_all[i]
        for ok in ks:
            if goal in I[0][:k]:
                flat_recalls[k] += 1
    
    flat_res = {"Setting": "Flat"}
    for ok in ks:
        flat_res[f"R@{k}"] = flat_recalls[k]/num_samples
    outcomes.append(flat_res)

    # Consider HNSW with completely different efSearch
    for ef in ef_search_values:
        index_hnsw.hnsw.efSearch = ef
        hnsw_recalls = {ok: 0 for ok in ks}
        for i, qv in enumerate(query_vectors_all):
            _, I = index_hnsw.search(qv, max(ks))
            goal = eval_indices_all[i]
            for ok in ks:
                if goal in I[0][:k]:
                    hnsw_recalls[k] += 1
        
        hnsw_res = {"Setting": f"HNSW (ef={ef})", "ef": ef}
        for ok in ks:
            hnsw_res[f"R@{k}"] = hnsw_recalls[k]/num_samples
        outcomes.append(hnsw_res)
    
    return outcomes

def format_table(measurement, outcomes):
    ks = [1, 5, 10, 15, 20]
    strains = []
    strains.append(f"nDatabase Dimension: {measurement}")
    strains.append("="*80)
    header = f"{'Index/efSearch':<20}"
    for ok in ks:
        header += f" | {'R@'+str(ok):<8}"
    strains.append(header)
    strains.append("-" * 80)
    for row in outcomes:
        line = f"{row['Setting']:<20}"
        for ok in ks:
            line += f" | {row[f'R@{k}']:<8.2f}"
        strains.append(line)
    strains.append("="*80)
    return "n".be part of(strains)

def essential(n):
    dataset_path = r"C:databaselaion_final.parquet"
    embeddings_path = r"C:databaseembeddings.npy"
    results_dir = r"C:outcomes"
    
    db_sizes = [50000, 100000, 150000, 200000]
    ef_search_values = [10, 20, 40, 80, 160]
    num_samples = n
    output_txt = os.path.be part of(results_dir, f"eval_results_{num_samples}.txt")
    output_png = os.path.be part of(results_dir, f"recall_vs_dbsize_{num_samples}.png")

    if not os.path.exists(dataset_path) or not os.path.exists(embeddings_path):
        print("Error: Dataset or embeddings not discovered.")
        return
    
    os.makedirs(results_dir, exist_ok=True)

    # Load All Information As soon as
    print("Loading base information...")
    df_all = pd.read_parquet(dataset_path)
    embeddings_all = np.load(embeddings_path).astype('float32')

    # Load CLIP mannequin as soon as
    print("Loading CLIP mannequin (ViT-B-32)...")
    mannequin, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
    tokenizer = open_clip.get_tokenizer('ViT-B-32')
    machine = "cuda" if torch.cuda.is_available() else "cpu"
    mannequin.to(machine)
    mannequin.eval()

    # Use samples legitimate for all subsets
    eval_indices = random.pattern(vary(min(db_sizes)), num_samples)
    print(f"Sampling {num_samples} queries for constant analysis...")

    # Generate question vectors
    query_vectors = []
    for idx in eval_indices:
        textual content = df_all.iloc[idx]['TEXT']
        text_tokens = tokenizer([text]).to(machine)
        with torch.no_grad():
            text_features = mannequin.encode_text(text_tokens)
            text_features /= text_features.norm(dim=-1, keepdim=True)
            query_vectors.append(text_features.cpu().numpy().astype('float32'))

    all_output_text = []
    # Accumulate all outcomes for plotting
    # construction: { 'R@1': { 'Flat': [val1, val2...], 'ef=10': [val1, val2...] }, ... }
    ks = [1, 5, 10, 15, 20]
    plot_data = {f"R@{ok}": { "Flat": [] } for ok in ks}
    for ef in ef_search_values:
        for ok in ks:
            plot_data[f"R@{k}"][f"HNSW ef={ef}"] = []

    for measurement in db_sizes:
        print(f"Evaluating with database measurement: {measurement}...")
        outcomes = evaluate_subset(measurement, embeddings_all, df_all, query_vectors, eval_indices, ef_search_values)
        table_str = format_table(measurement, outcomes)
        
        # Print to display
        print(table_str)
        all_output_text.append(table_str)

        # Accumulate for plot
        for row in outcomes:
            label = row["Setting"]
            if label == "Flat":
                for ok in ks:
                    plot_data[f"R@{k}"]["Flat"].append(row[f"R@{k}"])
            else:
                ef = row["ef"]
                for ok in ks:
                    plot_data[f"R@{k}"][f"HNSW ef={ef}"].append(row[f"R@{k}"])

    # Save textual content outcomes
    with open(output_txt, "w", encoding="utf-8") as f:
        f.write("n".be part of(all_output_text))
    print(f"nFinal outcomes saved to {output_txt}")

    # Create Particular person Plots for every Ok
    for ok in ks:
        plt.determine(figsize=(10, 6))
        k_key = f"R@{ok}"
        
        for label, values in plot_data[k_key].objects():
            linestyle = '--' if label == "Flat" else '-'
            marker = 'o' if label == "Flat" else 's'
            plt.plot(db_sizes, values, label=label, linestyle=linestyle, marker=marker)
        
        plt.title(f"Recall@{ok} vs Database Dimension")
        plt.xlabel("Database Dimension")
        plt.ylabel("Recall")
        plt.grid(True)
        plt.legend()
        
        output_png = os.path.be part of(results_dir, f"recall_vs_dbsize_{ok}.png")
        plt.tight_layout()
        plt.savefig(output_png)
        plt.shut()
        print(f"Plot saved to {output_png}")

if __name__ == "__main__":
    essential(500)

And the outcomes are the next:

Observations

1. For the Flat index (dotted line), Recall@5 and Recall@10 are within the vary of 0.70 – 0.85, as may be anticipated of actual life RAG purposes.
2. Flat index gives the very best Recall@ok throughout all database sizes and varieties a benchmark higher certain for HNSW.
3. At any given database measurement, Recall@ok will increase for a better ok. So for database measurement of 100k vectors, Recall@20 > Recall@15 > Recall@10 > Recall@5 > Recall@1. That is comprehensible as with a better ok, there’s extra chance that the bottom reality index is current within the retrieved set.
4. Each Flat and HNSW deteriorate constantly because the database measurement grows. It is because high-dimensional vector areas grow to be more and more crowded because the variety of vectors grows.
5. Efficiency improves for HNSW for greater ef_search values.
6. Because the database measurement approaches 200k, HNSW seems to degrade sooner than Flat search.

Does HNSW degrade sooner than Flat Search?

To view the relative efficiency of Flat vs HNSW indexes as database measurement grows, a barely completely different method is adopted:

1. The database indexes development and question choice course of stays similar as earlier than.
2. As an alternative of contemplating the bottom reality, we calculate the overlap between the Flat index and every of the HNSW ef_search outcomes for a given retrieval depend(ok).
3. 5 charts are drawn for every of the ok values, denoting the evolution of overlap as database measurement grows. For an ideal match with Flat index, the HNSW line will present a rating of 1. Extra importantly, if the degradation of HNSW outcomes is greater than Flat index, the line may have a unfavourable slope, else may have a horizontal or optimistic slope.

import pandas as pd
import numpy as np
import faiss
import torch
import open_clip
import os
import random
import matplotlib.pyplot as plt
import time

def evaluate_subset_compare(measurement, embeddings_all, df_all, query_vectors_all, ef_search_values):
    # Subset embeddings
    embeddings = embeddings_all[:size]
    dimension = embeddings.form[1]
    
    # Construct Indices in-memory for this subset measurement
    index_flat = faiss.IndexFlatL2(dimension)
    index_flat.add(embeddings)
    
    index_hnsw = faiss.IndexHNSWFlat(dimension, 16)
    index_hnsw.hnsw.efConstruction = 100
    index_hnsw.add(embeddings)

    num_samples = len(query_vectors_all)
    outcomes = []

    ks = [1, 5, 10, 15, 20]
    max_k = max(ks)

    # 1. Consider Flat as soon as for this subset
    flat_times = []
    flat_results_all = []
    for qv in query_vectors_all:
        start_t = time.perf_counter()
        _, I_flat_all = index_flat.search(qv, max_k)
        flat_times.append(time.perf_counter() - start_t)
        flat_results_all.append(I_flat_all[0])
    
    avg_flat_time_ms = (sum(flat_times) / num_samples) * 1000

    # 2. Consider HNSW relative to Flat
    for ef in ef_search_values:
        index_hnsw.hnsw.efSearch = ef
        
        hnsw_times = []
        # Monitor intersection counts for every ok
        overlap_counts = {ok: 0 for ok in ks}
        for i, qv in enumerate(query_vectors_all):
            # HNSW top-max_k
            start_t = time.perf_counter()
            _, I_hnsw_all = index_hnsw.search(qv, max_k)
            hnsw_times.append(time.perf_counter() - start_t)
            
            # Flat outcome was already pre-calculated
            I_flat_all = flat_results_all[i]
            
            for ok in ks:
                set_flat = set(I_flat_all[:k])
                set_hnsw = set(I_hnsw_all[0][:k])
                intersection = set_flat.intersection(set_hnsw)
                overlap_counts[k] += len(intersection) / ok
        
        avg_hnsw_time_ms = (sum(hnsw_times) / num_samples) * 1000
        
        hnsw_res = {
            "Setting": f"HNSW (ef={ef})", 
            "ef": ef,
            "FlatTime_ms": avg_flat_time_ms,
            "HNSWTime_ms": avg_hnsw_time_ms
        }
        for ok in ks:
            # Common over all queries
            hnsw_res[f"R@{k}"] = overlap_counts[k] / num_samples
        outcomes.append(hnsw_res)
    
    return outcomes

def format_all_tables(db_sizes, ef_search_values, all_results):
    ks = [1, 5, 10, 15, 20]
    strains = []
    
    # 1. Create one desk for every Recall@ok
    for ok in ks:
        k_label = f"R@{ok}"
        strains.append(f"nTable: {k_label} (HNSW Overlap with Flat)")
        strains.append("=" * (20 + 12 * len(db_sizes)))
        
        # Header
        header = f"{'ef_search':<18}"
        for measurement in db_sizes:
            header += f" | {measurement:<9}"
        strains.append(header)
        strains.append("-" * (20 + 12 * len(db_sizes)))
        
        # Rows (ef values)
        for ef in ef_search_values:
            row_str = f"{ef:<18}"
            for measurement in db_sizes:
                # Discover the outcome for this measurement and ef
                val = 0
                for r in all_results[size]:
                    if r.get('ef') == ef:
                        val = r.get(k_label, 0)
                        break
                row_str += f" | {val:<9.2f}"
            strains.append(row_str)
        strains.append("=" * (20 + 12 * len(db_sizes)))

    # 2. Create Search Time Desk
    strains.append("nTable: Common Search Time (ms)")
    strains.append("=" * (20 + 12 * len(db_sizes)))
    header = f"{'Index Setting':<18}"
    for measurement in db_sizes:
        header += f" | {measurement:<9}"
    strains.append(header)
    strains.append("-" * (20 + 12 * len(db_sizes)))
    
    # Flat Row
    row_flat = f"{'Flat Index':<18}"
    for measurement in db_sizes:
        # Flat time is similar for all ef in a measurement, so simply take any
        t = all_results[size][0]['FlatTime_ms']
        row_flat += f" | {t:<9.4f}"
    strains.append(row_flat)
    
    # HNSW Rows
    for ef in ef_search_values:
        row_str = f"HNSW (ef={ef:<3})"
        for measurement in db_sizes:
            t = 0
            for r in all_results[size]:
                if r.get('ef') == ef:
                    t = r.get('HNSWTime_ms', 0)
                    break
            row_str += f" | {t:<9.4f}"
        strains.append(row_str)
    strains.append("=" * (20 + 12 * len(db_sizes)))

    return "n".be part of(strains)

def essential(n):
    dataset_path = r"C:databaselaion_final.parquet"
    embeddings_path = r"C:databaseembeddings.npy"
    results_dir = r"C:outcomes"
    
    db_sizes = [50000, 100000, 150000, 200000]
    ef_search_values = [10, 20, 40, 80, 160]
    num_samples = n
    output_txt = os.path.be part of(results_dir, f"compare_results_{num_samples}.txt")
    output_png_prefix = "compare_vs_dbsize"

    if not os.path.exists(dataset_path) or not os.path.exists(embeddings_path):
        print("Error: Dataset or embeddings not discovered.")
        return
    
    os.makedirs(results_dir, exist_ok=True)

    # Load All Information As soon as
    print("Loading base information...")
    df_all = pd.read_parquet(dataset_path)
    embeddings_all = np.load(embeddings_path).astype('float32')

    # Load CLIP mannequin as soon as
    print("Loading CLIP mannequin (ViT-B-32)...")
    mannequin, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
    tokenizer = open_clip.get_tokenizer('ViT-B-32')
    machine = "cuda" if torch.cuda.is_available() else "cpu"
    mannequin.to(machine)
    mannequin.eval()

    # Use queries from the primary 50k rows
    eval_indices = random.pattern(vary(min(db_sizes)), num_samples)
    print(f"Sampling {num_samples} queries...")

    # Generate question vectors
    query_vectors = []
    for idx in eval_indices:
        textual content = df_all.iloc[idx]['TEXT']
        text_tokens = tokenizer([text]).to(machine)
        with torch.no_grad():
            text_features = mannequin.encode_text(text_tokens)
            text_features /= text_features.norm(dim=-1, keepdim=True)
            query_vectors.append(text_features.cpu().numpy().astype('float32'))

    all_results_data = {}
    ks = [1, 5, 10, 15, 20]
    plot_data = {f"R@{ok}": {} for ok in ks}
    for ef in ef_search_values:
        for ok in ks:
            plot_data[f"R@{k}"][f"ef={ef}"] = []

    for measurement in db_sizes:
        print(f"Evaluating with database measurement: {measurement}...")
        outcomes = evaluate_subset_compare(measurement, embeddings_all, df_all, query_vectors, ef_search_values)
        all_results_data[size] = outcomes

        # Accumulate for plot
        for row in outcomes:
            ef = row["ef"]
            for ok in ks:
                plot_data[f"R@{k}"][f"ef={ef}"].append(row[f"R@{k}"])

    # Format pivoted tables
    final_output_text = format_all_tables(db_sizes, ef_search_values, all_results_data)
    print(final_output_text)

    # Save textual content outcomes
    with open(output_txt, "w", encoding="utf-8") as f:
        f.write(final_output_text)
    print(f"nFinal outcomes saved to {output_txt}")

    # Create Particular person Plots for every Ok
    for ok in ks:
        plt.determine(figsize=(10, 6))
        k_key = f"R@{ok}"
        
        for label, values in plot_data[k_key].objects():
            plt.plot(db_sizes, values, label=label, marker='s')
        
        plt.title(f"HNSW vs Flat Overlap Recall@{ok} vs Database Dimension")
        plt.xlabel("Database Dimension")
        plt.ylabel("Overlap Ratio")
        plt.grid(True)
        plt.legend()
        
        output_png = os.path.be part of(results_dir, f"{output_png_prefix}_{ok}.png")
        plt.tight_layout()
        plt.savefig(output_png)
        plt.shut()
        print(f"Plot saved to {output_png}")

if __name__ == "__main__":
    essential(500)

And the outcomes are the next:

Observations

1. In all instances, the strains have a unfavourable slope, indicating that HNSW degrades sooner than the Flat index as database grows.
2. Larger ef_search values degrade slower than decrease values, which fall fairly sharply.
3. Larger ef_search values have vital overlap (>90%) with the benchmark flat search as in comparison with the decrease values.

Recall-latency trade-off

We all know that HNSW is quicker than Flat search. To see it in motion, I’ve additionally measured the typical latency within the code of the earlier part. Listed here are the typical search occasions (in ms):

Observations

1. HNSW is orders of magnitude sooner than flat search, which is the first motive for it to be the search algorithm of selection for nearly all vector databases.
2. Time taken by Flat search will increase virtually linearly with database measurement (O(N) complexity)
3. For a given ef_search worth (a row), HNSW time is sort of fixed. At this scale (200k vectors), HNSW latency stays almost fixed.
4. As ef_search will increase in a column, the HNSW time will increase very considerably. As an example, time taken for ef=160 is 3X that of ef=40

Tuning the RAG pipeline

The above evaluation exhibits that whereas HNSW is unquestionably the choice to undertake in a manufacturing state of affairs for latency causes, there’s a have to periodically tune the ef_search to keep up the latency-recall steadiness because the database grows. Some greatest practices that needs to be adopted are as follows:

1. Given the problem of measuring Recall@ok in a manufacturing database, preserve a check case repository of floor reality doc chunks and queries, which may be run at common intervals to verify retrieval high quality. We may begin with essentially the most frequent queries requested by the person, and chunks which might be wanted for a superb recall.
2. One other oblique solution to verify recall high quality can be to make use of a robust LLM to guage the standard of the retrieved context. As an alternative of asking “Did we get the perfect paperwork for the person question?”, which is troublesome to say exactly for a big database, we are able to ask a barely weaker query “Does the retrieved context truly include the reply to the person’s query?” and let the choose LLM reply to that.
3. Accumulate person suggestions in manufacturing. Consumer score of a response together with any guide correction can be utilized as a set off for efficiency tuning.
4. Whereas tuning ef_search, begin with a conservatively excessive worth, measure Recall@ok, then scale back till latency is appropriate.
5. Measure Recall on the top_k that the RAG makes use of, normally between 3 and 10. Take into account stress-free top_k to fifteen or 20 and let the LLM determine which chunks within the given context to make use of for the response throughout synthesis step. Assuming the context doesn’t grow to be too massive to slot in the LLM’s context window, such an method would allow a excessive recall whereas having a reasonable ef_search worth, thereby retaining latency low.

Hybrid RAG pipeline

HNSW tuning utilizing ef_search can not repair the problem of falling recall with growing database measurement past a degree. That’s as a result of vector search even utilizing a flat index, turns into noisy when too many vectors are packed shut collectively within the N dimensional house (N being the variety of dimensions output by the embedding mannequin). Because the charts within the above part present, recall drops by 10%+ as database grows from 50k to 200k. The dependable solution to preserve recall is to make use of metadata filtering (eg; utilizing a data graph), to establish potential doc ids and run retrieval just for these. I talk about this intimately in my article GraphRAG in Apply: Learn how to Construct Value-Environment friendly, Excessive-Recall Retrieval Techniques

Key Takeaways

• HNSW is the default retrieval algorithm in most vector databases, however it’s not often tuned or monitored in manufacturing RAG techniques.
• Retrieval high quality degrades silently because the vector database grows, even when latency stays steady.
• For a similar corpus measurement, Flat search constantly achieves greater Recall@ok than HNSW, serving as a helpful higher certain for analysis.
• HNSW recall degrades sooner than Flat seek for fastened ef_search values as database measurement will increase.
• Growing ef_search improves recall, however latency grows quickly, creating a pointy recall–latency trade-off.
• Merely tuning HNSW parameters is inadequate at scale—vector search itself turns into noisy in dense embedding areas.
• Hybrid RAG pipelines utilizing metadata filters (SQL, graphs, inverted indexes) are essentially the most dependable solution to preserve recall at scale.

Conclusion

HNSW has earned its place because the spine of recent vector databases—not as a result of it’s completely correct, however as a result of it’s quick sufficient to make large-scale semantic search sensible.

Nonetheless, in RAG techniques, pace with out recall is a false optimization.

This text exhibits that as vector databases develop, retrieval high quality deteriorates quietly—particularly beneath approximate search—whereas latency metrics stay deceptively steady. The result’s a system that seems wholesome from an infrastructure perspective, however progressively feeds weaker context to the LLM, growing hallucinations and lowering reply high quality.

The answer is to not abandon HNSW, nor to arbitrarily improve ef_search.

As an alternative, production-grade RAG techniques should:

• Measure retrieval high quality explicitly and repeatedly.
• Deal with Flat search as a recall baseline.
• Constantly rebalance recall and latency.
• And finally, transfer towards hybrid retrieval architectures that slender the search house earlier than vector similarity is utilized.

In case your RAG system’s solutions are getting worse as your information grows, the issue is probably not your LLM, your prompts, or your embeddings—however the retrieval algorithm you by no means realized you had been counting on.

Join with me and share your feedback at www.linkedin.com/in/partha-sarkar-lets-talk-AI

_{Photos used on this article are synthetically generated. LAOIN-Aesthetics dataset used beneath CC-BY 4.0 license. Figures and code created by me}