# CALDEX: Unified Single-Index Retrieval, State of the Art on Three of the World’s Hardest Benchmarks

The full study behind the results and methodology.

## Abstract

Reliable retrieval remains difficult for applications built on large language models, where a capable model given the wrong context hallucinates. This paper presents CALDEX, a unified single-index vector storage and retrieval system that finds the exact words and the semantic meaning of a query **in one pass**, rather than running two searches and merging their results. It keeps **no full-precision copy** of the vectors, and its whole index is **smaller than the raw vectors** it holds. CALDEX is evaluated on three of the world’s hardest benchmarks, Google DeepMind’s LIMIT [1], the Cohere 1M case of VectorDBBench [2] and the GloVe-100-angular dataset of ann-benchmarks [3], on 4 cores and 16 GB of memory. Results show that it is **state of the art on each**, and it offers a practical pathway toward reliable retrieval on the hardware a team already has.

**Index Terms**—vector search, approximate nearest neighbour search, hybrid retrieval, semantic search, BM25, recall, index size, queries per second, benchmarks

## I. Introduction

Vector search has become the retrieval layer of most applications built on large language models. For every question, retrieval finds the passages that relate to it and passes them to the model as context. When that context is wrong, a capable model hallucinates or gives an incorrect answer. As a result, the bottleneck on the way to reliable AI is often **not the model but the retrieval behind it**.

Today’s retrieval systems are **split in two**: one pipeline searches for the exact words of a query and another for its meaning, and the answers are merged before they reach the model, which adds another layer of chance to finding the right one. Besides accuracy, retrieval has a memory cost: many vector databases compress their vectors and keep the full-precision vectors beside them to rescore the results [4]–[7], so **the same data is stored twice**. This results in a significant gap: the right context is split across two searches, and the data behind it is paid for twice.

To fill this gap, this study presents CALDEX, **a single-index vector storage and retrieval system**. Its one-pass search finds both the exact word and the semantic meaning of every token, it keeps no full-precision copy, and its whole index is smaller than the raw vectors. The goal is to **return the right context in one search, in less memory than the data itself**, on the hardware a team already has.

### A. Previous Work

There are two main types of search in retrieval systems: keyword search, such as BM25, which matches the exact words of a query, and semantic search, which matches its meaning through embeddings. Keyword search works well when an answer shares the query’s words, but **misses answers phrased differently**. Semantic search finds those, but Google DeepMind’s researchers proved that **an embedding of a fixed size cannot return every combination of right answers** once a corpus is large enough, and released LIMIT to show it [1], [8].

Simultaneously, vector search has converged on graph indexes such as HNSW, built in hnswlib [9] and FAISS [10]. They work well on most data, but **their 8-bit forms save memory by giving up accuracy**, as USearch also does [11], and on data whose neighbourhoods are hard to find they return fewer of the true neighbours for the same effort [12]. There exists a significant gap in the formulation of a system that **finds exact words and meaning in one index, keeps no second copy of its vectors, and holds its accuracy on hard data**.

### B. Contributions

Unlike conventional approaches that run keyword and semantic search separately and merge their results as a sequential step, CALDEX **finds both in a single pass**, and each result carries **the file, the exact place in it and the passage itself**. It was evaluated on **4 cores and 16 GB of memory**, against open-source graph indexes on the same machine and the published results of the vector databases teams deploy.

### C. Why These Three Benchmarks

Three public benchmarks are the focus of this study, each chosen to test one place where retrieval fails: (1) **LIMIT**, Google DeepMind’s test of the answers that search by meaning cannot find, on which the best embedding models find fewer than one in five of the right answers even in their first hundred results [1], [8]; (2) **Cohere 1M**, VectorDBBench’s test of real text embeddings against the databases teams deploy [2], [13]; and (3) **GloVe 1.2M**, ann-benchmarks’ dataset on which a graph index most easily stops at the wrong neighbours, with the least room to be smaller than its vectors [3], [12]. Together they test **the answers a system can find at all, the cost of finding them at scale, and the rigour of the search underneath**.

## II. Benchmarks and Measures

Each benchmark was run on its own released data, queries and answer key, and scored the way it scores. Accuracy is measured throughout as **the share of right answers returned in the first k results**:

*Recall@k* = |*R*ₖ ∩ *G*| / |*G*| (1)

where

- *R*ₖ denotes the first k results returned for a question;
- *G* denotes the right answers to that question.

### 01 · A. LIMIT

Search by meaning turns every document into one embedding and returns the documents whose embeddings sit closest to the query’s. LIMIT was built to show the ceiling of this method, and it is deliberately simple, so when the best embedding models fail on it, the fault lies in the method, not the data [1]. It was chosen because it is **the clearest test of whether a system can get more from the same embeddings**. It contains **50,000 short profiles** of what people like and **1,000 questions** such as “Who likes Birch Trees?”, each with exactly **two right answers** [8]. CALDEX was run on the released data and compared with every system in the paper’s full-set results, each at its best, by (1) at **k = 2**, the paper’s strictest measure.

### 02 · B. Cohere 1M

Applications built on a language model typically turn each passage of their documents into an embedding with a commercial model, and search those embeddings for every question. Cohere 1M reproduces this workload, and it was chosen for three reasons. First, it is **real**: a million paragraphs of English Wikipedia, embedded by Cohere’s model in **768 dimensions** [13]. Second, it is **where memory costs**: with 768 numbers in every embedding, an index that keeps a second, full-precision copy pays for the data twice. Third, it is **the vector databases’ own test** in VectorDBBench [2], so CALDEX is measured on their ground, against their best published figures.

CALDEX was run on VectorDBBench’s own files, queries and answer key, and scored by (1) at **k = 100**. **Four measures** were taken: accuracy; the memory and disk an index needs together, as a multiple of the raw vectors; the time from raw vectors to a searchable index; and searches answered a second. Each database is shown **at its best published point**, from VectorDBBench or its vendor [2], [14]: the best accuracy of its latest version, its fastest speed, every load it has published, and its size where it publishes one [15]–[17]. CALDEX’s time to a searchable index is its build from the raw vectors. Open-source graph indexes were run on CALDEX’s machine and measured the same way, **16 searches at a time at the same accuracy**, CALDEX through its own server and each library called directly in code.

### 03 · C. GloVe 1.2M

To search quickly, an index keeps a graph, a map of which vectors lie near which, and walks it toward the answer. On GloVe this walk is hard: many of its queries have true neighbours barely closer than a crowd of others, more than in most datasets [12], so **a graph that takes shortcuts stops at the wrong place**. Its vectors are also short, so a graph’s links are a large share of the index, which makes **being smaller than the raw vectors hardest here**. It was chosen because it tests the rigour of a graph where graphs are most easily lost, against the whole field that ann-benchmarks runs on it.

GloVe-100-angular contains **1,183,514 word vectors** of **100 dimensions** [18], with an official answer key. Every index was built by Zerivek from ann-benchmarks’ data on the vectors padded to 128 dimensions, all on the same machine except three from an earlier run; sizes include the padding and are set against the raw vectors at 100 dimensions, the stricter baseline. Each index is counted in memory and on disk together. Memory is counted **on one basis for every library: its index**, which it must hold in memory to search, the least it can use. CALDEX’s is **its whole server process**, as measured while it served searches, so the difference counts against CALDEX. Disk is the index, with the full-precision vectors beside a compressed one, as on Cohere 1M. Accuracy is (1) at **k = 10**, and CALDEX’s speed is also compared with ann-benchmarks’ published speeds, one search at a time [19].

### D. What a Search Returns

The documentation of **17 search products**, from vector databases and search engines to cloud platforms and a memory layer, was reviewed [20]–[36]. Each returns an identifier, a score and what the application stored beside the vector; **none returns the exact place in the source a result came from**. CALDEX returns the file it came from, the exact place in it (a page, a moment in a recording, a message) and the passage itself, in every format it searches.

## III. Results and Discussion

This section evaluates CALDEX on the three benchmarks. Every result is set out on the [results page](https://www.zerivek.com/docs/benchmarks), in the tables and figures numbered as cited here.

### A. LIMIT

CALDEX places **89.15%** of the right answers in the first two results, 1,783 of 2,000, as shown in [Table 1](https://www.zerivek.com/docs/benchmarks#table-limit). The best system in the paper, keyword search (BM25), reaches **85.7%**, and the best embedding model **3.2%**. This indicates that meaning alone cannot find these answers and exact words alone come close, while **one index that finds both surpasses every system in the paper**.

### B. Cohere 1M

- **Accuracy:** **98.34%** of the true top 100 and 98.51% of the true top 10 on 4 cores, ahead of every database in [Figure 1](https://www.zerivek.com/docs/benchmarks#figure-cohere-recall); the next, Elastic Cloud on 8 vCPUs and 60 GB, finds 97.48%.
- **Time to a searchable index:** CALDEX’s build, **123.6 s**, is **faster than every one of the 26 loads** drawn from zvec’s own chart [2], [14], each a database on its own configuration, as shown in [Figure 2](https://www.zerivek.com/docs/benchmarks#figure-cohere-ready): the nearest are zvec’s 147.9 s and Milvus’s 581.8 s, and the slowest, Elastic Cloud’s, takes 26,410 s.
- **Memory and disk:** CALDEX needs **0.62× the raw vectors** in memory and on disk together, 0.34× in memory and 0.29× on disk with no full-precision copy, as shown in [Figure 3](https://www.zerivek.com/docs/benchmarks#figure-cohere-kept). On the same machine, FAISS’s 8-bit index needs 1.68× once the full-precision copy the databases keep beside it [4]–[7] is counted on disk, and the uncompressed hnswlib 2.16×. By their own documentation, the databases need 2.04× for Elasticsearch [15], at least 2.21× for Weaviate [16] and at least 2.19× for pgvector [17], counting only the side each documents where it documents one.
- **Speed:** CALDEX builds fastest, in **123.6 s**, against 275 s for hnswlib and 596 s for HNSW-SQ8, and answers **3,480 searches a second** at 97% of the true top 10 or more, more than either library. Against the databases’ own published speeds, it answers **2,398 a second** at 97.14% of the true top 100 with 8 clients over local HTTP, faster than Qdrant Cloud’s 1,242 at 94.74%, Pinecone’s 1,147 at 92.62%, turbopuffer’s 798 at 89.93%, Weaviate Cloud’s 68 at 99.09% and pgvector’s 11 at 88.98% [2], as shown in [Figure 4](https://www.zerivek.com/docs/benchmarks#figure-cohere-speed); Pinecone’s latest is one p2.x8 pod with 20 clients over a cloud network.

These results indicate that CALDEX **finds more of the true neighbours** than every database shown, **in less memory and disk** than any whose documentation states its own, and is **searchable sooner and answers faster** than those it is compared with.

### C. GloVe 1.2M

In memory and on disk together, CALDEX needs **1.77× the raw vectors** at **96.05% Recall@10**, 0.98× in memory and 0.79× on disk with no full-precision copy, as shown in [Table 2](https://www.zerivek.com/docs/benchmarks#table-glove). **Every other index needs more**, and the uncompressed ones the most, as they hold every vector at 32 bits in memory and on disk: HNSW-Flat 3.92×, hnswlib 3.30× and USearch (32-bit) 3.30×. The compressed ones lose accuracy, so the systems that use them keep the full-precision vectors on disk beside them to rescore [4]–[7]: FAISS HNSW-SQ8 3.00×, HNSW-PQ 2.68× and USearch (8-bit) 2.38×. On disk alone, CALDEX’s whole index is **0.79× the raw vectors**, 21% smaller than the vectors themselves, and **no other graph index has been published smaller than its vectors** at 95% Recall@10 or more.

On the same machine, CALDEX builds in **141 s** and answers **7,942 searches a second** at 90.23% Recall@10, against 4,378 for FAISS HNSW-SQ8, 4,166 for hnswlib and 1,997 for FAISS IVF-PQ. For each GiB of memory in use, at every accuracy from 85% to 94%, it answers **2.2–3.2× more searches a second** than the best of these indexes, and at 95% it is **the only one left**. One search at a time, it answers **1,395 a second**, faster than Redis’s 1,264 at 91.97%, Vespa’s 628 at 91.21%, Weaviate’s 456 at 92.15%, Lucene’s 277 at 90.7%, Qdrant’s 197 at 93.21% and pgvector’s 69 at 91.5% at their fastest published points [19], as shown in [Table 3](https://www.zerivek.com/docs/benchmarks#table-glove-published-speed). These results indicate that **CALDEX’s graph holds its accuracy where graphs are most easily lost**, while staying smaller than the data it holds.

## IV. Conclusion

This paper presented CALDEX, a unified single-index vector storage and retrieval system that replaces the two searches of today’s retrieval with one, keeps no full-precision copy, and is smaller than the raw vectors it holds. Evaluated on 4 cores and 16 GB of memory, it **surpassed every system** in Google DeepMind’s LIMIT results with **89.15%** of the right answers in the first two; on Cohere 1M it found **98.34%** of the true top 100 in an index **3.5× smaller** than its raw vectors, and answered faster than Qdrant Cloud, Pinecone, turbopuffer, Weaviate Cloud and pgvector; and on GloVe 1.2M it is **the only published graph index smaller than its vectors** at 95% Recall@10 or more.

The three benchmarks test retrieval where it breaks: LIMIT where meaning alone runs out, Cohere 1M where memory and speed decide what an application pays to run, and GloVe 1.2M where a graph is most easily lost. The same system **leads each time**. The results suggest that **a single index offers a practical alternative to hybrid pipelines** that search twice and store their data twice, delivering the right context in one pass, in less memory than the data itself, on the hardware a team already has.

## References

[1] O. Weller, M. Boratko, I. Naim, and J. Lee, “On the theoretical limitations of embedding-based retrieval,” in *Proc. Int. Conf. Learning Representations (ICLR)*; full-set results in Table 5 of arXiv:2508.21038. [Online]. Available: https://arxiv.org/abs/2508.21038

[2] Zilliz, “VectorDBBench: Benchmark for vector databases,” GitHub. [Computer software]. [Online]. Available: https://github.com/zilliztech/VectorDBBench

[3] M. Aumüller, E. Bernhardsson, and A. Faithfull, “ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms,” *Information Systems*, vol. 87, Art. no. 101374. [Online]. Available: https://doi.org/10.1016/j.is.2019.02.006

[4] Elastic, “Dense vector field type,” Elasticsearch Reference. [Online]. Available: https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector

[5] pgvector, “pgvector: Open-source vector similarity search for Postgres,” GitHub. [Computer software]. [Online]. Available: https://github.com/pgvector/pgvector

[6] Qdrant, “Quantization,” Qdrant Documentation. [Online]. Available: https://qdrant.tech/documentation/guides/quantization/

[7] Weaviate, “Compression (vector quantization),” Weaviate Documentation. [Online]. Available: https://docs.weaviate.io/weaviate/concepts/vector-quantization

[8] O. Weller, “LIMIT,” Hugging Face. [Data set]. [Online]. Available: https://huggingface.co/datasets/orionweller/LIMIT

[9] nmslib, “hnswlib: Header-only C++/python library for fast approximate nearest neighbors,” ver. 0.8.0, GitHub. [Computer software]. [Online]. Available: https://github.com/nmslib/hnswlib

[10] Facebook Research, “Faiss: A library for efficient similarity search and clustering of dense vectors,” ver. 1.15.1, GitHub. [Computer software]. [Online]. Available: https://github.com/facebookresearch/faiss

[11] Unum, “USearch: Fast open-source search and clustering engine for vectors and arbitrary objects,” ver. 2.26.2, GitHub. [Computer software]. [Online]. Available: https://github.com/unum-cloud/usearch

[12] M. Aumüller and M. Ceccarello, “The role of local intrinsic dimensionality in benchmarking nearest neighbor search,” in *Proc. Int. Conf. Similarity Search and Applications (SISAP)*, pp. 113–127. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-030-32047-8_11

[13] Cohere, “wikipedia-22-12,” Hugging Face. [Data set]. [Online]. Available: https://huggingface.co/datasets/Cohere/wikipedia-22-12

[14] Zvec, “Benchmarks,” zvec.org. [Online]. Available: https://zvec.org/en/docs/db/benchmarks/

[15] Elastic, “Tune approximate kNN search,” Elastic Docs. [Online]. Available: https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/approximate-knn-search

[16] Weaviate, “Resource planning,” Weaviate Documentation. [Online]. Available: https://docs.weaviate.io/weaviate/concepts/resources

[17] Amazon Web Services, “Scale pgvector with binary quantization on Amazon Aurora PostgreSQL,” AWS Database Blog. [Online]. Available: https://aws.amazon.com/blogs/database/scale-pgvector-with-binary-quantization-on-amazon-aurora-postgresql/

[18] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in *Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP)*, pp. 1532–1543. [Online]. Available: https://doi.org/10.3115/v1/D14-1162

[19] ANN-Benchmarks, “glove-100-angular (k = 10),” ann-benchmarks.com. [Online]. Available: https://ann-benchmarks.com/glove-100-angular_10_angular.html

[20] Pinecone, “Semantic search,” Pinecone Docs. [Online]. Available: https://docs.pinecone.io/guides/search/semantic-search

[21] Weaviate, “Search patterns and basics,” Weaviate Documentation. [Online]. Available: https://docs.weaviate.io/weaviate/search/basics

[22] Milvus, “Basic vector search,” Milvus Documentation. [Online]. Available: https://milvus.io/docs/single-vector-search.md

[23] Qdrant, “Query points,” Qdrant API Reference. [Online]. Available: https://api.qdrant.tech/api-reference/search/query-points

[24] Chroma, “Query and get,” Chroma Docs. [Online]. Available: https://docs.trychroma.com/docs/querying-collections/query-and-get

[25] turbopuffer, “Query documents,” turbopuffer Docs. [Online]. Available: https://turbopuffer.com/docs/query

[26] Endee, “Hybrid search with BM25,” Endee Docs. [Online]. Available: https://docs.endee.io/v2/tutorials/hybrid-search

[27] TopK, “True hybrid search,” TopK Docs. [Online]. Available: https://docs.topk.io/concepts/true-hybrid-search

[28] Elastic, “Highlighting,” Elasticsearch Reference. [Online]. Available: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/highlighting

[29] Algolia, “Highlighting in InstantSearch.js,” Algolia Docs. [Online]. Available: https://www.algolia.com/doc/guides/building-search-ui/resources/ui-and-ux-patterns/in-depth/highlighting-snippeting/

[30] Typesense, “Typesense vector search and embeddings,” Typesense Docs. [Online]. Available: https://typesense.org/docs/30.2/api/vector-search.html

[31] Vectara, “Query a corpus with advanced options,” Vectara Docs. [Online]. Available: https://docs.vectara.com/docs/rest-api/query-corpus

[32] Google Cloud, “Query public index to get nearest neighbors,” Google Cloud Documentation. [Online]. Available: https://docs.cloud.google.com/vertex-ai/docs/vector-search/query-index-public-endpoint

[33] Microsoft, “Add semantic ranking: Azure AI Search,” Microsoft Learn. [Online]. Available: https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request

[34] MongoDB, “Run vector search ANN and ENN queries,” MongoDB Docs. [Online]. Available: https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/

[35] Redis, “FT.HYBRID,” Redis Docs. [Online]. Available: https://redis.io/docs/latest/commands/ft.hybrid/

[36] HydraDB, “hydradb-mcp: MCP server enabling HydraDB memory operations (store, recall, search) in AI coding tools,” GitHub. [Computer software]. [Online]. Available: https://github.com/hydra-db/hydradb-mcp

Source: https://www.zerivek.com/docs/benchmarks/report