Benchmarks

Zvec is engineered for speed, scale, and efficiency — and has been battle-tested across demanding production workloads within Alibaba Group.

Below, we present benchmark results that demonstrate how our system performs under various workloads and configurations.

All tests were conducted in controlled environments using standardized datasets and widely accepted methodologies to ensure fairness, transparency, and reproducibility.

Performance Evaluation

We evaluate Zvec using VectorDBBench, an open-source benchmarking framework widely adopted in the vector database community.

Our evaluation focus on two standard datasets:

Cohere 1M: 1 million 768-dimensional vectors
Cohere 10M: 10 million 768-dimensional vectors

For each dataset, we measure the following key performance indicators:

Queries Per Second (QPS): Throughput under sustained load.
Recall: Accuracy of nearest neighbor retrieval, reflecting search quality.
Index Build Time (load duration): Time required to ingest and index the full dataset, indicating ingestion efficiency.

Cohere 10M Benchmark Results

QPS Cohere 10M

Recall Cohere 10M

Load Duration Cohere 10M

Cohere 1M Benchmark Results

QPS Cohere 1M

Recall Cohere 1M

Load Duration Cohere 1M

Reproducing the Benchmarks

Follow these steps to reproduce our benchmark results in your own environment.

Prepare environment

Launch an ECS Instance

We recommend using Ubuntu 24.04 as the operating system. Other OS choices may require adjustments to the commands in this guide.
- Create a g9i.4xlarge instance (16 vCPU, 64 GiB RAM) following this guide

Install System Dependencies

Install git if not already installed
```
apt-get update
apt install git
```

Install Python3.11 or higher

apt-get update
apt install python3-full python3-venv python3-dev

cd /opt
python3 -m venv venv
source venv/bin/activate

Install VectorDBBench

You may need to use our fork until our PR is merged by upstream.

# Clone VectorDBBench
git clone https://github.com/egolearner/VectorDBBench.git
cd VectorDBBench

# Install deps
pip install -U pip
pip install -e .

# If you experience slow downloads or connection issues, you can try Aliyun PyPI mirror
# pip install -U pip -i https://mirrors.aliyun.com/pypi/simple
# pip install -e . -i https://mirrors.aliyun.com/pypi/simple

Install zvec
```
pip install -u zvec
```

Run Benchmarks

Cohere 10M

Build Index

vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 --case-type Performance768D10M --num-concurrency 12,14,16,18,20 --quantize-type int8 --ef-search 118 --is-using-refiner

Run Benchmark

vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 --case-type Performance768D10M --num-concurrency 12,14,16,18,20 --quantize-type int8 --ef-search 118 --is-using-refiner --skip-drop-old --skip-load

Cohere 1M

Build Index

vectordbbench zvec --path Performance768D1M --db-label 16c64g-v0.1 --case-type Performance768D1M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 30 --ef-search 180

Run Benchmark

vectordbbench zvec --path Performance768D1M --db-label 16c64g-v0.1 --case-type Performance768D1M --num-concurrency 12,14,16,18,20 --quantize-type int8 --m 30 --ef-search 180 --skip-drop-old --skip-load

Benchmarks

On this page