Benchmarks
See how Zvec-Grep affects answer quality, context use, tool calls, and completion time in controlled agent evaluations.
Zvec-Grep is evaluated on complete agent tasks, not only isolated search latency. The benchmarks ask whether better retrieval helps an agent produce a strong answer with less context, fewer tool calls, and less time.
A controlled comparison
Every case compares the same task, agent, model, prompt, environment, and limits. The treatment adds only a prepared Zvec-Grep index, tools, and standard usage guidance. Index preparation is measured separately from agent execution time.
Results at a glance
| Workload | Quality | Input tokens | Tool calls | Agent time |
|---|---|---|---|---|
| Coding · SWE-QA-Bench | +1.50 pp | −47.3% | −58.6% | −37.5% |
| General text · BrowseComp-Plus | +0.33 pp | −37.56% | −43.52% | −38.58% |
The coding result covers 20 tasks across 11 repositories with three runs per profile. The general-text result covers 80 cases over 100,195 documents with two trials per profile.
What is measured
- Answer quality: task-specific judge score or accuracy.
- Input tokens: model context consumed while the agent works.
- Tool calls: search and inspection actions recorded in the trajectory.
- Agent wall time: task execution time, excluding Zvec-Grep index preparation.
Quality should stay stable or improve. Tokens, tool calls, and wall time should decrease.
Benchmark suites
SWE-QA-Bench
Measures repository-level, cross-file, and multi-hop software-engineering question answering. It is designed for questions where the relevant implementation is distributed across a codebase and the entry point is not known in advance.
Read the SWE-QA-Bench setup and methodology
BrowseComp-Plus
Measures multi-document evidence retrieval and answer accuracy over a large, fixed text corpus. It tests whether ranked semantic and lexical evidence reduces broad document scanning without lowering answer quality.
How to interpret the results
The results show the effect of adding Zvec-Grep to an agent workflow; they are not a raw CLI latency comparison. Agent behavior is stochastic, so repeated-run averages are more useful than a single trial.
The largest efficiency gains appear when a task requires semantic discovery, cross-file synthesis, or evidence spread across many documents. For a known literal or symbol, native ripgrep may still be the most direct route.
Reproduce the evaluation
Both suites pin their inputs and dependencies, keep generated artifacts outside the agent-visible workspace, and provide their own runner and reporting workflow.