Benchmarks
Dated measurements of Vectorless, each linked to the report that produced it, with what the numbers do not show.
Every figure here comes from a dated report in the engine repository's
docs/evaluations
folder. Each report names its harness, corpus and commands. Later reports
correct earlier ones, and say so.
Unless stated, the corpus is FinanceBench: 10-K filings from
PatronusAI/financebench, with the gold evidence pages that dataset provides,
40 questions over 19 filings.
What these numbers do not show
Read these before the tables.
- Retrieval is not answer accuracy. "Every gold page found" and hit@5 say the evidence was among the returned pages. They say nothing about whether an answer written from those pages was right. Do not set them against answer-accuracy figures such as PageIndex's published 98.7%.
- 1.9 s and 4.1 s come from benchmark tools, not the API. They call the navigator and ingest directly, without the HTTP layer, database reads or the answer. The production API answered in about 3.5–6 s.
- Jev is not deterministic. Repeated runs returned exactly the same set 0.375 of the time (Jaccard 0.834; see Judgement, not generation, 2026-09-25). Replay returns a stored answer; it does not make the engine deterministic.
- Multi-hop questions are unmeasured. FinanceBench is mostly single-evidence.
- The answer pilot is too small to compare. 27 of 30 answers were correct on 5 filings, graded with GLM-4.6 using the grading prompt from PageIndex's Mafin evaluation. It is not published as a report.
Ingest
| Claim | Result | Date | Report |
|---|---|---|---|
| Contents tree on Jev instead of a generative model | 12–143 s (mean 44 s) and $0.0003–0.0015 per filing, against 103–840 s and $0.017–0.095, with zero generative calls; title recall 0.961 against the generative trees | 2026-09-18 | TOC extraction on Jev |
| Placing entries on their pages (11 filings) | Gold pages inside a leaf 16/20 became 20/20; median leaf holding one 183 pages became 36; cost $0.246 became $0.0046 | 2026-09-18 | Page resolution |
| Whole corpus after the per-page text fix | 47/47 gold pages inside a leaf; 21/21 filings have a tree | 2026-09-18 | Retrieval navigation |
| Ingest time per filing (tool, not API) | Median 66.1 s became 4.1 s after the llmgate fix, still 47/47; $0.133 for 21 filings, about $0.0063 per filing | 2026-09-28 | Latency was our CPU |
Navigation
| Claim | Result | Date | Report |
|---|---|---|---|
| First Jev navigation run | 37/40 right section, 34/40 every gold page, $0.0029 per question | 2026-09-18 | Retrieval navigation |
| After leaf splitting | 40/40 right section, 36/40 every gold page, $0.0039 per question | 2026-09-19 | Leaf granularity |
| Query time (navigator called directly) | Median 50.5 s became 1.9 s after the llmgate fix, still 36/40; $0.00402 became $0.00388 per question | 2026-09-28 | Latency was our CPU |
The 50.5 s was not the model. llmgate's guard rebuilt its tokenizer on every token count (256 ms each); Jev answered in about 0.6 s throughout.
Against chunk-and-embed
FinanceBench, 40 questions, k=5, final run 2026-09-21.
| Vectorless | BGE-small embeddings | BM25 | |
|---|---|---|---|
| hit@5 | 0.925 | 0.375 | 0.200 |
| F1@5 | 0.498 | 0.170 | 0.096 |
| Answer text in the top-1 unit | 0.750 | 0.225 | 0.075 |
Report: Head-to-head against chunk-and-embed. The embedding and BM25 baselines answer in about 49 ms and cost nothing per query; the report gives the latency and cost columns in full.
Negative result: an embedding pre-filter does not help. At k=50, BGE-small puts every gold page in its results for only 0.875 of questions, below judgewalk's 0.925 at k=5, so it cannot be used to shrink what Jev reads (report, 2026-09-25; its latency figures are superseded by the 2026-09-28 report).
Against PageIndex
3M 2018 10-K (160 pages), 6 questions × 3 repeats, one laptop, run on 2026-10-08 against PageIndex 0.2.22 in local mode. Both systems used glm-4.5-air on the same provider for generation.
| Vectorless | PageIndex | |
|---|---|---|
| Retrieval, median | 1.70 s | 7.05 s |
| Full answer, median | 5.27 s | 10.51 s |
| Ingest, ready to query | 10.5–19.2 s | 204–210 s |
| Retrieved pages hold the answer | 18/18 | 17/18 |
Report: PageIndex vs Vectorless latency.
The retrieval models differ: Vectorless navigates with Jev, PageIndex with a
tool-calling chat model. When Vectorless navigates with the same generative
model (treewalk on glm-4.5-air), the full-answer gap narrows to 7.6 s against
10.5 s at the median, and PageIndex wins 2 of the 6 questions. Part of
PageIndex's ingest time on this machine came from a Type3-font fallback that
re-parsed every page on one core.
pdfgrab
The PDF parser has its own reports in the pdfgrab repository.
- Agreement with pdfplumber. 0.0000 pt position drift on both axes, and 103/103 minus signs kept on 3M 2018 10-K pages 56–60 (was 83/103). That is agreement with pdfplumber, not ground truth.
- Tables, ICDAR 2013, per-document F1. 0.442 against pdfplumber's 0.458, at 81 ms per document against 794 (2026-09-17). Given a correct grid, 0.935. An earlier figure of 0.362 was a pooled F1, not the competition's metric.