Evidence status: 17 November 2026. This article replaces the earlier English benchmark report and documents 30 benchmark iterations for SupraAgent's module and support-answering paths, long-context diagnostics, poisoned-source controls and local runtime instrumentation. The central result is not a new “world-best” score. It is a more inspectable benchmark: denominators are explicit, retrieval is separated from citation and lead placement, long inputs are split by evidence position, and experimental security controls are kept distinct from deployed protection.
SupraAgent is the assistant in SupraWorx. In the paths evaluated here, it searches product documentation, checks which evidence may be used and assembles a cited answer. The runs use deterministic local fixtures and make no provider calls. They show how these particular product components behaved on a frozen source tree. They do not establish correctness for real customer questions, production response times, language-model quality or superiority over another assistant.
What was evaluated
The cycle began from the shipped round-five benchmark and used thirty recorded work and decision iterations—not thirty statistically independent repetitions of one experiment. Accepted, rejected and inconclusive decisions remain in the machine-readable ledger. The evaluation contract required identical paired questions and retrieval budgets, separate quality and runtime measures, retention of failures in each denominator, and explicit labels when a holdout had already been opened.
Area | Evaluation unit | What the result can support |
|---|---|---|
Module functions and abstention | 192 development questions: 144 answerable and 48 deliberately unanswerable | Retrieval, citation, lead placement and “not found” behaviour on this fixed set |
Function-answer funnel | 96 answerable function questions within that development set | Where a known target is lost between retrieval, citation and the first answer line |
Pooled long-context position analysis | 256,000, 512,000 and 1,024,000 characters, pooled by evidence at the beginning, middle or end; 144 answerable retrieval cases and 96 gold-citation cases per position | Position sensitivity on the complete checked-in long-input fixture report; not a population estimate for customer documents |
256k instrumentation smoke | 114 local samples per measured path | Partial-read disclosure, scan reuse, latency percentiles and process-RSS snapshots on this machine |
Poisoned-source comparison | 96 matched synthetic trials per reported condition | Behaviour under the stated synthetic attacker and provenance labels; not incident probability |
Fresh-process load smoke | Two sequential jobs and two jobs under two-process concurrency | Instrumentation and a small throughput/latency trade-off; not a load test or production SLO |
Scope of this replacement
The previous article also reported separate contact-search and digital-twin fixtures, an earlier 612-document long-input suite and an 11,232-attempt planted-document campaign. Those results remain part of the earlier round-five evidence, but the final 182-test integration receipt for this thirty-iteration cycle covers the module-function, long-context, poisoning, provenance and process-load paths listed above. This replacement therefore does not silently carry the older contact or twin numbers forward as if they had been rerun on the final frozen tree.
Module answers: report the whole decision, not one success rate
The reproduced development baseline contains 192 questions. Of these, 144 have an answer in the documentation and 48 ask for a plausible function that is not documented. “Not found” is therefore evaluated as a binary decision across both groups. This prevents a system from appearing safer merely by refusing more answerable questions.
Measure | Combined retrieval | BM25 | Denominator |
|---|---|---|---|
Gold module in the first four retrieved results | 127/144 | 126/144 | 144 answerable development questions |
Correct “not found” decision | 23/48 | 20/48 | 48 unanswerable development questions |
False abstention | 5/144 | 9/144 | 144 answerable development questions |
“Not found” precision | 23/28 | 20/29 | All “not found” outputs from each method |
Specificity on answerable questions | 139/144 | 135/144 | 144 answerable development questions |
Balanced accuracy | 72.22% | 67.71% | Mean of sensitivity on the 48 unanswerable and specificity on the 144 answerable questions |
These figures do not select one universal winner. Combined retrieval has the higher balanced abstention score on this development set, while BM25 is stronger in the narrower target-function funnel below. The prior German route-lexicon candidate was rejected: its development gains did not survive the long-context, regression and tail-latency gates, so the default was not changed.
Where the requested function is lost
For 96 answerable function questions with a marked target, the benchmark now follows the same target through three stages. Combined retrieval admits 58/96 targets into the permitted evidence set, cites 55/58 admitted targets and puts 54/55 cited targets in the lead. BM25 admits 69/96, cites 66/69 and puts 63/66 in the lead.
The interpretation is narrower than “answer accuracy.” For combined retrieval, the largest observed loss is before citation: 38/96 targets never enter the admitted evidence set. Once a target is cited, only 1/55 misses the lead. This identifies a retrieval bottleneck on these fixtures; it does not independently judge whether the complete prose answer is correct or useful.
Long context: position is now a first-class result
Following Liu et al., Lost in the Middle, TACL 2024, the report no longer pools all evidence positions. It records beginning, middle and end separately, plus the worst position and the best-to-worst gap.
The stronger result comes from the complete checked-in long-input report pooled across 256k, 512k and 1.024M-character inputs. For combined retrieval, module hit@4 was 105/144 with evidence at the beginning, 103/144 in the middle and 98/144 at the end: a 4.9-percentage-point best-to-worst gap. Gold evidence was cited in 48/96 beginning cases, 46/96 middle cases and 48/96 end cases: a 2.1-point gap.
A separate, much smaller diagnostic smoke produced larger 50-point hit@4 and 25-point gold-citation gaps. It is retained only as a diagnostic that helped expose the need for positional reporting; the pooled counts above are the headline evidence. Neither result is an estimate for customer documents because the fixtures are deterministic and constructed.
The 256k A/A instrumentation smoke also makes partial reading visible. All 114/114 measured candidate cases were incomplete reads, and all 114 displayed the limit notice, so there were 0/114 missing notices among incomplete reads. There were no complete-read cases in this smoke; a false-positive notice rate is therefore not estimable. Prefix and tail scan reuse was 75%, with no uncached tail pass recorded. “Notice displayed” means the limitation was disclosed; it does not mean the unread portion was irrelevant.
Poisoned sources: a strong synthetic result that is not deployed protection
The threat model assumes that an attacker has managed to place manipulated text in the retrieval corpus. The benchmark measures several stages separately: whether poisoned evidence enters the top four, ranks first, is cited, contributes a canary value to the answer, or projects an instruction into the prompt. This design is informed directly by PoisonedRAG, USENIX Security 2025. Liu et al., USENIX Security 2024 motivates systematic attack/defence cells and explicit attacker goals.
The evaluated candidate is an opt-in ranking rule: evidence explicitly labelled verified ranks before evidence explicitly labelled unverified. In 96 matched synthetic trials, it produced the following comparison:
Measure | Legacy ranking | Tier rule on, labels present | Denominator |
|---|---|---|---|
Poisoned evidence in top four | 84/96 | 0/96 | 96 matched trials |
Poisoned evidence at rank one | 80/96 | 0/96 | 96 matched trials |
Poisoned evidence cited | 84/96 | 0/96 | 96 matched trials |
Canary adopted | 35/96 | 0/96 | 96 matched trials |
Instruction projected into prompt | 30/96 | 0/96 | 96 matched trials |
Clean module evidence in top four | 30/96 | 96/96 | 96 matched trials |
The negative control is decisive. When sources had no provenance label, switching the rule on reproduced the unsafe baseline exactly across another 96 trials. With the switch off, 32 labelled and 32 unlabelled ablation trials were byte- and semantically identical, with normalized output SHA-256 bbee5f77ff88969be207ae9b1f149460e7d98d4263bb820f5e127f5d251e469c.
This is not a production security claim. The product audit found no production indexer that originates the required tier, the switch remains off, and the poisoning set had already been opened during development. A fresh sealed poisoning holdout is still missing. The evidence therefore supports only this conditional statement: on the stated synthetic fixtures, with correct labels supplied by the benchmark, the ranking rule removed the measured contamination and projection events.
StruQ, USENIX Security 2025 is a design precedent for separating trusted instructions from untrusted data. SupraAgent's candidate is not an implementation of StruQ: it is a retrieval-ranking rule and does not include StruQ's structured-query front end or specially trained model.
Performance: distributions and trade-offs, not a speed headline
At 256k characters, the local instrumentation smoke records 114 samples per measured path and reports nearest-rank percentiles. For the candidate prompt path, p50/p95/p99 were 21.655/49.813/51.103 ms. For the candidate answer path, they were 55.241/88.084/94.040 ms. Process-RSS deltas were sampled outside the timed boundary: prompt p50/p95/p99 were 0/425,984/1,343,488 bytes; answer p50/p95/p99 were 65,536/770,048/1,146,880 bytes. These snapshots exclude transient peaks and are not comparable to absolute memory from another process or host.
The fresh-process runner was intentionally bounded. Two sequential jobs completed without failure, with wall-time p50 7.713 s and p95 10.884 s. With two jobs under two-process concurrency, aggregate throughput rose from 0.1075 to 0.1697 jobs/s, while median per-job wall time worsened from 7.713 to 11.778 s. All output digests matched. With only two jobs per condition, this is an instrumentation smoke and an observed throughput/latency trade-off—not evidence of a latency improvement, capacity limit or production service level.
What changed after thirty iterations
The benchmark now reports evidence position, worst-position quality and position gaps for long inputs.
Abstention now includes the complete confusion matrix, so correct refusals cannot hide false refusals.
The target-function path is split into retrieval, admission, citation and lead placement.
Incomplete long-document reads and their user-facing notices are measured separately.
Latency uses p50, p95 and p99 with sample counts; process memory is recorded outside the timing boundary.
Poisoning is measured at retrieval, citation, answer adoption and prompt projection, with switch-off and unlabeled-source negative controls.
Development data and previously opened holdouts are machine-labelled as ineligible for new blind promotion claims.
One product candidate was added behind an off-by-default flag: provenance-tier ranking. It is not ready for promotion because production tier assignment and a fresh sealed poisoning holdout are missing. The German retrieval candidate remained rejected. The more important delivered change is the evaluation contract and its guardrails.
Scientific basis
Peer-reviewed work guides the protocol; paper-reported gains are never counted as SupraAgent results.
Liu et al., Lost in the Middle, TACL 2024: motivates reporting by relevant-evidence position rather than only pooled long-context averages.
Bai et al., LongBench, ACL 2024: motivates diverse long-context tasks, lengths and languages.
Zhang et al., LongCite, Findings of ACL 2025: motivates evaluating response quality and fine-grained citation quality separately.
Buchmann et al., Attribute or Abstain, EMNLP 2024: motivates separate attribution and abstention measurements for long documents.
Liang et al., HELM, TMLR 2023: motivates transparent multi-metric evaluation by scenario.
Liu et al., Formalizing and Benchmarking Prompt Injection Attacks and Defenses, USENIX Security 2024: motivates systematic attack/defence evaluation.
Zou et al., PoisonedRAG, USENIX Security 2025: directly motivates injected-knowledge-base-text poisoning cells under explicit attacker capabilities.
Chen et al., StruQ, USENIX Security 2025: provides a design precedent for separating trusted instructions from untrusted data, without implying that the ranking candidate implements StruQ.
Smucker, Allan and Carterette, CIKM 2007: motivates paired significance and randomization methods for information-retrieval evaluation.
Reproducibility and evidence map
The final focused integration suite passed 182 tests and failed 0 in 276.04 seconds on frozen Git tree 389c55268e1846702e436b405642de1903984b8a. It covered long-context reporting, module-function evaluation, corpus poisoning, provenance tiers, fresh-process load, worker pinning and the thirty-iteration ledger. It made 0 provider calls. This focused result does not establish full application-suite health.
Claims in this article | Repository evidence |
|---|---|
Scope comparison with the previous English article, including the legacy contact, twin, 612-document and 11,232-attempt suites |
|
Development retrieval, abstention and target-function funnel |
|
Pooled long-context position and citation counts |
|
Small position diagnostic and partial-read disclosure |
|
Poisoning baseline, tier candidate, negative control and ablations |
|
Scan reuse, latency percentiles and RSS snapshots |
|
Fresh-process sequential and concurrent smoke |
|
Test count, duration, source hashes, provider calls and mandatory blockers |
|
Protocol, promotion gates and paper-to-method mapping |
|
What remains before stronger claims
The next evidence should come from a fresh sealed poisoning holdout, a production provenance scheme with independently audited labels, independent semantic and UX review, real customer-like questions and representative production load. Any stochastic language-model path would also require repeated seeded runs and uncertainty intervals. Until those exist, the defensible conclusion is deliberately narrow: the cycle improved what the benchmark can reveal, identified a retrieval bottleneck, quantified a modest pooled position gap while preserving the larger diagnostic as a warning, and found a promising but currently non-deployable provenance-ranking control.