ca. 9 Minuten

SupraAgent benchmark after 30 iterations: long context, source controls and performance

Veröffentlicht von · SupraTix GmbH
Veröffentlicht am · Aktualisiert am

SupraAgent benchmark after 30 iterations: long context, source controls and performance

What 30 benchmark iterations show about SupraAgent: explicit denominators, position-aware long-context results, complete abstention metrics, synthetic poisoning controls, and local latency and memory measurements—with clear limits instead of a single best score.

Evidence status: 17 November 2026. This article replaces the earlier English benchmark report and documents 30 benchmark iterations for SupraAgent's module and support-answering paths, long-context diagnostics, poisoned-source controls and local runtime instrumentation. The central result is not a new “world-best” score. It is a more inspectable benchmark: denominators are explicit, retrieval is separated from citation and lead placement, long inputs are split by evidence position, and experimental security controls are kept distinct from deployed protection.

SupraAgent is the assistant in SupraWorx. In the paths evaluated here, it searches product documentation, checks which evidence may be used and assembles a cited answer. The runs use deterministic local fixtures and make no provider calls. They show how these particular product components behaved on a frozen source tree. They do not establish correctness for real customer questions, production response times, language-model quality or superiority over another assistant.

What was evaluated

The cycle began from the shipped round-five benchmark and used thirty recorded work and decision iterations—not thirty statistically independent repetitions of one experiment. Accepted, rejected and inconclusive decisions remain in the machine-readable ledger. The evaluation contract required identical paired questions and retrieval budgets, separate quality and runtime measures, retention of failures in each denominator, and explicit labels when a holdout had already been opened.

Area

Evaluation unit

What the result can support

Module functions and abstention

192 development questions: 144 answerable and 48 deliberately unanswerable

Retrieval, citation, lead placement and “not found” behaviour on this fixed set

Function-answer funnel

96 answerable function questions within that development set

Where a known target is lost between retrieval, citation and the first answer line

Pooled long-context position analysis

256,000, 512,000 and 1,024,000 characters, pooled by evidence at the beginning, middle or end; 144 answerable retrieval cases and 96 gold-citation cases per position

Position sensitivity on the complete checked-in long-input fixture report; not a population estimate for customer documents

256k instrumentation smoke

114 local samples per measured path

Partial-read disclosure, scan reuse, latency percentiles and process-RSS snapshots on this machine

Poisoned-source comparison

96 matched synthetic trials per reported condition

Behaviour under the stated synthetic attacker and provenance labels; not incident probability

Fresh-process load smoke

Two sequential jobs and two jobs under two-process concurrency

Instrumentation and a small throughput/latency trade-off; not a load test or production SLO

Scope of this replacement

The previous article also reported separate contact-search and digital-twin fixtures, an earlier 612-document long-input suite and an 11,232-attempt planted-document campaign. Those results remain part of the earlier round-five evidence, but the final 182-test integration receipt for this thirty-iteration cycle covers the module-function, long-context, poisoning, provenance and process-load paths listed above. This replacement therefore does not silently carry the older contact or twin numbers forward as if they had been rerun on the final frozen tree.

Module answers: report the whole decision, not one success rate

The reproduced development baseline contains 192 questions. Of these, 144 have an answer in the documentation and 48 ask for a plausible function that is not documented. “Not found” is therefore evaluated as a binary decision across both groups. This prevents a system from appearing safer merely by refusing more answerable questions.

Measure

Combined retrieval

BM25

Denominator

Gold module in the first four retrieved results

127/144

126/144

144 answerable development questions

Correct “not found” decision

23/48

20/48

48 unanswerable development questions

False abstention

5/144

9/144

144 answerable development questions

“Not found” precision

23/28

20/29

All “not found” outputs from each method

Specificity on answerable questions

139/144

135/144

144 answerable development questions

Balanced accuracy

72.22%

67.71%

Mean of sensitivity on the 48 unanswerable and specificity on the 144 answerable questions

These figures do not select one universal winner. Combined retrieval has the higher balanced abstention score on this development set, while BM25 is stronger in the narrower target-function funnel below. The prior German route-lexicon candidate was rejected: its development gains did not survive the long-context, regression and tail-latency gates, so the default was not changed.

Where the requested function is lost

For 96 answerable function questions with a marked target, the benchmark now follows the same target through three stages. Combined retrieval admits 58/96 targets into the permitted evidence set, cites 55/58 admitted targets and puts 54/55 cited targets in the lead. BM25 admits 69/96, cites 66/69 and puts 63/66 in the lead.

The interpretation is narrower than “answer accuracy.” For combined retrieval, the largest observed loss is before citation: 38/96 targets never enter the admitted evidence set. Once a target is cited, only 1/55 misses the lead. This identifies a retrieval bottleneck on these fixtures; it does not independently judge whether the complete prose answer is correct or useful.

Long context: position is now a first-class result

Following Liu et al., Lost in the Middle, TACL 2024, the report no longer pools all evidence positions. It records beginning, middle and end separately, plus the worst position and the best-to-worst gap.

The stronger result comes from the complete checked-in long-input report pooled across 256k, 512k and 1.024M-character inputs. For combined retrieval, module hit@4 was 105/144 with evidence at the beginning, 103/144 in the middle and 98/144 at the end: a 4.9-percentage-point best-to-worst gap. Gold evidence was cited in 48/96 beginning cases, 46/96 middle cases and 48/96 end cases: a 2.1-point gap.

A separate, much smaller diagnostic smoke produced larger 50-point hit@4 and 25-point gold-citation gaps. It is retained only as a diagnostic that helped expose the need for positional reporting; the pooled counts above are the headline evidence. Neither result is an estimate for customer documents because the fixtures are deterministic and constructed.

The 256k A/A instrumentation smoke also makes partial reading visible. All 114/114 measured candidate cases were incomplete reads, and all 114 displayed the limit notice, so there were 0/114 missing notices among incomplete reads. There were no complete-read cases in this smoke; a false-positive notice rate is therefore not estimable. Prefix and tail scan reuse was 75%, with no uncached tail pass recorded. “Notice displayed” means the limitation was disclosed; it does not mean the unread portion was irrelevant.

Poisoned sources: a strong synthetic result that is not deployed protection

The threat model assumes that an attacker has managed to place manipulated text in the retrieval corpus. The benchmark measures several stages separately: whether poisoned evidence enters the top four, ranks first, is cited, contributes a canary value to the answer, or projects an instruction into the prompt. This design is informed directly by PoisonedRAG, USENIX Security 2025. Liu et al., USENIX Security 2024 motivates systematic attack/defence cells and explicit attacker goals.

The evaluated candidate is an opt-in ranking rule: evidence explicitly labelled verified ranks before evidence explicitly labelled unverified. In 96 matched synthetic trials, it produced the following comparison:

Measure

Legacy ranking

Tier rule on, labels present

Denominator

Poisoned evidence in top four

84/96

0/96

96 matched trials

Poisoned evidence at rank one

80/96

0/96

96 matched trials

Poisoned evidence cited

84/96

0/96

96 matched trials

Canary adopted

35/96

0/96

96 matched trials

Instruction projected into prompt

30/96

0/96

96 matched trials

Clean module evidence in top four

30/96

96/96

96 matched trials

The negative control is decisive. When sources had no provenance label, switching the rule on reproduced the unsafe baseline exactly across another 96 trials. With the switch off, 32 labelled and 32 unlabelled ablation trials were byte- and semantically identical, with normalized output SHA-256 bbee5f77ff88969be207ae9b1f149460e7d98d4263bb820f5e127f5d251e469c.

This is not a production security claim. The product audit found no production indexer that originates the required tier, the switch remains off, and the poisoning set had already been opened during development. A fresh sealed poisoning holdout is still missing. The evidence therefore supports only this conditional statement: on the stated synthetic fixtures, with correct labels supplied by the benchmark, the ranking rule removed the measured contamination and projection events.

StruQ, USENIX Security 2025 is a design precedent for separating trusted instructions from untrusted data. SupraAgent's candidate is not an implementation of StruQ: it is a retrieval-ranking rule and does not include StruQ's structured-query front end or specially trained model.

Performance: distributions and trade-offs, not a speed headline

At 256k characters, the local instrumentation smoke records 114 samples per measured path and reports nearest-rank percentiles. For the candidate prompt path, p50/p95/p99 were 21.655/49.813/51.103 ms. For the candidate answer path, they were 55.241/88.084/94.040 ms. Process-RSS deltas were sampled outside the timed boundary: prompt p50/p95/p99 were 0/425,984/1,343,488 bytes; answer p50/p95/p99 were 65,536/770,048/1,146,880 bytes. These snapshots exclude transient peaks and are not comparable to absolute memory from another process or host.

The fresh-process runner was intentionally bounded. Two sequential jobs completed without failure, with wall-time p50 7.713 s and p95 10.884 s. With two jobs under two-process concurrency, aggregate throughput rose from 0.1075 to 0.1697 jobs/s, while median per-job wall time worsened from 7.713 to 11.778 s. All output digests matched. With only two jobs per condition, this is an instrumentation smoke and an observed throughput/latency trade-off—not evidence of a latency improvement, capacity limit or production service level.

What changed after thirty iterations

  • The benchmark now reports evidence position, worst-position quality and position gaps for long inputs.

  • Abstention now includes the complete confusion matrix, so correct refusals cannot hide false refusals.

  • The target-function path is split into retrieval, admission, citation and lead placement.

  • Incomplete long-document reads and their user-facing notices are measured separately.

  • Latency uses p50, p95 and p99 with sample counts; process memory is recorded outside the timing boundary.

  • Poisoning is measured at retrieval, citation, answer adoption and prompt projection, with switch-off and unlabeled-source negative controls.

  • Development data and previously opened holdouts are machine-labelled as ineligible for new blind promotion claims.

One product candidate was added behind an off-by-default flag: provenance-tier ranking. It is not ready for promotion because production tier assignment and a fresh sealed poisoning holdout are missing. The German retrieval candidate remained rejected. The more important delivered change is the evaluation contract and its guardrails.

Scientific basis

Peer-reviewed work guides the protocol; paper-reported gains are never counted as SupraAgent results.

Reproducibility and evidence map

The final focused integration suite passed 182 tests and failed 0 in 276.04 seconds on frozen Git tree 389c55268e1846702e436b405642de1903984b8a. It covered long-context reporting, module-function evaluation, corpus poisoning, provenance tiers, fresh-process load, worker pinning and the thirty-iteration ledger. It made 0 provider calls. This focused result does not establish full application-suite health.

Claims in this article

Repository evidence

Scope comparison with the previous English article, including the legacy contact, twin, 612-document and 11,232-attempt suites

benchmarks/supraagent/2026-11-13-round-five/publication/article-en.html; live pre-rewrite snapshot SHA-256 52d1b0f81ced7ce178041b4091abba2aef1ab0fecb5d25d5d391e7d1e90e6547

Development retrieval, abstention and target-function funnel

benchmarks/supraagent/2026-11-16-thirty-iterations/iterations.json, iterations 1 and 14–18

Pooled long-context position and citation counts

benchmarks/supraagent/2026-11-13-round-five/runs/final/skill-who/L-chunked/candidate/answers.jsonl.gz, independently pooled by long_position for combined retrieval across 256k, 512k and 1.024M inputs

Small position diagnostic and partial-read disclosure

iterations.json, iterations 3–8; the notice false-positive rate is excluded because the disclosure smoke contains zero complete reads

Poisoning baseline, tier candidate, negative control and ablations

iterations.json, iterations 9–13 and 24–27

Scan reuse, latency percentiles and RSS snapshots

iterations.json, iterations 19–21

Fresh-process sequential and concurrent smoke

iterations.json, iterations 22–23

Test count, duration, source hashes, provider calls and mandatory blockers

benchmarks/supraagent/2026-11-16-thirty-iterations/integration-receipt.json

Protocol, promotion gates and paper-to-method mapping

benchmarks/supraagent/2026-11-16-thirty-iterations/README.md

What remains before stronger claims

The next evidence should come from a fresh sealed poisoning holdout, a production provenance scheme with independently audited labels, independent semantic and UX review, real customer-like questions and representative production load. Any stochastic language-model path would also require repeated seeded runs and uncertainty intervals. Until those exist, the defensible conclusion is deliberately narrow: the cycle improved what the benchmark can reveal, identified a retrieval bottleneck, quantified a modest pooled position gap while preserving the larger diagnostic as a warning, and found a promising but currently non-deployable provenance-ranking control.

Read the German benchmark article





SupraTix GmbH oder Partnergesellschaften - Alle Rechte vorbehalten.

Copyright © 2016–2026