Does the order of the passages change Jev's pick?

On this page

1. What I measured
2. Confidence identifies a more stable subset
3. The same order test across modes
4. Criterion labels and option order
5. Renaming passages in Choice
6. What confidence can tell me about correctness
7. Follow-up evidence and limits

In my reranking benchmark, I gave TypeSafe's Jev↗ thirty search results per query and used its answers to put the useful ones first. On ranking quality it kept up with Cohere Rerank 4 Pro. That leaves a different question: show Jev the same passages in a different order, or under different names, and does it still pick the same one?

I reversed, shuffled and renamed the passages, repeated identical requests, and changed the labels on the answer options. Then I checked whether Jev's own confidence tells you which picks hold still. These results first appeared on September 17 as an update to the benchmark article; they have their own page since September 23.

#What I measured

I tested passage permutations, repeated requests, and different labels for the same questions. These tests reuse the 1,617 queries from the benchmark whose original thirty candidates include at least one labelled-relevant passage, with the same candidates and 2,000-character truncation. The original and reversed Choice responses are from September 16; the new repeats, shuffles, and label experiments are from September 17. The saved Jev responses report jev-1.13.0.

Jev answered in the same four setups as in the benchmark:

The largest change to how I read the results is a distinction I needed to make explicit: ranking a passage first and choosing an answer are different measurements. Jev Choice can answer “none” while still assigning probabilities to every passage.

The ranking tables in the benchmark use the first definition, and so do the ranking flips below. The answer-correctness tables below use the second. “Correct” throughout means supported by the dataset's labels; unjudged passages count as not relevant.

#Confidence identifies a more stable subset

Here I group queries by the original response's confidence, then check whether reversing the passages changes the highest-ranked passage. The grouping uses information available before the second call.

At confidence 0.9 or above, the highest-ranked passage changes on 24 of 604 queries (4.0%), compared with 400 of 1,617 overall. That group covers 37.4% of this set. The other 62.6% still need a policy: another model, more evidence, review, or abstention. Filtering doesn't repair their rankings.

The confidence belongs to Jev's actual answer, which can be “none.” Within the same 604-query group, the actual answer changes on 17 queries (2.8%). Its labelled correctness is 563/604 (93.2%) in the original order and 556/604 (92.1%) after reversal. Those are different quantities from passage-ranking flips.

September 17 follow-up

What survives a confidence filter?

Raise the minimum returned confidence, then inspect which questions remain and how often reversing the candidate order changes the top-ranked passage.

These 1,617 queries come from the eight English datasets. Each has at least one labelled-relevant passage among its original thirty candidates. Answer-removed lists are excluded.

Loading the recorded questions…

Of the 400 ranking flips, 101 swap one labelled-relevant passage for another and 98 swap two passages without a relevant label. Another 111 change a labelled-correct ranking into an incorrect one, while 90 improve it. About half therefore change labelled correctness. The opposing changes partly cancel: ranking accuracy moves from 76.0% to 74.7%.

The lowest-confidence group contains 197 of the 400 flips (49.3%). A small gap between the top two passage probabilities is also associated with instability: when that gap is below 0.05, 77 of 115 queries (67.0%) flip.

Reversal produced the largest disagreement among the individual permutation comparisons I tested. Two seeded random orders disagree on 302/1,617 queries (18.7%). Averaging two orders, then comparing with the average of the other two, gives flip rates of 13.7%, 16.1%, and 16.8% across the three possible pairings. Their descriptive pooled rate is 15.5%; those pairings reuse the same queries and are not independent observations.

Ranking accuracy is 76.2%–76.5% for the two evaluated two-order averages and 77.3% for the four-order average. A policy that adds a second order only below confidence 0.9 still disagrees with its alternative-order counterpart on 14.0% of queries, and needs that second call for 62.6%. These are useful trade-offs to measure, rather than evidence that averaging or a threshold solves the problem.

#The same order test across modes

A ranking flip means a different passage receives the highest score. Equal scores retain the original BM25 order. The score-gap column counts flips where either run gives its preferred passage at least 0.10 more score than the other run's preferred passage. A numerical tolerance of 1e-10 includes decimal gaps that binary floating-point subtraction places just below 0.10. This is an exploratory diagnostic: Choice, Noul, and rubric scores have different meanings, so equal numerical gaps do not imply equal error severity.

All rows use the same 1,617 queries, with each query weighted equally. This differs from the benchmark's headline tables, which weight each dataset equally. The Qwen reversal results are the original saved control from the benchmark's Qwen test, not a newly repeated run.

For the original calls on this same query-weighted set, the estimated cost per 1,000 queries is $0.34 for Jev Choice, $0.43 for shared-request Noul, $0.46 for the rubric, and $0.80 for isolated-passage Noul. These use saved API usage and the recorded price assumptions. Qwen's corresponding $0.08 is an allocated GPU runtime estimate that excludes boot and idle time, so it is not an equivalent all-in deployment cost. None of these amounts has been reconciled with an invoice.

The matched thirty-yes/no comparison is therefore 19.6% versus 92.6% ranking flips, with labelled correctness changing on 8.8% versus 27.1%. It still concerns this small Qwen model, recipe, and prompt; it doesn't establish a limit for constrained decoding or larger models.

The shared-request Noul and rubric setups remain sensitive to order, despite each passage having its own output question. Many changes have small score gaps, but a near-tie can still put an unlabelled passage above a labelled answer.

An isolated-passage request contains no other passage whose position can change. Its reversal column is consequently not applicable, not a measured zero. Repeating those requests still changes the highest-ranked passage on 105/1,617 queries (6.5%); none of those repeat flips clears the 0.10 score-gap test. Small score changes and the tie-breaking rule still matter.

The new repeated-input comparisons span September 16 and 17. They combine repeat variability with any unobserved service changes across those dates; calling them a pure noise floor would overstate the control. Among the original 604 high-confidence Choice queries, the highest-ranked passage changed on one repeated request.

#Criterion labels and option order

A report from a user on TypeSafe's Discord prompted this check. I have left them unnamed. The measurements here are from my saved runs, rather than a reproduction of their particular examples.

I held the relevance instruction and positive/negative descriptions fixed, and changed the criterion labels: YES/NO, A/B, X/Y, and RELEVANT/IRRELEVANT, in both option orders. Noul plus exact duplicates of Noul and YES/NO brought the total to 11 variants per passage. Each query request contained all thirty passages and 330 questions. That gives 48,510 passage scores per variant. This tests labels and option order, not arbitrary rewrites of the instruction.

X/Y had the largest option-order effect among the tested pairs. Resending the packed request for 200 queries produced a similar shift, and the passage-level X/Y order shifts correlated at 0.87 across the two runs.

Noul in the packed request differed from the original shared-request Noul scores by 0.006 on average, the same rounded mean absolute difference as the repeated-input control. That comparison detected no additional average discrepancy from packing these variants together. It doesn't rule out individual or systematic effects.

Grouped by the original, separately recorded Noul score, the spread across the four canonical Choice label pairs was:

Label sensitivity was concentrated in the middle of the original score range. The extremes were more stable in this test. These are per-passage probabilities, not the Choice answer-confidence groups above.

AUROC measures how often a labelled-relevant passage receives a higher score than a passage without a relevant label, with ties receiving half credit. Noul had the highest AUROC and ranking score among the canonical variants here. I haven't established the uncertainty around these differences. Averaging the Choice variants did not improve the point estimate over canonical Noul.

Choice YES/NO was about 0.038 lower than Noul on average, but the per-passage absolute difference was larger and other labels behaved differently. I would keep the question type, labels, and order fixed when evaluating a threshold. There is no universal offset that makes the scores interchangeable.

#Renaming passages in Choice

This is a separate experiment: changing the passage identifiers in the pick-one question. It keeps the relevance instruction fixed.

Across the four shuffled naming runs, ranking accuracy ranged from 75.1% to 76.1%, and actual answer correctness within each run's own high-confidence group ranged from 92.5% to 94.8%. Those groups can contain different queries. This is not a result that every originally high-confidence query survives every naming change.

For the 466 queries with original confidence at least 0.95, all four naming schemes in the original passage order produced the same highest-ranked passage. Below confidence 0.5, they agreed on 68.4%. Renaming mattered less than shuffling here; it did not have zero effect.

The saved-response check also found three returned Choice answers that differed from the highest-probability passage in near-ties. Keeping the actual answer separate from a score-derived ranking avoids silently treating those as identical.

#What confidence can tell me about correctness

TypeSafe describes confidence↗ as a statistic derived from the shape of the answer distribution. It is not defined as the probability that the selected answer is correct. These buckets test its association with observed correctness; they do not establish that confidence 0.90 means 90% accuracy.

The following rates evaluate the actual returned Choice answer, including “none.” For the answer-removed lists, I removed all passages labelled relevant and refilled the list from later BM25 candidates. That does not establish that no genuinely relevant passage remains.

The pooled result uses the experiment's mixture of present and answer-removed lists. It isn't a deployment accuracy estimate. Answer-removed responses are available for 1,609 of the 1,617 eligible queries; one NFCorpus and seven TREC-COVID twins are unavailable. The intervals describe sampling uncertainty under the binomial calculation; they do not account for missed labels, shared passages, or dataset selection.

For the original lists containing a labelled answer:

Higher confidence generally identifies better answers, but the relationship is uneven. The broad ≥0.9 group combines the 85.5% result in the 0.90–0.95 slice with stronger results above it. A cutoff chosen after inspecting these results still needs validation on new data and the application's actual mix of questions.

A preliminary fourteen-case inspection accompanying these results reported twelve relevant picks, one probably relevant, and one unclear pick from answer-removed lists. Its sampling and adjudication were not documented as representative or independent, so I do not use it to correct the 56.8% result or estimate the share of errors caused by missing labels. The true semantic accuracy remains unmeasured.

Label density differs substantially across these datasets. In this query set, Natural Questions averages about 1.3 labelled-relevant passages per query across its corpus, versus about 493 in TREC-COVID. Even the latter is not exhaustive ground truth. For isolated-passage Jev Noul, the fraction of ≥0.9 scores labelled relevant is 34.5% on Natural Questions and 98.5% on TREC-COVID. Those differences also include different tasks and candidate distributions; they don't isolate a causal effect of annotation density.

The binary-output probability calibration summaries therefore need care. Across the pooled sample, X/Y has a smaller expected calibration error against the available labels than Noul (0.042 versus 0.075), while on TREC-COVID it has a larger one (0.292 versus 0.247). That does not establish which variant's probabilities best reflect true relevance on new data.

#Follow-up evidence and limits

The interactive rows contain both passage rankings and actual Choice answers, with input-file hashes. The order report, label report, and confidence report are saved analysis printouts, not raw model responses. Their shorthand such as “right,” “real flip,” or “noise floor” should be read with the definitions above.

The benchmark checkout contains the original cache/ responses, the follow-up order_runs/ and label_runs/ responses, the candidate labels, and the analysis scripts scripts/order_flips.py, scripts/label_eval.py, and scripts/calib_pooled.py. Those inputs, rather than this article's tables alone, are needed to reproduce the results.

These are exploratory results for one model version, one fixed relevance instruction, and selected public-dataset queries. Training-data overlap is unknown. Thresholds were inspected on this sample; I have not validated an automation policy on a held-out production workload. The label tests do not cover arbitrary instruction wording, prediction questions, other languages, or future model versions. The multiple-perspective search experiment remains untested.

The ranking comparison itself, with Cohere Rerank 4 Pro, zerank-2, the costs and the negation test, is in I gave Jev a reranker's job.