I gave Jev a reranker's job.

I gave TypeSafe's Jev↗ thirty search results and asked it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. I wanted to know how far I'd get using Jev's answers as ranking scores.

What got me interested in the first place was asking several things about the same text in one call. Does this passage support a claim? Does it contradict it? Does it describe an actual test? I'd like to search from each of those angles without paying to read the same pile of text again every time.

#Giving Jev a reranker's job

A reranker takes the passages your search system found and puts the useful ones first. Jev returns structured answers to questions, so I used those answers to score the passages and sorted them in code.

I tried these relevance-scoring setups:

For the rubric, I convert the probabilities over those four levels into a score. Each candidate gets its own question; all those questions share the passages in one request. Separate output questions do not isolate a passage from the rest of that state. Jev supports mixing these question types↗.

The first run covered eight English datasets: SciFact, FiQA, Natural Questions, NFCorpus, TREC-COVID, BRIGHT biology and economics, and a Python CodeSearchNet sample.

Every setup started with the same thirty candidates from BM25, a keyword-based search method. I cut each passage to its first 2,000 characters. We're comparing how well the models reorder those snippets.

Of 2,327 queries, 1,617 had a passage labelled relevant among the candidates. Those are the queries scored below. For the rest, the first search hadn't found any labelled evidence to move up the list.

#Jev kept up with the rerankers

On the average across eight datasets, Jev's rubric scored 0.692 and Cohere Pro scored 0.691. I wouldn't call that a win. The 95% interval for the difference runs from −0.009 to +0.012: either model could be ahead.

There's another way to average the same results. Count each query equally, instead of each dataset, and Cohere leads 0.756 to 0.738. Both are worth seeing:

The score is nDCG@10: it rewards putting relevant passages near the top of the first ten results. Higher is better. A score of 0.69 doesn't mean 69% of questions were answered correctly.

FiQA has 411 scored queries; BRIGHT economics has 38. Under the second average, FiQA gets much more say. Cohere did better on FiQA and Natural Questions, so that change matters. If I were choosing a model for an application, I'd want a query mix that looked like its actual searches.

Here's what the calls cost, scaled to 1,000 queries. Each dataset has equal weight in this table.

ZeroEntropy was cheapest among the model setups in this table. Jev's cost was between zerank-2 and Cohere Pro.

I also tried making Jev run a tournament between groups of passages. That scored 0.668 on the dataset average, below the rubric's 0.692. Asking yes/no in a separate call for each passage scored 0.670. All that tournament machinery lost to asking how useful each passage was.

A separate duel setup scored 0.580, but I had restricted it to the keyword search's top ten. Anything outside those ten couldn't compete for the top spots.

#More BRIGHT questions

The first run had only 39 scored biology queries and 38 economics queries. I added the other five BRIGHT StackExchange subsets, bringing the total to 307 scored queries across seven subjects. These extra results stay separate from the eight-dataset averages above.

Averaging across subjects gave Jev's rubric 0.493, DeepSeek's JSON scores 0.487, zerank-2 0.479, and Cohere Pro 0.461. Jev was ahead numerically, but its gap over DeepSeek had a 95% interval of −0.013 to +0.025. The larger test still didn't resolve that difference.

#What happens when the passage says “not”?

“Works offline” and “does not work offline” have most of their words in common. Put both in a search result and the reranker has to care about the difference.

NevIR↗ tests this with paired passages and two corresponding queries. I ran its 1,383 test pairs, or 2,766 individual queries, with only two passages to choose from each time.

A model has to rank the right passage strictly higher for both queries to get credit for a pair. A tie fails. Independent random choices would get 25% of pairs right.

Jev's rubric beat Cohere Pro by 4.2 percentage points, with a paired-bootstrap 95% interval of +1.5 to +6.9 points. This was its clearest win in the experiment.

Jev's Choice setup ended up below Cohere, though. How I asked it to do the job mattered. This NevIR run also kept the two passages in a fixed order; I haven't repeated it with those two passages swapped.

#Why I tried Harsha's Qwen recipe

Harsha Gundala posted his Qwen recipe↗ in response to Jev's launch, joking that TypeSafe had spent two years in stealth and he'd spent two hours. He advertised five-times-faster JSON workloads on a Mac, with no new training required.

That was worth trying. If an existing small model could give me useful relevance scores through the same sort of interface, I'd want to know before building around Jev. I already had labelled passages and the other models' answers, so I could check whether the scores actually improved search results.

Harsha published the original implementation for Apple Silicon using MLX↗. I ran Shreyansh Singh's Transformers/PyTorch port↗ on rented RTX 4090s. The underlying model, Qwen2.5-1.5B-Instruct↗, comes from Alibaba's Qwen team. Despite “1B” in the recipe's name, it's a 1.5-billion-parameter model. The port contains serving code and presets; it doesn't supply newly fine-tuned weights.

The naming can be confusing. In Diogo Almeida's introduction to Jev↗, the TypeSafe founder describes a training method called Reinforcement Learning for Calibrated Decisions, or RLCD. The Qwen repository's name includes RLCD too, but the recipe I tested uses existing Qwen weights with parallel constrained decoding. It doesn't reproduce TypeSafe's training method.

The serving idea is useful on its own. Process the shared prompt once, keep the model's internal representation of it, then reuse that work to evaluate the different fields together. Each field has a fixed set of allowed answers. The code scores those choices and assembles the JSON, avoiding the need to generate the whole object token by token. That's how the port documents its batched mode↗, which is the mode I used.

For reranking, each field can ask whether one passage answers the query. I used its score for true to sort the passages. Getting a valid boolean back is guaranteed by the format; putting the right passage first still depends on the model. The port also explicitly warns that its normalized scores aren't empirically calibrated probabilities of correctness. A score of 0.8 isn't evidence that a judgment will be right 80% of the time.

My experiment tests that ranking quality. The original demo used a four-bit model on a Mac; this port loads unquantized weights on the GPU. I haven't reproduced Harsha's speed comparison, and these ranking results don't settle it.

#From thirty passages to one

First I gave it all thirty passages, with a yes/no question for each one. It scored 0.255, below the 0.486 keyword order it started with. The four-level rubric did better at 0.340, but still made the ranking worse.

Thirty passages might be too much to ask of a small model at once. So I ran it again with one passage and one yes/no question per prompt, using the same inference function. Thirty separate prompts for each search query.

That brought it up to 0.471. A big improvement over the thirty-passage version, but no clear gain over keyword ranking. Its gap against BM25 was −0.015, with a 95% interval of −0.033 to +0.004. That doesn't prove they're equivalent; it means this run didn't establish an improvement.

I also reversed the order of the thirty passages. The yes/no version picked a different top-ranked passage on 1,498 of 1,617 queries (92.6%). My initial comparison was with Jev Choice: 400 (24.7%). Those are different question types. The September 17 follow-up matches the thirty-yes/no setup on both sides: Jev changed its highest-ranked passage on 317 queries (19.6%). The full mode comparison is below. These are passage rankings; Jev Choice's separate option to answer “none” isn't counted in that ranking metric.

The thirty-passage Qwen setup was very sensitive to order. The one-passage run tells me that simplifying the task helped a lot. It doesn't tell me whether the culprit was the longer context, keeping track of thirty passages, the prompt, or some combination. Both versions used the same decoding function, so I can't blame parallel decoding itself from this comparison.

I haven't run repeated identical Qwen requests. Reversing the input measures a different thing. I also haven't tested a larger Qwen model or trained one for relevance scoring. This is a result about the small model and prompts I ran, rather than a limit on what the approach could do.

I'm glad I ran the one-passage control. The 0.255 result on its own gives a very incomplete picture of what this model can do. The improvement belongs in the write-up, even though it still didn't give me a clear gain over BM25.

#Asking several questions at once

A passage that tells you you're wrong can still be an excellent search result. Its relevance score alone won't tell you whether it supports your claim or contradicts it.

For the interface I want, I'd ask separate questions about each passage:

Does this passage support the claim?
Does it contradict the claim?
Does it describe direct testing?
Does it apply to the situation I'm asking about?

That would give me several scores per passage. I could sort by support, look for counterarguments, or put more weight on direct testing. Once the scores are stored, changing their weights is just code; it doesn't need another model call.

zerank-2 also accepts instructions↗, so I could run separate reranking requests for those perspectives. With Jev, I can put several questions in one request and supply the text once.

#What sharing the text cost

To measure that, I kept thirty SciFact passages fixed and added search queries. Each query got two typed questions: choose the best passage, and judge whether any passage answers it. I made one request at each of the six batch sizes below.

Forty queries cost about 2.7 times as much as one. Dividing by forty gives about one-fifteenth of the cost per query in the single-query request. Extra questions still use input tokens, but the passages only had to be supplied once. That forty-query request took about 794 ms, measured once.

I tracked only the first query's choice, and it stayed at “none.” I haven't checked the quality of every answer in the batch. These requests show what sharing the passages cost; the support-versus-counterargument example still needs its own quality test.

#Where the context limit applies

Jev documents roughly 32,000 tokens for the supplied text and all the questions together. Add more questions and there's less room for passages.

Cohere Rerank 4↗ and zerank-2↗ also have roughly 32,000-token model windows. They evaluate the query with one document at a time, and their APIs can batch those evaluations. Ten documents in an API request therefore don't all have to fit in a single model window.

Jev can handle passages separately too: ask several questions about one passage, store the scores, then move to the next. If a question requires comparing passages together, those passages need to share the state for that evaluation.

The forty-query batch fit because it reused the same text. When I tried packing queries with their own separate document collections, I hit request limits.

#What I'd build next

For ordinary relevance ranking, I'd compare Jev, Cohere Pro, and zerank-2 on the queries the application will actually get. Cohere did better on the query-weighted average here, and zerank-2 cost less. Jev's rubric kept up and did particularly well on NevIR. The Qwen follow-up gave me another reason to check quality before treating a cheaper scoring method as a replacement.

I still want to build the search interface that got me into this: find the evidence, then let me ask what disagrees with it. Before trusting that, I'd run the support, contradiction, and direct-testing questions both together and separately, then compare the answers. The cost experiment makes me want to try it. It hasn't answered whether those judgments will be any good.

You can follow the benchmark question by question in the lab, including the Qwen follow-up: pick a dataset, open a query, and compare what each model put first. The one-passage runs and reversed-order control are there too. The repository has the code and saved responses.

#Experiment notes

The original measurements are from September 16, 2026; the Qwen follow-up was added September 17. Calls to Jev used jev-latest; responses reported version 1.13.0. Results can change when that alias points to a new model.

The original scored-query counts were SciFact 264, FiQA 411, Natural Questions 320, NFCorpus 239, TREC-COVID 50, BRIGHT biology 39, BRIGHT economics 38, and CodeSearchNet Python 256. Natural Questions and CodeSearchNet used samples; the Python corpus contained 1,000 functions. These scores describe this candidate-generation pipeline, rather than official full-benchmark runs.

Tied scores kept the original keyword-search order. The metric used the datasets' relevance labels and linear relevance gains. Those labels may be incomplete, and cutting passages to 2,000 characters can remove the relevant part.

Intervals used 10,000 paired bootstrap resamples: queries within each subject for retrieval, and passage pairs for NevIR. BRIGHT's subject set stayed fixed. I tried multiple configurations without adjusting the intervals for multiple comparisons, so treat the comparisons as exploratory. NevIR also reuses source passages across pairs, which introduces dependence between some samples.

The DeepSeek JSON baseline scored passages together with reasoning disabled. A separate one-token yes/no baseline returned saturated probabilities. That configuration needs investigating before drawing conclusions about chat models or log-probability reranking from its poor score.

Qwen used the port's run_parallel_generation function. Its separately documented tree-based decoding mode wasn't tested. Some long code-heavy prompts in the additional BRIGHT tests ran out of GPU memory when all thirty fields were evaluated together. Those rows used six fields at a time against the shared prompt and are marked in the saved responses.

Costs came from provider usage fields and the experiment's pricing assumptions. The cost table uses the original candidate lists for all 2,327 queries, including those without labelled evidence, with each dataset weighted equally. It excludes the separate answer-removed calls. I haven't checked these usage costs against invoices.

Cohere ran through OpenRouter; Jev and ZeroEntropy used their own endpoints. ZeroEntropy served many calls in its slower fallback mode. These timings mix model work with different serving paths, so they don't support a general claim about which model is fastest.

The tests also don't establish a safe confidence threshold for automating decisions. That would need measured error rates on the application, including searches where none of the supplied passages answers the question.

#Update, September 17: order, wording, and confidence

I followed up with passage permutations, repeated requests, and different labels for the same questions. These tests reuse the 1,617 queries whose original thirty candidates include at least one labelled-relevant passage, with the same candidates and 2,000-character truncation. The original and reversed Choice responses are from September 16; the new repeats, shuffles, and label experiments are from September 17. The saved Jev responses report jev-1.13.0.

The largest change to how I read the results is a distinction I needed to make explicit: ranking a passage first and choosing an answer are different measurements. Jev Choice can answer “none” while still assigning probabilities to every passage.

The ranking tables above use the first definition. The answer-correctness tables below use the second. “Correct” throughout means supported by the dataset's labels; unjudged passages count as not relevant.

#Confidence identifies a more stable subset

Here I group queries by the original response's confidence, then check whether reversing the passages changes the highest-ranked passage. The grouping uses information available before the second call.

At confidence 0.9 or above, the highest-ranked passage changes on 24 of 604 queries (4.0%), compared with 400 of 1,617 overall. That group covers 37.4% of this set. The other 62.6% still need a policy: another model, more evidence, review, or abstention. Filtering doesn't repair their rankings.

The confidence belongs to Jev's actual answer, which can be “none.” Within the same 604-query group, the actual answer changes on 17 queries (2.8%). Its labelled correctness is 563/604 (93.2%) in the original order and 556/604 (92.1%) after reversal. Those are different quantities from passage-ranking flips.

Of the 400 ranking flips, 101 swap one labelled-relevant passage for another and 98 swap two passages without a relevant label. Another 111 change a labelled-correct ranking into an incorrect one, while 90 improve it. About half therefore change labelled correctness. The opposing changes partly cancel: ranking accuracy moves from 76.0% to 74.7%.

The lowest-confidence group contains 197 of the 400 flips (49.3%). A small gap between the top two passage probabilities is also associated with instability: when that gap is below 0.05, 77 of 115 queries (67.0%) flip.

Reversal produced the largest disagreement among the individual permutation comparisons I tested. Two seeded random orders disagree on 302/1,617 queries (18.7%). Averaging two orders, then comparing with the average of the other two, gives flip rates of 13.7%, 16.1%, and 16.8% across the three possible pairings. Their descriptive pooled rate is 15.5%; those pairings reuse the same queries and are not independent observations.

Ranking accuracy is 76.2%–76.5% for the two evaluated two-order averages and 77.3% for the four-order average. A policy that adds a second order only below confidence 0.9 still disagrees with its alternative-order counterpart on 14.0% of queries, and needs that second call for 62.6%. These are useful trade-offs to measure, rather than evidence that averaging or a threshold solves the problem.

#The same order test across modes

A ranking flip means a different passage receives the highest score. Equal scores retain the original BM25 order. The score-gap column counts flips where either run gives its preferred passage at least 0.10 more score than the other run's preferred passage. A numerical tolerance of 1e-10 includes decimal gaps that binary floating-point subtraction places just below 0.10. This is an exploratory diagnostic: Choice, Noul, and rubric scores have different meanings, so equal numerical gaps do not imply equal error severity.

All rows use the same 1,617 queries, with each query weighted equally. This differs from the earlier headline tables that weight each dataset equally. The Qwen reversal results are the original saved control, not a newly repeated run.

For the original calls on this same query-weighted set, the estimated cost per 1,000 queries is $0.34 for Jev Choice, $0.43 for shared-request Noul, $0.46 for the rubric, and $0.80 for isolated-passage Noul. These use saved API usage and the recorded price assumptions. Qwen's corresponding $0.08 is an allocated GPU runtime estimate that excludes boot and idle time, so it is not an equivalent all-in deployment cost. None of these amounts has been reconciled with an invoice.

The matched thirty-yes/no comparison is therefore 19.6% versus 92.6% ranking flips, with labelled correctness changing on 8.8% versus 27.1%. It still concerns this small Qwen model, recipe, and prompt; it doesn't establish a limit for constrained decoding or larger models.

The shared-request Noul and rubric setups remain sensitive to order, despite each passage having its own output question. Many changes have small score gaps, but a near-tie can still put an unlabelled passage above a labelled answer.

An isolated-passage request contains no other passage whose position can change. Its reversal column is consequently not applicable, not a measured zero. Repeating those requests still changes the highest-ranked passage on 105/1,617 queries (6.5%); none of those repeat flips clears the 0.10 score-gap test. Small score changes and the tie-breaking rule still matter.

The new repeated-input comparisons span September 16 and 17. They combine repeat variability with any unobserved service changes across those dates; calling them a pure noise floor would overstate the control. Among the original 604 high-confidence Choice queries, the highest-ranked passage changed on one repeated request.

#Criterion labels and option order

A report from a user on TypeSafe's Discord prompted this check. I have left them unnamed. The measurements here are from my saved runs, rather than a reproduction of their particular examples.

I held the relevance instruction and positive/negative descriptions fixed, and changed the criterion labels: YES/NO, A/B, X/Y, and RELEVANT/IRRELEVANT, in both option orders. Noul plus exact duplicates of Noul and YES/NO brought the total to 11 variants per passage. Each query request contained all thirty passages and 330 questions. That gives 48,510 passage scores per variant. This tests labels and option order, not arbitrary rewrites of the instruction.

X/Y had the largest option-order effect among the tested pairs. Resending the packed request for 200 queries produced a similar shift, and the passage-level X/Y order shifts correlated at 0.87 across the two runs.

Noul in the packed request differed from the original shared-request Noul scores by 0.006 on average, the same rounded mean absolute difference as the repeated-input control. That comparison detected no additional average discrepancy from packing these variants together. It doesn't rule out individual or systematic effects.

Grouped by the original, separately recorded Noul score, the spread across the four canonical Choice label pairs was:

Label sensitivity was concentrated in the middle of the original score range. The extremes were more stable in this test. These are per-passage probabilities, not the Choice answer-confidence groups above.

AUROC measures how often a labelled-relevant passage receives a higher score than a passage without a relevant label, with ties receiving half credit. Noul had the highest AUROC and ranking score among the canonical variants here. I haven't established the uncertainty around these differences. Averaging the Choice variants did not improve the point estimate over canonical Noul.

Choice YES/NO was about 0.038 lower than Noul on average, but the per-passage absolute difference was larger and other labels behaved differently. I would keep the question type, labels, and order fixed when evaluating a threshold. There is no universal offset that makes the scores interchangeable.

#Renaming passages in Choice

This is a separate experiment: changing the passage identifiers in the pick-one question. It keeps the relevance instruction fixed.

Across the four shuffled naming runs, ranking accuracy ranged from 75.1% to 76.1%, and actual answer correctness within each run's own high-confidence group ranged from 92.5% to 94.8%. Those groups can contain different queries. This is not a result that every originally high-confidence query survives every naming change.

For the 466 queries with original confidence at least 0.95, all four naming schemes in the original passage order produced the same highest-ranked passage. Below confidence 0.5, they agreed on 68.4%. Renaming mattered less than shuffling here; it did not have zero effect.

The saved-response check also found three returned Choice answers that differed from the highest-probability passage in near-ties. Keeping the actual answer separate from a score-derived ranking avoids silently treating those as identical.

#What confidence can tell me about correctness

TypeSafe describes confidence↗ as a statistic derived from the shape of the answer distribution. It is not defined as the probability that the selected answer is correct. These buckets test its association with observed correctness; they do not establish that confidence 0.90 means 90% accuracy.

The following rates evaluate the actual returned Choice answer, including “none.” For the answer-removed lists, I removed all passages labelled relevant and refilled the list from later BM25 candidates. That does not establish that no genuinely relevant passage remains.

The pooled result uses the experiment's mixture of present and answer-removed lists. It isn't a deployment accuracy estimate. Answer-removed responses are available for 1,609 of the 1,617 eligible queries; one NFCorpus and seven TREC-COVID twins are unavailable. The intervals describe sampling uncertainty under the binomial calculation; they do not account for missed labels, shared passages, or dataset selection.

For the original lists containing a labelled answer:

Higher confidence generally identifies better answers, but the relationship is uneven. The broad ≥0.9 group combines the 85.5% result in the 0.90–0.95 slice with stronger results above it. A cutoff chosen after inspecting these results still needs validation on new data and the application's actual mix of questions.

A preliminary fourteen-case inspection accompanying these results reported twelve relevant picks, one probably relevant, and one unclear pick from answer-removed lists. Its sampling and adjudication were not documented as representative or independent, so I do not use it to correct the 56.8% result or estimate the share of errors caused by missing labels. The true semantic accuracy remains unmeasured.

Label density differs substantially across these datasets. In this query set, Natural Questions averages about 1.3 labelled-relevant passages per query across its corpus, versus about 493 in TREC-COVID. Even the latter is not exhaustive ground truth. For isolated-passage Jev Noul, the fraction of ≥0.9 scores labelled relevant is 34.5% on Natural Questions and 98.5% on TREC-COVID. Those differences also include different tasks and candidate distributions; they don't isolate a causal effect of annotation density.

The binary-output probability calibration summaries therefore need care. Across the pooled sample, X/Y has a smaller expected calibration error against the available labels than Noul (0.042 versus 0.075), while on TREC-COVID it has a larger one (0.292 versus 0.247). That does not establish which variant's probabilities best reflect true relevance on new data.

#Follow-up evidence and limits

The interactive rows contain both passage rankings and actual Choice answers, with input-file hashes. The order report, label report, and confidence report are saved analysis printouts, not raw model responses. Their shorthand such as “right,” “real flip,” or “noise floor” should be read with the definitions above.

The benchmark checkout contains the original cache/ responses, the follow-up order_runs/ and label_runs/ responses, the candidate labels, and the analysis scripts scripts/order_flips.py, scripts/label_eval.py, and scripts/calib_pooled.py. Those inputs, rather than this article's tables alone, are needed to reproduce the results.

These are exploratory results for one model version, one fixed relevance instruction, and selected public-dataset queries. Training-data overlap is unknown. Thresholds were inspected on this sample; I have not validated an automation policy on a held-out production workload. The label tests do not cover arbitrary instruction wording, prediction questions, other languages, or future model versions. The multiple-perspective search experiment remains untested.