I gave Jev a reranker's job.

I gave TypeSafe's Jev↗ thirty search results and asked it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. I wanted to know how far I'd get using Jev's answers as ranking scores.

What got me interested in the first place was asking several things about the same text in one call. Does this passage support a claim? Does it contradict it? Does it describe an actual test? I'd like to search from each of those angles without paying to read the same pile of text again every time.

#Giving Jev a reranker's job

A reranker takes the passages your search system found and puts the useful ones first. Jev returns structured answers to questions, so I used those answers to score the passages and sorted them in code.

I tried these relevance-scoring setups:

For the rubric, I convert the probabilities over those four levels into a score. Each candidate gets its own question; all those questions share the passages in one request. Separate output questions do not isolate a passage from the rest of that state. Jev supports mixing these question types↗.

The first run covered eight English datasets: SciFact, FiQA, Natural Questions, NFCorpus, TREC-COVID, BRIGHT biology and economics, and a Python CodeSearchNet sample.

Every setup started with the same thirty candidates from BM25, a keyword-based search method. I cut each passage to its first 2,000 characters. We're comparing how well the models reorder those snippets.

Of 2,327 queries, 1,617 had a passage labelled relevant among the candidates. Those are the queries scored below. For the rest, the first search hadn't found any labelled evidence to move up the list.

#Jev kept up with the rerankers

On the average across eight datasets, Jev's rubric scored 0.692 and Cohere Pro scored 0.691. I wouldn't call that a win. The 95% interval for the difference runs from −0.009 to +0.012: either model could be ahead.

There's another way to average the same results. Count each query equally, instead of each dataset, and Cohere leads 0.756 to 0.738. Both are worth seeing:

The score is nDCG@10: it rewards putting relevant passages near the top of the first ten results. Higher is better. A score of 0.69 doesn't mean 69% of questions were answered correctly.

FiQA has 411 scored queries; BRIGHT economics has 38. Under the second average, FiQA gets much more say. Cohere did better on FiQA and Natural Questions, so that change matters. If I were choosing a model for an application, I'd want a query mix that looked like its actual searches.

Here's what the calls cost, scaled to 1,000 queries. Each dataset has equal weight in this table.

ZeroEntropy was cheapest among the model setups in this table. Jev's cost was between zerank-2 and Cohere Pro.

I also tried making Jev run a tournament between groups of passages. That scored 0.668 on the dataset average, below the rubric's 0.692. Asking yes/no in a separate call for each passage scored 0.670. All that tournament machinery lost to asking how useful each passage was.

A separate duel setup scored 0.580, but I had restricted it to the keyword search's top ten. Anything outside those ten couldn't compete for the top spots.

#More BRIGHT questions

The first run had only 39 scored biology queries and 38 economics queries. I added the other five BRIGHT StackExchange subsets, bringing the total to 307 scored queries across seven subjects. These extra results stay separate from the eight-dataset averages above.

Averaging across subjects gave Jev's rubric 0.493, DeepSeek's JSON scores 0.487, zerank-2 0.479, and Cohere Pro 0.461. Jev was ahead numerically, but its gap over DeepSeek had a 95% interval of −0.013 to +0.025. The larger test still didn't resolve that difference.

#What happens when the passage says “not”?

“Works offline” and “does not work offline” have most of their words in common. Put both in a search result and the reranker has to care about the difference.

NevIR↗ tests this with paired passages and two corresponding queries. I ran its 1,383 test pairs, or 2,766 individual queries, with only two passages to choose from each time.

A model has to rank the right passage strictly higher for both queries to get credit for a pair. A tie fails. Independent random choices would get 25% of pairs right.

Jev's rubric beat Cohere Pro by 4.2 percentage points, with a paired-bootstrap 95% interval of +1.5 to +6.9 points. This was its clearest win in the experiment.

Jev's Choice setup ended up below Cohere, though. How I asked it to do the job mattered. This NevIR run also kept the two passages in a fixed order; I haven't repeated it with those two passages swapped.

#Why I tried Harsha's Qwen recipe

Harsha Gundala posted his Qwen recipe↗ in response to Jev's launch, joking that TypeSafe had spent two years in stealth and he'd spent two hours. He advertised five-times-faster JSON workloads on a Mac, with no new training required.

That was worth trying. If an existing small model could give me useful relevance scores through the same sort of interface, I'd want to know before building around Jev. I already had labelled passages and the other models' answers, so I could check whether the scores actually improved search results.

Harsha published the original implementation for Apple Silicon using MLX↗. I ran Shreyansh Singh's Transformers/PyTorch port↗ on rented RTX 4090s. The underlying model, Qwen2.5-1.5B-Instruct↗, comes from Alibaba's Qwen team. Despite “1B” in the recipe's name, it's a 1.5-billion-parameter model. The port contains serving code and presets; it doesn't supply newly fine-tuned weights.

The naming can be confusing. In Diogo Almeida's introduction to Jev↗, the TypeSafe founder describes a training method called Reinforcement Learning for Calibrated Decisions, or RLCD. The Qwen repository's name includes RLCD too, but the recipe I tested uses existing Qwen weights with parallel constrained decoding. It doesn't reproduce TypeSafe's training method.

The serving idea is useful on its own. Process the shared prompt once, keep the model's internal representation of it, then reuse that work to evaluate the different fields together. Each field has a fixed set of allowed answers. The code scores those choices and assembles the JSON, avoiding the need to generate the whole object token by token. That's how the port documents its batched mode↗, which is the mode I used.

For reranking, each field can ask whether one passage answers the query. I used its score for true to sort the passages. Getting a valid boolean back is guaranteed by the format; putting the right passage first still depends on the model. The port also explicitly warns that its normalized scores aren't empirically calibrated probabilities of correctness. A score of 0.8 isn't evidence that a judgment will be right 80% of the time.

My experiment tests that ranking quality. The original demo used a four-bit model on a Mac; this port loads unquantized weights on the GPU. I haven't reproduced Harsha's speed comparison, and these ranking results don't settle it.

#From thirty passages to one

First I gave it all thirty passages, with a yes/no question for each one. It scored 0.255, below the 0.486 keyword order it started with. The four-level rubric did better at 0.340, but still made the ranking worse.

Thirty passages might be too much to ask of a small model at once. So I ran it again with one passage and one yes/no question per prompt, using the same inference function. Thirty separate prompts for each search query.

That brought it up to 0.471. A big improvement over the thirty-passage version, but no clear gain over keyword ranking. Its gap against BM25 was −0.015, with a 95% interval of −0.033 to +0.004. That doesn't prove they're equivalent; it means this run didn't establish an improvement.

I also reversed the order of the thirty passages. The yes/no version picked a different top-ranked passage on 1,498 of 1,617 queries (92.6%). My initial comparison was with Jev Choice: 400 (24.7%). Those are different question types. The September 17 follow-up matches the thirty-yes/no setup on both sides: Jev changed its highest-ranked passage on 317 queries (19.6%). The full mode comparison is in the follow-up on passage order. These are passage rankings; Jev Choice's separate option to answer “none” isn't counted in that ranking metric.

The thirty-passage Qwen setup was very sensitive to order. The one-passage run tells me that simplifying the task helped a lot. It doesn't tell me whether the culprit was the longer context, keeping track of thirty passages, the prompt, or some combination. Both versions used the same decoding function, so I can't blame parallel decoding itself from this comparison.

I haven't run repeated identical Qwen requests. Reversing the input measures a different thing. I also haven't tested a larger Qwen model or trained one for relevance scoring. This is a result about the small model and prompts I ran, rather than a limit on what the approach could do.

I'm glad I ran the one-passage control. The 0.255 result on its own gives a very incomplete picture of what this model can do. The improvement belongs in the write-up, even though it still didn't give me a clear gain over BM25.

#Asking several questions at once

A passage that tells you you're wrong can still be an excellent search result. Its relevance score alone won't tell you whether it supports your claim or contradicts it.

For the interface I want, I'd ask separate questions about each passage:

Does this passage support the claim?
Does it contradict the claim?
Does it describe direct testing?
Does it apply to the situation I'm asking about?

That would give me several scores per passage. I could sort by support, look for counterarguments, or put more weight on direct testing. Once the scores are stored, changing their weights is just code; it doesn't need another model call.

zerank-2 also accepts instructions↗, so I could run separate reranking requests for those perspectives. With Jev, I can put several questions in one request and supply the text once.

#What sharing the text cost

To measure that, I kept thirty SciFact passages fixed and added search queries. Each query got two typed questions: choose the best passage, and judge whether any passage answers it. I made one request at each of the six batch sizes below.

Forty queries cost about 2.7 times as much as one. Dividing by forty gives about one-fifteenth of the cost per query in the single-query request. Extra questions still use input tokens, but the passages only had to be supplied once. That forty-query request took about 794 ms, measured once.

I tracked only the first query's choice, and it stayed at “none.” I haven't checked the quality of every answer in the batch. These requests show what sharing the passages cost; the support-versus-counterargument example still needs its own quality test.

#Where the context limit applies

Jev documents roughly 32,000 tokens for the supplied text and all the questions together. Add more questions and there's less room for passages.

Cohere Rerank 4↗ and zerank-2↗ also have roughly 32,000-token model windows. They evaluate the query with one document at a time, and their APIs can batch those evaluations. Ten documents in an API request therefore don't all have to fit in a single model window.

Jev can handle passages separately too: ask several questions about one passage, store the scores, then move to the next. If a question requires comparing passages together, those passages need to share the state for that evaluation.

The forty-query batch fit because it reused the same text. When I tried packing queries with their own separate document collections, I hit request limits.

#What I'd build next

For ordinary relevance ranking, I'd compare Jev, Cohere Pro, and zerank-2 on the queries the application will actually get. Cohere did better on the query-weighted average here, and zerank-2 cost less. Jev's rubric kept up and did particularly well on NevIR. The Qwen follow-up gave me another reason to check quality before treating a cheaper scoring method as a replacement.

I still want to build the search interface that got me into this: find the evidence, then let me ask what disagrees with it. Before trusting that, I'd run the support, contradiction, and direct-testing questions both together and separately, then compare the answers. The cost experiment makes me want to try it. It hasn't answered whether those judgments will be any good.

You can follow the benchmark question by question in the lab, including the Qwen follow-up: pick a dataset, open a query, and compare what each model put first. The one-passage runs and reversed-order control are there too. The repository has the code and saved responses.

#Experiment notes

The original measurements are from September 16, 2026; the Qwen follow-up was added September 17. Calls to Jev used jev-latest; responses reported version 1.13.0. Results can change when that alias points to a new model.

The original scored-query counts were SciFact 264, FiQA 411, Natural Questions 320, NFCorpus 239, TREC-COVID 50, BRIGHT biology 39, BRIGHT economics 38, and CodeSearchNet Python 256. Natural Questions and CodeSearchNet used samples; the Python corpus contained 1,000 functions. These scores describe this candidate-generation pipeline, rather than official full-benchmark runs.

Tied scores kept the original keyword-search order. The metric used the datasets' relevance labels and linear relevance gains. Those labels may be incomplete, and cutting passages to 2,000 characters can remove the relevant part.

Intervals used 10,000 paired bootstrap resamples: queries within each subject for retrieval, and passage pairs for NevIR. BRIGHT's subject set stayed fixed. I tried multiple configurations without adjusting the intervals for multiple comparisons, so treat the comparisons as exploratory. NevIR also reuses source passages across pairs, which introduces dependence between some samples.

The DeepSeek JSON baseline scored passages together with reasoning disabled. A separate one-token yes/no baseline returned saturated probabilities. That configuration needs investigating before drawing conclusions about chat models or log-probability reranking from its poor score.

Qwen used the port's run_parallel_generation function. Its separately documented tree-based decoding mode wasn't tested. Some long code-heavy prompts in the additional BRIGHT tests ran out of GPU memory when all thirty fields were evaluated together. Those rows used six fields at a time against the shared prompt and are marked in the saved responses.

Costs came from provider usage fields and the experiment's pricing assumptions. The cost table uses the original candidate lists for all 2,327 queries, including those without labelled evidence, with each dataset weighted equally. It excludes the separate answer-removed calls. I haven't checked these usage costs against invoices.

Cohere ran through OpenRouter; Jev and ZeroEntropy used their own endpoints. ZeroEntropy served many calls in its slower fallback mode. These timings mix model work with different serving paths, so they don't support a general claim about which model is fastest.

The tests also don't establish a safe confidence threshold for automating decisions. That would need measured error rates on the application, including searches where none of the supplied passages answers the question.