================================================================================================ 1. Recount of the published order test Jev Choice, forward vs reversed: top pick changed on 400/1617 = 24.7% (22.7% to 26.9%) Qwen 30 yes/no, forward vs reversed: top pick changed on 1498/1617 = 92.6% (91.3% to 93.8%) Qwen 30 yes/no in one call, reversed vs published: any flip 1498/1617 = 92.6% (91.3% to 93.8%) real flip (gap 0.10 or more in either run) 1478/1617 = 91.4% (89.9% to 92.7%) right/wrong changed 439/1617 = 27.1% (25.0% to 29.4%) Qwen published order top pick right 13.6% nDCG@10 0.215 Qwen reversed order top pick right 26.7% nDCG@10 0.399 2. Does Jev's own confidence warn you? (forward run's confidence, the only one you'd have) confidence 0.00 to 0.50: 345 queries, flipped 197/345 = 57.1% (51.8% to 62.2%); 49% of all flips confidence 0.50 to 0.70: 327 queries, flipped 107/327 = 32.7% (27.9% to 38.0%); 27% of all flips confidence 0.70 to 0.90: 341 queries, flipped 72/341 = 21.1% (17.1% to 25.8%); 18% of all flips confidence 0.90 to 0.99: 352 queries, flipped 21/352 = 6.0% (3.9% to 8.9%); 5% of all flips confidence 0.99 to 1.00: 252 queries, flipped 3/252 = 1.2% (0.4% to 3.4%); 1% of all flips confidence 0.9 or more: 604 of 1617 queries (37%), flipped 24/604 = 4.0% (2.7% to 5.8%) both runs 0.9 or more: flipped 8/484 = 1.7% (0.8% to 3.2%) by the gap between Jev's top two probabilities (forward run): gap 0.00 to 0.05: 115 queries, flipped 77/115 = 67.0% (57.9% to 74.9%) gap 0.05 to 0.20: 208 queries, flipped 105/208 = 50.5% (43.7% to 57.2%) gap 0.20 to 0.50: 331 queries, flipped 129/331 = 39.0% (33.9% to 44.3%) gap 0.50 to 1.00: 963 queries, flipped 89/963 = 9.2% (7.6% to 11.2%) 3. Do the flips change right into wrong? both picks right (harmless) 101 (25% of flips) right became wrong 111 (28% of flips) wrong became right 90 (22% of flips) both wrong 98 (24% of flips) flips that change whether the top pick is right: 201/1617 = 12.4% (10.9% to 14.1%) 4. Quality, same queries forward only top pick right 76.0% nDCG@10 0.733 reversed only top pick right 74.7% nDCG@10 0.729 average of forward and reversed top pick right 76.2% nDCG@10 0.737 ================================================================================================ 5. Follow-up Choice runs (1617 queries with all five runs) noise floor, same order asked again: 47/1617 = 2.9% (2.2% to 3.8%) one run each, forward vs reversed: 400/1617 = 24.7% (22.7% to 26.9%) one run each, shuffle A vs shuffle B: 302/1617 = 18.7% (16.9% to 20.6%) one run each, forward vs shuffle A: 332/1617 = 20.5% (18.6% to 22.6%) one run each, forward vs shuffle B: 306/1617 = 18.9% (17.1% to 20.9%) one run each, reversed vs shuffle A: 310/1617 = 19.2% (17.3% to 21.2%) one run each, reversed vs shuffle B: 313/1617 = 19.4% (17.5% to 21.4%) average of two orders vs average of two other orders: fwd+rev vs A+B: 222/1617 = 13.7% (12.1% to 15.5%) fwd+A vs rev+B: 261/1617 = 16.1% (14.4% to 18.0%) fwd+B vs rev+A: 271/1617 = 16.8% (15.0% to 18.7%) all three splits pooled: 754/4851 = 15.5% (14.6% to 16.6%) real flips, fwd+rev vs A+B: 178/1617 = 11.0% (9.6% to 12.6%) real flips, one run each: fwd vs rev 371/1617 = 22.9% (21.0% to 25.1%); A vs B 271/1617 = 16.8% (15.0% to 18.7%) confidence gate: trust one run at 0.9 or more, otherwise add a second order and average gated (fwd, rev) vs gated (A, B): 226/1617 = 14.0% (12.4% to 15.8%); second call needed on 63% of queries noise floor among 0.9-or-more queries: 1/604 = 0.2% (0.0% to 0.9%) quality on these queries: one order (forward) top pick right 76.0% nDCG@10 0.733 one order (shuffle A) top pick right 75.1% nDCG@10 0.731 two orders averaged (forward + reversed) top pick right 76.2% nDCG@10 0.737 two orders averaged (shuffle A + B) top pick right 76.5% nDCG@10 0.736 gated (forward, reversed if unsure) top pick right 76.2% nDCG@10 0.735 four orders averaged top pick right 77.3% nDCG@10 0.742 ================================================================================================ 6. Same four numbers for every mode: order changed vs same order asked again Choice (30 passages, pick one): 1617 queries with every run reversed order vs published: any flip 400/1617 = 24.7% (22.7% to 26.9%) real flip (gap 0.10 or more in either run) 371/1617 = 22.9% (21.0% to 25.1%) right/wrong changed 201/1617 = 12.4% (10.9% to 14.1%) same order asked again vs published (noise floor): any flip 47/1617 = 2.9% (2.2% to 3.8%) real flip (gap 0.10 or more in either run) 13/1617 = 0.8% (0.5% to 1.4%) right/wrong changed 24/1617 = 1.5% (1.0% to 2.2%) published run had its top two scores tied on 14/1617 = 0.9% (0.5% to 1.4%) published order top pick right 76.0% nDCG@10 0.733 reversed order top pick right 74.7% nDCG@10 0.729 both averaged top pick right 76.2% nDCG@10 0.737 reversed, only questions where Jev said 0.9 or more (604): any flip 24/604 = 4.0% (2.7% to 5.8%) real flip (gap 0.10 or more in either run) 17/604 = 2.8% (1.8% to 4.5%) right/wrong changed 19/604 = 3.1% (2.0% to 4.9%) asked again, only questions where Jev said 0.9 or more (604): any flip 1/604 = 0.2% (0.0% to 0.9%) real flip (gap 0.10 or more in either run) 0/604 = 0.0% (0.0% to 0.6%) right/wrong changed 1/604 = 0.2% (0.0% to 0.9%) reversed, only questions where Jev said 0.99 or more (252): any flip 3/252 = 1.2% (0.4% to 3.4%) real flip (gap 0.10 or more in either run) 2/252 = 0.8% (0.2% to 2.8%) right/wrong changed 3/252 = 1.2% (0.4% to 3.4%) asked again, only questions where Jev said 0.99 or more (252): any flip 0/252 = 0.0% (-0.0% to 1.5%) real flip (gap 0.10 or more in either run) 0/252 = 0.0% (-0.0% to 1.5%) right/wrong changed 0/252 = 0.0% (-0.0% to 1.5%) shuffle A vs shuffle B: any flip 302/1617 = 18.7% (16.9% to 20.6%) real flip (gap 0.10 or more in either run) 271/1617 = 16.8% (15.0% to 18.7%) right/wrong changed 133/1617 = 8.2% (7.0% to 9.7%) 30 yes/no questions in one call: 1617 queries with every run reversed order vs published: any flip 317/1617 = 19.6% (17.7% to 21.6%) real flip (gap 0.10 or more in either run) 97/1617 = 6.0% (4.9% to 7.3%) right/wrong changed 142/1617 = 8.8% (7.5% to 10.3%) same order asked again vs published (noise floor): any flip 92/1617 = 5.7% (4.7% to 6.9%) real flip (gap 0.10 or more in either run) 0/1617 = 0.0% (0.0% to 0.2%) right/wrong changed 35/1617 = 2.2% (1.6% to 3.0%) published run had its top two scores tied on 113/1617 = 7.0% (5.8% to 8.3%) published order top pick right 74.3% nDCG@10 0.737 reversed order top pick right 74.5% nDCG@10 0.734 both averaged top pick right 74.7% nDCG@10 0.737 30 rubric scores in one call: 1617 queries with every run reversed order vs published: any flip 301/1617 = 18.6% (16.8% to 20.6%) real flip (gap 0.10 or more in either run) 57/1617 = 3.5% (2.7% to 4.5%) right/wrong changed 138/1617 = 8.5% (7.3% to 10.0%) same order asked again vs published (noise floor): any flip 74/1617 = 4.6% (3.7% to 5.7%) real flip (gap 0.10 or more in either run) 0/1617 = 0.0% (0.0% to 0.2%) right/wrong changed 36/1617 = 2.2% (1.6% to 3.1%) published run had its top two scores tied on 128/1617 = 7.9% (6.7% to 9.3%) published order top pick right 74.6% nDCG@10 0.738 reversed order top pick right 74.8% nDCG@10 0.735 both averaged top pick right 75.0% nDCG@10 0.739 one yes/no call per passage (order cannot matter): 1617 queries with every run same order asked again vs published (noise floor): any flip 105/1617 = 6.5% (5.4% to 7.8%) real flip (gap 0.10 or more in either run) 0/1617 = 0.0% (0.0% to 0.2%) right/wrong changed 41/1617 = 2.5% (1.9% to 3.4%) published run had its top two scores tied on 145/1617 = 9.0% (7.7% to 10.5%) published re-ask test (SciFact, 100 questions, 2 repeats each, same order): jev-noul-batch: top pick changed 3 of 200 jev-choice: top pick changed 4 of 200 7. Quality of every Jev mode on the same queries (all published runs) jev-choice top pick right 76.0% nDCG@10 0.733 jev-noul-pair top pick right 71.2% nDCG@10 0.717 jev-noul-batch top pick right 74.3% nDCG@10 0.737 jev-score-batch top pick right 74.6% nDCG@10 0.738 jev-tournament top pick right 75.4% nDCG@10 0.706 $ per 1,000 queries: jev-choice 0.34, jev-noul-pair 0.80, jev-noul-batch 0.43, jev-score-batch 0.46, jev-tournament 0.45