A human example
Repository search returns several files for a duplicate-submission bug.
What the caller supplies
The agent supplies the bug description and candidate snippets with stable source IDs.
What happens next
Jev rates relevance. The agent reads the shortlist first and can retrieve every omitted snippet.
Illustrative example, not a recorded result.
Potential value: high
Repeated large tool results can consume a substantial share of a coding conversation. Better ordering could reduce unnecessary reading.
Evidence confidence: moderate
Two independent studies support testing passage ranking. They do not establish final coding-task quality or cost savings.
The rating describes support for this claim. It is separate from Jev's returned probability. How we assign ratings.
Evidence, including disagreement
- Rank retrieved passages before reading them. reported. Ranking gains depend on the dataset and how scores are averaged.
- Compare decision types before choosing a model. reported. A broad comparison supports task-specific testing, not one universal winner.
The next test
50 debugging or research tasks with labeled relevant passages and counterevidence; keep a fresh evaluation split.
Compare against
- Original retrieval order
- Deduplication only
- Direct agent selection
Measure
- Final answer quality
- Relevant evidence recall
- Host tokens
- Cache writes
- Full task time
Decision after the test
Adopt only if final answer quality is no worse within a prespecified tolerance and complete task cost or time improves.
The report will retain inputs, question versions, every attempt and failure examples. We will update the confidence rating after reviewing the result.
Use a related Engine recipe
Recipes are implementation starting points. Their presence does not mean the protocol above has passed.