What if instead of search queries being interpreted as a “bag of terms” we could see them as a “bag of decisions”?
\nWe know Jev acts as frontier quality reranker. As a fun experiment, I wanted to try another technique: Don’t just ask Jev “Is this relevant for {query}?”. Instead, let an LLM generate the query’s criteria: a rubric to assess relevance for just this query.
\nWe prompt an LLM with “generate yes / no question where the affirmative indicates relevance”. For the query desk for kids it might produce:
1 - Is the document about desks designed for children?\n2 - Does the document offer products or information on kids’ desks?\n3 - Does it mention child-sized or age-appropriate desk dimensions or ergonomics?\n4 - Does it reference desks for homework, study, or activities for kids?\n5 - Are features specific to children’s desks (e.g., height-adjustable for kids, safety/rounded edges, storage fo\nr school supplies) discussed?\n6 - Does it include buying guides, reviews, or comparisons of kids’ desks?\n7 - Does it provide pricing or availability for children’s desks?\n8 - Does it mention desks for specific child age ranges (e.g., toddlers, elementary, teens) as kids’ desks?\n9 - Are brands or models of children’s desks listed?\n10 - Does it cover where to buy or how to choose a desk for kids?Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance:
document = """\nThis adjustable study desk is perfect for kids ages 5–12.\nIts height grows with your child.\n"""\nresponse = client.decide(\n state=document,\n questions={\n "question_1": Noul(\n instructions="Is this document about desks designed for children?"\n ),\n "question_2": Noul(\n instructions="Does the document offer products or information on kids’ desks?"\n )\n ...\n },\n)Instead of a query being represented as a bag of terms, we have a bag of questions. Since Jev gives the probability of ‘yes’, we can simply sum the probability of each question being true as the rerank score:
\ntotal_probability = 0.0\nfor question_id, answer in response.answers.items():\n probability = answer.noul\n print(f"{question_id}: {probability:.2f}")\n total_probability += probabilityNow total_probability can be our reranker’s final score. We can add that to BM25, or just take it directly. In other words:
I ran an experiment comparing BM25 vs two variants (gory details here):
\nI then evaluated on two e-commerce datasets Wayfair WANDS and Amazon ESCI. How did it fair?
\nDoes this work? It seems to. Here’s some initial results:
\n| Dataset | Strategy name | Mean NDCG | Median NDCG |
|---|---|---|---|
| WANDS | bm25 | 0.5407 | 0.475 |
| WANDS | jev_reranker | 0.574 | 0.561 |
| WANDS | bag_of_decisions | 0.610 | 0.561 |
| ESCI | bm25 | 0.289 | 0.171 |
| ESCI | jev_reranker | 0.314 | 0.205 |
| ESCI | bag_of_decisions | 0.360 | 0.341 |
Or graphically
\nWe’re essentially doing per-query feature creation. Use an LLM as inspiration of what’s important to consider for this query. Then use Jev to measure whether we’re covering that criteria. We might wonder how we might modify the LLM call to provide more incisive + diverse questions then the ones above (IMO some are repetitive)
\nIt’s a fun approach. We seem to be able to train more and more decision models, possible even ones focused on specific domains. At least in this one toy experiment, it seems to be a promising approach worth deeper study. Enjoy!
","headings":[{"level":2,"text":"How does it stack up?","id":"how-does-it-stack-up"}]}}