Agents Lookup

Use cases / Decision models (Jev)

Shadow-test an open decision model next to Jev before switching

Send every production decision to both Jev and an open model, compare answers, and train the open one until it is good enough to take over.

DifficultyAdvanced
Setup time1-2 weeks
CategoryDecision models (Jev)

Author's figure: Author's figure: 35,812 Jev calls, $3.68 spent; on that day's test set the untrained Laya scored 44% and Jev 94% (Kitsuno). Reported by the builder, not measured by us.

The use case

Teams with steady decision traffic want lower cost or local control but cannot risk a worse model in production. This team logged every Jev request, sent the same requests to the open Laya model in shadow, and measured agreement on a test set. The untrained open model was far behind at first, so they trained it on their own questions and kept it running next to Jev on real requests. The switch only happens per question once the open model's accuracy is proven on logged data, and an engineer owns that decision.

Example workflow

  1. Log every Jev request and answer in production
  2. Send the same requests to Laya or another open model in shadow
  3. Build a labelled test set from logged traffic
  4. Fine-tune the open model on your own questions
  5. Switch question by question once accuracy matches, with an engineer's sign-off

Tools

Risks and guardrails

Sources

Written by Agents Lookup. Credits and official pages for the tools this use case names: