Shadow-test an open decision model next to Jev before switching
Send every production decision to both Jev and an open model, compare answers, and train the open one until it is good enough to take over.
Author's figure: Author's figure: 35,812 Jev calls, $3.68 spent; on that day's test set the untrained Laya scored 44% and Jev 94% (Kitsuno). Reported by the builder, not measured by us.
The use case
Teams with steady decision traffic want lower cost or local control but cannot risk a worse model in production. This team logged every Jev request, sent the same requests to the open Laya model in shadow, and measured agreement on a test set. The untrained open model was far behind at first, so they trained it on their own questions and kept it running next to Jev on real requests. The switch only happens per question once the open model's accuracy is proven on logged data, and an engineer owns that decision.
Example workflow
- Log every Jev request and answer in production
- Send the same requests to Laya or another open model in shadow
- Build a labelled test set from logged traffic
- Fine-tune the open model on your own questions
- Switch question by question once accuracy matches, with an engineer's sign-off
Tools
- Jev (TypeSafe AI)
- Laya
- Unsloth
- Modal
Risks and guardrails
- A shadow test set may not reflect rare cases
- Training data drawn from Jev can copy its errors
- Running two models doubles cost during the test
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Laya in shadow, next to Jev on every request by Kitsuno on Made with Jev: madewithjev.com/builds/laya-in-shadow
- Laya in shadow, next to Jev on every request: original post or repo by Kitsuno: kitsuno.ai/content/engineering/laya-in-shadow
- Unsloth (official page): unsloth.ai (checked 2026-11-19)
- Modal (official page): modal.com (checked 2026-11-19)