Check extracted document fields against the source text
After an LLM extracts fields from a document, Jev checks whether the source text actually supports each value.
Author's figure: Author reports the LLM extractor invented an unsupported value in 6 of 17 runs, and Jev scored the planted value at 0.02; 10 fields checked in about 0.9 s.. Reported by the builder, not measured by us.
The use case
Teams that use LLMs to pull fields from loan forms, claims or invoices sometimes get values the document never states. Jev cannot extract values, but it can answer whether the OCR text supports a given extracted value, with a probability. Fields with low support are flagged before they enter the system of record. The author also found Jev treats fax banners as content and cannot read checkbox layout, so those need cleanup or a person. An operator reviews every flagged field, and anything financial or legal is signed off by a person.
Example workflow
- OCR the document with LLMWhisperer or another OCR tool
- Extract fields with your existing LLM extractor
- Ask Jev one yes/no question per field: does the text support this value?
- Flag fields below a set probability
- Operator checks flagged fields against the original before saving
Tools
- Jev (TypeSafe AI)
- LLMWhisperer
- Python
Risks and guardrails
- Text-only reading misses checkboxes and layout
- Headers and banners can confuse the check
- A supported-looking value can still be wrong in the source
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Unstract blog by Arun Venkataswamy on Jev in document processing: unstract.com/blog/jev-and-where-a-system-one-model-fits-in-document-processing
- Hacker News submission: news.ycombinator.com/item