OCR read the invoice. Now what?

OCR can read an invoice surprisingly well and still put the wrong number in the wrong field. That’s the uncomfortable part of document automation: recognizing text is only the beginning.

An invoice parser has to decide that $1,284.50 is the total rather than the subtotal, that 08/12/2O26 is probably a date, and that a blurry description is not enough evidence to invent a GL account. Those decisions are where trust is won or lost.

I built the playground below around one fake invoice with deliberate OCR mistakes. The scores are hand-authored so the animation is repeatable; they are not vendor benchmarks or Intuit production numbers. At the default setting, a match above 82% is accepted, a nearby score is sent for review, and a weak match asks the user.

Auto-accept threshold = 82%

70–95%

Run matcherReset

Invoice PDFOCR overlay

Acme Supplies

Vend0r: Acme Coffee LLCOCR 71%

Inv #: INV-1042OCR 94%

Date: 08/12/2O26OCR 68%

Total due: $1,284.50OCR 91%

GL: 6100-Office Supp1iesOCR 63%

Page 1 of 1 · PO ref 88AOCR 88%

Schema fields

• Vendorpending
• Invoice #pending
• Datepending
• Amountpending
• GL accountpending

Active scores

Candidate scores appear when a field is being matched.

Run an operation above and every step lands here — scrub, replay, or slow it down.

Speed

0.5×1×1.5×2×

Run the matcher to walk OCR → candidates → confidence → auto / review / ask.

Move the threshold toward 95%. Invoice number and amount hold up, while the messier fields start asking for help. That is exactly what I want a threshold to do: change how often the system acts on its own, not make uncertainty disappear.

From a page to a field

I think about the pipeline in seven steps:

page image
  → OCR text and boxes
  → a few candidates for each field
  → normalization
  → ranking
  → accept, review, or ask
  → validation and write-back

An OCR-first model keeps the words and their positions explicit. Models such as LayoutLMv3 combine those with visual features. OCR-free models such as Donut go straight from pixels to structured text. Either way, the output can be wrong. “OCR-free” is an architecture choice, not a trust guarantee.

The failures usually fall into a few familiar buckets:

• The text is readable, but it came from the wrong box.
• The box is right, but a character is wrong: O instead of 0, for example.
• Nothing on the page is convincing enough.

That last case matters. “I don’t know” is a feature when the alternative is silently posting bad financial data.

Don’t score every scrap of text

Before ranking, I narrow the search. If I’m looking for invoice_total, I start near labels such as “Total due,” favor currency-shaped values, and pay attention to where totals usually sit on a page. For a date, I keep strings a date parser can actually understand.

Normalization needs to be field-aware. Replacing O with 0 can make sense inside an otherwise valid date. Applying that replacement to every vendor name would create new errors while fixing old ones. I also keep the original text and bounding boxes, even after joining or cleaning tokens, because someone will eventually need to explain where a value came from.

The demo uses a deliberately simple score:

score = 0.45 × lexical + 0.55 × semantic

I wouldn’t copy those weights into a real system. A useful ranker would also consider page layout, OCR confidence, and whether the value passes a type-specific parser:

s(field, candidate) =
    w_lex  × text_similarity
  + w_sem  × semantic_similarity
  + w_geo  × layout_prior
  + w_ocr  × ocr_confidence
  + w_type × parser_validity

Each signal catches something different. Fuzzy text matching helps with typos and stable patterns such as INV-####. Semantic similarity connects “Total due” with amount, but it should never be trusted to validate an account number. Layout gives useful context without hard-coding one vendor’s pixel coordinates. A deterministic parser can reject February 31 or a GL code that does not exist.

The mix should change by field. Amounts and IDs need stronger lexical and type checks. Free-text descriptions can lean more on meaning. There is no good reason for every field to share one global trust policy.

A score is not a probability

A model returning 0.93 does not mean it is right 93% of the time. To make that number useful, I’d take a held-out set, group predictions by score, compare confidence with actual accuracy, and fit a calibration map. This paper on neural-network calibration is a good place to start.

Then I’d choose thresholds from the cost of a mistake. For an amount field, the requirement might be:

precision(amount | score ≥ threshold) ≥ 99.5%

Precision alone can hide a useless system, though. If that target is reached by accepting only 12% of invoices, the model is safe but not doing much work. Always put coverage beside precision:

coverage = automatically accepted fields / all predicted fields

The real product choice is where to sit on that risk-versus-coverage curve.

Review and “I have no idea” are different states

If two candidates are close, or the top score lands just below the threshold, showing the likely answer for one-click confirmation is reasonable. If every candidate is weak, asking an open question is more honest. A prefilled guess in that situation only nudges the reviewer toward the model’s mistake.

I would also stop automatic write-back when:

• a high-impact action is hard to reverse
• a type check fails
• the document uses an unseen template, language, or currency
• fields disagree with one another

Cross-field checks catch errors that no individual score can. Does subtotal plus tax minus discount equal the total? Is the invoice date before the due date? Do the vendor and remittance details agree? Has the same vendor, invoice number, and amount already been recorded?

A failed check should send the document to review. It should not quietly “fix” the value and bury what happened.

Measure the fields people care about

“The model is 93% accurate” is almost meaningless if the misses are concentrated in amount and account fields. I’d report precision and recall by field, automation rate at a fixed precision target, user overrides, review time, and calibration error. I’d also slice by vendor, template, scan quality, and time.

Random page-level splits are especially misleading here. Two invoices from the same vendor can be nearly identical, so one ends up teaching the model how to pass the other. Splitting by vendor or template gives a more honest picture of how the system handles something new.

For every write, keep enough information to reconstruct the decision:

{
  "field": "invoice_total",
  "raw_text": "$1,284.5O",
  "normalized_value": { "currency": "USD", "minor_units": 128450 },
  "source_boxes": [[0.48, 0.72, 0.44, 0.07]],
  "model_version": "field-ranker-2026-08-24",
  "score": 0.993,
  "threshold": 0.995,
  "decision": "review"
}

That history is useful only if it is handled carefully. Documents may contain names, addresses, bank details, and other sensitive data. Encrypt retained artifacts, restrict access by tenant, and delete them on a defined schedule.

The standard I’d use is straightforward: if the software can’t show where a field came from, validate its type, and replay the decision later, it shouldn’t write that field without help.