Document AI
OCR Evaluation for Documents, Forms, and Tables
Choose an invoice OCR review threshold using payable-field errors and human review capacity.
For invoice processing, the decision is which documents can safely pass without a person checking the payable fields. Test the full invoice, not just average text accuracy. A higher confidence threshold may catch errors but create more human work. This is a method, not a BuildAI OCR test.
Set the two operating limits
Suppose the finance team can review 30% of invoices each day. It also requires no known wrong total to be paid automatically during the pilot. The first limit comes from reviewer capacity; the second is a pilot safety rule, not a proof of zero future errors. “Correct invoice” means supplier, invoice number, due date, currency, and total are all right and attached to the right rows. A misplaced table value is wrong even if its digits were read correctly.
Label the documents and sweep confidence
Collect held-out invoices from each supplier, scan type, language, and layout. Include unreadable pages and duplicate invoice numbers. Have a reviewer label required fields and document-level correctness. Run the same preprocessing for each candidate. Compare field errors, invoices sent for review, and wrong invoices allowed through at several confidence cutoffs. Google Document AI's evaluation guide explains how confidence thresholds move precision and recall; the exact confidence values cannot be compared across systems without calibration.
Worked example: choose a cutoff
Imagine a pilot set of 1,000 invoices. The table is hypothetical. At 0.90, no wrong total passes automatically and 25% need review, within capacity. At 0.98, the safety rule still passes but the review queue is too large.
| Confidence cutoff | Auto processed | Sent to review | Wrong totals auto processed | Decision |
|---|---|---|---|---|
| 0.70 | 900 | 100 (10%) | 3 | Unsafe pilot |
| 0.90 | 750 | 250 (25%) | 0 | Passes pilot rules |
| 0.98 | 550 | 450 (45%) | 0 | Review capacity exceeded |
Run the 0.90 cutoff on fresh invoices from every major supplier before using it. Zero observed wrong totals in 750 automatic invoices does not mean the true error rate is zero. A rough “rule of three” would still allow a rate around 3/750, or 0.4%, at a 95% upper bound under simple independent-sample assumptions. Supplier clusters weaken that shortcut, so report errors by supplier and keep a human audit sample in production. If review volume rises above 30%, add capacity or improve extraction instead of silently lowering the cutoff.
Record document ID, supplier, fields, confidence, route, corrected values, reviewer time, and pipeline version. Count failed pages and repeat invoices as well as successful documents.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.