Mitrix's engineering team explains how healthcare document AI works when the input is a fax, not a clean PDF: OCR, structured extraction, retrieval with citations and the review step.
Most medical-record AI demos start with a clean PDF. Real intake starts with a faxed referral scanned at an angle, a lab result pasted into an email thread, a 60-page history with three patient identifiers, and a nursing note that says “see above”.
A model can read that pile. The harder question is whether the output can be trusted in a clinical or billing workflow, where a wrong date, a missed code or a patient mismatch has consequences. That is an architecture problem more than a model problem.
Start with the documents
Before choosing a model, collect the documents. Pull 300 to 500 pages from the sources that actually feed the workflow: referrals, orders, prior authorizations, lab results, progress notes, insurance cards. Include the bad ones. A sample made of clean PDFs produces a demo that fails in week two.
Label a ground-truth set: document type, patient identifiers, dates, codes, orders, and the fields the downstream system needs. This set becomes the acceptance test for every model and prompt change.
The first technical decision is layout-aware OCR. Plain text extraction loses table structure, checkboxes and reading order. A document AI service that returns pages, blocks, tables and bounding boxes gives the extraction model something to work with. Handwriting and stamps still fail often enough that the review step has to assume they will.
A pipeline that produces defensible data
- Ingest with provenance. Store the original file, page images, source system, received time and a hash. Every extracted field keeps a pointer to the page and bounding box that produced it.
- Classify before extraction. A referral, an insurance card and a lab result need different schemas. Route by document type and send unknown types to a human queue.
- Extract into a strict schema. Define field names, types, allowed values and required fields. Validate the model output against a JSON schema before anything else reads it.
- Apply rules. Check dates against the episode, provider identifier (NPI) checksums, procedure and diagnosis codes, totals, member IDs and cross-field dependencies. Rules catch errors a model will happily reproduce.
- Route exceptions to review. The reviewer sees the source page and the extracted value side by side, edits in place, and every change is logged with a reason.
- Publish with a version. Downstream systems receive the cleaned record, confidence per field, the model version and the review status.
The goal is a pipeline where the automated portion is measurable and the manual portion is small, targeted and auditable.
What breaks first
Pages get mixed between patients. Fax bundles combine records from several people more often than anyone wants to admit. Check patient identifiers on every page. A mixed bundle often looks fine on page one. If a page does not match the bundle, quarantine it and flag the source.
Medication lists and lab tables lose their meaning when OCR reads them as a stream of words. Use table extraction that keeps row and column relationships, then validate totals and units against the rest of the record.
“Yes”, “No” and “N/A” boxes are easy to see and hard to read. Extract the label and the selected state together, and send ambiguous marks to review.
Templates change without notice. Payers and facilities update forms quietly. Monitor extraction confidence by document source. A sudden drop usually means a new template, not a model failure.
Where the output goes changes the bar
An extracted field that fills a read-only summary can tolerate a lower confidence than a field that triggers a claim. Map every field to its consumer before setting thresholds. Clinical review, billing, quality reporting and care coordination carry different costs when the data is wrong.
If the same extraction feeds several consumers, publish it once with field-level confidence and let each consumer apply its own rule. Duplicating the pipeline per team creates inconsistent numbers and twice the audit surface.
Retrieval that a clinician can check
RAG in healthcare usually fails for boring reasons. The retrieval set is wrong, the chunks lose the section they came from, or the answer cannot be traced to a page.
Index the record, not the open web. Keep patient ID, encounter, document type, date and access scope in the metadata. Chunk by document structure, such as section headings, tables and note boundaries, instead of fixed token windows. For codes, medication names and provider names, combine keyword search with embeddings. Pure vector search is weak on exact identifiers.
Every answer should cite the page and bounding box it came from. When the evidence is missing, the system should say so. A confident answer with no source is worse than no answer in a clinical setting.
Build an evaluation set of 50 to 100 real questions with known answers. Measure retrieval recall first. If the right passage is not in the context, answer quality does not matter.
Designing the review step
High-risk fields deserve a mandatory review: patient identity, medication, procedure code, date of service and authorization number. Lower-risk fields can pass automatically above a confidence threshold, but the threshold should come from measured error rates, not a vendor default.
The review interface matters more than most teams expect. Reviewers need the source page, the extracted field, the reason the model flagged it and a keyboard path through the queue. Batch review helps, but only after the queue is sorted by risk.
Track minutes per document, exception rate, correction rate and the reasons reviewers reject a field. Those numbers tell you where to improve the pipeline. A benchmark score will not show where the queue slows down.
PHI boundaries
Every vendor in the path needs a business associate agreement (BAA), a retention setting that matches your policy, and a clear answer on whether customer data is used for training. Private endpoints and regional processing help, but they do not replace access control.
Redaction is tempting and limited. Removing names and addresses can also remove the clinical meaning the extraction needs. Scoped access, encryption, short retention and audit logs are usually more useful than aggressive redaction.
Prompts and responses contain PHI. Treat logs as PHI, restrict who can read them, and set a retention period. If the pipeline routes between model providers, each provider must meet the same controls. Model fallback keeps the service up; it does not satisfy a compliance requirement.
Model choice is the smaller decision
Teams spend weeks comparing models and days on the schema. In production, the schema, validation rules and review queue determine whether the system is usable. A smaller model with a tight schema and good rules often beats a larger model asked to return free-form JSON.
Use a provider-agnostic orchestration layer if you need fallback or cost control, but keep the contract stable: same schema, same validation, same audit fields. Test every model change against the same labeled set. A model upgrade should not silently change extraction behavior.
What to measure in a pilot
Baseline the manual process before the pilot: time per document, error rate, rework and cost per 100 pages. Without a baseline, a successful pilot has nothing to compare against.
Then measure straight-through rate, field-level precision and recall, reviewer minutes, exception reasons and the time to onboard a new document type. The last one predicts operating cost better than any single accuracy number.
Run the pilot on one document type for four to six weeks. Read-only integration first. Write-back after the review loop is stable.
A scope that ships
Pick a document with volume, clear fields and a measurable downstream cost. Referral packets, DME orders and prior authorizations are usually better first choices than “all medical records”.
Start with 10 to 15 fields and two reviewer roles. Add document types after the review queue, audit trail and reconciliation reports work. Expanding scope is easier than recovering trust after a bad extraction reaches a claim.
Build, buy or wrap
Document AI platforms sell ingestion, OCR and a review UI. They are a reasonable start when your document types are common and your integration needs are light. They become expensive when you need custom schemas, payer-specific rules, EMR write-back or per-client isolation.
A custom pipeline makes sense when the extracted data feeds a regulated workflow, when you need field-level audit, or when the document mix is specific to your business. A hybrid approach often works: buy OCR and document classification, build extraction, validation and review around your workflow. Our AI/ML team works that way regularly.
Questions for a document AI vendor
- Can the platform return page, block and bounding-box coordinates for every field?
- Does it support a strict output schema, or only free-form text?
- Where is data processed, and is it used for training?
- Can we bring our own model or route between providers?
- How are corrections stored and reused in evaluation?
- What does the audit trail show, and who can read it?
- How do we export everything if we leave?
Demo accuracy on clean samples does not answer any of these questions.
How Mitrix builds document AI
Mitrix has built healthcare document pipelines that combine layout-aware OCR, LLM extraction, rules validation and a review interface. The same work includes RAG with citations over medical knowledge and integration with EMR and HL7 systems. We keep human review and audit in the loop, and we work inside the client’s HIPAA-aware controls rather than around them.
You can see the healthcare work we have shipped in the DME network case study. If you are evaluating a document AI pilot, bring the messiest 50 pages you have. That sample will tell you more than a vendor demo. Talk to us when you want a second opinion on the architecture.