ByHeartAI
Intermediate7 min read

Image Understanding & OCR

OCR is a specialist that reads characters and layout off an image. A VLM can talk about the whole scene. For anything that must be an exact string, OCR (or a document model) comes first.

Explain like I'm new to AI

Two jobs get mixed up:

  • What's in this picture? (a dog, a broken UI, a chart going down) → vision-language model
  • What letters are printed here? (invoice total, passport number, code) → OCR (optical character recognition), often with layout (tables, columns, reading order)
  1. 1. Detect text boxes
  2. 2. Read characters
  3. 3. Keep reading order / tables
  4. 4. Hand text to the LLM

Use when: Invoices, forms, receipts — you need the actual string and structure.

Watch out: Won't explain a chart or a meme. Garbage in, garbage out on blurry photos.

OCR extracts the letters. A VLM can explain the page. For money and IDs, extract then check — don't only ask the model.

Modern "document AI" combines both: layout + OCR + a language model that fills a JSON schema (see structured output).

Mental model

A human accountant reads the digits off the paper (OCR), then thinks about whether the expense is allowed (LLM). Skipping the first step and "vibing" the total is how $1,800 becomes $18,000.

How it works

OCR pipeline: detect text regions → recognize characters → order them (left-to-right, tables) → optional language model cleanup. Classic engines (Tesseract) and cloud document APIs still win on dense forms.

VLM path: feed the page image; ask questions. Strong on messy photos, whiteboards, memes, UIs. Weak as a ledger.

Best production pattern: extract (OCR/layout) → validate (schema, checksums, human confirm on money) → reason with the VLM on the image plus the extracted fields.

Real-world example

Expense app: OCR pulls total = 48.20 and merchant = …. The VLM checks the receipt photo looks like a real meal, not a screenshot of a meme. Neither step alone is enough.

Technical explanation

Layout matters as much as characters: a 3-column table dumped as a single string is garbage. Keep bounding boxes. For PDFs, native text (if the file has it) beats raster OCR.

Resolution and contrast dominate OCR errors. A VLM may "read" a blurry sign with a plausible guess — that's hallucinated literacy.

Screenshots of apps are a hybrid: OCR for copy-paste strings, VLM for "which tab is selected."

Common mistakes

Common mistake

Asking a VLM "what's the IBAN?" and wiring the answer to a payment. Extract, checksum, confirm.

  • OCR without layout on tables.
  • Skipping native PDF text and photographing a digital file.

When to use it

  • Forms, IDs, receipts, slides with numbers, any string that must round-trip exactly.

When NOT to use it

  • "Describe this landscape" — no letters to copy; use a VLM.

Alternatives

  • Human keying for high-stakes docs; barcode/QR instead of reading printed numbers.

Quick quiz

Question 1 of 3

When must you use OCR (or native PDF text) instead of only a VLM?

Question 2 of 3

Why does layout matter for OCR?

Question 3 of 3

True or false: you should wire a VLM's spoken IBAN straight into a payment.

Related concepts

  • Vision-Language Models — A vision-language model reads images as visual tokens beside your text, so one network can answer questions about what is in a picture.
  • What is Structured Output? — Structured output makes an LLM return data in a strict format like JSON that follows a schema, so software can reliably use its answers.
NextSpeech-to-Text & Text-to-Speech

Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI