ocr models explained

revisiting the ideal input 🏙️

welcome to introspection ft. harsehaj! ⭐️ i’m harsehaj — always up to something in social good x tech. scroll to the end for a daily roundup of unique opportunities. :)

ocr models explained 🏙️

ocr model = optical character recognition ai model.

and, a lot of open-sourced models (chandra, deepseek, etc.) have been shipped this week. naturally, it’s time for a crash course. perhaps a side project experiment next week too.

optical character recognition is the process of converting images of text (like scanned documents, street signs, handwritten notes) into machine-readable text. this is the enabling model for many applications like license plate recognition, scanning your checks instead of manual data entry, and assistive technology for visually impaired folks.

let’s work through this process with a hypothetical image input: a crumpled-up receipt.

we used to have to write scripts for every single step involved in understanding the text on a receipt, from cleaning the image to understanding where the text even is. it’s like hovering over a kid and telling them every little rule, even the ones that seem intuitive. now, reading for ocr models is a more intelligent process. we let it teach itself what it even means to read.

the first thing that happens is vision encoding. a vision transformer (ViT), or a CNN-based backbone, takes in the raw rgb pixels and begins converting them into a semantic map of the page. it figures out, on its own, how to ignore glare, straighten tilted lines, and focus on text regions. it will learn patterns like “this region looks like text,” or “this table grid likely separates structured data.”

in older ocr, you had separate code to clean noise and detect text. this logic lives in the encoder’s weights for state-of-the-art models.

in deepseek-ocr, this encoder goes one step further by compressing the image. the model reduces millions of pixels into a few dozen visual tokens, which are tiny semantic units representing everything important about the page. each visual token is like a thought bubble: “upper left paragraph,” “bold headline,” “figure caption.”

this is how deepseek achieves its ~10x compression. instead of passing thousands of text tokens downstream, it hands over a concise visual summary of the receipt. pixels as inputs, rather than text tokens.

once the encoder has transformed the visual world into tokens, the decoder takes over. hello, language modelling. the decoder (often a transformer language model) starts producing output tokens (characters or subwords) autoregressively, one at a time. most importantly, each prediction step looks back not only at the previously generated text, but across the visual field as well.

so for every letter of the receipt identified, the model is also asking: “where was that curve of a letter again?”, “is that bold word part of a header or a paragraph?” or “what comes next visually?”

this makes modern ocr models much more resilient. they won’t need to be retrained for each new font or style of handwriting, and they can handle contextual clues provided by bolding, underlines, colours, and more. it’s a step up from just images of words to the text version of those words. the model also learns with context by understanding meaning through outputting structured representations like json fields, document entities, and key-value pairs.

this is important for self-healing/correction. the model can use contextual understanding to fix recognition errors as it generates text.

tl;dr:

• encoder: turns pixels into vision tokens

• decoder: turns vision tokens into text (or structure)

LOL.

everything in between (noise reduction, segmentation, bounding boxes, normalization) happens implicitly through learning. no more explicit scripts.

the training happens end-to-end while feeding in raw images and supervising with text outputs. the model learns every operation that helps minimize the final loss, including how to “clean” and “segment” internally. this is why models like deepseek-ocr can outperform older ocr systems while using fewer tokens and less compute.

so, what if text should be fed as images? once a model’s visual encoder understands how to read, surely there’s no reason not to let it read everything that way.

today’s drops 🔍️

• vercel’s ship ai conference is happening right now (you can tune in virtually)

• business it internship @ hermes

• true ventures fellowship applications are open until nov 10th for rising college seniors

---

i also write a bi-monthly update focused on my work and project updates if you’d like to check that out here.

want to post an opportunity on my blog? shoot me a line at harsehajx@gmail.com.

teehee,
harsehaj