For years, extracting text from images meant Tesseract OCR — a trained model that works offline but chokes on handwriting, bad lighting, and complex layouts. Then multimodal LLMs arrived. Claude 3.5 Sonnet, GPT-4o, and Gemini 2.0 now read images natively. They understand context: a receipt isn't just text, it's a financial document with a date, vendor, total, and line items. A photo of hail damage on a roof isn't pixels; it's evidence of impact zones and extent. Handwritten notes become clean structured data. Aidxn now ships multimodal vision in three client tools: a receipt parser (vendor invoicing), a damage assessor (insurance claims), and a product cataloguer (e-commerce). The pattern is simple: image input → LLM vision → structured JSON output via tool use. Here's how to build it, cost it, and ship it.
Why Vision LLMs Beat Tesseract — The Gap
Tesseract is free and works offline. That's its advantage. But it fails in the real world. Handwriting, shadows, skewed photos, mixed text sizes, logos, checkboxes — Tesseract guesses badly or returns garbage. You spend hours writing regex to clean the output. Multimodal LLMs understand intent. Show Claude a photo of a roof with hail marks, and it identifies the damage zone and severity without training on 100k roof images. Feed it a rumpled receipt, and it extracts vendor, date, line items, and total even if the photo is blurry or sideways. The model reasons about context in ways pixel-to-text models cannot.
Cost-wise, vision LLMs are cheaper than you think. A 1024-pixel image costs 170 tokens with Claude. An invoice recognition API costs $0.10–$1.00 per image. A single Claude vision request (image + extraction prompt) runs ~$0.001 per image. At scale, multimodal LLMs win. Offline OCR's advantage evaporates when you're already running a backend. And quality? Not close. Vision LLM accuracy for structured extraction is 95%+; Tesseract is 70–80% with manual cleanup.
The Four Use Cases Aidxn Ships
1. Receipt Parsing for Vendor Invoicing
Upload a photo of an invoice. The LLM extracts vendor name, date, invoice number, line items, quantities, unit prices, tax, and total. Output is clean JSON that flows into your accounting system. No manual data entry.
const parseReceipt = async (imageBase64) => {
const response = await fetch("https://api.anthropic.com/v1/messages", {
method: "POST",
headers: {
"x-api-key": process.env.ANTHROPIC_API_KEY,
"content-type": "application/json",
"anthropic-version": "2023-06-01",
},
body: JSON.stringify({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
messages: [
{
role: "user",
content: [
{
type: "image",
source: {
type: "base64",
media_type: "image/jpeg",
data: imageBase64,
},
},
{
type: "text",
text: `Extract this receipt as JSON. Return: { vendor, date (YYYY-MM-DD), invoiceNumber, items: [{ name, quantity, unitPrice, total }], subtotal, tax, total }. If any field is missing, return null.`,
},
],
},
],
tools: [
{
name: "extract_receipt",
description: "Extract structured receipt data",
input_schema: {
type: "object",
properties: {
vendor: { type: "string" },
date: { type: "string" },
invoiceNumber: { type: "string" },
items: {
type: "array",
items: {
type: "object",
properties: {
name: { type: "string" },
quantity: { type: "number" },
unitPrice: { type: "number" },
total: { type: "number" },
},
},
},
tax: { type: "number" },
total: { type: "number" },
},
required: ["vendor", "date", "total"],
},
},
],
}),
});
const toolResult = response.content.find((c) => c.type === "tool_use");
return toolResult?.input;
};2. Damage Assessment for Insurance Claims
Homeowner photographs hail damage on their roof. Multimodal LLM analyzes the photo and returns: damage type, severity (minor/moderate/severe), impact zone location, and estimated coverage percentage. This feeds directly into claims workflows.
3. Document OCR Replacement
Scanned contracts, PDFs, legal docs — instead of running them through Tesseract, send them to Claude. It extracts text with 95% accuracy, preserves structure, and can answer follow-up questions ("extract all liability clauses"). No training, no fine-tuning. One API call.
4. Product Cataloguing via Photo
E-commerce: user uploads a product photo. LLM identifies the product name, category, key features, and estimated price range. Ideal for user-generated listings (marketplace apps) or internal inventory tagging.
Cost Math: When Multimodal Pays
Claude image cost: 170 tokens per 1024-pixel image. At $0.80 per million input tokens (Claude 3.5 Sonnet), one image costs ~$0.00014. A prompt + extraction adds 500–1000 tokens. Total: ~$0.0005–$0.001 per image. At 10k images/month, you're spending $5–$10 on vision. Third-party receipt APIs charge $0.10–$1.00 per image; at 10k images, that's $1,000–$10,000/month. Multimodal LLMs are 100–1000x cheaper.
Constraint: Claude processes images up to 20MB, and larger images incur more tokens. Optimize by resizing before upload. 1024x1024 is the sweet spot: high enough quality for receipts and damage photos, low enough token cost. If users upload 4MB phone photos, resize serverside to 1024px on the long edge — cuts tokens by 60%, same quality.
Implementation: Image → JSON → Tools
Three patterns: base64 upload (small images, embedded), URL fetch (remote images, no re-upload), or S3 presigned (large files, private documents). Aidxn ships base64 for receipts (small, instant), S3 presigned for damage assessments (HD photos, privacy-sensitive). Return structured JSON via tool use — the model commits to a schema, no parsing surprise text responses.
Six FAQs
Does Claude's vision work with PDFs?
Claude accepts `application/pdf` directly. One PDF page costs 170 tokens; each additional page adds ~50 tokens. For multi-page contracts, send each page separately in parallel requests — cheaper and faster than a single chunked PDF.
How accurate is vision OCR vs Tesseract?
Multimodal LLMs: 95%+ for clean documents, 85–90% for handwriting and skewed photos. Tesseract: 70–80% for clean documents, 40–50% for anything else. Vision LLM wins unless your input is always perfectly scanned and upright.
Can I batch process 100 images at once?
Technically yes, but parallelize instead. Send 10 requests concurrently; each waits 2–3 seconds. Batching (submitting 100 together) takes 3–5 minutes and ties up a connection. Use a job queue (Bull, Temporal, or Supabase Queues) for 1000+ images. Parallel is faster and more reliable.
What if the image is too dark or blurry for the LLM to read?
Claude returns `null` or a confidence penalty in the response. Catch it: if confidence is low, ask the user to re-upload a clearer photo. For mission-critical use (insurance damage), add manual review as a fallback — LLM extracts first, human validates if confidence is borderline.
Does vision work with animated GIFs or video frames?
Single image only. If you need to process video, extract frames client-side (via canvas), send frames individually. At 30fps, that's 1800 requests per minute — queue them instead of streaming.
Which LLM has the best vision: Claude, GPT-4o, or Gemini?
All three are excellent. Claude 3.5 Sonnet is fastest (200ms avg) and cheapest ($0.80/M tokens). GPT-4o is most accurate on complex documents (medical records, legal). Gemini 2.0 is free-tier friendly if you're prototyping. For production, benchmark your use case: spin up three parallel extractions on the same 50-image test set, measure accuracy and cost. Pick the winner. For most commercial work, Claude wins on speed + cost.
The Bottom Line
Multimodal LLMs are table stakes for any product handling documents, photos, or visual context. Stop training OCR models. Stop wrestling with Tesseract. Send the image to Claude with a structured JSON schema, get clean data back, move on. Cost is negligible (fractions of a cent), accuracy is high (95%+), and implementation time is one afternoon. For the invoice system Aidxn built last quarter, switching from Tesseract to Claude cut manual cleanup time by 80% and eliminated training costs entirely. Start with receipts or damage photos — highest ROI — then expand to product cataloguing and document extraction. See Aidxn Design services for guidance on scaling vision LLM systems to 100k+ images/month. For depth on streaming vision responses (real-time damage assessment video analysis), check Streaming LLM Responses — Why You Need Server-Sent Events Right Now.