receipt-extractor-lfm2.5-350m

Turn a receipt into structured JSON on the phone. This small model reads the OCR text of a receipt photo and returns line items, subtotal, tax, service charge/tip, discount and total as JSON. Built for an on-device, Splitwise-style "snap the bill and split it by item" app.

  • Base model: LiquidAI/LFM2.5-350M (354M parameters), fully fine-tuned
  • Input: OCR text of a receipt (one visual line per row), from Google ML Kit (Android), Apple Vision (iOS), PaddleOCR, etc.
  • Output: compact JSON: {"items":[{"name","qty","price"}], "subtotal", "tax", "service", "discount", "total"}
  • On-device: includes model.litertlm (int8, 364 MB) for Google's LiteRT-LM runtime (Android, iOS, desktop)
  • Speed: median 0.8 s, p90 1.9 s per receipt in the official LiteRT-LM runtime (int8, 4 CPU threads on a server CPU, excluding OCR). Expect similar or better on a recent phone; please benchmark on your target devices.

How it fits in an app

photo ──► on-device OCR (ML Kit / Apple Vision, ~0.3 s) ──► this model (~1 s) ──► JSON ──► sum check ──► split UI
                                                                                   └─ fails → ask the user to review

The sum check (do item prices + tax + service − discount equal the total?) is a strong confidence signal: in our tests, receipts that pass it were fully correct ~89% of the time, versus ~25% for receipts that fail it. receipt_utils.sum_check() in this repo implements it.

Quick start (Python / transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
import importlib.util, sys

repo = "Ajay0007/receipt-extractor-lfm2.5-350m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

# helper functions shipped with the model (prompt, OCR line grouping, JSON parsing, sum check)
spec = importlib.util.spec_from_file_location("receipt_utils", hf_hub_download(repo, "receipt_utils.py"))
ru = importlib.util.module_from_spec(spec); spec.loader.exec_module(ru)

ocr_text = """1 EGG TART 13,000
2 CHOCO CUSTARD PASTRY 24,000
REDBEAN BREAD 9,000
TOTAL 46,000
CASH 50,000
CHANGE 4,000"""

messages = [{"role": "user", "content": ru.build_prompt(ocr_text)}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_dict=True,
                                 return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
receipt = ru.parse_output(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
print(receipt)
print("sum check passed:", ru.sum_check(receipt))

Output for the example above:

{"items": [{"name": "EGG TART", "qty": "1", "price": "13,000"}, {"name": "CHOCO CUSTARD PASTRY", "qty": "2", "price": "24,000"}, {"name": "REDBEAN BREAD", "qty": "1", "price": "9,000"}], "total": "46,000"}

Use the prompt exactly as build_prompt writes it. The model was trained on that wording. Use greedy decoding (do_sample=False).

Getting OCR text in the right format

The model expects one visual receipt line per row, with the tokens on a line ordered left to right (so an item name and its price sit on the same row). Raw OCR engines return boxes, not lines. Use receipt_utils.ocr_boxes_to_text([(x_min, y_min, y_max, text), ...]), which is the same grouping used to build the training data. Port this ~20-line function to Kotlin/Swift for your app.

On-device: LiteRT-LM (Android / iOS)

model.litertlm is an int8 build (weights int8, activations fp32) made with litert-torch (dynamic_wi8_afp32, prefill 256, KV cache 1024 tokens). Measured on 100 receipts per set in the LiteRT-LM runtime (4 CPU threads): CORD split-ready 70% / total 94%, WildReceipt split-ready 31% / total 88%. That is within ~1 point of the bf16 weights on the same receipts, so int8 conversion costs essentially no accuracy. Latency: median 0.82 s, p90 1.90 s.

import litert_lm                                 # pip install litert-lm-api
from huggingface_hub import hf_hub_download
path = hf_hub_download("Ajay0007/receipt-extractor-lfm2.5-350m", "model.litertlm")
engine = litert_lm.Engine(path, backend=litert_lm.Backend.CPU(thread_count=4), max_num_tokens=2048)
conv = engine.create_conversation(sampler_config=litert_lm.SamplerConfig(top_k=1, temperature=0.0),
                                  max_output_tokens=768)
reply = conv.send_message(ru.build_prompt(ocr_text))
print(reply["content"] if isinstance(reply, dict) else reply)

For apps, load the same file with the LiteRT-LM Kotlin API (Android, stable) or Swift API (iOS), or use the Google AI Edge Gallery app to try it on a phone and see its speed.

Training data

Dataset Receipts used Region / language Labels used
CORD-v2 (CC-BY-4.0), train split 800 Indonesia (Indonesian/English, IDR) items, priced add-ons, subtotal, tax, service, discount, total
WildReceipt, train split 1,013 mostly US/EU photos (English, some German/French) items, subtotal, tax, tip→service, total
Total 1,813

Inputs were produced by running RapidOCR (PP-OCR) on the images, so the model learned from realistic, noisy OCR, not clean ground-truth text. CORD labels were corrected: priced add-ons stored as sub-items in CORD (e.g. "Extra Jelly 5,000") were restored, and a few annotation errors fixed. Store names, addresses, dates and payment lines (cash, change, card) are deliberately not extracted.

Training procedure

Full fine-tune (all weights) with plain supervised next-token loss on the assistant JSON only (prompt tokens masked).

Setting Value
Epochs 5
Learning rate / schedule 5e-5, cosine, 5% warmup
Batch size 8
Max sequence length 2,048 tokens
Precision bf16, gradient checkpointing
Hardware / time 1× NVIDIA A40, ~6 min
Target format compact JSON, null fields dropped (shorter output = faster on device)

We also tested LoRA (r=16, all linear layers, lr 2e-4): it matched the full fine-tune on CORD (70% vs 72% split-ready) and is a good option if you want small swappable adapters.

Evaluation

All numbers use OCR from RapidOCR, greedy decoding and the bf16 weights unless noted. Split-ready = every line item (name + price) and the total are exactly right, so the bill can be split with no edits. Item F1 = item matched on name (fuzzy ≥ 80) and exact price. Amounts are compared by value (60.000 = 60,000, 525 = 525.00).

This model

Test set Item F1 Total correct Split-ready
CORD-v2 test+val, corrected labels (200, Indonesia) 86.3% 95.9% 77.5%
WildReceipt test (472, US/EU photos) 57.3% 88.4% 30.9%
SROIE test (347, Malaysia, total only) — unseen dataset – 85.0% –
Real-world photos, Latin script (30, Wikimedia Commons, hand-verified) — unseen 29.5% 50.0% 13.3%
Real-world photos, CJK/Thai/Cyrillic (33, VLM-labeled) — unseen 17.1% 54.5% 12.1%

Compared with other small models (same pipeline, same test sets)

Model Trained on CORD split / total WildReceipt split / total SROIE total Photos split / total Size
Donut-base CORD-v2 (public, image→JSON) CORD 82 / 98 1 / 54 31 0 / 13 ~200M params
Gemma 3 270M CORD 64 / 93 4 / 48 33 0 / 20 ~290 MB
LFM2.5-350M CORD 73 / 94 2 / 40 19 0 / 13 364 MB
Qwen3-0.6B CORD 76 / 93 10 / 64 49 3 / 47 614 MB
LFM2-VL-450M (image→JSON) CORD 74 / 92 3 / 52 43 7 / 47 ~450M params
LFM2.5-350M — ours CORD-fixed + WildReceipt 78 / 96 31 / 88 85 13 / 50 364 MB
Qwen3-0.6B — ours CORD-fixed + WildReceipt 76 / 94 37 / 91 86 37 / 57 614 MB
LFM2-350M-Extract, zero-shot — 2 / 34 2 / 16 26 – / – 364 MB

Key takeaways:

  • Models trained only on CORD don't generalize. Every CORD-only model, including the public Donut-CORD, gets ≤10% split-ready on US/EU receipts. Adding WildReceipt raised WildReceipt totals from ~40–65% to ~90%, and SROIE (never seen in training) from ~20–50% to ~85%.
  • The int8 LiteRT build was evaluated separately; see the on-device section.
  • Seed-to-seed variation is about ±1–3 points; the 30-photo set has about ±9 points of noise.

Limitations

  • Real-world photos are still hard. On real phone photos, about 1 in 3 come out fully split-ready and about 1 in 2 have the right total. Typical failures:
    • totals labeled with keywords unseen in training ("Celkem", "SUMA PLN", "NET A PAYER") get omitted;
    • the OCR misses or garbles the total (OCR quality caps accuracy, and the model copies OCR errors);
    • item names and prices on separate lines of tall receipts get mis-paired.
  • Non-Latin scripts (Japanese, Chinese, Thai, Cyrillic) were not in the training data: totals are often right, but items rarely are.
  • Amounts are copied as printed (strings with the receipt's own separators and currency marks). Normalize them in the app (receipt_utils.to_number).
  • Handwritten bills, invoices and multi-page documents are out of scope.
  • Do not use the output for accounting or tax purposes without review.

Improve it for your users: fine-tune further on a few hundred labeled receipts from your target countries. In our experiments, adding diverse data mattered far more than model size.

License

LFM Open License v1.0: free for commercial use by organizations with < USD 10M annual revenue; larger organizations need a separate license from Liquid AI. Fine-tuning data: CORD-v2 (CC-BY-4.0) and WildReceipt. This repository contains modified weights (fine-tuned) of LiquidAI/LFM2.5-350M.

Downloads last month
319
Safetensors
Model size
0.4B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ajay0007/receipt-extractor-lfm2.5-350m

Finetuned
(112)
this model

Datasets used to train Ajay0007/receipt-extractor-lfm2.5-350m