|
|
---
|
|
|
license: apache-2.0
|
|
|
language:
|
|
|
- en
|
|
|
- vi
|
|
|
pipeline_tag: image-to-text
|
|
|
model-index:
|
|
|
- name: HTR-ConvText
|
|
|
results:
|
|
|
- task:
|
|
|
type: image-to-text
|
|
|
name: Handwritten Text Recognition
|
|
|
dataset:
|
|
|
name: IAM
|
|
|
type: iam
|
|
|
split: test
|
|
|
metrics:
|
|
|
- type: cer
|
|
|
value: 4.0
|
|
|
name: Test CER
|
|
|
- type: wer
|
|
|
value: 12.9
|
|
|
name: Test WER
|
|
|
- task:
|
|
|
type: image-to-text
|
|
|
name: Handwritten Text Recognition
|
|
|
dataset:
|
|
|
name: LAM
|
|
|
type: lam
|
|
|
split: test
|
|
|
metrics:
|
|
|
- type: cer
|
|
|
value: 2.7
|
|
|
name: Test CER
|
|
|
- type: wer
|
|
|
value: 7.0
|
|
|
name: Test WER
|
|
|
- task:
|
|
|
type: image-to-text
|
|
|
name: Handwritten Text Recognition
|
|
|
dataset:
|
|
|
name: READ2016
|
|
|
type: read2016
|
|
|
split: test
|
|
|
metrics:
|
|
|
- type: cer
|
|
|
value: 3.6
|
|
|
name: Test CER
|
|
|
- type: wer
|
|
|
value: 15.7
|
|
|
name: Test WER
|
|
|
- task:
|
|
|
type: image-to-text
|
|
|
name: Handwritten Text Recognition
|
|
|
dataset:
|
|
|
name: HANDS-VNOnDB
|
|
|
type: hands-vnondb
|
|
|
split: test
|
|
|
metrics:
|
|
|
- type: cer
|
|
|
value: 3.45
|
|
|
name: Test CER
|
|
|
- type: wer
|
|
|
value: 8.9
|
|
|
name: Test WER
|
|
|
---
|
|
|
---
|
|
|
# HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition
|
|
|
|
|
|
<div align="center"> <img src="image/architecture.png" alt="HTR-ConvText Architecture" width="800"/> </div>
|
|
|
|
|
|
<p align="center">
|
|
|
<a href="https://huggingface.co/DAIR-Group/HTR-ConvText">
|
|
|
<img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue">
|
|
|
</a>
|
|
|
<a href="https://github.com/DAIR-Group/HTR-ConvText">
|
|
|
<img alt="GitHub" src="https://img.shields.io/badge/GitHub-Repo-181717.svg?logo=github&logoColor=white">
|
|
|
</a>
|
|
|
<a href="https://github.com/DAIR-Group/HTR-ConvText/blob/main/LICENSE">
|
|
|
<img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-green">
|
|
|
</a>
|
|
|
<a href="https://arxiv.org/abs/2512.05021">
|
|
|
<img alt="arXiv" src="https://img.shields.io/badge/arXiv-2512.05021-b31b1b.svg">
|
|
|
</a>
|
|
|
</p>
|
|
|
|
|
|
## Highlights
|
|
|
|
|
|
HTR-ConvText is a novel hybrid architecture for Handwritten Text Recognition (HTR) that effectively balances local feature extraction with global contextual modeling. Designed to overcome the limitations of standard CTC-based decoding and data-hungry Transformers, HTR-ConvText delivers state-of-the-art performance with the following key features:
|
|
|
|
|
|
- **Hybrid CNN-ViT Architecture**: Seamlessly integrates a ResNet backbone with MobileViT blocks (MVP) and Conditional Positional Encoding, enabling the model to capture fine-grained stroke details while maintaining global spatial awareness.
|
|
|
- **Hierarchical ConvText Encoder**: A U-Net-like encoder structure that interleaves Multi-Head Self-Attention with Depthwise Convolutions. This design efficiently models both long-range dependencies and local structural patterns.
|
|
|
- **Textual Context Module (TCM)**: An innovative training-only auxiliary module that injects bidirectional linguistic priors into the visual encoder. This mitigates the conditional independence weakness of CTC decoding without adding any latency during inference.
|
|
|
- **State-of-the-Art Performance**: Outperforms existing methods on major benchmarks including IAM (English), READ2016 (German), LAM (Italian), and HANDS-VNOnDB (Vietnamese), specifically excelling in low-resource scenarios and complex diacritics.
|
|
|
|
|
|
## Model Overview
|
|
|
|
|
|
HTR-ConvText configurations and specifications:
|
|
|
|
|
|
| Feature | Specification |
|
|
|
| ------------------- | --------------------------------------------------- |
|
|
|
| Architecture Type | Hybrid CNN + Vision Transformer (Encoder-Only) |
|
|
|
| Parameters | ~65.9M |
|
|
|
| Backbone | ResNet-18 + MobileViT w/ Positional Encoding (MVP) |
|
|
|
| Encoder Layers | 8 ConvText Blocks (Hierarchical) |
|
|
|
| Attention Heads | 8 |
|
|
|
| Embedding Dimension | 512 |
|
|
|
| Image Input Size | 512×64 |
|
|
|
| Inference Strategy | Standard CTC Decoding (TCM is removed at inference) |
|
|
|
|
|
|
For more details, including ablation studies and theoretical proofs, please refer to our [Technical Report](https://arxiv.org/pdf/2512.05021).
|
|
|
|
|
|
## Performance
|
|
|
|
|
|
We evaluated HTR-ConvText across four diverse datasets. The model achieves new SOTA results with the lowest Character Error Rate (CER) and Word Error Rate (WER) without requiring massive synthetic pre-training.
|
|
|
|
|
|
| Dataset | Language | Ours CER (%) | HTR-VT | OrigamiNet | TrOCR | CRNN |
|
|
|
|-----------|-------------|--------------|--------|------------|-------|-------|
|
|
|
| IAM | English | 4.0 | 4.7 | 4.8 | 7.3 | 7.8 |
|
|
|
| LAM | Italian | 2.7 | 2.8 | 3.0 | 3.6 | 3.8 |
|
|
|
| READ2016 | German | 3.6 | 3.9 | - | - | 4.7 |
|
|
|
| VNOnDB | Vietnamese | 3.45 | 4.26 | 7.6 | - | 10.53 |
|
|
|
|
|
|
## Quickstart
|
|
|
|
|
|
### Instalation
|
|
|
|
|
|
1. **Clone the repository**
|
|
|
```cmd
|
|
|
git clone https://github.com/0xk0ry/HTR-ConvText.git
|
|
|
cd HTR-ConvText
|
|
|
```
|
|
|
2. **Create and activate a Python 3.9+ Conda environment**
|
|
|
```cmd
|
|
|
conda create -n htr-convtext python=3.9 -y
|
|
|
conda activate htr-convtext
|
|
|
```
|
|
|
3. **Install PyTorch** using the wheel that matches your CUDA driver (swap the index for CPU-only builds):
|
|
|
```cmd
|
|
|
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
|
|
|
```
|
|
|
4. **Install the remaining project requirements** (everything except PyTorch, which you already picked in step 3).
|
|
|
```cmd
|
|
|
pip install -r requirements.txt
|
|
|
```
|
|
|
|
|
|
The code was tested on Python 3.9 and PyTorch 2.9.1.
|
|
|
|
|
|
### Data Preparation
|
|
|
|
|
|
We provide split files (train.ln, val.ln, test.ln) for IAM, READ2016, LAM, and VNOnDB under data/. Organize your data as follows:
|
|
|
|
|
|
```
|
|
|
./data/iam/
|
|
|
├── train.ln
|
|
|
├── val.ln
|
|
|
├── test.ln
|
|
|
└── lines
|
|
|
├── a01-000u-00.png
|
|
|
├── a01-000u-00.txt
|
|
|
└── ...
|
|
|
```
|
|
|
|
|
|
### Training
|
|
|
|
|
|
We provide comprehensive scripts in the ./run/ directory. To train on the IAM dataset with the Textual Context Module (TCM) enabled:
|
|
|
|
|
|
```
|
|
|
# Using the provided script
|
|
|
bash run/iam.sh
|
|
|
|
|
|
# OR running directly via Python
|
|
|
python train.py \
|
|
|
--use-wandb \
|
|
|
--dataset iam \
|
|
|
--tcm-enable \
|
|
|
--exp-name "htr-convtext-iam" \
|
|
|
--img-size 512 64 \
|
|
|
--train-bs 32 \
|
|
|
--val-bs 8 \
|
|
|
--data-path /path/to/iam/lines/ \
|
|
|
--train-data-list data/iam/train.ln \
|
|
|
--val-data-list data/iam/val.ln \
|
|
|
--test-data-list data/iam/test.ln \
|
|
|
--nb-cls 80
|
|
|
```
|
|
|
|
|
|
### Inference / Evaluation
|
|
|
|
|
|
To evaluate a pre-trained checkpoint on the test set:
|
|
|
|
|
|
```
|
|
|
python test.py \
|
|
|
--resume ./checkpoints/best_CER.pth \
|
|
|
--dataset iam \
|
|
|
--img-size 512 64 \
|
|
|
--data-path /path/to/iam/lines/ \
|
|
|
--test-data-list data/iam/test.ln \
|
|
|
--nb-cls 80
|
|
|
```
|
|
|
|
|
|
## Citation
|
|
|
|
|
|
If you find our work helpful, please cite our paper:
|
|
|
|
|
|
```
|
|
|
@misc{truc2025htrconvtex,
|
|
|
title={HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition},
|
|
|
author={Pham Thach Thanh Truc and Dang Hoai Nam and Huynh Tong Dang Khoa and Vo Nguyen Le Duy},
|
|
|
year={2025},
|
|
|
eprint={2512.05021},
|
|
|
archivePrefix={arXiv},
|
|
|
primaryClass={cs.CV},
|
|
|
url={https://arxiv.org/abs/2512.05021},
|
|
|
}
|
|
|
```
|
|
|
|
|
|
## Acknowledgement
|
|
|
|
|
|
This project is inspired by and adapted from [HTR-VT](https://github.com/Intellindust-AI-Lab/HTR-VT). We gratefully acknowledge the authors for their open-source contributions. |
