Instructions to use zai-org/GLM-OCR with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-OCR with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zai-org/GLM-OCR")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForImageTextToText

tokenizer = AutoTokenizer.from_pretrained("zai-org/GLM-OCR")
model = AutoModelForImageTextToText.from_pretrained("zai-org/GLM-OCR")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use zai-org/GLM-OCR with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-OCR"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-OCR",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-OCR

SGLang

How to use zai-org/GLM-OCR with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-OCR" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-OCR",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-OCR" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-OCR",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-OCR with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-OCR
```

Performance Issue (Transformers)

#36

by THEMi11 - opened Feb 13

Discussion

THEMi11

Feb 13

Running this model with transformers as in example is extremely slow. ~30 seconds per image. Batching does not help. Running on h100.

useserreses

Feb 15

•

edited Feb 15

I encountered the same issue. In my case it was Patch convolution that took minutes to perform. I looked inside of the code and found out that each of the ~22k patches produces exactly one output voxel (1024, 1, 1, 1). So, basically, the convolution easily may be replaced with one matrix multiplication but I think cudnn spends a lot of time dispatching thousands of kernels.

Some monkey patching like this should work:

patch_embed = base_model.visual.patch_embed
proj = patch_embed.proj  # Conv3d(3, 1024, (2,14,14), stride=(2,14,14))

in_features = (
    patch_embed.in_channels
    * patch_embed.temporal_patch_size
    * patch_embed.patch_size ** 2
)  # 3 * 2 * 14 * 14 = 1176
embed_dim = patch_embed.embed_dim  # 1024
weight = proj.weight  # (1024, 3, 2, 14, 14)
bias = proj.bias      # (1024,)

def _fast_forward(hidden_states: torch.Tensor) -> torch.Tensor:
    target_dtype = weight.dtype
    hidden_states = hidden_states.reshape(-1, in_features).to(dtype=target_dtype)
    return F.linear(hidden_states, weight.reshape(embed_dim, -1), bias)

patch_embed.forward = _fast_forward

MarioWork

Feb 19

Hi,
I have the same issue. I needed 2 hours for 35 images on an A100

iyuge2

Z.ai org Feb 25

For deployment, we recommend using vLLM or SGLang, which offer better throughput and efficiency for inference in production environments.

If you just want to run quick experiments or integrate the model with minimal setup, you can use our SDK directly.

The SDK usage guide is available here: https://github.com/zai-org/GLM-OCR

iyuge2 changed discussion status to closed Feb 25

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment