Intel Gaudi

Intel Gaudi accelerators run PyTorch on the Intel Gaudi stack, with the device string hpu. Gaudi runs PyTorch in eager mode by default. Eager mode runs each operation as soon as it’s called, the same way PyTorch runs on a CPU or GPU, so Transformers models run on Gaudi without Gaudi-specific changes. To capture the model into a graph instead, compile it with torch.compile.

Installation

Install the Intel Gaudi software and its PyTorch package (habana-torch-plugin, imported as habana_frameworks) with the Intel Gaudi installation guide, plus Accelerate 1.5.0 or later.

Transformers uses int64 tensors for token IDs, so enable int64 support on Gaudi. Transformers’ Gaudi CI runs with the same setting.

export PT_ENABLE_INT64_SUPPORT=1

Check that Transformers can see the device.

from transformers import is_torch_hpu_available

print(is_torch_hpu_available())

It prints True when Gaudi is ready to use, and False with Accelerate older than 1.5.0, even if a Gaudi device is present.

Select devices

Choose which Gaudi devices a Docker container can use when you start it. Set HABANA_VISIBLE_MODULES to module IDs, which stay the same across reboots and driver reinstalls, or HABANA_VISIBLE_DEVICES to device indices, which can change. If you set neither, PyTorch uses any available Gaudi device on the server.

docker run --runtime=habana -e HABANA_VISIBLE_MODULES=0,1,2,3 ...

Run hl-smi -Q index,module_id -f csv to map device indices to module IDs. For a distributed workload on part of a server, Intel supports only 2-device and 4-device groups. See Multiple Dockers each with a single workload for the supported module ID combinations.

Run inference

Pass device_map="hpu" to from_pretrained(). pipeline() picks Gaudi automatically when you don’t pass device.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", device_map="hpu")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

inputs = tokenizer("Intel Gaudi is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quantize models

bitsandbytes quantizes models to 4 or 8 bits on Gaudi 2 and Gaudi 3. When you don’t pass device_map, it places the model on the current Gaudi device.

To fine-tune a model that doesn’t fit on one device, load it in 4-bit with bitsandbytes and train LoRA adapters with PEFT.

Train with Trainer

TrainingArguments detects Gaudi and sets the device to hpu, so Trainer trains on it without any device settings. All Gaudi generations support bf16.

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
)
print(training_args.device)

It prints an hpu device when Trainer will train on Gaudi.

Compile the model

Transformers wraps torch.compile on Gaudi, so any call with a backend other than hpu_backend or eager, including a @torch.compile decorator in modeling code, switches to hpu_backend and logs a warning. The wrapper applies to every torch.compile call, for inference or training.

To compile a model for training, set torch_compile=True in TrainingArguments, which sets torch_compile_backend to "hpu_backend".

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
    torch_compile=True,
)

Distributed training

Distributed training uses the HCCL collective backend. Transformers selects it on Gaudi, and you can also set it explicitly with ddp_backend="hccl".

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
    ddp_backend="hccl",
)

Launch one process per Gaudi device with torchrun or accelerate launch, after selecting devices. accelerate launch sets the number of processes to the number of Gaudi devices when you don’t pass --num_processes.

accelerate launch train.py

Next steps

• Read the Intel Gaudi documentation for hardware details, the software stack, and the Gaudi architecture.
• Try Optimum for Intel Gaudi for Gaudi-optimized model implementations for training and inference.