Accelerate documentation

DDP 通信钩子

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

DDP 通信钩子

分布式数据并行(DDP)通信钩子提供了一个通用接口,用于通过覆盖DistributedDataParallel中的原生 allreduce 来控制梯度在各工作进程之间的通信方式。系统提供了若干内置的通信钩子,用户可以轻松应用其中任意一个来优化通信。

  • FP16 压缩钩子:通过将梯度转换为半精度浮点格式(torch.float16)来压缩梯度,从而降低通信开销。
  • BF16 压缩钩子:与 FP16 类似,但使用 Brain Floating Point 格式(torch.bfloat16),在某些硬件上可能更高效。
  • PowerSGD 钩子:一种先进的梯度压缩算法,可提供高压缩率,并加速受带宽限制的分布式训练。

在本教程中,您将了解如何快速设置 DDP 通信钩子,并使用 Accelerate 提供的工具执行训练,其过程可以简单到只需添加一行新代码!这展示了如何使用 DDP 通信钩子,借助 Accelerate 库优化分布式训练中的梯度通信。

FP16 压缩钩子

PyTorch
Accelerate
import torch
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed.algorithms.ddp_comm_hooks import default_hooks
from accelerate.test_utils.testing import get_backend

device_type, _, _ = get_backend()
device_id = getattr(torch, device_type, torch.cuda).current_device()

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        return self.layer(x)

model = MyModel()
model = DDP(model, device_ids=[device_id])
model.register_comm_hook(state=None, hook=default_hooks.fp16_compress_hook)

# Training loop
for data, targets in data_loader:
    outputs = model(data)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

BF16 压缩钩子

BF16 压缩 Hook API 目前处于实验阶段,并且要求 NCCL 版本高于 2.9.6。

PyTorch
Accelerate
import torch
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed.algorithms.ddp_comm_hooks import default_hooks
from accelerate.test_utils.testing import get_backend

device_type, _, _ = get_backend()
device_id = getattr(torch, device_type, torch.cuda).current_device()

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        return self.layer(x)

model = MyModel()
model = DDP(model, device_ids=[device_id])
model.register_comm_hook(state=None, hook=default_hooks.bf16_compress_hook)

# Training loop
for data, targets in data_loader:
    outputs = model(data)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

PowerSGD 钩子

PowerSGD 通常需要与模型梯度大小相同的额外内存来启用误差反馈,这可以补偿有偏压缩通信并提高精度。

PyTorch
Accelerate
import torch
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed.algorithms.ddp_comm_hooks import powerSGD_hook
from accelerate.test_utils.testing import get_backend

device_type, _, _ = get_backend()
device_id = getattr(torch, device_type, torch.cuda).current_device()

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        return self.layer(x)

model = MyModel()
model = DDP(model, device_ids=[device_id])
state = powerSGD_hook.PowerSGDState(process_group=None)
model.register_comm_hook(state=state, hook=powerSGD_hook.powerSGD_hook)

# Training loop
for data, targets in data_loader:
    outputs = model(data)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

DDP 通信钩子实用工具

通信钩子还有两个额外的实用工具,用于支持可选功能。

comm_wrapper

comm_wrapper 是一个用于为通信钩子添加额外功能的选项。例如,它可以用于将 FP16 压缩与其他通信策略结合使用。当前支持的包装器包括 no、fp16 和 bf16。

from accelerate import Accelerator, DDPCommunicationHookType, DistributedDataParallelKwargs
import torch

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        return self.layer(x)

# DDP Communication Hook setup
ddp_kwargs = DistributedDataParallelKwargs(
    comm_hook=DDPCommunicationHookType.POWER_SGD,
    comm_wrapper=DDPCommunicationHookType.FP16
)
accelerator = Accelerator(kwargs_handlers=[ddp_kwargs])

model = MyModel()
optimizer = torch.optim.Adam(model.parameters())
data_loader = DataLoader(dataset, batch_size=16)

model, optimizer, data_loader = accelerator.prepare(model, optimizer, data_loader)

# Training loop
for data, targets in data_loader:
    outputs = model(data)
    loss = criterion(outputs, targets)
    accelerator.backward(loss)
    optimizer.step()
    optimizer.zero_grad()

comm_state_option

comm_state_option 允许你传递某些通信钩子所需的额外状态信息。这对于有状态的钩子(例如 PowerSGD)特别有用,因为它们需要在训练步骤之间维护超参数和内部状态。下面是一个展示如何使用 comm_state_option 配合 PowerSGD 钩子的示例。

from accelerate import Accelerator, DDPCommunicationHookType, DistributedDataParallelKwargs
import torch

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        return self.layer(x)

# DDP Communication Hook setup
ddp_kwargs = DistributedDataParallelKwargs(
    comm_hook=DDPCommunicationHookType.POWER_SGD,
    comm_state_option={"matrix_approximation_rank": 2}
)
accelerator = Accelerator(kwargs_handlers=[ddp_kwargs])

model = MyModel()
optimizer = torch.optim.Adam(model.parameters())
data_loader = DataLoader(dataset, batch_size=16)

model, optimizer, data_loader = accelerator.prepare(model, optimizer, data_loader)

# Training loop
for data, targets in data_loader:
    outputs = model(data)
    loss = criterion(outputs, targets)
    accelerator.backward(loss)
    optimizer.step()
    optimizer.zero_grad()

如需了解更高级的用法和额外的钩子,请参阅 PyTorch DDP Communication Hooks 文档。

Update on GitHub