Loss Scaling Free - ^new^

BF16 has the , so gradients rarely underflow — even without loss scaling. The tradeoff: less precision (7 vs 10 mantissa bits), but for most deep learning tasks, BF16’s precision is sufficient.

High-precision tasks, such as training Large Language Models (LLMs), often suffer from "spiky" loss curves. Scaling-free formats like BF16 are naturally more robust against these instabilities.

For years, the solution to this instability was . If you have ever trained a model in FP16, you’ve likely tweaked a "loss scaling factor," agonizing over whether to set it to static values or let the optimizer dynamically adjust it. loss scaling free

# Define the model model = nn.Sequential([...])

# PyTorch example with torch.autocast(device_type='cuda', dtype=torch.bfloat16): loss = model(input) loss.backward() # No loss scaling needed optimizer.step() BF16 has the , so gradients rarely underflow

# Apply static loss scaling scaled_loss = loss * 1.0

Dynamic loss scaling (automatic adjustment) solved some of this, but it added computational overhead and tuning complexity. Scaling-free formats like BF16 are naturally more robust

❌ :

Get more from Pinnacle with powerful add-ons