Software:DeepSpeed

From HandWiki
Short description: Microsoft open source library
DeepSpeed
DeepSpeed logo.svg
Original author(s)Microsoft Research
Developer(s)Microsoft
Initial releaseMay 18, 2020; 4 years ago (2020-05-18)
Stable release
v0.5.10 / January 14, 2022; 2 years ago (2022-01-14)
Repositorygithub.com/microsoft/DeepSpeed
Written inPython, CUDA, C++
TypeSoftware library
LicenseMIT License
Websitedeepspeed.ai

DeepSpeed is an open source deep learning optimization library for PyTorch.[1] The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.[6]

See also

References

Further reading

External links