纸上得来终觉浅，绝知此事要躬行。--陆游
Shallow learning from books will not lead to a deep understanding; real knowledge comes from practical experience.
We use the python, pytorch, pytorch.distributed to implement the famous LLM-distributed-training-methods.
We will try our best to coding everything to implement the distribued model training in deep learning.
Althouth we use PyTorch, we will try to avoid using encapsulated advanced features as much as possible, such as automatic differentiation, optimizers, gradient updates, etc.