-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #246 from Modalities/readme_updates
chore: README updates
- Loading branch information
Showing
6 changed files
with
246 additions
and
49 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,34 +16,78 @@ | |
| </a> | ||
| </div> | ||
|
|
||
| Modalities is a PyTorch-native framework for distributed training of Large Language Models (LLMs) and Foundation Models (FMs) at scale. Given the complexity of distributed training and rapid advancements in the field, we aim to provide a flexible and easy-to-use framework that enables researchers and practitioners to train and evaluate LLMs and FMs efficiently. Modalities is built on top of PyTorch and leverages the latest advancements in distributed training, such as Fully Sharded Data Parallel (FSDP), mixed precision training, Flash Attention and many more, to achieve state-of-the-art performance and throughput. | ||
|
|
||
| We successfully scaled Modalities up to 2048 GPUs on two HPC centers, namely [Leonardo Booster](https://leonardo-supercomputer.cineca.eu/hpc-system/) and [MareNostrum 5](https://www.bsc.es/ca/marenostrum/marenostrum-5), featuring Nvidia A100 and H100 GPUs, respectively. The results of our scaling experiments can be found [here](#scaling-experiments). | ||
|
|
||
| Besides its scalabilty, Modalities allows to seamlessly integrate new components and features, such as custom attention mechanisms, loss functions, optimizers or models. We provide a series of tutorials to help you get started with training and evaluating models using Modalities. We achieve this level of extensibility by having clear interfaces for each component type (e.g., model, optimizer, etc.), that a component must implement to be registered within in Modalities at runtime. | ||
|
|
||
| ## Getting Started | ||
| For training and evaluation a model, feel free to checkout [this](https://github.com/Modalities/modalities/blob/main/examples/getting_started/README.md) getting started tutorial, in which we train a small, 60M-parameter GPT model on a tiny subset of the Redpajama V2 dataset. | ||
|
|
||
| ## Installation | ||
|
|
||
| Create conda environment and activate it via | ||
| ``` | ||
| There are two ways to install modalities. If you want to use the latest nightly version, or if you want to modify the code base itself, we recommend installing modalities directly from source. | ||
|
|
||
| If you want to use Modalities as a library and register your custom components with Modalities, you can install it directly via pip which provides you with the latest stable version. | ||
|
|
||
|
|
||
| ### Option 1: Installation from source | ||
|
|
||
|
|
||
| Create a conda environment and activate it via | ||
|
|
||
| ```sh | ||
| conda create -n modalities python=3.10 | ||
| conda activate modalities | ||
| ``` | ||
| Either clone the repository via | ||
| ```sh | ||
| git clone [email protected]:Modalities/modalities.git | ||
| ``` | ||
| or download the repository as a zip file and extract it. | ||
| ``` | ||
| wget https://github.com/Modalities/modalities/archive/refs/heads/main.zip | ||
| unzip main.zip | ||
| ``` | ||
|
|
||
| then, install the repository via | ||
|
|
||
| Currently, the flash attention dependency cannot be installed without torch being installed beforehand. | ||
| Until the flash attention developers fix this, we have to run | ||
|
|
||
| ```sh | ||
| pip install torch | ||
| ``` | ||
| beforehand. | ||
|
|
||
| Afterwards, Modalities can be installed via | ||
|
|
||
| ```sh | ||
| cd modalities | ||
| pip install -e . | ||
| ``` | ||
|
|
||
| If you want to contribute, have a look at `CONTRIBUTING.md`. | ||
| ### Option 2: Installation via pip | ||
|
|
||
| To install modalities via pip, run | ||
|
|
||
| ```sh | ||
| pip install torch | ||
| pip install modalities | ||
| ``` | ||
|
|
||
| Note, that also here, torch has to be installed before installing Modalities due to flash attention's dependency management. | ||
|
|
||
|
|
||
| ## Usage | ||
| For running the training endpoint on multiple GPUs run `CUDA_VISIBLE_DEVICES=2,3 torchrun --nnodes 1 --nproc_per_node 2 --rdzv-endpoint=0.0.0.0:29502 src/modalities/__main__.py run --config_file_path config_files/config.yaml`. | ||
| For running the training endpoint on multiple GPUs run | ||
| ```sh | ||
| CUDA_VISIBLE_DEVICES=2,3 torchrun --nnodes 1 --nproc_per_node 2 --rdzv-endpoint=0.0.0.0:29502 modalities run --config_file_path config_files/config.yaml | ||
| ``` | ||
|
|
||
| Or, if you are a VsCode user, add this to your `launch.json`: | ||
| In the example above, we use `torchrun` to run the training endpoint on two GPUs. The `--nnodes` argument specifies the number of nodes in the cluster, `--nproc_per_node` specifies the number of processes per node, and `--rdzv-endpoint` specifies the rendezvous endpoint. The `modalities run` command specifies the training endpoint, and `--config_file_path` specifies the path to the configuration file. The configuraton file contains the exhaustive parameterization for all the training components (e.g., dataset, model, optimize, etc.), making training fully reproducible. A full list of all the components already available in Modalities can be found [here](docs/components/components.md). | ||
|
|
||
| Or, if you are a VSCode user, add this to your `launch.json` | ||
| ```json | ||
|
|
||
| { | ||
|
|
@@ -70,6 +114,7 @@ Or, if you are a VsCode user, add this to your `launch.json`: | |
| "envFile": "${workspaceFolder}/.env" | ||
| } | ||
| ``` | ||
| which will allow you to run the training endpoint directly from VSCode and debug it. | ||
|
|
||
| ## Supported Features | ||
| In the following, we list the already implemented, planned and in-progress features w.r.t. to improving downstream performance, throughput, multi-modality, and alignment. | ||
|
|
@@ -85,9 +130,14 @@ In the following, we list the already implemented, planned and in-progress featu | |
| | Memmap for efficient data loading | supported | Optimizes the data pipeline to reduce I/O bottlenecks. | | ||
| | Activation Checkpointing | supported | Saves intermediate activations to memory only at certain points during the forward pass and recomputes them during the backward pass, reducing memory usage at the cost of additional computation. | | ||
| | Flash Attention | supported | A highly optimized attention mechanism that significantly reduces the computational burden and memory footprint of attention calculations, enabling faster training and inference on large models. | | ||
| | Tensor Parallelism | prototype | Implementing vertical model sharding, as an efficient model parallelism technique| | ||
| | Sequence Parallelism | prototype | Variant of Tensor Parallelism that shard on the sequence dimension | | ||
| | FSDP 2 | prototype | Improved version of the original FSDP | | ||
| | Torch Compile | prototype | Speeds up tensor operations by JIT compiling tensor operations into optimized kernels | | ||
| | Deferred Initialisation | prototype | Instead of instantiating the model in CPU RAM, the modules are instantiated as fake tensors and operations are recorded. Once sharded (e.g., via FSDP), each rank only instantiates the local tensors by replaying the tensor operations.| | ||
| | Adaptive Batch Size Exploration | planned | Dynamically increases the training batch size during the training process to identify the maximum batch size that can be accommodated by a given GPU setup without causing memory overflow or performance degradation. | | ||
| | Node Failure Recovery | planned | Implements mechanisms to automatically detect and recover from failures (e.g., node or GPU failures) in distributed training environments, ensuring that training can continue with minimal interruption even if one or more nodes / GPUs in the cluster fail. | | ||
|
|
||
| | Loss Parallelism | planned | Reduces memory footprint and communication overhead by computing the loss locally on each rank. | | ||
|
|
||
|
|
||
| ### Downstream Performance Features | ||
|
|
@@ -107,60 +157,83 @@ In the following, we list the already implemented, planned and in-progress featu | |
| | Knowledge Distillation | planned | Transfers knowledge from a larger, complex model to a smaller, more efficient model, improving the smaller model's performance without the computational cost of the larger model.| | ||
| | Hyperparameter Optimization | planned | Grid search for various hyperparameter such as LR, Optimizer arguments etc. Also the integration of µP might be interesting | | ||
|
|
||
| ## Tutorials | ||
| Even though Modalities significantly simplifies LLM training, there is still some technical complexity left. We provide a series of tutorials to help you get started with training and evaluating models using Modalities. | ||
|
|
||
| - [Getting Started](examples/getting_started/README.md)</br> | ||
| Brief overview on how to get started with Modalities by training a small GPT model on a tiny subset of the Redpajama V2 dataset. | ||
|
|
||
| - [Library Usage](examples/library_usage/README.md)</br> | ||
| How to use Modalities as a library and register custom components with Modalities. | ||
|
|
||
| - [Modalities in 15mins] </br> | ||
| Jupyter notebook will be added soon | ||
|
|
||
| ## Entry Points | ||
| Modalities provides several entry points to interact with the framework. The following section lists the available entry points and their respective functionalities. | ||
|
|
||
|
|
||
| ### Model training | ||
|
|
||
| We use [click](https://click.palletsprojects.com/en/) as a tool to add new entry points and their CLI arguments. | ||
| For this we have a main entry point from which all other entry points are started. | ||
| For model pretraining, we have to pass a configuration file that specifies the model architecture, optimizer, dataset, dataloader, and other training components. Additionally, we specify the number of nodes, the number of processes per node, and the rendezvous endpoint. | ||
|
|
||
| The main entry point is `src/modalities/__main__.py:main()`. | ||
| We register other sub-entrypoints by using our main `click.group`, called `main`, as follows: | ||
| ```python | ||
| @main.command(name="my_new_entry_point") | ||
| Example: | ||
| ```sh | ||
| CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --rdzv-endpoint localhost:29515 \ | ||
| --nnodes 1 \ | ||
| --nproc_per_node 4 \ | ||
| $(which modalities) run --config_file_path configs/pretraining_config.yaml | ||
| ``` | ||
|
|
||
| See the following full example: | ||
| ```python | ||
| import click | ||
| import click_pathlib | ||
| Explanation: | ||
|
|
||
| * `CUDA_VISIBLE_DEVICES=0,1,2,3`: This environment variable specifies which GPUs will be used for the job. In this case, GPUs with IDs 0, 1, 2, 3 are selected for training. | ||
|
|
||
| @click.group() | ||
| def main() -> None: | ||
| pass | ||
| * `torchrun`: This is a utility from PyTorch used to launch distributed training. It automatically manages multiple processes for distributed training. | ||
|
|
||
| * `--rdzv-endpoint localhost:29515`: Specifies the rendezvous endpoint. Here, localhost is the machine's address, and 29515 is the port. The rendezvous endpoint coordinates the processes involved in distributed training. | ||
|
|
||
| config_option = click.option( | ||
| "--config_file_path", | ||
| type=click_pathlib.Path(exists=False), | ||
| required=True, | ||
| help="Path to a file with the YAML config file.", | ||
| ) | ||
| * `--nnodes 1`: Specifies the number of nodes to be used in the distributed setup. Since this is a single-node setup, 1 is used. | ||
|
|
||
| * `--nproc_per_node 4`: This argument tells torchrun how many processes to launch on each node. In this case, 4 processes are launched per node, corresponding to the 4 GPUs (IDs 0, 1, 2, 3) specified by CUDA_VISIBLE_DEVICES. | ||
|
|
||
| @main.command(name="do_stuff") | ||
| @config_option | ||
| @click.option( | ||
| "--my_cli_argument", | ||
| type=int, | ||
| required=True, | ||
| help="New integer argument", | ||
| ) | ||
| def entry_point_do_stuff(config_file_path: Path, my_cli_argument: int): | ||
| print(f"Do stuff with {config_file_path} and {my_cli_argument}...) | ||
| ... | ||
| * `$(which modalities) run`: This part dynamically finds the path to the Modalities executable and runs it. The run command triggers the main process to start the training. | ||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
| * `--config_file_path configs/pretraining_config.yaml`: The --config_file_path argument provides the path to the configuration file for the training job. In this example, the configuration is provided in configs/pretraining_config.yaml, which includes settings like model architecture, optimizer, dataset, dataloader and other training components. An example config file can be found [here](examples/getting_started/example_config.yaml). | ||
|
|
||
| ### Raw Training Dataset Indexation | ||
|
|
||
| The goal of the indexation process is to determine the starting byte position and length of each document in the raw data file. Subsequently, the index file is used to efficiently access the raw data during tokenization. | ||
|
|
||
| Example: | ||
| ```sh | ||
| modalities data create_raw_index --index_path data/preprocessed/fineweb_edu_num_docs_483606.idx \ | ||
| data/raw/fineweb_edu_num_docs_483606.jsonl | ||
| ``` | ||
| With | ||
| ```toml | ||
| [project.scripts] | ||
| modalities = "modalities.__main__:main" | ||
|
|
||
| Explanation: | ||
|
|
||
| The `modalities data create_raw_index` command triggers the process of creating the index from the raw data. The `--index_path` argument specifies the location where the generated index file will be saved. In this example, the index will be stored at `data/preprocessed/fineweb_edu_num_docs_483606.idx`. The last part, i.e., `data/raw/fineweb_edu_num_docs_483606.jsonl` is the input file in JSONL (JSON Lines) format containing the raw data. The command will process this file to create the index. | ||
|
|
||
| ### Raw Training Dataset Tokenization | ||
|
|
||
| Tokenization is the process of converting raw text data into a sequence of tokens that can be used as input to the model. The tokenization requires a configuration file, fully describing the tokenization process, making it fully reproducible. An example tokenization config can be found [here](examples/getting_started/example_dataset_config_train.yaml). | ||
|
|
||
| Example: | ||
| ```sh | ||
| modalities data pack_encoded_data configs/tokenization_config.yaml | ||
| ``` | ||
| in our `pyproject.toml`, we can start only main with `modalities` (which does nothing), or a specific sub-entrypoint e.g. `modalities do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. | ||
|
|
||
| Alternatively, directly use `src/modalities/__main__.py do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. | ||
| ### Inference | ||
|
|
||
| For inference on a model checkpoint, we have to pass a configuration file that specifies the full inference setup. An example inference config can be found [here](examples/getting_started/example_text_generation_config.yaml). | ||
|
|
||
| Example: | ||
|
|
||
| ```sh | ||
| modalities generate_text --config_file_path example_text_generation_config.yaml | ||
|
|
||
| ``` | ||
|
|
||
| ## Scaling Experiments | ||
|
|
||
|
|
||
Oops, something went wrong.