Skip to content

Commit

Permalink
Merge branch 'allegroai:main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
pollfly authored Sep 15, 2024
2 parents a3ffe57 + 00c903b commit 56dda31
Show file tree
Hide file tree
Showing 6 changed files with 37 additions and 28 deletions.
15 changes: 10 additions & 5 deletions docs/clearml_agent/clearml_agent_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,28 +39,33 @@ clearml-agent daemon <arguments> --stop

### Allocating Resources
To specify GPUs associated with the agent, add the `--gpus` flag.

:::info Docker Mode
Make sure to include the `--docker` flag, as GPU management through the agent is only supported in [Docker Mode](clearml_agent_execution_env.md#docker-mode).
:::

To execute multiple agents on the same machine (usually assigning GPU for the different agents), run:
```bash
clearml-agent daemon --detached --queue default --gpus 0
clearml-agent daemon --detached --queue default --gpus 1
clearml-agent daemon --gpus 0 --queue default --docker
clearml-agent daemon --gpus 1 --queue default --docker
```
To allocate more than one GPU, provide a list of allocated GPUs
```bash
clearml-agent daemon --gpus 0,1 --queue dual_gpu
clearml-agent daemon --gpus 0,1 --queue dual_gpu --docker
```

### Queue Prioritization
A single agent can listen to multiple queues. The priority is set by their order.

```bash
clearml-agent daemon --detached --queue high_q low_q --gpus 0
clearml-agent daemon --queue high_q low_q
```
This ensures the agent first tries to pull a Task from the `high_q` queue, and only if it is empty, the agent will try to pull
from the `low_q` queue.

To make sure an agent pulls from all queues equally, add the `--order-fairness` flag.
```bash
clearml-agent daemon --detached --queue group_a group_b --order-fairness --gpus 0
clearml-agent daemon --queue group_a group_b --order-fairness
```
It will make sure the agent will pull from the `group_a` queue, then from `group_b`, then back to `group_a`, etc. This ensures
that `group_a` or `group_b` will not be able to starve one another of resources.
Expand Down
11 changes: 6 additions & 5 deletions docs/getting_started/main.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@ title: What is ClearML?
slug: /
---

ClearML is an open source platform that automates and simplifies developing and managing machine learning solutions
for thousands of data science teams all over the world.
It is designed as an end-to-end MLOps and LLMOps suite allowing you to focus on developing your ML code and automation,
while ClearML ensures your work is reproducible and scalable.
ClearML is an open-source, end-to-end AI Platform designed to streamline AI adoption and the entire development lifecycle.
It supports every phase of AI development, from research to production, allowing users to
leverage any model, dataset, or architecture at scale. ClearML integrates seamlessly with existing tools,
frameworks, and infrastructures, offering unmatched flexibility and control for AI builders and DevOps teams building,
training, and deploying models at every scale on any AI infrastructure.


## Getting Started
Expand Down Expand Up @@ -95,7 +96,7 @@ Want a more in depth introduction to ClearML? Choose where you want to get start
<img src="/docs/latest/icons/ico-devops-engineer.svg" alt="DevOps Engineer logo" />
</i>
<h4>DevOps Engineers</h4>
<p>Learn learn how to deploy and configure a ClearML Server</p>
<p>Learn how to deploy and configure a ClearML Server</p>
<span class="btn-link">
<a href="./deploying_clearml/clearml_server#deployment">START HERE</a>
</span>
Expand Down
17 changes: 10 additions & 7 deletions docs/getting_started/mlops/mlops_first_steps.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,18 +53,21 @@ required python packages, and execute and monitor the process.
(or even multiple queues), but only a single agent will pull a Task to be executed.

:::tip Agent Deployment Modes
ClearML Agents can be deployed in Virtual Environment Mode or Docker Mode. In [virtual environment mode](../../clearml_agent/clearml_agent_execution_env.md),
the agent creates a new venv to execute an experiment. In [Docker mode](../../clearml_agent/clearml_agent_execution_env.md#docker-mode),
the agent executes an experiment inside a Docker container. For more information, see [Running Modes](../../fundamentals/agents_and_queues.md#running-modes).
ClearML Agents can be deployed in:
* [Virtual environment mode](../../clearml_agent/clearml_agent_execution_env.md): Agent creates a new venv to execute an experiment.
* [Docker mode](../../clearml_agent/clearml_agent_execution_env.md#docker-mode): Agent executes an experiment inside a
Docker container.

For more information, see [Running Modes](../../fundamentals/agents_and_queues.md#running-modes).
:::

## Clone an Experiment
Experiments already in the system can be reproduced for validation, or used as a baseline for further experimentation.
Experiments can be reproduced (cloned) for validation or as a baseline for further experimentation.
Cloning a task duplicates the task's configuration, but not its outputs.
**To clone an experiment in the ClearML WebApp:**
1. Click on any project card to open its [experiments table](../../webapp/webapp_exp_table.md)
1. Right-click one of the experiments on the table
1. Click on any project card to open its [experiments table](../../webapp/webapp_exp_table.md).
1. Right-click one of the experiments on the table .
1. Click **Clone** in the context menu, which will open a **CLONE EXPERIMENT** window.
1. Click **CLONE** in the window.
Expand Down Expand Up @@ -169,7 +172,7 @@ Logging models into the model repository is the easiest way to integrate the dev
Any model stored by a supported framework (Keras / TensorFlow / PyTorch / Joblib etc.) will be automatically logged into ClearML.
ClearML also supports methods to explicitly log models. Models can be automatically stored on a preferred storage medium
(s3 bucket, google storage, etc.).
(S3 bucket, Google storage, etc.).
#### Log Metrics
Log as many metrics as you want from your processes using the [Logger](../../fundamentals/logger.md) module. This
Expand Down
2 changes: 1 addition & 1 deletion docs/getting_started/mlops/mlops_second_steps.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ dataset.tags = []
new_dataset.tags = ['latest']
```

The new dataset inherits the contents of the datasets specified in `Dataset.create`'s `parents` argument.
The new dataset inherits the contents of the datasets specified in `Dataset.create`'s `parent_datasets` argument.
This not only helps trace back dataset changes with full genealogy, but also makes the storage more efficient,
since it only stores the changed and/or added files from the parent versions.
When you access the Dataset, it automatically merges the files from all parent versions
Expand Down
18 changes: 9 additions & 9 deletions docs/guides/reporting/explicit_reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,17 +85,17 @@ For example, if the Task ID is `9ed78536b91a44fbb3cc7a006128c1b0`, then the dire
In addition to ClearML automagical logging, the `clearml` Python
package contains methods for explicit reporting of plots, log text, media, and tables. These methods include:

* [Logger.report_histogram](../../references/sdk/logger.md#report_histogram)
* [Logger.report_confusion_matrix](../../references/sdk/logger.md#report_confusion_matrix)
* [Logger.report_scatter2d](../../references/sdk/logger.md#report_scatter2d)
* [Logger.report_scatter3d](../../references/sdk/logger.md#report_scatter3d)
* [Logger.report_surface](../../references/sdk/logger.md#report_surface)
* [`Logger.report_histogram`](../../references/sdk/logger.md#report_histogram)
* [`Logger.report_confusion_matrix`](../../references/sdk/logger.md#report_confusion_matrix)
* [`Logger.report_scatter2d`](../../references/sdk/logger.md#report_scatter2d)
* [`Logger.report_scatter3d`](../../references/sdk/logger.md#report_scatter3d)
* [`Logger.report_surface`](../../references/sdk/logger.md#report_surface)
(surface diagrams)
* [Logger.report_image](../../references/sdk/logger.md#report_image) - Report an image and upload its contents.
* [Logger.report_table](../../references/sdk/logger.md#report_table) - Report a table as a Pandas DataFrame, CSV file,
* [`Logger.report_image`](../../references/sdk/logger.md#report_image) - Report an image and upload its contents.
* [`Logger.report_table`](../../references/sdk/logger.md#report_table) - Report a table as a Pandas DataFrame, CSV file,
or URL for a CSV file.
* [Logger.report_media](../../references/sdk/logger.md#report_media) - Report media including images, audio, and video.
* [Logger.get_default_upload_destination](../../references/sdk/logger.md#get_default_upload_destination) - Retrieve the destination that is set for uploaded media.
* [`Logger.report_media`](../../references/sdk/logger.md#report_media) - Report media including images, audio, and video.
* [`Logger.get_default_upload_destination`](../../references/sdk/logger.md#get_default_upload_destination) - Retrieve the destination that is set for uploaded media.

### Get a Logger

Expand Down
2 changes: 1 addition & 1 deletion docs/release_notes/ver_1_7.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,5 +129,5 @@ an empty string
* Fix `report_histogram` - does not show "horizontal" orientation ([ClearML GitHub issue 699](https://github.com/allegroai/clearml/issues/699))
* Fix table reporting - when using `logger.report_table(title, series, iteration, etc)`, the `series` arg does not appear in UI ([ClearML GitHub issue 684](https://github.com/allegroai/clearml/issues/684))
* Fix artifacts (and models) use task's original name and not new name
* Fix very long filenames from s3 can't be downloaded (with `get_local_copy()`)
* Fix very long filenames from S3 can't be downloaded (with `get_local_copy()`)
* Fix overwrite of existing output models on pipeline task with `monitor_models` ([ClearML GitHub issue #757](https://github.com/allegroai/clearml/issues/757))

0 comments on commit 56dda31

Please sign in to comment.