Skip to content

Commit

Permalink
Fix doc references
Browse files Browse the repository at this point in the history
  • Loading branch information
GeorgesLorre committed Mar 19, 2024
1 parent 35160eb commit ea61a82
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 30 deletions.
7 changes: 3 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,9 @@ At a high level, Fondant consists of three main parts:
component type.


* The `/pipeline` directory which contains the modules for implementing a Fondant pipeline.
* `pipeline.py`: Defines the `Pipeline` class which is used to define the pipeline graph and the
pipeline run. The
implemented class is then consumed by the compiler to compile to a specific pipeline runner.
* The `/dataset` directory which contains the modules for implementing a Fondant pipeline.
* `dataset.py`: Defines the `Dataset` class which is used to define the graph. The
implemented class is then consumed by the compiler to compile to a specific runner.
This module also implements the
`ComponentOp` class which is used to define the component operation in the pipeline graph.
* `compiler.py`: Defines the `Compiler` class which is used to define the compiler that
Expand Down
6 changes: 3 additions & 3 deletions docs/guides/build_a_simple_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ pipeline = Pipeline(

??? "View a detailed reference of the options accepted by the `Pipeline` class"

::: fondant.pipeline.Pipeline.__init__
::: fondant.dataset.Pipeline.__init__
handler: python
options:
show_source: false
Expand Down Expand Up @@ -103,7 +103,7 @@ We provide three arguments to the `.read()` method:

??? "View a detailed reference of the `Pipeline.read()` method"

::: fondant.pipeline.Pipeline.read
::: fondant.dataset.Pipeline.read
handler: python
options:
show_source: false
Expand Down Expand Up @@ -171,7 +171,7 @@ english_images = images.apply(

??? "View a detailed reference of the `Dataset.apply()` method"

::: fondant.pipeline.pipeline.Dataset.apply
::: fondant.dataset.Dataset.apply
handler: python
options:
show_source: false
Expand Down
38 changes: 15 additions & 23 deletions docs/pipeline.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# Pipeline
# Dataset

A Fondant pipeline is a tool for building complex workflows by creating a Directed Acyclic Graph
(DAG) of different components that need to be executed. With Fondant, you can use both reusable
components and custom components, and chain them into a pipeline.
A Fondant Dataset is a checkpoint in a Directed Acyclic Graph
(DAG) of one or more different components that need to be executed. With Fondant, you can use both reusable
components and custom components, and chain them together.

## Composing a pipeline

Start by creating a `pipeline.py` file and adding the following code.
[//]: # (TODO update this section once we have the workspace)
## Composing a Pipeline

Start by creating a `pipeline.py` file and adding the following code.
```python
from fondant.pipeline import Pipeline

Expand All @@ -30,34 +31,25 @@ The base path can be:
* **A local directory**: only valid for the local runner, points to a local directory. This is
useful for local development.

!!! note "IMPORTANT"

Make sure the provided base_path already exists.

??? "View a detailed reference of the options accepted by the `Pipeline` class"

::: fondant.pipeline.Pipeline.__init__
handler: python
options:
show_source: false

### Adding a load component

You can read data into your pipeline by using the `Pipeline.read()` method with a load component.
You can read data into your pipeline by using the `Dataset.read()` method with a load component.

```python
dataset = pipeline.read(
dataset = Dataset.read(
"load_from_parquet",
arguments={
"dataset_uri": "path/to/dataset",
"n_rows_to_load": 100,
},
)
```
[//]: # (TODO: Add example of init from manifest)

??? "View a detailed reference of the `Pipeline.read()` method"
??? "View a detailed reference of the `Dataset.read()` method"

::: fondant.pipeline.Pipeline.read
::: fondant.dataset.Pipeline.read
handler: python
options:
show_source: false
Expand All @@ -68,7 +60,7 @@ graph. It returns a lazy `Dataset` instance which you can use to chain transform
### Adding transform components

```python
from fondant.pipeline import Resources
from fondant.dataset import Resources

dataset = dataset.apply(
"embed_text",
Expand All @@ -90,7 +82,7 @@ can choose the type of GPU as well.

??? "View a detailed reference of the `Dataset.apply()` method"

::: fondant.pipeline.pipeline.Dataset.apply
::: fondant.dataset.Dataset.apply
handler: python
options:
show_source: false
Expand All @@ -112,7 +104,7 @@ dataset = dataset.write(

??? "View a detailed reference of the `Dataset.write()` method"

::: fondant.pipeline.pipeline.Dataset.write
::: fondant.dataset.Dataset.write
handler: python
options:
show_source: false
Expand Down

0 comments on commit ea61a82

Please sign in to comment.