Skip to content

Commit

Permalink
[dagster-dbt] refactor fetch-row-count user-facing API (#21846)
Browse files Browse the repository at this point in the history
## Summary

Refactors the user-facing api to tell dbt to fetch row counts for users.

Makes `dbt.cli(...).stream()` return a `DbtEventIterator`, a thin
wrapper around `collections.abc.Iterator` which holds some custom
methods, in this case `fetch_row_counts`:

```python
# old 
yield from dbt.cli(["build"], context=context).enable_fetch_row_count().stream()

# new 
yield from dbt.cli(["build"], context=context).stream().fetch_row_counts()
```

The impl remains identical, but eventually we could move to a more
generic impl under the hood that lets users apply their own
transformations after materializations complete.

```python
yield from dbt.cli(["build"], context=context).stream().fetch_row_counts().map(lambda event: ...)
```

## Test Plan

Update unit test.
  • Loading branch information
benpankow authored May 15, 2024
1 parent 2489802 commit 94b8ff3
Show file tree
Hide file tree
Showing 2 changed files with 223 additions and 157 deletions.
Loading

0 comments on commit 94b8ff3

Please sign in to comment.