Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2266] [Feature] Make MAXIMUM_SEED_SIZE configurable #7117

Open
3 tasks done
acurtis-evi opened this issue Mar 3, 2023 · 2 comments · May be fixed by #7125 or #11177
Open
3 tasks done

[CT-2266] [Feature] Make MAXIMUM_SEED_SIZE configurable #7117

acurtis-evi opened this issue Mar 3, 2023 · 2 comments · May be fixed by #7125 or #11177
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors!

Comments

@acurtis-evi
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

There is a need to allow the 1MB seed limit to be relaxed

Slack thread for context
https://getdbt.slack.com/archives/C50NEBJGG/p1677856760823939

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

@acurtis-evi acurtis-evi added enhancement New feature or request triage labels Mar 3, 2023
@github-actions github-actions bot changed the title [Feature] Make MAXIMUM_SEED_SIZE configurable [CT-2266] [Feature] Make MAXIMUM_SEED_SIZE configurable Mar 3, 2023
@dbeatty10
Copy link
Contributor

Thanks for opening this issue and the associated PR @acurtis-evi !

Related to #6875

Current behavior

From the caveats to state comparison for seeds:

dbt stores a file hash of seed files that are <1 MB in size. If the contents of these seeds is modified, the seed will be included in state:modified.

Proposed behavior

Lightly edited from here:

  • add a DBT_MAXIMUM_SEED_SIZE env variable (expressed in MiB)
  • default to 1
  • setting DBT_MAXIMUM_SEED_SIZE to 0 would remove the limit entirely
    • i.e., if the derived constant MAXIMUM_SEED_SIZE is 0, then there is no limit to the seed size

Functional approval

@jtcohen6 indicated the following in the discussion in Slack:

Ok! I'm not strictly opposed to either/both of:

  • making this limit configurable (you accept slowness as trade-off)
  • updating the logic to better handle larger files — if one of you would be interested in contributing that (not something we'd be prioritizing soon ourselves)

@noppaz
Copy link

noppaz commented Mar 7, 2023

This aims to be solved in #7125.
There is a documentation update issue for it here dbt-labs/docs.getdbt.com#2958

@jtcohen6 jtcohen6 linked a pull request Dec 24, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors!
Projects
None yet
4 participants