Make MAXIMUM_SEED_SIZE_MIB configurable #7125

noppaz · 2023-03-05T09:53:19Z

resolves #7117
resolves #7124

Description

Increasing the maximum size of seed files where the content is hashed for state comparison will enable a greater use of deferred runs with updated seed file contents. By making this a configuration with an environment variable users are able to override the default 1 MiB limit.

Furthermore, to support reading larger files in memory constrained environments, a new method is added to read and compute file contents incrementally. As this is performed on bytes for better performance we continue using the previous UTF-8 content method for small files to not mess up current states where seed files are not stored with UTF-8.

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

cla-bot · 2023-03-05T09:53:24Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @noppaz

acurtis-evi · 2023-03-06T14:02:04Z

I added a PR against your PR for changes that I'd recommend. Added change for compatibility and merging our changes.

* Make MAXIMUM_SEED_SIZE configurable * Updated MAXIMUM_SEED_SIZE_NAME * Added changie * Update core/dbt/constants.py Co-authored-by: Doug Beatty <[email protected]> * Adding suggested change * Fixed comparison * Added comment * Added constants change and removed changie --------- Co-authored-by: Doug Beatty <[email protected]>

acurtis-evi · 2023-03-07T15:24:47Z

This looks good to me!

jtcohen6

Thanks for the PR @noppaz! A few comments after a first read through the code.

jtcohen6 · 2023-03-22T12:43:26Z

core/dbt/constants.py

-MAXIMUM_SEED_SIZE_NAME = "1MB"
+
+def get_max_seed_size():
+    mx = os.getenv("DBT_MAXIMUM_SEED_SIZE", "1")


I'd prefer that this config's name include the unit: DBT_MAXIMUM_SEED_SIZE_MIB

Rather than accessing an env var directly here, this should really be implemented as a "global config" (available in our Flags module). I don't think it would make very much sense to pass as a CLI flag (--maximum-seed-size), but I could very much see someone wanting to set this in "user config":

# profiles.yml -- for now config: maximum_seed_size_mb: 5

That means:

Implement it as a param (= CLI flag + env var)

Add it to UserConfig

I have addressed this in fa836bb. It required some additional moving of the usage of the global flag as it is not a constant anymore and the constants.py file is referenced much earlier in the import cycle.

jtcohen6 · 2023-03-22T12:47:14Z

core/dbt/constants.py

+DEFAULT_MAXIMUM_SEED_SIZE = 1 * 1024 * 1024
+MAXIMUM_SEED_SIZE = get_max_seed_size() * DEFAULT_MAXIMUM_SEED_SIZE


I don't think we need both of these constants defined. MAXIMUM_SEED_SIZE should just be get_max_seed_size() * 1024 * 1024. The get_max_seed_size() function will return the default value of 1 if the user does not set/override.

Thank you for checking @jtcohen6!

One reason for keeping DEFAULT_MAXIMUM_SEED_SIZE would be to be able to keep the logic of load_seed_source_file() in read_files.py. Since we do not convert the file from bytes in the new incremental logic we opted to continue computing the checksum with the "old" method for files smaller than 1 MiB so that we'd ensure backward compatibility with current manifests. This is since the checksum of a file encoded as anything other than UTF-8 will be different depending on if it gets UTF-8 encoded or is read as raw bytes.

The reason for not converting from bytes to UTF-8 string is performance and somewhat cleaner code but if we want to remove DEFAULT_MAXIMUM_SEED_SIZE maybe we should remove the elif and only compute the seed file hash incrementally instead. I see two options then:

Read the file as string and ensure UTF-8 encoding before hashing

Continue reading the raw bytes and accept that some projects with seed files that are not utf-8 might have a diff in their manifest deferral when upgrading to this version of dbt-core.

What's your thoughts on this?

I've implemented this suggestion with removal of DEFAULT_MAXIMUM_SEED_SIZE and went with alternative 1 above. The changes can be seen in 53021d9.

noppaz · 2023-05-03T06:11:43Z

Hey @jtcohen6, I was hoping we'd manage to find some more of your time after 1.5 release (which is looking extremely good btw). Would appreciate some more looks at this. I can't manage to tag @dbt-labs/oss-maintainers mentioned here, therefore pinging you directly.

dbeatty10 · 2023-05-17T15:42:15Z

@noppaz sorry for our silence here!

@jtcohen6 has limited availability currently, so I'm going to give a read through the discussion so far and try out this branch.

dbeatty10 · 2023-05-17T16:31:37Z

Will close and re-open to try to trigger the CI checks.

noppaz · 2023-05-19T18:40:39Z

@dbeatty10 thanks for the tests, I've made adjustments and I'm able to get them passing locally now so I think another round of tests and eyes on the code would be the next step.

noppaz · 2023-05-23T14:00:32Z

I see that the tests are failing on Windows only now. I think the issue is related to this comment.

However, this shines some light on what is kind of weird to me; the expected checksum is different on Windows and Linux/MacOS as can be seen in the tests from c2e2958. Should we not generate the same checksum so manifests can be correctly compared across platforms?

JavierLopezT · 2024-03-12T11:22:37Z

@noppaz @dbeatty10 Currently our CI/CD doesn't work because we have one seed bigger than 1MiB, so we would need this.

Is there any intention of finishing it up? Thank you very much!

noppaz · 2024-03-14T10:48:17Z

@noppaz @dbeatty10 Currently our CI/CD doesn't work because we have one seed bigger than 1MiB, so we would need this.

Is there any intention of finishing it up? Thank you very much!

Hey @JavierLopezT, with no intention of sounding bitter, I gave up on this work due to low interest from dbt Labs. Would be happy to finish it up if they'd be up for it. The problem remains for myself as well.

Co-authored by: Noah Holm <[email protected]> Co-authored by: Jeremy Cohen <[email protected]>

jtcohen6 · 2024-12-24T16:27:53Z

@noppaz We've heard some renewed interest in making seed size configurable (for the purposes of state:modified comparison), so I ported the changes from this PR into a new one:

Make MAXIMUM_SEED_SIZE_MIB configurable #11177

noppaz requested review from a team as code owners March 5, 2023 09:53

noppaz requested review from peterallenwebb and aranke March 5, 2023 09:53

compute seed file hash incrementally

880b035

noppaz force-pushed the incremental-seed-hash branch from da6c118 to 880b035 Compare March 6, 2023 08:33

cla-bot bot added the cla:yes label Mar 6, 2023

acurtis-evi mentioned this pull request Mar 6, 2023

Make MAXIMUM_SEED_SIZE configurable #7116

Closed

6 tasks

acurtis-evi and others added 2 commits March 7, 2023 14:44

update changelog

bfcabbf

noppaz changed the title ~~compute seed file hash incrementally~~ Make MAXIMUM_SEED_SIZE configurable Mar 7, 2023

add 0 option for no size limit

866191a

This was referenced Mar 7, 2023

Configurable MAXIMUM_SEED_SIZE for state hash comparison dbt-labs/docs.getdbt.com#2958

Closed

[CT-2271] [Feature] Compute seed file hashes incrementally #7124

Open

[CT-2266] [Feature] Make MAXIMUM_SEED_SIZE configurable #7117

Open

dbeatty10 added the ready_for_review Externally contributed PR has functional approval, ready for code review from Core engineering label Mar 7, 2023

Merge branch 'main' into incremental-seed-hash

a1afca5

acurtis-evi mentioned this pull request Mar 8, 2023

1.4.next acurtis-evi/dbt-core#5

Draft

6 tasks

jtcohen6 added the Team:Language label Mar 22, 2023

jtcohen6 reviewed Mar 22, 2023

View reviewed changes

noppaz added 2 commits April 10, 2023 15:52

add maximum_seed_size_mib to global config

fa836bb

remove DEFAULT_MAXIMUM_SEED_SIZE and always use incremental method

53021d9

noppaz requested a review from a team April 10, 2023 14:05

noppaz added 2 commits April 10, 2023 16:16

Merge branch 'main' into incremental-seed-hash

3b9b8cf

Merge branch 'main' into incremental-seed-hash

c2e2958

dbeatty10 closed this May 17, 2023

dbeatty10 reopened this May 17, 2023

dbeatty10 requested a review from a team as a code owner May 17, 2023 16:31

dbeatty10 requested review from emmyoop and removed request for a team May 17, 2023 16:31

strip start and end of contents

eb1ba52

noppaz changed the title ~~Make MAXIMUM_SEED_SIZE configurable~~ Make MAXIMUM_SEED_SIZE_MIB configurable May 19, 2023

jtcohen6 removed the Team:Language label Jul 19, 2023

emmyoop removed their request for review October 5, 2023 13:25

dbeatty10 added the community This PR is from a community member label Apr 3, 2024

graciegoheen assigned jtcohen6 Dec 19, 2024

jtcohen6 added a commit that referenced this pull request Dec 24, 2024

Reapply changes from #7125

d9db592

Co-authored by: Noah Holm <[email protected]> Co-authored by: Jeremy Cohen <[email protected]>

jtcohen6 pushed a commit that referenced this pull request Dec 24, 2024

Reapply changes from #7125

704120c

Co-authored by: Noah Holm <[email protected]> Co-authored by: Jeremy Cohen <[email protected]>

jtcohen6 mentioned this pull request Dec 24, 2024

Make MAXIMUM_SEED_SIZE_MIB configurable #11177

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make MAXIMUM_SEED_SIZE_MIB configurable #7125

Make MAXIMUM_SEED_SIZE_MIB configurable #7125

noppaz commented Mar 5, 2023 •

edited by dbeatty10

Loading

cla-bot bot commented Mar 5, 2023

acurtis-evi commented Mar 6, 2023 •

edited

Loading

acurtis-evi commented Mar 7, 2023

jtcohen6 left a comment

jtcohen6 Mar 22, 2023 •

edited

Loading

noppaz Apr 10, 2023

jtcohen6 Mar 22, 2023

noppaz Mar 22, 2023

noppaz Apr 10, 2023

noppaz commented May 3, 2023

dbeatty10 commented May 17, 2023

dbeatty10 commented May 17, 2023

noppaz commented May 19, 2023

noppaz commented May 23, 2023

JavierLopezT commented Mar 12, 2024 •

edited

Loading

noppaz commented Mar 14, 2024 •

edited

Loading

jtcohen6 commented Dec 24, 2024

		DEFAULT_MAXIMUM_SEED_SIZE = 1 * 1024 * 1024
		MAXIMUM_SEED_SIZE = get_max_seed_size() * DEFAULT_MAXIMUM_SEED_SIZE

Make MAXIMUM_SEED_SIZE_MIB configurable #7125

Are you sure you want to change the base?

Make MAXIMUM_SEED_SIZE_MIB configurable #7125

Conversation

noppaz commented Mar 5, 2023 • edited by dbeatty10 Loading

Description

Checklist

cla-bot bot commented Mar 5, 2023

acurtis-evi commented Mar 6, 2023 • edited Loading

acurtis-evi commented Mar 7, 2023

jtcohen6 left a comment

Choose a reason for hiding this comment

jtcohen6 Mar 22, 2023 • edited Loading

Choose a reason for hiding this comment

noppaz Apr 10, 2023

Choose a reason for hiding this comment

jtcohen6 Mar 22, 2023

Choose a reason for hiding this comment

noppaz Mar 22, 2023

Choose a reason for hiding this comment

noppaz Apr 10, 2023

Choose a reason for hiding this comment

noppaz commented May 3, 2023

dbeatty10 commented May 17, 2023

dbeatty10 commented May 17, 2023

noppaz commented May 19, 2023

noppaz commented May 23, 2023

JavierLopezT commented Mar 12, 2024 • edited Loading

noppaz commented Mar 14, 2024 • edited Loading

jtcohen6 commented Dec 24, 2024

noppaz commented Mar 5, 2023 •

edited by dbeatty10

Loading

acurtis-evi commented Mar 6, 2023 •

edited

Loading

jtcohen6 Mar 22, 2023 •

edited

Loading

JavierLopezT commented Mar 12, 2024 •

edited

Loading

noppaz commented Mar 14, 2024 •

edited

Loading