Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ETL tooling specifically for writing clean files #3

Open
mehalter opened this issue Nov 6, 2024 · 0 comments
Open

Improve ETL tooling specifically for writing clean files #3

mehalter opened this issue Nov 6, 2024 · 0 comments

Comments

@mehalter
Copy link
Member

mehalter commented Nov 6, 2024

We should make sure the write_clean_file is a relatively sane and generalizable format. At the very least there are a few use cases we should keep in mind:

  1. Generating object key prefix/path to easily handle the required structure for Glue crawler/partitioning.
  2. Easily give the user the ability to just specify a new file extension without worrying about the actual raw filename.

These could be considered generally useful utilities. So they should go in the glue module if that makes sense or they could live in the more general util module or somewhere similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant