Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate PTransform recipes (<=0.10pfr style) #169

Open
moradology opened this issue Feb 12, 2024 · 0 comments
Open

Deprecate PTransform recipes (<=0.10pfr style) #169

moradology opened this issue Feb 12, 2024 · 0 comments

Comments

@moradology
Copy link
Contributor

moradology commented Feb 12, 2024

People trying to commit to this project and create recipes are very likely to look up expected behavior of pipeline transformations (PTransforms) and they are very likely to get some nasty surprises as things are currently set up. It is extremely frustrating that Beam's pipeline definition DSL is so heavily dependent upon mutation of hidden state, but that appears to be a fundamental limitation of the pipeline API and a likely source for future bugs.

LET IT BE KNOWN: if you're ever writing a DSL for DAG construction, there's no good reason to avoid immutability. This doesn't need to be high performance! (I'm looking at you, beam devs...)

Anyway, we should probably torch anything that's going to introduce bugs and violate expectations of the underlying Beam libraries. The strategy I've proposed can be found here #168 and a compliant recipe can be found here:

@moradology moradology changed the title Deprecate PTransform recipes (>=0.10pfr style) Deprecate PTransform recipes (<=0.10pfr style) Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant