poc: support virtual embeds

[v0idpwn]

The question is if we should. WDYT?
Wrote this while answering a question on slack.

[josevalim]

How would a virtual embed be loaded/dumped? What is the use case? :)

[v0idpwn]

This is the question/use case. My suggestion was using :any + a struct and taking care of typing on your side, but I can see why one would expect support for virtual embeds and decided to check if we could support it easily.

[josevalim]

Right. but if we are going to include it in the docs, we need to explain when this would useful. Why do you have a virtual field in this style? What is the use case? When would we recommend it?

[SukhikhN]

Virtual embeds could be useful when an Ecto schema is used primarily as application's internal data structure and in the second place for storing it somewhere.

For example, I have a project where I use a schema with several embedded schemas for representing a game state that lives in a GenServer. Ecto changesets are very useful for mutating this state in response to player's actions.

Only part of the state should be stored (in ETS or database) to be restored when the GenServer (re)starts. So I made some fields virtual in order to skip them when dumping the state to a storage. But I'm forced to declare embeds as virtual maps or map arrays and write special functions for casting them and tracking their changes.

Another example: I cast an external API JSON data into a schema in order to make convenient transformations into internal data format and reject invalid data. Nested JSON objects are casted into embeds. This API uses almost same data structure for requests and responses, so when making a request to the API, I reuse the schema and dump it to JSON. The request structure has less fields than response, so again some fields of the schema are virtual and therefore are not dumped when I make a request. I can't use this approach if one of response embeds should not be send when using the schema for a request.

[SukhikhN]

Of course, this problems could be solved in other ways, but making embeds virtual seems more natural to me in this cases.

After all, if we can mark any embedded schema field as virtual so it would not be dumped/loaded, why should embeds be an exception?

[josevalim]

Thanks for the clarification. I would probably model your case with different schemas but it doesn't hurt supporting this for consistency, as long as the implementation is straight-forward.

@v0idpwn do you see this growing much in complexity?

[v0idpwn]

I don't think it will grow in complexity at all, will add some tests.

[pmargreff]

We have a project that would benefit from using embeds_many ... virtual: true. Our use case is the following:

We have a complex embedded field that we want to display in the UI more using forms directive. Let's say we have an embed for IP, called ip we achieve to render it to the users or APIs by marking the field as ip_formatted, :string, virtual: true, parsing in from and to this virtual field in between fetching from database and rendering it. So, we have both a detailed embed and a formatted version for easy display.

The challenge arises with more complex scenarios, like converting a list of embeds into a simpler list of embeds that are compatible with form validations and other features, such as sorting (we are not using liveview, and unfortunately, it isn't an option in this case). Our current workaround has been to use load_in_query: false, which functions similarly to virtual: true and is also supported in embeds_many.

It works, but it is not ideal (and I imagine not intended to be used in that way either). It always goes with this comment # load_in_query is to emulate virtual which is not compatible with embeds_many. That said, we would benefit from this work as well.

cc: @Lgdev07

[josevalim]

Sorry, but that is still unclear then, if you never write it down, what is the benefit of being an embedded? It could as well be field :children, :any, virtual: true. If you are not using changesets or queries, which is often the case with virtual fields, why is embedding giving you? It may be even worth asking if it should be coupled in the same schema/struct in the first place. :)

[pmargreff]

If you are not using changesets or queries...

We are using the changesets for validations and rendering errors against the "virtual embed". You have a point about having a different schema that is not persisted, however, in that case, we would need to duplicate many other fields where we don't do this type of normalization. Using any would also be an option, but it would make it more complex to reason about the structure when passing it through changesets for validation.