Serializing time event types from Dry::Struct events

[xhs345]

Hi, I implemented the event base class in my app on Dry::Struct as shown in some shared tutorials.

One thing I'm disliking about it is how objects types like Time, DateTime and Date are serialized when saved by ActiveRecord.
E.g. !ruby/object:DateTime 2020-08-07 14:27:32.000000000 -07:00) or even

!ruby/object:ActiveSupport::TimeWithZone
    utc: &3 2022-11-22 14:31:00.000000000 Z
    zone: *2
    time: *3

In my mind it would be better to save them are iso8601 string because a) they could be consumed platform agnostic and b) also take up a lot less space (right now I'm running into errors because the data in one event is too large). Only obvious downside I see is that I lose the milliseconds.

The biggest challenge I see at the moment is that in the load step I need to load the event class from the event_type and then map the event attributes to the data to deserialize the correct strings into Time objects again.

I saw the guides for custom mappers and serializers in the docs, so I was just wondering if a specific implementation for this issue already existed or if anyone has suggestions.

[mostlyobvious]

The reason why you're seeing YAML in event data column in event_store_events tables is that we chose it as a safe default to serialize various data types. Mostly working without any additional configuration and supporting all database engines RES works on (not only PostgreSQL). Like every choice, it has its drawbacks too. YAML is one side of the spectrum.

This is the line responsible to set it:

rails_event_store/rails_event_store/lib/rails_event_store/client.rb

Line 9 in 260de99

repository: RailsEventStoreActiveRecord::EventRepository.new(serializer: RubyEventStore::Serializers::YAML),

In ecommerce app, which runs on PostgreSQL, we chose to store data column with jsonb data type. To reap benefits from jsonb and also due to how ActiveRecord interacts with pg/jsonb we do not serialize data column there:
https://github.com/RailsEventStore/ecommerce/blob/f783f8d44ded7e756933b2eba001e905514457e0/infra/lib/infra/event_store.rb#L16
The outcome is similar to what JSON.dump does. For example, Time is represented as ISO string. But also brings some problems described in https://railseventstore.org/docs/v2/mapping_serialization/#configuring-a-different-mapper.

So this side of the spectrum most of the time requires describing event schemas with Dry::Struct, ActiveModel::Attributes or the like. It is additional work required by the developers. But also additional benefits — documentation.

Looking at this event class example from ecommerce, the attributes passed to initialize the instance of event class will be processed by its internal schema description. Coercing any ISO strings into Time if that was stated in event schema.

Going with Protocol Buffers means mandatory event schema, because the serialized output is binary and schema is needed to decode it. This is the other side of the spectrum.

Somewhere in the middle of the spectrum is the solution we'll be slowly introducing from next (v2.6.0) RES release.
There's already a component called Mapper that is responsible to transform event object into record ready to be persisted in database or scheduled into background jobs to process. This component is a pipeline of transformations (similar in concept to Rack middlewares). When reading event records from disk, they have to be processed by this pipeline until they become event objects again.

By introducing special transformation in the pipeline we have a way to transform particular objects before they even hit the serializer. To make it work in both directions, information about original type has to be preserved.
https://github.com/RailsEventStore/rails_event_store/pull/1103/files#diff-0aae494240dbcf777230301fced84eb9495e93bc01227505d8220470be605908
This is not an original idea, I believe ActiveJob already does it similarly.

In all approaches to transform objects into strings and back, information about desired type has to be preserved. In YAML that's encoded along data. In JSON it has to be stored aside, in code (event schema) or along event (metadata).

[mostlyobvious]

Btw. some time ago I've started "benchmarking" different approaches to event schemas — https://github.com/pawelpacana/res-event-schema-comparison

[xhs345]

Wow, thanks so much for the in-depth response, that gives me a lot to study. YAML as a default choice makes total sense.
After reading I reconsidered and now think I'll wait for the 2.6 release first to see how I can utilize the new functionality.

For our project we use MySQL, so sadly don't have JSONB available. I suppose I could still use JSON (since our version is newer than 5.7.8), but then I would have to still include the json serializer, which doesn't sound too bad.

So this side of the spectrum most of the time requires describing event schemas with Dry::Struct, ActiveModel::Attributes or the like. It is additional work required by the developers. But also additional benefits — documentation.

Yeah, since events are such an integral part of the system when using an event store it seems very irresponsible to me to not define a proper structure for them in code

[xhs345]

One more thing to add: Since I was struggling with the size of the field in the event I thought a bit more about potential solutions and one idea I got was to use Zlib to deflate and inflate the data as part of the serialization.
E.g. my string of comma separated floats can be reduced from 240k to 10k characters. I'm going to do it manually for now, but could be a neat solution for others who are having the same issue and it might warrant a better implementation.

[mostlyobvious]

Released: https://github.com/RailsEventStore/rails_event_store/releases/tag/v2.6.0

[xhs345]

Thanks!