Position not used anymore since RES 2.x?

[butsjoh]

Hi,

We started from scratch using ruby-event-store (2.0.1) and rails_event_store_active_record (2.0.1) and planning to upgrade to 2.1.0 soon. But we noticed that the position column is always nil while in previous version (before 2.0.1) it was filled in. Is this expected behaviour? We did not change anything related to expected_version or something. Can you point me to configuration that i need to check why this is happening or can you confirm that it is not really being used? Thnx

[butsjoh]

nvm we where using expected_version: :auto (the default) so it is normal position is not filled in

But i do have another question then related to this. It seems we cannot make the primary key of the event_store_events_in_streams table of type uuid cause it used for order. Is there any reason why it cannot be of type uuid and it needs to be a sequential number?

[mostlyobvious]

nvm we where using expected_version: :auto (the default) so it is normal position is not filled in

The default expection_version in append and publish is :any and indeed its result is NULL — more on the why can be found in https://railseventstore.org/docs/v2/expected_version/

It seems we cannot make the primary key of the event_store_events_in_streams table of type uuid cause it used for order.

Right, let's dissect the schema piece by piece (from postgresql perspective to pick just one):

                                       Table "public.event_store_events_in_streams"
   Column   |            Type             | Collation | Nullable |                          Default                          
------------+-----------------------------+-----------+----------+-----------------------------------------------------------
 id         | integer                     |           | not null | nextval('event_store_events_in_streams_id_seq'::regclass)
 stream     | character varying           |           | not null | 
 position   | integer                     |           |          | 
 event_id   | uuid                        |           | not null | 
 created_at | timestamp without time zone |           | not null | 
Indexes:
    "event_store_events_in_streams_pkey" PRIMARY KEY, btree (id)
    "index_event_store_events_in_streams_on_created_at" btree (created_at)
    "index_event_store_events_in_streams_on_stream_and_event_id" UNIQUE, btree (stream, event_id)
    "index_event_store_events_in_streams_on_stream_and_position" UNIQUE, btree (stream, "position")

The event_store_events_in_streams table holds information about streams that events belong to. One event can belong to multiple streams. In a single stream, particular event can only appear once. Streams are some form of indexing or grouping of events, tailored for the needs of the reader.

Thus the event_id column of type uuid referencing event, a stream column describing which stream it belongs to. And the UNIQUE, btree (stream, event_id) constraint to protect uniqueness of the event in a stream.

Events appear in a certain, stable order in a stream. This order can be decided at the moment of writing — the case of expected_version: :any. Or it can be controlled tightly by the application with optimistic concurrency control via expection_version: Integer.

The former uses NULL as a value in the position column . The latter uses integers in the position column. As I mentioned, the order needs to be stable. We need the exact same results on every read. In streams that fill in integers in position this simple — just order by position. In streams that allow NULL in position, we need a second column to decide on order. Primary key, being an integer, provides that quality of stable order decided by the moment of writing.

So finally we sort each stream by position first and id second and need both of them. Having a primary key of uuid type does not help with sorting (unless UUIDs are very specificly crafted to give monotonic order). It would have been an additional column (instead of reusing existing one), which would be a waste.

Does that answer your question? Would you do it differently?

[butsjoh]

Hi,

Thanks for the detailed answer, it was in the direction that i was assuming after reading through the code a bit more. Let me think about this a bit further before i can form an opinion on it. Will get back to you about this but for now this gives me the explanation i need to guide my thinking process in the right direction.

[butsjoh]

Without thinking it much threw yet. The event_store_events_in_streams table also has a created_at (with high precision). Could not that not be added as an alternative to sort on? So first order on position and then created_at? That way id can be whatever it needs to be and is not used for sorting at all. Or are there any downside or assumption that i am not seeing? Processing events in order is always going to be a tricky thing and currently we don't have a lot of practical experience yet with event sourcing (apart from reading the theory) to make a good decision. We are currently not using aggregate root which i can see from the documentation uses expected_version with explicit number. I think choosing how to work with expected_version is one of the things we should be aware of and not assume things to just be in order.

[butsjoh]

Also another argumentation to maybe allow sorting on created_at instead of id is in the situation where you use distribution databases (in our case coackroach) where maintaining sequential ids is not that easy. There the use of uuid as the primary key is mostly preferred. Maybe the suggestion would be to make the second sorting column optional (id or created_at)?

[mostlyobvious]

The event_store_events_in_streams table also has a created_at (with high precision). Could not that not be added as an alternative to sort on? So first order on position and then created_at? That way id can be whatever it needs to be and is not used for sorting at all. Or are there any downside or assumption that i am not seeing?

With sequential ids, the strict monotonicity is guaranteed. With high precision time column it could work good enough for some systems, but there is no guarantee. If you end with same timestamps for two records, what is the order? Postgresql in such case would rely on the order in which data is persisted on disk. Would that order remain after replication, on a different node? I can't tell. What about other RDBMSes?

These tradeoffs are fine in a custom repository, tailored for specific system and for a given set of assumptions and risks you're willing to take. I'd rather not incorporate it in RES itself, but I'm open for making custom repositories easier.

Also another argumentation to maybe allow sorting on created_at instead of id is in the situation where you use distribution databases (in our case coackroach) where maintaining sequential ids is not that easy.

Maintaining clocks in distributed systems is not easy as well: https://www.theverge.com/2012/11/26/3692392/google-spanner-atomic-clocks-GPS

[mostlyobvious]

We are currently not using aggregate root which i can see from the documentation uses expected_version with explicit number. I think choosing how to work with expected_version is one of the things we should be aware of and not assume things to just be in order.

Btw. have a look at https://github.com/arkency/aggregates to have an overview of different options/implementations of an aggregate than aggregate_root itself.