replace deprecated statefulMapConcat

[pjfanning]

I'm not sure if this is better but statefulMapConcat is deprecated.

The code is trying the take a Source[String, _] and to remove duplicates.

Both approaches involve building sets and those sets will consume a lot of memory if there is a lot of data. I don't think this is avoidable.

I am not a Cassandra expert but I think we might be able to get Cassandra to run 'DISTINCT' on the query. That could mean that we could remove the statefulMapConcat/statefulMap stage.

[nvollmar]

Cassandra only supports distinct queries on the partition key. Since both persistence_id and tag are the partition key we can't distinct query on the tag alone.

Since this is part of the reconciler that has to be manually called and is already marked as a rather expensive operation, I'd think this change is not too risky.

[nvollmar]

lgtm

[pjfanning]

Cassandra only supports distinct queries on the partition key. Since both persistence_id and tag are the partition key we can't distinct query on the tag alone.

Since this is part of the reconciler that has to be manually called and is already marked as a rather expensive operation, I'd think this change is not too risky.

@nvollmar I haven't tried it but if it was possible to sort the query result then the duplicate check would only need to know the last element as opposed to keeping a full set of visited tags. Do you think Cassandra is likely to allow this query to be sorted?

[nvollmar]

@pjfanning Cassandra does not allow to sort by arbitrary columns. You can define a cluster ordering of a table, but that also has limitations. A more "Cassandra way" to solve this would be using a dedicated table to keep all unique tags for example.

[Roiocam]

LGTM, only one style suggestion