Adding a new mapper type isn't possible.

[dhirendrafkl]

While we want to add new Mapper that would do mapping based on some conditional logic we found out that due to MapperType being an Enum, it isn't possible to customize it to add new mapper.

[aryaKetan]

Can you elaborate on what conditional logic you want to implement. We can have a discussion around that.

[dhirendrafkl]

Example

Table1
EntityId,Key,Value

Table2
EntityId,...

In table1 there are some keys which in merged table would become columns.

MergedTable
EntityId,KeyX,KeyY,...

We wanted this type of customization which wasn't possible due to aforementioned problem.

[aryaKetan]

So, let me see if I got the requirement correct.

You have table 1 as

ACC1, EMAIL, [email protected]
ACC1, PHONE, 900
ACC2, EMAIL, [email protected]

and you want to transform it into a table MergedTable as follows

ACC1 ,[email protected],900
ACC2,[email protected],NULL

Is my understanding correct?

[dhirendrafkl]

Yes. Absolutely right.
On Aug 13, 2015 8:53 AM, "Arya Ketan" [email protected] wrote:

So, let me see if I got the requirement correct.

You have table 1 as

ACC1, EMAIL, [email protected]
ACC1, PHONE, 900
ACC2, EMAIL, [email protected]

and you want to transform it into a table MergedTable as follows

ACC1 ,[email protected],900
ACC2,[email protected],NULL

Is my understanding correct?

—
Reply to this email directly or view it on GitHub
#28 (comment).

---

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. If you are not the intended recipient
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited. Although Flipkart has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments

[aryaKetan]

Ok. So the problem with your ask is that data is to transform your data from EAV model to row model. This may work only if updates/deletes to both rows of ACC1 happen in the same transaction. Even then, it would be complex to merge the rows into 1 row.
But more or less , updates within the same txn will never happen and there-fore you would have to maintain state for ACC1 for you to merge.

But actually, to do what you want to do, you dont need a mapper type interface.
This login can be implemented in the destination client processor.
Say for e.g. you want to insert into mysql the merged table.

just keep firing queries such as update MergedTable set KeyX='Val' where entityId = 'ACC1';
when you get KeyY, your query would be like
update MergedTable set KeyY='Val2' where entityId = 'ACC1';

and your Merged Table is ready.

Say if you want to insert the merged table into HBase,
then keep adding columns in the column family for the rowKey ACC1.
.

This can be done for any of the databases.

[anshul10s]

This is interesting approach Arya, works good for consumer stores which can accept data in columnar fashion and support ACID transaction.

Was wondering what will be the approach if I want to index these 2 columns in ES for a given entity. I can model it as 2 different child documents of entity, so that updates can be independent or read the document and update it.

Reading and updating will have consistency issues in corner cases, as ES is does not support ACID transaction. Another approach can be to treat these as a change singal and query the source of truth for all rows related to ACC1 and create the complete document for indexing.

[aryaKetan]

So, if we want to model our destination store in ES under the same conditions.
So the elasic search doc is like
{
"keyX" : "Val1",
"keyV" : "Val2"
}
cant we can do this by running groovy scripts for update or using the UpdateRequest ( in java client) ? We dont need to read doc and update. Now, even if you are reading it and updating, there is version support in ES update and if version does not match, it can return error

IMO ,this will not be doable only in K-V stores where-in in-place updates are not possible. ( couchbase - membase bucket)

[dhirendrafkl]

We've already overriden and solved it. Only concern is if someone wants to
add a mapper for some specific use case this won't be possible.

On Thu, Aug 13, 2015 at 11:51 AM, Arya Ketan [email protected]
wrote:

So, if we want to model our destination store in ES under the same
conditions.
So the elasic search doc is like
{
"keyX" : "Val1",
"keyV" : "Val2"
}
cant we can do this by running groovy scripts for update or using the
UpdateRequest ( in java client) ? We dont need to read doc and update. Now,
even if you are reading it and updating, there is version support in ES
update and if version does not match, it can return error

IMO ,this will not be doable only in K-V stores where-in in-place updates
are not possible. ( couchbase - membase bucket)

—
Reply to this email directly or view it on GitHub
#28 (comment).

---

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. If you are not the intended recipient
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited. Although Flipkart has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments

[regunathb]

@dhirendrafkl do you have a fix that will address this issue? Happy to accept a pull request if the solution is generic enough.

[dhirendrafkl]

@regunathb Solution was implemented seperately without forking from aesop. This can be found at https://github.com/Flipkart/erp-change-propagation/tree/master/fklogistics-migration/fklogistics-hbase-consumer