Removed Requirement of transactions on ORL, added SCN code.

[alosix]

There are 2 specific commits here that others might find useful. If it would help for me to move them over to another branch for the pull, let me know.

This removes the need to use transactions in mysql. Transactions still work, but individual updates will as well. This change also removes an NPE that happens when an un-watched table happens to have an update.
sovrn@a997948

This change has made the relay a lot easier to use:
sovrn@061475e

We only really need to watch for changes currently, so there's not much of a need to use the file based storage. I found that in-memory didn't work because it would always default the listener to bin file 1 at offset 4. For our longer running servers those files were purged quite some time ago. This change should work for both file and in-memory. If by the end of the method the scn is still 0 or -1, it connects to the DB to look for the current file and position, then turns that into an SCN. I used the raw JDBC code here because I didn't see any connection managers (ie c3p0 or dbcp) in the dependencies. Seemed silly to add one just for this.

This also removes the need to calculate the current SCN by hand when running the example code.

[phanindraganti]

Thanks Jason. Please give us around 2 days to review this code, and see how
to pull this in.

Offhand - one of the changes look good, which is the ability to query the
earliest available binlog file on server and stream from there.

The prototype adapter has a bug where in if two mysql statements are
present in same transaction, throws an cautionary exception. Not sure if
the patch related to "unwatched table" is addressessing that. If not, have
a patch to push myself. Will take a look.

Cheers
Phani

On Mon, Apr 28, 2014 at 11:37 AM, Jason Powers [email protected]:

There are 2 specific commits here that others might find useful. If it
would help for me to move them over to another branch for the pull, let me
know.

This removes the need to use transactions in mysql. Transactions still
work, but individual updates will as well. This change also removes an NPE
that happens when an un-watched table happens to have an update.
sovrn/databus@a997948sovrn@a997948

This change has made the relay a lot easier to use:
sovrn@061475ehttps://github.com/sovrn/databus/commit/061475e59c9f612347cd42fe979d83ed94b72ee5

We only really need to watch for changes currently, so there's not much of
a need to use the file based storage. I found that in-memory didn't work
because it would always default the listener to bin file 1 at offset 4. For
our longer running servers those files were purged quite some time ago.
This change should work for both file and in-memory. If by the end of the
method the scn is still 0 or -1, it connects to the DB to look for the
current file and position, then turns that into an SCN. I used the raw JDBC
code here because I didn't see any connection managers (ie c3p0 or dbcp) in
the dependencies. Seemed silly to add one just for this.

This also removes the need to calculate the current SCN by hand when

running the example code.

You can merge this Pull Request by running

git pull https://github.com/sovrn/databus master

Or view, comment on, or merge it at:

#32
Commit Summary

• Updates to use sovrn repo for builds
• Swapped to sovrn built version of open-replicator
• Removed link to sovrn nexus repo, moved to local init.gradle
• Removed the requirement to work in a transaction, added code to stop
  the NPE when an unwatched table comes through
• Changed version number to be unique to this fork
• Initial cut of code that will generate the current SCN if its not
  given one at startup

File Changes

• M
  databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/ORListener.javahttps://github.com/Removed Requirement of transactions on ORL, added SCN code. #32/files#diff-0(15)
• M
  databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/OpenReplicatorEventProducer.javahttps://github.com/Removed Requirement of transactions on ORL, added SCN code. #32/files#diff-1(55)
• M subprojects.gradlehttps://github.com/Removed Requirement of transactions on ORL, added SCN code. #32/files#diff-2(4)

Patch Links:

• https://github.com/linkedin/databus/pull/32.patch
• https://github.com/linkedin/databus/pull/32.diff

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/32
.

[alosix]

No problem,

Yeah, it sounds like the un-watched table code fixes the same thing that
you're talking about.

If it helps I can move them each over to separate branches and send pull
requests like that.

On Tue, Apr 29, 2014 at 8:44 AM, Phanindra Ganti
[email protected]:

Thanks Jason. Please give us around 2 days to review this code, and see
how
to pull this in.

Offhand - one of the changes look good, which is the ability to query the
earliest available binlog file on server and stream from there.

The prototype adapter has a bug where in if two mysql statements are
present in same transaction, throws an cautionary exception. Not sure if
the patch related to "unwatched table" is addressessing that. If not, have
a patch to push myself. Will take a look.

Cheers
Phani

On Mon, Apr 28, 2014 at 11:37 AM, Jason Powers [email protected]:

There are 2 specific commits here that others might find useful. If it
would help for me to move them over to another branch for the pull, let
me
know.

This removes the need to use transactions in mysql. Transactions still
work, but individual updates will as well. This change also removes an
NPE
that happens when an un-watched table happens to have an update.
sovrn/databus@a997948<
sovrn@a997948c651f85a496fb3f52e6ca608ec8ce6f52>

This change has made the relay a lot easier to use:
sovrn/databus@061475e<
sovrn@061475e59c9f612347cd42fe979d83ed94b72ee5>

We only really need to watch for changes currently, so there's not much
of
a need to use the file based storage. I found that in-memory didn't work
because it would always default the listener to bin file 1 at offset 4.
For
our longer running servers those files were purged quite some time ago.
This change should work for both file and in-memory. If by the end of
the
method the scn is still 0 or -1, it connects to the DB to look for the
current file and position, then turns that into an SCN. I used the raw
JDBC
code here because I didn't see any connection managers (ie c3p0 or dbcp)
in
the dependencies. Seemed silly to add one just for this.

This also removes the need to calculate the current SCN by hand when

running the example code.

You can merge this Pull Request by running

git pull https://github.com/sovrn/databus master

Or view, comment on, or merge it at:

#32
Commit Summary

• Updates to use sovrn repo for builds
• Swapped to sovrn built version of open-replicator
• Removed link to sovrn nexus repo, moved to local init.gradle
• Removed the requirement to work in a transaction, added code to stop
  the NPE when an unwatched table comes through
• Changed version number to be unique to this fork
• Initial cut of code that will generate the current SCN if its not
  given one at startup

File Changes

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/ORListener.java<
https://github.com/linkedin/databus/pull/32/files#diff-0>(15)

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/OpenReplicatorEventProducer.java<
https://github.com/linkedin/databus/pull/32/files#diff-1>(55)

• M subprojects.gradle<
  https://github.com/linkedin/databus/pull/32/files#diff-2>(4)

Patch Links:

• https://github.com/linkedin/databus/pull/32.patch
• https://github.com/linkedin/databus/pull/32.diff

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/32#issuecomment-41684621
.

[phanindraganti]

Jason,

  Can you please send the patch to pull in from the oldest available

binlog as a separate pull request ?

Thanks
Phani

On Tue, Apr 29, 2014 at 8:03 AM, Jason Powers [email protected]:

No problem,

Yeah, it sounds like the un-watched table code fixes the same thing that
you're talking about.

If it helps I can move them each over to separate branches and send pull
requests like that.

On Tue, Apr 29, 2014 at 8:44 AM, Phanindra Ganti
[email protected]:

Thanks Jason. Please give us around 2 days to review this code, and see
how
to pull this in.

Offhand - one of the changes look good, which is the ability to query
the
earliest available binlog file on server and stream from there.

The prototype adapter has a bug where in if two mysql statements are
present in same transaction, throws an cautionary exception. Not sure if
the patch related to "unwatched table" is addressessing that. If not,
have
a patch to push myself. Will take a look.

Cheers
Phani

On Mon, Apr 28, 2014 at 11:37 AM, Jason Powers [email protected]:

There are 2 specific commits here that others might find useful. If it
would help for me to move them over to another branch for the pull,
let
me
know.

This removes the need to use transactions in mysql. Transactions still
work, but individual updates will as well. This change also removes an
NPE
that happens when an un-watched table happens to have an update.
sovrn/databus@a997948<

sovrn@a997948c651f85a496fb3f52e6ca608ec8ce6f52>

This change has made the relay a lot easier to use:
sovrn/databus@061475e<

sovrn@061475e59c9f612347cd42fe979d83ed94b72ee5>

We only really need to watch for changes currently, so there's not
much
of
a need to use the file based storage. I found that in-memory didn't
work
because it would always default the listener to bin file 1 at offset
4.
For
our longer running servers those files were purged quite some time
ago.
This change should work for both file and in-memory. If by the end of
the
method the scn is still 0 or -1, it connects to the DB to look for the
current file and position, then turns that into an SCN. I used the raw
JDBC
code here because I didn't see any connection managers (ie c3p0 or
dbcp)
in
the dependencies. Seemed silly to add one just for this.

This also removes the need to calculate the current SCN by hand when

running the example code.

You can merge this Pull Request by running

git pull https://github.com/sovrn/databus master

Or view, comment on, or merge it at:

#32
Commit Summary

• Updates to use sovrn repo for builds
• Swapped to sovrn built version of open-replicator
• Removed link to sovrn nexus repo, moved to local init.gradle
• Removed the requirement to work in a transaction, added code to stop
  the NPE when an unwatched table comes through
• Changed version number to be unique to this fork
• Initial cut of code that will generate the current SCN if its not
  given one at startup

File Changes

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/ORListener.java<

https://github.com/linkedin/databus/pull/32/files#diff-0>(15)

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/OpenReplicatorEventProducer.java<

https://github.com/linkedin/databus/pull/32/files#diff-1>(55)

• M subprojects.gradle<
  https://github.com/linkedin/databus/pull/32/files#diff-2>(4)

Patch Links:

• https://github.com/linkedin/databus/pull/32.patch
• https://github.com/linkedin/databus/pull/32.diff

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32#issuecomment-41684621>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/32#issuecomment-41687657
.

[alosix]

No prob.

On Tue, May 6, 2014 at 7:39 AM, Phanindra Ganti [email protected]:

Jason,

Can you please send the patch to pull in from the oldest available
binlog as a separate pull request ?

Thanks
Phani

On Tue, Apr 29, 2014 at 8:03 AM, Jason Powers [email protected]:

No problem,

Yeah, it sounds like the un-watched table code fixes the same thing that
you're talking about.

If it helps I can move them each over to separate branches and send pull
requests like that.

On Tue, Apr 29, 2014 at 8:44 AM, Phanindra Ganti
[email protected]:

Thanks Jason. Please give us around 2 days to review this code, and
see
how
to pull this in.

Offhand - one of the changes look good, which is the ability to query
the
earliest available binlog file on server and stream from there.

The prototype adapter has a bug where in if two mysql statements are
present in same transaction, throws an cautionary exception. Not sure
if
the patch related to "unwatched table" is addressessing that. If not,
have
a patch to push myself. Will take a look.

Cheers
Phani

On Mon, Apr 28, 2014 at 11:37 AM, Jason Powers <
[email protected]>wrote:

There are 2 specific commits here that others might find useful. If
it
would help for me to move them over to another branch for the pull,
let
me
know.

This removes the need to use transactions in mysql. Transactions
still
work, but individual updates will as well. This change also removes
an
NPE
that happens when an un-watched table happens to have an update.
sovrn/databus@a997948<

sovrn@a997948c651f85a496fb3f52e6ca608ec8ce6f52>

This change has made the relay a lot easier to use:
sovrn/databus@061475e<

sovrn@061475e59c9f612347cd42fe979d83ed94b72ee5>

We only really need to watch for changes currently, so there's not
much
of
a need to use the file based storage. I found that in-memory didn't
work
because it would always default the listener to bin file 1 at offset
4.
For
our longer running servers those files were purged quite some time
ago.
This change should work for both file and in-memory. If by the end
of
the
method the scn is still 0 or -1, it connects to the DB to look for
the
current file and position, then turns that into an SCN. I used the
raw
JDBC
code here because I didn't see any connection managers (ie c3p0 or
dbcp)
in
the dependencies. Seemed silly to add one just for this.

This also removes the need to calculate the current SCN by hand when

running the example code.

You can merge this Pull Request by running

git pull https://github.com/sovrn/databus master

Or view, comment on, or merge it at:

#32
Commit Summary

• Updates to use sovrn repo for builds
• Swapped to sovrn built version of open-replicator
• Removed link to sovrn nexus repo, moved to local init.gradle
• Removed the requirement to work in a transaction, added code to
  stop
  the NPE when an unwatched table comes through
• Changed version number to be unique to this fork
• Initial cut of code that will generate the current SCN if its not
  given one at startup

File Changes

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/ORListener.java<

https://github.com/linkedin/databus/pull/32/files#diff-0>(15)

• M

databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/OpenReplicatorEventProducer.java<

https://github.com/linkedin/databus/pull/32/files#diff-1>(55)

• M subprojects.gradle<
  https://github.com/linkedin/databus/pull/32/files#diff-2>(4)

Patch Links:

• https://github.com/linkedin/databus/pull/32.patch
• https://github.com/linkedin/databus/pull/32.diff

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32#issuecomment-41684621>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/linkedin/databus/pull/32#issuecomment-41687657>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/32#issuecomment-42302709
.