Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PipeConsensus: complete consensus prodedure and pipe components with new thrift service #12355

Merged
merged 121 commits into from
May 31, 2024
Merged

PipeConsensus: complete consensus prodedure and pipe components with new thrift service #12355

merged 121 commits into from
May 31, 2024

Conversation

Pengzna
Copy link
Collaborator

@Pengzna Pengzna commented Apr 16, 2024 •
edited
Loading

@Pengzna Pengzna changed the title PipeConsensus: initialize thrift service PipeConsensus: initialize thrift service & connector and receiver logic Apr 16, 2024
@SteveYurongSu SteveYurongSu self-requested a review April 16, 2024 15:07
@SteveYurongSu SteveYurongSu self-assigned this Apr 16, 2024
Pengzna added 14 commits April 17, 2024 17:22
@Pengzna
fix review
b901cc7
@Pengzna
add comment and temporarily complete receiver
e8d183c
@Pengzna
feat: add clientManager and handshake logic
bae4893
@Pengzna
feat: construct pipeConsensus payload
a6493b7
@Pengzna
fix: client manager code style
463085b
@Pengzna
fix: manage event.referenceCount as review and add retry logic
87561fa
@Pengzna
fix: transfer request
c09f027
@Pengzna
feat: sync connector
13ce8e6
@Pengzna
feat: add new RPC service and client manager
dbf9c61
@Pengzna
feat: initialize handler
73dc7c9
@Pengzna
refactor: adjust new RPC service
d26acfb
@Pengzna
Merge remote-tracking branch 'base/master' into pipe-consensus
0461eac 
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/task/connection/PipeEventCollector.java
#	iotdb-core/node-commons/pom.xml
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/event/EnrichedEvent.java
@Pengzna
refactor: adopt tsFile dependency and remove RPC to new consensus int…
2fcc786 
…erface
@Pengzna
fix: redesign batch interface
8953c3c
OneSizeFitsQuorum
OneSizeFitsQuorum reviewed May 28, 2024
View reviewed changes
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/ConsensusFactory.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/ConsensusFactory.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/config/PipeConsensusConfig.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/pipe/PipeConsensus.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/ratis/RatisConsensus.java Outdated Show resolved Hide resolved
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/pipe/PipeConsensus.java Outdated Show resolved Hide resolved
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/DataRegionConsensusImpl.java Show resolved Hide resolved
...a/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java Show resolved Hide resolved
...a/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java Outdated Show resolved Hide resolved
...a/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java Outdated Show resolved Hide resolved
Caideyipi and others added 18 commits May 28, 2024 20:42
@Caideyipi
delete raw
f168839
@Caideyipi
Added IT
1ef6c77
@Caideyipi
Change IT
8cbf3f5
@Pengzna
fix
85b9e00
@Pengzna
fix review
15f29fe
@Pengzna
Merge pull request #2 from Caideyipi/consensus-apply-comment
54d3d8a 
Pipe Consensus: Applied partial comments & Added PipeConsensus to IT
@Pengzna
Merge remote-tracking branch 'refs/remotes/origin/pipe-consensus' int…
80bc979 
…o pipe-consensus

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/pipeconsensus/PipeConsensusReceiver.java
@Pengzna
fix all review
821395d
@SteveYurongSu
Merge branch 'master' of https://github.com/apache/iotdb into pr/12355
c130490
@Pengzna
revert IT and fix review
7ab7798
@Pengzna
Merge remote-tracking branch 'refs/remotes/origin/pipe-consensus' int…
41c7794 
…o pipe-consensus
@Pengzna
Merge remote-tracking branch 'refs/remotes/base/master' into pipe-con…
a7f0b22 
…sensus
@Pengzna
fix: build success
3295635
@Pengzna
fix review of ysc's part
47bb893
@Pengzna
fix assignProgressIndexForTsFileRecovery
514d124
@Pengzna
fix dependency
7341cbc
@Pengzna
fix dependency
3592e77
@Pengzna
fix dependency
2982404
Pengzna added 4 commits May 30, 2024 21:59
@Pengzna
fix review
465da4d
@Pengzna
Merge remote-tracking branch 'refs/remotes/base/master' into pipe-con…
5ce827b 
…sensus

# Conflicts:
#	iotdb-core/datanode/src/assembly/resources/conf/iotdb-datanode.properties
@Pengzna
fix merge conflict
2c42b21
@Pengzna
Merge remote-tracking branch 'refs/remotes/base/master' into pipe-con…
d7362e3 
…sensus

# Conflicts:
#	iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties
OneSizeFitsQuorum
OneSizeFitsQuorum reviewed May 31, 2024
View reviewed changes
Copy link
Contributor

@OneSizeFitsQuorum OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should add two more consensus class in line82
image

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties Outdated Show resolved Hide resolved
OneSizeFitsQuorum
OneSizeFitsQuorum approved these changes May 31, 2024
View reviewed changes
Copy link
Contributor

@OneSizeFitsQuorum OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@OneSizeFitsQuorum OneSizeFitsQuorum merged commit 9000bad into apache:master May 31, 2024
36 of 38 checks passed
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
@Pengzna @yschengzi
@Caideyipi @SteveYurongSu @OneSizeFitsQuorum @SzyWilliam
PipeConsensus: complete consensus prodedure and pipe components with …
c790c03 
…new thrift service (apache#12355)

Co-authored-by: yschengzi <[email protected]>
Co-authored-by: Caideyipi <[email protected]>
Co-authored-by: Steve Yurong Su <[email protected]>
Co-authored-by: OneSizeFitQuorum <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DanielWang2035 DanielWang2035 DanielWang2035 left review comments

@SzyWilliam SzyWilliam SzyWilliam left review comments

@Caideyipi Caideyipi Caideyipi approved these changes

@OneSizeFitsQuorum OneSizeFitsQuorum OneSizeFitsQuorum approved these changes

@SteveYurongSu SteveYurongSu Awaiting requested review from SteveYurongSu

Assignees

@SteveYurongSu SteveYurongSu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Pengzna @OneSizeFitsQuorum @SzyWilliam @DanielWang2035 @Caideyipi @SteveYurongSu @yschengzi