feat: Create initial Coordinator V2 service #444

morgsmccauley · 2023-12-12T01:55:52Z

This PR adds the initial Coordinator V2 service, which acts as the main driver for the Control Plane.

Coordinator Overview

Coordinator V2 will exist as a new standalone service with it's primary goal being ensuring the current registry configuration is mapped to the system. It's core logic is just an infinite loop which reads the registry, and sends necessary requests to the Block Streamer and Runner services to synchronise that config.

Block Streamer Changes

Some changes have been made to Block Streamer to enable the above:

version and redis_stream have been added to the proto so that Coordinator can configure them.
Support for ActionFunctionCall has been added - Initially I thought only ActionAny allowed, but the current registry has ActionFunctionCall and therefore needs to be supported.
last_published_block is now written to Redis to enable "Start from interruption"

What's not been done

I wanted to limit the scope of this PR as it was starting to get big, I'll address these tasks in follow up PRs:

Provisioning - Coordinator should check the status of provisioning, and act when the state isn't as expected. This can be done after Expose endpoint to control provisioning #426 is implemented.
Retry recoverable errors - Any error will be propagated cause the error to exit, this includes connection errors to the Block Streamer and Runner services. As these are very likely to occur (across restarts) we should retry these errors.
Environment configuration - There are many hard coded values (endpoints, registry contract, etc.) which should be configurable via the environment.
Logging and Metrics

…streams`

morgsmccauley · 2024-01-08T22:31:35Z

block-streamer/Cargo.toml

@@ -26,7 +26,7 @@ wildmatch = "2.1.1"

 registry-types = { path = "../registry/types", features = ["near-primitives"] }

-near-lake-framework = "0.7.4"
+near-lake-framework = "=0.7.4"


The existing mocks for near-lake-framework depend on behaviour which does not exist in the next version.

I'm pinning this for now, when we upgrade we'll need to find another way to mock it.

morgsmccauley · 2024-01-08T22:32:42Z

block-streamer/src/block_stream.rs

        }
        MatchingRule::Event { .. } => {
-            bail!("Event matching rule not yet supported for historical processing, function {:?} {:?}", indexer.account_id, indexer.function_name);


Since we have a single stream for historical/real-time we can no longer bail here. Instead we just log and continue to the 'real-time' portion. Which is essentially the same behaviour.

coordinator/src/main.rs

darunrs

Sorry it took a while to review! Mainly just have some small comments on the specifics. Looks good to me overall!

block-streamer/proto/block_streamer.proto

block-streamer/src/block_stream.rs

block-streamer/src/server/block_streamer_service.rs

coordinator/src/executors_handler.rs

coordinator/src/main.rs

coordinator/src/redis.rs

coordinator/src/registry.rs

coordinator/src/main.rs

darunrs

LGTM! Looks like the build failure is a rate limit issue, so that's resolvable... hopefully.

morgsmccauley linked an issue Dec 12, 2023 that may be closed by this pull request

Create Control service #421

Closed

morgsmccauley force-pushed the test/block-streamer branch from 0d49621 to 135c00d Compare December 14, 2023 00:19

Base automatically changed from test/block-streamer to main December 14, 2023 00:21

morgsmccauley force-pushed the feat/control-plane branch from 523545d to 47d2b24 Compare December 14, 2023 21:30

morgsmccauley changed the base branch from main to refactor/shared-registry-types December 14, 2023 21:30

Base automatically changed from refactor/shared-registry-types to main December 15, 2023 02:02

morgsmccauley force-pushed the feat/control-plane branch 4 times, most recently from 3241d19 to c971512 Compare December 21, 2023 02:42

morgsmccauley changed the base branch from main to 415-add-createdupdated-at-timestamps December 21, 2023 02:43

morgsmccauley force-pushed the feat/control-plane branch from cb2021b to 9c7f2b8 Compare December 21, 2023 03:45

morgsmccauley force-pushed the feat/control-plane branch from a4969ba to af1ff04 Compare January 3, 2024 08:20

Base automatically changed from 415-add-createdupdated-at-timestamps to main January 4, 2024 19:54

morgsmccauley force-pushed the feat/control-plane branch from af1ff04 to ebec14c Compare January 4, 2024 20:11

darunrs mentioned this pull request Jan 5, 2024

feat: Create Rust gRPC client for Runner #479

Closed

morgsmccauley force-pushed the feat/control-plane branch 2 times, most recently from 3dbc9ee to b76eb37 Compare January 8, 2024 02:04

morgsmccauley changed the base branch from main to 468-create-rust-grpc-client-for-runner January 8, 2024 02:04

morgsmccauley mentioned this pull request Jan 8, 2024

feat: Create rust GRPC client for Runner #491

Merged

morgsmccauley force-pushed the feat/control-plane branch from 6fd4af8 to bf00ccb Compare January 8, 2024 20:04

Base automatically changed from 468-create-rust-grpc-client-for-runner to main January 8, 2024 21:24

morgsmccauley added 8 commits January 9, 2024 10:24

feat: Create crate for coordinator control service

8e6e95d

feat: Fetch and print registry contract

c2f857b

feat: Used shared registry types

dd26099

refactor: Extract registry fetch to own mod

025b315

refactor: Use structured logging

3e05d9a

feat: Start all streams with stubbed data

4ba5296

feat: Start stream with filter specified in registry

126296d

feat: Add update_at/created_at fields to registry

64d835b

morgsmccauley added 9 commits January 9, 2024 10:24

chore: Use near-primitives feature for registry-types

6e89697

refactor: Rename synchronise_registry_config -> `synchronise_block_…

67e60fd

…streams`

feat: Start executors

83a2d92

feat: Add error context

2977df0

feat: Restart executors with mismatched versions

20b0fb4

feat: Stop executors not in registry

73609d8

refactor: Configure block stream redis stream from coordinator

a9cf247

fix: Enable (hack) mocking by pinning lake framework version

240ba3d

ci: Add checks for Coordinator

165111f

morgsmccauley force-pushed the feat/control-plane branch from 7135848 to 165111f Compare January 8, 2024 21:25

morgsmccauley commented Jan 8, 2024

View reviewed changes

refactor: Rename to last_published_block as it has not been executed

62a3334

morgsmccauley commented Jan 8, 2024

View reviewed changes

coordinator/src/main.rs Outdated Show resolved Hide resolved

morgsmccauley marked this pull request as ready for review January 8, 2024 22:37

morgsmccauley requested a review from a team as a code owner January 8, 2024 22:37

morgsmccauley changed the title ~~feat/control plane~~ feat: Create initial Coordinator V2 service Jan 8, 2024

morgsmccauley mentioned this pull request Jan 9, 2024

Create Control service #421

Closed

darunrs reviewed Jan 10, 2024

View reviewed changes

morgsmccauley added 4 commits January 11, 2024 14:25

chore: Remove "historical" name references from logging

d7aa296

fix: Publish blocks to correct redis stream

e373d4c

chore: Fix spelling

abc7272

fixup! fix: Publish blocks to correct redis stream

762a078

darunrs approved these changes Jan 11, 2024

View reviewed changes

morgsmccauley added 2 commits January 11, 2024 14:52

refactor: Rename for clarity

d250f1f

fix: Fix rate limiting on protoc setup

46745d2

morgsmccauley merged commit b76fac0 into main Jan 11, 2024
12 checks passed

morgsmccauley deleted the feat/control-plane branch January 11, 2024 02:02

morgsmccauley mentioned this pull request Jan 17, 2024

Prod Release 17/01/24 #513

Merged

morgsmccauley mentioned this pull request Apr 22, 2024

test stable branch git fix up #687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Create initial Coordinator V2 service #444

feat: Create initial Coordinator V2 service #444

morgsmccauley commented Dec 12, 2023 •

edited

Loading

morgsmccauley Jan 8, 2024

morgsmccauley Jan 8, 2024

darunrs left a comment

darunrs left a comment •

edited

Loading

feat: Create initial Coordinator V2 service #444

feat: Create initial Coordinator V2 service #444

Conversation

morgsmccauley commented Dec 12, 2023 • edited Loading

Coordinator Overview

Block Streamer Changes

What's not been done

morgsmccauley Jan 8, 2024

Choose a reason for hiding this comment

morgsmccauley Jan 8, 2024

Choose a reason for hiding this comment

darunrs left a comment

Choose a reason for hiding this comment

darunrs left a comment • edited Loading

Choose a reason for hiding this comment

morgsmccauley commented Dec 12, 2023 •

edited

Loading

darunrs left a comment •

edited

Loading