Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create Control service #421

Closed
Tracked by #396
morgsmccauley opened this issue Nov 21, 2023 · 1 comment · Fixed by #444, #453, #503 or #515
Closed
Tracked by #396

Create Control service #421

morgsmccauley opened this issue Nov 21, 2023 · 1 comment · Fixed by #444, #453, #503 or #515
Assignees
@morgsmccauley
Labels
component: Coordinator

Comments

@morgsmccauley
Copy link
Collaborator

morgsmccauley commented Nov 21, 2023 •
edited
Loading

This service will be solely responsible for ensure the rest of the system is working as expected. It will read the current configuration from the Registry contract, and issue relevant commands to bring this configuration to life, i.e. it will Coordinate the rest of the system.

To allow an seamless transition from Coordinator V1 to V2, this should exist as a new service. We can call it queryapi-coordinator-mainnet to avoid conflicts.

Coordinator V2 will run an infinite loop consisting of the following logic:

  1. Fetch current indexers from registry contract
  2. Fetch Executor statuses
  3. Fetch Streamer statuses
  4. For each indexer
    • Provisioning
      • if the indexer has not been provisioned: provision it (we should only check once)
    • Streams
      • if there is no stream: start the stream from last_processed_block
      • If there is no stream and no last_processed_block, start from created/updated block
      • if stream exists and version differs: restart the stream from start_block_height
      • else: do nothing
    • Executors
      • if there is no executor: start it with current config
      • if executor exists and version differs: restart with new config
      • else: do nothing
@morgsmccauley morgsmccauley mentioned this issue Nov 21, 2023
Closed
@morgsmccauley morgsmccauley changed the title Create Control service (Coordinator V2) Create Control service Nov 21, 2023
@morgsmccauley morgsmccauley self-assigned this Dec 12, 2023
@morgsmccauley morgsmccauley linked a pull request Dec 12, 2023 that will close this issue
feat: Create initial Coordinator V2 service #444
Merged
@morgsmccauley morgsmccauley mentioned this issue Dec 14, 2023
Merged
morgsmccauley added a commit that referenced this issue Dec 15, 2023
@morgsmccauley
refactor: Extract registry types in to own crate (#453)
af38be3 
This PR extract the types exposed by the registry contract in to its own
reusable crate: `registry_types`. This was done to avoid the duplication
across our different service which; fetch the JSON, and then construct
the Rust types.

## `registry_types` crate 
The registry types were already partially shared between Coordinator and
the registry, but I've expanded on this to include all types rather than
just a subset. I've created a new crate, rather than using the existing
`./indexer/indexer_rule_type` crate as some packages needed to be
upgraded, which caused conflicts across the `indexer/` cargo workspace -
which seemed unnecessary to fix given that it is being deprecated.

## Registry contract updates
`near-sdk` version `4.1.1` is no longer compilable because it depends on
a non-existent crate, hence the upgrade to `5.0.0-alpha.1`. Since I was
already making changes here, I decided to remove the migration related
code as it is no longer needed.

## Consuming `registry_types`
With the shared types in place, I've updated `block-streamer/` to use
them, removing the existing types which it duplicates. I also went ahead
and removed all types which were unused (left over from Alertexer).

`indexer_rule_type` depends on `near-sdk` version `4.1.1` (the broken
one), and since it's no longer shared, I removed that dependency so that
it can compile again.

Relates to #421
@morgsmccauley morgsmccauley linked a pull request Dec 17, 2023 that will close this issue
refactor: Extract registry types in to own crate #453
Merged
@morgsmccauley
Copy link
Collaborator Author

morgsmccauley commented Jan 9, 2024 •
edited
Loading

Initial Coordinator service is in #444, but there are some tasks which will be done in follow up PRs:

  • Provisioning - Coordinator should check the status of provisioning, and act when the state isn't as expected. This can be done after Expose endpoint to control provisioning #426 is implemented.
  • Retry recoverable errors - Any error will be propagated cause the error to exit, this includes connection errors to the Block Streamer and Runner services. As these are very likely to occur (across restarts) we should retry these errors.
  • Environment configuration - There are many hard coded values (endpoints, registry contract, etc.) which should be configurable via the environment.
  • Logging and Metrics
  • Handle executors and block streams in parallel

@morgsmccauley morgsmccauley linked a pull request Jan 16, 2024 that will close this issue
refactor: Configure coordinator/block-streamer via environment #503
Merged
@morgsmccauley morgsmccauley reopened this Jan 16, 2024
@morgsmccauley morgsmccauley linked a pull request Jan 17, 2024 that will close this issue
feat: Capture errors thrown within Coordinator #515
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@morgsmccauley morgsmccauley

Labels
component: Coordinator
Projects
None yet
Milestone
No milestone
1 participant
@morgsmccauley