chore: Task Workers Revamp #796

jsantell · 2024-01-24T19:53:25Z

Decoupled Worker Tasks

Worker tasks are now abstracted such that routes now send jobs via a JobClient decoupled from where the work is performed. A standalone noosphere_gateway::jobs::worker_queue module provides the work distribution logic, with each processor receiving a GatewayJobContext that provides access to sphere contexts, IPFS, and name resolvers when processing a job.

Worker jobs can now queue up additional tasks on success (IPFS Syndication -> Name Record Publish) (LinkRecord can propagate before blocks are available #606).
Jobs can be retried upon failure.
Some timeout logic has been implemented, but not yet ready.
Republishing name records is now its own job type. Eliminates the need for the scheduler to resolve the latest name record.

Minimize external queries

GatewayManager and routes were updated to minimize GM queries (#753).

In addition to a single, cacheable baseline query for scope, unauthenticated routes now has one less external query, and authenticated routes also has one less external query, with another query moved behind authorization.

Ultimately handler logic should not be changed, other than moving Push handler's queueing of name record publishing behind IPFS syndication.

Caveats/Concerns/Enhancements

We should do some profiling to measure lock contention across workers where appropriate amongst shared resources like KuboClient and the name resolver.
Job processors may need to be rethought in order to better take advantage of retries e.g. classifying a failure between retriables, invalid state/unretriable, and failed-but-ok-to-lazily-try-again-later.
~~Implement timeouts in worker queue.~~ Timeouts now implemented
Ipfs syndication revision parameter now optional (unused? to confirm)
Ensure name record provided to PUSH is still valid after Ipfs syndication completes due to record expiry

github-actions · 2024-01-31T19:11:02Z

Test flake analysis

No flakes detected 🎉

status	platform	features	toolchain
🟢	`ubuntu-latest`	`test-kubo,headers`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers,rocksdb`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers`	`nightly`
🟢	`macos-13`	`test-kubo,headers,rocksdb`	`stable`
🟢	`macos-13`	`test-kubo,headers`	`stable`
🟢	`windows-latest`	`test-kubo,headers`	`stable`

jsantell · 2024-02-01T19:37:01Z

Added explicit checks on existence of name records after syncing changes in tests, possibly clearing up a handful of intermittent failures.

Windows builds are having an unrelated issue: #805

jsantell · 2024-02-05T20:50:21Z

Added time out support, recreating the hanging worker thread and attempting to resubmit the job. Of note, the SingleTenantJobClient is set to only 1 attempt for now, until we normalize what an error is in the job processors

jsantell · 2024-02-05T22:42:30Z

rust/noosphere-gateway/src/single_tenant/job_client.rs

+const JOB_RETRIES: usize = 1;
+/// How many seconds before a job is considered broken,
+/// and potentially restarted.
+const TIMEOUT_SECONDS: u64 = 180;


Syndicating large histories could be longer than this timeout, and notably, name records' publishing timeframe is currently 2 minutes.

jsantell · 2024-02-20T20:23:44Z

Rebasing on #815

cdata · 2024-02-26T17:02:59Z

Ipfs syndication revision parameter now optional (unused? to confirm)

IIRC we now syndicate full spheres at a time to IPFS, so it isn't necessary to track this.

cdata

The bones look excellent, but I have some light opinions about some of the flesh :)

Great work so far!

rust/noosphere-gateway/src/jobs/worker_queue/mod.rs

rust/noosphere-gateway/src/jobs/worker_queue/queue_thread.rs

rust/noosphere-gateway/src/jobs/worker_queue/queue.rs

rust/noosphere-gateway/src/jobs/worker_queue/worker.rs

rust/noosphere-gateway/src/jobs/job_processor.rs

rust/noosphere-ns/src/utils.rs

rust/noosphere/tests/distributed_basic.rs

rust/noosphere-gateway/src/jobs/job_processor.rs

rust/noosphere-ns/src/utils.rs

rust/noosphere-gateway/src/jobs/worker_queue/queue.rs

rust/noosphere-gateway/src/jobs/worker_queue/queue_thread.rs

rust/noosphere-gateway/src/jobs/worker_queue/worker.rs

rust/noosphere-gateway/src/single_tenant/gateway_manager.rs

rust/noosphere-gateway/src/jobs/worker_queue/queue_thread.rs

jsantell · 2024-02-27T16:05:59Z

Updates (commit):

Changes/cleanups from feedback
Remove vestigial revision in IpfsSyndication job
Reduce indirection from cloning (e.g. use outer Arcs, not inner)
s/ContextResolver/SphereContextResolver
s/NameResolverExt/NameResolverPoller
s/WorkerQueueThread/WorkerQueueOrchestrator
More documentation for worker_queue
Updated retries to explicitly be "retrying" (defaulting to 0) rather than "number of attempts" (defaulting to 1)

cdata

LGTM, thanks for the hard work on this!

* Decouple worker tasks to run independently of the gateway. Fixes #720. * Reduce GatewayManager queries in the gateway handlers on request. Fixes #723. * Introduce polling helpers in tests ensuring async name record publication occurs before continuing the test.

jsantell force-pushed the task-workers branch 6 times, most recently from b8f9f6b to a24a2af Compare January 31, 2024 18:31

jsantell changed the title ~~[WIP] Task Workers~~ chore: Task Workers Revamp Jan 31, 2024

jsantell force-pushed the task-workers branch from a24a2af to 856263e Compare January 31, 2024 18:34

jsantell marked this pull request as ready for review January 31, 2024 18:50

jsantell requested a review from cdata January 31, 2024 18:50

jsantell force-pushed the task-workers branch 3 times, most recently from 959058a to 8edb673 Compare February 1, 2024 19:20

jsantell requested review from cdata and removed request for cdata February 1, 2024 19:55

jsantell force-pushed the task-workers branch from 8edb673 to 40fae49 Compare February 5, 2024 20:47

jsantell force-pushed the task-workers branch 2 times, most recently from a6ca055 to 30c88a1 Compare February 5, 2024 22:29

jsantell commented Feb 5, 2024

View reviewed changes

jsantell mentioned this pull request Feb 6, 2024

chore: Use default openssl on github CI and do not fail early on tests #805

Merged

jsantell force-pushed the task-workers branch 7 times, most recently from 8b2440e to c79d784 Compare February 12, 2024 18:34

jsantell force-pushed the task-workers branch from c79d784 to 5f120de Compare February 20, 2024 20:23

cdata suggested changes Feb 26, 2024

View reviewed changes

jsantell commented Feb 26, 2024

View reviewed changes

jsantell force-pushed the task-workers branch from 370d2fd to 8cadcfc Compare February 27, 2024 16:00

jsantell requested a review from cdata February 27, 2024 16:11

jsantell force-pushed the task-workers branch from 8cadcfc to 59fb09a Compare February 27, 2024 16:34

cdata approved these changes Feb 29, 2024

View reviewed changes

jsantell force-pushed the task-workers branch from 59fb09a to 1359c00 Compare February 29, 2024 17:49

jsantell force-pushed the task-workers branch from 1359c00 to b98fe6d Compare February 29, 2024 17:53

jsantell merged commit 887ebff into main Feb 29, 2024
21 of 22 checks passed

jsantell deleted the task-workers branch February 29, 2024 18:27

jsantell mentioned this pull request Mar 11, 2024

GatewayManager should conservatively use extract_counterpart() #753

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Task Workers Revamp #796

chore: Task Workers Revamp #796

jsantell commented Jan 24, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

jsantell commented Feb 1, 2024

jsantell commented Feb 5, 2024

jsantell Feb 5, 2024

jsantell commented Feb 20, 2024

cdata commented Feb 26, 2024

cdata left a comment

jsantell commented Feb 27, 2024 •

edited

Loading

cdata left a comment

chore: Task Workers Revamp #796

chore: Task Workers Revamp #796

Conversation

jsantell commented Jan 24, 2024 • edited Loading

Decoupled Worker Tasks

Minimize external queries

Caveats/Concerns/Enhancements

github-actions bot commented Jan 31, 2024 • edited Loading

Test flake analysis

jsantell commented Feb 1, 2024

jsantell commented Feb 5, 2024

jsantell Feb 5, 2024

Choose a reason for hiding this comment

jsantell commented Feb 20, 2024

cdata commented Feb 26, 2024

cdata left a comment

Choose a reason for hiding this comment

jsantell commented Feb 27, 2024 • edited Loading

cdata left a comment

Choose a reason for hiding this comment

jsantell commented Jan 24, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

jsantell commented Feb 27, 2024 •

edited

Loading