Skip to content

3. Job Processing Flow

Jump to bottom
creativeid00 edited this page Dec 22, 2021 · 17 revisions

Processing Flow Diagram

All queueing of Jobs/MacroJobs/MicroJobs are using a locking queue with timeout to be able to handle node drop outs/disconnects when joining a room for a MicroJob or creating a MacroJob room. When querying open jobs (Jobs,MacroJobs,MicroJobs) it is the union of jobs that have exceeded the timestamp lock & jobs that are have not been previously timestamp locked.

Job Placement

DApp creates a Job:

  1. DApp specifies Job code (shader/Tensorflow/OpenCL) URI to source from (IPFS/SFTP/WS/HTTPS).
  2. DApp specifies Job data URI to source from (IPFS/SFTP/WS/HTTPS).
  3. DApp fills a JobInfo structure, including Job slicing parameters and a seed for random number generator and places it into CRDT.
  4. Dapp Slices Job with slicing parameters and creates MacroJobs & MicroJobs with random ID's from random seed saving into the CRDT DB
  5. DApp doesn't need to wait until job completion as the Polygon (Matic) BlockChain Smart Contract will create a transaction block on Verify Job Function/Release Escrow call. The Dapp can poll into Polygon (Matic) blockchain to check job completion.

Processing node initialization

  1. Once a Job processor is started it is linked/synced to CRDT DB.
  2. Create or Join Jobs PubSub Channel (first bootstrap node should do this)

Job Processing

  1. Job processor asks for open jobs via the PubSub Channel, if none exist, loop here (step 3)
  2. Join Job{ID}/MacroJob PubSub channel

MacroJob Processing

  1. Job processor asks for open MacroJobs from the PubSub Channel (because CRDT DB might not be synced) Job processor asks for open MacroJobs from the PubSub Channel. CRDT is not used as MacroJobs storage.

  2. if MacroJob open, Join Job{id}/MacroJob{id} Channel, goto 'MicroJob Processing'.

    *** This can also in the future check for locality of processing nodes

  3. If no MacroJobs open goto 'Job Processing'.

  4. Mark CRDT DB with timeout timestamp/lock and create channel Job{id}/MacroJob{id} and notify PubSub Create in-memory MicroJob queue, set itself as the current queue owner. To create a validation MicroJob a random number generator should be used with seed equal to Job input seed + MacroJob index {id}

  5. Create PubSub channel Job{id}/MacroJob{id} and notify over Job PubSub channel about the created MacroJob channel.

  6. Job Processor joins MacroJob PubSub Channel

  7. Go to MicroJob Processing

MicroJob Processing

  1. Job Processor asks for open MicroJobs via PubSub, if none goto 'MacroJob Processing' Job Processor requests for in-memory MicroJob queue via PubSub. Once a Microjob queue owner receives the request it marks the requestor as the current queue owner and sends the queue to PubSub. Note: All Processors receive the updated in-memory queue but only the current queue owner can operate with it.
  2. Job processor marks MicroJob CRDT DB with timeout timestamp/lock and notifies PubSub Channel Job processor receives the MicrJobs queue. If it is marked as the current queue it checks if there is an open Microjob and marks it with with timeout timestamp/lock and notifies PubSub Channel
  3. Job processor applies the code algorithm for data blocks that are specified in the MicroJob.
  4. Job processor writes MicroJob processing results into CRDT when the MicroJob processing is finished. a. Data should be written to IPFS b. CRDT record should be written with Job{ID}/MacroJob{ID}/MicroJob{ID} = data: { wallet address, hashes of processed blocks } + hash (Job{ID}/MacroJob{ID}/MicroJob{ID} & data) c. Job processor publishes to Job channel, 1 MicroJob is complete
  5. Job Processor checks if final MicroJob for a Job, if so jump to Job Finalization
  6. Job processor checks if there are non-processed Micro Jobs and repeats p.1-6 until all Micro Jobs are processed.
  7. Go to 'Macro Job Processing'

Job Finalization.

  1. When a Job processor finishes a MicroJob processing it checks if Job's MicroJob count finished is equal to MicroJob count.

  2. if not goto Job Processing

  3. Last MicroJob scans MacroJobs and finds verifier hashes by checking all MicroJob hashes to see which matches the other 10 nodes hashes

  4. It sends this hash to the Smart Contract along with wallet addresses to release the Escrow

  5. Smart Contract ZkSnark algorithm can verify because {ID}'s are generated from RandomSeed, encrypted with zkSnark

    *** If smart contract fails to verify. Culprit can be figured out, by getting index of MacroJob verifier node and checking hashes against other non-verifier nodes to find the mismatch.

Clone this wiki locally