Skip to content

Concurrent Read With ReadSize

Jump to bottom
Laxmi Prasad Oli edited this page Mar 26, 2022 · 1 revision

Recently we changed protocol a bit by modifying ReadMarker(RM) fields. Previous RM used read counter in its fields to calculate and verify read token redeeming. Now we use read-size instead which avoids race condition with read counter. Below I would like to discuss problem with Read Counter and its solution with Read-Size and also the merits and demerits of both.

Read Counter

So you are a client and want to read a file data. You'd request the required data with start block and end block. Note that a block is always a 64KB of data. Consider your previous read counter is r. The difference endblock - startblock (dR) is added to previous read counter r, and sums to, say c. This c is added to the RM along with other fields and data is requested with this signed RM.

Blobber upon receiving download request does the following:

  1. Verifies the signature of request
  2. Verifies signature of RM
  3. Verifes current RM's read-counter is in sync with previous RM's read-counter
  4. Store current RM as a latest RM.

Note: A RM and its counter is not shared with other client

Now consider the client having two processes P1 and P2 requesting files.

  1. P1 request for latest read-counter, so does P2. Both process has received same read-counter, rc
  2. P1 added n to rc and P2 added m to rc. Both signed the RM and request data to the blobber.
  3. Say P2 request reached blobber first so it got served and blobber updated the read-counter to m + rc.
  4. P1 request reached blobber but read-counter is not in sync with the latest read-counter, so gets rejected with latest RM in the response
  5. P1 updates its read-counter and re-sends to blobber. If P2 had already updated read-counter then it will fail again. Thus stalling process P1.
  6. Read is therefore sequential.

So having multiple processes to read data would be meaningless. And this limitation is directly from the protocol

Redeeming read-marker

To redeem read-marker from the read pool of the client, blobber would send latest RM and can redeem all the pending tokens from the single RM.

Read Size

To allow concurrent read possible from the protocol, we changed RM's field from read-counter to read-size. So you are a client and want to read a file data. You'd request the required data with start block and end bock (dR), like previous. Requested read size is dR*64KB which is put in RM and signed.

Blobber upon receiving download request does the following:

  1. Verifies the signature of request
  2. Verifies signature of RM
  3. Vefifies against readmarker size < requested size
  4. Store current RM as new row.

Redeeming read-marker

To redeem read-marker from the read pool of the client, blobber would send all the RMs to redeem pending tokens of each RM

Comparison between ReadCounter and ReadSize

As we see with read-counter concurrent read from the blobbers is not possible. However there are some merits and demerits of both read-counter and read-size.

With read-counter a blobber can save single ReadMarkerEntity to redeem all the pending read-tokens. However this is not the case for read-size. Each RMs need to be stored in database as new row and can be deleted upon redeeming. This is not a big deal for a blobber as it also gets frequently cleared upon redeeming.

The subject of major concern is increase in number of requests to the blockchain. Read request is a common file operation and will result in massive requests generation. So for million blobbers, handling 100 read-requests per second in average, this would mean Hundered million requests in one go.

If we consider every blobber had worker setup to 10 second for redeeming read-tokens then in worst case i.e. all blobbers send redeem request at the same time, it would mean One Billion requests at one go.

Solution to Abundant Read-requests

I think we need to incentivize blobber for sending less requests as possible. We need to provide option for sending batch-request to redeem tokens. So this would be like:

  1. Blockchain provides interface to process redeem requests in batch
  2. Blobber's worker collects, say 1000, RMs and then sends to blockchain to redeem all 1000 RMs.

The other desirable possibility would be:

  1. Blobber collects all the RMs of a client.
  2. Client provides some form of proxy key to a blobber.
  3. Blobber uses this proxy key to merge all the RMs of client into one signed RM and sends to blockchain
  4. Blockchain would verify this new signature and transfer tokens from client's read-pool.

Even with Read-counter, blobber can choose to store each ReadMarkers and redeem tokens with each RMs, though it could have done it with single latest-RM, so as to attack blockchain. So we need to incentivize blobbers for sending less requests i.e. fees for transactions, f(t1 + t2 + t3 + ....) should be lesser than f(t1) + f(t2) + f(t3) + ...

Clone this wiki locally