refactor download file

[cnlangzi]

• download fragments by chunks in parallel requests (not Chunk on Protocols). It is https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
• download fragments in parallel requests from blobbers
• resume a download of a file that has been partially transferred.
• still need to support video-streaming . please check more detail with @moldis if there is any question. it is used on local or web video player
• pipe stream support (output bytes to io.Writer instead of file directly)
• support to reconstruct data on local with minimal fragments instead of download all fragments if it is possible . check here for detail

[cnlangzi]

some bugs need to be fixed too

• folder is not able to download. we need to get file list , and download them one by one
• please cancel download task if it is not able to download. eg auth ticket is incorrect or expired

[moldis]

@cnlangzi Not clear abound point #1. The ranged request has been implemented in the oldest algorithm, are you going to reimplement it?

[cnlangzi]

@cnlangzi Not clear abound point #1. The ranged request has been implemented in the oldest algorithm, are you going to reimplement it?

yes, it should work in parallel requests too.

[lpoli]

Task taken. Will start on Sunday

[cnlangzi]

@lpoli , please double check ReadMarker. it seem be not designed for parallel requests.

[lpoli]

1. Download fragments by chunks; So this is not supported by blobber yet? Please also explain about nature of parallel requests

2. What support for video-streaming?

3. Pipe stream is fine. Understood.

4. I checked Reed Solomon docs. It is to reconstruct missing shards. So you mean we need this for repairing data, right?

@cnlangzi @moldis @guruhubb

[cnlangzi]

1. Download fragments by chunks; So this is not supported by blobber yet? Please also explain about nature of parallel requests
2. What support for video-streaming?
3. Pipe stream is fine. Understood.
4. I checked Reed Solomon docs. It is to reconstruct missing shards. So you mean we need this for repairing data, right?

@cnlangzi @moldis @guruhubb

re 1: ignore parallel requests in a blobber download. Because parallel requests is not supported on ReadMarker.
re 2: https://docs.0chain.net/0chain/zbox-cli#video-streaming
re 4: for now, all fragments are downloaded. but it is unnecessary. for example, we can only download 4 fragments only, and reconstruct original file if an allocation is 4 data + 2 parity.

[lpoli]

Readmarker doesn't support parallel requests. Are you suggesting it is due to ReadCounter?

Regarding video-streaming; I am still not sure what support is required?

[cnlangzi]

https://docs.0chain.net/0chain/zbox-cli#video-streaming

re ReadMarker

ReadMarker is verified one by one like WriteMarker. it is great if it is supported. please double check code

re video-streaming

please check it on https://docs.0chain.net/0chain/zbox-cli#video-streaming. I don't know other documentation too. it seems is implemented by @moldis ? if yes, try to ping him for more detail

[lpoli]

@moldis said they are using block download.

Now it seems only work is to pipe download stream to some writer/reader.