Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Milestone: Reprovide Sweep in Go DHT #35

Open
yiannisbot opened this issue Aug 3, 2023 · 11 comments
Open

Milestone: Reprovide Sweep in Go DHT #35

yiannisbot opened this issue Aug 3, 2023 · 11 comments

Comments

@yiannisbot
Copy link
Member

ETA: 2023-11-30

Description: Tracking issue for the design, specification and implementation of Reprovide Sweep, as per the docs and discussions here:

Children:

@j-adel
Copy link

j-adel commented Jan 24, 2024

Hello. I was wondering if there are any updates on the timeline for the Reprovide Sweep. This feature is desperately needed for a large content providers.

@yiannisbot
Copy link
Member Author

Hi @j-adel, unfortunately, we've had to deprioritise this work stream due to some reorganisation of the team and its targets. Can you give some more details about the use-case and how pressing Reprovide Sweep is for your organisation? Given the right resources/capacity/funding, we could reconsider and bring this back up the priority list.

If it's more convenient, feel free to reach out to me at: [email protected]

@Rashkae2
Copy link

Rashkae2 commented Jan 24, 2024

I think it's already known, but if you have more than a few GB of content on an IPFS node, the default provide mechanism is unbelievable inefficient. It will generate lots of network traffic, keep open connections to hundreds of peers, and will not even be able to publish the content (to dht) fast enough to not fall off the network.

Accelerated DHT solves this, but is very problematic. The Hourly 10 minute sweep of the DHT will effectively DDoS your own network for a few minutes every hour. It is not something can can be left on a network connection used by other services/peers/home networks.

Personally, I have found the best way to work around both these limitations by implementing my own sweep via config files. Once every 24 hours or so, stop the deamon, swap in a config file with Accelerated dht, start the daemon for 45 minutes, (long enough to sweep the dht and republish all CID's).. stop deamon, swap in config file with Strategic providing, and restart. (config files need to be swapped because --config-file option doesn't seem to do anything whatsoever.)

This problem can't possibly he helping adoption.

@j-adel
Copy link

j-adel commented Feb 3, 2024

@yiannisbot That's unfortunate. I was commenting on behalf of one of the free access provider libraries. Several of them have tried to use IPFS as an alternative format for sharing books and articles as it lends itself quite nicely to many of the medium's demands. However, this issue in particular is causing issues in reproviding content to the network as there are hundreds of thousands of articles to reprovide every day. This issue is more apparent on lesser capable devices and for clients who want to help the service. I'm not sure what kind of resources can be offered but unfortunately financial doesn't seem likely.
However, I do want to emphasize how useful IPFS could be for sharing open-access documents if it resolves these large scale content providers and efficiency in dealing with them.

@pkieltyka
Copy link

hi all, is this ticket related to the reprovide sweep mention in the https://blog.ipfs.tech/2023-09-amino-refactoring/ blog post, "Amino (the Public IPFS DHT) is getting a facelift" ?

The blog post was very encouraging to solve some key issues on IPFS/Amino, is the roadmap changing as outlined in the blog post..?

@guillaumemichel
Copy link

is this ticket related to the reprovide sweep mention in the https://blog.ipfs.tech/2023-09-amino-refactoring/ blog post

yes

The blog post was very encouraging to solve some key issues on IPFS/Amino, is the roadmap changing as outlined in the blog post..?

@pkieltyka, yes our priorities have shifted, we need funding in order to continue this workstream. See #35 (comment)

@j-adel
Copy link

j-adel commented Feb 26, 2024

@guillaumemichel Is there the option to try to call for contributors to dig in and help out? Perhaps if the discussion of new features is more open and transparent more people would like to contribute to this and other features. I'd love to help out personally but I have zero experience with Go, unfortunately.

@guillaumemichel
Copy link

Yes, feel free to share it to potential contributors if you think they would be interested in working on the implementation 👍🏻

@pkieltyka
Copy link

pkieltyka commented Mar 16, 2024

I thought Protocol Labs was a well funded organization, and the blog post was published on the official IPFS blog, and seems this is a pretty critical component for IPFS to have a more robust network.. I'm confused whats going on over there.. IPFS has been in development for what, 15 years? .. why would the org behind IPFS post a roadmap on their official blog and then not follow through..? did they not consider the budget at the time of publishing the post?

@j-adel
Copy link

j-adel commented Mar 16, 2024

@pkieltyka 11 years in development, I think. And while this news is greatly disappointing, please remember that these are people doing hard work on an open-source service in a difficult economic environment.

@guillaumemichel
Copy link

@pkieltyka the team working on this development has nucleated out of Protocol Labs back in January 2024, and is now an independent entity. Hence, the team had to change its priorities to find funding.

Some background about Protocol Labs nucleations.

The blogpost was posted before the team was aware about nucleation timelines and conditions. I am sorry about the change of plans, we would have loved to pursue this workstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants