Welcome to storm-crawler Discussions!

[jnioche]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[rzo1]

Cool! Thanks for your work & efforts! 👍

[kkrugler]

Hi @jnioche - I've never tried GitHub Discussions before, this should be interesting, thanks 👍 .

I think I mentioned previously I've got a somewhat functional implementation of the IRLBot's DRUM (Disk Repository with Update Management), and I see from this SO discussion there are at least two other implementations, so seems like there's going to be a starting point for something scalable.

[jnioche]

hi @kkrugler; thanks for sharing. This should be useful for https://github.com/crawler-commons/url-frontier/discussions
As you might have seen, things are finally starting over there ;-)
will have a look at DRUM

[jnioche]

@kkrugler
https://github.com/mgledi/DRUMS -> requires records of equal sizes, so not really an option for storing URLs (and possibly associated metadata)
https://github.com/RovoMe/JDrum -> need to have a closer look but annoyingly it is not available as a Maven dependency

Would your implementation fit as a subproject of CrawlerCommons?