Concern about list growth

[StevenBlack]

👋🏻 Hello @krystian3w, @hawkeye116477, @azet12, and all contributors here.

As you know I include KADhosts.txt in all the various hosts files combinations produced by my repository at https://github.com/StevenBlack/hosts.

Together we achieve surprisingly massive distribution. I think this is mostly because people trust us given our track record and, collectively, our responsiveness.

Lately I am increasingly concerned about list growth. The growth rate is about 10,000 domains per month recently. Most growth comes from the KADhosts.txt component.

In the past KADhosts.txt grew but it also exhibited a healthy number of domain removals, from version to version.

Lately I have noticed that KADhosts.txt seems to be more of an "add-only bucket". In other words, I see fewer and fewer domain removals from KADhosts.txt.

I think all sources of hosts are prone to growing without bounds. Removing old domains is certainly harder than adding new domains. But culling domains is a sign of healthy curation.

Presently I don't have any suggestions, and I have no answers.

But if you guys become more aggressive in culling domains from KADhosts.txt, I would be very happy.

Here's the size of the base version of my hosts file, the one most people download, since 2018.

Cheers!

[StevenBlack]

@krystian3w is this supposed to be a "discussion" now? Or is this a "more polite" way of closing (burying?) an issue without acknowledgement or comment?

Because it is starting to feel like my concern isn't going to be acknowledged, and it's looking like no actual discussion is imminent. Consequently nothing is going to change without anyone actually saying that.

I need to know, disregarding what happens in the tail beyond the second standard deviation, is KADhosts an add-only bucket, now? Is curation stopped, now?

[krystian3w]

The proliferation is mainly due to the stalling removal of offline domains after the Polish CERT organisation = 2K+ domains to manually check.

[StevenBlack]

@krystian3w this is a graph of the size of the KADhosts.txt. The data points correspond to the dates I fetched KADhosts.txt for release builds.

[hawkeye116477]

Removing dead domains will be restored soon.

[hawkeye116477]

First step done. For now about 20k entries removed.

[StevenBlack]

Thank you @hawkeye116477 I noticed that 😄

[StevenBlack]

@hawkeye116477 and @krystian3w I want to thank you for the great effort you've made to reduce list size. I feel better 🙃 and I appreciate this very much!