Zookeeper history extremely bloated

[dgeswein-dlx]

Looking at Singularities usage of zookeeper, I see a LOT of activity in /singularity/slaves/[slaveId]/history/ . There are nearly 2 million entries. Our mesos cluster has external autoscaling, which means agents join and leave very regularly, so there is a lot of history. Will it harm anything if we clean out old entries from these history tables, or would this be problematic for Singularity?

[ssalinas]

Yes, anything that appears on the slaves page in the UI under dead or inactive can be safely removed from the list. There is an api for this as well which we use internally when autoscaling our own cluster. A DELETE to /api/slaves/slave/{slaveId} will remove it and clean that entry from zookeeper. That said, it would definitely make sense to add something that periodically cleans out the dead/inactive list

[dgeswein-dlx]

Ok, thanks, that's good to know. My situation is a little different in that I have active slaves with 129,000 entries in the history folder. Is there any harm in cleaning out old history entries and only keeping the most recent 10 in the slave history?

[ssalinas]

For dead/inactive slaves there is no harm in clearing any of them out. In fact we created endpoints to do so in bulk in #1795