PROPOSAL: Increase validator offline threshold to 7.6 hours

[zanglang]

Currently, a validator node will be jailed if it fails to sign 2500 blocks within a 5000 block window, or approximately 3 1/2 hours given the current average block time of 5.5 seconds. Whilst the Crypto.org chain does not implement downtime slashing, the downtime window may be too short considering that:

a) The majority of other large Cosmos chains have a significantly higher offline thresholds, even for those who are highly dependent on validator security for on-chain infrastructure:

Gravity Bridge - 5000 blocks ~ 8 hours
Cosmos Hub, Juno, Akash - 9500 blocks ~ 16 hours
Secret - 11250 blocks ~ 18 hours
Band - 28500 blocks ~ 21 hours
Axelar - 17500 blocks ~ 29 hours

b) The average block time is expected to continue dropping as more validators adopt the 3s "time_commit" parameter change merged in April 2021. This value was 6.2 seconds in November 2021, and 5.5 seconds at time of writing (source: http://bit.ly/3OGwNrM). Assuming full adoption of the new timeout_commit window brings this value down to ~4 seconds, the downtime window would be ~160 minutes.

c) Even though the chain does not implement downtime slashing, we may observe that due to the longer jailing times and Crypto.com Defi Wallet's ordering algorithms, previously jailed validators tend to be heavily penalized by massively shedding delegations, leading to them falling out of the active set.

Looking at the above points, I would like to propose raising the signed blocks window from 50% of 5000 blocks to at least 10000 blocks, i.e. approximately 7.6 hours.

min_signed_per_window: "0.500000000000000000"
signed_blocks_window: "10000"

[zkaizhi]

Thank you @zanglang, for submitting this proposal.

We are very much open to look at how we can make improvements to support an acceptable margin for the validator offline threshold.

Compared to the other cosmos ecosystem chains, we indeed seem to have a slightly stricter offline threshold that we require our validators to meet. Let us provide some more context on how we came to set these initial parameters:

• As set in our business continuity process (BCP), we have defined business critical processes to maintain a 99.95% high availability, and should therefore stay within a ~4H recovery time objective (RTO) (see for example: https://en.wikipedia.org/wiki/High_availability).
  If we look at the current block time and time_commit settings, this becomes roughly a ~3.8 H offline threshold.

• We currently do not enforce any slashing for downtime, but we do have a longer jail time as compared to other major cosmos ecosystems, to balance this out.

So we are open to discuss how to raise this offline threshold to around ~4H, either by:

1. Raising the signed_blocks_window
   Raising the signed_blocks_window from the current “5000” to “6000”. This will increase the threshold to ~4.6H, given the current average block time of 5.5 seconds. The consideration that block times (through time_commit) could further be decreased over time, is indeed possible. Therefore, we can always further discuss parameter changes to re-assess in case this scenario plays out.

min_signed_per_window: "0.500000000000000000"
signed_blocks_window: "6000"

2. Decreasing the min_signed_per_window
   Decreasing the min_signed_per_window to 0.4, this will result in 4.6H, but may have the negative consequence of “lowering the signing quality” during the same time window.

min_signed_per_window: "0.400000000000000000"
signed_blocks_window: "5000"

Finally, with respect to the DeFi wallet ranking, we are actively working on updating the uptime calculation, to be based on the last x number of blocks signed, instead of blocks since inception. Next to this, we will take into account validators that have been flagged for heavy misconduct. This should yield more fair ranking results, rewarding validators for their recent uptime, such that users can make a good selection when delegating to validators.

Hope this helps,
Feel free to share your thoughts on this proposal,
The Crypto.org Team

[fernando-galaxystaking]

I would like to take up the topic again, because I think it has been lost.

It honors CDC that an RTO of 4h (high availability) should be maintained in this chain. But is this necessary when looking at each individual validator? The service does not generally fail due to the failure of a single validator. The CDC validators are an exception due to their high voting power. These could cause a problem.

Everyone should be able to take a break of 3-5 hours. As a father with a full-time job, I personally always notice this on the plane or even while swimming. I always do both with a feeling of disturbance.
In the other chains where we are active, this value is around 48 hours.

Even the relationship between allowed downtime and jail time doesn't really fit either.
The most problems should be solved after 5-6 hours. However, after 3.5 hours you have to go to jail for 24 hours.
However, I don't want to completely absolve the responsibility of the validators either.

I would even suggest the following:

  "signed_blocks_window": "10000"
  "min_signed_per_window": "0.100000000000000000"
  "downtime_jail_duration": "43200s"

This would raise possible downtime to ~ 12.5 hours and the subsequent time in jail to 12 hours.
Those would still be tougher parameters than in the other chains.

[marco-novelli]

I agree with 12.5 hours downtime

medusanode

[fernando-galaxystaking]

After some feedback from other validators of our chain and the fact that CDC wants an RTO of ~4h and thus high availability, I am changing my recommendation to the following:

"signed_blocks_window": "5000"
"min_signed_per_window": "0.100000000000000000"
"downtime_jail_duration": "43200s"

Now the signed_blocks_window would remain the same. Just the min_signed_per_window would change.
But i didn't change my mind with the downtime_jail_duration, since it hasn't been discussed yet.