add fail-safe mode by default to avoid double-sign when the new pod started while the old one is terminating but is still up

[andy108369]

Add fail-safe mode by default to avoid double-sign when the new pod started while the old one is terminating but is still up for any reason.

The fail-safe mode would check whether the last few recent blocks (5, 10 or 50) haven't been signed by the validator before starting it.
And if it does sign the recent blocks, it would then fail safely with some message such as: echo "WARNING!!! There is an active validator with the same pubkey is validating the blocks! Safely exiting..."

It has happened to Dimokus as he woke up to a jailed validator because of that.

[andy108369]

I remember I was using this script for Crypto Org chain for checking the last N signatures.
So its parts can be leveraged for the fail-safe check.

#!/bin/bash
# filename: ./scr
address=$1
hist=$2

if [ -z "$hist" ] || [ -z $address ] || [ ! $hist -gt 0 ] ; then
    echo ""
    echo -e "Usage:\n\t$0 validator_address number_of_history_blocks"
    echo ""
    echo -e "Example:\n\t$0 0E0DA7A5005429525A080C026D8DB0E7C85C5A11 20"
    echo ""
    echo "Hint: you can find your validator with the command:"
    echo -e "\tjq -r '.address' ~/.chain-maind/config/priv_validator_key.json"
    echo ""
    exit
fi

cheight=$(curl -s http://127.0.0.1:26657/commit | jq -r .result.signed_header.header.height)
lheight=$(expr $cheight - $hist)

sigs=0
for (( height=$lheight ; height < $cheight ; height++)); do
    JSON=$( curl -s http://127.0.0.1:26657/block?height=$height )
    sig=$( echo $JSON | jq -r .result.block.last_commit.signatures | grep -c $address )
    TISO=$( echo $JSON | jq .result.block.header.time | sed 's/\..*Z/Z/' | jq '( fromdateiso8601 )')
    TS=$( date -d "@$TISO" +'%Y-%m-%d %H:%M:%S' )
    echo -e "Local time: $TS\tBlock#: $height\tSigned: $sig"
    sigs=$(( $sigs + $sig ))
done

perc=$( echo "$sigs / ( $hist / 100 )" | bc -l )
missed=$(( $hist - $sigs ))

printf '\nValidator %s signed %.2f%s of the last %d blocks.\n' $address $perc '%' $hist
echo "Number of missed signatures: ${missed}."

Check if validator is signing the blocks:

$ jq -r '.address' ~/.chain-maind/config/priv_validator_key.json
666A630D5E13294072F2D1ECA7655CAA4F84289F

$ ./scr 666A630D5E13294072F2D1ECA7655CAA4F84289F 5
Local time: 2021-03-26 16:44:53 Block#: 22055   Signed: 1
Local time: 2021-03-26 16:45:00 Block#: 22056   Signed: 1
Local time: 2021-03-26 16:45:06 Block#: 22057   Signed: 1
Local time: 2021-03-26 16:45:13 Block#: 22058   Signed: 1
Local time: 2021-03-26 16:45:19 Block#: 22059   Signed: 1

Validator 666A630D5E13294072F2D1ECA7655CAA4F84289F signed 100.00% of the last 5 blocks.
Number of missed signatures: 0.

[Dimokus88]

I remember I was using this script for Crypto Org chain for checking the last N signatures. So its parts can be leveraged for the fail-safe check.

#!/bin/bash
# filename: ./scr
address=$1
hist=$2

if [ -z "$hist" ] || [ -z $address ] || [ ! $hist -gt 0 ] ; then
    echo ""
    echo -e "Usage:\n\t$0 validator_address number_of_history_blocks"
    echo ""
    echo -e "Example:\n\t$0 0E0DA7A5005429525A080C026D8DB0E7C85C5A11 20"
    echo ""
    echo "Hint: you can find your validator with the command:"
    echo -e "\tjq -r '.address' ~/.chain-maind/config/priv_validator_key.json"
    echo ""
    exit
fi

cheight=$(curl -s http://127.0.0.1:26657/commit | jq -r .result.signed_header.header.height)
lheight=$(expr $cheight - $hist)

sigs=0
for (( height=$lheight ; height < $cheight ; height++)); do
    JSON=$( curl -s http://127.0.0.1:26657/block?height=$height )
    sig=$( echo $JSON | jq -r .result.block.last_commit.signatures | grep -c $address )
    TISO=$( echo $JSON | jq .result.block.header.time | sed 's/\..*Z/Z/' | jq '( fromdateiso8601 )')
    TS=$( date -d "@$TISO" +'%Y-%m-%d %H:%M:%S' )
    echo -e "Local time: $TS\tBlock#: $height\tSigned: $sig"
    sigs=$(( $sigs + $sig ))
done

perc=$( echo "$sigs / ( $hist / 100 )" | bc -l )
missed=$(( $hist - $sigs ))

printf '\nValidator %s signed %.2f%s of the last %d blocks.\n' $address $perc '%' $hist
echo "Number of missed signatures: ${missed}."

Check if validator is signing the blocks:

$ jq -r '.address' ~/.chain-maind/config/priv_validator_key.json
666A630D5E13294072F2D1ECA7655CAA4F84289F

$ ./scr 666A630D5E13294072F2D1ECA7655CAA4F84289F 5
Local time: 2021-03-26 16:44:53 Block#: 22055   Signed: 1
Local time: 2021-03-26 16:45:00 Block#: 22056   Signed: 1
Local time: 2021-03-26 16:45:06 Block#: 22057   Signed: 1
Local time: 2021-03-26 16:45:13 Block#: 22058   Signed: 1
Local time: 2021-03-26 16:45:19 Block#: 22059   Signed: 1

Validator 666A630D5E13294072F2D1ECA7655CAA4F84289F signed 100.00% of the last 5 blocks.
Number of missed signatures: 0.

Looks a awesome!
I think, this can implementation at running service in deployment!

[andy108369]

The only caveat is that one would have to use some known RPC node to query as one can't just query itself 127.0.0.1:26657 before starting the node...

I am wondering, maybe tendermint already has this kind of check built-in?

[Dimokus88]

i found value double_sign_check_height = 0 in config.toml . But i not sure about reliability.
But this parameter will look good as an addition to the script.

[andy108369]

i found value double_sign_check_height = 0 in config.toml . But i not sure about reliability. But this parameter will look good as an addition to the script.

yeah, it looks like exactly that! thank you!

https://docs.tendermint.com/v0.34/tendermint-core/configuration.html

[consensus]
# ...

# How many blocks to look back to check existence of the node's consensus votes before joining consensus
# When non-zero, the node will panic upon restart
# if the same consensus key was used to sign {double_sign_check_height} last blocks.
# So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic.
double_sign_check_height = 0

[Dimokus88]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned.
And double containers at the moment are quite common among providers.

[andy108369]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned. And double containers at the moment are quite common among providers.

I don't think the ban is permanent. Validator can always issue the tx slashing unjail to get out of the jail.

[Dimokus88]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned. And double containers at the moment are quite common among providers.

I don't think the ban is permanent. Validator can always issue the tx slashing unjail to get out of the jail.

A double signature is punishable by an eternal ban, it looks like the date unjail 9999-12-31

[andy108369]

For some reason some had a problem with that option (double_sign_check_height) enabled.
So I think it needs to be tested.

https://discord.com/channels/747885925232672829/751246770885099533/968578447884550146

[andy108369]

So I think it needs to be tested.

I've just tested it with SIFNODED_CONSENSUS_DOUBLE_SIGN_CHECK_HEIGHT=10, it worked very well:

11:37PM INF found signature from the same key height=9303106 idx=41 module=consensus sig={"block_id_flag":2,"signature":"REDACTED","timestamp":"2022-11-05T23:35:43.456919786Z","validator_address":"REDACTED"}
panic: Failed to start consensus state: found signature from the same key