Genesis ledger sync requested by Mina node

[georgeee]

Description

A seed node was restarted and spent some hours being out-of-sync.

After that, it was restarted again and peers were set in such a away to ensure connectivity to some peers with the most recent blockchain, among them whale-1.

Seed was not able to get the best tip immediately, but successfully loaded frontier, hence it got to transition frontier sync (not bootstrap). Seed figured out the tip of the frontier was out of date and launched sync_local_state with target jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF.

For some reason local state of the seed contains jwE6rXod6YKoPqGk2cqu9tuDXDtTm7v4pQhRW7Y7fv78MNh88Sx instead of jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF reported by whale-1 and frontier for the current epoch ledger, while agreeing on the next epoch ledger (jxR3LnYbsVuddwJUpzk4L5vAG1GYrZHJY5de1osHHVNHq8BzSWL).

This is epoch 1, so one would expect jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF to be available locally (as part of genesis package), and agree with the value from frontier's best tip.

Question: how is it possible that local state and frontier disagree on the current ledger and agree on the next ledger?\

All files extracted from seeds file system (logs, dbs): https://drive.google.com/file/d/1X9O0pKJpgQ8vpg6rkH2ed4RfT_Z7y-Wz/view?usp=sharing

Steps to Reproduce

Not clear

Expected Result

Synchronization happens

Actual Result

Synchronization is stalled forever.

Seed attempts to synchronize to one of genesis ledgers and her request to the whale-1 gets rejected (because this is an expected behavior for the genesis ledger):

[2024-12-12 13:29:02.288549] Requested_unknown_item (Sync ledger query with $hash, query: ["Num_accounts"], with error: $error)
$sender_id = <seed>
$hash = "jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF"
$error = {"string":"Refusing to answer sync ledger query for ledger_hash: 22812639317355063860959747045308626384239266729484000542770784884096638231286"}

Daemon version

7aefaa0

How frequently do you see this issue?

Rarely

What is the impact of this issue on your ability to run a node?

Blocker

Status

Max observed block height:              1
Max observed unvalidated block height:  0
Local uptime:                           4h30m12s
Chain id:                               97a481b4cd4289b7e38d9e737c88ca0ca1bbc3d174bbaa30a8578f576deac60e
Git SHA-1:                              7aefaa07b0c89e7f0b0ef2cffb985d2e3a0b08a2
Configuration directory:                /root/config
Peers:                                  4
User_commands sent:                     0
SNARK worker:                           None
SNARK work fee:                         100000000
Sync status:                            Bootstrap
Block producers running:                0
Coinbase receiver:                      Block producer
Consensus time now:                     epoch=1, slot=3052
Consensus mechanism:                    proof_of_stake

Additional information

A log line containing some information from the seed:

[2024-12-12 12:57:02.217613]Consensus__Proof_of_stake: Syncing local state; requesting $num_requested snapshots from peers
$host = "37.27.59.202"
$local_state = {"staking_epoch_snapshot":{"ledger_hash":"jwE6rXod6YKoPqGk2cqu9tuDXDtTm7v4pQhRW7Y7fv78MNh88Sx","delegators":{}},"next_epoch_snapshot":{"ledger_hash":"jxR3LnYbsVuddwJUpzk4L5vAG1GYrZHJY5de1osHHVNHq8BzSWL","delegators":{}},"last_checked_slot_and_epoch":{}}
$peer_id = "12D3KooWEaF1ED3t2FyLhPirybL83puLBSUfFTzeeTq16pCZyAEi"
$port = 10003
$peer_id = "12D3KooWEaF1ED3t2FyLhPirybL83puLBSUfFTzeeTq16pCZyAEi"
$port = 10003
$target_ledger_hash = "jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF"

[georgeee]

Okay, so jwE6rXod6YKoPqGk2cqu9tuDXDtTm7v4pQhRW7Y7fv78MNh88Sx is the staking ledger of genesis block, while jxyDwPtX5YeXVuRWrqG6vtTfmiwz5gEpehksSZizRmBtRxgvwkF is "next" for genesis, as expected

How could the local state happen to contain current ledger as staking of epoch 0 and next ledger as next of epoch 1?

[georgeee]

Same issue is observed on a number of other nodes in the cluster (plain-1 for example).