Check for gaps in block numbers and throw if found during migration

[alecps]

Fixes https://github.com/celo-org/celo-blockchain-planning/issues/832

Tested locally using baklava datadir with gaps

[palango]

Looks good, but I wonder if we should have a separate command to check a datadir before the migration, independent of the pre-migration. What do you think?

[alecps]

@palango Interesting idea. I need to think about that a bit more. My first reaction is that it would add an extra step + time to the migration process. I'm also trying to work out if it's even possible to have gaps in ancient blocks, as the ancient db is append only. Having a separate script could be a quick way to bring extra piece of mind. Will give this some more thought tomorrow

[palango]

@palango Interesting idea. I need to think about that a bit more. My first reaction is that it would add an extra step + time to the migration process. I'm also trying to work out if it's even possible to have gaps in ancient blocks, as the ancient db is append only. Having a separate script could be a quick way to bring extra piece of mind. Will give this some more thought tomorrow

Not a requirement, but it might be too late for a full resync if people run the pre-migration shortly before the schedules block.

[piersy]

My feeling is that a separate command for this is unnecessary, if we want people to check this earlier then we should just message them to run the pre-migration earlier.

[alecps]

I've observed some strange behavior that I want to look into more. It seems there may be an earlier gap that this branch does not detect that is preventing blocks from freezing beyond block 5,002,000. This branch only throws on a later gap that starts at block 5835918. It's possible this is a partial gap, where only receipt data is missing. Will re-open this for review when I've gotten to the bottom of what's going on

[alecps]

I added checks for all the necessary data (receipts, headers, total difficulty, etc.) for both ancient and non-ancient blocks. The script before only checked for gaps in headers. I haven't been able to test with gaps in ancients because I haven't been able to get a node to freeze blockls beyond the first gap. It's unclear whether it's possible to actually have a gap in the ancient data.

[alecps]

@palango After thinking about it more, I do actually think it would be nice to have a command for testing data integrity. Though it isn't strictly necessary, it might be appreciated by anyone who runs into data corruption issues and wants to quickly check whether the database they're using is okay before trying a second time. We could even consider recommending that people run it first as a precaution, but that might not be necessary. I think it might make sense to add in a follow up PR so that this one doesn't get too big. What do you think?

[palango]

@palango After thinking about it more, I do actually think it would be nice to have a command for testing data integrity. Though it isn't strictly necessary, it might be appreciated by anyone who runs into data corruption issues and wants to quickly check whether the database they're using is okay before trying a second time. We could even consider recommending that people run it first as a precaution, but that might not be necessary. I think it might make sense to add in a follow up PR so that this one doesn't get too big. What do you think?

Totally agree, this can be added separately. And maybe this doesn't need to be coupled to the migration at all, even though it would make sense to reuse code if possible.