Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helios fails to sync (invalid sync committee period) #388

Open
EzequielPostan opened this issue Oct 11, 2024 · 3 comments
Open

Helios fails to sync (invalid sync committee period) #388

EzequielPostan opened this issue Oct 11, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@EzequielPostan
Copy link

Hello, I am experimenting with Helios and noticed something that I need to understand and may be a bug.
I would like to use Helios to validate historical data, which means that I am interested in running Helios from the oldest checkpoint possible. Moreover, as time goes by, and more checkpoints are created, I will still want to start from that oldest possible checkpoint.

I started then Helios like this (I am testing changes to the code, so I am building the code)

$ cargo run  -- \
      --network mainnet  \
      --consensus-rpc https://www.lightclientdata.org \
      --execution-rpc $EXEC_RPC_WITH_API_KEY

I ran it in that way to trigger Helios to use the BaseConfig checkpoint

0xc7fc7b2f4b548bfc9305fa80bc1865ddc6eea4557f0a80507af5dc34db7bd9ce

which seems to be for this block at slot 8613568 (from Sep-24-2019 07:15:20 PM +UTC) which looks fine. And I know that in theory, I am be able to run with --checkpoint and use that checkpoint later to fix that as a starting one.

Now, the issue I have is that Helios fails to sync. This happens at "around" (below is the last update I see validated by Helios in my prints) the point where the finality update has

Attested header slot:10152347
Finalized header slot:Some(10152256)

Here are the normal logs:

$ cargo run  -- \
      --network mainnet  \
      --consensus-rpc https://www.lightclientdata.org \
      --execution-rpc $EXEC_RPC_WITH_API_KEY
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.25s
     Running `target/debug/helios --network mainnet --consensus-rpc 'https://www.lightclientdata.org' --execution-rpc 'https://eth-mainnet.g.alchemy.com/v2/<HERE IS MY API KEY>'`
ConsensusClient RPC: https://www.lightclientdata.org
Default checkpoint leaded 0xc7fc7b2f4b548bfc9305fa80bc1865ddc6eea4557f0a80507af5dc34db7bd9ce: 
2024-10-11T13:08:47.772539Z  INFO helios::rpc: rpc server started at 127.0.0.1:8545
2024-10-11T13:08:48.938388Z  WARN helios::consensus: checkpoint too old, consider using a more recent block
2024-10-11T13:08:53.951582Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:54.452449Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:54.954452Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:55.465156Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:55.972255Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:56.486164Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:57.000883Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:57.519062Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:58.033118Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:58.549821Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:59.079259Z  INFO helios::consensus: sync committee updated
2024-10-11T13:08:59.616485Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:00.144696Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:00.687047Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:01.226089Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:01.762875Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:02.312410Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:02.853935Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:03.398686Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:03.975107Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:04.573795Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:05.105404Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:05.627512Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:06.230685Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:06.800106Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:07.364466Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:07.934522Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:08.549117Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:09.127214Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:09.677936Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:10.204541Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:10.734456Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:11.259440Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:11.790895Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:12.316871Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:12.863380Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:13.392050Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:13.919872Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:14.453015Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:14.989828Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:15.570070Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:16.121032Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:16.644695Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:17.188080Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:17.713029Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:18.237662Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:18.764083Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:19.289330Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:19.814826Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:20.339830Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:20.864856Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:21.386316Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:21.934267Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:22.459005Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:22.995888Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:23.527148Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:24.052272Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:24.565964Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:25.090404Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:25.622298Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:26.147243Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:26.675490Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:27.207140Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:27.730732Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:28.259521Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:28.783406Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:29.336588Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:29.900849Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:30.468631Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:31.028030Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:31.593209Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:32.145113Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:32.705123Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:33.272768Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:33.852040Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:34.445757Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:35.031867Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:35.591344Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:36.138531Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:36.685892Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:37.256491Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:37.830011Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:38.377169Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:38.916183Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:39.475376Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:40.035589Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:40.576541Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:41.143514Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:41.707088Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:42.250476Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:42.820803Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:43.397150Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:43.957646Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:44.526665Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:45.085213Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:45.651808Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:46.213737Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:46.776256Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:47.348839Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:47.910653Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:48.469736Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:49.020085Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:49.581192Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:50.141864Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:50.713124Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:51.265271Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:51.816961Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:52.371762Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:53.002831Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:53.561575Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:54.121904Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:54.676716Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:55.232735Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:55.782460Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:56.351774Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:56.929865Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:57.551266Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:58.289911Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:58.896568Z  INFO helios::consensus: sync committee updated
2024-10-11T13:09:59.470696Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:00.047285Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:00.646327Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:01.194848Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:01.754387Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:02.319570Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:02.878364Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:03.423185Z  INFO helios::consensus: sync committee updated
2024-10-11T13:10:04.086961Z ERROR helios::consensus: sync failed err=invalid sync committee period

If I load a newer checkpoint, I know it works (as I saw recommended in #361) but for my use case this is not ideal. I saw this same issue in Holesky testnet (at a different slot of course as it is a different chain).

My questions are:

  • why is this failing? Is this a bug of the implementation? was there a change at certain ETH upgrades that "broke" the protocol?
  • can this be fixed?
  • Is there any suggested workaround you could find?
  • Do you know the oldest checkpoint I could use in case there is no workaround?

Thank you in advance for the responses

@ncitron
Copy link
Collaborator

ncitron commented Oct 11, 2024

Yes so the reason for this is that the networking spec defines a maximum number of light client updates you can fetch at once, which Nimbus (the node we use for the consensus rpc that we provide by default) enforces this. This means the maximum age of a checkpoint Helios can fetch is 128 * 27 hours = 144 days. This could be addressed by having Helios make multiple update request calls here. We haven't supported this in the past, since we suggest users start up with a somewhat recent checkpoint (and after first sync, helios caches new checkpoints to use, so as long as clients go online every so often after first sync, you are good). The reason we suggest this is there is a class of attacks known as "long range attacks" or "weak subjectivity attacks" that you become susceptible to if you use very old checkpoint. Fortunately for the light client protocol, these attacks are hard to pull of in practice. It shouldn't be to hard to calculate how many updates you need by finding the expected period (using expected_current_slot and dividing by 32 * 256) and subtracting the bootstrap period (using the bootstrap slot and diving by 32 * 256), which yields the number of periods we need. If it is greater than 128 we can just make multiple calls to get_updates to fetch the full list. Is this something you'd be interested in working on? If not, I can probably get to it soon.

@EzequielPostan
Copy link
Author

Thank you for the explanation @ncitron

I did the following changes based on your comments and seems to work, but I am not sure if I could be missing some corner cases.

Any place where I could read more on long range attacks/weak subjectivity? It is good for me to understand the potential risks for my intended use.
Once again, thank you for the responses in advance

@ncitron
Copy link
Collaborator

ncitron commented Oct 26, 2024

I think the main change I would make to that is we should provide exactly how many updates we need, rather than just doing the MAX for each time.

@ncitron ncitron added the bug Something isn't working label Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants
@EzequielPostan @ncitron and others