-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error "Neither last processed snapshot nor initial snapshot were found on disk" #53
Comments
Digged a bit deeper into this and it seems like the functionality for loading the initial snapshot changed in the commit 572cd0b. Do I need to include a json file in my container now that contains the initial snapshot? Or how does loading that initial snapshot work when running the snapshot-streaming the first time? I tried including that json using the old format:
But now I get this error: I tried several variations of the file but can't seem to make it work. Would it be possible to provide the correct json for that initial snapshot? |
hey @flannoo. please try fetching the snapshot from the network and set it as curl -X GET http://<nodeip>:<nodeport>/global-snapshots/latest -H "Accept: application/json" The snapshot must be a json-serialized signed snapshot (either full snapshot for |
Hi @marcinwadon , thank you for looking into this.
The only issue I have now, is that the genesis snapshot (with ordinal 0) is not imported in the opensearch cluster, as the app thinks it's already uploaded (since we define that as the lastSnapshot that was imported). Is there any way I can also upload the ordinal 0 snapshot? Or will I need to do this in a manual way somehow (asthe snapshot-streaming app isn't foreseen to handle this scenario perhaps)? Ordinals can't be negative numbers, so I can't change the ordinal of the "lastSnapshot.json" to -1. I want to run this locally as I want to test a change I did in the blockexplorer to add an API that retrieves all the wallet balances in a certain snapshot ordinal. Right now, it's not returning the correct balances as the genesis snapshot is not stored in opensearch (so I don't know if it's a bug in my code or if it's due to missing data). Thanks! |
Hi @flannoo. You can't force the current version of snapshot-streaming to process the genesis because the current version works only with the incremental snapshots and genesis is a full snapshot. The way it came to this is that previously the network was operating on global snapshots only, then we had a switch to using incremental snapshots which are lighter. The snapshot-streaming got adjusted to work on incrementals with and initial start from a last full snapshot and because all our envs already had full snapshots processed we didn't have an issue with no full snapshot persisted on disk at the time. Long story short we will have to add the processing of genesis (full snapshot) to the snapshot-streaming so that any new network can successfully use it. We are aware about the issue and we have a ticket for adding that functionality. Will let you know on the status of this ticket. BTW are you saying that after mocking the |
Thank you @TheMMaciek
Yes, that is correct. I fetch the latest FULL snapshot using the curl statement below and wrap the result in a "signed" json object together with empty hash & proofsHash fields and then the application is able to process the next (incremental) snapshots and upload them to opensearch. So I added the below line to the Dockerfile in the "start.sh" file, before it starts the java application (so the "lastSnapshot.json" will be present on the filesystem when the app starts)
This results in this kind of file (removed some content to keep it short)
The application then successfully uploads the subsequent (incremental) snapshots:
I'm using the tessellation repository and run the skaffold command to spin up the cluster on my local dev machine (so it spins up a new network every time I test, I don't have a permanent DEV cluster running) |
So perhaps a good way to solve this could be instead of throwing the error that no snapshots were found, that the application tries to retrieve the genesis full snapshot (0) using this URL: http://genesis-node-host:port/global-snapshots/0?full=true |
Yeah using this endpoint to pull the full snapshots is of course part of the solution. But also the whole mapping of full snapshot to the schema expected by opensearch also needs to be implemented - FYI the incremental and full snapshot schemas are different so you can't just propose full where incremental is expected. Currently this mapping for the full snapshots is not in the codebase it needs to be added - good thing is it shouldn't be too hard. |
Hi,
I've tried to setup the snapshot streaming on the latest supported tessellation version (v2.0.0-alpha.12), but I keep getting the error "Neither last processed snapshot nor initial snapshot were found on disk!".
Probably I'm missing something in my configuration, but I can't seem to figure it out.
In the "application.conf" file located in "kubernetes/snapshot-streaming" folder, I needed to add the following missing fields:
And I also needed to add some currency indexes in the opensearch properties in that application.conf file, I used these values (copied values from "src/main/resources/application.conf")
This is the complete application.config that I'm using:
After adding that and running the
skaffold dev --trigger=manual
, the application starts but then yields the error that no snapshots were found on disk.This is what I see in the logs:
Any ideas on what I'm missing or doing wrong?
It seems it's not connecting to my running tessellation cluster v2.0.0-alpha.12 (or it goes in error before trying to connect to it). I also tried updating the l0Peers ip property to the local IP address instead of the name "l0-initial-validator" but that doesn't seem to change anything
Any help is greatly appreciated :-)
Thanks!
The text was updated successfully, but these errors were encountered: