feat request: block checkpointing

[antazoey]

Overview

Prevent blocks from being re-checked.
Both a perf and can help prevent duplicating actions you really don't want to duplicate.

Specification

Either just do this by default and have a checkpointing kwargument or something else or some combination of things that achieve the same goal. Make it good tho plz.

Dependencies

Include links to any open issues that must be resolved before this feature can be implemented.

[fubuloubu]

More broadly, it would be good to have a system that let's the live runners "pick up where they left off" by recording the last block they successfully processed, and adding that to state (so users can trigger startup procedures such as handling past blocks based on that)

[mikeshultz]

This sort of has been completed in #33. It sounds like we don't want Silverback to handle missed or historical events, but we now provide the user with the last seen and processed blocks to do what they want with (which may include historical processing).

There's still a bit of a disconnect between websocket and polling runners though mentioned in #44 though.

We can probably close this unless we do want to diverge and start handling historical data.

[antazoey]

I agree, #33 seems like it fixes this.

[antazoey]

I guess this issue different than the one fixed.. I was hoping to have something like a flag for silverback that made the default value be the last checkpointed block num rather that having to do it explicitly.

It seems like we have the means to the functionality now though!

[mikeshultz]

I was hoping to have something like a flag for silverback that made the default value be the last checkpointed block num rather that having to do it explicitly.

I'm not really following you here. The default value of what? Explicitly do what?

[antazoey]

like if it crashes, and then I run the same silverback command again using like a special flag, and if i dont tell it any other block number, it automatically use the one from the last checkpoint

[antazoey]

thinking of similar to like pytest's --stepwise flag where it you run it again it picks up where it left off

[mikeshultz]

Like Silverback automatically replays history and schedules jobs? From discussions with @fubuloubu it sounded like we don't want this behavior. Let the users handle historical data if they want but leave silverback out of it.

There's still the undesirable behavior in the PollingRunner though, documented in #44. Maybe they need to get on the same page so we have a unified idea of Silverback.

[antazoey]

Well I meant just as flag, not like the default behavior of silverback. But yeah, I suppose I can script it out myself.

[mikeshultz]

you/we could potentially build a runner just for that, too. Something that specifically starts by fetching events/blocks from where it left off.

[antazoey]

you/we could potentially build a runner just for that, too. Something that specifically starts by fetching events/blocks from where it left off.

yeah i figured when i have time someday i will retry and access the best way.
And i can reopen the feature request if it makes sense or make a custom runner or even just show in docs how to setup to behave this way.