Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redesign seed sequencer #117

Open
folbricht opened this issue Aug 12, 2019 · 0 comments
Open

Redesign seed sequencer #117

folbricht opened this issue Aug 12, 2019 · 0 comments

Comments

@folbricht
Copy link
Owner

When running extract with seeds, there are a couple of issues that seem different but happen in the same piece of code. Fixing those requires a rewrite of the code that determines what seed should be used.

  1. The current algorithm tries to find the seed with the longest sequence of matching chunks, regardless if that sequence is aligned and can be reflinked or not. Ideally, the algorithm should always use "clone-able" ranges first and if there aren't any more, then also consider "copy-able". This will result in optimal storage efficiency and performance.

  2. When a seed is corrupt (the hash of a chunk doesn't match), the whole process fails. It'd be much better to either evict the whole seed and carry on without it, or to keep going and just get the damaged chunks by other means (other seeds or chunk store).

Getting both of those right and doing it all concurrently may require additional complexity, perhaps the self-seed, where chunks from earlier sections of the target file are copied/cloned to later sections, may need to be dropped and be replaced with a static execution plan calculated beforehand, not during execution.

RyuzakiKK added a commit to RyuzakiKK/desync that referenced this issue Dec 7, 2021
If we have a seed index for a file/device that is corrupted we should
try harder to continue anyway by discarding the invalid seed and
fallback to the potentially other seeds and/or by just using the store.

Partially addresses folbricht#117

Signed-off-by: Ludovico de Nittis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant