Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid copying of versioned files #30

Closed
varmar05 opened this issue Oct 28, 2019 · 4 comments
Closed

Avoid copying of versioned files #30

varmar05 opened this issue Oct 28, 2019 · 4 comments
Assignees
Labels
refactor Internal code refactoring

Comments

@varmar05
Copy link
Contributor

In case of geodiff supported files we should completely avoid copying files since they can be open with some clients in WAL mode and thus changes would be lost.

We need either to flush changes right before copy or not to copy at all (might be impossible in some cases). Also we need to investigate how robust is geodiff sync against data loss when some file is still open by 3rd client (e.g. modified between pull and push)

@varmar05
Copy link
Contributor Author

varmar05 commented Mar 6, 2020

We currently trigger sqlite checkpoint manually before push. Also we download geodiff changeset files whenever possible.
However, there are still some use cases with potential issues, e.g. changing table schema which geodiff can not handle, therefore we need more actions, e.g.:

  • close qgis layers before sync (short-term)
  • check existence of WAL files before pull and raise exception
  • create sync transactional so we can abort whole pull process if needed or retry individual failures (Retry for failed pull/push requests  #6)

@wonder-sk
Copy link
Contributor

There is SQLite API for copying sqlite databases which should be hopefully safer than verbatim copies and dealing with checkpoints/vacuum: https://sqlite.org/backup.html

@wonder-sk
Copy link
Contributor

The new geodiff 1.0 will have GEODIFF_makeCopySqlite(src, dst) which will allow safe copying. And we should not even need to do checkpoint/vacuum anymore.

@wonder-sk wonder-sk added the refactor Internal code refactoring label May 21, 2021
@erpas erpas self-assigned this Jun 9, 2021
@wonder-sk
Copy link
Contributor

Fixed in c03c33f and 348d77d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactor Internal code refactoring
Projects
None yet
Development

No branches or pull requests

3 participants