Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from tombstoning deleted objects to actually deleting them #1703

Open
snarfed opened this issue Jan 19, 2025 · 3 comments
Open

Switch from tombstoning deleted objects to actually deleting them #1703

snarfed opened this issue Jan 19, 2025 · 3 comments
Labels

Comments

@snarfed
Copy link
Owner

snarfed commented Jan 19, 2025

When we get a Delete, right now we set Object.deleted to True, and we have to check that flag everywhere. This originally seemed like a good idea, long ago when I was young and foolish, but I don't know that it's ever bought us anything, and it's a liability. We should switch to actually deleting.

@snarfed snarfed added the infra label Jan 19, 2025
@snarfed
Copy link
Owner Author

snarfed commented Jan 21, 2025

One catch: we can't delete Objects in Protocol.receive because send tasks need their copies. So we'd need to delay that delete, maybe with Object._expire.

@snarfed
Copy link
Owner Author

snarfed commented Jan 21, 2025

^ Done, via Object._expire.

I'm also running a workflow to delete all existing deleted Objects right now. 18M! Using https://cloud.google.com/dataflow/docs/guides/templates/provided/firestore-bulk-delete#gcloud , based on https://github.com/snarfed/bridgy#delete-old-responses :

gcloud dataflow jobs run 'Delete Object.deleted datastore entities' \
  --gcs-location gs://dataflow-templates-us-central1/latest/Firestore_to_Firestore_Delete \
  --region us-central1 \
  --parameters firestoreReadGqlQuery="SELECT __key__ FROM Object WHERE deleted = TRUE",firestoreReadProjectId=bridgy-federated,firestoreDeleteProjectId=bridgy-federated,firestoreHintNumWorkers=100

@snarfed
Copy link
Owner Author

snarfed commented Jan 22, 2025

Next step, delete all old Objects (eg >90d) that should now be expired due to our new logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant