Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete incomplete file uploads #1329

Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8946d5f
used prefetch_related method to optimize the query performed to fetch…
yaswanthsaivendra May 23, 2023
7d410fd
Revert "add direnv support (#1304)"
yaswanthsaivendra May 26, 2023
7179fdf
Revert "Revert "add direnv support (#1304)""
yaswanthsaivendra May 26, 2023
9cb5f7c
Revert "used prefetch_related method to optimize the query performed …
yaswanthsaivendra May 26, 2023
d2134a9
Merge branch 'coronasafe:master' into master
yaswanthsaivendra May 26, 2023
7000a38
used prefetch where we are querying the skills using UserAssigned Ser…
yaswanthsaivendra May 26, 2023
a75b98d
Revert "used prefetch where we are querying the skills using UserAssi…
yaswanthsaivendra May 27, 2023
f59ff6b
made neccessary changes for prefetching the skills
yaswanthsaivendra May 27, 2023
ec69b24
Merge branch 'coronasafe:master' into master
yaswanthsaivendra May 29, 2023
e6541ef
Revert "made neccessary changes for prefetching the skills"
yaswanthsaivendra May 30, 2023
81dfa52
delete incomplete file uploads
yaswanthsaivendra Jun 1, 2023
a354df9
updated code to do bulk delete
yaswanthsaivendra Jun 2, 2023
cf537e9
Merge branch 'coronasafe:master' into delete_incomplete_file_uploads
yaswanthsaivendra Jun 2, 2023
d5278bc
Merge branch 'delete_incomplete_file_uploads' of github.com:yaswanths…
yaswanthsaivendra Jun 2, 2023
897ec15
changed the hard delete to soft delete
yaswanthsaivendra Jun 7, 2023
c063d65
updated code to handle up upper limit by batching
yaswanthsaivendra Jun 8, 2023
89c76e1
Merge branch 'delete_incomplete_file_uploads' of github.com:yaswanths…
yaswanthsaivendra Jun 8, 2023
2e1ac00
minor fix
yaswanthsaivendra Jun 8, 2023
2a1c1e1
minor fix and added tests
yaswanthsaivendra Jun 8, 2023
c018e3c
Merge branch 'master' into delete_incomplete_file_uploads
sainak Jul 5, 2023
f4c1853
update task for celery 5.x
sainak Jul 5, 2023
551df2b
Merge branch 'master' into delete_incomplete_file_uploads
sainak Jul 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions care/facility/models/file_upload.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,13 @@ def get_object(self, bucket=settings.FILE_UPLOAD_BUCKET, **kwargs):
Key=f"{self.FileType(self.file_type).name}/{self.internal_name}",
**kwargs,
)

@staticmethod
def bulk_delete_objects(s3_keys):
s3 = boto3.client("s3", **cs_provider.get_client_config())
bucket = settings.FILE_UPLOAD_BUCKET

return s3.delete_objects(
Bucket=bucket,
Delete={"Objects": [{"Key": key} for key in s3_keys], "Quiet": True},
)
25 changes: 25 additions & 0 deletions care/facility/tasks/fileupload/delete_incomplete_file_uploads.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from datetime import timedelta

from celery.decorators import periodic_task
from celery.schedules import crontab
from django.utils import timezone

from care.facility.models.file_upload import FileUpload


@periodic_task(
run_every=crontab(minute="0", hour="0")
) # Run the task daily at midnight
def delete_incomplete_file_uploads():
yesterday = timezone.now() - timedelta(days=1)
incomplete_uploads = FileUpload.objects.filter(
created__date__lte=yesterday, upload_completed=False
yaswanthsaivendra marked this conversation as resolved.
Show resolved Hide resolved
)

s3_keys = [
f"{upload.FileType(upload.file_type).name/upload.internal_name}"
for upload in incomplete_uploads
]

FileUpload.bulk_delete_objects(s3_keys)
incomplete_uploads.update(deleted=True)