Upload ODK submission user photos to S3 for easy access #1744
Conversation
Sujanadh
added
enhancement
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… feat/upload-image-to-s3
spwoodcock
changed the title
| for idx, filename in enumerate(attachments): | ||
| s3_key = f"{s3_base_path}/{instance_id}/{idx+1}.jpeg" | ||
|
|
||
| if object_exists(s3_bucket, s3_key): |
There was a problem hiding this comment.
This will make a lot of calls to S3 potentially!
Not a problem with Minio, but slow if external S3.
Could we do a comparison for what the submission says should exist against the database table instead?
We would need the stat approach if we didn't gave the db table 👍
There was a problem hiding this comment.
ah makes sense! 🤔
|
|
||
| # Insert the record into submission_photos table | ||
| sql = text(""" | ||
| INSERT INTO submission_photos ( |
There was a problem hiding this comment.
Also this could be batched for efficiency.
It's generally good to not make db queries in a for loop - instead gather the data (images uploaded) there in a dict or something, then do a batch db table insert in one go 😄
| -- Start a transaction | ||
| BEGIN; | ||
|
|
||
| CREATE TABLE submission_photos ( |
There was a problem hiding this comment.
Check this SQL doesn't error if you run it twice 👍
Maybe create table if not exists will fix
|
for more information, see https://pre-commit.ci
|
Looks good! Merging 💫 |
What type of PR is this? (check all applicable)
Related Issue
- Extract user photos from submissions and make available via S3 #1701
- Add a database record of photos taken in the field #1729
Describe this PR
This PR creates a function to upload submission images to s3
submission_photostable with fieldsproject_id,task_id,submission_id, ands3_pathScreenshots
N/A
Alternative Approaches Considered
Did you attempt any other approaches that are not documented in code?
Review Guide
Checklist before requesting a review
[optional] What gif best describes this PR or how it makes you feel?