update ingestion script to support multiple S3 buckets

[boegel]

No description provided.

[bedroge]

This was already a bit ugly, but doesn't work as intended anymore at all now, as it will exit before the second bucket is being handled. Easy fix would be to change it in continue, but maybe it should be done a bit cleaner with a proper if/else or something

[boegel]

fixed in aed065a

[bedroge]

Gave it a quick try, and the new code seems to work fine, except that there is a bug in the existing find_tarballs function, it doesn't like empty buckets:

    for object in s3.list_objects_v2(Bucket=bucket)['Contents']
KeyError: 'Contents'

[bedroge]

Gave it a quick try, and the new code seems to work fine, except that there is a bug in the existing find_tarballs function, it doesn't like empty buckets:

    for object in s3.list_objects_v2(Bucket=bucket)['Contents']
KeyError: 'Contents'

This can actually be easily fixed by using:

       object['Key'] for object in s3.list_objects_v2(Bucket=bucket).get('Contents', [])

[boegel]

Gave it a quick try, and the new code seems to work fine, except that there is a bug in the existing find_tarballs function, it doesn't like empty buckets

fixed in cc63131

[boegel]

Gave it a quick try, and the new code seems to work fine, except that there is a bug in the existing find_tarballs function, it doesn't like empty buckets:

    for object in s3.list_objects_v2(Bucket=bucket)['Contents']
KeyError: 'Contents'

This can actually be easily fixed by using:

       object['Key'] for object in s3.list_objects_v2(Bucket=bucket).get('Contents', [])

OK, see ddef0c7

[bedroge]

Lgtm. One failing CI test, but that's completely unrelated to the changes made in this PR, so I'm ignoring it.

[boegel]

opened an issue for CI problem: #155