scan_bucket.py struggles when too many objects in the bucket

[cbp123]

We have a bucket with millions of objects in it. Because scan_bucket.py loads all objects in the bucket into memory before scanning them, it can freeze for a long time at startup. I imagine in the worst case it could start to run out of memory.

I modified the code to load and scan the objects in pages instead and it worked much better. If you think this is a better method, let me know and I can submit a PR.

[denniswebb]

Yes a PR would be great. Thanks.

[jdepp]

Hey @cbp123, any update on this?