Skip to content
forked from lathanh/s3nukem

A single-file Ruby script to delete Amazon S3 an bucket with many objects (millions) quickly by using multiple threads to retrieve and delete the individual objects.

License

Notifications You must be signed in to change notification settings

woz5999/s3nukem

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 

Repository files navigation

S3NUKEM

s3nukem is a slightly improved version of s3nuke, a Ruby script by Steve Eley that relatively quickly deletes an Amazon Web Services (AWS) Simple Storage Service (S3) bucket with many objects (millions) by using multiple threads to retrieve and delete the individual objects.

In my use case, s3cmd, which deletes with a single thread, deleted objects at a rate of about 1,800/minute (2.5 million / day). s3nukem, with 10 delete threads deleted objects at a rate of about 9,000/minute (13 million / day). My task of deleting 99 million objects went from 40 days to 7.6 days. More threads and Ruby 1.9 (I was using 1.8.5) would have probably completed the job even more quickly.

Improvements

  • The key retrieval thread will pause when the queue contains 1000 * thread_count items. The original script's queue would grow unabated, eating up memory unnecessarily.

  • All output is automatically flushed, which ensures you can keep an eye on progress in real-time

  • Added the number of seconds elapsed since the start of the script to output so you can calculate the rate at which items are being deleted.

Installation

You'll need:

  • Ruby Ruby 1.9 should work faster because of the native thread implementation (on the other hand, network/S3 latency may be your biggest bottleneck).

  • right_aws gem; dmarkow's version if you're running Ruby >= 1.9

      # Ruby < 1.9
      sudo gem install right_aws
    
      # Ruby >= 1.9
      sudo gem install dmarkow-right_aws --source http://gems.github.com
    

Download and make executable; e.g.,

# download
wget https://raw.github.com/lathanh/s3nukem/master/s3nukem
# or
curl -O https://raw.github.com/lathanh/s3nukem/master/s3nukem

# make executable
chmod 755 s3nukem

Obvious Warning

This script is intended to delete all of the items in an S3 bucket very quickly. You will not be prompted to ask you if you're sure. There is no undo.

Do not taunt Happy Fun Script.

Docker

docker run -ti --rm \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY  \
  s3nukem:latest \
    -a $AWS_ACCESS_KEY_ID \
    -s $AWS_SECRET_ACCESS_KEY \
    <bucket>

License

This script is released under the Apache License, Version 2.0. I really don't care what you do with it, so long as "sue me" is not on the agenda.

Credits

Original script by Steve Eley.

Improvements by Robert LaThanh.

About

A single-file Ruby script to delete Amazon S3 an bucket with many objects (millions) quickly by using multiple threads to retrieve and delete the individual objects.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Ruby 100.0%