Slower than tar

[jonatanblue]

It takes around two times longer to backup a directory compared to using tar on the command line.

dinghy:

# time ./dinghy.py --dirs=/var/lib/rundeck/logs backup --dest /tmp/backup-test --filename=test.tar.gz
2017-06-15 09:34:12,679 WARNING overriding default directories with /var/lib/rundeck/logs
2017-06-15 09:34:12,724 INFO counting files...
2017-06-15 09:34:13,987 INFO adding directory /var/lib/rundeck/logs
Processing |################################| 162445/162445

2017-06-15 09:39:01,148 INFO backup complete

real	4m49.105s
user	2m32.640s
sys	0m38.250s

tar:

# time tar czvf /tmp/backup-test/manual_logs.tar.gz /var/lib/rundeck/logs
...
real	2m22.006s
user	1m1.840s
sys	0m28.210s

[nvtkaszpir]

Try opening file as stream (it has its limitations, though), so try to change 'w:gz' to 'w|gz'.

https://docs.python.org/2/library/tarfile.html#tarfile.open

Or alternativey spawn system process.