WIP: CLI enhancements

[kristjaneerik]

stor cp, stor cpto

Support for multiple sources. New usage: stor cp [-h] [-r] SOURCE [SOURCE ...] DEST

Also adds stor cpto which reverses DEST and SOURCE(s) and is useful for e.g. find . -name '*.csv' | xargs stor cpto swift://tenant/container/prefix/.

stor rm

Support for multiple sources. New usage: stor rm [-h] [-r] PATH [PATH ...]

stor cat

Support for multiple sources. New usage: stor cat [-h] PATH [PATH ...]

stor ls, stor list, stor walkfiles

Defaulting to ./ if no argument given. Much fancier output, e.g.:

$ stor ls --sort-by-time --reverse --url  # same as stor ls -tru
Total: 469.0 KiB
  5.0 KiB  2017-07-10 11:45:49                    file:///Users/eerik/Documents/stor-dev/tmp.ncdu
101.5 KiB  2017-07-08 20:42:16                    file:///Users/eerik/Documents/stor-dev/.coverage
172.9 KiB  2017-07-08 10:49:10                    file:///Users/eerik/Documents/stor-dev/tmp.walkfiles
172.9 KiB  2017-07-08 10:49:01        text/plain  file:///Users/eerik/Documents/stor-dev/tmp.list
    708 B  2017-07-08 10:47:11                    file:///Users/eerik/Documents/stor-dev/testfile
  4.5 KiB  2017-07-08 00:38:46                    file:///Users/eerik/Documents/stor-dev/README.md
    198 B  2017-07-08 00:09:27        text/plain  file:///Users/eerik/Documents/stor-dev/requirements.txt
    643 B  2017-07-07 09:23:31                    file:///Users/eerik/Documents/stor-dev/tox.ini
     81 B  2017-07-06 12:04:19     text/x-python  file:///Users/eerik/Documents/stor-dev/setup.py
  1.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/setup.cfg
    144 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-setup.txt
     31 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-docs.txt
    142 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-dev.txt
  1.0 KiB  2017-07-06 12:04:19  application/x-sh  file:///Users/eerik/Documents/stor-dev/publish_docs.sh
    198 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/MANIFEST.in
  2.6 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/Makefile
  2.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/LICENSE
  2.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/CONTRIBUTING.md
    103 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/AUTHORS
    263 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/.travis.yml
    227 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/.gitignore
                                             DIR  file:///Users/eerik/Documents/stor-dev/stor.egg-info
                                             DIR  file:///Users/eerik/Documents/stor-dev/stor
                                             DIR  file:///Users/eerik/Documents/stor-dev/docs
                                             DIR  file:///Users/eerik/Documents/stor-dev/.venv
                                             DIR  file:///Users/eerik/Documents/stor-dev/.tox
                                             DIR  file:///Users/eerik/Documents/stor-dev/.git
                                             DIR  file:///Users/eerik/Documents/stor-dev/.eggs
                                             DIR  file:///Users/eerik/Documents/stor-dev/.cache

Misc changes

Enable Swift integration to be more flexible by adding more environment variables to control e.g. the tenant.

Note: Somewhat in progress -- stor.tests.test_integration_swift.SwiftIntegrationTest.test_copytree_multiple_to_from_dir might not work correctly and stor cpto doesn't have tests at the moment.

EDIT: This used to be just about extending stor cp, but I changed it to be a general CLI-enhancement PR.

[jtratner]

What do you think about aping the current man page for copy and adding a '-t', '--target-directory' flag and overriding the default help to look like this instead (so again it's more similar to POSIX copy):

usage: stor cp [-h] [-r] SOURCE [SOURCE ...] DEST
       stor cp [-r] -t DEST SOURCE [SOURCE ...]

Alias for copy. '-' is a special character that allows for using stdin as the
source.

optional arguments:
  -h, --help  show this help message and exit
  -r          Copy a directory and its subtree to the destination directory.
              Must be specified before any other flags.

[jtratner]

how do we want to handle mix of directories and files?

[kristjaneerik]

It would be really neat to have the -t argument, but I tried to express it in argparse and couldn't do it -- too hard to not have the positional DEST if the -t argument is set, at least with my (limited) knowledge of argparse. The way I did it would give you something like stor cp [-r] [-t DEST] SOURCE [SOURCE ...] [DEST] and you'd have to do validation of args after getting them, rather than having argparse do it.

I would like for it to handle both files and directories (just call isdir() on each SOURCE and choose appropriately? Is that expensive, does it matter?), but right now it has to be all one or the other, good catch. That way we could remove the -r flag, right?

[kristjaneerik]

I changed this to be a general CLI enhancement PR since many of the changes are in the same vain.

WIP!

[jtratner]

if you output something prefixed as file:/// does that work with stor? Just looking at your example...

[jtratner]

Just thinking about this some more: let's see if we can get this to look really nice and handle CLI output well for s3 and swift, and then I can work independently on getting this to work in a performant manner with many objects.

[jtratner]

One thing - the openstack object storage API (aka swift) actually returns a list of JSON objects that includes date modified and size, so you don't have to stat each file individually. We just didn't care about it in v1, so we throw away all that information. My suggestion would be to use that info - will be much much faster on the front end vs calling getsize() repeatedly.

I'm thinking we could add a caching contextmanager that only maintains the cache for the duration of the operation, so user never sees stale results) - especially would matter for not accidentally caching on retry. Or just stuff the data on to each object's dictionary when we return from list...

But it's probably easier to get this into a good state in terms of parameters and formatting (aiming to have defaults be what comes back from list command by default last-modified, size, (storage class?) for S3 and last-modified, size, content-type for Swift), then to harp on performance enhancement (I can try to do that later)