Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: CLI enhancements #34

Open
wants to merge 24 commits into
base: master
Choose a base branch
from

Conversation

kristjaneerik
Copy link
Collaborator

@kristjaneerik kristjaneerik commented Jul 7, 2017

stor cp, stor cpto

Support for multiple sources. New usage: stor cp [-h] [-r] SOURCE [SOURCE ...] DEST

Also adds stor cpto which reverses DEST and SOURCE(s) and is useful for e.g. find . -name '*.csv' | xargs stor cpto swift://tenant/container/prefix/.

stor rm

Support for multiple sources. New usage: stor rm [-h] [-r] PATH [PATH ...]

stor cat

Support for multiple sources. New usage: stor cat [-h] PATH [PATH ...]

stor ls, stor list, stor walkfiles

Defaulting to ./ if no argument given. Much fancier output, e.g.:

$ stor ls --sort-by-time --reverse --url  # same as stor ls -tru
Total: 469.0 KiB
  5.0 KiB  2017-07-10 11:45:49                    file:///Users/eerik/Documents/stor-dev/tmp.ncdu
101.5 KiB  2017-07-08 20:42:16                    file:///Users/eerik/Documents/stor-dev/.coverage
172.9 KiB  2017-07-08 10:49:10                    file:///Users/eerik/Documents/stor-dev/tmp.walkfiles
172.9 KiB  2017-07-08 10:49:01        text/plain  file:///Users/eerik/Documents/stor-dev/tmp.list
    708 B  2017-07-08 10:47:11                    file:///Users/eerik/Documents/stor-dev/testfile
  4.5 KiB  2017-07-08 00:38:46                    file:///Users/eerik/Documents/stor-dev/README.md
    198 B  2017-07-08 00:09:27        text/plain  file:///Users/eerik/Documents/stor-dev/requirements.txt
    643 B  2017-07-07 09:23:31                    file:///Users/eerik/Documents/stor-dev/tox.ini
     81 B  2017-07-06 12:04:19     text/x-python  file:///Users/eerik/Documents/stor-dev/setup.py
  1.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/setup.cfg
    144 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-setup.txt
     31 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-docs.txt
    142 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/requirements-dev.txt
  1.0 KiB  2017-07-06 12:04:19  application/x-sh  file:///Users/eerik/Documents/stor-dev/publish_docs.sh
    198 B  2017-07-06 12:04:19        text/plain  file:///Users/eerik/Documents/stor-dev/MANIFEST.in
  2.6 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/Makefile
  2.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/LICENSE
  2.3 KiB  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/CONTRIBUTING.md
    103 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/AUTHORS
    263 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/.travis.yml
    227 B  2017-07-06 12:04:19                    file:///Users/eerik/Documents/stor-dev/.gitignore
                                             DIR  file:///Users/eerik/Documents/stor-dev/stor.egg-info
                                             DIR  file:///Users/eerik/Documents/stor-dev/stor
                                             DIR  file:///Users/eerik/Documents/stor-dev/docs
                                             DIR  file:///Users/eerik/Documents/stor-dev/.venv
                                             DIR  file:///Users/eerik/Documents/stor-dev/.tox
                                             DIR  file:///Users/eerik/Documents/stor-dev/.git
                                             DIR  file:///Users/eerik/Documents/stor-dev/.eggs
                                             DIR  file:///Users/eerik/Documents/stor-dev/.cache

Misc changes

Enable Swift integration to be more flexible by adding more environment variables to control e.g. the tenant.

Note: Somewhat in progress -- stor.tests.test_integration_swift.SwiftIntegrationTest.test_copytree_multiple_to_from_dir might not work correctly and stor cpto doesn't have tests at the moment.

EDIT: This used to be just about extending stor cp, but I changed it to be a general CLI-enhancement PR.

New usage: stor cp [-h] [-r] SOURCE [SOURCE ...] DEST
Also adds ``stor cpto`` which reverses DEST and SOURCE(s).
Enable Swift integration to be more flexible by adding more environment
variables.

Sem-Ver: feature
@jtratner
Copy link
Contributor

jtratner commented Jul 7, 2017

What do you think about aping the current man page for copy and adding a '-t', '--target-directory' flag and overriding the default help to look like this instead (so again it's more similar to POSIX copy):

usage: stor cp [-h] [-r] SOURCE [SOURCE ...] DEST
       stor cp [-r] -t DEST SOURCE [SOURCE ...]

Alias for copy. '-' is a special character that allows for using stdin as the
source.

optional arguments:
  -h, --help  show this help message and exit
  -r          Copy a directory and its subtree to the destination directory.
              Must be specified before any other flags.

@jtratner
Copy link
Contributor

jtratner commented Jul 7, 2017

how do we want to handle mix of directories and files?

@kristjaneerik
Copy link
Collaborator Author

It would be really neat to have the -t argument, but I tried to express it in argparse and couldn't do it -- too hard to not have the positional DEST if the -t argument is set, at least with my (limited) knowledge of argparse. The way I did it would give you something like stor cp [-r] [-t DEST] SOURCE [SOURCE ...] [DEST] and you'd have to do validation of args after getting them, rather than having argparse do it.

I would like for it to handle both files and directories (just call isdir() on each SOURCE and choose appropriately? Is that expensive, does it matter?), but right now it has to be all one or the other, good catch. That way we could remove the -r flag, right?

@kristjaneerik kristjaneerik mentioned this pull request Jul 7, 2017
23 tasks
@kristjaneerik
Copy link
Collaborator Author

I changed this to be a general CLI enhancement PR since many of the changes are in the same vain.

WIP!

@kristjaneerik kristjaneerik changed the title Enable multiple SOURCE(s) for stor cp WIP: CLI enhancements Jul 8, 2017
Use fixed-width columns by default but add --tabs flag to get previous
behavior back.
Add --relative-time flag to give times relative to now, e.g.
"2.5 days [ago]".

Sem-Ver: feature
@jtratner
Copy link
Contributor

jtratner commented Jul 9, 2017

if you output something prefixed as file:/// does that work with stor? Just looking at your example...

@jtratner
Copy link
Contributor

jtratner commented Jul 10, 2017

Just thinking about this some more: let's see if we can get this to look really nice and handle CLI output well for s3 and swift, and then I can work independently on getting this to work in a performant manner with many objects.

@jtratner
Copy link
Contributor

One thing - the openstack object storage API (aka swift) actually returns a list of JSON objects that includes date modified and size, so you don't have to stat each file individually. We just didn't care about it in v1, so we throw away all that information. My suggestion would be to use that info - will be much much faster on the front end vs calling getsize() repeatedly.

I'm thinking we could add a caching contextmanager that only maintains the cache for the duration of the operation, so user never sees stale results) - especially would matter for not accidentally caching on retry. Or just stuff the data on to each object's dictionary when we return from list...

But it's probably easier to get this into a good state in terms of parameters and formatting (aiming to have defaults be what comes back from list command by default last-modified, size, (storage class?) for S3 and last-modified, size, content-type for Swift), then to harp on performance enhancement (I can try to do that later)

add -l/--simple-list to show the single column version
and -b/--use-bytes to display size in bytes
For local files, show last modified / size / mimetype / name
For Swift files, also show hash / ETag
For s3 files, also show storage class
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants