feature: Shell completions

[medihack]

Is your feature request related to a problem? Please describe.

It would be nice to have shell completions of the available duties and parameters for convenience.

Describe the solution you'd like

Something similar to what invoke has.

Describe alternatives you've considered

None

Additional context

None

Boost priority

• Boost priority in our backlog through Polar.sh. Higher pledge, higher priority.
• Minimum pledge by user/organization is $5, minimum amount for boost is $30.
• View all issues with pledges.
• We receive the funds once the issue is completed and confirmed by you.
• Features with the insiders label are released to sponsors first, and tied to a funding goal.

[medihack]

Uups ... you already created one.

[pawamoy]

Ah, let me keep this one then, it helps to have the actual author in the issue metadata 🙂

[medihack]

I hope that moving on to this issue will not result in losing my small funding. I have never heard of Polar.sh before, but I like any idea that can make Open-Source more sustainable.

[pawamoy]

Oh, so sorry, I missed that! Strangely the badge doesn't show your pledge (I thought it would do so) (it does now). Of course I'll make sure your pledge doesn't go to waste! Thank you so much ❤️

[pawamoy]

I setup issue boosting recently, as an experiment and after user feedback, and you're the first one to boost an issue 😄 I may not be the one resolving it, but rest assured it's now in the top of our backlog and will be handled very soon 😄

[medihack]

I setup issue boosting recently, as an experiment and after user feedback, and you're the first one to boost an issue 😄 I may not be the one resolving it, but rest assured it's now in the top of our backlog and will be handled very soon 😄

No hurry. I just noticed that you’re a significant contributor to open-source projects, which I truly admire (I contribute to various Open-Source projects myself). I’m particularly interested in the idea of making the open-source community more financially sustainable - it’s such an important topic.
I also wanted to say that I really like the duty project! With invoke appearing to be unmaintained, I think there’s a growing need for a well-supported alternative, and I believe duty could fill that gap in the long term.

[pawamoy]

Oh I didn't know invoke wasn't maintained! Lets make duty even better then 😄

[bswck]

What shells should we support?

[pawamoy]

Most common ones probably, if they are supported by the lib we'll use: bash, zsh, fish, powershell.

[bswck]

alrighty!

couple of questions from the technical POV, as i am implementing this—@pawamoy:

• are we fine with pulling in shellingham as an additional dependency for shell detection? i'm not considering just telling people to install pip install shellingham externally, because in the future we may want to integrate with something else, so it would be good to establish a stronger relationship with the packages we plan using to utilize for this feature.

  needs decision—should shellingham and possibly other deps (assuming no more as for now though) be extra (e.g. duty[tab]) or hard dependencies? my thinking is we should answer that with data:

  • most ppl use duty interactively → it could be reasonable to make those completion-related dependencies hard deps and not introduce additional complexity in code doing import guards and other unnecessary complicated availability checks
  • most ppl use duty in CI → it would be an unnecessary cost to pull in a dependency unlikely to be used and possibly slowing down things (though would that be a very huge cost really? otoh this is a purposefully simple project advertising itself mainly for the smart anti-IPC callable approach)

  misc:

  • typer's approach to that is just distributing another package with name typer-slim which seems the most suitable for CI. maybe instead of trying to tackle the extra problem (which would be easier if Support the empty-string extra pypa/setuptools#1503 had been implemented—that would render typer-slim unnecessary ;) ), we can also adopt that slim shady packaging practice. wdyt?

some other research notes

• invoke delivers its completions in a known shell by spawning itself with --complete global option and passing in the current completion context (what is being written in the terminal)
• typer delivers its completions in a known shell just like click, its base, which is similar to invoke but instead of exposing --complete as an option it depends on a specific environment variable that puts it in the "completion mode". moreover, click preserves the info about targeted shell up to the completion mode invocation, in order to let the results of completions vary based on the current shell via the shell completion classes API. pretty neat; would we want to modify our completions based on the current shell? most likely yeah?
• cappa delivers completions in a similar way, but doesn't smuggle the info about the targeted shell so you can't implement different completions for different shells easily

(i was wondering if we could save some time and not have to actually run the program to register commands only for completions, and instead have these commands somehow statically prepared from the last save; but ig it doesn't matter that much, it would be unreliable, we need to collect that info from runtime anyway)

[pawamoy]

Thanks for the analysis @bswck!

are we fine with pulling in shellingham as an additional dependency for shell detection?

I wonder if we really need shell detection. I believe many tools just expect a flag like --shell <bash|fish|zsh|powershell> to print out the relevant instructions. Let me check how they do that 🤔

we can also adopt that slim shady packaging practice. wdyt?

I had recent thoughts about this. Usually, projects start out as project-name and include dependencies for the CLI (cappa, typer, whatever). Then, if we want to optimize and reduce dependencies when they're not needed, we can refactor the library into a project-name-lib separate project, and depend on it in project-name. No breaking change, users who just want to quickly install/use/try the tool won't get any surprise (no missing CLI), and experienced users who want to reduce their dependencies will find that they can depend on project-name-lib instead. Finally, for TUIs and GUIs, additional projects can be created like project-name-tui and project-name-gui. Ideally, project-name will still provide the tui or gui subcommands (or --tui or --gui flags) and warn if the dependencies are missing. Extras can also be declared on project-name, like project-name[tui], project-name[gui] and project-name[all].

However here we probably always install duty to be used as a CLI (even in CI), so I don't think it's worth creating separate projects just to avoid the shell-detection library (if we decide to use one) and auto-completion library.

[pawamoy]

would we want to modify our completions based on the current shell? most likely yeah?

I'm not sure why we would want that. Our CLI shape doesn't change based on the shell (unless I'm missing something?), so we probably don't need to do that?

[pawamoy]

(i was wondering if we could save some time and not have to actually run the program to register commands only for completions, and instead have these commands somehow statically prepared from the last save; but ig it doesn't matter that much, it would be unreliable, we need to collect that info from runtime anyway)

Yeah, running the program in a Python interpreter each time we hit tab makes things slow. That's an issue with Python and dynamic CLIs, and I'm afraid there's not much we can do without over-engineering things. Happy to be proven otherwise, maybe there are completion frameworks that provide clever features to make things fast.

[pawamoy]

% pdm completion --help
Usage: pdm completion [-h] [shell]

Generate completion scripts for the given shell

Positional Arguments:
  shell       The shell to generate the scripts for. If not given, PDM will properly guess from `SHELL` env var.

Options:
  -h, --help  Show this help message and exit.

Seems good enough to me, WDYT?

[bswck]

makes sense. shellingham also suggests a routine for basic shell detection regarding the SHELL env var.
cool—pdm completion actually uses pycomplete which produces static completion scripts (all completions are known eagerly and listed in the script), which is only possible due to its API stability (not our case); a nice summary has been given here

[bswck]

I was tempted to try out argcomplete... but after diving into the implementation details it has, I'm honestly discouraged.

I think we can roll out our own, hardcoded solution

[pawamoy]

OK, sounds good to me! By hardcoded solution you mean scripts for each shell we want to support, and maybe a few updates to the Python CLI to return the necessary completion data?

[pawamoy]

@medihack what shell are you using?

[medihack]

@medihack what shell are you using?

I only use bash. Invoke seems to have also scripts for fish and zsh (https://github.com/pyinvoke/invoke/tree/main/invoke/completion)

[pawamoy]

Thanks! @bswck I'll consider it resolved if we only implement Bash and Zsh completions for now. With the code in place, it will be easy to add Fish and Powershell completions later if we want to (or if users request it). What do you both think?

[bswck]

@medihack @pawamoy please guide me: which way of finding completions do we consider desired for a simple use case below?

for a shell prompt duty b and exposed¹ duties that include

build
rebuild
upload-github

do we expect:

(1)
build (match by prefix only)

(2)
build upload-github (match by prefix & suffix)

or (3)
build upload-github rebuild (match by prefix, suffix, and inclusion; order respectively)

?

all options make sense.
I assume it is useful to get build rebuild as completions of duty bui.

Footnotes

1. I assume it is desired to be able to hide an entire duty from completions, presumably with @duty(shell_completions=False). ↩

[pawamoy]

I'd expect only build, at least for Bash. I realize now this was maybe what you meant with "would we want to modify our completions based on the current shell?", because I know Zsh is a bit more powerful regarding completions, and can probably be fed more than just build, and will choose itself what word to complete given user configuration. But I'm not entirely sure how that all works, so I don't know if the completed words returned by duty itself can be the same for each shell, or if, like the completion scripts that duty generates (will generate), they vary depending on the shell. Long sentence, sorry, let me know if it's unclear.