-
Notifications
You must be signed in to change notification settings - Fork 225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve inventory target name resolution #3291
Conversation
The documentation for the plan functions get_target and get_targets do not mention anything about wildcard support, but the Bolt documentation at https://www.puppet.com/docs/bolt/latest/running_bolt_commands#specify-targets-using-glob-matching mentions at the bottom that "Extended glob matching is only supported when specifying targets from the command-line interface. Plans do not support extended glob matching." - which seems to imply that basic glob matching should be supported. |
I'll have to debug the tests (i'll do this in a separate PR). Assuming all the tests pass I think this is a great improvement. i agree with your assessment and interpretation of the intent for matching. Thanks! |
95742a8
to
ebbcf8f
Compare
ebbcf8f
to
627c000
Compare
I made one more improvement to do an exact-match lookup instead of the slower wildcard if there are no wildcard characters. In my ~3,300 target string name |
closing and re-opening to trigger tests. |
Now that I got tests unblocked, you can see a couple cases we will need to address with this work. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Failing tests:
Failures:
1) Bolt::Target when parsing userinfo strips brackets from ipv6 addresses in a uri
Failure/Error: raise(WildcardError, target) if targets.empty?
Bolt::Inventory::Inventory::WildcardError:
Found 0 targets matching wildcard pattern ssh://[::1]:22
# ./lib/bolt/inventory/inventory.rb:160:in `resolve_name'
# ./lib/bolt/inventory/inventory.rb:180:in `block in expand_targets'
# ./lib/bolt/inventory/inventory.rb:179:in `map'
# ./lib/bolt/inventory/inventory.rb:179:in `expand_targets'
# ./lib/bolt/inventory/inventory.rb:99:in `get_targets'
# ./lib/bolt/inventory/inventory.rb:108:in `get_target'
# ./spec/unit/target_spec.rb:177:in `block (3 levels) in <top (required)>'
2) Bolt::Target when parsing userinfo returns set object with feature_set
Failure/Error: raise(WildcardError, target) if targets.empty?
Bolt::Inventory::Inventory::WildcardError:
Found 0 targets matching wildcard pattern ssh://[::1]:22
# ./lib/bolt/inventory/inventory.rb:160:in `resolve_name'
# ./lib/bolt/inventory/inventory.rb:180:in `block in expand_targets'
# ./lib/bolt/inventory/inventory.rb:179:in `map'
# ./lib/bolt/inventory/inventory.rb:179:in `expand_targets'
# ./lib/bolt/inventory/inventory.rb:99:in `get_targets'
# ./lib/bolt/inventory/inventory.rb:108:in `get_target'
# ./spec/unit/target_spec.rb:192:in `block (3 levels) in <top (required)>'
Finished in 18.04 seconds (files took 1.62 seconds to load)
[153](https://github.com/puppetlabs/bolt/actions/runs/8713085630/job/23900793508?pr=3291#step:7:154)6 examples, 2 failures
Failed examples:
rspec ./spec/unit/target_spec.rb:176 # Bolt::Target when parsing userinfo strips brackets from ipv6 addresses in a uri
rspec ./spec/unit/target_spec.rb:191 # Bolt::Target when parsing userinfo returns set object with feature_set
d6c5158
to
5a1dfa7
Compare
@donoghuc I've fixed these with what I believe to be a correct fix (if it looks like a URI, don't try wildcard matching on it) but one seemingly unrelated test failed. Maybe it would work if the CI run was re-started. |
triggering a CI run by opening and then closing this PR. |
Ah, so it looks like this actually exposed a small issue in our tests #3318 I think the fact that this now results in raising a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@seanmil I just pushed the test fix from Cas to main, if you rebase this on that and we see a green run we'll merge.
When resolving a large number of targets from a plan the match_wildcard? regex is compiled on the fly possibly hundreds or thousands (for a large inventory) of times per target name resolution. Switching to File.fnmatch provides a significant speed increase and seems to more accurately reflect the functionality stated in the Bolt documentation with regards to target matching. !feature * **Enable basic glob matching of target strings** Previously, only the '*' wildcard character would match when targets were resolved from areas other than the CLI (such as from within a Plan). With this change wildcard matching is switched to use Ruby's fnmatch with basic glob matching enabled. In addition to providing more wildcard matching options, it also provides a significant performance improvement over the prior implementation.
Performing wildcard target name resolution is fairly costly, having to search through all group names, target names, and target aliases, calling File.fnmatch on each one in order to find matches. In the case of the targets being non-wildcard strings much faster exact matching can be performed resulting in a significant performance improvement. !feature * **Optimize get_targets performance for exact-match cases** Attempt exact target name matches for strings passed to get_targets and only attempt the slower wildcard match if the string contains a valid glob wildcard character.
5a1dfa7
to
4d6e563
Compare
All green (finally!), thanks! |
Thank you! |
Update the target string name resolution behavior with two significant improvements: