Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

serialize filewise args to 'grep' and 'git grep' #153

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

nasamuffin
Copy link

@nasamuffin nasamuffin commented Mar 10, 2020

Issue #, if available:
Fixes #192

Description of changes:
There is a maximum number of args which can be passed to via the command line. To avoid it when invoking 'git-grep'
or 'grep', let's use xargs to invoke a new process when the number of args exceeds the system limit. Semantically the results should be identical.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@nasamuffin nasamuffin force-pushed the serialize-greps branch 2 times, most recently from 8c85efd to f1a905c Compare March 16, 2020 23:48
When running tests locally outside of a container (e.g. git clone
git-secrets; make test), if a user is already using git-secrets by
default in their system config and default gitdir template, tests which
expect not to have git-secrets installed will fail. Instead, let's
remove all hooks when we create a directory without the testbench's
template and ignore the system config which may contain patterns that
conflict with the testbench.
@jrn
Copy link

jrn commented Mar 30, 2020

(still figuring out the GitHub UI, sorry).

Can there be a test of the behavior with a high number of files?

@jrn
Copy link

jrn commented Mar 30, 2020

Just some coding style nits.

Still looking for tests as well.

@nasamuffin nasamuffin force-pushed the serialize-greps branch 2 times, most recently from b12b78c to 788c2f0 Compare April 2, 2020 23:22
Copy link

@jrn jrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

git-secrets Outdated Show resolved Hide resolved
git-secrets Outdated Show resolved Hide resolved
git-secrets Outdated Show resolved Hide resolved
git-secrets Outdated Show resolved Hide resolved
test/pre-commit.bats Outdated Show resolved Hide resolved
Copy link

@jrn jrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two nits left.

git-secrets Show resolved Hide resolved
git-secrets Outdated Show resolved Hide resolved
Passing a very long argument list to git-grep can cause it to fail;
indeed, it's possible for the list of paths passed by git-secrets to
either grep or git-grep to exceed the maximum number of arguments
allowed in a user's environment (`getconf ARG_MAX`). Instead, let xargs
check that the number of arguments won't exceed the system limit.

Signed-off-by: Emily Shaffer <[email protected]>
Copy link

@jrn jrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for your patient work.

Copy link

@jrn jrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more nit. Sorry I missed it before.

*) exit 255 ;;
esac' - \
-nwHEI "${options}" "${combined_patterns}" --
status=$?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May want this to be local. Or:

case $? in
  0) ...

}

# Performs a regular grep, taking into account patterns and recursion.
# Note: this function returns 1 on success, 0 on error.
regular_grep() {
local files=("${@}") patterns=$(load_patterns) action='skip'
local status=0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise, can avoid this temporary variable by using $? directly

@sparr
Copy link
Contributor

sparr commented Jun 21, 2023

@nasamuffin Could you rebase this without #155? It would be easier to proceed on the two changes independently. You might also rebase on master to replace my merge commit above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The invocations to git grep fails if the total length of filepaths causes the command line be over 32767
3 participants