Skip to content

Commit

Permalink
Add ::QueryString::Text to replace ::AutoEscape
Browse files Browse the repository at this point in the history
* Replace `::QueryString::AutoEscape` with `::QueryString::Text`
* Deprecate `::QueryString::AutoEscape` to prevent it from loading
* Move `=field:value` for `term` queries to `::QueryString::Text`
* Add `*field:value` for `wildcard` queries
* Add `~field:value` for `fuzzy` queries
* Add `/field:value` for `regexp` queries
* Add `+field:value` for `match_phrase` queries
* Automatically promote queries against `text` fields to `match` queries
  unless otherwise specified
* Add tests for the behavior
  • Loading branch information
reyjrar committed Nov 11, 2023
1 parent 6a80325 commit 37d0dd7
Show file tree
Hide file tree
Showing 11 changed files with 422 additions and 89 deletions.
74 changes: 66 additions & 8 deletions CopyIndexes.mkdn
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ es-copy-index.pl - Copy an index from one cluster to another

# VERSION

version 8.7
version 8.8

# SYNOPSIS

Expand Down Expand Up @@ -177,7 +177,17 @@ The **incident-rt1234-2013.01.11** index will now hold all the data from both of
The search string is pre-analyzed before being sent to ElasticSearch. The following plugins
work to manipulate the query string and provide richer, more complete syntax for CLI applications.

## App::ElasticSearch::Utilities::QueryString::AutoEscape
## App::ElasticSearch::Utilities::QueryString::Barewords

The following barewords are transformed:

or => OR
and => AND
not => NOT

## App::ElasticSearch::Utilities::QueryString::Text

### Terms Query via '='

Provide an '=' prefix to a query string parameter to promote that parameter to a `term` filter.

Expand All @@ -195,15 +205,63 @@ Is translated into:

{ term => { user_agent => "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1" } }

Which provides an exact match to the term in the query.
### Wildcard Query via '\*'

## App::ElasticSearch::Utilities::QueryString::Barewords
Provide an '\*' prefix to a query string parameter to promote that parameter to a `wildcard` filter.

The following barewords are transformed:
This uses the wild card match for text fields to making matching more intuitive.

or => OR
and => AND
not => NOT
E.g.:

*user_agent:"Mozilla*"

Is translated into:

{ wildcard => { user_agent => "Mozilla* } }

### Regexp Query via '/'

Provide an '/' prefix to a query string parameter to promote that parameter to a `regexp` filter.

If you want to use regexp matching for finding data, you can use:

/message:'\\bden(ial|ied|y)'

Is translated into:

{ regexp => { message => "\\bden(ial|ied|y)" } }

### Fuzzy Matching via '~'

Provide an '~' prefix to a query string parameter to promote that parameter to a `fuzzy` filter.

~message:deny

Is translated into:

{ fuzzy => { message => "deny" } }

### Phrase Matching via '+'

Provide an '+' prefix to a query string parameter to promote that parameter to a `match_phrase` filter.

+message:"login denied"

Is translated into:

{ match_phrase => { message => "login denied" } }

### Automatic Match Queries for Text Fields

If the field meta data is provided and the field is a `text` type, the query
will automatically be mapped to a `match` query.

# message field is text
message:"foo"

Is translated into:

{ match => { message => "foo" } }

## App::ElasticSearch::Utilities::QueryString::IP

Expand Down
2 changes: 1 addition & 1 deletion Maintenance.mkdn
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ es-daily-index-maintenance.pl - Run to prune old indexes and optimize existing

# VERSION

version 8.7
version 8.8

# SYNOPSIS

Expand Down
2 changes: 1 addition & 1 deletion README.mkdn
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch

# VERSION

version 8.7
version 8.8

# SYNOPSIS

Expand Down
74 changes: 66 additions & 8 deletions Searching.mkdn
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ es-search.pl - Provides a CLI for quick searches of data in ElasticSearch daily

# VERSION

version 8.7
version 8.8

# SYNOPSIS

Expand Down Expand Up @@ -386,7 +386,17 @@ es-search.pl - Search a logging cluster for information
The search string is pre-analyzed before being sent to ElasticSearch. The following plugins
work to manipulate the query string and provide richer, more complete syntax for CLI applications.

## App::ElasticSearch::Utilities::QueryString::AutoEscape
## App::ElasticSearch::Utilities::QueryString::Barewords

The following barewords are transformed:

or => OR
and => AND
not => NOT

## App::ElasticSearch::Utilities::QueryString::Text

### Terms Query via '='

Provide an '=' prefix to a query string parameter to promote that parameter to a `term` filter.

Expand All @@ -404,15 +414,63 @@ Is translated into:

{ term => { user_agent => "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1" } }

Which provides an exact match to the term in the query.
### Wildcard Query via '\*'

## App::ElasticSearch::Utilities::QueryString::Barewords
Provide an '\*' prefix to a query string parameter to promote that parameter to a `wildcard` filter.

The following barewords are transformed:
This uses the wild card match for text fields to making matching more intuitive.

or => OR
and => AND
not => NOT
E.g.:

*user_agent:"Mozilla*"

Is translated into:

{ wildcard => { user_agent => "Mozilla* } }

### Regexp Query via '/'

Provide an '/' prefix to a query string parameter to promote that parameter to a `regexp` filter.

If you want to use regexp matching for finding data, you can use:

/message:'\\bden(ial|ied|y)'

Is translated into:

{ regexp => { message => "\\bden(ial|ied|y)" } }

### Fuzzy Matching via '~'

Provide an '~' prefix to a query string parameter to promote that parameter to a `fuzzy` filter.

~message:deny

Is translated into:

{ fuzzy => { message => "deny" } }

### Phrase Matching via '+'

Provide an '+' prefix to a query string parameter to promote that parameter to a `match_phrase` filter.

+message:"login denied"

Is translated into:

{ match_phrase => { message => "login denied" } }

### Automatic Match Queries for Text Fields

If the field meta data is provided and the field is a `text` type, the query
will automatically be mapped to a `match` query.

# message field is text
message:"foo"

Is translated into:

{ match => { message => "foo" } }

## App::ElasticSearch::Utilities::QueryString::IP

Expand Down
11 changes: 10 additions & 1 deletion examples/es-parse-query-string.pl
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,19 @@

my %OPT;
GetOptions(\%OPT, qw(
or
or
field=s%
));
my $json = JSON->new->ascii->canonical(1)->pretty;
my %fields = ();
if( $OPT{field} ) {
foreach my $f ( keys %{ $OPT{field} } ) {
$fields{$f} = { type => $OPT{field}->{$f} },
}
output({color=>'yellow'}, "Fields: " . $json->encode(\%fields));
}
my $qs = App::ElasticSearch::Utilities::QueryString->new(
fields_meta => \%fields,
default_join => $OPT{or} ? 'OR' : 'AND',
);
my $query = $qs->expand_query_string(@ARGV);
Expand Down
14 changes: 10 additions & 4 deletions lib/App/ElasticSearch/Utilities/QueryString.pm
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,18 @@ sub _build_plugins {
my $globals = es_globals('plugins');
my $finder = Module::Pluggable::Object->new(
search_path => ['App::ElasticSearch::Utilities::QueryString',@{ $self->search_path }],
except => [qw(App::ElasticSearch::Utilities::QueryString::Plugin)],
except => [qw(
App::ElasticSearch::Utilities::QueryString::AutoEscape
App::ElasticSearch::Utilities::QueryString::Plugin
)],
instantiate => 'new',
);
my @plugins;
foreach my $p ( sort { $a->priority <=> $b->priority || $a->name cmp $b->name }
$finder->plugins( options => defined $globals ? $globals : {} )
$finder->plugins(
fields_meta => $self->fields_meta,
options => defined $globals ? $globals : {},
)
) {
debug(sprintf "Loaded %s with priority:%d", $p->name, $p->priority);
push @plugins, $p;
Expand Down Expand Up @@ -246,10 +252,10 @@ words to prevent syntax errors.
The search string is pre-analyzed before being sent to ElasticSearch. The following plugins
work to manipulate the query string and provide richer, more complete syntax for CLI applications.
=from_other App::ElasticSearch::Utilities::QueryString::AutoEscape / SYNOPSIS
=from_other App::ElasticSearch::Utilities::QueryString::BareWords / SYNOPSIS
=from_other App::ElasticSearch::Utilities::QueryString::Text / SYNOPSIS
=from_other App::ElasticSearch::Utilities::QueryString::IP / SYNOPSIS
=from_other App::ElasticSearch::Utilities::QueryString::Ranges / SYNOPSIS
Expand Down
62 changes: 0 additions & 62 deletions lib/App/ElasticSearch/Utilities/QueryString/AutoEscape.pm

This file was deleted.

14 changes: 13 additions & 1 deletion lib/App/ElasticSearch/Utilities/QueryString/Plugin.pm
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use warnings;
use Hash::Merge::Simple qw(clone_merge);
use Moo::Role;
use Ref::Util qw(is_arrayref is_hashref);
use Types::Standard qw( Str Int );
use Types::Standard qw( HashRef Str Int );

=attr name
Expand Down Expand Up @@ -41,6 +41,18 @@ has priority => (
);
sub _build_priority { 50; }

=attr fields_meta
A hash reference with the field data from L<App::ElasticSearch::Utilities::es_index_fields>.
=cut

has fields_meta => (
is => 'rw',
isa => HashRef,
default => sub { {} },
);

=head1 INTERFACE
=head2 handle_token()
Expand Down
Loading

0 comments on commit 37d0dd7

Please sign in to comment.