Pattern acceleration #87

elliVM · 2024-09-12T08:03:21Z

Pattern acceleration feature to activate bloom filtering on set regex pattern. Goal is to limit bloom filtering to certain patterns like UUID.

Flow:

Walker walks XML and for each search term regex checks tokenized term against tables in bloomdb excluding the filtertype table
Collect list of tables with a pattern match record from SQL metadata
For each table a temp table is created
For each temp table generate condition that compares temp table filter against parent table filter using bloommatch UDF
Combine conditions from each table, if all filters were null then also true
Combine conditions from each search term
Left join tables to NestedTopNQuery and use the final combined condition in query

Notes: Tokenizer max token count is set to 0 to get only major tokens since that is what dpf_03 currently uses to tokenize the bloom filter tables

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

src/main/java/com/teragrep/pth_06/planner/walker/ConditionWalker.java

src/main/java/com/teragrep/pth_06/planner/walker/conditions/ElementCondition.java

src/test/java/com/teragrep/pth_06/planner/walker/conditions/ElementConditionTest.java

… and IndexStatementConditionTest and tests for isBloomSearchCondition method

kortemik · 2024-09-13T07:49:54Z

src/main/java/com/teragrep/pth_06/config/ConditionConfig.java

    }

-    public ConditionConfig(DSLContext ctx, boolean streamQuery, boolean bloomEnabled, boolean withoutFilters) {


is this removing the feature that allows to search files without a filter?

Option was not used in github repo. Added option with implementation and tests

kortemik · 2024-09-13T07:56:21Z

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

+     * Inserts a filter for each filtertype inside parent table records. Filter is filled with search term token set,
+     * filter size is selected using parent table filtertype expected and fpp values.
+     */
+    private void insertFilters() {


please elaborate more what happens here, from the description it sounds like select all from specific filtertype into temp table

Method was poorly named. After refactoring TableFilters is responsible of category tables filters and inserting all filter types. class TableFilterTypesFromMetadata is used to fetch the tables filter types from metadata

kortemik · 2024-09-13T07:57:13Z

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

+        }
+    }
+
+    private void insertFilterFromRecord(


please elaborate more what happens here, it sounds like select filter from record into xyz

refactored into method filterBytesFromRecord in class TableFilters that creates a bloom filter with correct size and tokens.

kortemik · 2024-09-13T07:58:37Z

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

+    }
+
+    /**
+     * Generates a condition that returns true if this temp tables search term tokens might be contained in the parent


what is a parent table in this context?

Parent table was the bloom filter table that the temp table was created to match, renamed as originTable in new class CategoryTableImpl

kortemik · 2024-09-13T07:59:33Z

src/main/java/com/teragrep/pth_06/planner/PatternMatch.java

+
+import static com.teragrep.pth_06.jooq.generated.bloomdb.Bloomdb.BLOOMDB;
+
+public class PatternMatch {


what is the responsibility of objects in this class?

Responsibility was to get bloom filter tables that had record with a filtertype pattern that matched with any of the search term tokens. Refactored into two classes PatternMatchTables thet gets the tables from metadata and PatternMatchCondition that generates the condition used to do the pattern matching

src/main/java/com/teragrep/pth_06/planner/StreamDBClient.java

src/main/java/com/teragrep/pth_06/planner/walker/conditions/ElementCondition.java

…y and ensure single responsibility for objects

…ss to get pattern match tables

elliVM · 2024-09-18T07:38:29Z

Refactoring

New classes:

Class	Responsibility
PatternMatchTables	Finds bloomdb Tables that match a pattern condition
CategoryTableImpl	Temp table from a bloom filter table that can return a `CategoryTableCondition`
Created	`CategoryTable` that is created to database
WithFilterTypes	`CategoryTable` with its filters inserted
TableFilters	Inserts filters of a `CategoryTable`
TableFilterTypesFromMetadata	Fetches different filter types of a table from metadata
CategoryTableCondition	Condition that compares category tables filter bytes against boom filter table filter bytes with bloommatch UDF, selects the same size and bloom term id
PatternMatchCondition	Condition that check if any of given tokens match with `bloomdb.filtertype.pattern`

Other changes:

Many method naming changes
Added interfaces CategoryTable, TableRecords, BloomQueryCondition
Added missing withoutFilters option and implementation to IndexStatementCondition
Equality methods for all new classes
Tests for all new classes

eemhu · 2024-09-19T07:04:33Z

src/main/java/com/teragrep/pth_06/planner/Created.java

+
+/**
+ * Decorator that creates category table
+ */


The purpose of this object is not super clear to me, why not just call create() on the origin categoryTable?

Removed decorators in refactoring and just use CategoryTable interface methods create() and insertFilters().

src/test/java/com/teragrep/pth_06/planner/CategoryTableImplTest.java

src/test/java/com/teragrep/pth_06/planner/PatternMatchTablesTest.java

src/test/java/com/teragrep/pth_06/planner/TableFilterTypesFromMetadataResultTest.java

src/test/java/com/teragrep/pth_06/planner/TokenizedValueTest.java

src/test/java/com/teragrep/pth_06/planner/walker/conditions/CategoryTableConditionTest.java

…test assertions

eemhu · 2024-09-24T07:59:03Z

src/main/java/com/teragrep/pth_06/planner/walker/conditions/IndexStatementCondition.java

@@ -97,9 +97,9 @@ public Condition condition() {

            for (final Table<?> table : tableSet) {
                // create category temp table for this pattern match table
-                final CategoryTable categoryTable = new Created(
-                        new WithFilterTypes(new CategoryTableImpl(config, table, tokenizedValue))


Now that I think about it, perhaps the older way was better to reduce temporal coupling?

@kortemik what do you think?

yes, just with better naming.

kortemik · 2024-09-24T08:29:18Z

src/main/java/com/teragrep/pth_06/planner/walker/conditions/IndexStatementCondition.java

@@ -97,9 +97,9 @@ public Condition condition() {

            for (final Table<?> table : tableSet) {
                // create category temp table for this pattern match table
-                final CategoryTable categoryTable = new Created(
-                        new WithFilterTypes(new CategoryTableImpl(config, table, tokenizedValue))


yes, just with better naming.

…ategory table building and remove interfaces from tests

…ns using pattern from each filter_type_id

eemhu · 2024-09-24T12:53:09Z

src/main/java/com/teragrep/pth_06/planner/walker/conditions/IndexStatementCondition.java

            }
+            newCondition = config.withoutFilters() ? combinedNullFilterCondition : combinedTableCondition


as much as I like the ternary operator i don't think it is as clear as the if-statement

Changed to if-statement

src/test/java/com/teragrep/pth_06/planner/CategoryTableImplTest.java

…inserted to category table, some code cleanup

elliVM · 2024-09-30T05:12:00Z

Enabled token entanglement in tokenizer

elliVM added 2 commits September 12, 2024 10:54

merge squash pattern-acceleration-refactor

88df094

run jooq generate with new settings and spotless

d9730ae

kortemik requested a review from eemhu September 12, 2024 08:05

eemhu reviewed Sep 12, 2024

View reviewed changes

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java Outdated Show resolved Hide resolved

eemhu reviewed Sep 12, 2024

View reviewed changes

src/main/java/com/teragrep/pth_06/planner/walker/ConditionWalker.java Show resolved Hide resolved

rename BloomFilterTempTable tokenSet to searchTermTokenSet

dd42e06

eemhu reviewed Sep 12, 2024

View reviewed changes

src/main/java/com/teragrep/pth_06/planner/walker/conditions/ElementCondition.java Outdated Show resolved Hide resolved

make ConditionWalker and ValidElement classes final

b2aaf03

eemhu reviewed Sep 12, 2024

View reviewed changes

src/test/java/com/teragrep/pth_06/planner/walker/conditions/ElementConditionTest.java Show resolved Hide resolved

elliVM added 4 commits September 12, 2024 12:48

rename method isIndexStatement to isBloomSearchCondition

c1ae260

add indexstatement with bad connection tests for ElementConditionTest…

914ddff

… and IndexStatementConditionTest and tests for isBloomSearchCondition method

apply spotless

485b346

add javadoc to tests that use Condition.toString for equality testing

b3435ad

eemhu requested a review from kortemik September 12, 2024 12:52

kortemik reviewed Sep 13, 2024

View reviewed changes

elliVM added 4 commits September 16, 2024 15:28

refactoring iteration: multiple new classes and interfaces to simplif…

2916cd6

…y and ensure single responsibility for objects

apply spotless

be3c266

renaming of names and methods

1c31037

refactor ElementCondition to directly use IndexStatementCondition cla…

0c3ffcc

…ss to get pattern match tables

Renaming and some comments

0e00db0

kortemik requested review from eemhu and kortemik September 18, 2024 17:11