Implement matching stats for SQL #3610

jnsrnhld · 2024-10-25T10:58:17Z

No description provided.

This reverts commit 63a5832.

jnsrnhld · 2024-10-25T11:33:45Z

Der RestartTest schlägt aktuell fehl, mir ist nicht ganz klar, warum. Alle anderen Tests laufen (lokal) durch.

awildturtok · 2024-10-28T09:02:56Z

Prefix und PrefixRange funktionieren mit dem Ansatz im Prinzip auch. Was noch nicht funktionieren wird, ist ColumnEqual und dadurch implizite Vererbung. Da musst du dann das group-by erweitern um die ColumnEqual Spalte(n).

jnsrnhld · 2024-10-28T09:41:42Z

@awildturtok Vielleicht liegt hier ein Missverständnis vor: Prefix und PrefixRange sind bereits in diesem Ansatz und PR enthalten. Konzepte mit ColumnEqual werden nach "altem" Ansatz berechnet. Die Grundidee dieses neuen Ansatzes ist es ja, eine gewisse Art von Konzepten gesondert zu berechnen und die anderen nach "altem" Ansatz.

Gibt es denn außer EBM (Regional) noch große Konzeptbäume, die ColumnEqual-Conditions verwenden? Ansonsten wäre das Verbesserungspotential marginal, aber der Code würde nochmal deutlich komplexer werden.

thoniTUB · 2024-10-28T10:54:12Z

Es scheint ein Jackson Problem zu geben, kommt mir auch etwas bekannt vor

2024-10-27T14:47:44.6250900Z Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.bakdata.conquery.models.datasets.concepts.MatchingStats` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
2024-10-27T14:47:44.6254795Z  at [Source: (GZIPInputStream); byte offset: #91] (through reference chain: com.bakdata.conquery.models.datasets.concepts.tree.TreeConcept["matchingStats"])
2024-10-27T14:47:44.6256929Z 	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
2024-10-27T14:47:44.6259162Z 	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1887)
2024-10-27T14:47:44.6260953Z 	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
2024-10-27T14:47:44.6262849Z 	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1375)
2024-10-27T14:47:44.6264874Z 	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserialize(AbstractDeserializer.java:274)
2024-10-27T14:47:44.6266727Z 	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
2024-10-27T14:47:44.6268885Z 	at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserializeFromObject(SuperSonicBeanDeserializer.java:295)
2024-10-27T14:47:44.6270981Z 	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:220)
2024-10-27T14:47:44.6272685Z 	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
2024-10-27T14:47:44.6274636Z 	at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:116)
2024-10-27T14:47:44.6277028Z 	at com.fasterxml.jackson.databind.deser.std.StdDelegatingDeserializer.deserialize(StdDelegatingDeserializer.java:170)
2024-10-27T14:47:44.6279447Z 	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
2024-10-27T14:47:44.6281990Z 	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
2024-10-27T14:47:44.6284458Z 	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
2024-10-27T14:47:44.6286510Z 	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
2024-10-27T14:47:44.6288590Z 	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
2024-10-27T14:47:44.6290620Z 	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2125)
2024-10-27T14:47:44.6291946Z 	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1501)
2024-10-27T14:47:44.6293430Z 	at com.bakdata.conquery.io.storage.xodus.stores.SerializingStore.read(SerializingStore.java:344)
2024-10-27T14:47:44.6294674Z 	... 98 common frames omitted

awildturtok · 2024-10-28T10:57:51Z

Hier fehlen CPSTypes, sonst kann Jackson nicht zwischen den 2 Klassen unterscheiden. Vlt kannst du auch den Storage parametrisieren?

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/MatchingStats.java

backend/src/main/java/com/bakdata/conquery/mode/local/SqlMatchingStats.java

backend/src/main/java/com/bakdata/conquery/mode/local/SqlUpdateMatchingStatsJob.java

.../src/main/java/com/bakdata/conquery/models/datasets/concepts/conditions/PrefixCondition.java

backend/src/main/java/com/bakdata/conquery/models/worker/LocalNamespace.java

backend/src/main/java/com/bakdata/conquery/models/worker/Namespace.java

backend/src/main/java/com/bakdata/conquery/sql/execution/DefaultResultSetProcessor.java

backend/src/test/resources/shared/icd.test.json

backend/src/main/java/com/bakdata/conquery/mode/local/UpdateMatchingStatsSqlJob.java

backend/src/main/java/com/bakdata/conquery/mode/local/SqlUpdateMatchingStatsJob.java

jnsrnhld · 2024-11-05T12:29:48Z

@awildturtok Kannst du bitte deinen Stand nochmal pushen?

This reverts commit 20fc92e.

awildturtok

Sind vor allem aufräum sachen, sieht gut aus!

backend/src/main/java/com/bakdata/conquery/mode/local/UpdateMatchingStatsSqlJob.java

.../src/main/java/com/bakdata/conquery/models/datasets/concepts/conditions/PrefixCondition.java

jnsrnhld · 2024-11-06T13:55:27Z

Ich warte dann auf deinen Push bevor ich ggf. etwas ändere.

additionally minor simplification in UpdateMatchingStatsSqlJob

awildturtok · 2024-11-06T14:13:49Z

@jnsrnhld habe meine Anmerkungen direkt angepasst

jnsrnhld · 2024-11-06T15:00:23Z

@awildturtok Kannst du mich hier einmal anpingen wenn du fertig bist?

awildturtok · 2024-11-06T15:44:59Z

@jnsrnhld der zweite Fehldurchlauf war flaky, sollte jetzt passen

jnsrnhld · 2024-11-07T07:51:39Z

Also wenn ich jetzt nichts übersehen hab hast du alle deine Anmerkungen ja selber bearbeitet. Ist von meiner Seite noch was zu tun hier?

This reverts commit 9ab227e.

This reverts commit 98f5902.

jnsrnhld added 6 commits October 16, 2024 21:38

Implement MatchingStats for SQL mode

2a04364

Submit matchings stats collection tasks in parallel

f5955d5

Track time while calculating matching stats

467abcc

Deactivate mappings and filter job

63a5832

Draft for fast matching stats calculation of only-equal-condition trees

2e130ba

Revert "Deactivate mappings and filter job"

cecd791

This reverts commit 63a5832.

jnsrnhld added backend Project SQL labels Oct 25, 2024

jnsrnhld changed the title ~~Improve SQL matching stats performance~~ Implement matching stats for SQL Oct 25, 2024

jnsrnhld added 2 commits October 25, 2024 13:19

Revert changes to MetadataCollectionTest

7e21052

Revert changes to ConceptTreeNode

52d097e

jnsrnhld added 3 commits October 26, 2024 11:03

Simplify condition match methods

7f7523c

Use SqlMatchingStats::empty for more fluent API

a87950c

Remove parallel calls

de11866

jnsrnhld added 2 commits October 29, 2024 16:13

Use parallel in setAndAggregate

4797e82

Simplify SQL matching stats calc

a8bf45c

jnsrnhld commented Oct 30, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/MatchingStats.java Show resolved Hide resolved

jnsrnhld and others added 2 commits October 30, 2024 15:53

Add missing getter and setter

9bf3c17

Merge branch 'develop' into sql/matching-stats-perf

1c02de1

jnsrnhld requested a review from awildturtok October 30, 2024 15:08

awildturtok reviewed Oct 30, 2024

View reviewed changes

jnsrnhld added 4 commits October 31, 2024 08:54

Cleanup

944aca7

Remove kh-content.csv

7e60026

More cleanup

c23e4b6

Use ListenableFutures

c850c4e

Revert changes in UpdateMatchingStatsMessage

5be35d6

jnsrnhld commented Oct 31, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/mode/local/UpdateMatchingStatsSqlJob.java Outdated Show resolved Hide resolved

awildturtok reviewed Nov 5, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/mode/local/SqlUpdateMatchingStatsJob.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/mode/local/SqlUpdateMatchingStatsJob.java Outdated Show resolved Hide resolved

jnsrnhld force-pushed the sql/matching-stats-perf branch from c36164e to 5be35d6 Compare November 5, 2024 09:49

Draft

0988c8c

awildturtok force-pushed the sql/matching-stats-perf branch from 25c5bba to 213358f Compare November 5, 2024 13:03

Refactoring

d0b02b7

jnsrnhld force-pushed the sql/matching-stats-perf branch from 213358f to d0b02b7 Compare November 6, 2024 08:42

jnsrnhld and others added 4 commits November 6, 2024 09:51

Revert changes

20fc92e

Merge branch 'develop' into sql/matching-stats-perf

dba40f9

Revert "Revert changes"

2f2a4ae

This reverts commit 20fc92e.

Fix

edd2f73

jnsrnhld requested a review from awildturtok November 6, 2024 12:17

awildturtok reviewed Nov 6, 2024

View reviewed changes

cleanup CTCondition.getAuxillaryColumns

944fc14

additionally minor simplification in UpdateMatchingStatsSqlJob

ignore auxillary columns

6f4cd6a

jnsrnhld marked this pull request as ready for review November 7, 2024 07:51

awildturtok and others added 6 commits November 11, 2024 11:05

dont min/max when ungrouped

98f5902

Fix least/greatest usage when there is only 1 validity date

9ab227e

Revert "Fix least/greatest usage when there is only 1 validity date"

e794f67

This reverts commit 9ab227e.

Revert "dont min/max when ungrouped"

546b0e4

This reverts commit 98f5902.

Only use least/greatest for multiple validity dates

4c724a3

Merge branch 'develop' into sql/matching-stats-perf

073f10e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement matching stats for SQL #3610

Implement matching stats for SQL #3610

jnsrnhld commented Oct 25, 2024 •

edited

Loading

jnsrnhld commented Oct 25, 2024

awildturtok commented Oct 28, 2024

jnsrnhld commented Oct 28, 2024

thoniTUB commented Oct 28, 2024

awildturtok commented Oct 28, 2024

jnsrnhld commented Nov 5, 2024

awildturtok left a comment

jnsrnhld commented Nov 6, 2024

awildturtok commented Nov 6, 2024

jnsrnhld commented Nov 6, 2024

awildturtok commented Nov 6, 2024

jnsrnhld commented Nov 7, 2024

Implement matching stats for SQL #3610

Are you sure you want to change the base?

Implement matching stats for SQL #3610

Conversation

jnsrnhld commented Oct 25, 2024 • edited Loading

jnsrnhld commented Oct 25, 2024

awildturtok commented Oct 28, 2024

jnsrnhld commented Oct 28, 2024

thoniTUB commented Oct 28, 2024

awildturtok commented Oct 28, 2024

jnsrnhld commented Nov 5, 2024

awildturtok left a comment

Choose a reason for hiding this comment

jnsrnhld commented Nov 6, 2024

awildturtok commented Nov 6, 2024

jnsrnhld commented Nov 6, 2024

awildturtok commented Nov 6, 2024

jnsrnhld commented Nov 7, 2024

jnsrnhld commented Oct 25, 2024 •

edited

Loading