Adding support for defining filters on measures #3624

AdityaHegde · 2023-12-05T13:50:49Z

This is a backend only PR to support filters on measures

begelundmuller · 2023-12-11T17:30:59Z

proto/rill/runtime/v1/expressions.proto

+message Expression {
+  oneof expression {
+    google.protobuf.Value value = 1;
+    string column = 2;


Name suggestion: identifier and make it 1 instead of 2

begelundmuller · 2023-12-11T17:33:05Z

proto/rill/runtime/v1/expressions.proto

+  }
+}
+
+message ConditionExpression {


Suggestion: Just name it Condition

begelundmuller · 2023-12-11T17:33:41Z

proto/rill/runtime/v1/queries.proto

@@ -8,6 +8,7 @@ import "rill/runtime/v1/export_format.proto";
 import "rill/runtime/v1/schema.proto";
 import "rill/runtime/v1/time_grain.proto";
 import "validate/validate.proto";
+import "expressions.proto";


Full path like above, i.e. rill/runtime/v1/expressions.proto

Singular, i.e. expression.proto

begelundmuller · 2023-12-11T17:35:25Z

proto/rill/runtime/v1/expressions.proto

+  OPERATION_EQUALS = 1;
+  OPERATION_NOT_EQUALS = 2;
+  OPERATION_LESSER = 3;
+  OPERATION_LESSER_OR_EQUALS = 4;
+  OPERATION_GREATER = 5;
+  OPERATION_GREATER_OR_EQUALS = 6;


Is it more common to use "equals" or "equal"?

Also, should we consider the usual shorthands, like EQ, NEQ, LT, LTE, etc.?

begelundmuller · 2023-12-11T17:37:00Z

proto/rill/runtime/v1/expressions.proto

+  OPERATION_BETWEEN = 9;
+  OPERATION_NOT_BETWEEN = 10;


Redundant – would suggest using an "and" of two expressions instead. It's also ambiguous – i.e. are they inclusive or exclusive of the start and end values?

These are from duckdb docs: https://duckdb.org/docs/sql/expressions/comparison_operators.html#between-and-is-not-null
I think we can convert them to compound operations in UI and keep the API simple.

begelundmuller · 2023-12-11T17:38:52Z

proto/rill/runtime/v1/expressions.proto

@@ -0,0 +1,35 @@
+syntax = "proto3";


Question: not sure what I think, but do you think we should consider shorter type names given the nesting of expressions and how often we'll end up looking at them printed?

I.e. using terms like expr (expression), cond (condition), val (value), op (operation), args (operands), ident (identifer), eq (equal), etc.?

For example, MongoDB does that: https://www.mongodb.com/docs/manual/reference/operator/query/expr/.

I would also suggest looking out for prior art for expressions expressed in protobufs. Two good places to look are:

https://github.com/googleapis/googleapis

https://buf.build/explore

begelundmuller · 2023-12-11T17:43:25Z

proto/rill/runtime/v1/queries.proto

@@ -279,7 +280,8 @@ message MetricsViewAggregationRequest {
  TimeRange time_range = 12;
  google.protobuf.Timestamp time_start = 6; // Deprecated in favor of time_range
  google.protobuf.Timestamp time_end = 7; // Deprecated in favor of time_range
-  MetricsViewFilter filter = 8;
+  Expression filter = 8;
+  repeated MetricsViewColumnAlias aliases = 13;


Why are aliases needed in other cases than MetricsViewComparison? For the other cases, I think it's okay to require clients to use the real names (since there's no ambiguity around base vs. comparison vs. delta values).

The idea was to use the same measure in measures and filter. Aggregation allows for count and count distinct so I thought those could be defined in aliases and used in measures and filter fields.

begelundmuller · 2023-12-11T17:45:02Z

proto/rill/runtime/v1/queries.proto

+message MetricsViewColumnAlias {
+  string name = 1;
+  oneof alias { // Is this overkill to future proof this?
+    MetricsViewMeasureAlias measure_alias = 2;
+  }
+}


Yeah I think we should just flatten it. In light of the comment above, just one type MetricsViewComparisonMeasureAlias might be enough.

begelundmuller · 2023-12-11T17:46:17Z

proto/rill/runtime/v1/queries.proto

+    MEASURE_TYPE_COUNT = 1;
+    MEASURE_TYPE_COUNT_DISTINCT = 2;


These can already be defined with an alias through MetricsViewAggregationMeasure

begelundmuller · 2023-12-11T17:48:28Z

proto/rill/runtime/v1/queries.proto

@@ -279,7 +280,8 @@ message MetricsViewAggregationRequest {
  TimeRange time_range = 12;
  google.protobuf.Timestamp time_start = 6; // Deprecated in favor of time_range
  google.protobuf.Timestamp time_end = 7; // Deprecated in favor of time_range
-  MetricsViewFilter filter = 8;
+  Expression filter = 8;


Need to add support for HAVING clauses (different from WHERE clauses).

Maybe we refactor to Expression having and Expression where? Or to avoid that, could keep Expression filter and add Expression having or Expression measure_filter?

Ya i think separating will keep it cleaner. Combined filter was not clean in the code.

Also having makes more sense. We can think of using where for the main filter in the near future.

begelundmuller · 2023-12-12T11:03:13Z

proto/rill/runtime/v1/queries.proto

+message MetricsViewMeasureAlias {
+  enum MeasureType {
+    MEASURE_TYPE_UNSPECIFIED = 0;
+    MEASURE_TYPE_BASE_VALUE = 1;
+    MEASURE_TYPE_COMPARISON_VALUE = 2;
+    MEASURE_TYPE_ABS_DELTA = 3;
+    MEASURE_TYPE_REL_DELTA = 4;
+  }
+
+  string name = 1;
+  MeasureType type = 2;
+  repeated google.protobuf.Value args = 3;
+  string alias = 4;
+}


Would incorporate Comparison in the name since this is specific to that use case (no other API supports base/comparison/delta values)

Should reuse (and probably rename) the MetricsViewComparisonSortType type so we don't have two enums doing the same

begelundmuller · 2023-12-12T11:06:00Z

proto/rill/runtime/v1/queries.proto

+  Condition where = 8;
+  Condition having = 14;


Consider using Expression directly, simpler to have one entrypoint (and matches the name of the file). Also slightly more flexible in that you can put something like just WHERE true

proto/rill/runtime/v1/queries.proto

runtime/queries/metricsview_comparison_toplist.go

runtime/queries/metricsview.go

begelundmuller · 2023-12-14T17:54:53Z

runtime/queries/metricsview.go

+func dimensionAliases(mv *runtimev1.MetricsViewSpec) map[string]identifier {
+	aliases := map[string]identifier{}
+	for _, dim := range mv.Dimensions {
+		aliases[dim.Name] = identifier{safeName(metricsViewDimensionColumn(dim)), dim.Unnest}
+	}
+	return aliases
+}


The previous buildFilterClauseForMetricsViewFilter implementation avoided building a lookup map since the overhead of building it (allocating/assigning a ~20 element map, plus a similar number of allocations for safeName) would be much higher than just iterating over mv.Dimensions a few times (most filters apply to just one or two dimensions).

Always hard to balance optimization vs. convenience, and not sure how much this matters, but would consider staying with the format of buildFilterClauseForMetricsViewFilter (i.e. passing mv and aliases directly as args; and maybe use a boolean like isHaving to indicate which to use for lookups).

Ya makes sense. We should really look at having some global cache per metrics view. Things would be very clean with a map. But for now i have updated to use mv and aliases

runtime/queries/metricsview.go

runtime/queries/filterutil.go

runtime/queries/metricsview.go

runtime/queries/metricsview_comparison_toplist.go

begelundmuller · 2023-12-15T14:28:33Z

runtime/queries/metricsview_comparison_toplist.go

+			if q.Having != nil {
+				// having clause needs selected columns to either be in group by or be aggregations.
+				// so adding additional sum() around measure and comparison columns
+				columnsTuple = fmt.Sprintf(
+					"sum(base.%[1]s) as %[1]s, sum(comparison.%[1]s) AS %[2]s, sum(base.%[1]s - comparison.%[1]s) AS %[3]s, sum((base.%[1]s - comparison.%[1]s)/comparison.%[1]s::DOUBLE) AS %[4]s",
+					safeName(m.Name),
+					safeName(m.Name+"__previous"),
+					safeName(m.Name+"__delta_abs"),
+					safeName(m.Name+"__delta_rel"),
+				)
+			} else {
+				columnsTuple = fmt.Sprintf(
+					"base.%[1]s, comparison.%[1]s AS %[2]s, base.%[1]s - comparison.%[1]s AS %[3]s, (base.%[1]s - comparison.%[1]s)/comparison.%[1]s::DOUBLE AS %[4]s",
+					safeName(m.Name),
+					safeName(m.Name+"__previous"),
+					safeName(m.Name+"__delta_abs"),
+					safeName(m.Name+"__delta_rel"),
+				)
+			}


Is this needed because the actual aggregation happens in a sub-query? If yes, then instead of adding a new layer of aggregations, couldn't we just use a WHERE clause in the outer select instead? (Since a HAVING clause is basically equivalent to a WHERE clause in a wrapped query, or am I missing something?)

If the aggregation is needed, consider using ANY_VALUE or LAST, which I believe we use for such cases in other places (less ambiguous than SUM).

Good point. I have updated to use an outer where clause.

begelundmuller · 2023-12-18T10:53:50Z

runtime/queries/metricsview.go

+			case runtimev1.MetricsViewComparisonMeasureType_METRICS_VIEW_COMPARISON_MEASURE_TYPE_UNSPECIFIED,
+				runtimev1.MetricsViewComparisonMeasureType_METRICS_VIEW_COMPARISON_MEASURE_TYPE_BASE_VALUE:
+				// using `measure_0` as is causing ambiguity error in duckdb
+				if dialect == drivers.DialectDuckDB {
+					return "base." + safeName(alias.Name), true
+				}


The columnIdentifierExpression function is not specific to the MetricsViewComparison API, so this doesn't seem safe?

Also, is the clause being templated in the correct place? I don't believe this was an issue before, so wondering if the ambiguity is due to some other issue

Added an outer query for duckdb to get rid of the ambiguity.

runtime/queries/metricsview_comparison_toplist_test.go

runtime/queries/metricsview_timeseries_test.go

runtime/testruntime/testruntime.go

AdityaHegde added 5 commits December 5, 2023 19:20

Initial commit

2f56587

Using enum for expression and adding to metricsview_aggregation

102ac29

Merge branch 'main' into adityahegde/filters-on-measures

ccfeb0e

Comparison in measure filters

d439c54

Adding to metricsview toplist

655ab2e

AdityaHegde self-assigned this Dec 7, 2023

Updated proto spec

4808486

begelundmuller reviewed Dec 11, 2023

View reviewed changes

Add generic expression syntax

74bddbd

begelundmuller reviewed Dec 12, 2023

View reviewed changes

AdityaHegde added 7 commits December 13, 2023 12:30

Final proto draft

3bc151a

Integrate and test where filter

d0d5228

Integrate where clause everywhere

0f9c159

Merge branch 'main' into adityahegde/filters-on-measures

171a7f2

Adding some tests for having clauses

9d9b016

Add backwards compatibility for filter

aa1b379

fix lint

e9ff93b

AdityaHegde force-pushed the adityahegde/filters-on-measures branch from ebe7390 to e9ff93b Compare December 14, 2023 07:26

Adding alias support in metricsview_comparison_toplist

a828601

AdityaHegde marked this pull request as ready for review December 14, 2023 08:16

AdityaHegde requested a review from begelundmuller December 14, 2023 08:16

Apply sum() only when having is enabled

5a7526a

AdityaHegde force-pushed the adityahegde/filters-on-measures branch from b0b4a36 to 5a7526a Compare December 14, 2023 08:39

Fix lint

38db0ff

begelundmuller requested changes Dec 14, 2023

View reviewed changes

AdityaHegde added 3 commits December 15, 2023 12:49

PR comments

0ddcba5

Move filter helpers to pkg/expressionpb

f02b422

refactor filter builder to not use map

a4086af

AdityaHegde force-pushed the adityahegde/filters-on-measures branch from 2ee87d4 to 2d27256 Compare December 15, 2023 12:20

Backwards compatibility for older sort types

1e49865

AdityaHegde force-pushed the adityahegde/filters-on-measures branch from 2d27256 to 1e49865 Compare December 15, 2023 12:48

begelundmuller requested changes Dec 15, 2023

View reviewed changes

PR comments

0d02580

begelundmuller approved these changes Dec 15, 2023

View reviewed changes

Testing on druid and fixes

025a62d

AdityaHegde force-pushed the adityahegde/filters-on-measures branch from af2c562 to 025a62d Compare December 18, 2023 06:43

begelundmuller requested changes Dec 18, 2023

View reviewed changes

AdityaHegde added 3 commits December 18, 2023 17:07

Merge branch 'main' into adityahegde/filters-on-measures

b5754b3

Using outer query

d1186d1

Removing unused testdata

1832ebf

AdityaHegde requested a review from begelundmuller December 18, 2023 15:40

begelundmuller approved these changes Dec 18, 2023

View reviewed changes

begelundmuller merged commit 07cadb9 into main Dec 18, 2023
5 checks passed

begelundmuller deleted the adityahegde/filters-on-measures branch December 18, 2023 16:24

This was referenced Dec 19, 2023

filter out dimension names that do not have dimension values #3726

Closed

move incomplete filters out of dashboard store and into a dedicated piece of state #3728

Merged

begelundmuller mentioned this pull request Dec 28, 2023

Runtime: Fix missing security policy row filters for new filter expressions #3753

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for defining filters on measures #3624

Adding support for defining filters on measures #3624

AdityaHegde commented Dec 5, 2023 •

edited

Loading

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

AdityaHegde Dec 12, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

AdityaHegde Dec 12, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

begelundmuller Dec 11, 2023

AdityaHegde Dec 12, 2023

AdityaHegde Dec 12, 2023

begelundmuller Dec 12, 2023

begelundmuller Dec 12, 2023

begelundmuller Dec 14, 2023

AdityaHegde Dec 15, 2023

begelundmuller Dec 15, 2023

AdityaHegde Dec 15, 2023

begelundmuller Dec 18, 2023

begelundmuller Dec 18, 2023

AdityaHegde Dec 18, 2023

Adding support for defining filters on measures #3624

Adding support for defining filters on measures #3624

Conversation

AdityaHegde commented Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdityaHegde commented Dec 5, 2023 •

edited

Loading