Resolves #3013: Support UDF #2995

pengpeng-lu · 2024-12-05T19:51:15Z

No description provided.

This reverts commit 3a5a44b.

pengpeng-lu · 2024-12-20T04:50:38Z

...rd-layer-core/src/main/java/com/apple/foundationdb/record/query/expressions/Comparisons.java

@@ -38,7 +38,6 @@
 import com.apple.foundationdb.record.metadata.expressions.InvertibleFunctionKeyExpression;
 import com.apple.foundationdb.record.metadata.expressions.TupleFieldsHelper;
 import com.apple.foundationdb.record.planprotos.PComparison;
-import com.apple.foundationdb.record.planprotos.PComparison.PComparisonType;


just changing package in this file

It looks like you just removed the import and then had to add package qualifiers to all the references. Unless I am getting it wrong, please undo as the code was more concise.

pengpeng-lu · 2024-12-20T04:50:49Z

...ore/src/main/java/com/apple/foundationdb/record/query/plan/IndexKeyValueToPartialRecord.java

@@ -40,7 +40,6 @@
 import com.apple.foundationdb.record.planprotos.PIndexKeyValueToPartialRecord.PFieldCopier;
 import com.apple.foundationdb.record.planprotos.PIndexKeyValueToPartialRecord.PFieldWithValueCopier;
 import com.apple.foundationdb.record.planprotos.PIndexKeyValueToPartialRecord.PMessageCopier;
-import com.apple.foundationdb.record.planprotos.PIndexKeyValueToPartialRecord.PTupleSource;


just changing package.

Same here. Please undo!

pengpeng-lu · 2024-12-20T04:51:15Z

...er-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/BuiltInFunction.java

@@ -63,7 +63,7 @@ public abstract class BuiltInFunction<T extends Typed> {
     * @param parameterTypes The type of the parameter(s).
     * @param encapsulationFunction An encapsulation of the function's runtime computation.
     */
-    protected BuiltInFunction(@Nonnull final String functionName, @Nonnull final List<Type> parameterTypes, @Nonnull final EncapsulationFunction<T> encapsulationFunction) {
+    public BuiltInFunction(@Nonnull final String functionName, @Nonnull final List<Type> parameterTypes, @Nonnull final EncapsulationFunction<T> encapsulationFunction) {


to let other repos use this creator class

This constructor can only be called from subclasses as this class is abstract. Every subclass is allowed to call this constructor regardless if in downstream or in this module. Please undo unless I misunderstood.

problem solved after MacroFunction introduced.

pengpeng-lu · 2024-12-20T04:51:23Z

...in/java/com/apple/foundationdb/record/query/plan/cascades/KeyExpressionExpansionVisitor.java

@@ -21,7 +21,7 @@
 package com.apple.foundationdb.record.query.plan.cascades;

 import com.apple.foundationdb.record.RecordCoreException;
-import com.apple.foundationdb.record.RecordMetaDataProto;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;


just changing package.

pengpeng-lu · 2024-12-20T04:51:34Z

.../src/main/java/com/apple/foundationdb/record/query/plan/cascades/NullableArrayTypeUtils.java

@@ -20,7 +20,7 @@

 package com.apple.foundationdb.record.query.plan.cascades;

-import com.apple.foundationdb.record.RecordMetaDataProto;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;


just changing package.

pengpeng-lu · 2024-12-20T04:52:06Z

...ore/src/test/java/com/apple/foundationdb/record/metadata/MetaDataEvolutionValidatorTest.java

@@ -22,6 +22,7 @@

 import com.apple.foundationdb.async.RankedSet;
 import com.apple.foundationdb.record.RecordCoreException;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;


just change package in this folder (metadata)

pengpeng-lu · 2024-12-20T04:52:23Z

...c/test/java/com/apple/foundationdb/record/metadata/expressions/LiteralKeyExpressionTest.java

@@ -20,7 +20,7 @@

 package com.apple.foundationdb.record.metadata.expressions;

-import com.apple.foundationdb.record.RecordMetaDataProto;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;


just changing package in this file

pengpeng-lu · 2024-12-20T04:52:41Z

.../src/test/java/com/apple/foundationdb/record/provider/foundationdb/FDBMetaDataStoreTest.java

@@ -26,6 +26,7 @@
 import com.apple.foundationdb.record.RecordMetaData;
 import com.apple.foundationdb.record.RecordMetaDataBuilder;
 import com.apple.foundationdb.record.RecordMetaDataOptionsProto;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;


just changing package from here on.

normen662

Hi, I gave a bunch of comments. I am unable to find logic that calls MacroFunctionValue.call() meaning that code is untested on this level. Please write test cases to test the expansion.

normen662 · 2025-01-09T08:04:05Z

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaDataBuilder.java

@@ -222,6 +229,11 @@ private void loadProtoExceptRecords(@Nonnull RecordMetaDataProto.MetaData metaDa
                typeBuilder.setRecordTypeKey(LiteralKeyExpression.fromProtoValue(typeProto.getExplicitKey()));
            }
        }
+        PlanSerializationContext serializationContext = new PlanSerializationContext(DefaultPlanSerializationRegistry.INSTANCE,
+                PlanHashable.CURRENT_FOR_CONTINUATION);


I think this is fine.

normen662 · 2025-01-09T08:05:14Z

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaDataBuilder.java

+        udfMap.put(udf.getFunctionName(), udf);
+    }
+
+    public void addUdfs(@Nonnull Collection<Udf> udfs) {


Make this an Iterable<? extends Udf>. That gives you the most generic type you actually need.

normen662 · 2025-01-09T08:06:26Z

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/Key.java

@@ -22,7 +22,7 @@

 import com.apple.foundationdb.annotation.API;
 import com.apple.foundationdb.annotation.SpotBugsSuppressWarnings;
-import com.apple.foundationdb.record.RecordMetaDataProto;
+import com.apple.foundationdb.record.RecordKeyExpressionProto;
 import com.apple.foundationdb.record.logging.LogMessageKeys;


Can you ask Alec to review the proto stuff as well?

normen662 · 2025-01-09T08:07:21Z

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/Udf.java

+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2015-2024 Apple Inc. and the FoundationDB project authors


Suggested change

* Copyright 2015-2024 Apple Inc. and the FoundationDB project authors

* Copyright 2015-2025 Apple Inc. and the FoundationDB project authors

normen662 · 2025-01-09T08:09:50Z

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaDataBuilder.java

+        PlanSerializationContext serializationContext = new PlanSerializationContext(DefaultPlanSerializationRegistry.INSTANCE,
+                PlanHashable.CURRENT_FOR_CONTINUATION);
+        for (RecordMetaDataProto.Udf udf: metaDataProto.getUdfsList()) {
+            udfMap.put(udf.getFunctionName(), new Udf(udf.getFunctionName(), Value.fromValueProto(serializationContext, udf.getFunctionValue())));


Your Udf class should get a fromProto(...) static method that you hand the proto to make an instance of Udf from it. That's more idiomatic.

normen662 · 2025-01-09T08:19:49Z

...c/main/java/com/apple/foundationdb/record/query/plan/cascades/values/MacroFunctionValue.java

+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2015-2024 Apple Inc. and the FoundationDB project authors


Suggested change

* Copyright 2015-2024 Apple Inc. and the FoundationDB project authors

* Copyright 2015-2025 Apple Inc. and the FoundationDB project authors

normen662 · 2025-01-09T08:21:51Z

...c/main/java/com/apple/foundationdb/record/query/plan/cascades/values/MacroFunctionValue.java

+public class MacroFunctionValue extends AbstractValue {
+
+    @Nonnull
+    private final List<Value> argList;


The style is to name a list of(unless ambiguous) just the plural of the things the list holds and to add the type of it to the name (in the planner). Also, avoid contractions and abbreviations. So this one should be something like argumentValues

this one becomes "parameterTypes" (same as BuiltInFunction), and "parameterIdentifiers" in the new MacroFunction class.

normen662 · 2025-01-09T08:22:18Z

...c/main/java/com/apple/foundationdb/record/query/plan/cascades/values/MacroFunctionValue.java

+    private final List<Value> argList;
+
+    @Nonnull
+    private final Value body;


This one should be bodyValue

normen662 · 2025-01-09T08:24:58Z

...c/main/java/com/apple/foundationdb/record/query/plan/cascades/values/MacroFunctionValue.java

+    @Nonnull
+    private final Value body;
+
+    private MacroFunctionValue(@Nonnull final List<Value> argList, @Nonnull final Value underlying) {


Please make names line up. Also lists are defensively copied into ImmutableList using copyOf() which does not make a copy if the list handed in is already an ImmutableList

normen662 · 2025-01-09T08:30:52Z

fdb-record-layer-core/src/main/proto/record_query_runtime.proto


 option java_outer_classname = "RecordQueryRuntimeProto";
 option java_multiple_files = true;

-message PEnumLightValue {


I am sure there is a reason, but why did this move back to the plan protos?

because some "PValue" depends on "PComparableObject"
if we keep "PComparableObject " in record_query_runtime.proto, then
record_query_plan depends on record_query_runtime.proto,
record_query_runtime depends on record_metadata.proto,
record_metadata.proto depends on record_query_plan (because Udf uses PValue)
so there is a circular dependency.

so I moved all stuff related to "PComparableObject" into record_query_plan.proto

normen662 · 2025-01-09T08:49:34Z

...c/main/java/com/apple/foundationdb/record/query/plan/cascades/values/MacroFunctionValue.java

+ * The function takes in a list of arguments as QuantifiedObjectValue, of which alias = uniqueId, resultType = argument type.
+ * and a body value, executing the function is by replacing the QuantifiedObjectValue in the body with the actual argument.
+ */
+public class MacroFunctionValue extends AbstractValue {


This is a comment I wrote after I actually already had submitted the review. Something did not sit right here.

I think you do need to have some sort of helper structure to hold the argument list and the body of the macro, but it does need to be and actually should not be a Value itself. This Value is never a real Value that exists at runtime that you can evaluate, etc. In concept this class is exactly like a BuiltinFunction except:

BuiltinFunctions cannot be serialized/deserialized

Macros like what you have in mind are not builtin.

What can be done with relative ease is to create a class Function that gets everything that is currently in BuiltinFunction. Then make an empty BuiltinFunction that extends from it and represents the superclass for all actually builtin functions. Then also have a class, let's call it MacroFunction also extend Function. MacroFunction adds protobuf serializability. Notice that the encapsulationFunction in Function is exactly what you named call(...).

MacroFunction is in effect a factory for Values. MacroFunction is the thing that you need to keep in the udf map you have in the metadata. Since MacroFunction still needs to know the body and that's a Value all of the refactoring you did for the protos is still very much needed.

foundationdb-ci · 2025-01-10T00:59:59Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: d058c09
Duration 0:12:33
Result: ❌ FAILED
Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-01-11T02:14:05Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: e11b315
Duration 0:13:01
Result: ❌ FAILED
Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-01-13T08:35:00Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: 29b9cb2
Duration 0:46:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

alecgrieser

I focused on the protobuf changes in this review, and they mostly look good to me, with a few suggestions. I could take a closer look at the rest of the PR if desired

alecgrieser · 2025-01-14T18:19:09Z

fdb-record-layer-core/src/main/proto/record_key_expression.proto

+ */
+syntax = "proto2";
+
+package com.apple.foundationdb.record;


Hm, there's a bit of a decision to be made here. By leaving the package as com.apple.foundationdb.record, that means that this file shares a package with record_metadata.proto (etc). That might not be a bad thing, but I could also see us wanting to update this to something unique like com.apple.foundationdb.record.expressions. I don't have too strong an opinion actually, and maybe leaving it scoped to the same level as the other files is fine. It certainly decreases the diff.

yeah I think it makes sense to use package com.apple.foundationdb.record.expressions, I updated the PR. It doesn't increase more diff luckily :)

alecgrieser · 2025-01-14T18:26:12Z

fdb-record-layer-core/src/main/proto/record_metadata.proto

@@ -97,99 +99,12 @@ message MetaData {
  optional bool uses_subspace_key_counter = 11;
  repeated JoinedRecordType joined_record_types = 12;
  repeated UnnestedRecordType unnested_record_types = 13;
+  repeated Udf udfs = 14;


I'm trying to think a little bit about how we'd evolve this forward in the future. The feature implemented here is for scalar valued functions, which works pretty well with our Value abstraction. If we wanted to implement table-valued functions I think we'd need slightly different machinery, and I'm trying to think if we'd want to have one "UDF" message that had a oneof over the different function types, or if we'd want to have different ScalarValuedFunction and TableValuedFunction messages at this level.

My inclination is to think that you'd want them to be separate, which makes me think that we'd want to call the Udf message here something like ScalarValuedFunction in anticipation of some future function types

Great point! I agree, I changed the Udf to be ScalarValuedFunction.

alecgrieser · 2025-01-14T18:30:22Z

fdb-record-layer-core/src/main/proto/record_query_plan.proto

  }
 }

+message PMacroFunctionValue {
+  optional string function_name = 1;
+  repeated PValue arguments = 2;


Should this be a repeated string instead of a repeated PValue? Does using a PValue here mean we record type signature information, and that's what we want to preserve?

arguments are preserved as QuantifiedObjectValue, which holds a Type resultType and a unique identifier (a uuid).

foundationdb-ci · 2025-01-14T23:44:30Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: 52b3ba4
Duration 0:47:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-01-14T23:46:36Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: 91e6117
Duration 0:07:51
Result: ❌ FAILED
Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-01-15T01:24:18Z

Result of fdb-record-layer-pr on Linux CentOS 7

Commit ID: 262f03b
Duration 0:47:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

pengpeng-lu added 30 commits July 28, 2023 13:19

save

3a5a44b

Revert "save"

53416bf

This reverts commit 3a5a44b.

Merge remote-tracking branch 'upstream/main'

3f88bb4

Merge remote-tracking branch 'upstream/main'

773e0dd

Merge remote-tracking branch 'upstream/main'

fa472d7

Merge remote-tracking branch 'upstream/main'

4c32432

Merge remote-tracking branch 'upstream/main'

ccd8e8d

Merge remote-tracking branch 'upstream/main'

403a4a2

Merge remote-tracking branch 'upstream/main'

cada59a

Merge remote-tracking branch 'upstream/main'

3b02019

Merge remote-tracking branch 'upstream/main'

138a7fc

Merge branch 'main' of github.com:pengpeng-lu/fdb-record-layer

f8b3658

Merge remote-tracking branch 'upstream/main'

b4edc0a

Merge remote-tracking branch 'upstream/main'

c42c0ce

Merge remote-tracking branch 'upstream/main'

a570479

Merge remote-tracking branch 'upstream/main'

f4e5cac

Merge remote-tracking branch 'upstream/main'

c10ec5b

Merge remote-tracking branch 'upstream/main'

0d9cc50

Merge remote-tracking branch 'upstream/main'

057fee0

Merge remote-tracking branch 'upstream/main'

37c8641

Merge remote-tracking branch 'upstream/main'

a52d15a

Merge remote-tracking branch 'upstream/main'

74c24fb

Merge remote-tracking branch 'upstream/main'

2f33668

Merge remote-tracking branch 'upstream/main'

01c4d01

Merge remote-tracking branch 'upstream/main'

8c70b2a

Merge remote-tracking branch 'upstream/main'

5d59fa2

Merge remote-tracking branch 'upstream/main'

7e84176

Merge remote-tracking branch 'upstream/main'

0d54f1e

Merge remote-tracking branch 'upstream/main'

543a79c

Merge remote-tracking branch 'upstream/main'

8fd8208

pengpeng-lu commented Dec 20, 2024

View reviewed changes

pengpeng-lu changed the title ~~Support UDF~~ Resolves #3013: Support UDF Jan 3, 2025

normen662 requested changes Jan 9, 2025

View reviewed changes

normen662 reviewed Jan 9, 2025

View reviewed changes

pengpeng-lu requested a review from alecgrieser January 9, 2025 19:29

pengpeng-lu added 4 commits January 9, 2025 12:14

save

523859e

Merge remote-tracking branch 'upstream/main'

18312ed

merge main

0a70d6e

MacroFunction

d058c09

pengpeng-lu added 2 commits January 10, 2025 10:20

save

67fad06

add test

e11b315

pmd

29b9cb2

pengpeng-lu requested a review from normen662 January 13, 2025 07:48

alecgrieser reviewed Jan 14, 2025

View reviewed changes

pengpeng-lu added 2 commits January 14, 2025 14:56

comments

52b3ba4

change package

91e6117

fix test compile error

262f03b

	* Copyright 2015-2024 Apple Inc. and the FoundationDB project authors
	* Copyright 2015-2025 Apple Inc. and the FoundationDB project authors

Resolves #3013: Support UDF #2995

Are you sure you want to change the base?

Resolves #3013: Support UDF #2995

Conversation

pengpeng-lu commented Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normen662 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normen662 Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

foundationdb-ci commented Jan 10, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

foundationdb-ci commented Jan 11, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

foundationdb-ci commented Jan 13, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

alecgrieser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pengpeng-lu Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foundationdb-ci commented Jan 14, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

foundationdb-ci commented Jan 14, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

foundationdb-ci commented Jan 15, 2025

Result of fdb-record-layer-pr on Linux CentOS 7

pengpeng-lu commented Dec 5, 2024 •

edited

Loading

normen662 Jan 9, 2025 •

edited

Loading

pengpeng-lu Jan 14, 2025 •

edited

Loading