Squashed commit of the following:

commit 05c1785 Author: lziq <[email protected]> Date: Wed Oct 19 10:11:34 2022 -0700 Applied comments in PR 822 (#836) commit 8255a07 Author: John Ed Quinn <[email protected]> Date: Tue Oct 18 14:15:15 2022 -0700 Set next dev version (#846) commit bc4dc47 Author: R. C. Howell <[email protected]> Date: Tue Oct 18 13:13:53 2022 -0700 Bumps partiql-isl-kotlin to 0.2.2 (#845) commit 55de098 Author: Alan Cai <[email protected]> Date: Mon Oct 17 14:01:40 2022 -0700 Update conformance test runner skip list with ported `pts` tests (#829) commit d8bcf1f Author: R. C. Howell <[email protected]> Date: Mon Oct 17 09:15:21 2022 -0700 v0.8.0 (#844) Creates 0.8.0 commit commit b4b7e1f Author: R. C. Howell <[email protected]> Date: Fri Oct 14 16:52:49 2022 -0700 Moves partiql-grammar to org.partiql.lang.syntax.antlr so we don't vend a separate jar (#843) commit 28d960f Author: yliuuuu <[email protected]> Date: Thu Oct 13 16:07:57 2022 -0700 fix subquery execution (#842) commit 0b8d14a Author: Arash Maymandi <[email protected]> Date: Thu Oct 13 16:04:55 2022 -0700 Final changes for REPLACE/UPSERT/INSERT EXCLUDED (#831) This commit includes the following: - Support for INSERT INTO tbl ... ON CONFLICT DO REPLACE EXCLUDED and REPLACE INTO tbl ... evaluation. - Support for INSERT INTO tbl ... ON CONFLICT DO UPDATE EXCLUDED logical planning. - The reason that evaluation is excluded is b/c the logic is for update evaluation requires implementing merge with existing values which makes it more completed; since it's not part of the related ask (attached issue) we defer the implementation for now - Adds parsing support for AS alias for UPSERT INTO and REPLACE INTO statements. commit 13f622f Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 15:50:06 2022 -0700 Cleans up SORT and UNPIVOT factories (#839) commit 17acf86 Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 15:47:41 2022 -0700 Updates changelog (#840) commit 376bc18 Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 14:45:45 2022 -0700 Adds support for aggregations and grouping to the planner (#821) commit 8be7a2d Author: Alan Cai <[email protected]> Date: Wed Oct 12 13:37:59 2022 -0700 Update README.md following addition of partiql-tests git submodule (#838) commit 2210658 Merge: 5715a88 9fdc92c Author: Vladimir Gapeyev <[email protected]> Date: Wed Oct 12 10:40:25 2022 -0700 Merge pull request #797 from partiql/graph-match-expr Switched from MATCH-as-FROM-source to MATCH-as-expression commit 9fdc92c Author: Vladimir Gapeyev <[email protected]> Date: Wed Oct 12 10:36:39 2022 -0700 Cleanup commit 0dfc9ec Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 11 14:09:36 2022 -0700 Put back tests from the reverted commit, adjusting the success status as appropriate. commit a7b42bf Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 11 10:58:32 2022 -0700 Revert "Allow unparethesized MATCH in more places." This reverts commit af6f6c3. It's hard or impossible to match this grammar addition in the partiql-lang-rust edition of the grammar. commit 5715a88 Author: Josh Pschorr <[email protected]> Date: Fri Oct 7 13:41:00 2022 -0700 Default sort & null spec in evaluator, not parser (#834) commit af6f6c3 Author: Vladimir Gapeyev <[email protected]> Date: Thu Oct 6 23:11:09 2022 -0700 Allow unparethesized MATCH in more places. commit 9b26e9c Author: Vladimir Gapeyev <[email protected]> Date: Thu Oct 6 16:00:58 2022 -0700 Grammar tweaks to not require (and prohibit) parentheses around a pattern with commas. Instead, require commas around the whole MATCH expression in this case. commit 1ce2669 Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 4 22:49:22 2022 -0700 A couple more parsing tests, involving a bare edge pattern -[]->. commit e552b92 Merge: 02b07cb a9e5a4e Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 4 17:36:45 2022 -0700 Merge branch 'main' into graph-match-expr. Resolved a conflict in lang/src/org/partiql/lang/eval/physical/PhysicalPlanCompilerImpl.kt commit a9e5a4e Author: lziq <[email protected]> Date: Tue Oct 4 15:32:00 2022 -0700 Removed snapshot from version (#828) commit a0209df Author: R. C. Howell <[email protected]> Date: Mon Oct 3 13:57:55 2022 -0700 Adds PIVOT to PartiQLPlanner (#817) commit 9a159a0 Author: yliuuuu <[email protected]> Date: Mon Oct 3 10:45:01 2022 -0700 Adds UNPIVOT operator to PartiQLPlanner (#815) * support `unpivot` operator in planner commit 07f4a27 Author: John Ed Quinn <[email protected]> Date: Fri Sep 30 17:02:57 2022 -0400 Adds SORT operator to planner (#793) commit f80e015 Author: Alan Cai <[email protected]> Date: Wed Sep 28 14:26:35 2022 -0700 Add Kotlin conformance test runner (#809) commit 882d7d2 Author: R. C. Howell <[email protected]> Date: Tue Sep 27 17:11:55 2022 -0700 Remove state from planner (#813) commit a871ba5 Author: R. C. Howell <[email protected]> Date: Tue Sep 27 10:42:13 2022 -0700 Moves partiql-isl-kotlin to a library of partiql-lang-kotlin (#805) commit cd16bf3 Author: lziq <[email protected]> Date: Mon Sep 26 15:38:08 2022 -0700 Removed type names from ExprValueType (#808) commit 2e19331 Author: R. C. Howell <[email protected]> Date: Mon Sep 26 08:23:20 2022 -0700 Renames PhysicalExprToThunkConverter as PhysicalPlanCompiler (#801) commit 706acd9 Author: lziq <[email protected]> Date: Fri Sep 23 10:03:51 2022 -0700 Marked TypedOpBehavior.LEGACY as deprecated (#796) commit 02b07cb Author: Vladimir Gapeyev <[email protected]> Date: Fri Sep 23 09:58:05 2022 -0700 Lint fixes. commit f29f894 Author: Vladimir Gapeyev <[email protected]> Date: Thu Sep 22 21:16:56 2022 -0700 Switched from MATCH-as-FROM-source to MATCH-as-expression. commit 4b3ec94 Author: Vladimir Gapeyev <[email protected]> Date: Sun Sep 18 15:34:08 2022 -0700 Rename AST node graph_match_expr ~~> gpml_pattern, and matchExpr ~~> gpmlPattern in the grammar. The new name is a tad more appropriate for the node's contents and will be clashing less with the upcoming new expression form graph_match. commit 58e908d Author: Arash Maymandi <[email protected]> Date: Thu Sep 22 15:32:20 2022 -0700 Add experimental REPLACE/UPSERT INTO (#788) * Add experimental REPLACE/UPSERT INTO Adding the parsing capability for `REPLACE INTO` and `REPLACE INTO`. See the PR and associated Issue for more details. commit 8bfaf20 Author: Arash Maymandi <[email protected]> Date: Thu Sep 22 14:38:16 2022 -0700 Add logical plan support for DO REPLACE EXCLUDED (#792) commit 96de8e7 Author: John Ed Quinn <[email protected]> Date: Wed Sep 21 14:45:58 2022 -0700 Adds projection alias support to ORDER BY clause (#740) * Adds projection alias support to ORDER BY clause commit 001a99b Author: Arash Maymandi <[email protected]> Date: Wed Sep 21 12:59:42 2022 -0700 Add an additional example for ExprFunction (#786) * Add an additional example for ExprFunction Adds an example for using a custom function for working with elements of lists and struct members. commit 58dd2cb Author: lziq <[email protected]> Date: Tue Sep 20 15:23:10 2022 -0700 Refactor stivt code (#791) commit 74b6cf6 Author: John Ed Quinn <[email protected]> Date: Mon Sep 19 10:49:12 2022 -0700 Splits large tests to allow parallelization and speedup in builds (#779) commit 16ccbc1 Author: lziq <[email protected]> Date: Fri Sep 16 09:52:18 2022 -0700 Refactor STIVT Code Style (#782)
partiql · Oct 21, 2022 · 56ab702 · 56ab702
1 parent 0a71ac4
commit 56ab702
Show file tree

Hide file tree

Showing 211 changed files with 9,104 additions and 1,858 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -19,6 +19,8 @@ jobs:
         java: [11, 17]
     steps:
       - uses: actions/checkout@v2
+        with:
+          submodules: recursive
       - name: Use Java ${{ matrix.java }}
         uses: actions/setup-java@v1
         with:
@@ -30,6 +32,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
+        with:
+          submodules: recursive
       - name: Use Java 17
         uses: actions/setup-java@v1
         with:
@@ -50,17 +54,17 @@ jobs:
           file: lang/build/reports/jacoco/test/jacocoTestReport.xml
           flags: LANG
 
-      - name: Upload PTS coverage
+      - name: Upload PARTIQL_PTS coverage
         uses: codecov/codecov-action@v3
         with:
-          file: pts/build/reports/jacoco/test/jacocoTestReport.xml
-          flags: PTS
+          file: partiql-pts/build/reports/jacoco/test/jacocoTestReport.xml
+          flags: PARTIQL_PTS
 
-      - name: Upload TEST_SCRIPT coverage
+      - name: Upload PARTIQL_TEST_SCRIPT coverage
         uses: codecov/codecov-action@v3
         with:
-          file: testscript/build/reports/jacoco/test/jacocoTestReport.xml
-          flags: TEST_SCRIPT
+          file: partiql-testscript/build/reports/jacoco/test/jacocoTestReport.xml
+          flags: PARTIQL_TEST_SCRIPT
 
       - name: Upload EXAMPLES coverage
         uses: codecov/codecov-action@v3

diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "test/partiql-tests"]
+	path = test/partiql-tests
+	url = https://github.com/partiql/partiql-tests.git
diff --git a/CHANGELOG.md b/CHANGELOG.md
diff --git a/README.md b/README.md
@@ -31,7 +31,7 @@ This project is published to [Maven Central](https://search.maven.org/artifact/o
 
 | Group ID | Artifact ID | Recommended Version |
 |----------|-------------|---------------------|
-| `org.partiql` | `partiql-lang-kotlin` | `0.7.0`| 
+| `org.partiql` | `partiql-lang-kotlin` | `0.8.0`| 
 
 
 For Maven builds, add the following to your `pom.xml`:
@@ -62,7 +62,21 @@ Be sure to replace `${version}` with the desired version.
 
 **Pre-requisite**: Building this project requires Java 11+.
 
-To build this project, clone this repository and from its root directory execute:
+This project uses a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to pull in 
+[partiql-tests](https://github.com/partiql/partiql-tests). The easiest way to pull everything in is to clone the 
+repository recursively:
+
+```bash
+$ git clone --recursive https://github.com/partiql/partiql-lang-kotlin.git
+```
+
+You can also initialize the submodules as follows:
+
+```bash
+$ git submodule update --init --recursive
+```
+
+To build this project, from the root directory execute:
 
 ```shell
 ./gradlew build

diff --git a/build.gradle b/build.gradle
@@ -36,7 +36,7 @@ allprojects {
 
 subprojects {
     group = 'org.partiql'
-    version = '0.7.1-SNAPSHOT'
+    version = '0.8.1-SNAPSHOT'
 }
 
 buildDir = new File(rootProject.projectDir, "gradle-build/" + project.name)
@@ -61,18 +61,6 @@ configure(subprojects.findAll { it.name != 'example' }) {
 
 subprojects { proj ->
 
-    /* Note: Module `partiql-grammar` is an exception to the configurations below since the configurations do not apply.
-     *   - Specifically:
-     *     - the module does not have a Test directory
-     *     - the module does not contain Java code
-     *     - the module follows a conventional multi-project and multi-language source structure
-     *       - AKA -- `src/main` should be the sources root, not `src`
-     *   - Without the exception, the build fails as Gradle requires Kotlin classes, and the ANTLR sources cannot be found.
-     */
-    if (proj.name == "partiql-grammar") {
-        return
-    }
-
     apply plugin: "org.jetbrains.kotlin.jvm"
     tasks.withType(Test) {
         jvmArgs += ['-Duser.language=en', '-Duser.country=US']
@@ -91,7 +79,10 @@ subprojects { proj ->
             afterEvaluate {
                 classDirectories.setFrom(files(classDirectories.files.collect {
                     // exclude PIG generated files.
-                    fileTree(dir: it, exclude: ["org/partiql/lang/domains/*.class"])
+                    fileTree(dir: it, exclude: [
+                            "org/partiql/lang/domains/*.class",
+                            "org/partiql/ionschema/model/*.class"
+                    ])
                 }))
             }
         }
@@ -101,53 +92,76 @@ subprojects { proj ->
         dependsOn.add(ktlintCheck)
     }
 
-    plugins.withId('java', { _ ->
-        sourceSets {
-            main.java.srcDirs = ["src"]
-            main.resources.srcDirs = ["resources"]
-            test.java.srcDirs = ["test"]
-            test.resources.srcDirs = ["test-resources"]
+    tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
+        kotlinOptions {
+            jvmTarget = "1.8"
+            apiVersion = "1.4"
+            languageVersion = "1.4"
+        }
+    }
+
+    ktlint {
+        filter {
+            // excluding multiple files from ktlint was somewhat of a challenge using other approaches..
+            // We use the "workaround" described here: https://github.com/JLLeitschuh/ktlint-gradle/issues/222
+            exclude({it.file.absolutePath.contains("/generated-src/")})
         }
+    }
 
+    plugins.withId('java', { _ ->
         java {
             sourceCompatibility = JavaVersion.VERSION_1_8
             targetCompatibility = JavaVersion.VERSION_1_8
         }
     })
+
+    // Include generated version-hash build properties file in the main source sets
     plugins.withId('org.jetbrains.kotlin.jvm', { _ ->
         def generatedVersionDir = "${buildDir}/generated-version"
-
         sourceSets {
-            main.kotlin.srcDirs = ["src"]
-            test.kotlin.srcDirs = ["test"]
-
             main {
                 output.dir(generatedVersionDir, builtBy: 'generateVersionAndHashProperties')
             }
         }
-
         // generates a build properties file with the current PartiQL version and most recent commit hash
         task generateVersionAndHashProperties {
             doLast {
                 def propertiesFile = file "$generatedVersionDir/partiql.properties"
                 propertiesFile.parentFile.mkdirs()
                 def properties = new Properties()
-
                 // get current PartiQL version
                 properties.setProperty("version", version.toString())
-
                 // get most recent short commit hash
                 def commitHash = new ByteArrayOutputStream()
                 exec {
                     commandLine 'git', 'rev-parse', '--short', 'HEAD'
                     standardOutput = commitHash
                 }
                 properties.setProperty("commit", commitHash.toString().trim())
-
                 propertiesFile.withWriter { properties.store(it, null) }
             }
         }
         processResources.dependsOn generateVersionAndHashProperties
     })
-}
 
+    if (project.name == "partiql-isl" || project.name == "partiql-tests-runner") {
+        // skip subprojects using the conventional Gradle layout
+        return
+    }
+
+    plugins.withId('java', { _ ->
+        sourceSets {
+            main.java.srcDirs = ["src"]
+            main.resources.srcDirs = ["resources"]
+            test.java.srcDirs = ["test"]
+            test.resources.srcDirs = ["test-resources"]
+        }
+    })
+
+    plugins.withId('org.jetbrains.kotlin.jvm', { _ ->
+        sourceSets {
+            main.kotlin.srcDirs = ["src"]
+            test.kotlin.srcDirs = ["test"]
+        }
+    })
+}
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
@@ -6,5 +6,5 @@ repositories {
 dependencies {
     // Adding this dependency here in buildSrc exposes it to all build.gradle files in this project
     // which allows PIG to be invoked by the :lang project at build time.
-    implementation 'org.partiql:partiql-ir-generator:0.5.0'
+    implementation 'org.partiql:partiql-ir-generator:0.6.0'
 }
diff --git a/examples/src/kotlin/org/partiql/examples/MergeKeyValues.kt b/examples/src/kotlin/org/partiql/examples/MergeKeyValues.kt
@@ -0,0 +1,89 @@
+package org.partiql.examples
+
+import org.partiql.lang.eval.BindingCase
+import org.partiql.lang.eval.BindingName
+import org.partiql.lang.eval.EvaluationSession
+import org.partiql.lang.eval.ExprFunction
+import org.partiql.lang.eval.ExprValue
+import org.partiql.lang.eval.ExprValueFactory
+import org.partiql.lang.eval.ExprValueType
+import org.partiql.lang.eval.StructOrdering
+import org.partiql.lang.eval.namedValue
+import org.partiql.lang.eval.stringValue
+import org.partiql.lang.types.FunctionSignature
+import org.partiql.lang.types.StaticType
+import java.lang.Exception
+import kotlin.collections.HashMap
+
+abstract class MergeKeysBaseExprFunction(
+    val valueFactory: ExprValueFactory,
+) : ExprFunction
+
+/**
+ * For the Given [ExprValue] representing collection of structs, merges key/values based on the given inputs in flatten list
+ * for values.
+ *
+ * E.g.
+ * Given:
+ *  [
+ *      {'Name':'certificate','Values':['abc', 'cde']},
+ *      {'Name':'certificate','Values':['ghj', 'klu']},
+ *      {'Name':'test','Values':['ghj', 'klu']}
+ *  ]
+ *  'Name' as mergeKey
+ *  'Values' as valueKey
+ *
+ *  Expected:
+ *  [
+ *      {'test': ['ghj', 'klu']},
+ *      {'certificate': ['abc', 'cde', 'ghj', 'klu']}
+ *  ]
+ */
+class MergeKeyValues(valueFactory: ExprValueFactory) :
+    MergeKeysBaseExprFunction(valueFactory) {
+    override val signature = FunctionSignature(
+        name = "merge_key_values",
+        requiredParameters = listOf(
+            StaticType.unionOf(StaticType.BAG, StaticType.LIST, StaticType.SEXP),
+            StaticType.STRING,
+            StaticType.STRING
+        ),
+        returnType = StaticType.LIST
+    )
+
+    override fun callWithRequired(session: EvaluationSession, required: List<ExprValue>): ExprValue {
+        val mergeKey = required[1].stringValue()
+        val valueKey = required[2].stringValue()
+        val result = HashMap<String, MutableList<ExprValue>>()
+
+        required[0].forEach {
+            if (it.type != ExprValueType.STRUCT) {
+                throw Exception("All elements on input collection must be of type struct. Erroneous value: $it")
+            }
+
+            val binding = it.bindings[BindingName(mergeKey, BindingCase.INSENSITIVE)]
+            val bindingValue = binding!!.stringValue()
+            val valueElem = it.bindings[BindingName(valueKey, BindingCase.INSENSITIVE)]
+
+            if (valueElem != null) {
+                if (result[bindingValue] == null) {
+                    result[bindingValue] = mutableListOf(valueElem)
+                } else {
+                    result[bindingValue]!!.add(valueElem)
+                }
+            }
+        }
+
+        val keys = result.keys.map { valueFactory.newString(it) }
+        val values = result.values.map { valueFactory.newList(it).flatten() }
+
+        val listOfStructs = keys.zip(values)
+            .map {
+                valueFactory.newStruct(
+                    listOf(valueFactory.newList(it.second).namedValue(it.first)).asSequence(),
+                    StructOrdering.UNORDERED
+                )
+            }
+        return valueFactory.newList(listOfStructs)
+    }
+}