Squashed commit of the following:

commit c272f1b Merge: d667635 aa199ef Author: Vladimir Gapeyev <[email protected]> Date: Mon Oct 31 16:19:19 2022 -0700 Merge pull request #861 from partiql/v0.8.1 Branching for 0.8.1 release. commit aa199ef Merge: 025347c d667635 Author: Vladimir Gapeyev <[email protected]> Date: Mon Oct 31 16:12:16 2022 -0700 Merge branch 'main' into v0.8.1 Resolved conflicts in CHANGELOG.md commit d667635 Author: John Ed Quinn <[email protected]> Date: Mon Oct 31 13:25:12 2022 -0700 Adds simple auto-completion to the CLI (#855) commit 025347c Author: Vladimir Gapeyev <[email protected]> Date: Fri Oct 28 22:02:53 2022 +0000 We do have to release partiql-isl-kotlin alongside partiql-lang-kotlin, after all. Otherwise, release to Maven does not work out with the current nested-project setup. commit 7cc25a4 Author: Vladimir Gapeyev <[email protected]> Date: Fri Oct 28 11:12:29 2022 -0700 Revert ISL release. commit d003273 Author: Vladimir Gapeyev <[email protected]> Date: Fri Oct 28 08:47:49 2022 -0700 Branching for 0.8.1 release. commit a4c20f2 Author: John Ed Quinn <[email protected]> Date: Thu Oct 27 15:41:32 2022 -0700 Updates the CLI Shell Highlighter to use ANTLR-generated lexer/parser (#854) commit ab88148 Author: Alan Cai <[email protected]> Date: Thu Oct 27 12:55:10 2022 -0700 Add `partiql-lang-kotlin` version migration modules (#852) commit cce34d2 Author: lziq <[email protected]> Date: Mon Oct 24 15:51:39 2022 -0700 Set `TypedOpParameter` to be `HONOR_PARAMETER` by default (#783) commit ebbe6dc Author: John Ed Quinn <[email protected]> Date: Thu Oct 20 14:01:00 2022 -0700 Update ISL to SNAPSHOT (#851) commit a298d93 Author: Arash Maymandi <[email protected]> Date: Thu Oct 20 12:13:52 2022 -0700 Add redaction for INSERT/UPSERT/REPLACE INTO << >> (#850) This commit enables statement redaction for INSERT, REPLACE, and UPSERT. See the following for more details: #849 commit fb4b29f Merge: 05c1785 8f8d6d9 Author: Vladimir Gapeyev <[email protected]> Date: Thu Oct 20 10:31:39 2022 -0700 Merge pull request #847 from partiql/issue-616 Makes SqlException.errorContext non-nullable, resolving Issue #616. commit 8f8d6d9 Author: Vladimir Gapeyev <[email protected]> Date: Wed Oct 19 11:32:54 2022 -0700 Attempting to fix divergence in the git submodule test/partiql-tests. commit 05c1785 Author: lziq <[email protected]> Date: Wed Oct 19 10:11:34 2022 -0700 Applied comments in PR 822 (#836) commit 89dacc2 Merge: 7ab757c 8255a07 Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 18 17:17:22 2022 -0700 Merge branch 'main' into issue-616 commit 8255a07 Author: John Ed Quinn <[email protected]> Date: Tue Oct 18 14:15:15 2022 -0700 Set next dev version (#846) commit 7ab757c Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 18 13:14:43 2022 -0700 Makes SqlException.errorContext non-nullable, resolving Issue #616. commit bc4dc47 Author: R. C. Howell <[email protected]> Date: Tue Oct 18 13:13:53 2022 -0700 Bumps partiql-isl-kotlin to 0.2.2 (#845) commit 55de098 Author: Alan Cai <[email protected]> Date: Mon Oct 17 14:01:40 2022 -0700 Update conformance test runner skip list with ported `pts` tests (#829) commit d8bcf1f Author: R. C. Howell <[email protected]> Date: Mon Oct 17 09:15:21 2022 -0700 v0.8.0 (#844) Creates 0.8.0 commit commit b4b7e1f Author: R. C. Howell <[email protected]> Date: Fri Oct 14 16:52:49 2022 -0700 Moves partiql-grammar to org.partiql.lang.syntax.antlr so we don't vend a separate jar (#843) commit 28d960f Author: yliuuuu <[email protected]> Date: Thu Oct 13 16:07:57 2022 -0700 fix subquery execution (#842) commit 0b8d14a Author: Arash Maymandi <[email protected]> Date: Thu Oct 13 16:04:55 2022 -0700 Final changes for REPLACE/UPSERT/INSERT EXCLUDED (#831) This commit includes the following: - Support for INSERT INTO tbl ... ON CONFLICT DO REPLACE EXCLUDED and REPLACE INTO tbl ... evaluation. - Support for INSERT INTO tbl ... ON CONFLICT DO UPDATE EXCLUDED logical planning. - The reason that evaluation is excluded is b/c the logic is for update evaluation requires implementing merge with existing values which makes it more completed; since it's not part of the related ask (attached issue) we defer the implementation for now - Adds parsing support for AS alias for UPSERT INTO and REPLACE INTO statements. commit 13f622f Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 15:50:06 2022 -0700 Cleans up SORT and UNPIVOT factories (#839) commit 17acf86 Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 15:47:41 2022 -0700 Updates changelog (#840) commit 376bc18 Author: John Ed Quinn <[email protected]> Date: Thu Oct 13 14:45:45 2022 -0700 Adds support for aggregations and grouping to the planner (#821) commit 8be7a2d Author: Alan Cai <[email protected]> Date: Wed Oct 12 13:37:59 2022 -0700 Update README.md following addition of partiql-tests git submodule (#838) commit 2210658 Merge: 5715a88 9fdc92c Author: Vladimir Gapeyev <[email protected]> Date: Wed Oct 12 10:40:25 2022 -0700 Merge pull request #797 from partiql/graph-match-expr Switched from MATCH-as-FROM-source to MATCH-as-expression commit 9fdc92c Author: Vladimir Gapeyev <[email protected]> Date: Wed Oct 12 10:36:39 2022 -0700 Cleanup commit 0dfc9ec Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 11 14:09:36 2022 -0700 Put back tests from the reverted commit, adjusting the success status as appropriate. commit a7b42bf Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 11 10:58:32 2022 -0700 Revert "Allow unparethesized MATCH in more places." This reverts commit af6f6c3. It's hard or impossible to match this grammar addition in the partiql-lang-rust edition of the grammar. commit 5715a88 Author: Josh Pschorr <[email protected]> Date: Fri Oct 7 13:41:00 2022 -0700 Default sort & null spec in evaluator, not parser (#834) commit af6f6c3 Author: Vladimir Gapeyev <[email protected]> Date: Thu Oct 6 23:11:09 2022 -0700 Allow unparethesized MATCH in more places. commit 9b26e9c Author: Vladimir Gapeyev <[email protected]> Date: Thu Oct 6 16:00:58 2022 -0700 Grammar tweaks to not require (and prohibit) parentheses around a pattern with commas. Instead, require commas around the whole MATCH expression in this case. commit 1ce2669 Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 4 22:49:22 2022 -0700 A couple more parsing tests, involving a bare edge pattern -[]->. commit e552b92 Merge: 02b07cb a9e5a4e Author: Vladimir Gapeyev <[email protected]> Date: Tue Oct 4 17:36:45 2022 -0700 Merge branch 'main' into graph-match-expr. Resolved a conflict in lang/src/org/partiql/lang/eval/physical/PhysicalPlanCompilerImpl.kt commit a9e5a4e Author: lziq <[email protected]> Date: Tue Oct 4 15:32:00 2022 -0700 Removed snapshot from version (#828) commit a0209df Author: R. C. Howell <[email protected]> Date: Mon Oct 3 13:57:55 2022 -0700 Adds PIVOT to PartiQLPlanner (#817) commit 9a159a0 Author: yliuuuu <[email protected]> Date: Mon Oct 3 10:45:01 2022 -0700 Adds UNPIVOT operator to PartiQLPlanner (#815) * support `unpivot` operator in planner commit 07f4a27 Author: John Ed Quinn <[email protected]> Date: Fri Sep 30 17:02:57 2022 -0400 Adds SORT operator to planner (#793) commit 02b07cb Author: Vladimir Gapeyev <[email protected]> Date: Fri Sep 23 09:58:05 2022 -0700 Lint fixes. commit f29f894 Author: Vladimir Gapeyev <[email protected]> Date: Thu Sep 22 21:16:56 2022 -0700 Switched from MATCH-as-FROM-source to MATCH-as-expression. commit 4b3ec94 Author: Vladimir Gapeyev <[email protected]> Date: Sun Sep 18 15:34:08 2022 -0700 Rename AST node graph_match_expr ~~> gpml_pattern, and matchExpr ~~> gpmlPattern in the grammar. The new name is a tad more appropriate for the node's contents and will be clashing less with the upcoming new expression form graph_match.
partiql · Nov 1, 2022 · 2c08d17 · 2c08d17
1 parent dbd8c50
commit 2c08d17
Show file tree

Hide file tree

Showing 114 changed files with 5,383 additions and 1,194 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,8 +4,49 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+<!-- Template: after a release, copy and paste out below
 ## [Unreleased]
 
+### Added
+
+### Changed
+
+### Deprecated
+
+### Fixed
+
+### Removed
+
+### Security
+-->
+
+
+## [Unreleased]
+
+### Added
+- Adds simple auto-completion to the CLI.
+
+### Changed
+- Now `CompileOption` uses `TypedOpParameter.HONOR_PARAMETERS` as default.
+- Updates the CLI Shell Highlighter to use the ANTLR generated lexer/parser for highlighting user queries
+
+### Deprecated
+
+### Fixed
+
+### Removed
+
+### Security
+
+
+## [0.8.1] - 2022-10-28
+
+### Added
+- Extends statement redaction to support `INSERT/REPLACE/UPSERT INTO`.
+
+
+## [0.8.0] - 2022-10-14
+
 ### Added
 - `CHANGELOG.md` with back-filling of the previous releases to the change log to provide more visibility on unreleased
   changes and make the release process easier by using the `unreleased` section of change log. The `CONTRIBUTING.md`
@@ -14,8 +55,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   on the changes that include these and the alternatives that have been considered.
 - README.md badges for GitHub Actions status, codecov, and license
 - An experimental (pending [#15](https://github.com/partiql/partiql-docs/issues/15)) embedding of a subset of
-  the [GPML (Graph Pattern Matching Language)](https://arxiv.org/abs/2112.06217) graph query into the `FROM` clause,
-  supporting. The use within the grammar is based on the assumption of a new graph data type being added to the
+  the [GPML (Graph Pattern Matching Language)](https://arxiv.org/abs/2112.06217) graph query, as a new expression
+  form `<expr> MATCH <gpml_pattern>`, which can be used as a bag-of-structs data source in the `FROM` clause.   
+  The use within the grammar is based on the assumption of a new graph data type being added to the
   specification of data types within PartiQL, and should be considered experimental until the semantics of the graph
   data type are specified.
   - basic and abbreviated node and edge patterns (section 4.1 of the GPML paper)
@@ -41,10 +83,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
       [#707](https://github.com/partiql/partiql-lang-kotlin/issues/707), [#683](https://github.com/partiql/partiql-lang-kotlin/issues/683),
       and [#730](https://github.com/partiql/partiql-lang-kotlin/issues/730)
 - Parsing of `INSERT` DML with `ON CONFLICT DO REPLACE EXCLUDED` based on [RFC-0011](https://github.com/partiql/partiql-docs/blob/main/RFCs/0011-partiql-insert.md)
-- Adds experimental parsing of `REPLACE INTO` and `UPSERT INTO` DML commands pending approval of the following RFC for moving out of experimental:
-  - https://github.com/partiql/partiql-docs/issues/27
-- Logical plan representation of `INSERT` DML with `ON CONFLICT DO REPLACE EXCLUDED` based on [RFC-0011](https://github.com/partiql/partiql-docs/blob/main/RFCs/0011-partiql-insert.md)
+- Adds a subset of `REPLACE INTO` and `UPSERT INTO` parsing based on [RFC-0030](https://github.com/partiql/partiql-docs/blob/main/RFCs/0030-partiql-upsert-replace.md)
+  - Parsing of target attributes is not supported yet and is pending [#841](https://github.com/partiql/partiql-lang-kotlin/issues/841)
+- Logical plan representation and evaluation support for `INSERT` DML with `ON CONFLICT DO REPLACE EXCLUDED` and `REPLACE INTO` based on [RFC-0011](https://github.com/partiql/partiql-docs/blob/main/RFCs/0011-partiql-insert.md)
+- Logical plan representation of `INSERT` DML with `ON CONFLICT DO UPDATE EXCLUDED` and `UPSERT INTO` based on [RFC-0011](https://github.com/partiql/partiql-docs/blob/main/RFCs/0011-partiql-insert.md)
 - Enabled projection alias support for ORDER BY clause
+- Adds support for PIVOT in the planner consistent with `EvaluatingCompiler`
 
 #### Experimental Planner Additions
 
@@ -70,9 +114,15 @@ stage in the `PlannerPipeline` and to generate performance metrics for the indiv
   query author, e.g. `true and x.id = 42` -> `x.id = 42`), `true and true` -> `true`, etc.
   - `RemoveUselessFiltersPass`, which removes useless filters introduced by the previous pass or by the query author 
   (e.g. `(filter (lit true) <bexpr>))` -> `<bexpr>`.
+- Add support for `UNPIVOT`, the behavior is expected to be compatible with the `evaluating compiler`.
+- Adds support for GROUP BY (aggregations, group keys, etc)
+- Adds support for ORDER BY in Planner
 
 ### Changed
 - The default parser for all components of PartiQL is now the PartiQLParser -- see the deprecation of `SqlParser`
+- Parsing of `ORDER BY` clauses will no longer populate the AST with defaults for the 'sort specification'
+  (i.e., `ASC` or `DESC`) or 'nulls specification' (i.e., `NULLS FIRST` or `NULLS LAST`) when the are not provided in
+  the query text. Defaulting of sort order is moved to the evaluator.
 
 ### Deprecated
 - Deprecates `SqlLexer` and `SqlParser` to be replaced with the `PartiQLParserBuilder`.
@@ -128,13 +178,6 @@ stage in the `PlannerPipeline` and to generate performance metrics for the indiv
   A client program may be interrupted by `NoSuchFieldError` exception.
 - [breaking change] Removal of `NodeMetadata` from `org.partiql.lang.eval`:
   A client program may be interrupted by `NoClassDefFoundError` exception.
-- [breaking change] Removal of the following classes from `org.partiql.lang.eval.like`:
-  - `CodepointCheckpointIterator`
-  - `PatternPart`
-  - `PatternPart.AnyOneChar`
-  - `PatternPart.ExactChars`
-  - `PatternPartKt`
-    A client program may be interrupted by NoClassDefFoundError exception.
 
 ### Fixed
 - Fix `write_file` CLI function; the old function required the input to be a `string`, but it must be a generic type.

diff --git a/README.md b/README.md
@@ -29,9 +29,9 @@ GitHub issues list for details.)
 
 This project is published to [Maven Central](https://search.maven.org/artifact/org.partiql/partiql-lang-kotlin).
 
-| Group ID | Artifact ID | Recommended Version |
-|----------|-------------|---------------------|
-| `org.partiql` | `partiql-lang-kotlin` | `0.7.0`| 
+| Group ID      | Artifact ID           | Recommended Version |
+|---------------|-----------------------|---------------------|
+| `org.partiql` | `partiql-lang-kotlin` | `0.8.1`             | 
 
 
 For Maven builds, add the following to your `pom.xml`:
@@ -62,7 +62,21 @@ Be sure to replace `${version}` with the desired version.
 
 **Pre-requisite**: Building this project requires Java 11+.
 
-To build this project, clone this repository and from its root directory execute:
+This project uses a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to pull in 
+[partiql-tests](https://github.com/partiql/partiql-tests). The easiest way to pull everything in is to clone the 
+repository recursively:
+
+```bash
+$ git clone --recursive https://github.com/partiql/partiql-lang-kotlin.git
+```
+
+You can also initialize the submodules as follows:
+
+```bash
+$ git submodule update --init --recursive
+```
+
+To build this project, from the root directory execute:
 
 ```shell
 ./gradlew build

diff --git a/build.gradle b/build.gradle
@@ -36,7 +36,7 @@ allprojects {
 
 subprojects {
     group = 'org.partiql'
-    version = '0.7.1-SNAPSHOT'
+    version = '0.8.1'
 }
 
 buildDir = new File(rootProject.projectDir, "gradle-build/" + project.name)
@@ -61,18 +61,6 @@ configure(subprojects.findAll { it.name != 'example' }) {
 
 subprojects { proj ->
 
-    /* Note: Module `partiql-grammar` is an exception to the configurations below since the configurations do not apply.
-     *   - Specifically:
-     *     - the module does not have a Test directory
-     *     - the module does not contain Java code
-     *     - the module follows a conventional multi-project and multi-language source structure
-     *       - AKA -- `src/main` should be the sources root, not `src`
-     *   - Without the exception, the build fails as Gradle requires Kotlin classes, and the ANTLR sources cannot be found.
-     */
-    if (proj.name == "partiql-grammar") {
-        return
-    }
-
     apply plugin: "org.jetbrains.kotlin.jvm"
     tasks.withType(Test) {
         jvmArgs += ['-Duser.language=en', '-Duser.country=US']
@@ -156,8 +144,8 @@ subprojects { proj ->
         processResources.dependsOn generateVersionAndHashProperties
     })
 
-    if (project.name == "partiql-isl" || project.name == "partiql-tests-runner") {
-        // skip since partiql-isl uses the conventional Gradle layout
+    if (project.name == "partiql-isl" || project.name == "partiql-tests-runner" || project.parent.name.contains("-migration")) {
+        // skip subprojects using the conventional Gradle layout
         return
     }
 

diff --git a/cli/src/org/partiql/shell/CompleterDefault.kt b/cli/src/org/partiql/shell/CompleterDefault.kt
@@ -0,0 +1,42 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates.  All rights reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License").
+ * You may not use this file except in compliance with the License.
+ * A copy of the License is located at:
+ *
+ *      http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
+ * language governing permissions and limitations under the License.
+ */
+
+package org.partiql.shell
+
+import org.jline.reader.Completer
+import org.jline.reader.impl.completer.AggregateCompleter
+import org.jline.reader.impl.completer.ArgumentCompleter
+import org.jline.reader.impl.completer.NullCompleter
+import org.jline.reader.impl.completer.StringsCompleter
+
+internal class CompleterDefault : AggregateCompleter(getAllCommands()) {
+    companion object {
+        private fun getAllCommands(): List<Completer> = listOf(
+            buildCompoundCommand("SELECT"),
+            buildCompoundCommand("REMOVE"),
+            buildCompoundCommand("UPSERT"),
+            buildCompoundCommand("REPLACE"),
+            buildCompoundCommand("EXEC"),
+            buildCompoundCommand("CREATE", listOf("INDEX", "TABLE")),
+            buildCompoundCommand("DROP", listOf("INDEX", "TABLE")),
+            buildCompoundCommand("INSERT", listOf("INTO")),
+        )
+
+        private fun buildCompoundCommand(primary: String, targets: List<String> = emptyList()) = ArgumentCompleter(
+            StringsCompleter(primary),
+            StringsCompleter(targets),
+            NullCompleter.INSTANCE
+        )
+    }
+}
diff --git a/cli/src/org/partiql/shell/Shell.kt b/cli/src/org/partiql/shell/Shell.kt
@@ -24,8 +24,9 @@ import org.jline.reader.History
 import org.jline.reader.LineReader
 import org.jline.reader.LineReaderBuilder
 import org.jline.reader.UserInterruptException
-import org.jline.reader.impl.completer.NullCompleter
+import org.jline.reader.impl.completer.AggregateCompleter
 import org.jline.terminal.TerminalBuilder
+import org.jline.utils.AttributedString
 import org.jline.utils.AttributedStringBuilder
 import org.jline.utils.AttributedStyle
 import org.jline.utils.InfoCmp
@@ -67,6 +68,11 @@ private const val HELP = """
 !clear              Clears the screen
 """
 
+private val SUCCESS: AttributedStyle = AttributedStyle.DEFAULT.foreground(AttributedStyle.GREEN)
+private val ERROR: AttributedStyle = AttributedStyle.DEFAULT.foreground(AttributedStyle.RED)
+private val INFO: AttributedStyle = AttributedStyle.DEFAULT
+private val WARN: AttributedStyle = AttributedStyle.DEFAULT.foreground(AttributedStyle.YELLOW)
+
 private val EXIT_DELAY: Duration = Duration(3000)
 
 /**
@@ -127,10 +133,15 @@ class Shell(
             this.config.isMonochrome -> null
             else -> ShellHighlighter()
         }
+        val completer = AggregateCompleter(CompleterDefault())
         val reader = LineReaderBuilder.builder()
             .terminal(terminal)
             .parser(ShellParser)
-            .completer(NullCompleter())
+            .completer(completer)
+            .option(LineReader.Option.GROUP_PERSIST, true)
+            .option(LineReader.Option.AUTO_LIST, true)
+            .option(LineReader.Option.CASE_INSENSITIVE, true)
+            .variable(LineReader.LIST_MAX, 10)
             .highlighter(highlighter)
             .expander(ShellExpander)
             .variable(LineReader.HISTORY_FILE, homeDir.resolve(".partiql/.history"))
@@ -290,6 +301,16 @@ private fun History.Entry.pretty(): String {
         .toAnsi()
 }
 
+private fun ansi(string: String, style: AttributedStyle) = AttributedString(string, style).toAnsi()
+
+private fun PrintStream.success(string: String) = this.println(ansi(string, SUCCESS))
+
+private fun PrintStream.error(string: String) = this.println(ansi(string, ERROR))
+
+private fun PrintStream.info(string: String) = this.println(ansi(string, INFO))
+
+private fun PrintStream.warn(string: String) = this.println(ansi(string, WARN))
+
 private class ThreadInterrupter : Closeable {
     private val thread = Thread.currentThread()