Add new impl

Matts966 · Apr 16, 2022 · 7b24fc8 · 7b24fc8
1 parent 74ce04b
commit 7b24fc8
Show file tree

Hide file tree

Showing 11 changed files with 419 additions and 208 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -0,0 +1,66 @@
+name: Build
+on: push
+env:
+  cache-version: v1.0.2
+jobs:
+  linux:
+    name: Test the repository on Linux
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout the repository
+        uses: actions/checkout@v2
+        with:
+          ref: ${{ github.ref }}
+      - name: Cache
+        uses: pat-s/[email protected]
+        with:
+          path: ~/.cache/bazel
+          key: ${{ env.cache-version }}-${{ runner.os }}-bazelisk-build-${{ hashFiles('./**') }}
+          restore-keys: ${{ env.cache-version }}-${{ runner.os }}-bazelisk-build-
+      - name: Setup
+        run: |
+          sudo apt-get update
+          sudo apt-get install --no-install-recommends -y gcc-9 g++-9
+          sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 900 \
+            --slave /usr/bin/g++ g++ /usr/bin/g++-9
+          bazelisk test --test_output=errors //zetasql/public:sql_formatter_test
+          bazelisk build //zetasql/tools/zetasql-formatter:format
+          sudo cp bazel-bin/zetasql/tools/zetasql-formatter/format zetasql-formatter
+          zip zetasql-formatter_linux_x86_64.zip zetasql-formatter
+          ./zetasql-formatter .
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          name: Debug
+          tag_name: debug
+          files: zetasql-formatter_linux_x86_64.zip
+          prerelease: true
+  macos:
+    name: Test the repository
+    runs-on: macos-10.15
+    steps:
+      - name: Checkout the repository
+        uses: actions/checkout@v2
+        with:
+          ref: ${{ github.ref }}
+      - name: Cache
+        uses: pat-s/[email protected]
+        with:
+          path: ~/.cache/bazel
+          key: ${{ env.cache-version }}-${{ runner.os }}-bazelisk-build-${{ hashFiles('./**') }}
+          restore-keys: ${{ env.cache-version }}-${{ runner.os }}-bazelisk-build-
+      - name: Test the repository
+        run: |
+          export TEST_TMPDIR=~/.cache/bazel
+          CC=g++ bazelisk test --test_output=errors //zetasql/public:sql_formatter_test
+          CC=g++ bazelisk build //zetasql/tools/zetasql-formatter:format
+          sudo cp bazel-bin/zetasql/tools/zetasql-formatter/format zetasql-formatter
+          zip zetasql-formatter_darwin_amd64.zip zetasql-formatter
+          ./zetasql-formatter .
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          name: Debug
+          tag_name: debug
+          files: zetasql-formatter_darwin_amd64.zip
+          prerelease: true
diff --git a/README.md b/README.md
@@ -1,89 +1,37 @@
-## ZetaSQL - Analyzer Framework for SQL
+## ZetaSQL Formatter
 
-ZetaSQL defines a language (grammar, types, data model, and semantics) as well
-as a parser and analyzer.  It is not itself a database or query engine. Instead
-it is intended to be used by multiple engines wanting to provide consistent
-behavior for all semantic analysis, name resolution, type checking, implicit
-casting, etc. Specific query engines may not implement all features in the
-ZetaSQL language and may give errors if specific features are not supported. For
-example, engine A may not support any updates and engine B may not support
-analytic functions.
+[![release](https://github.com/Matts966/zetasql-formatter/workflows/release/badge.svg?event=create)](https://github.com/Matts966/zetasql-formatter/actions?query=event%3Acreate+workflow%3Arelease+)
+[![test](https://github.com/Matts966/zetasql-formatter/workflows/test/badge.svg?branch=formatter)](https://github.com/Matts966/zetasql-formatter/actions?query=branch%main+workflow%3Atest+)
 
-[ZetaSQL Language Guide](docs/README.md)
+<p align="center">
+  <img src="./docs/changes.png">
+</p>
 
-[ZetaSQL ResolvedAST API](docs/resolved_ast.md)
+This repository is forked from [google/zetasql](https://github.com/google/zetasql) and provides SQL formatter with preserved comments. This formatter can be applied to mainly BigQuery and SpanSQL.
 
-## Status of Project and Roadmap
+## Quick Start
 
-This codebase is being open sourced in multiple phases:
+```bash
+# To install for MacOSX
+wget https://github.com/Matts966/zetasql-formatter/releases/latest/download/zetasql-formatter_darwin_amd64.zip \
+  && sudo unzip zetasql-formatter_darwin_amd64.zip -d /usr/local/bin
+```
 
-1. Parser and Analyzer **Complete**
-   - Initial release includes only a subset of tests
-2. Reference Implementation **In Progress**
-   - Base capability **Complete**
-   - Function library **In Progress**
-3. Compliance Tests **Complete**
-   - includes framework for validating compliance of arbitrary engines
-4. Misc tooling
-   - Improved Formatter **In Progress**
+```bash
+# To install for Linux
+wget https://github.com/Matts966/zetasql-formatter/releases/latest/download/zetasql-formatter_linux_x86_64.zip \
+  && sudo unzip zetasql-formatter_linux_x86_64.zip -d /usr/local/bin
+```
 
-Multiplatform support is planned for the following platforms:
-
- - Linux (Ubuntu 1804 _with gcc8_ is our reference platform, but others may work).
- - MacOS (Experimental)
- - Windows (version TDB)
-
-Until all this code is released, we cannot provide any guarantees of API
-stability and cannot accept contributions. We will also be releasing more
-documentation over time, particular related to developing engines with this
-framework. Documentation on the [language](docs/) itself is fairly
-complete.
-
-
-## Flags
-ZetaSQL uses the Abseil [Flags](https://abseil.io/blog/20190509-flags) library
-to handle commandline flags. Unless otherwise documented, all flags are for
-debugging purposes only and may change, stop working or be removed at any time.
-
-
-## How to Build
-
-ZetaSQL uses [bazel](https://bazel.build) for building and dependency
-resolution. After installing bazel (we maintain support for 1.0,
-but other versions may work), simply run:
-
-```bazel build ...```
-
-## How to add as a Dependency in bazel
-See the (WORKSPACE) file, as it is a little unusual.
-
-### With docker
- TODO: Add docker build script.
-
-## Example Usage
-A very basic command line tool is available to run simple queries with the
-reference implementation:
-```bazel run //zetasql/tools/execute_query:execute_query -- "select 1 + 1;"```
-
-The reference implementation is not yet completely released and currently
-supports only a subset of functions and types.
-
-## Differential Privacy
-For questions, documentation and examples of ZetaSQLs implementation of
-Differential Privacy, please check out
-(https://github.com/google/differential-privacy).
-
-## Versions
-
-ZetaSQL makes no guarantees regarding compatibility between releases.
-Breaking changes may be made at any time. Our releases are numbered based
-on the date of the commit the release is cut from. The number format is
-YYYY.MM.n, where YYYY is the year, MM is the two digit month, and n is a
-sequence number within the time period.
+```bash
+# To apply formatter
+zetasql-formatter [paths]
+```
 
 ## License
 
 [Apache License 2.0](LICENSE)
 
-## Support Disclaimer
-This is not an officially supported Google product.
+## Sponsors
+
+The development of this formatter is sponsored by the Japan Data Science Consortium.
diff --git a/zetasql/parser/gen_extra_files.py b/zetasql/parser/gen_extra_files.py
@@ -71,6 +71,8 @@ class ParseTreeVisitor {
        public:
         virtual ~ParseTreeVisitor() {}
         virtual void visit(const ASTNode *node, void* data) = 0;
+        virtual void visitStart(const ASTNode *node, void* data) {};
+        virtual void visitEnd(const ASTNode *node, void* data) {};
       ''')
   for cls in concrete_classes:
     yield ('  virtual void visit{0}(const {0}* node, void* data) = 0;\n\n'
@@ -154,7 +156,9 @@ def GeneerateParseTreeAcceptMethods(
   for cls in concrete_classes:
     yield textwrap.dedent('''\
         void {0}::Accept(ParseTreeVisitor* visitor, void* data) const {{
+          visitor->visitStart(this, data);
           visitor->visit{0}(this, data);
+          visitor->visitEnd(this, data);
         }}
 
         ''').format(cls)

diff --git a/zetasql/parser/parser.h b/zetasql/parser/parser.h
@@ -22,6 +22,7 @@
 #include <utility>
 #include <variant>
 #include <vector>
+#include <deque>
 
 #include "zetasql/base/arena.h"
 #include "zetasql/parser/ast_node_kind.h"
@@ -258,6 +259,8 @@ absl::Status ParseExpression(const ParseResumeLocation& resume_location,
 // Unparse a given AST back to a canonical SQL string and return it.
 // Works for any AST node.
 std::string Unparse(const ASTNode* root);
+std::string UnparseWithComments(const ASTNode* root, std::deque<std::pair<std::string,
+                    ParseLocationPoint>>& parse_tokens);
 
 // Parse the first few keywords from <input> (ignoring whitespace, comments and
 // hints) to determine what kind of statement it is (if it is valid).