Merge pull request #10 from hasura/documentation

Add project documentation and README
hasura · Jun 10, 2024 · 4c6fabe · 4c6fabe
2 parents 68f4209 + 1b99ad9
commit 4c6fabe
Show file tree

Hide file tree

Showing 14 changed files with 2,673 additions and 2,873 deletions.
diff --git a/README.md b/README.md
@@ -1,48 +1,123 @@
-# Hasura Elasticsearch Connector
-
-## Get started
-This guide will help you perform the setup of this connector in local
-
-### Clone Repository
-Clone connector from [ndc-elasticsearch](https://github.com/hasura/ndc-elasticsearch/) github repository
-```
-git clone https://github.com/hasura/ndc-elasticsearch.git
-```
-
-### Set Environment Variables
-Set the following environment variables for connector
-```
-ELASTICSEARCH_URL=${YOUR_ELASTICSEARCH_URL}
-ELASTICSEARCH_USERNAME=${YOUR_ELASTICSEARCH_USERNAME}
-ELASTICSEARCH_PASSWORD=${YOUR_ELASTICSEARCH_PASSWORD}
-ELASTICSEARCH_API_KEY=${YOUR_ELASTICSEARCH_API_KEY}
-ELASTICSEARCH_CA_CERT_PATH=${YOUR_ELASTICSEARCH_CA_CERT_PATH}
-ELASTICSEARCH_INDEX_PATTERN=${REGEX_PATTERN_OF_INDICES}
-```
-Note: One can set either username and password or api key in env variables for authentication.
-
-### Build connector executable file
-```go
-go build
-```
-
-### Update configuration
-Run the update cli command to update the `configuration.json` file
-
-```
-ndc-elasticsearch update
-```
-
-### Run the connector locally
-Run the following command to start your connector
-
-```
-ndc-elasticsearch serve
-```
-Connector server will be up and running at http://localhost:8080
-
-### Verify
-```
-curl http://localhost:8080/schema
-```
-Send request to `query` endpoint for queries
+# Elasticsearch Connector
+
+<a href="https://hasura.io/"><img src="./docs/logo.png" align="right" width="200"></a>
+
+[![Docs](https://img.shields.io/badge/docs-v3.x-brightgreen.svg?style=flat)](https://hasura.io/docs/3.0/latest/connectors/elasticsearch/)
+[![ndc-hub](https://img.shields.io/badge/ndc--hub-elasticsearch-blue.svg?style=flat)](https://hasura.io/connectors/ndc-elasticsearch)
+[![License](https://img.shields.io/badge/license-Apache--2.0-purple.svg?style=flat)](LICENSE.txt)
+[![Status](https://img.shields.io/badge/status-alpha-yellow.svg?style=flat)](./readme.md)
+
+With this connector, Hasura allows you to instantly create a real-time GraphQL API on top of your documents in Elasticsearch. This connector supports Elasticsearch functionalities listed in the table below, allowing for efficient and scalable data operations. Additionally, you will benefit from all the powerful features of Hasura’s Data Delivery Network (DDN) platform, including query pushdown capabilities that delegate all query operations to the Elasticsearch, thereby enhancing query optimization and performance.
+
+This connector is built using the [Go Data Connector SDK](https://github.com/hasura/ndc-sdk-go) and implements the [Data Connector Spec](https://github.com/hasura/ndc-spec).
+
+- [Connector information in the Hasura Hub](https://hasura.io/connectors/elasticsearch)
+- [Hasura V3 Documentation](https://hasura.io/docs/3.0)
+
+## Features
+
+Below, you'll find a matrix of all supported features for the Elasticsearch connector:
+
+<!-- DocDB matrix -->
+
+| Feature                         | Supported | Notes |
+| ------------------------------- | --------- | ----- |
+| Native Queries + Logical Models | ❌         |       |
+| Simple Object Query             | ✅         |       |
+| Filter / Search                 | ✅         |       |
+| Simple Aggregation              | ✅         |       |
+| Sort                            | ✅         |       |
+| Paginate                        | ✅         |       |
+| Nested Objects                  | ✅         |       |
+| Nested Arrays                   | ✅         |       |
+| Nested Filtering                | ❌         |       |
+| Nested Sorting                  | ❌         |       |
+| Nested Relationships            | ❌         |       |
+
+
+## Before you get Started
+
+1. Create a [Hasura Cloud account](https://console.hasura.io)
+2. Install the [CLI](https://hasura.io/docs/3.0/cli/installation/)
+3. [Create a project](https://hasura.io/docs/3.0/getting-started/create-a-project)
+
+## Using the connector
+
+To use the Elasticsearch connector, follow these steps in a Hasura project:
+
+1. Add the connector:
+
+   ```bash
+   ddn add connector-manifest es_connector --subgraph app --hub-connector hasura/elasticsearch --type cloud
+   ```
+
+   In the snippet above, we've used the subgraph `app` as it's available by default; however, you can change this
+   value to match any [subgraph](https://hasura.io/docs/3.0/project-configuration/subgraphs) which you've created in your project.
+
+2. Add your Elasticsearch credentials:
+
+   Open your project in your text editor and open the `base.env.yaml` file in the root of your project. Then, add
+   `ES_CONNECTOR_URL`, `ES_CONNECTOR_USERNAME` and `ES_CONNECTOR_PASSWORD` environment variables under the `app` subgraph:
+
+   ```yaml
+   supergraph: {}
+   subgraphs:
+     app:
+       ES_CONNECTOR_URL: "<YOUR_ELASTICSEARCH_URL>"
+       ES_CONNECTOR_USERNAME: "<YOUR_ELASTICSEARCH_USERNAME>"
+       ES_CONNECTOR_PASSWORD: "<YOUR_ELASTICSEARCH_PASSWORD>"
+   ```
+
+   Next, update your `/app/es_connector/connector/es_connector.build.hml` file to reference this new environment
+   variable:
+
+   ```yaml
+   # other configuration above
+   ELASTICSEARCH_URL:
+     valueFromEnv: ES_CONNECTOR_URL
+   ELASTICSEARCH_USERNAME:
+     valueFromEnv: ES_CONNECTOR_USERNAME
+   ELASTICSEARCH_PASSWORD:
+     valueFromEnv: ES_CONNECTOR_PASSWORD
+   ```
+
+   Notice, when we use an environment variable, we must change the key to `valueFromEnv` instead of `value`. This tells
+   Hasura DDN to look for the value in the environment variable we've defined instead of using the value directly.
+
+3. Update the connector manifest and the connector link
+
+   These two steps will (1) allow Hasura to introspect your data source and complete the configuration and (2) deploy the
+   connector to Hasura DDN:
+
+   ```bash
+   ddn update connector-manifest es_connector
+   ```
+
+   ```bash
+   ddn update connector-link es_connector
+   ```
+
+4. Add Environment Variables
+
+To configure the connector, the following environment variables need to be set:
+
+| Environment Variable          | Description                                                                                                     | Required | Example Value                                                  |
+| ----------------------------- | --------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------- |
+| `ELASTICSEARCH_URL`           | The comma-separated list of Elasticsearch host addresses for connection                                         | Yes      | `https://example.es.gcp.cloud.es.io:9200`                      |
+| `ELASTICSEARCH_USERNAME`      | The username for authenticating to the Elasticsearch cluster                                                    | Yes      | `admin`                                                        |
+| `ELASTICSEARCH_PASSWORD`      | The password for the Elasticsearch user account                                                                 | Yes      | `default`                                                      |
+| `ELASTICSEARCH_API_KEY`       | The Elasticsearch API key for authenticating to the Elasticsearch cluster                                       | No       | `ABCzYWk0NEI0aDRxxxxxxxxxx1k6LWVQa2gxMUpRTUstbjNwTFIzbGoyUQ==` |
+| `ELASTICSEARCH_CA_CERT_PATH`  | The path to the Certificate Authority (CA) certificate for verifying the Elasticsearch server's SSL certificate | No       | `/etc/connector/cacert.pem`                                    |
+| `ELASTICSEARCH_INDEX_PATTERN` | The pattern for matching Elasticsearch indices, potentially including wildcards, used by the connector          | No       | `hasura*`                                                      |
+
+## Documentation
+
+View the full documentation for the Elasticsearch connector [here](./docs/index.md).
+
+## Contributing
+
+Check out our [contributing guide](./docs/contributing.md) for more details.
+
+## License
+
+The Elasticsearch connector is available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -3,7 +3,7 @@ services:
   ndc-elasticsearch:
     build: .
     volumes:
-      - ./configuration.json:/etc/connector/configuration.json
+      - ./resources/configuration.json:/etc/connector/configuration.json
       - certs:/usr/share/elasticsearch/config/certs
     ports:
       - 8080:8080

diff --git a/docs/architecture.md b/docs/architecture.md
@@ -0,0 +1,81 @@
+# General Architecture of the Elasticsearch Connector
+
+### Overview
+
+The Elasticsearch Connector takes a `QueryRequest`, which contains information about the query a user would like to run, translates it to an Elasticsearch DSL (Domain Specific Language) query, executes it against Elasticsearch, and returns the results as a `QueryResponse`.
+
+### Components
+
+The connector has three main components:
+
+1. **prepareElasticsearchQuery**
+2. **Search**
+3. **prepareResponse**
+
+### Details
+
+#### 1. prepareElasticsearchQuery
+
+Transforms `QueryRequest` into Elasticsearch DSL (Domain Specific Language) queries.
+
+**Functionality:**
+- Parse the incoming `QueryRequest`.
+- Construct the DSL query with parameters like source, size, from, sort, and query.
+
+**API:**
+
+```go
+func prepareElasticsearchQuery(
+  ctx context.Context, 
+  request *schema.QueryRequest, 
+  state *types.State
+) (map[string]interface{}, error)
+```
+
+#### 2. Search
+
+Executes the DSL query against Elasticsearch and returns results.
+
+**Functionality:**
+- Execute queries on the specified Elasticsearch index.
+- Fetch search results.
+
+**API:**
+
+```go
+func (e *Client) Search(
+  ctx context.Context,
+  index string, 
+  body map[string]interface{}
+) (map[string]interface{}, error)
+```
+
+#### 3. prepareResponse
+
+Converts Elasticsearch search results into `QueryResponse`.
+
+**Functionality:**
+- Traverse each field in the Elasticsearch response.
+- Construct and return the QueryResponse, ensuring it conforms to the expected fields.
+
+**API:**
+
+```go
+func prepareResponse(
+  ctx context.Context, 
+  response map[string]interface{}
+) *schema.RowSet
+```
+## Workflow
+
+1. **Receive `QueryRequest`:**
+   - Incoming `QueryRequest` are intercepted by the connector.
+
+2. **Prepare Query:**
+   - The `prepareElasticsearchQuery` component processes the incoming query, translating it into an Elasticsearch DSL query.
+
+3. **Execute Query:**
+   - The `Search` component executes the DSL query against Elasticsearch and retrieves the raw response.
+
+4. **Prepare Response:**
+   - The `prepareResponse` component transforms the raw Elasticsearch response into a `QueryResponse`.
diff --git a/docs/code-of-conduct.md b/docs/code-of-conduct.md
@@ -0,0 +1,60 @@
+# Hasura GraphQL Engine Community Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make
+participation in our project and our community a harassment-free experience for everyone, regardless of age, body size,
+disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education,
+socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming, inclusive and gender-neutral language (example: instead of "Hey guys", you could use "Hey folks" or
+  "Hey all")
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances 
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take
+appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits,
+issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any
+contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the
+project or its community. Examples of representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed representative at an online or offline
+event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at
+[email protected]. All complaints will be reviewed and investigated and will result in a response that is deemed
+necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to
+the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent
+repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at
+https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
diff --git a/docs/contributing.md b/docs/contributing.md
@@ -0,0 +1,39 @@
+# Contributing
+
+_First_: if you feel insecure about how to start contributing, feel free to ask us on our
+[Discord channel](https://discordapp.com/invite/hasura) in the #contrib channel. You can also just go ahead with your contribution and we'll give you feedback. Don't worry - the worst that can happen is that you'll be politely asked to change something. We appreciate any contributions, and we don't want a wall of rules to stand in the way of that.
+
+However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover what we're looking for. By addressing the points below, the chances that we can quickly merge or address your contributions will increase.
+
+## 1. Code of conduct
+
+Please follow our [Code of conduct](./code-of-conduct.md) in the context of any contributions made to Hasura.
+
+## 2. CLA
+
+For all contributions, a CLA (Contributor License Agreement) needs to be signed
+[here](https://cla-assistant.io/hasura/<repo>) before (or after) the pull request has been submitted. A bot will prompt contributors to sign the CLA via a pull request comment, if necessary.
+
+## 3. Ways of contributing
+
+### Reporting an Issue
+
+- Make sure you test against the latest released cloud version. It is possible that we may have already fixed the bug you're experiencing.
+- Provide steps to reproduce the issue, including Database (e.g. Postgres) version and Hasura DDN version.
+- Please include logs, if relevant.
+- Create a [issue](https://github.com/hasura/[connectorName]/issues/new/choose).
+
+### Working on an issue
+
+- We use the [fork-and-branch git workflow](https://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/).
+- Please make sure there is an issue associated with the work that you're doing.
+- If you're working on an issue, please comment that you are doing so to prevent duplicate work by others also.
+- Squash your commits and refer to the issue using `fix #<issue-no>` or `close #<issue-no>` in the commit message, at the end. For example: `resolve answers to everything (fix #42)` or `resolve answers to everything, fix #42`
+- Rebase master with your branch before submitting a pull request.
+
+## 6. Commit messages
+
+- The first line should be a summary of the changes, not exceeding 50 characters, followed by an optional body which has more details about the changes. Refer to [this link](https://github.com/erlang/otp/wiki/writing-good-commit-messages) for more information on writing good commit messages.
+- Use the imperative present tense: "add/fix/change", not "added/fixed/changed" nor "adds/fixes/changes".
+- Don't capitalize the first letter of the summary line.
+- Don't add a period/dot (.) at the end of the summary line.