Merge branch 'feast-dev:master' into feast-ibis

tokoko · Feb 22, 2024 · 91e16f0 · 91e16f0
2 parents f1781b4 + 1cc94f2
commit 91e16f0
Show file tree

Hide file tree

Showing 105 changed files with 2,170 additions and 1,012 deletions.
diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml
@@ -86,7 +86,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.8" ]
+        python-version: [ "3.8", "3.10" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/pr_local_integration_tests.yml b/.github/workflows/pr_local_integration_tests.yml
@@ -19,7 +19,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.8" ]
+        python-version: [ "3.8", "3.10" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml
@@ -12,8 +12,6 @@ jobs:
         exclude:
           - os: macOS-latest
             python-version: "3.9"
-          - os: macOS-latest
-            python-version: "3.10"
     env:
       OS: ${{ matrix.os }}
       PYTHON: ${{ matrix.python-version }}

diff --git a/Makefile b/Makefile
@@ -332,7 +332,7 @@ format-python:
 	cd ${ROOT_DIR}/sdk/python; python -m black --target-version py38 feast tests
 
 lint-python:
-	cd ${ROOT_DIR}/sdk/python; python -m mypy
+	cd ${ROOT_DIR}/sdk/python; python -m mypy feast
 	cd ${ROOT_DIR}/sdk/python; python -m isort feast/ tests/ --check-only
 	cd ${ROOT_DIR}/sdk/python; python -m flake8 feast/ tests/
 	cd ${ROOT_DIR}/sdk/python; python -m black --check feast tests

diff --git a/community/maintainers.md b/community/maintainers.md
@@ -9,7 +9,12 @@ In alphabetical order
 | Name           | GitHub Username  | Email                       | Organization       |
 | -------------- | ---------------- |-----------------------------| ------------------ |
 | Achal Shah     | `achals`         | [email protected]            | Tecton             |
-| Willem Pienaar | `woop`           | [email protected]      | Tecton             |
+| Edson Tirelli   | `etirelli`       | [email protected]          | Red Hat          |
+| Francisco Javier Arceo   | `franciscojavierarceo`       | [email protected]          | Affirm          |
+| Hao Xu   | `HaoXuAI`       | [email protected]          | JPMorgan          |
+| Jeremy Ary   | `jeremyary`       | [email protected]          | Red Hat          |
+| Shuchu Han  | `shuchu`       | [email protected]          | Independent          |
+| Willem Pienaar | `woop`           | [email protected]      | Cleric             |
 | Zhiling Chen   | `zhilingc`       | [email protected]          | GetGround          |
 
 ## Emeritus Maintainers

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -99,6 +99,7 @@
   * [MySQL (contrib)](reference/online-stores/mysql.md)
   * [Rockset (contrib)](reference/online-stores/rockset.md)
   * [Hazelcast (contrib)](reference/online-stores/hazelcast.md)
+  * [ScyllaDB (contrib)](reference/online-stores/scylladb.md)
 * [Providers](reference/providers/README.md)
   * [Local](reference/providers/local.md)
   * [Google Cloud Platform](reference/providers/google-cloud-platform.md)

diff --git a/docs/project/development-guide.md b/docs/project/development-guide.md
@@ -154,6 +154,16 @@ pip install -e ".[dev]"
 
 This will allow the installed feast version to automatically reflect changes to your local development version of Feast without needing to reinstall everytime you make code changes.
 
+10. Compile the protubufs
+```sh
+make compile-protos-python
+```
+
+11. Spin up Docker Image
+```sh
+docker build -t docker-whale -f ./sdk/python/feast/infra/feature_servers/multicloud/Dockerfile .
+```
+
 ### Code Style & Linting
 Feast Python SDK / CLI codebase:
 - Conforms to [Black code style](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html)

diff --git a/docs/reference/data-sources/overview.md b/docs/reference/data-sources/overview.md
@@ -19,13 +19,13 @@ Details for each specific data source can be found [here](README.md).
 Below is a matrix indicating which data sources support which types.
 
 | | File | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino |
-| :-------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
-| `bytes`     | yes | yes | yes | yes | yes | yes | yes |
-| `string`    | yes | yes | yes | yes | yes | yes | yes |
-| `int32`     | yes | yes | yes | yes | yes | yes | yes |
-| `int64`     | yes | yes | yes | yes | yes | yes | yes |
-| `float32`   | yes | yes | yes | yes | yes | yes | yes |
-| `float64`   | yes | yes | yes | yes | yes | yes | yes |
-| `bool`      | yes | yes | yes | yes | yes | yes | yes |
-| `timestamp` | yes | yes | yes | yes | yes | yes | yes |
-| array types | yes | yes | no  | no  | yes | yes | no  |
+| :-------------------------------- | :-- | :-- |:----------| :-- | :-- | :-- | :-- |
+| `bytes`     | yes | yes | yes       | yes | yes | yes | yes |
+| `string`    | yes | yes | yes       | yes | yes | yes | yes |
+| `int32`     | yes | yes | yes       | yes | yes | yes | yes |
+| `int64`     | yes | yes | yes       | yes | yes | yes | yes |
+| `float32`   | yes | yes | yes       | yes | yes | yes | yes |
+| `float64`   | yes | yes | yes       | yes | yes | yes | yes |
+| `bool`      | yes | yes | yes       | yes | yes | yes | yes |
+| `timestamp` | yes | yes | yes       | yes | yes | yes | yes |
+| array types | yes | yes | yes       | no  | yes | yes | no  |
diff --git a/docs/reference/data-sources/snowflake.md b/docs/reference/data-sources/snowflake.md
@@ -46,5 +46,5 @@ The full set of configuration options is available [here](https://rtd.feast.dev/
 
 ## Supported Types
 
-Snowflake data sources support all eight primitive types, but currently do not support array types.
+Snowflake data sources support all eight primitive types. Array types are also supported but not with type inference.
 For a comparison against other batch data sources, please see [here](overview.md#functionality-matrix).
diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md
@@ -54,4 +54,6 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
 [hazelcast.md](hazelcast.md)
 {% endcontent-ref %}
 
-
+{% content-ref url="scylladb.md" %}
+[scylladb.md](scylladb.md)
+{% endcontent-ref %}
diff --git a/docs/reference/online-stores/scylladb.md b/docs/reference/online-stores/scylladb.md
@@ -0,0 +1,94 @@
+# ScyllaDB Cloud online store
+
+## Description
+
+ScyllaDB is a low-latency and high-performance Cassandra-compatible (uses CQL) database. You can use the existing Cassandra connector to use ScyllaDB as an online store in Feast.
+
+The [ScyllaDB](https://www.scylladb.com/) online store provides support for materializing feature values into a ScyllaDB or [ScyllaDB Cloud](https://www.scylladb.com/product/scylla-cloud/) cluster for serving online features real-time.
+
+## Getting started
+
+Install Feast with Cassandra support:
+```bash
+pip install "feast[cassandra]"
+```
+
+Create a new Feast project:
+```bash
+feast init REPO_NAME -t cassandra
+```
+
+### Example (ScyllaDB)
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: scylla_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: cassandra
+    hosts:
+        - 172.17.0.2
+    keyspace: feast
+    username: scylla
+    password: password
+```
+{% endcode %}
+
+### Example (ScyllaDB Cloud)
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: scylla_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: cassandra
+    hosts:
+        - node-0.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
+        - node-1.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
+        - node-2.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
+    keyspace: feast
+    username: scylla
+    password: password
+```
+{% endcode %}
+
+
+The full set of configuration options is available in [CassandraOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store.CassandraOnlineStoreConfig).
+For a full explanation of configuration options please look at file
+`sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md`.
+
+Storage specifications can be found at `docs/specs/online_store_format.md`.
+
+## Functionality Matrix
+
+The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
+Below is a matrix indicating which functionality is supported by the Cassandra plugin.
+
+|                                                           | Cassandra |
+| :-------------------------------------------------------- | :-------- |
+| write feature values to the online store                  | yes       |
+| read feature values from the online store                 | yes       |
+| update infrastructure (e.g. tables) in the online store   | yes       |
+| teardown infrastructure (e.g. tables) in the online store | yes       |
+| generate a plan of infrastructure changes                 | yes       |
+| support for on-demand transforms                          | yes       |
+| readable by Python SDK                                    | yes       |
+| readable by Java                                          | no        |
+| readable by Go                                            | no        |
+| support for entityless feature views                      | yes       |
+| support for concurrent writing to the same key            | no        |
+| support for ttl (time to live) at retrieval               | no        |
+| support for deleting expired data                         | no        |
+| collocated by feature view                                | yes       |
+| collocated by feature service                             | no        |
+| collocated by entity key                                  | no        |
+
+To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
+
+## Resources
+
+* [Sample application with ScyllaDB](https://feature-store.scylladb.com/stable/)
+* [ScyllaDB website](https://www.scylladb.com/)
+* [ScyllaDB Cloud documentation](https://cloud.docs.scylladb.com/stable/)