From cff63624cd7eb862435f442dd9691fd336fcbe07 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 12:20:02 +1000 Subject: [PATCH 01/16] Distribution RFC --- rfcs/0002-distribution.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 rfcs/0002-distribution.md diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md new file mode 100644 index 00000000..05808570 --- /dev/null +++ b/rfcs/0002-distribution.md @@ -0,0 +1,29 @@ +# Connector Package Distribution + +## Purpose + +Connector API, definition and packaging are specified respectively by: + +* [NDC specification](http://hasura.github.io/ndc-spec/) +* [Deployment Specification](https://github.com/hasura/ndc-hub/blob/main/rfcs/0000-deployment.md) +* [Packaging Specification (WIP)](https://github.com/hasura/ndc-hub/pull/89/files) + +This new distribution specification details how connector packages are intended to be owned, stored, indexed, searched, fetched and automatically published. + +The intuition for this system is inspired by other package management systems such as NPM, Cabal, etc. + +## Out of Scope for this RFC + +TODO + +## Proposal + +### Ownership +### Storage +### Indexing +### Discoverability +### Access +### Publication + + + From 08b2e824088c387cc7301cd7dfb7f343cad234f6 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 13:01:15 +1000 Subject: [PATCH 02/16] More details --- rfcs/0002-distribution.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 05808570..de7b94f3 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -14,16 +14,30 @@ The intuition for this system is inspired by other package management systems su ## Out of Scope for this RFC -TODO +The format of the metadata used for indexing, etc. + ## Proposal +While the precursor specifications outline the structure and mechanisms of packaging, this RFC details how the packages are owned, distributed, and indexed. A layered solution is outlined from storage up to user-applications and how they can be leveraged by CI in order to automatically publish updated versions and scrape topics for community contribution discovery. + +This solution enables the following UX scenarios: + +* Authors are granted system credentials and API tokens +* Packages are manually published via CLI by authors along with metadata +* Packages are automatically published by authors via CI and API in connector repositories +* Packages browsed and searched for by Hasura V3 users +* Packages are referenced in Hasura V3 projects +* Packages are fetched for local usage in Hasura V3 projects + ### Ownership ### Storage +### API ### Indexing ### Discoverability ### Access ### Publication +### Validation From f15953b9e5ceab5237c1cd2a40122993a3e08d58 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 14:29:04 +1000 Subject: [PATCH 03/16] WIP --- rfcs/0002-distribution.md | 65 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index de7b94f3..50c5316e 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -12,10 +12,15 @@ This new distribution specification details how connector packages are intended The intuition for this system is inspired by other package management systems such as NPM, Cabal, etc. +There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha) + + ## Out of Scope for this RFC The format of the metadata used for indexing, etc. +Authentication mechanisms. + ## Proposal @@ -31,13 +36,69 @@ This solution enables the following UX scenarios: * Packages are fetched for local usage in Hasura V3 projects ### Ownership + +Ownership is granted on an Organisation -> Package -> Author -> Version hierarchy. + +Users will be granted roles that authorize operations withing this hierarchy. + +An initial draft of roles (from most, to least privileged) is: + +* Operations - Global Access for System Operations +* Auditor - Global Read-Only Access +* Admin - Organisation Administrator - Create Packages +* Owner - Package Administrator +* Author - Package Contributor via Releases +* Public - General Public (default) + +All roles (except auditor) can grant lesser role privileges to users in their domains. + +Only Operations can grant the Operations role. + +All changes within the system are logged and can be viewed by the auditor role. + ### Storage + +Package definitions take the form described in the packaging spec. These need to be stored. The storage mechanism can be described abstractly: + +* The storage system is philosophically idempotent and content addressable (hashes are included in assets). +* Upload is available to the system internally, and able to be delegated to authors via pre-authorized URIs. +* Stable read-only (fetch) URIs exist for the stored location of packages +* Indexes are maintained outside of storage, but minimal metadata is maintained for system-administration purposes + +While this abstract definition is useful for system-requirements, in practice our initial implementation will use Google Cloud Buckets. + +### Database + +The database backing the API provides all of the APIs state management capabilities outside of package archive storage (as described in "Storage"). + +The initial implementation of the Database will be Postgres. + ### API + +The API provides the user-interaction layer that mediates the database and storage components. No direct user interaction should occur with either the database, or storage, except for the case when the API delegates a storage interaction - such as providing an author a pre-authorized URL for publication, or providing a Hasura V3 project user a public storage URL for a package definition. + +The various functions of the API are described as follows: + +#### Administration + +* TODO + +#### Authoring + +* TODO + +#### Discoverability + +* TODO + +#### Acquisition + +* TODO + ### Indexing ### Discoverability ### Access ### Publication -### Validation - +### Validation (TODO: What did I mean by this?) From d92c44695aee65466bfcfd7ff9ce0da51c53ec44 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 14:47:44 +1000 Subject: [PATCH 04/16] WIP --- rfcs/0002-distribution.md | 83 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 80 insertions(+), 3 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 50c5316e..5f735284 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -1,5 +1,8 @@ # Connector Package Distribution +This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions. + + ## Purpose Connector API, definition and packaging are specified respectively by: @@ -17,9 +20,11 @@ There was a previous implementation of these concepts as described (TODO: Get do ## Out of Scope for this RFC -The format of the metadata used for indexing, etc. +The following are not described in this specification: -Authentication mechanisms. +* The format of the metadata used for indexing, etc +* Authentication mechanisms +* Verification policies and procedures ## Proposal @@ -77,6 +82,8 @@ The initial implementation of the Database will be Postgres. The API provides the user-interaction layer that mediates the database and storage components. No direct user interaction should occur with either the database, or storage, except for the case when the API delegates a storage interaction - such as providing an author a pre-authorized URL for publication, or providing a Hasura V3 project user a public storage URL for a package definition. +The API will be implemented via a Hasura V3 instance. (TODO: Check if V3 has the capabilities to implement this yet, or if we should start with a V2 instance for stability reasons) + The various functions of the API are described as follows: #### Administration @@ -95,10 +102,80 @@ The various functions of the API are described as follows: * TODO + +### Applications / CLI + +The Hasura V3 CLI will provide a consistent and convenient interface to interacting with the API. + + +### CI + +Contributors may leverage the API or CLI in their CI workflows in order to automate the publication of new versions of their packages. + + +### Community Topic Consumption + +Hasura may automatically crawl pre-defined locations (such as Github) in order to collect third-party community contributions without authors needing to explicitly create new pull-requests, etc. + +For example: Github Topics can be use to search for e.g. "#hasura-v3-packge" and if the repository contains valid package definitions, consume these and index them. + +Please see previous work: TODO: Previous topic collection proof-of-concept + +*Open-Question: How would a community crawled package transition to an explicitly managed package? How would ownership be established and transitioned, etc?* + + ### Indexing + +The technical considerations were described in the "API" section, however, users should be able to leverage indexes on the following properties: + +* Name +* Version +* Date of Publication +* Organisation +* Author +* Tags +* Category +* Free-text description +* Related packages +* Underlying DB (or Data solution) +* Any metadata in the Package description +* Verification status + ### Discoverability + +The discoverability component will simply hard-code various permutations of the "Indexing" criteria to provide the user browsable lists of packages. + +These could include lists such as: + +* Most popular packages +* Most recent packages +* Verified packages +* Etc. + ### Access + +Roles should be deliberately narrow in their scope, with higher level roles being able to grant lower-level roles, but not able to perform their duties implicitly. + + ### Publication -### Validation (TODO: What did I mean by this?) + +The publication of packages should be performed via the API by users who have the Author role. + +The mechanism used is a three-step process: + +* Request pre-authorized storage URI for a package version via API +* Upload the package definition archive via the storage URI +* Set the metadata for the package version + +This can be done via a single user-interaction if an application (such as Hasura V3 CLI) abstracts these three steps. + + +### Verification + +Verification status procedures and policies Organisation/Package should be defined further in subsequent specifications. + +Verification information should describe what has been checked and to what it applies. A separate table should be kept for this information, not just a bit on existing resources. + +This is for granularity and auditing purposes. From 96b6f1f6116b0936bae647d2fca9474b3588ea7f Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 14:59:43 +1000 Subject: [PATCH 05/16] WIP --- rfcs/0002-distribution.md | 56 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 4 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 5f735284..fe055b1d 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -2,6 +2,18 @@ This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions. +## Items Outstanding in this Specification (TODO) + +The following items are intended to be fleshed-out in this specification prior to approval: + +* Data-formats +* URI locations +* Identifiers +* Assignment of implementation +* Dependencies + +In addition, all "TODO" references should be replaced before finalization. + ## Purpose @@ -72,6 +84,11 @@ Package definitions take the form described in the packaging spec. These need to While this abstract definition is useful for system-requirements, in practice our initial implementation will use Google Cloud Buckets. +Storage conventions will be followed so that our system could initially predict the location of packages and we can incrementally transition to API based package access in service of rapid delivery. + +Storage location convention will initially be: `ORG/PACKAGE/VERSION/SHA/org-packge-version-sha.tar.gz` + + ### Database The database backing the API provides all of the APIs state management capabilities outside of package archive storage (as described in "Storage"). @@ -86,21 +103,37 @@ The API will be implemented via a Hasura V3 instance. (TODO: Check if V3 has the The various functions of the API are described as follows: +#### Operation + +* System health monitoring +* Restarts +* Resource allocation + #### Administration -* TODO +* Creation of organisations +* Creation of users +* Creation of packages +* Assignment of roles +* Verification +* Revocation of content +* Redirection of resources #### Authoring -* TODO +* Creation of packages +* Publication of new versions of packages +* Association of new metadata with organisation/package/versions +* Request for verification +* Revocation (requests?) of package versions #### Discoverability -* TODO +* Search for package by metadata filters #### Acquisition -* TODO +* Request for package download URI ### Applications / CLI @@ -179,3 +212,18 @@ Verification information should describe what has been checked and to what it ap This is for granularity and auditing purposes. +### System Abuse Scenarios and Mitigation + +Any publicly accessible APIs with publication capabilities have the potential to be abused and as such we should attempt to predict and mitigate the scenarios that we can anticipate: + +* Identity misappropriation +* Inappropriate content +* Leaked credentials +* Denial of service +* IP harvesting by third parties and competitors +* Incorrect application of verification +* Recycling of content +* Unintentional mistakes +* Spam / Reflection +* Etc. + From 7f49afbeae99d7f4e013212ea50e9226b3e4d446 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:05:47 +1000 Subject: [PATCH 06/16] revocation concerns --- rfcs/0002-distribution.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index fe055b1d..20b4f25a 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -11,6 +11,7 @@ The following items are intended to be fleshed-out in this specification prior t * Identifiers * Assignment of implementation * Dependencies +* Revocation concerns In addition, all "TODO" references should be replaced before finalization. From 094f085db1593c56130b32bc21c59f45e2a9c264 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:13:52 +1000 Subject: [PATCH 07/16] WIP --- rfcs/0002-distribution.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 20b4f25a..5c7f691f 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -141,6 +141,14 @@ The various functions of the API are described as follows: The Hasura V3 CLI will provide a consistent and convenient interface to interacting with the API. +The CLI commands will include the following: + +* hasura3 package create +* hasura3 package publish +* hasura3 package revoke +* hasura3 package search +* hasura3 package fetch + ### CI From c6451c62dd7504e155232e1b158c89ad31fd9106 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:22:21 +1000 Subject: [PATCH 08/16] follow-up-actions --- rfcs/0002-distribution.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 5c7f691f..41865b71 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -16,6 +16,15 @@ The following items are intended to be fleshed-out in this specification prior t In addition, all "TODO" references should be replaced before finalization. +## Follow-Up Work + +After implementation of this system, follow-up actions should be performed: + +* Migration and deprecation of previous hub API +* Publication of new compatible versions of all available connectors +* TODO + + ## Purpose Connector API, definition and packaging are specified respectively by: @@ -228,6 +237,7 @@ Any publicly accessible APIs with publication capabilities have the potential to * Identity misappropriation * Inappropriate content * Leaked credentials +* Squatting * Denial of service * IP harvesting by third parties and competitors * Incorrect application of verification From 8c31ba4ec752d34c30a06b6c82915d6cc3fc1a76 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:25:57 +1000 Subject: [PATCH 09/16] title update --- rfcs/0002-distribution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 41865b71..0ed599c2 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -1,4 +1,4 @@ -# Connector Package Distribution +# Connector Package Distribution RFC This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions. From ad433cfb0d20d1c3abaa59165b685195fd2d6864 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:33:47 +1000 Subject: [PATCH 10/16] moving purpose higher in document --- rfcs/0002-distribution.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 0ed599c2..bda8f17f 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -2,6 +2,22 @@ This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions. + +## Purpose + +Connector API, definition and packaging are specified respectively by: + +* [NDC specification](http://hasura.github.io/ndc-spec/) +* [Deployment Specification](https://github.com/hasura/ndc-hub/blob/main/rfcs/0000-deployment.md) +* [Packaging Specification (WIP)](https://github.com/hasura/ndc-hub/pull/89/files) + +This new distribution specification details how connector packages are intended to be owned, stored, indexed, searched, fetched and automatically published. + +The intuition for this system is inspired by other package management systems such as NPM, Cabal, etc. + +There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha) + + ## Items Outstanding in this Specification (TODO) The following items are intended to be fleshed-out in this specification prior to approval: @@ -25,21 +41,6 @@ After implementation of this system, follow-up actions should be performed: * TODO -## Purpose - -Connector API, definition and packaging are specified respectively by: - -* [NDC specification](http://hasura.github.io/ndc-spec/) -* [Deployment Specification](https://github.com/hasura/ndc-hub/blob/main/rfcs/0000-deployment.md) -* [Packaging Specification (WIP)](https://github.com/hasura/ndc-hub/pull/89/files) - -This new distribution specification details how connector packages are intended to be owned, stored, indexed, searched, fetched and automatically published. - -The intuition for this system is inspired by other package management systems such as NPM, Cabal, etc. - -There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha) - - ## Out of Scope for this RFC The following are not described in this specification: From 75357f3dda15483d778db8a6e8cc9f17cdcf1807 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Thu, 15 Feb 2024 15:38:08 +1000 Subject: [PATCH 11/16] heading levels --- rfcs/0002-distribution.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index bda8f17f..87134ef1 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -18,7 +18,7 @@ The intuition for this system is inspired by other package management systems su There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha) -## Items Outstanding in this Specification (TODO) +### Items Outstanding in this Specification (TODO) The following items are intended to be fleshed-out in this specification prior to approval: @@ -31,8 +31,7 @@ The following items are intended to be fleshed-out in this specification prior t In addition, all "TODO" references should be replaced before finalization. - -## Follow-Up Work +### Follow-Up Work After implementation of this system, follow-up actions should be performed: @@ -40,8 +39,7 @@ After implementation of this system, follow-up actions should be performed: * Publication of new compatible versions of all available connectors * TODO - -## Out of Scope for this RFC +### Out of Scope for this RFC The following are not described in this specification: From 07446ff7913e03f6c2a1814f4754d4fedc370af5 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Fri, 16 Feb 2024 14:52:51 +1000 Subject: [PATCH 12/16] rationale --- rfcs/0002-distribution.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 87134ef1..0ad68be3 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -48,6 +48,32 @@ The following are not described in this specification: * Verification policies and procedures +## Motivation and Why and How this Differs from Existing Solutions + +First, where are the current and proposed system different? + +| Layer | Current System | Proposed System | Difference | +| --- | --- | --- | --- | +| Package Definition Storage | Github Tag | .tar.gz | Currently the package definitions are stored in a directory hierarchy on a github branch/tag | +| Database | Postgres | Postgres | Same infrastructure with a new schema | +| API | Hasura | Hasura | Same infrastructure with different roles and actions | +| CLI | N/A | Hasura V3 CLI Plugin | No current CLI interactions available | +| Registry | ndc-hub/registry | No central component in proposal | There is no central definition source outside of the DB/Storage layer in the proposal | +| Third-party CI | No current component | Usage via CLI/API | Third-parties are able to integrate into their CI via API | +| Topic Scraping | No current component | Scheduled/Webhook trigger of topic ingestion of community connectors | Arbitrary ingestion is possible via the API/CLI | + +The current system's GraphQL: https://data.pro.arusah.com/ - Looking at all the graphql queries starting with connector_ should give all the things that we have so far. + +Google cloud function (https://github.com/hasura/connectors-cloud-integration/tree/main/sync_connector_hub_data) runs every 24 hours and scrapes registry and inserts into DB. + +Issues with the current system: + +* Pull-based ingestion of registry - Changes should ideally propagate instantaneously +* No ability to treat connectors independently - Need to PR to central registry +* Artefacts are not distributable outside of Github references - Tied to Github +* TODO + + ## Proposal While the precursor specifications outline the structure and mechanisms of packaging, this RFC details how the packages are owned, distributed, and indexed. A layered solution is outlined from storage up to user-applications and how they can be leveraged by CI in order to automatically publish updated versions and scrape topics for community contribution discovery. @@ -61,6 +87,7 @@ This solution enables the following UX scenarios: * Packages are referenced in Hasura V3 projects * Packages are fetched for local usage in Hasura V3 projects + ### Ownership Ownership is granted on an Organisation -> Package -> Author -> Version hierarchy. From 44d54f89fc2b02b782c7f239704c5a2e97672fa8 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Tue, 20 Feb 2024 08:16:40 +1000 Subject: [PATCH 13/16] milestones --- rfcs/0002-distribution.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 0ad68be3..292b5119 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -31,13 +31,36 @@ The following items are intended to be fleshed-out in this specification prior t In addition, all "TODO" references should be replaced before finalization. +### Delivery Roadmap + +Milestone 1 - Definition Links: + +* Where there are currently git tag references: + * Add link and checksum to package definition archive to DB Schema + * Add link and checksum to package definition archive to Github metadata format + +Milestone 2 - Topic Tributary: + +* Add a CI process to ingest connectors tagged with a topic in addition to registry connectors + +Milestone 3 - User and Role updates: + +* Extend the DB schema to include user/role information +* Set up auth to allow signup/login/token flows +* Extend the API permissions to allow role and resource based access to data +* Add validation actions to authoring flows + +Milestone 4 - CLI: + +* Create a new CLI plugin to manage interaction with the API + + ### Follow-Up Work After implementation of this system, follow-up actions should be performed: -* Migration and deprecation of previous hub API * Publication of new compatible versions of all available connectors -* TODO + ### Out of Scope for this RFC From ed19027b88ba051c889915924d11591d82377850 Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Tue, 20 Feb 2024 08:19:57 +1000 Subject: [PATCH 14/16] roadmap --- rfcs/0002-distribution.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 292b5119..6e11056b 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -17,6 +17,8 @@ The intuition for this system is inspired by other package management systems su There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha) +This proposal intends to allow the existing system to be extended to support new functionality and not require a migration. + ### Items Outstanding in this Specification (TODO) @@ -31,8 +33,11 @@ The following items are intended to be fleshed-out in this specification prior t In addition, all "TODO" references should be replaced before finalization. + ### Delivery Roadmap +The delivery of the changes outlined in this RFC can be rolled out incrementally, and this can be paused or stopped at any stage without disruption to the current system. + Milestone 1 - Definition Links: * Where there are currently git tag references: @@ -59,7 +64,8 @@ Milestone 4 - CLI: After implementation of this system, follow-up actions should be performed: -* Publication of new compatible versions of all available connectors +* Publication of package definition archives for all connectors +* Publication of new versions of all available connectors linking to archives ### Out of Scope for this RFC @@ -132,6 +138,7 @@ Only Operations can grant the Operations role. All changes within the system are logged and can be viewed by the auditor role. + ### Storage Package definitions take the form described in the packaging spec. These need to be stored. The storage mechanism can be described abstractly: @@ -154,6 +161,7 @@ The database backing the API provides all of the APIs state management capabilit The initial implementation of the Database will be Postgres. + ### API The API provides the user-interaction layer that mediates the database and storage components. No direct user interaction should occur with either the database, or storage, except for the case when the API delegates a storage interaction - such as providing an author a pre-authorized URL for publication, or providing a Hasura V3 project user a public storage URL for a package definition. From 09dc46930a503bc33d53d7fb5895b3d90f383d7d Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Tue, 20 Feb 2024 21:07:00 +1000 Subject: [PATCH 15/16] example MD change for MS 1 --- rfcs/0002-distribution.md | 44 +++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 6e11056b..93abea6b 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -38,24 +38,60 @@ In addition, all "TODO" references should be replaced before finalization. The delivery of the changes outlined in this RFC can be rolled out incrementally, and this can be paused or stopped at any stage without disruption to the current system. -Milestone 1 - Definition Links: +#### Milestone 1 - Definition Links: * Where there are currently git tag references: * Add link and checksum to package definition archive to DB Schema * Add link and checksum to package definition archive to Github metadata format -Milestone 2 - Topic Tributary: +The initial change to the metadata format would look as follows for the [Postgres Connector](https://github.com/hasura/ndc-hub/blob/lyndon/distribution-rfc/registry/postgres/metadata.json): + +```json +{ + "overview": { ... }, + "author": { ... }, + "is_verified": true, + "is_hosted_by_hasura": true, + // New stanza + "packages": [ + { + "uri": "https://foobar.com/releases/postgres-postgresql-v0.2.0-9283dh9283u...hd092ujdf2ued.tar.gz", + "checksum": { + "type": "sha256", + "value": "9283dh9283u...hd092ujdf2ued" + }, + // Optional link from package to source + "source": { + "hash": "98801634b0e1396c933188eef88178952f412a8c", + } + } + ] + "source_code": { + "is_open_source": true, + "repository": "https://github.com/hasura/ndc-postgres", + "version": [ + { + "tag": "v0.2.0", + "hash": "98801634b0e1396c933188eef88178952f412a8c", + "is_verified": true + } + ] + } +} +``` + +#### Milestone 2 - Topic Tributary: * Add a CI process to ingest connectors tagged with a topic in addition to registry connectors -Milestone 3 - User and Role updates: +#### Milestone 3 - User and Role updates: * Extend the DB schema to include user/role information * Set up auth to allow signup/login/token flows * Extend the API permissions to allow role and resource based access to data * Add validation actions to authoring flows -Milestone 4 - CLI: +#### Milestone 4 - CLI: * Create a new CLI plugin to manage interaction with the API From 11330f05035e95d102ded1da1e59f4a7f7a2996f Mon Sep 17 00:00:00 2001 From: Lyndon Maydwell Date: Tue, 20 Feb 2024 21:11:28 +1000 Subject: [PATCH 16/16] update --- rfcs/0002-distribution.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/rfcs/0002-distribution.md b/rfcs/0002-distribution.md index 93abea6b..6d25a5b0 100644 --- a/rfcs/0002-distribution.md +++ b/rfcs/0002-distribution.md @@ -80,6 +80,15 @@ The initial change to the metadata format would look as follows for the [Postgre } ``` +While package definition releases can be hosted at any URL, some convenient locations could include: + +* Under the github releases artefacts +* As a standalone github repository using the "download archive" feature +* In a Hasura Google cloud bucket (for Hasura authors) + +We will establish conventions for this that make authoring as streamlined as possible. + + #### Milestone 2 - Topic Tributary: * Add a CI process to ingest connectors tagged with a topic in addition to registry connectors