From 2322bda70a1b5db258e19ca0e10af2f77bce74fa Mon Sep 17 00:00:00 2001 From: Dominik Kutra Date: Tue, 2 Jul 2024 14:29:50 +0200 Subject: [PATCH 1/8] add review instructions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fynn Beuttenmüller --- README.md | 53 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2550c5c1..8a13f482 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,53 @@ # collection -This repository is used to manage the resources displayed on [bioimage.io](http://bioimage.io). +This repository is used to manage the resources displayed on [bioimage.io][bioimageio]. -Most users will not directly dispatch the workflows defined in this reporitory, but should instead login on [bioimage.io](http://bioimage.io) and use the front-end to interact with the bioimage.io collection. +Most users will not directly dispatch the workflows defined in this reporitory, but should instead login on [bioimage.io][bioimageio] and use the front-end to interact with the bioimage.io collection. We currently do not have a workflow for direct upload, but publicly available resource packages may be staged with a stage workflow dispatch. -## Lifecycle of a draft +## Maintaining the bioimage.io Collection + +In order to update or add new resources to the [bioimage.io][bioimageio] collection, they have to undergo review. +Current reviewers are listed in [`bioimageio_collection_config.json`][review-config], under section `reviewers`. + +### Reviewer Onboarding + +1. Open a pull request, adding a person to the [list of reviewers][review-config], see https://github.com/bioimage-io/collection/pull/75 for an example. + * one public email address is required + * github id can be found using `https://api.github.com/users/` +1. Once the pull request has been merged, the new reviewer can + * accept resource drafts + * request changes on resource drafts + * upload a new version for any resource + +### Review Process + +The review process technically starts after a user uploaded a _resource package_. +Such a _resource package_ could e.g. be a newly uploaded _model package_, or _notebook package_, or an updated version of any existing resource. +Typically, uploaders would go via [bioimage.io/upload][upload]. +Alternatively, any direct link to a downloadable resource package (`.zip`-file) would work. +The latter option is reserved for members of this repository (or the bioimageio org). + +#### Staging + +Now the stage workflow needs to be dispatched. +If the resource package was uploaded via the bioimage.io website, this is initiated automatically. +In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-action] ("run workflow"). +Staging unpacks the files from the zipped resource package to our public S3. +Once unpacked, the staged _resource draft_ is automatically tested ([test action][test-action] is triggerd automatically at the end of the stage action). + +#### Testing + +Staged resource drafts are automatically tested: +* Is their metadata valid? +* Can test outputs be reproduced from test inputs? +* Are linked URLs available? +* ... + +Once the -1. unpacking: Uploaded resource packages are unpacked and their individuell files uploaded to our public S3 storage. -2. testing: Staged resource drafts are automatically tested: Is their metadata valid? Can test outputs be reproduced from test inputs? Are linked URLs available? 3. awaiting reviewe: After the tests have concluded the bioimageio reviewers are notified. 4. The reviewer will result in a) changes requested: Please upload an updated draft (which overwrites the current draft). @@ -30,3 +67,9 @@ graph TD; ar--->accepted[4b: accepted] accepted-->published[published: (draft is deleted)] ``` + +[bioimageio]: https://bioimage.io +[review-config]: https://github.com/bioimage-io/collection/blob/main/bioimageio_collection_config.json +[staging-action]: https://github.com/bioimage-io/collection/actions/workflows/stage.yaml +[test-action]: https://github.com/bioimage-io/collection/actions/workflows/test.yaml +[upload]: https://bioimage.io/#/upload From 22f52f7544512e095437db4bc497f6682a3ae073 Mon Sep 17 00:00:00 2001 From: Dominik Kutra Date: Tue, 2 Jul 2024 15:04:14 +0200 Subject: [PATCH 2/8] More updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fynn Beuttenmüller --- README.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8a13f482..22f6af69 100644 --- a/README.md +++ b/README.md @@ -35,19 +35,30 @@ The latter option is reserved for members of this repository (or the bioimageio Now the stage workflow needs to be dispatched. If the resource package was uploaded via the bioimage.io website, this is initiated automatically. -In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-action] ("run workflow"). +In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-action] ("run workflow")[^1]. Staging unpacks the files from the zipped resource package to our public S3. -Once unpacked, the staged _resource draft_ is automatically tested ([test action][test-action] is triggerd automatically at the end of the stage action). +Once unpacked, the staged _resource draft_ is automatically tested ([test workflow][test-action] is triggerd automatically at the end of the stage action). #### Testing Staged resource drafts are automatically tested: + * Is their metadata valid? * Can test outputs be reproduced from test inputs? * Are linked URLs available? * ... -Once the +Tests can also be triggered (via github api or manually) by dispatching the [test workflow][test-action][^2]. + +Once the tests are completed, a reviewer needs to take a look. +The uploader gets a notification via email. +An overview of all pending _resource drafts_ can be found at https://bioimageio-uploader.netlify.app/#/status. +A _draft_ is identified by its concept id (`id` from the `rdf.yaml`). + +#### Review + +Reviewers should check the models for technical correctness (aided by CI, see [Testing section](#testing)) and contents/metadata of the resource. +For models, reviewers can use [the model documentation][model-docs] as a guide. 3. awaiting reviewe: After the tests have concluded the bioimageio reviewers are notified. 4. The reviewer will result in @@ -68,7 +79,11 @@ graph TD; accepted-->published[published: (draft is deleted)] ``` +[^1]: Parameters to this workflow are `Bioimage.io resource identifier` (`id` from the `rdf.yaml`), and `Download URL of the resource package zip-file`, which should contain a publicly reachable url to a _resource package_ `.zip`-file. +[^2]: Parameters to this workflow are `Bioimage.io resource concept` (`id` from the `rdf.yaml`), and `Published version or 'draft'` (optional, usually `draft`). + [bioimageio]: https://bioimage.io +[model-docs]: https://bioimage.io/docs/#/guides/developers-guide?id=model-documentation [review-config]: https://github.com/bioimage-io/collection/blob/main/bioimageio_collection_config.json [staging-action]: https://github.com/bioimage-io/collection/actions/workflows/stage.yaml [test-action]: https://github.com/bioimage-io/collection/actions/workflows/test.yaml From 212d5f171b1d23a0084303396fc41e5fcdc38259 Mon Sep 17 00:00:00 2001 From: Dominik Kutra Date: Tue, 2 Jul 2024 15:28:33 +0200 Subject: [PATCH 3/8] even more review documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fynn Beuttenmüller --- README.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 22f6af69..28d02bd0 100644 --- a/README.md +++ b/README.md @@ -35,9 +35,9 @@ The latter option is reserved for members of this repository (or the bioimageio Now the stage workflow needs to be dispatched. If the resource package was uploaded via the bioimage.io website, this is initiated automatically. -In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-action] ("run workflow")[^1]. +In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-w] ("run workflow")[^1]. Staging unpacks the files from the zipped resource package to our public S3. -Once unpacked, the staged _resource draft_ is automatically tested ([test workflow][test-action] is triggerd automatically at the end of the stage action). +Once unpacked, the staged _resource draft_ is automatically tested ([test workflow][test-wf] is triggerd automatically at the end of the stage action). #### Testing @@ -48,7 +48,7 @@ Staged resource drafts are automatically tested: * Are linked URLs available? * ... -Tests can also be triggered (via github api or manually) by dispatching the [test workflow][test-action][^2]. +Tests can also be triggered (via github api or manually) by dispatching the [test workflow][test-wf][^2]. Once the tests are completed, a reviewer needs to take a look. The uploader gets a notification via email. @@ -60,10 +60,17 @@ A _draft_ is identified by its concept id (`id` from the `rdf.yaml`). Reviewers should check the models for technical correctness (aided by CI, see [Testing section](#testing)) and contents/metadata of the resource. For models, reviewers can use [the model documentation][model-docs] as a guide. -3. awaiting reviewe: After the tests have concluded the bioimageio reviewers are notified. -4. The reviewer will result in - a) changes requested: Please upload an updated draft (which overwrites the current draft). - b) accepted: The resource will be published (and the draft deleted). +Reviewers can: + + * _request changes_: + A contributor is expected to upload an updated (fixed) draft (which overwrites the current draft). + This can be done either by the website, or the stage workflow. + Important is to keep the `id` the same. +* _accept_: + Accepting the _resource draft_ via the web interface triggers the [publish workflow][publish-wf], which creates a new unique _resource version_. + As a result, the resource is published, the draft deleted and, thus, the _resource_ is available via the [bioimage.io][bioimageio] website. + The [backup workflow][backup-wf] will upload/publish the _resource version_ to zenodo using the bioimage.io bot account (tagged with [`backup.bioimage.io`][zenodo-overview]). + Additionally an 'error' status may be shown if an exception occured. This also may be the case for invalid inputs. @@ -76,15 +83,18 @@ graph TD; ar--->cr[4a: changes requestd] cr-->unpacking ar--->accepted[4b: accepted] - accepted-->published[published: (draft is deleted)] + accepted-->published[published: draft is deleted] ``` [^1]: Parameters to this workflow are `Bioimage.io resource identifier` (`id` from the `rdf.yaml`), and `Download URL of the resource package zip-file`, which should contain a publicly reachable url to a _resource package_ `.zip`-file. [^2]: Parameters to this workflow are `Bioimage.io resource concept` (`id` from the `rdf.yaml`), and `Published version or 'draft'` (optional, usually `draft`). +[backup-wf]: https://github.com/bioimage-io/collection/actions/workflows/backup.yaml [bioimageio]: https://bioimage.io [model-docs]: https://bioimage.io/docs/#/guides/developers-guide?id=model-documentation [review-config]: https://github.com/bioimage-io/collection/blob/main/bioimageio_collection_config.json -[staging-action]: https://github.com/bioimage-io/collection/actions/workflows/stage.yaml -[test-action]: https://github.com/bioimage-io/collection/actions/workflows/test.yaml +[publish-wf]: https://github.com/bioimage-io/collection/actions/workflows/publish.yaml +[staging-wf]: https://github.com/bioimage-io/collection/actions/workflows/stage.yaml +[test-wf]: https://github.com/bioimage-io/collection/actions/workflows/test.yaml [upload]: https://bioimage.io/#/upload +[zenodo-overview]: https://zenodo.org/search?q=metadata.subjects.subject%3A%22backup.bioimage.io%22&l=list&p=1&s=10&sort=bestmatch From 9bf5c1a1209e7a4c8a5edf05ec2e49edd4792ef7 Mon Sep 17 00:00:00 2001 From: Dominik Kutra Date: Tue, 2 Jul 2024 15:33:51 +0200 Subject: [PATCH 4/8] Update README.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fynn Beuttenmüller --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 28d02bd0..d0a46f75 100644 --- a/README.md +++ b/README.md @@ -66,11 +66,17 @@ Reviewers can: A contributor is expected to upload an updated (fixed) draft (which overwrites the current draft). This can be done either by the website, or the stage workflow. Important is to keep the `id` the same. +* _apply minor changes_: + For minor changes it is often not necessary to involve the contributor directly. + Such changes could be e.g. obvious typos in some of the fields. + Reviewers can apply changes locally to the _resource package_ and re-upload (using the same `id`), or, given S3 credentials, files could also be changed in place. + Latter way of changing requires manually triggering the tests (see [Testing section](#testing)). * _accept_: Accepting the _resource draft_ via the web interface triggers the [publish workflow][publish-wf], which creates a new unique _resource version_. As a result, the resource is published, the draft deleted and, thus, the _resource_ is available via the [bioimage.io][bioimageio] website. The [backup workflow][backup-wf] will upload/publish the _resource version_ to zenodo using the bioimage.io bot account (tagged with [`backup.bioimage.io`][zenodo-overview]). - + + Additionally an 'error' status may be shown if an exception occured. This also may be the case for invalid inputs. From e07f21ab07e3198711db9ac3ec2f517a9568c256 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fynn=20Beuttenm=C3=BCller?= Date: Tue, 2 Jul 2024 16:33:14 +0200 Subject: [PATCH 5/8] Update README.md --- README.md | 53 +++++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index d0a46f75..3d7de878 100644 --- a/README.md +++ b/README.md @@ -23,21 +23,36 @@ Current reviewers are listed in [`bioimageio_collection_config.json`][review-con * request changes on resource drafts * upload a new version for any resource -### Review Process +### Adding a bioimage.io resource -The review process technically starts after a user uploaded a _resource package_. -Such a _resource package_ could e.g. be a newly uploaded _model package_, or _notebook package_, or an updated version of any existing resource. +A bioimage.io resource is created by a user uploading a _resource package_. +Such a _resource package_ could e.g. be a newly uploaded _model package_, or _notebook package_. Typically, uploaders would go via [bioimage.io/upload][upload]. Alternatively, any direct link to a downloadable resource package (`.zip`-file) would work. The latter option is reserved for members of this repository (or the bioimageio org). +Once available online the _resource package_ is staged (see [Staging section](#staging)), tested (see [Testing section](#testing)), and reviewed (see [Review section](#review)). + + +graph TD; + stage[stage] + test[test] + cr[request changes] + publish[accept/publish] + backup[backup to Zenodo] + + stage-->test + test-->cr + cr-->stage + test-->publish + publish-->backup #### Staging -Now the stage workflow needs to be dispatched. -If the resource package was uploaded via the bioimage.io website, this is initiated automatically. -In case of a url to a resource package, the `stage` workflow needs to be [dispatched manually, or via github api][staging-w] ("run workflow")[^1]. +Given a download URL to a _resource package_ the `stage` workflow needs to be [dispatched manually, or via github api][staging-wf] ("run workflow")[^1]. +If the _resource package_ was uploaded via the bioimage.io website, this is initiated automatically by the uploader service. + Staging unpacks the files from the zipped resource package to our public S3. -Once unpacked, the staged _resource draft_ is automatically tested ([test workflow][test-wf] is triggerd automatically at the end of the stage action). +Once unpacked, the staged _resource draft_ is automatically tested (the [test workflow][test-wf] is dispatched automatically at the end of the stage workflow). #### Testing @@ -50,14 +65,19 @@ Staged resource drafts are automatically tested: Tests can also be triggered (via github api or manually) by dispatching the [test workflow][test-wf][^2]. -Once the tests are completed, a reviewer needs to take a look. -The uploader gets a notification via email. +Once the tests are completed, the uploader gets a notification via email that their draft is awaiting review; now a reviewer needs to take a look. + An overview of all pending _resource drafts_ can be found at https://bioimageio-uploader.netlify.app/#/status. A _draft_ is identified by its concept id (`id` from the `rdf.yaml`). #### Review Reviewers should check the models for technical correctness (aided by CI, see [Testing section](#testing)) and contents/metadata of the resource. + +To this end it can be helpful to check the logs displayed at https://bioimageio-uploader.netlify.app/#/status/. +There information about automated workflow steps and validation outcome is logged. +Additionally an 'error' status may be shown if an exception occured in the GitHub workflow run producing the log. + For models, reviewers can use [the model documentation][model-docs] as a guide. Reviewers can: @@ -77,21 +97,6 @@ Reviewers can: The [backup workflow][backup-wf] will upload/publish the _resource version_ to zenodo using the bioimage.io bot account (tagged with [`backup.bioimage.io`][zenodo-overview]). - -Additionally an 'error' status may be shown if an exception occured. -This also may be the case for invalid inputs. - -```mermaid -graph TD; - unpacking[1: unpacking]-->unpacked[unpacked] - unpacked-->testing[2: testing] - testing-->ar[3: awaiting review] - ar--->cr[4a: changes requestd] - cr-->unpacking - ar--->accepted[4b: accepted] - accepted-->published[published: draft is deleted] -``` - [^1]: Parameters to this workflow are `Bioimage.io resource identifier` (`id` from the `rdf.yaml`), and `Download URL of the resource package zip-file`, which should contain a publicly reachable url to a _resource package_ `.zip`-file. [^2]: Parameters to this workflow are `Bioimage.io resource concept` (`id` from the `rdf.yaml`), and `Published version or 'draft'` (optional, usually `draft`). From cf99b3f26844a183b9a9291601cb817d58290270 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fynn=20Beuttenm=C3=BCller?= Date: Tue, 2 Jul 2024 16:44:27 +0200 Subject: [PATCH 6/8] Update workflow input descriptions --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3d7de878..2c921a71 100644 --- a/README.md +++ b/README.md @@ -97,8 +97,8 @@ Reviewers can: The [backup workflow][backup-wf] will upload/publish the _resource version_ to zenodo using the bioimage.io bot account (tagged with [`backup.bioimage.io`][zenodo-overview]). -[^1]: Parameters to this workflow are `Bioimage.io resource identifier` (`id` from the `rdf.yaml`), and `Download URL of the resource package zip-file`, which should contain a publicly reachable url to a _resource package_ `.zip`-file. -[^2]: Parameters to this workflow are `Bioimage.io resource concept` (`id` from the `rdf.yaml`), and `Published version or 'draft'` (optional, usually `draft`). +[^1]: Parameters to this workflow are `bioimage.io resource concept ID` (`id` from the `rdf.yaml`), and `Download URL of the resource package zip-file`, which should contain a publicly reachable URL to a _resource package_ `.zip`-file. +[^2]: Parameters to this workflow are `Bioimage.io resource concept ID` (`id` from the `rdf.yaml`), and `Published version or 'draft'` (optional, usually `draft`). [backup-wf]: https://github.com/bioimage-io/collection/actions/workflows/backup.yaml [bioimageio]: https://bioimage.io From 50db77c5b103635306cca4eba9a6e627bbc3202b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fynn=20Beuttenm=C3=BCller?= Date: Tue, 2 Jul 2024 16:56:35 +0200 Subject: [PATCH 7/8] fix concept_id status page link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2c921a71..e2332739 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ A _draft_ is identified by its concept id (`id` from the `rdf.yaml`). Reviewers should check the models for technical correctness (aided by CI, see [Testing section](#testing)) and contents/metadata of the resource. -To this end it can be helpful to check the logs displayed at https://bioimageio-uploader.netlify.app/#/status/. +To this end it can be helpful to check the logs displayed at `https://bioimageio-uploader.netlify.app/#/status/`. There information about automated workflow steps and validation outcome is logged. Additionally an 'error' status may be shown if an exception occured in the GitHub workflow run producing the log. From 28dbe089ea9b871a28c201307e922bcebdf67312 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fynn=20Beuttenm=C3=BCller?= Date: Tue, 2 Jul 2024 16:59:05 +0200 Subject: [PATCH 8/8] mention website use case first --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e2332739..f0787f10 100644 --- a/README.md +++ b/README.md @@ -48,8 +48,9 @@ graph TD; #### Staging -Given a download URL to a _resource package_ the `stage` workflow needs to be [dispatched manually, or via github api][staging-wf] ("run workflow")[^1]. -If the _resource package_ was uploaded via the bioimage.io website, this is initiated automatically by the uploader service. +If the _resource package_ was uploaded via the bioimage.io website, the staging of the new resource draft is initiated automatically by the uploader service. +Otherwise, given a download URL to a _resource package_, the `stage` workflow needs to be [dispatched manually, or via github api][staging-wf] ("run workflow")[^1]. + Staging unpacks the files from the zipped resource package to our public S3. Once unpacked, the staged _resource draft_ is automatically tested (the [test workflow][test-wf] is dispatched automatically at the end of the stage workflow).