[CI] Auto distribution of connect as a rpk managed plugin

.github/workflows/test_plugin_uploader.yml

@@ -0,0 +1,56 @@
+name: Test Plugin Uploader
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  unit-test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: install deps
+        working-directory: resources/plugin_uploader
+        run: pip install -r requirements_dev.txt
+
+      - name: run unit tests
+        working-directory: resources/plugin_uploader
+        run: pytest .
+
+  ruff-lint:
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - uses: chartboost/ruff-action@v1
+        with:
+          version: 0.4.10
+          src: './resources/plugin_uploader'
+
+  pyright-type-check:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.12'
+
+    - name: install deps
+      working-directory: resources/plugin_uploader
+      run: pip install -r requirements_dev.txt
+
+    - name: run pyright
+      working-directory: resources/plugin_uploader
+      run: pyright

.gitignore

@@ -9,3 +9,4 @@ release_notes.md
 .idea
 .vscode
 .op
+__pycache__

resources/plugin_uploader/README.md

@@ -0,0 +1,90 @@
+# Plugin uploader
+
+## Description
+
+```
+Usage: plugin_uploader.py [OPTIONS] COMMAND [ARGS]...
+
+CLI tool to upload/index goreleaser-built binaries to/in S3.
+
+Options:
+--help  Show this message and exit.
+
+Commands:
+upload-archives  Create tar.gz archives from binaries and upload to S3
+upload-manifest  Create manifest.json and upload to S3
+
+`plugin_uploader.py` is used to upload the binaries generated by goreleaser to S3 in a manner that is consumable by RPK as a plugin.
+
+```
+
+## Install
+
+`pip install -r requirements.txt`
+
+## How to use
+
+Primary use case is in GitHub Actions in response to creation of a GitHub release.
+
+Here are more details about running this in other contexts (e.g. local testing, manual release interventions)
+
+It's expected that you have used goreleaser to build a set of binaries for a given release tag (such as following a
+GitHub release tag creation).
+
+Goreleaser creates a `$DIST` directory at the project root dir containing all built binaries and two JSON files:
+
+* `$DIST/<build-name>-<os>-<arch>/<binary-filename>`
+* ...
+* `$DIST/artifacts.json`
+* `$DIST/metadata.json`
+
+
+### Create archives from binaries and upload them
+
+Locate the `artifact.json` and `metadata.json` files produced by goreleaser.
+E.g. `$DIST/artifacts.json`, `$DIST/metadata.json`.
+
+`$DIST="dist"`, by default, but may be overridden to something else in goreleaser configs.
+
+```shell
+./plugin_uploader.py upload-archives \
+                        --artifacts-file=$DIST/artifacts.json \
+                        --metadata-file=$DIST/metadata.json \
+                        --project-root-dir=<PROJECT_ROOT> \
+                        --region=<AWS_REGION> \
+                        --bucket=<AWS_S3_BUCKET> \
+                        --plugin=<PLUGIN_NAME> \
+                        --goos=<OS1,OS2,...> \
+                        --goarch=<ARCH1,ARCH2,...>
+```
+
+`PROJECT_ROOT` should be the root directory of the Golang project (by default, wherer `.goreleaser.yml` lives)
+
+`PLUGIN_NAME` should match the `<build-name>` as defined in goreleaser configs.
+It's assumed that the binary filename is always `redpanda-<build-name>`. E.g. for `connect`:
+
+* `build-name` is `connect`
+* Binary filename is `redpanda-connect`
+
+A binary is included for archival / upload only if it matches some `--goos` AND some `--goarch`.
+
+`--dry-run` is available for skipping final S3 upload step.
+
+### Create manifest.json and upload it
+
+This lists all archives for the specific plugin and constructs a `manifest.json` from the listing.
+
+This should be run after uploading any archives.
+
+```shell
+./plugin_uploader.py upload-manifest \
+                        --region=<AWS_REGION> \
+                        --bucket=<AWS_S3_BUCKET> \
+                        --plugin=<PLUGIN_NAME> \
+                        --repo-hostname=<REPO_HOSTNAME>
+```
+
+`--repo-hostname` is used for generating the right public facing download URLs for archives in the plugin repo. E.g.
+`rpk-plugins.redpanda.com`
+
+`--dry-run` is available for skipping the final S3 upload step.