feat: filter extension registry

README.md

@@ -14,13 +14,9 @@ The build step uses a [Docker Engine](https://docs.docker.com/engine/) (even a [
 
 The (experimental) [builder service](#builder-service) makes the use of extensions even easier, no tools need to be installed except for k6x. The [builder service](#builder-service) builds the k6 binary on the fly.
 
-**asciicast (native builder)**
+**asciicast**
 
-[![asciicast](https://asciinema.org/a/610337.svg)](https://asciinema.org/a/610337)
-
-**asciicast (docker builder)**
-
-[![asciicast](https://asciinema.org/a/611493.svg)](https://asciinema.org/a/611493)
+[![asciicast](https://asciinema.org/a/612785.svg)](https://asciinema.org/a/612785)
 
 ## Prerequisites
 
@@ -106,7 +102,13 @@ The k6 subcommands are extended with some global command line flags related to b
   ```
   k6x run --with k6/x/mock script.js
   ```
-
+
+- `--filter expr` [jmespath](https://jmespath.org/) syntax extension registry [filter](#filtering) (default: `[*]`)
+
+  ```
+  k6x run --filter "[?contains(tiers,'Official')]" script.js
+  ```
+
 - `--builder list` a comma-separated list of builders (default: `service,native,docker`), available builders:
   - `service` this builder uses the builder service if it is specified (in the `K6X_BUILDER_SERVICE` environment variable), otherwise the next builder will be used without error
   - `native` this builder uses the installed go compiler if available, otherwise the next builder is used without error
@@ -128,6 +130,7 @@ Some new subcommands will also appear, which are related to building the k6 bina
   Flags:
     -o, --out name  output extension name
     --bin-dir path  folder for custom k6 binary (default: .)
+    --filter expr   jmespath syntax extension registry filter (default: [*])
     --builder list  comma separated list of builders (default: service,native,docker)
     -h, --help      display this help
   ```
@@ -151,6 +154,7 @@ Some new subcommands will also appear, which are related to building the k6 bina
 
   Flags:
     --addr address  listen address (default: 127.0.0.1:8787)
+    --filter expr   jmespath syntax extension registry filter (default: [*])
     --builder list  comma separated list of builders
 
     -h, --help      display this help
@@ -165,6 +169,7 @@ Some new subcommands will also appear, which are related to building the k6 bina
     --platform list    comma separated list of platforms (default: linux/amd64,linux/arm64,windows/amd64,windows/arm64,darwin/amd64,darwin/arm64)
     --stars number     minimum number of repository stargazers (default: 5)
     --with dependency  dependency and version constraints (default: latest version of k6 and registered extensions)
+    --filter expr      jmespath syntax extension registry filter (default: [*])
     --builder list     comma separated list of builders (default: service,local,docker)
     -h, --help         display this help
   ```
@@ -191,6 +196,24 @@ The build step is done using the go compiler included in the image. The partial
 
 The k6x docker builder (`--builder docker`) also uses this docker image. It creates a local volume called `k6x-cache` and mounts it to the `/cache` path. Thanks to this, the docker build runs almost at the same speed as the native build (apart from the first build).
 
+### Filtering
+
+In certain runtime environments, the use of arbitrary extensions is not allowed. There is a need to limit the extensions that can be used.
+
+This use case can be solved most flexibly by narrowing down the extension registry. The content of the [extension registry](https://github.com/grafana/k6-docs/blob/main/src/data/doc-extensions/extensions.json) can be narrowed using a [jmespath](https://jmespath.org/) syntax filter expression. Extensions can be filtered based on any property. 
+
+*allow only officially supported extensions*
+
+```
+k6x --filter "[?contains(tiers,'Official')]" run script.js
+```
+
+*allow only cloud enabled extensions*
+
+```
+k6x --filter "[?cloudEnabled == true]" run script.js
+```
+
 ## Appendix
 
 ### How It Works
@@ -359,6 +382,6 @@ https://example.com/linux/amd64/[email protected],[email protected],k6/x/[email protected],to
 
 Based on the platform parameters (`goos`, `goarch`) and dependencies, the service prepares the k6 binary.
 
-Since the response (the k6 binary) depends only on the request path, it can be easily cached. The service therefore sets a sufficiently long caching period (at least one year) in the response, as well as the usual cache headers (e.g. `ETag`). By placing a caching proxy in front of the service, it can be ensured that the actual k6 binary build takes place only once for each parameter combination.
+Since the response (the k6 binary) depends only on the request path, it can be easily cached. The service therefore sets a sufficiently long caching period (at least three month) in the response, as well as the usual cache headers (e.g. `ETag`). By placing a caching proxy in front of the service, it can be ensured that the actual k6 binary build takes place only once for each parameter combination.
 
 The advantage of the solution is that the k6 binary is created on the fly, only for the parameter combinations that are actually used. Since the service preserves the go cache between builds, a specific build happens quickly enough.

go.mod

@@ -12,6 +12,7 @@ require (
 	github.com/fatih/color v1.15.0
 	github.com/google/go-github/v55 v55.0.0
 	github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79
+	github.com/jmespath/go-jmespath v0.4.0
 	github.com/magefile/mage v1.15.0
 	github.com/mattn/go-colorable v0.1.13
 	github.com/mattn/go-isatty v0.0.18

go.sum

@@ -163,6 +163,10 @@ github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
+github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
+github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@@ -560,6 +564,8 @@ gopkg.in/mattn/go-colorable.v0 v0.1.0/go.mod h1:BVJlBXzARQxdi3nZo6f6bnl5yR20/tOL
 gopkg.in/mattn/go-isatty.v0 v0.0.4/go.mod h1:wt691ab7g0X4ilKZNmMII3egK0bTxl37fEn/Fwbd8gc=
 gopkg.in/mattn/go-runewidth.v0 v0.0.4/go.mod h1:BmXejnxvhwdaATwiJbB1vZ2dtXkQKZGu9yLFCZb4msQ=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

internal/cmd/cmd_build.go

@@ -41,6 +41,7 @@ Flags:
   -o, --out name     output extension name
   --bin-dir path     folder for custom k6 binary (default: {{.bin}})
   --with dependency  additional dependency and version constraints
+  --filter expr      jmespath syntax extension registry filter (default: [*])
   --builder list     comma separated list of builders (default: {{.builders}})
   --no-color         disable colored output  
   -h, --help         display this help

internal/cmd/cmd_preload.go

@@ -55,6 +55,7 @@ Flags:
   --platform list    comma separated list of platforms (default: {{.platforms}})
   --stars number     minimum number of repository stargazers (default: 5)
   --with dependency  dependency and version constraints (default: latest version of k6 and registered extensions)
+  --filter expr      jmespath syntax extension registry filter (default: [*])
   --builder list     comma separated list of builders (default: {{.builders}})
   -h, --help         display this help
 `

internal/cmd/cmd_service.go

@@ -82,7 +82,9 @@ Usage:
   {{.appname}} service [flags]
 
 Flags:
-  --addr address  listen address (default: 127.0.0.1:8787)
+  --addr address     listen address (default: 127.0.0.1:8787)
+  --filter expr      jmespath syntax extension registry filter (default: [*])
+  --builder list     comma separated list of builders (default: {{.builders}})
 
   -h, --help      display this help
 `

internal/cmd/main.go

@@ -45,6 +45,8 @@ func main(
 		return exitErr, err
 	}
 
+	defer opts.spinner.Stop()
+
 	initLogger(opts)
 
 	c := make(chan os.Signal, 1)
@@ -56,7 +58,11 @@ func main(
 		}
 	}()
 
-	res := resolver.NewWithCacheDir(opts.dirs.http)
+	res, err := resolver.New(opts.dirs.http, opts.filter)
+	if err != nil {
+		return exitErr, err
+	}
+
 	cmd := filepath.Join(opts.dirs.bin, k6Binary)
 
 	if opts.deps() {
@@ -135,12 +141,15 @@ const (
 
 	otherUsage = `
 Launcher Commands:
-  deps   Print k6 and extension dependencies
-  build  Build custom k6 binary with extensions
+  deps    Print k6 and extension dependencies
+  build   Build custom k6 binary with extensions
+  service Start k6x builder service
+  preload Preload (go) build cache
 
 Launcher Flags:
   --bin-dir path     cache folder for k6 binary (default: {{.bin}})
   --with dependency  additional dependency and version constraints
+  --filter expr      jmespath syntax extension registry filter (default: [*])
   --builder list     comma separated list of builders (default: {{.builders}})
   --clean            remove cached k6 binary
   --dry              do not run k6 command

internal/cmd/options.go

@@ -47,6 +47,7 @@ type options struct {
 	clean   bool
 	dry     bool
 	engines []builder.Engine
+	filter  string
 	out     []string
 	with    dependency.Dependencies
 	addr    string
@@ -92,7 +93,7 @@ func cleanargv(argv []string) []string {
 			continue
 		}
 
-		if arg == "--bin-dir" || arg == "--builder" || arg == "--with" {
+		if arg == "--bin-dir" || arg == "--builder" || arg == "--with" || arg == "--filter" {
 			i++
 			continue
 		}
@@ -115,6 +116,13 @@ func newFlagSet(opts *options) *pflag.FlagSet {
 
 	flag.StringArrayVarP(&opts.out, "out", "o", []string{}, "")
 
+	filter := os.Getenv(strings.ToUpper(opts.appname) + "_FILTER") //nolint:forbidigo
+	if len(filter) == 0 {
+		filter = "[*]"
+	}
+
+	flag.StringVar(&opts.filter, "filter", filter, "")
+
 	// deps command
 	flag.BoolVar(&opts.resolve, "resolve", false, "")
 	flag.BoolVar(&opts.json, "json", false, "")

internal/resolver/gh.go

@@ -7,7 +7,6 @@ package resolver
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"net/http"
 	"os"
@@ -17,27 +16,34 @@ import (
 	"github.com/google/go-github/v55/github"
 	"github.com/gregjones/httpcache"
 	"github.com/gregjones/httpcache/diskcache"
+	"github.com/jmespath/go-jmespath"
 	"github.com/szkiba/k6x/internal/dependency"
 )
 
 type ghResolver struct {
 	client *github.Client
+	filter *jmespath.JMESPath
 }
 
-func NewWithCacheDir(cachedir string) Resolver {
+func New(cachedir string, filter string) (Resolver, error) {
 	transport := httpcache.NewTransport(diskcache.New(cachedir))
 
 	client := &http.Client{Transport: newTransport(transport)}
 
-	return NewWithHTTPClient(client)
-}
+	res := new(ghResolver)
 
-func NewWithHTTPClient(client *http.Client) Resolver {
-	return NewWithGitHubClient(github.NewClient(client))
-}
+	res.client = github.NewClient(client)
+
+	if len(filter) != 0 {
+		query, err := jmespath.Compile(filter)
+		if err != nil {
+			return nil, err
+		}
 
-func NewWithGitHubClient(client *github.Client) Resolver {
-	return &ghResolver{client: client}
+		res.filter = query
+	}
+
+	return res, nil
 }
 
 func (res *ghResolver) Resolve(
@@ -75,9 +81,8 @@ func (res *ghResolver) getRegistry(ctx context.Context) (*extensionRegistry, err
 		return nil, err
 	}
 
-	reg := new(extensionRegistry)
-
-	if err = json.Unmarshal([]byte(str), reg); err != nil {
+	reg, err := parseExtensionRegistry([]byte(str), res.filter)
+	if err != nil {
 		return nil, err
 	}
 

internal/resolver/registry.go

@@ -6,9 +6,11 @@
 package resolver
 
 import (
+	"encoding/json"
 	"net/url"
 	"strings"
 
+	"github.com/jmespath/go-jmespath"
 	"github.com/szkiba/k6x/internal/dependency"
 )
 
@@ -22,6 +24,44 @@ type registeredExtension struct {
 	Type []string `json:"type,omitempty"`
 }
 
+func applyFilter(src []byte, filter *jmespath.JMESPath) ([]byte, error) {
+	if filter == nil {
+		return src, nil
+	}
+
+	loose := new(struct {
+		Extensions interface{} `json:"extensions"`
+	})
+
+	if err := json.Unmarshal(src, loose); err != nil {
+		return nil, err
+	}
+
+	data, err := filter.Search(loose.Extensions)
+	if err != nil {
+		return nil, err
+	}
+
+	loose.Extensions = data
+
+	return json.Marshal(loose)
+}
+
+func parseExtensionRegistry(src []byte, filter *jmespath.JMESPath) (*extensionRegistry, error) {
+	bin, err := applyFilter(src, filter)
+	if err != nil {
+		return nil, err
+	}
+
+	reg := new(extensionRegistry)
+
+	if err := json.Unmarshal(bin, reg); err != nil {
+		return nil, err
+	}
+
+	return reg, nil
+}
+
 func (reg *extensionRegistry) toModules() dependency.Modules {
 	mods := make(dependency.Modules)
 

releases/v0.4.0.md

@@ -8,7 +8,8 @@ k6x `v0.4.0` is here 🎉!
 
 Main new features:
 
- - Builder Service [#6](https://github.com/szkiba/k6x/issues/6)
+ - Builder Service [#17](https://github.com/szkiba/k6x/issues/17)
+ - Filter Extension Registry [#20](https://github.com/szkiba/k6x/issues/20)
 
 ## Builder Service
 
@@ -83,3 +84,22 @@ Based on the platform parameters (`goos`, `goarch`) and dependencies, the servic
 Since the response (the k6 binary) depends only on the request path, it can be easily cached. The service therefore sets a sufficiently long caching period (at least one year) in the response, as well as the usual cache headers (e.g. `ETag`). By placing a caching proxy in front of the service, it can be ensured that the actual k6 binary build takes place only once for each parameter combination.
 
 The advantage of the solution is that the k6 binary is created on the fly, only for the parameter combinations that are actually used. Since the service preserves the go cache between builds, a specific build happens quickly enough.
+
+## Filter Extension Registry
+
+In certain runtime environments, the use of arbitrary extensions is not allowed. There is a need to limit the extensions that can be used.
+
+This use case can be solved most flexibly by narrowing down the extension registry. The content of the [extension registry](https://github.com/grafana/k6-docs/blob/main/src/data/doc-extensions/extensions.json) can be narrowed using a [jmespath](https://jmespath.org/) syntax filter expression. Extensions can be filtered based on any property. 
+
+*allow only officially supported extensions*
+
+```
+k6x --filter "[?contains(tiers,'Official')]" run script.js
+```
+
+*allow only cloud enabled extensions*
+
+```
+k6x --filter "[?cloudEnabled == true]" run script.js
+```
+