[Proposal] Migrate pipeline to Dagger Go SDK

[rossf7]

Before the break I got a tip from @dipankardas011 about this newish k3s module for dagger
https://daggerverse.dev/mod/github.com/marcosnils/daggerverse/k3s

This got me thinking dagger could help with some pain points I have with the current pipeline

• Pipeline logic can't easily be run locally
• PRs from forked repos fail because they can't access the kubeconfig secret (for very sensible security reasons :)

I did some hacking on this over the holidays and got our pipeline running in a k3s container managed by the module.

Benefits

• Whole pipeline including deploying falco and creating benchmark jobs can be run locally in a kind cluster
• Whole pipeline can be developed in go. GitHub Actions workflow just becomes a wrapper that calls the dagger pipeline
• Automated tests can be run with full stack (including kepler in estimation mode) in a k3s container on a GitHub Actions runner
• k3s test clusters use same flux manifests as prod cluster (just needs patches to helmrelease values for volume mounts)

Weaknesses

• There is a learning curve for dagger
• Barrier to entry for new contributors may be slightly higher (having go and docker knowledge helps)

Alternatives considered

act is another tool for running actions workflows locally but using it with kind needs a "kitchen sink" docker image that is 18Gb!
nektos/act#335

Due to size I've split the changes into 2 PRs. #152 ports the current pipeline to dagger. #153 adds the integration test and docs for running the pipeline locally.

I did invest some time here because I think its a good solution for us. However it was also fun to learn something new and I'll probably use dagger for some of my own projects. So if we don't go this direction no worries! ☺️

[locomundo]

This is a great initiative, Ross!! It will make it much easier to work on GR! I will take a look at the PRs. Thank you!

[rossf7]

We discussed dagger in this weeks meeting and I demoed progress so far. Next steps are

• Main objective is to make onboarding experience for new contributors as smooth as possible. So need to improve docs and add makefile (or use nix ;)
• Secondary objective is level up go code and add linter.

Please add feedback in this issue or PR #153. I closed #152 because 2 PRs was too confusing!

We'll discuss again during next meeting (22nd Jan) to review progress and decide if we want to use dagger.