Skip to content

Commit

Permalink
feat(providers): first implementation of a 1password connect provider (
Browse files Browse the repository at this point in the history
…#164)

* feat(providers): first implementation of a 1password connect provider

Signed-off-by: Fabian Jucker <[email protected]>

* style: remove newline

Signed-off-by: Fabian Jucker <[email protected]>

* test: add basic integration test

Signed-off-by: Fabian Jucker <[email protected]>

---------

Signed-off-by: Fabian Jucker <[email protected]>
  • Loading branch information
juckerf authored Sep 29, 2023
1 parent 601350d commit 904d391
Show file tree
Hide file tree
Showing 8 changed files with 218 additions and 19 deletions.
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ It supports various backends including:
- [Google Sheets](#google-sheets)
- [SOPS](https://github.com/getsops/sops)-encrypted files
- Terraform State
- 1Password Connect
- CredHub(Coming soon)

- Use `vals eval -f refs.yaml` to replace all the `ref`s in the file to actual values and secrets.
Expand Down Expand Up @@ -210,6 +211,7 @@ Please see the [relevant unit test cases](https://github.com/helmfile/vals/blob/
- [Azure Key Vault](#azure-key-vault)
- [EnvSubst](#envsubst)
- [GitLab](#gitlab)
- [1Password Connect](#1password-connect)

Please see [pkg/providers](https://github.com/helmfile/vals/tree/master/pkg/providers) for the implementations of all the providers. The package names corresponds to the URI schemes.

Expand Down Expand Up @@ -628,6 +630,27 @@ Examples:
- `ref+gitlab://gitlab.com/11111/password`
- `ref+gitlab://my-gitlab.org/11111/password?ssl_verify=true&scheme=https`
### 1Password Connect
For this provider to work you require a working and accessible [1Password connect server](https://developer.1password.com/docs/connect).
The following env vars have to be configured:
- `OP_CONNECT_HOST`
- `OP_CONNET_TOKEN`
1Password is organized in vaults and items.
An item can have multiple fields with or without a section. Labels can be set on fields and sections.
Vaults, items, sections and labels can be accessed by ID or by label/name (and IDs and labels can be mixed and matched in one URL).
If a section does not have a label the field is only accessible via the section ID. This does not hold true for some default fields which may have no section at all (e.g.username and password for a `Login` item).
*Caution: vals-expressions are parsed as URIs. For the 1Password connect provider the host component of the URI identifies the vault (by ID or name). Therefore vaults containing certain characters not allowed in the host component (e.g. whitespaces, see [RFC-3986](https://www.rfc-editor.org/rfc/rfc3986#section-3.2.2) for details) can only be accessed by ID.*
Examples:
- `ref+onepasswordconnect://VAULT_ID/ITEM_ID#/[SECTION_ID.]FIELD_ID`
- `ref+onepasswordconnect://VAULT_LABEL/ITEM_LABEL#/[SECTION_LABEL.]FIELD_LABEL`
- `ref+onepasswordconnect://VAULT_LABEL/ITEM_ID#/[SECTION_LABEL.]FIELD_ID`
## Advanced Usages
### Discriminating config and secrets
Expand Down
5 changes: 5 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ require (
cloud.google.com/go/iam v1.1.1 // indirect
cloud.google.com/go/kms v1.15.2 // indirect
filippo.io/age v1.1.1 // indirect
github.com/1Password/connect-sdk-go v1.5.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.0.1 // indirect
Expand Down Expand Up @@ -105,14 +106,18 @@ require (
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
github.com/urfave/cli v1.22.14 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
golang.org/x/crypto v0.13.0 // indirect
golang.org/x/net v0.15.0 // indirect
golang.org/x/sync v0.3.0 // indirect
Expand Down
11 changes: 11 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ cloud.google.com/go/storage v1.33.0 h1:PVrDOkIC8qQVa1P3SXGpQvfuJhN2LHOoyZvWs8D2X
cloud.google.com/go/storage v1.33.0/go.mod h1:Hhh/dogNRGca7IWv1RC2YqEn0c0G77ctA/OxflYkiD8=
filippo.io/age v1.1.1 h1:pIpO7l151hCnQ4BdyBujnGP2YlUo0uj6sAVNHGBvXHg=
filippo.io/age v1.1.1/go.mod h1:l03SrzDUrBkdBx8+IILdnn2KZysqQdbEBUQ4p3sqEQE=
github.com/1Password/connect-sdk-go v1.5.3 h1:KyjJ+kCKj6BwB2Y8tPM1Ixg5uIS6HsB0uWA8U38p/Uk=
github.com/1Password/connect-sdk-go v1.5.3/go.mod h1:5rSymY4oIYtS4G3t0oMkGAXBeoYiukV3vkqlnEjIDJs=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.2 h1:t5+QXLCK9SVi0PPdaY0PrFvYUo24KwA0QwxnaHRSVd4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.2/go.mod h1:bjGvMhVMb+EEm3VRNQawDMUyMMjo+S5ewNjflkep/0Q=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.1 h1:LNHhpdK7hzUcx/k1LIcuh5k7k1LGIWLQfCjaneSj7Fc=
Expand Down Expand Up @@ -268,6 +270,8 @@ github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 h1:rzf0wL0CHVc8CEsgyygG0
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/image-spec v1.0.2 h1:9yCKha/T5XdGtO0q9Q9a6T5NUCsTn/DrBg0D7ufOcFM=
github.com/opencontainers/runc v1.1.5 h1:L44KXEpKmfWDcS02aeGm8QNTFXTo2D+8MYGDIJ/GDEs=
github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
github.com/ory/dockertest/v3 v3.10.0 h1:4K3z2VMe8Woe++invjaTB7VRyQXQy5UY+loujO4aNE4=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
Expand All @@ -288,13 +292,18 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/uber/jaeger-client-go v2.30.0+incompatible h1:D6wyKGCecFaSRUpo8lCVbaOOb6ThwMmTEbhRwtKR97o=
github.com/uber/jaeger-client-go v2.30.0+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=
github.com/uber/jaeger-lib v2.4.1+incompatible h1:td4jdvLcExb4cBISKIpHuGoVXh+dVKhn2Um6rjCsSsg=
github.com/uber/jaeger-lib v2.4.1+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U=
github.com/urfave/cli v1.22.14 h1:ebbhrRiGK2i4naQJr+1Xj92HXZCrK7MsyTS/ob3HnAk=
github.com/urfave/cli v1.22.14/go.mod h1:X0eDS6pD6Exaclxm99NJ3FiCDRED7vIHpx2mDOHLvkA=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f h1:J9EGpcZtP0E/raorCMxlFGSTBrsSlaDGf3jU/qvAE2c=
Expand All @@ -303,6 +312,8 @@ github.com/xeipuuv/gojsonschema v1.2.0 h1:LhYJRs+L4fBtjZUfuSZIKGeVu0QRy8e5Xi7D17
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
Expand Down
90 changes: 90 additions & 0 deletions pkg/providers/onepasswordconnect/onepasswordconnect.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
package onepasswordconnect

import (
"errors"
"fmt"
"strings"

"github.com/1Password/connect-sdk-go/connect"
"gopkg.in/yaml.v3"

"github.com/helmfile/vals/pkg/api"
)

type provider struct {
client connect.Client
}

// New creates a new 1Password Connect provider
func New(cfg api.StaticConfig) *provider {
p := &provider{}

return p
}

// Get secret string from 1Password Connect
func (p *provider) GetString(key string) (string, error) {
var err error

splits := strings.Split(key, "/")
if len(splits) < 2 {
return "", fmt.Errorf("invalid URI: %v", errors.New("vault or item missing"))
}

client, err := connect.NewClientFromEnvironment()
if err != nil {
return "", fmt.Errorf("storage.NewClient: %v", err)
}

p.client = client

item, err := client.GetItem(splits[1], splits[0])
if err != nil {
return "", fmt.Errorf("error retrieving item: %v", err)
}

var data = make(map[string]string)
// fill map with all possible ID/Label combinations for value
for _, f := range item.Fields {
data[f.ID] = f.Value
// if no section on field (default fields on some item types) use value directly
if f.Section == nil {
data[f.Label] = f.Value
} else {
if f.Section.Label != "" {
var key = strings.Join([]string{f.Section.Label, f.Label}, ".")
data[key] = f.Value
key = strings.Join([]string{f.Section.Label, f.ID}, ".")
data[key] = f.Value
}
key = strings.Join([]string{f.Section.ID, f.Label}, ".")
data[key] = f.Value
key = strings.Join([]string{f.Section.ID, f.ID}, ".")
data[key] = f.Value
}
}
var yamlData []byte
yamlData, err = yaml.Marshal(data)
if err != nil {
return "", fmt.Errorf("yaml.Marshal: %v", err)
}

return (string)(yamlData), nil
}

// Convert yaml to map interface and return the requested keys
func (p *provider) GetStringMap(key string) (map[string]interface{}, error) {
yamlData, err := p.GetString(key)
if err != nil {
fmt.Println(err)
return nil, err
}

m := map[string]interface{}{}

if err := yaml.Unmarshal([]byte(yamlData), &m); err != nil {
return nil, err
}

return m, nil
}
3 changes: 3 additions & 0 deletions pkg/stringmapprovider/stringmapprovider.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"github.com/helmfile/vals/pkg/providers/awssecrets"
"github.com/helmfile/vals/pkg/providers/azurekeyvault"
"github.com/helmfile/vals/pkg/providers/gcpsecrets"
"github.com/helmfile/vals/pkg/providers/onepasswordconnect"
"github.com/helmfile/vals/pkg/providers/sops"
"github.com/helmfile/vals/pkg/providers/ssm"
"github.com/helmfile/vals/pkg/providers/vault"
Expand All @@ -34,6 +35,8 @@ func New(l *log.Logger, provider api.StaticConfig) (api.LazyLoadedStringMapProvi
return azurekeyvault.New(provider), nil
case "awskms":
return awskms.New(provider), nil
case "onepasswordconnect":
return onepasswordconnect.New(provider), nil
}

return nil, fmt.Errorf("failed initializing string-map provider from config: %v", provider)
Expand Down
3 changes: 3 additions & 0 deletions pkg/stringprovider/stringprovider.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/helmfile/vals/pkg/providers/gcpsecrets"
"github.com/helmfile/vals/pkg/providers/gcs"
"github.com/helmfile/vals/pkg/providers/gitlab"
"github.com/helmfile/vals/pkg/providers/onepasswordconnect"
"github.com/helmfile/vals/pkg/providers/s3"
"github.com/helmfile/vals/pkg/providers/sops"
"github.com/helmfile/vals/pkg/providers/ssm"
Expand Down Expand Up @@ -52,6 +53,8 @@ func New(l *log.Logger, provider api.StaticConfig) (api.LazyLoadedStringProvider
return azurekeyvault.New(provider), nil
case "gitlab":
return gitlab.New(provider), nil
case "onepasswordconnect":
return onepasswordconnect.New(provider), nil
}

return nil, fmt.Errorf("failed initializing string provider from config: %v", provider)
Expand Down
43 changes: 24 additions & 19 deletions vals.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import (
"github.com/helmfile/vals/pkg/providers/gcs"
"github.com/helmfile/vals/pkg/providers/gitlab"
"github.com/helmfile/vals/pkg/providers/googlesheets"
"github.com/helmfile/vals/pkg/providers/onepasswordconnect"
"github.com/helmfile/vals/pkg/providers/s3"
"github.com/helmfile/vals/pkg/providers/sops"
"github.com/helmfile/vals/pkg/providers/ssm"
Expand Down Expand Up @@ -62,25 +63,26 @@ const (
// secret cache size
defaultCacheSize = 512

ProviderVault = "vault"
ProviderS3 = "s3"
ProviderGCS = "gcs"
ProviderGitLab = "gitlab"
ProviderSSM = "awsssm"
ProviderKms = "awskms"
ProviderSecretsManager = "awssecrets"
ProviderSOPS = "sops"
ProviderEcho = "echo"
ProviderFile = "file"
ProviderGCPSecretManager = "gcpsecrets"
ProviderGoogleSheets = "googlesheets"
ProviderTFState = "tfstate"
ProviderTFStateGS = "tfstategs"
ProviderTFStateS3 = "tfstates3"
ProviderTFStateAzureRM = "tfstateazurerm"
ProviderTFStateRemote = "tfstateremote"
ProviderAzureKeyVault = "azurekeyvault"
ProviderEnvSubst = "envsubst"
ProviderVault = "vault"
ProviderS3 = "s3"
ProviderGCS = "gcs"
ProviderGitLab = "gitlab"
ProviderSSM = "awsssm"
ProviderKms = "awskms"
ProviderSecretsManager = "awssecrets"
ProviderSOPS = "sops"
ProviderEcho = "echo"
ProviderFile = "file"
ProviderGCPSecretManager = "gcpsecrets"
ProviderGoogleSheets = "googlesheets"
ProviderTFState = "tfstate"
ProviderTFStateGS = "tfstategs"
ProviderTFStateS3 = "tfstates3"
ProviderTFStateAzureRM = "tfstateazurerm"
ProviderTFStateRemote = "tfstateremote"
ProviderAzureKeyVault = "azurekeyvault"
ProviderEnvSubst = "envsubst"
ProviderOnePasswordConnect = "onepasswordconnect"
)

var (
Expand Down Expand Up @@ -228,6 +230,9 @@ func (r *Runtime) prepare() (*expansion.ExpandRegexMatch, error) {
case ProviderEnvSubst:
p := envsubst.New(conf)
return p, nil
case ProviderOnePasswordConnect:
p := onepasswordconnect.New(conf)
return p, nil
}
return nil, fmt.Errorf("no provider registered for scheme %q", scheme)
}
Expand Down
59 changes: 59 additions & 0 deletions vals_onepasswordconnect_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package vals

import (
"fmt"
"testing"

"github.com/google/go-cmp/cmp"
)

func TestValues_OnePasswordConnect_EvalTemplate(t *testing.T) {
// TODO
// create vault and item for testing
// op vault create vals-test
// op item create --vault vals-test --title=vals-test [email protected] password=secret --category=login

// Pre-requisite:
// Setup 1Password connect service with access to `vals-test` vault: https://developer.1password.com/docs/connect/

// set up service principal credentials in the environment:
// "OP_CONNECT_TOKEN": "...",
// "OP_CONNECT_HOST": "...",

type testcase struct {
template map[string]interface{}
expected map[string]interface{}
}
vaultLabel := "vals-test"
itemLabel := "vals-test"

testcases := []testcase{
{
template: map[string]interface{}{
"foo": "FOO",
"username": fmt.Sprintf("ref+onepasswordconnect://%s/%s#/username", vaultLabel, itemLabel),
"password": fmt.Sprintf("ref+onepasswordconnect://%s/%s#/password", vaultLabel, itemLabel),
},
expected: map[string]interface{}{
"foo": "FOO",
"username": "[email protected]",
"password": "secret",
},
},
}

for i := range testcases {
tc := testcases[i]
t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
vals, err := Eval(tc.template)
if err != nil {
t.Fatalf("%v", err)
}

diff := cmp.Diff(tc.expected, vals)
if diff != "" {
t.Errorf("unxpected diff: %s", diff)
}
})
}
}

0 comments on commit 904d391

Please sign in to comment.