feat(admission-controller): Accept URL for custom region customers [SSPROD-32839]

[IgorEulalio]

What this PR does / why we need it:

Checklist

• Title of the PR starts with type and scope, (e.g. feat(agent,node-analyzer,sysdig-deploy):)
• Chart Version bumped for the respective charts
• Variables are documented in the README.md (or README.tpl in some charts)
• Check GithubAction checks (like lint) to avoid merge-check stoppers
• All test files are added in the tests folder of their respective chart and have a "_test" suffix

[github-actions]

PR title does not comply with regex: ^(\w*)(?:\(([\w\$\.\,\-\*\s]*)\))?\:\s?(.*)$!
Check PR guidelines at https://github.com/sysdiglabs/charts/blob/master/README.md#pull-requests

All good!

[AlbertoBarba]

Shouldn't this include be inside the first if?

[IgorEulalio]

This is being validated for the entire define because, in the else scenario, it also should be validated, so instead of validating inside both ifs I'm validating before them.

[AlbertoBarba]

Since .Values.sysdig.apiEndpoint should be used only if custom region has been chosen, i'm not sure why we are still checking it here 🤔

[IgorEulalio]

I don't know if I properly understood your comment, but apiEndpoint isn't only used if custom region has been chosen.

If custom region has been chosen we require either url or apiEndpoint, but for non-custom regions we still provide the option to receive apiEndpoint, then in the last case we fallback to default endpoint from _regions.tpl.

[IgorEulalio]

@AlbertoBarba I'm moving our thread to here so we can take a look at this in the future if needed :)

So, with this code, we achieve the following:

{{/*
Determine Secure endpoint based on provided region or .Values.sysdig.apiEndpoint
*/}}
{{- define "admissionController.apiEndpoint" -}}
    {{ include "chart.validateSysdigApiEndpoint" . }}
    {{- if eq .Values.global.sysdig.region "custom" -}}
        {{- $apiEndpoint := coalesce .Values.sysdig.url .Values.sysdig.apiEndpoint -}}
        {{- required "A valid Sysdig API endpoint (.apiEndpoint or .url) is required for custom region" $apiEndpoint -}}
    {{- else }}
        {{- if .Values.sysdig.apiEndpoint -}}
            {{- printf "%s" .Values.sysdig.apiEndpoint -}}
        {{- else -}}
            {{- if hasKey ((include "sysdig.regions" .) | fromYaml) .Values.global.sysdig.region -}}
                {{- include "sysdig.secureApiEndpoint" . -}}
            {{- else -}}
                {{- required "A valid .Values.sysdig.apiEndpoint or a known region is required" .Values.sysdig.apiEndpoint -}}
            {{- end -}}
        {{- end -}}
    {{- end -}}
{{- end -}}

If the region is custom, allow customers to specify either url or apiEndpoint -> if region is custom, previous behavior only allows customers to specify apiEndpoint, not url.
If the region is not custom, still allow a customer to override SECURE_URL providing apiEndpoint -> same behavior as before. Test here enforcing this behavior.
If the region is not custom and apiEndpoint is not provided, we should receive a region that is present in the map sysdig.regions, otherwise we'll fail saying: pick a valid region or define your own endpoint.
On top of everything, if we specify apiEndpoint, it will be validated against http:// and https:// prefixes, since it is concatenated with https:// in the configmap definition here: That is a breaking change, though if customers have apiEndpoint specified with either http:// or https:// prefixes, AC shouldn't be working 🙂

cc: @airadier

[airadier]

@AlbertoBarba I'm moving our thread to here so we can take a look at this in the future if needed :)

So, with this code, we achieve the following:

{{/*
Determine Secure endpoint based on provided region or .Values.sysdig.apiEndpoint
*/}}
{{- define "admissionController.apiEndpoint" -}}
    {{ include "chart.validateSysdigApiEndpoint" . }}
    {{- if eq .Values.global.sysdig.region "custom" -}}
        {{- $apiEndpoint := coalesce .Values.sysdig.url .Values.sysdig.apiEndpoint -}}
        {{- required "A valid Sysdig API endpoint (.apiEndpoint or .url) is required for custom region" $apiEndpoint -}}
    {{- else }}
        {{- if .Values.sysdig.apiEndpoint -}}
            {{- printf "%s" .Values.sysdig.apiEndpoint -}}
        {{- else -}}
            {{- if hasKey ((include "sysdig.regions" .) | fromYaml) .Values.global.sysdig.region -}}
                {{- include "sysdig.secureApiEndpoint" . -}}
            {{- else -}}
                {{- required "A valid .Values.sysdig.apiEndpoint or a known region is required" .Values.sysdig.apiEndpoint -}}
            {{- end -}}
        {{- end -}}
    {{- end -}}
{{- end -}}

If the region is custom, allow customers to specify either url or apiEndpoint -> if region is custom, previous behavior only allows customers to specify apiEndpoint, not url. If the region is not custom, still allow a customer to override SECURE_URL providing apiEndpoint -> same behavior as before. Test here enforcing this behavior. If the region is not custom and apiEndpoint is not provided, we should receive a region that is present in the map sysdig.regions, otherwise we'll fail saying: pick a valid region or define your own endpoint. On top of everything, if we specify apiEndpoint, it will be validated against http:// and https:// prefixes, since it is concatenated with https:// in the configmap definition here: That is a breaking change, though if customers have apiEndpoint specified with either http:// or https:// prefixes, AC shouldn't be working 🙂

cc: @airadier

Looks good to me. We are keeping the existing behavior, adding additional checks, and solving the problem reported in SSPROD-32839 that wouldn't allow using "url" with "custom" region.

[AlbertoBarba]

@IgorEulalio @airadier My only concern is the fact that customers should now provide a region by design.
If a customer provides a "sysdig region" we auto-populate those URLs/endpoints in every chart.
Otherwise, if the customer specifies "custom" as a region, he'll have to populate each URL/endpoint on his own.
These changes are going to introduce a mixed approach that gives me some doubts.
@aroberts87 @mavimo Any thoughts about it?

[airadier]

@IgorEulalio @airadier My only concern is the fact that customers should now provide a region by design.
If a customer provides a "sysdig region" we auto-populate those URLs/endpoints in every chart.
Otherwise, if the customer specifies "custom" as a region, he'll have to populate each URL/endpoint on his own.
These changes are going to introduce a mixed approach that gives me some doubts.
@aroberts87 @mavimo Any thoughts about it?

@AlbertoBarba isn't this the current and expected behavior?

• If apiEndpoint is provided, it will override the region endpoints
• In AC, we support providing URL for backwards compatibility, although apiEndpoint is supported preferred as it is the standard across other sysdig-deploy components
• If the user provides a region, the apiEndpointis automatically computed, so it is not mandatory
• If the user provides custom as the region (i.e. for On-Prem) then apiEndpoint becomes mandatory

[AlbertoBarba]

• If apiEndpoint is provided, it will override the region endpoints
• In AC, we support providing URL for backwards compatibility, although apiEndpoint is supported preferred as it is the standard across other sysdig-deploy components

@airadier AFAIK when we introduces the common region resolution the flow became:

• If the user specify a valid region, we always use defaults endpoints (specified in https://github.com/sysdiglabs/charts/blob/master/charts/common/templates/_regions.tpl).
• If the user specify custom region, apiEndpoint will be used

While in this PR we'll add the "Set a valid region, but be able to override the endpoint anyway".

I'll wait for @aroberts87 opinion since he was the one who introduced the common region resolution

[airadier]

While in this PR we'll add the "Set a valid region, but be able to override the endpoint anyway".

I'll wait for @aroberts87 opinion since he was the one who introduced the common region resolution

If that's the case, then I agree, let's align the behavior with the rest of the components. So if I understand correctly the main difference is that region will take priority over apiEndpoint, and apiEndpoint should only be considered when region is custom?

[IgorEulalio]

While in this PR we'll add the "Set a valid region, but be able to override the endpoint anyway".

This isn't something that is being introduced in this PR, the only change regarding the entire region definition workflow is that if we define custom region, we expect either url or apiEndpoint, whereas in the old behavior we were expecting only apiEndpoint.

The ability to override the apiEndpoint without using custom region is already there:
https://github.com/sysdiglabs/charts/blob/master/charts/admission-controller/templates/_helpers.tpl#L117

I can see that agent uses the exactly same mechanism to define collectorEndpoint:
https://github.com/sysdiglabs/charts/blob/master/charts/agent/templates/_helpers.tpl#L264

[aroberts87]

Hey all, sorry for the delay. Just returning from holiday. To address the intent/priority of the region and various apiEndpoint flags... The intent of global.sysdig.region was to provide a universal way to configure the API and collector endpoints across all subcharts for SaaS users. If the user had an OnPrem installation they would be able to specify the custom region code and fallback to specifying the API and collector endpoints values for each component/chart. I've done a bit of testing and validated what @IgorEulalio found that it is possible to supply both a valid region code for global.sysdig.region as well as override the value in each subchart as if a region of custom had been provided. In this moment I'm not entirely sure why anyone would want to do this, but I do appreciate @IgorEulalio for recognizing this behavior and ensuring it remains unchanged.

For my own sake, I guess I am still unclear about what this change is attempting to fix and why it's only been discovered now, as the region code introduction is quite old at this point. Am I correct in understanding that we sorta-kinda made a breaking change six months ago (#1118) where we migrated from .Values.url to .Values.apiEndpoint (but let .Values.url continue to work as an undocumented option) and now we're trying to make that key a first-class citizen again? I took a look at the associated ticket and it seems like the main complaint here is that we have existing users who have not migrated from .Values.url to .Values.apiEndpoint and we're trying to engineer around requesting them to migrate their configurations to the new paradigm by reviving a deprecated chart parameter. If that's what we're doing here I'm not sure if this is what we should be doing, but as I said previously, I'm still a bit unclear on why this change is necessary in the first place.

[airadier]

Am I correct in understanding that we sorta-kinda made a breaking change six months ago (#1118) where we migrated from .Values.url to .Values.apiEndpoint (but let .Values.url continue to work as an undocumented option) and now we're trying to make that key a first-class citizen again?

I think so. Not really making it first-class citizen again, but that change, according to the ticket, breaks the backwards configuration when using custom region and the url parameter. This change makes sure that this is still working.

Also @IgorEulalio introduced an additional check to make sure that apiEndpoint doesn't include the redundant https:// prefix, as it is already added by the template or the component if needed.

[IgorEulalio]

TBH is also unclear for me how much impact this is bringing to customers, I definitely agree that this is engineering solution to save customer from a migration.

Though, if we take a look at the change, we are not exactly making it available again, it is already there, customers can today specify only url and that will replace the SECURE_URL value in configmap:

SECURE_URL: "{{ .Values.sysdig.url | default (printf "https://%s" (include "admissionController.apiEndpoint" .)) }}"

So at the end of today we only deprecated it for scenarios where customers are using custom regions, besides removing it from the docs.

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

This PR has been closed due to inactivity.