Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[init-database] - Issues to connect to the database when workload is meshed using Istio #1838

Open
3 tasks done
tchellomello opened this issue Apr 22, 2024 · 4 comments · May be fixed by #1880
Open
3 tasks done

[init-database] - Issues to connect to the database when workload is meshed using Istio #1838

tchellomello opened this issue Apr 22, 2024 · 4 comments · May be fixed by #1880
Labels
community needs_triage

Comments

@tchellomello
Copy link
Contributor

tchellomello commented Apr 22, 2024 •
edited
Loading

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

With the feature introduced by the ffba1b4, a InitContainer named init-database was introduced to validate the database status by calling the wait-for-migrations script. I really liked this approach though, so we have a dedicated task for it.

Therefore, if the workload/namespace is meshed with Istio, the initContainer requires an special UID to be captured by the Istio, otherwise the initContainer will not be able to reach out to the database. Note that this issue does not happen with a regular pod, as the sidecar will get injected as expected, whereas on the initContainer there is no sidecar yet.

To fix the issue, we need to expose to this deployment the ability to override the securityContext to runAsUser.

image

To address this issue, I stopped the operator (to avoid reconciliation) and manually patch the awx-task deployment which then works as expected.

❯ kubectl get deploy awx-toca-task -o json | jq '.spec.template.spec.initContainers.[0].securityContext'
{
  "runAsUser": 1337
}

❯ kubectl images -n awx
[Summary]: 1 namespaces, 3 pods, 15 containers and 4 different images
+---------------------------------+-------------------------+--------------------------------+
|               Pod               |        Container        |             Image              |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-migration-24.2.0-kg4m6 | migration-job           | quay.io/ansible/awx:24.2.0     |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-task-5bdc6ff596-xsj4z  | redis                   | docker.io/redis:7              |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-task           | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-ee             | quay.io/ansible/awx-ee:24.2.0  |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-rsyslog        | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | istio-proxy             | docker.io/istio/proxyv2:1.21.1 |
+                                 +-------------------------+                                +
|                                 | (init) istio-validation |                                |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init-database    | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init-receptor    | quay.io/ansible/awx-ee:24.2.0  |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-web-6f4657cbfb-gzhz5   | redis                   | docker.io/redis:7              |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-web            | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+                                +
|                                 | awx-toca-rsyslog        |                                |
+                                 +-------------------------+--------------------------------+
|                                 | istio-proxy             | docker.io/istio/proxyv2:1.21.1 |
+                                 +-------------------------+                                +
|                                 | (init) istio-validation |                                |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init             | quay.io/ansible/awx-ee:24.2.0  |
+---------------------------------+-------------------------+--------------------------------+

❯ kubectl get pods -n awx
NAME                              READY   STATUS      RESTARTS   AGE
awx-toca-migration-24.2.0-kg4m6   0/1     Completed   0          61m
awx-toca-task-5bdc6ff596-xsj4z    5/5     Running     0          50m
awx-toca-web-6f4657cbfb-gzhz5     4/4     Running     0          53m

AWX Operator version

2.15.0

AWX version

24.2.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

1.29

Modifications

yes

Steps to reproduce

  1. Mesh the namespace using Istio
  2. Deploy workload

Expected results

Work as expect

Actual results

Fails to connect to the database due to initContainer being unable to reach out to the network.

Additional information

N/A

Operator Logs

N/A

Tasks
Beta Give feedback
No tasks being tracked yet.
@RaceFPV
Copy link

RaceFPV commented May 17, 2024

+1 just ran into this same issue today, task never comes online due to init containers firing before the istio sidecar is online.

@RaceFPV RaceFPV linked a pull request May 28, 2024 that will close this issue
Open
@mick1627
Copy link

mick1627 commented Jun 3, 2024

Adding support for task_label to set sidecar.istio.io/inject to false could be a solution.
There is a helm variable task_annotations for annotations but using annotation is deprecated https://istio.io/latest/docs/reference/config/annotations/#SidecarInject

@RaceFPV
Copy link

RaceFPV commented Jun 3, 2024

I don't think thats a reasonable solution as our end goal here is to run istio successfully while using awx-operator, which we can do now we just have to pause controller enforcement to do so until a proper patch is released.

@maarous
Copy link

maarous commented Sep 11, 2024

A more proper solution is to create inject.istio.io/templates in istiod that allows the injection of whatever you need in awx-task or any other object. (so you can inject securityContext even with controller enforcement)
And than use task_label to add the new custom label template to force the injection in awx-task pod.

https://github.com/istio/istio/blob/master/manifests/charts/istio-control/istio-discovery/values.yaml#L137-L162
https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#custom-templates-experimental

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community needs_triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

DRAFT: initial runasuser testing for istio support, fixes #1838 RaceFPV/awx-operator
4 participants
@tchellomello @RaceFPV @mick1627 @maarous