Issues with the default libraries of OPA Gatekeeper

[vruask]

Hey all, I'm trying to configure the constraint template taken from https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/host-namespaces/template.yaml along with its constraint taken from https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/host-namespaces/samples/psp-host-namespace/constraint.yaml

Procedure followed:

1. Used kubectl apply -f template.yaml to configure the template. Got the response constrainttemplate.templates.gatekeeper.sh/k8spsphostnamespace created
2. Used kubectl apply -f constraint.yaml to configure the constraint. Got the response k8spsphostnamespace.constraints.gatekeeper.sh/psp-host-namespace created
3. Tried installing the below Pod using kubectl apply -f samplepod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-server
  labels:
    app: httpd-host-namespace
spec:
  hostPID: true
  hostIPC: true
  containers:
  - name: httpd
    image: httpd

Expected behavior: The pod is prevented from being created, because the hostPID and hostIPC = true. The pod
should be created only if the value is false.
Actual behavior: Pod gets created: pod/httpd-server created. The same behavior occurs, if hostPID and hostIPC = false. The logs are attached for both audit and controller-manager pods.
audit.log
controller-man.log

Not sure what I'm doing wrong here, since its basically the same template. Looking forward to your response!

[maxsmythe]

Can you run kubectl get -f constraint.yaml -oyaml to see if there are audit results reported on the constraint and if the webhook pods are reporting it as ingested?

If there are audit results, then we know the problem is with the webhook specifically, not with the constraint or template.

If the webhook isn't functioning and the webhook pods look healthy, the usual explanation is some kind of networking error, like a firewall.

[maxsmythe]

Hi!

That response shows that the Pod is healthy: it's returning a json object that says "I don't understand the format of your input", which is expected, since we aren't sending any input to the HTTP endpoint.

This points to the API server being unable to connect to the service for some reason. Usually the answer is that the cluster's networking has a firewall somewhere. GCP private clusters is the only case of this I'm aware of, unfortunately diagnosing specific networking issues for clusters is outside of what we can do.

Here is the link to known issues/fixes for specific cluster providers, in case they apply for you:

https://open-policy-agent.github.io/gatekeeper/website/docs/vendor-specific

[vruask]

Hi @maxsmythe

Thanks for your prompt answers!

I have an update: This issue does NOT occur for the below policies

• https://github.com/open-policy-agent/gatekeeper/blob/master/demo/basic/constraints/all_ns_must_have_gatekeeper.yaml - This policy, when applied, successfully blocks the namespace from getting created, if the gatekeeper label is not present.
• https://github.com/open-policy-agent/gatekeeper/blob/master/demo/agilebank/constraints/owner_must_be_provided.yaml - This policy also gives the error when NS don't have the owner label.

Error from server ([all-must-have-owner] All namespaces must have an `owner` label that points to your company username): error when
creating ".\\disallowed_owner.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [all-must-have-owner] All 
namespaces must have an `owner` label that points to your company username

So far, I see 2 possibilities here (maybe there are more):

• Since the above two policies are working fine, maybe there aren't any networking-related issues (Correct me if I'm wrong here) ?
• Maybe the issue only occurs with certain policies ?

I have also ruled out any firewall-related restrictions on our cluster. Could you please maybe direct me to the correct log location that you mentioned in your previous post, where these networking-related issues can be identified ? BTW, the K8s cluster is on EKS.

Thank you!

[maxsmythe]

Interesting. A successful call to the webhook rules out a firewall issue.

After you've gotten a rejection from the all-must-have-owner constraint, can you try to create that non-compliant pod again and see if it gets rejected? I'm wondering if Gatekeeper was still bootstrapping the first time you tested.

The logs I mentioned would be the Kubernetes API Server logs. I'm not sure how/if those can be accessed on EKS.

[vruask]

Hi @maxsmythe

If you're asking whether the policy is successfully enforced even in subsequent attempts, then yes. I tried to add the non-compliant pod multiple times, but gatekeeper keeps blocking the pod from being created.

Error from server ([all-must-have-owner] All namespaces must have an `owner` label that points to your company username): error when creating ".\\gatekeeper-library\\library\\general\\requiredlabels\\samples\\all-must-have-owner\\example_disallowed.yaml": admission webhook "validation.gatekeeper.sh" denied the request: 
[all-must-have-owner] All namespaces must have an `owner` label that points to your company username

Hope this helps. Thank you!

[maxsmythe]

I'm asking if the following sequence of events happens:

• You kubectl apply -f gatekeeper-library/library/general/requiredlabels/samples/all-must-have-owner/example_disallowed.yaml and receive a rejection message
• After receiving a rejection creating the namespace, run: kubectl apply -f kubectl apply -f samplepod.yaml and let me know if it gets rejected.

My suspicion is maybe you attempted to create the pod before Gatekeeper had ingested the constraint/template. This tests that theory.