adds openshift project key preflight check

[stitakis]

fixes #589

[renedupont]

Looks in general good to me. Just one question: Was the oc client approach easier than using the openshift API or how did you decide that?

[stitakis]

@renedupont thanks a lot for help!

[clemensutschig]

I still believe this is WRONG! ... we tried to decouple OCP as much as possible .. and now we have next to jenkins a direct connection. WHY? @stitakis @michaelsauter

what is the usecase for this?

[michaelsauter]

I'd suggest we revert this PR, and then discuss what problem it is trying to solve, and what approach should be taken to solve the issue. #589 hints at some issues with E2E tests. @stitakis can you explain what those are? If this PR is "just" solving E2E test issues, it is a huge pill to swallow ...

[stitakis]

@clemensutschig @michaelsauter what, what is wrong here? :-) Can you please shed some light on the decoupling on purpose of prov app from OCP. Asking because obviously I wasn't aware of this. From my point of view, I don't see any disavantage on consuming the openshift api to verify if a project key exists (it is not for E2E tests). The ocp api is stable and the dependency in one direction, from ProvApp to ocp api. I'd like understand better the architecture and decisions taken in the past. Happy to discuss on the phone.

[michaelsauter]

So if it is not for E2E tests, what is it then for?

By adding "OpenShift knowledge" to the prov app you are increasing the scope of the prov app. This should be done with great care, as it is already a monster.

The prov app does not create OpenShift projects at the moment, but delegates this to Jenkins. The process is completely asynchronous - the prov app never checks the result of the Jenkins Pipeline. I don't think there is anything wrong with triggering the Jenkins Pipeline even if the OpenShift project exists. I'm not sure what its behaviour is right now, but there are just two options: it should either fail because the project already exists (which is OK) or succeed because the project already exists (which is also OK).

Allowing to re-run and fill in the gaps even has benefits: Assume one part of the provisioning breaks (e.g. Jira/Bitbucket exists, but OpenShift doesn't), then one can just run the job again. And vice-versa.

[stitakis]

Thanks a lot for the quick feedback!

I understand that you are worried and take great care of the ods architecture. I appreciate this a lot and make me feel proud to be part of this great team. :-)

Imho in terms of implementation the benefits outweights the cost in this case. I'll explain why...

...this change extends the preflight checks to verify the project key in openshift. It is a plain preflight check.
In an enterprise environment correcting errors cost a lot of effort, time, energy and trust in IT products.
That's why avoiding errors is very important and the reason why preflight checks were created. To avoid fixing errors of incomplete provisioned projects.
Imagine something goes wrong in SNOW for whatever reason (wrong request, bug, etc...). The preflight checks are there to verify that all precondition are met and to give early detailed feedback to the consumer. This helps operations to know as early as possible where in the chain something was wrong.
Make sense?

[clemensutschig]

but why do you want a pre-flight check on OCP - if the project exists, the jenkins job will start and fail - done ... what more do we need?

[stitakis]

The ProvApp does not have the capability to get/query the job status. It triggers a fire-and-forget request to Jenkins over the webhook proxy. It cannot return job status error to the API consumer.

Therefore, this new preflight check is there to verify the project key and give early detailed error to the API consumer. If the project key exists in OCP the ProvApp will return an error to the API caller.

[clemensutschig]

@stitakis - but what is it needed for?! ...

[stitakis]

@clemensutschig to verify that the project key does not exists before triggering the provisioning... did I understand your last question? I guess no... please give me a quick call, I would be happy to chat with you :-)

[clemensutschig]

@clemensutschig to verify that the project key does not exists before triggering the provisioning... did I understand your last question? I guess no... please give me a quick call, I would be happy to chat with you :-)

@stitakis - I just dont get why we need a check for this? - the jenkins job will fail - (after successfull start) - and the (internal) project will be saved. so all good. where is the issue here? If it's purely for testing, then I recommend to check that within the test - and delete the project

[segator]

is very commons provapp creates projects partially wrong(maybe were able to create jira project and then fails on bitbucket or in OCP), the correct thing here is check everything before start creating. I agree with you that not a good idea that provapp knows about ocp, maybe then webhookproxy(that is the component that provapp is aware) should have the option to ask for if project exists.

[clemensutschig]

is very commons provapp creates projects partially wrong(maybe were able to create jira project and then fails on bitbucket or in OCP),

I am confused - there is cleanup etc in place - and by now its pretty resilient by now .. I rather fix this - instead introduce the above.,

[segator]

i don't trust this cleanup, we disabled it, it destroyed 3 productive projects in the past and caused a lot of problems...
a preflight check is more robust and secure, in case anyway the preflight check fails I prefer to fix the project by hand, automatic destroying operations are dangerous.

[clemensutschig]

i don't trust this cleanup, we disabled it, it destroyed 3 productive projects in the past and caused a lot of problems...
a preflight check is more robust and secure, in case anyway the preflight check fails I prefer to fix the project by hand, automatic destroying operations are dangerous.

there was ONE bug - that we fixed - in a cornercase ... (with an existing project) - that should never have happened. can you please elaborate on whats not working?

[segator]

I'm not saying it's not working now, is that I'm not feel confident using cleanup feature, is to risky and I think a better approach is to have a preflight check.

[clemensutschig]

I'm not saying it's not working now, is that I'm not feel confident using cleanup feature, is to risky and I think a better approach is to have a preflight check.

@stitakis / @segator - then please spend the time on writing (more) solid testcases around it :)

[stitakis]

@clemensutschig @segator yes, we need to spend more time in testing the delete project logic.
But I still don't understand why ProvApp should not query OCP (in this case getting project keys).
Of course ProvApp should never trigger in OCP an operation that changes its state. For that we have the Jenkins and jenkins-shared-lib.
But I still don't see an issue in querying OCP for information. Can you please explain why ProvApp shouldn't get information from OCP?

[michaelsauter]

What I'm not really getting is:

is very commons provapp creates projects partially wrong(maybe were able to create jira project and then fails on bitbucket or in OCP), the correct thing here is check everything before start creating

So if something goes wrong, wouldn't it be nice if I can just re-run the job and it fills in what is missing? I don't think we need cleanup here.

I think the best way forward is to have all "actions" triggered by the prov app to be idempotent, meaning I can run the job as often as I want. Each job should check if what it tries to create already exists, and skip.

Why do we need cleanup for the "broken provision" use case?

[michaelsauter]

@stitakis Let's first agree on why we need this query in the first place. I don't get that yet.

[stitakis]

@michaelsauter, it is a preflight check. The why is explained above. Happy to talk with you about it...should I call you?

[clemensutschig]

there is two usecaes

1. people want to cleanup a project all together - this is the functionality also exposed via API
2. in case something goes wrong (clean up mess that was created) - which internally uses functionality from (1)