decision in the matter of cloudmon/stackmon approach #576
Conversation
Signed-off-by: Piotr Bigos <[email protected]>
piobig2871
changed the title
Signed-off-by: Piotr Bigos <[email protected]>
Signed-off-by: Piotr Bigos <[email protected]>
|
|
||
| ## Decision | ||
|
|
||
| By opting for Gherkin test scenarios with Python mapping API calls to OpenStack, we aim to address the |
There was a problem hiding this comment.
we could link to Gherkin here, maybe, for people not familiar with it (myself included)?
I believe this should be an official source? https://cucumber.io/docs/gherkin/
There was a problem hiding this comment.
@artificial-intelligence this is a very good idea. I have not thought about that, thanks for pointing it out. I can also write a sentence or two about behave framework (https://behave.readthedocs.io/en/latest/) that we are using and link the documentation there. Would that be helpful as well?
There was a problem hiding this comment.
Yes that would be nice, I'm only lightly familiar with the topic (I think I read some articles about it some years ago).
Signed-off-by: Piotr Bigos <[email protected]>
Signed-off-by: Piotr Bigos <[email protected]>
| @@ -0,0 +1,43 @@ | |||
| --- | |||
| title: Evaluate current state of CloudMon | |||
There was a problem hiding this comment.
Maybe a better fitting title for the SCS DR would be "Evaluating the deployment of CloudMon in SCS infrastructure"?
| ## Motivation | ||
|
|
||
| One such solution, the [Cloudmon](https://stackmon.org/) project, presents challenges that may hinder its | ||
| suitability for organizations seeking efficient and reliable monitoring capabilities. This introduction |
There was a problem hiding this comment.
Should we list the challenges? The tool itself does good work - for infrastructures of a bigger scale like OTC. We need to compare it to our use case to put the challenges into context
| ## Decision | ||
|
|
||
| By opting for [Gherkin](https://cucumber.io/docs/gherkin/) test scenarios with Python mapping API calls to OpenStack, | ||
| we aim to address the complexities and shortcomings of the Cloudmon project while ensuring our monitoring and testing |
There was a problem hiding this comment.
we aim to address the complexities and shortcomings of the Cloudmon project
See comment above - what do these complexities and shortcomings consist of?
Signed-off-by: Piotr Bigos <[email protected]>
…kmon approach Signed-off-by: Piotr Bigos <[email protected]>
Co-authored-by: Matej Feder <[email protected]> Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr Bigos <[email protected]>
| against utilizing the Cloudmon project and instead embraces a more streamlined and effective approach involving Gherkin | ||
| test scenarios and mapping Python API calls to interact with OpenStack resources. By addressing the complexities and | ||
| shortcomings of the Cloudmon project, SCS organization aims to adopt a monitoring solution that not only meets but | ||
| exceeds our requirements for simplicity, reliability, and ease of use. |
There was a problem hiding this comment.
Honestly this phrasing is more then subjective. After dozens of hours spend explaining the documentation, setup scenarios, testing capabilities, workshops this statement is very offensive. SCS organization entered the evaluation and asked for multiple workshops to understand the tooling and offered help in improving the documentation. There was absolutely no output.
What you offer as an alternative is another "one of thousands" of namely universal testing frameworks that were not designed to monitor cloud environment and are not really dealing with measuring low level performance (of the API itself).
|
|
||
| Our approach was to base on a technical concept. This document serves as a proposal, with the final decision | ||
| subject to discussion with SCS team members. We propose a behavior-driven system based on solutions using Java framework | ||
| Cucumber, utilizing the Gherkin domain-specific language for defining executable specifications. This approach ensures |
There was a problem hiding this comment.
So what you offer is clearly controversial to the objective you stated above: simplicity. Taking Java and a widely unknown framework with even less known DSL is going to harm simplicity and understand-ability of the solution. It brings enormous amount of efforts required by operators to learn those things and be able to extend it.
There was a problem hiding this comment.
main idea of stackmon is in using ansible as a test scenario that literally anybody is able to read/understand/replay locally/extend (endlessly)
| Cucumber, utilizing the Gherkin domain-specific language for defining executable specifications. This approach ensures | ||
| clear, human-readable test behavior definitions, facilitating participation from both developers and non-technical | ||
| contributors. Considering the team's proficiency in Python, the language's simplicity and clarity, alignment with | ||
| OpenStack's ecosystem, and the robust support from the Python community, it's evident that Python presents a superior |
There was a problem hiding this comment.
I do not get how you come from Java mentioned above to the python
| ## Challanges | ||
|
|
||
| During our assessment of the Cloudmon/Stackmon project, we encountered significant challenges related to documentation, | ||
| particularly regarding lack of examples for configuration setups and usage guidelines. The lack of comprehensive |
There was a problem hiding this comment.
This doesn't sound fair. You were offered dedicated session, pointed to the relevant documentation (and as mentioned above it was communicated towards the CloudMon team that taking those workshops you will help improving the documentation), pointed to detailed explanation of the configuration, pointed to the live production configuration. If production configuration is not considered as "lack of examples" I do not know what to say more. If production testing scenarios of a very big OpenStack based public cloud with much more services then vanila SCS comes with are not sufficient it shows me the evaluation was performed not very detailed.
|
|
||
| ## Decision | ||
|
|
||
| By opting for [Gherkin](https://cucumber.io/docs/gherkin/) test scenarios with Python mapping API calls to OpenStack, |
There was a problem hiding this comment.
OpenTelekomCloud is using alternative framework (Robot Framework) for doing system acceptance testing. This comes with a huge effort (which you do not explicitly mention here, but is read between the lines) of developing this "python mapping API calls to OpenStack". This is a huge and complex work that can be avoided by relying on existing tooling for OpenStack. And btw, RobotFramework (at OTC) and behavior driven testing is great in testing, but is absolutely unsuitable for health and performance monitoring
|
As a general comment on how to approach system design decisions: From this text, it is not clear to me, why exactly is cucumber superior. I don't think it's a matter of the implementation language. You should be able to show concrete examples what cucumber actually does better than cloudmon, so you could convince me, as a reader. I would also expect a proposal to use new technology to list the shortcomings of said technology as well, because in my experience every technological decision has, of course, up- and downsides. Presenting only upsides is showing me, that we either are not aware of the downsides, which is dangerous, because it means we will learn of the downsides much later, and they might be even worse then the previous solution. In rare cases there are no downsides and you have a straight up better technical solution, which should be spelled out as well in the text, showing that at least it was thought about if there are downsides to the new solution. If I don't see such things, it's a red flag to me, as it's shows a clear bias in the decision, because when we are honest, of course we all, as humans, like one technology more than the other for a variety of subjective reasons like prior familiarity, trendiness, marketing and last but not least, technical merit. I think though it's important that we know about our own biases and at least make an attempt to counter them by an honest technical analysis of the up- and downsides of a given software. So either not enough research was done, to be able to list up- and downsides of all solutions. Both are understandable things that happen to all of us, still I think if we want to make better decisions we should try to avoid that. Thanks for reading, if you made it this far. |
| In the fast-paced environment of modern cloud computing, ensuring the reliability and performance | ||
| of cloud infrastructure is paramount for organizations. Effective monitoring and testing framework |
There was a problem hiding this comment.
Unfortunately I have many problems with this.
The complete text lacks a clearer definition of how e.g. Cloudmon "monitors" "reliability and performance" - what that even means in this context, because the context is unclear.
First, it only talks about "Cloud Computing", from guessing and stuff I know about Cloudmon I think this is about the IaaS Layer of a Cloud, but it isn't even mentioned anywhere.
I think this should be spelled out.
Second, neither "reliability" or "performance" or "monitoring" are really defined here, and used intermixed, which poses some problems, as you can see when looking at my next questions.
|
|
||
| In the fast-paced environment of modern cloud computing, ensuring the reliability and performance | ||
| of cloud infrastructure is paramount for organizations. Effective monitoring and testing framework | ||
| are essential tools in this endeavor, enabling proactive identification and resolution of issues before |
There was a problem hiding this comment.
e.g. most monitoring doesn't really allow proactive resolution of issues.
Monitoring is just analysing live data after the fact. e.g. you can see an API is slow in monitoring, or you can see a disk filling up. But all the activities monitoring allows a DevOps Team to do are reactive. You notice a problem in monitoring, then you react.
A testing framework, on the other hand, can of course be used to detect problems e.g. before deployment.
| Our approach was to base on a technical concept. This document serves as a proposal, with the final decision | ||
| subject to discussion with SCS team members. We propose a behavior-driven system based on solutions using Java framework |
There was a problem hiding this comment.
There are many different testing frameworks and testing strategies, which are not really differentiated in this text. You propose a behavior-driven test concept using cucumber, but somehow the document fails to mention any up- or downsides of behavior driven testing or of this specific framework.
There is no reasoning I have read in this text why behavior driven testing is superior to the cloudmon approach.
And I don't doubt that there might be advantages to it, but those need to be clearly spelled out somewhere.
This text also talks about a "technical concept", where can I see that? It isn't linked here, or included. If it is the basis for the decision, it surely should be included?
I only know both technologies superficially, for the record, that is cloudmon and cucumber.
|
What's the status here? |
No description provided.