Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Added es and cerebro services, rudimentary unstract-metrics lib added #473

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

chandrasekharan-zipstack
Copy link
Contributor

@chandrasekharan-zipstack chandrasekharan-zipstack commented Jul 10, 2024

What

  • Added es and cerebro services as part of the optional profile unstract-metrics
  • Added a simple unstract-metrics library that wraps elasticsearch_dsl to add and query metrics
  • Added some pytests for the new library methods

NOTE: Understood now that index creation need not happen through the regular code flow so some of the changes here would be invalid, will continue my development assuming that the index is already created and focus on exposing methods to aid easier collection of metrics for each sub-process / process type

How

  • Made use of elasticsearch_dsl which is a high level lib that helps interact with es

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

  • No, new changes added under optional profile and not used anywhere

Env Config

  • Added below envs to new library
ES_URL=http://localhost:9200
ES_CLOUD_ID=
ES_API_KEY=
COLLECT_UNSTRACT_METRICS=False

Relevant Docs

Dependencies Versions

  • elasticsearch-dsl~=8.14.0 but not included in any existing services yet

Notes on Testing

  • Tried setting it up locally and able to view the index and its mapping after inserting the metrics

Screenshots

  1. Docker compose stats with the new services
    image
  2. Able to access the es instance through cerebro
    image
  3. Added some unit tests
    image

Checklist

I have read and understood the Contribution Guidelines.

```


## Using Cerebro: An Elasticsearch Web Admin tool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chandrasekharan-zipstack Create a confluence page for Cerebro no need to include here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaseemjaskp Wondering if we should mention it in README so that users can install for admin tasks if they wish? Are there any other recommended tools?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hari-kuriakose kibana maybe?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hari-kuriakose @chandrasekharan-zipstack Elasicsearch official one is Kibana. we can mention that in readme if required. https://github.com/lmenezes/cerebro is not up to date with official ES.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaseemjaskp if cerebro is not upto date with official ES do you think we should also try kibana?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chandrasekharan-zipstack try out Kibana. but no need into include in docker-compose file

Copy link

@chandrasekharan-zipstack chandrasekharan-zipstack changed the title feat: Added es and cerebro services, rudimentary unstract-metrics lib added WIP: Added es and cerebro services, rudimentary unstract-metrics lib added Jul 18, 2024
Copy link

sonarqubecloud bot commented Sep 3, 2024

Copy link

sonarqubecloud bot commented Nov 3, 2024

Copy link
Contributor

github-actions bot commented Nov 3, 2024

filepath function $$\textcolor{#23d18b}{\tt{passed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_logs}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_client\_init}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_run\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{9}}$$ $$\textcolor{#23d18b}{\tt{9}}$$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants