[HWORKS-284] Documentation for exporting logs #142
Conversation
|
|
||
| ## Introduction | ||
| Hopsworks collects services and applications logs to [Logstash](https://www.elastic.co/logstash/) which then forwards them to OpenSearch for indexing. | ||
| Often organizations already have logging systems in place so streaming Hopsworks logs is necessary. |
There was a problem hiding this comment.
so exporting HOpsworks logs is necessary?
| Often organizations already have logging systems in place so streaming Hopsworks logs is necessary. | ||
|
|
||
| ## Prerequisites | ||
| To configure Logstash streaming logs outside of Hopsworks you will need SSH access to the cluster (Logstash node). Also, depending on the target system you might |
There was a problem hiding this comment.
To configure Logstash to stream logs.
| need authentication tokens or opening firewall rules. | ||
|
|
||
| ## Export logs | ||
| Logstash is a well established log collection service with many output [plugins](https://www.elastic.co/guide/en/logstash/7.17/output-plugins.html) available. |
There was a problem hiding this comment.
Logstash is not a log collection service, it's just a processing pipeline.
| Logstash process logs in *pipelines* where each pipeline is responsible for a logical group of logs. In Hopsworks we have multiple pipelines and their configuration files are under `/srv/hops/logstash/config` | ||
|
|
||
| ### Export services logs | ||
| To stream various services' logs outside of Hopsworks you will need to **create another pipeline** similar to `services`. |
There was a problem hiding this comment.
similar to the services pipeline.
| !!! note | ||
| Take a note of the pipeline address as we will use it in Step 2 | ||
|
|
||
| At the end of the file is the `output` section which currently forwards them to OpenSearch. Replace the output section with a sample block such as |
There was a problem hiding this comment.
forwards the logs to OpenSearch
| pipeline.batch.size: 50 | ||
| ``` | ||
|
|
||
| **Instruct** the services pipeline to push logs also in the newly created pipeline by appending to `services-intake` for example: |
There was a problem hiding this comment.
also to the newly created pipeline
|
|
||
|
|
||
| ### Export Spark logs | ||
| To stream applications' logs to another system the Steps are fairly similar to exporting services logs but need some additional configuration. |
|
|
||
|
|
||
| ### Export Spark logs | ||
| To stream applications' logs to another system the Steps are fairly similar to exporting services logs but need some additional configuration. |
There was a problem hiding this comment.
I would also specify Spark applications logs so users don't confuse with Hopsworks application.
| Finally you should restart Logstash `sudo systemctl restart logstash` | ||
|
|
||
| ## Conclusion | ||
| It is not easy to write a guide for a task that can be achieved in many different ways but in this guide we gave solid |
There was a problem hiding this comment.
Users don't care that it's not easy. Just refer them to the different plugins configuration to understand how they need to configure their new pipeline to send data wherever they need to.
https://hopsworks.atlassian.net/browse/HWORKS-284