Skip to content

Commit

Permalink
Merge pull request #1557 from splunk/repo-sync
Browse files Browse the repository at this point in the history
Pulling refs/heads/main into main
  • Loading branch information
aurbiztondo-splunk authored Oct 1, 2024
2 parents c31f25b + 8291a98 commit d1b3e21
Show file tree
Hide file tree
Showing 5 changed files with 106 additions and 4 deletions.
1 change: 1 addition & 0 deletions _includes/gdi/otel-receivers-table.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
* :ref:`apache-receiver`
* :ref:`apache-spark-receiver`
* :ref:`azureeventhub-receiver`
* :ref:`carbon-receiver`
* :ref:`cloudfoundry-receiver`
Expand Down
8 changes: 4 additions & 4 deletions gdi/monitors-databases/apache-spark.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ Apache Spark
============

.. meta::
:description: Use this Splunk Observability Cloud integration for the Apache Sparck clusters monitor. See benefits, install, configuration, and metrics
:description: Use this Splunk Observability Cloud integration for the Apache Spark clusters monitor. See benefits, install, configuration, and metrics

The Splunk Distribution of OpenTelemetry Collector uses the Smart Agent receiver with the
Apache Spark monitor type to monitor Apache Spark clusters. It does not
support fetching metrics from Spark Structured Streaming.
.. note:: If you're using the Splunk Distribution of the OpenTelemetry Collector and want to collect Apache Spark cluster metrics, use the native OTel component :ref:`apache-spark-receiver`.

The Splunk Distribution of the OpenTelemetry Collector uses the Smart Agent receiver with the Apache Spark monitor type to monitor Apache Spark clusters. It does not support fetching metrics from Spark Structured Streaming.

For the following cluster modes, the integration only supports HTTP
endpoints:
Expand Down
3 changes: 3 additions & 0 deletions gdi/opentelemetry/components.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ The Splunk Distribution of the OpenTelemetry Collector includes and supports the
* - :ref:`apache-receiver` (``apache``)
- Fetches stats from a Apache Web Server.
- Metrics
* - :ref:`apache-spark-receiver` (``apachespark``)
- Fetches metrics for an Apache Spark cluster through the Apache Spark REST API.
- Metrics
* - :ref:`azureeventhub-receiver` (``azureeventhub``)
- Pulls logs from an Azure event hub.
- Logs
Expand Down
1 change: 1 addition & 0 deletions gdi/opentelemetry/components/a-components-receivers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Collector components: Receivers
:hidden:

apache-receiver
apache-spark-receiver
azureeventhub-receiver
carbon-receiver
cloudfoundry-receiver
Expand Down
97 changes: 97 additions & 0 deletions gdi/opentelemetry/components/apache-spark-receiver.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
.. _apache-spark-receiver:

*******************************
Apache Spark receiver
*******************************

.. meta::
:description: The Apache Spark receiver fetches metrics for an Apache Spark cluster through the Apache Spark REST API.

The Apache Spark receiver monitors Apache Spark clusters and the applications running on them through the collection of performance metrics like memory utilization, CPU utilization, shuffle operations, and more. The supported pipeline type is ``metrics``. See :ref:`otel-data-processing` for more information.

.. note:: Out-of-the-box dashboards and navigators aren't supported for the Apache Spark receiver yet, but are planned for a future release.

The receiver retrieves metrics through the Apache Spark REST API using the following endpoints: ``/metrics/json``, ``/api/v1/applications/[app-id]/stages``, ``/api/v1/applications/[app-id]/executors``, and ``/api/v1/applications/[app-id]/jobs endpoints``.

Prerequisites
======================

This receiver supports Apache Spark versions 3.3.2 or higher.

Get started
======================

Follow these steps to configure and activate the component:

1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:

- :ref:`otel-install-linux`
- :ref:`otel-install-windows`
- :ref:`otel-install-k8s`

2. Configure the receiver as described in the next section.
3. Restart the Collector.

Sample configuration
--------------------------------

To activate the Apache Spark receiver, add ``apachespark`` to the ``receivers`` section of your configuration file:

.. code-block:: yaml
receivers:
apachespark:
collection_interval: 60s
endpoint: http://localhost:4040
application_names:
- PythonStatusAPIDemo
- PythonLR
To complete the configuration, include the receiver in the ``metrics`` pipeline of the ``service`` section of your configuration file:

.. code:: yaml
service:
pipelines:
metrics:
receivers: [apachespark]
Configuration options
-----------------------

The following settings are optional:

* ``collection_interval``. ``60s`` by default. Sets the interval this receiver collects metrics on.

* This value must be a string readable by Golang's ``time.ParseDuration``. Learn more at Go's official documentation :new-page:`ParseDuration function <https://pkg.go.dev/time#ParseDuration>`.

* Valid time units are ``ns``, ``us`` (or ``µs``), ``ms``, ``s``, ``m``, ``h``.

* .. include:: /_includes/gdi/collector-settings-initialdelay.rst

* ``endpoint``. ``http://localhost:4040`` by default. Apache Spark endpoint to connect to in the form of ``[http][://]{host}[:{port}]``.

* ``application_names``. An array of Spark application names for which metrics are collected from. If no application names are specified, metrics are collected for all Spark applications running on the cluster at the specified endpoint.

Settings
======================

The full list of settings exposed for this receiver are documented in the :new-page:`Apache Spark receiver config repo <https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/apachesparkreceiver/config.go>` in GitHub.

Metrics
======================

The following metrics, resource attributes, and attributes are available.

.. note:: The SignalFx exporter excludes some available metrics by default. Learn more about default metric filters in :ref:`list-excluded-metrics`.

.. raw:: html

<div class="metrics-component" category="included" url="https://raw.githubusercontent.com/splunk/collector-config-tools/main/metric-metadata/apachesparkreceiver.yaml"></div>

.. include:: /_includes/activate-deactivate-native-metrics.rst

Troubleshooting
======================

.. include:: /_includes/troubleshooting-components.rst

0 comments on commit d1b3e21

Please sign in to comment.