Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pulling refs/heads/main into main #1473

Merged
merged 23 commits into from
Aug 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
36bc5be
Add initial draft for SLO custom metrics scenario
trangl-splunk Jul 29, 2024
f01f75e
Typo
trangl-splunk Jul 29, 2024
62918b5
Add filters and screenshot
trangl-splunk Jul 29, 2024
d5c4b91
Fix numbered steps
trangl-splunk Jul 29, 2024
a0462b8
Fix table formatting
trangl-splunk Jul 29, 2024
1d3b48a
Fix table formatting
trangl-splunk Jul 29, 2024
b7c366d
Change field names
trangl-splunk Jul 29, 2024
24b36a8
Update text formatting
trangl-splunk Jul 29, 2024
7f44fa8
Fix table error
trangl-splunk Jul 29, 2024
54dc3a7
Update field name formatting
trangl-splunk Jul 29, 2024
56d6d57
Crop screenshot
trangl-splunk Jul 29, 2024
e422d86
Add missing word
trangl-splunk Jul 29, 2024
8c8400e
Fix step numbering
trangl-splunk Jul 29, 2024
d939e00
Resize table columns
trangl-splunk Jul 29, 2024
a091658
Wording
trangl-splunk Jul 29, 2024
6187043
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
b57ee43
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
4779a2d
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
4c4b471
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
31d2eef
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
1eff25f
Update alerts-detectors-notifications/slo/custom-metric-scenario.rst
trangl-splunk Aug 5, 2024
88a34da
Merge pull request #2196 from splunk/trangl-o11ydocs-6351-slo-custom-…
trangl-splunk Aug 5, 2024
cb27d75
Merge branch 'main' into repo-sync
aurbiztondo-splunk Aug 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 14 additions & 14 deletions alerts-detectors-notifications/slo/create-slo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,20 @@ Follow these steps to create an SLO.

* - :strong:`Field name`
- :strong:`Actions`
* - Metric type
* - :guilabel:`Metric type`
- Select :guilabel:`Service & endpoint` from the dropdown menu
* - Environment
* - :guilabel:`Environment`
- Open the dropdown menu and check the boxes for the environments where you want to apply this SLO
* - Service\:\endpoint
* - :guilabel:`Service:endpoint`
- * Search for the service you want to create an SLO for
* (Optional) Add an endpoint for the selected service
* - Indicator type
* - :guilabel:`Indicator type`
- Select either success rate or latency to use as the measurement for your SLO target:

* Request success: Measure the proportion of requests that result in a successful response over the duration of the compliance window

* Request latency: Measure the proportion of requests that load within the specified latency over the duration of the compliance window
* - Filters
* - :guilabel:`Filters`
- Enter any additional dimension names and values you want to apply this SLO to. Alternatively, use the ``NOT`` filter, represented by an exclamation point ( ! ), to exclude any dimension values from this SLO configuration.

To use a custom metric as the system health indicator for your SLI configuration, follow these steps:
Expand All @@ -55,11 +55,11 @@ Follow these steps to create an SLO.

* - :strong:`Field name`
- :strong:`Actions`
* - Metric type
* - :guilabel:`Metric type`
- Select :guilabel:`Custom metric` from the dropdown menu
* - Numerator metric
* - :guilabel:`Good events (numerator)`
- Search for the metric you want to use for the success request count
* - Denominator metric
* - :guilabel:`Total events (denominator)`
- Search for the metric you want to use for the total request count

.. note:: Custom metric SLO works by calculating the percentage of successful requests over a given compliance period. This calculation works better for counter and histogram metrics than for gauge metrics. Gauge metrics are not suitable for custom metric SLO, so you might get confusing data when selecting gauge metrics in your configuration.
Expand All @@ -73,11 +73,11 @@ Follow these steps to create an SLO.

* - :strong:`Field name`
- :strong:`Actions`
* - Target (%)
* - :guilabel:`Target (%)`
- Enter the target you want to set for this SLO.
* - Latency (ms)
* - :guilabel:`Latency (ms)`
- Only available and required for request latency SLI type. Enter the target loading time for your service requests.
* - Compliance window
* - :guilabel:`Compliance window`
- Select a compliance window for this SLO from the dropdown menu.

#. Set up alerting for your SLO. You can subscribe to get notifications for the following alerts.
Expand All @@ -89,12 +89,12 @@ Follow these steps to create an SLO.

* - :strong:`Alert`
- :strong:`Description`
* - Breach event
* - :guilabel:`Breach event`
- | Alerts when the service level indicator (SLI) doesn't meet the target over the specified compliance window.
| :strong:`Note:` Breach event alerting is selected by default and always runs in the background.
* - Error budget
* - :guilabel:`Error budget`
- Alerts when the remaining error budget is less than 10% of the estimated error budget for the compliance window.
* - Burn rate
* - :guilabel:`Burn rate`
- Alerts when the rate of consumption of your SLO error budget exceeds a healthy threshold for the specified compliance window. To learn more, see :ref:`burn-rate-alerts`.

#. Splunk Observability Cloud automatically generates a name for your SLO. You can change this auto-generated name, as long as the SLO name is unique.
Expand Down
82 changes: 82 additions & 0 deletions alerts-detectors-notifications/slo/custom-metric-scenario.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
.. _custom-metric-slo-scenario:

*********************************************************************************************
Scenario: Kai configures a service level objective (SLO) based on a Synthetics check
*********************************************************************************************


.. meta::
:description: This Splunk service level objective (SLO) scenario describes how to configure an SLO based on a Synthetics check

Kai, a site reliability engineer at Buttercup Games, has created a Synthetics Browser test for the Emby service on the Buttercup Games website. Kai didn't instrument the Emby service but wants to monitor the health of this service.

To measure and track the health metrics of the Emby service, Kai configures a custom metric SLO based on the Synthetics Browser test.

Use custom metric as service level indicator (SLI)
======================================================

From the :guilabel:`Detectors & SLOs` page, Kai configures the SLI and sets up a target for their SLO. Kai follows these steps:

#. Kai wants to use custom metrics as the system health indicators, so they select the :guilabel:`Custom metric` from the :guilabel:`Metric type` menu.
#. Kai enters the custom metrics they want to measure in the following fields:

.. list-table::
:header-rows: 1
:widths: 10 20 30 40

* - Field
- Metric name
- Filters
- Description

* - :guilabel:`Good events (numerator)`
- :strong:`synthetics.run.count`
- Kai adds the following filters for this metric:

* :strong:`test = Emby check`
* :strong:`success = true`
- Kai uses the :strong:`success = true` filter to count the number of successful requests for the Emby service on the Buttercup Games website.

* - :guilabel:`Total events (denominator)`
- :strong:`synthetics.run.count`
- Kai adds the following filter for this metric:

* :strong:`test = Emby check`
- Kai uses the same metric name and the :strong:`test = Emby check` filter to track the same Synthetics Browser test. However, Kai doesn't include the :strong:`success = true` dimension filter in order to count the number of total requests for the Emby service on the Buttercup Games website.

#. Kai enters the following fields to define a target for their SLO:

.. list-table::
:header-rows: 1
:widths: 32 30 50

* - Field
- Value
- Description

* - :guilabel:`Target (%)`
- :strong:`99.9`
- Kai wants to measure the Emby service successful request rate against a 99.9% target.

* - :guilabel:`Compliance window`
- :strong:`Last 30 days`
- Kai wants to track this SLO over the past 30 days.

#. Kai subscribes to receive an alert whenever there is a breach event for the SLO target.

.. image:: /_images/images-slo/custom-metric-slo-scenario.png
:width: 100%
:alt: This image shows Kai's SLO configuration using the ``synthetics.run.count`` metric and appropriate filters.


Summary
=======================

Kai set up an SLO and subscribed to breach event alerts that let them know when the Emby service on the Buttercup Games website fails to meet the request rate target. Kai managed to use custom metrics from an existing Synthetic Browser test to configure the SLO, despite not instrumenting the service themself.

Learn more
=======================

For more information about creating an SLO, see :ref:`create-slo`.

For more information about the Synthetics Browser test, see :ref:`browser-test`.
1 change: 1 addition & 0 deletions alerts-detectors-notifications/slo/slo-intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Introduction to service level objective (SLO) management in Splunk Observability
Measure and track your service health metrics with SLOs <create-slo>
View and manage SLOs <view-slo>
burn-rate-alerts
Scenario: Kai configures an SLO based on a Synthetics check <custom-metric-scenario>

Service level objective (SLO) management is a service level monitoring experience that helps you align your business needs with your engineering reliability goals.

Expand Down
Loading