Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AWS] Use namespace for GetListMetrics when exists #41022

Merged
merged 6 commits into from
Oct 8, 2024
Merged

[AWS] Use namespace for GetListMetrics when exists #41022

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a419b52
Use namespace for GetListMetrics when exists
kaiyan-sheng Sep 27, 2024
4143efb
add changelog
kaiyan-sheng Sep 27, 2024
95df0a6
fix lint
kaiyan-sheng Sep 27, 2024
d3be852
check len(namespaceDetailTotal) instead
kaiyan-sheng Oct 2, 2024
f83c542
update documentation
kaiyan-sheng Oct 7, 2024
2c0dde2
Merge branch 'main' into cloudwatch_delay
kaiyan-sheng Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Remove excessive info-level logs in cgroups setup {pull}40491[40491]
- Add missing ECS Cloud fields in GCP `metrics` metricset when using `exclude_labels: true` {issue}40437[40437] {pull}40467[40467]
- Add AWS OwningAccount support for cross account monitoring {issue}40570[40570] {pull}40691[40691]
- Use namespace for GetListMetrics when exists in AWS {pull}41022[41022]
- Fix http server helper SSL config. {pull}39405[39405]

*Osquerybeat*
Expand Down
3 changes: 2 additions & 1 deletion metricbeat/docs/modules/aws.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,8 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
| IAM ListAccountAliases | 1 | Once on startup
| STS GetCallerIdentity | 1 | Once on startup
| EC2 DescribeRegions| 1 | Once on startup
| CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics without specifying namespace in configuration | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics with specific namespaces in configuration | Total number of results / ListMetrics max page size * number of unique namespaces | Per region per collection period
| CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
|===
`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
Expand Down
3 changes: 2 additions & 1 deletion x-pack/metricbeat/module/aws/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,8 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
| IAM ListAccountAliases | 1 | Once on startup
| STS GetCallerIdentity | 1 | Once on startup
| EC2 DescribeRegions| 1 | Once on startup
| CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics without specifying namespace in configuration | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics with specific namespaces in configuration | Total number of results / ListMetrics max page size * number of unique namespaces | Per region per collection period
| CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
|===
`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
Expand Down
20 changes: 16 additions & 4 deletions x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,22 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error {
continue
}

// retrieve all the details for all the metrics available in the current region
listMetricsOutput, err := aws.GetListMetricsOutput("*", regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s: %w", regionName, err)
// retrieve all the details for all the metrics available in the current region when no namespace is specified
// otherwise only retrieve metrics from the specific namespaces from the config
var listMetricsOutput []aws.MetricWithID
if len(namespaceDetailTotal) == 0 {
listMetricsOutput, err = aws.GetListMetricsOutput("*", regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s and namespace %s: %w", regionName, "*", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to include the namespace in this error message?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I think that this is can be a good integration test to include it. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error message here shows namespace is * because in this if statement we are calling ListMetrics with * all namespaces. The else statement part is the one with a specific namespace.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas, most integration tests start the metricset/input and check that they produce a non-zero number of events.

What kind of check do you have in mind?

What do you think about creating a dedicated issue to enhance the integration tests for the cloudwatch metricset?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaiyan-sheng, this change may increase the number of API calls because we're making one API call for each namespace in the config.

However, it shouldn't make a difference since I don't expect configurations to have more than 1-3 namespaces, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmoog #41146 FYI

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmoog Yeah that's the original thought when we decide to use "*" instead of individual namespace in this API call. But I think at this point, the benefit of using specific namespaces with a higher number of API calls wins.
I will update the document here: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-aws.html#aws-api-requests

Copy link
Contributor Author

@kaiyan-sheng kaiyan-sheng Oct 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually already have tests around this but since we are mocking the ListMetrics API and what's returning from the API, the test is no use for distinguish namespaces. As we discussed earlier, I will merge this PR to get the fix in but look into the test later. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmoog I tested the pr again with the config below to double check how many times this ListMetrics API will be called. Turned out we are calling it once because namespaces are the same in both parts under metrics:

- module: aws
  period: 60s
  metricsets:
    - cloudwatch
  data_granularity: 60s
  include_linked_accounts: true
  regions:
    - ap-southeast-1
  metrics:
    - dimensions:
        - name: CanaryName
          value: canary1
      namespace: CloudWatchSynthetics
    - dimensions:
        - name: CanaryName
          value: canary2
      namespace: CloudWatchSynthetics

}
} else {
for namespace := range namespaceDetailTotal {
listMetricsOutputPerNamespace, err := aws.GetListMetricsOutput(namespace, regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s and namespace %s: %w", regionName, namespace, err)
}
listMetricsOutput = append(listMetricsOutput, listMetricsOutputPerNamespace...)
}
}

if len(listMetricsOutput) == 0 {
Expand Down
Loading