o365: Add Microsoft Reports data-stream

[kcreddy]

Proposed commit message

Adds following Microsoft 365 Usage Reports to Office 365 integration using Microsoft Graph API.

• Microsoft Teams User Activity User Detail: ref
• Viva Engage Groups Activity Group Detail: ref
• Office365 Groups Activity Group Detail: ref
• SharePoint Site Usage Site Detail: ref
• OneDrive Usage Account Detail: ref

Reference issue: #12054.

Other changes:

1. Update docs to indicate the integration now supports both logs and metrics.
2. Update kibana.version to 8.15 to utilise latest CEL macros.

Note

To reviewers:
Following decisions are taken during this work:

1. Make generic reports data-streams instead of data-stream per report. Users are allowed to configure which reports to fetch.
2. Downloaded CSV files contains spaces inside field names, which are replaced by underscore. The field names remain otherwise unaltered to maintain integrity with raw data.
3. System tests are removed in commit as the config url path doesn't support a regex. We require a wildcard pattern on date, to avoid failure during daily-CI-runs.
4. Reason for adding transforms is documented in README.
5. The transform should ideally run once every 24h because the reports are only available once a day. But due to the limitation of transform's maximum frequency, it is updated to 1h.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Screenshots to PR and integration.
• Update o365 documentation to include both logs and metrics.
• Add transform.
• Add data validation logic to CEL as performed by Microsoft.
• Generic reports instead of multiple data-streams.

How to test this PR locally

Pipeline Tests:
eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=reports

--- Test results for package: o365 - START ---
╭─────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ o365    │ reports     │ pipeline  │ (ingest pipeline warnings test-office365-groups-activity-group.log)   │ PASS   │   270.8875ms │
│ o365    │ reports     │ pipeline  │ (ingest pipeline warnings test-onedrive-usage-account.log)            │ PASS   │ 294.903333ms │
│ o365    │ reports     │ pipeline  │ (ingest pipeline warnings test-sharepoint-site-usage-site.log)        │ PASS   │ 279.616917ms │
│ o365    │ reports     │ pipeline  │ (ingest pipeline warnings test-teams-user-activity-user.log)          │ PASS   │ 304.869084ms │
│ o365    │ reports     │ pipeline  │ (ingest pipeline warnings test-viva-engage-groups-activity-group.log) │ PASS   │ 353.329375ms │
│ o365    │ reports     │ pipeline  │ test-office365-groups-activity-group.log                              │ PASS   │  88.770792ms │
│ o365    │ reports     │ pipeline  │ test-onedrive-usage-account.log                                       │ PASS   │  59.335084ms │
│ o365    │ reports     │ pipeline  │ test-sharepoint-site-usage-site.log                                   │ PASS   │  59.270125ms │
│ o365    │ reports     │ pipeline  │ test-teams-user-activity-user.log                                     │ PASS   │     76.312ms │
│ o365    │ reports     │ pipeline  │ test-viva-engage-groups-activity-group.log                            │ PASS   │  63.989709ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: o365 - END   ---
Done

Related issues

• Relates [o365] Add additional reports to Office 365 integration #12054

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[kcreddy]

Removed system tests as the regex is unsupported in path.

[chrisberkhout]

I think regular expressions can be used, but with different syntax.

The elastic/stream README says:

path: the path to match. It can use gorilla/mux parameters patterns

The mux doc says:

Paths can have variables. They are defined using the format {name} or {name:pattern}. If a regular expression pattern is not defined, the matched variable will be anything until the next slash. For example:

r := mux.NewRouter()
r.HandleFunc("/products/{key}", ProductHandler)
r.HandleFunc("/articles/{category}/", ArticlesCategoryHandler)
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
90.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5e9dee5

History

• 💚 Build #19880 succeeded 38afd95
• 💔 Build #19870 failed 69e6adc
• 💔 Build #19857 failed b6d564a
• 💔 Build #19823 failed 758ae82
• 💚 Build #19616 succeeded 9a9f595
• 💔 Build #19614 failed ceb4f0e

cc @kcreddy

[kcreddy]

Moving to draft as it needs PM clarification how we want to split the data (across integrations/data-streams).

[chrisberkhout]

Partial review with a few comments. Read the README and looked at the overall structure.

Moving to draft as it needs PM clarification how we want to split the data (across integrations/data-streams).

I like it as it is. I'm not sure that splitting reports into separate data streams would help in any way, especially since you can already configure which reports you want and the transforms create separate destination indices for each. For the de-duplication, transforms always seem like a heavy-handed solution, but they might be the best solution here.

[chrisberkhout]

Suggested change

	|------------------|:-------:|
	|------------------|-------|

This is a very nice table to have in the README!

Looks better with both columns left-aligned:

[chrisberkhout]

If this file is broken out into separate files for each report, then each transform can use one of those files without changes. Currently they're all different:

find -wholename '*/reports/*/fields.yml' -or -wholename '*/transform/*/fields.yml' | sort | xargs md5sum

377cac0e355ca11527037e46271238bd  ./data_stream/reports/fields/fields.yml
5b233259d6615b83105cf4053896b768  ./elasticsearch/transform/latest_office365_groups_activity_group/fields/fields.yml
78c15f9ad25f60975de6ffd5196e971d  ./elasticsearch/transform/latest_onedrive_usage_account/fields/fields.yml
a2f7e13dbfca735b495940306347750f  ./elasticsearch/transform/latest_sharepoint_site_usage_site/fields/fields.yml
a65fa2cf69f1a8aa619815fddb0a1871  ./elasticsearch/transform/latest_teams_user_activity_user/fields/fields.yml
8106e9736f725f106b863c4b233de965  ./elasticsearch/transform/latest_viva_engage_groups_activity_group/fields/fields.yml

[kcreddy]

Thanks Chris. I like that method, will incorporate it.

[chrisberkhout]

Another very helpful table.

I'd say Source indices: metrics-o365.reports-* rather than .ds-metrics-o365.reports-*.

metrics-o365.reports-* is the value used in source.index in transform.yml.

[kcreddy]

I like it as it is. I'm not sure that splitting reports into separate data streams would help in any way, especially since you can already configure which reports you want and the transforms create separate destination indices for each.

The real concern is whether to add reports inside Microsoft Office 365 integration or create multiple integrations i.e., 1 integration per Microsoft 365 entity, such as OneDrive, to enable customers to onboard entity-specific integrations.

[kcreddy]

Closing in favour of #12256