Skip to content

Commit

Permalink
[Cisco Duo] Integration updates (elastic#11200)
Browse files Browse the repository at this point in the history
Added new data stream telephony_v2 to support the new v2 API endpoint.

Updated the auth data stream to migrate from the HTTPJSON to CEL.

Review documentation and dashboards
  • Loading branch information
chemamartinez authored Oct 1, 2024
1 parent 27f35c8 commit b0c20d8
Show file tree
Hide file tree
Showing 36 changed files with 2,454 additions and 703 deletions.
42 changes: 36 additions & 6 deletions packages/cisco_duo/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cisco Duo

The Cisco Duo integration collects and parses data from the [Cisco Duo Admin APIs](https://duo.com/docs/adminapi).
The Cisco Duo integration collects and parses data from the [Cisco Duo Admin APIs](https://duo.com/docs/adminapi). The Duo Admin API provides programmatic access to the administrative functionality of Duo Security's two-factor authentication platform.

## Compatibility

Expand All @@ -16,11 +16,33 @@ In order to ingest data from the Cisco Duo Admin API you must:
- For this integration you will require **Grant read information** and **Grant read log** permissions.
- Make sure you have whitelisted your IP Address.

## Note
More details for each step can be found at [First steps](https://duo.com/docs/adminapi#first-steps).

While setting up the interval take care of following.
- `Interval has to be greater than 1m.`
- `Larger values of interval might cause delay in data ingestion.`
## Data streams

The Cisco Duo integration collects logs for the following types of events.

- [**Administrator Logs**](https://duo.com/docs/adminapi#administrator-logs)
- [**Authentication Logs**](https://duo.com/docs/adminapi#authentication-logs)
- [**Offline Enrollment Logs**](https://duo.com/docs/adminapi#offline-enrollment-logs)
- [**Summary**](https://duo.com/docs/adminapi#retrieve-summary)
- [**Telephony Logs**](https://duo.com/docs/adminapi#telephony-logs)
- [**Telephony Logs (legacy)**](https://duo.com/docs/adminapi#telephony-logs-(legacy-v1))

## V2 Handlers

Cisco Duo has implemented v2 handlers for some endpoints. In these cases, the API v1 handler remains supported, but will be limited or deprecated in the future.

From data streams listed above, v2 handlers are supported for Authentication and Telephony Logs at the moment. It is recommended to migrate data streams to the v2 endpoints when they become available.

## Configuration

The following considerations should be taken into account when configuring the integration.

- Interval has to be greater or equal than `1m`.
- The Duo Admin API retrieves records from the last 180 days up to as recently as two minutes before the API request. Consider this when configuring the `Initial interval` parameter for the v2 API endpoints, as it doesn't support `d` as a suffix, its maximum value is `4320h` which corresponds to that 180 days.
- For v2 API endpoints, a new parameter `limit` has been added to control the number of records per response. Default value is 100 and can be incresead until 1000.
- Larger values of interval might cause delay in data ingestion.

## Logs

Expand Down Expand Up @@ -62,4 +84,12 @@ This is the `telephony` dataset.

{{event "telephony"}}

{{fields "telephony"}}
{{fields "telephony"}}

### Telephony v2

This is the `telephony_v2` dataset.

{{event "telephony_v2"}}

{{fields "telephony_v2"}}
60 changes: 60 additions & 0 deletions packages/cisco_duo/_dev/deploy/docker/files/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,63 @@ rules:
{"response": [
{"context":"administrator login","credits":5,"isotimestamp":"2021-07-22T12:59:30+00:00","phone":"+121234512345","timestamp":1626958770,"type":"phone"},{"context":"verify","credits":1,"isotimestamp":"2021-08-16T06:03:32+00:00","phone":"+121234512345","timestamp":1629093812,"type":"sms"},{"context": "authentication","credits": 1,"isotimestamp":"2020-03-20T15:38:12+00:00","phone":"+121234512345","timestamp":1584718692,"type":"sms"}
], "stat": "OK"}
- path: /admin/v2/logs/telephony
methods: ["GET"]
query_params:
next_offset: "1666714065304,5bf1a860-fe39-49e3-be29-217659663a74"
responses:
- status_code: 200
body: |-
{
"stat": "OK",
"response": {
"items": [
{
"context": "administrator login",
"credits": 0,
"phone": "+13135559542",
"telephony_id": "5bf1a860-fe39-49e3-be29-217659663a74",
"ts": "2022-10-25T16:07:45.304526+00:00",
"txid": "fb0c129b-f994-4d3d-953b-c3e764272eb7",
"type": "sms"
}
],
"metadata": {
"total_objects": 1
}
}
}
- path: /admin/v2/logs/telephony
methods: ["GET"]
responses:
- status_code: 200
body: |-
{
"stat": "OK",
"response": {
"items": [
{
"context": "enrollment",
"credits": 1,
"phone": "+12125556707",
"telephony_id": "220f89ff-bff8-4466-b6cb-b7787940ce68",
"ts": "2023-03-21T22:34:49.466370+00:00",
"txid": "2f5d34d3-053f-422c-9dd4-77a5d58706b1",
"type": "sms"
},
{
"context": "authentication",
"credits": 2,
"phone": "+17345551311",
"telephony_id": "60799fee-f08f-4ba8-971f-4e53b3473e9a",
"ts": "2023-01-26T20:54:12.573580+00:00",
"txid": "373bd5f3-1e42-4a5d-aefa-b33ae278fac8",
"type": "phone"
}
],
"metadata": {
"next_offset": "1666714065304,5bf1a860-fe39-49e3-be29-217659663a74",
"total_objects": 2
}
}
}
11 changes: 11 additions & 0 deletions packages/cisco_duo/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
# newer versions go on top
- version: "2.0.0"
changes:
- description: Migrate to CEL input for data streams that use v2 API.
type: enhancement
link: https://github.com/elastic/integrations/pull/11200
- description: Add Telephony v2 data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/11200
- description: Update dashboards and documentation.
type: enhancement
link: https://github.com/elastic/integrations/pull/11200
- version: "1.26.0"
changes:
- description: "Allow @custom pipeline access to event.original without setting preserve_original_event."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ vars:
hostname: http://{{Hostname}}:{{Port}}
secret_key: 40_characters_long_secret_key
integration_key: temp_integration_key
enable_request_tracer: true
data_stream:
vars:
preserve_original_event: true
enable_request_tracer: true
162 changes: 162 additions & 0 deletions packages/cisco_duo/data_stream/auth/agent/stream/cel.yml.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
config_version: 2
interval: {{interval}}
resource.url: {{hostname}}

{{#if enable_request_tracer}}
resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
resource.tracer.maxbackups: 5
resource.tracer.maxsize: 5
{{/if}}

regexp:
"next_offset_timestamp": '^\d+'

state:
url: {{hostname}}
integration_key: {{integration_key}}
secret_key: {{secret_key}}
limit: {{limit}}
initial_interval: {{initial_interval}}
want_more: false
redact:
fields:
- secret_key

program: |
(
state.want_more ?
state
:
state.with({
"mintime": state.?cursor.last_published.orValue(int(now - duration(state.initial_interval)) * 1000),
"maxtime": int(now - duration("2m")) * 1000,
"date": now.format(time_layout.RFC1123Z),
})
).as(state, state.with(
request(
"GET",
state.?want_more.orValue(false) ?
state.next_url
:
state.url.trim_right("/") + "/admin/v2/logs/authentication?" + {
"limit": [string(int(state.limit))],
"maxtime": [string(int(state.maxtime))],
"mintime": [string(int(state.mintime))],
"sort": ["ts:asc"],
}.format_query()
).with(
{
"Header": {
"Content-Type": ["application/x-www-form-urlencoded"],
"Date": [state.date],
"Authorization": ["Basic " + (
state.integration_key + ":" + (
[
state.date,
"GET",
state.url.trim_prefix("https://"),
"/admin/v2/logs/authentication",
{
"limit": [string(int(state.limit))],
"maxtime": [string(int(state.maxtime))],
"mintime": [string(int(state.mintime))],
?"next_offset": has(state.next_offset) ?
optional.of([string(state.next_offset)])
:
optional.none(),
"sort": ["ts:asc"],
}.format_query()
].join("\n")
.hmac("sha1", bytes(state.secret_key))
.hex()
)
).base64()],
},
}
).do_request().as(resp, (resp.StatusCode == 200) ?
bytes(resp.Body).decode_json().as(body, has(body.?response.authlogs) && size(body.response.authlogs) > 0 ?
{
"events": body.response.authlogs.map(item,
{
"message": item.encode_json(),
}
),
"url": state.url,
"integration_key": state.integration_key,
"secret_key": state.secret_key,
"limit": state.limit,
"mintime": state.mintime,
"maxtime": state.maxtime,
"date": now.format(time_layout.RFC1123Z),
"want_more": has(body.?response.?metadata.next_offset),
?"next_offset": (has(body.?response.?metadata.next_offset) && body.response.metadata.next_offset != null) ?
optional.of(string(body.response.metadata.next_offset))
:
optional.none(),
"next_url": (has(body.?response.?metadata.next_offset) && body.response.metadata.next_offset != null) ?
(
state.url.trim_right("/") + "/admin/v2/logs/authentication?" + {
"limit": [string(int(state.limit))],
"maxtime": [string(int(state.maxtime))],
"mintime": [string(int(state.mintime))],
"next_offset": [string(body.response.metadata.next_offset)],
"sort": ["ts:asc"],
}.format_query()
)
:
state.url,
"cursor": {
?"last_published": (has(body.?response.?metadata.next_offset) && body.response.metadata.next_offset != null) ?
optional.of(body.response.metadata.next_offset.re_find("next_offset_timestamp"))
:
optional.none(),
}
}
:
{
"events":[],
"want_more": false,
}

)
:
bytes(resp.Body).decode_json().as(body,
{
"events": {
"error": {
"code": has(body.code) ? string(body.code) : string(resp.StatusCode),
"id": string(resp.Status),
"message": "GET: " +
(
(has(body.message) && body.message != "") ?
string(body.message) +
(has(body.message_detail) ?
": " + string(body.message_detail)
:
""
)
:
string(resp.Status) + " (" + string(resp.StatusCode) + ")"
),
},
},
"want_more": false,
}
)
)
))

tags:
{{#if preserve_original_event}}
- preserve_original_event
{{/if}}
{{#each tags as |tag|}}
- {{tag}}
{{/each}}
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if processors}}
processors:
{{processors}}
{{/if}}
20 changes: 18 additions & 2 deletions packages/cisco_duo/data_stream/auth/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,24 @@
type: logs
title: Cisco Duo authentication logs
streams:
- input: httpjson
- input: cel
enabled: false
vars:
- name: initial_interval
type: text
title: Initial Interval
multi: false
show_user: false
required: true
default: 4320h
description: How far back to pull Telephony logs from the Cisco Duo API. Maximum interval is 180 days (4320 hours). Supported units for this parameter are h/m/s.
- name: limit
type: integer
title: Limit
description: Maximum number of records to fetch on each request. Max is 1000.
show_user: false
required: true
default: 100
- name: tags
type: text
title: Tags
Expand All @@ -28,6 +44,6 @@ streams:
show_user: false
description: >-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
template_path: httpjson.yml.hbs
template_path: cel.yml.hbs
title: Cisco Duo authentication logs
description: Collect Cisco Duo authentication logs
Loading

0 comments on commit b0c20d8

Please sign in to comment.