Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added PanOS 11 syslog standard fields; repaired broken field extracts & name collisions #294

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

jwiley80
Copy link

@jwiley80 jwiley80 commented May 9, 2023

Description

Added PanOS 11 syslog standard fields; repaired broken field extracts & name collisions

How Has This Been Tested?

Tested in Splunk against large-scale existing pan:* data flows

Types of changes

  • Breaking change 1: In default/transforms.conf, [extract_userid] previously omitted the "src_user" field early in the message, causing all fields following it to parse incorrectly. This change includes the "src_user" field correctly, and causes all fields after it to parse correctly. The may cause breakage for some users, dashboards, or other use cases dependent the currently incorrect field assignments.

  • Breaking change 2: In default/transforms.conf, [extract_config] previously included the "devicegroup_level3" and "devicegroup_level4" fields that do not exist in the log data. All fields following these extracts have been parsing incorrectly. This change correctly excludes the "devicegroup_level3" and "devicegroup_level4" fields to match the data correctly, and causes all fields after that point to parse correctly. The may cause breakage for some users, dashboards, or other use cases dependent the currently incorrect field assignments.

  • Bug fixes:

In props.conf

  1. pan:config - field aliases and evals added for CIM mapping and compatibility
  2. pan:globalprotect - field aliases and evals added for CIM mapping and compatibility; aliased "time_generated" to "generated_time" for consistency with other pan sourcetype naming
  3. pan:hipmatch - field aliases and evals added for CIM mapping and compatibility
  4. pan:system - field aliases and evals added for CIM mapping and compatibility, added description extracts
  5. pan:userid - added Field Aliases to match corrected Transforms extracts

In transforms.conf

  1. extract_traffic - updated to PanOS11 syslog fields in TechDocs; used "host_id", "host_serial", "nssai_sd" and "nssai_sst" instead of current "hostid", "serialnumber", "nsdsai_sd" and "nsdsai_sst" in TechDocs
  2. extract_threat - updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs
  3. extract_system - updated to PanOS11 syslog fields in TechDocs (only added "high_res_timestamp")
  4. extract_hipmatch- updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs
  5. extract_globalprotect - updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs. In extract_globalprotect, the old version uses 'serial_number' for this field, which collides with field 3, which is the 'dvc_serial', not the serial of the src/user asset being described in the log

Note:

  1. The fields "host_id" and "host_serial" in extract_threat, extract_traffic, extract_globalprotect, and extract_hipmatch are extremely useful for asset correlation, and needs to be consistently named for analysis.
  2. I changed field names from 'dst...' to 'dest...' for Splunk CIM compatibility. Both can be used with field aliasing if preferred, but dest is more consistent with the existing Splunk CIM.

paulmnguyen and others added 12 commits March 24, 2023 14:06
pan:system corrected dvc coalesce, added description extracts
…onfig data

Removed "devicegroup_level3" and "devicegroup_level4" fields, which do not exist in the config data, and cause all later fields to parse incorrectly.  

Added PanOS 11 updated fields at end
extract_threat, extract_traffic, extract_globalprotect, and extract_hipmatch all contain the fields "host_id" and "host_serial" - this is extremely useful for asset correlation, and needs to be consistently named for analysis.   In extract_globalprotect, the old version uses 'serial_number' for this field, which collides with field 3, which is the 'dvc_serial', not the serial of the src/user asset being described in the log
hipmatch fieldalias & eval updates for CIM consistency
field aliases and evals added for CIM compatibility
@welcome-to-palo-alto-networks
Copy link

🎉 Thanks for opening this pull request! We really appreciate contributors like you! 🙌

@jwiley80 jwiley80 closed this May 9, 2023
@jwiley80 jwiley80 reopened this May 9, 2023
@paulmnguyen paulmnguyen self-assigned this May 9, 2023
@paulmnguyen paulmnguyen added the add-on Related to the Splunk Add-On label May 9, 2023
@btorresgil
Copy link
Member

Thanks for the PR! Seeing a lot of good changes here. With the breaking changes we'd have to do a major release with comprehensive release note so give us some time to go through everything and plan.

@jwiley80 Can you remove any lines you commented out and do another commit/push? We'll see the lines are removed in the diff during review but having them still exist commented makes the diff harder to parse.

Thanks again!

@paulmnguyen paulmnguyen force-pushed the develop branch 2 times, most recently from de4dfdc to d7bd687 Compare May 17, 2023 20:58
@jwiley80
Copy link
Author

Any progress on this? I'm not sure if you're waiting on something from me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add-on Related to the Splunk Add-On
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants