Skip to content

Commit

Permalink
make sev and deprecated_v1_fields obsolete. bump to 5.0.0
Browse files Browse the repository at this point in the history
  • Loading branch information
@jsvd
jsvd committed Aug 1, 2017
1 parent 472bb81 commit a781cf9
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 260 deletions.
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 5.0.0
- move `sev` and `deprecated_v1_fields` fields from deprecated to obsolete

## 4.1.2
- added mapping for outcome = eventOutcome from CEF whitepaper (ref:p26/39)

Expand Down
21 changes: 5 additions & 16 deletions docs/index.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,12 @@ This setting allows the following character sequences to have special meaning:
* `\\n` (backslash "n") - means newline (ASCII 0x0A)

[id="plugins-{type}s-{plugin}-deprecated_v1_fields"]
===== `deprecated_v1_fields` (DEPRECATED)
===== `deprecated_v1_fields` (OBSOLETE)

* DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
* OBSOLETE WARNING: This configuration item is obsolete and will prevent the pipeline from starting if used
* Value type is <<boolean,boolean>>
* There is no default value for this setting.

Set this flag if you want to have both v1 and v2 fields indexed at the same time. Note that this option will increase
the index size and data stored in outputs like Elasticsearch
This option is available to ease transition to new schema

[id="plugins-{type}s-{plugin}-fields"]
===== `fields`

Expand Down Expand Up @@ -106,20 +102,13 @@ Device product field in CEF header. The new value can include `%{foo}` strings
to help you build a new value from other parts of the event.

[id="plugins-{type}s-{plugin}-sev"]
===== `sev` (DEPRECATED)
===== `sev` (OBSOLETE)

* DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
* OBSOLETE WARNING: This configuration item is obsolete and will prevent the pipeline from starting.
* Value type is <<string,string>>
* There is no default value for this setting.

Deprecated severity field for CEF header. The new value can include `%{foo}` strings
to help you build a new value from other parts of the event.

This field is used only if :severity is unchanged set to the default value.

Defined as field of type string to allow sprintf. The value will be validated
to be an integer in the range from 0 to 10 (including).
All invalid values will be mapped to the default of 6.
Obsolete severity field for CEF header use :severity instead.

[id="plugins-{type}s-{plugin}-severity"]
===== `severity`
Expand Down
24 changes: 3 additions & 21 deletions lib/logstash/codecs/cef.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,8 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
# to help you build a new value from other parts of the event.
config :name, :validate => :string, :default => "Logstash"

# Deprecated severity field for CEF header. The new value can include `%{foo}` strings
# to help you build a new value from other parts of the event.
#
# This field is used only if :severity is unchanged set to the default value.
#
# Defined as field of type string to allow sprintf. The value will be validated
# to be an integer in the range from 0 to 10 (including).
# All invalid values will be mapped to the default of 6.
config :sev, :validate => :string, :deprecated => "This setting is being deprecated, use :severity instead."
# Obsolete severity field for CEF header
config :sev, :validate => :string, :obsolete => "This setting is obsolete, use :severity instead."

# Severity field in CEF header. The new value can include `%{foo}` strings
# to help you build a new value from other parts of the event.
Expand All @@ -53,10 +46,7 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
# Fields to be included in CEV extension part as key/value pairs
config :fields, :validate => :array, :default => []

# Set this flag if you want to have both v1 and v2 fields indexed at the same time. Note that this option will increase
# the index size and data stored in outputs like Elasticsearch
# This option is available to ease transition to new schema
config :deprecated_v1_fields, :validate => :boolean, :deprecated => "This setting is being deprecated"
config :deprecated_v1_fields, :validate => :boolean, :obsolete => "This setting is obsolete"

# If your input puts a delimiter between each CEF event, you'll want to set
# this to be that delimiter.
Expand Down Expand Up @@ -133,9 +123,6 @@ def handle(data, &block)
split_data = data.split /(?<=[^\\]\\\\)[\|]|(?<!\\)[\|]/

# To be invoked when config settings is set to TRUE for V1 field names (cef_ext.<fieldname>) the following code might be removed in upcoming Codec revision
if deprecated_v1_fields
handle_v1_fields(event, split_data)
end

# To be invoked with default config settings to utilise the new field name formatting and flatten out the JSON document
# Store header fields
Expand Down Expand Up @@ -213,12 +200,7 @@ def encode(event)
name = sanitize_header_field(event.sprintf(@name))
name = self.class.get_config["name"][:default] if name == ""

# :sev is deprecated and therefore only considered if :severity equals the default setting or is invalid
severity = sanitize_severity(event, @severity)
if severity == self.class.get_config["severity"][:default] && @sev
# Use deprecated setting sev
severity = sanitize_severity(event, @sev)
end

# Should also probably set the fields sent
header = ["CEF:0", vendor, product, version, signature, name, severity].join("|")
Expand Down
2 changes: 1 addition & 1 deletion logstash-codec-cef.gemspec
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Gem::Specification.new do |s|

s.name = 'logstash-codec-cef'
s.version = '4.1.3'
s.version = '5.0.0'
s.platform = 'java'
s.licenses = ['Apache License (2.0)']
s.summary = "CEF codec to parse and encode CEF formated logs"
Expand Down
222 changes: 0 additions & 222 deletions spec/codecs/cef_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -209,54 +209,6 @@
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=[0-9TZ.:-]+$/m)
end

it "should use severity (instead of depricated sev), if severity is set)" do
codec.on_event{|data, newdata| results << newdata}
codec.sev = "4"
codec.severity = "5"
codec.fields = []
event = LogStash::Event.new("foo" => "bar")
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|5\|$/m)
end

it "should use deprecated sev, if severity is not set (equals default value)" do
codec.on_event{|data, newdata| results << newdata}
codec.sev = "4"
codec.fields = []
event = LogStash::Event.new("foo" => "bar")
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
end

it "should use deprecated sev, if severity is explicitly set to default value)" do
codec.on_event{|data, newdata| results << newdata}
codec.sev = "4"
codec.severity = "6"
codec.fields = []
event = LogStash::Event.new("foo" => "bar")
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
end

it "should use deprecated sev, if severity is invalid" do
codec.on_event{|data, newdata| results << newdata}
codec.sev = "4"
codec.severity = ""
codec.fields = []
event = LogStash::Event.new("foo" => "bar")
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
end

it "should use default value, if severity is not set and sev is invalid" do
codec.on_event{|data, newdata| results << newdata}
codec.sev = ""
codec.fields = []
event = LogStash::Event.new("foo" => "bar")
codec.encode(event)
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
end
end

context "sanitize header field" do
Expand Down Expand Up @@ -567,180 +519,6 @@ def validate(e)
end
end
end

end

context "decode with deprecated version option" do
let (:message) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
let(:options) {
{
"deprecated_v1_fields" => true
}
}

subject(:codec) { LogStash::Codecs::CEF.new(options) }

def validate(e)
insist { e.is_a?(LogStash::Event) }
insist { e.get('cef_version') } == "0"
insist { e.get('cef_device_version') } == "1.0"
insist { e.get('cef_sigid') } == "100"
insist { e.get('cef_name') } == "trojan successfully stopped"
insist { e.get('cef_severity') } == "10"
insist { e.get('cefVersion') } == "0"
insist { e.get('deviceVersion') } == "1.0"
insist { e.get('deviceEventClassId') } == "100"
insist { e.get('name') } == "trojan successfully stopped"
insist { e.get('severity') } == "10"
end

it "should parse the cef headers" do
subject.decode(message) do |e|
validate(e)
ext = e.get('cef_ext')
insist { e.get("cef_vendor") } == "security"
insist { e.get("cef_product") } == "threatmanager"
insist { e.get("deviceVendor") } == "security"
insist { e.get("deviceProduct") } == "threatmanager"
end
end

it "should parse the cef body" do
subject.decode(message) do |e|
ext = e.get('cef_ext')
insist { ext['src'] } == "10.0.0.192"
insist { ext['dst'] } == "12.121.122.82"
insist { ext['spt'] } == "1232"
insist { e.get("sourceAddress")} == "10.0.0.192"
insist { e.get("destinationAddress") } == "12.121.122.82"
insist { e.get("sourcePort") } == "1232"
end
end

let (:no_ext) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|" }
it "should be OK with no extension dictionary" do
subject.decode(no_ext) do |e|
validate(e)
insist { e.get("cef_ext") } == nil
end
end

let (:missing_headers) { "CEF:0|||1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
it "should be OK with missing CEF headers (multiple pipes in sequence)" do
subject.decode(missing_headers) do |e|
validate(e)
insist { e.get("cef_vendor") } == ""
insist { e.get("cef_product") } == ""
insist { e.get("deviceVendor") } == ""
insist { e.get("deviceProduct") } == ""
end
end

let (:leading_whitespace) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10| src=10.0.0.192 dst=12.121.122.82 spt=1232" }
it "should strip leading whitespace from the message" do
subject.decode(leading_whitespace) do |e|
validate(e)
end
end

let (:escaped_pipes) { 'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this\|has an escaped pipe' }
it "should be OK with escaped pipes in the message" do
subject.decode(escaped_pipes) do |e|
ext = e.get('cef_ext')
insist { ext['moo'] } == 'this\|has an escaped pipe'
end
end

let (:pipes_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this|has an pipe'}
it "should be OK with not escaped pipes in the message" do
subject.decode(pipes_in_message) do |e|
ext = e.get('cef_ext')
insist { ext['moo'] } == 'this|has an pipe'
end
end

let (:escaped_equal_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this \=has escaped \= equals\='}
it "should be OK with escaped equal in the message" do
subject.decode(escaped_equal_in_message) do |e|
ext = e.get('cef_ext')
insist { ext['moo'] } == 'this =has escaped = equals='
end
end

let (:escaped_backslash_in_header) {'CEF:0|secu\\\\rity|threat\\\\manager|1.\\\\0|10\\\\0|tro\\\\jan successfully stopped|\\\\10|'}
it "should be OK with escaped backslash in the headers" do
subject.decode(escaped_backslash_in_header) do |e|
insist { e.get("cef_version") } == '0'
insist { e.get("cef_vendor") } == 'secu\\rity'
insist { e.get("cef_product") } == 'threat\\manager'
insist { e.get("cef_device_version") } == '1.\\0'
insist { e.get("cef_sigid") } == '10\\0'
insist { e.get("cef_name") } == 'tro\\jan successfully stopped'
insist { e.get("cef_severity") } == '\\10'
end
end

let (:escaped_backslash_in_header_edge_case) {'CEF:0|security\\\\\\||threatmanager\\\\|1.0|100|trojan successfully stopped|10|'}
it "should be OK with escaped backslash in the headers (edge case: escaped slash in front of pipe)" do
subject.decode(escaped_backslash_in_header_edge_case) do |e|
validate(e)
insist { e.get("cef_vendor") } == 'security\\|'
insist { e.get("cef_product") } == 'threatmanager\\'
end
end

let (:escaped_pipes_in_header) {'CEF:0|secu\\|rity|threatmanager\\||1.\\|0|10\\|0|tro\\|jan successfully stopped|\\|10|'}
it "should be OK with escaped pipes in the headers" do
subject.decode(escaped_pipes_in_header) do |e|
insist { e.get("cef_version") } == '0'
insist { e.get("cef_vendor") } == 'secu|rity'
insist { e.get("cef_product") } == 'threatmanager|'
insist { e.get("cef_device_version") } == '1.|0'
insist { e.get("cef_sigid") } == '10|0'
insist { e.get("cef_name") } == 'tro|jan successfully stopped'
insist { e.get("cef_severity") } == '|10'
insist { e.get("cefVersion") } == '0'
insist { e.get("deviceVendor") } == 'secu|rity'
insist { e.get("deviceProduct") } == 'threatmanager|'
insist { e.get("deviceVersion") } == '1.|0'
insist { e.get("deviceEventClassId") } == '10|0'
insist { e.get("name") } == 'tro|jan successfully stopped'
insist { e.get("severity") } == '|10'
end
end

let (:escaped_backslash_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this \\\\has escaped \\\\ backslashs\\\\'}
it "should be OK with escaped backslashs in the message" do
subject.decode(escaped_backslash_in_message) do |e|
ext = e.get('cef_ext')
insist { ext['moo'] } == 'this \\has escaped \\ backslashs\\'
end
end

let (:equal_in_header) {'CEF:0|security|threatmanager=equal|1.0|100|trojan successfully stopped|10|'}
it "should be OK with equal in the headers" do
subject.decode(equal_in_header) do |e|
validate(e)
insist { e.get("cef_product") } == "threatmanager=equal"
end
end

let (:syslog) { "Syslogdate Sysloghost CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
it "Should detect headers before CEF starts" do
subject.decode(syslog) do |e|
validate(e)
insist { e.get('syslog') } == 'Syslogdate Sysloghost'
end
end

context "when payload is not in CEF" do
let (:message) { "potatoes" }
it "Should detect headers before CEF starts" do
subject.decode(message) do |e|
insist { e.get('tags') } == ['_cefparsefailure']
end
end
end
end

context "encode and decode" do
Expand Down

0 comments on commit a781cf9

Please sign in to comment.