rust/sip: register parser for tcp v9 #9893

glongo · 2023-11-25T16:18:47Z

Make sure these boxes are signed before submitting your Pull Request -- thank you.

I have read the contributing guide lines at
https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html
I have signed the Open Information Security Foundation contribution agreement at
https://suricata.io/about/contribution-agreement/ (note: this is only required once)
I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/3351

Describe changes:

Documentation updated
Implements pattern matching for protocol detection
Renames append_request and append_response to build_tx_request and build_tx_response
Removes input_len >= 3 condition
SIP version characters are enforced to strictly match SIP/2.0

Provide values to any of the below to override the defaults.

To use a pull request use a branch name like pr/N where N is the
pull request number.

Alternatively, SV_BRANCH may also be a link to an
OISF/suricata-verify pull-request.

SV_REPO=
SV_BRANCH=pr/1494
SU_REPO=
SU_BRANCH=
LIBHTP_REPO=
LIBHTP_BRANCH=

Accepts valid characters as defined in RFC3261.

The `is_version_char` function incorrectly allowed characters that are not part of the valid SIP version "SIP/2.0". For instance, 'HTTP/1.1' was mistakenly accepted as a valid SIP version, although it's not. This commit fixes the issue by updating the condition to strictly check for the correct version string.

This patch lets the parser to work over tcp protocol, taking care of handling data before calling the request/response parsers. Ticket OISF#3351.

This patch permits to set a direction when a new transaction is created in order to avoid 'signature shadowing' as reported by Eric Leblond in commit 5aaf507

This permits to detect SIP protocol using pattern matching as well.

codecov · 2023-11-25T16:35:56Z

Codecov Report

Merging #9893 (769de02) into master (d005fff) will decrease coverage by 0.11%.
The diff coverage is 86.20%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9893      +/-   ##
==========================================
- Coverage   82.45%   82.35%   -0.11%     
==========================================
  Files         972      972              
  Lines      273057   273260     +203     
==========================================
- Hits       225156   225047     -109     
- Misses      47901    48213     +312

Flag	Coverage Δ
fuzzcorpus	`64.21% <59.59%> (-0.16%)`	⬇️
suricata-verify	`61.06% <84.89%> (-0.04%)`	⬇️
unittests	`62.90% <29.50%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

catenacyber · 2023-11-27T10:58:38Z

SIP version characters are enforced to strictly match SIP/2.0

Why so ?
I think we want to match on past and future versions as well...

catenacyber · 2023-11-27T11:00:49Z

rust/src/sip/parser.rs

@@ -78,7 +78,7 @@ fn is_request_uri_char(b: u8) -> bool {

 #[inline]
 fn is_version_char(b: u8) -> bool {
-    is_alphanumeric(b) || b"./".contains(&b)
+    b"SIP/2.0".contains(&b)


I think we rather want the caller of is_version_char begin by tag("SIP/"), what do you think ?

This will also match 0022.PSI for instance

I agree with you, it's much better tagging SIP/ string first and then taking the chars until \r\n is found.

catenacyber · 2023-11-27T11:01:41Z

rust/src/sip/sip.rs

@@ -518,6 +518,59 @@ pub unsafe extern "C" fn rs_sip_parse_response_tcp(
    state.parse_response_tcp(flow, stream_slice)
 }

+fn register_pattern_probe(proto: u8) -> i8 {
+    let methods: Vec<&str> = vec![
+        "REGISTER\0",


I do not think you need the final 0 in rust, do you ?

if you look at register_pattern_probe in rust/src/smb/smb.rs you'll see that the patterns contains \0.

Looks that SMB can be optimized as well then. We just use a buffer and its length, and you specify the - 1 for the length to get rid of this 0, right ?

Yes, it's right.

It seems that the same logic is applied in bittorrent_dht.rs

const BITTORRENT_DHT_PAYLOAD_PREFIX: &[u8] = b"d1:ad2:id20:\0"; if AppLayerProtoDetectPMRegisterPatternCS( IPPROTO_UDP, ALPROTO_BITTORRENT_DHT, BITTORRENT_DHT_PAYLOAD_PREFIX.as_ptr() as *const c_char, BITTORRENT_DHT_PAYLOAD_PREFIX.len() as u16 - 1, 0, crate::core::Direction::ToServer.into(), ) < 0 { SCLogDebug!("Failed to register protocol detection pattern for direction TOSERVER"); };

https://redmine.openinfosecfoundation.org/issues/6583 ;-)

I've made some tests, and it looks like the null character is mandatory; otherwise, the pattern matching won't work.

Patterns (such as BITTORRENT_DHT_PAYLOAD_PREFIX) are used in C functions, like DetectContentDataParse , that takes a string as input.

I believe it's not worth making changes around the code just to remove the null byte from the patterns.

catenacyber · 2023-11-27T11:02:27Z

rust/src/sip/sip.rs

+    unsafe {
+        for method in methods {
+            let depth = (method.len() - 1) as u16;
+            r |= AppLayerProtoDetectPMRegisterPatternCSwPP(


do we really need probing after this pattern ?

Plus this commit should reference https://redmine.openinfosecfoundation.org/issues/5047 right ?

Well, I think that probing it's not needed at this point and yes, the commit should point to the redmine ticket. (missed it.)

So, I think it is rather AppLayerProtoDetectPMRegisterPatternCS to be used

catenacyber · 2023-11-27T11:03:51Z

doc/userguide/upgrade.rst

+  Also, SIP protocol is detected using pattern matching and not only
+  probing parser.
+- ``SIP_PORTS`` variable has been introduced in suricata.yaml
+- Application layer's ``sip`` counter has been split into ``sip_tcp`` and ``sip_udp``.


this should mention that this is for the stats events.
cc @jufajardini

glongo · 2023-11-27T13:42:15Z

SIP version characters are enforced to strictly match SIP/2.0

Why so ? I think we want to match on past and future versions as well...

IIRC SIP/2.0 is the only version so far, but it's better to tag and parse it as we already said

glongo · 2023-11-28T13:43:54Z

Replaced by #9909

glongo added 8 commits November 24, 2023 23:29

rust/sip: rustfmt sip module

5451fdb

sip/parser: accept valid chars

961a62d

Accepts valid characters as defined in RFC3261.

rust/sip: register parser for tcp

a4f7db1

This patch lets the parser to work over tcp protocol, taking care of handling data before calling the request/response parsers. Ticket OISF#3351.

rust/sip: add direction to transaction

c49e208

This patch permits to set a direction when a new transaction is created in order to avoid 'signature shadowing' as reported by Eric Leblond in commit 5aaf507

suricata.yaml: define SIP_PORTS

96b34a7

rust/sip: register pattern matching

fbb6c7a

This permits to detect SIP protocol using pattern matching as well.

doc: add upgrade section for 8

769de02

glongo requested review from jasonish, jufajardini, victorjulien and a team as code owners November 25, 2023 16:18

This was referenced Nov 25, 2023

rust/sip: register parser for tcp v8 #9880

Closed

sip: add tests for sip over tcp v3 OISF/suricata-verify#1494

Closed

catenacyber reviewed Nov 27, 2023

View reviewed changes

glongo closed this Nov 28, 2023

catenacyber mentioned this pull request Nov 29, 2023

rust/sip: register parser for tcp v10 #9909

Closed

3 tasks

glongo deleted the dev-sip-tcp-v9 branch February 28, 2024 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rust/sip: register parser for tcp v9 #9893

rust/sip: register parser for tcp v9 #9893

glongo commented Nov 25, 2023

codecov bot commented Nov 25, 2023

catenacyber commented Nov 27, 2023

catenacyber Nov 27, 2023

catenacyber Nov 27, 2023

glongo Nov 27, 2023

catenacyber Nov 27, 2023

glongo Nov 27, 2023

catenacyber Nov 27, 2023

glongo Nov 27, 2023

glongo Nov 27, 2023

catenacyber Nov 27, 2023

glongo Nov 27, 2023

catenacyber Nov 27, 2023

glongo Nov 27, 2023

catenacyber Nov 27, 2023

catenacyber Nov 27, 2023

jufajardini Nov 27, 2023

glongo commented Nov 27, 2023

glongo commented Nov 28, 2023

rust/sip: register parser for tcp v9 #9893

rust/sip: register parser for tcp v9 #9893

Conversation

glongo commented Nov 25, 2023

Provide values to any of the below to override the defaults.

codecov bot commented Nov 25, 2023

Codecov Report

catenacyber commented Nov 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glongo commented Nov 27, 2023

glongo commented Nov 28, 2023