From 990912b1d8b18ac1f11114db4e21b0300b884b6e Mon Sep 17 00:00:00 2001
From: Dean Balandin <dean.metoh@gmail.com>
Date: Tue, 27 Jun 2023 12:40:37 +0000
Subject: [PATCH 001/462] stream: decouple stream.bypass dependency from tls
 bypass

Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.
---
 src/stream-tcp.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/src/stream-tcp.c b/src/stream-tcp.c
index 99dcd299530c..c97a23f2f91d 100644
--- a/src/stream-tcp.c
+++ b/src/stream-tcp.c
@@ -5472,17 +5472,13 @@ int StreamTcpPacket (ThreadVars *tv, Packet *p, StreamTcpThread *stt,
         }
 
         if (ssn->flags & STREAMTCP_FLAG_BYPASS) {
-            /* we can call bypass callback, if enabled */
-            if (StreamTcpBypassEnabled()) {
-                PacketBypassCallback(p);
-            }
-
-        /* if stream is dead and we have no detect engine at all, bypass. */
+            PacketBypassCallback(p);
         } else if (g_detect_disabled &&
                 (ssn->client.flags & STREAMTCP_STREAM_FLAG_NOREASSEMBLY) &&
                 (ssn->server.flags & STREAMTCP_STREAM_FLAG_NOREASSEMBLY) &&
                 StreamTcpBypassEnabled())
         {
+            /* if stream is dead and we have no detect engine at all, bypass. */
             SCLogDebug("bypass as stream is dead and we have no rules");
             PacketBypassCallback(p);
         }

From 6107332c4cf8ee3dc965b2cc0424d6b04e5fc2bc Mon Sep 17 00:00:00 2001
From: Dean Balandin <dean.metoh@gmail.com>
Date: Tue, 27 Jun 2023 12:53:36 +0000
Subject: [PATCH 002/462] userguide: update encrypted traffic bypass

Update documentation to reflect the new features and changes.
---
 doc/userguide/configuration/suricata-yaml.rst | 24 +++++++++----------
 .../performance/ignoring-traffic.rst          |  7 +++---
 2 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index bde02112b179..f22dbbec1481 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -1657,13 +1657,13 @@ as raw ``content`` inspection will still be disabled. There is no point in doing
 pattern matching on traffic known to be encrypted. Inspection for (encrypted)
 Heartbleed and other protocol anomalies still happens.
 
-When ``encryption-handling`` is set to ``bypass``, all processing of this session is
-stopped. No further parsing and inspection happens. If ``stream.bypass`` is enabled
-this will lead to the flow being bypassed, either inside Suricata or by the
-capture method if it supports it and is configured for it.
+When ``encryption-handling`` is set to ``bypass``, all processing of this
+session is stopped. No further parsing and inspection happens. This will also
+lead to the flow being bypassed, either inside Suricata or by the capture method
+if it supports it and is configured for it.
 
-Finally, if ``encryption-handling`` is set to ``full``, Suricata will process the
-flow as normal, without inspection limitations or bypass.
+Finally, if ``encryption-handling`` is set to ``full``, Suricata will process
+the flow as normal, without inspection limitations or bypass.
 
 The option has replaced the ``no-reassemble`` option. If ``no-reassemble`` is
 present, and ``encryption-handling`` is not, ``false`` is interpreted as
@@ -1989,12 +1989,12 @@ are typically provided through the command line, are contained in the node
 parameters. There are two ways to specify arguments: lengthy and short.
 Dashes are omitted when describing the arguments. This setup node can be
 used to set up the memory configuration, accessible NICs, and other EAL-related
-parameters, among other things. The node `dpdk.eal-params` also supports 
-multiple arguments of the same type. This can be useful for EAL arguments 
-such as `--vdev`, `--allow`, or `--block`. Values for these EAL arguments 
-are specified as a comma-separated list. 
-An example of such usage can be found in the example above where the `allow` 
-argument only makes `0000:3b:00.0` and `0000:3b:00.1` accessible to Suricata. 
+parameters, among other things. The node `dpdk.eal-params` also supports
+multiple arguments of the same type. This can be useful for EAL arguments
+such as `--vdev`, `--allow`, or `--block`. Values for these EAL arguments
+are specified as a comma-separated list.
+An example of such usage can be found in the example above where the `allow`
+argument only makes `0000:3b:00.0` and `0000:3b:00.1` accessible to Suricata.
 arguments with list node. such as --vdev, --allow, --block eal options.
 The definition of lcore affinity as an EAL
 parameter is a standard practice. However, lcore parameters like `-l`, `-c`,
diff --git a/doc/userguide/performance/ignoring-traffic.rst b/doc/userguide/performance/ignoring-traffic.rst
index a2c7a8825528..712d1ff0b1e1 100644
--- a/doc/userguide/performance/ignoring-traffic.rst
+++ b/doc/userguide/performance/ignoring-traffic.rst
@@ -74,9 +74,10 @@ encrypted traffic
 -----------------
 
 The TLS app layer parser has the ability to stop processing encrypted traffic
-after the initial handshake. By setting the `app-layer.protocols.tls.encryption-handling`
-option to `bypass` the rest of this flow is ignored. If flow bypass is enabled,
-the bypass is done in the kernel or in hardware.
+after the initial handshake. By setting the
+`app-layer.protocols.tls.encryption-handling` option to `bypass` the rest of
+this flow is ignored. The bypass is done in the kernel or in hardware, similar
+to how flow bypass is done.
 
 .. _bypass:
 

From a284f01c1ddc39e9710f9992322bafda6d3e9f1d Mon Sep 17 00:00:00 2001
From: Thomas Winter <Thomas.Winter@alliedtelesis.co.nz>
Date: Mon, 15 May 2023 14:18:47 +1200
Subject: [PATCH 003/462] iprep: fix parsing ip-rep data with carriage return

Commit e7c0f0ad91fd removed uses of atoi with a new number parsing
functions. This broke parsing ip-reputation data files that contained
trailing carriage returns as it was being included in the number
string to convert.

Bug: #6243.
---
 src/reputation.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/reputation.c b/src/reputation.c
index b9f2186d0100..75f3ba0c3fa4 100644
--- a/src/reputation.c
+++ b/src/reputation.c
@@ -282,7 +282,8 @@ static int SRepSplitLine(SRepCIDRTree *cidr_ctx, char *line, Address *ip, uint8_
     char *origline = line;
 
     while (i < (int)line_len) {
-        if (line[i] == ',' || line[i] == '\n' || line[i] == '\0' || i == (int)(line_len - 1)) {
+        if (line[i] == ',' || line[i] == '\n' || line[i] == '\r' || line[i] == '\0' ||
+                i == (int)(line_len - 1)) {
             line[i] = '\0';
 
             ptrs[idx] = line;

From b235e85c689254a4ad9b930d6562e3f2167f91ff Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 30 Aug 2023 11:24:24 +0200
Subject: [PATCH 004/462] rust: fix clippy warnings for version 1.72.0

Includes using the right prototype for C SRepCatGetByShortname
---
 rust/src/applayer.rs         | 2 +-
 rust/src/detect/byte_math.rs | 4 ++--
 rust/src/detect/iprep.rs     | 5 +++--
 rust/src/ffi/base64.rs       | 2 +-
 rust/src/pgsql/parser.rs     | 2 +-
 5 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/rust/src/applayer.rs b/rust/src/applayer.rs
index 33fa83a92d00..255fa1593c2b 100644
--- a/rust/src/applayer.rs
+++ b/rust/src/applayer.rs
@@ -44,7 +44,7 @@ impl StreamSlice {
     #[cfg(test)]
     pub fn from_slice(slice: &[u8], flags: u8, offset: u64) -> Self {
         Self {
-            input: slice.as_ptr() as *const u8,
+            input: slice.as_ptr(),
             input_len: slice.len() as u32,
             flags,
             offset
diff --git a/rust/src/detect/byte_math.rs b/rust/src/detect/byte_math.rs
index 0cc60e52bfd7..80bd3d5ee178 100644
--- a/rust/src/detect/byte_math.rs
+++ b/rust/src/detect/byte_math.rs
@@ -432,7 +432,7 @@ pub unsafe extern "C" fn ScByteMathParse(c_arg: *const c_char) -> *mut DetectByt
         }
     };
     match parse_bytemath(arg) {
-        Ok((_, detect)) => return Box::into_raw(Box::new(detect)) as *mut DetectByteMathData,
+        Ok((_, detect)) => return Box::into_raw(Box::new(detect)),
         Err(_) => return std::ptr::null_mut(),
     }
 }
@@ -440,7 +440,7 @@ pub unsafe extern "C" fn ScByteMathParse(c_arg: *const c_char) -> *mut DetectByt
 #[no_mangle]
 pub unsafe extern "C" fn ScByteMathFree(ptr: *mut DetectByteMathData) {
     if !ptr.is_null() {
-        let _ = Box::from_raw(ptr as *mut DetectByteMathData);
+        let _ = Box::from_raw(ptr);
     }
 }
 
diff --git a/rust/src/detect/iprep.rs b/rust/src/detect/iprep.rs
index 4018ea97a45e..16f5d9d5d15e 100644
--- a/rust/src/detect/iprep.rs
+++ b/rust/src/detect/iprep.rs
@@ -24,6 +24,7 @@ use nom7::Err;
 use nom7::IResult;
 
 use std::ffi::{CStr, CString};
+use std::os::raw::c_char;
 use std::str::FromStr;
 
 #[repr(u8)]
@@ -71,7 +72,7 @@ pub fn is_alphanumeric_or_slash(chr: char) -> bool {
 }
 
 extern "C" {
-    pub fn SRepCatGetByShortname(name: *const i8) -> u8;
+    pub fn SRepCatGetByShortname(name: *const c_char) -> u8;
 }
 
 pub fn detect_parse_iprep(i: &str) -> IResult<&str, DetectIPRepData> {
@@ -84,7 +85,7 @@ pub fn detect_parse_iprep(i: &str) -> IResult<&str, DetectIPRepData> {
     let (i, name) = take_while(is_alphanumeric_or_slash)(i)?;
     // copy as to have final zero
     let namez = CString::new(name).unwrap();
-    let cat = unsafe { SRepCatGetByShortname(namez.as_ptr() as *const i8) };
+    let cat = unsafe { SRepCatGetByShortname(namez.as_ptr()) };
     if cat == 0 {
         return Err(Err::Error(make_error(i, ErrorKind::MapOpt)));
     }
diff --git a/rust/src/ffi/base64.rs b/rust/src/ffi/base64.rs
index 0019a6ff2b6f..ea72a344c393 100644
--- a/rust/src/ffi/base64.rs
+++ b/rust/src/ffi/base64.rs
@@ -46,7 +46,7 @@ pub unsafe extern "C" fn Base64Encode(
     if encoded.len() + 1 > *output_len as usize {
         return Base64ReturnCode::SC_BASE64_OVERFLOW;
     }
-    let output = std::slice::from_raw_parts_mut(&mut *(output as *mut u8), *output_len as usize);
+    let output = std::slice::from_raw_parts_mut(&mut *output, *output_len as usize);
     output[0..encoded.len()].copy_from_slice(encoded.as_bytes());
     output[encoded.len()] = 0;
     *output_len = encoded.len() as c_ulong;
diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index bb1a9ea09e35..ae07d5d5a078 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -593,7 +593,7 @@ pub fn pgsql_parse_startup_packet(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, b) = take(len - PGSQL_LENGTH_FIELD)(i)?;
     let (_, message) =
         match proto_major {
-            1 | 2 | 3 => {
+            1..=3 => {
                 let (b, proto_major) = be_u16(b)?;
                 let (b, proto_minor) = be_u16(b)?;
                 let (b, params) = pgsql_parse_startup_parameters(b)?;

From 5bdbc1a313a8489a012ec1a50859bdd58a899067 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 30 Aug 2023 11:43:07 +0200
Subject: [PATCH 005/462] rdp: do not use zero-bit bitflag

cf https://docs.rs/bitflags/latest/bitflags/#zero-bit-flags

As warned by clippy 1.72.0
---
 rust/src/rdp/parser.rs | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/rust/src/rdp/parser.rs b/rust/src/rdp/parser.rs
index 604d10a19c4d..a8004e290b96 100644
--- a/rust/src/rdp/parser.rs
+++ b/rust/src/rdp/parser.rs
@@ -160,7 +160,8 @@ pub enum Protocol {
 // rdp-spec, section 2.2.1.1.1
 bitflags! {
     pub struct ProtocolFlags: u32 {
-        const PROTOCOL_RDP = Protocol::ProtocolRdp as u32;
+        //Protocol::ProtocolRdp is 0 as always supported
+        //and bitflags crate does not like zero-bit flags
         const PROTOCOL_SSL = Protocol::ProtocolSsl as u32;
         const PROTOCOL_HYBRID = Protocol::ProtocolHybrid as u32;
         const PROTOCOL_RDSTLS = Protocol::ProtocolRdsTls as u32;
@@ -1089,7 +1090,7 @@ mod tests_negotiate_49350 {
                 cookie: None,
                 negotiation_request: Some(NegotiationRequest {
                     flags: NegotiationRequestFlags::empty(),
-                    protocols: ProtocolFlags::PROTOCOL_RDP,
+                    protocols: ProtocolFlags { bits: Protocol::ProtocolRdp as u32 },
                 }),
                 data: Vec::new(),
             }),
@@ -1179,7 +1180,7 @@ mod tests_core_49350 {
             ),
             client_dig_product_id: Some(String::from("")),
             connection_hint: Some(ConnectionHint::ConnectionHintNotProvided),
-            server_selected_protocol: Some(ProtocolFlags::PROTOCOL_RDP),
+            server_selected_protocol: Some(ProtocolFlags { bits: Protocol::ProtocolRdp as u32 }),
             desktop_physical_width: None,
             desktop_physical_height: None,
             desktop_orientation: None,

From b67ff4badf02ee72e40ec193c9da4d99207fef7f Mon Sep 17 00:00:00 2001
From: Yatin Kanetkar <yatin.kanetkar@gmail.com>
Date: Sat, 19 Aug 2023 13:10:33 -0400
Subject: [PATCH 006/462] dhcp: Log Vendor Client Identifier (dhcp option 60)

* Log vendor client identifier (dhcp option 60) if extended dhcp
logging is turned on. This required the `vendor_client_identifier` to
be added to the json schema. Validation done using an SV Test
* Added `requested_ip` to the json schema as well, since it was
missed. My SV test failed without it.

Feature #4587
---
 etc/schema.json         | 6 ++++++
 rust/src/dhcp/dhcp.rs   | 1 +
 rust/src/dhcp/logger.rs | 6 ++++++
 3 files changed, 13 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index 1b49cf5af1fc..efd17092f68c 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -560,12 +560,18 @@
                 "renewal_time": {
                     "type": "integer"
                 },
+                "requested_ip":{
+                    "type": "string"
+                },
                 "subnet_mask": {
                     "type": "string"
                 },
                 "type": {
                     "type": "string"
                 },
+                "vendor_class_identifier":{
+                    "type": "string"
+                },
                 "dns_servers": {
                     "type": "array",
                     "minItems": 1,
diff --git a/rust/src/dhcp/dhcp.rs b/rust/src/dhcp/dhcp.rs
index 5afa1efb376d..b69b675b8ce9 100644
--- a/rust/src/dhcp/dhcp.rs
+++ b/rust/src/dhcp/dhcp.rs
@@ -42,6 +42,7 @@ pub const DHCP_OPT_TYPE: u8 = 53;
 pub const DHCP_OPT_PARAMETER_LIST: u8 = 55;
 pub const DHCP_OPT_RENEWAL_TIME: u8 = 58;
 pub const DHCP_OPT_REBINDING_TIME: u8 = 59;
+pub const DHCP_OPT_VENDOR_CLASS_ID: u8 = 60;
 pub const DHCP_OPT_CLIENT_ID: u8 = 61;
 pub const DHCP_OPT_END: u8 = 255;
 
diff --git a/rust/src/dhcp/logger.rs b/rust/src/dhcp/logger.rs
index 8423064946cb..b29e2158ef95 100644
--- a/rust/src/dhcp/logger.rs
+++ b/rust/src/dhcp/logger.rs
@@ -168,6 +168,12 @@ impl DHCPLogger {
                                 self.log_opt_routers(js, option)?;
                             }
                         }
+                        DHCP_OPT_VENDOR_CLASS_ID => {
+                            if self.extended && !option.data.is_empty(){
+                                js.set_string_from_bytes("vendor_class_identifier",
+                                                         &option.data)?;
+                            }
+                        }
                         _ => {}
                     }
                 }

From 541cafa40a1a6a7d183f9105f61160c986e2e3e0 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 30 Aug 2023 14:48:56 +0200
Subject: [PATCH 007/462] config/flow: fix division by zero

Fixes: 805b07fa4236 ("src: checks to avoid divisions by zero")

Coverity id: 1539152

Ticket: #5920
Ticket: #6255
---
 src/flow.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/flow.c b/src/flow.c
index 891a5fb9b94d..9783b7883b0b 100644
--- a/src/flow.c
+++ b/src/flow.c
@@ -606,10 +606,11 @@ void FlowInitConfig(bool quiet)
             FatalError("Invalid value for flow.hash-size: NULL");
         }
 
-        if (StringParseUint32(&configval, 10, strlen(conf_val), conf_val) > 0 || configval == 0) {
+        if (StringParseUint32(&configval, 10, strlen(conf_val), conf_val) && configval != 0) {
             flow_config.hash_size = configval;
         } else {
-            FatalError("Invalid value for flow.hash-size");
+            FatalError("Invalid value for flow.hash-size. Must be a numeric value in the range "
+                       "1-4294967295");
         }
     }
     if ((ConfGet("flow.prealloc", &conf_val)) == 1)

From 783d07007f9845c1040f6ecff555732d8fa4da4e Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 3 Aug 2023 13:03:41 +0000
Subject: [PATCH 008/462] napatech: fix compilation errors in SCLog calls

Since f8474344cdd00e3d128ffc3ec6d7e465bbe2894d, there is an extra
argument to SCLog which indicates the module and subsystem
identifier.  The Napatech vendor code is missing this argument,
which is fixed here.
---
 src/source-napatech.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index b3d969f8fe13..db6c61fc68ce 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -740,23 +740,23 @@ static void RecommendNUMAConfig(SCLogLevel log_level)
     }
 
     if (set_cpu_affinity) {
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__,
+        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module,
                 "Minimum host buffers that should be defined in ntservice.ini:");
 
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, "   NUMA Node 0: %d",
+        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 0: %d",
                 (SC_ATOMIC_GET(numa0_count)));
 
         if (numa_max_node() >= 1)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__,
-                    "   NUMA Node 1: %d ", (SC_ATOMIC_GET(numa1_count)));
+            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 1: %d ",
+                    (SC_ATOMIC_GET(numa1_count)));
 
         if (numa_max_node() >= 2)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__,
-                    "   NUMA Node 2: %d ", (SC_ATOMIC_GET(numa2_count)));
+            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 2: %d ",
+                    (SC_ATOMIC_GET(numa2_count)));
 
         if (numa_max_node() >= 3)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__,
-                    "   NUMA Node 3: %d ", (SC_ATOMIC_GET(numa3_count)));
+            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 3: %d ",
+                    (SC_ATOMIC_GET(numa3_count)));
 
         snprintf(string0, 16, "[%d, 16, 0]", SC_ATOMIC_GET(numa0_count));
         snprintf(string1, 16, (numa_max_node() >= 1 ? ",[%d, 16, 1]" : ""),
@@ -766,9 +766,8 @@ static void RecommendNUMAConfig(SCLogLevel log_level)
         snprintf(string3, 16, (numa_max_node() >= 3 ? ",[%d, 16, 3]" : ""),
                 SC_ATOMIC_GET(numa3_count));
 
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__,
-                "E.g.: HostBuffersRx=%s%s%s%s", string0, string1, string2,
-                string3);
+        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module,
+                "E.g.: HostBuffersRx=%s%s%s%s", string0, string1, string2, string3);
     } else if (log_level == SC_LOG_ERROR) {
         SCLogError("Or, try running /opt/napatech3/bin/ntpl -e \"delete=all\" to clean-up stream "
                    "NUMA config.");

From c6da59d57ba2725be1b7d33632d77b3fbd2850a3 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 3 Aug 2023 12:18:05 +0000
Subject: [PATCH 009/462] napatech: fix missing header includes

---
 src/source-napatech.c | 2 ++
 src/util-napatech.c   | 1 +
 2 files changed, 3 insertions(+)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index db6c61fc68ce..d2fab8add5e8 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -26,6 +26,7 @@
  *
  */
 #include "suricata-common.h"
+#include "action-globals.h"
 #include "decode.h"
 #include "packet.h"
 #include "suricata.h"
@@ -39,6 +40,7 @@
 #include "tmqh-packetpool.h"
 #include "util-napatech.h"
 #include "source-napatech.h"
+#include "runmode-napatech.h"
 
 #ifndef HAVE_NAPATECH
 
diff --git a/src/util-napatech.c b/src/util-napatech.c
index c8e5e3658058..b23bd1316e7b 100644
--- a/src/util-napatech.c
+++ b/src/util-napatech.c
@@ -33,6 +33,7 @@
 #include "tm-threads.h"
 #include "util-napatech.h"
 #include "source-napatech.h"
+#include "runmode-napatech.h"
 
 #ifdef NAPATECH_ENABLE_BYPASS
 

From 658bbbc078a688faa0820d13790215dc09564576 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 3 Aug 2023 12:30:42 +0000
Subject: [PATCH 010/462] napatech: fix compilation with SCTIME usage

This replaces the broken compilation due to the change of SCTime_t
into a structure: 9fbe68364259ea71fcd0d22521afcaddefdc744d.
---
 src/source-napatech.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index d2fab8add5e8..6bbfc979bfb3 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -949,23 +949,19 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
          */
         switch (NT_NET_GET_PKT_TIMESTAMP_TYPE(packet_buffer)) {
             case NT_TIMESTAMP_TYPE_NATIVE_UNIX:
-                p->ts = SCTIME_FROM_SECS(pkt_ts / 100000000);
-                p->ts += SCTIME_FROM_USECS(
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts / 100000000),
                         ((pkt_ts % 100000000) / 100) + ((pkt_ts % 100) > 50 ? 1 : 0));
                 break;
             case NT_TIMESTAMP_TYPE_PCAP:
-                p->ts = SCTIME_FROM_SECS(pkt_ts >> 32);
-                p->ts += SCTIME_FROM_USECS(pkt_ts & 0xFFFFFFFF);
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts >> 32), pkt_ts & 0xFFFFFFFF);
                 break;
             case NT_TIMESTAMP_TYPE_PCAP_NANOTIME:
-                p->ts = SCTIME_FROM_SECS(pkt_ts >> 32);
-                p->ts += SCTIME_FROM_USECS(
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts >> 32),
                         ((pkt_ts & 0xFFFFFFFF) / 1000) + ((pkt_ts % 1000) > 500 ? 1 : 0));
                 break;
             case NT_TIMESTAMP_TYPE_NATIVE_NDIS:
                 /* number of seconds between 1/1/1601 and 1/1/1970 */
-                p->ts = SCTIME_FROM_SECS((pkt_ts / 100000000) - 11644473600);
-                p->ts += SCTIME_FROM_USECS(
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS((pkt_ts / 100000000) - 11644473600),
                         ((pkt_ts % 100000000) / 100) + ((pkt_ts % 100) > 50 ? 1 : 0));
                 break;
             default:

From 23e53865131e9c6a947e6f540359d95b4228ac81 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 3 Aug 2023 12:40:13 +0000
Subject: [PATCH 011/462] napatech: fix thread flags with THV_RUNNING

This update the Napatech vendor module with changes introduced in
13beba141c98debc4d7e29081c91a799362f19fb that introduces THV_RUNNING.
---
 src/util-napatech.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/src/util-napatech.c b/src/util-napatech.c
index b23bd1316e7b..74ff82fda714 100644
--- a/src/util-napatech.c
+++ b/src/util-napatech.c
@@ -643,7 +643,7 @@ static void *NapatechStatsLoop(void *arg)
                      "active streams.");
     }
 
-    TmThreadsSetFlag(tv, THV_INIT_DONE);
+    TmThreadsSetFlag(tv, THV_INIT_DONE | THV_RUNNING);
     while (1) {
         if (TmThreadsCheckFlag(tv, THV_KILL)) {
             SCLogDebug("NapatechStatsLoop THV_KILL detected");
@@ -1027,7 +1027,7 @@ static void *NapatechBufMonitorLoop(void *arg)
         exit(EXIT_FAILURE);
     }
 
-    TmThreadsSetFlag(tv, THV_INIT_DONE);
+    TmThreadsSetFlag(tv, THV_INIT_DONE | THV_RUNNING);
     while (1) {
         if (TmThreadsCheckFlag(tv, THV_KILL)) {
             SCLogDebug("NapatechBufMonitorLoop THV_KILL detected");
@@ -1217,7 +1217,6 @@ void NapatechStartStats(void)
         FatalError("Failed to spawn thread for NapatechBufMonitor - Killing engine.");
     }
 
-
     return;
 }
 

From d7aa7a063ff004a35a116cb39e47bc18dd378e5a Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 3 Aug 2023 15:32:10 +0200
Subject: [PATCH 012/462] napatech: fix warnings with ByteExtractStringUint8

The WARN_UNUSED attribute has been added to ByteExtractStringUint8
in commit 698816811406572c443ca1e95c309d292f489376.  The return
value is now handled and appropriate errors printed.
---
 src/util-napatech.c | 30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

diff --git a/src/util-napatech.c b/src/util-napatech.c
index 74ff82fda714..e8c842fabc4c 100644
--- a/src/util-napatech.c
+++ b/src/util-napatech.c
@@ -1454,8 +1454,16 @@ uint32_t NapatechSetupTraffic(uint32_t first_stream, uint32_t last_stream)
             if (strchr(port->val, '-')) {
                 stream_spec = CONFIG_SPECIFIER_RANGE;
 
-                ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val);
-                ByteExtractStringUint8(&ports_spec.second[iteration], 10, 0, strchr(port->val, '-')+1);
+                if (ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val) == -1) {
+                    FatalError("Invalid value '%s' in napatech.ports specification in conf file.",
+                            port->val);
+                }
+
+                if (ByteExtractStringUint8(&ports_spec.second[iteration], 10, 0,
+                            strchr(port->val, '-') + 1) == -1) {
+                    FatalError("Invalid value '%s' in napatech.ports specification in conf file.",
+                            port->val);
+                }
 
                 if (ports_spec.first[iteration] == ports_spec.second[iteration]) {
                     if (is_inline) {
@@ -1533,8 +1541,17 @@ uint32_t NapatechSetupTraffic(uint32_t first_stream, uint32_t last_stream)
                 }
                 stream_spec = CONFIG_SPECIFIER_RANGE;
 
-                ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val);
-                ByteExtractStringUint8(&ports_spec.second[iteration], 10, 0, strchr(port->val, '-') + 1);
+                if (ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val) == -1) {
+                    FatalError("Invalid value '%s' in napatech.ports specification in conf file.",
+                            port->val);
+                }
+
+                if (ByteExtractStringUint8(&ports_spec.second[iteration], 10, 0,
+                            strchr(port->val, '-') + 1) == -1) {
+                    FatalError("Invalid value '%s' in napatech.ports specification in conf file.",
+                            port->val);
+                }
+
                 snprintf(ports_spec.str, sizeof (ports_spec.str), "(%d..%d)", ports_spec.first[iteration], ports_spec.second[iteration]);
             } else {
                 /* check that the sting in the config file is correctly specified */
@@ -1544,7 +1561,10 @@ uint32_t NapatechSetupTraffic(uint32_t first_stream, uint32_t last_stream)
                 }
                 stream_spec = CONFIG_SPECIFIER_INDIVIDUAL;
 
-                ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val);
+                if (ByteExtractStringUint8(&ports_spec.first[iteration], 10, 0, port->val) == -1) {
+                    FatalError("Invalid value '%s' in napatech.ports specification in conf file.",
+                            port->val);
+                }
 
                 /* Determine the ports to use on the NTPL assign statement*/
                 if (iteration == 0) {

From 95ecbd117832c62c099b9da5762c454f6540a7e2 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Mon, 14 Aug 2023 14:30:31 +0200
Subject: [PATCH 013/462] configure: move -lntapi to LIBS variable

Previously -lntapi was appended to LDFLAGS which did not work with
all build environments.
---
 configure.ac | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 21304786cfc2..1fba3ebeefb9 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1826,7 +1826,8 @@
 
     if test "$enable_napatech" = "yes"; then
         CPPFLAGS="${CPPFLAGS} -I${with_napatech_includes}"
-        LDFLAGS="${LDFLAGS} -L${with_napatech_libraries} -lntapi"
+        LDFLAGS="${LDFLAGS} -L${with_napatech_libraries}"
+        LIBS="${LIBS} -lntapi"
         AC_CHECK_HEADER(nt.h,NAPATECH="yes",NAPATECH="no")
         if test "$NAPATECH" != "no"; then
             NAPATECH=""

From 8c1ccc1cfe39717500a1e75400a9a0570e6d14df Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Tue, 15 Aug 2023 10:24:05 +0200
Subject: [PATCH 014/462] napatech: fix shadowed global is_inline warning

---
 src/source-napatech.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index 6bbfc979bfb3..6c035ad7bf2b 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -400,7 +400,7 @@ static NtFlowStream_t InitFlowStream(int adapter, int stream_id)
  * \return Error code indicating success (1) or failure (0).
  *
  */
-static int ProgramFlow(Packet *p, int is_inline)
+static int ProgramFlow(Packet *p, int inline_mode)
 {
     NtFlow_t flow_match;
     memset(&flow_match, 0, sizeof(flow_match));
@@ -586,7 +586,7 @@ static int ProgramFlow(Packet *p, int is_inline)
     if (PacketCheckAction(p, ACTION_DROP)) {
         flow_match.keySetId = NAPATECH_FLOWTYPE_DROP;
     } else {
-        if (is_inline) {
+        if (inline_mode) {
             flow_match.keySetId = NAPATECH_FLOWTYPE_PASS;
         } else {
             flow_match.keySetId = NAPATECH_FLOWTYPE_DROP;

From c4059a7f02bdc4254affa228f3dba853c557b831 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Mon, 14 Aug 2023 12:03:41 +0200
Subject: [PATCH 015/462] napatech: generalise numa config recommending

Previous implementation hardcoded up to 4 NUMA nodes.
We support arbitrary number of NUMA nodes now.

Note that this commit also removes the old SCLog logging
calls.  But since the logic has changed, these have been replaced
directly with new code.
---
 src/source-napatech.c | 147 ++++++++++++++++++++----------------------
 1 file changed, 71 insertions(+), 76 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index 6c035ad7bf2b..f646c42030d1 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -115,10 +115,11 @@ SC_ATOMIC_DECLARE(uint16_t, total_tallied);
  * are running*/
 SC_ATOMIC_DECLARE(uint16_t, stream_count);
 
-SC_ATOMIC_DECLARE(uint16_t, numa0_count);
-SC_ATOMIC_DECLARE(uint16_t, numa1_count);
-SC_ATOMIC_DECLARE(uint16_t, numa2_count);
-SC_ATOMIC_DECLARE(uint16_t, numa3_count);
+typedef struct NapatechNumaDetect_ {
+    SC_ATOMIC_DECLARE(uint16_t, count);
+} NapatechNumaDetect;
+
+NapatechNumaDetect *numa_detect = NULL;
 
 SC_ATOMIC_DECLARE(uint64_t, flow_callback_cnt);
 SC_ATOMIC_DECLARE(uint64_t, flow_callback_handled_pkts);
@@ -127,35 +128,63 @@ SC_ATOMIC_DECLARE(uint64_t, flow_callback_tcp_pkts);
 SC_ATOMIC_DECLARE(uint64_t, flow_callback_unhandled_pkts);
 
 /**
- * \brief Register the Napatech  receiver (reader) module.
+ * \brief Initialize the Napatech receiver (reader) module for globals.
  */
-void TmModuleNapatechStreamRegister(void)
+static TmEcode NapatechStreamInit(void)
 {
-    tmm_modules[TMM_RECEIVENAPATECH].name = "NapatechStream";
-    tmm_modules[TMM_RECEIVENAPATECH].ThreadInit = NapatechStreamThreadInit;
-    tmm_modules[TMM_RECEIVENAPATECH].Func = NULL;
-    tmm_modules[TMM_RECEIVENAPATECH].PktAcqLoop = NapatechPacketLoop;
-    tmm_modules[TMM_RECEIVENAPATECH].PktAcqBreakLoop = NULL;
-    tmm_modules[TMM_RECEIVENAPATECH].ThreadExitPrintStats = NapatechStreamThreadExitStats;
-    tmm_modules[TMM_RECEIVENAPATECH].ThreadDeinit = NapatechStreamThreadDeinit;
-    tmm_modules[TMM_RECEIVENAPATECH].cap_flags = SC_CAP_NET_RAW;
-    tmm_modules[TMM_RECEIVENAPATECH].flags = TM_FLAG_RECEIVE_TM;
+    int i;
 
     SC_ATOMIC_INIT(total_packets);
     SC_ATOMIC_INIT(total_drops);
     SC_ATOMIC_INIT(total_tallied);
     SC_ATOMIC_INIT(stream_count);
 
-    SC_ATOMIC_INIT(numa0_count);
-    SC_ATOMIC_INIT(numa1_count);
-    SC_ATOMIC_INIT(numa2_count);
-    SC_ATOMIC_INIT(numa3_count);
+    numa_detect = SCMalloc(sizeof(*numa_detect) * (numa_max_node() + 1));
+    if (numa_detect == NULL) {
+        FatalError("Failed to allocate memory for numa detection array: %s", strerror(errno));
+    }
+
+    for (i = 0; i <= numa_max_node(); ++i) {
+        SC_ATOMIC_INIT(numa_detect[i].count);
+    }
 
     SC_ATOMIC_INIT(flow_callback_cnt);
     SC_ATOMIC_INIT(flow_callback_handled_pkts);
     SC_ATOMIC_INIT(flow_callback_udp_pkts);
     SC_ATOMIC_INIT(flow_callback_tcp_pkts);
     SC_ATOMIC_INIT(flow_callback_unhandled_pkts);
+
+    return TM_ECODE_OK;
+}
+
+/**
+ * \brief Deinitialize the Napatech receiver (reader) module for globals.
+ */
+static TmEcode NapatechStreamDeInit(void)
+{
+    if (numa_detect != NULL) {
+        SCFree(numa_detect);
+    }
+
+    return TM_ECODE_OK;
+}
+
+/**
+ * \brief Register the Napatech  receiver (reader) module.
+ */
+void TmModuleNapatechStreamRegister(void)
+{
+    tmm_modules[TMM_RECEIVENAPATECH].name = "NapatechStream";
+    tmm_modules[TMM_RECEIVENAPATECH].ThreadInit = NapatechStreamThreadInit;
+    tmm_modules[TMM_RECEIVENAPATECH].Func = NULL;
+    tmm_modules[TMM_RECEIVENAPATECH].PktAcqLoop = NapatechPacketLoop;
+    tmm_modules[TMM_RECEIVENAPATECH].PktAcqBreakLoop = NULL;
+    tmm_modules[TMM_RECEIVENAPATECH].ThreadExitPrintStats = NapatechStreamThreadExitStats;
+    tmm_modules[TMM_RECEIVENAPATECH].ThreadDeinit = NapatechStreamThreadDeinit;
+    tmm_modules[TMM_RECEIVENAPATECH].cap_flags = SC_CAP_NET_RAW;
+    tmm_modules[TMM_RECEIVENAPATECH].flags = TM_FLAG_RECEIVE_TM;
+    tmm_modules[TMM_RECEIVENAPATECH].Init = NapatechStreamInit;
+    tmm_modules[TMM_RECEIVENAPATECH].DeInit = NapatechStreamDeInit;
 }
 
 /**
@@ -729,51 +758,31 @@ static int GetNumaNode(void)
  * \param log_level of the currently running instance.
  *
  */
-static void RecommendNUMAConfig(SCLogLevel log_level)
+static void RecommendNUMAConfig(void)
 {
-    char string0[16];
-    char string1[16];
-    char string2[16];
-    char string3[16];
+    char *buffer, *p;
     int set_cpu_affinity = 0;
 
+    p = buffer = SCCalloc(sizeof(char), (32 * (numa_max_node() + 1) + 1));
+    if (buffer == NULL) {
+        FatalError("Failed to allocate memory for temporary buffer: %s", strerror(errno));
+    }
+
     if (ConfGetBool("threading.set-cpu-affinity", &set_cpu_affinity) != 1) {
         set_cpu_affinity = 0;
     }
 
     if (set_cpu_affinity) {
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module,
-                "Minimum host buffers that should be defined in ntservice.ini:");
-
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 0: %d",
-                (SC_ATOMIC_GET(numa0_count)));
-
-        if (numa_max_node() >= 1)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 1: %d ",
-                    (SC_ATOMIC_GET(numa1_count)));
-
-        if (numa_max_node() >= 2)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 2: %d ",
-                    (SC_ATOMIC_GET(numa2_count)));
-
-        if (numa_max_node() >= 3)
-            SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module, "   NUMA Node 3: %d ",
-                    (SC_ATOMIC_GET(numa3_count)));
-
-        snprintf(string0, 16, "[%d, 16, 0]", SC_ATOMIC_GET(numa0_count));
-        snprintf(string1, 16, (numa_max_node() >= 1 ? ",[%d, 16, 1]" : ""),
-                SC_ATOMIC_GET(numa1_count));
-        snprintf(string2, 16, (numa_max_node() >= 2 ? ",[%d, 16, 2]" : ""),
-                SC_ATOMIC_GET(numa2_count));
-        snprintf(string3, 16, (numa_max_node() >= 3 ? ",[%d, 16, 3]" : ""),
-                SC_ATOMIC_GET(numa3_count));
-
-        SCLog(log_level, __FILE__, __FUNCTION__, __LINE__, _sc_module,
-                "E.g.: HostBuffersRx=%s%s%s%s", string0, string1, string2, string3);
-    } else if (log_level == SC_LOG_ERROR) {
-        SCLogError("Or, try running /opt/napatech3/bin/ntpl -e \"delete=all\" to clean-up stream "
-                   "NUMA config.");
+        SCLogPerf("Minimum host buffers that should be defined in ntservice.ini:");
+        for (int i = 0; i <= numa_max_node(); ++i) {
+            SCLogPerf("   NUMA Node %d: %d", i, SC_ATOMIC_GET(numa_detect[i].count));
+            p += snprintf(p, 32, "%s[%d, 16, %d]", (i == 0 ? "" : ","),
+                    SC_ATOMIC_GET(numa_detect[i].count), i);
+        }
+        SCLogPerf("E.g.: HostBuffersRx=%s", buffer);
     }
+
+    SCFree(buffer);
 }
 
 /**
@@ -820,21 +829,9 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
 
     if (is_autoconfig) {
         numa_node = GetNumaNode();
-        switch (numa_node) {
-        case 0:
-            SC_ATOMIC_ADD(numa0_count, 1);
-            break;
-        case 1:
-            SC_ATOMIC_ADD(numa1_count, 1);
-            break;
-        case 2:
-            SC_ATOMIC_ADD(numa2_count, 1);
-            break;
-        case 3:
-            SC_ATOMIC_ADD(numa3_count, 1);
-            break;
-        default:
-            break;
+
+        if (numa_node <= numa_max_node()) {
+            SC_ATOMIC_ADD(numa_detect[numa_node].count, 1);
         }
 
         if (ConfGetBool("threading.set-cpu-affinity", &set_cpu_affinity) != 1) {
@@ -845,7 +842,6 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
             NapatechSetupNuma(ntv->stream_id, numa_node);
         }
 
-        numa_node = GetNumaNode();
         SC_ATOMIC_ADD(stream_count, 1);
         if (SC_ATOMIC_GET(stream_count) == NapatechGetNumConfiguredStreams()) {
 
@@ -866,14 +862,13 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
             closer = 1;
 
             if (status == 0x20002061) {
-                SCLogError("Check host buffer configuration in ntservice.ini.");
-                RecommendNUMAConfig(SC_LOG_ERROR);
-                exit(EXIT_FAILURE);
-
+                FatalError("Check host buffer configuration in ntservice.ini"
+                           " or try running /opt/napatech3/bin/ntpl -e "
+                           "\"delete=all\" to clean-up stream NUMA config.");
             } else if (status == 0x20000008) {
                 FatalError("Check napatech.ports in the suricata config file.");
             }
-            RecommendNUMAConfig(SC_LOG_PERF);
+            RecommendNUMAConfig();
             SCLogNotice("Napatech packet input engine started.");
         }
     } // is_autoconfig

From 405fc580ea748a0ee2c37d46aad9a8a841cfa510 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Wed, 16 Aug 2023 10:30:37 +0200
Subject: [PATCH 016/462] napatech: remove superfluous log messages

---
 src/source-napatech.c | 15 ++++++++-------
 src/util-napatech.c   |  2 --
 2 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index f646c42030d1..6654fbc60047 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -813,13 +813,14 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
 
 #ifdef NAPATECH_ENABLE_BYPASS
     NtFlowStream_t flow_stream[MAX_ADAPTERS] = { 0 };
-
-    /* Get a FlowStream handle for each adapter so we can efficiently find the
-     * correct handle corresponding to the port on which a packet is received.
-     */
-    int adapter = 0;
-    for (adapter = 0; adapter < NapatechGetNumAdapters(); ++adapter) {
-        flow_stream[adapter] = InitFlowStream(adapter, ntv->stream_id);
+    if (NapatechUseHWBypass()) {
+        /* Get a FlowStream handle for each adapter so we can efficiently find the
+         * correct handle corresponding to the port on which a packet is received.
+         */
+        int adapter = 0;
+        for (adapter = 0; adapter < NapatechGetNumAdapters(); ++adapter) {
+            flow_stream[adapter] = InitFlowStream(adapter, ntv->stream_id);
+        }
     }
 #endif
 
diff --git a/src/util-napatech.c b/src/util-napatech.c
index e8c842fabc4c..affcd49cbb87 100644
--- a/src/util-napatech.c
+++ b/src/util-napatech.c
@@ -1409,8 +1409,6 @@ uint32_t NapatechSetupTraffic(uint32_t first_stream, uint32_t last_stream)
 #ifdef NAPATECH_ENABLE_BYPASS
     if (NapatechUseHWBypass()) {
         SCLogInfo("Napatech Hardware Bypass enabled.");
-    } else {
-        SCLogInfo("Napatech Hardware Bypass available but disabled.");
     }
 #else
     if (NapatechUseHWBypass()) {

From 1e1b3a4edacc3da1272992d0a6a5203654766998 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Wed, 16 Aug 2023 10:36:16 +0200
Subject: [PATCH 017/462] napatech: fix incorrect fmt specifiers for log

---
 src/source-napatech.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index 6654fbc60047..d1d57f856a10 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -689,7 +689,7 @@ TmEcode NapatechStreamThreadInit(ThreadVars *tv, const void *initdata, void **da
 
     DatalinkSetGlobalType(LINKTYPE_ETHERNET);
 
-    SCLogDebug("Started processing packets from NAPATECH  Stream: %lu", ntv->stream_id);
+    SCLogDebug("Started processing packets from NAPATECH  Stream: %u", ntv->stream_id);
 
     *data = (void *) ntv;
     SCReturnInt(TM_ECODE_OK);
@@ -888,7 +888,7 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
         StatsSetupPrivate(tv);
         StatsSetUI64(tv, hba_pkt, 0);
     }
-    SCLogDebug("Opening NAPATECH Stream: %lu for processing", ntv->stream_id);
+    SCLogDebug("Opening NAPATECH Stream: %u for processing", ntv->stream_id);
 
     if ((status = NT_NetRxOpen(&(ntv->rx_stream), "SuricataStream",
             NT_NET_INTERFACE_PACKET, ntv->stream_id, ntv->hba)) != NT_SUCCESS) {

From a4756138cf402b1994f808f02ae5ba889455115d Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Wed, 16 Aug 2023 13:30:01 +0200
Subject: [PATCH 018/462] napatech: print NUMA recommendation early

When thread affinity is set, the NUMA configuration specified in
the napatech.ini configuration could be incorrect and then fail.
This fails before the recommended configuration is printed, which
is pretty unhelpful.
---
 src/source-napatech.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index d1d57f856a10..44201a2bb361 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -845,6 +845,9 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
 
         SC_ATOMIC_ADD(stream_count, 1);
         if (SC_ATOMIC_GET(stream_count) == NapatechGetNumConfiguredStreams()) {
+            /* Print the recommended NUMA configuration early because it
+             * can fail with "No available hostbuffers" in NapatechSetupTraffic */
+            RecommendNUMAConfig();
 
 #ifdef NAPATECH_ENABLE_BYPASS
             if (ConfGetBool("napatech.inline", &is_inline) == 0) {
@@ -869,7 +872,6 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
             } else if (status == 0x20000008) {
                 FatalError("Check napatech.ports in the suricata config file.");
             }
-            RecommendNUMAConfig();
             SCLogNotice("Napatech packet input engine started.");
         }
     } // is_autoconfig

From 185f605d11b26a98b6ad0e854a1d4d24969703e7 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Mon, 21 Aug 2023 11:05:57 +0200
Subject: [PATCH 019/462] napatech: fix null-dereference of packet

---
 src/source-napatech.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index 44201a2bb361..5d5e1f674d15 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -927,17 +927,16 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
         }
 
         Packet *p = PacketGetFromQueueOrAlloc();
-#ifdef NAPATECH_ENABLE_BYPASS
-        p->ntpv.bypass = 0;
-#endif
-
-        p->ntpv.rx_stream = ntv->rx_stream;
-
         if (unlikely(p == NULL)) {
             NT_NetRxRelease(ntv->rx_stream, packet_buffer);
             SCReturnInt(TM_ECODE_FAILED);
         }
 
+#ifdef NAPATECH_ENABLE_BYPASS
+        p->ntpv.bypass = 0;
+#endif
+        p->ntpv.rx_stream = ntv->rx_stream;
+
         pkt_ts = NT_NET_GET_PKT_TIMESTAMP(packet_buffer);
 
         /*

From 0ddc44f4c116945e0a8fa846d4f2c5ba5a7d8b63 Mon Sep 17 00:00:00 2001
From: Arne Welzel <arne.welzel@corelight.com>
Date: Sun, 20 Aug 2023 17:32:47 +0200
Subject: [PATCH 020/462] community-id: Fix IPv6 address sorting not respecting
 byte order

When comparing IPv6 addresses based on uint32_t chunks, one needs to
apply ntohl() conversion to the individual parts, otherwise on little
endian systems individual bytes are compared in the wrong order.
Avoid this all and leverage memcmp(), it'll short circuit on the first
differing byte and its return values tells us which address sorts lower.

Bug: #6276
---
 src/output-json.c | 17 ++---------------
 1 file changed, 2 insertions(+), 15 deletions(-)

diff --git a/src/output-json.c b/src/output-json.c
index d19bc3da1545..5d4255cd2897 100644
--- a/src/output-json.c
+++ b/src/output-json.c
@@ -638,18 +638,6 @@ static bool CalculateCommunityFlowIdv4(const Flow *f,
     return false;
 }
 
-static inline bool FlowHashRawAddressIPv6LtU32(const uint32_t *a, const uint32_t *b)
-{
-    for (int i = 0; i < 4; i++) {
-        if (a[i] < b[i])
-            return true;
-        if (a[i] > b[i])
-            break;
-    }
-
-    return false;
-}
-
 static bool CalculateCommunityFlowIdv6(const Flow *f,
         const uint16_t seed, unsigned char *base64buf)
 {
@@ -673,9 +661,8 @@ static bool CalculateCommunityFlowIdv6(const Flow *f,
     dp = htons(dp);
 
     ipv6.seed = htons(seed);
-    if (FlowHashRawAddressIPv6LtU32(f->src.addr_data32, f->dst.addr_data32) ||
-            ((memcmp(&f->src, &f->dst, sizeof(f->src)) == 0) && sp < dp))
-    {
+    int cmp_r = memcmp(&f->src, &f->dst, sizeof(f->src));
+    if ((cmp_r < 0) || (cmp_r == 0 && sp < dp)) {
         memcpy(&ipv6.src, &f->src.addr_data32, 16);
         memcpy(&ipv6.dst, &f->dst.addr_data32, 16);
         ipv6.sp = sp;

From e65c0524144fe0e246dc629926e97770ea70f228 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:10:49 -0400
Subject: [PATCH 021/462] build/nss: Remove libnss from CI

---
 .github/workflows/builds.yml | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index af17d82f79cb..eb71ea23bd21 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -2095,7 +2095,6 @@ jobs:
               libjansson-dev \
               libjansson4 \
               liblua5.1-dev \
-              libnss3-dev \
               libnspr4-dev \
               libnuma-dev \
               liblz4-dev \
@@ -2179,7 +2178,6 @@ jobs:
               libmagic-dev \
               libjansson-dev \
               libjansson4 \
-              libnss3-dev \
               libnspr4-dev \
               liblz4-dev \
               libssl-dev \
@@ -2259,7 +2257,6 @@ jobs:
               libjansson-dev \
               libjansson4 \
               liblua5.1-dev \
-              libnss3-dev \
               libnspr4-dev \
               libnuma-dev \
               liblz4-dev \

From 30b5338af3a796de5d4e382e6556339fd365d56a Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 6 Sep 2023 13:22:42 +0200
Subject: [PATCH 022/462] fuzz: enable by default all protocols

That means DNP3, ENIP and NFS

Ticket: #6189
---
 src/app-layer-detect-proto.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index 4ae69350df54..c7f902edc22f 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -1902,6 +1902,11 @@ int AppLayerProtoDetectConfProtoDetectionEnabledDefault(
     if (RunmodeIsUnittests())
         goto enabled;
 
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    // so that fuzzig takes place for DNP3 and such
+    default_enabled = true;
+#endif
+
     r = snprintf(param, sizeof(param), "%s%s%s", "app-layer.protocols.",
                  alproto, ".enabled");
     if (r < 0) {

From 82758fb09f8035c0a723332813831d87037559ed Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 5 Sep 2023 15:06:24 -0600
Subject: [PATCH 023/462] configure: check for new enough sphinx-build

We need a recent version of Sphinx to build the documentation in
7.0. Check for a minimum version of 3.4.3. If older, do not build the
docs which is the same behavior when sphinx-build is not found.

Bug: #6297
---
 configure.ac | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 1fba3ebeefb9..743770464993 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2229,8 +2229,23 @@ fi
         AC_DEFINE([CLS],[64],[L1 cache line size])
     fi
 
-# sphinx for documentation
+# sphinx-build for documentation, and also check for a new enough version
     AC_PATH_PROG(HAVE_SPHINXBUILD, sphinx-build, "no")
+    if test "$HAVE_SPHINXBUILD" != "no"; then
+        MIN_SPHINX_BUILD_VERSION="3.4.3"
+        sphinx_build_version=$($HAVE_SPHINXBUILD --version 2>&1 | cut -d' ' -f2-)
+        AC_MSG_CHECKING([for sphinx-build >= $MIN_SPHINX_BUILD_VERSION])
+        AS_VERSION_COMPARE([$sphinx_build_version], [$MIN_SPHINX_BUILD_VERSION],
+            [
+                AC_MSG_RESULT([no, documentation will not be built])
+                HAVE_SPHINXBUILD="no"
+            ],
+            [], [])
+        if test "$HAVE_SPHINXBUILD" != "no"; then
+            AC_MSG_RESULT(yes)
+        fi
+    fi
+
     if test "$HAVE_SPHINXBUILD" = "no"; then
        enable_sphinxbuild=no
        if test -e "$srcdir/doc/userguide/suricata.1"; then

From ae3b1a9e36dc57e72eff7ca3d4e4d1441d36335f Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 5 Sep 2023 15:16:43 -0600
Subject: [PATCH 024/462] configure: more idiomatic autoconf for sphinx-build
 checks

- Use SPHINX_BUILD instead of HAVE_SPHINX_BUILD, as here we're
  actually using the path of the program.

- Wrap some elements in [] as is done in modern idiomatic autoconf
---
 configure.ac              | 16 ++++++++--------
 doc/userguide/Makefile.am |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/configure.ac b/configure.ac
index 743770464993..72adef1b9187 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2230,29 +2230,29 @@ fi
     fi
 
 # sphinx-build for documentation, and also check for a new enough version
-    AC_PATH_PROG(HAVE_SPHINXBUILD, sphinx-build, "no")
-    if test "$HAVE_SPHINXBUILD" != "no"; then
+    AC_PATH_PROG([SPHINX_BUILD], [sphinx-build], [no])
+    if test "$SPHINX_BUILD" != "no"; then
         MIN_SPHINX_BUILD_VERSION="3.4.3"
-        sphinx_build_version=$($HAVE_SPHINXBUILD --version 2>&1 | cut -d' ' -f2-)
+        sphinx_build_version=$($SPHINX_BUILD --version 2>&1 | cut -d' ' -f2-)
         AC_MSG_CHECKING([for sphinx-build >= $MIN_SPHINX_BUILD_VERSION])
         AS_VERSION_COMPARE([$sphinx_build_version], [$MIN_SPHINX_BUILD_VERSION],
             [
                 AC_MSG_RESULT([no, documentation will not be built])
-                HAVE_SPHINXBUILD="no"
+                SPHINX_BUILD="no"
             ],
             [], [])
-        if test "$HAVE_SPHINXBUILD" != "no"; then
-            AC_MSG_RESULT(yes)
+        if test "$SPHINX_BUILD" != "no"; then
+            AC_MSG_RESULT([yes])
         fi
     fi
 
-    if test "$HAVE_SPHINXBUILD" = "no"; then
+    if test "$SPHINX_BUILD" = "no"; then
        enable_sphinxbuild=no
        if test -e "$srcdir/doc/userguide/suricata.1"; then
            have_suricata_man=yes
        fi
     fi
-    AM_CONDITIONAL([HAVE_SPHINXBUILD], [test "x$enable_sphinxbuild" != "xno"])
+    AM_CONDITIONAL([SPHINX_BUILD], [test "x$enable_sphinxbuild" != "xno"])
     AM_CONDITIONAL([HAVE_SURICATA_MAN], [test "x$have_suricata_man" = "xyes"])
 
 # pdflatex for the pdf version of the user manual
diff --git a/doc/userguide/Makefile.am b/doc/userguide/Makefile.am
index 8fcfa861771c..bd157920cfac 100644
--- a/doc/userguide/Makefile.am
+++ b/doc/userguide/Makefile.am
@@ -37,7 +37,7 @@ if HAVE_SURICATA_MAN
 dist_man1_MANS = suricata.1 suricatasc.1 suricatactl.1 suricatactl-filestore.1
 endif
 
-if HAVE_SPHINXBUILD
+if SPHINX_BUILD
 dist_man1_MANS = suricata.1 suricatasc.1 suricatactl.1 suricatactl-filestore.1
 
 if HAVE_PDFLATEX
@@ -92,4 +92,4 @@ clean-local:
 	rm -f $(top_builddir)/doc/userguide/suricata*.1
 	rm -f $(top_builddir)/doc/userguide/userguide.pdf
 
-endif # HAVE_SPHINXBUILD
+endif # SPHINX_BUILD

From 2b57179d65ecd0c29b1031e4ac23ef5296d60046 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 6 Sep 2023 08:51:49 -0600
Subject: [PATCH 025/462] readthedocs: pin theme to sphinx_rtd_theme

ReadTheDocs changed the default theme.
---
 doc/userguide/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/userguide/conf.py b/doc/userguide/conf.py
index 47fec8ec6c24..cf87f19c311e 100644
--- a/doc/userguide/conf.py
+++ b/doc/userguide/conf.py
@@ -143,6 +143,7 @@ def setup(app):
         else:
             app.add_stylesheet('css/suricata.css')
 else:
+    html_theme = 'sphinx_rtd_theme'
     html_context = {
         'css_files': [
             'https://media.readthedocs.org/css/sphinx_rtd_theme.css',

From 00e00254eae205bad5d4cfbf6c9e69f944faaf69 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 5 Sep 2023 14:49:34 +0200
Subject: [PATCH 026/462] spm/hs: don't exit on bad patterns

A bad pattern in a rule that hyperscan would fail to compile would
exit Suricata. This could happen during a rule reload as well.

In case of a untrusted ruleset, this could potentially be used to
shut down the sensor.

Commit 7d0851b0c2 already blocks the only know case, but this patch
is more defensive.

Ticket: #6195.
---
 src/util-spm-hs.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/util-spm-hs.c b/src/util-spm-hs.c
index 62862be230ca..cfcb8acd52a9 100644
--- a/src/util-spm-hs.c
+++ b/src/util-spm-hs.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2016 Open Information Security Foundation
+/* Copyright (C) 2016-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -84,7 +84,7 @@ static int HSBuildDatabase(const uint8_t *needle, uint16_t needle_len,
         SCLogError("Unable to compile '%s' with Hyperscan, "
                    "returned %d.",
                 expr, err);
-        exit(EXIT_FAILURE);
+        return -1;
     }
 
     SCFree(expr);
@@ -96,7 +96,7 @@ static int HSBuildDatabase(const uint8_t *needle, uint16_t needle_len,
         /* If scratch allocation failed, this is not recoverable:  other SPM
          * contexts may need this scratch space. */
         SCLogError("Unable to alloc scratch for Hyperscan, returned %d.", err);
-        exit(EXIT_FAILURE);
+        return -1;
     }
     global_thread_ctx->ctx = scratch;
     sctx->db = db;

From c6afee64d510daa9f383b160f3abd194ee74a15b Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 30 Aug 2023 21:35:08 +0200
Subject: [PATCH 027/462] smtp: fix null deref with config option body md5

Ticket: #6279

If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
---
 src/util-decode-mime.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/util-decode-mime.c b/src/util-decode-mime.c
index 7ee5263b769c..b22a8c2e6f61 100644
--- a/src/util-decode-mime.c
+++ b/src/util-decode-mime.c
@@ -1766,6 +1766,12 @@ static int FindMimeHeader(const uint8_t *buf, uint32_t blen,
         state->body_begin = 1;
         state->body_end = 0;
 
+        // Begin the body md5 computation if config asks so
+        if (MimeDecGetConfig()->body_md5 && state->md5_ctx == NULL) {
+            state->md5_ctx = SCMd5New();
+            SCMd5Update(state->md5_ctx, buf, blen + state->current_line_delimiter_len);
+        }
+
         ret = ProcessBodyLine(buf, blen, state);
         if (ret != MIME_DEC_OK) {
             SCLogDebug("Error: ProcessBodyLine() function failed");

From 8553d567d269d164a97db4a3c479ce00f3ac8474 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 13 Sep 2023 11:47:03 -0300
Subject: [PATCH 028/462] release: 7.0.1; update changelog

---
 ChangeLog    | 34 ++++++++++++++++++++++++++++++++++
 configure.ac |  2 +-
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index f96b3b9d7c91..051331a87c76 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,37 @@
+7.0.1 -- 2023-09-13
+
+Security #6279: Crash in SMTP parser during parsing of email
+Security #6195: process exit in hyperscan error handling 
+Bug #6276: community-id: Fix IPv6 address sorting not respecting byte order
+Bug #6256: eve: crash if output dir isn't writeable
+Bug #6255: flow: possible divide by zero at start up
+Bug #6247: pcre: parsing crash in multi-tenant multi-loader setup
+Bug #6244: tcp: RST with data used in reassembly
+Bug #6243: Parsing ip-reputation reputation config files now rejects CR and CR+LF
+Bug #6240: pcap/file: negative pcap file timestamps lead to weird output
+Bug #6233: dpdk: fix overall threads check for IPS mode
+Bug #6232: dpdk: treat unknown socket value as a valid value
+Bug #6222: Decode-events of IPv6 GRE are not triggered
+Bug #6201: multi-tenancy: crash under test mode when tenant signature load fails
+Bug #6191: if protocol dcerpc first packet type is Alter_context, it will not parse dcerpc
+Bug #6095: windows: lua script path truncated
+Bug #6094: eve/stats: memcap_pressure and memcap_pressure_max not logged
+Bug #6044: detect: multi-tenancy leaks memory if more than 1 tenant registered
+Bug #5870: ips/af-packet: crash when copy-iface is the same as the interface
+Bug #5619: dpdk/ips: crash at shutdown with mlx
+Bug #5443: ftp-data: failed assertion
+Bug #4881: alert event incorrectly log stored files
+Optimization #6265: threading: set a higher default stack size for threads
+Optimization #6263: mpm/ac: reduce stack usage
+Optimization #5920: investigate: check and fix unhandled divisions by 0
+Optimization #3637: Performance impact of Cisco Fabricpath
+Feature #6267: multi-tenancy: reload-tenants command
+Feature #6230: stats: add drop reason counters
+Feature #4756: capture: support ips stats for all IPS capture methods
+Feature #4587: dhcp: vendor class indentifier support
+Documentation #6231: userguide: add installation from Ubuntu PPA section
+Documentation #6124: userguide: add instructions/explanation for (not) running suricata with root
+
 7.0.0 -- 2023-07-18
 
 Bug #6212: file.magic: rule reload can lead to crashes
diff --git a/configure.ac b/configure.ac
index 72adef1b9187..fe1068ab0ddf 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-    AC_INIT([suricata],[7.0.1-dev])
+    AC_INIT([suricata],[7.0.1])
     m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])AM_SILENT_RULES([yes])
     AC_CONFIG_HEADERS([src/autoconf.h])
     AC_CONFIG_SRCDIR([src/suricata.c])

From 908f49eef1cdb2b85f74a8f955b51dfddbdbd27d Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 14 Sep 2023 14:22:04 +0200
Subject: [PATCH 029/462] version: start development towards 7.0.2

---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index fe1068ab0ddf..8bb752715f53 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-    AC_INIT([suricata],[7.0.1])
+    AC_INIT([suricata],[7.0.2-dev])
     m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])AM_SILENT_RULES([yes])
     AC_CONFIG_HEADERS([src/autoconf.h])
     AC_CONFIG_SRCDIR([src/suricata.c])

From 904f0ddeeeb1bdb4a686f991cf090a47dd84249e Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 22 Aug 2023 10:17:24 -0400
Subject: [PATCH 030/462] stats: Track stream reassembly drops

Issue: 6235
---
 etc/schema.json             | 6 +++++-
 src/decode.c                | 4 ++++
 src/decode.h                | 1 +
 src/stream-tcp-reassemble.c | 2 +-
 4 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/etc/schema.json b/etc/schema.json
index efd17092f68c..28182cb95d98 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -40,7 +40,8 @@
             "type": "integer"
         },
         "host": {
-            "$comment": "May change to sensor_name in the future, or become user configurable: https://redmine.openinfosecfoundation.org/issues/4919",
+            "$comment":
+                    "May change to sensor_name in the future, or become user configurable: https://redmine.openinfosecfoundation.org/issues/4919",
             "description": "the sensor-name, if configured",
             "type": "string"
         },
@@ -4095,6 +4096,9 @@
                                 "stream_midstream": {
                                     "type": "integer"
                                 },
+                                "stream_reassembly": {
+                                    "type": "integer"
+                                },
                                 "nfq_error": {
                                     "type": "integer"
                                 },
diff --git a/src/decode.c b/src/decode.c
index b49b29838cab..5cdeeead6b96 100644
--- a/src/decode.c
+++ b/src/decode.c
@@ -804,6 +804,8 @@ const char *PacketDropReasonToString(enum PacketDropReason r)
             return "stream memcap";
         case PKT_DROP_REASON_STREAM_MIDSTREAM:
             return "stream midstream";
+        case PKT_DROP_REASON_STREAM_REASSEMBLY:
+            return "stream reassembly";
         case PKT_DROP_REASON_APPLAYER_ERROR:
             return "applayer error";
         case PKT_DROP_REASON_APPLAYER_MEMCAP:
@@ -842,6 +844,8 @@ static const char *PacketDropReasonToJsonString(enum PacketDropReason r)
             return "ips.drop_reason.stream_memcap";
         case PKT_DROP_REASON_STREAM_MIDSTREAM:
             return "ips.drop_reason.stream_midstream";
+        case PKT_DROP_REASON_STREAM_REASSEMBLY:
+            return "ips.drop_reason.stream_reassembly";
         case PKT_DROP_REASON_APPLAYER_ERROR:
             return "ips.drop_reason.applayer_error";
         case PKT_DROP_REASON_APPLAYER_MEMCAP:
diff --git a/src/decode.h b/src/decode.h
index fe42924bb628..dedfbb09efd0 100644
--- a/src/decode.h
+++ b/src/decode.h
@@ -401,6 +401,7 @@ enum PacketDropReason {
     PKT_DROP_REASON_STREAM_ERROR,
     PKT_DROP_REASON_STREAM_MEMCAP,
     PKT_DROP_REASON_STREAM_MIDSTREAM,
+    PKT_DROP_REASON_STREAM_REASSEMBLY,
     PKT_DROP_REASON_NFQ_ERROR,    /**< no nfq verdict, must be error */
     PKT_DROP_REASON_INNER_PACKET, /**< drop issued by inner (tunnel) packet */
     PKT_DROP_REASON_MAX,
diff --git a/src/stream-tcp-reassemble.c b/src/stream-tcp-reassemble.c
index 135b22485c04..737b222d53e2 100644
--- a/src/stream-tcp-reassemble.c
+++ b/src/stream-tcp-reassemble.c
@@ -2017,7 +2017,7 @@ int StreamTcpReassembleHandleSegment(ThreadVars *tv, TcpReassemblyThreadCtx *ra_
             SCLogDebug("StreamTcpReassembleHandleSegmentHandleData error");
             /* failure can only be because of memcap hit, so see if this should lead to a drop */
             ExceptionPolicyApply(
-                    p, stream_config.reassembly_memcap_policy, PKT_DROP_REASON_STREAM_MEMCAP);
+                    p, stream_config.reassembly_memcap_policy, PKT_DROP_REASON_STREAM_REASSEMBLY);
             SCReturnInt(-1);
         }
 

From 252e8dbb32b23c0bc20e6bb5690c5516a4213e9b Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 7 Sep 2023 11:53:55 -0600
Subject: [PATCH 031/462] conf: fix include handling from arrays

Includes from an "include" array were being loaded into the wrong
parent as the logic for array handing in include context was not
updated.

If we are descending into an array in include context, pass through
the current parent so the included configuration is included where it
is expected.

Bug: #6300
---
 src/conf-yaml-loader.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/conf-yaml-loader.c b/src/conf-yaml-loader.c
index 9312a189e8a4..1bd107e0c1c9 100644
--- a/src/conf-yaml-loader.c
+++ b/src/conf-yaml-loader.c
@@ -374,8 +374,9 @@ static int ConfYamlParse(yaml_parser_t *parser, ConfNode *parent, int inseq, int
         }
         else if (event.type == YAML_SEQUENCE_START_EVENT) {
             SCLogDebug("event.type=YAML_SEQUENCE_START_EVENT; state=%d", state);
-            if (ConfYamlParse(parser, node, 1, rlevel, state == CONF_INCLUDE ? CONF_INCLUDE : 0) !=
-                    0)
+            /* If we're processing a list of includes, use the current parent. */
+            if (ConfYamlParse(parser, state == CONF_INCLUDE ? parent : node, 1, rlevel,
+                        state == CONF_INCLUDE ? CONF_INCLUDE : 0) != 0)
                 goto fail;
             node->is_seq = 1;
             state = CONF_KEY;

From 72ee505aa025b53956c6a3c2428c36db904207e6 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 3 Aug 2023 15:18:51 +0530
Subject: [PATCH 032/462] conf: check if node value is Null

Bug: #6303, #6302
---
 src/conf.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/conf.c b/src/conf.c
index 58a1db5c122a..f2bf978e8c25 100644
--- a/src/conf.c
+++ b/src/conf.c
@@ -354,6 +354,8 @@ int ConfGetChildValue(const ConfNode *base, const char *name, const char **vptr)
         return 0;
     }
     else {
+        if (node->val == NULL)
+            return 0;
         *vptr = node->val;
         return 1;
     }

From cdcb1b3263985b4106cab5e515c4887b2a30af8c Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Wed, 13 Sep 2023 15:20:38 +0530
Subject: [PATCH 033/462] util/ioctl: cut vain check on
 GetIfaceMaxHWHeaderLength

---
 src/util-ioctl.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/src/util-ioctl.c b/src/util-ioctl.c
index 4926cefde8e6..96ebe0b5ee98 100644
--- a/src/util-ioctl.c
+++ b/src/util-ioctl.c
@@ -139,10 +139,6 @@ int GetIfaceMaxPacketSize(const char *pcap_dev)
             return 0;
     }
     int ll_header = GetIfaceMaxHWHeaderLength(pcap_dev);
-    if (ll_header == -1) {
-        /* be conservative, choose a big one */
-        ll_header = 16;
-    }
     return ll_header + mtu;
 }
 

From 4639a62eb7b132fe73fda19ec3cb0921f1424c52 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 10 Aug 2023 21:54:39 +0530
Subject: [PATCH 034/462] util/ioctl: rename pcap_dev to dev

---
 src/util-ioctl.c | 38 ++++++++++++++------------------------
 1 file changed, 14 insertions(+), 24 deletions(-)

diff --git a/src/util-ioctl.c b/src/util-ioctl.c
index 96ebe0b5ee98..97f8dd378f71 100644
--- a/src/util-ioctl.c
+++ b/src/util-ioctl.c
@@ -55,26 +55,16 @@
  *
  * \param Name of a network interface
  */
-static int GetIfaceMaxHWHeaderLength(const char *pcap_dev)
+static int GetIfaceMaxHWHeaderLength(const char *dev)
 {
-    if ((!strcmp("eth", pcap_dev))
-            ||
-            (!strcmp("br", pcap_dev))
-            ||
-            (!strcmp("bond", pcap_dev))
-            ||
-            (!strcmp("wlan", pcap_dev))
-            ||
-            (!strcmp("tun", pcap_dev))
-            ||
-            (!strcmp("tap", pcap_dev))
-            ||
-            (!strcmp("lo", pcap_dev))) {
+    if ((!strcmp("eth", dev)) || (!strcmp("br", dev)) || (!strcmp("bond", dev)) ||
+            (!strcmp("wlan", dev)) || (!strcmp("tun", dev)) || (!strcmp("tap", dev)) ||
+            (!strcmp("lo", dev))) {
         /* Add possible VLAN tag or Qing headers */
         return 8 + ETHERNET_HEADER_LEN;
     }
 
-    if (!strcmp("ppp", pcap_dev))
+    if (!strcmp("ppp", dev))
         return SLL_HEADER_LEN;
     /* SLL_HEADER_LEN is the biggest one and
        add possible VLAN tag and Qing headers */
@@ -88,29 +78,29 @@ static int GetIfaceMaxHWHeaderLength(const char *pcap_dev)
  * \param Name of link
  * \retval -1 in case of error, 0 if MTU can not be found
  */
-int GetIfaceMTU(const char *pcap_dev)
+int GetIfaceMTU(const char *dev)
 {
 #if defined SIOCGIFMTU
     struct ifreq ifr;
     int fd;
 
-    (void)strlcpy(ifr.ifr_name, pcap_dev, sizeof(ifr.ifr_name));
+    (void)strlcpy(ifr.ifr_name, dev, sizeof(ifr.ifr_name));
     fd = socket(AF_INET, SOCK_DGRAM, 0);
     if (fd == -1) {
         return -1;
     }
 
     if (ioctl(fd, SIOCGIFMTU, (char *)&ifr) < 0) {
-        SCLogWarning("Failure when trying to get MTU via ioctl for '%s': %s (%d)", pcap_dev,
+        SCLogWarning("Failure when trying to get MTU via ioctl for '%s': %s (%d)", dev,
                 strerror(errno), errno);
         close(fd);
         return -1;
     }
     close(fd);
-    SCLogInfo("%s: MTU %d", pcap_dev, ifr.ifr_mtu);
+    SCLogInfo("%s: MTU %d", dev, ifr.ifr_mtu);
     return ifr.ifr_mtu;
 #elif defined OS_WIN32
-    return GetIfaceMTUWin32(pcap_dev);
+    return GetIfaceMTUWin32(dev);
 #else
     /* ioctl is not defined, let's pretend returning 0 is ok */
     return 0;
@@ -127,18 +117,18 @@ int GetIfaceMTU(const char *pcap_dev)
  * \param Name of a network interface
  * \retval 0 in case of error
  */
-int GetIfaceMaxPacketSize(const char *pcap_dev)
+int GetIfaceMaxPacketSize(const char *dev)
 {
-    if ((pcap_dev == NULL) || strlen(pcap_dev) == 0)
+    if ((dev == NULL) || strlen(dev) == 0)
         return 0;
 
-    int mtu = GetIfaceMTU(pcap_dev);
+    int mtu = GetIfaceMTU(dev);
     switch (mtu) {
         case 0:
         case -1:
             return 0;
     }
-    int ll_header = GetIfaceMaxHWHeaderLength(pcap_dev);
+    int ll_header = GetIfaceMaxHWHeaderLength(dev);
     return ll_header + mtu;
 }
 

From 572f8a3da6241a71aecc1cd5d5b6e75035cf426d Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 10 Aug 2023 21:04:56 +0530
Subject: [PATCH 035/462] util/ioctl: use LiveDevice to retrieve name

The fn GetIfaceMaxPacketSize now uses LiveDevice object as a param
instead of a string. This was done to keep the logic of checking for the
device to this function itself instead of having callers first determine
whether the device exists or not.
This also falls in line with the changes made to avoid excessive MTU
logs in the following commit.

Related to redmine ticket 5831.
---
 src/source-af-packet.c | 4 ++--
 src/source-pcap.c      | 2 +-
 src/suricata.c         | 3 ++-
 src/util-ioctl.c       | 8 ++++++--
 src/util-ioctl.h       | 2 +-
 5 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/src/source-af-packet.c b/src/source-af-packet.c
index 0c50ed219aa6..317c8704e5af 100644
--- a/src/source-af-packet.c
+++ b/src/source-af-packet.c
@@ -1573,7 +1573,7 @@ sockaddr_ll) + ETH_HLEN) - ETH_HLEN);
     int snaplen = default_packet_size;
 
     if (snaplen == 0) {
-        snaplen = GetIfaceMaxPacketSize(ptv->iface);
+        snaplen = GetIfaceMaxPacketSize(ptv->livedev);
         if (snaplen <= 0) {
             SCLogWarning("%s: unable to get MTU, setting snaplen default of 1514", ptv->iface);
             snaplen = 1514;
@@ -1607,7 +1607,7 @@ static int AFPComputeRingParamsV3(AFPThreadVars *ptv)
     int snaplen = default_packet_size;
 
     if (snaplen == 0) {
-        snaplen = GetIfaceMaxPacketSize(ptv->iface);
+        snaplen = GetIfaceMaxPacketSize(ptv->livedev);
         if (snaplen <= 0) {
             SCLogWarning("%s: unable to get MTU, setting snaplen default of 1514", ptv->iface);
             snaplen = 1514;
diff --git a/src/source-pcap.c b/src/source-pcap.c
index a36c2b646faa..f916d69354c7 100644
--- a/src/source-pcap.c
+++ b/src/source-pcap.c
@@ -514,7 +514,7 @@ static TmEcode ReceivePcapThreadInit(ThreadVars *tv, const void *initdata, void
 
     if (pcapconfig->snaplen == 0) {
         /* We set snaplen if we can get the MTU */
-        ptv->pcap_snaplen = GetIfaceMaxPacketSize(pcapconfig->iface);
+        ptv->pcap_snaplen = GetIfaceMaxPacketSize(ptv->livedev);
     } else {
         ptv->pcap_snaplen = pcapconfig->snaplen;
     }
diff --git a/src/suricata.c b/src/suricata.c
index 7fe469acf24d..d9adcaf07b26 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -2486,7 +2486,8 @@ static int ConfigGetCaptureValue(SCInstance *suri)
                             dev[len-1] = '\0';
                         }
                     }
-                    unsigned int iface_max_packet_size = GetIfaceMaxPacketSize(dev);
+                    LiveDevice *ld = LiveGetDevice(dev);
+                    unsigned int iface_max_packet_size = GetIfaceMaxPacketSize(ld);
                     if (iface_max_packet_size > default_packet_size)
                         default_packet_size = iface_max_packet_size;
                 }
diff --git a/src/util-ioctl.c b/src/util-ioctl.c
index 97f8dd378f71..399751b05679 100644
--- a/src/util-ioctl.c
+++ b/src/util-ioctl.c
@@ -114,11 +114,15 @@ int GetIfaceMTU(const char *dev)
  * for the link. In case of uncertainty, it will output a
  * majorant to be sure avoid the cost of dynamic allocation.
  *
- * \param Name of a network interface
+ * \param LiveDevice object
  * \retval 0 in case of error
  */
-int GetIfaceMaxPacketSize(const char *dev)
+int GetIfaceMaxPacketSize(LiveDevice *ld)
 {
+    if (ld == NULL)
+        return 0;
+
+    const char *dev = ld->dev;
     if ((dev == NULL) || strlen(dev) == 0)
         return 0;
 
diff --git a/src/util-ioctl.h b/src/util-ioctl.h
index 2d0c74740dd8..24f897487511 100644
--- a/src/util-ioctl.h
+++ b/src/util-ioctl.h
@@ -25,7 +25,7 @@
 #include "util-device.h"
 
 int GetIfaceMTU(const char *pcap_dev);
-int GetIfaceMaxPacketSize(const char *pcap_dev);
+int GetIfaceMaxPacketSize(LiveDevice *ld);
 int GetIfaceOffloading(const char *dev, int csum, int other);
 int GetIfaceRSSQueuesNum(const char *pcap_dev);
 #ifdef SIOCGIFFLAGS

From 2fa0fac2897628a8c1e4c11c888447a5d55b8df8 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 10 Aug 2023 21:05:45 +0530
Subject: [PATCH 036/462] af-packet: fetch mtu info once

With the current layout and fn calls, it was seen that once in the
beginning after the MTU was found and displayed to the user, when the
threads spawned, each thread displayed MTU info as a part of
AFPPeersListAdd fn. This happened in AF_PACKET IPS mode and led to
excessive MTU logs.
Save this info in the LiveDevice struct and avoid calling the unneeded
fns later on.

Bug 5831
---
 src/source-af-packet.c | 19 ++++++++++++-------
 src/util-device.h      |  1 +
 src/util-ioctl.c       |  1 +
 3 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/src/source-af-packet.c b/src/source-af-packet.c
index 317c8704e5af..6112cb9d8869 100644
--- a/src/source-af-packet.c
+++ b/src/source-af-packet.c
@@ -492,7 +492,6 @@ static TmEcode AFPPeersListAdd(AFPThreadVars *ptv)
     SCEnter();
     AFPPeer *peer = SCMalloc(sizeof(AFPPeer));
     AFPPeer *pitem;
-    int mtu, out_mtu;
 
     if (unlikely(peer == NULL)) {
         SCReturnInt(TM_ECODE_FAILED);
@@ -527,12 +526,18 @@ static TmEcode AFPPeersListAdd(AFPThreadVars *ptv)
                 continue;
             peer->peer = pitem;
             pitem->peer = peer;
-            mtu = GetIfaceMTU(ptv->iface);
-            out_mtu = GetIfaceMTU(ptv->out_iface);
-            if (mtu != out_mtu) {
-                SCLogWarning("MTU on %s (%d) and %s (%d) are not equal, "
-                             "transmission of packets bigger than %d will fail.",
-                        ptv->iface, mtu, ptv->out_iface, out_mtu, MIN(out_mtu, mtu));
+
+            LiveDevice *iface = ptv->livedev;
+            DEBUG_VALIDATE_BUG_ON(iface == NULL);
+            DEBUG_VALIDATE_BUG_ON(strcmp(iface->dev, ptv->iface) != 0);
+            LiveDevice *out_iface = LiveGetDevice(ptv->out_iface);
+            if (out_iface == NULL)
+                FatalError("AF_PACKET device %s not found. Aborting..", ptv->out_iface);
+            if (iface->mtu != out_iface->mtu) {
+                SCLogWarning("MTU on %s (%d) and %s (%d) are not equal, transmission of packets "
+                             "bigger than %d will fail.",
+                        iface->dev, iface->mtu, out_iface->dev, out_iface->mtu,
+                        MIN(out_iface->mtu, iface->mtu));
             }
             peerslist.peered += 2;
             break;
diff --git a/src/util-device.h b/src/util-device.h
index 51ddf7d5257a..0f756b78ca3a 100644
--- a/src/util-device.h
+++ b/src/util-device.h
@@ -49,6 +49,7 @@ typedef struct {
 typedef struct LiveDevice_ {
     char *dev;  /**< the device (e.g. "eth0") */
     char dev_short[MAX_DEVNAME + 1];
+    int mtu; /* MTU of the device */
     bool tenant_id_set;
 
     uint16_t id;
diff --git a/src/util-ioctl.c b/src/util-ioctl.c
index 399751b05679..f39662bd4d47 100644
--- a/src/util-ioctl.c
+++ b/src/util-ioctl.c
@@ -132,6 +132,7 @@ int GetIfaceMaxPacketSize(LiveDevice *ld)
         case -1:
             return 0;
     }
+    ld->mtu = mtu;
     int ll_header = GetIfaceMaxHWHeaderLength(dev);
     return ll_header + mtu;
 }

From f293823ab3f169ca07eb7e587a01737e15041b8d Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 14 Sep 2023 10:29:17 -0400
Subject: [PATCH 037/462] detect: Count buffer id once

This commit removes a second, unnecessary increment of the de_ctx buffer
id.

Issue: 5211
---
 src/detect-engine.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 3810cb0a7af4..009741c503fb 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -1893,7 +1893,6 @@ int DetectEngineBufferTypeGetByIdTransforms(
     BUG_ON(HashListTableAdd(de_ctx->buffer_type_hash_name, (void *)map, 0) != 0);
     BUG_ON(HashListTableAdd(de_ctx->buffer_type_hash_id, (void *)map, 0) != 0);
     SCLogDebug("buffer %s registered with id %d, parent %d", map->name, map->id, map->parent_id);
-    de_ctx->buffer_type_id++;
 
     if (map->frame) {
         DetectFrameInspectEngineCopy(de_ctx, map->parent_id, map->id, &map->transforms);

From 01d232b6be153700d443337f728c2f664159cd85 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 14 Sep 2023 10:30:13 -0400
Subject: [PATCH 038/462] detect/rule: Use de_ctx buffer id

This commit uses the detect engine buffer id instead of the global
value.

Issue: 5211
---
 src/detect-engine-build.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index aa12ab56a8a8..80f5d89757e1 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -640,7 +640,7 @@ static json_t *RulesGroupPrintSghStats(const DetectEngineCtx *de_ctx, const SigG
     uint32_t mpms_min = 0;
     uint32_t mpms_max = 0;
 
-    int max_buffer_type_id = DetectBufferTypeMaxId() + 1;
+    int max_buffer_type_id = de_ctx->buffer_type_id;
 
     struct {
         uint32_t total;
@@ -755,6 +755,7 @@ static json_t *RulesGroupPrintSghStats(const DetectEngineCtx *de_ctx, const SigG
             if (w > mpms_max)
                 mpms_max = w;
 
+            BUG_ON(mpm_list >= max_buffer_type_id);
             mpm_stats[mpm_list].total += w;
             mpm_stats[mpm_list].cnt++;
             if (mpm_stats[mpm_list].min == 0 || w < mpm_stats[mpm_list].min)

From 59d050a0674f746a0aff1bc35a9dc743d07d232f Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 14 Sep 2023 10:31:05 -0400
Subject: [PATCH 039/462] detect/gen: Minor cleanup

Issue: 5211
---
 src/detect-engine-build.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 80f5d89757e1..e957b50bdb07 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -668,10 +668,8 @@ static json_t *RulesGroupPrintSghStats(const DetectEngineCtx *de_ctx, const SigG
 
     json_t *js_array = json_array();
 
-    const Signature *s;
-    uint32_t x;
-    for (x = 0; x < sgh->init->sig_cnt; x++) {
-        s = sgh->init->match_array[x];
+    for (uint32_t x = 0; x < sgh->init->sig_cnt; x++) {
+        const Signature *s = sgh->init->match_array[x];
         if (s == NULL)
             continue;
 

From 3eac0f15c0cff65a7f81f4d6b04426d6599a24ad Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Fri, 15 Sep 2023 06:08:38 -0400
Subject: [PATCH 040/462] dpdk: stop devices immediately after Suricata stop
 command

To better represent port stats of Suricata stop the device
right after Suricata is deemed to shut down.
While Suricata deinitialization happened, the device kept
receiving packets. But because Suricata was no longer
interested in the packets, the device accounted these packets
as missed and it could have alter true stats of Suricata.
---
 src/source-dpdk.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/src/source-dpdk.c b/src/source-dpdk.c
index 36a6d222e2c6..ddbb2976c961 100644
--- a/src/source-dpdk.c
+++ b/src/source-dpdk.c
@@ -212,7 +212,7 @@ static void DevicePostStartPMDSpecificActions(DPDKThreadVars *ptv, const char *d
         i40eDeviceSetRSS(ptv->port_id, ptv->threads);
 }
 
-static void DevicePreStopPMDSpecificActions(DPDKThreadVars *ptv, const char *driver_name)
+static void DevicePreClosePMDSpecificActions(DPDKThreadVars *ptv, const char *driver_name)
 {
     if (strcmp(driver_name, "net_bonding") == 0) {
         driver_name = BondingDeviceDriverGet(ptv->port_id);
@@ -378,6 +378,12 @@ static TmEcode ReceiveDPDKLoop(ThreadVars *tv, void *data, void *slot)
     while (1) {
         if (unlikely(suricata_ctl_flags != 0)) {
             SCLogDebug("Stopping Suricata!");
+            if (ptv->queue_id == 0) {
+                rte_eth_dev_stop(ptv->port_id);
+                if (ptv->copy_mode == DPDK_COPY_MODE_TAP || ptv->copy_mode == DPDK_COPY_MODE_IPS) {
+                    rte_eth_dev_stop(ptv->out_port_id);
+                }
+            }
             DPDKDumpCounters(ptv);
             break;
         }
@@ -668,12 +674,7 @@ static TmEcode ReceiveDPDKThreadDeinit(ThreadVars *tv, void *data)
             SCReturnInt(TM_ECODE_FAILED);
         }
 
-        DevicePreStopPMDSpecificActions(ptv, dev_info.driver_name);
-    }
-
-    rte_eth_dev_stop(ptv->port_id);
-    if (ptv->copy_mode == DPDK_COPY_MODE_TAP || ptv->copy_mode == DPDK_COPY_MODE_IPS) {
-        rte_eth_dev_stop(ptv->out_port_id);
+        DevicePreClosePMDSpecificActions(ptv, dev_info.driver_name);
     }
 
     ptv->pkt_mempool = NULL; // MP is released when device is closed

From af4bb917dc9842229445683b5ce2f955faa464c2 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Fri, 15 Sep 2023 06:15:23 -0400
Subject: [PATCH 041/462] dpdk: reset stats just before the start of packet
 receive loop

While Suricata initializes, the device must be started to e.g.
apply rte_flow rules on some devices. But in the meantime, the
NIC started receiving packets but accounted those as missed.
Stats reset was added to better represent true packet drop.
---
 src/source-dpdk.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/source-dpdk.c b/src/source-dpdk.c
index ddbb2976c961..54503e212271 100644
--- a/src/source-dpdk.c
+++ b/src/source-dpdk.c
@@ -373,8 +373,10 @@ static TmEcode ReceiveDPDKLoop(ThreadVars *tv, void *data, void *slot)
     // Indicate that the thread is actually running its application level code (i.e., it can poll
     // packets)
     TmThreadsSetFlag(tv, THV_RUNNING);
-
     PacketPoolWait();
+
+    rte_eth_stats_reset(ptv->port_id);
+    rte_eth_xstats_reset(ptv->port_id);
     while (1) {
         if (unlikely(suricata_ctl_flags != 0)) {
             SCLogDebug("Stopping Suricata!");

From 22ffdbb1b32222fd9be7efb9efe96fce653d5904 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 4 Sep 2023 15:51:16 +0200
Subject: [PATCH 042/462] mime: process chunk as soon as possible

In the case stream depth gets reached afterwards, it cannot be
processed after that.

Ticket: #6367
---
 src/util-decode-mime.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/util-decode-mime.c b/src/util-decode-mime.c
index b22a8c2e6f61..8e8b426c44a8 100644
--- a/src/util-decode-mime.c
+++ b/src/util-decode-mime.c
@@ -1375,17 +1375,22 @@ static int ProcessBase64BodyLine(const uint8_t *buf, uint32_t len,
          * size. We strip of spaces this while storing it in bvremain */
         if (consumed_bytes == 0 && leftover_bytes > B64_BLOCK) {
             DEBUG_VALIDATE_BUG_ON(state->bvr_len != 0);
-            return ProcessBase64BodyLineCopyRemainder(buf, len, offset, state);
+            ret = ProcessBase64BodyLineCopyRemainder(buf, len, offset, state);
+            break;
         } else if (leftover_bytes > 0 && leftover_bytes <= B64_BLOCK) {
             /* If remaining is 4 by this time, we encountered spaces during processing */
             DEBUG_VALIDATE_BUG_ON(state->bvr_len != 0);
-            return ProcessBase64BodyLineCopyRemainder(buf, len, offset + consumed_bytes, state);
+            ret = ProcessBase64BodyLineCopyRemainder(buf, len, offset + consumed_bytes, state);
+            break;
         }
 
         /* Update counts */
         remaining = leftover_bytes;
         offset += consumed_bytes;
     }
+    if (ret == MIME_DEC_OK && state->data_chunk_len > 0) {
+        ret = ProcessDecodedDataChunk(state->data_chunk, state->data_chunk_len, state);
+    }
     return ret;
 }
 

From 27a665546bb5a71633d5fa0396b2ba46eb119e13 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Fri, 8 Sep 2023 10:04:31 -0400
Subject: [PATCH 043/462] detect/bytejump: Change DoMatch signature to return
 bool

Issue: 4624

Change the function signature of byte-jump's domatch from an int to a
bool to avoid ambiguity handling return values.
---
 src/detect-bytejump.c                  | 22 +++++++++++-----------
 src/detect-bytejump.h                  | 12 +++---------
 src/detect-engine-content-inspection.c |  4 ++--
 3 files changed, 16 insertions(+), 22 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 63d70aad5919..3896e9d3bcb3 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -132,10 +132,10 @@ static bool DetectBytejumpValidateNbytes(const DetectBytejumpData *data, int32_t
  *  \param m byte jump sigmatch
  *  \param payload ptr to the payload
  *  \param payload_len length of the payload
- *  \retval 1 match
- *  \retval 0 no match
+ *  \retval true match
+ *  \retval false no match
  */
-int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
+bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         const SigMatchCtx *ctx, const uint8_t *payload, uint32_t payload_len, uint16_t flags,
         int32_t nbytes, int32_t offset)
 {
@@ -148,7 +148,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     int extbytes;
 
     if (payload_len == 0) {
-        SCReturnInt(0);
+        SCReturnBool(false);
     }
 
     /* Validate the number of bytes we are testing
@@ -161,7 +161,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
             SCLogDebug("Invalid byte_jump nbytes "
                        "seen in byte_jump - %d",
                     nbytes);
-            SCReturnInt(0);
+            SCReturnBool(false);
         }
     }
 
@@ -177,7 +177,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
 
         /* No match if there is no relative base */
         if (ptr == NULL || len <= 0) {
-            SCReturnInt(0);
+            SCReturnBool(false);
         }
     }
     else {
@@ -190,7 +190,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         SCLogDebug("Data not within payload "
                    "pkt=%p, ptr=%p, len=%d, nbytes=%d",
                 payload, ptr, len, nbytes);
-        SCReturnInt(0);
+        SCReturnBool(false);
     }
 
     /* Extract the byte data */
@@ -198,7 +198,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         extbytes = ByteExtractStringUint64(&val, data->base, nbytes, (const char *)ptr);
         if(extbytes <= 0) {
             SCLogDebug("error extracting %d bytes of string data: %d", nbytes, extbytes);
-            SCReturnInt(0);
+            SCReturnBool(false);
         }
     }
     else {
@@ -206,7 +206,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         extbytes = ByteExtractUint64(&val, endianness, (uint16_t)nbytes, ptr);
         if (extbytes != nbytes) {
             SCLogDebug("error extracting %d bytes of numeric data: %d", nbytes, extbytes);
-            SCReturnInt(0);
+            SCReturnBool(false);
         }
     }
 
@@ -239,7 +239,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         SCLogDebug("Jump location (%" PRIu64 ") is not within "
                    "payload (%" PRIu32 ")",
                 val, payload_len);
-        SCReturnInt(0);
+        SCReturnBool(false);
     }
 
 #ifdef DEBUG
@@ -252,7 +252,7 @@ int DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     /* Adjust the detection context to the jump location. */
     det_ctx->buffer_offset = val;
 
-    SCReturnInt(1);
+    SCReturnBool(true);
 }
 
 static int DetectBytejumpMatch(DetectEngineThreadCtx *det_ctx,
diff --git a/src/detect-bytejump.h b/src/detect-bytejump.h
index e1850ec77041..f8ee530b3864 100644
--- a/src/detect-bytejump.h
+++ b/src/detect-bytejump.h
@@ -68,16 +68,10 @@ void DetectBytejumpRegister (void);
  * \param p pointer to the current packet
  * \param m pointer to the sigmatch that we will cast into DetectBytejumpData
  *
- * \retval -1 error
- * \retval  0 no match
- * \retval  1 match
- *
- * \todo The return seems backwards.  We should return a non-zero error code.
- *       One of the error codes is "no match".  As-is if someone accidentally
- *       does: if (DetectBytejumpMatch(...)) { match }, then they catch an
- *       error as a match.
+ * \retval  false no match
+ * \retval  true
  */
-int DetectBytejumpDoMatch(DetectEngineThreadCtx *, const Signature *, const SigMatchCtx *,
+bool DetectBytejumpDoMatch(DetectEngineThreadCtx *, const Signature *, const SigMatchCtx *,
         const uint8_t *, uint32_t, uint16_t, int32_t, int32_t);
 
 #endif /* __DETECT_BYTEJUMP_H__ */
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 77ebb3f82723..ae7102f9c061 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -534,8 +534,8 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
                       DETECT_BYTEJUMP_LITTLE: 0);
         }
 
-        if (DetectBytejumpDoMatch(
-                    det_ctx, s, smd->ctx, buffer, buffer_len, bjflags, nbytes, offset) != 1) {
+        if (!DetectBytejumpDoMatch(
+                    det_ctx, s, smd->ctx, buffer, buffer_len, bjflags, nbytes, offset)) {
             goto no_match;
         }
 

From 2bf9d0fdf9778b48c3db8d39e51c6129e19213a3 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Fri, 8 Sep 2023 10:09:52 -0400
Subject: [PATCH 044/462] detect/bytejump: Improve negative post_offset
 handling.

Issue: 4624

Handle negative post_offset values that jump before the buffer as though
they refer to the buffer start.
---
 src/detect-bytejump.c | 50 +++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 19 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 3896e9d3bcb3..ca1b72534845 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -168,9 +168,11 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     /* Calculate the ptr value for the bytejump and length remaining in
      * the packet from that point.
      */
+    ptr = payload;
+    len = payload_len;
     if (flags & DETECT_BYTEJUMP_RELATIVE) {
-        ptr = payload + det_ctx->buffer_offset;
-        len = payload_len - det_ctx->buffer_offset;
+        ptr += det_ctx->buffer_offset;
+        len -= det_ctx->buffer_offset;
 
         ptr += offset;
         len -= offset;
@@ -181,14 +183,14 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         }
     }
     else {
-        ptr = payload + offset;
-        len = payload_len - offset;
+        ptr += offset;
+        len -= offset;
     }
 
     /* Verify the to-be-extracted data is within the packet */
     if (ptr < payload || nbytes > len) {
         SCLogDebug("Data not within payload "
-                   "pkt=%p, ptr=%p, len=%d, nbytes=%d",
+                   "pkt=%p, ptr=%p, len=%" PRIi32 ", nbytes=%" PRIi32,
                 payload, ptr, len, nbytes);
         SCReturnBool(false);
     }
@@ -210,7 +212,8 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         }
     }
 
-    SCLogDebug("VAL: (%" PRIu64 " x %" PRIu32 ") + %d + %" PRId32, val, data->multiplier, extbytes, data->post_offset);
+    SCLogDebug("VAL: (%" PRIu64 " x %" PRIu32 ") + %" PRIi32 " + %" PRId32, val, data->multiplier,
+            extbytes, data->post_offset);
 
     /* Adjust the jump value based on flags */
     val *= data->multiplier;
@@ -220,25 +223,31 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
         }
     }
     val += data->post_offset;
+    SCLogDebug("val: %" PRIi64 " post_offset: %" PRIi32, val, data->post_offset);
 
+    const uint8_t *jumpptr;
     /* Calculate the jump location */
     if (flags & DETECT_BYTEJUMP_BEGIN) {
-        SCLogDebug("NEWVAL: payload %p + %" PRIu64, payload, val);
+        jumpptr = payload + (int64_t)val;
+        SCLogDebug("NEWVAL: payload %p + %" PRIi64 " = %p\n", payload, (int64_t)val, jumpptr + val);
     } else if (flags & DETECT_BYTEJUMP_END) {
-        val = payload_len + val;
-        SCLogDebug("NEWVAL: payload %p + %" PRIu32 " - %" PRIu64, payload, payload_len, val);
+        jumpptr = payload + payload_len + (int64_t)val;
+        SCLogDebug(
+                "NEWVAL: payload %p + %" PRIu32 " + %" PRIi64, payload, payload_len, (int64_t)val);
     } else {
-        val += (ptr - payload) + extbytes;
-        SCLogDebug("NEWVAL: ptr %p + %" PRIu64, ptr, val);
+        jumpptr = ptr + (int64_t)val + extbytes;
+        SCLogDebug("NEWVAL: ptr %p + %" PRIi64 " = %p\n", ptr, val, jumpptr);
     }
 
     /* Validate that the jump location is still in the packet
      * \todo Should this validate it is still in the *payload*?
      */
-    if (val >= payload_len) {
-        SCLogDebug("Jump location (%" PRIu64 ") is not within "
-                   "payload (%" PRIu32 ")",
-                val, payload_len);
+    if (jumpptr < payload) {
+        jumpptr = payload;
+        SCLogDebug("jump location is before buffer start; resetting to buffer start");
+    } else if (jumpptr >= (payload + payload_len)) {
+        SCLogDebug("Jump location (%" PRIu64 ") is not within payload (%" PRIu32 ")",
+                payload_len + val, payload_len);
         SCReturnBool(false);
     }
 
@@ -250,7 +259,8 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
 #endif /* DEBUG */
 
     /* Adjust the detection context to the jump location. */
-    det_ctx->buffer_offset = val;
+    DEBUG_VALIDATE_BUG_ON(jumpptr < payload);
+    det_ctx->buffer_offset = jumpptr - payload;
 
     SCReturnBool(true);
 }
@@ -479,7 +489,8 @@ static DetectBytejumpData *DetectBytejumpParse(
         if (*offset == NULL)
             goto error;
     } else {
-        if (StringParseInt32(&data->offset, 0, (uint16_t)strlen(args[1]), args[1]) <= 0) {
+        if (StringParseI32RangeCheck(
+                    &data->offset, 10, (uint16_t)strlen(args[1]), args[1], -65535, 65535) <= 0) {
             SCLogError("Malformed offset: %s", optstr);
             goto error;
         }
@@ -518,11 +529,12 @@ static DetectBytejumpData *DetectBytejumpParse(
                 goto error;
             }
         } else if (strncasecmp("post_offset ", args[i], 12) == 0) {
-            if (StringParseInt32(&data->post_offset, 10, (uint16_t)strlen(args[i]) - 12,
-                        args[i] + 12) <= 0) {
+            if (StringParseI32RangeCheck(&data->post_offset, 10, (uint16_t)strlen(args[i]) - 12,
+                        args[i] + 12, -65535, 65535) <= 0) {
                 SCLogError("Malformed post_offset: %s", optstr);
                 goto error;
             }
+            SCLogDebug("post_offset: %s [%d]", optstr, data->post_offset);
         } else if (strcasecmp("dce", args[i]) == 0) {
             data->flags |= DETECT_BYTEJUMP_DCE;
         } else {

From ad5f41c95c327a7b39df4d74e3a9a4cc9646f096 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 28 Sep 2023 21:17:00 +0200
Subject: [PATCH 045/462] detect: remove code writing unused values

Coveridy ID 1546822 1546823 and 1546824
---
 src/detect-engine-address.c | 1 -
 src/detect-engine-build.c   | 2 --
 src/util-threshold-config.c | 1 -
 3 files changed, 4 deletions(-)

diff --git a/src/detect-engine-address.c b/src/detect-engine-address.c
index 287517194735..ac10e142dedc 100644
--- a/src/detect-engine-address.c
+++ b/src/detect-engine-address.c
@@ -1289,7 +1289,6 @@ int DetectAddressTestConfVars(void)
         }
 
         DetectAddressHeadFree(gh);
-        gh = NULL;
         DetectAddressHeadFree(ghn);
         ghn = NULL;
     }
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index e957b50bdb07..8b7621271983 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1645,8 +1645,6 @@ int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, Detect
             /* when a group's sigs are added to the joingr, we can free it */
             gr->next = NULL;
             DetectPortFree(de_ctx, gr);
-            gr = NULL;
-
         /* append */
         } else {
             gr->next = NULL;
diff --git a/src/util-threshold-config.c b/src/util-threshold-config.c
index 74e58fbc8c76..5d762a8f7091 100644
--- a/src/util-threshold-config.c
+++ b/src/util-threshold-config.c
@@ -510,7 +510,6 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
                 if (sm != NULL) {
                     SigMatchRemoveSMFromList(s, sm, DETECT_SM_LIST_THRESHOLD);
                     SigMatchFree(de_ctx, sm);
-                    sm = NULL;
                 }
             }
 

From 77e1134ee144eb558d5fd644297dbee0627611ab Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 28 Sep 2023 16:15:14 -0600
Subject: [PATCH 046/462] readme: formatting

Use consistent header style and wrap at 80 chars for better reading in
a terminal.
---
 README.md | 130 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 78 insertions(+), 52 deletions(-)

diff --git a/README.md b/README.md
index 4c1f5ad213bb..3c8e3d65a627 100644
--- a/README.md
+++ b/README.md
@@ -1,118 +1,144 @@
-Suricata
-========
+# Suricata
 
 [![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/suricata.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:suricata)
 [![codecov](https://codecov.io/gh/OISF/suricata/branch/master/graph/badge.svg?token=QRyyn2BSo1)](https://codecov.io/gh/OISF/suricata)
 
-Introduction
-------------
+## Introduction
 
-[Suricata](https://suricata.io) is a network IDS, IPS and NSM engine developed by the [OISF](https://oisf.net) and the Suricata community.
+[Suricata](https://suricata.io) is a network IDS, IPS and NSM engine
+developed by the [OISF](https://oisf.net) and the Suricata community.
 
-Installation
-------------
+## Installation
 
 https://docs.suricata.io/en/latest/install.html
 
-User Guide
-----------
+## User Guide
 
-You can follow the [Suricata user guide](https://docs.suricata.io/en/latest/) to get started.
+You can follow the [Suricata user
+guide](https://docs.suricata.io/en/latest/) to get started.
 
-Contributing
-------------
+## Contributing
 
 We're happily taking patches and other contributions. Please see our
-[Contribution Process](https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html)
+[Contribution
+Process](https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html)
 for how to get started.
 
-Suricata is a complex piece of software dealing with mostly untrusted input. Mishandling this input will have serious consequences:
+Suricata is a complex piece of software dealing with mostly untrusted
+input. Mishandling this input will have serious consequences:
 
 * in IPS mode a crash may knock a network offline;
-* in passive mode a compromise of the IDS may lead to loss of critical and confidential data;
+* in passive mode a compromise of the IDS may lead to loss of critical
+  and confidential data;
 * missed detection may lead to undetected compromise of the network.
 
-In other words, we think the stakes are pretty high, especially since in many common cases the IDS/IPS will be directly reachable by an attacker.
+In other words, we think the stakes are pretty high, especially since
+in many common cases the IDS/IPS will be directly reachable by an
+attacker.
 
-For this reason, we have developed a QA process that is quite extensive. A consequence is that contributing to Suricata can be a somewhat lengthy process.
+For this reason, we have developed a QA process that is quite
+extensive. A consequence is that contributing to Suricata can be a
+somewhat lengthy process.
 
 On a high level, the steps are:
 
-1. GitHub-CI based checks. This runs automatically when a pull request is made.
-
+1. GitHub-CI based checks. This runs automatically when a pull request
+   is made.
 2. Review by devs from the team and community
-
-3. QA runs from private QA setups. These are private due to the nature of the test traffic.
-
+3. QA runs from private QA setups. These are private due to the nature
+   of the test traffic.
 
 ### Overview of Suricata's QA steps
 
-OISF team members are able to submit builds to our private QA setup. It will run a series of build tests and a regression suite to confirm no existing features break.
+OISF team members are able to submit builds to our private QA
+setup. It will run a series of build tests and a regression suite to
+confirm no existing features break.
 
-The final QA runs takes a few hours minimally, and generally runs overnight. It currently runs:
+The final QA runs takes a few hours minimally, and generally runs
+overnight. It currently runs:
 
-- extensive build tests on different OS', compilers, optimization levels, configure features
+- extensive build tests on different OS', compilers, optimization
+  levels, configure features
 - static code analysis using cppcheck, scan-build
-- runtime code analysis using valgrind, AddressSanitizer, LeakSanitizer
+- runtime code analysis using valgrind, AddressSanitizer,
+  LeakSanitizer
 - regression tests for past bugs
 - output validation of logging
 - unix socket testing
 - pcap based fuzz testing using ASAN and LSAN
 - traffic replay based IDS and IPS tests
 
-Next to these tests, based on the type of code change further tests can be run manually:
+Next to these tests, based on the type of code change further tests
+can be run manually:
 
 - traffic replay testing (multi-gigabit)
 - large pcap collection processing (multi-terabytes)
 - fuzz testing (might take multiple days or even weeks)
 - pcap based performance testing
 - live performance testing
-- various other manual tests based on evaluation of the proposed changes
-
-
-It's important to realize that almost all of the tests above are used as acceptance tests. If something fails, it's up to you to address this in your code.
-
-
-One step of the QA is currently run post-merge. We submit builds to the Coverity Scan program. Due to limitations of this (free) service, we can submit once a day max.
-Of course it can happen that after the merge the community will find issues. For both cases we request you to help address the issues as they may come up.
+- various other manual tests based on evaluation of the proposed
+  changes
 
+It's important to realize that almost all of the tests above are used
+as acceptance tests. If something fails, it's up to you to address
+this in your code.
 
+One step of the QA is currently run post-merge. We submit builds to
+the Coverity Scan program. Due to limitations of this (free) service,
+we can submit once a day max.  Of course it can happen that after the
+merge the community will find issues. For both cases we request you to
+help address the issues as they may come up.
 
-
-### FAQ
+## FAQ
 
 __Q: Will you accept my PR?__
 
-A: That depends on a number of things, including the code quality. With new features it also depends on whether the team and/or the community think the feature is useful, how much it affects other code and features, the risk of performance regressions, etc.
-
+A: That depends on a number of things, including the code
+quality. With new features it also depends on whether the team and/or
+the community think the feature is useful, how much it affects other
+code and features, the risk of performance regressions, etc.
 
 __Q: When will my PR be merged?__
 
-A: It depends, if it's a major feature or considered a high risk change, it will probably go into the next major version.
-
+A: It depends, if it's a major feature or considered a high risk
+change, it will probably go into the next major version.
 
 __Q: Why was my PR closed?__
 
-A: As documented in the [Suricata GitHub workflow](https://docs.suricata.io/en/latest/devguide/codebase/contributing/github-pr-workflow.html),
+A: As documented in the [Suricata GitHub
+workflow](https://docs.suricata.io/en/latest/devguide/codebase/contributing/github-pr-workflow.html),
 we expect a new pull request for every change.
 
-Normally, the team (or community) will give feedback on a pull request after which
-it is expected to be replaced by an improved PR. So look at the comments. If you
-disagree with the comments we can still discuss them in the closed PR.
-
-If the PR was closed without comments it's likely due to QA failure. If the GitHub-CI checks failed, the PR should be fixed right away. No need for a discussion about it, unless you believe the QA failure is incorrect.
+Normally, the team (or community) will give feedback on a pull request
+after which it is expected to be replaced by an improved PR. So look
+at the comments. If you disagree with the comments we can still
+discuss them in the closed PR.
 
+If the PR was closed without comments it's likely due to QA
+failure. If the GitHub-CI checks failed, the PR should be fixed right
+away. No need for a discussion about it, unless you believe the QA
+failure is incorrect.
 
 __Q: the compiler/code analyser/tool is wrong, what now?__
 
-A: To assist in the automation of the QA, we're not accepting warnings or errors to stay. In some cases this could mean that we add a suppression if the tool supports that (e.g. valgrind, DrMemory). Some warnings can be disabled. In some exceptional cases the only 'solution' is to refactor the code to work around a static code checker limitation false positive. While frustrating, we prefer this over leaving warnings in the output. Warnings tend to get ignored and then increase risk of hiding other warnings.
-
+A: To assist in the automation of the QA, we're not accepting warnings
+or errors to stay. In some cases this could mean that we add a
+suppression if the tool supports that (e.g. valgrind, DrMemory). Some
+warnings can be disabled. In some exceptional cases the only
+'solution' is to refactor the code to work around a static code
+checker limitation false positive. While frustrating, we prefer this
+over leaving warnings in the output. Warnings tend to get ignored and
+then increase risk of hiding other warnings.
 
 __Q: I think your QA test is wrong__
 
-A: If you really think it is, we can discuss how to improve it. But don't come to this conclusion too quickly, more often it's the code that turns out to be wrong.
-
+A: If you really think it is, we can discuss how to improve it. But
+don't come to this conclusion too quickly, more often it's the code
+that turns out to be wrong.
 
 __Q: do you require signing of a contributor license agreement?__
 
-A: Yes, we do this to keep the ownership of Suricata in one hand: the Open Information Security Foundation. See http://suricata.io/about/open-source/ and http://suricata.io/about/contribution-agreement/
+A: Yes, we do this to keep the ownership of Suricata in one hand: the
+Open Information Security Foundation. See
+http://suricata.io/about/open-source/ and
+http://suricata.io/about/contribution-agreement/

From c8ee69c8df6a2b2d034c82aed4e29a511cd520da Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 28 Sep 2023 16:19:51 -0600
Subject: [PATCH 047/462] readme: add a resources section

Consolidate a few items into a resources section, and add few more
items, most importantly the bug tracker as it can't currently be found
from our GitHub presence.
---
 README.md | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/README.md b/README.md
index 3c8e3d65a627..364ff0e6df97 100644
--- a/README.md
+++ b/README.md
@@ -8,14 +8,13 @@
 [Suricata](https://suricata.io) is a network IDS, IPS and NSM engine
 developed by the [OISF](https://oisf.net) and the Suricata community.
 
-## Installation
+## Resources
 
-https://docs.suricata.io/en/latest/install.html
-
-## User Guide
-
-You can follow the [Suricata user
-guide](https://docs.suricata.io/en/latest/) to get started.
+- [Home Page](https://suricata.io)
+- [Bug Tracker](https://redmine.openinfosecfoundation.org/projects/suricata)
+- [User Guide](https://docs.suricata.io)
+- [Installation Guide](https://docs.suricata.io/en/latest/install.html)
+- [User Support Forum](https://forum.suricata.io)
 
 ## Contributing
 

From bb15a8f76675520329d2deecb10523460d21e98c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 28 Sep 2023 16:25:41 -0600
Subject: [PATCH 048/462] readme: minor cleanups

---
 README.md | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/README.md b/README.md
index 364ff0e6df97..2b1a213bcbcb 100644
--- a/README.md
+++ b/README.md
@@ -26,10 +26,10 @@ for how to get started.
 Suricata is a complex piece of software dealing with mostly untrusted
 input. Mishandling this input will have serious consequences:
 
-* in IPS mode a crash may knock a network offline;
+* in IPS mode a crash may knock a network offline
 * in passive mode a compromise of the IDS may lead to loss of critical
-  and confidential data;
-* missed detection may lead to undetected compromise of the network.
+  and confidential data
+* missed detection may lead to undetected compromise of the network
 
 In other words, we think the stakes are pretty high, especially since
 in many common cases the IDS/IPS will be directly reachable by an
@@ -118,7 +118,7 @@ failure. If the GitHub-CI checks failed, the PR should be fixed right
 away. No need for a discussion about it, unless you believe the QA
 failure is incorrect.
 
-__Q: the compiler/code analyser/tool is wrong, what now?__
+__Q: The compiler/code analyser/tool is wrong, what now?__
 
 A: To assist in the automation of the QA, we're not accepting warnings
 or errors to stay. In some cases this could mean that we add a
@@ -135,7 +135,7 @@ A: If you really think it is, we can discuss how to improve it. But
 don't come to this conclusion too quickly, more often it's the code
 that turns out to be wrong.
 
-__Q: do you require signing of a contributor license agreement?__
+__Q: Do you require signing of a contributor license agreement?__
 
 A: Yes, we do this to keep the ownership of Suricata in one hand: the
 Open Information Security Foundation. See

From 3ecb923db121d8d20da2405f6087db250f2b4028 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 14 Sep 2023 11:44:19 -0300
Subject: [PATCH 049/462] detect/analyzer: add more details for ipopts

In addition to the ipopts keyword name, also log the ip option that was
matched on.

Task #6348
---
 src/detect-engine-analyzer.c | 10 ++++++++++
 src/detect-ipopts.c          | 33 +++++++++++++++++++++++++++++++++
 src/detect-ipopts.h          |  2 ++
 3 files changed, 45 insertions(+)

diff --git a/src/detect-engine-analyzer.c b/src/detect-engine-analyzer.c
index 1735ce35bb85..a37afabb0f00 100644
--- a/src/detect-engine-analyzer.c
+++ b/src/detect-engine-analyzer.c
@@ -39,6 +39,7 @@
 #include "detect-bytetest.h"
 #include "detect-flow.h"
 #include "detect-tcp-flags.h"
+#include "detect-ipopts.h"
 #include "feature.h"
 #include "util-print.h"
 #include "util-time.h"
@@ -851,6 +852,15 @@ static void DumpMatches(RuleAnalyzer *ctx, JsonBuilder *js, const SigMatchData *
                 jb_close(js);
                 break;
             }
+            case DETECT_IPOPTS: {
+                const DetectIpOptsData *cd = (const DetectIpOptsData *)smd->ctx;
+
+                jb_open_object(js, "ipopts");
+                const char *flag = IpOptsFlagToString(cd->ipopt);
+                jb_set_string(js, "option", flag);
+                jb_close(js);
+                break;
+            }
         }
         jb_close(js);
 
diff --git a/src/detect-ipopts.c b/src/detect-ipopts.c
index 07e6b7eac9b2..105751c388a4 100644
--- a/src/detect-ipopts.c
+++ b/src/detect-ipopts.c
@@ -119,6 +119,39 @@ struct DetectIpOpts_ {
     { NULL, 0 },
 };
 
+/**
+ * \brief Return human readable value for ipopts flag
+ *
+ * \param flag uint16_t DetectIpOptsData ipopts flag value
+ */
+const char *IpOptsFlagToString(uint16_t flag)
+{
+    switch (flag) {
+        case IPV4_OPT_FLAG_RR:
+            return "rr";
+        case IPV4_OPT_FLAG_LSRR:
+            return "lsrr";
+        case IPV4_OPT_FLAG_EOL:
+            return "eol";
+        case IPV4_OPT_FLAG_NOP:
+            return "nop";
+        case IPV4_OPT_FLAG_TS:
+            return "ts";
+        case IPV4_OPT_FLAG_SEC:
+            return "sec";
+        case IPV4_OPT_FLAG_ESEC:
+            return "esec";
+        case IPV4_OPT_FLAG_SSRR:
+            return "ssrr";
+        case IPV4_OPT_FLAG_SID:
+            return "satid";
+        case 0xffff:
+            return "any";
+        default:
+            return NULL;
+    }
+}
+
 /**
  * \internal
  * \brief This function is used to match ip option on a packet with those passed via ipopts:
diff --git a/src/detect-ipopts.h b/src/detect-ipopts.h
index 4089ea5ad655..a4009252d0b5 100644
--- a/src/detect-ipopts.h
+++ b/src/detect-ipopts.h
@@ -45,5 +45,7 @@ typedef struct DetectIpOptsData_ {
 
 void DetectIpOptsRegister (void);
 
+const char *IpOptsFlagToString(uint16_t flag);
+
 #endif /*__DETECT_IPOPTS_H__ */
 

From 299ee6ed5561f01575150b436d5db31485dab146 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 27 Sep 2023 14:15:18 +0200
Subject: [PATCH 050/462] detect: check if signature uses too many buffers

Ticket: #6104

The approach in master branch is to change the prototype of
SigMatchAppendSMToList so that it allocates itself the new SigMatch
This approach requires to change all the 100-ish calls to
SigMatchAppendSMToList and is thus quite a big change.

For branch 7, we still wanted to avoid the buffer overflow, but
did not want such an intrusive change, and still wanted to make
the signature invalid. Instead of changing the prototype of the
function, we make it return early, and set a flag in the signature
which can be later checked by SigValidate
---
 src/detect-parse.c | 8 +++++++-
 src/detect.h       | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index 33d739300dd8..2e798d7b1cbf 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -473,7 +473,8 @@ void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
                 s->init_data->curbuf == NULL) {
             if (SignatureInitDataBufferCheckExpand(s) < 0) {
                 SCLogError("failed to expand rule buffer array");
-                // return -1; TODO error handle
+                s->init_data->init_flags |= SIG_FLAG_INIT_OVERFLOW;
+                return;
             }
 
             /* initialize new buffer */
@@ -1904,6 +1905,11 @@ static int SigValidate(DetectEngineCtx *de_ctx, Signature *s)
         SCReturnInt(0);
     }
 
+    if (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW) {
+        SCLogError("rule %u tries to use too many buffers", s->id);
+        SCReturnInt(0);
+    }
+
     bool has_frame = false;
     bool has_app = false;
     bool has_pkt = false;
diff --git a/src/detect.h b/src/detect.h
index fd299c5047b2..69a5524e583c 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -283,6 +283,7 @@ typedef struct DetectPort_ {
     BIT_U32(8) /**< priority is explicitly set by the priority keyword */
 #define SIG_FLAG_INIT_FILEDATA              BIT_U32(9)  /**< signature has filedata keyword */
 #define SIG_FLAG_INIT_JA3                   BIT_U32(10) /**< signature has ja3 keyword */
+#define SIG_FLAG_INIT_OVERFLOW              BIT_U32(11) /**< signature has overflown buffers */
 
 /* signature mask flags */
 /** \note: additions should be added to the rule analyzer as well */

From 96a0e7016fd36458db52b1fca81b9f4a61e24105 Mon Sep 17 00:00:00 2001
From: Travis Green <travis@travisgreen.net>
Date: Wed, 28 Aug 2019 22:16:28 -0600
Subject: [PATCH 051/462] doc: add tcp flags documentation

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/header-keywords.rst | 54 +++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/doc/userguide/rules/header-keywords.rst b/doc/userguide/rules/header-keywords.rst
index 3b45788e07b8..36d1437647f3 100644
--- a/doc/userguide/rules/header-keywords.rst
+++ b/doc/userguide/rules/header-keywords.rst
@@ -293,6 +293,60 @@ Example of tos with a negated value:
 TCP keywords
 ------------
 
+tcp.flags
+^^^^^^^^^
+
+The tcp.flags keyword checks for specific `TCP flag bits
+<https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_segment_structure>`_.
+
+The following flag bits may be checked:
+
+====  ====================================
+Flag  Description
+====  ====================================
+F     FIN - Finish
+S     SYN - Synchronize sequence numbers
+R     RST - Reset
+P     PSH - Push
+A     ACK - Acknowledgment
+U     URG - Urgent
+C     CWR - Congestion Window Reduced
+E     ECE - ECN-Echo
+0     No TCP Flags Set
+====  ====================================
+
+The following modifiers can be set to change the match criteria:
+
+========  ===================================
+Modifier  Description
+========  ===================================
+``+``     match on the bits, plus any others
+``*``     match if any of the bits are set
+``!``     match if the bits are not set
+========  ===================================
+
+To handle writing rules for session initiation packets such as ECN where a SYN
+packet is sent with CWR and ECE flags set, an option mask may be used by
+appending a comma and masked values. For example, a rule that checks for a SYN
+flag, regardless of the values of the reserved bits is ``tcp.flags:S,CE;``
+
+Format of tcp.flags::
+
+    tcp.flags:[modifier]<test flags>[,<ignore flags>];
+    tcp.flags:[!|*|+]<FSRPAUCE0>[,<FSRPAUCE>];
+
+Example::
+
+  alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"Example tcp.flags sig"; \
+ :example-rule-emphasis:`tcp.flags:FPU,CE;` classtype:misc-activity; sid:1; rev:1;)
+
+It is also possible to use the `tcp.flags` content as a fast_pattern by using the `prefilter` keyword. For more information on `prefilter` usage see :doc:`prefilter-keywords`
+
+Example::
+
+  alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"Example tcp.flags sig"; \
+ :example-rule-emphasis:`tcp.flags:FPU,CE; prefilter;` classtype:misc-activity; sid:1; rev:1;)  
+
 seq
 ^^^
 The seq keyword can be used in a signature to check for a specific TCP

From 915707090762a0be84fc38ec458baebb05f92252 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 27 Sep 2023 13:08:33 +0200
Subject: [PATCH 052/462] quic: v2 support per rfc 9369

Ticket: #4968
---
 rust/src/quic/crypto.rs | 40 ++++++++++++++++++++++++++++++----------
 rust/src/quic/parser.rs | 13 +++++++++++++
 2 files changed, 43 insertions(+), 10 deletions(-)

diff --git a/rust/src/quic/crypto.rs b/rust/src/quic/crypto.rs
index 9b2857efef24..d40e84527468 100644
--- a/rust/src/quic/crypto.rs
+++ b/rust/src/quic/crypto.rs
@@ -32,10 +32,15 @@ pub const AES128_IV_LEN: usize = 12;
 pub struct HeaderProtectionKey(Aes128);
 
 impl HeaderProtectionKey {
-    fn new(secret: &[u8]) -> Self {
+    fn new(secret: &[u8], version: u32) -> Self {
         let hk = Hkdf::<Sha256>::from_prk(secret).unwrap();
         let mut secret = [0u8; AES128_KEY_LEN];
-        hkdf_expand_label(&hk, b"quic hp", &mut secret, AES128_KEY_LEN as u16);
+        let quichp = if version == 0x6b3343cf {
+            b"quicv2 hp" as &[u8]
+        } else {
+            b"quic hp" as &[u8]
+        };
+        hkdf_expand_label(&hk, quichp, &mut secret, AES128_KEY_LEN as u16);
         return Self(Aes128::new(GenericArray::from_slice(&secret)));
     }
 
@@ -70,17 +75,27 @@ pub struct PacketKey {
 }
 
 impl PacketKey {
-    fn new(secret: &[u8]) -> Self {
+    fn new(secret: &[u8], version: u32) -> Self {
         let hk = Hkdf::<Sha256>::from_prk(secret).unwrap();
         let mut secret = [0u8; AES128_KEY_LEN];
-        hkdf_expand_label(&hk, b"quic key", &mut secret, AES128_KEY_LEN as u16);
+        let quickey = if version == 0x6b3343cf {
+            b"quicv2 key" as &[u8]
+        } else {
+            b"quic key" as &[u8]
+        };
+        hkdf_expand_label(&hk, quickey, &mut secret, AES128_KEY_LEN as u16);
         let key = Aes128Gcm::new(GenericArray::from_slice(&secret));
 
         let mut r = PacketKey {
             key,
             iv: [0u8; AES128_IV_LEN],
         };
-        hkdf_expand_label(&hk, b"quic iv", &mut r.iv, AES128_IV_LEN as u16);
+        let quiciv = if version == 0x6b3343cf {
+            b"quicv2 iv" as &[u8]
+        } else {
+            b"quic iv" as &[u8]
+        };
+        hkdf_expand_label(&hk, quiciv, &mut r.iv, AES128_IV_LEN as u16);
         return r;
     }
 
@@ -111,10 +126,10 @@ pub struct DirectionalKeys {
 }
 
 impl DirectionalKeys {
-    fn new(secret: &[u8]) -> Self {
+    fn new(secret: &[u8], version: u32) -> Self {
         Self {
-            header: HeaderProtectionKey::new(secret),
-            packet: PacketKey::new(secret),
+            header: HeaderProtectionKey::new(secret, version),
+            packet: PacketKey::new(secret, version),
         }
     }
 }
@@ -163,6 +178,11 @@ pub fn quic_keys_initial(version: u32, client_dst_connection_id: &[u8]) -> Optio
             0x38, 0x76, 0x2c, 0xf7, 0xf5, 0x59, 0x34, 0xb3, 0x4d, 0x17, 0x9a, 0xe6, 0xa4, 0xc8,
             0x0c, 0xad, 0xcc, 0xbb, 0x7f, 0x0a,
         ],
+        0x6b3343cf => &[
+            // https://www.rfc-editor.org/rfc/rfc9369.html#section-3.3.1
+            0x0d, 0xed, 0xe3, 0xde, 0xf7, 0x00, 0xa6, 0xdb, 0x81, 0x93, 0x81, 0xbe, 0x6e, 0x26,
+            0x9d, 0xcb, 0xf9, 0xbd, 0x2e, 0xd9,
+        ],
         _ => {
             return None;
         }
@@ -174,7 +194,7 @@ pub fn quic_keys_initial(version: u32, client_dst_connection_id: &[u8]) -> Optio
     hkdf_expand_label(&hk, b"server in", &mut server_secret, 32);
 
     return Some(QuicKeys {
-        local: DirectionalKeys::new(&server_secret),
-        remote: DirectionalKeys::new(&client_secret),
+        local: DirectionalKeys::new(&server_secret, version),
+        remote: DirectionalKeys::new(&client_secret, version),
     });
 }
diff --git a/rust/src/quic/parser.rs b/rust/src/quic/parser.rs
index 5d9d700080b0..126973633bac 100644
--- a/rust/src/quic/parser.rs
+++ b/rust/src/quic/parser.rs
@@ -44,6 +44,7 @@ impl QuicVersion {
     pub const Q044: QuicVersion = QuicVersion(0x51303434);
     pub const Q045: QuicVersion = QuicVersion(0x51303435);
     pub const Q046: QuicVersion = QuicVersion(0x51303436);
+    pub const V2: QuicVersion = QuicVersion(0x6b3343cf);
 
     fn is_gquic(&self) -> bool {
         *self == QuicVersion::Q043
@@ -61,6 +62,7 @@ impl From<QuicVersion> for String {
             QuicVersion(0x51303434) => "Q044".to_string(),
             QuicVersion(0x51303435) => "Q045".to_string(),
             QuicVersion(0x51303436) => "Q046".to_string(),
+            QuicVersion(0x6b3343cf) => "v2".to_string(),
             QuicVersion(x) => format!("{:x}", x),
         }
     }
@@ -286,7 +288,18 @@ impl QuicHeader {
                             return Err(nom7::Err::Error(QuicError::InvalidPacket));
                         }
                     }
+                } else if version == QuicVersion::V2 {
+                    match (first & 0x30) >> 4 {
+                        0x01 => QuicType::Initial,
+                        0x02 => QuicType::ZeroRTT,
+                        0x03 => QuicType::Handshake,
+                        0x00 => QuicType::Retry,
+                        _ => {
+                            return Err(nom7::Err::Error(QuicError::InvalidPacket));
+                        }
+                    }
                 } else {
+                    // consider as Quic version 1 (and latest drafts)
                     match (first & 0x30) >> 4 {
                         0x00 => QuicType::Initial,
                         0x01 => QuicType::ZeroRTT,

From 0a4011655f5bf7e1047223f53710562c1a9fbc2d Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Wed, 27 Sep 2023 12:06:37 +0530
Subject: [PATCH 053/462] doc/code-submission: add commit sign guide

---
 .../devguide/codebase/contributing/code-submission-process.rst   | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/userguide/devguide/codebase/contributing/code-submission-process.rst b/doc/userguide/devguide/codebase/contributing/code-submission-process.rst
index 06eb84eee79a..22bf16046a40 100644
--- a/doc/userguide/devguide/codebase/contributing/code-submission-process.rst
+++ b/doc/userguide/devguide/codebase/contributing/code-submission-process.rst
@@ -19,6 +19,7 @@ Commits
     * Description, wrapped at ~72 characters
 #. Commits should be individually compilable, starting with the oldest commit. Make sure that
    each commit can be built if it and the preceding commits in the PR are used.
+#. Commits should be authored with the format: "FirstName LastName <name@example.com>"
 
 Information that needs to be part of a commit (if applicable):
 

From 4ab4f711de1ad3424499f21a145f7f68d48c7173 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 4 Oct 2023 14:47:24 -0300
Subject: [PATCH 054/462] doc/install: link to devguide's install from git

Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.

Kept that reference, since it might still be useful for non-Ubuntu
cases.
---
 doc/userguide/install.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/install.rst b/doc/userguide/install.rst
index c484d0edd2bf..8eb0248013d9 100644
--- a/doc/userguide/install.rst
+++ b/doc/userguide/install.rst
@@ -364,5 +364,9 @@ Suricata packages.
 Advanced Installation
 ---------------------
 
-Various installation guides for installing from GIT and for other operating systems are maintained at:
+If you are using Ubuntu, you can follow
+:doc:`devguide/codebase/installation-from-git`.
+
+For other various installation guides for installing from GIT and for other operating
+systems, please check (bear in mind that those may be somewhat outdated):
 https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricata_Installation

From ffed5eb3d3b79d276a72abc84b7286c74ed56ba8 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 4 Oct 2023 14:50:45 -0300
Subject: [PATCH 055/462] doc/quickstart: add software-properties instruction

This is indicated in the `Installation` section, but not in the
quickstart, and it felt like a valid addition, here, too.
---
 doc/userguide/quickstart.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/userguide/quickstart.rst b/doc/userguide/quickstart.rst
index 899de35fd4c3..d2bdda0fe8c0 100644
--- a/doc/userguide/quickstart.rst
+++ b/doc/userguide/quickstart.rst
@@ -11,6 +11,7 @@ It's assumed that you run a recent Ubuntu release as the official PPA can then
 be used for the installation. To install the latest stable Suricata version, follow
 the steps::
 
+    sudo apt-get install software-properties-common
     sudo add-apt-repository ppa:oisf/suricata-stable
     sudo apt update
     sudo apt install suricata jq

From 7406ac0fa595658c70ed3f13cf79656f2b0d290a Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Wed, 4 Oct 2023 13:14:41 +0530
Subject: [PATCH 056/462] output/email: use SCCalloc for OutputJsonEmailCtx

email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on

In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.

Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.

Bug 6380
---
 src/output-json-smtp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/output-json-smtp.c b/src/output-json-smtp.c
index cc3003907585..f7674687c5c4 100644
--- a/src/output-json-smtp.c
+++ b/src/output-json-smtp.c
@@ -122,7 +122,7 @@ static OutputInitResult OutputSmtpLogInitSub(ConfNode *conf, OutputCtx *parent_c
     OutputInitResult result = { NULL, false };
     OutputJsonCtx *ojc = parent_ctx->data;
 
-    OutputJsonEmailCtx *email_ctx = SCMalloc(sizeof(OutputJsonEmailCtx));
+    OutputJsonEmailCtx *email_ctx = SCCalloc(1, sizeof(OutputJsonEmailCtx));
     if (unlikely(email_ctx == NULL))
         return result;
 

From 14a4c6c696ea8c3eef0b6fb5fb53918e4efe0ad1 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 3 Oct 2023 11:19:02 +0200
Subject: [PATCH 057/462] rust: update brotli decompressor crate

cf https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=59687
---
 rust/Cargo.lock.in | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rust/Cargo.lock.in b/rust/Cargo.lock.in
index a52fca5254d8..1800ba44f0ca 100644
--- a/rust/Cargo.lock.in
+++ b/rust/Cargo.lock.in
@@ -155,9 +155,9 @@ dependencies = [
 
 [[package]]
 name = "brotli-decompressor"
-version = "2.3.4"
+version = "2.3.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4b6561fd3f895a11e8f72af2cb7d22e08366bebc2b6b57f7744c4bda27034744"
+checksum = "503a0bcf59056a66c55d8eefd05e9c0f00f9c9cdddbb6bd499623ce49100da43"
 dependencies = [
  "alloc-no-stdlib",
  "alloc-stdlib",

From 1a132f454a64f699118dafcdfccb0687317b435e Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 5 Oct 2023 13:13:27 -0300
Subject: [PATCH 058/462] docs: adjust readthedocs config to new options

Our documentation was failing to build, seems connected to the new way
of indicating build options (cf
https://readthedocs.org/projects/suricata/builds/22112658/,
https://docs.readthedocs.io/en/stable/config-file/v2.html#build,
and https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os).

Added the build.os required new field, and adjusted the way python
version is passed.

For the new configuration style for read the docs, one of the ways to
pass extra configuration for python is having a requirements file.
---
 .readthedocs.yaml              | 17 ++++++++++++-----
 doc/userguide/requirements.txt |  1 +
 2 files changed, 13 insertions(+), 5 deletions(-)
 create mode 100644 doc/userguide/requirements.txt

diff --git a/.readthedocs.yaml b/.readthedocs.yaml
index 4455c2901e55..e545e9cf3822 100644
--- a/.readthedocs.yaml
+++ b/.readthedocs.yaml
@@ -1,11 +1,18 @@
 # Required by Read The Docs
 version: 2
 
-formats: all
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
 
 python:
-  version: "3.8"
+  install:
+    - requirements: doc/userguide/requirements.txt
+
+sphinx:
+  builder: html
+  configuration: doc/userguide/conf.py
+  fail_on_warning: false
 
-  # Use an empty install section to avoid RTD from picking up a non-python
-  # requirements.txt file.
-  install: []
+formats: all
diff --git a/doc/userguide/requirements.txt b/doc/userguide/requirements.txt
new file mode 100644
index 000000000000..483a4e9600bd
--- /dev/null
+++ b/doc/userguide/requirements.txt
@@ -0,0 +1 @@
+sphinx_rtd_theme

From aa0db7bf9efbe0f120bf0a0318686616dfb189c7 Mon Sep 17 00:00:00 2001
From: Bruno Franca <Brunofrancadevsec@gmail.com>
Date: Sat, 7 Oct 2023 18:26:31 -0300
Subject: [PATCH 059/462] detect-ssh-proto-version: use FAIL macros in tests

Task #6337
---
 src/detect-ssh-proto-version.c | 39 +++++++++++-----------------------
 1 file changed, 12 insertions(+), 27 deletions(-)

diff --git a/src/detect-ssh-proto-version.c b/src/detect-ssh-proto-version.c
index 9115d8affb83..d357d3f801d1 100644
--- a/src/detect-ssh-proto-version.c
+++ b/src/detect-ssh-proto-version.c
@@ -286,12 +286,11 @@ static int DetectSshVersionTestParse01 (void)
 {
     DetectSshVersionData *ssh = NULL;
     ssh = DetectSshVersionParse(NULL, "1.0");
-    if (ssh != NULL && strncmp((char *) ssh->ver, "1.0", 3) == 0) {
-        DetectSshVersionFree(NULL, ssh);
-        return 1;
-    }
+    FAIL_IF_NULL(ssh);
+    FAIL_IF_NOT(strncmp((char *)ssh->ver, "1.0", 3) == 0);
+    DetectSshVersionFree(NULL, ssh);
 
-    return 0;
+    PASS;
 }
 
 /**
@@ -302,12 +301,10 @@ static int DetectSshVersionTestParse02 (void)
 {
     DetectSshVersionData *ssh = NULL;
     ssh = DetectSshVersionParse(NULL, "2_compat");
-    if (ssh->flags & SSH_FLAG_PROTOVERSION_2_COMPAT) {
-        DetectSshVersionFree(NULL, ssh);
-        return 1;
-    }
+    FAIL_IF_NOT(ssh->flags & SSH_FLAG_PROTOVERSION_2_COMPAT);
+    DetectSshVersionFree(NULL, ssh);
 
-    return 0;
+    PASS;
 }
 
 /**
@@ -318,27 +315,15 @@ static int DetectSshVersionTestParse03 (void)
 {
     DetectSshVersionData *ssh = NULL;
     ssh = DetectSshVersionParse(NULL, "2_com");
-    if (ssh != NULL) {
-        DetectSshVersionFree(NULL, ssh);
-        return 0;
-    }
+    FAIL_IF_NOT_NULL(ssh);
     ssh = DetectSshVersionParse(NULL, "");
-    if (ssh != NULL) {
-        DetectSshVersionFree(NULL, ssh);
-        return 0;
-    }
+    FAIL_IF_NOT_NULL(ssh);
     ssh = DetectSshVersionParse(NULL, ".1");
-    if (ssh != NULL) {
-        DetectSshVersionFree(NULL, ssh);
-        return 0;
-    }
+    FAIL_IF_NOT_NULL(ssh);
     ssh = DetectSshVersionParse(NULL, "lalala");
-    if (ssh != NULL) {
-        DetectSshVersionFree(NULL, ssh);
-        return 0;
-    }
+    FAIL_IF_NOT_NULL(ssh);
 
-    return 1;
+    PASS;
 }
 
 

From 535938d7f67715ec67ab0c8dd99aa3a670f89d0a Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Thu, 5 Oct 2023 21:04:26 +0000
Subject: [PATCH 060/462] doc: add tls.cert_chain_len docs

Ticket: #6386

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/tls-keywords.rst | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/doc/userguide/rules/tls-keywords.rst b/doc/userguide/rules/tls-keywords.rst
index a97ce3f32e4a..dc28c97cd583 100644
--- a/doc/userguide/rules/tls-keywords.rst
+++ b/doc/userguide/rules/tls-keywords.rst
@@ -278,3 +278,27 @@ Example::
     tls.random_bytes; content:"|57 5d 77 02 07 c2 9d be 24 01 cc f0 5d cd e1 d2 a5 86 9c 4a 3e ee 38 db 55 1a d9 bc|"; sid: 200076;)
 
 ``tls.random_bytes`` is a sticky buffer.
+
+tls.cert_chain_len
+------------------
+
+Matches on the TLS certificate chain length.
+
+tls.cert_chain_len supports `<, >, <>, !` and using an exact value.
+
+Example::
+
+  alert tls any any -> any any (msg:"cert chain exact value"; \
+ tls.cert_chain_len:1; classtype:misc-activity; sid:1; rev:1;)
+
+  alert tls any any -> any any (msg:"cert chain less than value"; \
+ tls.cert_chain_len:<2; classtype:misc-activity; sid:2; rev:1;)
+
+  alert tls any any -> any any (msg:"cert chain greater than value"; \
+ tls.cert_chain_len:>0; classtype:misc-activity; sid:2; rev:1;)
+
+  alert tls any any -> any any (msg:"cert chain greater than less than value";\
+ tls.cert_chain_len:0<>2; classtype:misc-activity; sid:3; rev:1;)
+
+  alert tls any any -> any any (msg:"cert chain not value"; \
+ tls.cert_chain_len:!2; classtype:misc-activity; sid:4; rev:1;)

From cf8b630ed2e62116989b3d27ab8213b28217e0ea Mon Sep 17 00:00:00 2001
From: Comfort Amaechi <comfortamaechi619@gmail.com>
Date: Sat, 7 Oct 2023 19:38:22 -0400
Subject: [PATCH 061/462] userguide: cover install-full and install-conf

Ticket: #6342
---
 doc/userguide/install.rst | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/install.rst b/doc/userguide/install.rst
index 8eb0248013d9..ad7a9e44747f 100644
--- a/doc/userguide/install.rst
+++ b/doc/userguide/install.rst
@@ -154,7 +154,7 @@ Recommended::
     cargo install --force cbindgen
 
 Compilation
-"""""""""""
+^^^^^^^^^^^
 
 Follow these steps from your Suricata directory::
 
@@ -165,6 +165,32 @@ Follow these steps from your Suricata directory::
     make -j8 # j is for paralleling, you may de/increase depending on your CPU
     make install # to install your Suricata compiled binary
 
+Auto-Setup
+^^^^^^^^^^
+
+You can also use the available auto-setup features of Suricata:
+
+::
+
+    ./configure && make && sudo make install-conf
+
+*make install-conf* would do the regular "make install" and then it would automatically
+create/setup all the necessary directories and ``suricata.yaml`` for you.
+
+::
+
+    ./configure && make && sudo make install-rules
+
+*make install-rules* would do the regular "make install" and then it would automatically
+download and set up the latest ruleset from Emerging Threats available for Suricata.
+
+::
+
+    ./configure && make && sudo make install-full
+
+*make install-full* would combine everything mentioned above (install-conf and install-rules)
+and will present you with a ready-to-run (configured and set-up) Suricata.
+
 .. _install-binary-packages:
 
 Binary packages

From 1bcea5a992ab80cd4ae658732029ffa06ff280f3 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Wed, 16 Aug 2023 23:51:10 +0200
Subject: [PATCH 062/462] dpdk: add hugepage hint to lower the amount of
 reserved hugepages

If a user allocates too many hugepages and those are largely not used
then Suricata suggests that the user can lower the amount of hugepages
and therefore save memory for other purposes.

Ticket: #5966
---
 configure.ac          |  1 +
 src/suricata-common.h |  4 ++
 src/suricata.c        |  2 +
 src/util-dpdk.c       | 99 +++++++++++++++++++++++++++++++++++++++++++
 src/util-dpdk.h       |  1 +
 5 files changed, 107 insertions(+)

diff --git a/configure.ac b/configure.ac
index 8bb752715f53..cf6b8f625256 100644
--- a/configure.ac
+++ b/configure.ac
@@ -137,6 +137,7 @@
     AC_CHECK_HEADERS([getopt.h])
     AC_CHECK_HEADERS([limits.h netdb.h netinet/in.h poll.h sched.h signal.h])
     AC_CHECK_HEADERS([stdarg.h stdint.h stdio.h stdlib.h stdbool.h string.h strings.h sys/ioctl.h])
+    AC_CHECK_HEADERS([math.h])
     AC_CHECK_HEADERS([syslog.h sys/prctl.h sys/socket.h sys/stat.h sys/syscall.h])
     AC_CHECK_HEADERS([sys/time.h time.h unistd.h sys/param.h])
     AC_CHECK_HEADERS([sys/ioctl.h linux/if_ether.h linux/if_packet.h linux/filter.h])
diff --git a/src/suricata-common.h b/src/suricata-common.h
index 47d578f72780..fe8ec179dff3 100644
--- a/src/suricata-common.h
+++ b/src/suricata-common.h
@@ -280,6 +280,10 @@ typedef unsigned char u_char;
 #include <magic.h>
 #endif
 
+#ifdef HAVE_MATH_H
+#include <math.h>
+#endif
+
 /* we need this to stringify the defines which are supplied at compiletime see:
    http://gcc.gnu.org/onlinedocs/gcc-3.4.1/cpp/Stringification.html#Stringification */
 #define xstr(s) str(s)
diff --git a/src/suricata.c b/src/suricata.c
index d9adcaf07b26..30e6490826c7 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -3037,6 +3037,8 @@ int SuricataMain(int argc, char **argv)
 
     PostRunStartedDetectSetup(&suricata);
 
+    DPDKEvaluateHugepages();
+
     SCPledge();
     SuricataMainLoop(&suricata);
 
diff --git a/src/util-dpdk.c b/src/util-dpdk.c
index c9c1d73d0314..089aa45674ae 100644
--- a/src/util-dpdk.c
+++ b/src/util-dpdk.c
@@ -24,6 +24,7 @@
 #include "suricata.h"
 #include "util-dpdk.h"
 #include "util-debug.h"
+#include "util-byte.h"
 
 void DPDKCleanupEAL(void)
 {
@@ -65,6 +66,104 @@ void DPDKFreeDevice(LiveDevice *ldev)
 #endif
 }
 
+static FILE *HugepagesMeminfoOpen(void)
+{
+    FILE *fp = fopen("/proc/meminfo", "r");
+    if (fp == NULL) {
+        SCLogInfo("Can't analyze hugepage usage: failed to open /proc/meminfo");
+    }
+    return fp;
+}
+
+static void HugepagesMeminfoClose(FILE *fp)
+{
+    if (fp) {
+        fclose(fp);
+    }
+}
+
+/**
+ * Parsing values of meminfo
+ *
+ * \param fp Opened file pointer for reading of file /proc/meminfo at beginning
+ * \param keyword Entry to look for e.g. "HugePages_Free:"
+ * \return n Value of the entry
+ * \return -1 On error
+ *
+ */
+static int32_t MemInfoParseValue(FILE *fp, const char *keyword)
+{
+    char path[256], value_str[64];
+    int32_t value = -1;
+
+    while (fscanf(fp, "%255s", path) != EOF) {
+        if (strcmp(path, keyword) == 0) {
+            if (fscanf(fp, "%63s", value_str) == EOF) {
+                SCLogDebug("%s: not followed by any number", keyword);
+                break;
+            }
+
+            if (StringParseInt32(&value, 10, 23, value_str) < 0) {
+                SCLogDebug("Failed to convert %s from /proc/meminfo", keyword);
+                value = -1;
+            }
+            break;
+        }
+    }
+    return value;
+}
+
+static void MemInfoEvaluateHugepages(FILE *fp)
+{
+    int32_t free_hugepages = MemInfoParseValue(fp, "HugePages_Free:");
+    if (free_hugepages < 0) {
+        SCLogInfo("HugePages_Free information not found in /proc/meminfo");
+        return;
+    }
+
+    rewind(fp);
+
+    int32_t total_hugepages = MemInfoParseValue(fp, "HugePages_Total:");
+    if (total_hugepages < 0) {
+        SCLogInfo("HugePages_Total information not found in /proc/meminfo");
+        return;
+    } else if (total_hugepages == 0) {
+        SCLogInfo("HugePages_Total equals to zero");
+        return;
+    }
+
+    float free_hugepages_ratio = (float)free_hugepages / (float)total_hugepages;
+    if (free_hugepages_ratio > 0.5) {
+        SCLogInfo("%" PRIu32 " of %" PRIu32
+                  " of hugepages are free - number of hugepages can be lowered to e.g. %.0lf",
+                free_hugepages, total_hugepages, ceil((total_hugepages - free_hugepages) * 1.15));
+    }
+}
+
+static void MemInfoWith(void (*callback)(FILE *))
+{
+    FILE *fp = HugepagesMeminfoOpen();
+    if (fp) {
+        callback(fp);
+        HugepagesMeminfoClose(fp);
+    }
+}
+
+void DPDKEvaluateHugepages(void)
+{
+    if (run_mode != RUNMODE_DPDK)
+        return;
+
+#ifdef HAVE_DPDK
+    if (rte_eal_has_hugepages() == 0) { // hugepages disabled
+        SCLogPerf("Hugepages not enabled - enabling hugepages can improve performance");
+        return;
+    }
+#endif
+
+    MemInfoWith(MemInfoEvaluateHugepages);
+}
+
 #ifdef HAVE_DPDK
 
 /**
diff --git a/src/util-dpdk.h b/src/util-dpdk.h
index 1fb3532f5d4d..a94f46225217 100644
--- a/src/util-dpdk.h
+++ b/src/util-dpdk.h
@@ -121,6 +121,7 @@ void DPDKCleanupEAL(void);
 
 void DPDKCloseDevice(LiveDevice *ldev);
 void DPDKFreeDevice(LiveDevice *ldev);
+void DPDKEvaluateHugepages(void);
 
 #ifdef HAVE_DPDK
 const char *DPDKGetPortNameByPortID(uint16_t pid);

From 9dc83b6a43489e0b66dd494a9a48a46cb34ea782 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Wed, 23 Aug 2023 07:57:50 +0200
Subject: [PATCH 063/462] dpdk: add hugepage hint on low number of hugepages

If a user doesn't allocate/allocates too little hugepages,
Suricata fails to start and outputs a hint to increase
number of hugepages (if enabled).

Ticket: #5966
---
 src/runmode-dpdk.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/src/runmode-dpdk.c b/src/runmode-dpdk.c
index feba401b4130..2cdf5cb32505 100644
--- a/src/runmode-dpdk.c
+++ b/src/runmode-dpdk.c
@@ -1522,11 +1522,17 @@ static void *ParseDpdkConfigAndConfigureDevice(const char *iface)
 
     if (retval < 0) { // handles both configure attempts
         iconf->DerefFunc(iconf);
-        retval = rte_eal_cleanup();
-        if (retval != 0)
+        if (rte_eal_cleanup() != 0)
             FatalError("EAL cleanup failed: %s", strerror(-retval));
 
-        FatalError("%s: failed to configure", iface);
+        if (retval == -ENOMEM) {
+            FatalError("%s: memory allocation failed - consider"
+                       "%s freeing up some memory.",
+                    iface,
+                    rte_eal_has_hugepages() != 0 ? " increasing the number of hugepages or" : "");
+        } else {
+            FatalError("%s: failed to configure", iface);
+        }
     }
 
     SC_ATOMIC_RESET(iconf->ref);

From 6a4df6eb30de45a4e50f7a4700959c4c7c75b7ad Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Wed, 4 Oct 2023 16:57:52 +0200
Subject: [PATCH 064/462] dpdk: support new 23.11 DPDK bonding API

Ticket: #6381
---
 src/util-dpdk-bonding.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/util-dpdk-bonding.c b/src/util-dpdk-bonding.c
index 2dda0927a735..59b92ba8294a 100644
--- a/src/util-dpdk-bonding.c
+++ b/src/util-dpdk-bonding.c
@@ -54,7 +54,12 @@ uint16_t BondingMemberDevicesGet(
         uint16_t bond_pid, uint16_t bonded_devs[], uint16_t bonded_devs_length)
 {
 #ifdef HAVE_DPDK_BOND
+#if RTE_VERSION >= RTE_VERSION_NUM(23, 11, 0, 0)
+    int32_t len = rte_eth_bond_members_get(bond_pid, bonded_devs, bonded_devs_length);
+#else
     int32_t len = rte_eth_bond_slaves_get(bond_pid, bonded_devs, bonded_devs_length);
+#endif /* RTE_VERSION >= RTE_VERSION_NUM(23, 11, 0, 0) */
+
     if (len == 0)
         FatalError("%s: no bonded devices found", DPDKGetPortNameByPortID(bond_pid));
     else if (len < 0)

From c53086575af04e2bbea46a0b6033dbda406998ee Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Wed, 4 Oct 2023 16:59:39 +0200
Subject: [PATCH 065/462] dpdk: update DPDK builder versions

---
 .github/workflows/builds.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index eb71ea23bd21..d900b7b051bf 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -1961,7 +1961,7 @@ jobs:
     needs: [ prepare-deps, prepare-cbindgen ]
     strategy:
       matrix:
-        dpdk_version: [ 22.11.1, 21.11.3, 20.11.7, 19.11.14 ]
+        dpdk_version: [ 22.11.3, 21.11.5, 20.11.9, 19.11.14 ]
     steps:
 
       # Cache Rust stuff.

From e9c1ca2804ce6b4a588a11a12ea76b47d48a3769 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 10 Oct 2023 12:09:09 +0200
Subject: [PATCH 066/462] detect: fix legacy modifiers leading to multi-buffer

Fix non-continious matches with content and pcre modifiers setting up
multiple buffers.

To address this store whether a buffer is multi-capable and if not reuse
an earlier buffer if possible.

Bug: #6397.

Fixes: ad88efc2d868 ("detect: support multi buffer matching")
---
 src/detect-engine.c |  5 +++++
 src/detect-parse.c  | 47 ++++++++++++++++++++++++++++++++++-----------
 src/detect.h        |  2 ++
 3 files changed, 43 insertions(+), 11 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 009741c503fb..d8f9f1880e56 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -1444,6 +1444,8 @@ int DetectBufferSetActiveList(DetectEngineCtx *de_ctx, Signature *s, const int l
     s->init_data->curbuf->id = list;
     s->init_data->curbuf->head = NULL;
     s->init_data->curbuf->tail = NULL;
+    s->init_data->curbuf->multi_capable =
+            DetectEngineBufferTypeSupportsMultiInstanceGetById(de_ctx, list);
     SCLogDebug("new: idx %u list %d set up curbuf %p", s->init_data->buffer_index - 1, list,
             s->init_data->curbuf);
 
@@ -1470,6 +1472,7 @@ int DetectBufferGetActiveList(DetectEngineCtx *de_ctx, Signature *s)
         if (new_list == -1) {
             SCReturnInt(-1);
         }
+        int base_list = s->init_data->list;
         SCLogDebug("new_list %d", new_list);
         s->init_data->list = new_list;
         s->init_data->list_set = false;
@@ -1482,6 +1485,8 @@ int DetectBufferGetActiveList(DetectEngineCtx *de_ctx, Signature *s)
                 return -1;
             }
             s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
+            s->init_data->curbuf->multi_capable =
+                    DetectEngineBufferTypeSupportsMultiInstanceGetById(de_ctx, base_list);
         }
         if (s->init_data->curbuf == NULL) {
             SCLogError("failed to setup buffer");
diff --git a/src/detect-parse.c b/src/detect-parse.c
index 2e798d7b1cbf..b696b2055cb2 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -289,19 +289,32 @@ int DetectEngineContentModifierBufferSetup(DetectEngineCtx *de_ctx,
             SCLogError("no matches for previous buffer");
             return -1;
         }
-        if (SignatureInitDataBufferCheckExpand(s) < 0) {
-            SCLogError("failed to expand rule buffer array");
-            return -1;
+        bool reuse_buffer = false;
+        if (s->init_data->curbuf != NULL && (int)s->init_data->curbuf->id != sm_list) {
+            for (uint32_t x = 0; x < s->init_data->buffer_index; x++) {
+                if (s->init_data->buffers[x].id == (uint32_t)sm_list) {
+                    s->init_data->curbuf = &s->init_data->buffers[x];
+                    reuse_buffer = true;
+                    break;
+                }
+            }
         }
 
-        /* initialize a new buffer */
-        s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
-        s->init_data->curbuf->id = sm_list;
-        s->init_data->curbuf->head = NULL;
-        s->init_data->curbuf->tail = NULL;
-        SCLogDebug("idx %u list %d set up curbuf %p s->init_data->buffer_index %u",
-                s->init_data->buffer_index - 1, sm_list, s->init_data->curbuf,
-                s->init_data->buffer_index);
+        if (!reuse_buffer) {
+            if (SignatureInitDataBufferCheckExpand(s) < 0) {
+                SCLogError("failed to expand rule buffer array");
+                return -1;
+            }
+
+            /* initialize a new buffer */
+            s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
+            s->init_data->curbuf->id = sm_list;
+            s->init_data->curbuf->head = NULL;
+            s->init_data->curbuf->tail = NULL;
+            SCLogDebug("idx %u list %d set up curbuf %p s->init_data->buffer_index %u",
+                    s->init_data->buffer_index - 1, sm_list, s->init_data->curbuf,
+                    s->init_data->buffer_index);
+        }
     }
 
     /* transfer the sm from the pmatch list to sm_list */
@@ -469,6 +482,18 @@ void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
             SCLogDebug("reset: list %d != s->init_data->list %d", list, s->init_data->list);
             s->init_data->list = DETECT_SM_LIST_NOTSET;
         }
+
+        if (s->init_data->curbuf != NULL && (int)s->init_data->curbuf->id != list) {
+            for (uint32_t x = 0; x < s->init_data->buffer_index; x++) {
+                if (s->init_data->buffers[x].id == (uint32_t)list &&
+                        !s->init_data->buffers[x].multi_capable) {
+                    SCLogDebug("reusing buffer %u as it isn't multi-capable", x);
+                    s->init_data->curbuf = &s->init_data->buffers[x];
+                    break;
+                }
+            }
+        }
+
         if ((s->init_data->curbuf != NULL && (int)s->init_data->curbuf->id != list) ||
                 s->init_data->curbuf == NULL) {
             if (SignatureInitDataBufferCheckExpand(s) < 0) {
diff --git a/src/detect.h b/src/detect.h
index 69a5524e583c..04dd49a65a75 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -515,6 +515,8 @@ typedef struct SignatureInitDataBuffer_ {
     bool sm_init; /**< initialized by sigmatch, which is likely something like `urilen:10; http.uri;
                      content:"abc";`. These need to be in the same list. Unset once `http.uri` is
                      set up. */
+    bool multi_capable; /**< true if we can have multiple instances of this buffer, so e.g. for
+                           http.uri. */
     /* sig match list */
     SigMatch *head;
     SigMatch *tail;

From 15947f21736662ca5997dbc075b4ec9a7f5a304d Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 13 Oct 2023 13:47:05 +0200
Subject: [PATCH 067/462] detect: inspect all packets in multi-layer tunneling

When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.

This would only happen for single and workers, due to how the pipelines
are constructed.

The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.

This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.

Bug: #6402.
---
 src/tm-threads.c | 9 ++++++---
 src/tm-threads.h | 8 ++++++--
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/src/tm-threads.c b/src/tm-threads.c
index 63bf3be19db4..b173cb84f442 100644
--- a/src/tm-threads.c
+++ b/src/tm-threads.c
@@ -142,9 +142,11 @@ TmEcode TmThreadsSlotVarRun(ThreadVars *tv, Packet *p, TmSlot *slot)
             TmThreadsSlotProcessPktFail(tv, s, NULL);
             return TM_ECODE_FAILED;
         }
-
-        if (TmThreadsProcessDecodePseudoPackets(tv, &tv->decode_pq, s->slot_next) != TM_ECODE_OK) {
-            return TM_ECODE_FAILED;
+        if (s->tm_flags & TM_FLAG_DECODE_TM) {
+            if (TmThreadsProcessDecodePseudoPackets(tv, &tv->decode_pq, s->slot_next) !=
+                    TM_ECODE_OK) {
+                return TM_ECODE_FAILED;
+            }
         }
     }
 
@@ -661,6 +663,7 @@ void TmSlotSetFuncAppend(ThreadVars *tv, TmModule *tm, const void *data)
     /* we don't have to check for the return value "-1".  We wouldn't have
      * received a TM as arg, if it didn't exist */
     slot->tm_id = TmModuleGetIDForTM(tm);
+    slot->tm_flags |= tm->flags;
 
     tv->tmm_flags |= tm->flags;
     tv->cap_flags |= tm->cap_flags;
diff --git a/src/tm-threads.h b/src/tm-threads.h
index ec791bea47be..4ca55f9bc72c 100644
--- a/src/tm-threads.h
+++ b/src/tm-threads.h
@@ -63,14 +63,18 @@ typedef struct TmSlot_ {
 
     SC_ATOMIC_DECLARE(void *, slot_data);
 
+    /** copy of the TmModule::flags */
+    uint8_t tm_flags;
+
+    /* store the thread module id */
+    int tm_id;
+
     TmEcode (*SlotThreadInit)(ThreadVars *, const void *, void **);
     void (*SlotThreadExitPrintStats)(ThreadVars *, void *);
     TmEcode (*SlotThreadDeinit)(ThreadVars *, void *);
 
     /* data storage */
     const void *slot_initdata;
-    /* store the thread module id */
-    int tm_id;
 
 } TmSlot;
 

From 986a4417c6cf28ebc2485e20dbb567feddc2a1f7 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 5 Oct 2023 09:18:50 +0200
Subject: [PATCH 068/462] detect: error early when too many buffers

Ticket: #6104

To get a chance to clean properly, before we leak memory.
---
 src/detect-parse.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index b696b2055cb2..d9800f0a2f34 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -499,16 +499,18 @@ void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
             if (SignatureInitDataBufferCheckExpand(s) < 0) {
                 SCLogError("failed to expand rule buffer array");
                 s->init_data->init_flags |= SIG_FLAG_INIT_OVERFLOW;
-                return;
+                // SignatureInitDataBufferCheckExpand should not fail in this case
+                DEBUG_VALIDATE_BUG_ON(s->init_data->curbuf == NULL);
+                // keep curbuf even with wrong id as we error on this signature
+            } else {
+                /* initialize new buffer */
+                s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
+                s->init_data->curbuf->id = list;
+                /* buffer set up by sigmatch is tracked in case we add a stickybuffer for the
+                 * same list. */
+                s->init_data->curbuf->sm_init = true;
+                SCLogDebug("s->init_data->buffer_index %u", s->init_data->buffer_index);
             }
-
-            /* initialize new buffer */
-            s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
-            s->init_data->curbuf->id = list;
-            /* buffer set up by sigmatch is tracked in case we add a stickybuffer for the
-             * same list. */
-            s->init_data->curbuf->sm_init = true;
-            SCLogDebug("s->init_data->buffer_index %u", s->init_data->buffer_index);
         }
         BUG_ON(s->init_data->curbuf == NULL);
 
@@ -1015,8 +1017,11 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
         /* setup may or may not add a new SigMatch to the list */
         setup_ret = st->Setup(de_ctx, s, NULL);
     }
-    if (setup_ret < 0) {
+    if (setup_ret < 0 || (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW)) {
         SCLogDebug("\"%s\" failed to setup", st->name);
+        if (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW) {
+            SCLogError("rule %u tries to use too many buffers", s->id);
+        }
 
         /* handle 'silent' error case */
         if (setup_ret == -2) {
@@ -1930,11 +1935,6 @@ static int SigValidate(DetectEngineCtx *de_ctx, Signature *s)
         SCReturnInt(0);
     }
 
-    if (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW) {
-        SCLogError("rule %u tries to use too many buffers", s->id);
-        SCReturnInt(0);
-    }
-
     bool has_frame = false;
     bool has_app = false;
     bool has_pkt = false;

From 737bc4f219ea36b4da4ffbebef15b0619dffbca1 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 11 Sep 2023 16:49:48 +0200
Subject: [PATCH 069/462] mime: avoid quadratic complexity in MimeDecAddEntity

Ticket: #6306

Keep a reference to last child, consume a bit more RAM to save CPU
---
 src/util-decode-mime.c | 10 ++++------
 src/util-decode-mime.h |  1 +
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/src/util-decode-mime.c b/src/util-decode-mime.c
index 8e8b426c44a8..5e7a8d5713f4 100644
--- a/src/util-decode-mime.c
+++ b/src/util-decode-mime.c
@@ -384,7 +384,7 @@ static MimeDecUrl * MimeDecAddUrl(MimeDecEntity *entity, uint8_t *url, uint32_t
  */
 MimeDecEntity * MimeDecAddEntity(MimeDecEntity *parent)
 {
-    MimeDecEntity *curr, *node = SCMalloc(sizeof(MimeDecEntity));
+    MimeDecEntity *node = SCMalloc(sizeof(MimeDecEntity));
     if (unlikely(node == NULL)) {
         return NULL;
     }
@@ -394,12 +394,10 @@ MimeDecEntity * MimeDecAddEntity(MimeDecEntity *parent)
     if (parent != NULL) {
         if (parent->child == NULL) {
             parent->child = node;
+            parent->last_child = node;
         } else {
-            curr = parent->child;
-            while (curr->next != NULL) {
-                curr = curr->next;
-            }
-            curr->next = node;
+            parent->last_child->next = node;
+            parent->last_child = node;
         }
     }
 
diff --git a/src/util-decode-mime.h b/src/util-decode-mime.h
index 0dde13ca0496..cc79d98a6eb0 100644
--- a/src/util-decode-mime.h
+++ b/src/util-decode-mime.h
@@ -144,6 +144,7 @@ typedef struct MimeDecEntity {
     uint8_t *msg_id;  /**< Quick access pointer to message Id */
     struct MimeDecEntity *next;  /**< Pointer to list of sibling entities */
     struct MimeDecEntity *child;  /**< Pointer to list of child entities */
+    struct MimeDecEntity *last_child; /**< Pointer to tail of the list of child entities */
 } MimeDecEntity;
 
 /**

From d07e20c0a3e7d021507130fe57f7248f924e519f Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Wed, 18 Oct 2023 19:44:53 +0530
Subject: [PATCH 070/462] release: 7.0.2; update changelog

---
 ChangeLog    | 23 +++++++++++++++++++++++
 configure.ac |  2 +-
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index 051331a87c76..c5ffe05113c1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,26 @@
+7.0.2 -- 2023-10-18
+
+Security #6306: mime: quadratic complexity in MimeDecAddEntity
+Bug #6402: detect: multi-level tunneling inspection fails
+Bug #6397: detect: multiple legacy buffer selection leading to multi-buffer
+Bug #6381: DPDK 23.11 changed function name of Bond API
+Bug #6380: email: disabled fields in suricata.yaml also get logged
+Bug #6303: conf: an empty child node is not checked for NULL
+Bug #6300: config: includes provided as a sequence are loaded into the wrong parent configuration node
+Bug #6297: configure/docs: check for a supported version of sphinx-build
+Bug #6104: detect/multi-buffer: Heap-buffer-overflow in SigMatchAppendSMToList
+Bug #6009: dpdk: incorrect final stats
+Bug #5831: af-packet/ips: excessive mtu log messages
+Bug #5211: detect/frames: crash with detect.profiling.grouping.dump-to-disk
+Bug #4624: byte_jump with negative post_offset before start of buffer failure
+Feature #6367: SMTP: do not delay mime chunk processing
+Feature #5966: dpdk: Analyze hugepage allocation on startup
+Feature #4968: QUIC v2 support
+Task #6348: detect/analyzer: add more details for the ipopts keyword
+Task #6235: decode: add drop reason for stream reassembly memcap
+Documentation #6349: userguide: add section about tcp.flags
+Documentation #6342: userguide: cover install-full and install-conf in the install page
+
 7.0.1 -- 2023-09-13
 
 Security #6279: Crash in SMTP parser during parsing of email
diff --git a/configure.ac b/configure.ac
index cf6b8f625256..3193deb27697 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-    AC_INIT([suricata],[7.0.2-dev])
+    AC_INIT([suricata],[7.0.2])
     m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])AM_SILENT_RULES([yes])
     AC_CONFIG_HEADERS([src/autoconf.h])
     AC_CONFIG_SRCDIR([src/suricata.c])

From 2fe2d82506f5697d45ce28642bd3bb3780f3b369 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 19 Oct 2023 16:13:19 +0200
Subject: [PATCH 071/462] version: start development towards 7.0.3

---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 3193deb27697..b377a1da9f2d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-    AC_INIT([suricata],[7.0.2])
+    AC_INIT([suricata],[7.0.3-dev])
     m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])AM_SILENT_RULES([yes])
     AC_CONFIG_HEADERS([src/autoconf.h])
     AC_CONFIG_SRCDIR([src/suricata.c])

From 0e5fdbb8fb919e06988cdae8737214d304287d36 Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Wed, 11 Oct 2023 21:42:47 +0000
Subject: [PATCH 072/462] doc: be consistent with the use of "sudo"

Issue: #5720
---
 doc/userguide/install.rst | 66 ++++++++++++++++++++++++---------------
 1 file changed, 41 insertions(+), 25 deletions(-)

diff --git a/doc/userguide/install.rst b/doc/userguide/install.rst
index ad7a9e44747f..5c28aef020da 100644
--- a/doc/userguide/install.rst
+++ b/doc/userguide/install.rst
@@ -90,18 +90,20 @@ Rust support::
 Ubuntu/Debian
 """""""""""""
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 Minimal::
 
     # Installed Rust and cargo as indicated above
-    apt-get install build-essential git libjansson-dev libpcap-dev \
+    sudo apt-get install build-essential git libjansson-dev libpcap-dev \
                     libpcre2-dev libtool libyaml-dev make pkg-config zlib1g-dev
     # On most distros installing cbindgen with package manager should be enough
-    apt-get install cbindgen # alternative: cargo install --force cbindgen
+    sudo apt-get install cbindgen # alternative: cargo install --force cbindgen
 
 Recommended::
 
     # Installed Rust and cargo as indicated above
-    apt-get install autoconf automake build-essential ccache clang curl git \
+    sudo apt-get install autoconf automake build-essential ccache clang curl git \
                     gosu jq libbpf-dev libcap-ng0 libcap-ng-dev libelf-dev \
                     libevent-dev libgeoip-dev libhiredis-dev libjansson-dev \
                     liblua5.1-dev libmagic-dev libnet1-dev libpcap-dev \
@@ -112,39 +114,41 @@ Recommended::
 
 Extra for iptables/nftables IPS integration::
 
-    apt-get install libnetfilter-queue-dev libnetfilter-queue1  \
+    sudo apt-get install libnetfilter-queue-dev libnetfilter-queue1  \
                     libnetfilter-log-dev libnetfilter-log1      \
                     libnfnetlink-dev libnfnetlink0
 
 CentOS, AlmaLinux, RockyLinux, Fedora, etc
 """"""""""""""""""""""""""""""""""""""""""
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 To install all minimal dependencies, it is required to enable extra package
 repository in most distros. You can enable it possibly by
 one of the following ways::
 
-    dnf -y update
-    dnf -y install dnf-plugins-core
+    sudo dnf -y update
+    sudo dnf -y install dnf-plugins-core
     # AlmaLinux 8
-    dnf config-manager --set-enabled powertools
+    sudo dnf config-manager --set-enabled powertools
     # AlmaLinux 9
-    dnf config-manager --set-enable crb
+    sudo dnf config-manager --set-enable crb
     # Oracle Linux 8
-    dnf config-manager --set-enable ol8_codeready_builder
+    sudo dnf config-manager --set-enable ol8_codeready_builder
     # Oracle Linux 9
-    dnf config-manager --set-enable ol9_codeready_builder
+    sudo dnf config-manager --set-enable ol9_codeready_builder
 
 Minimal::
 
     # Installed Rust and cargo as indicated above
-    dnf install -y gcc gcc-c++ git jansson-devel libpcap-devel libtool \
+    sudo dnf install -y gcc gcc-c++ git jansson-devel libpcap-devel libtool \
                    libyaml-devel make pcre2-devel which zlib-devel
     cargo install --force cbindgen
 
 Recommended::
 
     # Installed Rust and cargo as indicated above
-    dnf install -y autoconf automake diffutils file-devel gcc gcc-c++ git \
+    sudo dnf install -y autoconf automake diffutils file-devel gcc gcc-c++ git \
                    jansson-devel jq libcap-ng-devel libevent-devel \
                    libmaxminddb-devel libnet-devel libnetfilter_queue-devel \
                    libnfnetlink-devel libpcap-devel libtool libyaml-devel \
@@ -202,6 +206,8 @@ Ubuntu from Personal Package Archives (PPA)
 For Ubuntu, OISF maintains a PPA ``suricata-stable`` that always contains the
 latest stable release.
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 Setup to install the latest stable Suricata::
 
     sudo apt-get install software-properties-common
@@ -236,6 +242,8 @@ To remove Suricata from your system::
 Getting Debug or Pre-release Versions
 """""""""""""""""""""""""""""""""""""
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 If you want Suricata with built-in (enabled) debugging, you can install the
 debug package::
 
@@ -256,6 +264,8 @@ Suricata will then always be the latest release, stable or beta.
 Daily Releases
 """"""""""""""
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 If you would like to help test the daily build packages from our latest git(dev)
 repository, the same procedures as above apply, just using another PPA,
 ``suricata-daily``::
@@ -281,6 +291,8 @@ repository, the same procedures as above apply, just using another PPA,
 Debian
 ^^^^^^
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 In Debian 9 (stretch) and later do::
 
     sudo apt-get install suricata
@@ -314,29 +326,31 @@ Installing From Package Repositories
 CentOS, RHEL, AlmaLinux, RockyLinux, etc Version 8+
 '''''''''''''''''''''''''''''''''''''''''''''''''''
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 .. code-block:: none
 
-   dnf install epel-release dnf-plugins-core
-   dnf copr enable @oisf/suricata-7.0
-   dnf install suricata
+   sudo dnf install epel-release dnf-plugins-core
+   sudo dnf copr enable @oisf/suricata-7.0
+   sudo dnf install suricata
 
 CentOS 7
 ''''''''
 
 .. code-block:: none
 
-   yum install epel-release yum-plugin-copr
-   yum copr enable @oisf/suricata-7.0
-   yum install suricata
+   sudo yum install epel-release yum-plugin-copr
+   sudo yum copr enable @oisf/suricata-7.0
+   sudo yum install suricata
 
 Fedora
 ''''''
 
 .. code-block:: none
 
-    dnf install dnf-plugins-core
-    dnf copr enable @oisf/suricata-7.0
-    dnf install suricata
+    sudo dnf install dnf-plugins-core
+    sudo dnf copr enable @oisf/suricata-7.0
+    sudo dnf install suricata
 
 Additional Notes for RPM Installations
 """"""""""""""""""""""""""""""""""""""
@@ -357,21 +371,23 @@ Starting Suricata On-Boot
 
 The Suricata RPMs are configured to run from Systemd.
 
+.. note:: The following instructions require ``sudo`` to be installed.
+
 To start Suricata::
 
-  systemctl start suricata
+  sudo systemctl start suricata
 
 To stop Suricata::
 
-  systemctl stop suricata
+  sudo systemctl stop suricata
 
 To have Suricata start on-boot::
 
-  systemctl enable suricata
+  sudo systemctl enable suricata
 
 To reload rules::
 
-   systemctl reload suricata
+  sudo systemctl reload suricata
 
 .. _install-advanced:
 

From 5c0af0b203de47b8a8bb5d22c8e2871fe9104469 Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Wed, 11 Oct 2023 19:24:03 +0000
Subject: [PATCH 073/462] rust/doc: add docstring to rust module files.

Issue: #4584
---
 rust/src/applayer.rs             |  2 +-
 rust/src/applayertemplate/mod.rs |  2 ++
 rust/src/asn1/mod.rs             |  2 ++
 rust/src/bittorrent_dht/mod.rs   |  2 ++
 rust/src/common.rs               |  2 ++
 rust/src/conf.rs                 |  2 ++
 rust/src/core.rs                 |  2 +-
 rust/src/dcerpc/mod.rs           |  2 ++
 rust/src/detect/mod.rs           |  2 ++
 rust/src/dhcp/mod.rs             |  2 ++
 rust/src/dns/mod.rs              |  2 ++
 rust/src/ffi/mod.rs              |  2 ++
 rust/src/filecontainer.rs        |  2 ++
 rust/src/filetracker.rs          | 23 +++++++++++------------
 rust/src/frames.rs               |  2 ++
 rust/src/ftp/mod.rs              |  2 ++
 rust/src/http2/mod.rs            |  2 ++
 rust/src/ike/mod.rs              |  2 ++
 rust/src/jsonbuilder.rs          |  2 ++
 rust/src/kerberos.rs             |  2 ++
 rust/src/krb/mod.rs              |  2 ++
 rust/src/lib.rs                  |  5 +++++
 rust/src/log.rs                  |  2 ++
 rust/src/lua.rs                  |  2 ++
 rust/src/lzma.rs                 |  2 ++
 rust/src/mime/mod.rs             |  2 ++
 rust/src/modbus/mod.rs           |  2 ++
 rust/src/mqtt/mod.rs             |  2 ++
 rust/src/nfs/mod.rs              |  2 ++
 rust/src/ntp/mod.rs              |  2 ++
 rust/src/pgsql/mod.rs            |  2 +-
 rust/src/plugin.rs               |  2 ++
 rust/src/quic/mod.rs             |  2 ++
 rust/src/rdp/mod.rs              |  2 +-
 rust/src/rfb/mod.rs              |  2 ++
 rust/src/sip/mod.rs              |  2 ++
 rust/src/smb/mod.rs              |  2 ++
 rust/src/snmp/mod.rs             |  2 ++
 rust/src/ssh/mod.rs              |  2 ++
 rust/src/telnet/mod.rs           |  2 ++
 rust/src/tftp/mod.rs             |  2 ++
 rust/src/util.rs                 |  2 ++
 rust/src/x509/mod.rs             |  2 ++
 43 files changed, 94 insertions(+), 16 deletions(-)

diff --git a/rust/src/applayer.rs b/rust/src/applayer.rs
index 255fa1593c2b..97db321e2249 100644
--- a/rust/src/applayer.rs
+++ b/rust/src/applayer.rs
@@ -15,7 +15,7 @@
  * 02110-1301, USA.
  */
 
-//! Parser registration functions and common interface
+//! Parser registration functions and common interface module.
 
 use std;
 use crate::core::{self,DetectEngineState,Flow,AppLayerEventType,AppProto,Direction};
diff --git a/rust/src/applayertemplate/mod.rs b/rust/src/applayertemplate/mod.rs
index 63f4ed139e87..e22bd68e391b 100644
--- a/rust/src/applayertemplate/mod.rs
+++ b/rust/src/applayertemplate/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Application layer template parser and logger module.
+
 mod parser;
 pub mod template;
 /* TEMPLATE_START_REMOVE */
diff --git a/rust/src/asn1/mod.rs b/rust/src/asn1/mod.rs
index 7c52b4cb8563..4b77b0ca28d5 100644
--- a/rust/src/asn1/mod.rs
+++ b/rust/src/asn1/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! ASN.1 parser module.
+
 use der_parser::ber::{parse_ber_recursive, BerObject, BerObjectContent, Tag};
 use nom7::Err;
 use std::convert::TryFrom;
diff --git a/rust/src/bittorrent_dht/mod.rs b/rust/src/bittorrent_dht/mod.rs
index 0c79ecc92fa5..9e0d033d3003 100644
--- a/rust/src/bittorrent_dht/mod.rs
+++ b/rust/src/bittorrent_dht/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! BitTorrent DHT application layer, logger and parser module.
+
 pub mod bittorrent_dht;
 pub mod logger;
 pub mod parser;
diff --git a/rust/src/common.rs b/rust/src/common.rs
index a8880b2c4c02..1d10bbe443d0 100644
--- a/rust/src/common.rs
+++ b/rust/src/common.rs
@@ -1,3 +1,5 @@
+//! Utility library module for commonly used strings, hexadecimals and other elements.
+
 use super::build_slice;
 use crate::jsonbuilder::HEX;
 use std::ffi::CString;
diff --git a/rust/src/conf.rs b/rust/src/conf.rs
index b176d5f71f20..50acf6cae895 100644
--- a/rust/src/conf.rs
+++ b/rust/src/conf.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Module for retrieving configuration details.
+
 use std::os::raw::c_char;
 use std::os::raw::c_void;
 use std::os::raw::c_int;
diff --git a/rust/src/core.rs b/rust/src/core.rs
index 5b0a67afc0fb..abb27ea578fe 100644
--- a/rust/src/core.rs
+++ b/rust/src/core.rs
@@ -15,7 +15,7 @@
  * 02110-1301, USA.
  */
 
-// This file exposes items from the core "C" code to Rust.
+//! This module exposes items from the core "C" code to Rust.
 
 use std;
 use crate::filecontainer::*;
diff --git a/rust/src/dcerpc/mod.rs b/rust/src/dcerpc/mod.rs
index 7765e044ca81..800d2ade7311 100644
--- a/rust/src/dcerpc/mod.rs
+++ b/rust/src/dcerpc/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! DCE/RPC protocol parser, logger and detection module.
+
 pub mod dcerpc;
 pub mod dcerpc_udp;
 pub mod parser;
diff --git a/rust/src/detect/mod.rs b/rust/src/detect/mod.rs
index 84bd28e2863d..41c7ff2455bd 100644
--- a/rust/src/detect/mod.rs
+++ b/rust/src/detect/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Module for rule parsing.
+
 pub mod byte_math;
 pub mod error;
 pub mod iprep;
diff --git a/rust/src/dhcp/mod.rs b/rust/src/dhcp/mod.rs
index bc859d7b54ba..fd783d961ff2 100644
--- a/rust/src/dhcp/mod.rs
+++ b/rust/src/dhcp/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! DHCP parser, detection and logger module.
+
 pub mod dhcp;
 pub mod parser;
 pub mod logger;
diff --git a/rust/src/dns/mod.rs b/rust/src/dns/mod.rs
index 054bb50b259a..b0ca00ffc9d4 100644
--- a/rust/src/dns/mod.rs
+++ b/rust/src/dns/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! DNS parser, detection, logger and application layer module.
+
 pub mod detect;
 pub mod dns;
 pub mod log;
diff --git a/rust/src/ffi/mod.rs b/rust/src/ffi/mod.rs
index ff9f7d1642c3..e97e6c98c639 100644
--- a/rust/src/ffi/mod.rs
+++ b/rust/src/ffi/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Module that exposes C bindings to the Suricata Rust library.
+
 pub mod hashing;
 pub mod base64;
 pub mod strings;
diff --git a/rust/src/filecontainer.rs b/rust/src/filecontainer.rs
index c51daaa8f139..3a8bde5f7d5b 100644
--- a/rust/src/filecontainer.rs
+++ b/rust/src/filecontainer.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! This module handles file container operations (open, append, close).
+
 use std::ptr;
 use std::os::raw::{c_void};
 
diff --git a/rust/src/filetracker.rs b/rust/src/filetracker.rs
index ad1c4c0c3a2c..3ae65eecb559 100644
--- a/rust/src/filetracker.rs
+++ b/rust/src/filetracker.rs
@@ -15,18 +15,17 @@
  * 02110-1301, USA.
  */
 
-/**
- *  \file
- *  \author Victor Julien <victor@inliniac.net>
- *
- * Tracks chunk based file transfers. Chunks may be transferred out
- * of order, but cannot be transferred in parallel. So only one
- * chunk at a time.
- *
- * GAP handling. If a data gap is encountered, the file is truncated
- * and new data is no longer pushed down to the lower level APIs.
- * The tracker does continue to follow the file.
- */
+//! Gap handling and Chunk-based file transfer tracker module.
+//!
+//! GAP handling. If a data gap is encountered, the file is truncated
+//! and new data is no longer pushed down to the lower level APIs.
+//! The tracker does continue to follow the file
+//
+//! Tracks chunk based file transfers. Chunks may be transferred out
+//! of order, but cannot be transferred in parallel. So only one
+//! chunk at a time.
+//!
+//! Author: Victor Julien <victor@inliniac.net>
 
 use crate::core::*;
 use std::collections::HashMap;
diff --git a/rust/src/frames.rs b/rust/src/frames.rs
index 3311a4b36f48..3a45d014b472 100644
--- a/rust/src/frames.rs
+++ b/rust/src/frames.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Module for bindings to the Suricata C frame API.
+
 use crate::applayer::StreamSlice;
 use crate::core::Flow;
 #[cfg(not(test))]
diff --git a/rust/src/ftp/mod.rs b/rust/src/ftp/mod.rs
index 1a60ca470c34..3839c9661e0d 100644
--- a/rust/src/ftp/mod.rs
+++ b/rust/src/ftp/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! FTP parser and application layer module.
+
 use nom7::bytes::complete::{tag, take_until};
 use nom7::character::complete::{digit1, multispace0};
 use nom7::combinator::{complete, map_res, opt, verify};
diff --git a/rust/src/http2/mod.rs b/rust/src/http2/mod.rs
index f1d6f30bb274..910e968105d0 100644
--- a/rust/src/http2/mod.rs
+++ b/rust/src/http2/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! HTTP/2 parser, detection, logger and application layer module.
+
 #![allow(clippy::result_unit_err)]
 
 mod decompression;
diff --git a/rust/src/ike/mod.rs b/rust/src/ike/mod.rs
index 4f8114d2a0af..366688eae2ea 100644
--- a/rust/src/ike/mod.rs
+++ b/rust/src/ike/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! IKE parser, detection, logger and application layer module.
+
 // written by Pierre Chifflier  <chifflier@wzdftpd.net>
 
 extern crate ipsec_parser;
diff --git a/rust/src/jsonbuilder.rs b/rust/src/jsonbuilder.rs
index c1f466563be7..82be09953c70 100644
--- a/rust/src/jsonbuilder.rs
+++ b/rust/src/jsonbuilder.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+ //! Module for building JSON documents.
+
 #![allow(clippy::missing_safety_doc)]
 
 use std::cmp::max;
diff --git a/rust/src/kerberos.rs b/rust/src/kerberos.rs
index 0a5e9517abfe..e7c51cc2f28f 100644
--- a/rust/src/kerberos.rs
+++ b/rust/src/kerberos.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Kerberos parser wrapper module.
+
 use nom7::IResult;
 use nom7::error::{ErrorKind, ParseError};
 use nom7::number::streaming::le_u16;
diff --git a/rust/src/krb/mod.rs b/rust/src/krb/mod.rs
index c8be867a0936..ca6237d7bb6b 100644
--- a/rust/src/krb/mod.rs
+++ b/rust/src/krb/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Kerberos-v5 application layer, logger and detection module.
+
 // written by Pierre Chifflier  <chifflier@wzdftpd.net>
 
 pub mod krb5;
diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index 4c238538273a..36098ffadd0b 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -15,6 +15,11 @@
  * 02110-1301, USA.
  */
 
+//! Suricata is a network intrusion prevention and monitoring engine.
+//!
+//! Suricata is a hybrid C and Rust application. What is found here are
+//! the components written in Rust.
+
 #![cfg_attr(feature = "strict", deny(warnings))]
 
 // Allow these patterns as its a style we like.
diff --git a/rust/src/log.rs b/rust/src/log.rs
index 05344b6e6cd9..744169a97039 100644
--- a/rust/src/log.rs
+++ b/rust/src/log.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Logging utility module.
+
 use std;
 use std::ffi::CString;
 use std::path::Path;
diff --git a/rust/src/lua.rs b/rust/src/lua.rs
index 87cb8b3acf9c..4fce2a824944 100644
--- a/rust/src/lua.rs
+++ b/rust/src/lua.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Lua wrapper module.
+
 use std::os::raw::c_char;
 use std::os::raw::c_int;
 use std::os::raw::c_long;
diff --git a/rust/src/lzma.rs b/rust/src/lzma.rs
index b16c4e7b5255..e10d803a6e2c 100644
--- a/rust/src/lzma.rs
+++ b/rust/src/lzma.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! lzma decompression utility module.
+
 use lzma_rs::decompress::{Options, Stream};
 use lzma_rs::error::Error;
 use std::io::{Cursor, Write};
diff --git a/rust/src/mime/mod.rs b/rust/src/mime/mod.rs
index 399500db9ac2..6f4a9bc21301 100644
--- a/rust/src/mime/mod.rs
+++ b/rust/src/mime/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! MIME protocol parser module.
+
 use crate::common::nom7::take_until_and_consume;
 use nom7::branch::alt;
 use nom7::bytes::streaming::{tag, take_until, take_while};
diff --git a/rust/src/modbus/mod.rs b/rust/src/modbus/mod.rs
index 50e65939047f..6f3c434a0444 100644
--- a/rust/src/modbus/mod.rs
+++ b/rust/src/modbus/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Modbus application layer, logger, parser and detection module.
+
 pub mod detect;
 pub mod log;
 pub mod modbus;
diff --git a/rust/src/mqtt/mod.rs b/rust/src/mqtt/mod.rs
index c0225ecec31c..aefcc33e092b 100644
--- a/rust/src/mqtt/mod.rs
+++ b/rust/src/mqtt/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! MQTT application layer, detection, logger and parser module.
+
 pub mod detect;
 pub mod logger;
 pub mod mqtt;
diff --git a/rust/src/nfs/mod.rs b/rust/src/nfs/mod.rs
index 17cfc724d3e8..2f6fe84df917 100644
--- a/rust/src/nfs/mod.rs
+++ b/rust/src/nfs/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! NFS application layer, parser, logger module.
+
 pub mod types;
 pub mod rpc_records;
 pub mod nfs_records;
diff --git a/rust/src/ntp/mod.rs b/rust/src/ntp/mod.rs
index 35f1a70f2e1b..30ff834304cb 100644
--- a/rust/src/ntp/mod.rs
+++ b/rust/src/ntp/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! NTP application layer and parser module.
+
 // written by Pierre Chifflier  <chifflier@wzdftpd.net>
 
 pub mod ntp;
diff --git a/rust/src/pgsql/mod.rs b/rust/src/pgsql/mod.rs
index 054fd4886598..a4c79c03192f 100644
--- a/rust/src/pgsql/mod.rs
+++ b/rust/src/pgsql/mod.rs
@@ -15,7 +15,7 @@
  * 02110-1301, USA.
  */
 
-//! PostgreSQL parser and application layer
+//! PostgreSQL parser, logger and application layer module.
 //!
 //! written by Juliana Fajardini <jufajardini@oisf.net>
 
diff --git a/rust/src/plugin.rs b/rust/src/plugin.rs
index f9daef7de439..ad214aaa47f5 100644
--- a/rust/src/plugin.rs
+++ b/rust/src/plugin.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Plugin utility module.
+
 pub fn init() {
     unsafe {
         let context = super::core::SCGetContext();
diff --git a/rust/src/quic/mod.rs b/rust/src/quic/mod.rs
index 91693059dccb..8a8f1bbd7fce 100644
--- a/rust/src/quic/mod.rs
+++ b/rust/src/quic/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! QUIC application layer, parser, detection and logger module.
+
 mod crypto;
 mod cyu;
 pub mod detect;
diff --git a/rust/src/rdp/mod.rs b/rust/src/rdp/mod.rs
index ddfc511d4fa6..dc83db829940 100644
--- a/rust/src/rdp/mod.rs
+++ b/rust/src/rdp/mod.rs
@@ -15,7 +15,7 @@
  * 02110-1301, USA.
  */
 
-//! RDP parser and application layer
+//! RDP parser, logger and application layer module.
 //!
 //! written by Zach Kelly <zach.kelly@lmco.com>
 
diff --git a/rust/src/rfb/mod.rs b/rust/src/rfb/mod.rs
index 68d37ec8a0eb..050ee7709b70 100644
--- a/rust/src/rfb/mod.rs
+++ b/rust/src/rfb/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! RFB protocol parser, logger and detection module.
+
 // Author: Frank Honza <frank.honza@dcso.de>
 
 pub mod detect;
diff --git a/rust/src/sip/mod.rs b/rust/src/sip/mod.rs
index 33eec45ccdb9..de63aaa52018 100755
--- a/rust/src/sip/mod.rs
+++ b/rust/src/sip/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! SIP protocol parser, detection and logger module.
+
 // written by Giuseppe Longo <giuseppe@glongo.it>
 
 pub mod detect;
diff --git a/rust/src/smb/mod.rs b/rust/src/smb/mod.rs
index ca1eb7771fd0..5b74f1ca4e02 100644
--- a/rust/src/smb/mod.rs
+++ b/rust/src/smb/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! SMB application layer, detection, logger and parser module.
+
 pub mod error;
 pub mod smb_records;
 pub mod smb_status;
diff --git a/rust/src/snmp/mod.rs b/rust/src/snmp/mod.rs
index 3bb90ab2f35d..7c6ceb35884b 100644
--- a/rust/src/snmp/mod.rs
+++ b/rust/src/snmp/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! SNMP application layer, parser, detection and logger module.
+
 // written by Pierre Chifflier  <chifflier@wzdftpd.net>
 
 extern crate snmp_parser;
diff --git a/rust/src/ssh/mod.rs b/rust/src/ssh/mod.rs
index 12efccd0a554..ff506e9439a1 100644
--- a/rust/src/ssh/mod.rs
+++ b/rust/src/ssh/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! SSH application layer, logger, detection and parser module.
+
 pub mod detect;
 pub mod logger;
 mod parser;
diff --git a/rust/src/telnet/mod.rs b/rust/src/telnet/mod.rs
index 2dfa97a2f1b3..38685c795443 100644
--- a/rust/src/telnet/mod.rs
+++ b/rust/src/telnet/mod.rs
@@ -15,5 +15,7 @@
  * 02110-1301, USA.
  */
 
+//! Telnet application layer and parser module.
+
 pub mod telnet;
 mod parser;
diff --git a/rust/src/tftp/mod.rs b/rust/src/tftp/mod.rs
index 7c3d292a00f8..6ae29ac90143 100644
--- a/rust/src/tftp/mod.rs
+++ b/rust/src/tftp/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! TFTP parser, logger and application layer module.
+
 // written by Clément Galland <clement.galland@epita.fr>
 
 pub mod tftp;
diff --git a/rust/src/util.rs b/rust/src/util.rs
index a0933689164e..d7109464f773 100644
--- a/rust/src/util.rs
+++ b/rust/src/util.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Utility module.
+
 use std::ffi::CStr;
 use std::os::raw::c_char;
 
diff --git a/rust/src/x509/mod.rs b/rust/src/x509/mod.rs
index fa9706e90211..c87928cf17a8 100644
--- a/rust/src/x509/mod.rs
+++ b/rust/src/x509/mod.rs
@@ -15,6 +15,8 @@
  * 02110-1301, USA.
  */
 
+//! Module for SSL/TLS X.509 certificates parser and decoder.
+
 // written by Pierre Chifflier  <chifflier@wzdftpd.net>
 
 use crate::common::rust_string_to_c;

From 54de0450f43cd424c53f43811ef3ba65290411aa Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Thu, 12 Oct 2023 11:13:41 +0000
Subject: [PATCH 074/462] rust: remove cbindgen:ignore on frames module

This directive is no longer required, and does
mess up the rustdoc description of the module.
---
 rust/src/lib.rs | 1 -
 1 file changed, 1 deletion(-)

diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index 36098ffadd0b..da2859637783 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -75,7 +75,6 @@ pub mod conf;
 pub mod jsonbuilder;
 #[macro_use]
 pub mod applayer;
-/// cbindgen:ignore
 pub mod frames;
 pub mod filecontainer;
 pub mod filetracker;

From 3b1558946dff9e0466910273c2588e3071d19cb5 Mon Sep 17 00:00:00 2001
From: Liza Opar <oparliza@gmail.com>
Date: Thu, 12 Oct 2023 14:07:48 +0300
Subject: [PATCH 075/462] misc: improve code documentation

Task #6383
---
 src/app-layer-dnp3.c       | 6 ++++++
 src/app-layer-dnp3.h       | 6 ++++++
 src/app-layer-htp-libhtp.c | 6 +++++-
 src/app-layer-htp-libhtp.h | 6 +++++-
 src/app-layer-htp-mem.h    | 5 +++++
 src/app-layer-smb.c        | 6 ++++++
 src/app-layer-smb.h        | 6 ++++++
 src/app-layer.h            | 4 ++++
 8 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/src/app-layer-dnp3.c b/src/app-layer-dnp3.c
index 21abfaaeb6da..9501b9f5ea57 100644
--- a/src/app-layer-dnp3.c
+++ b/src/app-layer-dnp3.c
@@ -15,6 +15,12 @@
  * 02110-1301, USA.
  */
 
+/**
+ * \file
+ *
+ * DNP3 protocol implementation
+ */
+
 #include "suricata-common.h"
 #include "suricata.h"
 #include "stream.h"
diff --git a/src/app-layer-dnp3.h b/src/app-layer-dnp3.h
index 6445631d977f..aae07f9c8095 100644
--- a/src/app-layer-dnp3.h
+++ b/src/app-layer-dnp3.h
@@ -15,6 +15,12 @@
  * 02110-1301, USA.
  */
 
+/**
+ * \file
+ *
+ * DNP3 application layer protocol header file
+ */
+
 #ifndef __APP_LAYER_DNP3_H__
 #define __APP_LAYER_DNP3_H__
 
diff --git a/src/app-layer-htp-libhtp.c b/src/app-layer-htp-libhtp.c
index f7daf70c92ba..2fbd5eae3390 100644
--- a/src/app-layer-htp-libhtp.c
+++ b/src/app-layer-htp-libhtp.c
@@ -36,7 +36,11 @@
  ***************************************************************************/
 
 /**
- * Anoop Saldanha <anoopsaldanha@gmail.com>
+ * \file
+ *
+ * \author Anoop Saldanha <anoopsaldanha@gmail.com>
+ *
+ * APIs from libhtp 0.5.x.
  */
 
 #include "suricata-common.h"
diff --git a/src/app-layer-htp-libhtp.h b/src/app-layer-htp-libhtp.h
index c4a3c991f74b..574dda4134dc 100644
--- a/src/app-layer-htp-libhtp.h
+++ b/src/app-layer-htp-libhtp.h
@@ -36,7 +36,11 @@
  ***************************************************************************/
 
 /**
- * Anoop Saldanha <anoopsaldanha@gmail.com>
+ * \file
+ *
+ * \author Anoop Saldanha <anoopsaldanha@gmail.com>
+ *
+ * APIs from libhtp 0.5.x.
  */
 
 #ifndef __APP_LAYER_HTP_LIBHTP__H__
diff --git a/src/app-layer-htp-mem.h b/src/app-layer-htp-mem.h
index 5df67f824d87..01cefe6750e0 100644
--- a/src/app-layer-htp-mem.h
+++ b/src/app-layer-htp-mem.h
@@ -15,6 +15,11 @@
  * 02110-1301, USA.
  */
 
+/**
+ * \file
+ *
+ * Memory management functions for HTP in the application layer
+ */
 
 void HTPParseMemcap(void);
 void *HTPMalloc(size_t size);
diff --git a/src/app-layer-smb.c b/src/app-layer-smb.c
index e5c49d62e9c0..0c6102e83e5f 100644
--- a/src/app-layer-smb.c
+++ b/src/app-layer-smb.c
@@ -15,6 +15,12 @@
  * 02110-1301, USA.
  */
 
+/**
+ * \file
+ *
+ * SMB protocol handling
+ */
+
 #include "suricata-common.h"
 #include "suricata.h"
 
diff --git a/src/app-layer-smb.h b/src/app-layer-smb.h
index 147f0c01a2ec..39ae04115440 100644
--- a/src/app-layer-smb.h
+++ b/src/app-layer-smb.h
@@ -15,6 +15,12 @@
  * 02110-1301, USA.
  */
 
+/**
+ * \file
+ *
+ * SMB protocol handling
+ */
+
 #ifndef __APP_LAYER_SMB_H__
 #define __APP_LAYER_SMB_H__
 
diff --git a/src/app-layer.h b/src/app-layer.h
index d08d785d73d1..cbe2fbc9af84 100644
--- a/src/app-layer.h
+++ b/src/app-layer.h
@@ -16,8 +16,12 @@
  */
 
 /**
+ * \file
+ *
  * \author Victor Julien <victor@inliniac.net>
  * \author Anoop Saldanha <anoopsaldanha@gmail.com>
+ *
+ *  Application layer handling and protocols implementation
  */
 
 #ifndef __APP_LAYER_H__

From 47a11c7ea4083d71057cb5ab333165fa6ff16422 Mon Sep 17 00:00:00 2001
From: Hadiqa Alamdar Bukhari <hadiqaalamdar@gmail.com>
Date: Fri, 20 Oct 2023 00:40:50 +0500
Subject: [PATCH 076/462] util-misc: Convert unittests to new FAIL/PASS API

Task #6345
---
 src/util-misc.c | 782 ++++++++++++------------------------------------
 1 file changed, 194 insertions(+), 588 deletions(-)

diff --git a/src/util-misc.c b/src/util-misc.c
index 38c4f9dd8d06..4380e694a8a0 100644
--- a/src/util-misc.c
+++ b/src/util-misc.c
@@ -252,227 +252,129 @@ static int UtilMiscParseSizeStringTest01(void)
 
     str = "10";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = "10kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = "10gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240UL) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240UL);
 
     /* space start */
 
     str = " 10";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = " 10kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = " 10gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space end */
 
     str = "10 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = "10kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = "10gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space start - space end */
 
     str = " 10 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = " 10kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = " 10gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space between number and scale */
 
@@ -480,452 +382,257 @@ static int UtilMiscParseSizeStringTest01(void)
 
     str = "10";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = "10 kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = "10 gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space start */
 
     str = " 10";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = " 10 kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = " 10 gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space end */
 
     str = "10 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = "10 kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = "10 mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = "10 gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* space start - space end */
 
     str = " 10 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10);
 
     str = " 10 kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024);
 
     str = " 10 mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10 * 1024 * 1024);
 
     str = " 10 gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10737418240) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10737418240);
 
     /* no space */
 
     str = "10.5";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = "10.5kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = "10.5gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space start */
 
     str = " 10.5";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = " 10.5kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = " 10.5gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space end */
 
     str = "10.5 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = "10.5kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = "10.5gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space start - space end */
 
     str = " 10.5 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = " 10.5kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = " 10.5gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space between number and scale */
 
@@ -933,235 +640,134 @@ static int UtilMiscParseSizeStringTest01(void)
 
     str = "10.5";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = "10.5 kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = "10.5 gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
-
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space start */
 
     str = " 10.5";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = " 10.5 kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 Kb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 KB";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 mb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = " 10.5 gb";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space end */
 
     str = "10.5 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = "10.5 kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = "10.5 mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = "10.5 gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* space start - space end */
 
     str = " 10.5 ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5);
 
     str = " 10.5 kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 Kb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 KB ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024);
 
     str = " 10.5 mb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024);
 
     str = " 10.5 gb ";
     result = 0;
-    if (ParseSizeString(str, &result) > 0) {
-        goto error;
-    }
-    if (result != 10.5 * 1024 * 1024 * 1024) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString(str, &result) > 0);
+    FAIL_IF(result != 10.5 * 1024 * 1024 * 1024);
 
     /* Should fail on unknown units. */
-    if (ParseSizeString("32eb", &result) > 0) {
-        goto error;
-    }
+    FAIL_IF(ParseSizeString("32eb", &result) > 0);
 
-    return 1;
- error:
-    return 0;
+    PASS;
 }
 
 void UtilMiscRegisterTests(void)

From 4ff52f26d70584dbcb234c6a7c261df13022a266 Mon Sep 17 00:00:00 2001
From: Hadiqa Alamdar Bukhari <hadiqaalamdar@gmail.com>
Date: Sat, 21 Oct 2023 02:25:50 +0500
Subject: [PATCH 077/462] flow-bit: Convert unittests to new FAIL/PASS API

Task #6329
---
 src/flow-bit.c | 138 ++++++++++++++-----------------------------------
 1 file changed, 39 insertions(+), 99 deletions(-)

diff --git a/src/flow-bit.c b/src/flow-bit.c
index 3662611ec8cf..f59e7eae943e 100644
--- a/src/flow-bit.c
+++ b/src/flow-bit.c
@@ -138,69 +138,51 @@ void FlowBitFree(FlowBit *fb)
 #ifdef UNITTESTS
 static int FlowBitTest01 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
     FlowBitAdd(&f, 0);
 
-    FlowBit *fb = FlowBitGet(&f,0);
-    if (fb != NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 0);
+    FAIL_IF_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest02 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
-    FlowBit *fb = FlowBitGet(&f,0);
-    if (fb == NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 0);
+    FAIL_IF_NOT_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest03 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
     FlowBitAdd(&f, 0);
 
     FlowBit *fb = FlowBitGet(&f,0);
-    if (fb == NULL) {
-        printf("fb == NULL although it was just added: ");
-        goto end;
-    }
+    FAIL_IF_NULL(fb);
 
     FlowBitRemove(&f, 0);
 
-    fb = FlowBitGet(&f,0);
-    if (fb != NULL) {
-        printf("fb != NULL although it was just removed: ");
-        goto end;
-    } else {
-        ret = 1;
-    }
-end:
+    fb = FlowBitGet(&f, 0);
+    FAIL_IF_NOT_NULL(fb);
+
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest04 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -209,18 +191,15 @@ static int FlowBitTest04 (void)
     FlowBitAdd(&f, 2);
     FlowBitAdd(&f, 3);
 
-    FlowBit *fb = FlowBitGet(&f,0);
-    if (fb != NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 0);
+    FAIL_IF_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest05 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -229,18 +208,15 @@ static int FlowBitTest05 (void)
     FlowBitAdd(&f, 2);
     FlowBitAdd(&f, 3);
 
-    FlowBit *fb = FlowBitGet(&f,1);
-    if (fb != NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 1);
+    FAIL_IF_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest06 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -249,18 +225,15 @@ static int FlowBitTest06 (void)
     FlowBitAdd(&f, 2);
     FlowBitAdd(&f, 3);
 
-    FlowBit *fb = FlowBitGet(&f,2);
-    if (fb != NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 2);
+    FAIL_IF_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest07 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -269,18 +242,15 @@ static int FlowBitTest07 (void)
     FlowBitAdd(&f, 2);
     FlowBitAdd(&f, 3);
 
-    FlowBit *fb = FlowBitGet(&f,3);
-    if (fb != NULL)
-        ret = 1;
+    FlowBit *fb = FlowBitGet(&f, 3);
+    FAIL_IF_NULL(fb);
 
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest08 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -290,27 +260,19 @@ static int FlowBitTest08 (void)
     FlowBitAdd(&f, 3);
 
     FlowBit *fb = FlowBitGet(&f,0);
-    if (fb == NULL)
-        goto end;
+    FAIL_IF_NULL(fb);
 
     FlowBitRemove(&f,0);
 
-    fb = FlowBitGet(&f,0);
-    if (fb != NULL) {
-        printf("fb != NULL even though it was removed: ");
-        goto end;
-    }
+    fb = FlowBitGet(&f, 0);
+    FAIL_IF_NOT_NULL(fb);
 
-    ret = 1;
-end:
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest09 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -320,27 +282,19 @@ static int FlowBitTest09 (void)
     FlowBitAdd(&f, 3);
 
     FlowBit *fb = FlowBitGet(&f,1);
-    if (fb == NULL)
-        goto end;
+    FAIL_IF_NULL(fb);
 
     FlowBitRemove(&f,1);
 
-    fb = FlowBitGet(&f,1);
-    if (fb != NULL) {
-        printf("fb != NULL even though it was removed: ");
-        goto end;
-    }
+    fb = FlowBitGet(&f, 1);
+    FAIL_IF_NOT_NULL(fb);
 
-    ret = 1;
-end:
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest10 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -350,27 +304,19 @@ static int FlowBitTest10 (void)
     FlowBitAdd(&f, 3);
 
     FlowBit *fb = FlowBitGet(&f,2);
-    if (fb == NULL)
-        goto end;
+    FAIL_IF_NULL(fb);
 
     FlowBitRemove(&f,2);
 
-    fb = FlowBitGet(&f,2);
-    if (fb != NULL) {
-        printf("fb != NULL even though it was removed: ");
-        goto end;
-    }
+    fb = FlowBitGet(&f, 2);
+    FAIL_IF_NOT_NULL(fb);
 
-    ret = 1;
-end:
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 static int FlowBitTest11 (void)
 {
-    int ret = 0;
-
     Flow f;
     memset(&f, 0, sizeof(Flow));
 
@@ -380,21 +326,15 @@ static int FlowBitTest11 (void)
     FlowBitAdd(&f, 3);
 
     FlowBit *fb = FlowBitGet(&f,3);
-    if (fb == NULL)
-        goto end;
+    FAIL_IF_NULL(fb);
 
     FlowBitRemove(&f,3);
 
-    fb = FlowBitGet(&f,3);
-    if (fb != NULL) {
-        printf("fb != NULL even though it was removed: ");
-        goto end;
-    }
+    fb = FlowBitGet(&f, 3);
+    FAIL_IF_NOT_NULL(fb);
 
-    ret = 1;
-end:
     GenericVarFree(f.flowvar);
-    return ret;
+    PASS;
 }
 
 #endif /* UNITTESTS */

From 68d3c0c3885e742b0eae17d9a4e42a6b2c8584fd Mon Sep 17 00:00:00 2001
From: Hadiqa Alamdar Bukhari <hadiqaalamdar@gmail.com>
Date: Mon, 23 Oct 2023 20:25:28 +0500
Subject: [PATCH 078/462] detect-tcp-window: Convert unittests to new FAIL/PASS
 API

Task #6339
---
 src/detect-tcp-window.c | 55 ++++++++++++++---------------------------
 1 file changed, 18 insertions(+), 37 deletions(-)

diff --git a/src/detect-tcp-window.c b/src/detect-tcp-window.c
index 9f5c56270bb5..3a8526b890f3 100644
--- a/src/detect-tcp-window.c
+++ b/src/detect-tcp-window.c
@@ -226,15 +226,13 @@ void DetectWindowFree(DetectEngineCtx *de_ctx, void *ptr)
  */
 static int DetectWindowTestParse01 (void)
 {
-    int result = 0;
     DetectWindowData *wd = NULL;
     wd = DetectWindowParse(NULL, "35402");
-    if (wd != NULL &&wd->size==35402) {
-        DetectWindowFree(NULL, wd);
-        result = 1;
-    }
+    FAIL_IF_NULL(wd);
+    FAIL_IF_NOT(wd->size == 35402);
 
-    return result;
+    DetectWindowFree(NULL, wd);
+    PASS;
 }
 
 /**
@@ -242,19 +240,14 @@ static int DetectWindowTestParse01 (void)
  */
 static int DetectWindowTestParse02 (void)
 {
-    int result = 0;
     DetectWindowData *wd = NULL;
     wd = DetectWindowParse(NULL, "!35402");
-    if (wd != NULL) {
-        if (wd->negated == 1 && wd->size==35402) {
-            result = 1;
-        } else {
-            printf("expected wd->negated=1 and wd->size=35402\n");
-        }
-        DetectWindowFree(NULL, wd);
-    }
+    FAIL_IF_NULL(wd);
+    FAIL_IF_NOT(wd->negated == 1);
+    FAIL_IF_NOT(wd->size == 35402);
 
-    return result;
+    DetectWindowFree(NULL, wd);
+    PASS;
 }
 
 /**
@@ -262,17 +255,12 @@ static int DetectWindowTestParse02 (void)
  */
 static int DetectWindowTestParse03 (void)
 {
-    int result = 0;
     DetectWindowData *wd = NULL;
     wd = DetectWindowParse(NULL, "");
-    if (wd == NULL) {
-        result = 1;
-    } else {
-        printf("expected a NULL pointer (It was an empty string)\n");
-    }
-    DetectWindowFree(NULL, wd);
+    FAIL_IF_NOT_NULL(wd);
 
-    return result;
+    DetectWindowFree(NULL, wd);
+    PASS;
 }
 
 /**
@@ -280,16 +268,12 @@ static int DetectWindowTestParse03 (void)
  */
 static int DetectWindowTestParse04 (void)
 {
-    int result = 0;
     DetectWindowData *wd = NULL;
     wd = DetectWindowParse(NULL, "1235402");
-    if (wd != NULL) {
-        printf("expected a NULL pointer (It was exceeding the MAX window size)\n");
-        DetectWindowFree(NULL, wd);
-    }else
-        result=1;
+    FAIL_IF_NOT_NULL(wd);
 
-    return result;
+    DetectWindowFree(NULL, wd);
+    PASS;
 }
 
 /**
@@ -297,7 +281,6 @@ static int DetectWindowTestParse04 (void)
  */
 static int DetectWindowTestPacket01 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"Hi all!";
     uint16_t buflen = strlen((char *)buf);
     Packet *p[3];
@@ -305,8 +288,7 @@ static int DetectWindowTestPacket01 (void)
     p[1] = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
     p[2] = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_ICMP);
 
-    if (p[0] == NULL || p[1] == NULL ||p[2] == NULL)
-        goto end;
+    FAIL_IF(p[0] == NULL || p[1] == NULL || p[2] == NULL);
 
     /* TCP wwindow = 40 */
     p[0]->tcph->th_win = htons(40);
@@ -327,11 +309,10 @@ static int DetectWindowTestPacket01 (void)
                               {0, 1},
                               /* packet 2 should not match */
                               {0, 0} };
-    result = UTHGenericTest(p, 3, sigs, sid, (uint32_t *) results, 2);
+    FAIL_IF(UTHGenericTest(p, 3, sigs, sid, (uint32_t *)results, 2) == 0);
 
     UTHFreePackets(p, 3);
-end:
-    return result;
+    PASS;
 }
 
 /**

From 54d8f45afc3136db3150e343fdaf9f9bb3859cc8 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 23 Oct 2023 15:19:45 -0300
Subject: [PATCH 079/462] userguide: add proper label to RPM install section

Use a reference label that is stable, instead of one that could change
in case a new section is added above it.
---
 doc/userguide/install.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/userguide/install.rst b/doc/userguide/install.rst
index 5c28aef020da..b3d39d216a0b 100644
--- a/doc/userguide/install.rst
+++ b/doc/userguide/install.rst
@@ -310,6 +310,8 @@ For Debian 10 (buster), for instance, run the following as ``root``::
     apt-get update
     apt-get install suricata -t buster-backports
 
+.. _RPM packages:
+
 CentOS, AlmaLinux, RockyLinux, Fedora, etc
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

From a9851430e2d7dd6c919f7332b71f222b889b835f Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 23 Oct 2023 15:25:44 -0300
Subject: [PATCH 080/462] github: improve template CLA request info

Indicate that the CLA only has to be signed once, as we have had
contributors think that was required for each new PR.
---
 .github/PULL_REQUEST_TEMPLATE.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 35d68550164c..15977b06f540 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,7 +1,9 @@
 Make sure these boxes are signed before submitting your Pull Request -- thank you.
 
-- [ ] I have read the contributing guide lines at https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html
-- [ ] I have signed the Open Information Security Foundation contribution agreement at https://suricata.io/about/contribution-agreement/
+- [ ] I have read the contributing guide lines at
+   https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html
+- [ ] I have signed the Open Information Security Foundation contribution agreement at
+   https://suricata.io/about/contribution-agreement/ (note: this is only required once)
 - [ ] I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)
 
 Link to [redmine](https://redmine.openinfosecfoundation.org/projects/suricata/issues) ticket:

From 292fda88b4a7dd9f099cf50e044c601cd5f00f9d Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Mon, 25 Sep 2023 15:37:07 +0200
Subject: [PATCH 081/462] unix-manager: prioritize the shutdown check

Make sure Suricata is in the running state before
you attempt to execute commands on the Unix sockets.
UnixMain is being called in an infinite loop where
TmThreadsCheckFlag(th_v, THV_KILL) is checked for the
deinit phase. However, it may take some time between
the start of Suricata's deinitialization and
the receipt of THV_KILL flag in the Unix thread.

In between this time period, the Unix manager can still
perform select() operation on the Unix socket while
the socket being already deinitialized.

Likely with a longer time span between the initial shutdown
command and actual closing of Unix sockets resulted in
an error of invalid file descriptors.

Ticket: #6272
---
 src/unix-manager.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/src/unix-manager.c b/src/unix-manager.c
index 9893553f429e..9fb5bd7935bc 100644
--- a/src/unix-manager.c
+++ b/src/unix-manager.c
@@ -628,6 +628,13 @@ static int UnixMain(UnixCommand * this)
     UnixClient *uclient;
     UnixClient *tclient;
 
+    if (suricata_ctl_flags & SURICATA_STOP) {
+        TAILQ_FOREACH_SAFE (uclient, &this->clients, next, tclient) {
+            UnixCommandClose(this, uclient->fd);
+        }
+        return 1;
+    }
+
     /* Wait activity on the socket */
     FD_ZERO(&select_set);
     FD_SET(this->socket, &select_set);
@@ -649,13 +656,6 @@ static int UnixMain(UnixCommand * this)
         return 0;
     }
 
-    if (suricata_ctl_flags & SURICATA_STOP) {
-        TAILQ_FOREACH_SAFE(uclient, &this->clients, next, tclient) {
-            UnixCommandClose(this, uclient->fd);
-        }
-        return 1;
-    }
-
     /* timeout: continue */
     if (ret == 0) {
         return 1;

From ffd769d178ccb8042cec620f24ad09087259b7bc Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Fri, 27 Oct 2023 09:09:43 -0400
Subject: [PATCH 082/462] detect/bytejump: Remove unused "Match" function

Issue: 4623

DetectBytejumpMatch is no longer used -- it's counterpart --
DetectByteJumpDoMatch is and will remain.
---
 src/detect-bytejump.c | 116 +-----------------------------------------
 1 file changed, 1 insertion(+), 115 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index ca1b72534845..21bbc3209bb1 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -61,8 +61,6 @@
 
 static DetectParseRegex parse_regex;
 
-static int DetectBytejumpMatch(DetectEngineThreadCtx *det_ctx,
-                        Packet *p, const Signature *s, const SigMatchCtx *ctx);
 static DetectBytejumpData *DetectBytejumpParse(
         DetectEngineCtx *de_ctx, const char *optstr, char **nbytes, char **offset);
 static int DetectBytejumpSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr);
@@ -76,7 +74,7 @@ void DetectBytejumpRegister (void)
     sigmatch_table[DETECT_BYTEJUMP].name = "byte_jump";
     sigmatch_table[DETECT_BYTEJUMP].desc = "allow the ability to select a <num of bytes> from an <offset> and move the detection pointer to that position";
     sigmatch_table[DETECT_BYTEJUMP].url = "/rules/payload-keywords.html#byte-jump";
-    sigmatch_table[DETECT_BYTEJUMP].Match = DetectBytejumpMatch;
+    sigmatch_table[DETECT_BYTEJUMP].Match = NULL;
     sigmatch_table[DETECT_BYTEJUMP].Setup = DetectBytejumpSetup;
     sigmatch_table[DETECT_BYTEJUMP].Free  = DetectBytejumpFree;
 #ifdef UNITTESTS
@@ -265,118 +263,6 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     SCReturnBool(true);
 }
 
-static int DetectBytejumpMatch(DetectEngineThreadCtx *det_ctx,
-                        Packet *p, const Signature *s, const SigMatchCtx *ctx)
-{
-    const DetectBytejumpData *data = (const DetectBytejumpData *)ctx;
-    const uint8_t *ptr = NULL;
-    const uint8_t *jumpptr = NULL;
-    uint16_t len = 0;
-    uint64_t val = 0;
-    int extbytes;
-
-    if (p->payload_len == 0) {
-        return 0;
-    }
-
-    /* Calculate the ptr value for the bytejump and length remaining in
-     * the packet from that point.
-     */
-    if (data->flags & DETECT_BYTEJUMP_RELATIVE) {
-        ptr = p->payload + det_ctx->buffer_offset;
-        DEBUG_VALIDATE_BUG_ON(p->payload_len - det_ctx->buffer_offset > UINT16_MAX);
-        len = (uint16_t)(p->payload_len - det_ctx->buffer_offset);
-
-        /* No match if there is no relative base */
-        if (ptr == NULL || len == 0) {
-            return 0;
-        }
-
-        ptr += data->offset;
-        len -= data->offset;
-    }
-    else {
-        ptr = p->payload + data->offset;
-        DEBUG_VALIDATE_BUG_ON(p->payload_len - data->offset > UINT16_MAX);
-        len = (uint16_t)(p->payload_len - data->offset);
-    }
-
-    /* Verify the to-be-extracted data is within the packet */
-    if (ptr < p->payload || data->nbytes > len) {
-        SCLogDebug("Data not within packet "
-               "payload=%p, ptr=%p, len=%d, nbytes=%d",
-               p->payload, ptr, len, data->nbytes);
-        return 0;
-    }
-
-    /* Extract the byte data */
-    if (data->flags & DETECT_BYTEJUMP_STRING) {
-        extbytes = ByteExtractStringUint64(&val, data->base,
-                                           data->nbytes, (const char *)ptr);
-        if (extbytes <= 0) {
-            SCLogDebug("error extracting %d bytes of string data: %d",
-                    data->nbytes, extbytes);
-            return -1;
-        }
-    }
-    else {
-        int endianness = (data->flags & DETECT_BYTEJUMP_LITTLE) ? BYTE_LITTLE_ENDIAN : BYTE_BIG_ENDIAN;
-        extbytes = ByteExtractUint64(&val, endianness, data->nbytes, ptr);
-        if (extbytes != data->nbytes) {
-            SCLogDebug("error extracting %d bytes of numeric data: %d",
-                    data->nbytes, extbytes);
-            return -1;
-        }
-    }
-
-    //printf("VAL: (%" PRIu64 " x %" PRIu32 ") + %d + %" PRId32 "\n", val, data->multiplier, extbytes, data->post_offset);
-
-    /* Adjust the jump value based on flags */
-    val *= data->multiplier;
-    if (data->flags & DETECT_BYTEJUMP_ALIGN) {
-        if ((val % 4) != 0) {
-            val += 4 - (val % 4);
-        }
-    }
-    val += data->post_offset;
-
-    /* Calculate the jump location */
-    if (data->flags & DETECT_BYTEJUMP_BEGIN) {
-        jumpptr = p->payload + val;
-        //printf("NEWVAL: payload %p + %ld = %p\n", p->payload, val, jumpptr);
-    }
-    else {
-        val += extbytes;
-        jumpptr = ptr + val;
-        //printf("NEWVAL: ptr %p + %ld = %p\n", ptr, val, jumpptr);
-    }
-
-
-    /* Validate that the jump location is still in the packet
-     * \todo Should this validate it is still in the *payload*?
-     */
-    if ((jumpptr < p->payload) || (jumpptr >= p->payload + p->payload_len)) {
-        SCLogDebug("Jump location (%p) is not within "
-               "packet (%p-%p)", jumpptr, p->payload, p->payload + p->payload_len - 1);
-        return 0;
-    }
-
-#ifdef DEBUG
-    if (SCLogDebugEnabled()) {
-        const uint8_t *sptr = (data->flags & DETECT_BYTEJUMP_BEGIN) ? p->payload
-                                                              : ptr;
-        SCLogDebug("jumping %" PRId64 " bytes from %p (%08x) to %p (%08x)",
-               val, sptr, (int)(sptr - p->payload),
-               jumpptr, (int)(jumpptr - p->payload));
-    }
-#endif /* DEBUG */
-
-    /* Adjust the detection context to the jump location. */
-    det_ctx->buffer_offset = jumpptr - p->payload;
-
-    return 1;
-}
-
 static DetectBytejumpData *DetectBytejumpParse(
         DetectEngineCtx *de_ctx, const char *optstr, char **nbytes_str, char **offset)
 {

From f363b99fd7592824dbcbec465f1968c6f615ccaa Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Fri, 27 Oct 2023 09:10:47 -0400
Subject: [PATCH 083/462] detect/bytejump: Improve end-of-buffer handling

Issue: 4623

This commit addresses the issues reported in issue 4623 when the jump
value points at the last byte in the buffer.
---
 src/detect-bytejump.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 21bbc3209bb1..1c851b719234 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -166,24 +166,19 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     /* Calculate the ptr value for the bytejump and length remaining in
      * the packet from that point.
      */
-    ptr = payload;
-    len = payload_len;
+    ptr = payload + offset;
+    len = payload_len - offset;
     if (flags & DETECT_BYTEJUMP_RELATIVE) {
         ptr += det_ctx->buffer_offset;
         len -= det_ctx->buffer_offset;
 
-        ptr += offset;
-        len -= offset;
+        SCLogDebug("[relative] after: ptr %p [len %d]", ptr, len);
 
         /* No match if there is no relative base */
-        if (ptr == NULL || len <= 0) {
+        if (ptr == NULL || (nbytes && len <= 0)) {
             SCReturnBool(false);
         }
     }
-    else {
-        ptr += offset;
-        len -= offset;
-    }
 
     /* Verify the to-be-extracted data is within the packet */
     if (ptr < payload || nbytes > len) {
@@ -243,7 +238,7 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     if (jumpptr < payload) {
         jumpptr = payload;
         SCLogDebug("jump location is before buffer start; resetting to buffer start");
-    } else if (jumpptr >= (payload + payload_len)) {
+    } else if (jumpptr > (payload + payload_len)) {
         SCLogDebug("Jump location (%" PRIu64 ") is not within payload (%" PRIu32 ")",
                 payload_len + val, payload_len);
         SCReturnBool(false);

From 804c5b737bd70b98d3e922c668ef49cb87a9a0a0 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Fri, 27 Oct 2023 16:58:08 -0600
Subject: [PATCH 084/462] runmodes: remove obsolete references to pcap auto
 modes

These auto modes were remove many years ago. Also cleanup the wording
a little.

Task: #6427
---
 src/runmode-pcap-file.c | 7 ++-----
 src/runmode-pcap.c      | 7 ++-----
 2 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/src/runmode-pcap-file.c b/src/runmode-pcap-file.c
index e03592514fee..7c5bfcc4c38a 100644
--- a/src/runmode-pcap-file.c
+++ b/src/runmode-pcap-file.c
@@ -42,11 +42,8 @@ void RunModeFilePcapRegister(void)
     RunModeRegisterNewRunMode(RUNMODE_PCAP_FILE, "single", "Single threaded pcap file mode",
             RunModeFilePcapSingle, NULL);
     RunModeRegisterNewRunMode(RUNMODE_PCAP_FILE, "autofp",
-            "Multi threaded pcap file mode.  Packets from "
-            "each flow are assigned to a single detect thread, "
-            "unlike \"pcap-file-auto\" where packets from "
-            "the same flow can be processed by any detect "
-            "thread",
+            "Multi-threaded pcap file mode. Packets from each flow are assigned to a consistent "
+            "detection thread",
             RunModeFilePcapAutoFp, NULL);
 
     return;
diff --git a/src/runmode-pcap.c b/src/runmode-pcap.c
index bfefe3ec0e41..21c32846b737 100644
--- a/src/runmode-pcap.c
+++ b/src/runmode-pcap.c
@@ -41,11 +41,8 @@ void RunModeIdsPcapRegister(void)
     RunModeRegisterNewRunMode(RUNMODE_PCAP_DEV, "single", "Single threaded pcap live mode",
             RunModeIdsPcapSingle, NULL);
     RunModeRegisterNewRunMode(RUNMODE_PCAP_DEV, "autofp",
-            "Multi threaded pcap live mode.  Packets from "
-            "each flow are assigned to a single detect thread, "
-            "unlike \"pcap_live_auto\" where packets from "
-            "the same flow can be processed by any detect "
-            "thread",
+            "Multi-threaded pcap live mode. Packets from each flow are assigned to a consistent "
+            "detection thread",
             RunModeIdsPcapAutoFp, NULL);
     RunModeRegisterNewRunMode(RUNMODE_PCAP_DEV, "workers",
             "Workers pcap live mode, each thread does all"

From d4e4bdac905d70ed36cb778eecf5b2178e8aba22 Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Mon, 30 Oct 2023 01:13:17 +0000
Subject: [PATCH 085/462] detect/bytejump: convert unittests to FAIL/PASS

Issue: #6328
---
 src/detect-bytejump.c | 259 +++++++++++++++---------------------------
 1 file changed, 89 insertions(+), 170 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 1c851b719234..b0b034774636 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -632,15 +632,12 @@ static int g_dce_stub_data_buffer_id = 0;
  */
 static int DetectBytejumpTestParse01(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL, "4,0", NULL, NULL);
-    if (data != NULL) {
-        DetectBytejumpFree(NULL, data);
-        result = 1;
-    }
+    FAIL_IF_NULL(data);
 
-    return result;
+    DetectBytejumpFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -648,23 +645,18 @@ static int DetectBytejumpTestParse01(void)
  */
 static int DetectBytejumpTestParse02(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL, "4, 0", NULL, NULL);
-    if (data != NULL) {
-        if (   (data->nbytes == 4)
-            && (data->offset == 0)
-            && (data->multiplier == 1)
-            && (data->post_offset == 0)
-            && (data->flags == 0)
-            && (data->base == DETECT_BYTEJUMP_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytejumpFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->multiplier == 1);
+    FAIL_IF_NOT(data->post_offset == 0);
+    FAIL_IF_NOT(data->flags == 0);
+    FAIL_IF_NOT(data->base == DETECT_BYTEJUMP_BASE_UNSET);
 
-    return result;
+    DetectBytejumpFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -672,30 +664,23 @@ static int DetectBytejumpTestParse02(void)
  */
 static int DetectBytejumpTestParse03(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL,
             " 4,0 , relative , little, string, "
             "dec, align, from_beginning",
             NULL, NULL);
-    if (data != NULL) {
-        if (   (data->nbytes == 4)
-            && (data->offset == 0)
-            && (data->multiplier == 1)
-            && (data->post_offset == 0)
-            && (data->flags == ( DETECT_BYTEJUMP_RELATIVE
-                                |DETECT_BYTEJUMP_LITTLE
-                                |DETECT_BYTEJUMP_STRING
-                                |DETECT_BYTEJUMP_ALIGN
-                                |DETECT_BYTEJUMP_BEGIN))
-            && (data->base == DETECT_BYTEJUMP_BASE_DEC))
-        {
-            result = 1;
-        }
-        DetectBytejumpFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->multiplier == 1);
+    FAIL_IF_NOT(data->post_offset == 0);
+    FAIL_IF_NOT(data->flags ==
+                (DETECT_BYTEJUMP_RELATIVE | DETECT_BYTEJUMP_LITTLE | DETECT_BYTEJUMP_STRING |
+                        DETECT_BYTEJUMP_ALIGN | DETECT_BYTEJUMP_BEGIN));
+    FAIL_IF_NOT(data->base == DETECT_BYTEJUMP_BASE_DEC);
 
-    return result;
+    DetectBytejumpFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -706,31 +691,24 @@ static int DetectBytejumpTestParse03(void)
  */
 static int DetectBytejumpTestParse04(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL,
             " 4,0 , relative , little, string, "
             "dec, align, from_beginning , "
             "multiplier 2 , post_offset -16 ",
             NULL, NULL);
-    if (data != NULL) {
-        if (   (data->nbytes == 4)
-            && (data->offset == 0)
-            && (data->multiplier == 2)
-            && (data->post_offset == -16)
-            && (data->flags == ( DETECT_BYTEJUMP_RELATIVE
-                                |DETECT_BYTEJUMP_LITTLE
-                                |DETECT_BYTEJUMP_ALIGN
-                                |DETECT_BYTEJUMP_STRING
-                                |DETECT_BYTEJUMP_BEGIN))
-            && (data->base == DETECT_BYTEJUMP_BASE_DEC))
-        {
-            result = 1;
-        }
-        DetectBytejumpFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->multiplier == 2);
+    FAIL_IF_NOT(data->post_offset == -16);
+    FAIL_IF_NOT(data->flags ==
+                (DETECT_BYTEJUMP_RELATIVE | DETECT_BYTEJUMP_LITTLE | DETECT_BYTEJUMP_ALIGN |
+                        DETECT_BYTEJUMP_STRING | DETECT_BYTEJUMP_BEGIN));
+    FAIL_IF_NOT(data->base == DETECT_BYTEJUMP_BASE_DEC);
 
-    return result;
+    DetectBytejumpFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -738,17 +716,14 @@ static int DetectBytejumpTestParse04(void)
  */
 static int DetectBytejumpTestParse05(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL,
             " 4,0 , relative , little, dec, "
             "align, from_beginning",
             NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -756,14 +731,11 @@ static int DetectBytejumpTestParse05(void)
  */
 static int DetectBytejumpTestParse06(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL, "9, 0", NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -771,14 +743,11 @@ static int DetectBytejumpTestParse06(void)
  */
 static int DetectBytejumpTestParse07(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL, "24, 0, string, dec", NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -786,14 +755,11 @@ static int DetectBytejumpTestParse07(void)
  */
 static int DetectBytejumpTestParse08(void)
 {
-    int result = 0;
     DetectBytejumpData *data = NULL;
     data = DetectBytejumpParse(NULL, "4, 0xffffffffffffffff", NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -856,11 +822,11 @@ static int DetectBytejumpTestParse10(void)
     FAIL_IF_NOT(sm->type == DETECT_BYTEJUMP);
 
     DetectBytejumpData *bd = (DetectBytejumpData *)sm->ctx;
-    if (!(bd->flags & DETECT_BYTEJUMP_DCE) && !(bd->flags & DETECT_BYTEJUMP_RELATIVE) &&
-            (bd->flags & DETECT_BYTEJUMP_STRING) && (bd->flags & DETECT_BYTEJUMP_BIG) &&
-            (bd->flags & DETECT_BYTEJUMP_LITTLE)) {
-        goto end;
-    }
+    FAIL_IF_NOT(bd->flags & DETECT_BYTEJUMP_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTEJUMP_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_LITTLE);
 
     s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any "
                                       "(msg:\"Testing bytejump_body\"; "
@@ -879,11 +845,11 @@ static int DetectBytejumpTestParse10(void)
     FAIL_IF_NOT(sm->type == DETECT_BYTEJUMP);
 
     bd = (DetectBytejumpData *)sm->ctx;
-    if (!(bd->flags & DETECT_BYTEJUMP_DCE) && !(bd->flags & DETECT_BYTEJUMP_RELATIVE) &&
-            (bd->flags & DETECT_BYTEJUMP_STRING) && (bd->flags & DETECT_BYTEJUMP_BIG) &&
-            (bd->flags & DETECT_BYTEJUMP_LITTLE)) {
-        goto end;
-    }
+    FAIL_IF_NOT(bd->flags & DETECT_BYTEJUMP_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTEJUMP_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_LITTLE);
 
     s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any "
                                       "(msg:\"Testing bytejump_body\"; "
@@ -902,13 +868,12 @@ static int DetectBytejumpTestParse10(void)
     FAIL_IF_NOT(sm->type == DETECT_BYTEJUMP);
 
     bd = (DetectBytejumpData *)sm->ctx;
-    if ((bd->flags & DETECT_BYTEJUMP_DCE) && !(bd->flags & DETECT_BYTEJUMP_RELATIVE) &&
-            (bd->flags & DETECT_BYTEJUMP_STRING) && (bd->flags & DETECT_BYTEJUMP_BIG) &&
-            (bd->flags & DETECT_BYTEJUMP_LITTLE)) {
-        goto end;
-    }
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTEJUMP_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTEJUMP_LITTLE);
 
-end:
     DetectEngineCtxFree(de_ctx);
     PASS;
 }
@@ -919,12 +884,10 @@ static int DetectBytejumpTestParse10(void)
 static int DetectBytejumpTestParse11(void)
 {
     DetectEngineCtx *de_ctx = NULL;
-    int result = 1;
     Signature *s = NULL;
 
     de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
+    FAIL_IF_NULL(de_ctx);
 
     de_ctx->flags |= DE_QUIET;
     s = SigInit(de_ctx, "alert tcp any any -> any any "
@@ -933,10 +896,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,string,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -944,10 +904,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_sub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,big,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -955,10 +912,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,little,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -966,10 +920,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,string,hex,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -977,10 +928,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,string,dec,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -988,10 +936,7 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,string,oct,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytejump_body\"; "
@@ -999,17 +944,12 @@ static int DetectBytejumpTestParse11(void)
                 "dce_stub_data; "
                 "content:\"one\"; byte_jump:4,0,align,multiplier 2, "
                 "post_offset -16,from_beginning,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
- end:
     SigGroupCleanup(de_ctx);
     SigCleanSignatures(de_ctx);
     DetectEngineCtxFree(de_ctx);
-
-    return result;
+    PASS;
 }
 
 /**
@@ -1073,7 +1013,6 @@ static int DetectBytejumpTestParse14(void)
  */
 static int DetectByteJumpTestPacket01 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"GET /AllWorkAndNoPlayMakesWillADullBoy HTTP/1.0"
                     "User-Agent: Wget/1.11.4"
                     "Accept: */*"
@@ -1084,18 +1023,16 @@ static int DetectByteJumpTestPacket01 (void)
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (msg:\"pcre + byte_test + "
     "relative\"; pcre:\"/AllWorkAndNoPlayMakesWillADullBoy/\"; byte_jump:1,6,"
     "relative,string,dec; content:\"0\"; sid:134; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig);
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig));
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 /**
@@ -1105,7 +1042,6 @@ static int DetectByteJumpTestPacket01 (void)
  */
 static int DetectByteJumpTestPacket02 (void)
 {
-    int result = 0;
     uint8_t buf[] = { 0x00, 0x00, 0x00, 0x77, 0xff, 0x53,
                     0x4d, 0x42, 0x2f, 0x00, 0x00, 0x00, 0x00, 0x18,
                     0x01, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -1117,23 +1053,20 @@ static int DetectByteJumpTestPacket02 (void)
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (msg:\"byte_jump with byte_jump"
                  " + relative\"; byte_jump:1,13; byte_jump:4,0,relative; "
                  "content:\"|48 00 00|\"; within:3; sid:144; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig);
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig));
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 static int DetectByteJumpTestPacket03(void)
 {
-    int result = 0;
     uint8_t *buf = NULL;
     uint16_t buflen = 0;
     buf = SCMalloc(4);
@@ -1147,20 +1080,18 @@ static int DetectByteJumpTestPacket03(void)
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (msg:\"byte_jump\"; "
         "byte_jump:1,214748364; sid:1; rev:1;)";
 
-    result = !UTHPacketMatchSig(p, sig);
+    FAIL_IF(UTHPacketMatchSig(p, sig));
 
     UTHFreePacket(p);
+    FAIL_IF_NULL(buf);
 
-end:
-    if (buf != NULL)
-        SCFree(buf);
-    return result;
+    SCFree(buf);
+    PASS;
 }
 
 /**
@@ -1168,22 +1099,19 @@ static int DetectByteJumpTestPacket03(void)
  */
 static int DetectByteJumpTestPacket04 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"XYZ04abcdABCD";
     uint16_t buflen = strlen((char *)buf);
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (content:\"XYZ\"; byte_jump:2,0,relative,string,dec; content:\"ABCD\"; distance:0; within:4; sid:1; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig);
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig));
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 /**
@@ -1191,22 +1119,19 @@ static int DetectByteJumpTestPacket04 (void)
  */
 static int DetectByteJumpTestPacket05 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"XYZ04abcdABCD";
     uint16_t buflen = strlen((char *)buf);
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (content:\"XYZ\"; byte_jump:2,0,relative,string,dec; content:\"cdABCD\"; within:6; sid:1; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig) ? 0 : 1;
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig) ? 0 : 1);
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 /**
@@ -1214,22 +1139,19 @@ static int DetectByteJumpTestPacket05 (void)
  */
 static int DetectByteJumpTestPacket06 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"XX04abcdABCD";
     uint16_t buflen = strlen((char *)buf);
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (content:\"XX\"; byte_jump:2,0,relative,string,dec,from_beginning; content:\"ABCD\"; distance:4; within:4; sid:1; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig);
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig));
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 /**
@@ -1237,22 +1159,19 @@ static int DetectByteJumpTestPacket06 (void)
  */
 static int DetectByteJumpTestPacket07 (void)
 {
-    int result = 0;
     uint8_t *buf = (uint8_t *)"XX04abcdABCD";
     uint16_t buflen = strlen((char *)buf);
     Packet *p;
     p = UTHBuildPacket((uint8_t *)buf, buflen, IPPROTO_TCP);
 
-    if (p == NULL)
-        goto end;
+    FAIL_IF_NULL(p);
 
     char sig[] = "alert tcp any any -> any any (content:\"XX\"; byte_jump:2,0,relative,string,dec,from_beginning; content:\"abcdABCD\"; distance:0; within:8; sid:1; rev:1;)";
 
-    result = UTHPacketMatchSig(p, sig) ? 1 : 0;
+    FAIL_IF_NOT(UTHPacketMatchSig(p, sig) ? 1 : 0);
 
     UTHFreePacket(p);
-end:
-    return result;
+    PASS;
 }
 
 /**

From a240a93b6931c94485d336cdc340e16929437a01 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Fri, 27 Oct 2023 10:19:31 -0600
Subject: [PATCH 086/462] dns/eve: use default formats if formats is empty

If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.

I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.

So if no formats are specified, use the default of all.

Bug: #6420
---
 src/output-json-dns.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/src/output-json-dns.c b/src/output-json-dns.c
index 0b6589d5c1bf..cd3ccac29db8 100644
--- a/src/output-json-dns.c
+++ b/src/output-json-dns.c
@@ -512,15 +512,23 @@ static void JsonDnsLogInitFilters(LogDnsFileCtx *dnslog_ctx, ConfNode *conf)
         if (dnslog_ctx->flags & LOG_ANSWERS) {
             ConfNode *format;
             if ((format = ConfNodeLookupChild(conf, "formats")) != NULL) {
-                dnslog_ctx->flags &= ~LOG_FORMAT_ALL;
+                uint64_t flags = 0;
                 ConfNode *field;
                 TAILQ_FOREACH (field, &format->head, next) {
                     if (strcasecmp(field->val, "detailed") == 0) {
-                        dnslog_ctx->flags |= LOG_FORMAT_DETAILED;
+                        flags |= LOG_FORMAT_DETAILED;
                     } else if (strcasecmp(field->val, "grouped") == 0) {
-                        dnslog_ctx->flags |= LOG_FORMAT_GROUPED;
+                        flags |= LOG_FORMAT_GROUPED;
+                    } else {
+                        SCLogWarning("Invalid JSON DNS log format: %s", field->val);
                     }
                 }
+                if (flags) {
+                    dnslog_ctx->flags &= ~LOG_FORMAT_ALL;
+                    dnslog_ctx->flags |= flags;
+                } else {
+                    SCLogWarning("Empty EVE DNS format array, using defaults");
+                }
             } else {
                 dnslog_ctx->flags |= LOG_FORMAT_ALL;
             }

From 7d60bb71397fa2c23eea06decdc1f6761deabfae Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 30 Oct 2023 12:16:33 -0600
Subject: [PATCH 087/462] dns/eve: make removed v1 style a warning, not an
 error

We don't error out in this case, but instead default to v2. So use a
warning instead of an error.
---
 src/output-json-dns.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/output-json-dns.c b/src/output-json-dns.c
index cd3ccac29db8..020e27853a9e 100644
--- a/src/output-json-dns.c
+++ b/src/output-json-dns.c
@@ -486,7 +486,7 @@ static void JsonDnsCheckVersion(ConfNode *conf)
                     break;
                 case 1:
                     if (!v1_deprecation_warned) {
-                        SCLogError("DNS EVE v1 logging has been removed, will use v2");
+                        SCLogWarning("DNS EVE v1 logging has been removed, will use v2");
                         v1_deprecation_warned = true;
                     }
                     break;

From 3a40ed5641772b8f7620cbbaa64f1a62c27f017b Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Tue, 31 Oct 2023 00:09:43 +0000
Subject: [PATCH 088/462] detect/bytetest: convert unittests to FAIL/PASS

Issue: #6332
---
 src/detect-bytetest.c | 534 +++++++++++++++---------------------------
 1 file changed, 189 insertions(+), 345 deletions(-)

diff --git a/src/detect-bytetest.c b/src/detect-bytetest.c
index 3eeba0f0f0e6..27070ffa36ff 100644
--- a/src/detect-bytetest.c
+++ b/src/detect-bytetest.c
@@ -759,15 +759,11 @@ static int g_dce_stub_data_buffer_id = 0;
  */
 static int DetectBytetestTestParse01(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, =, 1 , 0", NULL, NULL, NULL);
-    if (data != NULL) {
-        DetectBytetestFree(NULL, data);
-        result = 1;
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -775,23 +771,18 @@ static int DetectBytetestTestParse01(void)
  */
 static int DetectBytetestTestParse02(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, !=, 1, 0", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 4)
-            && (data->value == 1)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 1);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -799,24 +790,19 @@ static int DetectBytetestTestParse02(void)
  */
 static int DetectBytetestTestParse03(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, !=, 1, 0, relative", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 4)
-            && (data->value == 1)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->flags == DETECT_BYTETEST_RELATIVE)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 1);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_RELATIVE);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -824,24 +810,18 @@ static int DetectBytetestTestParse03(void)
  */
 static int DetectBytetestTestParse04(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, !=, 1, 0, string, oct", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 4)
-            && (data->value == 1)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->flags == DETECT_BYTETEST_STRING)
-            && (data->base == DETECT_BYTETEST_BASE_OCT))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 1);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_STRING);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_OCT);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -849,23 +829,17 @@ static int DetectBytetestTestParse04(void)
  */
 static int DetectBytetestTestParse05(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, =, 1, 0, string, dec", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 4)
-            && (data->value == 1)
-            && (data->offset == 0)
-            && (data->flags == DETECT_BYTETEST_STRING)
-            && (data->base == DETECT_BYTETEST_BASE_DEC))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 1);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_STRING);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_DEC);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -873,23 +847,17 @@ static int DetectBytetestTestParse05(void)
  */
 static int DetectBytetestTestParse06(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, >, 1, 0, string, hex", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_GT)
-            && (data->nbytes == 4)
-            && (data->value == 1)
-            && (data->offset == 0)
-            && (data->flags == DETECT_BYTETEST_STRING)
-            && (data->base == DETECT_BYTETEST_BASE_HEX))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_GT);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 1);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_STRING);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_HEX);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -897,23 +865,17 @@ static int DetectBytetestTestParse06(void)
  */
 static int DetectBytetestTestParse07(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, <, 5, 0, big", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_LT)
-            && (data->nbytes == 4)
-            && (data->value == 5)
-            && (data->offset == 0)
-            && (data->flags & DETECT_BYTETEST_BIG)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_LT);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags & DETECT_BYTETEST_BIG);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -921,23 +883,18 @@ static int DetectBytetestTestParse07(void)
  */
 static int DetectBytetestTestParse08(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, <, 5, 0, little", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_LT)
-            && (data->nbytes == 4)
-            && (data->value == 5)
-            && (data->offset == 0)
-            && (data->flags == DETECT_BYTETEST_LITTLE)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_LT);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_LITTLE);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -945,23 +902,17 @@ static int DetectBytetestTestParse08(void)
  */
 static int DetectBytetestTestParse09(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, !, 5, 0", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 4)
-            && (data->value == 5)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
-
-    return result;
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -969,24 +920,19 @@ static int DetectBytetestTestParse09(void)
  */
 static int DetectBytetestTestParse10(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("	4 , ! &, 5	, 0 , little ", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_AND)
-            && (data->nbytes == 4)
-            && (data->value == 5)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->flags == DETECT_BYTETEST_LITTLE)
-            && (data->base == DETECT_BYTETEST_BASE_UNSET))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_AND);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_LITTLE);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_UNSET);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -994,26 +940,20 @@ static int DetectBytetestTestParse10(void)
  */
 static int DetectBytetestTestParse11(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4,!^,5,0,little,string,relative,hex", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_OR)
-            && (data->nbytes == 4)
-            && (data->value == 5)
-            && (data->offset == 0)
-            && (data->neg_op)
-            && (data->flags == (DETECT_BYTETEST_LITTLE
-                                |DETECT_BYTETEST_STRING
-                                |DETECT_BYTETEST_RELATIVE))
-            && (data->base == DETECT_BYTETEST_BASE_HEX))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_OR);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->neg_op);
+    FAIL_IF_NOT(data->flags ==
+                (DETECT_BYTETEST_LITTLE | DETECT_BYTETEST_STRING | DETECT_BYTETEST_RELATIVE));
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_HEX);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -1021,14 +961,11 @@ static int DetectBytetestTestParse11(void)
  */
 static int DetectBytetestTestParse12(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, =, 1, 0, hex", NULL, NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -1036,14 +973,10 @@ static int DetectBytetestTestParse12(void)
  */
 static int DetectBytetestTestParse13(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("9, =, 1, 0", NULL, NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
-
-    return result;
+    FAIL_IF_NOT_NULL(data);
+    PASS;
 }
 
 /**
@@ -1051,23 +984,18 @@ static int DetectBytetestTestParse13(void)
  */
 static int DetectBytetestTestParse14(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("23,=,0xffffffffffffffffULL,0,string,oct", NULL, NULL, NULL);
-    if (data != NULL) {
-        if (   (data->op == DETECT_BYTETEST_OP_EQ)
-            && (data->nbytes == 23)
-            && (data->value == 0xffffffffffffffffULL)
-            && (data->offset == 0)
-            && (data->flags == DETECT_BYTETEST_STRING)
-            && (data->base == DETECT_BYTETEST_BASE_OCT))
-        {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_EQ);
+    FAIL_IF_NOT(data->nbytes == 23);
+    FAIL_IF_NOT(data->value == 0xffffffffffffffffULL);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags == DETECT_BYTETEST_STRING);
+    FAIL_IF_NOT(data->base == DETECT_BYTETEST_BASE_OCT);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -1075,14 +1003,11 @@ static int DetectBytetestTestParse14(void)
  */
 static int DetectBytetestTestParse15(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("24, =, 0xffffffffffffffffULL, 0, string", NULL, NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -1090,14 +1015,11 @@ static int DetectBytetestTestParse15(void)
  */
 static int DetectBytetestTestParse16(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4,=,0,0xffffffffffffffffULL", NULL, NULL, NULL);
-    if (data == NULL) {
-        result = 1;
-    }
+    FAIL_IF_NOT_NULL(data);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -1105,21 +1027,17 @@ static int DetectBytetestTestParse16(void)
  */
 static int DetectBytetestTestParse17(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, <, 5, 0, dce", NULL, NULL, NULL);
-    if (data != NULL) {
-        if ( (data->op == DETECT_BYTETEST_OP_LT) &&
-             (data->nbytes == 4) &&
-             (data->value == 5) &&
-             (data->offset == 0) &&
-             (data->flags & DETECT_BYTETEST_DCE) ) {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_LT);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF_NOT(data->flags & DETECT_BYTETEST_DCE);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -1127,21 +1045,17 @@ static int DetectBytetestTestParse17(void)
  */
 static int DetectBytetestTestParse18(void)
 {
-    int result = 0;
     DetectBytetestData *data = NULL;
     data = DetectBytetestParse("4, <, 5, 0", NULL, NULL, NULL);
-    if (data != NULL) {
-        if ( (data->op == DETECT_BYTETEST_OP_LT) &&
-             (data->nbytes == 4) &&
-             (data->value == 5) &&
-             (data->offset == 0) &&
-             !(data->flags & DETECT_BYTETEST_DCE) ) {
-            result = 1;
-        }
-        DetectBytetestFree(NULL, data);
-    }
+    FAIL_IF_NULL(data);
+    FAIL_IF_NOT(data->op == DETECT_BYTETEST_OP_LT);
+    FAIL_IF_NOT(data->nbytes == 4);
+    FAIL_IF_NOT(data->value == 5);
+    FAIL_IF_NOT(data->offset == 0);
+    FAIL_IF(data->flags & DETECT_BYTETEST_DCE);
 
-    return result;
+    DetectBytetestFree(NULL, data);
+    PASS;
 }
 
 /**
@@ -1150,26 +1064,20 @@ static int DetectBytetestTestParse18(void)
 static int DetectBytetestTestParse19(void)
 {
     Signature *s = SigAlloc();
-    if (s == NULL)
-        return 0;
+    FAIL_IF_NULL(s);
 
-    int result = 1;
+    FAIL_IF(DetectSignatureSetAppProto(s, ALPROTO_DCERPC) < 0);
 
-    if (DetectSignatureSetAppProto(s, ALPROTO_DCERPC) < 0) {
-        SigFree(NULL, s);
-        return 0;
-    }
-
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,dce") == 0);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,string,dce") == -1);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,big,dce") == -1);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,little,dce") == -1);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,hex,dce") == -1);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,oct,dce") == -1);
-    result &= (DetectBytetestSetup(NULL, s, "1,=,1,6,dec,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,dce") == 0);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,string,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,big,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,little,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,hex,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,oct,dce") == -1);
+    FAIL_IF_NOT(DetectBytetestSetup(NULL, s, "1,=,1,6,dec,dce") == -1);
 
     SigFree(NULL, s);
-    return result;
+    PASS;
 }
 
 /**
@@ -1178,13 +1086,11 @@ static int DetectBytetestTestParse19(void)
 static int DetectBytetestTestParse20(void)
 {
     DetectEngineCtx *de_ctx = NULL;
-    int result = 1;
     Signature *s = NULL;
     DetectBytetestData *bd = NULL;
 
     de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
+    FAIL_IF_NULL(de_ctx);
 
     de_ctx->flags |= DE_QUIET;
     de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
@@ -1193,10 +1099,8 @@ static int DetectBytetestTestParse20(void)
                                "dce_stub_data; "
                                "content:\"one\"; distance:0; "
                                "byte_test:1,=,1,6,relative,dce; sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(de_ctx->sig_list);
+
     s = de_ctx->sig_list;
 
     SigMatch *sm = DetectBufferGetFirstSigMatch(s, g_dce_stub_data_buffer_id);
@@ -1205,15 +1109,12 @@ static int DetectBytetestTestParse20(void)
     sm = sm->next;
     FAIL_IF_NOT(sm->type == DETECT_BYTETEST);
     bd = (DetectBytetestData *)sm->ctx;
-    if (!(bd->flags & DETECT_BYTETEST_DCE) &&
-        !(bd->flags & DETECT_BYTETEST_RELATIVE) &&
-        (bd->flags & DETECT_BYTETEST_STRING) &&
-        (bd->flags & DETECT_BYTETEST_BIG) &&
-        (bd->flags & DETECT_BYTETEST_LITTLE) &&
-        (bd->neg_op) ) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_LITTLE);
+    FAIL_IF(bd->neg_op);
 
     s->next = SigInit(de_ctx, "alert tcp any any -> any any "
                       "(msg:\"Testing bytetest_body\"; "
@@ -1221,10 +1122,8 @@ static int DetectBytetestTestParse20(void)
                       "dce_stub_data; "
                       "content:\"one\"; distance:0; "
                       "byte_test:1,=,1,6,relative,dce; sid:1;)");
-    if (s->next == NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(s->next);
+
     s = s->next;
 
     sm = DetectBufferGetFirstSigMatch(s, g_dce_stub_data_buffer_id);
@@ -1232,15 +1131,12 @@ static int DetectBytetestTestParse20(void)
     FAIL_IF_NULL(sm->next);
     sm = sm->next;
     bd = (DetectBytetestData *)sm->ctx;
-    if (!(bd->flags & DETECT_BYTETEST_DCE) &&
-        !(bd->flags & DETECT_BYTETEST_RELATIVE) &&
-        (bd->flags & DETECT_BYTETEST_STRING) &&
-        (bd->flags & DETECT_BYTETEST_BIG) &&
-        (bd->flags & DETECT_BYTETEST_LITTLE) &&
-        (bd->neg_op) ) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_LITTLE);
+    FAIL_IF(bd->neg_op);
 
     s->next = SigInit(de_ctx, "alert tcp any any -> any any "
                       "(msg:\"Testing bytetest_body\"; "
@@ -1248,32 +1144,26 @@ static int DetectBytetestTestParse20(void)
                       "dce_stub_data; "
                       "content:\"one\"; distance:0; "
                       "byte_test:1,=,1,6,relative; sid:1;)");
-    if (s->next == NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(s->next);
+
     s = s->next;
     sm = DetectBufferGetFirstSigMatch(s, g_dce_stub_data_buffer_id);
     FAIL_IF_NULL(sm);
     FAIL_IF_NULL(sm->next);
     sm = sm->next;
     bd = (DetectBytetestData *)sm->ctx;
-    if ((bd->flags & DETECT_BYTETEST_DCE) &&
-        !(bd->flags & DETECT_BYTETEST_RELATIVE) &&
-        (bd->flags & DETECT_BYTETEST_STRING) &&
-        (bd->flags & DETECT_BYTETEST_BIG) &&
-        (bd->flags & DETECT_BYTETEST_LITTLE) &&
-        (bd->neg_op) ) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF(bd->flags & DETECT_BYTETEST_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_LITTLE);
+    FAIL_IF(bd->neg_op);
 
- end:
     SigGroupCleanup(de_ctx);
     SigCleanSignatures(de_ctx);
     DetectEngineCtxFree(de_ctx);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -1282,119 +1172,83 @@ static int DetectBytetestTestParse20(void)
 static int DetectBytetestTestParse21(void)
 {
     DetectEngineCtx *de_ctx = NULL;
-    int result = 1;
     Signature *s = NULL;
 
     de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
+    FAIL_IF_NULL(de_ctx);
 
     de_ctx->flags |= DE_QUIET;
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,string,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,big,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,little,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,hex,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,dec,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,oct,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,string,hex,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,big,string,hex,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,big,string,oct,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,little,string,hex,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
     s = SigInit(de_ctx, "alert tcp any any -> any any "
                 "(msg:\"Testing bytetest_body\"; "
                 "dce_iface:3919286a-b10c-11d0-9ba8-00c04fd92ef5; "
                 "content:\"one\"; byte_test:1,=,1,6,big,string,dec,dce; sid:1;)");
-    if (s != NULL) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NOT_NULL(s);
 
- end:
     SigGroupCleanup(de_ctx);
     SigCleanSignatures(de_ctx);
     DetectEngineCtxFree(de_ctx);
 
-    return result;
+    PASS;
 }
 
 /**
@@ -1403,44 +1257,34 @@ static int DetectBytetestTestParse21(void)
 static int DetectBytetestTestParse22(void)
 {
     DetectEngineCtx *de_ctx = NULL;
-    int result = 0;
     Signature *s = NULL;
     DetectBytetestData *bd = NULL;
 
     de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
+    FAIL_IF_NULL(de_ctx);
 
     de_ctx->flags |= DE_QUIET;
     de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
                                "(file_data; byte_test:1,=,1,6,relative; sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        printf("sig parse failed: ");
-        goto end;
-    }
+    FAIL_IF_NULL(de_ctx->sig_list);
 
     s = de_ctx->sig_list;
     SigMatch *sm = DetectBufferGetFirstSigMatch(s, g_file_data_buffer_id);
     FAIL_IF_NULL(sm);
     FAIL_IF_NOT(sm->type == DETECT_BYTETEST);
     bd = (DetectBytetestData *)sm->ctx;
-    if (bd->flags & DETECT_BYTETEST_DCE &&
-        bd->flags & DETECT_BYTETEST_RELATIVE &&
-        (bd->flags & DETECT_BYTETEST_STRING) &&
-        (bd->flags & DETECT_BYTETEST_BIG) &&
-        (bd->flags & DETECT_BYTETEST_LITTLE) &&
-        (bd->neg_op) ) {
-        printf("wrong flags: ");
-        goto end;
-    }
+    FAIL_IF(bd->flags & DETECT_BYTETEST_DCE);
+    FAIL_IF_NOT(bd->flags & DETECT_BYTETEST_RELATIVE);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_STRING);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_BIG);
+    FAIL_IF(bd->flags & DETECT_BYTETEST_LITTLE);
+    FAIL_IF(bd->neg_op);
 
-    result = 1;
- end:
     SigGroupCleanup(de_ctx);
     SigCleanSignatures(de_ctx);
     DetectEngineCtxFree(de_ctx);
 
-    return result;
+    PASS;
 }
 
 /**

From 80f13b93aad9a66a82485d7d31fbc7eec085ad31 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Sep 2023 09:48:15 +0200
Subject: [PATCH 089/462] detect/urilen: fix discontinue matching logic

Actually discontinue matching.

Fixes: 21f9cc3a39a0 ("discontinue matching on buffer if urilen returns a match failure.")
---
 src/detect-engine-content-inspection.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index ae7102f9c061..8c5feb61a226 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -656,7 +656,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
             goto match;
         }
 
-        det_ctx->discontinue_matching = 0;
+        det_ctx->discontinue_matching = 1;
 
         goto no_match;
 #ifdef HAVE_LUA

From 6307a4d4b93d000d4c5e19434a6dd3958946dcd8 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 11 Sep 2023 07:05:48 +0200
Subject: [PATCH 090/462] host/iprep: run all timeout logic

Run all timeout logic if iprep is in use as well.

Minor code cleanups.

Bug: #6436.
---
 src/host-timeout.c | 32 +++++++-------------------------
 1 file changed, 7 insertions(+), 25 deletions(-)

diff --git a/src/host-timeout.c b/src/host-timeout.c
index 75918f3ebadd..8542d5f78abb 100644
--- a/src/host-timeout.c
+++ b/src/host-timeout.c
@@ -53,9 +53,7 @@ uint32_t HostGetActiveCount(void)
  */
 static int HostHostTimedOut(Host *h, SCTime_t ts)
 {
-    int tags = 0;
-    int thresholds = 0;
-    int vars = 0;
+    int busy = 0;
 
     /** never prune a host that is used by a packet
      *  we are currently processing in one of the threads */
@@ -63,28 +61,12 @@ static int HostHostTimedOut(Host *h, SCTime_t ts)
         return 0;
     }
 
-    if (h->iprep) {
-        if (SRepHostTimedOut(h) == 0)
-            return 0;
-
-        SCLogDebug("host %p reputation timed out", h);
-    }
-
-    if (TagHostHasTag(h) && TagTimeoutCheck(h, ts) == 0) {
-        tags = 1;
-    }
-    if (ThresholdHostHasThreshold(h) && ThresholdHostTimeoutCheck(h, ts) == 0) {
-        thresholds = 1;
-    }
-    if (HostHasHostBits(h) && HostBitsTimedoutCheck(h, ts) == 0) {
-        vars = 1;
-    }
-
-    if (tags || thresholds || vars)
-        return 0;
-
-    SCLogDebug("host %p timed out", h);
-    return 1;
+    busy |= (h->iprep && SRepHostTimedOut(h) == 0);
+    busy |= (TagHostHasTag(h) && TagTimeoutCheck(h, ts) == 0);
+    busy |= (ThresholdHostHasThreshold(h) && ThresholdHostTimeoutCheck(h, ts) == 0);
+    busy |= (HostHasHostBits(h) && HostBitsTimedoutCheck(h, ts) == 0);
+    SCLogDebug("host %p %s", h, busy ? "still active" : "timed out");
+    return !busy;
 }
 
 /**

From 0dda7f535cc109c9bf77239d29574b34878e5ad8 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 12 Sep 2023 12:13:52 +0200
Subject: [PATCH 091/462] flow/timeout: no need to wait for packetpool

The timeout logic no longer passes packets around, so don't depend
on the packet pool.

Bug: #6292.
---
 src/flow-timeout.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/src/flow-timeout.c b/src/flow-timeout.c
index 91dd872375e4..6a9b707c2186 100644
--- a/src/flow-timeout.c
+++ b/src/flow-timeout.c
@@ -370,8 +370,6 @@ static inline void FlowForceReassemblyForHash(void)
 {
     for (uint32_t idx = 0; idx < flow_config.hash_size; idx++) {
         FlowBucket *fb = &flow_hash[idx];
-
-        PacketPoolWaitForN(9);
         FBLOCK_LOCK(fb);
 
         Flow *f = fb->head;
@@ -380,7 +378,6 @@ static inline void FlowForceReassemblyForHash(void)
         /* we need to loop through all the flows in the queue */
         while (f != NULL) {
             Flow *next_f = f->next;
-            PacketPoolWaitForN(3);
 
             FLOWLOCK_WRLOCK(f);
 

From 6ae37b06f1f40e569e529f8c2965116c1da6f9f8 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 12 Sep 2023 12:15:54 +0200
Subject: [PATCH 092/462] packetpool: remove WaitForN logic as it is unused

---
 src/tmqh-packetpool.c | 56 -------------------------------------------
 src/tmqh-packetpool.h |  1 -
 2 files changed, 57 deletions(-)

diff --git a/src/tmqh-packetpool.c b/src/tmqh-packetpool.c
index 90d52bf787a4..e7bacf57b29e 100644
--- a/src/tmqh-packetpool.c
+++ b/src/tmqh-packetpool.c
@@ -81,62 +81,6 @@ void PacketPoolWait(void)
         cc_barrier();
 }
 
-/** \brief Wait until we have the requested amount of packets in the pool
- *
- *  In some cases waiting for packets is undesirable. Especially when
- *  a wait would happen under a lock of some kind, other parts of the
- *  engine could have to wait.
- *
- *  This function only returns when at least N packets are in our pool.
- *
- *  If counting in our pool's main stack didn't give us the number we
- *  are seeking, we check if the return stack is filled and add those
- *  to our main stack. Then we retry.
- *
- *  \param n number of packets needed
- */
-void PacketPoolWaitForN(int n)
-{
-    PktPool *my_pool = GetThreadPacketPool();
-
-    while (1) {
-        PacketPoolWait();
-
-        /* count packets in our stack */
-        int i = 0;
-        Packet *p, *pp;
-        pp = p = my_pool->head;
-        while (p != NULL) {
-            if (++i == n)
-                return;
-
-            pp = p;
-            p = p->next;
-        }
-
-        /* check return stack, return to our pool and retry counting */
-        if (my_pool->return_stack.head != NULL) {
-            SCMutexLock(&my_pool->return_stack.mutex);
-            /* Move all the packets from the locked return stack to the local stack. */
-            if (pp) {
-                pp->next = my_pool->return_stack.head;
-            } else {
-                my_pool->head = my_pool->return_stack.head;
-            }
-            my_pool->return_stack.head = NULL;
-            SC_ATOMIC_RESET(my_pool->return_stack.sync_now);
-            SCMutexUnlock(&my_pool->return_stack.mutex);
-
-        /* or signal that we need packets and wait */
-        } else {
-            SCMutexLock(&my_pool->return_stack.mutex);
-            SC_ATOMIC_ADD(my_pool->return_stack.sync_now, 1);
-            SCCondWait(&my_pool->return_stack.cond, &my_pool->return_stack.mutex);
-            SCMutexUnlock(&my_pool->return_stack.mutex);
-        }
-    }
-}
-
 /** \brief a initialized packet
  *
  *  \warning Use *only* at init, not at packet runtime
diff --git a/src/tmqh-packetpool.h b/src/tmqh-packetpool.h
index a48fb23b5f63..2e9672d4458c 100644
--- a/src/tmqh-packetpool.h
+++ b/src/tmqh-packetpool.h
@@ -72,7 +72,6 @@ void TmqhReleasePacketsToPacketPool(PacketQueue *);
 void TmqhPacketpoolRegister(void);
 Packet *PacketPoolGetPacket(void);
 void PacketPoolWait(void);
-void PacketPoolWaitForN(int n);
 void PacketPoolReturnPacket(Packet *p);
 void PacketPoolInit(void);
 void PacketPoolInitEmpty(void);

From 087ca49e397a4643ff1e8762e4a02277737caef1 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 12 Sep 2023 12:27:03 +0200
Subject: [PATCH 093/462] packetpool: return one packet as well on sync now

If a thread is hitting the packet pool return on a 'sync_now' return
the packet also if it is the first packet since the last flush.

Bug: #6435.
---
 src/tmqh-packetpool.c | 27 +++++++++++++++------------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/src/tmqh-packetpool.c b/src/tmqh-packetpool.c
index e7bacf57b29e..411af684ee2e 100644
--- a/src/tmqh-packetpool.c
+++ b/src/tmqh-packetpool.c
@@ -172,18 +172,21 @@ void PacketPoolReturnPacket(Packet *p)
         my_pool->head = p;
     } else {
         PktPool *pending_pool = my_pool->pending_pool;
-        if (pending_pool == NULL) {
-            /* No pending packet, so store the current packet. */
-            p->next = NULL;
-            my_pool->pending_pool = pool;
-            my_pool->pending_head = p;
-            my_pool->pending_tail = p;
-            my_pool->pending_count = 1;
-        } else if (pending_pool == pool) {
-            /* Another packet for the pending pool list. */
-            p->next = my_pool->pending_head;
-            my_pool->pending_head = p;
-            my_pool->pending_count++;
+        if (pending_pool == NULL || pending_pool == pool) {
+            if (pending_pool == NULL) {
+                /* No pending packet, so store the current packet. */
+                p->next = NULL;
+                my_pool->pending_pool = pool;
+                my_pool->pending_head = p;
+                my_pool->pending_tail = p;
+                my_pool->pending_count = 1;
+            } else if (pending_pool == pool) {
+                /* Another packet for the pending pool list. */
+                p->next = my_pool->pending_head;
+                my_pool->pending_head = p;
+                my_pool->pending_count++;
+            }
+
             if (SC_ATOMIC_GET(pool->return_stack.sync_now) || my_pool->pending_count > max_pending_return_packets) {
                 /* Return the entire list of pending packets. */
                 SCMutexLock(&pool->return_stack.mutex);

From dc40a139acb3c66f5d34074c240a1f81dec3f002 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 14 Sep 2023 06:49:31 +0200
Subject: [PATCH 094/462] packetpool: signal waiter within lock

Needed for predictable scheduling. From pthread_cond_signal man page:

"The pthread_cond_signal() or pthread_cond_broadcast() functions may
 be called by a thread whether or not it currently owns the mutex that
 threads calling pthread_cond_wait() or pthread_cond_timedwait() have
 associated with the condition variable during their waits; however, if
 predictable scheduling behaviour is required, then that mutex is locked
 by the thread calling pthread_cond_signal() or pthread_cond_broadcast()."
---
 src/tmqh-packetpool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/tmqh-packetpool.c b/src/tmqh-packetpool.c
index 411af684ee2e..85946517953a 100644
--- a/src/tmqh-packetpool.c
+++ b/src/tmqh-packetpool.c
@@ -193,8 +193,8 @@ void PacketPoolReturnPacket(Packet *p)
                 my_pool->pending_tail->next = pool->return_stack.head;
                 pool->return_stack.head = my_pool->pending_head;
                 SC_ATOMIC_RESET(pool->return_stack.sync_now);
-                SCMutexUnlock(&pool->return_stack.mutex);
                 SCCondSignal(&pool->return_stack.cond);
+                SCMutexUnlock(&pool->return_stack.mutex);
                 /* Clear the list of pending packets to return. */
                 my_pool->pending_pool = NULL;
                 my_pool->pending_head = NULL;

From 5b4ba0fe46f5e4ec79f9d6388e0458b25c31f8c4 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Fri, 8 Sep 2023 11:13:01 +0200
Subject: [PATCH 095/462] privs: hint the user of unset user/group name

Ticket: #6278
---
 src/util-privs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/util-privs.c b/src/util-privs.c
index 7e6f20bb1a7a..b08516b650e0 100644
--- a/src/util-privs.c
+++ b/src/util-privs.c
@@ -153,6 +153,11 @@ int SCGetUserID(const char *user_name, const char *group_name, uint32_t *uid, ui
     uint32_t groupid = 0;
     struct passwd *pw;
 
+    if (user_name == NULL || strlen(user_name) == 0) {
+        FatalError("no user name was provided - ensure it is specified either in the configuration "
+                   "file (run-as.user) or in command-line arguments (--user)");
+    }
+
     /* Get the user ID */
     if (isdigit((unsigned char)user_name[0]) != 0) {
         if (ByteExtractStringUint32(&userid, 10, 0, (const char *)user_name) < 0) {
@@ -216,6 +221,11 @@ int SCGetGroupID(const char *group_name, uint32_t *gid)
     uint32_t grpid = 0;
     struct group *gp;
 
+    if (group_name == NULL || strlen(group_name) == 0) {
+        FatalError("no group name was provided - ensure it is specified either in the "
+                   "configuration file (run-as.group) or in command-line arguments (--group)");
+    }
+
     /* Get the group ID */
     if (isdigit((unsigned char)group_name[0]) != 0) {
         if (ByteExtractStringUint32(&grpid, 10, 0, (const char *)group_name) < 0) {

From 5300cb625e6198d275e0cc86688944033b87ba65 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Fri, 8 Sep 2023 11:13:26 +0200
Subject: [PATCH 096/462] privs: refactor SCGetUser/GroupID to void functions

SCGetUserID/SCGetGroupID either FatalErrored out or
returned zero. As a result, the functions got refactored
into non-returning void functions.
---
 src/suricata.c   | 13 ++-----------
 src/util-privs.c | 12 ++++--------
 src/util-privs.h |  4 ++--
 3 files changed, 8 insertions(+), 21 deletions(-)

diff --git a/src/suricata.c b/src/suricata.c
index 30e6490826c7..7f979a7fbfcc 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -2155,20 +2155,11 @@ static int InitRunAs(SCInstance *suri)
     }
     /* Get the suricata user ID to given user ID */
     if (suri->do_setuid == TRUE) {
-        if (SCGetUserID(suri->user_name, suri->group_name,
-                        &suri->userid, &suri->groupid) != 0) {
-            SCLogError("failed in getting user ID");
-            return TM_ECODE_FAILED;
-        }
-
+        SCGetUserID(suri->user_name, suri->group_name, &suri->userid, &suri->groupid);
         sc_set_caps = TRUE;
     /* Get the suricata group ID to given group ID */
     } else if (suri->do_setgid == TRUE) {
-        if (SCGetGroupID(suri->group_name, &suri->groupid) != 0) {
-            SCLogError("failed in getting group ID");
-            return TM_ECODE_FAILED;
-        }
-
+        SCGetGroupID(suri->group_name, &suri->groupid);
         sc_set_caps = TRUE;
     }
 #endif
diff --git a/src/util-privs.c b/src/util-privs.c
index b08516b650e0..8210cc8065d9 100644
--- a/src/util-privs.c
+++ b/src/util-privs.c
@@ -145,9 +145,9 @@ void SCDropCaps(ThreadVars *tv)
  * \param   uid         pointer to the user id in which result will be stored
  * \param   gid         pointer to the group id in which result will be stored
  *
- * \retval  upon success it return 0
+ * \retval  FatalError on a failure
  */
-int SCGetUserID(const char *user_name, const char *group_name, uint32_t *uid, uint32_t *gid)
+void SCGetUserID(const char *user_name, const char *group_name, uint32_t *uid, uint32_t *gid)
 {
     uint32_t userid = 0;
     uint32_t groupid = 0;
@@ -204,8 +204,6 @@ int SCGetUserID(const char *user_name, const char *group_name, uint32_t *uid, ui
 
     *uid = userid;
     *gid = groupid;
-
-    return 0;
 }
 
 /**
@@ -214,9 +212,9 @@ int SCGetUserID(const char *user_name, const char *group_name, uint32_t *uid, ui
  * \param   group_name  pointer to the given group name
  * \param   gid         pointer to the group id in which result will be stored
  *
- * \retval  upon success it return 0
+ * \retval  FatalError on a failure
  */
-int SCGetGroupID(const char *group_name, uint32_t *gid)
+void SCGetGroupID(const char *group_name, uint32_t *gid)
 {
     uint32_t grpid = 0;
     struct group *gp;
@@ -244,8 +242,6 @@ int SCGetGroupID(const char *group_name, uint32_t *gid)
     endgrent();
 
     *gid = grpid;
-
-    return 0;
 }
 
 #ifdef __OpenBSD__
diff --git a/src/util-privs.h b/src/util-privs.h
index 64518814c5d2..454533963d34 100644
--- a/src/util-privs.h
+++ b/src/util-privs.h
@@ -90,8 +90,8 @@ void SCDropMainThreadCaps(uint32_t , uint32_t );
 #define SCDropMainThreadCaps(...)
 #endif /* HAVE_LIBCAP_NG */
 
-int SCGetUserID(const char *, const char *, uint32_t *, uint32_t *);
-int SCGetGroupID(const char *, uint32_t *);
+void SCGetUserID(const char *, const char *, uint32_t *, uint32_t *);
+void SCGetGroupID(const char *, uint32_t *);
 
 #ifdef __OpenBSD__
 int SCPledge(void);

From c8a7204b159553d338a6294218e696a72efdb4db Mon Sep 17 00:00:00 2001
From: Kirjan Kohuladas <kohuladaskirjan@gmail.com>
Date: Wed, 1 Nov 2023 15:29:57 -0400
Subject: [PATCH 097/462] doc/rule-profiling: fix suricatasc typo

---
 doc/userguide/rule-management/rule-profiling.rst | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/doc/userguide/rule-management/rule-profiling.rst b/doc/userguide/rule-management/rule-profiling.rst
index 0f33dfe26cd8..2c9926005c29 100644
--- a/doc/userguide/rule-management/rule-profiling.rst
+++ b/doc/userguide/rule-management/rule-profiling.rst
@@ -6,11 +6,11 @@ can be activated on demand from the unix socket and dumped from it.
 
 To start profiling ::
 
- surictasc -c ruleset-profile-start
+ suricatasc -c ruleset-profile-start
 
 To stop profiling ::
 
- surictasc -c ruleset-profile-stop
+ suricatasc -c ruleset-profile-stop
 
 To dump profiling ::
 
@@ -18,9 +18,9 @@ To dump profiling ::
 
 A typical scenario to get rules performance would be ::
 
- surictasc -c ruleset-profile-start
+ suricatasc -c ruleset-profile-start
  sleep 30
- surictasc -c ruleset-profile-stop
+ suricatasc -c ruleset-profile-stop
  suricatasc -c ruleset-profile
 
 On busy systems, using the sampling capability to capture performance

From ae72ce77fa9c57b8e462cca1130a7c01cd2d4144 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 6 Nov 2023 14:42:42 +0100
Subject: [PATCH 098/462] detect: parse units for integers

Ticket: #6423

Especially for filesize, instead of just a number, a signature
can use a number and a unit such as kb, mb or Gb
---
 rust/src/detect/uint.rs | 73 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 69 insertions(+), 4 deletions(-)

diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index 6cf31b2dbcf1..3d6a5baab0ca 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -16,7 +16,7 @@
  */
 
 use nom7::branch::alt;
-use nom7::bytes::complete::{is_a, tag, take_while};
+use nom7::bytes::complete::{is_a, tag, tag_no_case, take_while};
 use nom7::character::complete::digit1;
 use nom7::combinator::{all_consuming, map_opt, opt, value, verify};
 use nom7::error::{make_error, ErrorKind};
@@ -46,20 +46,54 @@ pub struct DetectUintData<T> {
 }
 
 pub trait DetectIntType:
-    std::str::FromStr + std::cmp::PartialOrd + num::PrimInt + num::Bounded
+    std::str::FromStr
+    + std::cmp::PartialOrd
+    + num::PrimInt
+    + num::Bounded
+    + num::ToPrimitive
+    + num::FromPrimitive
 {
 }
 impl<T> DetectIntType for T where
-    T: std::str::FromStr + std::cmp::PartialOrd + num::PrimInt + num::Bounded
+    T: std::str::FromStr
+        + std::cmp::PartialOrd
+        + num::PrimInt
+        + num::Bounded
+        + num::ToPrimitive
+        + num::FromPrimitive
 {
 }
 
+pub fn detect_parse_uint_unit(i: &str) -> IResult<&str, u64> {
+    let (i, unit) = alt((
+        value(1024, tag_no_case("kb")),
+        value(1024 * 1024, tag_no_case("mb")),
+        value(1024 * 1024 * 1024, tag_no_case("gb")),
+    ))(i)?;
+    return Ok((i, unit));
+}
+
+pub fn detect_parse_uint_with_unit<T: DetectIntType>(i: &str) -> IResult<&str, T> {
+    let (i, arg1) = map_opt(digit1, |s: &str| s.parse::<T>().ok())(i)?;
+    let (i, unit) = opt(detect_parse_uint_unit)(i)?;
+    if arg1 >= T::one() {
+        if let Some(u) = unit {
+            if T::max_value().to_u64().unwrap() / u < arg1.to_u64().unwrap() {
+                return Err(Err::Error(make_error(i, ErrorKind::Verify)));
+            }
+            let ru64 = arg1 * T::from_u64(u).unwrap();
+            return Ok((i, ru64));
+        }
+    }
+    Ok((i, arg1))
+}
+
 pub fn detect_parse_uint_start_equal<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
     let (i, _) = opt(tag("="))(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
-    let (i, arg1) = map_opt(digit1, |s: &str| s.parse::<T>().ok())(i)?;
+    let (i, arg1) = detect_parse_uint_with_unit(i)?;
     Ok((
         i,
         DetectUintData {
@@ -368,3 +402,34 @@ pub unsafe extern "C" fn rs_detect_u16_free(ctx: &mut DetectUintData<u16>) {
     // Just unbox...
     std::mem::drop(Box::from_raw(ctx));
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_parse_uint_unit() {
+        match detect_parse_uint::<u64>(" 2kb") {
+            Ok((_, val)) => {
+                assert_eq!(val.arg1, 2048);
+            }
+            Err(_) => {
+                assert!(false);
+            }
+        }
+        match detect_parse_uint::<u8>("2kb") {
+            Ok((_, _val)) => {
+                assert!(false);
+            }
+            Err(_) => {}
+        }
+        match detect_parse_uint::<u32>("3MB") {
+            Ok((_, val)) => {
+                assert_eq!(val.arg1, 3 * 1024 * 1024);
+            }
+            Err(_) => {
+                assert!(false);
+            }
+        }
+    }
+}

From 46a46e5b1f4909446e6f727f41b2eaeb3cb34bbd Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 6 Nov 2023 16:38:27 +0100
Subject: [PATCH 099/462] http2: event on mismatch between authority and host

Ticket: #6425
---
 rules/http2-events.rules |  1 +
 rust/src/http2/http2.rs  | 17 +++++++++++++++++
 2 files changed, 18 insertions(+)

diff --git a/rules/http2-events.rules b/rules/http2-events.rules
index c7a88b0c2b8f..868943a77bed 100644
--- a/rules/http2-events.rules
+++ b/rules/http2-events.rules
@@ -18,3 +18,4 @@ alert http2 any any -> any any (msg:"SURICATA HTTP2 failed decompression"; flow:
 alert http2 any any -> any any (msg:"SURICATA HTTP2 invalid range header"; flow:established; app-layer-event:http2.invalid_range; classtype:protocol-command-decode; sid:2290010; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 variable-length integer overflow"; flow:established; app-layer-event:http2.header_integer_overflow; classtype:protocol-command-decode; sid:2290011; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 too many streams"; flow:established; app-layer-event:http2.too_many_streams; classtype:protocol-command-decode; sid:2290012; rev:1;)
+alert http2 any any -> any any (msg:"SURICATA HTTP2 authority host mismatch"; flow:established,to_server; app-layer-event:http2.authority_host_mismatch; classtype:protocol-command-decode; sid:2290013; rev:1;)
diff --git a/rust/src/http2/http2.rs b/rust/src/http2/http2.rs
index 326030f9bbe3..bbaeddb40434 100644
--- a/rust/src/http2/http2.rs
+++ b/rust/src/http2/http2.rs
@@ -203,9 +203,25 @@ impl HTTP2Transaction {
     }
 
     fn handle_headers(&mut self, blocks: &[parser::HTTP2FrameHeaderBlock], dir: Direction) {
+        let mut authority = None;
+        let mut host = None;
         for block in blocks {
             if block.name == b"content-encoding" {
                 self.decoder.http2_encoding_fromvec(&block.value, dir);
+            } else if block.name.eq_ignore_ascii_case(b":authority") {
+                authority = Some(&block.value);
+            } else if block.name.eq_ignore_ascii_case(b"host") {
+                host = Some(&block.value);
+            }
+        }
+        if let Some(a) = authority {
+            if let Some(h) = host {
+                if !a.eq_ignore_ascii_case(h) {
+                    // The event is triggered only if both headers
+                    // are in the same frame to avoid excessive
+                    // complexity at runtime.
+                    self.set_event(HTTP2Event::AuthorityHostMismatch);
+                }
             }
         }
     }
@@ -383,6 +399,7 @@ pub enum HTTP2Event {
     InvalidRange,
     HeaderIntegerOverflow,
     TooManyStreams,
+    AuthorityHostMismatch,
 }
 
 pub struct HTTP2DynTable {

From b6cd66f41dff37bcd349bb968e6cbda4e3565a2d Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 8 Nov 2023 11:12:59 +0100
Subject: [PATCH 100/462] http2: update brotli crate

Fixes debug assertion found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63144
---
 rust/Cargo.lock.in | 191 ++++++++++++++++++++++++---------------------
 rust/Cargo.toml.in |   2 +-
 2 files changed, 104 insertions(+), 89 deletions(-)

diff --git a/rust/Cargo.lock.in b/rust/Cargo.lock.in
index 1800ba44f0ca..da0d6307e3b6 100644
--- a/rust/Cargo.lock.in
+++ b/rust/Cargo.lock.in
@@ -77,7 +77,7 @@ dependencies = [
  "asn1-rs-impl",
  "displaydoc",
  "nom",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
  "rusticata-macros",
  "thiserror",
  "time",
@@ -89,8 +89,8 @@ version = "0.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "726535892e8eae7e70657b4c8ea93d26b8553afb1ce617caee529ef96d7dee6c"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
  "synstructure",
 ]
@@ -101,8 +101,8 @@ version = "0.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2777730b2039ac0f95f093556e61b6d26cebed5393ca6f152717777cec3a42ed"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
 ]
 
@@ -144,9 +144,9 @@ dependencies = [
 
 [[package]]
 name = "brotli"
-version = "3.3.4"
+version = "3.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1a0b1dbcc8ae29329621f8d4f0d835787c1c38bb1401979b49d13b0b305ff68"
+checksum = "516074a47ef4bce09577a3b379392300159ce5b1ba2e501ff1c819950066100f"
 dependencies = [
  "alloc-no-stdlib",
  "alloc-stdlib",
@@ -155,9 +155,9 @@ dependencies = [
 
 [[package]]
 name = "brotli-decompressor"
-version = "2.3.5"
+version = "2.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "503a0bcf59056a66c55d8eefd05e9c0f00f9c9cdddbb6bd499623ce49100da43"
+checksum = "4e2e4afe60d7dd600fdd3de8d0f08c2b7ec039712e3b6137ff98b7004e82de4f"
 dependencies = [
  "alloc-no-stdlib",
  "alloc-stdlib",
@@ -192,9 +192,9 @@ dependencies = [
 
 [[package]]
 name = "cpufeatures"
-version = "0.2.9"
+version = "0.2.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a17b76ff3a4162b0b27f354a0c87015ddad39d35f9c0c36607a3bdd175dde1f1"
+checksum = "ce420fe07aecd3e67c5f910618fe65e94158f6dcc0adf44e00d69ce2bdfe0fd0"
 dependencies = [
  "libc",
 ]
@@ -248,8 +248,8 @@ version = "0.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c73af209b6a5dc8ca7cbaba720732304792cddc933cfea3d74509c2b1ef2f436"
 dependencies = [
- "num-bigint 0.4.3",
- "num-traits 0.2.16",
+ "num-bigint 0.4.4",
+ "num-traits 0.2.17",
  "syn 1.0.109",
 ]
 
@@ -261,7 +261,7 @@ checksum = "4cddf120f700b411b2b02ebeb7f04dc0b7c8835909a6c2f52bf72ed0dd3433b2"
 dependencies = [
  "der-oid-macro",
  "nom",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
  "rusticata-macros",
 ]
 
@@ -274,8 +274,8 @@ dependencies = [
  "asn1-rs",
  "displaydoc",
  "nom",
- "num-bigint 0.4.3",
- "num-traits 0.2.16",
+ "num-bigint 0.4.4",
+ "num-traits 0.2.17",
  "rusticata-macros",
 ]
 
@@ -296,9 +296,9 @@ version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "487585f4d0c6655fe74905e2504d8ad6908e4db67f744eb140876906c2f3175d"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
- "syn 2.0.28",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
+ "syn 2.0.39",
 ]
 
 [[package]]
@@ -325,17 +325,17 @@ version = "0.1.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "aa4da3c766cd7a0db8242e326e9e4e081edd567072893ed320008189715366a4"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
  "synstructure",
 ]
 
 [[package]]
 name = "flate2"
-version = "1.0.26"
+version = "1.0.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b9429470923de8e8cbd4d2dc513535400b4b3fef0319fb5c4e1f520a7bef743"
+checksum = "46303f565772937ffe1d394a4fac6f411c6013172fadde9dcdb1e147a086940e"
 dependencies = [
  "crc32fast",
  "miniz_oxide",
@@ -353,9 +353,9 @@ dependencies = [
 
 [[package]]
 name = "getrandom"
-version = "0.2.10"
+version = "0.2.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "be4136b2a15dd319360be1c07d9933517ccf0be8f16bf62a3bee4f0d618df427"
+checksum = "fe9006bed769170c11f845cf00c7c1e9092aeb3f268e007c3e760ac68008070f"
 dependencies = [
  "cfg-if",
  "libc",
@@ -431,9 +431,9 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
 
 [[package]]
 name = "libc"
-version = "0.2.147"
+version = "0.2.150"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b4668fb0ea861c1df094127ac5f1da3409a82116a4ba74fca2e58ef927159bb3"
+checksum = "89d92a4743f9a61002fae18374ed11e7973f530cb3a3255fb354818118b2203c"
 
 [[package]]
 name = "lzma-rs"
@@ -447,10 +447,11 @@ dependencies = [
 
 [[package]]
 name = "md-5"
-version = "0.10.5"
+version = "0.10.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6365506850d44bff6e2fbcb5176cf63650e48bd45ef2fe2665ae1570e0f4b9ca"
+checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
 dependencies = [
+ "cfg-if",
  "digest",
 ]
 
@@ -502,8 +503,8 @@ version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cd0b9a93a84b0d3ec3e70e02d332dc33ac6dfac9cde63e17fcb77172dededa62"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
 ]
 
@@ -528,7 +529,7 @@ dependencies = [
  "num-integer",
  "num-iter",
  "num-rational",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -539,18 +540,18 @@ checksum = "090c7f9998ee0ff65aa5b723e4009f7b217707f1fb5ea551329cc4d6231fb304"
 dependencies = [
  "autocfg",
  "num-integer",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
 name = "num-bigint"
-version = "0.4.3"
+version = "0.4.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f93ab6289c7b344a8a9f60f88d80aa20032336fe78da341afc91c8a2341fc75f"
+checksum = "608e7659b5c3d7cba262d894801b9ec9d00de989e8a82bd4bef91d08da45cdc0"
 dependencies = [
  "autocfg",
  "num-integer",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -560,7 +561,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b6b19411a9719e753aff12e5187b74d60d3dc449ec3f4dc21e3989c3f554bc95"
 dependencies = [
  "autocfg",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -581,7 +582,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "225d3389fb3509a24c93f5c29eb6bde2586b98d9f016636dff58d7c6f7569cd9"
 dependencies = [
  "autocfg",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -592,7 +593,7 @@ checksum = "7d03e6c028c5dc5cac6e2dec0efda81fc887605bb3d884578bb6d6bf7514e252"
 dependencies = [
  "autocfg",
  "num-integer",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -604,7 +605,7 @@ dependencies = [
  "autocfg",
  "num-bigint 0.2.6",
  "num-integer",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
@@ -613,14 +614,14 @@ version = "0.1.43"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "92e5113e9fd4cc14ded8e499429f396a20f98c772a47cc8622a736e1ec843c31"
 dependencies = [
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
 ]
 
 [[package]]
 name = "num-traits"
-version = "0.2.16"
+version = "0.2.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f30b0abd723be7e2ffca1272140fac1a2f084c77ec3e123c192b66af1ee9e6c2"
+checksum = "39e3200413f237f41ab11ad6d161bc7239c84dcb631773ccd7de3dfe4b5c267c"
 dependencies = [
  "autocfg",
 ]
@@ -641,8 +642,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dcbff9bc912032c62bf65ef1d5aea88983b420f4f839db1e9b0c281a25c9c799"
 dependencies = [
  "proc-macro-crate",
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
 ]
 
@@ -747,9 +748,9 @@ dependencies = [
 
 [[package]]
 name = "proc-macro2"
-version = "1.0.66"
+version = "1.0.69"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "18fb31db3f9bddb2ea821cde30a9f70117e3f119938b5ee630b7403aa6e2ead9"
+checksum = "134c189feb4956b20f6f547d2cf727d4c0fe06722b20a0eec87ed445a97f92da"
 dependencies = [
  "unicode-ident",
 ]
@@ -765,11 +766,11 @@ dependencies = [
 
 [[package]]
 name = "quote"
-version = "1.0.32"
+version = "1.0.33"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "50f3b39ccfb720540debaa0164757101c08ecb8d326b15358ce76a62c7e85965"
+checksum = "5267fca4496028628a95160fc423a33e8b2e6af8a5302579e322e4b520293cae"
 dependencies = [
- "proc-macro2 1.0.66",
+ "proc-macro2 1.0.69",
 ]
 
 [[package]]
@@ -859,8 +860,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "49a585d3c22887d23bb06dd602b8ce96c2a716e1fa89beec8bfb49e466f2d643"
 dependencies = [
  "proc-macro-crate",
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
 ]
 
@@ -878,15 +879,29 @@ dependencies = [
 
 [[package]]
 name = "serde"
-version = "1.0.179"
+version = "1.0.192"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bca2a08484b285dcb282d0f67b26cadc0df8b19f8c12502c13d966bf9482f001"
+dependencies = [
+ "serde_derive",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.192"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0a5bf42b8d227d4abf38a1ddb08602e229108a517cd4e5bb28f9c7eaafdce5c0"
+checksum = "d6c7207fbec9faa48073f3e3074cbe553af6ea512d7c21ba46e434e70ea9fbc1"
+dependencies = [
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
+ "syn 2.0.39",
+]
 
 [[package]]
 name = "sha1"
-version = "0.10.5"
+version = "0.10.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f04293dc80c3993519f2d7f6f511707ee7094fe0c6d3406feb330cdb3540eba3"
+checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
 dependencies = [
  "cfg-if",
  "cpufeatures",
@@ -895,9 +910,9 @@ dependencies = [
 
 [[package]]
 name = "sha2"
-version = "0.10.7"
+version = "0.10.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "479fb9d862239e610720565ca91403019f2f00410f1864c5aa7479b950a76ed8"
+checksum = "793db75ad2bcafc3ffa7c68b215fee268f537982cd901d132f89c6343f3a3dc8"
 dependencies = [
  "cfg-if",
  "cpufeatures",
@@ -906,9 +921,9 @@ dependencies = [
 
 [[package]]
 name = "siphasher"
-version = "0.3.10"
+version = "0.3.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bd3e3206899af3f8b12af284fafc038cc1dc2b41d1b89dd17297221c5d225de"
+checksum = "38b58827f4464d87d377d175e90bf58eb00fd8716ff0a62f80356b5e61555d0d"
 
 [[package]]
 name = "snmp-parser"
@@ -930,7 +945,7 @@ checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"
 
 [[package]]
 name = "suricata"
-version = "7.0.1-dev"
+version = "7.0.3-dev"
 dependencies = [
  "aes",
  "aes-gcm",
@@ -958,7 +973,7 @@ dependencies = [
  "ntp-parser",
  "num",
  "num-derive",
- "num-traits 0.2.16",
+ "num-traits 0.2.17",
  "regex",
  "sawp",
  "sawp-modbus",
@@ -976,11 +991,11 @@ dependencies = [
 
 [[package]]
 name = "suricata-derive"
-version = "7.0.1-dev"
+version = "7.0.3-dev"
 dependencies = [
  "proc-macro-crate",
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
 ]
 
@@ -1001,19 +1016,19 @@ version = "1.0.109"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "unicode-ident",
 ]
 
 [[package]]
 name = "syn"
-version = "2.0.28"
+version = "2.0.39"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04361975b3f5e348b2189d8dc55bc942f278b2d482a6a0365de5bdd62d351567"
+checksum = "23e78b90f2fcf45d3e842032ce32e3f2d1545ba6636271dcbf24fa306d87be7a"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "unicode-ident",
 ]
 
@@ -1023,8 +1038,8 @@ version = "0.12.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f36bdaa60a83aca3921b5259d5400cbf5e90fc51931376a9bd4a0eb79aa7210f"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
  "unicode-xid 0.2.4",
 ]
@@ -1036,30 +1051,30 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "956044ef122917dde830c19dec5f76d0670329fde4104836d62ebcb14f4865f1"
 dependencies = [
  "cfg-if",
- "proc-macro2 1.0.66",
- "quote 1.0.32",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
  "syn 1.0.109",
  "version_check",
 ]
 
 [[package]]
 name = "thiserror"
-version = "1.0.44"
+version = "1.0.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "611040a08a0439f8248d1990b111c95baa9c704c805fa1f62104b39655fd7f90"
+checksum = "f9a7210f5c9a7156bb50aa36aed4c95afb51df0df00713949448cf9e97d382d2"
 dependencies = [
  "thiserror-impl",
 ]
 
 [[package]]
 name = "thiserror-impl"
-version = "1.0.44"
+version = "1.0.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "090198534930841fab3a5d1bb637cde49e339654e606195f8d9c76eeb081dc96"
+checksum = "266b2e40bc00e5a6c09c3584011e08b06f123c00362c92b975ba9843aaaa14b8"
 dependencies = [
- "proc-macro2 1.0.66",
- "quote 1.0.32",
- "syn 2.0.28",
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
+ "syn 2.0.39",
 ]
 
 [[package]]
@@ -1105,15 +1120,15 @@ dependencies = [
 
 [[package]]
 name = "typenum"
-version = "1.16.0"
+version = "1.17.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "497961ef93d974e23eb6f433eb5fe1b7930b659f06d12dec6fc44a8f554c0bba"
+checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825"
 
 [[package]]
 name = "unicode-ident"
-version = "1.0.11"
+version = "1.0.12"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "301abaae475aa91687eb82514b328ab47a211a533026cb25fc3e519b86adfc3c"
+checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
 
 [[package]]
 name = "unicode-xid"
@@ -1163,9 +1178,9 @@ checksum = "c168940144dd21fd8046987c16a46a33d5fc84eec29ef9dcddc2ac9e31526b7c"
 
 [[package]]
 name = "x509-parser"
-version = "0.15.0"
+version = "0.15.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bab0c2f54ae1d92f4fcb99c0b7ccf0b1e3451cbd395e5f115ccbdbcb18d4f634"
+checksum = "7069fba5b66b9193bd2c5d3d4ff12b839118f6bcbef5328efafafb5395cf63da"
 dependencies = [
  "asn1-rs",
  "data-encoding",
diff --git a/rust/Cargo.toml.in b/rust/Cargo.toml.in
index 303994aac491..0bac7e159e0a 100644
--- a/rust/Cargo.toml.in
+++ b/rust/Cargo.toml.in
@@ -34,7 +34,7 @@ num-derive = "~0.2.5"
 num-traits = "~0.2.14"
 widestring = "~0.4.3"
 flate2 = "~1.0.19"
-brotli = "~3.3.0"
+brotli = "~3.4.0"
 hkdf = "~0.12.3"
 aes = "~0.7.5"
 aes-gcm = "~0.9.4"

From 6249722589f9a10a4771ef4f7a8cbabbd8935ccc Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 9 Nov 2023 14:19:59 +0100
Subject: [PATCH 101/462] http2: normalize host when there is user info

Ticket: 6479
---
 rust/src/http2/detect.rs | 62 +++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 30 deletions(-)

diff --git a/rust/src/http2/detect.rs b/rust/src/http2/detect.rs
index 99261adc4c41..1c595a0cb0f8 100644
--- a/rust/src/http2/detect.rs
+++ b/rust/src/http2/detect.rs
@@ -613,13 +613,28 @@ fn http2_lower(value: &[u8]) -> Option<Vec<u8>> {
 }
 
 // returns a tuple with the value and its size
-fn http2_normalize_host(value: &[u8]) -> (Option<Vec<u8>>, usize) {
-    match value.iter().position(|&x| x == b':') {
+fn http2_normalize_host(value: &[u8]) -> &[u8] {
+    match value.iter().position(|&x| x == b'@') {
         Some(i) => {
-            return (http2_lower(&value[..i]), i);
+            let value = &value[i+1..];
+            match value.iter().position(|&x| x == b':') {
+                Some(i) => {
+                    return &value[..i];
+                }
+                None => {
+                    return value;
+                }
+            }
         }
         None => {
-            return (http2_lower(value), value.len());
+            match value.iter().position(|&x| x == b':') {
+                Some(i) => {
+                    return &value[..i];
+                }
+                None => {
+                    return value;
+                }
+            }
         }
     }
 }
@@ -632,7 +647,7 @@ pub unsafe extern "C" fn rs_http2_tx_get_host_norm(
         let r = http2_normalize_host(value);
         // r is a tuple with the value and its size
         // this is useful when we only take a substring (before the port)
-        match r.0 {
+        match http2_lower(r) {
             Some(normval) => {
                 // In case we needed some normalization,
                 // the transaction needs to take ownership of this normalized host
@@ -640,12 +655,12 @@ pub unsafe extern "C" fn rs_http2_tx_get_host_norm(
                 let idx = tx.escaped.len() - 1;
                 let resvalue = &tx.escaped[idx];
                 *buffer = resvalue.as_ptr(); //unsafe
-                *buffer_len = r.1 as u32;
+                *buffer_len = resvalue.len() as u32;
                 return 1;
             }
             None => {
-                *buffer = value.as_ptr(); //unsafe
-                *buffer_len = r.1 as u32;
+                *buffer = r.as_ptr(); //unsafe
+                *buffer_len = r.len() as u32;
                 return 1;
             }
         }
@@ -1008,32 +1023,19 @@ mod tests {
     fn test_http2_normalize_host() {
         let buf0 = "aBC.com:1234".as_bytes();
         let r0 = http2_normalize_host(buf0);
-        match r0.0 {
-            Some(r) => {
-                assert_eq!(r, "abc.com".as_bytes().to_vec());
-            }
-            None => {
-                panic!("Result should not have been None");
-            }
-        }
+        assert_eq!(r0, "aBC.com".as_bytes().to_vec());
         let buf1 = "oisf.net".as_bytes();
         let r1 = http2_normalize_host(buf1);
-        match r1.0 {
-            Some(r) => {
-                panic!("Result should not have been None, not {:?}", r);
-            }
-            None => {}
-        }
-        assert_eq!(r1.1, "oisf.net".len());
+        assert_eq!(r1, "oisf.net".as_bytes().to_vec());
         let buf2 = "localhost:3000".as_bytes();
         let r2 = http2_normalize_host(buf2);
-        match r2.0 {
-            Some(r) => {
-                panic!("Result should not have been None, not {:?}", r);
-            }
-            None => {}
-        }
-        assert_eq!(r2.1, "localhost".len());
+        assert_eq!(r2, "localhost".as_bytes().to_vec());
+        let buf3 = "user:pass@localhost".as_bytes();
+        let r3 = http2_normalize_host(buf3);
+        assert_eq!(r3, "localhost".as_bytes().to_vec());
+        let buf4 = "user:pass@localhost:123".as_bytes();
+        let r4 = http2_normalize_host(buf4);
+        assert_eq!(r4, "localhost".as_bytes().to_vec());
     }
 
     #[test]

From f2b47bb0dc5e33b9ec0ed9e31e1f454c134c15e1 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 30 Oct 2023 17:05:50 -0600
Subject: [PATCH 102/462] eve: remove some dead code

---
 src/suricata-plugin.h  | 1 -
 src/util-logopenfile.h | 1 -
 2 files changed, 2 deletions(-)

diff --git a/src/suricata-plugin.h b/src/suricata-plugin.h
index dec87bb64e9e..3296b51397b8 100644
--- a/src/suricata-plugin.h
+++ b/src/suricata-plugin.h
@@ -61,7 +61,6 @@ typedef struct SCEveFileType_ {
     TAILQ_ENTRY(SCEveFileType_) entries;
 } SCEveFileType;
 
-bool SCPluginRegisterEveFileType(SCEveFileType *);
 bool SCRegisterEveFileType(SCEveFileType *);
 
 typedef struct SCCapturePlugin_ {
diff --git a/src/util-logopenfile.h b/src/util-logopenfile.h
index 9439c1036ea4..bbb5211cda77 100644
--- a/src/util-logopenfile.h
+++ b/src/util-logopenfile.h
@@ -72,7 +72,6 @@ typedef struct LogFilePluginCtx_ {
 typedef struct LogFileCtx_ {
     union {
         FILE *fp;
-        void *plugin_data;
 #ifdef HAVE_LIBHIREDIS
         void *redis;
 #endif

From 327c629253a07446294f903e88e57aa1a44be117 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 1 Nov 2023 16:57:39 -0600
Subject: [PATCH 103/462] outputs: call plugin ThreadDeinit, not Deinit

With the change to the hash table for tracking threaded loggers, this
call is now called once per thread, so should be changed to the
ThreadDeinit, as that is not longer being called.

Then call Deinit for the primary logger. In threaded mode this would be
the parent, its just the logger in non-threaded mode.

Bug: #6438
---
 src/util-logopenfile.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/util-logopenfile.c b/src/util-logopenfile.c
index edcd9900286e..feca63f44afd 100644
--- a/src/util-logopenfile.c
+++ b/src/util-logopenfile.c
@@ -865,8 +865,8 @@ int LogFileFreeCtx(LogFileCtx *lf_ctx)
         SCReturnInt(0);
     }
 
-    if (lf_ctx->type == LOGFILE_TYPE_PLUGIN) {
-        lf_ctx->plugin.plugin->Deinit(lf_ctx->plugin.init_data);
+    if (lf_ctx->type == LOGFILE_TYPE_PLUGIN && lf_ctx->parent != NULL) {
+        lf_ctx->plugin.plugin->ThreadDeinit(lf_ctx->plugin.init_data, lf_ctx->plugin.thread_data);
     }
 
     if (lf_ctx->threaded) {
@@ -902,6 +902,13 @@ int LogFileFreeCtx(LogFileCtx *lf_ctx)
         OutputUnregisterFileRotationFlag(&lf_ctx->rotation_flag);
     }
 
+    /* Deinitialize output plugins. We only want to call this for the
+     * parent of threaded output, or always for non-threaded
+     * output. */
+    if (lf_ctx->type == LOGFILE_TYPE_PLUGIN && lf_ctx->parent == NULL) {
+        lf_ctx->plugin.plugin->Deinit(lf_ctx->plugin.init_data);
+    }
+
     memset(lf_ctx, 0, sizeof(*lf_ctx));
     SCFree(lf_ctx);
 

From 741ba51c1e9d8d12dc8b3e5cee1f0805b8ef97af Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 16 Sep 2023 11:17:58 +0200
Subject: [PATCH 104/462] github-ci: Fedora 37 to 39; use packaged cbindgen

---
 .github/workflows/builds.yml | 36 +++++++++++++-----------------------
 1 file changed, 13 insertions(+), 23 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index d900b7b051bf..62748b10e985 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -1014,11 +1014,11 @@ jobs:
       - run: suricata-update -V
       - run: suricatasc -h
 
-  # Fedora 37 build using Clang.
-  fedora-37-clang:
-    name: Fedora 37 (clang, debug, asan, wshadow, rust-strict, systemd)
+  # Fedora 39 build using Clang.
+  fedora-39-clang:
+    name: Fedora 39 (clang, debug, asan, wshadow, rust-strict, systemd)
     runs-on: ubuntu-latest
-    container: fedora:37
+    container: fedora:39
     needs: [prepare-deps, prepare-cbindgen]
     steps:
 
@@ -1041,6 +1041,7 @@ jobs:
                 autoconf \
                 automake \
                 cargo \
+                cbindgen \
                 ccache \
                 clang \
                 diffutils \
@@ -1082,12 +1083,6 @@ jobs:
           path: prep
       - run: tar xf prep/libhtp.tar.gz
       - run: tar xf prep/suricata-update.tar.gz
-      - name: Setup cbindgen
-        run: |
-          mkdir -p $HOME/.cargo/bin
-          cp prep/cbindgen $HOME/.cargo/bin
-          chmod 755 $HOME/.cargo/bin/cbindgen
-          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: CC="clang" CFLAGS="$DEFAULT_CFLAGS -Wshadow -fsanitize=address -fno-omit-frame-pointer" ./configure --enable-debug --enable-unittests --disable-shared --enable-rust-strict --enable-hiredis --enable-nfqueue --enable-lua
         env:
@@ -1118,11 +1113,11 @@ jobs:
       # Check compilation against systemd
       - run: ldd src/suricata | grep libsystemd &> /dev/null
 
-  # Fedora 37 build using GCC.
-  fedora-37-gcc:
-    name: Fedora 37 (gcc, debug, asan, wshadow, rust-strict)
+  # Fedora 39 build using GCC.
+  fedora-39-gcc:
+    name: Fedora 39 (gcc, debug, asan, wshadow, rust-strict)
     runs-on: ubuntu-latest
-    container: fedora:37
+    container: fedora:39
     needs: [prepare-deps, prepare-cbindgen]
     steps:
 
@@ -1138,6 +1133,7 @@ jobs:
                 autoconf \
                 automake \
                 cargo \
+                cbindgen \
                 ccache \
                 diffutils \
                 file-devel \
@@ -1175,12 +1171,6 @@ jobs:
           path: prep
       - run: tar xf prep/libhtp.tar.gz
       - run: tar xf prep/suricata-update.tar.gz
-      - name: Setup cbindgen
-        run: |
-          mkdir -p $HOME/.cargo/bin
-          cp prep/cbindgen $HOME/.cargo/bin
-          chmod 755 $HOME/.cargo/bin/cbindgen
-          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: ./configure --enable-debug --enable-unittests --disable-shared --enable-rust-strict --enable-hiredis --enable-nfqueue
         env:
@@ -1213,10 +1203,10 @@ jobs:
   # This job builds and tests Suricata as a non-root user as some
   # issues only show up when not running as root, and by default all
   # jobs in GitHub actions are run as root inside the container.
-  fedora-37-non-root:
-    name: Fedora 37 (non-root, debug, clang, asan, wshadow, rust-strict, systemd)
+  fedora-39-non-root:
+    name: Fedora 39 (non-root, debug, clang, asan, wshadow, rust-strict, systemd)
     runs-on: ubuntu-latest
-    container: fedora:37
+    container: fedora:39
     needs: [prepare-deps]
     steps:
       - run: |

From de14e3d0b5d262a27238d08efc02f898c9a600cb Mon Sep 17 00:00:00 2001
From: daniel zhao <danielzhao2029@gmail.com>
Date: Tue, 14 Nov 2023 19:04:10 +0800
Subject: [PATCH 105/462] detect/flow: fix DETECT_FLOW_FLAG_ESTABLISHED check

Ticket: #6448
---
 src/detect-flow.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/src/detect-flow.c b/src/detect-flow.c
index 09787515722c..fdee0779e458 100644
--- a/src/detect-flow.c
+++ b/src/detect-flow.c
@@ -227,6 +227,10 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                 if (fd->flags & DETECT_FLOW_FLAG_ESTABLISHED) {
                     SCLogError("DETECT_FLOW_FLAG_ESTABLISHED flag is already set");
                     goto error;
+                } else if (fd->flags & DETECT_FLOW_FLAG_NOT_ESTABLISHED) {
+                    SCLogError("cannot set DETECT_FLOW_FLAG_ESTABLISHED, "
+                               "DETECT_FLOW_FLAG_NOT_ESTABLISHED already set");
+                    goto error;
                 } else if (fd->flags & DETECT_FLOW_FLAG_STATELESS) {
                     SCLogError("DETECT_FLOW_FLAG_STATELESS already set");
                     goto error;
@@ -236,7 +240,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                 if (fd->flags & DETECT_FLOW_FLAG_NOT_ESTABLISHED) {
                     SCLogError("DETECT_FLOW_FLAG_NOT_ESTABLISHED flag is already set");
                     goto error;
-                } else if (fd->flags & DETECT_FLOW_FLAG_NOT_ESTABLISHED) {
+                } else if (fd->flags & DETECT_FLOW_FLAG_ESTABLISHED) {
                     SCLogError("cannot set DETECT_FLOW_FLAG_NOT_ESTABLISHED, "
                                "DETECT_FLOW_FLAG_ESTABLISHED already set");
                     goto error;
@@ -946,6 +950,19 @@ static int DetectFlowTestParse21 (void)
     PASS;
 }
 
+/**
+ * \test DetectFlowTestParse22 is a test for setting the established,not_established flow opts both
+ */
+static int DetectFlowTestParse22(void)
+{
+    DetectFlowData *fd = NULL;
+    fd = DetectFlowParse(NULL, "established,not_established");
+    FAIL_IF_NOT_NULL(fd);
+    fd = DetectFlowParse(NULL, "not_established,established");
+    FAIL_IF_NOT_NULL(fd);
+    PASS;
+}
+
 static int DetectFlowSigTest01(void)
 {
     uint8_t *buf = (uint8_t *)"supernovaduper";
@@ -1104,6 +1121,7 @@ static void DetectFlowRegisterTests(void)
     UtRegisterTest("DetectFlowTestParse20", DetectFlowTestParse20);
     UtRegisterTest("DetectFlowTestParseNocase20", DetectFlowTestParseNocase20);
     UtRegisterTest("DetectFlowTestParse21", DetectFlowTestParse21);
+    UtRegisterTest("DetectFlowTestParse22", DetectFlowTestParse22);
     UtRegisterTest("DetectFlowTestParseNotEstablished",
         DetectFlowTestParseNotEstablished);
     UtRegisterTest("DetectFlowTestParseNoFrag", DetectFlowTestParseNoFrag);

From 6bb882c4c0157cb3edee561860baa4854a510619 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 8 Sep 2023 10:01:41 +0200
Subject: [PATCH 106/462] macset: remove dead flow init/cleanup code

FlowInit() will only be called on a newly allocated, or a fully cleaned
up flow, so no existing storage will exist.

The only caller of `FLOW_RECYCLE` first calls `FlowFreeStorage()`, so
the reset logic in `FLOW_RECYCLE` can never trigger.

Remove now unused MacSetReset logic.
---
 src/flow-util.c   | 10 +++-------
 src/flow-util.h   |  6 ------
 src/util-macset.c | 18 ------------------
 3 files changed, 3 insertions(+), 31 deletions(-)

diff --git a/src/flow-util.c b/src/flow-util.c
index 3572c0823f4e..dc6a7103a6bd 100644
--- a/src/flow-util.c
+++ b/src/flow-util.c
@@ -200,13 +200,9 @@ void FlowInit(Flow *f, const Packet *p)
     f->timeout_at = timeout_at;
 
     if (MacSetFlowStorageEnabled()) {
-        MacSet *ms = FlowGetStorageById(f, MacSetGetFlowStorageID());
-        if (ms != NULL) {
-            MacSetReset(ms);
-        } else {
-            ms = MacSetInit(10);
-            FlowSetStorageById(f, MacSetGetFlowStorageID(), ms);
-        }
+        DEBUG_VALIDATE_BUG_ON(FlowGetStorageById(f, MacSetGetFlowStorageID()) != NULL);
+        MacSet *ms = MacSetInit(10);
+        FlowSetStorageById(f, MacSetGetFlowStorageID(), ms);
     }
 
     SCReturn;
diff --git a/src/flow-util.h b/src/flow-util.h
index 4bdb9e2d3e12..3d0d978b5a76 100644
--- a/src/flow-util.h
+++ b/src/flow-util.h
@@ -115,12 +115,6 @@
         (f)->sgh_toclient = NULL;                                                                  \
         GenericVarFree((f)->flowvar);                                                              \
         (f)->flowvar = NULL;                                                                       \
-        if (MacSetFlowStorageEnabled()) {                                                          \
-            MacSet *ms = FlowGetStorageById((f), MacSetGetFlowStorageID());                        \
-            if (ms != NULL) {                                                                      \
-                MacSetReset(ms);                                                                   \
-            }                                                                                      \
-        }                                                                                          \
         RESET_COUNTERS((f));                                                                       \
     } while (0)
 
diff --git a/src/util-macset.c b/src/util-macset.c
index 3f540a23a1d3..9853a3241680 100644
--- a/src/util-macset.c
+++ b/src/util-macset.c
@@ -259,14 +259,6 @@ int MacSetSize(const MacSet *ms)
     return size;
 }
 
-void MacSetReset(MacSet *ms)
-{
-    if (ms == NULL)
-        return;
-    ms->state[MAC_SET_SRC] = ms->state[MAC_SET_DST] = EMPTY_SET;
-    ms->last[MAC_SET_SRC] = ms->last[MAC_SET_DST] = 0;
-}
-
 void MacSetFree(MacSet *ms)
 {
     size_t total_free = 0;
@@ -334,16 +326,6 @@ static int MacSetTest01(void)
     ret = MacSetForEach(ms, CheckTest1Membership, &i);
     FAIL_IF_NOT(ret == 0);
 
-    MacSetReset(ms);
-    FAIL_IF_NOT(MacSetSize(ms) == 0);
-
-    MacSetAdd(ms, addr2, addr3);
-    FAIL_IF_NOT(MacSetSize(ms) == 2);
-
-    i = 1;
-    ret = MacSetForEach(ms, CheckTest1Membership, &i);
-    FAIL_IF_NOT(ret == 0);
-
     MacSetFree(ms);
     PASS;
 }

From 2f4027c1178b7fe4966ef5f5e951c4f084de28df Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 16 Nov 2023 09:13:50 +0100
Subject: [PATCH 107/462] version: start work on 8.0.0

---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index b377a1da9f2d..1908c76ab76a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-    AC_INIT([suricata],[7.0.3-dev])
+    AC_INIT([suricata],[8.0.0-dev])
     m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])AM_SILENT_RULES([yes])
     AC_CONFIG_HEADERS([src/autoconf.h])
     AC_CONFIG_SRCDIR([src/suricata.c])

From 8c5310aefda9e919a0d76efad2295d1916a17780 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 26 May 2023 08:58:43 +0200
Subject: [PATCH 108/462] doc: quic in eve/schema

Ticket: #6076
---
 etc/schema.json | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index 28182cb95d98..c194017ddf6f 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -2971,15 +2971,18 @@
             "optional": true,
             "properties": {
                 "cyu": {
+                    "description": "ja3-like fingerprint for versions of QUIC before standardization",
                     "type": "array",
                     "minItems": 1,
                     "items": {
                         "type": "object",
                         "properties": {
                             "hash": {
+                                "description": "cyu hash hex representation",
                                 "type": "string"
                             },
                             "string": {
+                                "description": "cyu hash string representation",
                                 "type": "string"
                             }
                         },
@@ -2987,18 +2990,22 @@
                     }
                 },
                 "extensions": {
+                    "description": "list of extensions in hello",
                     "type": "array",
                     "minItems": 1,
                     "items": {
                         "type": "object",
                         "properties": {
                             "name": {
+                                "description": "human-friendly name of the extension",
                                 "type": "string"
                             },
                             "type": {
+                                "description": "integer identifier of the extension",
                                 "type": "integer"
                             },
                             "values": {
+                                "description": "extension values",
                                 "type": "array",
                                 "minItems": 1,
                                 "items": {
@@ -3010,38 +3017,47 @@
                     }
                 },
                 "ja3": {
+                    "description": "ja3 from client, as in TLS",
                     "type": "object",
                     "optional": true,
                     "properties": {
                         "hash": {
+                            "description": "ja3 hex representation",
                             "type": "string"
                         },
                         "string": {
+                            "description": "ja3 string representation",
                             "type": "string"
                         }
                     },
                     "additionalProperties": false
                 },
                 "ja3s": {
+                    "description": "ja3 from server, as in TLS",
                     "type": "object",
                     "optional": true,
                     "properties": {
                         "hash": {
+                            "description": "ja3s hex representation",
                             "type": "string"
                         },
                         "string": {
+                            "description": "ja3s string representation",
                             "type": "string"
                         }
                     },
                     "additionalProperties": false
                 },
                 "sni": {
+                    "description": "Server Name Indication",
                     "type": "string"
                 },
                 "ua": {
+                    "description": "User Agent for versions of QUIC before standardization",
                     "type": "string"
                 },
                 "version": {
+                    "description": "Quic protocol version",
                     "type": "string"
                 }
             },

From ab9b6e30b128d0c498df6cef0d30ca4ad412ed92 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 20 Jun 2023 16:20:34 +0200
Subject: [PATCH 109/462] detect: adds flow integer keywords

Ticket: #6164

flow.pkts_toclient
flow.pkts_toserver
flow.bytes_toclient
flow.bytes_toserver
---
 doc/userguide/rules/flow-keywords.rst |  83 +++++++-
 src/Makefile.am                       |   2 +
 src/detect-engine-register.c          |   5 +
 src/detect-engine-register.h          |   4 +
 src/detect-flow-pkts.c                | 278 ++++++++++++++++++++++++++
 src/detect-flow-pkts.h                |  26 +++
 6 files changed, 397 insertions(+), 1 deletion(-)
 create mode 100644 src/detect-flow-pkts.c
 create mode 100644 src/detect-flow-pkts.h

diff --git a/doc/userguide/rules/flow-keywords.rst b/doc/userguide/rules/flow-keywords.rst
index bb0269299a19..6d451ce82aab 100644
--- a/doc/userguide/rules/flow-keywords.rst
+++ b/doc/userguide/rules/flow-keywords.rst
@@ -290,6 +290,7 @@ flow.age
 --------
 
 Flow age in seconds (integer)
+This keyword does not wait for the end of the flow, but will be checked at each packet.
 
 Syntax::
 
@@ -305,4 +306,84 @@ Signature example::
 
  alert tcp any any -> any any (msg:"Flow longer than one hour"; flow.age:>3600; flowbits: isnotset, onehourflow; flowbits: onehourflow, name; sid:1; rev:1;)
 
-In this example, we combine `flow.age` and `flowbits` to get an alert on the first packet after the flow's age is older than one hour.
\ No newline at end of file
+In this example, we combine `flow.age` and `flowbits` to get an alert on the first packet after the flow's age is older than one hour.
+
+flow.pkts_toclient
+------------------
+
+Flow number of packets to client (integer)
+This keyword does not wait for the end of the flow, but will be checked at each packet.
+
+Syntax::
+
+ flow.pkts_toclient: [op]<number>
+
+The number of packets can be matched exactly, or compared using the _op_ setting::
+
+ flow.pkts_toclient:3    # exactly 3
+ flow.pkts_toclient:<3   # smaller than 3
+ flow.pkts_toclient:>=2  # greater than or equal to 2
+
+Signature example::
+
+ alert ip any any -> any any (msg:"Flow has 20 packets"; flow.pkts_toclient:20; sid:1;)
+
+flow.pkts_toserver
+------------------
+
+Flow number of packets to server (integer)
+This keyword does not wait for the end of the flow, but will be checked at each packet.
+
+Syntax::
+
+ flow.pkts_toserver: [op]<number>
+
+The number of packets can be matched exactly, or compared using the _op_ setting::
+
+ flow.pkts_toserver:3    # exactly 3
+ flow.pkts_toserver:<3   # smaller than 3
+ flow.pkts_toserver:>=2  # greater than or equal to 2
+
+Signature example::
+
+ alert ip any any -> any any (msg:"Flow has 20 packets"; flow.pkts_toserver:20; sid:1;)
+
+flow.bytes_toclient
+-------------------
+
+Flow number of bytes to client (integer)
+This keyword does not wait for the end of the flow, but will be checked at each packet.
+
+Syntax::
+
+ flow.bytes_toclient: [op]<number>
+
+The number of packets can be matched exactly, or compared using the _op_ setting::
+
+ flow.bytes_toclient:3    # exactly 3
+ flow.bytes_toclient:<3   # smaller than 3
+ flow.bytes_toclient:>=2  # greater than or equal to 2
+
+Signature example::
+
+ alert ip any any -> any any (msg:"Flow has less than 2000 bytes"; flow.bytes_toclient:<2000; sid:1;)
+
+flow.bytes_toserver
+-------------------
+
+Flow number of bytes to server (integer)
+This keyword does not wait for the end of the flow, but will be checked at each packet.
+
+Syntax::
+
+ flow.bytes_toserver: [op]<number>
+
+The number of packets can be matched exactly, or compared using the _op_ setting::
+
+ flow.bytes_toserver:3    # exactly 3
+ flow.bytes_toserver:<3   # smaller than 3
+ flow.bytes_toserver:>=2  # greater than or equal to 2
+
+Signature example::
+
+ alert ip any any -> any any (msg:"Flow has less than 2000 bytes"; flow.bytes_toserver:<2000; sid:1;)
\ No newline at end of file
diff --git a/src/Makefile.am b/src/Makefile.am
index 48a5ce850ce2..c5b2fe52e894 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -166,6 +166,7 @@ noinst_HEADERS = \
 	detect-flowbits.h \
 	detect-flow.h \
 	detect-flow-age.h \
+	detect-flow-pkts.h \
 	detect-flowint.h \
 	detect-flowvar.h \
 	detect-fragbits.h \
@@ -777,6 +778,7 @@ libsuricata_c_a_SOURCES = \
 	detect-flowbits.c \
 	detect-flow.c \
 	detect-flow-age.c \
+	detect-flow-pkts.c \
 	detect-flowint.c \
 	detect-flowvar.c \
 	detect-fragbits.c \
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index df6e4a738ffc..d570510949f4 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -114,6 +114,7 @@
 #include "detect-rev.h"
 #include "detect-flow.h"
 #include "detect-flow-age.h"
+#include "detect-flow-pkts.h"
 #include "detect-tcp-window.h"
 #include "detect-ftpbounce.h"
 #include "detect-isdataat.h"
@@ -561,6 +562,10 @@ void SigTableSetup(void)
     DetectReplaceRegister();
     DetectFlowRegister();
     DetectFlowAgeRegister();
+    DetectFlowPktsToClientRegister();
+    DetectFlowPktsToServerRegister();
+    DetectFlowBytesToClientRegister();
+    DetectFlowBytesToServerRegister();
     DetectWindowRegister();
     DetectRpcRegister();
     DetectFtpbounceRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 7d6c457ef9b0..24a0e56cce1b 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -110,6 +110,10 @@ enum DetectKeywordId {
     DETECT_FRAME,
 
     DETECT_FLOW_AGE,
+    DETECT_FLOW_PKTS_TO_CLIENT,
+    DETECT_FLOW_PKTS_TO_SERVER,
+    DETECT_FLOW_BYTES_TO_CLIENT,
+    DETECT_FLOW_BYTES_TO_SERVER,
 
     DETECT_AL_TLS_VERSION,
     DETECT_AL_TLS_SUBJECT,
diff --git a/src/detect-flow-pkts.c b/src/detect-flow-pkts.c
new file mode 100644
index 000000000000..7066b0b3bbc4
--- /dev/null
+++ b/src/detect-flow-pkts.c
@@ -0,0 +1,278 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#include "suricata-common.h"
+#include "rust.h"
+#include "detect-flow-pkts.h"
+#include "detect-engine.h"
+#include "detect-engine-prefilter.h"
+#include "detect-engine-uint.h"
+#include "detect-parse.h"
+
+static int DetectFlowPktsToClientMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const Signature *s, const SigMatchCtx *ctx)
+{
+    if (p->flow == NULL) {
+        return 0;
+    }
+    uint32_t nb = p->flow->tosrcpktcnt;
+
+    const DetectU32Data *du32 = (const DetectU32Data *)ctx;
+    return DetectU32Match(nb, du32);
+}
+
+static void DetectFlowPktsToClientFree(DetectEngineCtx *de_ctx, void *ptr)
+{
+    rs_detect_u32_free(ptr);
+}
+
+static int DetectFlowPktsToClientSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
+{
+    DetectU32Data *du32 = DetectU32Parse(rawstr);
+    if (du32 == NULL)
+        return -1;
+
+    SigMatch *sm = SigMatchAlloc();
+    if (sm == NULL) {
+        DetectFlowPktsToClientFree(de_ctx, du32);
+        return -1;
+    }
+
+    sm->type = DETECT_FLOW_PKTS_TO_CLIENT;
+    sm->ctx = (SigMatchCtx *)du32;
+
+    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    s->flags |= SIG_FLAG_REQUIRE_PACKET;
+
+    return 0;
+}
+
+static void PrefilterPacketFlowPktsToClientMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const void *pectx)
+{
+    const PrefilterPacketHeaderCtx *ctx = pectx;
+    if (!PrefilterPacketHeaderExtraMatch(ctx, p))
+        return;
+
+    DetectU32Data du32;
+    du32.mode = ctx->v1.u8[0];
+    du32.arg1 = ctx->v1.u32[1];
+    du32.arg2 = ctx->v1.u32[2];
+    if (DetectFlowPktsToClientMatch(det_ctx, p, NULL, (const SigMatchCtx *)&du32)) {
+        PrefilterAddSids(&det_ctx->pmq, ctx->sigs_array, ctx->sigs_cnt);
+    }
+}
+
+static int PrefilterSetupFlowPktsToClient(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
+{
+    return PrefilterSetupPacketHeader(de_ctx, sgh, DETECT_FLOW_PKTS_TO_CLIENT,
+            PrefilterPacketU32Set, PrefilterPacketU32Compare, PrefilterPacketFlowPktsToClientMatch);
+}
+
+static bool PrefilterFlowPktsToClientIsPrefilterable(const Signature *s)
+{
+    return PrefilterIsPrefilterableById(s, DETECT_FLOW_PKTS_TO_CLIENT);
+}
+
+void DetectFlowPktsToClientRegister(void)
+{
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].name = "flow.pkts_toclient";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].desc = "match flow number of packets to client";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].url = "/rules/flow-keywords.html#flow-pkts_toclient";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].Match = DetectFlowPktsToClientMatch;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].Setup = DetectFlowPktsToClientSetup;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].Free = DetectFlowPktsToClientFree;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].SupportsPrefilter =
+            PrefilterFlowPktsToClientIsPrefilterable;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_CLIENT].SetupPrefilter = PrefilterSetupFlowPktsToClient;
+}
+
+static int DetectFlowPktsToServerMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const Signature *s, const SigMatchCtx *ctx)
+{
+    if (p->flow == NULL) {
+        return 0;
+    }
+    uint32_t nb = p->flow->todstpktcnt;
+
+    const DetectU32Data *du32 = (const DetectU32Data *)ctx;
+    return DetectU32Match(nb, du32);
+}
+
+static void DetectFlowPktsToServerFree(DetectEngineCtx *de_ctx, void *ptr)
+{
+    rs_detect_u32_free(ptr);
+}
+
+static int DetectFlowPktsToServerSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
+{
+    DetectU32Data *du32 = DetectU32Parse(rawstr);
+    if (du32 == NULL)
+        return -1;
+
+    SigMatch *sm = SigMatchAlloc();
+    if (sm == NULL) {
+        DetectFlowPktsToServerFree(de_ctx, du32);
+        return -1;
+    }
+
+    sm->type = DETECT_FLOW_PKTS_TO_SERVER;
+    sm->ctx = (SigMatchCtx *)du32;
+
+    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    s->flags |= SIG_FLAG_REQUIRE_PACKET;
+
+    return 0;
+}
+
+static void PrefilterPacketFlowPktsToServerMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const void *pectx)
+{
+    const PrefilterPacketHeaderCtx *ctx = pectx;
+    if (!PrefilterPacketHeaderExtraMatch(ctx, p))
+        return;
+
+    DetectU32Data du32;
+    du32.mode = ctx->v1.u8[0];
+    du32.arg1 = ctx->v1.u32[1];
+    du32.arg2 = ctx->v1.u32[2];
+    if (DetectFlowPktsToServerMatch(det_ctx, p, NULL, (const SigMatchCtx *)&du32)) {
+        PrefilterAddSids(&det_ctx->pmq, ctx->sigs_array, ctx->sigs_cnt);
+    }
+}
+
+static int PrefilterSetupFlowPktsToServer(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
+{
+    return PrefilterSetupPacketHeader(de_ctx, sgh, DETECT_FLOW_PKTS_TO_SERVER,
+            PrefilterPacketU32Set, PrefilterPacketU32Compare, PrefilterPacketFlowPktsToServerMatch);
+}
+
+static bool PrefilterFlowPktsToServerIsPrefilterable(const Signature *s)
+{
+    return PrefilterIsPrefilterableById(s, DETECT_FLOW_PKTS_TO_SERVER);
+}
+
+void DetectFlowPktsToServerRegister(void)
+{
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].name = "flow.pkts_toserver";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].desc = "match flow number of packets to server";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].url = "/rules/flow-keywords.html#flow-pkts_toserver";
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].Match = DetectFlowPktsToServerMatch;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].Setup = DetectFlowPktsToServerSetup;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].Free = DetectFlowPktsToServerFree;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].SupportsPrefilter =
+            PrefilterFlowPktsToServerIsPrefilterable;
+    sigmatch_table[DETECT_FLOW_PKTS_TO_SERVER].SetupPrefilter = PrefilterSetupFlowPktsToServer;
+}
+
+static int DetectFlowBytesToClientMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const Signature *s, const SigMatchCtx *ctx)
+{
+    if (p->flow == NULL) {
+        return 0;
+    }
+    uint64_t nb = p->flow->tosrcbytecnt;
+
+    const DetectU64Data *du64 = (const DetectU64Data *)ctx;
+    return DetectU64Match(nb, du64);
+}
+
+static void DetectFlowBytesToClientFree(DetectEngineCtx *de_ctx, void *ptr)
+{
+    rs_detect_u64_free(ptr);
+}
+
+static int DetectFlowBytesToClientSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
+{
+    DetectU64Data *du64 = DetectU64Parse(rawstr);
+    if (du64 == NULL)
+        return -1;
+
+    SigMatch *sm = SigMatchAlloc();
+    if (sm == NULL) {
+        DetectFlowBytesToClientFree(de_ctx, du64);
+        return -1;
+    }
+
+    sm->type = DETECT_FLOW_BYTES_TO_CLIENT;
+    sm->ctx = (SigMatchCtx *)du64;
+
+    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    s->flags |= SIG_FLAG_REQUIRE_PACKET;
+
+    return 0;
+}
+
+void DetectFlowBytesToClientRegister(void)
+{
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].name = "flow.bytes_toclient";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].desc = "match flow number of bytes to client";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].url =
+            "/rules/flow-keywords.html#flow-bytes_toclient";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].Match = DetectFlowBytesToClientMatch;
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].Setup = DetectFlowBytesToClientSetup;
+    sigmatch_table[DETECT_FLOW_BYTES_TO_CLIENT].Free = DetectFlowBytesToClientFree;
+}
+
+static int DetectFlowBytesToServerMatch(
+        DetectEngineThreadCtx *det_ctx, Packet *p, const Signature *s, const SigMatchCtx *ctx)
+{
+    if (p->flow == NULL) {
+        return 0;
+    }
+    uint64_t nb = p->flow->todstbytecnt;
+
+    const DetectU64Data *du64 = (const DetectU64Data *)ctx;
+    return DetectU64Match(nb, du64);
+}
+
+static void DetectFlowBytesToServerFree(DetectEngineCtx *de_ctx, void *ptr)
+{
+    rs_detect_u64_free(ptr);
+}
+
+static int DetectFlowBytesToServerSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
+{
+    DetectU64Data *du64 = DetectU64Parse(rawstr);
+    if (du64 == NULL)
+        return -1;
+
+    SigMatch *sm = SigMatchAlloc();
+    if (sm == NULL) {
+        DetectFlowBytesToServerFree(de_ctx, du64);
+        return -1;
+    }
+
+    sm->type = DETECT_FLOW_BYTES_TO_SERVER;
+    sm->ctx = (SigMatchCtx *)du64;
+
+    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    s->flags |= SIG_FLAG_REQUIRE_PACKET;
+
+    return 0;
+}
+
+void DetectFlowBytesToServerRegister(void)
+{
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].name = "flow.bytes_toserver";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].desc = "match flow number of bytes to server";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].url =
+            "/rules/flow-keywords.html#flow-bytes_toserver";
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].Match = DetectFlowBytesToServerMatch;
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].Setup = DetectFlowBytesToServerSetup;
+    sigmatch_table[DETECT_FLOW_BYTES_TO_SERVER].Free = DetectFlowBytesToServerFree;
+}
diff --git a/src/detect-flow-pkts.h b/src/detect-flow-pkts.h
new file mode 100644
index 000000000000..ddc5cd172b2b
--- /dev/null
+++ b/src/detect-flow-pkts.h
@@ -0,0 +1,26 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#ifndef __DETECT_FLOW_PKTS_H__
+#define __DETECT_FLOW_PKTS_H__
+
+void DetectFlowPktsToClientRegister(void);
+void DetectFlowPktsToServerRegister(void);
+void DetectFlowBytesToClientRegister(void);
+void DetectFlowBytesToServerRegister(void);
+
+#endif /* __DETECT_FLOW_PKTS_H__ */

From 7b0a5dae6049cc49fee8f04a245a309aed7eaff3 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 28 Sep 2023 19:19:10 +0200
Subject: [PATCH 110/462] napatech: remove deprecated hba support

---
 src/runmode-napatech.c | 12 ------------
 src/source-napatech.c  | 37 +------------------------------------
 src/source-napatech.h  |  1 -
 3 files changed, 1 insertion(+), 49 deletions(-)

diff --git a/src/runmode-napatech.c b/src/runmode-napatech.c
index cb8f560ea350..3d503b965573 100644
--- a/src/runmode-napatech.c
+++ b/src/runmode-napatech.c
@@ -194,14 +194,6 @@ static void *NapatechConfigParser(const char *device)
         return NULL;
     }
 
-    /* Set the host buffer allowance for this stream
-     * Right now we just look at the global default - there is no per-stream hba configuration
-     */
-    if (ConfGetInt("napatech.hba", &conf->hba) == 0) {
-        conf->hba = -1;
-    } else {
-        SCLogWarning("Napatech Host Buffer Allocation (hba) will be deprecated in Suricata v7.0.");
-    }
     return (void *) conf;
 }
 
@@ -235,10 +227,6 @@ static int NapatechInit(int runmode)
         FatalError("Failed to allocate memory for NAPATECH device.");
     }
 
-    if ((ConfGetInt("napatech.hba", &conf->hba) != 0) && (conf->hba > 0)) {
-        SCLogInfo("Host Buffer Allowance: %d", (int) conf->hba);
-    }
-
     if (use_hw_bypass) {
 #ifdef NAPATECH_ENABLE_BYPASS
         if (NapatechVerifyBypassSupport()) {
diff --git a/src/source-napatech.c b/src/source-napatech.c
index 5d5e1f674d15..572fa748ec72 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -88,7 +88,6 @@ typedef struct NapatechThreadVars_
     ThreadVars *tv;
     NtNetStreamRx_t rx_stream;
     uint16_t stream_id;
-    int hba;
     TmSlot *slot;
 } NapatechThreadVars;
 
@@ -685,7 +684,6 @@ TmEcode NapatechStreamThreadInit(ThreadVars *tv, const void *initdata, void **da
     memset(ntv, 0, sizeof (NapatechThreadVars));
     ntv->stream_id = stream_id;
     ntv->tv = tv;
-    ntv->hba = conf->hba;
 
     DatalinkSetGlobalType(LINKTYPE_ETHERNET);
 
@@ -800,9 +798,6 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
     uint64_t pkt_ts;
     NtNetBuf_t packet_buffer;
     NapatechThreadVars *ntv = (NapatechThreadVars *) data;
-    uint64_t hba_pkt_drops = 0;
-    uint64_t hba_byte_drops = 0;
-    uint16_t hba_pkt = 0;
     int numa_node = -1;
     int set_cpu_affinity = 0;
     int closer = 0;
@@ -880,20 +875,10 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
             "Napatech Packet Loop Started - cpu: %3d, cpu_numa: %3d   stream: %3u ",
             sched_getcpu(), numa_node, ntv->stream_id);
 
-    if (ntv->hba > 0) {
-        char *s_hbad_pkt = SCCalloc(1, 32);
-        if (unlikely(s_hbad_pkt == NULL)) {
-            FatalError("Failed to allocate memory for NAPATECH stream counter.");
-        }
-        snprintf(s_hbad_pkt, 32, "nt%d.hba_drop", ntv->stream_id);
-        hba_pkt = StatsRegisterCounter(s_hbad_pkt, tv);
-        StatsSetupPrivate(tv);
-        StatsSetUI64(tv, hba_pkt, 0);
-    }
     SCLogDebug("Opening NAPATECH Stream: %u for processing", ntv->stream_id);
 
     if ((status = NT_NetRxOpen(&(ntv->rx_stream), "SuricataStream",
-            NT_NET_INTERFACE_PACKET, ntv->stream_id, ntv->hba)) != NT_SUCCESS) {
+            NT_NET_INTERFACE_PACKET, ntv->stream_id, -1)) != NT_SUCCESS) {
 
         NAPATECH_ERROR(status);
         SCFree(ntv);
@@ -969,22 +954,6 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
                 SCReturnInt(TM_ECODE_FAILED);
         }
 
-        if (unlikely(ntv->hba > 0)) {
-            NtNetRx_t stat_cmd;
-            stat_cmd.cmd = NT_NETRX_READ_CMD_STREAM_DROP;
-            /* Update drop counter */
-            if (unlikely((status = NT_NetRxRead(ntv->rx_stream, &stat_cmd)) != NT_SUCCESS)) {
-                NAPATECH_ERROR(status);
-                SCLogInfo("Couldn't retrieve drop statistics from the RX stream: %u",
-                        ntv->stream_id);
-            } else {
-                hba_pkt_drops = stat_cmd.u.streamDrop.pktsDropped;
-
-                StatsSetUI64(tv, hba_pkt, hba_pkt_drops);
-            }
-            StatsSyncCountersIfSignalled(tv);
-        }
-
 #ifdef NAPATECH_ENABLE_BYPASS
         p->ntpv.dyn3 = _NT_NET_GET_PKT_DESCR_PTR_DYN3(packet_buffer);
         p->BypassPacketsFlow = (NapatechIsBypassSupported() ? NapatechBypassCallback : NULL);
@@ -1019,10 +988,6 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
         NapatechDeleteFilters();
     }
 
-    if (unlikely(ntv->hba > 0)) {
-        SCLogInfo("Host Buffer Allowance Drops - pkts: %ld,  bytes: %ld", hba_pkt_drops, hba_byte_drops);
-    }
-
     SCReturnInt(TM_ECODE_OK);
 }
 
diff --git a/src/source-napatech.h b/src/source-napatech.h
index 2e3fb9462fb7..c638f89dae64 100644
--- a/src/source-napatech.h
+++ b/src/source-napatech.h
@@ -34,7 +34,6 @@ void TmModuleNapatechDecodeRegister(void);
 struct NapatechStreamDevConf
 {
     uint16_t stream_id;
-    intmax_t hba;
 };
 
 int NapatechSetPortmap(int port, int peer);

From 9865164e75decf58a3ab1da2ff3a161e639dd8b6 Mon Sep 17 00:00:00 2001
From: Ralph Eastwood <rea@napatech.com>
Date: Thu, 5 Oct 2023 08:02:19 +0000
Subject: [PATCH 111/462] napatech: update docs to remove hba reference

---
 doc/userguide/capture-hardware/napatech.rst | 23 ---------------------
 src/source-napatech.c                       |  6 +++---
 2 files changed, 3 insertions(+), 26 deletions(-)

diff --git a/doc/userguide/capture-hardware/napatech.rst b/doc/userguide/capture-hardware/napatech.rst
index e382de42947b..22acd9abfce0 100644
--- a/doc/userguide/capture-hardware/napatech.rst
+++ b/doc/userguide/capture-hardware/napatech.rst
@@ -391,14 +391,6 @@ that is being processed, the following counters will be output in stats.log:
 This is useful for fine-grain debugging to determine if a specific CPU core or
 thread is falling behind resulting in dropped packets.
 
-If hba is enabled the following counter will also be provided:
-
-- napa<streamid>.hba_drop: the number of packets dropped because the host buffer allowance high-water mark was reached.
-
-In addition to counters host buffer utilization is tracked and logged. This is also useful for
-debugging. Log messages are output for both Host and On-Board buffers when reach 25, 50, 75
-percent of utilization. Corresponding messages are output when utilization decreases.
-
 Debugging:
 
 For debugging configurations it is useful to see what traffic is flowing as well as what streams are
@@ -419,15 +411,6 @@ Napatech configuration options:
 These are the Napatech options available in the Suricata configuration file::
 
   napatech:
-    # The Host Buffer Allowance for all streams
-    # (-1 = OFF, 1 - 100 = percentage of the host buffer that can be held back)
-    # This may be enabled when sharing streams with another application.
-    # Otherwise, it should be turned off.
-    #
-    # Note: hba will be deprecated in Suricata 7
-    #
-    #hba: -1
-
     # When use_all_streams is set to "yes" the initialization code will query
     # the Napatech service for all configured streams and listen on all of them.
     # When set to "no" the streams config array will be used.
@@ -516,12 +499,6 @@ These are the Napatech options available in the Suricata configuration file::
     #
     hashmode: hash5tuplesorted
 
-*Note: hba is useful only when a stream is shared with another application. When hba is enabled packets will be dropped
-(i.e. not delivered to Suricata) when the host-buffer utilization reaches the high-water mark indicated by the hba value.
-This insures that, should Suricata get behind in its packet processing, the other application will still receive all
-of the packets. If this is enabled without another application sharing the stream it will result in sub-optimal packet
-buffering.*
-
 Make sure that there are enough host-buffers declared in ``ntservice.ini`` to
 accommodate the number of cores/streams being used.
 
diff --git a/src/source-napatech.c b/src/source-napatech.c
index 572fa748ec72..071d9ae68416 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -797,7 +797,7 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
     char error_buffer[100];
     uint64_t pkt_ts;
     NtNetBuf_t packet_buffer;
-    NapatechThreadVars *ntv = (NapatechThreadVars *) data;
+    NapatechThreadVars *ntv = (NapatechThreadVars *)data;
     int numa_node = -1;
     int set_cpu_affinity = 0;
     int closer = 0;
@@ -877,8 +877,8 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
 
     SCLogDebug("Opening NAPATECH Stream: %u for processing", ntv->stream_id);
 
-    if ((status = NT_NetRxOpen(&(ntv->rx_stream), "SuricataStream",
-            NT_NET_INTERFACE_PACKET, ntv->stream_id, -1)) != NT_SUCCESS) {
+    if ((status = NT_NetRxOpen(&(ntv->rx_stream), "SuricataStream", NT_NET_INTERFACE_PACKET,
+                 ntv->stream_id, -1)) != NT_SUCCESS) {
 
         NAPATECH_ERROR(status);
         SCFree(ntv);

From ad96382cf236d8c2e5c053fd56f83833c0926087 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 10 Oct 2023 08:02:12 -0400
Subject: [PATCH 112/462] output/null: Add the null output device

This commit adds the null output device; to use, set the filetype
to "nullsink" for each output that should discard and never persist
logs/alerts/etc.

This is implemented as an "internal eve output plugin" just like the
syslog eve output type.
---
 src/Makefile.am       |  2 +
 src/output-eve-null.c | 85 +++++++++++++++++++++++++++++++++++++++++++
 src/output-eve-null.h | 25 +++++++++++++
 src/output-json.c     |  4 ++
 4 files changed, 116 insertions(+)
 create mode 100644 src/output-eve-null.c
 create mode 100644 src/output-eve-null.h

diff --git a/src/Makefile.am b/src/Makefile.am
index c5b2fe52e894..8f6d9ced9908 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -386,6 +386,7 @@ noinst_HEADERS = \
 	log-tlslog.h \
 	log-tlsstore.h \
 	output-eve-stream.h \
+	output-eve-null.h \
 	output-filedata.h \
 	output-file.h \
 	output-filestore.h \
@@ -1039,6 +1040,7 @@ libsuricata_c_a_SOURCES = \
 	output-json-tftp.c \
 	output-json-tls.c \
 	output-eve-syslog.c \
+	output-eve-null.c \
 	output-lua.c \
 	output-packet.c \
 	output-stats.c \
diff --git a/src/output-eve-null.c b/src/output-eve-null.c
new file mode 100644
index 000000000000..1b62b96b36cb
--- /dev/null
+++ b/src/output-eve-null.c
@@ -0,0 +1,85 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Jeff Lucovsky <jlucovsky@oisf.net>
+ *
+ * File-like output for logging: null/discard device
+ */
+
+#include "suricata-common.h" /* errno.h, string.h, etc. */
+
+#include "output.h" /* DEFAULT_LOG_* */
+#include "output-eve-null.h"
+
+#ifdef OS_WIN32
+void NullLogInitialize(void)
+{
+}
+#else /* !OS_WIN32 */
+
+#define OUTPUT_NAME "nullsink"
+
+static int NullLogInit(ConfNode *conf, bool threaded, void **init_data)
+{
+    *init_data = NULL;
+    return 0;
+}
+
+static int NullLogWrite(const char *buffer, int buffer_len, void *init_data, void *thread_data)
+{
+    return 0;
+}
+
+static int NullLogThreadInit(void *init_data, int thread_id, void **thread_data)
+{
+    *thread_data = NULL;
+    return 0;
+}
+
+static int NullLogThreadDeInit(void *init_data, void *thread_data)
+{
+    return 0;
+}
+
+static void NullLogDeInit(void *init_data)
+{
+}
+
+void NullLogInitialize(void)
+{
+    SCLogDebug("Registering the %s logger", OUTPUT_NAME);
+
+    SCEveFileType *file_type = SCCalloc(1, sizeof(SCEveFileType));
+
+    if (file_type == NULL) {
+        FatalError("Unable to allocate memory for eve file type %s", OUTPUT_NAME);
+    }
+
+    file_type->name = OUTPUT_NAME;
+    file_type->Init = NullLogInit;
+    file_type->Deinit = NullLogDeInit;
+    file_type->Write = NullLogWrite;
+    file_type->ThreadInit = NullLogThreadInit;
+    file_type->ThreadDeinit = NullLogThreadDeInit;
+    if (!SCRegisterEveFileType(file_type)) {
+        FatalError("Failed to register EVE file type: %s", OUTPUT_NAME);
+    }
+}
+#endif /* !OS_WIN32 */
diff --git a/src/output-eve-null.h b/src/output-eve-null.h
new file mode 100644
index 000000000000..9fd331347f04
--- /dev/null
+++ b/src/output-eve-null.h
@@ -0,0 +1,25 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Jeff Lucovsky <jlucovsky@oisf.net>
+ *
+ * File-like output for logging: null/discard device
+ */
+void NullLogInitialize(void);
diff --git a/src/output-json.c b/src/output-json.c
index 5d4255cd2897..1dd2f948aba3 100644
--- a/src/output-json.c
+++ b/src/output-json.c
@@ -47,7 +47,10 @@
 #include "app-layer-parser.h"
 #include "util-classification-config.h"
 #include "util-syslog.h"
+
+/* Internal output plugins */
 #include "output-eve-syslog.h"
+#include "output-eve-null.h"
 
 #include "output.h"
 #include "output-json.h"
@@ -98,6 +101,7 @@ void OutputJsonRegister (void)
     // Register output file types that use the new eve filetype registration
     // API.
     SyslogInitialize();
+    NullLogInitialize();
 }
 
 json_t *SCJsonString(const char *val)

From 9d8eec453a11bf03e96ebe5215d8474c6b8ef348 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Mon, 16 Oct 2023 08:28:50 -0400
Subject: [PATCH 113/462] general: Remove vi formatting directives

---
 src/output-eve-syslog.c | 1 -
 src/util-log-redis.c    | 1 -
 src/util-logopenfile.c  | 1 -
 3 files changed, 3 deletions(-)

diff --git a/src/output-eve-syslog.c b/src/output-eve-syslog.c
index a19d2589449e..5d71b5d807d1 100644
--- a/src/output-eve-syslog.c
+++ b/src/output-eve-syslog.c
@@ -1,4 +1,3 @@
-/* vi: set et ts=4: */
 /* Copyright (C) 2021 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
diff --git a/src/util-log-redis.c b/src/util-log-redis.c
index b729c6be763e..5f590d2c6933 100644
--- a/src/util-log-redis.c
+++ b/src/util-log-redis.c
@@ -1,4 +1,3 @@
-/* vi: set et ts=4: */
 /* Copyright (C) 2007-2021 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
diff --git a/src/util-logopenfile.c b/src/util-logopenfile.c
index feca63f44afd..1b1986490658 100644
--- a/src/util-logopenfile.c
+++ b/src/util-logopenfile.c
@@ -1,4 +1,3 @@
-/* vi: set et ts=4: */
 /* Copyright (C) 2007-2022 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of

From 6a4184303508cee619e70341ff03d2bd5f51ff5e Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Sun, 22 Oct 2023 10:05:49 -0400
Subject: [PATCH 114/462] detect/tenants: Add tenant context to rule loads

Issue: 1520

This commit adds the tenant id for context to rule and .config file
loads.
---
 src/detect-engine-build.c        | 16 +++++++++++-----
 src/detect-engine-loader.c       | 17 ++++++++++++++---
 src/detect-engine.c              | 20 +++++++++++---------
 src/detect-engine.h              |  2 +-
 src/util-classification-config.c |  8 ++++++--
 src/util-reference-config.c      |  8 ++++++--
 src/util-threshold-config.c      |  6 +++++-
 7 files changed, 54 insertions(+), 23 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 8b7621271983..af59884cced1 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1496,11 +1496,17 @@ int SigAddressPrepareStage1(DetectEngineCtx *de_ctx)
     }
 
     if (!(de_ctx->flags & DE_QUIET)) {
-        SCLogInfo("%" PRIu32 " signatures processed. %" PRIu32 " are IP-only "
-                "rules, %" PRIu32 " are inspecting packet payload, %"PRIu32
-                " inspect application layer, %"PRIu32" are decoder event only",
-                de_ctx->sig_cnt, cnt_iponly, cnt_payload, cnt_applayer,
-                cnt_deonly);
+        if (strlen(de_ctx->config_prefix) > 0)
+            SCLogInfo("tenant id %d: %" PRIu32 " signatures processed. %" PRIu32 " are IP-only "
+                      "rules, %" PRIu32 " are inspecting packet payload, %" PRIu32
+                      " inspect application layer, %" PRIu32 " are decoder event only",
+                    de_ctx->tenant_id, de_ctx->sig_cnt, cnt_iponly, cnt_payload, cnt_applayer,
+                    cnt_deonly);
+        else
+            SCLogInfo("%" PRIu32 " signatures processed. %" PRIu32 " are IP-only "
+                      "rules, %" PRIu32 " are inspecting packet payload, %" PRIu32
+                      " inspect application layer, %" PRIu32 " are decoder event only",
+                    de_ctx->sig_cnt, cnt_iponly, cnt_payload, cnt_applayer, cnt_deonly);
 
         SCLogConfig("building signature grouping structure, stage 1: "
                "preprocessing rules... complete");
diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 3ef29b9b40f1..ae01f406e9ec 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -245,7 +245,11 @@ static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern,
         if (strcmp("/dev/null", fname) == 0)
             return 0;
 #endif
-        SCLogConfig("Loading rule file: %s", fname);
+        if (strlen(de_ctx->config_prefix) > 0) {
+            SCLogConfig("tenant id %d: Loading rule file: %s", de_ctx->tenant_id, fname);
+        } else {
+            SCLogConfig("Loading rule file: %s", fname);
+        }
         r = DetectLoadSigFile(de_ctx, fname, good_sigs, bad_sigs);
         if (r < 0) {
             ++(st->bad_files);
@@ -347,8 +351,15 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, int sig_file_excl
         }
     } else {
         /* we report the total of files and rules successfully loaded and failed */
-        SCLogInfo("%" PRId32 " rule files processed. %" PRId32 " rules successfully loaded, %" PRId32 " rules failed",
-            sig_stat->total_files, sig_stat->good_sigs_total, sig_stat->bad_sigs_total);
+        if (strlen(de_ctx->config_prefix) > 0)
+            SCLogInfo("tenant id %d:  %" PRId32 " rule files processed. %" PRId32
+                      " rules successfully loaded, %" PRId32 " rules failed",
+                    de_ctx->tenant_id, sig_stat->total_files, sig_stat->good_sigs_total,
+                    sig_stat->bad_sigs_total);
+        else
+            SCLogInfo("%" PRId32 " rule files processed. %" PRId32
+                      " rules successfully loaded, %" PRId32 " rules failed",
+                    sig_stat->total_files, sig_stat->good_sigs_total, sig_stat->bad_sigs_total);
     }
 
     if ((sig_stat->bad_sigs_total || sig_stat->bad_files) && de_ctx->failure_fatal) {
diff --git a/src/detect-engine.c b/src/detect-engine.c
index d8f9f1880e56..e50a6fa505ad 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2462,7 +2462,8 @@ static int DetectEngineReloadThreads(DetectEngineCtx *new_de_ctx)
     return -1;
 }
 
-static DetectEngineCtx *DetectEngineCtxInitReal(enum DetectEngineType type, const char *prefix)
+static DetectEngineCtx *DetectEngineCtxInitReal(
+        enum DetectEngineType type, const char *prefix, uint32_t tenant_id)
 {
     DetectEngineCtx *de_ctx = SCMalloc(sizeof(DetectEngineCtx));
     if (unlikely(de_ctx == NULL))
@@ -2474,6 +2475,7 @@ static DetectEngineCtx *DetectEngineCtxInitReal(enum DetectEngineType type, cons
     de_ctx->sigerror = NULL;
     de_ctx->type = type;
     de_ctx->filemagic_thread_ctx_id = -1;
+    de_ctx->tenant_id = tenant_id;
 
     if (type == DETECT_ENGINE_TYPE_DD_STUB || type == DETECT_ENGINE_TYPE_MT_STUB) {
         de_ctx->version = DetectEngineGetVersion();
@@ -2547,25 +2549,25 @@ static DetectEngineCtx *DetectEngineCtxInitReal(enum DetectEngineType type, cons
 
 DetectEngineCtx *DetectEngineCtxInitStubForMT(void)
 {
-    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_MT_STUB, NULL);
+    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_MT_STUB, NULL, 0);
 }
 
 DetectEngineCtx *DetectEngineCtxInitStubForDD(void)
 {
-    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_DD_STUB, NULL);
+    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_DD_STUB, NULL, 0);
 }
 
 DetectEngineCtx *DetectEngineCtxInit(void)
 {
-    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_NORMAL, NULL);
+    return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_NORMAL, NULL, 0);
 }
 
-DetectEngineCtx *DetectEngineCtxInitWithPrefix(const char *prefix)
+DetectEngineCtx *DetectEngineCtxInitWithPrefix(const char *prefix, uint32_t tenant_id)
 {
     if (prefix == NULL || strlen(prefix) == 0)
         return DetectEngineCtxInit();
     else
-        return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_NORMAL, prefix);
+        return DetectEngineCtxInitReal(DETECT_ENGINE_TYPE_NORMAL, prefix, tenant_id);
 }
 
 static void DetectEngineCtxFreeThreadKeywordData(DetectEngineCtx *de_ctx)
@@ -3841,7 +3843,7 @@ static int DetectEngineMultiTenantLoadTenant(uint32_t tenant_id, const char *fil
         goto error;
     }
 
-    de_ctx = DetectEngineCtxInitWithPrefix(prefix);
+    de_ctx = DetectEngineCtxInitWithPrefix(prefix, tenant_id);
     if (de_ctx == NULL) {
         SCLogError("initializing detection engine "
                    "context failed.");
@@ -3901,7 +3903,7 @@ static int DetectEngineMultiTenantReloadTenant(uint32_t tenant_id, const char *f
         goto error;
     }
 
-    DetectEngineCtx *new_de_ctx = DetectEngineCtxInitWithPrefix(prefix);
+    DetectEngineCtx *new_de_ctx = DetectEngineCtxInitWithPrefix(prefix, tenant_id);
     if (new_de_ctx == NULL) {
         SCLogError("initializing detection engine "
                    "context failed.");
@@ -4759,7 +4761,7 @@ int DetectEngineReload(const SCInstance *suri)
     }
 
     /* get new detection engine */
-    new_de_ctx = DetectEngineCtxInitWithPrefix(prefix);
+    new_de_ctx = DetectEngineCtxInitWithPrefix(prefix, old_de_ctx->tenant_id);
     if (new_de_ctx == NULL) {
         SCLogError("initializing detection engine "
                    "context failed.");
diff --git a/src/detect-engine.h b/src/detect-engine.h
index a1732b16a993..02e784ee973c 100644
--- a/src/detect-engine.h
+++ b/src/detect-engine.h
@@ -88,7 +88,7 @@ void DetectEngineBufferTypeSupportsMpm(DetectEngineCtx *de_ctx, const char *name
 void DetectEngineBufferTypeSupportsTransformations(DetectEngineCtx *de_ctx, const char *name);
 
 /* prototypes */
-DetectEngineCtx *DetectEngineCtxInitWithPrefix(const char *prefix);
+DetectEngineCtx *DetectEngineCtxInitWithPrefix(const char *prefix, uint32_t tenant_id);
 DetectEngineCtx *DetectEngineCtxInit(void);
 DetectEngineCtx *DetectEngineCtxInitStubForDD(void);
 DetectEngineCtx *DetectEngineCtxInitStubForMT(void);
diff --git a/src/util-classification-config.c b/src/util-classification-config.c
index be42469e6d4a..9d7ed05bde32 100644
--- a/src/util-classification-config.c
+++ b/src/util-classification-config.c
@@ -363,8 +363,12 @@ static bool SCClassConfParseFile(DetectEngineCtx *de_ctx, FILE *fd)
     }
 
 #ifdef UNITTESTS
-    SCLogInfo("Added \"%d\" classification types from the classification file",
-              de_ctx->class_conf_ht->count);
+    if (de_ctx != NULL && strlen(de_ctx->config_prefix) > 0)
+        SCLogInfo("tenant id %d: Added \"%d\" classification types from the classification file",
+                de_ctx->tenant_id, de_ctx->class_conf_ht->count);
+    else
+        SCLogInfo("Added \"%d\" classification types from the classification file",
+                de_ctx->class_conf_ht->count);
 #endif
 
     return errors == 0;
diff --git a/src/util-reference-config.c b/src/util-reference-config.c
index 0a3109825229..0e5c51ea141e 100644
--- a/src/util-reference-config.c
+++ b/src/util-reference-config.c
@@ -335,8 +335,12 @@ static bool SCRConfParseFile(DetectEngineCtx *de_ctx, FILE *fd)
     }
 
 #ifdef UNITTESTS
-    SCLogInfo("Added \"%d\" reference types from the reference.config file",
-              de_ctx->reference_conf_ht->count);
+    if (de_ctx != NULL && strlen(de_ctx->config_prefix) > 0)
+        SCLogInfo("tenant id %d: Added \"%d\" reference types from the reference.config file",
+                de_ctx->tenant_id, de_ctx->reference_conf_ht->count);
+    else
+        SCLogInfo("Added \"%d\" reference types from the reference.config file",
+                de_ctx->reference_conf_ht->count);
 #endif /* UNITTESTS */
     return true;
 }
diff --git a/src/util-threshold-config.c b/src/util-threshold-config.c
index 5d762a8f7091..0e5caf83265f 100644
--- a/src/util-threshold-config.c
+++ b/src/util-threshold-config.c
@@ -1042,7 +1042,11 @@ int SCThresholdConfParseFile(DetectEngineCtx *de_ctx, FILE *fp)
         }
     }
 
-    SCLogInfo("Threshold config parsed: %d rule(s) found", rule_num);
+    if (de_ctx != NULL && strlen(de_ctx->config_prefix) > 0)
+        SCLogInfo("tenant id %d: Threshold config parsed: %d rule(s) found", de_ctx->tenant_id,
+                rule_num);
+    else
+        SCLogInfo("Threshold config parsed: %d rule(s) found", rule_num);
     return 0;
 }
 

From ffd559cd8e941dafaa2f143437a1253abb77c745 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 9 Nov 2023 05:38:44 -0500
Subject: [PATCH 115/462] detect/transform: Add case-change transform constants

Add the constants for the to_lowercase and to_uppercase transforms

Issue: 6439
---
 src/detect-engine-register.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 24a0e56cce1b..92acd84f044b 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -325,6 +325,8 @@ enum DetectKeywordId {
     DETECT_TRANSFORM_PCREXFORM,
     DETECT_TRANSFORM_URL_DECODE,
     DETECT_TRANSFORM_XOR,
+    DETECT_TRANSFORM_TOLOWER,
+    DETECT_TRANSFORM_TOUPPER,
 
     DETECT_AL_IKE_EXCH_TYPE,
     DETECT_AL_IKE_SPI_INITIATOR,

From e5c2f9a56dee82fc498d85aec68d028b89237544 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 9 Nov 2023 05:39:33 -0500
Subject: [PATCH 116/462] detect/transform: Add case changing transforms

This commit adds the implementation for the case changing transforms:
to_lowercase and to_uppercase

Issue: 6439
---
 src/Makefile.am                   |   2 +
 src/detect-transform-casechange.c | 169 ++++++++++++++++++++++++++++++
 src/detect-transform-casechange.h |  31 ++++++
 3 files changed, 202 insertions(+)
 create mode 100644 src/detect-transform-casechange.c
 create mode 100644 src/detect-transform-casechange.h

diff --git a/src/Makefile.am b/src/Makefile.am
index 8f6d9ced9908..a125e2a432ba 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -338,6 +338,7 @@ noinst_HEADERS = \
 	detect-tls-version.h \
 	detect-tls-random.h \
 	detect-tos.h \
+	detect-transform-casechange.h \
 	detect-transform-compress-whitespace.h \
 	detect-transform-dotprefix.h \
 	detect-transform-md5.h \
@@ -949,6 +950,7 @@ libsuricata_c_a_SOURCES = \
 	detect-tls-version.c \
 	detect-tls-random.c \
 	detect-tos.c \
+	detect-transform-casechange.c \
 	detect-transform-compress-whitespace.c \
 	detect-transform-dotprefix.c \
 	detect-transform-md5.c \
diff --git a/src/detect-transform-casechange.c b/src/detect-transform-casechange.c
new file mode 100644
index 000000000000..851030828ced
--- /dev/null
+++ b/src/detect-transform-casechange.c
@@ -0,0 +1,169 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Jeff Lucovsky <jlucovsky@oisf.net>
+ *
+ * Implements case changing transforms
+ */
+
+#include "suricata-common.h"
+#include "detect.h"
+#include "detect-engine.h"
+#include "detect-parse.h"
+#include "detect-transform-casechange.h"
+
+/**
+ *  \internal
+ *  \brief Register the to_lowercase transform
+ *  \param det_ctx detection engine ctx
+ *  \param s signature
+ *  \param optstr options string
+ *  \retval 0 ok
+ *  \retval -1 failure
+ */
+static int DetectTransformToLowerSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
+{
+    SCEnter();
+
+    int r = DetectSignatureAddTransform(s, DETECT_TRANSFORM_TOLOWER, NULL);
+
+    SCReturnInt(r);
+}
+
+/**
+ *  \internal
+ *  \brief Apply the to_lowercase keyword to the last pattern match
+ *  \param buffer Inspection buffer
+ *  \param optstr options string
+ */
+static void DetectTransformToLower(InspectionBuffer *buffer, void *options)
+{
+    const uint8_t *input = buffer->inspect;
+    const uint32_t input_len = buffer->inspect_len;
+
+    if (input_len == 0) {
+        return;
+    }
+
+    uint8_t output[input_len];
+    for (uint32_t i = 0; i < input_len; i++) {
+        output[i] = u8_tolower(input[i]);
+    }
+
+    InspectionBufferCopy(buffer, output, input_len);
+}
+/**
+ *  \internal
+ *  \brief Register the to_upperrcase transform
+ *  \param det_ctx detection engine ctx
+ *  \param s signature
+ *  \param optstr options string
+ *  \retval 0 ok
+ *  \retval -1 failure
+ */
+static int DetectTransformToUpperSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
+{
+    SCEnter();
+
+    int r = DetectSignatureAddTransform(s, DETECT_TRANSFORM_TOUPPER, NULL);
+
+    SCReturnInt(r);
+}
+
+/**
+ *  \internal
+ *  \brief Apply the to_uppercase keyword to the last pattern match
+ *  \param buffer Inspection buffer
+ *  \param optstr options string
+ */
+static void DetectTransformToUpper(InspectionBuffer *buffer, void *options)
+{
+    const uint8_t *input = buffer->inspect;
+    const uint32_t input_len = buffer->inspect_len;
+
+    if (input_len == 0) {
+        return;
+    }
+
+    uint8_t output[input_len];
+    for (uint32_t i = 0; i < input_len; i++) {
+        output[i] = u8_toupper(input[i]);
+    }
+
+    InspectionBufferCopy(buffer, output, input_len);
+}
+
+/*
+ * \internal
+ * \brief Check if content is compatible with transform
+ *
+ * If the content contains any lowercase characters, than it is not compatible.
+ */
+static bool TransformToUpperValidate(const uint8_t *content, uint16_t content_len, void *options)
+{
+    if (content) {
+        for (uint32_t i = 0; i < content_len; i++) {
+            if (islower(*content++)) {
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
+/*
+ * \internal
+ * \brief Check if content is compatible with transform
+ *
+ * If the content contains any uppercase characters, than it is not compatible.
+ */
+static bool TransformToLowerValidate(const uint8_t *content, uint16_t content_len, void *options)
+{
+    if (content) {
+        for (uint32_t i = 0; i < content_len; i++) {
+            if (isupper(*content++)) {
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
+void DetectTransformToUpperRegister(void)
+{
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].name = "to_uppercase";
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].desc = "convert buffer to uppercase";
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].url = "/rules/transforms.html#to_uppercase";
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].Transform = DetectTransformToUpper;
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].TransformValidate = TransformToUpperValidate;
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].Setup = DetectTransformToUpperSetup;
+    sigmatch_table[DETECT_TRANSFORM_TOUPPER].flags |= SIGMATCH_NOOPT;
+}
+
+void DetectTransformToLowerRegister(void)
+{
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].name = "to_lowercase";
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].desc = "convert buffer to lowercase";
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].url = "/rules/transforms.html#to_lowercase";
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].Transform = DetectTransformToLower;
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].TransformValidate = TransformToLowerValidate;
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].Setup = DetectTransformToLowerSetup;
+    sigmatch_table[DETECT_TRANSFORM_TOLOWER].flags |= SIGMATCH_NOOPT;
+}
diff --git a/src/detect-transform-casechange.h b/src/detect-transform-casechange.h
new file mode 100644
index 000000000000..db6275b915f7
--- /dev/null
+++ b/src/detect-transform-casechange.h
@@ -0,0 +1,31 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Jeff Lucovsky <jlucovsky@oisf.net>
+ */
+
+#ifndef __DETECT_TRANSFORM_CASECHANGE_H
+#define __DETECT_TRANSFORM_CASECHANGE_H
+
+/* prototypes */
+void DetectTransformToLowerRegister(void);
+void DetectTransformToUpperRegister(void);
+
+#endif /* __DETECT_TRANSFORM_CASECHANGE_H */

From 9ee55d23949a1bb42e6b48429d720fcc6c15147b Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 9 Nov 2023 06:22:55 -0500
Subject: [PATCH 117/462] doc/transform: Document case-changing transforms.

Issue: 6439
---
 doc/userguide/rules/transforms.rst | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/doc/userguide/rules/transforms.rst b/doc/userguide/rules/transforms.rst
index 91ab2ef5fc9e..f52bac7f3eea 100644
--- a/doc/userguide/rules/transforms.rst
+++ b/doc/userguide/rules/transforms.rst
@@ -68,6 +68,18 @@ compress_whitespace
 
 Compresses all consecutive whitespace into a single space.
 
+to_lowercase
+------------
+
+Converts the buffer to lowercase and passes the value on.
+
+This example alerts if ``http.uri`` contains ``this text has been converted to lowercase``
+
+Example::
+
+    alert http any any -> any any (http.uri; to_lowercase; \
+        content:"this text has been converted to lowercase"; sid:1;)
+
 to_md5
 ------
 
@@ -79,6 +91,18 @@ Example::
     alert http any any -> any any (http_request_line; to_md5; \
         content:"|54 A9 7A 8A B0 9C 1B 81 37 25 22 14 51 D3 F9 97|"; sid:1;)
 
+to_uppercase
+------------
+
+Converts the buffer to uppercase and passes the value on.
+
+This example alerts if ``http.uri`` contains ``THIS TEXT HAS BEEN CONVERTED TO LOWERCASE``
+
+Example::
+
+    alert http any any -> any any (http.uri; to_uppercase; \
+        content:"THIS TEXT HAS BEEN CONVERTED TO UPPERCASE"; sid:1;)
+
 to_sha1
 ---------
 
@@ -134,3 +158,4 @@ Example::
 
     alert http any any -> any any (msg:"HTTP with xor"; http.uri; \
         xor:"0d0ac8ff"; content:"password="; sid:1;)
+

From 1110a86cb9bc3815aec2db1b5cc3253d63db53f5 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 9 Nov 2023 08:35:46 -0500
Subject: [PATCH 118/462] detect/transform: Register case-change transforms

Issue: 6439
---
 src/detect-engine-register.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index d570510949f4..af247a1b4fa7 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -245,6 +245,7 @@
 #include "detect-transform-pcrexform.h"
 #include "detect-transform-urldecode.h"
 #include "detect-transform-xor.h"
+#include "detect-transform-casechange.h"
 
 #include "util-rule-vars.h"
 
@@ -701,6 +702,8 @@ void SigTableSetup(void)
     DetectTransformPcrexformRegister();
     DetectTransformUrlDecodeRegister();
     DetectTransformXorRegister();
+    DetectTransformToLowerRegister();
+    DetectTransformToUpperRegister();
 
     DetectFileHandlerRegister();
 

From a46779d866b1b121adc73164215ba6437f53c208 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 14 Nov 2023 08:23:43 -0500
Subject: [PATCH 119/462] detect/transform: Clarify transformation validation

Issue: 6439

Clarify the transform validation step. When a transform indicates that
the content/byte-array is not compatible, validation will stop.

Content is incompatible is some cases -- e.g., following the
to_lowercase transform with content containing uppercase characters.
An alert is not possible since the content contains uppercase and the
transform has converted the buffer into all lowercase.
---
 src/detect-engine.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index e50a6fa505ad..3754c713225a 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -1690,8 +1690,8 @@ void InspectionBufferCopy(InspectionBuffer *buffer, uint8_t *buf, uint32_t buf_l
  *  transform may validate that it's compatible with the transform.
  *
  *  When a transform indicates the byte array is incompatible, none of the
- *  subsequent transforms, if any, are invoked. This means the first positive
- *  validation result terminates the loop.
+ *  subsequent transforms, if any, are invoked. This means the first validation
+ *  failure terminates the loop.
  *
  *  \param de_ctx Detection engine context.
  *  \param sm_list The SM list id.

From e3cd0d073f18a9d760e332852d53bce080ea96f0 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 7 Nov 2023 17:23:23 +0100
Subject: [PATCH 120/462] http2: app-layer event for userinfo in uri

Ticket: #6426

as per RFC 9113
":authority" MUST NOT include the deprecated userinfo subcomponent
for "http" or "https" schemed URIs.
---
 rules/http2-events.rules | 1 +
 rust/src/http2/http2.rs  | 6 ++++++
 2 files changed, 7 insertions(+)

diff --git a/rules/http2-events.rules b/rules/http2-events.rules
index 868943a77bed..7cceaf24c307 100644
--- a/rules/http2-events.rules
+++ b/rules/http2-events.rules
@@ -19,3 +19,4 @@ alert http2 any any -> any any (msg:"SURICATA HTTP2 invalid range header"; flow:
 alert http2 any any -> any any (msg:"SURICATA HTTP2 variable-length integer overflow"; flow:established; app-layer-event:http2.header_integer_overflow; classtype:protocol-command-decode; sid:2290011; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 too many streams"; flow:established; app-layer-event:http2.too_many_streams; classtype:protocol-command-decode; sid:2290012; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 authority host mismatch"; flow:established,to_server; app-layer-event:http2.authority_host_mismatch; classtype:protocol-command-decode; sid:2290013; rev:1;)
+alert http2 any any -> any any (msg:"SURICATA HTTP2 user info in uri"; flow:established,to_server; app-layer-event:http2.userinfo_in_uri; classtype:protocol-command-decode; sid:2290014; rev:1;)
diff --git a/rust/src/http2/http2.rs b/rust/src/http2/http2.rs
index bbaeddb40434..14d7b47dfb03 100644
--- a/rust/src/http2/http2.rs
+++ b/rust/src/http2/http2.rs
@@ -210,6 +210,11 @@ impl HTTP2Transaction {
                 self.decoder.http2_encoding_fromvec(&block.value, dir);
             } else if block.name.eq_ignore_ascii_case(b":authority") {
                 authority = Some(&block.value);
+                if block.value.iter().any(|&x| x == b'@') {
+                    // it is forbidden by RFC 9113 to have userinfo in this field
+                    // when in HTTP1 we can have user:password@domain.com
+                    self.set_event(HTTP2Event::UserinfoInUri);
+                }
             } else if block.name.eq_ignore_ascii_case(b"host") {
                 host = Some(&block.value);
             }
@@ -400,6 +405,7 @@ pub enum HTTP2Event {
     HeaderIntegerOverflow,
     TooManyStreams,
     AuthorityHostMismatch,
+    UserinfoInUri,
 }
 
 pub struct HTTP2DynTable {

From 58c7a438ed22e82f7ed1fd853676fd789ff82f05 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 9 Sep 2023 17:38:17 +0200
Subject: [PATCH 121/462] detect/flow: optimize only_stream/no_stream options

Until now the implementation would scan the stream, fallback to the
packet payload in exception cases, then keep track of where the match
was and in the flow match logic reject the match if it was in the wrong
buffer.

This patch simplifies this logic, by refusing to inspect the packet
payload when `only_stream` is set.

To do this the `only_stream`/`no_stream` options are now translated
to the pseudo protocols `tcp-stream` and `tcp-pkt` at parsing, so that
the `flow` keyword doesn't have to evaluate these conditions anymore.
---
 src/detect-engine.c |   4 +
 src/detect-flow.c   | 305 ++++++++++++++++++++++----------------------
 src/detect.h        |   4 +
 3 files changed, 163 insertions(+), 150 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 3754c713225a..4059ffaf5261 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -1970,6 +1970,10 @@ static int DetectEngineInspectRulePayloadMatches(
             if (!(s->flags & SIG_FLAG_REQUIRE_PACKET) && (p->flags & PKT_STREAM_ADD)) {
                 return false;
             }
+            if (s->flags & SIG_FLAG_REQUIRE_STREAM_ONLY) {
+                SCLogDebug("SIG_FLAG_REQUIRE_STREAM_ONLY, so no match");
+                return false;
+            }
             if (DetectEngineInspectPacketPayload(de_ctx, det_ctx, s, p->flow, p) != 1) {
                 return false;
             }
diff --git a/src/detect-flow.c b/src/detect-flow.c
index fdee0779e458..9b0627cdfb12 100644
--- a/src/detect-flow.c
+++ b/src/detect-flow.c
@@ -82,12 +82,11 @@ void DetectFlowRegister (void)
 /**
  * \param pflags packet flags (p->flags)
  * \param pflowflags packet flow flags (p->flowflags)
- * \param tflags detection flags (det_ctx->flags)
  * \param dflags detect flow flags
  * \param match_cnt number of matches to trigger
  */
-static inline int FlowMatch(const uint32_t pflags, const uint8_t pflowflags, const uint16_t tflags,
-        const uint16_t dflags, const uint16_t match_cnt)
+static inline int FlowMatch(const uint32_t pflags, const uint8_t pflowflags, const uint16_t dflags,
+        const uint16_t match_cnt)
 {
     uint8_t cnt = 0;
 
@@ -113,14 +112,6 @@ static inline int FlowMatch(const uint32_t pflags, const uint8_t pflowflags, con
         cnt++;
     }
 
-    if (tflags & DETECT_ENGINE_THREAD_CTX_STREAM_CONTENT_MATCH) {
-        if (dflags & DETECT_FLOW_FLAG_ONLYSTREAM)
-            cnt++;
-    } else {
-        if (dflags & DETECT_FLOW_FLAG_NOSTREAM)
-            cnt++;
-    }
-
     return (match_cnt == cnt) ? 1 : 0;
 }
 
@@ -154,7 +145,7 @@ int DetectFlowMatch (DetectEngineThreadCtx *det_ctx, Packet *p,
 
     const DetectFlowData *fd = (const DetectFlowData *)ctx;
 
-    const int ret = FlowMatch(p->flags, p->flowflags, det_ctx->flags, fd->flags, fd->match_cnt);
+    const int ret = FlowMatch(p->flags, p->flowflags, fd->flags, fd->match_cnt);
     SCLogDebug("returning %" PRId32 " fd->match_cnt %" PRId32 " fd->flags 0x%02X p->flowflags 0x%02X",
         ret, fd->match_cnt, fd->flags, p->flowflags);
     SCReturnInt(ret);
@@ -165,11 +156,13 @@ int DetectFlowMatch (DetectEngineThreadCtx *det_ctx, Packet *p,
  *
  * \param de_ctx Pointer to the detection engine context
  * \param flowstr Pointer to the user provided flow options
+ * \param[out] parse_flags keyword flags only used during parsing
  *
  * \retval fd pointer to DetectFlowData on success
  * \retval NULL on failure
  */
-static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flowstr)
+static DetectFlowData *DetectFlowParse(
+        DetectEngineCtx *de_ctx, const char *flowstr, uint16_t *parse_flags)
 {
     DetectFlowData *fd = NULL;
     char *args[3] = {NULL,NULL,NULL};
@@ -219,8 +212,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
     fd->flags = 0;
     fd->match_cnt = 0;
 
-    int i;
-    for (i = 0; i < (ret - 1); i++) {
+    for (int i = 0; i < (ret - 1); i++) {
         if (args[i]) {
             /* inspect our options and set the flags */
             if (strcasecmp(args[i], "established") == 0) {
@@ -236,6 +228,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_ESTABLISHED;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "not_established") == 0) {
                 if (fd->flags & DETECT_FLOW_FLAG_NOT_ESTABLISHED) {
                     SCLogError("DETECT_FLOW_FLAG_NOT_ESTABLISHED flag is already set");
@@ -246,6 +239,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_NOT_ESTABLISHED;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "stateless") == 0) {
                 if (fd->flags & DETECT_FLOW_FLAG_STATELESS) {
                     SCLogError("DETECT_FLOW_FLAG_STATELESS flag is already set");
@@ -256,6 +250,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_STATELESS;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "to_client") == 0 || strcasecmp(args[i], "from_server") == 0) {
                 if (fd->flags & DETECT_FLOW_FLAG_TOCLIENT) {
                     SCLogError("cannot set DETECT_FLOW_FLAG_TOCLIENT flag is already set");
@@ -265,6 +260,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_TOCLIENT;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "to_server") == 0 || strcasecmp(args[i], "from_client") == 0){
                 if (fd->flags & DETECT_FLOW_FLAG_TOSERVER) {
                     SCLogError("cannot set DETECT_FLOW_FLAG_TOSERVER flag is already set");
@@ -274,26 +270,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_TOSERVER;
-            } else if (strcasecmp(args[i], "only_stream") == 0) {
-                if (fd->flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
-                    SCLogError("cannot set only_stream flag is already set");
-                    goto error;
-                } else if (fd->flags & DETECT_FLOW_FLAG_NOSTREAM) {
-                    SCLogError(
-                            "cannot set only_stream flag, DETECT_FLOW_FLAG_NOSTREAM already set");
-                    goto error;
-                }
-                fd->flags |= DETECT_FLOW_FLAG_ONLYSTREAM;
-            } else if (strcasecmp(args[i], "no_stream") == 0) {
-                if (fd->flags & DETECT_FLOW_FLAG_NOSTREAM) {
-                    SCLogError("cannot set no_stream flag is already set");
-                    goto error;
-                } else if (fd->flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
-                    SCLogError(
-                            "cannot set no_stream flag, DETECT_FLOW_FLAG_ONLYSTREAM already set");
-                    goto error;
-                }
-                fd->flags |= DETECT_FLOW_FLAG_NOSTREAM;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "no_frag") == 0) {
                 if (fd->flags & DETECT_FLOW_FLAG_NO_FRAG) {
                     SCLogError("cannot set no_frag flag is already set");
@@ -303,6 +280,7 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_NO_FRAG;
+                fd->match_cnt++;
             } else if (strcasecmp(args[i], "only_frag") == 0) {
                 if (fd->flags & DETECT_FLOW_FLAG_ONLY_FRAG) {
                     SCLogError("cannot set only_frag flag is already set");
@@ -312,13 +290,34 @@ static DetectFlowData *DetectFlowParse (DetectEngineCtx *de_ctx, const char *flo
                     goto error;
                 }
                 fd->flags |= DETECT_FLOW_FLAG_ONLY_FRAG;
+                fd->match_cnt++;
+
+                /* special case: these only affect parsing, not matching */
+
+            } else if (strcasecmp(args[i], "only_stream") == 0) {
+                if (*parse_flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
+                    SCLogError("cannot set only_stream flag is already set");
+                    goto error;
+                } else if (*parse_flags & DETECT_FLOW_FLAG_NOSTREAM) {
+                    SCLogError(
+                            "cannot set only_stream flag, DETECT_FLOW_FLAG_NOSTREAM already set");
+                    goto error;
+                }
+                *parse_flags |= DETECT_FLOW_FLAG_ONLYSTREAM;
+            } else if (strcasecmp(args[i], "no_stream") == 0) {
+                if (*parse_flags & DETECT_FLOW_FLAG_NOSTREAM) {
+                    SCLogError("cannot set no_stream flag is already set");
+                    goto error;
+                } else if (*parse_flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
+                    SCLogError(
+                            "cannot set no_stream flag, DETECT_FLOW_FLAG_ONLYSTREAM already set");
+                    goto error;
+                }
+                *parse_flags |= DETECT_FLOW_FLAG_NOSTREAM;
             } else {
                 SCLogError("invalid flow option \"%s\"", args[i]);
                 goto error;
             }
-
-            fd->match_cnt++;
-            //printf("args[%" PRId32 "]: %s match_cnt: %" PRId32 " flags: 0x%02X\n", i, args[i], fd->match_cnt, fd->flags);
         }
     }
     pcre2_match_data_free(match);
@@ -377,13 +376,15 @@ int DetectFlowSetupImplicit(Signature *s, uint32_t flags)
  */
 int DetectFlowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *flowstr)
 {
+    uint16_t parse_flags = 0;
+
     /* ensure only one flow option */
     if (s->init_data->init_flags & SIG_FLAG_INIT_FLOW) {
         SCLogError("A signature may have only one flow option.");
         return -1;
     }
 
-    DetectFlowData *fd = DetectFlowParse(de_ctx, flowstr);
+    DetectFlowData *fd = DetectFlowParse(de_ctx, flowstr, &parse_flags);
     if (fd == NULL)
         return -1;
 
@@ -403,14 +404,8 @@ int DetectFlowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *flowstr)
         s->flags |= SIG_FLAG_TOSERVER;
         s->flags |= SIG_FLAG_TOCLIENT;
     }
-    if (fd->flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
-        s->flags |= SIG_FLAG_REQUIRE_STREAM;
-    }
-    if (fd->flags & DETECT_FLOW_FLAG_NOSTREAM) {
-        s->flags |= SIG_FLAG_REQUIRE_PACKET;
-    } else if (fd->flags == DETECT_FLOW_FLAG_TOSERVER ||
-               fd->flags == DETECT_FLOW_FLAG_TOCLIENT)
-    {
+    if (fd->flags == 0 || fd->flags == DETECT_FLOW_FLAG_TOSERVER ||
+            fd->flags == DETECT_FLOW_FLAG_TOCLIENT) {
         /* no direct flow is needed for just direction,
          * no sigmatch is needed either. */
         SigMatchFree(de_ctx, sm);
@@ -422,6 +417,13 @@ int DetectFlowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *flowstr)
     if (sm != NULL) {
         SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     }
+
+    if (parse_flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
+        s->flags |= (SIG_FLAG_REQUIRE_STREAM | SIG_FLAG_REQUIRE_STREAM_ONLY);
+    }
+    if (parse_flags & DETECT_FLOW_FLAG_NOSTREAM) {
+        s->flags |= SIG_FLAG_REQUIRE_PACKET;
+    }
     return 0;
 
 error:
@@ -450,7 +452,7 @@ PrefilterPacketFlowMatch(DetectEngineThreadCtx *det_ctx, Packet *p, const void *
     if (!PrefilterPacketHeaderExtraMatch(ctx, p))
         return;
 
-    if (FlowMatch(p->flags, p->flowflags, det_ctx->flags, ctx->v1.u16[0], ctx->v1.u16[1])) {
+    if (FlowMatch(p->flags, p->flowflags, ctx->v1.u16[0], ctx->v1.u16[1])) {
         PrefilterAddSids(&det_ctx->pmq, ctx->sigs_array, ctx->sigs_cnt);
     }
 }
@@ -502,9 +504,10 @@ static bool PrefilterFlowIsPrefilterable(const Signature *s)
  */
 static int DetectFlowTestParse01 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "established");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "established", &parsed_flags);
     FAIL_IF_NULL(fd);
+    FAIL_IF_NOT(parsed_flags == 0);
     DetectFlowFree(NULL, fd);
     PASS;
 }
@@ -514,8 +517,8 @@ static int DetectFlowTestParse01 (void)
  */
 static int DetectFlowTestParse02 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "established");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "established", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_ESTABLISHED &&
         fd->match_cnt == 1);
@@ -527,8 +530,8 @@ static int DetectFlowTestParse02 (void)
  */
 static int DetectFlowTestParse03 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "stateless");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "stateless", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_STATELESS && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -540,8 +543,8 @@ static int DetectFlowTestParse03 (void)
  */
 static int DetectFlowTestParse04 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "to_client");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "to_client", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOCLIENT && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -553,8 +556,8 @@ static int DetectFlowTestParse04 (void)
  */
 static int DetectFlowTestParse05 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "to_server");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "to_server", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOSERVER && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -566,8 +569,8 @@ static int DetectFlowTestParse05 (void)
  */
 static int DetectFlowTestParse06 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_server", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOCLIENT && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -579,8 +582,8 @@ static int DetectFlowTestParse06 (void)
  */
 static int DetectFlowTestParse07 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_client");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_client", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOSERVER && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -592,8 +595,8 @@ static int DetectFlowTestParse07 (void)
  */
 static int DetectFlowTestParse08 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "established,to_client");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "established,to_client", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED && fd->flags & DETECT_FLOW_FLAG_TOCLIENT && fd->match_cnt == 2);
     DetectFlowFree(NULL, fd);
@@ -605,8 +608,8 @@ static int DetectFlowTestParse08 (void)
  */
 static int DetectFlowTestParse09 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "to_client,stateless");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "to_client,stateless", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -620,8 +623,8 @@ static int DetectFlowTestParse09 (void)
  */
 static int DetectFlowTestParse10 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server,stateless");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_server,stateless", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -635,8 +638,8 @@ static int DetectFlowTestParse10 (void)
  */
 static int DetectFlowTestParse11 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, " from_server , stateless ");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, " from_server , stateless ", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -651,8 +654,8 @@ static int DetectFlowTestParse11 (void)
  */
 static int DetectFlowTestParseNocase01 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "ESTABLISHED");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "ESTABLISHED", &parsed_flags);
     FAIL_IF_NULL(fd);
     DetectFlowFree(NULL, fd);
     PASS;
@@ -663,8 +666,8 @@ static int DetectFlowTestParseNocase01 (void)
  */
 static int DetectFlowTestParseNocase02 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "ESTABLISHED");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "ESTABLISHED", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_ESTABLISHED &&
         fd->match_cnt == 1);
@@ -677,10 +680,11 @@ static int DetectFlowTestParseNocase02 (void)
  */
 static int DetectFlowTestParseNocase03 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "STATELESS");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "STATELESS", &parsed_flags);
     FAIL_IF_NULL(fd);
-    FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_STATELESS && fd->match_cnt == 1);         DetectFlowFree(NULL, fd);
+    FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_STATELESS && fd->match_cnt == 1);
+    DetectFlowFree(NULL, fd);
     PASS;
 }
 
@@ -689,8 +693,8 @@ static int DetectFlowTestParseNocase03 (void)
  */
 static int DetectFlowTestParseNocase04 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "TO_CLIENT");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "TO_CLIENT", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOCLIENT && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -702,8 +706,8 @@ static int DetectFlowTestParseNocase04 (void)
  */
 static int DetectFlowTestParseNocase05 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "TO_SERVER");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "TO_SERVER", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOSERVER && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -715,8 +719,8 @@ static int DetectFlowTestParseNocase05 (void)
  */
 static int DetectFlowTestParseNocase06 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "FROM_SERVER");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "FROM_SERVER", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOCLIENT && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -728,8 +732,8 @@ static int DetectFlowTestParseNocase06 (void)
  */
 static int DetectFlowTestParseNocase07 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "FROM_CLIENT");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "FROM_CLIENT", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags == DETECT_FLOW_FLAG_TOSERVER && fd->match_cnt == 1);
     DetectFlowFree(NULL, fd);
@@ -741,8 +745,8 @@ static int DetectFlowTestParseNocase07 (void)
  */
 static int DetectFlowTestParseNocase08 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "ESTABLISHED,TO_CLIENT");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "ESTABLISHED,TO_CLIENT", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -756,8 +760,8 @@ static int DetectFlowTestParseNocase08 (void)
  */
 static int DetectFlowTestParseNocase09 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "TO_CLIENT,STATELESS");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "TO_CLIENT,STATELESS", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -771,8 +775,8 @@ static int DetectFlowTestParseNocase09 (void)
  */
 static int DetectFlowTestParseNocase10 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "FROM_SERVER,STATELESS");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "FROM_SERVER,STATELESS", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -786,8 +790,8 @@ static int DetectFlowTestParseNocase10 (void)
  */
 static int DetectFlowTestParseNocase11 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, " FROM_SERVER , STATELESS ");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, " FROM_SERVER , STATELESS ", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_STATELESS &&
         fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
@@ -801,8 +805,8 @@ static int DetectFlowTestParseNocase11 (void)
  */
 static int DetectFlowTestParse12 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server:stateless");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_server:stateless", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -812,8 +816,8 @@ static int DetectFlowTestParse12 (void)
  */
 static int DetectFlowTestParse13 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "invalidoptiontest");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "invalidoptiontest", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -823,8 +827,8 @@ static int DetectFlowTestParse13 (void)
  */
 static int DetectFlowTestParse14 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -834,8 +838,8 @@ static int DetectFlowTestParse14 (void)
  */
 static int DetectFlowTestParse15 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "established,stateless");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "established,stateless", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -845,8 +849,8 @@ static int DetectFlowTestParse15 (void)
  */
 static int DetectFlowTestParse16 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "to_client,to_server");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "to_client,to_server", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -857,8 +861,8 @@ static int DetectFlowTestParse16 (void)
  */
 static int DetectFlowTestParse17 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "to_client,from_server");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "to_client,from_server", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -868,13 +872,13 @@ static int DetectFlowTestParse17 (void)
  */
 static int DetectFlowTestParse18 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server,established,only_stream");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd =
+            DetectFlowParse(NULL, "from_server,established,only_stream", &parsed_flags);
     FAIL_IF_NULL(fd);
-    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED &&
-        fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
-        fd->flags & DETECT_FLOW_FLAG_ONLYSTREAM &&
-        fd->match_cnt == 3);
+    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED && fd->flags & DETECT_FLOW_FLAG_TOCLIENT);
+    FAIL_IF_NOT(parsed_flags == DETECT_FLOW_FLAG_ONLYSTREAM);
+    FAIL_IF_NOT(fd->match_cnt == 2);
     DetectFlowFree(NULL, fd);
     PASS;
 }
@@ -884,13 +888,13 @@ static int DetectFlowTestParse18 (void)
  */
 static int DetectFlowTestParseNocase18 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "FROM_SERVER,ESTABLISHED,ONLY_STREAM");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd =
+            DetectFlowParse(NULL, "FROM_SERVER,ESTABLISHED,ONLY_STREAM", &parsed_flags);
     FAIL_IF_NULL(fd);
-    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED &&
-        fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
-        fd->flags & DETECT_FLOW_FLAG_ONLYSTREAM &&
-        fd->match_cnt == 3);
+    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED && fd->flags & DETECT_FLOW_FLAG_TOCLIENT);
+    FAIL_IF_NOT(parsed_flags == DETECT_FLOW_FLAG_ONLYSTREAM);
+    FAIL_IF_NOT(fd->match_cnt == 2);
     DetectFlowFree(NULL, fd);
     PASS;
 }
@@ -901,8 +905,9 @@ static int DetectFlowTestParseNocase18 (void)
  */
 static int DetectFlowTestParse19 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server,established,only_stream,a");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd =
+            DetectFlowParse(NULL, "from_server,established,only_stream,a", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -912,13 +917,12 @@ static int DetectFlowTestParse19 (void)
  */
 static int DetectFlowTestParse20 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server,established,no_stream");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_server,established,no_stream", &parsed_flags);
     FAIL_IF_NULL(fd);
-    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED &&
-        fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
-        fd->flags & DETECT_FLOW_FLAG_NOSTREAM &&
-        fd->match_cnt == 3);
+    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED && fd->flags & DETECT_FLOW_FLAG_TOCLIENT);
+    FAIL_IF_NOT(parsed_flags == DETECT_FLOW_FLAG_NOSTREAM);
+    FAIL_IF_NOT(fd->match_cnt == 2);
     DetectFlowFree(NULL, fd);
     PASS;
 }
@@ -928,13 +932,12 @@ static int DetectFlowTestParse20 (void)
  */
 static int DetectFlowTestParseNocase20 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "FROM_SERVER,ESTABLISHED,NO_STREAM");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "FROM_SERVER,ESTABLISHED,NO_STREAM", &parsed_flags);
     FAIL_IF_NULL(fd);
-    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED &&
-        fd->flags & DETECT_FLOW_FLAG_TOCLIENT &&
-        fd->flags & DETECT_FLOW_FLAG_NOSTREAM &&
-        fd->match_cnt == 3);
+    FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ESTABLISHED && fd->flags & DETECT_FLOW_FLAG_TOCLIENT);
+    FAIL_IF_NOT(parsed_flags == DETECT_FLOW_FLAG_NOSTREAM);
+    FAIL_IF_NOT(fd->match_cnt == 2);
     DetectFlowFree(NULL, fd);
     PASS;
 }
@@ -944,8 +947,8 @@ static int DetectFlowTestParseNocase20 (void)
  */
 static int DetectFlowTestParse21 (void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "from_server,a,no_stream");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "from_server,a,no_stream", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -955,10 +958,10 @@ static int DetectFlowTestParse21 (void)
  */
 static int DetectFlowTestParse22(void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "established,not_established");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "established,not_established", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
-    fd = DetectFlowParse(NULL, "not_established,established");
+    fd = DetectFlowParse(NULL, "not_established,established", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -1005,8 +1008,8 @@ static int DetectFlowSigTest01(void)
  */
 static int DetectFlowTestParseNotEstablished(void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "not_established");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "not_established", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_NOT_ESTABLISHED);
     DetectFlowFree(NULL, fd);
@@ -1018,8 +1021,8 @@ static int DetectFlowTestParseNotEstablished(void)
  */
 static int DetectFlowTestParseNoFrag(void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "no_frag");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "no_frag", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_NO_FRAG);
     DetectFlowFree(NULL, fd);
@@ -1031,8 +1034,8 @@ static int DetectFlowTestParseNoFrag(void)
  */
 static int DetectFlowTestParseOnlyFrag(void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "only_frag");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "only_frag", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ONLY_FRAG);
     DetectFlowFree(NULL, fd);
@@ -1044,8 +1047,8 @@ static int DetectFlowTestParseOnlyFrag(void)
  */
 static int DetectFlowTestParseNoFragOnlyFrag(void)
 {
-    DetectFlowData *fd = NULL;
-    fd = DetectFlowParse(NULL, "no_frag,only_frag");
+    uint16_t parsed_flags = 0;
+    DetectFlowData *fd = DetectFlowParse(NULL, "no_frag,only_frag", &parsed_flags);
     FAIL_IF_NOT_NULL(fd);
     PASS;
 }
@@ -1055,14 +1058,15 @@ static int DetectFlowTestParseNoFragOnlyFrag(void)
  */
 static int DetectFlowTestNoFragMatch(void)
 {
+    uint16_t parsed_flags = 0;
     uint32_t pflags = 0;
-    DetectFlowData *fd = DetectFlowParse(NULL, "no_frag");
+    DetectFlowData *fd = DetectFlowParse(NULL, "no_frag", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_NO_FRAG);
     FAIL_IF_NOT(fd->match_cnt == 1);
-    FAIL_IF_NOT(FlowMatch(pflags, 0, 0, fd->flags, fd->match_cnt));
+    FAIL_IF_NOT(FlowMatch(pflags, 0, fd->flags, fd->match_cnt));
     pflags |= PKT_REBUILT_FRAGMENT;
-    FAIL_IF(FlowMatch(pflags, 0, 0, fd->flags, fd->match_cnt));
+    FAIL_IF(FlowMatch(pflags, 0, fd->flags, fd->match_cnt));
     PASS;
 }
 
@@ -1071,14 +1075,15 @@ static int DetectFlowTestNoFragMatch(void)
  */
 static int DetectFlowTestOnlyFragMatch(void)
 {
+    uint16_t parsed_flags = 0;
     uint32_t pflags = 0;
-    DetectFlowData *fd = DetectFlowParse(NULL, "only_frag");
+    DetectFlowData *fd = DetectFlowParse(NULL, "only_frag", &parsed_flags);
     FAIL_IF_NULL(fd);
     FAIL_IF_NOT(fd->flags & DETECT_FLOW_FLAG_ONLY_FRAG);
     FAIL_IF_NOT(fd->match_cnt == 1);
-    FAIL_IF(FlowMatch(pflags, 0, 0, fd->flags, fd->match_cnt));
+    FAIL_IF(FlowMatch(pflags, 0, fd->flags, fd->match_cnt));
     pflags |= PKT_REBUILT_FRAGMENT;
-    FAIL_IF_NOT(FlowMatch(pflags, 0, 0, fd->flags, fd->match_cnt));
+    FAIL_IF_NOT(FlowMatch(pflags, 0, fd->flags, fd->match_cnt));
     PASS;
 }
 
diff --git a/src/detect.h b/src/detect.h
index 04dd49a65a75..882dc93bcd5e 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -246,6 +246,10 @@ typedef struct DetectPort_ {
 
 #define SIG_FLAG_FLUSH                  BIT_U32(12) /**< detection logic needs stream flush notification */
 
+#define SIG_FLAG_REQUIRE_STREAM_ONLY                                                               \
+    BIT_U32(13) /**< signature is requiring stream match. Stream match is not optional, so no      \
+                   fallback to packet payload. */
+
 // vacancies
 
 #define SIG_FLAG_REQUIRE_FLOWVAR        BIT_U32(17) /**< signature can only match if a flowbit, flowvar or flowint is available. */

From 4a079541b2cb55bb9582d011cac9027dee0eb825 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 14 Nov 2023 06:44:11 +0100
Subject: [PATCH 122/462] detect: fix inspect engine return codes

Use proper inspect engine codes instead of bool.
---
 src/detect-engine.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 4059ffaf5261..edabd0b0a313 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -1928,7 +1928,7 @@ static int DetectEngineInspectRulePacketMatches(
         if (sigmatch_table[smd->type].Match(det_ctx, p, s, smd->ctx) <= 0) {
             KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
             SCLogDebug("no match");
-            return false;
+            return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
         }
         KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
         if (smd->is_last) {
@@ -1937,7 +1937,7 @@ static int DetectEngineInspectRulePacketMatches(
         }
         smd++;
     }
-    return true;
+    return DETECT_ENGINE_INSPECT_SIG_MATCH;
 }
 
 static int DetectEngineInspectRulePayloadMatches(
@@ -1968,22 +1968,22 @@ static int DetectEngineInspectRulePayloadMatches(
             /* skip if we don't have to inspect the packet and segment was
              * added to stream */
             if (!(s->flags & SIG_FLAG_REQUIRE_PACKET) && (p->flags & PKT_STREAM_ADD)) {
-                return false;
+                return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
             }
             if (s->flags & SIG_FLAG_REQUIRE_STREAM_ONLY) {
                 SCLogDebug("SIG_FLAG_REQUIRE_STREAM_ONLY, so no match");
-                return false;
+                return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
             }
             if (DetectEngineInspectPacketPayload(de_ctx, det_ctx, s, p->flow, p) != 1) {
-                return false;
+                return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
             }
         }
     } else {
         if (DetectEngineInspectPacketPayload(de_ctx, det_ctx, s, p->flow, p) != 1) {
-            return false;
+            return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
         }
     }
-    return true;
+    return DETECT_ENGINE_INSPECT_SIG_MATCH;
 }
 
 bool DetectEnginePktInspectionRun(ThreadVars *tv,
@@ -1994,8 +1994,8 @@ bool DetectEnginePktInspectionRun(ThreadVars *tv,
     SCEnter();
 
     for (DetectEnginePktInspectionEngine *e = s->pkt_inspect; e != NULL; e = e->next) {
-        if (e->v1.Callback(det_ctx, e, s, p, alert_flags) == false) {
-            SCLogDebug("sid %u: e %p Callback returned false", s->id, e);
+        if (e->v1.Callback(det_ctx, e, s, p, alert_flags) != DETECT_ENGINE_INSPECT_SIG_MATCH) {
+            SCLogDebug("sid %u: e %p Callback returned no match", s->id, e);
             return false;
         }
         SCLogDebug("sid %u: e %p Callback returned true", s->id, e);

From 3cad7cfa56f34ddfbd96867f7fd20aec297535ee Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 16 Nov 2023 12:37:08 +0100
Subject: [PATCH 123/462] unittests: free packet using PacketFree

Update SigTest17 which left a dangling pointer.
---
 src/tests/detect.c         |  3 ---
 src/util-unittest-helper.c | 23 +----------------------
 2 files changed, 1 insertion(+), 25 deletions(-)

diff --git a/src/tests/detect.c b/src/tests/detect.c
index 302666ac853a..37dc2cfe9bf7 100644
--- a/src/tests/detect.c
+++ b/src/tests/detect.c
@@ -1064,14 +1064,11 @@ static int SigTest17 (void)
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p);
 
     uint32_t capid = VarNameStoreLookupByName("http_host", VAR_TYPE_PKT_VAR);
-
     PktVar *pv_hn = PktVarGet(p, capid);
     FAIL_IF_NULL(pv_hn);
-
     FAIL_IF(pv_hn->value_len != 15);
     FAIL_IF_NOT(memcmp(pv_hn->value, "one.example.org", pv_hn->value_len) == 0);
 
-    PktVarFree(pv_hn);
     DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
     DetectEngineCtxFree(de_ctx);
     ConfDeInit();
diff --git a/src/util-unittest-helper.c b/src/util-unittest-helper.c
index 80356cf82e2b..48d2a045c19b 100644
--- a/src/util-unittest-helper.c
+++ b/src/util-unittest-helper.c
@@ -486,28 +486,7 @@ void UTHFreePacket(Packet *p)
 {
     if (p == NULL)
         return;
-#if 0 // VJ we now use one buffer
-    switch (p->proto) {
-        case IPPROTO_UDP:
-            if (p->udph != NULL)
-                SCFree(p->udph);
-            if (p->ip4h != NULL)
-                SCFree(p->ip4h);
-        break;
-        case IPPROTO_TCP:
-            if (p->tcph != NULL)
-                SCFree(p->tcph);
-            if (p->ip4h != NULL)
-                SCFree(p->ip4h);
-        break;
-        case IPPROTO_ICMP:
-            if (p->ip4h != NULL)
-                SCFree(p->ip4h);
-        break;
-        /* TODO: Add more protocols */
-    }
-#endif
-    SCFree(p);
+    PacketFree(p);
 }
 
 void UTHAssignFlow(Packet *p, Flow *f)

From 4a02a14df1be3821042b1c60e3722b114d26fa14 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 15 Nov 2023 10:13:14 +0100
Subject: [PATCH 124/462] doc/userguide: document host table yaml settings

---
 doc/userguide/configuration/suricata-yaml.rst | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index c19ed48b3d0e..0b39705d896b 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -1256,6 +1256,37 @@ network inspection.
 
 .. image:: suricata-yaml/IDS_chunk_size.png
 
+
+Host Tracking
+-------------
+
+.. _suricata-yaml-host-settings:
+
+
+The Host table is used for tracking per IP address. This is used for tracking
+per IP thresholding, per IP tagging, storing `iprep` data and storing `hostbit`.
+
+Settings
+~~~~~~~~
+
+The configuration allows specifying the following settings: `hash-size`,  `prealloc` and `memcap`.
+
+.. code-block:: yaml
+
+    host:
+      hash-size: 4096
+      prealloc: 1000
+      memcap: 32mb
+
+* `hash-size`: size of the hash table in number of rows
+* `prealloc`: number of `Host` objects preallocated for efficiency
+* `memcap`: max memory use for hosts, including the hash table size
+
+Hosts are evicted from the hash table by the Flow Manager thread when all
+data in the host is expired (tag, threshold, etc). Hosts with iprep will
+not expire.
+
+
 Application Layer Parsers
 -------------------------
 

From 6b2c33990f2c61643d94c74396f930e465305b38 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 15 Nov 2023 09:11:32 +0100
Subject: [PATCH 125/462] doc/userguide: add tag keyword page

Ticket: #3015.
---
 doc/userguide/rules/index.rst |   3 +-
 doc/userguide/rules/tag.rst   | 133 ++++++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+), 1 deletion(-)
 create mode 100644 doc/userguide/rules/tag.rst

diff --git a/doc/userguide/rules/index.rst b/doc/userguide/rules/index.rst
index e0d4378dfa5b..76266b334581 100644
--- a/doc/userguide/rules/index.rst
+++ b/doc/userguide/rules/index.rst
@@ -42,4 +42,5 @@ Suricata Rules
    datasets
    lua-detection
    differences-from-snort
-   multi-buffer-matching
\ No newline at end of file
+   multi-buffer-matching
+   tag
diff --git a/doc/userguide/rules/tag.rst b/doc/userguide/rules/tag.rst
new file mode 100644
index 000000000000..1b057cb6bd81
--- /dev/null
+++ b/doc/userguide/rules/tag.rst
@@ -0,0 +1,133 @@
+Tag
+===
+
+The `tag` keyword allows tagging of the current and future packets.
+
+Tagged packets can be logged in `EVE` and conditional PCAP logging.
+
+Tagging is limited to a scope: `host` or `session` (flow). When using `host` a
+direction can be specified: `src` or `dst`. Tagging will then occur based on the
+`src` or `dst` IP address of the packet generating the alert.
+
+Tagging is further controlled by count: `packets`, `bytes` or `seconds`. If the
+count is ommited built-in defaults will be used:
+
+- for `session`: 256 packets
+- for `host`: 256 packets for the destination IP of the packet triggering the alert
+
+The `tag` keyword can appear multiple times in a rule.
+
+Syntax
+~~~~~~
+
+::
+
+    tag:<scope>[,<count>, <metric>[,<direction>]];
+
+Values for `scope`: `session` and `host`
+Values for `metric`: `packets`, `bytes`, `seconds`
+Values for `direction`: `src` and `dst`
+
+.. note:: "direction" can only be specified if scope is "host" and both "count"
+   and "metric" are also specified.
+
+Examples
+~~~~~~~~
+
+Keyword::
+
+    tag:session;                # tags next 256 packets in the flow
+    tag:host;                   # tags next 256 packets for the dst ip of the alert
+    tag:host,100,packets,src;   # tags next 100 packets for src ip of the alert
+    tag:host,3600,seconds,dst;  # tags packets for dst host for the next hour
+
+Full rule examples:
+
+.. container:: example-rule
+
+   alert dns any any -> any any (dns.query; content:"evil"; tag:host,60,seconds,src; sid:1;)
+
+.. container:: example-rule
+
+   alert http any any -> any any (http.method; content:"POST"; tag:session; sid:1;)
+
+How to Use Tags
+~~~~~~~~~~~~~~~
+
+EVE
+"""
+
+Tags can be set to generate `EVE` `tag` records:
+
+.. code-block:: yaml
+
+    outputs:
+      - eve-log:
+          enabled: yes
+          filename: eve.json
+          types:
+            - alert:
+                tagged-packets: true
+
+The tagged packets will then be logged with `event_type`: `packet`:
+
+.. code-block:: json
+
+    {
+      "timestamp": "2020-06-03T10:29:17.850417+0000",
+      "flow_id": 1576832511820424,
+      "event_type": "packet",
+      "src_ip": "192.168.0.27",
+      "src_port": 54634,
+      "dest_ip": "192.168.0.103",
+      "dest_port": 22,
+      "proto": "TCP",
+      "pkt_src": "wire/pcap",
+      "packet": "CAAn6mWJAPSNvfrHCABFAAAogkVAAIAG9rfAqAAbwKgAZ9VqABZvnJXH5Zf6aFAQEAljEwAAAAAAAAAA",
+      "packet_info": {
+        "linktype": 1
+      }
+    }
+
+EVE: :ref:`Eve JSON Output <eve-json-output>`
+
+Conditional PCAP Logging
+""""""""""""""""""""""""
+
+Using the conditional PCAP logging option the tag keyword can control which
+packets are logged by the PCAP logging.
+
+.. code-block:: yaml
+
+    outputs:
+      - pcap-log:
+          enabled: yes
+          filename: log.pcap
+          limit: 1000mb
+          max-files: 2000
+          compression: none
+          mode: normal
+          use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets
+          honor-pass-rules: no # If set to "yes", flows in which a pass rule matched will stop being logged.
+          # Use "all" to log all packets or use "alerts" to log only alerted packets and flows or "tag"
+          # to log only flow tagged via the "tag" keyword
+          conditional: tag
+
+PCAP Logging: :ref:`PCAP log <suricata_yaml_pcap_log>`
+
+Tracking by Host/Flow
+~~~~~~~~~~~~~~~~~~~~~
+
+When the tags are using the `session` scope, the tag is added to the
+`Flow` structure. If a packet has no flow, no tagging will happen. No
+errors/warnings are generated for this.
+
+See :ref:`Flow Settings <suricata-yaml-flow-settings>` for managing flow
+limits and resources.
+
+When tags are using the `host` scope, the tag is stored with a `Host`
+object in the host table. The Host table size will affect effectiveness
+of per host tags.
+
+See :ref:`Host Settings <suricata-yaml-host-settings>` for managing host
+table size.

From 3b826fff6840b24e54a086606a14185745b6d3a3 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 15 Nov 2023 19:18:08 +0100
Subject: [PATCH 126/462] detect/tag: reuse result of previous host lookup

Minor optimization that could lead to a reduction in host table
lookups if more than one host feature is in use.
---
 src/detect-engine-tag.c | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/src/detect-engine-tag.c b/src/detect-engine-tag.c
index aceb5f10c6f4..e6a4134048e4 100644
--- a/src/detect-engine-tag.c
+++ b/src/detect-engine-tag.c
@@ -493,6 +493,26 @@ static void TagHandlePacketHost(Host *host, Packet *p)
     }
 }
 
+static Host *GetLockedSrcHost(Packet *p)
+{
+    if (p->host_src == NULL) {
+        p->host_src = HostLookupHostFromHash(&p->src);
+    } else {
+        HostLock(p->host_src);
+    }
+    return p->host_src;
+}
+
+static Host *GetLockedDstHost(Packet *p)
+{
+    if (p->host_dst == NULL) {
+        p->host_dst = HostLookupHostFromHash(&p->dst);
+    } else {
+        HostLock(p->host_dst);
+    }
+    return p->host_dst;
+}
+
 /**
  * \brief Search tags for src and dst. Update entries of the tag, remove if necessary
  *
@@ -516,20 +536,22 @@ void TagHandlePacket(DetectEngineCtx *de_ctx,
         TagHandlePacketFlow(p->flow, p);
     }
 
-    Host *src = HostLookupHostFromHash(&p->src);
-    if (src) {
+    Host *src = GetLockedSrcHost(p);
+    if (src != NULL) {
         if (TagHostHasTag(src)) {
-            TagHandlePacketHost(src,p);
+            TagHandlePacketHost(src, p);
         }
-        HostRelease(src);
+        HostUnlock(src);
     }
-    Host *dst = HostLookupHostFromHash(&p->dst);
-    if (dst) {
+
+    Host *dst = GetLockedDstHost(p);
+    if (dst != NULL) {
         if (TagHostHasTag(dst)) {
-            TagHandlePacketHost(dst,p);
+            TagHandlePacketHost(dst, p);
         }
-        HostRelease(dst);
+        HostUnlock(dst);
     }
+
     SCReturn;
 }
 

From 68a2fcaad3abcd503246feca730dc2da1ff91af2 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 20 Sep 2023 10:46:23 +0200
Subject: [PATCH 127/462] mpm: thread ctx cleanups

Remove unused thread ctx' from AC variants

Use single thread store in detection.

Minor cleanups.
---
 src/app-layer-detect-proto.c        |   4 +-
 src/app-layer-ftp.c                 |   2 +-
 src/app-layer-smtp.c                |   2 +-
 src/detect-dns-query.c              |   5 +-
 src/detect-engine-frame.c           |   4 +-
 src/detect-engine-mpm.c             |   3 +-
 src/detect-engine-payload.c         |   5 +-
 src/detect-engine-prefilter.c       |   8 +-
 src/detect-engine.c                 |   5 --
 src/detect-file-data.c              |   3 +-
 src/detect-filemagic.c              |   3 +-
 src/detect-filename.c               |   3 +-
 src/detect-http-client-body.c       |   2 +-
 src/detect-http-header.c            |   8 +-
 src/detect-http-raw-header.c        |   4 +-
 src/detect-http2.c                  |   5 +-
 src/detect-ike-vendor.c             |   2 +-
 src/detect-krb5-cname.c             |   5 +-
 src/detect-krb5-sname.c             |   5 +-
 src/detect-mqtt-subscribe-topic.c   |   5 +-
 src/detect-mqtt-unsubscribe-topic.c |   5 +-
 src/detect-quic-cyu-hash.c          |   2 +-
 src/detect-quic-cyu-string.c        |   2 +-
 src/detect-tls-certs.c              |   5 +-
 src/detect.h                        |   7 +-
 src/util-mpm-ac-bs.c                | 126 +---------------------------
 src/util-mpm-ac-bs.h                |   7 --
 src/util-mpm-ac-ks.c                | 115 +------------------------
 src/util-mpm-ac-ks.h                |   8 --
 src/util-mpm-ac.c                   | 123 +--------------------------
 src/util-mpm-ac.h                   |   7 --
 src/util-mpm.c                      |  11 ++-
 src/util-mpm.h                      |   1 +
 33 files changed, 61 insertions(+), 441 deletions(-)

diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index c7f902edc22f..77f3c648c0da 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -1998,7 +1998,7 @@ AppLayerProtoDetectThreadCtx *AppLayerProtoDetectGetCtxThread(void)
         for (j = 0; j < 2; j++) {
             mpm_ctx = &alpd_ctx.ctx_ipp[i].ctx_pm[j].mpm_ctx;
             mpm_tctx = &alpd_tctx->mpm_tctx[i][j];
-            mpm_table[mpm_ctx->mpm_type].InitThreadCtx(mpm_ctx, mpm_tctx);
+            MpmInitThreadCtx(mpm_tctx, mpm_ctx->mpm_type);
         }
     }
 
@@ -2028,7 +2028,7 @@ void AppLayerProtoDetectDestroyCtxThread(AppLayerProtoDetectThreadCtx *alpd_tctx
         for (dir = 0; dir < 2; dir++) {
             mpm_ctx = &alpd_ctx.ctx_ipp[ipproto_map].ctx_pm[dir].mpm_ctx;
             mpm_tctx = &alpd_tctx->mpm_tctx[ipproto_map][dir];
-            mpm_table[mpm_ctx->mpm_type].DestroyThreadCtx(mpm_ctx, mpm_tctx);
+            MpmDestroyThreadCtx(mpm_tctx, mpm_ctx->mpm_type);
         }
     }
     PmqFree(&alpd_tctx->pmq);
diff --git a/src/app-layer-ftp.c b/src/app-layer-ftp.c
index 3db448279073..c0a815e31dae 100644
--- a/src/app-layer-ftp.c
+++ b/src/app-layer-ftp.c
@@ -277,7 +277,7 @@ static void FTPLocalStorageFree(void *ptr)
         }
 
         if (td->ftp_mpm_thread_ctx != NULL) {
-            mpm_table[FTP_MPM].DestroyThreadCtx(ftp_mpm_ctx, td->ftp_mpm_thread_ctx);
+            MpmDestroyThreadCtx(td->ftp_mpm_thread_ctx, FTP_MPM);
             SCFree(td->ftp_mpm_thread_ctx);
         }
 
diff --git a/src/app-layer-smtp.c b/src/app-layer-smtp.c
index bf93c4517877..7b921324ae8e 100644
--- a/src/app-layer-smtp.c
+++ b/src/app-layer-smtp.c
@@ -1590,7 +1590,7 @@ static void SMTPLocalStorageFree(void *ptr)
         }
 
         if (td->smtp_mpm_thread_ctx != NULL) {
-            mpm_table[SMTP_MPM].DestroyThreadCtx(smtp_mpm_ctx, td->smtp_mpm_thread_ctx);
+            MpmDestroyThreadCtx(td->smtp_mpm_thread_ctx, SMTP_MPM);
             SCFree(td->smtp_mpm_thread_ctx);
         }
 
diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index 354c4f834411..fd2c7450853e 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -165,9 +165,8 @@ static void PrefilterTxDnsQuery(DetectEngineThreadCtx *det_ctx, const void *pect
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-engine-frame.c b/src/detect-engine-frame.c
index 2d2b13ed396b..722263d45390 100644
--- a/src/detect-engine-frame.c
+++ b/src/detect-engine-frame.c
@@ -126,7 +126,7 @@ static int FrameStreamDataPrefilterFunc(
         // PrintRawDataFp(stdout, data, data_len);
 
         (void)mpm_table[mpm_ctx->mpm_type].Search(
-                mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
         SCLogDebug("det_ctx->pmq.rule_id_array_cnt %u", det_ctx->pmq.rule_id_array_cnt);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
     }
@@ -167,7 +167,7 @@ static void PrefilterMpmFrame(DetectEngineThreadCtx *det_ctx, const void *pectx,
 
         if (data != NULL && data_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
             SCLogDebug("det_ctx->pmq.rule_id_array_cnt %u", det_ctx->pmq.rule_id_array_cnt);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
         }
diff --git a/src/detect-engine-mpm.c b/src/detect-engine-mpm.c
index f091a3dadaa0..48c4da115a45 100644
--- a/src/detect-engine-mpm.c
+++ b/src/detect-engine-mpm.c
@@ -895,8 +895,7 @@ void PatternMatchThreadPrint(MpmThreadCtx *mpm_thread_ctx, uint16_t mpm_matcher)
 void PatternMatchThreadDestroy(MpmThreadCtx *mpm_thread_ctx, uint16_t mpm_matcher)
 {
     SCLogDebug("mpm_thread_ctx %p, mpm_matcher %"PRIu16"", mpm_thread_ctx, mpm_matcher);
-    if (mpm_table[mpm_matcher].DestroyThreadCtx != NULL)
-        mpm_table[mpm_matcher].DestroyThreadCtx(NULL, mpm_thread_ctx);
+    MpmDestroyThreadCtx(mpm_thread_ctx, mpm_matcher);
 }
 void PatternMatchThreadPrepare(MpmThreadCtx *mpm_thread_ctx, uint16_t mpm_matcher)
 {
diff --git a/src/detect-engine-payload.c b/src/detect-engine-payload.c
index 5ce63109ac0a..ef92e68629f8 100644
--- a/src/detect-engine-payload.c
+++ b/src/detect-engine-payload.c
@@ -64,9 +64,8 @@ static int StreamMpmFunc(
         smd->det_ctx->stream_mpm_cnt++;
         smd->det_ctx->stream_mpm_size += data_len;
 #endif
-        (void)mpm_table[smd->mpm_ctx->mpm_type].Search(smd->mpm_ctx,
-                &smd->det_ctx->mtcs, &smd->det_ctx->pmq,
-                data, data_len);
+        (void)mpm_table[smd->mpm_ctx->mpm_type].Search(
+                smd->mpm_ctx, &smd->det_ctx->mtc, &smd->det_ctx->pmq, data, data_len);
         PREFILTER_PROFILING_ADD_BYTES(smd->det_ctx, data_len);
     }
     return 0;
diff --git a/src/detect-engine-prefilter.c b/src/detect-engine-prefilter.c
index fd1b691ee2d9..3c33071e7211 100644
--- a/src/detect-engine-prefilter.c
+++ b/src/detect-engine-prefilter.c
@@ -731,8 +731,8 @@ static void PrefilterMpm(DetectEngineThreadCtx *det_ctx, const void *pectx, Pack
     //PrintRawDataFp(stdout, data, data_len);
 
     if (data != NULL && data_len >= mpm_ctx->minlen) {
-        (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+        (void)mpm_table[mpm_ctx->mpm_type].Search(
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
     }
 }
@@ -801,8 +801,8 @@ static void PrefilterMpmPkt(DetectEngineThreadCtx *det_ctx,
     //PrintRawDataFp(stdout, data, data_len);
 
     if (data != NULL && data_len >= mpm_ctx->minlen) {
-        (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+        (void)mpm_table[mpm_ctx->mpm_type].Search(
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
     }
 }
diff --git a/src/detect-engine.c b/src/detect-engine.c
index edabd0b0a313..6fa894c0c794 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -3211,8 +3211,6 @@ static TmEcode DetectEngineThreadCtxInitForMT(ThreadVars *tv, DetectEngineThread
 static TmEcode ThreadCtxDoInit (DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx)
 {
     PatternMatchThreadPrepare(&det_ctx->mtc, de_ctx->mpm_matcher);
-    PatternMatchThreadPrepare(&det_ctx->mtcs, de_ctx->mpm_matcher);
-    PatternMatchThreadPrepare(&det_ctx->mtcu, de_ctx->mpm_matcher);
 
     PmqSetup(&det_ctx->pmq);
 
@@ -3463,8 +3461,6 @@ static void DetectEngineThreadCtxFree(DetectEngineThreadCtx *det_ctx)
     /** \todo get rid of this static */
     if (det_ctx->de_ctx != NULL) {
         PatternMatchThreadDestroy(&det_ctx->mtc, det_ctx->de_ctx->mpm_matcher);
-        PatternMatchThreadDestroy(&det_ctx->mtcs, det_ctx->de_ctx->mpm_matcher);
-        PatternMatchThreadDestroy(&det_ctx->mtcu, det_ctx->de_ctx->mpm_matcher);
     }
 
     PmqFree(&det_ctx->pmq);
@@ -3552,7 +3548,6 @@ void DetectEngineThreadCtxInfo(ThreadVars *t, DetectEngineThreadCtx *det_ctx)
 {
     /* XXX */
     PatternMatchThreadPrint(&det_ctx->mtc, det_ctx->de_ctx->mpm_matcher);
-    PatternMatchThreadPrint(&det_ctx->mtcu, det_ctx->de_ctx->mpm_matcher);
 }
 
 static uint32_t DetectKeywordCtxHashFunc(HashListTable *ht, void *data, uint16_t datalen)
diff --git a/src/detect-file-data.c b/src/detect-file-data.c
index e26654e8b9e8..1f162d0d5ce8 100644
--- a/src/detect-file-data.c
+++ b/src/detect-file-data.c
@@ -471,8 +471,7 @@ static void PrefilterTxFiledata(DetectEngineThreadCtx *det_ctx, const void *pect
 
             if (buffer->inspect_len >= mpm_ctx->minlen) {
                 uint32_t prev_rule_id_array_cnt = det_ctx->pmq.rule_id_array_cnt;
-                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                        &det_ctx->mtcu, &det_ctx->pmq,
+                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx, &det_ctx->mtc, &det_ctx->pmq,
                         buffer->inspect, buffer->inspect_len);
                 PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
 
diff --git a/src/detect-filemagic.c b/src/detect-filemagic.c
index b1c53593dced..d816b8c53dfe 100644
--- a/src/detect-filemagic.c
+++ b/src/detect-filemagic.c
@@ -376,8 +376,7 @@ static void PrefilterTxFilemagic(DetectEngineThreadCtx *det_ctx, const void *pec
                 continue;
 
             if (buffer->inspect_len >= mpm_ctx->minlen) {
-                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                        &det_ctx->mtcu, &det_ctx->pmq,
+                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx, &det_ctx->mtc, &det_ctx->pmq,
                         buffer->inspect, buffer->inspect_len);
                 PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
             }
diff --git a/src/detect-filename.c b/src/detect-filename.c
index fc6e2fc7189e..5eb446af5134 100644
--- a/src/detect-filename.c
+++ b/src/detect-filename.c
@@ -313,8 +313,7 @@ static void PrefilterTxFilename(DetectEngineThreadCtx *det_ctx, const void *pect
                 continue;
 
             if (buffer->inspect_len >= mpm_ctx->minlen) {
-                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                        &det_ctx->mtcu, &det_ctx->pmq,
+                (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx, &det_ctx->mtc, &det_ctx->pmq,
                         buffer->inspect, buffer->inspect_len);
                 PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
             }
diff --git a/src/detect-http-client-body.c b/src/detect-http-client-body.c
index 41b2552e9b99..32c407a00aeb 100644
--- a/src/detect-http-client-body.c
+++ b/src/detect-http-client-body.c
@@ -373,7 +373,7 @@ static void PrefilterTxHttpRequestBody(DetectEngineThreadCtx *det_ctx, const voi
 
     if (buffer->inspect_len >= mpm_ctx->minlen) {
         (void)mpm_table[mpm_ctx->mpm_type].Search(
-                mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
     }
 }
diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index e5101f9276b0..9d4b187a9f25 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -268,8 +268,8 @@ static void PrefilterMpmHttpHeader(DetectEngineThreadCtx *det_ctx, const void *p
     //PrintRawDataFp(stdout, data, data_len);
 
     if (data != NULL && data_len >= mpm_ctx->minlen) {
-        (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+        (void)mpm_table[mpm_ctx->mpm_type].Search(
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
     }
 }
@@ -520,7 +520,7 @@ static void PrefilterTxHttp2Header(DetectEngineThreadCtx *det_ctx, const void *p
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
@@ -658,7 +658,7 @@ static void PrefilterTxHttp1Header(DetectEngineThreadCtx *det_ctx, const void *p
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-http-raw-header.c b/src/detect-http-raw-header.c
index 946c2233e5c2..1494f02d22d7 100644
--- a/src/detect-http-raw-header.c
+++ b/src/detect-http-raw-header.c
@@ -262,8 +262,8 @@ static void PrefilterMpmHttpHeaderRaw(DetectEngineThreadCtx *det_ctx, const void
     //PrintRawDataFp(stdout, data, data_len);
 
     if (data != NULL && data_len >= mpm_ctx->minlen) {
-        (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                &det_ctx->mtcu, &det_ctx->pmq, data, data_len);
+        (void)mpm_table[mpm_ctx->mpm_type].Search(
+                mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, data, data_len);
         PREFILTER_PROFILING_ADD_BYTES(det_ctx, data_len);
     }
 }
diff --git a/src/detect-http2.c b/src/detect-http2.c
index 40cbe3e3a78e..ec4840afe306 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -687,9 +687,8 @@ static void PrefilterTxHttp2HName(DetectEngineThreadCtx *det_ctx, const void *pe
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-ike-vendor.c b/src/detect-ike-vendor.c
index 54418e0fe01a..1af41bac23d8 100644
--- a/src/detect-ike-vendor.c
+++ b/src/detect-ike-vendor.c
@@ -105,7 +105,7 @@ static void PrefilterTxIkeVendor(DetectEngineThreadCtx *det_ctx, const void *pec
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
         local_id++;
diff --git a/src/detect-krb5-cname.c b/src/detect-krb5-cname.c
index d6f653beed18..632df0ea5dd8 100644
--- a/src/detect-krb5-cname.c
+++ b/src/detect-krb5-cname.c
@@ -157,9 +157,8 @@ static void PrefilterTxKrb5CName(DetectEngineThreadCtx *det_ctx, const void *pec
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index e4ccc6c2432e..19d3c6716116 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -157,9 +157,8 @@ static void PrefilterTxKrb5SName(DetectEngineThreadCtx *det_ctx, const void *pec
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-mqtt-subscribe-topic.c b/src/detect-mqtt-subscribe-topic.c
index c2793bb13a80..258dc0b4cf6d 100644
--- a/src/detect-mqtt-subscribe-topic.c
+++ b/src/detect-mqtt-subscribe-topic.c
@@ -158,9 +158,8 @@ static void PrefilterTxMQTTSubscribeTopic(DetectEngineThreadCtx *det_ctx, const
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
         local_id++;
diff --git a/src/detect-mqtt-unsubscribe-topic.c b/src/detect-mqtt-unsubscribe-topic.c
index 0ff49ea6d0d3..2c1cb02c4234 100644
--- a/src/detect-mqtt-unsubscribe-topic.c
+++ b/src/detect-mqtt-unsubscribe-topic.c
@@ -158,9 +158,8 @@ static void PrefilterTxMQTTUnsubscribeTopic(DetectEngineThreadCtx *det_ctx, cons
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
         local_id++;
diff --git a/src/detect-quic-cyu-hash.c b/src/detect-quic-cyu-hash.c
index 8b094aaa1d61..a475a23f1e5e 100644
--- a/src/detect-quic-cyu-hash.c
+++ b/src/detect-quic-cyu-hash.c
@@ -155,7 +155,7 @@ static void PrefilterTxQuicHash(DetectEngineThreadCtx *det_ctx, const void *pect
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-quic-cyu-string.c b/src/detect-quic-cyu-string.c
index cf1164c40fde..53775d0ffc20 100644
--- a/src/detect-quic-cyu-string.c
+++ b/src/detect-quic-cyu-string.c
@@ -147,7 +147,7 @@ static void PrefilterTxQuicString(DetectEngineThreadCtx *det_ctx, const void *pe
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
             (void)mpm_table[mpm_ctx->mpm_type].Search(
-                    mpm_ctx, &det_ctx->mtcu, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index a0204377373e..cccc695c91cf 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -232,9 +232,8 @@ static void PrefilterTxTlsCerts(DetectEngineThreadCtx *det_ctx, const void *pect
             break;
 
         if (buffer->inspect_len >= mpm_ctx->minlen) {
-            (void)mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
-                    &det_ctx->mtcu, &det_ctx->pmq,
-                    buffer->inspect, buffer->inspect_len);
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
             PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
         }
 
diff --git a/src/detect.h b/src/detect.h
index 882dc93bcd5e..a4d9ef360227 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1179,12 +1179,7 @@ typedef struct DetectEngineThreadCtx_ {
     SignatureNonPrefilterStore *non_pf_store_ptr;
     uint32_t non_pf_store_cnt;
 
-    /** pointer to the current mpm ctx that is stored
-     *  in a rule group head -- can be either a content
-     *  or uricontent ctx. */
-    MpmThreadCtx mtc;   /**< thread ctx for the mpm */
-    MpmThreadCtx mtcu;  /**< thread ctx for uricontent mpm */
-    MpmThreadCtx mtcs;  /**< thread ctx for stream mpm */
+    MpmThreadCtx mtc; /**< thread ctx for the mpm */
     PrefilterRuleStore pmq;
 
     /** SPM thread context used for scanning. This has been cloned from the
diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c
index 0bc5efd4bcd4..304894b71250 100644
--- a/src/util-mpm-ac-bs.c
+++ b/src/util-mpm-ac-bs.c
@@ -63,9 +63,7 @@
 #include "util-validate.h"
 
 void SCACBSInitCtx(MpmCtx *);
-void SCACBSInitThreadCtx(MpmCtx *, MpmThreadCtx *);
 void SCACBSDestroyCtx(MpmCtx *);
-void SCACBSDestroyThreadCtx(MpmCtx *, MpmThreadCtx *);
 int SCACBSAddPatternCI(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
                        uint32_t, SigIntId, uint8_t);
 int SCACBSAddPatternCS(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
@@ -74,7 +72,6 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx);
 uint32_t SCACBSSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                       PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen);
 void SCACBSPrintInfo(MpmCtx *mpm_ctx);
-void SCACBSPrintSearchStats(MpmThreadCtx *mpm_thread_ctx);
 void SCACBSRegisterTests(void);
 
 /* a placeholder to denote a failure transition in the goto table */
@@ -98,18 +95,13 @@ void MpmACBSRegister(void)
 {
     mpm_table[MPM_AC_BS].name = "ac-bs";
     mpm_table[MPM_AC_BS].InitCtx = SCACBSInitCtx;
-    mpm_table[MPM_AC_BS].InitThreadCtx = SCACBSInitThreadCtx;
     mpm_table[MPM_AC_BS].DestroyCtx = SCACBSDestroyCtx;
-    mpm_table[MPM_AC_BS].DestroyThreadCtx = SCACBSDestroyThreadCtx;
     mpm_table[MPM_AC_BS].AddPattern = SCACBSAddPatternCS;
     mpm_table[MPM_AC_BS].AddPatternNocase = SCACBSAddPatternCI;
     mpm_table[MPM_AC_BS].Prepare = SCACBSPreparePatterns;
     mpm_table[MPM_AC_BS].Search = SCACBSSearch;
     mpm_table[MPM_AC_BS].PrintCtx = SCACBSPrintInfo;
-    mpm_table[MPM_AC_BS].PrintThreadCtx = SCACBSPrintSearchStats;
     mpm_table[MPM_AC_BS].RegisterUnittests = SCACBSRegisterTests;
-
-    return;
 }
 
 /**
@@ -948,28 +940,6 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
     return -1;
 }
 
-/**
- * \brief Init the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- * \param matchsize      We don't need this.
- */
-void SCACBSInitThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    memset(mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-
-    mpm_thread_ctx->ctx = SCMalloc(sizeof(SCACBSThreadCtx));
-    if (mpm_thread_ctx->ctx == NULL) {
-        exit(EXIT_FAILURE);
-    }
-    memset(mpm_thread_ctx->ctx, 0, sizeof(SCACBSThreadCtx));
-    mpm_thread_ctx->memory_cnt++;
-    mpm_thread_ctx->memory_size += sizeof(SCACBSThreadCtx);
-
-    return;
-}
-
 /**
  * \brief Initialize the AC context.
  *
@@ -1003,26 +973,6 @@ void SCACBSInitCtx(MpmCtx *mpm_ctx)
     SCReturn;
 }
 
-/**
- * \brief Destroy the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- */
-void SCACBSDestroyThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    SCACBSPrintSearchStats(mpm_thread_ctx);
-
-    if (mpm_thread_ctx->ctx != NULL) {
-        SCFree(mpm_thread_ctx->ctx);
-        mpm_thread_ctx->ctx = NULL;
-        mpm_thread_ctx->memory_cnt--;
-        mpm_thread_ctx->memory_size -= sizeof(SCACBSThreadCtx);
-    }
-
-    return;
-}
-
 /**
  * \brief Destroy the mpm context.
  *
@@ -1356,19 +1306,6 @@ int SCACBSAddPatternCS(MpmCtx *mpm_ctx, uint8_t *pat, uint16_t patlen,
     return MpmAddPattern(mpm_ctx, pat, patlen, offset, depth, pid, sid, flags);
 }
 
-void SCACBSPrintSearchStats(MpmThreadCtx *mpm_thread_ctx)
-{
-
-#ifdef SC_AC_BS_COUNTERS
-    SCACBSThreadCtx *ctx = (SCACBSThreadCtx *)mpm_thread_ctx->ctx;
-    printf("AC Thread Search stats (ctx %p)\n", ctx);
-    printf("Total calls: %" PRIu32 "\n", ctx->total_calls);
-    printf("Total matches: %" PRIu64 "\n", ctx->total_matches);
-#endif /* SC_AC_BS_COUNTERS */
-
-    return;
-}
-
 void SCACBSPrintInfo(MpmCtx *mpm_ctx)
 {
     SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
@@ -1405,7 +1342,6 @@ static int SCACBSTest01(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1424,7 +1360,6 @@ static int SCACBSTest01(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1439,7 +1374,6 @@ static int SCACBSTest02(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abce", 4, 0, 0, 0, 0, 0);
@@ -1457,7 +1391,6 @@ static int SCACBSTest02(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1472,7 +1405,6 @@ static int SCACBSTest03(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1494,7 +1426,6 @@ static int SCACBSTest03(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1509,7 +1440,6 @@ static int SCACBSTest04(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdegh", 6, 0, 0, 1, 0, 0);
@@ -1528,7 +1458,6 @@ static int SCACBSTest04(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1543,7 +1472,6 @@ static int SCACBSTest05(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -1562,7 +1490,6 @@ static int SCACBSTest05(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1577,7 +1504,6 @@ static int SCACBSTest06(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     PmqSetup(&pmq);
@@ -1594,7 +1520,6 @@ static int SCACBSTest06(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1609,7 +1534,6 @@ static int SCACBSTest07(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* should match 30 times */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"A", 1, 0, 0, 0, 0, 0);
@@ -1639,7 +1563,6 @@ static int SCACBSTest07(void)
         printf("135 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1654,7 +1577,6 @@ static int SCACBSTest08(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1671,7 +1593,6 @@ static int SCACBSTest08(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1686,7 +1607,6 @@ static int SCACBSTest09(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ab", 2, 0, 0, 0, 0, 0);
@@ -1703,7 +1623,6 @@ static int SCACBSTest09(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1718,7 +1637,6 @@ static int SCACBSTest10(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcdefgh", 8, 0, 0, 0, 0, 0);
@@ -1740,7 +1658,6 @@ static int SCACBSTest10(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1755,7 +1672,6 @@ static int SCACBSTest11(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"he", 2, 0, 0, 1, 0, 0) == -1)
         goto end;
@@ -1786,10 +1702,9 @@ static int SCACBSTest11(void)
                           strlen(buf)) == 2);
 
  end:
-    SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
-    PmqFree(&pmq);
-    return result;
+     SCACBSDestroyCtx(&mpm_ctx);
+     PmqFree(&pmq);
+     return result;
 }
 
 static int SCACBSTest12(void)
@@ -1802,7 +1717,6 @@ static int SCACBSTest12(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"wxyz", 4, 0, 0, 0, 0, 0);
@@ -1822,7 +1736,6 @@ static int SCACBSTest12(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1837,7 +1750,6 @@ static int SCACBSTest13(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCD";
@@ -1856,7 +1768,6 @@ static int SCACBSTest13(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1871,7 +1782,6 @@ static int SCACBSTest14(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDE";
@@ -1890,7 +1800,6 @@ static int SCACBSTest14(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1905,7 +1814,6 @@ static int SCACBSTest15(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDEF";
@@ -1924,7 +1832,6 @@ static int SCACBSTest15(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1939,7 +1846,6 @@ static int SCACBSTest16(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABC";
@@ -1958,7 +1864,6 @@ static int SCACBSTest16(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1973,7 +1878,6 @@ static int SCACBSTest17(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzAB";
@@ -1992,7 +1896,6 @@ static int SCACBSTest17(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2007,7 +1910,6 @@ static int SCACBSTest18(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcde"
@@ -2031,7 +1933,6 @@ static int SCACBSTest18(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2046,7 +1947,6 @@ static int SCACBSTest19(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
@@ -2079,7 +1979,6 @@ static int SCACBSTest20(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAA"
@@ -2104,7 +2003,6 @@ static int SCACBSTest20(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2119,7 +2017,6 @@ static int SCACBSTest21(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2136,7 +2033,6 @@ static int SCACBSTest21(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2151,7 +2047,6 @@ static int SCACBSTest22(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -2171,7 +2066,6 @@ static int SCACBSTest22(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2186,7 +2080,6 @@ static int SCACBSTest23(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2203,7 +2096,6 @@ static int SCACBSTest23(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2218,7 +2110,6 @@ static int SCACBSTest24(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2235,7 +2126,6 @@ static int SCACBSTest24(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2250,7 +2140,6 @@ static int SCACBSTest25(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -2269,7 +2158,6 @@ static int SCACBSTest25(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2284,7 +2172,6 @@ static int SCACBSTest26(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 1, 0, 0);
@@ -2302,7 +2189,6 @@ static int SCACBSTest26(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2317,7 +2203,6 @@ static int SCACBSTest27(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ONE", 3, 0, 0, 0, 0, 0);
@@ -2335,7 +2220,6 @@ static int SCACBSTest27(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2350,7 +2234,6 @@ static int SCACBSTest28(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"one", 3, 0, 0, 0, 0, 0);
@@ -2368,7 +2251,6 @@ static int SCACBSTest28(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2383,7 +2265,6 @@ static int SCACBSTest29(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-    SCACBSInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcde", 5, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdef", 5, 0, 0, 1, 0, 0);
@@ -2403,7 +2284,6 @@ static int SCACBSTest29(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACBSDestroyCtx(&mpm_ctx);
-    SCACBSDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
diff --git a/src/util-mpm-ac-bs.h b/src/util-mpm-ac-bs.h
index 80ac0f4b06e8..d1135b1cfea6 100644
--- a/src/util-mpm-ac-bs.h
+++ b/src/util-mpm-ac-bs.h
@@ -71,11 +71,4 @@ typedef struct SCACBSCtx_ {
     uint16_t single_state_size;
 } SCACBSCtx;
 
-typedef struct SCACBSThreadCtx_ {
-    /* the total calls we make to the search function */
-    uint32_t total_calls;
-    /* the total patterns that we ended up matching against */
-    uint64_t total_matches;
-} SCACBSThreadCtx;
-
 void MpmACBSRegister(void);
diff --git a/src/util-mpm-ac-ks.c b/src/util-mpm-ac-ks.c
index 9b2ab07d3262..5f47cf495da5 100644
--- a/src/util-mpm-ac-ks.c
+++ b/src/util-mpm-ac-ks.c
@@ -85,9 +85,7 @@
 #if __BYTE_ORDER == __LITTLE_ENDIAN
 
 void SCACTileInitCtx(MpmCtx *);
-void SCACTileInitThreadCtx(MpmCtx *, MpmThreadCtx *);
 void SCACTileDestroyCtx(MpmCtx *);
-void SCACTileDestroyThreadCtx(MpmCtx *, MpmThreadCtx *);
 int SCACTileAddPatternCI(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
                          uint32_t, SigIntId, uint8_t);
 int SCACTileAddPatternCS(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
@@ -97,7 +95,6 @@ uint32_t SCACTileSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         PrefilterRuleStore *pmq, const uint8_t *buf,
                         uint32_t buflen);
 void SCACTilePrintInfo(MpmCtx *mpm_ctx);
-void SCACTilePrintSearchStats(MpmThreadCtx *mpm_thread_ctx);
 void SCACTileRegisterTests(void);
 
 uint32_t SCACTileSearchLarge(const SCACTileSearchCtx *ctx, MpmThreadCtx *mpm_thread_ctx,
@@ -961,25 +958,6 @@ int SCACTilePreparePatterns(MpmCtx *mpm_ctx)
     return -1;
 }
 
-/**
- * \brief Init the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- */
-void SCACTileInitThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    memset(mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-
-    mpm_thread_ctx->ctx = SCMalloc(sizeof(SCACTileThreadCtx));
-    if (mpm_thread_ctx->ctx == NULL) {
-        exit(EXIT_FAILURE);
-    }
-    memset(mpm_thread_ctx->ctx, 0, sizeof(SCACTileThreadCtx));
-    mpm_thread_ctx->memory_cnt++;
-    mpm_thread_ctx->memory_size += sizeof(SCACTileThreadCtx);
-}
-
 /**
  * \brief Initialize the AC context.
  *
@@ -1024,24 +1002,6 @@ void SCACTileInitCtx(MpmCtx *mpm_ctx)
     SCACTileGetConfig();
 }
 
-/**
- * \brief Destroy the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- */
-void SCACTileDestroyThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    SCACTilePrintSearchStats(mpm_thread_ctx);
-
-    if (mpm_thread_ctx->ctx != NULL) {
-        SCFree(mpm_thread_ctx->ctx);
-        mpm_thread_ctx->ctx = NULL;
-        mpm_thread_ctx->memory_cnt--;
-        mpm_thread_ctx->memory_size -= sizeof(SCACTileThreadCtx);
-    }
-}
-
 static void SCACTileDestroyInitCtx(MpmCtx *mpm_ctx)
 {
     SCACTileSearchCtx *search_ctx = (SCACTileSearchCtx *)mpm_ctx->ctx;
@@ -1421,16 +1381,6 @@ int SCACTileAddPatternCS(MpmCtx *mpm_ctx, uint8_t *pat, uint16_t patlen,
                               pid, sid, flags);
 }
 
-void SCACTilePrintSearchStats(MpmThreadCtx *mpm_thread_ctx)
-{
-#ifdef SC_AC_TILE_COUNTERS
-    SCACTileThreadCtx *ctx = (SCACTileThreadCtx *)mpm_thread_ctx->ctx;
-    printf("AC Thread Search stats (ctx %p)\n", ctx);
-    printf("Total calls: %" PRIu32 "\n", ctx->total_calls);
-    printf("Total matches: %" PRIu64 "\n", ctx->total_matches);
-#endif /* SC_AC_TILE_COUNTERS */
-}
-
 void SCACTilePrintInfo(MpmCtx *mpm_ctx)
 {
     SCACTileSearchCtx *search_ctx = (SCACTileSearchCtx *)mpm_ctx->ctx;
@@ -1461,15 +1411,12 @@ void MpmACTileRegister(void)
 {
     mpm_table[MPM_AC_KS].name = "ac-ks";
     mpm_table[MPM_AC_KS].InitCtx = SCACTileInitCtx;
-    mpm_table[MPM_AC_KS].InitThreadCtx = SCACTileInitThreadCtx;
     mpm_table[MPM_AC_KS].DestroyCtx = SCACTileDestroyCtx;
-    mpm_table[MPM_AC_KS].DestroyThreadCtx = SCACTileDestroyThreadCtx;
     mpm_table[MPM_AC_KS].AddPattern = SCACTileAddPatternCS;
     mpm_table[MPM_AC_KS].AddPatternNocase = SCACTileAddPatternCI;
     mpm_table[MPM_AC_KS].Prepare = SCACTilePreparePatterns;
     mpm_table[MPM_AC_KS].Search = SCACTileSearch;
     mpm_table[MPM_AC_KS].PrintCtx = SCACTilePrintInfo;
-    mpm_table[MPM_AC_KS].PrintThreadCtx = SCACTilePrintSearchStats;
     mpm_table[MPM_AC_KS].RegisterUnittests = SCACTileRegisterTests;
 }
 
@@ -1489,7 +1436,6 @@ static int SCACTileTest01(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1508,7 +1454,6 @@ static int SCACTileTest01(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1523,7 +1468,6 @@ static int SCACTileTest02(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abce", 4, 0, 0, 0, 0, 0);
@@ -1541,7 +1485,6 @@ static int SCACTileTest02(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1556,7 +1499,6 @@ static int SCACTileTest03(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1578,7 +1520,6 @@ static int SCACTileTest03(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1593,7 +1534,6 @@ static int SCACTileTest04(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdegh", 6, 0, 0, 1, 0, 0);
@@ -1612,7 +1552,6 @@ static int SCACTileTest04(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1627,7 +1566,6 @@ static int SCACTileTest05(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -1646,7 +1584,6 @@ static int SCACTileTest05(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1661,7 +1598,6 @@ static int SCACTileTest06(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     PmqSetup(&pmq);
@@ -1678,7 +1614,6 @@ static int SCACTileTest06(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1693,7 +1628,6 @@ static int SCACTileTest07(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* should match 30 times */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"A", 1, 0, 0, 0, 0, 0);
@@ -1723,7 +1657,6 @@ static int SCACTileTest07(void)
         printf("135 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1738,7 +1671,6 @@ static int SCACTileTest08(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1755,7 +1687,6 @@ static int SCACTileTest08(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1770,7 +1701,6 @@ static int SCACTileTest09(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ab", 2, 0, 0, 0, 0, 0);
@@ -1787,7 +1717,6 @@ static int SCACTileTest09(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1802,7 +1731,6 @@ static int SCACTileTest10(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcdefgh", 8, 0, 0, 0, 0, 0);
@@ -1824,7 +1752,6 @@ static int SCACTileTest10(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1839,7 +1766,6 @@ static int SCACTileTest11(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"he", 2, 0, 0, 1, 0, 0) == -1)
         goto end;
@@ -1870,10 +1796,9 @@ static int SCACTileTest11(void)
                               strlen(buf)) == 2);
 
  end:
-    SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
-    PmqFree(&pmq);
-    return result;
+     SCACTileDestroyCtx(&mpm_ctx);
+     PmqFree(&pmq);
+     return result;
 }
 
 static int SCACTileTest12(void)
@@ -1886,7 +1811,6 @@ static int SCACTileTest12(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"wxyz", 4, 0, 0, 0, 0, 0);
@@ -1906,7 +1830,6 @@ static int SCACTileTest12(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1921,7 +1844,6 @@ static int SCACTileTest13(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCD";
@@ -1940,7 +1862,6 @@ static int SCACTileTest13(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1955,7 +1876,6 @@ static int SCACTileTest14(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDE";
@@ -1974,7 +1894,6 @@ static int SCACTileTest14(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1989,7 +1908,6 @@ static int SCACTileTest15(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDEF";
@@ -2008,7 +1926,6 @@ static int SCACTileTest15(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2023,7 +1940,6 @@ static int SCACTileTest16(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABC";
@@ -2042,7 +1958,6 @@ static int SCACTileTest16(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2057,7 +1972,6 @@ static int SCACTileTest17(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzAB";
@@ -2076,7 +1990,6 @@ static int SCACTileTest17(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2091,7 +2004,6 @@ static int SCACTileTest18(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcde"
@@ -2115,7 +2027,6 @@ static int SCACTileTest18(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2130,7 +2041,6 @@ static int SCACTileTest19(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
@@ -2149,7 +2059,6 @@ static int SCACTileTest19(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2164,7 +2073,6 @@ static int SCACTileTest20(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAA"
@@ -2189,7 +2097,6 @@ static int SCACTileTest20(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2204,7 +2111,6 @@ static int SCACTileTest21(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2221,7 +2127,6 @@ static int SCACTileTest21(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2236,7 +2141,6 @@ static int SCACTileTest22(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -2256,7 +2160,6 @@ static int SCACTileTest22(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2271,7 +2174,6 @@ static int SCACTileTest23(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2288,7 +2190,6 @@ static int SCACTileTest23(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2303,7 +2204,6 @@ static int SCACTileTest24(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2320,7 +2220,6 @@ static int SCACTileTest24(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2335,7 +2234,6 @@ static int SCACTileTest25(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -2354,7 +2252,6 @@ static int SCACTileTest25(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2369,7 +2266,6 @@ static int SCACTileTest26(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 1, 0, 0);
@@ -2387,7 +2283,6 @@ static int SCACTileTest26(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2402,7 +2297,6 @@ static int SCACTileTest27(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ONE", 3, 0, 0, 0, 0, 0);
@@ -2420,7 +2314,6 @@ static int SCACTileTest27(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2435,7 +2328,6 @@ static int SCACTileTest28(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC_KS);
-    SCACTileInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"one", 3, 0, 0, 0, 0, 0);
@@ -2453,7 +2345,6 @@ static int SCACTileTest28(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACTileDestroyCtx(&mpm_ctx);
-    SCACTileDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
diff --git a/src/util-mpm-ac-ks.h b/src/util-mpm-ac-ks.h
index 05e0168b45d9..4979f84241a4 100644
--- a/src/util-mpm-ac-ks.h
+++ b/src/util-mpm-ac-ks.h
@@ -145,14 +145,6 @@ typedef struct SCACTileSearchCtx_ {
 
 } SCACTileSearchCtx;
 
-
-typedef struct SCACTileThreadCtx_ {
-    /* the total calls we make to the search function */
-    uint32_t total_calls;
-    /* the total patterns that we ended up matching against */
-    uint64_t total_matches;
-} SCACTileThreadCtx;
-
 void MpmACTileRegister(void);
 
 #endif /* __UTIL_MPM_AC_KS__H__ */
diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index 961031e3a918..6d1d44b30d14 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -63,9 +63,7 @@
 #include "util-validate.h"
 
 void SCACInitCtx(MpmCtx *);
-void SCACInitThreadCtx(MpmCtx *, MpmThreadCtx *);
 void SCACDestroyCtx(MpmCtx *);
-void SCACDestroyThreadCtx(MpmCtx *, MpmThreadCtx *);
 int SCACAddPatternCI(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
                      uint32_t, SigIntId, uint8_t);
 int SCACAddPatternCS(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
@@ -74,7 +72,6 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx);
 uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                     PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen);
 void SCACPrintInfo(MpmCtx *mpm_ctx);
-void SCACPrintSearchStats(MpmThreadCtx *mpm_thread_ctx);
 void SCACRegisterTests(void);
 
 /* a placeholder to denote a failure transition in the goto table */
@@ -819,28 +816,6 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx)
     return -1;
 }
 
-/**
- * \brief Init the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- * \param matchsize      We don't need this.
- */
-void SCACInitThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    memset(mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-
-    mpm_thread_ctx->ctx = SCMalloc(sizeof(SCACThreadCtx));
-    if (mpm_thread_ctx->ctx == NULL) {
-        exit(EXIT_FAILURE);
-    }
-    memset(mpm_thread_ctx->ctx, 0, sizeof(SCACThreadCtx));
-    mpm_thread_ctx->memory_cnt++;
-    mpm_thread_ctx->memory_size += sizeof(SCACThreadCtx);
-
-    return;
-}
-
 /**
  * \brief Initialize the AC context.
  *
@@ -874,26 +849,6 @@ void SCACInitCtx(MpmCtx *mpm_ctx)
     SCReturn;
 }
 
-/**
- * \brief Destroy the mpm thread context.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- */
-void SCACDestroyThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
-{
-    SCACPrintSearchStats(mpm_thread_ctx);
-
-    if (mpm_thread_ctx->ctx != NULL) {
-        SCFree(mpm_thread_ctx->ctx);
-        mpm_thread_ctx->ctx = NULL;
-        mpm_thread_ctx->memory_cnt--;
-        mpm_thread_ctx->memory_size -= sizeof(SCACThreadCtx);
-    }
-
-    return;
-}
-
 /**
  * \brief Destroy the mpm context.
  *
@@ -1153,19 +1108,6 @@ int SCACAddPatternCS(MpmCtx *mpm_ctx, uint8_t *pat, uint16_t patlen,
     return MpmAddPattern(mpm_ctx, pat, patlen, offset, depth, pid, sid, flags);
 }
 
-void SCACPrintSearchStats(MpmThreadCtx *mpm_thread_ctx)
-{
-
-#ifdef SC_AC_COUNTERS
-    SCACThreadCtx *ctx = (SCACThreadCtx *)mpm_thread_ctx->ctx;
-    printf("AC Thread Search stats (ctx %p)\n", ctx);
-    printf("Total calls: %" PRIu32 "\n", ctx->total_calls);
-    printf("Total matches: %" PRIu64 "\n", ctx->total_matches);
-#endif /* SC_AC_COUNTERS */
-
-    return;
-}
-
 void SCACPrintInfo(MpmCtx *mpm_ctx)
 {
     SCACCtx *ctx = (SCACCtx *)mpm_ctx->ctx;
@@ -1197,15 +1139,12 @@ void MpmACRegister(void)
 {
     mpm_table[MPM_AC].name = "ac";
     mpm_table[MPM_AC].InitCtx = SCACInitCtx;
-    mpm_table[MPM_AC].InitThreadCtx = SCACInitThreadCtx;
     mpm_table[MPM_AC].DestroyCtx = SCACDestroyCtx;
-    mpm_table[MPM_AC].DestroyThreadCtx = SCACDestroyThreadCtx;
     mpm_table[MPM_AC].AddPattern = SCACAddPatternCS;
     mpm_table[MPM_AC].AddPatternNocase = SCACAddPatternCI;
     mpm_table[MPM_AC].Prepare = SCACPreparePatterns;
     mpm_table[MPM_AC].Search = SCACSearch;
     mpm_table[MPM_AC].PrintCtx = SCACPrintInfo;
-    mpm_table[MPM_AC].PrintThreadCtx = SCACPrintSearchStats;
     mpm_table[MPM_AC].RegisterUnittests = SCACRegisterTests;
 
     return;
@@ -1226,7 +1165,6 @@ static int SCACTest01(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1245,7 +1183,6 @@ static int SCACTest01(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1260,7 +1197,6 @@ static int SCACTest02(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abce", 4, 0, 0, 0, 0, 0);
@@ -1278,7 +1214,6 @@ static int SCACTest02(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1293,7 +1228,6 @@ static int SCACTest03(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1315,7 +1249,6 @@ static int SCACTest03(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1330,7 +1263,6 @@ static int SCACTest04(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdegh", 6, 0, 0, 1, 0, 0);
@@ -1349,7 +1281,6 @@ static int SCACTest04(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1364,7 +1295,6 @@ static int SCACTest05(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -1383,7 +1313,6 @@ static int SCACTest05(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1398,7 +1327,6 @@ static int SCACTest06(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
     PmqSetup(&pmq);
@@ -1415,7 +1343,6 @@ static int SCACTest06(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1430,7 +1357,6 @@ static int SCACTest07(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* should match 30 times */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"A", 1, 0, 0, 0, 0, 0);
@@ -1460,7 +1386,6 @@ static int SCACTest07(void)
         printf("135 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1475,7 +1400,6 @@ static int SCACTest08(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1492,7 +1416,6 @@ static int SCACTest08(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1507,7 +1430,6 @@ static int SCACTest09(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ab", 2, 0, 0, 0, 0, 0);
@@ -1524,7 +1446,6 @@ static int SCACTest09(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1539,7 +1460,6 @@ static int SCACTest10(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcdefgh", 8, 0, 0, 0, 0, 0);
@@ -1561,7 +1481,6 @@ static int SCACTest10(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1576,7 +1495,6 @@ static int SCACTest11(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"he", 2, 0, 0, 1, 0, 0) == -1)
         goto end;
@@ -1607,10 +1525,9 @@ static int SCACTest11(void)
                           strlen(buf)) == 2);
 
  end:
-    SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
-    PmqFree(&pmq);
-    return result;
+     SCACDestroyCtx(&mpm_ctx);
+     PmqFree(&pmq);
+     return result;
 }
 
 static int SCACTest12(void)
@@ -1623,7 +1540,6 @@ static int SCACTest12(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"wxyz", 4, 0, 0, 0, 0, 0);
@@ -1643,7 +1559,6 @@ static int SCACTest12(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1658,7 +1573,6 @@ static int SCACTest13(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCD";
@@ -1677,7 +1591,6 @@ static int SCACTest13(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1692,7 +1605,6 @@ static int SCACTest14(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDE";
@@ -1711,7 +1623,6 @@ static int SCACTest14(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1726,7 +1637,6 @@ static int SCACTest15(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDEF";
@@ -1745,7 +1655,6 @@ static int SCACTest15(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1760,7 +1669,6 @@ static int SCACTest16(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzABC";
@@ -1779,7 +1687,6 @@ static int SCACTest16(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1794,7 +1701,6 @@ static int SCACTest17(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcdefghijklmnopqrstuvwxyzAB";
@@ -1813,7 +1719,6 @@ static int SCACTest17(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1828,7 +1733,6 @@ static int SCACTest18(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     const char pat[] = "abcde"
@@ -1852,7 +1756,6 @@ static int SCACTest18(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1867,7 +1770,6 @@ static int SCACTest19(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
@@ -1886,7 +1788,6 @@ static int SCACTest19(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1901,7 +1802,6 @@ static int SCACTest20(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     const char pat[] = "AAAAA"
@@ -1926,7 +1826,6 @@ static int SCACTest20(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1941,7 +1840,6 @@ static int SCACTest21(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -1958,7 +1856,6 @@ static int SCACTest21(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -1973,7 +1870,6 @@ static int SCACTest22(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
@@ -1993,7 +1889,6 @@ static int SCACTest22(void)
         printf("2 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2008,7 +1903,6 @@ static int SCACTest23(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2025,7 +1919,6 @@ static int SCACTest23(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2040,7 +1933,6 @@ static int SCACTest24(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 1 */
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
@@ -2057,7 +1949,6 @@ static int SCACTest24(void)
         printf("1 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2072,7 +1963,6 @@ static int SCACTest25(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
@@ -2091,7 +1981,6 @@ static int SCACTest25(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2106,7 +1995,6 @@ static int SCACTest26(void)
     memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     MpmAddPatternCI(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 0, 0, 0);
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 1, 0, 0);
@@ -2124,7 +2012,6 @@ static int SCACTest26(void)
         printf("3 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2139,7 +2026,6 @@ static int SCACTest27(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ONE", 3, 0, 0, 0, 0, 0);
@@ -2157,7 +2043,6 @@ static int SCACTest27(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
@@ -2172,7 +2057,6 @@ static int SCACTest28(void)
     memset(&mpm_ctx, 0, sizeof(MpmCtx));
     memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
     MpmInitCtx(&mpm_ctx, MPM_AC);
-    SCACInitThreadCtx(&mpm_ctx, &mpm_thread_ctx);
 
     /* 0 match */
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"one", 3, 0, 0, 0, 0, 0);
@@ -2190,7 +2074,6 @@ static int SCACTest28(void)
         printf("0 != %" PRIu32 " ",cnt);
 
     SCACDestroyCtx(&mpm_ctx);
-    SCACDestroyThreadCtx(&mpm_ctx, &mpm_thread_ctx);
     PmqFree(&pmq);
     return result;
 }
diff --git a/src/util-mpm-ac.h b/src/util-mpm-ac.h
index 46d0238fed43..3e8ec9db5bb2 100644
--- a/src/util-mpm-ac.h
+++ b/src/util-mpm-ac.h
@@ -77,13 +77,6 @@ typedef struct SCACCtx_ {
 
 } SCACCtx;
 
-typedef struct SCACThreadCtx_ {
-    /* the total calls we make to the search function */
-    uint32_t total_calls;
-    /* the total patterns that we ended up matching against */
-    uint64_t total_matches;
-} SCACThreadCtx;
-
 void MpmACRegister(void);
 
 #endif /* __UTIL_MPM_AC__H__ */
diff --git a/src/util-mpm.c b/src/util-mpm.c
index 1e05097ae584..0bacc9330b1c 100644
--- a/src/util-mpm.c
+++ b/src/util-mpm.c
@@ -197,7 +197,16 @@ void MpmFactoryDeRegisterAllMpmCtxProfiles(DetectEngineCtx *de_ctx)
 
 void MpmInitThreadCtx(MpmThreadCtx *mpm_thread_ctx, uint16_t matcher)
 {
-    mpm_table[matcher].InitThreadCtx(NULL, mpm_thread_ctx);
+    if (mpm_table[matcher].InitThreadCtx != NULL) {
+        mpm_table[matcher].InitThreadCtx(NULL, mpm_thread_ctx);
+    }
+}
+
+void MpmDestroyThreadCtx(MpmThreadCtx *mpm_thread_ctx, const uint16_t matcher)
+{
+    if (mpm_table[matcher].DestroyThreadCtx != NULL) {
+        mpm_table[matcher].DestroyThreadCtx(NULL, mpm_thread_ctx);
+    }
 }
 
 void MpmInitCtx(MpmCtx *mpm_ctx, uint8_t matcher)
diff --git a/src/util-mpm.h b/src/util-mpm.h
index 4ddd4de49659..87eec5e793a9 100644
--- a/src/util-mpm.h
+++ b/src/util-mpm.h
@@ -184,6 +184,7 @@ void MpmRegisterTests(void);
 
 void MpmInitCtx(MpmCtx *mpm_ctx, uint8_t matcher);
 void MpmInitThreadCtx(MpmThreadCtx *mpm_thread_ctx, uint16_t);
+void MpmDestroyThreadCtx(MpmThreadCtx *mpm_thread_ctx, const uint16_t matcher);
 
 int MpmAddPatternCS(struct MpmCtx_ *mpm_ctx, uint8_t *pat, uint16_t patlen,
                     uint16_t offset, uint16_t depth,

From 0c55fe3515413fda036d701e9b23c79173fe1f8c Mon Sep 17 00:00:00 2001
From: Sascha Steinbiss <satta@debian.org>
Date: Wed, 11 Oct 2023 22:25:46 +0200
Subject: [PATCH 128/462] detect: add mqtt.connect.protocolstring

Ticket:  OISF#6396
---
 doc/userguide/rules/mqtt-keywords.rst     | 13 ++++
 rust/src/mqtt/detect.rs                   | 20 +++++
 src/Makefile.am                           |  2 +
 src/detect-engine-register.c              |  2 +
 src/detect-engine-register.h              |  1 +
 src/detect-mqtt-connect-protocol-string.c | 94 +++++++++++++++++++++++
 src/detect-mqtt-connect-protocol-string.h | 29 +++++++
 7 files changed, 161 insertions(+)
 create mode 100644 src/detect-mqtt-connect-protocol-string.c
 create mode 100644 src/detect-mqtt-connect-protocol-string.h

diff --git a/doc/userguide/rules/mqtt-keywords.rst b/doc/userguide/rules/mqtt-keywords.rst
index 21776ac357cb..058a17b7ffde 100644
--- a/doc/userguide/rules/mqtt-keywords.rst
+++ b/doc/userguide/rules/mqtt-keywords.rst
@@ -163,6 +163,19 @@ Examples::
 ``mqtt.connect.password`` is a 'sticky buffer' and can be used as ``fast_pattern``.
 
 
+mqtt.connect.protocol_string
+----------------------------
+
+Match on the protocol string in the MQTT CONNECT message. In contrast to ``mqtt.protocol_version`` this is a property that is only really relevant in the initial CONNECT communication and never used again; hence it is organized under ``mqtt.connect``.
+
+Examples::
+
+  mqtt.connect.protocol_string; content:"MQTT";
+  mqtt.connect.protocol_string; content:"MQIsdp";
+
+``mqtt.connect.protocol_string`` is a 'sticky buffer' and can be used as ``fast_pattern``.
+
+
 mqtt.connect.username
 ---------------------
 
diff --git a/rust/src/mqtt/detect.rs b/rust/src/mqtt/detect.rs
index b47a84f74409..df0c78e8497f 100644
--- a/rust/src/mqtt/detect.rs
+++ b/rust/src/mqtt/detect.rs
@@ -231,6 +231,26 @@ pub unsafe extern "C" fn rs_mqtt_tx_get_connect_willmessage(
     return 0;
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn rs_mqtt_tx_get_connect_protocol_string(
+    tx: &MQTTTransaction, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
+    for msg in tx.msg.iter() {
+        if let MQTTOperation::CONNECT(ref cv) = msg.op {
+            let p = &cv.protocol_string;
+            if !p.is_empty() {
+                *buffer = p.as_ptr();
+                *buffer_len = p.len() as u32;
+                return 1;
+            }
+        }
+    }
+
+    *buffer = ptr::null();
+    *buffer_len = 0;
+    return 0;
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn rs_mqtt_tx_get_connack_sessionpresent(
     tx: &MQTTTransaction, session_present: *mut bool,
diff --git a/src/Makefile.am b/src/Makefile.am
index a125e2a432ba..f8033de41b88 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -250,6 +250,7 @@ noinst_HEADERS = \
 	detect-mqtt-connect-clientid.h \
 	detect-mqtt-connect-flags.h \
 	detect-mqtt-connect-password.h \
+	detect-mqtt-connect-protocol-string.h \
 	detect-mqtt-connect-username.h \
 	detect-mqtt-connect-willmessage.h \
 	detect-mqtt-connect-willtopic.h \
@@ -862,6 +863,7 @@ libsuricata_c_a_SOURCES = \
 	detect-mqtt-connect-clientid.c \
 	detect-mqtt-connect-flags.c \
 	detect-mqtt-connect-password.c \
+	detect-mqtt-connect-protocol-string.c \
 	detect-mqtt-connect-username.c \
 	detect-mqtt-connect-willmessage.c \
 	detect-mqtt-connect-willtopic.c \
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index af247a1b4fa7..bd8f66519683 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -219,6 +219,7 @@
 #include "detect-mqtt-connect-clientid.h"
 #include "detect-mqtt-connect-username.h"
 #include "detect-mqtt-connect-password.h"
+#include "detect-mqtt-connect-protocol-string.h"
 #include "detect-mqtt-connect-willtopic.h"
 #include "detect-mqtt-connect-willmessage.h"
 #include "detect-mqtt-connack-sessionpresent.h"
@@ -677,6 +678,7 @@ void SigTableSetup(void)
     DetectMQTTConnectClientIDRegister();
     DetectMQTTConnectUsernameRegister();
     DetectMQTTConnectPasswordRegister();
+    DetectMQTTConnectProtocolStringRegister();
     DetectMQTTConnectWillTopicRegister();
     DetectMQTTConnectWillMessageRegister();
     DetectMQTTConnackSessionPresentRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 92acd84f044b..abc1a403dd09 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -299,6 +299,7 @@ enum DetectKeywordId {
     DETECT_AL_MQTT_CONNECT_CLIENTID,
     DETECT_AL_MQTT_CONNECT_USERNAME,
     DETECT_AL_MQTT_CONNECT_PASSWORD,
+    DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING,
     DETECT_AL_MQTT_CONNECT_WILLTOPIC,
     DETECT_AL_MQTT_CONNECT_WILLMESSAGE,
     DETECT_AL_MQTT_CONNACK_SESSION_PRESENT,
diff --git a/src/detect-mqtt-connect-protocol-string.c b/src/detect-mqtt-connect-protocol-string.c
new file mode 100644
index 000000000000..421b293845b7
--- /dev/null
+++ b/src/detect-mqtt-connect-protocol-string.c
@@ -0,0 +1,94 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ *
+ * \author Sascha Steinbiss <sascha@steinbiss.name>
+ *
+ * Implements the mqtt.connect.protocolstring sticky buffer
+ */
+
+#include "suricata-common.h"
+#include "detect.h"
+#include "detect-parse.h"
+#include "detect-engine.h"
+#include "detect-engine-mpm.h"
+#include "detect-engine-prefilter.h"
+#include "detect-mqtt-connect-protocol-string.h"
+#include "rust.h"
+
+#define KEYWORD_NAME "mqtt.connect.protocol_string"
+#define KEYWORD_DOC  "mqtt-keywords.html#mqtt-connect-protocol_string"
+#define BUFFER_NAME  "mqtt.connect.protocol_string"
+#define BUFFER_DESC  "MQTT CONNECT protocol string"
+static int g_buffer_id = 0;
+
+static int DetectMQTTConnectProtocolStringSetup(
+        DetectEngineCtx *de_ctx, Signature *s, const char *arg)
+{
+    if (DetectBufferSetActiveList(de_ctx, s, g_buffer_id) < 0)
+        return -1;
+
+    if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
+        return -1;
+
+    return 0;
+}
+
+static InspectionBuffer *GetData(DetectEngineThreadCtx *det_ctx,
+        const DetectEngineTransforms *transforms, Flow *_f, const uint8_t _flow_flags, void *txv,
+        const int list_id)
+{
+    InspectionBuffer *buffer = InspectionBufferGet(det_ctx, list_id);
+    if (buffer->inspect == NULL) {
+        const uint8_t *b = NULL;
+        uint32_t b_len = 0;
+
+        if (rs_mqtt_tx_get_connect_protocol_string(txv, &b, &b_len) != 1)
+            return NULL;
+        if (b == NULL || b_len == 0)
+            return NULL;
+
+        InspectionBufferSetup(det_ctx, list_id, buffer, b, b_len);
+        InspectionBufferApplyTransforms(buffer, transforms);
+    }
+    return buffer;
+}
+
+void DetectMQTTConnectProtocolStringRegister(void)
+{
+    /* mqtt.connect.protocol_string sticky buffer */
+    sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].name = KEYWORD_NAME;
+    sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].desc =
+            "sticky buffer to match on the MQTT CONNECT protocol string";
+    sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].url = "/rules/" KEYWORD_DOC;
+    sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].Setup =
+            DetectMQTTConnectProtocolStringSetup;
+    sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].flags |= SIGMATCH_NOOPT;
+
+    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
+            DetectEngineInspectBufferGeneric, GetData);
+
+    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
+
+    DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
+
+    g_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
+
+    SCLogDebug("registering " BUFFER_NAME " rule option");
+}
diff --git a/src/detect-mqtt-connect-protocol-string.h b/src/detect-mqtt-connect-protocol-string.h
new file mode 100644
index 000000000000..3bbb21ab10d4
--- /dev/null
+++ b/src/detect-mqtt-connect-protocol-string.h
@@ -0,0 +1,29 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Sascha Steinbiss <sascha@steinbiss.name>
+ */
+
+#ifndef __DETECT_MQTT_CONNECT_PROTOCOLSTRING_H__
+#define __DETECT_MQTT_CONNECT_PROTOCOLSTRING_H__
+
+void DetectMQTTConnectProtocolStringRegister(void);
+
+#endif /* __DETECT_MQTT_CONNECT_PROTOCOLSTRING_H__ */

From d07e7f6862a3dfa732c6ebbafbbd42873f48d45a Mon Sep 17 00:00:00 2001
From: Sascha Steinbiss <satta@debian.org>
Date: Wed, 11 Oct 2023 22:25:55 +0200
Subject: [PATCH 129/462] detect: fix typo

---
 src/detect-mqtt-type.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-mqtt-type.c b/src/detect-mqtt-type.c
index 3bc7f1e4f593..c55938c78c2d 100644
--- a/src/detect-mqtt-type.c
+++ b/src/detect-mqtt-type.c
@@ -43,7 +43,7 @@ void MQTTTypeRegisterTests(void);
 void DetectMQTTTypeFree(DetectEngineCtx *de_ctx, void *);
 
 /**
- * \brief Registration function for ipopts: keyword
+ * \brief Registration function for mqtt.type: keyword
  */
 void DetectMQTTTypeRegister (void)
 {

From 491f5dcc319a99b06fe42a6235c0f5fd63abd820 Mon Sep 17 00:00:00 2001
From: Comfort Amaechi <comfortamaechi619@gmail.com>
Date: Mon, 6 Nov 2023 05:36:50 -0500
Subject: [PATCH 130/462] util-memcmp: Convert unittests to new FAIL/PASS API

Ticket: #6107
---
 src/util-memcmp.c | 104 ++++++++++++++++------------------------------
 1 file changed, 36 insertions(+), 68 deletions(-)

diff --git a/src/util-memcmp.c b/src/util-memcmp.c
index bf3d8e56b341..7113b82dd60c 100644
--- a/src/util-memcmp.c
+++ b/src/util-memcmp.c
@@ -38,10 +38,8 @@ static int MemcmpTest01 (void)
     uint8_t a[] = "abcd";
     uint8_t b[] = "abcd";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest02 (void)
@@ -49,10 +47,8 @@ static int MemcmpTest02 (void)
     uint8_t a[] = "abcdabcdabcdabcd";
     uint8_t b[] = "abcdabcdabcdabcd";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest03 (void)
@@ -60,10 +56,8 @@ static int MemcmpTest03 (void)
     uint8_t a[] = "abcdabcd";
     uint8_t b[] = "abcdabcd";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest04 (void)
@@ -72,12 +66,9 @@ static int MemcmpTest04 (void)
     uint8_t b[] = "abcD";
 
     int r = SCMemcmp(a, b, sizeof(a)-1);
-    if (r != 1) {
-        printf("%s != %s, but memcmp returned %d: ", a, b, r);
-        return 0;
-    }
+    FAIL_IF(r != 1);
 
-    return 1;
+    PASS;
 }
 
 static int MemcmpTest05 (void)
@@ -85,10 +76,8 @@ static int MemcmpTest05 (void)
     uint8_t a[] = "abcdabcdabcdabcd";
     uint8_t b[] = "abcDabcdabcdabcd";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 1)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 1);
+    PASS;
 }
 
 static int MemcmpTest06 (void)
@@ -96,10 +85,8 @@ static int MemcmpTest06 (void)
     uint8_t a[] = "abcdabcd";
     uint8_t b[] = "abcDabcd";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 1)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 1);
+    PASS;
 }
 
 static int MemcmpTest07 (void)
@@ -107,10 +94,8 @@ static int MemcmpTest07 (void)
     uint8_t a[] = "abcd";
     uint8_t b[] = "abcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest08 (void)
@@ -118,10 +103,8 @@ static int MemcmpTest08 (void)
     uint8_t a[] = "abcdabcdabcdabcd";
     uint8_t b[] = "abcdabcdabcdabcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest09 (void)
@@ -129,10 +112,8 @@ static int MemcmpTest09 (void)
     uint8_t a[] = "abcdabcd";
     uint8_t b[] = "abcdabcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 static int MemcmpTest10 (void)
@@ -140,10 +121,8 @@ static int MemcmpTest10 (void)
     uint8_t a[] = "abcd";
     uint8_t b[] = "Zbcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 1)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 1);
+    PASS;
 }
 
 static int MemcmpTest11 (void)
@@ -151,10 +130,8 @@ static int MemcmpTest11 (void)
     uint8_t a[] = "abcdabcdabcdabcd";
     uint8_t b[] = "Zbcdabcdabcdabcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 1)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 1);
+    PASS;
 }
 
 static int MemcmpTest12 (void)
@@ -162,10 +139,8 @@ static int MemcmpTest12 (void)
     uint8_t a[] = "abcdabcd";
     uint8_t b[] = "Zbcdabcde";
 
-    if (SCMemcmp(a, b, sizeof(a)-1) != 1)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmp(a, b, sizeof(a) - 1) != 1);
+    PASS;
 }
 
 static int MemcmpTest13 (void)
@@ -173,10 +148,8 @@ static int MemcmpTest13 (void)
     uint8_t a[] = "abcdefgh";
     uint8_t b[] = "AbCdEfGhIjK";
 
-    if (SCMemcmpLowercase(a, b, sizeof(a)-1) != 0)
-        return 0;
-
-    return 1;
+    FAIL_IF(SCMemcmpLowercase(a, b, sizeof(a) - 1) != 0);
+    PASS;
 }
 
 #include "util-cpu.h"
@@ -216,10 +189,9 @@ static int MemcmpTest14 (void)
     SCLogInfo("ticks passed %"PRIu64, ticks_end - ticks_start);
 
     printf("r1 %d\n", r1);
-    if (r1 != (51 * TEST_RUNS))
-        return 0;
+    FAIL_IF(r1 != (51 * TEST_RUNS));
 #endif
-    return 1;
+    PASS;
 }
 
 static int MemcmpTest15 (void)
@@ -255,10 +227,9 @@ static int MemcmpTest15 (void)
     SCLogInfo("ticks passed %"PRIu64, ticks_end - ticks_start);
 
     printf("r2 %d\n", r2);
-    if (r2 != (51 * TEST_RUNS))
-        return 0;
+    FAIL_IF(r2 != (51 * TEST_RUNS));
 #endif
-    return 1;
+    PASS;
 }
 
 static int MemcmpTest16 (void)
@@ -294,10 +265,9 @@ static int MemcmpTest16 (void)
     SCLogInfo("ticks passed %"PRIu64, ticks_end - ticks_start);
 
     printf("r3 %d\n", r3);
-    if (r3 != (51 * TEST_RUNS))
-        return 0;
+    FAIL_IF(r3 != (51 * TEST_RUNS));
 #endif
-    return 1;
+    PASS;
 }
 
 static int MemcmpTest17 (void)
@@ -333,10 +303,9 @@ static int MemcmpTest17 (void)
     SCLogInfo("ticks passed %"PRIu64, ticks_end - ticks_start);
 
     printf("r4 %d\n", r4);
-    if (r4 != (51 * TEST_RUNS))
-        return 0;
+    FAIL_IF(r4 != (51 * TEST_RUNS));
 #endif
-    return 1;
+    PASS;
 }
 
 struct MemcmpTest18Tests {
@@ -370,12 +339,11 @@ static int MemcmpTest18 (void)
 
     while (t && t->a != NULL) {
 
-        if (SCMemcmpLowercase(t->a, t->b, strlen(t->a)-1) != t->result)
-            return 0;
+        FAIL_IF(SCMemcmpLowercase(t->a, t->b, strlen(t->a) - 1) != t->result);
         t++;
     }
 
-    return 1;
+    PASS;
 }
 
 #endif /* UNITTESTS */

From 310dcd1dc46f113711084e08edc7d8b392c8a047 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 09:17:14 -0400
Subject: [PATCH 131/462] general: Use bool instead of int for condition fns

This commit changes the conditional logging functions to use bool rather
than int values.
---
 src/alert-debuglog.c       |  4 ++--
 src/alert-fastlog.c        |  6 +++---
 src/alert-syslog.c         |  4 ++--
 src/app-layer-ssh.c        |  2 +-
 src/app-layer-ssh.h        |  2 +-
 src/log-pcap.c             | 22 +++++++---------------
 src/log-tlsstore.c         | 12 ++++++------
 src/output-eve-stream.c    |  6 +++---
 src/output-json-alert.c    |  7 ++-----
 src/output-json-anomaly.c  |  2 +-
 src/output-json-drop.c     | 21 ++++++++++-----------
 src/output-json-frame.c    |  8 ++++----
 src/output-json-metadata.c |  7 ++-----
 src/output-lua.c           | 14 ++++++--------
 src/output-packet.c        |  2 +-
 src/output-packet.h        |  2 +-
 src/output-tx.c            |  3 +--
 src/output-tx.h            |  3 ++-
 18 files changed, 55 insertions(+), 72 deletions(-)

diff --git a/src/alert-debuglog.c b/src/alert-debuglog.c
index dfcd2938d3ab..e0a0802051e1 100644
--- a/src/alert-debuglog.c
+++ b/src/alert-debuglog.c
@@ -468,9 +468,9 @@ static OutputInitResult AlertDebugLogInitCtx(ConfNode *conf)
     return result;
 }
 
-static int AlertDebugLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool AlertDebugLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
-    return (p->alerts.cnt ? TRUE : FALSE);
+    return (p->alerts.cnt > 0);
 }
 
 static int AlertDebugLogLogger(ThreadVars *tv, void *thread_data, const Packet *p)
diff --git a/src/alert-fastlog.c b/src/alert-fastlog.c
index bfb7f8ffe8c4..8cd4a3c58aa8 100644
--- a/src/alert-fastlog.c
+++ b/src/alert-fastlog.c
@@ -71,7 +71,7 @@ TmEcode AlertFastLogThreadDeinit(ThreadVars *, void *);
 void AlertFastLogRegisterTests(void);
 static void AlertFastLogDeInitCtx(OutputCtx *);
 
-int AlertFastLogCondition(ThreadVars *tv, void *thread_data, const Packet *p);
+static bool AlertFastLogCondition(ThreadVars *tv, void *thread_data, const Packet *p);
 int AlertFastLogger(ThreadVars *tv, void *data, const Packet *p);
 
 void AlertFastLogRegister(void)
@@ -87,9 +87,9 @@ typedef struct AlertFastLogThread_ {
     LogFileCtx* file_ctx;
 } AlertFastLogThread;
 
-int AlertFastLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool AlertFastLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
-    return (p->alerts.cnt ? TRUE : FALSE);
+    return (p->alerts.cnt > 0);
 }
 
 static inline void AlertFastLogOutputAlert(AlertFastLogThread *aft, char *buffer,
diff --git a/src/alert-syslog.c b/src/alert-syslog.c
index fa585811bb5d..df0be1a94a63 100644
--- a/src/alert-syslog.c
+++ b/src/alert-syslog.c
@@ -367,9 +367,9 @@ static TmEcode AlertSyslogDecoderEvent(ThreadVars *tv, const Packet *p, void *da
     return TM_ECODE_OK;
 }
 
-static int AlertSyslogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool AlertSyslogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
-    return (p->alerts.cnt > 0 ? TRUE : FALSE);
+    return (p->alerts.cnt > 0);
 }
 
 static int AlertSyslogLogger(ThreadVars *tv, void *thread_data, const Packet *p)
diff --git a/src/app-layer-ssh.c b/src/app-layer-ssh.c
index 0cf404c8736e..71bc786ad6b4 100644
--- a/src/app-layer-ssh.c
+++ b/src/app-layer-ssh.c
@@ -71,7 +71,7 @@ static int SSHRegisterPatternsForProtocolDetection(void)
     return 0;
 }
 
-int SSHTxLogCondition(ThreadVars * tv, const Packet * p, void *state, void *tx, uint64_t tx_id)
+bool SSHTxLogCondition(ThreadVars *tv, const Packet *p, void *state, void *tx, uint64_t tx_id)
 {
     return rs_ssh_tx_get_log_condition(tx);
 }
diff --git a/src/app-layer-ssh.h b/src/app-layer-ssh.h
index 8dbb3be817ef..996cc260c735 100644
--- a/src/app-layer-ssh.h
+++ b/src/app-layer-ssh.h
@@ -28,7 +28,7 @@
 void RegisterSSHParsers(void);
 void SSHParserRegisterTests(void);
 
-int SSHTxLogCondition(ThreadVars *, const Packet *, void *state, void *tx, uint64_t tx_id);
+bool SSHTxLogCondition(ThreadVars *, const Packet *, void *state, void *tx, uint64_t tx_id);
 
 #endif /* __APP_LAYER_SSH_H__ */
 
diff --git a/src/log-pcap.c b/src/log-pcap.c
index 1e1b6da1fb55..7f16ddc7924b 100644
--- a/src/log-pcap.c
+++ b/src/log-pcap.c
@@ -206,7 +206,7 @@ static TmEcode PcapLogDataDeinit(ThreadVars *, void *);
 static void PcapLogFileDeInitCtx(OutputCtx *);
 static OutputInitResult PcapLogInitCtx(ConfNode *);
 static void PcapLogProfilingDump(PcapLogData *);
-static int PcapLogCondition(ThreadVars *, void *, const Packet *);
+static bool PcapLogCondition(ThreadVars *, void *, const Packet *);
 
 void PcapLogRegister(void)
 {
@@ -226,7 +226,7 @@ void PcapLogRegister(void)
     (prof).total += (UtilCpuGetTicks() - pcaplog_profile_ticks); \
     (prof).cnt++
 
-static int PcapLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool PcapLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
     PcapLogThreadData *ptd = (PcapLogThreadData *)thread_data;
 
@@ -235,29 +235,21 @@ static int PcapLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
         case LOGMODE_COND_ALL:
             break;
         case LOGMODE_COND_ALERTS:
-            if (p->alerts.cnt || (p->flow && FlowHasAlerts(p->flow))) {
-                return TRUE;
-            } else {
-                return FALSE;
-            }
+            return (p->alerts.cnt || (p->flow && FlowHasAlerts(p->flow)));
             break;
         case LOGMODE_COND_TAG:
-            if (p->flags & (PKT_HAS_TAG | PKT_FIRST_TAG)) {
-                return TRUE;
-            } else {
-                return FALSE;
-            }
+            return (p->flags & (PKT_HAS_TAG | PKT_FIRST_TAG));
             break;
     }
 
     if (p->flags & PKT_PSEUDO_STREAM_END) {
-        return FALSE;
+        return false;
     }
 
     if (IS_TUNNEL_PKT(p) && !IS_TUNNEL_ROOT_PKT(p)) {
-        return FALSE;
+        return false;
     }
-    return TRUE;
+    return true;
 }
 
 /**
diff --git a/src/log-tlsstore.c b/src/log-tlsstore.c
index 4e1d54a5fe3a..969044553673 100644
--- a/src/log-tlsstore.c
+++ b/src/log-tlsstore.c
@@ -226,15 +226,15 @@ static void LogTlsLogPem(LogTlsStoreLogThread *aft, const Packet *p, SSLState *s
  *  \brief Condition function for TLS logger
  *  \retval bool true or false -- log now?
  */
-static int LogTlsStoreCondition(ThreadVars *tv, const Packet *p, void *state,
-                                void *tx, uint64_t tx_id)
+static bool LogTlsStoreCondition(
+        ThreadVars *tv, const Packet *p, void *state, void *tx, uint64_t tx_id)
 {
     if (p->flow == NULL) {
-        return FALSE;
+        return false;
     }
 
     if (!(PKT_IS_TCP(p))) {
-        return FALSE;
+        return false;
     }
 
     SSLState *ssl_state = (SSLState *)state;
@@ -250,9 +250,9 @@ static int LogTlsStoreCondition(ThreadVars *tv, const Packet *p, void *state,
             ssl_state->server_connp.cert0_subject == NULL)
         goto dontlog;
 
-    return TRUE;
+    return true;
 dontlog:
-    return FALSE;
+    return false;
 }
 
 static int LogTlsStoreLogger(ThreadVars *tv, void *thread_data, const Packet *p,
diff --git a/src/output-eve-stream.c b/src/output-eve-stream.c
index 446fb3e60e18..919505dce70d 100644
--- a/src/output-eve-stream.c
+++ b/src/output-eve-stream.c
@@ -282,7 +282,7 @@ static void LogStream(const TcpStream *stream, JsonBuilder *js)
  * \param data  Pointer to the EveStreamLogThread struct
  * \param p     Pointer the packet which is being logged
  *
- * \retval 0 on succes
+ * \retval 0 on success
  */
 static int EveStreamLogger(ThreadVars *tv, void *thread_data, const Packet *p)
 {
@@ -422,9 +422,9 @@ static int EveStreamLogger(ThreadVars *tv, void *thread_data, const Packet *p)
  * \param tv    Pointer the current thread variables
  * \param p     Pointer the packet which is tested
  *
- * \retval bool TRUE or FALSE
+ * \retval bool true or false
  */
-static int EveStreamLogCondition(ThreadVars *tv, void *data, const Packet *p)
+static bool EveStreamLogCondition(ThreadVars *tv, void *data, const Packet *p)
 {
     EveStreamLogThread *td = data;
     EveStreamOutputCtx *ctx = td->stream_ctx;
diff --git a/src/output-json-alert.c b/src/output-json-alert.c
index a7df1065509e..c3886231c001 100644
--- a/src/output-json-alert.c
+++ b/src/output-json-alert.c
@@ -947,12 +947,9 @@ static int JsonAlertLogger(ThreadVars *tv, void *thread_data, const Packet *p)
     return 0;
 }
 
-static int JsonAlertLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool JsonAlertLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
-    if (p->alerts.cnt || (p->flags & PKT_HAS_TAG)) {
-        return TRUE;
-    }
-    return FALSE;
+    return (p->alerts.cnt || (p->flags & PKT_HAS_TAG));
 }
 
 static TmEcode JsonAlertLogThreadInit(ThreadVars *t, const void *initdata, void **data)
diff --git a/src/output-json-anomaly.c b/src/output-json-anomaly.c
index 606ead0e6224..ffe931a73ed6 100644
--- a/src/output-json-anomaly.c
+++ b/src/output-json-anomaly.c
@@ -279,7 +279,7 @@ static int JsonAnomalyLogger(ThreadVars *tv, void *thread_data, const Packet *p)
     return AnomalyJson(tv, aft, p);
 }
 
-static int JsonAnomalyLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool JsonAnomalyLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
     return p->events.cnt > 0 ||
            (p->app_layer_events && p->app_layer_events->cnt > 0) ||
diff --git a/src/output-json-drop.c b/src/output-json-drop.c
index 56484c36d43b..edce3793d100 100644
--- a/src/output-json-drop.c
+++ b/src/output-json-drop.c
@@ -340,49 +340,48 @@ static int JsonDropLogger(ThreadVars *tv, void *thread_data, const Packet *p)
     return 0;
 }
 
-
 /**
  * \brief Check if we need to drop-log this packet
  *
  * \param tv    Pointer the current thread variables
  * \param p     Pointer the packet which is tested
  *
- * \retval bool TRUE or FALSE
+ * \retval bool true or false
  */
-static int JsonDropLogCondition(ThreadVars *tv, void *data, const Packet *p)
+static bool JsonDropLogCondition(ThreadVars *tv, void *data, const Packet *p)
 {
     if (!EngineModeIsIPS()) {
         SCLogDebug("engine is not running in inline mode, so returning");
-        return FALSE;
+        return false;
     }
     if (PKT_IS_PSEUDOPKT(p)) {
         SCLogDebug("drop log doesn't log pseudo packets");
-        return FALSE;
+        return false;
     }
 
     if (!(PacketCheckAction(p, ACTION_DROP))) {
-        return FALSE;
+        return false;
     }
 
     if (g_droplog_flows_start && p->flow != NULL) {
-        int ret = FALSE;
+        bool ret = false;
 
         /* for a flow that will be dropped fully, log just once per direction */
         if (p->flow->flags & FLOW_ACTION_DROP) {
             if (PKT_IS_TOSERVER(p) && !(p->flow->flags & FLOW_TOSERVER_DROP_LOGGED))
-                ret = TRUE;
+                ret = true;
             else if (PKT_IS_TOCLIENT(p) && !(p->flow->flags & FLOW_TOCLIENT_DROP_LOGGED))
-                ret = TRUE;
+                ret = true;
         }
 
         /* if drop is caused by signature, log anyway */
         if (p->alerts.drop.action != 0)
-            ret = TRUE;
+            ret = true;
 
         return ret;
     }
 
-    return TRUE;
+    return true;
 }
 
 void JsonDropLogRegister (void)
diff --git a/src/output-json-frame.c b/src/output-json-frame.c
index d23ba92ac380..b7aaabc1dea9 100644
--- a/src/output-json-frame.c
+++ b/src/output-json-frame.c
@@ -376,15 +376,15 @@ static int JsonFrameLogger(ThreadVars *tv, void *thread_data, const Packet *p)
     return FrameJson(tv, aft, p);
 }
 
-static int JsonFrameLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
+static bool JsonFrameLogCondition(ThreadVars *tv, void *thread_data, const Packet *p)
 {
     if (p->flow == NULL || p->flow->alproto == ALPROTO_UNKNOWN)
-        return FALSE;
+        return false;
 
     if ((p->proto == IPPROTO_TCP || p->proto == IPPROTO_UDP) && p->flow->alparser != NULL) {
         FramesContainer *frames_container = AppLayerFramesGetContainer(p->flow);
         if (frames_container == NULL)
-            return FALSE;
+            return false;
 
         Frames *frames;
         if (PKT_IS_TOSERVER(p)) {
@@ -394,7 +394,7 @@ static int JsonFrameLogCondition(ThreadVars *tv, void *thread_data, const Packet
         }
         return (frames->cnt != 0);
     }
-    return FALSE;
+    return false;
 }
 
 static TmEcode JsonFrameLogThreadInit(ThreadVars *t, const void *initdata, void **data)
diff --git a/src/output-json-metadata.c b/src/output-json-metadata.c
index 231ff1d5d3e2..772b2ccb06f2 100644
--- a/src/output-json-metadata.c
+++ b/src/output-json-metadata.c
@@ -87,12 +87,9 @@ static int JsonMetadataLogger(ThreadVars *tv, void *thread_data, const Packet *p
     return MetadataJson(tv, aft, p);
 }
 
-static int JsonMetadataLogCondition(ThreadVars *tv, void *data, const Packet *p)
+static bool JsonMetadataLogCondition(ThreadVars *tv, void *data, const Packet *p)
 {
-    if (p->pktvar) {
-        return TRUE;
-    }
-    return FALSE;
+    return p->pktvar != NULL;
 }
 
 void JsonMetadataLogRegister (void)
diff --git a/src/output-lua.c b/src/output-lua.c
index cc93a2de4a54..28ba3e9f91b7 100644
--- a/src/output-lua.c
+++ b/src/output-lua.c
@@ -162,7 +162,7 @@ static int LuaStreamingLogger(ThreadVars *tv, void *thread_data, const Flow *f,
  *
  *  A single call to this function will run one script for a single
  *  packet. If it is called, it means that the registered condition
- *  function has returned TRUE.
+ *  function has returned true.
  *
  *  The script is called once for each alert stored in the packet.
  *
@@ -215,11 +215,9 @@ static int LuaPacketLoggerAlerts(ThreadVars *tv, void *thread_data, const Packet
     SCReturnInt(0);
 }
 
-static int LuaPacketConditionAlerts(ThreadVars *tv, void *data, const Packet *p)
+static bool LuaPacketConditionAlerts(ThreadVars *tv, void *data, const Packet *p)
 {
-    if (p->alerts.cnt > 0)
-        return TRUE;
-    return FALSE;
+    return (p->alerts.cnt > 0);
 }
 
 /** \internal
@@ -227,7 +225,7 @@ static int LuaPacketConditionAlerts(ThreadVars *tv, void *data, const Packet *p)
  *
  *  A single call to this function will run one script for a single
  *  packet. If it is called, it means that the registered condition
- *  function has returned TRUE.
+ *  function has returned true.
  *
  *  The script is called once for each packet.
  *
@@ -265,9 +263,9 @@ static int LuaPacketLogger(ThreadVars *tv, void *thread_data, const Packet *p)
     SCReturnInt(0);
 }
 
-static int LuaPacketCondition(ThreadVars *tv, void *data, const Packet *p)
+static bool LuaPacketCondition(ThreadVars *tv, void *data, const Packet *p)
 {
-    return TRUE;
+    return true;
 }
 
 /** \internal
diff --git a/src/output-packet.c b/src/output-packet.c
index 232be2697e5a..d42d1033cade 100644
--- a/src/output-packet.c
+++ b/src/output-packet.c
@@ -105,7 +105,7 @@ static TmEcode OutputPacketLog(ThreadVars *tv, Packet *p, void *thread_data)
     while (logger && store) {
         DEBUG_VALIDATE_BUG_ON(logger->LogFunc == NULL || logger->ConditionFunc == NULL);
 
-        if ((logger->ConditionFunc(tv, store->thread_data, (const Packet *)p)) == TRUE) {
+        if (logger->ConditionFunc(tv, store->thread_data, (const Packet *)p)) {
             PACKET_PROFILING_LOGGER_START(p, logger->logger_id);
             logger->LogFunc(tv, store->thread_data, (const Packet *)p);
             PACKET_PROFILING_LOGGER_END(p, logger->logger_id);
diff --git a/src/output-packet.h b/src/output-packet.h
index adba55689122..1e468669f137 100644
--- a/src/output-packet.h
+++ b/src/output-packet.h
@@ -32,7 +32,7 @@ typedef int (*PacketLogger)(ThreadVars *, void *thread_data, const Packet *);
 /** packet logger condition function pointer type,
  *  must return true for packets that should be logged
  */
-typedef int (*PacketLogCondition)(ThreadVars *, void *thread_data, const Packet *);
+typedef bool (*PacketLogCondition)(ThreadVars *, void *thread_data, const Packet *);
 
 int OutputRegisterPacketLogger(LoggerId logger_id, const char *name,
     PacketLogger LogFunc, PacketLogCondition ConditionFunc, OutputCtx *,
diff --git a/src/output-tx.c b/src/output-tx.c
index 18a34e78a734..8eb6a842a656 100644
--- a/src/output-tx.c
+++ b/src/output-tx.c
@@ -299,8 +299,7 @@ static void OutputTxLogCallLoggers(ThreadVars *tv, OutputTxLoggerThreadData *op_
                 SCLogDebug("EOF, so log now");
             } else {
                 if (logger->LogCondition) {
-                    int r = logger->LogCondition(tv, p, alstate, tx, tx_id);
-                    if (r == FALSE) {
+                    if (!logger->LogCondition(tv, p, alstate, tx, tx_id)) {
                         SCLogDebug("conditions not met, not logging");
                         goto next_logger;
                     }
diff --git a/src/output-tx.h b/src/output-tx.h
index 8d58156bef8d..88c12ff25f68 100644
--- a/src/output-tx.h
+++ b/src/output-tx.h
@@ -35,7 +35,8 @@ typedef int (*TxLogger)(ThreadVars *, void *thread_data, const Packet *, Flow *f
 /** tx logger condition function pointer type,
  *  must return true for tx that should be logged
  */
-typedef int (*TxLoggerCondition)(ThreadVars *, const Packet *, void *state, void *tx, uint64_t tx_id);
+typedef bool (*TxLoggerCondition)(
+        ThreadVars *, const Packet *, void *state, void *tx, uint64_t tx_id);
 
 int OutputRegisterTxLogger(LoggerId id, const char *name, AppProto alproto,
         TxLogger LogFunc,

From 051a14acd30b31eccc3ca96a58240cd7be89142d Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 09:30:09 -0400
Subject: [PATCH 132/462] general/bool: Use bool for file support

---
 src/app-layer-parser.c | 6 ++----
 src/app-layer-parser.h | 2 +-
 2 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/src/app-layer-parser.c b/src/app-layer-parser.c
index 7783c076b65b..572e15f628cc 100644
--- a/src/app-layer-parser.c
+++ b/src/app-layer-parser.c
@@ -1184,16 +1184,14 @@ uint64_t AppLayerParserGetTransactionActive(const Flow *f,
     SCReturnCT(active_id, "uint64_t");
 }
 
-int AppLayerParserSupportsFiles(uint8_t ipproto, AppProto alproto)
+bool AppLayerParserSupportsFiles(uint8_t ipproto, AppProto alproto)
 {
     // Custom case for only signature-only protocol so far
     if (alproto == ALPROTO_HTTP) {
         return AppLayerParserSupportsFiles(ipproto, ALPROTO_HTTP1) ||
                AppLayerParserSupportsFiles(ipproto, ALPROTO_HTTP2);
     }
-    if (alp_ctx.ctxs[FlowGetProtoMapping(ipproto)][alproto].GetTxFiles != NULL)
-        return TRUE;
-    return FALSE;
+    return alp_ctx.ctxs[FlowGetProtoMapping(ipproto)][alproto].GetTxFiles != NULL;
 }
 
 AppLayerTxData *AppLayerParserGetTxData(uint8_t ipproto, AppProto alproto, void *tx)
diff --git a/src/app-layer-parser.h b/src/app-layer-parser.h
index d27a08c85119..e9f8cf55e925 100644
--- a/src/app-layer-parser.h
+++ b/src/app-layer-parser.h
@@ -253,7 +253,7 @@ uint64_t AppLayerParserGetTransactionActive(const Flow *f, AppLayerParserState *
 
 uint8_t AppLayerParserGetFirstDataDir(uint8_t ipproto, AppProto alproto);
 
-int AppLayerParserSupportsFiles(uint8_t ipproto, AppProto alproto);
+bool AppLayerParserSupportsFiles(uint8_t ipproto, AppProto alproto);
 
 AppLayerTxData *AppLayerParserGetTxData(uint8_t ipproto, AppProto alproto, void *tx);
 uint64_t AppLayerParserGetTxDetectFlags(AppLayerTxData *txd, const uint8_t dir);

From 9bd2b7425d04fb4427d1acedd7ef403bd840172f Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 09:59:06 -0400
Subject: [PATCH 133/462] general/bool: Change Suricata int to bool

Change Suricata operational values from int to bool.
---
 src/detect-engine-loader.c    |  4 ++--
 src/detect-engine.c           |  4 ++--
 src/detect-parse.c            |  4 ++--
 src/detect.h                  |  2 +-
 src/respond-reject-libnet11.c |  2 +-
 src/suricata.c                | 30 +++++++++++++++---------------
 src/suricata.h                |  6 +++---
 src/util-landlock.c           |  2 +-
 src/util-privs.c              |  4 ++--
 9 files changed, 29 insertions(+), 29 deletions(-)

diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index ae01f406e9ec..40919568503a 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -274,7 +274,7 @@ static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern,
  *  \param sig_file_exclusive File passed in 'sig_file' should be loaded exclusively.
  *  \retval -1 on error
  */
-int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, int sig_file_exclusive)
+int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, bool sig_file_exclusive)
 {
     SCEnter();
 
@@ -297,7 +297,7 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, int sig_file_excl
     }
 
     /* ok, let's load signature files from the general config */
-    if (!(sig_file != NULL && sig_file_exclusive == TRUE)) {
+    if (!(sig_file != NULL && sig_file_exclusive)) {
         rule_files = ConfGetNode(varname);
         if (rule_files != NULL) {
             if (!ConfNodeIsSequence(rule_files)) {
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 6fa894c0c794..0fc2df6869b0 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -3859,7 +3859,7 @@ static int DetectEngineMultiTenantLoadTenant(uint32_t tenant_id, const char *fil
         goto error;
     }
 
-    if (SigLoadSignatures(de_ctx, NULL, 0) < 0) {
+    if (SigLoadSignatures(de_ctx, NULL, false) < 0) {
         SCLogError("Loading signatures failed.");
         goto error;
     }
@@ -3919,7 +3919,7 @@ static int DetectEngineMultiTenantReloadTenant(uint32_t tenant_id, const char *f
         goto error;
     }
 
-    if (SigLoadSignatures(new_de_ctx, NULL, 0) < 0) {
+    if (SigLoadSignatures(new_de_ctx, NULL, false) < 0) {
         SCLogError("Loading signatures failed.");
         goto error;
     }
diff --git a/src/detect-parse.c b/src/detect-parse.c
index d9800f0a2f34..bf54f9359a0e 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -128,7 +128,7 @@ void DetectFileRegisterFileProtocols(DetectFileHandlerTableElmt *reg)
 /* Table with all SigMatch registrations */
 SigTableElmt sigmatch_table[DETECT_TBLSIZE];
 
-extern int sc_set_caps;
+extern bool sc_set_caps;
 
 static void SigMatchTransferSigMatchAcrossLists(SigMatch *sm,
         SigMatch **src_sm_list, SigMatch **src_sm_list_tail,
@@ -1174,7 +1174,7 @@ static int SigParseActionRejectValidate(const char *action)
 {
 #ifdef HAVE_LIBNET11
 #if defined HAVE_LIBCAP_NG && !defined HAVE_LIBNET_CAPABILITIES
-    if (sc_set_caps == TRUE) {
+    if (sc_set_caps) {
         SCLogError("Libnet 1.1 is "
                    "incompatible with POSIX based capabilities with privs dropping. "
                    "For rejects to work, run as root/super user.");
diff --git a/src/detect.h b/src/detect.h
index a4d9ef360227..ced030067070 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1559,7 +1559,7 @@ void SigRegisterTests(void);
 
 void DisableDetectFlowFileFlags(Flow *f);
 char *DetectLoadCompleteSigPath(const DetectEngineCtx *, const char *sig_file);
-int SigLoadSignatures (DetectEngineCtx *, char *, int);
+int SigLoadSignatures(DetectEngineCtx *, char *, bool);
 void SigMatchSignatures(ThreadVars *th_v, DetectEngineCtx *de_ctx,
                        DetectEngineThreadCtx *det_ctx, Packet *p);
 
diff --git a/src/respond-reject-libnet11.c b/src/respond-reject-libnet11.c
index 95ff114484db..634fa3b8b545 100644
--- a/src/respond-reject-libnet11.c
+++ b/src/respond-reject-libnet11.c
@@ -62,7 +62,7 @@ uint16_t g_reject_dev_mtu = 0;
 
 /** set to true in main if we're setting caps. We need it here if we're using
   * reject rules as libnet 1.1 is not compatible with caps. */
-extern int sc_set_caps;
+extern bool sc_set_caps;
 
 #include <libnet.h>
 
diff --git a/src/suricata.c b/src/suricata.c
index 7f979a7fbfcc..fd069e6e5ddd 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -189,7 +189,7 @@ uint16_t max_pending_packets;
 int g_detect_disabled = 0;
 
 /** set caps or not */
-int sc_set_caps = FALSE;
+bool sc_set_caps = false;
 
 bool g_system = false;
 
@@ -1075,7 +1075,7 @@ static void SCInstanceInit(SCInstance *suri, const char *progname)
 
     memset(suri->pcap_dev, 0, sizeof(suri->pcap_dev));
     suri->sig_file = NULL;
-    suri->sig_file_exclusive = FALSE;
+    suri->sig_file_exclusive = false;
     suri->pid_filename = NULL;
     suri->regex_arg = NULL;
 
@@ -1084,8 +1084,8 @@ static void SCInstanceInit(SCInstance *suri, const char *progname)
 #ifndef OS_WIN32
     suri->user_name = NULL;
     suri->group_name = NULL;
-    suri->do_setuid = FALSE;
-    suri->do_setgid = FALSE;
+    suri->do_setuid = false;
+    suri->do_setgid = false;
 #endif /* OS_WIN32 */
     suri->userid = 0;
     suri->groupid = 0;
@@ -1605,7 +1605,7 @@ static TmEcode ParseCommandLine(int argc, char** argv, SCInstance *suri)
                 return TM_ECODE_FAILED;
 #else
                 suri->user_name = optarg;
-                suri->do_setuid = TRUE;
+                suri->do_setuid = true;
 #endif /* HAVE_LIBCAP_NG */
             } else if (strcmp((long_opts[option_index]).name, "group") == 0) {
 #ifndef HAVE_LIBCAP_NG
@@ -1614,7 +1614,7 @@ static TmEcode ParseCommandLine(int argc, char** argv, SCInstance *suri)
                 return TM_ECODE_FAILED;
 #else
                 suri->group_name = optarg;
-                suri->do_setgid = TRUE;
+                suri->do_setgid = true;
 #endif /* HAVE_LIBCAP_NG */
             } else if (strcmp((long_opts[option_index]).name, "erf-in") == 0) {
                 suri->run_mode = RUNMODE_ERF_FILE;
@@ -1972,7 +1972,7 @@ static TmEcode ParseCommandLine(int argc, char** argv, SCInstance *suri)
                 return TM_ECODE_FAILED;
             }
             suri->sig_file = optarg;
-            suri->sig_file_exclusive = TRUE;
+            suri->sig_file_exclusive = true;
             break;
         case 'u':
 #ifdef UNITTESTS
@@ -2142,25 +2142,25 @@ static int InitRunAs(SCInstance *suri)
 #ifndef OS_WIN32
     /* Try to get user/group to run suricata as if
        command line as not decide of that */
-    if (suri->do_setuid == FALSE && suri->do_setgid == FALSE) {
+    if (!suri->do_setuid && !suri->do_setgid) {
         const char *id;
         if (ConfGet("run-as.user", &id) == 1) {
-            suri->do_setuid = TRUE;
+            suri->do_setuid = true;
             suri->user_name = id;
         }
         if (ConfGet("run-as.group", &id) == 1) {
-            suri->do_setgid = TRUE;
+            suri->do_setgid = true;
             suri->group_name = id;
         }
     }
     /* Get the suricata user ID to given user ID */
-    if (suri->do_setuid == TRUE) {
+    if (suri->do_setuid) {
         SCGetUserID(suri->user_name, suri->group_name, &suri->userid, &suri->groupid);
-        sc_set_caps = TRUE;
-    /* Get the suricata group ID to given group ID */
-    } else if (suri->do_setgid == TRUE) {
+        sc_set_caps = true;
+        /* Get the suricata group ID to given group ID */
+    } else if (suri->do_setgid) {
         SCGetGroupID(suri->group_name, &suri->groupid);
-        sc_set_caps = TRUE;
+        sc_set_caps = true;
     }
 #endif
     return TM_ECODE_OK;
diff --git a/src/suricata.h b/src/suricata.h
index 957134b92c06..9d275edc5dd0 100644
--- a/src/suricata.h
+++ b/src/suricata.h
@@ -126,7 +126,7 @@ typedef struct SCInstance_ {
 
     char pcap_dev[128];
     char *sig_file;
-    int sig_file_exclusive;
+    bool sig_file_exclusive;
     char *pid_filename;
     char *regex_arg;
 
@@ -135,8 +135,8 @@ typedef struct SCInstance_ {
 #ifndef OS_WIN32
     const char *user_name;
     const char *group_name;
-    uint8_t do_setuid;
-    uint8_t do_setgid;
+    bool do_setuid;
+    bool do_setgid;
 #endif /* OS_WIN32 */
     uint32_t userid;
     uint32_t groupid;
diff --git a/src/util-landlock.c b/src/util-landlock.c
index d14e1bed0aeb..258993c30d5c 100644
--- a/src/util-landlock.c
+++ b/src/util-landlock.c
@@ -244,7 +244,7 @@ void LandlockSandboxing(SCInstance *suri)
             LandlockSandboxingWritePath(ruleset, LOCAL_STATE_DIR "/run/suricata/");
         }
     }
-    if (suri->sig_file_exclusive == FALSE) {
+    if (!suri->sig_file_exclusive) {
         const char *rule_path;
         if (ConfGet("default-rule-path", &rule_path) == 1 && rule_path) {
             LandlockSandboxingReadPath(ruleset, rule_path);
diff --git a/src/util-privs.c b/src/util-privs.c
index 8210cc8065d9..3a1ea485159a 100644
--- a/src/util-privs.c
+++ b/src/util-privs.c
@@ -44,7 +44,7 @@
 #include "runmodes.h"
 
 /** flag indicating if we'll be using caps */
-extern int sc_set_caps;
+extern bool sc_set_caps;
 
 /** our current runmode */
 extern int run_mode;
@@ -54,7 +54,7 @@ extern int run_mode;
  */
 void SCDropMainThreadCaps(uint32_t userid, uint32_t groupid)
 {
-    if (sc_set_caps == FALSE)
+    if (!sc_set_caps)
         return;
 
     capng_clear(CAPNG_SELECT_BOTH);

From 2016d68f414787808f6f24829d0c5a2707ff3a17 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:09:02 -0400
Subject: [PATCH 134/462] stream/bool: Use bool for StreamTcpInlineMode

---
 src/stream-tcp-list.c       |  6 +++---
 src/stream-tcp-reassemble.c | 12 ++++++------
 src/stream-tcp.c            |  4 ++--
 src/stream-tcp.h            |  2 +-
 4 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/stream-tcp-list.c b/src/stream-tcp-list.c
index 164825440f0c..2b8a4d079cef 100644
--- a/src/stream-tcp-list.c
+++ b/src/stream-tcp-list.c
@@ -779,7 +779,7 @@ static inline uint64_t GetLeftEdge(Flow *f, TcpSession *ssn, TcpStream *stream)
     if (use_raw) {
         uint64_t raw_progress = STREAM_RAW_PROGRESS(stream);
 
-        if (StreamTcpInlineMode() == TRUE) {
+        if (StreamTcpInlineMode()) {
             uint32_t chunk_size = (stream == &ssn->client) ?
                 stream_config.reassembly_toserver_chunk_size :
                 stream_config.reassembly_toclient_chunk_size;
@@ -834,14 +834,14 @@ static inline uint64_t GetLeftEdge(Flow *f, TcpSession *ssn, TcpStream *stream)
         last_ack_abs += (stream->last_ack - stream->base_seq);
     }
     /* in IDS mode we shouldn't see the base_seq pass last_ack */
-    DEBUG_VALIDATE_BUG_ON(last_ack_abs < left_edge && StreamTcpInlineMode() == FALSE && !f->ffr &&
+    DEBUG_VALIDATE_BUG_ON(last_ack_abs < left_edge && !StreamTcpInlineMode() && !f->ffr &&
                           ssn->state < TCP_CLOSED);
     left_edge = MIN(left_edge, last_ack_abs);
 
     /* if we're told to look for overlaps with different data we should
      * consider data that is ack'd as well. Injected packets may have
      * been ack'd or injected packet may be too late. */
-    if (StreamTcpInlineMode() == FALSE && check_overlap_different_data) {
+    if (!StreamTcpInlineMode() && check_overlap_different_data) {
         const uint32_t window = stream->window ? stream->window : 4096;
         if (window < left_edge)
             left_edge -= window;
diff --git a/src/stream-tcp-reassemble.c b/src/stream-tcp-reassemble.c
index 737b222d53e2..06992da791a5 100644
--- a/src/stream-tcp-reassemble.c
+++ b/src/stream-tcp-reassemble.c
@@ -421,7 +421,7 @@ uint64_t StreamTcpGetAcked(const TcpStream *stream)
 uint64_t StreamDataRightEdge(const TcpStream *stream, const bool eof)
 {
     uint64_t right_edge = STREAM_BASE_OFFSET(stream) + stream->segs_right_edge - stream->base_seq;
-    if (!eof && StreamTcpInlineMode() == FALSE) {
+    if (!eof && !StreamTcpInlineMode()) {
         right_edge = MIN(GetAbsLastAck(stream), right_edge);
     }
     return right_edge;
@@ -430,7 +430,7 @@ uint64_t StreamDataRightEdge(const TcpStream *stream, const bool eof)
 uint64_t StreamTcpGetUsable(const TcpStream *stream, const bool eof)
 {
     uint64_t right_edge = StreamingBufferGetConsecutiveDataRightEdge(&stream->sb);
-    if (!eof && StreamTcpInlineMode() == FALSE) {
+    if (!eof && !StreamTcpInlineMode()) {
         right_edge = MIN(GetAbsLastAck(stream), right_edge);
     }
     return right_edge;
@@ -496,7 +496,7 @@ static int StreamTcpReassemblyConfig(bool quiet)
     if (overlap_diff_data) {
         StreamTcpReassembleConfigEnableOverlapCheck();
     }
-    if (StreamTcpInlineMode() == TRUE) {
+    if (StreamTcpInlineMode()) {
         StreamTcpReassembleConfigEnableOverlapCheck();
     }
 
@@ -1189,7 +1189,7 @@ static inline uint32_t AdjustToAcked(const Packet *p,
     uint32_t adjusted = data_len;
 
     /* get window of data that is acked */
-    if (StreamTcpInlineMode() == FALSE) {
+    if (!StreamTcpInlineMode()) {
         SCLogDebug("ssn->state %s", StreamTcpStateAsString(ssn->state));
         if (data_len == 0 || ((ssn->state < TCP_CLOSED ||
                                       (ssn->state == TCP_CLOSED &&
@@ -1481,7 +1481,7 @@ bool StreamReassembleRawHasDataReady(TcpSession *ssn, Packet *p)
                          STREAMTCP_STREAM_FLAG_DISABLE_RAW))
         return false;
 
-    if (StreamTcpInlineMode() == FALSE) {
+    if (!StreamTcpInlineMode()) {
         const uint64_t segs_re_abs =
                 STREAM_BASE_OFFSET(stream) + stream->segs_right_edge - stream->base_seq;
         if (STREAM_RAW_PROGRESS(stream) == segs_re_abs) {
@@ -1860,7 +1860,7 @@ int StreamReassembleRaw(TcpSession *ssn, const Packet *p,
                         uint64_t *progress_out, bool respect_inspect_depth)
 {
     /* handle inline separately as the logic is very different */
-    if (StreamTcpInlineMode() == TRUE) {
+    if (StreamTcpInlineMode()) {
         return StreamReassembleRawInline(ssn, p, Callback, cb_data, progress_out);
     }
 
diff --git a/src/stream-tcp.c b/src/stream-tcp.c
index d76a0593a0d2..d41110aac5c7 100644
--- a/src/stream-tcp.c
+++ b/src/stream-tcp.c
@@ -6851,9 +6851,9 @@ int StreamTcpBypassEnabled(void)
  *  \retval 0 no
  *  \retval 1 yes
  */
-int StreamTcpInlineMode(void)
+bool StreamTcpInlineMode(void)
 {
-    return (stream_config.flags & STREAMTCP_INIT_FLAG_INLINE) ? 1 : 0;
+    return (stream_config.flags & STREAMTCP_INIT_FLAG_INLINE);
 }
 
 
diff --git a/src/stream-tcp.h b/src/stream-tcp.h
index 324671245995..ff8a0998cb4e 100644
--- a/src/stream-tcp.h
+++ b/src/stream-tcp.h
@@ -191,7 +191,7 @@ void StreamTcpSessionCleanup(TcpSession *ssn);
 void StreamTcpStreamCleanup(TcpStream *stream);
 /* check if bypass is enabled */
 int StreamTcpBypassEnabled(void);
-int StreamTcpInlineMode(void);
+bool StreamTcpInlineMode(void);
 
 int TcpSessionPacketSsnReuse(const Packet *p, const Flow *f, const void *tcp_ssn);
 

From d2c46110d6055960730ceaa4efaeb14c3c97947d Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:36:15 -0400
Subject: [PATCH 135/462] pcap/bool: Use bool type for is_private

---
 src/log-pcap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/log-pcap.c b/src/log-pcap.c
index 7f16ddc7924b..f8c8bf716562 100644
--- a/src/log-pcap.c
+++ b/src/log-pcap.c
@@ -141,7 +141,6 @@ typedef struct PcapLogCompressionData_ {
 typedef struct PcapLogData_ {
     int use_stream_depth;       /**< use stream depth i.e. ignore packets that reach limit */
     int honor_pass_rules;       /**< don't log if pass rules have matched */
-    int is_private;             /**< TRUE if ctx is thread local */
     SCMutex plog_lock;
     uint64_t pkt_cnt;		    /**< total number of packets */
     struct pcap_pkthdr *h;      /**< pcap header struct */
@@ -155,6 +154,7 @@ typedef struct PcapLogData_ {
     uint64_t profile_data_size; /**< track in bytes how many bytes we wrote */
     uint32_t file_cnt;          /**< count of pcap files we currently have */
     uint32_t max_files;         /**< maximum files to use in ring buffer mode */
+    bool is_private;            /**< true if ctx is thread local */
     LogModeConditionalType
             conditional; /**< log all packets or just packets and flows with alerts */
 
@@ -763,7 +763,7 @@ static PcapLogData *PcapLogDataCopy(const PcapLogData *pl)
     copy->suffix = pl->suffix;
 
     /* settings TODO move to global cfg struct */
-    copy->is_private = TRUE;
+    copy->is_private = true;
     copy->mode = pl->mode;
     copy->max_files = pl->max_files;
     copy->use_ringbuffer = pl->use_ringbuffer;

From 8f2a3ea7be8087201e14ab2de25dffe7de6afe68 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:41:18 -0400
Subject: [PATCH 136/462] prefilter/bool: Use bool values for is_last

---
 src/detect-engine-prefilter.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/detect-engine-prefilter.c b/src/detect-engine-prefilter.c
index 3c33071e7211..e40a5175dafb 100644
--- a/src/detect-engine-prefilter.c
+++ b/src/detect-engine-prefilter.c
@@ -453,7 +453,7 @@ void PrefilterSetupRuleGroup(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
             el->pectx = NULL; // e now owns the ctx
             e->gid = el->gid;
             if (el->next == NULL) {
-                e->is_last = TRUE;
+                e->is_last = true;
             }
             e++;
         }
@@ -477,7 +477,7 @@ void PrefilterSetupRuleGroup(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
             el->pectx = NULL; // e now owns the ctx
             e->gid = el->gid;
             if (el->next == NULL) {
-                e->is_last = TRUE;
+                e->is_last = true;
             }
             e++;
         }
@@ -579,7 +579,7 @@ void PrefilterSetupRuleGroup(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
             el->pectx = NULL; // e now owns the ctx
             e->gid = el->gid;
             if (el->next == NULL) {
-                e->is_last = TRUE;
+                e->is_last = true;
             }
             e++;
         }

From 84b2d665d63f9f632a51bb42767266e6421cae80 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:41:44 -0400
Subject: [PATCH 137/462] detect/bool: Use bool type for unittests

---
 src/detect-engine-address.c | 48 ++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 24 deletions(-)

diff --git a/src/detect-engine-address.c b/src/detect-engine-address.c
index ac10e142dedc..7819969e6ef3 100644
--- a/src/detect-engine-address.c
+++ b/src/detect-engine-address.c
@@ -1842,12 +1842,12 @@ DetectAddress *DetectAddressLookupInHead(const DetectAddressHead *gh, Address *a
 
 #ifdef UNITTESTS
 
-static int UTHValidateDetectAddress(DetectAddress *ad, const char *one, const char *two)
+static bool UTHValidateDetectAddress(DetectAddress *ad, const char *one, const char *two)
 {
     char str1[46] = "", str2[46] = "";
 
     if (ad == NULL)
-        return FALSE;
+        return false;
 
     switch(ad->ip.family) {
         case AF_INET:
@@ -1858,15 +1858,15 @@ static int UTHValidateDetectAddress(DetectAddress *ad, const char *one, const ch
 
             if (strcmp(str1, one) != 0) {
                 SCLogInfo("%s != %s", str1, one);
-                return FALSE;
+                return false;
             }
 
             if (strcmp(str2, two) != 0) {
                 SCLogInfo("%s != %s", str2, two);
-                return FALSE;
+                return false;
             }
 
-            return TRUE;
+            return true;
             break;
 
         case AF_INET6:
@@ -1877,19 +1877,19 @@ static int UTHValidateDetectAddress(DetectAddress *ad, const char *one, const ch
 
             if (strcmp(str1, one) != 0) {
                 SCLogInfo("%s != %s", str1, one);
-                return FALSE;
+                return false;
             }
 
             if (strcmp(str2, two) != 0) {
                 SCLogInfo("%s != %s", str2, two);
-                return FALSE;
+                return false;
             }
 
-            return TRUE;
+            return true;
             break;
     }
 
-    return FALSE;
+    return false;
 }
 
 typedef struct UTHValidateDetectAddressHeadRange_ {
@@ -1903,7 +1903,7 @@ static int UTHValidateDetectAddressHead(DetectAddressHead *gh, int nranges, UTHV
     int have = 0;
 
     if (gh == NULL)
-        return FALSE;
+        return false;
 
     DetectAddress *ad = NULL;
     ad = gh->ipv4_head;
@@ -1912,17 +1912,17 @@ static int UTHValidateDetectAddressHead(DetectAddressHead *gh, int nranges, UTHV
     while (have < expect) {
         if (ad == NULL) {
             printf("bad head: have %d ranges, expected %d: ", have, expect);
-            return FALSE;
+            return false;
         }
 
-        if (UTHValidateDetectAddress(ad, expectations[have].one, expectations[have].two) == FALSE)
-            return FALSE;
+        if (!UTHValidateDetectAddress(ad, expectations[have].one, expectations[have].two))
+            return false;
 
         ad = ad->next;
         have++;
     }
 
-    return TRUE;
+    return true;
 }
 
 static int AddressTestParse01(void)
@@ -4130,7 +4130,7 @@ static int AddressTestAddressGroupSetup38(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "![192.168.0.0/16,!192.168.14.0/24]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 3, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 3, expectations))
                 result = 1;
         }
 
@@ -4151,7 +4151,7 @@ static int AddressTestAddressGroupSetup39(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[![192.168.0.0/16,!192.168.14.0/24]]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 3, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 3, expectations))
                 result = 1;
         }
 
@@ -4171,7 +4171,7 @@ static int AddressTestAddressGroupSetup40(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[![192.168.0.0/16,[!192.168.14.0/24]]]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 3, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 3, expectations))
                 result = 1;
         }
 
@@ -4191,7 +4191,7 @@ static int AddressTestAddressGroupSetup41(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[![192.168.0.0/16,![192.168.14.0/24]]]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 3, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 3, expectations))
                 result = 1;
         }
 
@@ -4209,7 +4209,7 @@ static int AddressTestAddressGroupSetup42(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[2001::/3]");
         if (r == 0) {
-            if (UTHValidateDetectAddressHead(gh, 1, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 1, expectations))
                 result = 1;
         }
 
@@ -4228,7 +4228,7 @@ static int AddressTestAddressGroupSetup43(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[2001::/3,!3000::/5]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 2, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 2, expectations))
                 result = 1;
         }
 
@@ -4246,7 +4246,7 @@ static int AddressTestAddressGroupSetup44(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "3ffe:ffff:7654:feda:1245:ba98:3210:4562/96");
         if (r == 0) {
-            if (UTHValidateDetectAddressHead(gh, 1, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 1, expectations))
                 result = 1;
         }
 
@@ -4282,7 +4282,7 @@ static int AddressTestAddressGroupSetup46(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[![192.168.0.0/16,![192.168.1.0/24,192.168.3.0/24]]]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 4, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 4, expectations))
                 result = 1;
         }
 
@@ -4305,7 +4305,7 @@ static int AddressTestAddressGroupSetup47(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[![192.168.0.0/16,![192.168.1.0/24,192.168.3.0/24],!192.168.5.0/24]]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 5, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 5, expectations))
                 result = 1;
         }
 
@@ -4327,7 +4327,7 @@ static int AddressTestAddressGroupSetup48(void)
     if (gh != NULL) {
         int r = DetectAddressParse(NULL, gh, "[192.168.0.0/16,![192.168.1.0/24,192.168.3.0/24],!192.168.5.0/24]");
         if (r == 1) {
-            if (UTHValidateDetectAddressHead(gh, 4, expectations) == TRUE)
+            if (UTHValidateDetectAddressHead(gh, 4, expectations))
                 result = 1;
         }
 

From 28c950cef5981f2d6aab492bb241e2ada582432d Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 10:50:40 -0400
Subject: [PATCH 138/462] htp/bool: Use bool instead of int

---
 src/app-layer-htp-libhtp.c | 2 +-
 src/app-layer-htp-libhtp.h | 2 +-
 src/app-layer-htp.c        | 4 ++--
 src/app-layer-htp.h        | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/app-layer-htp-libhtp.c b/src/app-layer-htp-libhtp.c
index 2fbd5eae3390..dcc4a92b8bb0 100644
--- a/src/app-layer-htp-libhtp.c
+++ b/src/app-layer-htp-libhtp.c
@@ -61,7 +61,7 @@
  * \param uri_include_all boolean to indicate if scheme, username/password,
                           hostname and port should be part of the buffer
  */
-bstr *SCHTPGenerateNormalizedUri(htp_tx_t *tx, htp_uri_t *uri, int uri_include_all)
+bstr *SCHTPGenerateNormalizedUri(htp_tx_t *tx, htp_uri_t *uri, bool uri_include_all)
 {
     if (uri == NULL)
         return NULL;
diff --git a/src/app-layer-htp-libhtp.h b/src/app-layer-htp-libhtp.h
index 574dda4134dc..b08cda508ece 100644
--- a/src/app-layer-htp-libhtp.h
+++ b/src/app-layer-htp-libhtp.h
@@ -48,6 +48,6 @@
 
 #include "suricata-common.h"
 
-bstr *SCHTPGenerateNormalizedUri(htp_tx_t *tx, htp_uri_t *uri, int uri_include_all);
+bstr *SCHTPGenerateNormalizedUri(htp_tx_t *tx, htp_uri_t *uri, bool uri_include_all);
 
 #endif /* __APP_LAYER_HTP_LIBHTP__H__ */
diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index b576ba3b7b97..000fc88bbd0c 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -2463,7 +2463,7 @@ static int HTPCallbackResponseHeaderData(htp_tx_data_t *tx_data)
  */
 static void HTPConfigSetDefaultsPhase1(HTPCfgRec *cfg_prec)
 {
-    cfg_prec->uri_include_all = FALSE;
+    cfg_prec->uri_include_all = false;
     cfg_prec->request.body_limit = HTP_CONFIG_DEFAULT_REQUEST_BODY_LIMIT;
     cfg_prec->response.body_limit = HTP_CONFIG_DEFAULT_RESPONSE_BODY_LIMIT;
     cfg_prec->request.inspect_min_size = HTP_CONFIG_DEFAULT_REQUEST_INSPECT_MIN_SIZE;
@@ -2785,7 +2785,7 @@ static void HTPConfigParseParameters(HTPCfgRec *cfg_prec, ConfNode *s,
                                                 HTP_DECODER_URL_PATH,
                                                 ConfValIsTrue(p->val));
         } else if (strcasecmp("uri-include-all", p->name) == 0) {
-            cfg_prec->uri_include_all = ConfValIsTrue(p->val);
+            cfg_prec->uri_include_all = (1 == ConfValIsTrue(p->val));
             SCLogDebug("uri-include-all %s",
                     cfg_prec->uri_include_all ? "enabled" : "disabled");
         } else if (strcasecmp("query-plusspace-decode", p->name) == 0) {
diff --git a/src/app-layer-htp.h b/src/app-layer-htp.h
index c8c3a7f7b987..dee5c17e833e 100644
--- a/src/app-layer-htp.h
+++ b/src/app-layer-htp.h
@@ -157,8 +157,6 @@ typedef struct HTPCfgRec_ {
     htp_cfg_t           *cfg;
     struct HTPCfgRec_   *next;
 
-    int                 uri_include_all; /**< use all info in uri (bool) */
-
     /** max size of the client body we inspect */
     int                 randomize;
     int                 randomize_range;
@@ -171,6 +169,8 @@ typedef struct HTPCfgRec_ {
 
     HTPCfgDir request;
     HTPCfgDir response;
+
+    bool uri_include_all; /**< use all info in uri (bool) */
 } HTPCfgRec;
 
 /** Struct used to hold chunks of a body on a request */

From 0a716afadce2331ad351d9bec9c3ca195059ca71 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 12:52:21 -0400
Subject: [PATCH 139/462] run/bool: Use bool for threading value

---
 src/runmodes.c | 13 ++++++++-----
 src/runmodes.h |  2 +-
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/src/runmodes.c b/src/runmodes.c
index 348adfa25d43..852155332d75 100644
--- a/src/runmodes.c
+++ b/src/runmodes.c
@@ -72,7 +72,7 @@
 #include "suricata-plugin.h"
 
 int debuglog_enabled = 0;
-int threading_set_cpu_affinity = FALSE;
+bool threading_set_cpu_affinity = false;
 uint64_t threading_set_stack_size = 0;
 
 /* Runmode Global Thread Names */
@@ -982,12 +982,15 @@ float threading_detect_ratio = 1;
  */
 void RunModeInitializeThreadSettings(void)
 {
-    threading_set_cpu_affinity = FALSE;
-    if ((ConfGetBool("threading.set-cpu-affinity", &threading_set_cpu_affinity)) == 0) {
-        threading_set_cpu_affinity = FALSE;
+    int affinity = 0;
+    if ((ConfGetBool("threading.set-cpu-affinity", &affinity)) == 0) {
+        threading_set_cpu_affinity = false;
+    } else {
+        threading_set_cpu_affinity = affinity == 1;
     }
+
     /* try to get custom cpu mask value if needed */
-    if (threading_set_cpu_affinity == TRUE) {
+    if (threading_set_cpu_affinity) {
         AffinitySetupLoadFromConfig();
     }
     if ((ConfGetFloat("threading.detect-thread-ratio", &threading_detect_ratio)) != 1) {
diff --git a/src/runmodes.h b/src/runmodes.h
index 57b31b6d4054..668896dc17c5 100644
--- a/src/runmodes.h
+++ b/src/runmodes.h
@@ -100,7 +100,7 @@ bool IsRunModeSystem(enum RunModes run_mode_to_check);
 void RunModeEnablesBypassManager(void);
 int RunModeNeedsBypassManager(void);
 
-extern int threading_set_cpu_affinity;
+extern bool threading_set_cpu_affinity;
 extern float threading_detect_ratio;
 extern uint64_t threading_set_stack_size;
 

From 36e5792501fb34c1ec0067af7738ea0359116d09 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 12:54:27 -0400
Subject: [PATCH 140/462] debug/bool: Switch use_color to a bool

---
 src/util-debug.c | 4 ++--
 src/util-debug.h | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/util-debug.c b/src/util-debug.c
index 80509ae7ca0b..c62a0104dda3 100644
--- a/src/util-debug.c
+++ b/src/util-debug.c
@@ -341,7 +341,7 @@ static const char *SCTransformModule(const char *module_name, int *dn_len)
  *
  * \retval 0 on success; else a negative value on error
  */
-static SCError SCLogMessageGetBuffer(SCTime_t tval, int color, SCLogOPType type, char *buffer,
+static SCError SCLogMessageGetBuffer(SCTime_t tval, bool color, SCLogOPType type, char *buffer,
         size_t buffer_size, const char *log_format, const SCLogLevel log_level, const char *file,
         const unsigned int line, const char *function, const char *module, const char *message)
 {
@@ -952,7 +952,7 @@ static inline SCLogOPIfaceCtx *SCLogInitConsoleOPIface(const char *log_format,
 
 #ifndef OS_WIN32
     if (isatty(fileno(stdout)) && isatty(fileno(stderr))) {
-        iface_ctx->use_color = TRUE;
+        iface_ctx->use_color = true;
     }
 #endif
 
diff --git a/src/util-debug.h b/src/util-debug.h
index 2a0864cdd83a..296cf896519c 100644
--- a/src/util-debug.h
+++ b/src/util-debug.h
@@ -115,7 +115,7 @@ typedef struct SCLogOPBuffer_ {
 typedef struct SCLogOPIfaceCtx_ {
     SCLogOPIface iface;
 
-    int16_t use_color;
+    bool use_color;
     SCLogOPType type;
 
     /* the output file to be used if the interface is SC_LOG_IFACE_FILE */

From 924c59448f6d9b6243fee3e8a2c177889d11e3ba Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 27 Jul 2023 13:00:31 -0400
Subject: [PATCH 141/462] bool: Remove TRUE/FALSE

---
 src/suricata-common.h | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/src/suricata-common.h b/src/suricata-common.h
index fe8ec179dff3..4aad25a252f4 100644
--- a/src/suricata-common.h
+++ b/src/suricata-common.h
@@ -30,9 +30,6 @@
 #define DBG_PERF
 #endif
 
-#define TRUE   1
-#define FALSE  0
-
 #define _GNU_SOURCE
 #define __USE_GNU
 

From 2b9603d94ddb0ff1c103ce964f22d8bc63f4392c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 15 Nov 2023 14:48:40 -0600
Subject: [PATCH 142/462] github-ci: cancel previous builds workflow for branch

On a push of the same branch, cancel the previous running builds.yml
workflow.
---
 .github/workflows/builds.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 62748b10e985..adb2a7f2330e 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -12,6 +12,10 @@ on:
       SV_REPO:
       SV_BRANCH:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 permissions: read-all
 
 env:

From bec1d8ca9f5574a90e570fa1ec1042f31842d914 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 16 Nov 2023 07:33:35 -0600
Subject: [PATCH 143/462] github-ci: don't add author names/emails to new
 author comment

The new author details will still be available in the artifact, we're
just not calling them out in a nighly visible pull request comment.
---
 .github/workflows/authors-done.yml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/.github/workflows/authors-done.yml b/.github/workflows/authors-done.yml
index b1700e1d030d..d0fdf4491c45 100644
--- a/.github/workflows/authors-done.yml
+++ b/.github/workflows/authors-done.yml
@@ -44,8 +44,7 @@ jobs:
           script: |
             let fs = require('fs');
             let issue_number = Number(fs.readFileSync('./pr-number.txt'));
-            let new_authors = String(fs.readFileSync('./new-authors.txt'));
-            let msg = 'NOTE: This PR may contain new authors:\n\n```\n' + new_authors + '```';
+            let msg = 'NOTE: This PR may contain new authors.';
             await github.rest.issues.createComment({
               owner: context.repo.owner,
               repo: context.repo.repo,

From 58fb559594f8e00e9c63dcb9fee167d694689fd4 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Tue, 14 Nov 2023 06:46:31 -0300
Subject: [PATCH 144/462] userguide: document flow_id, with examples

Flow_id explanation expanded from version shared by Peter Manev.

Task #6445
---
 doc/userguide/output/eve/eve-json-format.rst | 310 ++++++++++++++++++-
 doc/userguide/output/eve/eve-json-output.rst |   2 +
 2 files changed, 311 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/output/eve/eve-json-format.rst b/doc/userguide/output/eve/eve-json-format.rst
index a253e046cf7f..d3cc6eefbafa 100644
--- a/doc/userguide/output/eve/eve-json-format.rst
+++ b/doc/userguide/output/eve/eve-json-format.rst
@@ -43,7 +43,307 @@ All the JSON log types share a common structure:
 ::
 
 
-  {"timestamp":"2009-11-24T21:27:09.534255","event_type":"TYPE", ...tuple... ,"TYPE":{ ... type specific content ... }}
+  {"timestamp":"2009-11-24T21:27:09.534255","flow_id":ID_NUMBER, "event_type":"TYPE", ...tuple... ,"TYPE":{ ... type specific content ... }}
+
+Field: flow_id
+~~~~~~~~~~~~~~
+
+Correlates the network protocol,  flow logs EVE data and any evidence that
+Suricata has logged to an ``alert`` event and that alert's metadata, as well as
+to ``fileinfo``/file transaction and anomaly logs, if available. The same correlation
+and logs are produced regardless if there is an alert, for any session/flow.
+
+The ability to correlate EVE logs belonging to a specific session/flow was
+introduced in 2014 (see `commit f1185d051c21 <https://github.com/OISF/suricata/
+commit/f1185d051c210ca0daacdddbe865a51af24f4ea3>`_).
+
+Further below, you can see several examples of events logged by Suricata: an
+:ref:`alert<eve-format-alert>` for an ``HTTP`` rule, ``fileinfo``, :ref:`http<eve-format-http>`,
+:ref:`anomaly<eve-format-anomaly>`, and :ref:`flow<eve-format-flow>` events, all
+easily correlated using the ``flow_id`` EVE field::
+
+    $ jq 'select(.flow_id==1676750115612680)' eve.json
+
+Event type: ``alert``::
+
+    {
+      "timestamp": "2023-09-18T06:13:41.532140+0000",
+      "flow_id": 1676750115612680,
+      "pcap_cnt": 130,
+      "event_type": "alert",
+      "src_ip": "142.11.240.191",
+      "src_port": 35361,
+      "dest_ip": "192.168.100.237",
+      "dest_port": 49175,
+      "proto": "TCP",
+      "pkt_src": "wire/pcap",
+      "ether": {
+        "src_mac": "52:54:00:36:3e:ff",
+        "dest_mac": "12:a9:86:6c:77:de"
+      },
+      "tx_id": 1,
+      "alert": {
+        "action": "allowed",
+        "gid": 1,
+        "signature_id": 2045001,
+        "rev": 1,
+        "signature": "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound",
+        "category": "A Network Trojan was detected",
+        "severity": 1,
+        "metadata": {
+          "affected_product": [
+            "Windows_XP_Vista_7_8_10_Server_32_64_Bit"
+          ],
+          "attack_target": [
+            "Client_Endpoint"
+          ],
+          "created_at": [
+            "2023_04_17"
+          ],
+          "deployment": [
+            "Perimeter"
+          ],
+          "former_category": [
+            "ATTACK_RESPONSE"
+          ],
+          "signature_severity": [
+            "Major"
+          ],
+          "updated_at": [
+            "2023_04_18"
+          ]
+        }
+      },
+      "http": {
+        "hostname": "142.11.240.191",
+        "http_port": 35361,
+        "url": "/",
+        "http_content_type": "text/xml",
+        "http_method": "POST",
+        "protocol": "HTTP/1.1",
+        "status": 200,
+        "length": 5362
+      },
+      "files": [
+        {
+          "filename": "/",
+          "gaps": false,
+          "state": "CLOSED",
+          "stored": false,
+          "size": 5362,
+          "tx_id": 1
+        }
+      ],
+      "app_proto": "http",
+      "direction": "to_client",
+      "flow": {
+        "pkts_toserver": 13,
+        "pkts_toclient": 12,
+        "bytes_toserver": 1616,
+        "bytes_toclient": 8044,
+        "start": "2023-09-18T06:13:33.324862+0000",
+        "src_ip": "192.168.100.237",
+        "dest_ip": "142.11.240.191",
+        "src_port": 49175,
+        "dest_port": 35361
+      }
+    }
+
+Event type: ``fileinfo``::
+
+    {
+      "timestamp": "2023-09-18T06:13:33.903924+0000",
+      "flow_id": 1676750115612680,
+      "pcap_cnt": 70,
+      "event_type": "fileinfo",
+      "src_ip": "192.168.100.237",
+      "src_port": 49175,
+      "dest_ip": "142.11.240.191",
+      "dest_port": 35361,
+      "proto": "TCP",
+      "pkt_src": "wire/pcap",
+      "ether": {
+        "src_mac": "12:a9:86:6c:77:de",
+        "dest_mac": "52:54:00:36:3e:ff"
+      },
+      "http": {
+        "hostname": "142.11.240.191",
+        "http_port": 35361,
+        "url": "/",
+        "http_content_type": "text/xml",
+        "http_method": "POST",
+        "protocol": "HTTP/1.1",
+        "status": 200,
+        "length": 212
+      },
+      "app_proto": "http",
+      "fileinfo": {
+        "filename": "/",
+        "gaps": false,
+        "state": "CLOSED",
+        "stored": false,
+        "size": 137,
+        "tx_id": 0
+      }
+    }
+
+Event type: ``HTTP``::
+
+    {
+      "timestamp": "2023-09-18T06:13:33.903924+0000",
+      "flow_id": 1676750115612680,
+      "pcap_cnt": 70,
+      "event_type": "http",
+      "src_ip": "192.168.100.237",
+      "src_port": 49175,
+      "dest_ip": "142.11.240.191",
+      "dest_port": 35361,
+      "proto": "TCP",
+      "pkt_src": "wire/pcap",
+      "ether": {
+        "src_mac": "12:a9:86:6c:77:de",
+        "dest_mac": "52:54:00:36:3e:ff"
+      },
+      "tx_id": 0,
+      "http": {
+        "hostname": "142.11.240.191",
+        "http_port": 35361,
+        "url": "/",
+        "http_content_type": "text/xml",
+        "http_method": "POST",
+        "protocol": "HTTP/1.1",
+        "status": 200,
+        "length": 212,
+        "request_headers": [
+          {
+            "name": "Content-Type",
+            "value": "text/xml; charset=utf-8"
+          },
+          {
+            "name": "SOAPAction",
+            "value": "\"http://tempuri.org/Endpoint/CheckConnect\""
+          },
+          {
+            "name": "Host",
+            "value": "142.11.240.191:35361"
+          },
+          {
+            "name": "Content-Length",
+            "value": "137"
+          },
+          {
+            "name": "Expect",
+            "value": "100-continue"
+          },
+          {
+            "name": "Accept-Encoding",
+            "value": "gzip, deflate"
+          },
+          {
+            "name": "Connection",
+            "value": "Keep-Alive"
+          }
+        ],
+        "response_headers": [
+          {
+            "name": "Content-Length",
+            "value": "212"
+          },
+          {
+            "name": "Content-Type",
+            "value": "text/xml; charset=utf-8"
+          },
+          {
+            "name": "Server",
+            "value": "Microsoft-HTTPAPI/2.0"
+          },
+          {
+            "name": "Date",
+            "value": "Mon, 18 Sep 2023 06:13:33 GMT"
+          }
+        ]
+      }
+    }
+
+Event type: ``anomaly``::
+
+    {
+      "timestamp": "2023-09-18T06:13:58.882971+0000",
+      "flow_id": 1676750115612680,
+      "pcap_cnt": 2878,
+      "event_type": "anomaly",
+      "src_ip": "192.168.100.237",
+      "src_port": 49175,
+      "dest_ip": "142.11.240.191",
+      "dest_port": 35361,
+      "proto": "TCP",
+      "pkt_src": "wire/pcap",
+      "ether": {
+        "src_mac": "12:a9:86:6c:77:de",
+        "dest_mac": "52:54:00:36:3e:ff"
+      },
+      "tx_id": 3,
+      "anomaly": {
+        "app_proto": "http",
+        "type": "applayer",
+        "event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
+        "layer": "proto_parser"
+      }
+    }
+
+
+Event type: ``flow``::
+
+    {
+      "timestamp": "2023-09-18T06:13:21.216460+0000",
+      "flow_id": 1676750115612680,
+      "event_type": "flow",
+      "src_ip": "192.168.100.237",
+      "src_port": 49175,
+      "dest_ip": "142.11.240.191",
+      "dest_port": 35361,
+      "proto": "TCP",
+      "app_proto": "http",
+      "flow": {
+        "pkts_toserver": 3869,
+        "pkts_toclient": 1523,
+        "bytes_toserver": 3536402,
+        "bytes_toclient": 94102,
+        "start": "2023-09-18T06:13:33.324862+0000",
+        "end": "2023-09-18T06:14:13.752399+0000",
+        "age": 40,
+        "state": "closed",
+        "reason": "shutdown",
+        "alerted": true
+      },
+      "ether": {
+        "dest_macs": [
+          "52:54:00:36:3e:ff"
+        ],
+        "src_macs": [
+          "12:a9:86:6c:77:de"
+        ]
+      },
+      "tcp": {
+        "tcp_flags": "1e",
+        "tcp_flags_ts": "1e",
+        "tcp_flags_tc": "1a",
+        "syn": true,
+        "rst": true,
+        "psh": true,
+        "ack": true,
+        "state": "closed",
+        "ts_max_regions": 1,
+        "tc_max_regions": 1
+      }
+    }
+
+.. note::
+   It is possible to have even more detailed alert records, by enabling for
+   instance logging http-body, or alert metadata (:ref:`alert output<eve-output-alert>`).
+
+Examples come from pcap found at https://app.any.run/tasks/ce7ca983-9e4b-4251-a7c3-fefa3da02ebe/.
+
 
 Event types
 ~~~~~~~~~~~
@@ -86,6 +386,8 @@ generated the event.
           omitted from internal "pseudo" packets such as flow timeout
           packets.
 
+.. _eve-format-alert:
+
 Event type: Alert
 -----------------
 
@@ -191,6 +493,8 @@ If pcap log capture is active in `multi` mode, a `capture_file` key will be adde
 with value being the full path of the pcap file where the corresponding packets
 have been extracted.
 
+.. _eve-format-anomaly:
+
 Event type: Anomaly
 -------------------
 
@@ -304,6 +608,8 @@ Examples
       }
     }
 
+.. _eve-format-http:
+
 Event type: HTTP
 ----------------
 
@@ -1345,6 +1651,8 @@ Example of SSH logging:
      }
   }
 
+.. _eve-format-flow:
+
 Event type: Flow
 ----------------
 
diff --git a/doc/userguide/output/eve/eve-json-output.rst b/doc/userguide/output/eve/eve-json-output.rst
index 512672f87d94..2730f543bbf5 100644
--- a/doc/userguide/output/eve/eve-json-output.rst
+++ b/doc/userguide/output/eve/eve-json-output.rst
@@ -53,6 +53,8 @@ Output types::
       #    enabled: yes ## set enable to yes to enable query pipelining
       #    batch-size: 10 ## number of entry to keep in buffer
 
+.. _eve-output-alert:
+
 Alerts
 ~~~~~~
 

From a649a92afd4ed71b264ee1df408875322d9dfaf9 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 16 Nov 2023 14:37:32 -0300
Subject: [PATCH 145/462] userguide: update tls not_after/not_before mentions

Our tls fields not_after and not_before are actually logged as
`notafter` and `notbefore`, but were documented with the underscore.

Update the documentation, since updating the log format itself would be
a breaking change.

Task #5494
---
 doc/userguide/output/eve/eve-json-format.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/userguide/output/eve/eve-json-format.rst b/doc/userguide/output/eve/eve-json-format.rst
index d3cc6eefbafa..0c7bc867aeb6 100644
--- a/doc/userguide/output/eve/eve-json-format.rst
+++ b/doc/userguide/output/eve/eve-json-format.rst
@@ -1041,8 +1041,8 @@ If extended logging is enabled the following fields are also included:
 * "fingerprint": The (SHA1) fingerprint of the TLS certificate
 * "sni": The Server Name Indication (SNI) extension sent by the client
 * "version": The SSL/TLS version used
-* "not_before": The NotBefore field from the TLS certificate
-* "not_after": The NotAfter field from the TLS certificate
+* "notbefore": The NotBefore field from the TLS certificate
+* "notafter": The NotAfter field from the TLS certificate
 * "ja3": The JA3 fingerprint consisting of both a JA3 hash and a JA3 string
 * "ja3s": The JA3S fingerprint consisting of both a JA3 hash and a JA3 string
 

From d2b25af3f4a21806efc3e38ebbe6e66b7dc5c0d4 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 30 Oct 2023 17:25:12 -0600
Subject: [PATCH 146/462] examples: add an example plugin of an eve filetype

This is an example of what adding plugin examples to the Suricata repo
could look like.

This plugin is an example plugin for an EVE filetype. It could be
extended to support outputs like Redis, syslog, etc.

There is one issue with adding plugins like this to an autotools
project, the project can't be built with --disable-shared, which is
more of an autotools limitation, and not really a Suricata issue.
Suricata built with --disable-shared will load plugins just fine.

Note that the examples directory was added as DIST_SUBDIRS as we don't
want normal builds to recurse into it and attempt to build the plugin,
its just an example, but we still need to keep distcheck happy.
---
 .github/workflows/builds.yml                  |  12 +
 Makefile.am                                   |   1 +
 configure.ac                                  |   2 +
 examples/plugins/README.md                    |   6 +
 examples/plugins/c-json-filetype/.gitignore   |   2 +
 examples/plugins/c-json-filetype/Makefile.am  |  17 ++
 .../plugins/c-json-filetype/Makefile.example  |  18 ++
 examples/plugins/c-json-filetype/README.md    | 123 +++++++++
 examples/plugins/c-json-filetype/filetype.c   | 243 ++++++++++++++++++
 9 files changed, 424 insertions(+)
 create mode 100644 examples/plugins/README.md
 create mode 100644 examples/plugins/c-json-filetype/.gitignore
 create mode 100644 examples/plugins/c-json-filetype/Makefile.am
 create mode 100644 examples/plugins/c-json-filetype/Makefile.example
 create mode 100644 examples/plugins/c-json-filetype/README.md
 create mode 100644 examples/plugins/c-json-filetype/filetype.c

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index adb2a7f2330e..93708415294a 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -288,6 +288,18 @@ jobs:
           test -e /usr/local/lib/suricata/python/suricata/update/configs/modify.conf
           test -e /usr/local/lib/suricata/python/suricata/update/configs/threshold.in
           test -e /usr/local/lib/suricata/python/suricata/update/configs/update.yaml
+      - name: Build C json filetype plugin
+        working-directory: examples/plugins/c-json-filetype
+        run: make
+      - name: Check C json filetype plugin
+        run: test -e examples/plugins/c-json-filetype/.libs/json-filetype.so.0.0.0
+      - name: Installing headers and library
+        run: |
+          make install-headers
+          make install-library
+      - name: Test plugin build with Makefile.example
+        working-directory: examples/plugins/c-json-filetype
+        run: PATH=/usr/local/bin:$PATH make -f Makefile.example
 
   almalinux-9-templates:
     name: AlmaLinux 9 Test Templates
diff --git a/Makefile.am b/Makefile.am
index 67963ed32fcf..b7a221f49299 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -10,6 +10,7 @@ EXTRA_DIST = ChangeLog COPYING LICENSE suricata.yaml.in \
 	     scripts/generate-images.sh
 SUBDIRS = $(HTP_DIR) rust src qa rules doc contrib etc python ebpf \
           $(SURICATA_UPDATE_DIR)
+DIST_SUBDIRS = examples/plugins/c-json-filetype $(SUBDIRS)
 
 CLEANFILES = stamp-h[0-9]*
 
diff --git a/configure.ac b/configure.ac
index 1908c76ab76a..1eb054d98c1e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2613,6 +2613,8 @@ AC_CONFIG_FILES(suricata.yaml etc/Makefile etc/suricata.logrotate etc/suricata.s
 AC_CONFIG_FILES(python/Makefile python/suricata/config/defaults.py)
 AC_CONFIG_FILES(ebpf/Makefile)
 AC_CONFIG_FILES(libsuricata-config)
+AC_CONFIG_FILES(examples/plugins/c-json-filetype/Makefile)
+
 AC_OUTPUT
 
 SURICATA_BUILD_CONF="Suricata Configuration:
diff --git a/examples/plugins/README.md b/examples/plugins/README.md
new file mode 100644
index 000000000000..8e47b6e7ffc2
--- /dev/null
+++ b/examples/plugins/README.md
@@ -0,0 +1,6 @@
+# Example Plugins
+
+## c-json-filetype
+
+An example plugin of an EVE/JSON filetype plugin. This type of plugin
+is useful if you want to send EVE output to custom destinations.
diff --git a/examples/plugins/c-json-filetype/.gitignore b/examples/plugins/c-json-filetype/.gitignore
new file mode 100644
index 000000000000..f5bb3af73f70
--- /dev/null
+++ b/examples/plugins/c-json-filetype/.gitignore
@@ -0,0 +1,2 @@
+*.so
+*.la
diff --git a/examples/plugins/c-json-filetype/Makefile.am b/examples/plugins/c-json-filetype/Makefile.am
new file mode 100644
index 000000000000..d5e912b8a469
--- /dev/null
+++ b/examples/plugins/c-json-filetype/Makefile.am
@@ -0,0 +1,17 @@
+plugindir = ${libdir}/suricata/plugins
+
+if BUILD_SHARED_LIBRARY
+plugin_LTLIBRARIES = json-filetype.la
+json_filetype_la_LDFLAGS = -module -shared
+json_filetype_la_SOURCES = filetype.c
+
+json_filetype_la_CPPFLAGS = -I$(abs_top_srcdir)/rust/gen -I$(abs_top_srcdir)/rust/dist
+
+else
+
+all-local:
+	@echo
+	@echo "Shared library support must be enabled to build plugins."
+	@echo
+
+endif
diff --git a/examples/plugins/c-json-filetype/Makefile.example b/examples/plugins/c-json-filetype/Makefile.example
new file mode 100644
index 000000000000..6d514aab1445
--- /dev/null
+++ b/examples/plugins/c-json-filetype/Makefile.example
@@ -0,0 +1,18 @@
+SRCS :=		filetype.c
+
+LIBSURICATA_CONFIG ?= libsuricata-config
+
+CPPFLAGS +=	`$(LIBSURICATA_CONFIG) --cflags`
+CPPFLAGS +=	-DSURICATA_PLUGIN -I.
+CPPFLAGS +=	"-D__SCFILENAME__=\"$(*F)\""
+
+OBJS :=		$(SRCS:.c=.o)
+
+filetype.so: $(OBJS)
+	$(CC) -fPIC -shared -o $@ $(OBJS)
+
+%.o: %.c
+	$(CC) -fPIC $(CPPFLAGS) -c -o $@ $<
+
+clean:
+	rm -f *.o *.so *~
diff --git a/examples/plugins/c-json-filetype/README.md b/examples/plugins/c-json-filetype/README.md
new file mode 100644
index 000000000000..2f7978977dbd
--- /dev/null
+++ b/examples/plugins/c-json-filetype/README.md
@@ -0,0 +1,123 @@
+# Example EVE Filetype Plugin
+
+## Building
+
+If in the Suricata source directory, this plugin can be built by
+running `make` and installed with `make install`.
+
+Note that Suricata must have been built without `--disable-shared`.
+
+## Building Standalone
+
+The file `Makefile.example` is an example of how you might build a
+plugin that is distributed separately from the Suricata source code.
+
+It has the following dependencies:
+
+- Suricata is installed
+- The Suricata library is installed: `make install-library`
+- The Suricata development headers are installed: `make install-headers`
+- The program `libsuricata-config` is in your path (installed with
+  `make install-library`)
+
+The run: `make -f Makefile.example`
+
+Before building this plugin you will need to build and install Suricata from the
+git master branch and install the development tools and headers:
+
+- `make install-library`
+- `make install-headers`
+
+then make sure the newly installed tool `libsuricata-config` can be
+found in your path, for example:
+```
+libsuricata-config --cflags
+```
+
+Then a simple `make` should build this plugin.
+
+Or if the Suricata installation is not in the path, a command like the following
+can be used:
+
+```
+PATH=/opt/suricata/bin:$PATH make
+```
+
+## Usage
+
+To run the plugin, first add the path to the plugin you just compiled to
+your `suricata.yaml`, for example:
+```
+plugins:
+  - /usr/lib/suricata/plugins/json-filetype.so
+```
+
+Then add an output for the plugin:
+```
+outputs:
+  - eve-log:
+      enabled: yes
+      filetype: json-filetype-plugin
+      threaded: true
+      types:
+        - dns
+        - tls
+        - http
+```
+
+In the example above we use the name specified in the plugin as the `filetype`
+and specify that all `dns`, `tls` and `http` log entries should be sent to the
+plugin.
+
+## Details
+
+This plugin demonstrates a Suricata JSON/EVE output plugin
+(file-type). The idea of a Suricata EVE output plugin is to provide a
+file like interface for the handling of rendered JSON logs. This is
+useful for custom destinations not builtin to Suricata or if the
+formatted JSON requires some post-processing.
+
+Note: EVE output plugins are not that useful just for reformatting the
+JSON output as the plugin does need to handle writing to a file once
+the file type has been delegated to the plugin.
+
+### Registering a Plugin
+
+All Suricata plugins make themselves known to Suricata by using a
+function named `SCPluginRegister` which is called after Suricata loads
+the plugin shared object file. This function must return a `SCPlugin`
+struct which contains basic information about the plugin.  For
+example:
+
+```c
+const SCPlugin PluginRegistration = {
+    .name = "eve-filetype",
+    .author = "Jason Ish",
+    .license = "GPLv2",
+    .Init = TemplateInit,
+};
+
+const SCPlugin *SCPluginRegister() {
+    return &PluginRegistration;
+}
+```
+
+### Initializing a Plugin
+
+After the plugin has been registered, the `Init` callback will be called. This
+is where the plugin will set itself up as a specific type of plugin such as an
+EVE output, or a capture method.
+
+This plugins registers itself as an EVE file type using the
+`SCRegisterEveFileType` struct. To register as an EVE file type the
+following must be provided:
+
+* name: This is the name of the output which will be used in the eve filetype
+  field in `suricata.yaml` to enable this output.
+* Init: The callback called when the output is "opened".
+* Deinit: The callback called the output is "closed".
+* ThreadInit: Callback called to initialize per thread data (if threaded).
+* ThreadDeinit: Callback called to deinitialize per thread data (if threaded).
+* Write: The callback called when an EVE record is to be "written".
+
+Please see the code in `filetype.c` for more details about this functions.
diff --git a/examples/plugins/c-json-filetype/filetype.c b/examples/plugins/c-json-filetype/filetype.c
new file mode 100644
index 000000000000..9c81d7f03267
--- /dev/null
+++ b/examples/plugins/c-json-filetype/filetype.c
@@ -0,0 +1,243 @@
+/* Copyright (C) 2020-2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#include "suricata-common.h"
+#include "suricata-plugin.h"
+#include "util-mem.h"
+#include "util-debug.h"
+
+#define FILETYPE_NAME "json-filetype-plugin"
+
+static int FiletypeThreadInit(void *ctx, int thread_id, void **thread_data);
+static int FiletypeThreadDeinit(void *ctx, void *thread_data);
+
+/**
+ * Per thread context data for each logging thread.
+ */
+typedef struct ThreadData_ {
+    /** The thread ID, for demonstration purposes only. */
+    int thread_id;
+
+    /** The number of records logged on this thread. */
+    uint64_t count;
+} ThreadData;
+
+/**
+ * A context object for each eve logger using this output.
+ */
+typedef struct Context_ {
+    /** Verbose, or print to stdout. */
+    int verbose;
+
+    /** A thread context to use when not running in threaded mode. */
+    ThreadData *thread;
+} Context;
+
+/**
+ * This function is called to initialize the output, it can be somewhat thought
+ * of like opening a file.
+ *
+ * \param conf The EVE configuration node using this output.
+ *
+ * \param threaded If true the EVE subsystem is running in threaded mode.
+ *
+ * \param data A pointer where context data can be stored relevant to this
+ *      output.
+ *
+ * Eve output plugins need to be thread aware as the threading happens at lower
+ * level than the EVE output, so a flag is provided here to notify the plugin if
+ * threading is enabled or not.
+ *
+ * If the plugin does not work with threads disabled, or enabled, this function
+ * should return -1.
+ *
+ * Note for upgrading a plugin from 6.0 to 7.0: The ConfNode in 7.0 is the
+ * configuration for the eve instance, not just a node named after the plugin.
+ * This allows the plugin to get more context about what it is logging.
+ */
+static int FiletypeInit(ConfNode *conf, bool threaded, void **data)
+{
+    SCLogNotice("Initializing template eve output plugin: threaded=%d", threaded);
+    Context *context = SCCalloc(1, sizeof(Context));
+    if (context == NULL) {
+        return -1;
+    }
+
+    /* Verbose by default. */
+    int verbose = 1;
+
+    /* An example of how you can access configuration data from a
+     * plugin. */
+    if (conf && (conf = ConfNodeLookupChild(conf, "eve-template")) != NULL) {
+        if (!ConfGetChildValueBool(conf, "verbose", &verbose)) {
+            verbose = 1;
+        } else {
+            SCLogNotice("Read verbose configuration value of %d", verbose);
+        }
+    }
+    context->verbose = verbose;
+
+    if (!threaded) {
+        /* We're not running in threaded mode so allocate a thread context here
+         * to avoid duplication of context data such as file pointers, database
+         * connections, etc. */
+        if (FiletypeThreadInit(context, 0, (void **)&context->thread) != 0) {
+            SCFree(context);
+            return -1;
+        }
+    }
+    *data = context;
+    return 0;
+}
+
+/**
+ * This function is called when the output is closed.
+ *
+ * This will be called after ThreadDeinit is called for each thread.
+ *
+ * \param data The data allocated in FiletypeInit. It should be cleaned up and
+ *      deallocated here.
+ */
+static void FiletypeDeinit(void *data)
+{
+    printf("TemplateClose\n");
+    Context *ctx = data;
+    if (ctx != NULL) {
+        if (ctx->thread) {
+            FiletypeThreadDeinit(ctx, (void *)ctx->thread);
+        }
+        SCFree(ctx);
+    }
+}
+
+/**
+ * Initialize per thread context.
+ *
+ * \param ctx The context created in TemplateInitOutput.
+ *
+ * \param thread_id An identifier for this thread.
+ *
+ * \param thread_data Pointer where thread specific context can be stored.
+ *
+ * When the EVE output is running in threaded mode this will be called once for
+ * each output thread with a unique thread_id. For regular file logging in
+ * threaded mode Suricata uses the thread_id to construct the files in the form
+ * of "eve.<thread_id>.json". This plugin may want to do similar, or open
+ * multiple connections to whatever the final logging location might be.
+ *
+ * In the case of non-threaded EVE logging this function is NOT called by
+ * Suricata, but instead this plugin chooses to use this method to create a
+ * default (single) thread context.
+ */
+static int FiletypeThreadInit(void *ctx, int thread_id, void **thread_data)
+{
+    ThreadData *tdata = SCCalloc(1, sizeof(ThreadData));
+    if (tdata == NULL) {
+        SCLogError("Failed to allocate thread data");
+        return -1;
+    }
+    tdata->thread_id = thread_id;
+    *thread_data = tdata;
+    SCLogNotice(
+            "Initialized thread %03d (pthread_id=%" PRIuMAX ")", tdata->thread_id, pthread_self());
+    return 0;
+}
+
+/**
+ * Deinitialize a thread.
+ *
+ * This is where any cleanup per thread should be done including free'ing of the
+ * thread_data if needed.
+ */
+static int FiletypeThreadDeinit(void *ctx, void *thread_data)
+{
+    if (thread_data == NULL) {
+        // Nothing to do.
+        return 0;
+    }
+
+    ThreadData *tdata = thread_data;
+    SCLogNotice(
+            "Deinitializing thread %d: records written: %" PRIu64, tdata->thread_id, tdata->count);
+    SCFree(tdata);
+    return 0;
+}
+
+/**
+ * This method is called with formatted Eve JSON data.
+ *
+ * \param buffer Formatted JSON buffer \param buffer_len Length of formatted
+ * JSON buffer \param data Data set in Init callback \param thread_data Data set
+ * in ThreadInit callbacl
+ *
+ * Do not block in this thread, it will cause packet loss. Instead of outputting
+ * to any resource that may block it might be best to enqueue the buffers for
+ * further processing which will require copying of the provided buffer.
+ */
+static int FiletypeWrite(const char *buffer, int buffer_len, void *data, void *thread_data)
+{
+    Context *ctx = data;
+    ThreadData *thread = thread_data;
+
+    /* The thread_data could be null which is valid, or it could be that we are
+     * in single threaded mode. */
+    if (thread == NULL) {
+        thread = ctx->thread;
+    }
+
+    thread->count++;
+
+    if (ctx->verbose) {
+        SCLogNotice("Received write with thread_data %p: %s", thread_data, buffer);
+    }
+    return 0;
+}
+
+/**
+ * Called by Suricata to initialize the module. This module registers
+ * new file type to the JSON logger.
+ */
+void PluginInit(void)
+{
+    SCEveFileType *my_output = SCCalloc(1, sizeof(SCEveFileType));
+    my_output->name = FILETYPE_NAME;
+    my_output->Init = FiletypeInit;
+    my_output->Deinit = FiletypeDeinit;
+    my_output->ThreadInit = FiletypeThreadInit;
+    my_output->ThreadDeinit = FiletypeThreadDeinit;
+    my_output->Write = FiletypeWrite;
+    if (!SCRegisterEveFileType(my_output)) {
+        FatalError("Failed to register filetype plugin: %s", FILETYPE_NAME);
+    }
+}
+
+const SCPlugin PluginRegistration = {
+    .name = FILETYPE_NAME,
+    .author = "FirstName LastName <name@example.org>",
+    .license = "GPL-2.0-only",
+    .Init = PluginInit,
+};
+
+/**
+ * The function called by Suricata after loading this plugin.
+ *
+ * A pointer to a populated SCPlugin struct must be returned.
+ */
+const SCPlugin *SCPluginRegister()
+{
+    return &PluginRegistration;
+}

From 7d40a9f178a3b8d686b1f33bfc1f6e21a662ccb3 Mon Sep 17 00:00:00 2001
From: Thomas Winter <Thomas.Winter@alliedtelesis.co.nz>
Date: Thu, 27 Apr 2023 16:08:46 +1200
Subject: [PATCH 147/462] rule-reload: Release excess memory freed during
 engine reload

The hot reload results in large chunks of memory being freed as the
as the old signature tables are discarded. Help the memory management
system along by telling to release as much memory as it can at this
point.

Bug: #6454.
---
 configure.ac          | 7 +++++++
 src/detect-engine.c   | 9 +++++++++
 src/suricata-common.h | 4 ++++
 3 files changed, 20 insertions(+)

diff --git a/configure.ac b/configure.ac
index 1eb054d98c1e..c2bed717616c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -220,6 +220,13 @@
             #include <sys/random.h> 
             ])
 
+    AC_CHECK_HEADERS([malloc.h])
+    AC_CHECK_DECL([malloc_trim],
+        AC_DEFINE([HAVE_MALLOC_TRIM], [1], [Use malloc_trim]),
+        [], [
+            #include <malloc.h> 
+            ])
+
     OCFLAGS=$CFLAGS
     CFLAGS=""
     AC_CHECK_FUNCS([strlcpy strlcat])
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 0fc2df6869b0..25e76445edd9 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -4797,6 +4797,15 @@ int DetectEngineReload(const SCInstance *suri)
     SCLogDebug("old_de_ctx should have been freed");
 
     SCLogNotice("rule reload complete");
+
+#ifdef HAVE_MALLOC_TRIM
+    /* The reload process potentially frees up large amounts of memory.
+     * Encourage the memory management system to reclaim as much as it
+     * can.
+     */
+    malloc_trim(0);
+#endif
+
     return 0;
 }
 
diff --git a/src/suricata-common.h b/src/suricata-common.h
index 4aad25a252f4..297a6fc4521e 100644
--- a/src/suricata-common.h
+++ b/src/suricata-common.h
@@ -212,6 +212,10 @@ typedef unsigned char u_char;
 #include <netdb.h>
 #endif
 
+#if HAVE_MALLOC_H
+#include <malloc.h>
+#endif
+
 #if __CYGWIN__
 #if !defined _X86_ && !defined __x86_64
 #define _X86_

From c50002978d7a4cc0d100095eaa082e4a8183a8a4 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Thu, 16 Nov 2023 13:49:50 +0000
Subject: [PATCH 148/462] doc: update file.data keyword documentation

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/file-keywords.rst | 30 +++++++++++++++++++++++++++
 doc/userguide/rules/http-keywords.rst |  2 +-
 src/detect-file-data.c                |  2 +-
 3 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/doc/userguide/rules/file-keywords.rst b/doc/userguide/rules/file-keywords.rst
index a9b24deafe07..c708ee746c0d 100644
--- a/doc/userguide/rules/file-keywords.rst
+++ b/doc/userguide/rules/file-keywords.rst
@@ -5,6 +5,36 @@ Suricata comes with several rule keywords to match on various file
 properties. They depend on properly configured
 :doc:`../file-extraction/file-extraction`.
 
+file.data
+---------
+
+The ``file.data`` sticky buffer matches on contents of files that are 
+seen in flows that Suricata evaluates. The various payload keywords can
+be used (e.g. ``startswith``, ``nocase`` and ``bsize``) with ``file.data``.
+
+Example::
+
+  alert smtp any any -> any any (msg:"smtp app layer file.data example"; \
+ file.data; content:"example file content"; sid:1; rev:1)
+
+  alert http any any -> any any (msg:"http app layer file.data example"; \
+ file.data; content:"example file content"; sid:2; rev:1)
+
+  alert http2 any any -> any any (msg:"http2 app layer file.data example"; \
+ file.data; content:"example file content"; sid:3; rev:1;)
+
+  alert nfs any any -> any any (msg:"nfs app layer file.data example"; \
+ file.data; content:" "; sid:5; rev:1)
+
+  alert ftp-data any any -> any any (msg:"ftp app layer file.data example"; \
+ file.data; content:"example file content"; sid:6; rev:1;)
+
+  alert tcp any any -> any any (msg:"tcp file.data example"; \
+ file.data; content:"example file content"; sid:4; rev:1)
+
+**Note** file_data is the legacy notation but can still be used.
+
+
 file.name
 ---------
 
diff --git a/doc/userguide/rules/http-keywords.rst b/doc/userguide/rules/http-keywords.rst
index 001c0f542e84..0c0f652ad397 100644
--- a/doc/userguide/rules/http-keywords.rst
+++ b/doc/userguide/rules/http-keywords.rst
@@ -838,7 +838,7 @@ Notes
    than 1k, 'content:!"<html"; depth:1024;' can only match if the
    pattern '<html' is absent from the first inspected chunk.
 
--  ``file.data`` can also be used with SMTP
+-  Refer to :doc:`file-keywords` for additional information.
 
 Multiple Buffer Matching
 ~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/src/detect-file-data.c b/src/detect-file-data.c
index 1f162d0d5ce8..ec1069678f99 100644
--- a/src/detect-file-data.c
+++ b/src/detect-file-data.c
@@ -73,7 +73,7 @@ void DetectFiledataRegister(void)
     sigmatch_table[DETECT_FILE_DATA].name = "file.data";
     sigmatch_table[DETECT_FILE_DATA].alias = "file_data";
     sigmatch_table[DETECT_FILE_DATA].desc = "make content keywords match on file data";
-    sigmatch_table[DETECT_FILE_DATA].url = "/rules/http-keywords.html#file-data";
+    sigmatch_table[DETECT_FILE_DATA].url = "/rules/file-keywords.html#file-data";
     sigmatch_table[DETECT_FILE_DATA].Setup = DetectFiledataSetup;
 #ifdef UNITTESTS
     sigmatch_table[DETECT_FILE_DATA].RegisterTests = DetectFiledataRegisterTests;

From 180459eeb6c248b70a8b20b1e14f829318bb242d Mon Sep 17 00:00:00 2001
From: Eric Leblond <el@stamus-networks.com>
Date: Sat, 18 Nov 2023 22:07:47 +0100
Subject: [PATCH 149/462] detect/stream_size: fix prefiltering registration

Ticket: #6551
---
 src/detect-stream_size.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-stream_size.c b/src/detect-stream_size.c
index 86aabd77c8e8..50cd15af2d35 100644
--- a/src/detect-stream_size.c
+++ b/src/detect-stream_size.c
@@ -216,7 +216,7 @@ static bool PrefilterPacketStreamSizeCompare(PrefilterPacketHeaderValue v, void
 
 static int PrefilterSetupStreamSize(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
 {
-    return PrefilterSetupPacketHeader(de_ctx, sgh, DETECT_TCPMSS, PrefilterPacketStreamSizeSet,
+    return PrefilterSetupPacketHeader(de_ctx, sgh, DETECT_STREAM_SIZE, PrefilterPacketStreamSizeSet,
             PrefilterPacketStreamSizeCompare, PrefilterPacketStreamsizeMatch);
 }
 

From 55a4e128847224c2271fc6eff8ac16329e7583a2 Mon Sep 17 00:00:00 2001
From: Joseph Reilly <joser17@vt.edu>
Date: Tue, 1 Aug 2023 12:42:48 +0000
Subject: [PATCH 150/462] af-xdp: detach XDP program early

To mitigate a bug with AF_XDP sockets in high traffic scenarios, the XDP program must be detatched before
the sockets are closed. This issue happens when large ammounts of traffic are sent to suricata and
the XDP program is not removed before AF_XDP sockets are closed. I believe this is a race
condition bug as detailed here: https://bugzilla.kernel.org/show_bug.cgi?id=217712

Further investigation shows this may be a bug exclusive to the driver/AMD processor combination.

This commit addresses the bug by ensuring the first thread to run the deinit function
removes the XDP program, which fixes the bug as detailed in the bugzilla link.

Bug #6238
---
 src/source-af-xdp.c | 63 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 56 insertions(+), 7 deletions(-)

diff --git a/src/source-af-xdp.c b/src/source-af-xdp.c
index f68f7f5724e9..8f4b6071bd75 100644
--- a/src/source-af-xdp.c
+++ b/src/source-af-xdp.c
@@ -61,8 +61,10 @@
 #include "util-validate.h"
 
 #ifdef HAVE_AF_XDP
-#include <xdp/xsk.h>
 #include <net/if.h>
+#include <bpf/libbpf.h>
+#include <xdp/xsk.h>
+#include <xdp/libxdp.h>
 #endif
 
 #if HAVE_LINUX_IF_ETHER_H
@@ -113,7 +115,9 @@ TmEcode NoAFXDPSupportExit(ThreadVars *tv, const void *initdata, void **data)
 #else /* We have AF_XDP support */
 
 #define POLL_TIMEOUT      100
-#define NUM_FRAMES        XSK_RING_PROD__DEFAULT_NUM_DESCS
+#define NUM_FRAMES_PROD   XSK_RING_PROD__DEFAULT_NUM_DESCS
+#define NUM_FRAMES_CONS   XSK_RING_CONS__DEFAULT_NUM_DESCS
+#define NUM_FRAMES        NUM_FRAMES_PROD
 #define FRAME_SIZE        XSK_UMEM__DEFAULT_FRAME_SIZE
 #define MEM_BYTES         (NUM_FRAMES * FRAME_SIZE * 2)
 #define RECONNECT_TIMEOUT 500000
@@ -636,14 +640,14 @@ static TmEcode ReceiveAFXDPThreadInit(ThreadVars *tv, const void *initdata, void
     ptv->threads = afxdpconfig->threads;
 
     /* Socket configuration */
-    ptv->xsk.cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
-    ptv->xsk.cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
+    ptv->xsk.cfg.rx_size = NUM_FRAMES_CONS;
+    ptv->xsk.cfg.tx_size = NUM_FRAMES_PROD;
     ptv->xsk.cfg.xdp_flags = afxdpconfig->mode;
     ptv->xsk.cfg.bind_flags = afxdpconfig->bind_flags;
 
     /* UMEM configuration */
-    ptv->umem.cfg.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS * 2;
-    ptv->umem.cfg.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
+    ptv->umem.cfg.fill_size = NUM_FRAMES_PROD * 2;
+    ptv->umem.cfg.comp_size = NUM_FRAMES_CONS;
     ptv->umem.cfg.frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
     ptv->umem.cfg.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM;
     ptv->umem.cfg.flags = afxdpconfig->mem_alignment;
@@ -824,15 +828,60 @@ static TmEcode ReceiveAFXDPLoop(ThreadVars *tv, void *data, void *slot)
     SCReturnInt(TM_ECODE_OK);
 }
 
+/**
+ * \brief function to unload an AF_XDP program
+ *
+ */
+static void RunModeAFXDPRemoveProg(char *iface_name)
+{
+    unsigned int ifindex = if_nametoindex(iface_name);
+
+    struct xdp_multiprog *progs = xdp_multiprog__get_from_ifindex(ifindex);
+    if (progs == NULL) {
+        return;
+    }
+    enum xdp_attach_mode mode = xdp_multiprog__attach_mode(progs);
+
+    struct xdp_program *prog = NULL;
+
+    // loop through the multiprogram struct, removing all the programs
+    for (prog = xdp_multiprog__next_prog(NULL, progs); prog;
+            prog = xdp_multiprog__next_prog(prog, progs)) {
+        int ret = xdp_program__detach(prog, ifindex, mode, 0);
+        if (ret) {
+            SCLogDebug("Error: cannot detatch XDP program: %s\n", strerror(errno));
+        }
+    }
+
+    prog = xdp_multiprog__main_prog(progs);
+    if (xdp_program__is_attached(prog, ifindex) != XDP_MODE_UNSPEC) {
+        int ret = xdp_program__detach(prog, ifindex, mode, 0);
+        if (ret) {
+            SCLogDebug("Error: cannot detatch XDP program: %s\n", strerror(errno));
+        }
+    }
+}
+
 /**
  * \brief DeInit function closes af-xdp socket at exit.
  * \param tv pointer to ThreadVars
  * \param data pointer that gets cast into AFXDPPThreadVars for ptv
  */
+static SCMutex sync_deinit = SCMUTEX_INITIALIZER;
+
 static TmEcode ReceiveAFXDPThreadDeinit(ThreadVars *tv, void *data)
 {
     AFXDPThreadVars *ptv = (AFXDPThreadVars *)data;
 
+    /*
+     * If AF_XDP is enabled, the program must be detached before the AF_XDP sockets
+     * are closed to mitigate a bug that causes an IO_PAGEFAULT in linux kernel
+     * version 5.19, unknown as of now what other versions this affects.
+     */
+    SCMutexLock(&sync_deinit);
+    RunModeAFXDPRemoveProg(ptv->iface);
+    SCMutexUnlock(&sync_deinit);
+
     if (ptv->xsk.xsk) {
         xsk_socket__delete(ptv->xsk.xsk);
         ptv->xsk.xsk = NULL;
@@ -924,4 +973,4 @@ static TmEcode DecodeAFXDPThreadDeinit(ThreadVars *tv, void *data)
 /* eof */
 /**
  * @}
- */
+ */
\ No newline at end of file

From 32cce122e1d8bb568cd5ff1b0db51c488cfb9f58 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 9 Nov 2023 09:57:58 +0100
Subject: [PATCH 151/462] detect: header_lowercase transform

Ticket: 6290
---
 doc/userguide/rules/transforms.rst      | 15 +++++
 src/Makefile.am                         |  2 +
 src/detect-engine-register.c            |  2 +
 src/detect-engine-register.h            |  1 +
 src/detect-transform-header-lowercase.c | 88 +++++++++++++++++++++++++
 src/detect-transform-header-lowercase.h | 30 +++++++++
 6 files changed, 138 insertions(+)
 create mode 100644 src/detect-transform-header-lowercase.c
 create mode 100644 src/detect-transform-header-lowercase.h

diff --git a/doc/userguide/rules/transforms.rst b/doc/userguide/rules/transforms.rst
index f52bac7f3eea..0067ace1de8b 100644
--- a/doc/userguide/rules/transforms.rst
+++ b/doc/userguide/rules/transforms.rst
@@ -159,3 +159,18 @@ Example::
     alert http any any -> any any (msg:"HTTP with xor"; http.uri; \
         xor:"0d0ac8ff"; content:"password="; sid:1;)
 
+header_lowercase
+----------------
+
+This transform is meant for HTTP/1 HTTP/2 header names normalization.
+It lowercases the header names, while keeping untouched the header values.
+
+The implementation uses a state machine :
+- it lowercases until it finds ``:```
+- it does not change until it finds a new line and switch back to first state
+
+This example alerts for both HTTP/1 and HTTP/2 with a authorization header
+Example::
+
+    alert http any any -> any any (msg:"HTTP authorization"; http.header_names; \
+        header_lowercase; content:"authorization:"; sid:1;)
diff --git a/src/Makefile.am b/src/Makefile.am
index f8033de41b88..21e1dfe5fbeb 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -342,6 +342,7 @@ noinst_HEADERS = \
 	detect-transform-casechange.h \
 	detect-transform-compress-whitespace.h \
 	detect-transform-dotprefix.h \
+	detect-transform-header-lowercase.h \
 	detect-transform-md5.h \
 	detect-transform-pcrexform.h \
 	detect-transform-sha1.h \
@@ -955,6 +956,7 @@ libsuricata_c_a_SOURCES = \
 	detect-transform-casechange.c \
 	detect-transform-compress-whitespace.c \
 	detect-transform-dotprefix.c \
+	detect-transform-header-lowercase.c \
 	detect-transform-md5.c \
 	detect-transform-pcrexform.c \
 	detect-transform-sha1.c \
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index bd8f66519683..0f459eccb67b 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -247,6 +247,7 @@
 #include "detect-transform-urldecode.h"
 #include "detect-transform-xor.h"
 #include "detect-transform-casechange.h"
+#include "detect-transform-header-lowercase.h"
 
 #include "util-rule-vars.h"
 
@@ -706,6 +707,7 @@ void SigTableSetup(void)
     DetectTransformXorRegister();
     DetectTransformToLowerRegister();
     DetectTransformToUpperRegister();
+    DetectTransformHeaderLowercaseRegister();
 
     DetectFileHandlerRegister();
 
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index abc1a403dd09..273aa10d7c9b 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -328,6 +328,7 @@ enum DetectKeywordId {
     DETECT_TRANSFORM_XOR,
     DETECT_TRANSFORM_TOLOWER,
     DETECT_TRANSFORM_TOUPPER,
+    DETECT_TRANSFORM_HEADER_LOWERCASE,
 
     DETECT_AL_IKE_EXCH_TYPE,
     DETECT_AL_IKE_SPI_INITIATOR,
diff --git a/src/detect-transform-header-lowercase.c b/src/detect-transform-header-lowercase.c
new file mode 100644
index 000000000000..7c776201b308
--- /dev/null
+++ b/src/detect-transform-header-lowercase.c
@@ -0,0 +1,88 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Philippe Antoine <contact@catenacyber.fr>
+ *
+ * Implements the header_lowercase transform keyword with option support
+ */
+
+#include "suricata-common.h"
+#include "detect.h"
+#include "detect-engine.h"
+#include "detect-parse.h"
+#include "detect-transform-header-lowercase.h"
+
+/**
+ *  \internal
+ *  \brief Apply the header_lowercase keyword to the last pattern match
+ *  \param det_ctx detection engine ctx
+ *  \param s signature
+ *  \param optstr options string
+ *  \retval 0 ok
+ *  \retval -1 failure
+ */
+static int DetectTransformHeaderLowercaseSetup(
+        DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
+{
+    SCEnter();
+    int r = DetectSignatureAddTransform(s, DETECT_TRANSFORM_HEADER_LOWERCASE, NULL);
+    SCReturnInt(r);
+}
+
+static void DetectTransformHeaderLowercase(InspectionBuffer *buffer, void *options)
+{
+    const uint8_t *input = buffer->inspect;
+    const uint32_t input_len = buffer->inspect_len;
+    if (input_len == 0) {
+        return;
+    }
+    uint8_t output[input_len];
+
+    // state 0 is header name, 1 is header value
+    int state = 0;
+    for (uint32_t i = 0; i < input_len; i++) {
+        if (state == 0) {
+            if (input[i] == ':') {
+                output[i] = input[i];
+                state = 1;
+            } else {
+                output[i] = u8_tolower(input[i]);
+            }
+        } else {
+            output[i] = input[i];
+            if (input[i] == '\n') {
+                state = 0;
+            }
+        }
+    }
+    InspectionBufferCopy(buffer, output, input_len);
+}
+
+void DetectTransformHeaderLowercaseRegister(void)
+{
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].name = "header_lowercase";
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].desc =
+            "modify buffer via lowercaseing header names";
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].url =
+            "/rules/transforms.html#header_lowercase";
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].Transform = DetectTransformHeaderLowercase;
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].Setup = DetectTransformHeaderLowercaseSetup;
+    sigmatch_table[DETECT_TRANSFORM_HEADER_LOWERCASE].flags |= SIGMATCH_NOOPT;
+}
diff --git a/src/detect-transform-header-lowercase.h b/src/detect-transform-header-lowercase.h
new file mode 100644
index 000000000000..aca7f874a499
--- /dev/null
+++ b/src/detect-transform-header-lowercase.h
@@ -0,0 +1,30 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Philippe Antoine <contact@catenacyber.fr>
+ */
+
+#ifndef __DETECT_TRANSFORM_HEADER_LOWERCASE_H__
+#define __DETECT_TRANSFORM_HEADER_LOWERCASE_H__
+
+/* prototypes */
+void DetectTransformHeaderLowercaseRegister(void);
+
+#endif /* __DETECT_TRANSFORM_HEADER_LOWERCASE_H__ */

From 90c17652a3ee2e7cad7e61515b389fa3b55996c2 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 12 Jun 2023 22:17:26 +0200
Subject: [PATCH 152/462] rust: remove unused

Ticket: #4083
---
 rust/src/smb/smb2_records.rs | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/rust/src/smb/smb2_records.rs b/rust/src/smb/smb2_records.rs
index 4a7721cdaa0a..7de9e6607dda 100644
--- a/rust/src/smb/smb2_records.rs
+++ b/rust/src/smb/smb2_records.rs
@@ -32,11 +32,6 @@ pub struct Smb2SecBlobRecord<'a> {
     pub data: &'a [u8],
 }
 
-pub fn parse_smb2_sec_blob(i: &[u8]) -> IResult<&[u8], Smb2SecBlobRecord> {
-    let (i, data) = rest(i)?;
-    Ok((i, Smb2SecBlobRecord { data }))
-}
-
 #[derive(Debug, PartialEq, Eq)]
 pub struct Smb2RecordDir {
     pub request: bool,

From 0b6b015e26e92d88ebfd7ddac6ee34646fc0a57c Mon Sep 17 00:00:00 2001
From: Philippe Antoine <contact@catenacyber.fr>
Date: Thu, 11 May 2023 10:02:32 +0200
Subject: [PATCH 153/462] output/alert: rewrite code for app-layer properties

Especially fix setup-app-layer script to not forget this part

This allows, for simple loggers, to have a unique definition
of the actual logging function with the jsonbuilder.
This way, alerts, files, and app-layer event can share the code
to output the same data.

Ticket: #3827
---
 rust/src/applayertemplate/logger.rs |   2 +
 rust/src/bittorrent_dht/logger.rs   |   2 +
 rust/src/http2/logger.rs            |   5 +-
 rust/src/quic/logger.rs             |   4 +-
 rust/src/snmp/log.rs                |   4 +-
 rust/src/ssh/logger.rs              |   2 +
 scripts/setup-app-layer.py          |   4 +
 src/app-layer-ftp.c                 |  11 +-
 src/app-layer-ftp.h                 |   2 +-
 src/output-json-alert.c             | 229 ++--------------------------
 src/output-json-bittorrent-dht.c    |   2 -
 src/output-json-dnp3.c              |  21 +++
 src/output-json-dnp3.h              |   1 +
 src/output-json-dns.c               |  21 ++-
 src/output-json-dns.h               |   3 +-
 src/output-json-file.c              |  21 ++-
 src/output-json-ftp.c               |  11 +-
 src/output-json-http2.c             |  15 --
 src/output-json-http2.h             |   1 -
 src/output-json-modbus.c            |  13 --
 src/output-json-modbus.h            |   1 -
 src/output-json-mqtt.c              |  12 +-
 src/output-json-mqtt.h              |   2 +-
 src/output-json-quic.c              |  13 --
 src/output-json-quic.h              |   1 -
 src/output-json-rfb.c               |  13 --
 src/output-json-rfb.h               |   2 -
 src/output-json-sip.c               |  11 --
 src/output-json-sip.h               |   2 -
 src/output-json-snmp.c              |   4 +-
 src/output-json-ssh.c               |   2 -
 src/output-json-template.c          |   2 -
 src/output-json-tls.c               |  14 +-
 src/output-json-tls.h               |   2 +-
 src/output.c                        |  52 +++++++
 src/output.h                        |   9 ++
 36 files changed, 177 insertions(+), 339 deletions(-)

diff --git a/rust/src/applayertemplate/logger.rs b/rust/src/applayertemplate/logger.rs
index 0105526fc0f1..766a07acdb9d 100644
--- a/rust/src/applayertemplate/logger.rs
+++ b/rust/src/applayertemplate/logger.rs
@@ -20,12 +20,14 @@ use crate::jsonbuilder::{JsonBuilder, JsonError};
 use std;
 
 fn log_template(tx: &TemplateTransaction, js: &mut JsonBuilder) -> Result<(), JsonError> {
+    js.open_object("template")?;
     if let Some(ref request) = tx.request {
         js.set_string("request", request)?;
     }
     if let Some(ref response) = tx.response {
         js.set_string("response", response)?;
     }
+    js.close()?;
     Ok(())
 }
 
diff --git a/rust/src/bittorrent_dht/logger.rs b/rust/src/bittorrent_dht/logger.rs
index 2cfb9270ef67..74ea7c59ba57 100644
--- a/rust/src/bittorrent_dht/logger.rs
+++ b/rust/src/bittorrent_dht/logger.rs
@@ -48,6 +48,7 @@ fn print_ip_addr(addr: &[u8]) -> std::string::String {
 fn log_bittorrent_dht(
     tx: &BitTorrentDHTTransaction, js: &mut JsonBuilder,
 ) -> Result<(), JsonError> {
+    js.open_object("bittorrent_dht")?;
     js.set_hex("transaction_id", &tx.transaction_id)?;
     if let Some(client_version) = &tx.client_version {
         js.set_hex("client_version", client_version)?;
@@ -125,6 +126,7 @@ fn log_bittorrent_dht(
         }
         js.close()?;
     };
+    js.close()?;
     Ok(())
 }
 
diff --git a/rust/src/http2/logger.rs b/rust/src/http2/logger.rs
index d25f852c43ab..099112b1aeb4 100644
--- a/rust/src/http2/logger.rs
+++ b/rust/src/http2/logger.rs
@@ -192,6 +192,7 @@ fn log_http2_frames(frames: &[HTTP2Frame], js: &mut JsonBuilder) -> Result<bool,
 }
 
 fn log_http2(tx: &HTTP2Transaction, js: &mut JsonBuilder) -> Result<bool, JsonError> {
+    js.open_object("http")?;
     js.set_string("version", "2")?;
 
     let mut common: HashMap<HeaderName, &Vec<u8>> = HashMap::new();
@@ -261,8 +262,8 @@ fn log_http2(tx: &HTTP2Transaction, js: &mut JsonBuilder) -> Result<bool, JsonEr
     let has_response = log_http2_frames(&tx.frames_tc, js)?;
     js.close()?;
 
-    // Close http2.
-    js.close()?;
+    js.close()?; // http2
+    js.close()?; // http
 
     return Ok(has_request || has_response || has_headers);
 }
diff --git a/rust/src/quic/logger.rs b/rust/src/quic/logger.rs
index e03ebdd6bf21..8cb08830e4d0 100644
--- a/rust/src/quic/logger.rs
+++ b/rust/src/quic/logger.rs
@@ -88,7 +88,7 @@ fn quic_tls_extension_name(e: u16) -> Option<String> {
     }
 }
 
-fn log_template(tx: &QuicTransaction, js: &mut JsonBuilder) -> Result<(), JsonError> {
+fn log_quic(tx: &QuicTransaction, js: &mut JsonBuilder) -> Result<(), JsonError> {
     js.open_object("quic")?;
     if tx.header.ty != QuicType::Short {
         js.set_string("version", String::from(tx.header.version).as_str())?;
@@ -153,5 +153,5 @@ pub unsafe extern "C" fn rs_quic_to_json(
     tx: *mut std::os::raw::c_void, js: &mut JsonBuilder,
 ) -> bool {
     let tx = cast_pointer!(tx, QuicTransaction);
-    log_template(tx, js).is_ok()
+    log_quic(tx, js).is_ok()
 }
diff --git a/rust/src/snmp/log.rs b/rust/src/snmp/log.rs
index 83414816c466..5707f30ccb4e 100644
--- a/rust/src/snmp/log.rs
+++ b/rust/src/snmp/log.rs
@@ -39,6 +39,7 @@ fn str_of_pdu_type(t:&PduType) -> Cow<str> {
 
 fn snmp_log_response(jsb: &mut JsonBuilder, tx: &mut SNMPTransaction) -> Result<(), JsonError>
 {
+    jsb.open_object("snmp")?;
     jsb.set_uint("version", tx.version as u64)?;
     if tx.encrypted {
         jsb.set_string("pdu_type", "encrypted")?;
@@ -71,11 +72,12 @@ fn snmp_log_response(jsb: &mut JsonBuilder, tx: &mut SNMPTransaction) -> Result<
         }
     }
 
+    jsb.close()?;
     return Ok(());
 }
 
 #[no_mangle]
-pub extern "C" fn rs_snmp_log_json_response(jsb: &mut JsonBuilder, tx: &mut SNMPTransaction) -> bool
+pub extern "C" fn rs_snmp_log_json_response(tx: &mut SNMPTransaction, jsb: &mut JsonBuilder) -> bool
 {
     snmp_log_response(jsb, tx).is_ok()
 }
diff --git a/rust/src/ssh/logger.rs b/rust/src/ssh/logger.rs
index 9bc7d7c33f39..008c6cb4517a 100644
--- a/rust/src/ssh/logger.rs
+++ b/rust/src/ssh/logger.rs
@@ -19,6 +19,7 @@ use super::ssh::SSHTransaction;
 use crate::jsonbuilder::{JsonBuilder, JsonError};
 
 fn log_ssh(tx: &SSHTransaction, js: &mut JsonBuilder) -> Result<bool, JsonError> {
+    js.open_object("ssh")?;
     if tx.cli_hdr.protover.is_empty() && tx.srv_hdr.protover.is_empty() {
         return Ok(false);
     }
@@ -58,6 +59,7 @@ fn log_ssh(tx: &SSHTransaction, js: &mut JsonBuilder) -> Result<bool, JsonError>
         }
         js.close()?;
     }
+    js.close()?;
     return Ok(true);
 }
 
diff --git a/scripts/setup-app-layer.py b/scripts/setup-app-layer.py
index 72f28c986c66..d8426634bca8 100755
--- a/scripts/setup-app-layer.py
+++ b/scripts/setup-app-layer.py
@@ -200,6 +200,10 @@ def logger_patch_output_c(proto):
     output = io.StringIO()
     inlines = open(filename).readlines()
     for i, line in enumerate(inlines):
+        if line.find("ALPROTO_TEMPLATE") > -1:
+            new_line = line.replace("TEMPLATE", proto.upper()).replace(
+                    "template", proto.lower())
+            output.write(new_line)
         if line.find("output-json-template.h") > -1:
             output.write(line.replace("template", proto.lower()))
         if line.find("/* Template JSON logger.") > -1:
diff --git a/src/app-layer-ftp.c b/src/app-layer-ftp.c
index c0a815e31dae..f46a4a967e24 100644
--- a/src/app-layer-ftp.c
+++ b/src/app-layer-ftp.c
@@ -1405,13 +1405,10 @@ uint16_t JsonGetNextLineFromBuffer(const char *buffer, const uint16_t len)
     return c == NULL ? len : (uint16_t)(c - buffer + 1);
 }
 
-void EveFTPDataAddMetadata(const Flow *f, JsonBuilder *jb)
+bool EveFTPDataAddMetadata(void *vtx, JsonBuilder *jb)
 {
-    const FtpDataState *ftp_state = NULL;
-    if (f->alstate == NULL)
-        return;
-
-    ftp_state = (FtpDataState *)f->alstate;
+    const FtpDataState *ftp_state = (FtpDataState *)vtx;
+    jb_open_object(jb, "ftp_data");
 
     if (ftp_state->file_name) {
         jb_set_string_from_bytes(jb, "filename", ftp_state->file_name, ftp_state->file_len);
@@ -1426,6 +1423,8 @@ void EveFTPDataAddMetadata(const Flow *f, JsonBuilder *jb)
         default:
             break;
     }
+    jb_close(jb);
+    return true;
 }
 
 /**
diff --git a/src/app-layer-ftp.h b/src/app-layer-ftp.h
index f79c5c9e7675..fb71d6b52de7 100644
--- a/src/app-layer-ftp.h
+++ b/src/app-layer-ftp.h
@@ -190,7 +190,7 @@ uint64_t FTPMemuseGlobalCounter(void);
 uint64_t FTPMemcapGlobalCounter(void);
 
 uint16_t JsonGetNextLineFromBuffer(const char *buffer, const uint16_t len);
-void EveFTPDataAddMetadata(const Flow *f, JsonBuilder *jb);
+bool EveFTPDataAddMetadata(void *vtx, JsonBuilder *jb);
 
 #endif /* __APP_LAYER_FTP_H__ */
 
diff --git a/src/output-json-alert.c b/src/output-json-alert.c
index c3886231c001..ad9d236f3090 100644
--- a/src/output-json-alert.c
+++ b/src/output-json-alert.c
@@ -137,164 +137,6 @@ static int AlertJsonDumpStreamSegmentCallback(
     return 1;
 }
 
-static void AlertJsonTls(const Flow *f, JsonBuilder *js)
-{
-    SSLState *ssl_state = (SSLState *)FlowGetAppState(f);
-    if (ssl_state) {
-        jb_open_object(js, "tls");
-
-        JsonTlsLogJSONExtended(js, ssl_state);
-
-        jb_close(js);
-    }
-
-    return;
-}
-
-static void AlertJsonSsh(const Flow *f, JsonBuilder *js)
-{
-    void *ssh_state = FlowGetAppState(f);
-    if (ssh_state) {
-        JsonBuilderMark mark = { 0, 0, 0 };
-        void *tx_ptr = rs_ssh_state_get_tx(ssh_state, 0);
-        jb_get_mark(js, &mark);
-        jb_open_object(js, "ssh");
-        if (rs_ssh_log_json(tx_ptr, js)) {
-            jb_close(js);
-        } else {
-            jb_restore_mark(js, &mark);
-        }
-    }
-
-    return;
-}
-
-static void AlertJsonHttp2(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    void *h2_state = FlowGetAppState(f);
-    if (h2_state) {
-        void *tx_ptr = rs_http2_state_get_tx(h2_state, tx_id);
-        if (tx_ptr) {
-            JsonBuilderMark mark = { 0, 0, 0 };
-            jb_get_mark(js, &mark);
-            jb_open_object(js, "http");
-            if (rs_http2_log_json(tx_ptr, js)) {
-                jb_close(js);
-            } else {
-                jb_restore_mark(js, &mark);
-            }
-        }
-    }
-
-    return;
-}
-
-static void AlertJsonDnp3(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    DNP3State *dnp3_state = (DNP3State *)FlowGetAppState(f);
-    if (dnp3_state) {
-        DNP3Transaction *tx = AppLayerParserGetTx(IPPROTO_TCP, ALPROTO_DNP3,
-            dnp3_state, tx_id);
-        if (tx) {
-            JsonBuilderMark mark = { 0, 0, 0 };
-            jb_get_mark(js, &mark);
-            bool logged = false;
-            jb_open_object(js, "dnp3");
-            if (tx->is_request && tx->done) {
-                jb_open_object(js, "request");
-                JsonDNP3LogRequest(js, tx);
-                jb_close(js);
-                logged = true;
-            }
-            if (!tx->is_request && tx->done) {
-                jb_open_object(js, "response");
-                JsonDNP3LogResponse(js, tx);
-                jb_close(js);
-                logged = true;
-            }
-            if (logged) {
-                /* Close dnp3 object. */
-                jb_close(js);
-            } else {
-                jb_restore_mark(js, &mark);
-            }
-        }
-    }
-}
-
-static void AlertJsonDns(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    void *dns_state = (void *)FlowGetAppState(f);
-    if (dns_state) {
-        void *txptr = AppLayerParserGetTx(f->proto, ALPROTO_DNS,
-                                          dns_state, tx_id);
-        if (txptr) {
-            jb_open_object(js, "dns");
-            JsonBuilder *qjs = JsonDNSLogQuery(txptr);
-            if (qjs != NULL) {
-                jb_set_object(js, "query", qjs);
-                jb_free(qjs);
-            }
-            JsonBuilder *ajs = JsonDNSLogAnswer(txptr);
-            if (ajs != NULL) {
-                jb_set_object(js, "answer", ajs);
-                jb_free(ajs);
-            }
-            jb_close(js);
-        }
-    }
-    return;
-}
-
-static void AlertJsonSNMP(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    void *snmp_state = (void *)FlowGetAppState(f);
-    if (snmp_state != NULL) {
-        void *tx = AppLayerParserGetTx(f->proto, ALPROTO_SNMP, snmp_state,
-                tx_id);
-        if (tx != NULL) {
-            jb_open_object(js, "snmp");
-            rs_snmp_log_json_response(js, tx);
-            jb_close(js);
-        }
-    }
-}
-
-static void AlertJsonRDP(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    void *rdp_state = (void *)FlowGetAppState(f);
-    if (rdp_state != NULL) {
-        void *tx = AppLayerParserGetTx(f->proto, ALPROTO_RDP, rdp_state,
-                tx_id);
-        if (tx != NULL) {
-            JsonBuilderMark mark = { 0, 0, 0 };
-            jb_get_mark(js, &mark);
-            if (!rs_rdp_to_json(tx, js)) {
-                jb_restore_mark(js, &mark);
-            }
-        }
-    }
-}
-
-static void AlertJsonBitTorrentDHT(const Flow *f, const uint64_t tx_id, JsonBuilder *js)
-{
-    void *bittorrent_dht_state = (void *)FlowGetAppState(f);
-    if (bittorrent_dht_state != NULL) {
-        void *tx =
-                AppLayerParserGetTx(f->proto, ALPROTO_BITTORRENT_DHT, bittorrent_dht_state, tx_id);
-        if (tx != NULL) {
-            JsonBuilderMark mark = { 0, 0, 0 };
-            jb_get_mark(js, &mark);
-            jb_open_object(js, "bittorrent_dht");
-            if (rs_bittorrent_dht_logger_log(tx, js)) {
-                jb_close(js);
-            } else {
-                jb_restore_mark(js, &mark);
-            }
-        }
-    }
-}
-
 static void AlertJsonSourceTarget(const Packet *p, const PacketAlert *pa,
                                   JsonBuilder *js, JsonAddrInfo *addr)
 {
@@ -471,7 +313,21 @@ static void AlertAddAppLayer(const Packet *p, JsonBuilder *jb,
         const uint64_t tx_id, const uint16_t option_flags)
 {
     const AppProto proto = FlowGetAppProtocol(p->flow);
+    EveJsonSimpleAppLayerLogger *al = SCEveJsonSimpleGetLogger(proto);
     JsonBuilderMark mark = { 0, 0, 0 };
+    if (al && al->LogTx) {
+        void *state = FlowGetAppState(p->flow);
+        if (state) {
+            void *tx = AppLayerParserGetTx(p->flow->proto, proto, state, tx_id);
+            if (tx) {
+                jb_get_mark(jb, &mark);
+                if (!al->LogTx(tx, jb)) {
+                    jb_restore_mark(jb, &mark);
+                }
+            }
+        }
+        return;
+    }
     switch (proto) {
         case ALPROTO_HTTP1:
             // TODO: Could result in an empty http object being logged.
@@ -486,12 +342,6 @@ static void AlertAddAppLayer(const Packet *p, JsonBuilder *jb,
             }
             jb_close(jb);
             break;
-        case ALPROTO_TLS:
-            AlertJsonTls(p->flow, jb);
-            break;
-        case ALPROTO_SSH:
-            AlertJsonSsh(p->flow, jb);
-            break;
         case ALPROTO_SMTP:
             jb_get_mark(jb, &mark);
             jb_open_object(jb, "smtp");
@@ -535,63 +385,12 @@ static void AlertAddAppLayer(const Packet *p, JsonBuilder *jb,
                 jb_restore_mark(jb, &mark);
             }
             break;
-        case ALPROTO_SIP:
-            JsonSIPAddMetadata(jb, p->flow, tx_id);
-            break;
-        case ALPROTO_RFB:
-            jb_get_mark(jb, &mark);
-            if (!JsonRFBAddMetadata(p->flow, tx_id, jb)) {
-                jb_restore_mark(jb, &mark);
-            }
-            break;
-        case ALPROTO_FTPDATA:
-            jb_get_mark(jb, &mark);
-            jb_open_object(jb, "ftp_data");
-            EveFTPDataAddMetadata(p->flow, jb);
-            jb_close(jb);
-            break;
-        case ALPROTO_DNP3:
-            AlertJsonDnp3(p->flow, tx_id, jb);
-            break;
-        case ALPROTO_HTTP2:
-            AlertJsonHttp2(p->flow, tx_id, jb);
-            break;
-        case ALPROTO_DNS:
-            AlertJsonDns(p->flow, tx_id, jb);
-            break;
         case ALPROTO_IKE:
             jb_get_mark(jb, &mark);
             if (!EveIKEAddMetadata(p->flow, tx_id, jb)) {
                 jb_restore_mark(jb, &mark);
             }
             break;
-        case ALPROTO_MQTT:
-            jb_get_mark(jb, &mark);
-            if (!JsonMQTTAddMetadata(p->flow, tx_id, jb)) {
-                jb_restore_mark(jb, &mark);
-            }
-            break;
-        case ALPROTO_QUIC:
-            jb_get_mark(jb, &mark);
-            if (!JsonQuicAddMetadata(p->flow, tx_id, jb)) {
-                jb_restore_mark(jb, &mark);
-            }
-            break;
-        case ALPROTO_SNMP:
-            AlertJsonSNMP(p->flow, tx_id, jb);
-            break;
-        case ALPROTO_RDP:
-            AlertJsonRDP(p->flow, tx_id, jb);
-            break;
-        case ALPROTO_MODBUS:
-            jb_get_mark(jb, &mark);
-            if (!JsonModbusAddMetadata(p->flow, tx_id, jb)) {
-                jb_restore_mark(jb, &mark);
-            }
-            break;
-        case ALPROTO_BITTORRENT_DHT:
-            AlertJsonBitTorrentDHT(p->flow, tx_id, jb);
-            break;
         default:
             break;
     }
diff --git a/src/output-json-bittorrent-dht.c b/src/output-json-bittorrent-dht.c
index 08b7dc4d722c..066df78f61fb 100644
--- a/src/output-json-bittorrent-dht.c
+++ b/src/output-json-bittorrent-dht.c
@@ -65,11 +65,9 @@ static int JsonBitTorrentDHTLogger(ThreadVars *tv, void *thread_data, const Pack
         return TM_ECODE_FAILED;
     }
 
-    jb_open_object(js, "bittorrent_dht");
     if (!rs_bittorrent_dht_logger_log(tx, js)) {
         goto error;
     }
-    jb_close(js);
 
     OutputJsonBuilderBuffer(js, thread->ctx);
     jb_free(js);
diff --git a/src/output-json-dnp3.c b/src/output-json-dnp3.c
index 97b1e92e00ce..4336e04e070c 100644
--- a/src/output-json-dnp3.c
+++ b/src/output-json-dnp3.c
@@ -210,6 +210,27 @@ void JsonDNP3LogResponse(JsonBuilder *js, DNP3Transaction *dnp3tx)
     jb_close(js);
 }
 
+bool AlertJsonDnp3(void *vtx, JsonBuilder *js)
+{
+    DNP3Transaction *tx = (DNP3Transaction *)vtx;
+    bool logged = false;
+    jb_open_object(js, "dnp3");
+    if (tx->is_request && tx->done) {
+        jb_open_object(js, "request");
+        JsonDNP3LogRequest(js, tx);
+        jb_close(js);
+        logged = true;
+    }
+    if (!tx->is_request && tx->done) {
+        jb_open_object(js, "response");
+        JsonDNP3LogResponse(js, tx);
+        jb_close(js);
+        logged = true;
+    }
+    jb_close(js);
+    return logged;
+}
+
 static int JsonDNP3LoggerToServer(ThreadVars *tv, void *thread_data,
     const Packet *p, Flow *f, void *state, void *vtx, uint64_t tx_id)
 {
diff --git a/src/output-json-dnp3.h b/src/output-json-dnp3.h
index 85d02ff1011c..6f81026780d9 100644
--- a/src/output-json-dnp3.h
+++ b/src/output-json-dnp3.h
@@ -24,5 +24,6 @@ void JsonDNP3LogRequest(JsonBuilder *js, DNP3Transaction *);
 void JsonDNP3LogResponse(JsonBuilder *js, DNP3Transaction *);
 
 void JsonDNP3LogRegister(void);
+bool AlertJsonDnp3(void *vtx, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_DNP3_H__ */
diff --git a/src/output-json-dns.c b/src/output-json-dns.c
index 020e27853a9e..b27c67feb240 100644
--- a/src/output-json-dns.c
+++ b/src/output-json-dns.c
@@ -263,7 +263,7 @@ typedef struct LogDnsLogThread_ {
     OutputJsonThreadCtx *ctx;
 } LogDnsLogThread;
 
-JsonBuilder *JsonDNSLogQuery(void *txptr)
+static JsonBuilder *JsonDNSLogQuery(void *txptr)
 {
     JsonBuilder *queryjb = jb_new_array();
     if (queryjb == NULL) {
@@ -292,7 +292,7 @@ JsonBuilder *JsonDNSLogQuery(void *txptr)
     return queryjb;
 }
 
-JsonBuilder *JsonDNSLogAnswer(void *txptr)
+static JsonBuilder *JsonDNSLogAnswer(void *txptr)
 {
     if (!rs_dns_do_log_answer(txptr, LOG_ALL_RRTYPES)) {
         return NULL;
@@ -304,6 +304,23 @@ JsonBuilder *JsonDNSLogAnswer(void *txptr)
     }
 }
 
+bool AlertJsonDns(void *txptr, JsonBuilder *js)
+{
+    jb_open_object(js, "dns");
+    JsonBuilder *qjs = JsonDNSLogQuery(txptr);
+    if (qjs != NULL) {
+        jb_set_object(js, "query", qjs);
+        jb_free(qjs);
+    }
+    JsonBuilder *ajs = JsonDNSLogAnswer(txptr);
+    if (ajs != NULL) {
+        jb_set_object(js, "answer", ajs);
+        jb_free(ajs);
+    }
+    jb_close(js);
+    return true;
+}
+
 static int JsonDnsLoggerToServer(ThreadVars *tv, void *thread_data,
     const Packet *p, Flow *f, void *alstate, void *txptr, uint64_t tx_id)
 {
diff --git a/src/output-json-dns.h b/src/output-json-dns.h
index 1e19427361b8..f46cad011089 100644
--- a/src/output-json-dns.h
+++ b/src/output-json-dns.h
@@ -26,7 +26,6 @@
 
 void JsonDnsLogRegister(void);
 
-JsonBuilder *JsonDNSLogQuery(void *txptr) __attribute__((nonnull));
-JsonBuilder *JsonDNSLogAnswer(void *txptr) __attribute__((nonnull));
+bool AlertJsonDns(void *vtx, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_DNS_H__ */
diff --git a/src/output-json-file.c b/src/output-json-file.c
index 3b015ea88e06..1018be06ee80 100644
--- a/src/output-json-file.c
+++ b/src/output-json-file.c
@@ -123,6 +123,7 @@ JsonBuilder *JsonBuildFileInfoRecord(const Packet *p, const File *ff, void *tx,
         return NULL;
 
     JsonBuilderMark mark = { 0, 0, 0 };
+    EveJsonSimpleAppLayerLogger *al;
     switch (p->flow->alproto) {
         case ALPROTO_HTTP1:
             jb_open_object(js, "http");
@@ -172,13 +173,19 @@ JsonBuilder *JsonBuildFileInfoRecord(const Packet *p, const File *ff, void *tx,
                 jb_restore_mark(js, &mark);
             }
             break;
-        case ALPROTO_HTTP2:
-            jb_get_mark(js, &mark);
-            jb_open_object(js, "http");
-            if (EveHTTP2AddMetadata(p->flow, tx_id, js)) {
-                jb_close(js);
-            } else {
-                jb_restore_mark(js, &mark);
+        default:
+            al = SCEveJsonSimpleGetLogger(p->flow->alproto);
+            if (al && al->LogTx) {
+                void *state = FlowGetAppState(p->flow);
+                if (state) {
+                    tx = AppLayerParserGetTx(p->flow->proto, p->flow->alproto, state, tx_id);
+                    if (tx) {
+                        jb_get_mark(js, &mark);
+                        if (!al->LogTx(tx, js)) {
+                            jb_restore_mark(js, &mark);
+                        }
+                    }
+                }
             }
             break;
     }
diff --git a/src/output-json-ftp.c b/src/output-json-ftp.c
index ece9344bf409..9fb8b8c622db 100644
--- a/src/output-json-ftp.c
+++ b/src/output-json-ftp.c
@@ -57,6 +57,7 @@ static void EveFTPLogCommand(FTPTransaction *tx, JsonBuilder *jb)
             return;
         }
     }
+    jb_open_object(jb, "ftp");
     jb_set_string(jb, "command", tx->command_descriptor->command_name);
     uint32_t min_length = tx->command_descriptor->command_length + 1; /* command + space */
     if (tx->request_length > min_length) {
@@ -149,6 +150,7 @@ static void EveFTPLogCommand(FTPTransaction *tx, JsonBuilder *jb)
     } else {
         JB_SET_FALSE(jb, "reply_truncated");
     }
+    jb_close(jb);
 }
 
 
@@ -169,17 +171,14 @@ static int JsonFTPLogger(ThreadVars *tv, void *thread_data,
     JsonBuilder *jb =
             CreateEveHeaderWithTxId(p, LOG_DIR_FLOW, event_type, NULL, tx_id, thread->ctx);
     if (likely(jb)) {
-        jb_open_object(jb, event_type);
         if (f->alproto == ALPROTO_FTPDATA) {
-            EveFTPDataAddMetadata(f, jb);
+            if (!EveFTPDataAddMetadata(vtx, jb)) {
+                goto fail;
+            }
         } else {
             EveFTPLogCommand(tx, jb);
         }
 
-        if (!jb_close(jb)) {
-            goto fail;
-        }
-
         OutputJsonBuilderBuffer(jb, thread);
 
         jb_free(jb);
diff --git a/src/output-json-http2.c b/src/output-json-http2.c
index d762e76d0665..7165ae8f6302 100644
--- a/src/output-json-http2.c
+++ b/src/output-json-http2.c
@@ -61,19 +61,6 @@ typedef struct JsonHttp2LogThread_ {
     OutputJsonThreadCtx *ctx;
 } JsonHttp2LogThread;
 
-
-bool EveHTTP2AddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *jb)
-{
-    void *state = FlowGetAppState(f);
-    if (state) {
-        void *tx = AppLayerParserGetTx(f->proto, ALPROTO_HTTP2, state, tx_id);
-        if (tx) {
-            return rs_http2_log_json(tx, jb);
-        }
-    }
-    return false;
-}
-
 static int JsonHttp2Logger(ThreadVars *tv, void *thread_data, const Packet *p,
                          Flow *f, void *state, void *txptr, uint64_t tx_id)
 {
@@ -88,11 +75,9 @@ static int JsonHttp2Logger(ThreadVars *tv, void *thread_data, const Packet *p,
     if (unlikely(js == NULL))
         return 0;
 
-    jb_open_object(js, "http");
     if (!rs_http2_log_json(txptr, js)) {
         goto end;
     }
-    jb_close(js);
     OutputJsonBuilderBuffer(js, aft->ctx);
 end:
     jb_free(js);
diff --git a/src/output-json-http2.h b/src/output-json-http2.h
index 66bf2ade968e..88ba420ab2df 100644
--- a/src/output-json-http2.h
+++ b/src/output-json-http2.h
@@ -25,6 +25,5 @@
 #define __OUTPUT_JSON_HTTP2_H__
 
 void JsonHttp2LogRegister(void);
-bool EveHTTP2AddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *jb);
 
 #endif /* __OUTPUT_JSON_HTTP2_H__ */
diff --git a/src/output-json-modbus.c b/src/output-json-modbus.c
index ace8c061f92d..9e508ead9acc 100644
--- a/src/output-json-modbus.c
+++ b/src/output-json-modbus.c
@@ -136,19 +136,6 @@ static TmEcode JsonModbusLogThreadDeinit(ThreadVars *t, void *data)
     return TM_ECODE_OK;
 }
 
-bool JsonModbusAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js)
-{
-    void *state = FlowGetAppState(f);
-    if (state) {
-        void *tx = AppLayerParserGetTx(f->proto, ALPROTO_MODBUS, state, tx_id);
-        if (tx) {
-            return rs_modbus_to_json(tx, js);
-        }
-    }
-
-    return false;
-}
-
 void JsonModbusLogRegister(void)
 {
     /* Register as an eve sub-module. */
diff --git a/src/output-json-modbus.h b/src/output-json-modbus.h
index 9bde2dae57a3..2b07e4eb2d5c 100644
--- a/src/output-json-modbus.h
+++ b/src/output-json-modbus.h
@@ -19,6 +19,5 @@
 #define __OUTPUT_JSON_MODBUS_H__
 
 void JsonModbusLogRegister(void);
-bool JsonModbusAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_MODBUS_H__ */
diff --git a/src/output-json-mqtt.c b/src/output-json-mqtt.c
index 9ea890508070..2f600343e20d 100644
--- a/src/output-json-mqtt.c
+++ b/src/output-json-mqtt.c
@@ -59,17 +59,9 @@ typedef struct LogMQTTLogThread_ {
     OutputJsonThreadCtx *ctx;
 } LogMQTTLogThread;
 
-bool JsonMQTTAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js)
+bool JsonMQTTAddMetadata(void *vtx, JsonBuilder *js)
 {
-    MQTTState *state = FlowGetAppState(f);
-    if (state) {
-        MQTTTransaction *tx = AppLayerParserGetTx(f->proto, ALPROTO_MQTT, state, tx_id);
-        if (tx) {
-            return rs_mqtt_logger_log(tx, MQTT_DEFAULTS, js);
-        }
-    }
-
-    return false;
+    return rs_mqtt_logger_log(vtx, MQTT_DEFAULTS, js);
 }
 
 static int JsonMQTTLogger(ThreadVars *tv, void *thread_data,
diff --git a/src/output-json-mqtt.h b/src/output-json-mqtt.h
index 1acb4e107faf..42d66f48680d 100644
--- a/src/output-json-mqtt.h
+++ b/src/output-json-mqtt.h
@@ -25,6 +25,6 @@
 #define __OUTPUT_JSON_MQTT_H__
 
 void JsonMQTTLogRegister(void);
-bool JsonMQTTAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js);
+bool JsonMQTTAddMetadata(void *vtx, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_MQTT_H__ */
diff --git a/src/output-json-quic.c b/src/output-json-quic.c
index fdf2d0f09340..830ac78fdfbb 100644
--- a/src/output-json-quic.c
+++ b/src/output-json-quic.c
@@ -140,19 +140,6 @@ static TmEcode JsonQuicLogThreadDeinit(ThreadVars *t, void *data)
     return TM_ECODE_OK;
 }
 
-bool JsonQuicAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js)
-{
-    void *state = FlowGetAppState(f);
-    if (state) {
-        void *tx = AppLayerParserGetTx(f->proto, ALPROTO_QUIC, state, tx_id);
-        if (tx) {
-            return rs_quic_to_json(tx, js);
-        }
-    }
-
-    return false;
-}
-
 void JsonQuicLogRegister(void)
 {
     /* Register as an eve sub-module. */
diff --git a/src/output-json-quic.h b/src/output-json-quic.h
index 2448d5063a34..48e38185f2bd 100644
--- a/src/output-json-quic.h
+++ b/src/output-json-quic.h
@@ -22,7 +22,6 @@
 #ifndef __OUTPUT_JSON_QUIC_H__
 #define __OUTPUT_JSON_QUIC_H__
 
-bool JsonQuicAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js);
 void JsonQuicLogRegister(void);
 
 #endif /* __OUTPUT_JSON_QUIC_H__ */
diff --git a/src/output-json-rfb.c b/src/output-json-rfb.c
index cc12d2f1bbdc..e2b832bece13 100644
--- a/src/output-json-rfb.c
+++ b/src/output-json-rfb.c
@@ -46,19 +46,6 @@
 
 #include "rust-bindings.h"
 
-bool JsonRFBAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js)
-{
-    void *state = FlowGetAppState(f);
-    if (state) {
-        RFBTransaction *tx = AppLayerParserGetTx(f->proto, ALPROTO_RFB, state, tx_id);
-        if (tx) {
-            return rs_rfb_logger_log(tx, js);
-        }
-    }
-
-    return false;
-}
-
 static int JsonRFBLogger(ThreadVars *tv, void *thread_data,
     const Packet *p, Flow *f, void *state, void *tx, uint64_t tx_id)
 {
diff --git a/src/output-json-rfb.h b/src/output-json-rfb.h
index 1264ee3f6b4b..7e4e48ebd4c8 100644
--- a/src/output-json-rfb.h
+++ b/src/output-json-rfb.h
@@ -26,6 +26,4 @@
 
 void JsonRFBLogRegister(void);
 
-bool JsonRFBAddMetadata(const Flow *f, uint64_t tx_id, JsonBuilder *js);
-
 #endif /* __OUTPUT_JSON_RFB_H__ */
diff --git a/src/output-json-sip.c b/src/output-json-sip.c
index 8297be1cc3eb..7dd442cf6aba 100644
--- a/src/output-json-sip.c
+++ b/src/output-json-sip.c
@@ -48,17 +48,6 @@
 
 #include "rust.h"
 
-void JsonSIPAddMetadata(JsonBuilder *js, const Flow *f, uint64_t tx_id)
-{
-    SIPState *state = FlowGetAppState(f);
-    if (state) {
-        SIPTransaction *tx = AppLayerParserGetTx(f->proto, ALPROTO_SIP, state, tx_id);
-        if (tx) {
-            rs_sip_log_json(tx, js);
-        }
-    }
-}
-
 static int JsonSIPLogger(ThreadVars *tv, void *thread_data,
     const Packet *p, Flow *f, void *state, void *tx, uint64_t tx_id)
 {
diff --git a/src/output-json-sip.h b/src/output-json-sip.h
index 60145dab5b98..0d2c53fa50df 100644
--- a/src/output-json-sip.h
+++ b/src/output-json-sip.h
@@ -26,6 +26,4 @@
 
 void JsonSIPLogRegister(void);
 
-void JsonSIPAddMetadata(JsonBuilder *js, const Flow *f, uint64_t tx_id);
-
 #endif /* __OUTPUT_JSON_SIP_H__ */
diff --git a/src/output-json-snmp.c b/src/output-json-snmp.c
index 27545b6f6903..cbf0a7c992e4 100644
--- a/src/output-json-snmp.c
+++ b/src/output-json-snmp.c
@@ -59,11 +59,9 @@ static int JsonSNMPLogger(ThreadVars *tv, void *thread_data,
         return TM_ECODE_FAILED;
     }
 
-    jb_open_object(jb, "snmp");
-    if (!rs_snmp_log_json_response(jb, snmptx)) {
+    if (!rs_snmp_log_json_response(snmptx, jb)) {
         goto error;
     }
-    jb_close(jb);
 
     OutputJsonBuilderBuffer(jb, thread);
 
diff --git a/src/output-json-ssh.c b/src/output-json-ssh.c
index 5ec70142f634..45a8d8eab333 100644
--- a/src/output-json-ssh.c
+++ b/src/output-json-ssh.c
@@ -64,11 +64,9 @@ static int JsonSshLogger(ThreadVars *tv, void *thread_data, const Packet *p,
     if (unlikely(js == NULL))
         return 0;
 
-    jb_open_object(js, "ssh");
     if (!rs_ssh_log_json(txptr, js)) {
         goto end;
     }
-    jb_close(js);
     OutputJsonBuilderBuffer(js, thread);
 
 end:
diff --git a/src/output-json-template.c b/src/output-json-template.c
index 76d42ad834e6..2ca48b7ae373 100644
--- a/src/output-json-template.c
+++ b/src/output-json-template.c
@@ -74,11 +74,9 @@ static int JsonTemplateLogger(ThreadVars *tv, void *thread_data, const Packet *p
         return TM_ECODE_FAILED;
     }
 
-    jb_open_object(js, "template");
     if (!rs_template_logger_log(tx, js)) {
         goto error;
     }
-    jb_close(js);
 
     OutputJsonBuilderBuffer(js, thread->ctx);
     jb_free(js);
diff --git a/src/output-json-tls.c b/src/output-json-tls.c
index 9771f4d1cd7c..7460a32f2574 100644
--- a/src/output-json-tls.c
+++ b/src/output-json-tls.c
@@ -392,8 +392,9 @@ static void JsonTlsLogJSONCustom(OutputTlsCtx *tls_ctx, JsonBuilder *js,
     }
 }
 
-void JsonTlsLogJSONExtended(JsonBuilder *tjs, SSLState * state)
+static bool JsonTlsLogJSONExtendedAux(void *vtx, JsonBuilder *tjs)
 {
+    SSLState *state = (SSLState *)vtx;
     JsonTlsLogJSONBasic(tjs, state);
 
     /* tls serial */
@@ -425,6 +426,15 @@ void JsonTlsLogJSONExtended(JsonBuilder *tjs, SSLState * state)
         JsonTlsLogClientCert(tjs, &state->client_connp, false, false);
         jb_close(tjs);
     }
+    return true;
+}
+
+bool JsonTlsLogJSONExtended(void *vtx, JsonBuilder *tjs)
+{
+    jb_open_object(tjs, "tls");
+    bool r = JsonTlsLogJSONExtendedAux(vtx, tjs);
+    jb_close(tjs);
+    return r;
 }
 
 static int JsonTlsLogger(ThreadVars *tv, void *thread_data, const Packet *p,
@@ -459,7 +469,7 @@ static int JsonTlsLogger(ThreadVars *tv, void *thread_data, const Packet *p,
     }
     /* log extended */
     else if (tls_ctx->flags & LOG_TLS_EXTENDED) {
-        JsonTlsLogJSONExtended(js, ssl_state);
+        JsonTlsLogJSONExtendedAux(ssl_state, js);
     }
     /* log basic */
     else {
diff --git a/src/output-json-tls.h b/src/output-json-tls.h
index 737e6233ef10..42f706b91d3f 100644
--- a/src/output-json-tls.h
+++ b/src/output-json-tls.h
@@ -29,6 +29,6 @@ void JsonTlsLogRegister(void);
 #include "app-layer-ssl.h"
 
 void JsonTlsLogJSONBasic(JsonBuilder *js, SSLState *ssl_state);
-void JsonTlsLogJSONExtended(JsonBuilder *js, SSLState *ssl_state);
+bool JsonTlsLogJSONExtended(void *vtx, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_TLS_H__ */
diff --git a/src/output.c b/src/output.c
index c13ab4862eda..031831966e17 100644
--- a/src/output.c
+++ b/src/output.c
@@ -67,6 +67,8 @@
 #include "log-stats.h"
 #include "output-json-nfs.h"
 #include "output-json-ftp.h"
+// for misplaced EveFTPDataAddMetadata
+#include "app-layer-ftp.h"
 #include "output-json-tftp.h"
 #include "output-json-smb.h"
 #include "output-json-ike.h"
@@ -1126,3 +1128,53 @@ void OutputRegisterLoggers(void)
     /* BitTorrent DHT JSON logger */
     JsonBitTorrentDHTLogRegister();
 }
+
+static EveJsonSimpleAppLayerLogger simple_json_applayer_loggers[ALPROTO_MAX] = {
+    { ALPROTO_UNKNOWN, NULL },
+    { ALPROTO_HTTP1, NULL }, // special: uses some options flags
+    { ALPROTO_FTP, NULL },   // TODO missing
+    { ALPROTO_SMTP, NULL },  // special: uses state
+    { ALPROTO_TLS, JsonTlsLogJSONExtended },
+    { ALPROTO_SSH, rs_ssh_log_json },
+    { ALPROTO_IMAP, NULL },   // protocol detection only
+    { ALPROTO_JABBER, NULL }, // no parser, no logging
+    { ALPROTO_SMB, NULL },    // special: uses state
+    { ALPROTO_DCERPC, NULL }, // TODO missing
+    { ALPROTO_IRC, NULL },    // no parser, no logging
+    { ALPROTO_DNS, AlertJsonDns },
+    { ALPROTO_MODBUS, (EveJsonSimpleTxLogFunc)rs_modbus_to_json },
+    { ALPROTO_ENIP, NULL }, // no logging
+    { ALPROTO_DNP3, AlertJsonDnp3 },
+    { ALPROTO_NFS, NULL }, // special: uses state
+    { ALPROTO_NTP, NULL }, // no logging
+    { ALPROTO_FTPDATA, EveFTPDataAddMetadata },
+    { ALPROTO_TFTP, NULL }, // TODO missing
+    { ALPROTO_IKE, NULL },  // special: uses state
+    { ALPROTO_KRB5, NULL }, // TODO missing
+    { ALPROTO_QUIC, rs_quic_to_json },
+    { ALPROTO_DHCP, NULL }, // TODO missing
+    { ALPROTO_SNMP, (EveJsonSimpleTxLogFunc)rs_snmp_log_json_response },
+    { ALPROTO_SIP, (EveJsonSimpleTxLogFunc)rs_sip_log_json },
+    { ALPROTO_RFB, rs_rfb_logger_log },
+    { ALPROTO_MQTT, JsonMQTTAddMetadata },
+    { ALPROTO_PGSQL, NULL },  // TODO missing
+    { ALPROTO_TELNET, NULL }, // no logging
+    { ALPROTO_TEMPLATE, rs_template_logger_log },
+    { ALPROTO_RDP, (EveJsonSimpleTxLogFunc)rs_rdp_to_json },
+    { ALPROTO_HTTP2, rs_http2_log_json },
+    { ALPROTO_BITTORRENT_DHT, rs_bittorrent_dht_logger_log },
+    { ALPROTO_HTTP, NULL }, // signature protocol, not for app-layer logging
+    { ALPROTO_FAILED, NULL },
+#ifdef UNITTESTS
+    { ALPROTO_TEST, NULL },
+#endif /* UNITESTS */
+};
+
+EveJsonSimpleAppLayerLogger *SCEveJsonSimpleGetLogger(AppProto alproto)
+{
+    if (alproto < ALPROTO_MAX) {
+        BUG_ON(simple_json_applayer_loggers[alproto].proto != alproto);
+        return &simple_json_applayer_loggers[alproto];
+    }
+    return NULL;
+}
diff --git a/src/output.h b/src/output.h
index 5c2d7bc90e62..815b2f20ed73 100644
--- a/src/output.h
+++ b/src/output.h
@@ -208,4 +208,13 @@ void OutputLoggerExitPrintStats(ThreadVars *, void *);
 void OutputSetupActiveLoggers(void);
 void OutputClearActiveLoggers(void);
 
+typedef bool (*EveJsonSimpleTxLogFunc)(void *, struct JsonBuilder *);
+
+typedef struct EveJsonSimpleAppLayerLogger {
+    AppProto proto;
+    EveJsonSimpleTxLogFunc LogTx;
+} EveJsonSimpleAppLayerLogger;
+
+EveJsonSimpleAppLayerLogger *SCEveJsonSimpleGetLogger(AppProto alproto);
+
 #endif /* ! __OUTPUT_H__ */

From 3b1b163ee6b0a4c382af761be90c0d53f0654402 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 16 Nov 2023 09:49:58 +0100
Subject: [PATCH 154/462] output/ftp: have ftp properties in alerts

Ticket: 6500
---
 src/output-json-ftp.c | 9 +++++----
 src/output-json-ftp.h | 1 +
 src/output.c          | 4 ++--
 3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/src/output-json-ftp.c b/src/output-json-ftp.c
index 9fb8b8c622db..34422f72f4af 100644
--- a/src/output-json-ftp.c
+++ b/src/output-json-ftp.c
@@ -46,15 +46,16 @@
 #include "app-layer-ftp.h"
 #include "output-json-ftp.h"
 
-static void EveFTPLogCommand(FTPTransaction *tx, JsonBuilder *jb)
+bool EveFTPLogCommand(void *vtx, JsonBuilder *jb)
 {
+    FTPTransaction *tx = vtx;
     /* Preallocate array objects to simplify failure case */
     JsonBuilder *js_resplist = NULL;
     if (!TAILQ_EMPTY(&tx->response_list)) {
         js_resplist = jb_new_array();
 
         if (unlikely(js_resplist == NULL)) {
-            return;
+            return false;
         }
     }
     jb_open_object(jb, "ftp");
@@ -151,6 +152,7 @@ static void EveFTPLogCommand(FTPTransaction *tx, JsonBuilder *jb)
         JB_SET_FALSE(jb, "reply_truncated");
     }
     jb_close(jb);
+    return true;
 }
 
 
@@ -166,7 +168,6 @@ static int JsonFTPLogger(ThreadVars *tv, void *thread_data,
     } else {
         event_type = "ftp";
     }
-    FTPTransaction *tx = vtx;
 
     JsonBuilder *jb =
             CreateEveHeaderWithTxId(p, LOG_DIR_FLOW, event_type, NULL, tx_id, thread->ctx);
@@ -176,7 +177,7 @@ static int JsonFTPLogger(ThreadVars *tv, void *thread_data,
                 goto fail;
             }
         } else {
-            EveFTPLogCommand(tx, jb);
+            EveFTPLogCommand(vtx, jb);
         }
 
         OutputJsonBuilderBuffer(jb, thread);
diff --git a/src/output-json-ftp.h b/src/output-json-ftp.h
index acba5539e1c6..704defd9585c 100644
--- a/src/output-json-ftp.h
+++ b/src/output-json-ftp.h
@@ -25,5 +25,6 @@
 #define __OUTPUT_JSON_FTP_H__
 
 void JsonFTPLogRegister(void);
+bool EveFTPLogCommand(void *vtx, JsonBuilder *js);
 
 #endif /* __OUTPUT_JSON_FTP_H__ */
diff --git a/src/output.c b/src/output.c
index 031831966e17..d57622ff39c9 100644
--- a/src/output.c
+++ b/src/output.c
@@ -1132,8 +1132,8 @@ void OutputRegisterLoggers(void)
 static EveJsonSimpleAppLayerLogger simple_json_applayer_loggers[ALPROTO_MAX] = {
     { ALPROTO_UNKNOWN, NULL },
     { ALPROTO_HTTP1, NULL }, // special: uses some options flags
-    { ALPROTO_FTP, NULL },   // TODO missing
-    { ALPROTO_SMTP, NULL },  // special: uses state
+    { ALPROTO_FTP, EveFTPLogCommand },
+    { ALPROTO_SMTP, NULL }, // special: uses state
     { ALPROTO_TLS, JsonTlsLogJSONExtended },
     { ALPROTO_SSH, rs_ssh_log_json },
     { ALPROTO_IMAP, NULL },   // protocol detection only

From 8a09bff0aa1e87a3c8e9e156bd36e31d2e7306ce Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 16 Nov 2023 09:52:12 +0100
Subject: [PATCH 155/462] output/tftp: have tftp properties in alerts

Ticket: 6501
---
 rust/src/tftp/log.rs   | 2 ++
 src/output-json-tftp.c | 2 --
 src/output.c           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/rust/src/tftp/log.rs b/rust/src/tftp/log.rs
index b4837036a156..f6e63531a07e 100644
--- a/rust/src/tftp/log.rs
+++ b/rust/src/tftp/log.rs
@@ -24,6 +24,7 @@ fn tftp_log_request(tx: &mut TFTPTransaction,
                     jb: &mut JsonBuilder)
                     -> Result<(), JsonError>
 {
+    jb.open_object("tftp")?;
     match tx.opcode {
         1 => jb.set_string("packet", "read")?,
         2 => jb.set_string("packet", "write")?,
@@ -31,6 +32,7 @@ fn tftp_log_request(tx: &mut TFTPTransaction,
     };
     jb.set_string("file", tx.filename.as_str())?;
     jb.set_string("mode", tx.mode.as_str())?;
+    jb.close()?;
     Ok(())
 }
 
diff --git a/src/output-json-tftp.c b/src/output-json-tftp.c
index 4fff67a8b696..a0bc9ee1809e 100644
--- a/src/output-json-tftp.c
+++ b/src/output-json-tftp.c
@@ -58,11 +58,9 @@ static int JsonTFTPLogger(ThreadVars *tv, void *thread_data,
         return TM_ECODE_FAILED;
     }
 
-    jb_open_object(jb, "tftp");
     if (unlikely(!rs_tftp_log_json_request(tx, jb))) {
         goto error;
     }
-    jb_close(jb);
 
     OutputJsonBuilderBuffer(jb, thread);
 
diff --git a/src/output.c b/src/output.c
index d57622ff39c9..5aa341d2cbd4 100644
--- a/src/output.c
+++ b/src/output.c
@@ -1148,7 +1148,7 @@ static EveJsonSimpleAppLayerLogger simple_json_applayer_loggers[ALPROTO_MAX] = {
     { ALPROTO_NFS, NULL }, // special: uses state
     { ALPROTO_NTP, NULL }, // no logging
     { ALPROTO_FTPDATA, EveFTPDataAddMetadata },
-    { ALPROTO_TFTP, NULL }, // TODO missing
+    { ALPROTO_TFTP, (EveJsonSimpleTxLogFunc)rs_tftp_log_json_request },
     { ALPROTO_IKE, NULL },  // special: uses state
     { ALPROTO_KRB5, NULL }, // TODO missing
     { ALPROTO_QUIC, rs_quic_to_json },

From e38b9de6a2fe8ca53f19585f84b24ae31ee720c9 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 16 Nov 2023 09:55:03 +0100
Subject: [PATCH 156/462] output/krb5: have krb5 properties in alerts

Ticket: 5977
---
 rust/src/krb/log.rs    | 4 +++-
 src/output-json-krb5.c | 4 +---
 src/output.c           | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/rust/src/krb/log.rs b/rust/src/krb/log.rs
index 7cb952581bc7..58c0d64b4893 100644
--- a/rust/src/krb/log.rs
+++ b/rust/src/krb/log.rs
@@ -22,6 +22,7 @@ use crate::krb::krb5::{KRB5Transaction,test_weak_encryption};
 
 fn krb5_log_response(jsb: &mut JsonBuilder, tx: &mut KRB5Transaction) -> Result<(), JsonError>
 {
+    jsb.open_object("krb5")?;
     match tx.error_code {
         Some(c) => {
             jsb.set_string("msg_type", &format!("{:?}", tx.msg_type))?;
@@ -63,12 +64,13 @@ fn krb5_log_response(jsb: &mut JsonBuilder, tx: &mut KRB5Transaction) -> Result<
         jsb.set_string("ticket_encryption", &refs)?;
         jsb.set_bool("ticket_weak_encryption", test_weak_encryption(x))?;
     }
+    jsb.close()?;
 
     return Ok(());
 }
 
 #[no_mangle]
-pub extern "C" fn rs_krb5_log_json_response(jsb: &mut JsonBuilder, tx: &mut KRB5Transaction) -> bool
+pub extern "C" fn rs_krb5_log_json_response(tx: &mut KRB5Transaction, jsb: &mut JsonBuilder) -> bool
 {
     krb5_log_response(jsb, tx).is_ok()
 }
diff --git a/src/output-json-krb5.c b/src/output-json-krb5.c
index 5e6fbad5ecd1..9fc45c5d3c53 100644
--- a/src/output-json-krb5.c
+++ b/src/output-json-krb5.c
@@ -59,11 +59,9 @@ static int JsonKRB5Logger(ThreadVars *tv, void *thread_data,
         return TM_ECODE_FAILED;
     }
 
-    jb_open_object(jb, "krb5");
-    if (!rs_krb5_log_json_response(jb, krb5tx)) {
+    if (!rs_krb5_log_json_response(krb5tx, jb)) {
         goto error;
     }
-    jb_close(jb);
 
     OutputJsonBuilderBuffer(jb, thread);
 
diff --git a/src/output.c b/src/output.c
index 5aa341d2cbd4..149dda58c284 100644
--- a/src/output.c
+++ b/src/output.c
@@ -1149,8 +1149,8 @@ static EveJsonSimpleAppLayerLogger simple_json_applayer_loggers[ALPROTO_MAX] = {
     { ALPROTO_NTP, NULL }, // no logging
     { ALPROTO_FTPDATA, EveFTPDataAddMetadata },
     { ALPROTO_TFTP, (EveJsonSimpleTxLogFunc)rs_tftp_log_json_request },
-    { ALPROTO_IKE, NULL },  // special: uses state
-    { ALPROTO_KRB5, NULL }, // TODO missing
+    { ALPROTO_IKE, NULL }, // special: uses state
+    { ALPROTO_KRB5, (EveJsonSimpleTxLogFunc)rs_krb5_log_json_response },
     { ALPROTO_QUIC, rs_quic_to_json },
     { ALPROTO_DHCP, NULL }, // TODO missing
     { ALPROTO_SNMP, (EveJsonSimpleTxLogFunc)rs_snmp_log_json_response },

From c272a646c5ae739d18901776cc5a940afd3d3d38 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 7 Sep 2023 11:00:42 +0200
Subject: [PATCH 157/462] detect: SigMatchAppendSMToList can fail

Ticket: #6104

And failures should be handled to say that the rule failed to load

Reverts the fix by 299ee6ed5561f01575150b436d5db31485dab146
that was simple, but not complete (memory leak),
to have this bigger API change which simplifies code.
---
 src/detect-app-layer-event.c                 |  21 ++--
 src/detect-app-layer-protocol.c              |  11 +-
 src/detect-asn1.c                            |  11 +-
 src/detect-base64-decode.c                   |   8 +-
 src/detect-bsize.c                           |   9 +-
 src/detect-bypass.c                          |   9 +-
 src/detect-byte-extract.c                    |  11 +-
 src/detect-bytejump.c                        |   8 +-
 src/detect-bytemath.c                        |   8 +-
 src/detect-bytetest.c                        |   8 +-
 src/detect-cipservice.c                      |  26 +---
 src/detect-config.c                          |  13 +-
 src/detect-content.c                         |   7 +-
 src/detect-csum.c                            | 126 +++++++------------
 src/detect-datarep.c                         |  12 +-
 src/detect-dataset.c                         |  12 +-
 src/detect-dce-iface.c                       |   9 +-
 src/detect-dce-opnum.c                       |   9 +-
 src/detect-detection-filter.c                |  13 +-
 src/detect-dhcp-leasetime.c                  |  11 +-
 src/detect-dhcp-rebinding-time.c             |  11 +-
 src/detect-dhcp-renewal-time.c               |  11 +-
 src/detect-dnp3.c                            |  34 +----
 src/detect-dns-opcode.c                      |   8 +-
 src/detect-dsize.c                           |  12 +-
 src/detect-engine-event.c                    |   9 +-
 src/detect-file-hash-common.c                |  13 +-
 src/detect-filesize.c                        |  13 +-
 src/detect-filestore.c                       |  28 ++---
 src/detect-flow-age.c                        |   9 +-
 src/detect-flow-pkts.c                       |  36 ++----
 src/detect-flow.c                            |  20 ++-
 src/detect-flowbits.c                        |  19 ++-
 src/detect-flowint.c                         |  19 ++-
 src/detect-flowvar.c                         |  25 ++--
 src/detect-fragbits.c                        |  11 +-
 src/detect-fragoffset.c                      |  16 +--
 src/detect-ftpbounce.c                       |  14 +--
 src/detect-ftpdata.c                         |   8 +-
 src/detect-geoip.c                           |  14 +--
 src/detect-hostbits.c                        |  19 ++-
 src/detect-http2.c                           |  54 ++------
 src/detect-icmp-id.c                         |  16 +--
 src/detect-icmp-seq.c                        |  16 +--
 src/detect-icmpv6-mtu.c                      |   9 +-
 src/detect-icode.c                           |  13 +-
 src/detect-id.c                              |  10 +-
 src/detect-ike-chosen-sa.c                   |  11 +-
 src/detect-ike-exch-type.c                   |  11 +-
 src/detect-ike-key-exchange-payload-length.c |  12 +-
 src/detect-ike-nonce-payload-length.c        |  12 +-
 src/detect-ipopts.c                          |  15 +--
 src/detect-ipproto.c                         |  10 +-
 src/detect-iprep.c                           |  14 +--
 src/detect-isdataat.c                        |   8 +-
 src/detect-itype.c                           |  16 +--
 src/detect-krb5-errcode.c                    |  13 +-
 src/detect-krb5-msgtype.c                    |  13 +-
 src/detect-krb5-ticket-encryption.c          |  13 +-
 src/detect-lua.c                             |  13 +-
 src/detect-mark.c                            |  14 +--
 src/detect-modbus.c                          |  13 +-
 src/detect-mqtt-connack-sessionpresent.c     |  13 +-
 src/detect-mqtt-connect-flags.c              |  13 +-
 src/detect-mqtt-flags.c                      |  13 +-
 src/detect-mqtt-protocol-version.c           |  13 +-
 src/detect-mqtt-qos.c                        |  13 +-
 src/detect-mqtt-reason-code.c                |  13 +-
 src/detect-mqtt-type.c                       |  13 +-
 src/detect-nfs-procedure.c                   |  12 +-
 src/detect-nfs-version.c                     |  11 +-
 src/detect-parse.c                           |  23 ++--
 src/detect-parse.h                           |   2 +-
 src/detect-pcre.c                            |   9 +-
 src/detect-pktvar.c                          |   9 +-
 src/detect-replace.c                         |   8 +-
 src/detect-rfb-secresult.c                   |  15 +--
 src/detect-rfb-sectype.c                     |  11 +-
 src/detect-rpc.c                             |  15 +--
 src/detect-sameip.c                          |  14 +--
 src/detect-snmp-pdu_type.c                   |  12 +-
 src/detect-snmp-version.c                    |  12 +-
 src/detect-ssh-proto-version.c               |  14 +--
 src/detect-ssh-software-version.c            |  14 +--
 src/detect-ssl-state.c                       |  13 +-
 src/detect-ssl-version.c                     |  14 +--
 src/detect-stream_size.c                     |   9 +-
 src/detect-tag.c                             |  11 +-
 src/detect-tcp-ack.c                         |  15 +--
 src/detect-tcp-flags.c                       |  15 +--
 src/detect-tcp-seq.c                         |  15 +--
 src/detect-tcp-window.c                      |  16 +--
 src/detect-tcpmss.c                          |   9 +-
 src/detect-template.c                        |   9 +-
 src/detect-template2.c                       |   9 +-
 src/detect-threshold.c                       |  15 +--
 src/detect-tls-cert-validity.c               |  43 ++-----
 src/detect-tls-certs.c                       |   8 +-
 src/detect-tls-version.c                     |  14 +--
 src/detect-tls.c                             |  39 ++----
 src/detect-tos.c                             |   9 +-
 src/detect-ttl.c                             |   9 +-
 src/detect-urilen.c                          |  21 ++--
 src/detect-xbits.c                           |  17 ++-
 src/detect.h                                 |   1 -
 src/util-threshold-config.c                  |  84 ++++---------
 106 files changed, 495 insertions(+), 1169 deletions(-)

diff --git a/src/detect-app-layer-event.c b/src/detect-app-layer-event.c
index bf306d363d39..d5eb0117218e 100644
--- a/src/detect-app-layer-event.c
+++ b/src/detect-app-layer-event.c
@@ -278,20 +278,19 @@ static int DetectAppLayerEventSetup(DetectEngineCtx *de_ctx, Signature *s, const
     }
     SCLogDebug("data->event_id %u", data->event_id);
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_APP_LAYER_EVENT;
-    sm->ctx = (SigMatchCtx *)data;
-
     if (event_type == APP_LAYER_EVENT_TYPE_PACKET) {
-        SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+        if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_APP_LAYER_EVENT, (SigMatchCtx *)data,
+                    DETECT_SM_LIST_MATCH) == NULL) {
+            goto error;
+        }
     } else {
         if (DetectSignatureSetAppProto(s, data->alproto) != 0)
             goto error;
 
-        SigMatchAppendSMToList(s, sm, g_applayer_events_list_id);
+        if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_APP_LAYER_EVENT, (SigMatchCtx *)data,
+                    g_applayer_events_list_id) == NULL) {
+            goto error;
+        }
         s->flags |= SIG_FLAG_APPLAYER;
     }
 
@@ -301,10 +300,6 @@ static int DetectAppLayerEventSetup(DetectEngineCtx *de_ctx, Signature *s, const
     if (data) {
         DetectAppLayerEventFree(de_ctx, data);
     }
-    if (sm) {
-        sm->ctx = NULL;
-        SigMatchFree(de_ctx, sm);
-    }
     return -1;
 }
 
diff --git a/src/detect-app-layer-protocol.c b/src/detect-app-layer-protocol.c
index 26a5ce6235aa..182f6d0faeb3 100644
--- a/src/detect-app-layer-protocol.c
+++ b/src/detect-app-layer-protocol.c
@@ -141,7 +141,6 @@ static int DetectAppLayerProtocolSetup(DetectEngineCtx *de_ctx,
         Signature *s, const char *arg)
 {
     DetectAppLayerProtocolData *data = NULL;
-    SigMatch *sm = NULL;
 
     if (s->alproto != ALPROTO_UNKNOWN) {
         SCLogError("Either we already "
@@ -169,14 +168,10 @@ static int DetectAppLayerProtocolSetup(DetectEngineCtx *de_ctx,
         }
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_APP_LAYER_PROTOCOL, (SigMatchCtx *)data,
+                DETECT_SM_LIST_MATCH) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_APP_LAYER_PROTOCOL;
-    sm->ctx = (void *)data;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     return 0;
 
 error:
diff --git a/src/detect-asn1.c b/src/detect-asn1.c
index e255e057afe5..5b3a3a2229b2 100644
--- a/src/detect-asn1.c
+++ b/src/detect-asn1.c
@@ -127,19 +127,12 @@ static int DetectAsn1Setup(DetectEngineCtx *de_ctx, Signature *s, const char *as
     if (ad == NULL)
         return -1;
 
-    /* Okay so far so good, lets get this into a SigMatch
-     * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ASN1, (SigMatchCtx *)ad, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         DetectAsn1Free(de_ctx, ad);
         return -1;
     }
 
-    sm->type = DETECT_ASN1;
-    sm->ctx = (SigMatchCtx *)ad;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
-
     return 0;
 }
 
diff --git a/src/detect-base64-decode.c b/src/detect-base64-decode.c
index 25fdf10e70c4..2794509a430f 100644
--- a/src/detect-base64-decode.c
+++ b/src/detect-base64-decode.c
@@ -191,7 +191,6 @@ static int DetectBase64DecodeSetup(DetectEngineCtx *de_ctx, Signature *s,
     uint8_t relative = 0;
     DetectBase64Decode *data = NULL;
     int sm_list;
-    SigMatch *sm = NULL;
     SigMatch *pm = NULL;
 
     if (str != NULL) {
@@ -226,13 +225,10 @@ static int DetectBase64DecodeSetup(DetectEngineCtx *de_ctx, Signature *s,
         }
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BASE64_DECODE, (SigMatchCtx *)data, sm_list) ==
+            NULL) {
         goto error;
     }
-    sm->type = DETECT_BASE64_DECODE;
-    sm->ctx = (SigMatchCtx *)data;
-    SigMatchAppendSMToList(s, sm, sm_list);
 
     if (!data->bytes) {
         data->bytes = BASE64_DECODE_MAX;
diff --git a/src/detect-bsize.c b/src/detect-bsize.c
index 3b3efe87b7ff..f69e20851839 100644
--- a/src/detect-bsize.c
+++ b/src/detect-bsize.c
@@ -199,7 +199,6 @@ static int SigParseGetMaxBsize(DetectU64Data *bsz)
 static int DetectBsizeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *sizestr)
 {
     SCEnter();
-    SigMatch *sm = NULL;
 
     if (DetectBufferGetActiveList(de_ctx, s) == -1)
         SCReturnInt(-1);
@@ -212,13 +211,9 @@ static int DetectBsizeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *
     if (bsz == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BSIZE, (SigMatchCtx *)bsz, list) == NULL) {
         goto error;
-    sm->type = DETECT_BSIZE;
-    sm->ctx = (void *)bsz;
-
-    SigMatchAppendSMToList(s, sm, list);
+    }
 
     SCReturnInt(0);
 
diff --git a/src/detect-bypass.c b/src/detect-bypass.c
index a0eb534b84db..51c5d2835160 100644
--- a/src/detect-bypass.c
+++ b/src/detect-bypass.c
@@ -69,7 +69,6 @@ void DetectBypassRegister(void)
 
 static int DetectBypassSetup(DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
-    SigMatch *sm = NULL;
 
     if (s->flags & SIG_FLAG_FILESTORE) {
         SCLogError("bypass can't work with filestore keyword");
@@ -77,13 +76,9 @@ static int DetectBypassSetup(DetectEngineCtx *de_ctx, Signature *s, const char *
     }
     s->flags |= SIG_FLAG_BYPASS;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BYPASS, NULL, DETECT_SM_LIST_POSTMATCH) == NULL) {
         return -1;
-
-    sm->type = DETECT_BYPASS;
-    sm->ctx = NULL;
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    }
 
     return 0;
 }
diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index ec9b27fc6406..5c69e4442df7 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -531,7 +531,6 @@ static inline DetectByteExtractData *DetectByteExtractParse(DetectEngineCtx *de_
  */
 static int DetectByteExtractSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg)
 {
-    SigMatch *sm = NULL;
     SigMatch *prev_pm = NULL;
     DetectByteExtractData *data = NULL;
     int ret = -1;
@@ -609,14 +608,10 @@ static int DetectByteExtractSetup(DetectEngineCtx *de_ctx, Signature *s, const c
     if (data->local_id > de_ctx->byte_extract_max_local_id)
         de_ctx->byte_extract_max_local_id = data->local_id;
 
-
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BYTE_EXTRACT, (SigMatchCtx *)data, sm_list) ==
+            NULL) {
         goto error;
-    sm->type = DETECT_BYTE_EXTRACT;
-    sm->ctx = (void *)data;
-    SigMatchAppendSMToList(s, sm, sm_list);
-
+    }
 
     if (!(data->flags & DETECT_BYTE_EXTRACT_FLAG_RELATIVE))
         goto okay;
diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index b0b034774636..37c01ed8c30b 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -469,7 +469,6 @@ static DetectBytejumpData *DetectBytejumpParse(
 
 static int DetectBytejumpSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
-    SigMatch *sm = NULL;
     SigMatch *prev_pm = NULL;
     DetectBytejumpData *data = NULL;
     char *offset = NULL;
@@ -569,12 +568,9 @@ static int DetectBytejumpSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         offset = NULL;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BYTEJUMP, (SigMatchCtx *)data, sm_list) == NULL) {
         goto error;
-    sm->type = DETECT_BYTEJUMP;
-    sm->ctx = (SigMatchCtx *)data;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     if (!(data->flags & DETECT_BYTEJUMP_RELATIVE))
         goto okay;
diff --git a/src/detect-bytemath.c b/src/detect-bytemath.c
index 9064b06fcf69..a2880216cffa 100644
--- a/src/detect-bytemath.c
+++ b/src/detect-bytemath.c
@@ -279,7 +279,6 @@ static DetectByteMathData *DetectByteMathParse(
  */
 static int DetectByteMathSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg)
 {
-    SigMatch *sm = NULL;
     SigMatch *prev_pm = NULL;
     DetectByteMathData *data;
     char *rvalue = NULL;
@@ -393,12 +392,9 @@ static int DetectByteMathSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         de_ctx->byte_extract_max_local_id = data->local_id;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BYTEMATH, (SigMatchCtx *)data, sm_list) == NULL) {
         goto error;
-    sm->type = DETECT_BYTEMATH;
-    sm->ctx = (void *)data;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     if (!(data->flags & DETECT_BYTEMATH_FLAG_RELATIVE))
         goto okay;
diff --git a/src/detect-bytetest.c b/src/detect-bytetest.c
index 27070ffa36ff..481eb51136db 100644
--- a/src/detect-bytetest.c
+++ b/src/detect-bytetest.c
@@ -582,7 +582,6 @@ static DetectBytetestData *DetectBytetestParse(
 
 static int DetectBytetestSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
-    SigMatch *sm = NULL;
     SigMatch *prev_pm = NULL;
     char *value = NULL;
     char *offset = NULL;
@@ -696,12 +695,9 @@ static int DetectBytetestSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         nbytes = NULL;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_BYTETEST, (SigMatchCtx *)data, sm_list) == NULL) {
         goto error;
-    sm->type = DETECT_BYTETEST;
-    sm->ctx = (SigMatchCtx *)data;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     if (!(data->flags & DETECT_BYTETEST_RELATIVE))
         goto okay;
diff --git a/src/detect-cipservice.c b/src/detect-cipservice.c
index 00b9a75ca099..494e1e17520f 100644
--- a/src/detect-cipservice.c
+++ b/src/detect-cipservice.c
@@ -208,7 +208,6 @@ static int DetectCipServiceSetup(DetectEngineCtx *de_ctx, Signature *s,
     SCEnter();
 
     DetectCipServiceData *cipserviced = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_ENIP) != 0)
         return -1;
@@ -217,21 +216,15 @@ static int DetectCipServiceSetup(DetectEngineCtx *de_ctx, Signature *s,
     if (cipserviced == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_CIPSERVICE, (SigMatchCtx *)cipserviced,
+                g_cip_buffer_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_CIPSERVICE;
-    sm->ctx = (void *) cipserviced;
-
-    SigMatchAppendSMToList(s, sm, g_cip_buffer_id);
+    }
     SCReturnInt(0);
 
 error:
     if (cipserviced != NULL)
         DetectCipServiceFree(de_ctx, cipserviced);
-    if (sm != NULL)
-        SCFree(sm);
     SCReturnInt(-1);
 }
 
@@ -378,7 +371,6 @@ static int DetectEnipCommandSetup(DetectEngineCtx *de_ctx, Signature *s,
         const char *rulestr)
 {
     DetectEnipCommandData *enipcmdd = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_ENIP) != 0)
         return -1;
@@ -387,21 +379,15 @@ static int DetectEnipCommandSetup(DetectEngineCtx *de_ctx, Signature *s,
     if (enipcmdd == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_ENIPCOMMAND, (SigMatchCtx *)enipcmdd, g_enip_buffer_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_ENIPCOMMAND;
-    sm->ctx = (void *) enipcmdd;
-
-    SigMatchAppendSMToList(s, sm, g_enip_buffer_id);
+    }
     SCReturnInt(0);
 
 error:
     if (enipcmdd != NULL)
         DetectEnipCommandFree(de_ctx, enipcmdd);
-    if (sm != NULL)
-        SCFree(sm);
     SCReturnInt(-1);
 }
 
diff --git a/src/detect-config.c b/src/detect-config.c
index ae215dd2161c..7ad8c88dca68 100644
--- a/src/detect-config.c
+++ b/src/detect-config.c
@@ -170,7 +170,6 @@ static int DetectConfigSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     SCEnter();
 
     DetectConfigData *fd = NULL;
-    SigMatch *sm = NULL;
     int res = 0;
     size_t pcre2len;
 #if 0
@@ -182,10 +181,6 @@ static int DetectConfigSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     }
 #endif
     pcre2_match_data *match = NULL;
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-    sm->type = DETECT_CONFIG;
 
     if (str == NULL || strlen(str) == 0) {
         SCLogError("config keywords need arguments");
@@ -297,8 +292,10 @@ static int DetectConfigSetup (DetectEngineCtx *de_ctx, Signature *s, const char
         s->flags |= SIG_FLAG_APPLAYER;
     }
 
-    sm->ctx = (SigMatchCtx*)fd;
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_CONFIG, (SigMatchCtx *)fd, DETECT_SM_LIST_POSTMATCH) == NULL) {
+        goto error;
+    }
 
     pcre2_match_data_free(match);
     return 0;
@@ -307,8 +304,6 @@ static int DetectConfigSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     if (match) {
         pcre2_match_data_free(match);
     }
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-content.c b/src/detect-content.c
index 5bbe9e9b3cae..5f34ffd13df1 100644
--- a/src/detect-content.c
+++ b/src/detect-content.c
@@ -353,12 +353,9 @@ int DetectContentSetup(DetectEngineCtx *de_ctx, Signature *s, const char *conten
         }
     }
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_CONTENT, (SigMatchCtx *)cd, sm_list) == NULL) {
         goto error;
-    sm->ctx = (void *)cd;
-    sm->type = DETECT_CONTENT;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     return 0;
 
diff --git a/src/detect-csum.c b/src/detect-csum.c
index 6129635a48f2..8947725f4d26 100644
--- a/src/detect-csum.c
+++ b/src/detect-csum.c
@@ -274,16 +274,9 @@ static int DetectIPV4CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectIPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_IPV4_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -291,15 +284,16 @@ static int DetectIPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_IPV4_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectIPV4CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectIPV4CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -371,16 +365,9 @@ static int DetectTCPV4CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectTCPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_TCPV4_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -388,15 +375,16 @@ static int DetectTCPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_TCPV4_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectTCPV4CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectTCPV4CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -468,16 +456,9 @@ static int DetectTCPV6CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectTCPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_TCPV6_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -485,15 +466,16 @@ static int DetectTCPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_TCPV6_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectTCPV6CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectTCPV6CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -565,16 +547,9 @@ static int DetectUDPV4CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectUDPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_UDPV4_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -582,15 +557,16 @@ static int DetectUDPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_UDPV4_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectUDPV4CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectUDPV4CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -662,16 +638,9 @@ static int DetectUDPV6CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectUDPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_UDPV6_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -679,15 +648,16 @@ static int DetectUDPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (void *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_UDPV6_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectUDPV6CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectUDPV6CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -757,16 +727,9 @@ static int DetectICMPV4CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectICMPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_ICMPV4_CSUM;
-
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
     memset(cd, 0, sizeof(DetectCsumData));
@@ -774,15 +737,16 @@ static int DetectICMPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const ch
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_ICMPV4_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectICMPV4CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectICMPV4CsumFree(de_ctx, cd);
 
     return -1;
 }
@@ -857,13 +821,6 @@ static int DetectICMPV6CsumMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectICMPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char *csum_str)
 {
     DetectCsumData *cd = NULL;
-    SigMatch *sm = NULL;
-
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_ICMPV6_CSUM;
 
     if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
         goto error;
@@ -872,15 +829,16 @@ static int DetectICMPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const ch
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
 
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_ICMPV6_CSUM, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (cd != NULL) DetectICMPV6CsumFree(de_ctx, cd);
-    if (sm != NULL) SCFree(sm);
+    if (cd != NULL)
+        DetectICMPV6CsumFree(de_ctx, cd);
 
     return -1;
 }
diff --git a/src/detect-datarep.c b/src/detect-datarep.c
index c9cc1795938a..5b959b4023a3 100644
--- a/src/detect-datarep.c
+++ b/src/detect-datarep.c
@@ -292,7 +292,6 @@ static int SetupLoadPath(const DetectEngineCtx *de_ctx,
 
 static int DetectDatarepSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
-    SigMatch *sm = NULL;
     char cmd_str[16] = "", name[64] = "";
     enum DatasetTypes type = DATASET_TYPE_NOTSET;
     char load[PATH_MAX] = "";
@@ -352,20 +351,15 @@ static int DetectDatarepSetup (DetectEngineCtx *de_ctx, Signature *s, const char
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
 
-    sm->type = DETECT_DATAREP;
-    sm->ctx = (SigMatchCtx *)cd;
-    SigMatchAppendSMToList(s, sm, list);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_DATAREP, (SigMatchCtx *)cd, list) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (cd != NULL)
         SCFree(cd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-dataset.c b/src/detect-dataset.c
index 3d2964605e6e..f6d0d844e2e5 100644
--- a/src/detect-dataset.c
+++ b/src/detect-dataset.c
@@ -344,7 +344,6 @@ static int SetupSavePath(const DetectEngineCtx *de_ctx,
 int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectDatasetData *cd = NULL;
-    SigMatch *sm = NULL;
     uint8_t cmd = 0;
     uint64_t memcap = 0;
     uint32_t hashsize = 0;
@@ -424,20 +423,15 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
 
-    sm->type = DETECT_DATASET;
-    sm->ctx = (SigMatchCtx *)cd;
-    SigMatchAppendSMToList(s, sm, list);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_DATASET, (SigMatchCtx *)cd, list) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (cd != NULL)
         SCFree(cd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-dce-iface.c b/src/detect-dce-iface.c
index 844e7bc1499a..a85248e0afc7 100644
--- a/src/detect-dce-iface.c
+++ b/src/detect-dce-iface.c
@@ -154,15 +154,10 @@ static int DetectDceIfaceSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         return -1;
     }
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_DCE_IFACE, did, g_dce_generic_list_id) == NULL) {
+        DetectDceIfaceFree(de_ctx, did);
         return -1;
     }
-
-    sm->type = DETECT_DCE_IFACE;
-    sm->ctx = did;
-
-    SigMatchAppendSMToList(s, sm, g_dce_generic_list_id);
     return 0;
 }
 
diff --git a/src/detect-dce-opnum.c b/src/detect-dce-opnum.c
index d10b175490e3..782d33666655 100644
--- a/src/detect-dce-opnum.c
+++ b/src/detect-dce-opnum.c
@@ -142,16 +142,11 @@ static int DetectDceOpnumSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         return -1;
     }
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_DCE_OPNUM, (SigMatchCtx *)dod, g_dce_generic_list_id) == NULL) {
         DetectDceOpnumFree(de_ctx, dod);
         return -1;
     }
-
-    sm->type = DETECT_DCE_OPNUM;
-    sm->ctx = (void *)dod;
-
-    SigMatchAppendSMToList(s, sm, g_dce_generic_list_id);
     return 0;
 }
 
diff --git a/src/detect-detection-filter.c b/src/detect-detection-filter.c
index 29c5183dc80f..b55d663b68eb 100644
--- a/src/detect-detection-filter.c
+++ b/src/detect-detection-filter.c
@@ -220,7 +220,6 @@ static int DetectDetectionFilterSetup(DetectEngineCtx *de_ctx, Signature *s, con
 {
     SCEnter();
     DetectThresholdData *df = NULL;
-    SigMatch *sm = NULL;
     SigMatch *tmpm = NULL;
 
     /* checks if there's a previous instance of threshold */
@@ -240,22 +239,16 @@ static int DetectDetectionFilterSetup(DetectEngineCtx *de_ctx, Signature *s, con
     if (df == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_DETECTION_FILTER, (SigMatchCtx *)df,
+                DETECT_SM_LIST_THRESHOLD) == NULL) {
         goto error;
-
-    sm->type = DETECT_DETECTION_FILTER;
-    sm->ctx = (SigMatchCtx *)df;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_THRESHOLD);
+    }
 
     return 0;
 
 error:
     if (df)
         SCFree(df);
-    if (sm)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-dhcp-leasetime.c b/src/detect-dhcp-leasetime.c
index dfa2c193302f..fea0d108fd58 100644
--- a/src/detect-dhcp-leasetime.c
+++ b/src/detect-dhcp-leasetime.c
@@ -93,14 +93,11 @@ static int DetectDHCPLeaseTimeSetup(DetectEngineCtx *de_ctx, Signature *s, const
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_DHCP_LEASETIME;
-    sm->ctx = (void *)dd;
 
-    SigMatchAppendSMToList(s, sm, g_buffer_id);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_DHCP_LEASETIME, (SigMatchCtx *)dd, g_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-dhcp-rebinding-time.c b/src/detect-dhcp-rebinding-time.c
index 3d63427eacb1..8d546376a394 100644
--- a/src/detect-dhcp-rebinding-time.c
+++ b/src/detect-dhcp-rebinding-time.c
@@ -93,14 +93,11 @@ static int DetectDHCPRebindingTimeSetup(DetectEngineCtx *de_ctx, Signature *s, c
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_DHCP_REBINDING_TIME;
-    sm->ctx = (void *)dd;
 
-    SigMatchAppendSMToList(s, sm, g_buffer_id);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_DHCP_REBINDING_TIME, (SigMatchCtx *)dd, g_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-dhcp-renewal-time.c b/src/detect-dhcp-renewal-time.c
index 9a38555a0d28..20ee763d9b90 100644
--- a/src/detect-dhcp-renewal-time.c
+++ b/src/detect-dhcp-renewal-time.c
@@ -93,14 +93,11 @@ static int DetectDHCPRenewalTimeSetup(DetectEngineCtx *de_ctx, Signature *s, con
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_DHCP_RENEWAL_TIME;
-    sm->ctx = (void *)dd;
 
-    SigMatchAppendSMToList(s, sm, g_buffer_id);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_DHCP_RENEWAL_TIME, (SigMatchCtx *)dd, g_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-dnp3.c b/src/detect-dnp3.c
index 208dec7c3a29..6d92596c1d73 100644
--- a/src/detect-dnp3.c
+++ b/src/detect-dnp3.c
@@ -205,7 +205,6 @@ static int DetectDNP3FuncSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 {
     SCEnter();
     DetectDNP3 *dnp3 = NULL;
-    SigMatch *sm = NULL;
     uint8_t function_code;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_DNP3) != 0)
@@ -222,23 +221,16 @@ static int DetectDNP3FuncSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     }
     dnp3->function_code = function_code;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_DNP3FUNC, (SigMatchCtx *)dnp3,
+                g_dnp3_match_buffer_id) == NULL) {
         goto error;
     }
-    sm->type = DETECT_AL_DNP3FUNC;
-    sm->ctx = (void *)dnp3;
-
-    SigMatchAppendSMToList(s, sm, g_dnp3_match_buffer_id);
 
     SCReturnInt(0);
 error:
     if (dnp3 != NULL) {
         SCFree(dnp3);
     }
-    if (sm != NULL) {
-        SCFree(sm);
-    }
     SCReturnInt(-1);
 }
 
@@ -291,7 +283,6 @@ static int DetectDNP3IndSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 {
     SCEnter();
     DetectDNP3 *detect = NULL;
-    SigMatch *sm = NULL;
     uint16_t flags;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_DNP3) != 0)
@@ -308,22 +299,16 @@ static int DetectDNP3IndSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     }
     detect->ind_flags = flags;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_DNP3IND, (SigMatchCtx *)detect,
+                g_dnp3_match_buffer_id) == NULL) {
         goto error;
     }
-    sm->type = DETECT_AL_DNP3IND;
-    sm->ctx = (void *)detect;
-    SigMatchAppendSMToList(s, sm, g_dnp3_match_buffer_id);
 
     SCReturnInt(0);
 error:
     if (detect != NULL) {
         SCFree(detect);
     }
-    if (sm != NULL) {
-        SCFree(sm);
-    }
     SCReturnInt(-1);
 }
 
@@ -366,7 +351,6 @@ static int DetectDNP3ObjSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     uint8_t group;
     uint8_t variation;
     DetectDNP3 *detect = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_DNP3) != 0)
         return -1;
@@ -382,22 +366,16 @@ static int DetectDNP3ObjSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     detect->obj_group = group;
     detect->obj_variation = variation;
 
-    sm = SigMatchAlloc();
-    if (unlikely(sm == NULL)) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_DNP3OBJ, (SigMatchCtx *)detect,
+                g_dnp3_match_buffer_id) == NULL) {
         goto fail;
     }
-    sm->type = DETECT_AL_DNP3OBJ;
-    sm->ctx = (void *)detect;
-    SigMatchAppendSMToList(s, sm, g_dnp3_match_buffer_id);
 
     SCReturnInt(1);
 fail:
     if (detect != NULL) {
         SCFree(detect);
     }
-    if (sm != NULL) {
-        SCFree(sm);
-    }
     SCReturnInt(0);
 }
 
diff --git a/src/detect-dns-opcode.c b/src/detect-dns-opcode.c
index 4c69753a83e0..853b01f0097d 100644
--- a/src/detect-dns-opcode.c
+++ b/src/detect-dns-opcode.c
@@ -41,15 +41,11 @@ static int DetectDnsOpcodeSetup(DetectEngineCtx *de_ctx, Signature *s,
         return -1;
     }
 
-    SigMatch *sm = SigMatchAlloc();
-    if (unlikely(sm == NULL)) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_DNS_OPCODE, (SigMatchCtx *)detect,
+                dns_opcode_list_id) == NULL) {
         goto error;
     }
 
-    sm->type = DETECT_AL_DNS_OPCODE;
-    sm->ctx = (void *)detect;
-    SigMatchAppendSMToList(s, sm, dns_opcode_list_id);
-    
     SCReturnInt(0);
 
 error:
diff --git a/src/detect-dsize.c b/src/detect-dsize.c
index 4336e3546b9f..bf095cd4fda5 100644
--- a/src/detect-dsize.c
+++ b/src/detect-dsize.c
@@ -119,7 +119,6 @@ static int DetectDsizeMatch (DetectEngineThreadCtx *det_ctx, Packet *p,
 static int DetectDsizeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectU16Data *dd = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectGetLastSMFromLists(s, DETECT_DSIZE, -1)) {
         SCLogError("Can't use 2 or more dsizes in "
@@ -137,18 +136,13 @@ static int DetectDsizeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL){
-        SCLogError("Failed to allocate memory for SigMatch");
+    SigMatch *sm = SigMatchAppendSMToList(
+            de_ctx, s, DETECT_DSIZE, (SigMatchCtx *)dd, DETECT_SM_LIST_MATCH);
+    if (sm == NULL) {
         rs_detect_u16_free(dd);
         goto error;
     }
 
-    sm->type = DETECT_DSIZE;
-    sm->ctx = (SigMatchCtx *)dd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
-
     SCLogDebug("dd->arg1 %" PRIu16 ", dd->arg2 %" PRIu16 ", dd->mode %" PRIu8 "", dd->arg1,
             dd->arg2, dd->mode);
     /* tell the sig it has a dsize to speed up engine init */
diff --git a/src/detect-engine-event.c b/src/detect-engine-event.c
index 82f838446ffa..5bbd5711259f 100644
--- a/src/detect-engine-event.c
+++ b/src/detect-engine-event.c
@@ -211,16 +211,11 @@ static int DetectEngineEventSetupDo(
 
     SCLogDebug("rawstr %s %u", rawstr, de->event);
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, smtype, (SigMatchCtx *)de, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         SCFree(de);
         return -1;
     }
-
-    sm->type = smtype;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     return 0;
 }
 
diff --git a/src/detect-file-hash-common.c b/src/detect-file-hash-common.c
index 3d35c9de0646..b028bff74bf1 100644
--- a/src/detect-file-hash-common.c
+++ b/src/detect-file-hash-common.c
@@ -321,7 +321,6 @@ int DetectFileHashSetup(
         DetectEngineCtx *de_ctx, Signature *s, const char *str, uint16_t type, int list)
 {
     DetectFileHashData *filehash = NULL;
-    SigMatch *sm = NULL;
 
     filehash = DetectFileHashParse(de_ctx, str, type);
     if (filehash == NULL)
@@ -329,14 +328,10 @@ int DetectFileHashSetup(
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = type;
-    sm->ctx = (void *)filehash;
 
-    SigMatchAppendSMToList(s, sm, list);
+    if (SigMatchAppendSMToList(de_ctx, s, type, (SigMatchCtx *)filehash, list) == NULL) {
+        goto error;
+    }
 
     s->file_flags |= FILE_SIG_NEED_FILE;
 
@@ -355,8 +350,6 @@ int DetectFileHashSetup(
 error:
     if (filehash != NULL)
         DetectFileHashFree(de_ctx, filehash);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-filesize.c b/src/detect-filesize.c
index 05caba8568c1..c29957d2870b 100644
--- a/src/detect-filesize.c
+++ b/src/detect-filesize.c
@@ -123,20 +123,15 @@ static int DetectFilesizeSetup (DetectEngineCtx *de_ctx, Signature *s, const cha
 {
     SCEnter();
     DetectU64Data *fsd = NULL;
-    SigMatch *sm = NULL;
 
     fsd = DetectU64Parse(str);
     if (fsd == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FILESIZE, (SigMatchCtx *)fsd, g_file_match_list_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_FILESIZE;
-    sm->ctx = (SigMatchCtx *)fsd;
-
-    SigMatchAppendSMToList(s, sm, g_file_match_list_id);
+    }
 
     s->file_flags |= (FILE_SIG_NEED_FILE|FILE_SIG_NEED_SIZE);
     SCReturnInt(0);
@@ -144,8 +139,6 @@ static int DetectFilesizeSetup (DetectEngineCtx *de_ctx, Signature *s, const cha
 error:
     if (fsd != NULL)
         DetectFilesizeFree(de_ctx, fsd);
-    if (sm != NULL)
-        SCFree(sm);
     SCReturnInt(-1);
 }
 
diff --git a/src/detect-filestore.c b/src/detect-filestore.c
index c53a93d78dd2..4efa59209967 100644
--- a/src/detect-filestore.c
+++ b/src/detect-filestore.c
@@ -349,7 +349,6 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
     }
 
     DetectFilestoreData *fd = NULL;
-    SigMatch *sm = NULL;
     char *args[3] = {NULL,NULL,NULL};
     int res = 0;
     size_t pcre2len;
@@ -361,12 +360,6 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
         return -1;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_FILESTORE;
-
     if (str != NULL && strlen(str) > 0) {
         char str_0[32];
         char str_1[32];
@@ -455,25 +448,22 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
             if (fd->scope == 0)
                 fd->scope = FILESTORE_SCOPE_DEFAULT;
         }
-
-        sm->ctx = (SigMatchCtx*)fd;
-    } else {
-        sm->ctx = (SigMatchCtx*)NULL;
     }
 
     if (s->alproto == ALPROTO_HTTP1 || s->alproto == ALPROTO_HTTP) {
         AppLayerHtpNeedFileInspection();
     }
 
-    SigMatchAppendSMToList(s, sm, g_file_match_list_id);
-    s->filestore_ctx = (const DetectFilestoreData *)sm->ctx;
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FILESTORE, (SigMatchCtx *)fd, g_file_match_list_id) == NULL) {
+        goto error;
+    }
+    s->filestore_ctx = fd;
 
-    sm = SigMatchAlloc();
-    if (unlikely(sm == NULL))
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FILESTORE_POSTMATCH, NULL, DETECT_SM_LIST_POSTMATCH) == NULL) {
         goto error;
-    sm->type = DETECT_FILESTORE_POSTMATCH;
-    sm->ctx = NULL;
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    }
 
     s->flags |= SIG_FLAG_FILESTORE;
 
@@ -486,8 +476,6 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
     if (match) {
         pcre2_match_data_free(match);
     }
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-flow-age.c b/src/detect-flow-age.c
index 7fc0f4d1e881..06ea3d9f9315 100644
--- a/src/detect-flow-age.c
+++ b/src/detect-flow-age.c
@@ -46,16 +46,11 @@ static int DetectFlowAgeSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     if (du32 == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FLOW_AGE, (SigMatchCtx *)du32, DETECT_SM_LIST_MATCH) == NULL) {
         DetectFlowAgeFree(de_ctx, du32);
         return -1;
     }
-
-    sm->type = DETECT_FLOW_AGE;
-    sm->ctx = (SigMatchCtx *)du32;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-flow-pkts.c b/src/detect-flow-pkts.c
index 7066b0b3bbc4..ef5ab2d32a44 100644
--- a/src/detect-flow-pkts.c
+++ b/src/detect-flow-pkts.c
@@ -46,16 +46,11 @@ static int DetectFlowPktsToClientSetup(DetectEngineCtx *de_ctx, Signature *s, co
     if (du32 == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOW_PKTS_TO_CLIENT, (SigMatchCtx *)du32,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectFlowPktsToClientFree(de_ctx, du32);
         return -1;
     }
-
-    sm->type = DETECT_FLOW_PKTS_TO_CLIENT;
-    sm->ctx = (SigMatchCtx *)du32;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -124,16 +119,11 @@ static int DetectFlowPktsToServerSetup(DetectEngineCtx *de_ctx, Signature *s, co
     if (du32 == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOW_PKTS_TO_SERVER, (SigMatchCtx *)du32,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectFlowPktsToServerFree(de_ctx, du32);
         return -1;
     }
-
-    sm->type = DETECT_FLOW_PKTS_TO_SERVER;
-    sm->ctx = (SigMatchCtx *)du32;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -202,16 +192,11 @@ static int DetectFlowBytesToClientSetup(DetectEngineCtx *de_ctx, Signature *s, c
     if (du64 == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOW_BYTES_TO_CLIENT, (SigMatchCtx *)du64,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectFlowBytesToClientFree(de_ctx, du64);
         return -1;
     }
-
-    sm->type = DETECT_FLOW_BYTES_TO_CLIENT;
-    sm->ctx = (SigMatchCtx *)du64;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -251,16 +236,11 @@ static int DetectFlowBytesToServerSetup(DetectEngineCtx *de_ctx, Signature *s, c
     if (du64 == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOW_BYTES_TO_SERVER, (SigMatchCtx *)du64,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectFlowBytesToServerFree(de_ctx, du64);
         return -1;
     }
-
-    sm->type = DETECT_FLOW_BYTES_TO_SERVER;
-    sm->ctx = (SigMatchCtx *)du64;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-flow.c b/src/detect-flow.c
index 9b0627cdfb12..696e5013a03e 100644
--- a/src/detect-flow.c
+++ b/src/detect-flow.c
@@ -388,13 +388,7 @@ int DetectFlowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *flowstr)
     if (fd == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_FLOW;
-    sm->ctx = (SigMatchCtx *)fd;
-
+    bool appendsm = true;
     /* set the signature direction flags */
     if (fd->flags & DETECT_FLOW_FLAG_TOSERVER) {
         s->flags |= SIG_FLAG_TOSERVER;
@@ -408,14 +402,18 @@ int DetectFlowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *flowstr)
             fd->flags == DETECT_FLOW_FLAG_TOCLIENT) {
         /* no direct flow is needed for just direction,
          * no sigmatch is needed either. */
-        SigMatchFree(de_ctx, sm);
-        sm = NULL;
+        appendsm = false;
     } else {
         s->init_data->init_flags |= SIG_FLAG_INIT_FLOW;
     }
 
-    if (sm != NULL) {
-        SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (appendsm) {
+        if (SigMatchAppendSMToList(
+                    de_ctx, s, DETECT_FLOW, (SigMatchCtx *)fd, DETECT_SM_LIST_MATCH) == NULL) {
+            goto error;
+        }
+    } else if (fd != NULL) {
+        DetectFlowFree(de_ctx, fd);
     }
 
     if (parse_flags & DETECT_FLOW_FLAG_ONLYSTREAM) {
diff --git a/src/detect-flowbits.c b/src/detect-flowbits.c
index 144eb89f8849..b04c271dc548 100644
--- a/src/detect-flowbits.c
+++ b/src/detect-flowbits.c
@@ -276,7 +276,6 @@ static int DetectFlowbitParse(const char *str, char *cmd, int cmd_len, char *nam
 int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectFlowbitsData *cd = NULL;
-    SigMatch *sm = NULL;
     uint8_t fb_cmd = 0;
     char fb_cmd_str[16] = "", fb_name[256] = "";
 
@@ -339,12 +338,6 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
     }
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_FLOWBITS;
-    sm->ctx = (SigMatchCtx *)cd;
 
     switch (fb_cmd) {
         /* case DETECT_FLOWBITS_CMD_NOALERT can't happen here */
@@ -352,14 +345,20 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
         case DETECT_FLOWBITS_CMD_ISNOTSET:
         case DETECT_FLOWBITS_CMD_ISSET:
             /* checks, so packet list */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOWBITS, (SigMatchCtx *)cd,
+                        DETECT_SM_LIST_MATCH) == NULL) {
+                goto error;
+            }
             break;
 
         case DETECT_FLOWBITS_CMD_SET:
         case DETECT_FLOWBITS_CMD_UNSET:
         case DETECT_FLOWBITS_CMD_TOGGLE:
             /* modifiers, only run when entire sig has matched */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOWBITS, (SigMatchCtx *)cd,
+                        DETECT_SM_LIST_POSTMATCH) == NULL) {
+                goto error;
+            }
             break;
 
         // suppress coverity warning as scan-build-7 warns w/o this.
@@ -373,8 +372,6 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
 error:
     if (cd != NULL)
         DetectFlowbitFree(de_ctx, cd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-flowint.c b/src/detect-flowint.c
index 6a28e980ce26..224eb650dc64 100644
--- a/src/detect-flowint.c
+++ b/src/detect-flowint.c
@@ -369,7 +369,6 @@ static DetectFlowintData *DetectFlowintParse(DetectEngineCtx *de_ctx, const char
 static int DetectFlowintSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectFlowintData *sfd = NULL;
-    SigMatch *sm = NULL;
 
     sfd = DetectFlowintParse(de_ctx, rawstr);
     if (sfd == NULL)
@@ -377,18 +376,15 @@ static int DetectFlowintSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_FLOWINT;
-    sm->ctx = (SigMatchCtx *)sfd;
 
     switch (sfd->modifier) {
         case FLOWINT_MODIFIER_SET:
         case FLOWINT_MODIFIER_ADD:
         case FLOWINT_MODIFIER_SUB:
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOWINT, (SigMatchCtx *)sfd,
+                        DETECT_SM_LIST_POSTMATCH) == NULL) {
+                goto error;
+            }
             break;
 
         case FLOWINT_MODIFIER_LT:
@@ -399,7 +395,10 @@ static int DetectFlowintSetup(DetectEngineCtx *de_ctx, Signature *s, const char
         case FLOWINT_MODIFIER_GT:
         case FLOWINT_MODIFIER_ISSET:
         case FLOWINT_MODIFIER_NOTSET:
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOWINT, (SigMatchCtx *)sfd,
+                        DETECT_SM_LIST_MATCH) == NULL) {
+                goto error;
+            }
             break;
         default:
             goto error;
@@ -410,8 +409,6 @@ static int DetectFlowintSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 error:
     if (sfd)
         DetectFlowintFree(de_ctx, sfd);
-    if (sm)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-flowvar.c b/src/detect-flowvar.c
index 4386a38caa12..38c8dc062919 100644
--- a/src/detect-flowvar.c
+++ b/src/detect-flowvar.c
@@ -117,7 +117,6 @@ int DetectFlowvarMatch (DetectEngineThreadCtx *det_ctx, Packet *p,
 static int DetectFlowvarSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectFlowvarData *fd = NULL;
-    SigMatch *sm = NULL;
     char varname[64], varcontent[64];
     int res = 0;
     size_t pcre2len;
@@ -184,14 +183,11 @@ static int DetectFlowvarSetup (DetectEngineCtx *de_ctx, Signature *s, const char
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (unlikely(sm == NULL))
-        goto error;
-
-    sm->type = DETECT_FLOWVAR;
-    sm->ctx = (SigMatchCtx *)fd;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FLOWVAR, (SigMatchCtx *)fd, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     SCFree(content);
     return 0;
@@ -199,8 +195,6 @@ static int DetectFlowvarSetup (DetectEngineCtx *de_ctx, Signature *s, const char
 error:
     if (fd != NULL)
         DetectFlowvarDataFree(de_ctx, fd);
-    if (sm != NULL)
-        SCFree(sm);
     if (content != NULL)
         SCFree(content);
     return -1;
@@ -265,7 +259,6 @@ int DetectVarStoreMatch(DetectEngineThreadCtx *det_ctx,
  */
 int DetectFlowvarPostMatchSetup(DetectEngineCtx *de_ctx, Signature *s, uint32_t idx)
 {
-    SigMatch *sm = NULL;
     DetectFlowvarData *fv = NULL;
 
     fv = SCMalloc(sizeof(DetectFlowvarData));
@@ -277,14 +270,10 @@ int DetectFlowvarPostMatchSetup(DetectEngineCtx *de_ctx, Signature *s, uint32_t
     fv->idx = idx;
     fv->post_match = true;
 
-    sm = SigMatchAlloc();
-    if (unlikely(sm == NULL))
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLOWVAR_POSTMATCH, (SigMatchCtx *)fv,
+                DETECT_SM_LIST_POSTMATCH) == NULL) {
         goto error;
-
-    sm->type = DETECT_FLOWVAR_POSTMATCH;
-    sm->ctx = (SigMatchCtx *)fv;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    }
     return 0;
 error:
     if (fv != NULL)
diff --git a/src/detect-fragbits.c b/src/detect-fragbits.c
index 0c266557864b..a9657641424c 100644
--- a/src/detect-fragbits.c
+++ b/src/detect-fragbits.c
@@ -287,20 +287,15 @@ static DetectFragBitsData *DetectFragBitsParse (const char *rawstr)
 static int DetectFragBitsSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectFragBitsData *de = NULL;
-    SigMatch *sm = NULL;
 
     de = DetectFragBitsParse(rawstr);
     if (de == NULL)
         return -1;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_FRAGBITS, (SigMatchCtx *)de, DETECT_SM_LIST_MATCH) == NULL) {
         goto error;
-
-    sm->type = DETECT_FRAGBITS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-fragoffset.c b/src/detect-fragoffset.c
index f32f06005a49..b4b21ff58ede 100644
--- a/src/detect-fragoffset.c
+++ b/src/detect-fragoffset.c
@@ -230,25 +230,21 @@ static DetectFragOffsetData *DetectFragOffsetParse (DetectEngineCtx *de_ctx, con
 static int DetectFragOffsetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *fragoffsetstr)
 {
     DetectFragOffsetData *fragoff = NULL;
-    SigMatch *sm = NULL;
 
     fragoff = DetectFragOffsetParse(de_ctx, fragoffsetstr);
     if (fragoff == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) goto error;
-
-    sm->type = DETECT_FRAGOFFSET;
-    sm->ctx = (SigMatchCtx *)fragoff;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FRAGOFFSET, (SigMatchCtx *)fragoff,
+                DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (fragoff != NULL) DetectFragOffsetFree(de_ctx, fragoff);
-    if (sm != NULL) SCFree(sm);
+    if (fragoff != NULL)
+        DetectFragOffsetFree(de_ctx, fragoff);
     return -1;
 
 }
diff --git a/src/detect-ftpbounce.c b/src/detect-ftpbounce.c
index 318f72cf3b58..79b0f1b579e2 100644
--- a/src/detect-ftpbounce.c
+++ b/src/detect-ftpbounce.c
@@ -207,18 +207,9 @@ int DetectFtpbounceSetup(DetectEngineCtx *de_ctx, Signature *s, const char *ftpb
 {
     SCEnter();
 
-    SigMatch *sm = NULL;
-
     if (DetectSignatureSetAppProto(s, ALPROTO_FTP) != 0)
         return -1;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) {
-        return -1;
-    }
-
-    sm->type = DETECT_FTPBOUNCE;
-
     /* We don't need to allocate any data for ftpbounce here.
      *
      * TODO: As a suggestion, maybe we can add a flag in the flow
@@ -228,8 +219,9 @@ int DetectFtpbounceSetup(DetectEngineCtx *de_ctx, Signature *s, const char *ftpb
      * without breaking the connection, so I guess we can make it a bit faster
      * with a flow flag set lookup in the Match function.
      */
-    sm->ctx = NULL;
 
-    SigMatchAppendSMToList(s, sm, g_ftp_request_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FTPBOUNCE, NULL, g_ftp_request_list_id) == NULL) {
+        return -1;
+    }
     SCReturnInt(0);
 }
diff --git a/src/detect-ftpdata.c b/src/detect-ftpdata.c
index c07847dff3f8..ce9e5c3c211c 100644
--- a/src/detect-ftpdata.c
+++ b/src/detect-ftpdata.c
@@ -191,15 +191,11 @@ static int DetectFtpdataSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     if (ftpcommandd == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FTPDATA, (SigMatchCtx *)ftpcommandd,
+                g_ftpdata_buffer_id) == NULL) {
         DetectFtpdataFree(de_ctx, ftpcommandd);
         return -1;
     }
-    sm->type = DETECT_FTPDATA;
-    sm->ctx = (void *)ftpcommandd;
-
-    SigMatchAppendSMToList(s, sm, g_ftpdata_buffer_id);
     return 0;
 }
 
diff --git a/src/detect-geoip.c b/src/detect-geoip.c
index d3f6c14ecdaf..e31e9fd518b4 100644
--- a/src/detect-geoip.c
+++ b/src/detect-geoip.c
@@ -409,21 +409,17 @@ static DetectGeoipData *DetectGeoipDataParse (DetectEngineCtx *de_ctx, const cha
 static int DetectGeoipSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
     DetectGeoipData *geoipdata = NULL;
-    SigMatch *sm = NULL;
 
     geoipdata = DetectGeoipDataParse(de_ctx, optstr);
     if (geoipdata == NULL)
         goto error;
 
     /* Get this into a SigMatch and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_GEOIP;
-    sm->ctx = (SigMatchCtx *)geoipdata;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_GEOIP, (SigMatchCtx *)geoipdata, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -431,8 +427,6 @@ static int DetectGeoipSetup(DetectEngineCtx *de_ctx, Signature *s, const char *o
 error:
     if (geoipdata != NULL)
         DetectGeoipDataFree(de_ctx, geoipdata);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
diff --git a/src/detect-hostbits.c b/src/detect-hostbits.c
index 764bf62805c1..571510325aea 100644
--- a/src/detect-hostbits.c
+++ b/src/detect-hostbits.c
@@ -331,7 +331,6 @@ static int DetectHostbitParse(const char *str, char *cmd, int cmd_len,
 int DetectHostbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectXbitsData *cd = NULL;
-    SigMatch *sm = NULL;
     uint8_t fb_cmd = 0;
     uint8_t hb_dir = 0;
     char fb_cmd_str[16] = "", fb_name[256] = "";
@@ -406,12 +405,6 @@ int DetectHostbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_HOSTBITS;
-    sm->ctx = (void *)cd;
 
     switch (fb_cmd) {
         /* case DETECT_XBITS_CMD_NOALERT can't happen here */
@@ -419,14 +412,20 @@ int DetectHostbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
         case DETECT_XBITS_CMD_ISNOTSET:
         case DETECT_XBITS_CMD_ISSET:
             /* checks, so packet list */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_HOSTBITS, (SigMatchCtx *)cd,
+                        DETECT_SM_LIST_MATCH) == NULL) {
+                goto error;
+            }
             break;
 
         case DETECT_XBITS_CMD_SET:
         case DETECT_XBITS_CMD_UNSET:
         case DETECT_XBITS_CMD_TOGGLE:
             /* modifiers, only run when entire sig has matched */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_HOSTBITS, (SigMatchCtx *)cd,
+                        DETECT_SM_LIST_POSTMATCH) == NULL) {
+                goto error;
+            }
             break;
 
         // suppress coverity warning as scan-build-7 warns w/o this.
@@ -440,8 +439,6 @@ int DetectHostbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
 error:
     if (cd != NULL)
         SCFree(cd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-http2.c b/src/detect-http2.c
index ec4840afe306..9991b85ad9ac 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -263,17 +263,12 @@ static int DetectHTTP2frametypeSetup (DetectEngineCtx *de_ctx, Signature *s, con
         return -1;
     *http2ft = frame_type;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_FRAMETYPE, (SigMatchCtx *)http2ft,
+                g_http2_match_buffer_id) == NULL) {
         DetectHTTP2frametypeFree(NULL, http2ft);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_FRAMETYPE;
-    sm->ctx = (SigMatchCtx *)http2ft;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
@@ -348,17 +343,12 @@ static int DetectHTTP2errorcodeSetup (DetectEngineCtx *de_ctx, Signature *s, con
         return -1;
     *http2ec = error_code;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_ERRORCODE, (SigMatchCtx *)http2ec,
+                g_http2_match_buffer_id) == NULL) {
         DetectHTTP2errorcodeFree(NULL, http2ec);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_ERRORCODE;
-    sm->ctx = (SigMatchCtx *)http2ec;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
@@ -415,17 +405,12 @@ static int DetectHTTP2prioritySetup (DetectEngineCtx *de_ctx, Signature *s, cons
     if (prio == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_PRIORITY, (SigMatchCtx *)prio,
+                g_http2_match_buffer_id) == NULL) {
         rs_detect_u8_free(prio);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_PRIORITY;
-    sm->ctx = (SigMatchCtx *)prio;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
@@ -482,17 +467,12 @@ static int DetectHTTP2windowSetup (DetectEngineCtx *de_ctx, Signature *s, const
     if (wu == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_WINDOW, (SigMatchCtx *)wu,
+                g_http2_match_buffer_id) == NULL) {
         rs_detect_u32_free(wu);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_WINDOW;
-    sm->ctx = (SigMatchCtx *)wu;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
@@ -539,17 +519,12 @@ static int DetectHTTP2sizeUpdateSetup (DetectEngineCtx *de_ctx, Signature *s, co
     if (su == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_SIZEUPDATE, (SigMatchCtx *)su,
+                g_http2_match_buffer_id) == NULL) {
         DetectHTTP2settingsFree(NULL, su);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_SIZEUPDATE;
-    sm->ctx = (SigMatchCtx *)su;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
@@ -596,17 +571,12 @@ static int DetectHTTP2settingsSetup (DetectEngineCtx *de_ctx, Signature *s, cons
     if (http2set == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_HTTP2_SETTINGS, (SigMatchCtx *)http2set,
+                g_http2_match_buffer_id) == NULL) {
         DetectHTTP2settingsFree(NULL, http2set);
         return -1;
     }
 
-    sm->type = DETECT_HTTP2_SETTINGS;
-    sm->ctx = (SigMatchCtx *)http2set;
-
-    SigMatchAppendSMToList(s, sm, g_http2_match_buffer_id);
-
     return 0;
 }
 
diff --git a/src/detect-icmp-id.c b/src/detect-icmp-id.c
index aee14bc377e9..301b2e76830f 100644
--- a/src/detect-icmp-id.c
+++ b/src/detect-icmp-id.c
@@ -241,25 +241,21 @@ static DetectIcmpIdData *DetectIcmpIdParse (DetectEngineCtx *de_ctx, const char
 static int DetectIcmpIdSetup (DetectEngineCtx *de_ctx, Signature *s, const char *icmpidstr)
 {
     DetectIcmpIdData *iid = NULL;
-    SigMatch *sm = NULL;
 
     iid = DetectIcmpIdParse(de_ctx, icmpidstr);
     if (iid == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) goto error;
-
-    sm->type = DETECT_ICMP_ID;
-    sm->ctx = (SigMatchCtx *)iid;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_ICMP_ID, (SigMatchCtx *)iid, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (iid != NULL) DetectIcmpIdFree(de_ctx, iid);
-    if (sm != NULL) SCFree(sm);
+    if (iid != NULL)
+        DetectIcmpIdFree(de_ctx, iid);
     return -1;
 
 }
diff --git a/src/detect-icmp-seq.c b/src/detect-icmp-seq.c
index 18a53fa68c26..ca32d2cec7ac 100644
--- a/src/detect-icmp-seq.c
+++ b/src/detect-icmp-seq.c
@@ -244,24 +244,20 @@ static DetectIcmpSeqData *DetectIcmpSeqParse (DetectEngineCtx *de_ctx, const cha
 static int DetectIcmpSeqSetup (DetectEngineCtx *de_ctx, Signature *s, const char *icmpseqstr)
 {
     DetectIcmpSeqData *iseq = NULL;
-    SigMatch *sm = NULL;
 
     iseq = DetectIcmpSeqParse(de_ctx, icmpseqstr);
     if (iseq == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) goto error;
-
-    sm->type = DETECT_ICMP_SEQ;
-    sm->ctx = (SigMatchCtx *)iseq;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_ICMP_SEQ, (SigMatchCtx *)iseq, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
-    if (iseq != NULL) DetectIcmpSeqFree(de_ctx, iseq);
-    if (sm != NULL) SCFree(sm);
+    if (iseq != NULL)
+        DetectIcmpSeqFree(de_ctx, iseq);
     return -1;
 
 }
diff --git a/src/detect-icmpv6-mtu.c b/src/detect-icmpv6-mtu.c
index ecb87343d1a5..f84e484e90ae 100644
--- a/src/detect-icmpv6-mtu.c
+++ b/src/detect-icmpv6-mtu.c
@@ -114,16 +114,11 @@ static int DetectICMPv6mtuSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
     if (icmpv6mtud == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ICMPV6MTU, (SigMatchCtx *)icmpv6mtud,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectICMPv6mtuFree(de_ctx, icmpv6mtud);
         return -1;
     }
-
-    sm->type = DETECT_ICMPV6MTU;
-    sm->ctx = (SigMatchCtx *)icmpv6mtud;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
     s->proto.flags |= DETECT_PROTO_IPV6;
 
diff --git a/src/detect-icode.c b/src/detect-icode.c
index 3a601c286df7..1e7d1cc060af 100644
--- a/src/detect-icode.c
+++ b/src/detect-icode.c
@@ -118,18 +118,14 @@ static int DetectICodeSetup(DetectEngineCtx *de_ctx, Signature *s, const char *i
 {
 
     DetectU8Data *icd = NULL;
-    SigMatch *sm = NULL;
 
     icd = DetectU8Parse(icodestr);
     if (icd == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) goto error;
-
-    sm->type = DETECT_ICODE;
-    sm->ctx = (SigMatchCtx *)icd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ICODE, (SigMatchCtx *)icd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -137,7 +133,6 @@ static int DetectICodeSetup(DetectEngineCtx *de_ctx, Signature *s, const char *i
 error:
     if (icd != NULL)
         rs_detect_u8_free(icd);
-    if (sm != NULL) SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-id.c b/src/detect-id.c
index 52392885a554..6725b7c1367e 100644
--- a/src/detect-id.c
+++ b/src/detect-id.c
@@ -191,7 +191,6 @@ static DetectIdData *DetectIdParse (const char *idstr)
 int DetectIdSetup (DetectEngineCtx *de_ctx, Signature *s, const char *idstr)
 {
     DetectIdData *id_d = NULL;
-    SigMatch *sm = NULL;
 
     id_d = DetectIdParse(idstr);
     if (id_d == NULL)
@@ -199,16 +198,11 @@ int DetectIdSetup (DetectEngineCtx *de_ctx, Signature *s, const char *idstr)
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ID, (SigMatchCtx *)id_d, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         DetectIdFree(de_ctx, id_d);
         return -1;
     }
-
-    sm->type = DETECT_ID;
-    sm->ctx = (SigMatchCtx *)id_d;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
     return 0;
 }
diff --git a/src/detect-ike-chosen-sa.c b/src/detect-ike-chosen-sa.c
index 59d245de7611..0ae8d400cba6 100644
--- a/src/detect-ike-chosen-sa.c
+++ b/src/detect-ike-chosen-sa.c
@@ -211,14 +211,11 @@ static int DetectIkeChosenSaSetup(DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_IKE_CHOSEN_SA;
-    sm->ctx = (void *)dd;
 
-    SigMatchAppendSMToList(s, sm, g_ike_chosen_sa_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_IKE_CHOSEN_SA, (SigMatchCtx *)dd,
+                g_ike_chosen_sa_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-ike-exch-type.c b/src/detect-ike-exch-type.c
index 3beb2c3a2519..38d4218d7faa 100644
--- a/src/detect-ike-exch-type.c
+++ b/src/detect-ike-exch-type.c
@@ -115,14 +115,11 @@ static int DetectIkeExchTypeSetup(DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_IKE_EXCH_TYPE;
-    sm->ctx = (SigMatchCtx *)ike_exch_type;
 
-    SigMatchAppendSMToList(s, sm, g_ike_exch_type_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_IKE_EXCH_TYPE, (SigMatchCtx *)ike_exch_type,
+                g_ike_exch_type_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-ike-key-exchange-payload-length.c b/src/detect-ike-key-exchange-payload-length.c
index 998948f4827c..4caad8038717 100644
--- a/src/detect-ike-key-exchange-payload-length.c
+++ b/src/detect-ike-key-exchange-payload-length.c
@@ -121,14 +121,12 @@ static int DetectIkeKeyExchangePayloadLengthSetup(
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_IKE_KEY_EXCHANGE_PAYLOAD_LENGTH;
-    sm->ctx = (SigMatchCtx *)key_exchange_payload_length;
 
-    SigMatchAppendSMToList(s, sm, g_ike_key_exch_payload_length_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_IKE_KEY_EXCHANGE_PAYLOAD_LENGTH,
+                (SigMatchCtx *)key_exchange_payload_length,
+                g_ike_key_exch_payload_length_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-ike-nonce-payload-length.c b/src/detect-ike-nonce-payload-length.c
index 91bc6c200cac..fbb3a903366a 100644
--- a/src/detect-ike-nonce-payload-length.c
+++ b/src/detect-ike-nonce-payload-length.c
@@ -115,14 +115,12 @@ static int DetectIkeNoncePayloadLengthSetup(
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_IKE_NONCE_PAYLOAD_LENGTH;
-    sm->ctx = (SigMatchCtx *)nonce_payload_length;
 
-    SigMatchAppendSMToList(s, sm, g_ike_nonce_payload_length_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_IKE_NONCE_PAYLOAD_LENGTH,
+                (SigMatchCtx *)nonce_payload_length,
+                g_ike_nonce_payload_length_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-ipopts.c b/src/detect-ipopts.c
index 105751c388a4..e4e9e22a36a3 100644
--- a/src/detect-ipopts.c
+++ b/src/detect-ipopts.c
@@ -243,27 +243,22 @@ static DetectIpOptsData *DetectIpOptsParse (const char *rawstr)
 static int DetectIpOptsSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectIpOptsData *de = NULL;
-    SigMatch *sm = NULL;
 
     de = DetectIpOptsParse(rawstr);
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_IPOPTS, (SigMatchCtx *)de, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_IPOPTS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (de) SCFree(de);
-    if (sm) SCFree(sm);
+    if (de)
+        SCFree(de);
     return -1;
 }
 
diff --git a/src/detect-ipproto.c b/src/detect-ipproto.c
index 51aac4f173bc..e5a0c7969b2f 100644
--- a/src/detect-ipproto.c
+++ b/src/detect-ipproto.c
@@ -188,7 +188,6 @@ static int DetectIPProtoTypePresentForOP(Signature *s, uint8_t op)
  */
 static int DetectIPProtoSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
-    SigMatch *sm = NULL;
     int i;
 
     DetectIPProtoData *data = DetectIPProtoParse(optstr);
@@ -414,12 +413,10 @@ static int DetectIPProtoSetup(DetectEngineCtx *de_ctx, Signature *s, const char
             break;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_IPPROTO, (SigMatchCtx *)data, DETECT_SM_LIST_MATCH) == NULL) {
         goto error;
-    sm->type = DETECT_IPPROTO;
-    sm->ctx = (void *)data;
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -430,7 +427,6 @@ static int DetectIPProtoSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     return -1;
 }
 
-
 void DetectIPProtoRemoveAllSMs(DetectEngineCtx *de_ctx, Signature *s)
 {
     SigMatch *sm = s->init_data->smlists[DETECT_SM_LIST_MATCH];
diff --git a/src/detect-iprep.c b/src/detect-iprep.c
index cc3d9a04c9bc..068619405bce 100644
--- a/src/detect-iprep.c
+++ b/src/detect-iprep.c
@@ -213,7 +213,6 @@ static int DetectIPRepMatch (DetectEngineThreadCtx *det_ctx, Packet *p,
 
 int DetectIPRepSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
-    SigMatch *sm = NULL;
 
     DetectIPRepData *cd = rs_detect_iprep_parse(rawstr);
     if (cd == NULL) {
@@ -225,22 +224,17 @@ int DetectIPRepSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_IPREP;
-    sm->ctx = (SigMatchCtx *)cd;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_IPREP, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
     if (cd != NULL)
         DetectIPRepFree(de_ctx, cd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-isdataat.c b/src/detect-isdataat.c
index e0858f1354a4..7b4d629ad3a1 100644
--- a/src/detect-isdataat.c
+++ b/src/detect-isdataat.c
@@ -211,7 +211,6 @@ static DetectIsdataatData *DetectIsdataatParse (DetectEngineCtx *de_ctx, const c
  */
 int DetectIsdataatSetup (DetectEngineCtx *de_ctx, Signature *s, const char *isdataatstr)
 {
-    SigMatch *sm = NULL;
     SigMatch *prev_pm = NULL;
     DetectIsdataatData *idad = NULL;
     char *offset = NULL;
@@ -273,12 +272,9 @@ int DetectIsdataatSetup (DetectEngineCtx *de_ctx, Signature *s, const char *isda
         goto end;
     }
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ISDATAAT, (SigMatchCtx *)idad, sm_list) == NULL) {
         goto end;
-    sm->type = DETECT_ISDATAAT;
-    sm->ctx = (SigMatchCtx *)idad;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     if (!(idad->flags & ISDATAAT_RELATIVE)) {
         ret = 0;
diff --git a/src/detect-itype.c b/src/detect-itype.c
index d8168600f5d2..3f8da9568aec 100644
--- a/src/detect-itype.c
+++ b/src/detect-itype.c
@@ -129,25 +129,21 @@ static int DetectITypeSetup(DetectEngineCtx *de_ctx, Signature *s, const char *i
 {
 
     DetectU8Data *itd = NULL;
-    SigMatch *sm = NULL;
 
     itd = DetectITypeParse(de_ctx, itypestr);
     if (itd == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL) goto error;
-
-    sm->type = DETECT_ITYPE;
-    sm->ctx = (SigMatchCtx *)itd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ITYPE, (SigMatchCtx *)itd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (itd != NULL) DetectITypeFree(de_ctx, itd);
-    if (sm != NULL) SCFree(sm);
+    if (itd != NULL)
+        DetectITypeFree(de_ctx, itd);
     return -1;
 }
 
diff --git a/src/detect-krb5-errcode.c b/src/detect-krb5-errcode.c
index 30c516f8d273..f9d22cbede5d 100644
--- a/src/detect-krb5-errcode.c
+++ b/src/detect-krb5-errcode.c
@@ -176,7 +176,6 @@ static DetectKrb5ErrCodeData *DetectKrb5ErrCodeParse (const char *krb5str)
 static int DetectKrb5ErrCodeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *krb5str)
 {
     DetectKrb5ErrCodeData *krb5d = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_KRB5) != 0)
         return -1;
@@ -185,22 +184,16 @@ static int DetectKrb5ErrCodeSetup (DetectEngineCtx *de_ctx, Signature *s, const
     if (krb5d == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_KRB5_ERRCODE, (SigMatchCtx *)krb5d,
+                g_krb5_err_code_list_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_KRB5_ERRCODE;
-    sm->ctx = (void *)krb5d;
-
-    SigMatchAppendSMToList(s, sm, g_krb5_err_code_list_id);
+    }
 
     return 0;
 
 error:
     if (krb5d != NULL)
         DetectKrb5ErrCodeFree(de_ctx, krb5d);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-krb5-msgtype.c b/src/detect-krb5-msgtype.c
index 0dd800d6be58..4e2ae85848ed 100644
--- a/src/detect-krb5-msgtype.c
+++ b/src/detect-krb5-msgtype.c
@@ -173,7 +173,6 @@ static DetectKrb5MsgTypeData *DetectKrb5MsgTypeParse (const char *krb5str)
 static int DetectKrb5MsgTypeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *krb5str)
 {
     DetectKrb5MsgTypeData *krb5d = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_KRB5) != 0)
         return -1;
@@ -182,22 +181,16 @@ static int DetectKrb5MsgTypeSetup (DetectEngineCtx *de_ctx, Signature *s, const
     if (krb5d == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_KRB5_MSGTYPE, (SigMatchCtx *)krb5d,
+                g_krb5_msg_type_list_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_KRB5_MSGTYPE;
-    sm->ctx = (void *)krb5d;
-
-    SigMatchAppendSMToList(s, sm, g_krb5_msg_type_list_id);
+    }
 
     return 0;
 
 error:
     if (krb5d != NULL)
         DetectKrb5MsgTypeFree(de_ctx, krb5d);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-krb5-ticket-encryption.c b/src/detect-krb5-ticket-encryption.c
index ea1444e30d36..e3550084ffb6 100644
--- a/src/detect-krb5-ticket-encryption.c
+++ b/src/detect-krb5-ticket-encryption.c
@@ -44,7 +44,6 @@ static int DetectKrb5TicketEncryptionSetup(
         DetectEngineCtx *de_ctx, Signature *s, const char *krb5str)
 {
     DetectKrb5TicketEncryptionData *krb5d = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_KRB5) != 0)
         return -1;
@@ -53,22 +52,16 @@ static int DetectKrb5TicketEncryptionSetup(
     if (krb5d == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_KRB5_TICKET_ENCRYPTION, (SigMatchCtx *)krb5d,
+                g_krb5_ticket_encryption_list_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_KRB5_TICKET_ENCRYPTION;
-    sm->ctx = (void *)krb5d;
-
-    SigMatchAppendSMToList(s, sm, g_krb5_ticket_encryption_list_id);
+    }
 
     return 0;
 
 error:
     if (krb5d != NULL)
         DetectKrb5TicketEncryptionFree(de_ctx, krb5d);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-lua.c b/src/detect-lua.c
index dfb26dcbe698..4f66fa7395ab 100644
--- a/src/detect-lua.c
+++ b/src/detect-lua.c
@@ -1013,7 +1013,6 @@ static int DetectLuaSetupPrime(DetectEngineCtx *de_ctx, DetectLuaData *ld, const
 static int DetectLuaSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectLuaData *lua = NULL;
-    SigMatch *sm = NULL;
 
     /* First check if Lua rules are enabled, by default Lua in rules
      * is disabled. */
@@ -1047,12 +1046,6 @@ static int DetectLuaSetup (DetectEngineCtx *de_ctx, Signature *s, const char *st
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_LUA;
-    sm->ctx = (SigMatchCtx *)lua;
 
     int list = -1;
     if (lua->alproto == ALPROTO_UNKNOWN) {
@@ -1118,15 +1111,15 @@ static int DetectLuaSetup (DetectEngineCtx *de_ctx, Signature *s, const char *st
         goto error;
     }
 
-    SigMatchAppendSMToList(s, sm, list);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_LUA, (SigMatchCtx *)lua, list) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
     if (lua != NULL)
         DetectLuaFree(de_ctx, lua);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mark.c b/src/detect-mark.c
index b6a46a2a5413..90ed7750a4e5 100644
--- a/src/detect-mark.c
+++ b/src/detect-mark.c
@@ -203,18 +203,14 @@ static int DetectMarkSetup (DetectEngineCtx *de_ctx, Signature *s, const char *r
     if (data == NULL) {
         return -1;
     }
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
-        DetectMarkDataFree(de_ctx, data);
-        return -1;
-    }
-
-    sm->type = DETECT_MARK;
-    sm->ctx = (SigMatchCtx *)data;
 
     /* Append it to the list of post match, so the mark is set if the
      * full signature matches. */
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_MARK, (SigMatchCtx *)data, DETECT_SM_LIST_POSTMATCH) == NULL) {
+        DetectMarkDataFree(de_ctx, data);
+        return -1;
+    }
     return 0;
 #endif
 }
diff --git a/src/detect-modbus.c b/src/detect-modbus.c
index b010500a143c..f4e6d4fd03ff 100644
--- a/src/detect-modbus.c
+++ b/src/detect-modbus.c
@@ -85,7 +85,6 @@ static int DetectModbusSetup(DetectEngineCtx *de_ctx, Signature *s, const char *
 {
     SCEnter();
     DetectModbusRust *modbus = NULL;
-    SigMatch        *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MODBUS) != 0)
         return -1;
@@ -96,22 +95,16 @@ static int DetectModbusSetup(DetectEngineCtx *de_ctx, Signature *s, const char *
     }
 
     /* Okay so far so good, lets get this into a SigMatch and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_MODBUS, (SigMatchCtx *)modbus, g_modbus_buffer_id) == NULL) {
         goto error;
-
-    sm->type    = DETECT_AL_MODBUS;
-    sm->ctx     = (void *) modbus;
-
-    SigMatchAppendSMToList(s, sm, g_modbus_buffer_id);
+    }
 
     SCReturnInt(0);
 
 error:
     if (modbus != NULL)
         DetectModbusFree(de_ctx, modbus);
-    if (sm != NULL)
-        SCFree(sm);
     SCReturnInt(-1);
 }
 
diff --git a/src/detect-mqtt-connack-sessionpresent.c b/src/detect-mqtt-connack-sessionpresent.c
index 7ec902f1172c..4b29158b1f89 100644
--- a/src/detect-mqtt-connack-sessionpresent.c
+++ b/src/detect-mqtt-connack-sessionpresent.c
@@ -156,7 +156,6 @@ static bool *DetectMQTTConnackSessionPresentParse(const char *rawstr)
 static int DetectMQTTConnackSessionPresentSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     bool *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -165,22 +164,16 @@ static int DetectMQTTConnackSessionPresentSetup (DetectEngineCtx *de_ctx, Signat
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_CONNACK_SESSION_PRESENT, (SigMatchCtx *)de,
+                mqtt_connack_session_present_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_CONNACK_SESSION_PRESENT;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_connack_session_present_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-connect-flags.c b/src/detect-mqtt-connect-flags.c
index 49bfae6f4b52..ce543ecdaa41 100644
--- a/src/detect-mqtt-connect-flags.c
+++ b/src/detect-mqtt-connect-flags.c
@@ -214,7 +214,6 @@ static DetectMQTTConnectFlagsData *DetectMQTTConnectFlagsParse(const char *rawst
 static int DetectMQTTConnectFlagsSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectMQTTConnectFlagsData *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -223,22 +222,16 @@ static int DetectMQTTConnectFlagsSetup(DetectEngineCtx *de_ctx, Signature *s, co
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_CONNECT_FLAGS, (SigMatchCtx *)de,
+                mqtt_connect_flags_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_CONNECT_FLAGS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_connect_flags_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-flags.c b/src/detect-mqtt-flags.c
index 799e1668e404..d0614061416d 100644
--- a/src/detect-mqtt-flags.c
+++ b/src/detect-mqtt-flags.c
@@ -198,7 +198,6 @@ static DetectMQTTFlagsData *DetectMQTTFlagsParse(const char *rawstr)
 static int DetectMQTTFlagsSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectMQTTFlagsData *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -207,22 +206,16 @@ static int DetectMQTTFlagsSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_FLAGS, (SigMatchCtx *)de, mqtt_flags_id) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_FLAGS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_flags_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-protocol-version.c b/src/detect-mqtt-protocol-version.c
index 39a9ce67d6f9..6ba183d75c8a 100644
--- a/src/detect-mqtt-protocol-version.c
+++ b/src/detect-mqtt-protocol-version.c
@@ -106,7 +106,6 @@ static int DetectMQTTProtocolVersionMatch(DetectEngineThreadCtx *det_ctx,
  */
 static int DetectMQTTProtocolVersionSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
-    SigMatch *sm = NULL;
     DetectU8Data *de = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
@@ -116,22 +115,16 @@ static int DetectMQTTProtocolVersionSetup(DetectEngineCtx *de_ctx, Signature *s,
     if (de == NULL)
         return -1;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_PROTOCOL_VERSION, (SigMatchCtx *)de,
+                mqtt_protocol_version_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_PROTOCOL_VERSION;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_protocol_version_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         rs_detect_u8_free(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-qos.c b/src/detect-mqtt-qos.c
index 07aa834dc20d..a00eaee185a2 100644
--- a/src/detect-mqtt-qos.c
+++ b/src/detect-mqtt-qos.c
@@ -135,7 +135,6 @@ static uint8_t *DetectMQTTQosParse(const char *rawstr)
 static int DetectMQTTQosSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     uint8_t *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -144,22 +143,16 @@ static int DetectMQTTQosSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_QOS, (SigMatchCtx *)de, mqtt_qos_id) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_QOS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_qos_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-reason-code.c b/src/detect-mqtt-reason-code.c
index 085c9c047c9f..e6ecba44cc26 100644
--- a/src/detect-mqtt-reason-code.c
+++ b/src/detect-mqtt-reason-code.c
@@ -151,7 +151,6 @@ static uint8_t *DetectMQTTReasonCodeParse(const char *rawstr)
 static int DetectMQTTReasonCodeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     uint8_t *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -160,22 +159,16 @@ static int DetectMQTTReasonCodeSetup (DetectEngineCtx *de_ctx, Signature *s, con
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_REASON_CODE, (SigMatchCtx *)de,
+                mqtt_reason_code_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_REASON_CODE;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_reason_code_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-mqtt-type.c b/src/detect-mqtt-type.c
index c55938c78c2d..5e23a509ca7c 100644
--- a/src/detect-mqtt-type.c
+++ b/src/detect-mqtt-type.c
@@ -140,7 +140,6 @@ static uint8_t *DetectMQTTTypeParse(const char *rawstr)
 static int DetectMQTTTypeSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     uint8_t *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_MQTT) < 0)
         return -1;
@@ -149,22 +148,16 @@ static int DetectMQTTTypeSetup (DetectEngineCtx *de_ctx, Signature *s, const cha
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_MQTT_TYPE, (SigMatchCtx *)de, mqtt_type_id) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_AL_MQTT_TYPE;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, mqtt_type_id);
+    }
 
     return 0;
 
 error:
     if (de != NULL)
         SCFree(de);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-nfs-procedure.c b/src/detect-nfs-procedure.c
index 08d69f7d6371..24c1563df18f 100644
--- a/src/detect-nfs-procedure.c
+++ b/src/detect-nfs-procedure.c
@@ -153,7 +153,6 @@ static int DetectNfsProcedureSetup (DetectEngineCtx *de_ctx, Signature *s,
                                    const char *rawstr)
 {
     DetectU32Data *dd = NULL;
-    SigMatch *sm = NULL;
 
     SCLogDebug("\'%s\'", rawstr);
 
@@ -168,15 +167,12 @@ static int DetectNfsProcedureSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_NFS_PROCEDURE;
-    sm->ctx = (void *)dd;
 
     SCLogDebug("low %u hi %u", dd->arg1, dd->arg2);
-    SigMatchAppendSMToList(s, sm, g_nfs_request_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_NFS_PROCEDURE, (SigMatchCtx *)dd,
+                g_nfs_request_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-nfs-version.c b/src/detect-nfs-version.c
index 5b4f3b82def8..99c88149a73e 100644
--- a/src/detect-nfs-version.c
+++ b/src/detect-nfs-version.c
@@ -152,15 +152,12 @@ static int DetectNfsVersionSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_NFS_VERSION;
-    sm->ctx = (void *)dd;
 
     SCLogDebug("low %u hi %u", dd->arg1, dd->arg2);
-    SigMatchAppendSMToList(s, sm, g_nfs_request_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_NFS_VERSION, (SigMatchCtx *)dd,
+                g_nfs_request_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-parse.c b/src/detect-parse.c
index bf54f9359a0e..ba3c17d789ae 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -447,8 +447,16 @@ void SigTableApplyStrictCommandLineOption(const char *str)
  * \param new  The sig match to append.
  * \param list The list to append to.
  */
-void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
+SigMatch *SigMatchAppendSMToList(
+        DetectEngineCtx *de_ctx, Signature *s, uint16_t type, SigMatchCtx *ctx, const int list)
 {
+    SigMatch *new = SigMatchAlloc();
+    if (new == NULL)
+        return NULL;
+
+    new->type = type;
+    new->ctx = ctx;
+
     if (new->type == DETECT_CONTENT) {
         s->init_data->max_content_list_id = MAX(s->init_data->max_content_list_id, (uint32_t)list);
     }
@@ -498,10 +506,9 @@ void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
                 s->init_data->curbuf == NULL) {
             if (SignatureInitDataBufferCheckExpand(s) < 0) {
                 SCLogError("failed to expand rule buffer array");
-                s->init_data->init_flags |= SIG_FLAG_INIT_OVERFLOW;
-                // SignatureInitDataBufferCheckExpand should not fail in this case
-                DEBUG_VALIDATE_BUG_ON(s->init_data->curbuf == NULL);
-                // keep curbuf even with wrong id as we error on this signature
+                new->ctx = NULL;
+                SigMatchFree(de_ctx, new);
+                return NULL;
             } else {
                 /* initialize new buffer */
                 s->init_data->curbuf = &s->init_data->buffers[s->init_data->buffer_index++];
@@ -530,6 +537,7 @@ void SigMatchAppendSMToList(Signature *s, SigMatch *new, const int list)
                     sigmatch_table[sm->type].name, sm->idx);
         }
     }
+    return new;
 }
 
 void SigMatchRemoveSMFromList(Signature *s, SigMatch *sm, int sm_list)
@@ -1017,11 +1025,8 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
         /* setup may or may not add a new SigMatch to the list */
         setup_ret = st->Setup(de_ctx, s, NULL);
     }
-    if (setup_ret < 0 || (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW)) {
+    if (setup_ret < 0) {
         SCLogDebug("\"%s\" failed to setup", st->name);
-        if (s->init_data->init_flags & SIG_FLAG_INIT_OVERFLOW) {
-            SCLogError("rule %u tries to use too many buffers", s->id);
-        }
 
         /* handle 'silent' error case */
         if (setup_ret == -2) {
diff --git a/src/detect-parse.h b/src/detect-parse.h
index a7f2c4d17df7..2eecd286f631 100644
--- a/src/detect-parse.h
+++ b/src/detect-parse.h
@@ -75,7 +75,7 @@ SigMatchData* SigMatchList2DataArray(SigMatch *head);
 void SigParseRegisterTests(void);
 Signature *DetectEngineAppendSig(DetectEngineCtx *, const char *);
 
-void SigMatchAppendSMToList(Signature *, SigMatch *, int);
+SigMatch *SigMatchAppendSMToList(DetectEngineCtx *, Signature *, uint16_t, SigMatchCtx *, int);
 void SigMatchRemoveSMFromList(Signature *, SigMatch *, int);
 int SigMatchListSMBelongsTo(const Signature *, const SigMatch *);
 
diff --git a/src/detect-pcre.c b/src/detect-pcre.c
index ce5155f7e238..913d782f4afe 100644
--- a/src/detect-pcre.c
+++ b/src/detect-pcre.c
@@ -865,7 +865,6 @@ static int DetectPcreSetup (DetectEngineCtx *de_ctx, Signature *s, const char *r
 {
     SCEnter();
     DetectPcreData *pd = NULL;
-    SigMatch *sm = NULL;
     int parsed_sm_list = DETECT_SM_LIST_NOTSET;
     char capture_names[1024] = "";
     AppProto alproto = ALPROTO_UNKNOWN;
@@ -918,12 +917,10 @@ static int DetectPcreSetup (DetectEngineCtx *de_ctx, Signature *s, const char *r
     if (sm_list == -1)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    SigMatch *sm = SigMatchAppendSMToList(de_ctx, s, DETECT_PCRE, (SigMatchCtx *)pd, sm_list);
+    if (sm == NULL) {
         goto error;
-    sm->type = DETECT_PCRE;
-    sm->ctx = (void *)pd;
-    SigMatchAppendSMToList(s, sm, sm_list);
+    }
 
     for (uint8_t x = 0; x < pd->idx; x++) {
         if (DetectFlowvarPostMatchSetup(de_ctx, s, pd->capids[x]) < 0)
diff --git a/src/detect-pktvar.c b/src/detect-pktvar.c
index a9e24168a6fa..7166188eb256 100644
--- a/src/detect-pktvar.c
+++ b/src/detect-pktvar.c
@@ -152,15 +152,10 @@ static int DetectPktvarSetup (DetectEngineCtx *de_ctx, Signature *s, const char
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (unlikely(sm == NULL)) {
-        DetectPktvarFree(de_ctx, cd);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_PKTVAR, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         goto error;
     }
-    sm->type = DETECT_PKTVAR;
-    sm->ctx = (SigMatchCtx *)cd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
 
     pcre2_match_data_free(match);
     return 0;
diff --git a/src/detect-replace.c b/src/detect-replace.c
index eae52a5a29ac..147c3e94d8b1 100644
--- a/src/detect-replace.c
+++ b/src/detect-replace.c
@@ -156,15 +156,9 @@ int DetectReplaceSetup(DetectEngineCtx *de_ctx, Signature *s, const char *replac
     SCFree(content);
     content = NULL;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (unlikely(sm == NULL)) {
-        SCFree(ud->replace);
-        ud->replace = NULL;
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_REPLACE, NULL, DETECT_SM_LIST_POSTMATCH) == NULL) {
         goto error;
     }
-    sm->type = DETECT_REPLACE;
-    sm->ctx = NULL;
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
     return 0;
 
 error:
diff --git a/src/detect-rfb-secresult.c b/src/detect-rfb-secresult.c
index ff82d98fa690..403c16d08aa5 100644
--- a/src/detect-rfb-secresult.c
+++ b/src/detect-rfb-secresult.c
@@ -210,7 +210,6 @@ static DetectRfbSecresultData *DetectRfbSecresultParse (const char *rawstr)
 static int DetectRfbSecresultSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectRfbSecresultData *de = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_RFB) < 0)
         return -1;
@@ -219,20 +218,16 @@ static int DetectRfbSecresultSetup (DetectEngineCtx *de_ctx, Signature *s, const
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_RFB_SECRESULT, (SigMatchCtx *)de, rfb_secresult_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_RFB_SECRESULT;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, rfb_secresult_id);
+    }
 
     return 0;
 
 error:
-    if (de) SCFree(de);
-    if (sm) SCFree(sm);
+    if (de)
+        SCFree(de);
     return -1;
 }
 
diff --git a/src/detect-rfb-sectype.c b/src/detect-rfb-sectype.c
index 400ee5cb087c..d942a4503a49 100644
--- a/src/detect-rfb-sectype.c
+++ b/src/detect-rfb-sectype.c
@@ -127,14 +127,11 @@ static int DetectRfbSectypeSetup (DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_RFB_SECTYPE;
-    sm->ctx = (void *)dd;
 
-    SigMatchAppendSMToList(s, sm, g_rfb_sectype_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_RFB_SECTYPE, (SigMatchCtx *)dd,
+                g_rfb_sectype_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-rpc.c b/src/detect-rpc.c
index 2739d6218caf..07f29569000f 100644
--- a/src/detect-rpc.c
+++ b/src/detect-rpc.c
@@ -266,26 +266,21 @@ static DetectRpcData *DetectRpcParse (DetectEngineCtx *de_ctx, const char *rpcst
 int DetectRpcSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rpcstr)
 {
     DetectRpcData *rd = NULL;
-    SigMatch *sm = NULL;
 
     rd = DetectRpcParse(de_ctx, rpcstr);
     if (rd == NULL) goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_RPC, (SigMatchCtx *)rd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_RPC;
-    sm->ctx = (SigMatchCtx *)rd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (rd != NULL) DetectRpcFree(de_ctx, rd);
-    if (sm != NULL) SCFree(sm);
+    if (rd != NULL)
+        DetectRpcFree(de_ctx, rd);
     return -1;
 
 }
diff --git a/src/detect-sameip.c b/src/detect-sameip.c
index 4bf03b0cd888..2ed72cf00e3a 100644
--- a/src/detect-sameip.c
+++ b/src/detect-sameip.c
@@ -92,26 +92,18 @@ static int DetectSameipMatch(DetectEngineThreadCtx *det_ctx,
  */
 static int DetectSameipSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
-    SigMatch *sm = NULL;
 
     /* Get this into a SigMatch and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_SAMEIP;
-    sm->ctx = NULL;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_SAMEIP, NULL, DETECT_SM_LIST_MATCH) == NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
-
 }
 
 #ifdef UNITTESTS
diff --git a/src/detect-snmp-pdu_type.c b/src/detect-snmp-pdu_type.c
index d053c29a792d..243d6c323be8 100644
--- a/src/detect-snmp-pdu_type.c
+++ b/src/detect-snmp-pdu_type.c
@@ -180,7 +180,6 @@ static int DetectSNMPPduTypeSetup (DetectEngineCtx *de_ctx, Signature *s,
                                    const char *rawstr)
 {
     DetectSNMPPduTypeData *dd = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_SNMP) != 0)
         return -1;
@@ -193,15 +192,12 @@ static int DetectSNMPPduTypeSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_SNMP_PDU_TYPE;
-    sm->ctx = (void *)dd;
 
     SCLogDebug("snmp.pdu_type %d", dd->pdu_type);
-    SigMatchAppendSMToList(s, sm, g_snmp_pdu_type_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SNMP_PDU_TYPE, (SigMatchCtx *)dd,
+                g_snmp_pdu_type_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-snmp-version.c b/src/detect-snmp-version.c
index 57359c091bd6..64029659381e 100644
--- a/src/detect-snmp-version.c
+++ b/src/detect-snmp-version.c
@@ -132,7 +132,6 @@ static int DetectSNMPVersionSetup (DetectEngineCtx *de_ctx, Signature *s,
                                    const char *rawstr)
 {
     DetectU32Data *dd = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_SNMP) != 0)
         return -1;
@@ -145,15 +144,12 @@ static int DetectSNMPVersionSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_SNMP_VERSION;
-    sm->ctx = (void *)dd;
 
     SCLogDebug("snmp.version %d", dd->arg1);
-    SigMatchAppendSMToList(s, sm, g_snmp_version_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SNMP_VERSION, (SigMatchCtx *)dd,
+                g_snmp_version_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
diff --git a/src/detect-ssh-proto-version.c b/src/detect-ssh-proto-version.c
index d357d3f801d1..1ca99e620287 100644
--- a/src/detect-ssh-proto-version.c
+++ b/src/detect-ssh-proto-version.c
@@ -233,7 +233,6 @@ static DetectSshVersionData *DetectSshVersionParse (DetectEngineCtx *de_ctx, con
 static int DetectSshVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectSshVersionData *ssh = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_SSH) != 0)
         return -1;
@@ -244,21 +243,16 @@ static int DetectSshVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_SSH_PROTOVERSION;
-    sm->ctx = (void *)ssh;
 
-    SigMatchAppendSMToList(s, sm, g_ssh_banner_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SSH_PROTOVERSION, (SigMatchCtx *)ssh,
+                g_ssh_banner_list_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (ssh != NULL)
         DetectSshVersionFree(de_ctx, ssh);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
diff --git a/src/detect-ssh-software-version.c b/src/detect-ssh-software-version.c
index 5fec33ac0eef..c2ba4ba888ef 100644
--- a/src/detect-ssh-software-version.c
+++ b/src/detect-ssh-software-version.c
@@ -220,7 +220,6 @@ static DetectSshSoftwareVersionData *DetectSshSoftwareVersionParse (DetectEngine
 static int DetectSshSoftwareVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectSshSoftwareVersionData *ssh = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_SSH) != 0)
         return -1;
@@ -231,21 +230,16 @@ static int DetectSshSoftwareVersionSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_SSH_SOFTWAREVERSION;
-    sm->ctx = (void *)ssh;
 
-    SigMatchAppendSMToList(s, sm, g_ssh_banner_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SSH_SOFTWAREVERSION, (SigMatchCtx *)ssh,
+                g_ssh_banner_list_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (ssh != NULL)
         DetectSshSoftwareVersionFree(de_ctx, ssh);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
diff --git a/src/detect-ssl-state.c b/src/detect-ssl-state.c
index 3f2df48db7aa..fd60f045a4c3 100644
--- a/src/detect-ssl-state.c
+++ b/src/detect-ssl-state.c
@@ -303,7 +303,6 @@ static DetectSslStateData *DetectSslStateParse(const char *arg)
 static int DetectSslStateSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg)
 {
     DetectSslStateData *ssd = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
@@ -312,21 +311,15 @@ static int DetectSslStateSetup(DetectEngineCtx *de_ctx, Signature *s, const char
     if (ssd == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SSL_STATE, (SigMatchCtx *)ssd,
+                g_tls_generic_list_id) == NULL) {
         goto error;
-
-    sm->type = DETECT_AL_SSL_STATE;
-    sm->ctx = (SigMatchCtx*)ssd;
-
-    SigMatchAppendSMToList(s, sm, g_tls_generic_list_id);
+    }
     return 0;
 
 error:
     if (ssd != NULL)
         DetectSslStateFree(de_ctx, ssd);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-ssl-version.c b/src/detect-ssl-version.c
index b4e142bd7389..1326da49dd1c 100644
--- a/src/detect-ssl-version.c
+++ b/src/detect-ssl-version.c
@@ -290,7 +290,6 @@ static DetectSslVersionData *DetectSslVersionParse(DetectEngineCtx *de_ctx, cons
 static int DetectSslVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectSslVersionData *ssl = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
@@ -301,21 +300,16 @@ static int DetectSslVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_SSL_VERSION;
-    sm->ctx = (void *)ssl;
 
-    SigMatchAppendSMToList(s, sm, g_tls_generic_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_SSL_VERSION, (SigMatchCtx *)ssl,
+                g_tls_generic_list_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (ssl != NULL)
         DetectSslVersionFree(de_ctx, ssl);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-stream_size.c b/src/detect-stream_size.c
index 50cd15af2d35..196439aa3131 100644
--- a/src/detect-stream_size.c
+++ b/src/detect-stream_size.c
@@ -146,16 +146,11 @@ static int DetectStreamSizeSetup (DetectEngineCtx *de_ctx, Signature *s, const c
     if (sd == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_STREAM_SIZE, (SigMatchCtx *)sd, DETECT_SM_LIST_MATCH) == NULL) {
         DetectStreamSizeFree(de_ctx, sd);
         return -1;
     }
-
-    sm->type = DETECT_STREAM_SIZE;
-    sm->ctx = (SigMatchCtx *)sd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     return 0;
 }
 
diff --git a/src/detect-tag.c b/src/detect-tag.c
index c31b44088d7d..bab756b3b601 100644
--- a/src/detect-tag.c
+++ b/src/detect-tag.c
@@ -303,17 +303,12 @@ int DetectTagSetup(DetectEngineCtx *de_ctx, Signature *s, const char *tagstr)
     if (td == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    /* Append it to the list of tags */
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_TAG, (SigMatchCtx *)td, DETECT_SM_LIST_TMATCH) ==
+            NULL) {
         DetectTagDataFree(de_ctx, td);
         return -1;
     }
-
-    sm->type = DETECT_TAG;
-    sm->ctx = (SigMatchCtx *)td;
-
-    /* Append it to the list of tags */
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_TMATCH);
     return 0;
 }
 
diff --git a/src/detect-tcp-ack.c b/src/detect-tcp-ack.c
index d34911120b5c..b2e35ca813d2 100644
--- a/src/detect-tcp-ack.c
+++ b/src/detect-tcp-ack.c
@@ -110,24 +110,19 @@ static int DetectAckMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectAckSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
     DetectAckData *data = NULL;
-    SigMatch *sm = NULL;
 
     data = SCMalloc(sizeof(DetectAckData));
     if (unlikely(data == NULL))
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_ACK;
-
     if (StringParseUint32(&data->ack, 10, 0, optstr) < 0) {
         goto error;
     }
-    sm->ctx = (SigMatchCtx*)data;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ACK, (SigMatchCtx *)data, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -135,8 +130,6 @@ static int DetectAckSetup(DetectEngineCtx *de_ctx, Signature *s, const char *opt
 error:
     if (data)
         SCFree(data);
-    if (sm)
-        SigMatchFree(de_ctx, sm);
     return -1;
 
 }
diff --git a/src/detect-tcp-flags.c b/src/detect-tcp-flags.c
index 183ae96f6792..04caed0209a2 100644
--- a/src/detect-tcp-flags.c
+++ b/src/detect-tcp-flags.c
@@ -480,27 +480,22 @@ static DetectFlagsData *DetectFlagsParse (const char *rawstr)
 static int DetectFlagsSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectFlagsData *de = NULL;
-    SigMatch *sm = NULL;
 
     de = DetectFlagsParse(rawstr);
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_FLAGS, (SigMatchCtx *)de, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         goto error;
-
-    sm->type = DETECT_FLAGS;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (de) SCFree(de);
-    if (sm) SCFree(sm);
+    if (de)
+        SCFree(de);
     return -1;
 }
 
diff --git a/src/detect-tcp-seq.c b/src/detect-tcp-seq.c
index 20d7c6d97d57..0a34f5633de9 100644
--- a/src/detect-tcp-seq.c
+++ b/src/detect-tcp-seq.c
@@ -105,24 +105,19 @@ static int DetectSeqMatch(DetectEngineThreadCtx *det_ctx,
 static int DetectSeqSetup (DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
 {
     DetectSeqData *data = NULL;
-    SigMatch *sm = NULL;
 
     data = SCMalloc(sizeof(DetectSeqData));
     if (unlikely(data == NULL))
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_SEQ;
-
     if (StringParseUint32(&data->seq, 10, 0, optstr) < 0) {
         goto error;
     }
-    sm->ctx = (SigMatchCtx*)data;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_SEQ, (SigMatchCtx *)data, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
@@ -130,8 +125,6 @@ static int DetectSeqSetup (DetectEngineCtx *de_ctx, Signature *s, const char *op
 error:
     if (data)
         SCFree(data);
-    if (sm)
-        SigMatchFree(de_ctx, sm);
     return -1;
 
 }
diff --git a/src/detect-tcp-window.c b/src/detect-tcp-window.c
index 3a8526b890f3..c0a7bb7e1b05 100644
--- a/src/detect-tcp-window.c
+++ b/src/detect-tcp-window.c
@@ -181,28 +181,24 @@ static DetectWindowData *DetectWindowParse(DetectEngineCtx *de_ctx, const char *
 static int DetectWindowSetup (DetectEngineCtx *de_ctx, Signature *s, const char *windowstr)
 {
     DetectWindowData *wd = NULL;
-    SigMatch *sm = NULL;
 
     wd = DetectWindowParse(de_ctx, windowstr);
     if (wd == NULL) goto error;
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_WINDOW;
-    sm->ctx = (SigMatchCtx *)wd;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_WINDOW, (SigMatchCtx *)wd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
+        goto error;
+    }
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
 
 error:
-    if (wd != NULL) DetectWindowFree(de_ctx, wd);
-    if (sm != NULL) SCFree(sm);
+    if (wd != NULL)
+        DetectWindowFree(de_ctx, wd);
     return -1;
 
 }
diff --git a/src/detect-tcpmss.c b/src/detect-tcpmss.c
index 1ed04d349943..c04a9be09ecc 100644
--- a/src/detect-tcpmss.c
+++ b/src/detect-tcpmss.c
@@ -105,16 +105,11 @@ static int DetectTcpmssSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     if (tcpmssd == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_TCPMSS, (SigMatchCtx *)tcpmssd, DETECT_SM_LIST_MATCH) == NULL) {
         DetectTcpmssFree(de_ctx, tcpmssd);
         return -1;
     }
-
-    sm->type = DETECT_TCPMSS;
-    sm->ctx = (SigMatchCtx *)tcpmssd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-template.c b/src/detect-template.c
index 693e4bde821b..5e09170d82a3 100644
--- a/src/detect-template.c
+++ b/src/detect-template.c
@@ -192,16 +192,11 @@ static int DetectTemplateSetup (DetectEngineCtx *de_ctx, Signature *s, const cha
     if (templated == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_TEMPLATE, (SigMatchCtx *)templated,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectTemplateFree(de_ctx, templated);
         return -1;
     }
-
-    sm->type = DETECT_TEMPLATE;
-    sm->ctx = (void *)templated;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-template2.c b/src/detect-template2.c
index 7b554599a362..df93a535e6b9 100644
--- a/src/detect-template2.c
+++ b/src/detect-template2.c
@@ -112,16 +112,11 @@ static int DetectTemplate2Setup (DetectEngineCtx *de_ctx, Signature *s, const ch
     if (template2d == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_TEMPLATE2, (SigMatchCtx *)template2d,
+                DETECT_SM_LIST_MATCH) == NULL) {
         DetectTemplate2Free(de_ctx, template2d);
         return -1;
     }
-
-    sm->type = DETECT_TEMPLATE2;
-    sm->ctx = (SigMatchCtx *)template2d;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
 
     return 0;
diff --git a/src/detect-threshold.c b/src/detect-threshold.c
index 95a09633b2ba..768447204267 100644
--- a/src/detect-threshold.c
+++ b/src/detect-threshold.c
@@ -239,7 +239,6 @@ static DetectThresholdData *DetectThresholdParse(const char *rawstr)
 static int DetectThresholdSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
     DetectThresholdData *de = NULL;
-    SigMatch *sm = NULL;
     SigMatch *tmpm = NULL;
 
     /* checks if there is a previous instance of detection_filter */
@@ -259,20 +258,16 @@ static int DetectThresholdSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
     if (de == NULL)
         goto error;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_THRESHOLD, (SigMatchCtx *)de, DETECT_SM_LIST_THRESHOLD) == NULL) {
         goto error;
-
-    sm->type = DETECT_THRESHOLD;
-    sm->ctx = (SigMatchCtx *)de;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_THRESHOLD);
+    }
 
     return 0;
 
 error:
-    if (de) SCFree(de);
-    if (sm) SCFree(sm);
+    if (de)
+        SCFree(de);
     return -1;
 }
 
diff --git a/src/detect-tls-cert-validity.c b/src/detect-tls-cert-validity.c
index 63939b849286..3720d287db5c 100644
--- a/src/detect-tls-cert-validity.c
+++ b/src/detect-tls-cert-validity.c
@@ -414,7 +414,6 @@ static int DetectTlsExpiredSetup (DetectEngineCtx *de_ctx, Signature *s,
                                   const char *rawstr)
 {
     DetectTlsValidityData *dd = NULL;
-    SigMatch *sm = NULL;
 
     SCLogDebug("\'%s\'", rawstr);
 
@@ -429,25 +428,20 @@ static int DetectTlsExpiredSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
 
     dd->mode = DETECT_TLS_VALIDITY_EX;
     dd->type = DETECT_TLS_TYPE_NOTAFTER;
     dd->epoch = 0;
     dd->epoch2 = 0;
 
-    sm->type = DETECT_AL_TLS_EXPIRED;
-    sm->ctx = (void *)dd;
-
-    SigMatchAppendSMToList(s, sm, g_tls_validity_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_EXPIRED, (SigMatchCtx *)dd,
+                g_tls_validity_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     DetectTlsValidityFree(de_ctx, dd);
-    if (sm)
-        SCFree(sm);
     return -1;
 }
 
@@ -465,7 +459,6 @@ static int DetectTlsValidSetup (DetectEngineCtx *de_ctx, Signature *s,
                                 const char *rawstr)
 {
     DetectTlsValidityData *dd = NULL;
-    SigMatch *sm = NULL;
 
     SCLogDebug("\'%s\'", rawstr);
 
@@ -480,25 +473,20 @@ static int DetectTlsValidSetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
 
     dd->mode = DETECT_TLS_VALIDITY_VA;
     dd->type = DETECT_TLS_TYPE_NOTAFTER;
     dd->epoch = 0;
     dd->epoch2 = 0;
 
-    sm->type = DETECT_AL_TLS_VALID;
-    sm->ctx = (void *)dd;
-
-    SigMatchAppendSMToList(s, sm, g_tls_validity_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_VALID, (SigMatchCtx *)dd,
+                g_tls_validity_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     DetectTlsValidityFree(de_ctx, dd);
-    if (sm)
-        SCFree(sm);
     return -1;
 }
 
@@ -555,7 +543,6 @@ static int DetectTlsValiditySetup (DetectEngineCtx *de_ctx, Signature *s,
                                    const char *rawstr, uint8_t type)
 {
     DetectTlsValidityData *dd = NULL;
-    SigMatch *sm = NULL;
 
     SCLogDebug("\'%s\'", rawstr);
 
@@ -570,31 +557,25 @@ static int DetectTlsValiditySetup (DetectEngineCtx *de_ctx, Signature *s,
 
     /* okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
 
     if (type == DETECT_TLS_TYPE_NOTBEFORE) {
         dd->type = DETECT_TLS_TYPE_NOTBEFORE;
-        sm->type = DETECT_AL_TLS_NOTBEFORE;
     }
     else if (type == DETECT_TLS_TYPE_NOTAFTER) {
         dd->type = DETECT_TLS_TYPE_NOTAFTER;
-        sm->type = DETECT_AL_TLS_NOTAFTER;
     }
     else {
         goto error;
     }
 
-    sm->ctx = (void *)dd;
-
-    SigMatchAppendSMToList(s, sm, g_tls_validity_buffer_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_NOTAFTER, (SigMatchCtx *)dd,
+                g_tls_validity_buffer_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     DetectTlsValidityFree(de_ctx, dd);
-    if (sm)
-        SCFree(sm);
     return -1;
 }
 
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index cccc695c91cf..e994c9e2b0e8 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -341,15 +341,11 @@ static int DetectTLSCertChainLenSetup(DetectEngineCtx *de_ctx, Signature *s, con
         return -1;
     }
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, KEYWORD_ID, (SigMatchCtx *)dd, g_tls_cert_buffer_id) ==
+            NULL) {
         rs_detect_u32_free(dd);
         return -1;
     }
-    sm->type = KEYWORD_ID;
-    sm->ctx = (void *)dd;
-
-    SigMatchAppendSMToList(s, sm, g_tls_cert_buffer_id);
     return 0;
 }
 
diff --git a/src/detect-tls-version.c b/src/detect-tls-version.c
index cba1f55e95d1..f3a119d5a226 100644
--- a/src/detect-tls-version.c
+++ b/src/detect-tls-version.c
@@ -232,7 +232,6 @@ static DetectTlsVersionData *DetectTlsVersionParse (DetectEngineCtx *de_ctx, con
 static int DetectTlsVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectTlsVersionData *tls = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
@@ -243,22 +242,17 @@ static int DetectTlsVersionSetup (DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_TLS_VERSION;
-    sm->ctx = (void *)tls;
 
-    SigMatchAppendSMToList(s, sm, g_tls_generic_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_VERSION, (SigMatchCtx *)tls,
+                g_tls_generic_list_id) == NULL) {
+        goto error;
+    }
 
     return 0;
 
 error:
     if (tls != NULL)
         DetectTlsVersionFree(de_ctx, tls);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
diff --git a/src/detect-tls.c b/src/detect-tls.c
index 71e45696cd9c..8a9c98fac795 100644
--- a/src/detect-tls.c
+++ b/src/detect-tls.c
@@ -298,7 +298,6 @@ static DetectTlsData *DetectTlsSubjectParse (DetectEngineCtx *de_ctx, const char
 static int DetectTlsSubjectSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectTlsData *tls = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
@@ -309,21 +308,16 @@ static int DetectTlsSubjectSetup (DetectEngineCtx *de_ctx, Signature *s, const c
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_TLS_SUBJECT;
-    sm->ctx = (void *)tls;
 
-    SigMatchAppendSMToList(s, sm, g_tls_cert_list_id);
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_AL_TLS_SUBJECT, (SigMatchCtx *)tls, g_tls_cert_list_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (tls != NULL)
         DetectTlsSubjectFree(de_ctx, tls);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
@@ -494,7 +488,6 @@ static DetectTlsData *DetectTlsIssuerDNParse(DetectEngineCtx *de_ctx, const char
 static int DetectTlsIssuerDNSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
     DetectTlsData *tls = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
@@ -505,21 +498,16 @@ static int DetectTlsIssuerDNSetup (DetectEngineCtx *de_ctx, Signature *s, const
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_AL_TLS_ISSUERDN;
-    sm->ctx = (void *)tls;
 
-    SigMatchAppendSMToList(s, sm, g_tls_cert_list_id);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_ISSUERDN, (SigMatchCtx *)tls,
+                g_tls_cert_list_id) == NULL) {
+        goto error;
+    }
     return 0;
 
 error:
     if (tls != NULL)
         DetectTlsIssuerDNFree(de_ctx, tls);
-    if (sm != NULL)
-        SCFree(sm);
     return -1;
 
 }
@@ -594,19 +582,16 @@ static void DetectTlsFingerprintFree(DetectEngineCtx *de_ctx, void *ptr)
  */
 static int DetectTlsStoreSetup (DetectEngineCtx *de_ctx, Signature *s, const char *str)
 {
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_TLS) != 0)
         return -1;
 
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        return -1;
-
-    sm->type = DETECT_AL_TLS_STORE;
     s->flags |= SIG_FLAG_TLSSTORE;
 
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_TLS_STORE, NULL, DETECT_SM_LIST_POSTMATCH) ==
+            NULL) {
+        return -1;
+    }
     return 0;
 }
 
diff --git a/src/detect-tos.c b/src/detect-tos.c
index 002ff9c927c9..e8c1fe6f62d7 100644
--- a/src/detect-tos.c
+++ b/src/detect-tos.c
@@ -185,16 +185,11 @@ static int DetectTosSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg
     if (tosd == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_TOS, (SigMatchCtx *)tosd, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         DetectTosFree(de_ctx, tosd);
         return -1;
     }
-
-    sm->type = DETECT_TOS;
-    sm->ctx = (SigMatchCtx *)tosd;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
     return 0;
 }
diff --git a/src/detect-ttl.c b/src/detect-ttl.c
index 02309b4f04de..6d0a25311803 100644
--- a/src/detect-ttl.c
+++ b/src/detect-ttl.c
@@ -116,16 +116,11 @@ static int DetectTtlSetup (DetectEngineCtx *de_ctx, Signature *s, const char *tt
     if (ttld == NULL)
         return -1;
 
-    SigMatch *sm = SigMatchAlloc();
-    if (sm == NULL) {
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_TTL, (SigMatchCtx *)ttld, DETECT_SM_LIST_MATCH) ==
+            NULL) {
         DetectTtlFree(de_ctx, ttld);
         return -1;
     }
-
-    sm->type = DETECT_TTL;
-    sm->ctx = (SigMatchCtx *)ttld;
-
-    SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
     s->flags |= SIG_FLAG_REQUIRE_PACKET;
     return 0;
 }
diff --git a/src/detect-urilen.c b/src/detect-urilen.c
index a0736b3bf1d5..67acda5d5c58 100644
--- a/src/detect-urilen.c
+++ b/src/detect-urilen.c
@@ -102,7 +102,6 @@ static int DetectUrilenSetup (DetectEngineCtx *de_ctx, Signature *s, const char
 {
     SCEnter();
     DetectUrilenData *urilend = NULL;
-    SigMatch *sm = NULL;
 
     if (DetectSignatureSetAppProto(s, ALPROTO_HTTP) != 0)
         return -1;
@@ -110,16 +109,18 @@ static int DetectUrilenSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     urilend = DetectUrilenParse(urilenstr);
     if (urilend == NULL)
         goto error;
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-    sm->type = DETECT_AL_URILEN;
-    sm->ctx = (void *)urilend;
 
-    if (urilend->raw_buffer)
-        SigMatchAppendSMToList(s, sm, g_http_raw_uri_buffer_id);
-    else
-        SigMatchAppendSMToList(s, sm, g_http_uri_buffer_id);
+    if (urilend->raw_buffer) {
+        if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_URILEN, (SigMatchCtx *)urilend,
+                    g_http_raw_uri_buffer_id) == NULL) {
+            goto error;
+        }
+    } else {
+        if (SigMatchAppendSMToList(de_ctx, s, DETECT_AL_URILEN, (SigMatchCtx *)urilend,
+                    g_http_uri_buffer_id) == NULL) {
+            goto error;
+        }
+    }
 
     SCReturnInt(0);
 
diff --git a/src/detect-xbits.c b/src/detect-xbits.c
index 4fae4414819f..92b86ba9da0b 100644
--- a/src/detect-xbits.c
+++ b/src/detect-xbits.c
@@ -335,7 +335,6 @@ static int DetectXbitParse(DetectEngineCtx *de_ctx,
 
 int DetectXbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 {
-    SigMatch *sm = NULL;
     DetectXbitsData *cd = NULL;
 
     int result = DetectXbitParse(de_ctx, rawstr, &cd);
@@ -349,12 +348,6 @@ int DetectXbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-    sm = SigMatchAlloc();
-    if (sm == NULL)
-        goto error;
-
-    sm->type = DETECT_XBITS;
-    sm->ctx = (void *)cd;
 
     switch (cd->cmd) {
         /* case DETECT_XBITS_CMD_NOALERT can't happen here */
@@ -362,14 +355,20 @@ int DetectXbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
         case DETECT_XBITS_CMD_ISNOTSET:
         case DETECT_XBITS_CMD_ISSET:
             /* checks, so packet list */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_MATCH);
+            if (SigMatchAppendSMToList(
+                        de_ctx, s, DETECT_XBITS, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
+                goto error;
+            }
             break;
 
         case DETECT_XBITS_CMD_SET:
         case DETECT_XBITS_CMD_UNSET:
         case DETECT_XBITS_CMD_TOGGLE:
             /* modifiers, only run when entire sig has matched */
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_POSTMATCH);
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_XBITS, (SigMatchCtx *)cd,
+                        DETECT_SM_LIST_POSTMATCH) == NULL) {
+                goto error;
+            }
             break;
     }
 
diff --git a/src/detect.h b/src/detect.h
index ced030067070..a3cd161fa654 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -287,7 +287,6 @@ typedef struct DetectPort_ {
     BIT_U32(8) /**< priority is explicitly set by the priority keyword */
 #define SIG_FLAG_INIT_FILEDATA              BIT_U32(9)  /**< signature has filedata keyword */
 #define SIG_FLAG_INIT_JA3                   BIT_U32(10) /**< signature has ja3 keyword */
-#define SIG_FLAG_INIT_OVERFLOW              BIT_U32(11) /**< signature has overflown buffers */
 
 /* signature mask flags */
 /** \note: additions should be added to the rule analyzer as well */
diff --git a/src/util-threshold-config.c b/src/util-threshold-config.c
index 0e5caf83265f..b093467b398a 100644
--- a/src/util-threshold-config.c
+++ b/src/util-threshold-config.c
@@ -224,7 +224,6 @@ static int SetupSuppressRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid,
         const char *th_ip)
 {
     Signature *s = NULL;
-    SigMatch *sm = NULL;
     DetectThresholdData *de = NULL;
 
     BUG_ON(parsed_type != TYPE_SUPPRESS);
@@ -266,15 +265,10 @@ static int SetupSuppressRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid,
             if (unlikely(de == NULL))
                 goto error;
 
-            sm = SigMatchAlloc();
-            if (sm == NULL) {
-                SCLogError("Error allocating SigMatch");
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_THRESHOLD, (SigMatchCtx *)de,
+                        DETECT_SM_LIST_SUPPRESS) == NULL) {
                 goto error;
             }
-
-            sm->type = DETECT_THRESHOLD;
-            sm->ctx = (void *)de;
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_SUPPRESS);
         }
     } else if (id == 0 && gid > 0)    {
         if (parsed_track == TRACK_RULE) {
@@ -295,16 +289,10 @@ static int SetupSuppressRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid,
             if (unlikely(de == NULL))
                 goto error;
 
-            sm = SigMatchAlloc();
-            if (sm == NULL) {
-                SCLogError("Error allocating SigMatch");
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_THRESHOLD, (SigMatchCtx *)de,
+                        DETECT_SM_LIST_SUPPRESS) == NULL) {
                 goto error;
             }
-
-            sm->type = DETECT_THRESHOLD;
-            sm->ctx = (void *)de;
-
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_SUPPRESS);
         }
     } else if (id > 0 && gid == 0) {
         SCLogError("Can't use a event config that has "
@@ -327,16 +315,10 @@ static int SetupSuppressRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid,
             if (unlikely(de == NULL))
                 goto error;
 
-            sm = SigMatchAlloc();
-            if (sm == NULL) {
-                SCLogError("Error allocating SigMatch");
+            if (SigMatchAppendSMToList(de_ctx, s, DETECT_THRESHOLD, (SigMatchCtx *)de,
+                        DETECT_SM_LIST_SUPPRESS) == NULL) {
                 goto error;
             }
-
-            sm->type = DETECT_THRESHOLD;
-            sm->ctx = (void *)de;
-
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_SUPPRESS);
         }
     }
 
@@ -377,8 +359,7 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
     /* Install it */
     if (id == 0 && gid == 0) {
         for (s = de_ctx->sig_list; s != NULL; s = s->next) {
-            sm = DetectGetLastSMByListId(s,
-                    DETECT_SM_LIST_THRESHOLD, DETECT_THRESHOLD, -1);
+            sm = DetectGetLastSMByListId(s, DETECT_SM_LIST_THRESHOLD, DETECT_THRESHOLD, -1);
             if (sm != NULL) {
                 SCLogWarning("signature sid:%" PRIu32 " has "
                              "an event var set.  The signature event var is "
@@ -411,19 +392,14 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
             de->new_action = parsed_new_action;
             de->timeout = parsed_timeout;
 
-            sm = SigMatchAlloc();
-            if (sm == NULL) {
-                SCLogError("Error allocating SigMatch");
-                goto error;
-            }
-
+            uint16_t smtype = DETECT_THRESHOLD;
             if (parsed_type == TYPE_RATE)
-                sm->type = DETECT_DETECTION_FILTER;
-            else
-                sm->type = DETECT_THRESHOLD;
-            sm->ctx = (void *)de;
+                smtype = DETECT_DETECTION_FILTER;
 
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_THRESHOLD);
+            if (SigMatchAppendSMToList(
+                        de_ctx, s, smtype, (SigMatchCtx *)de, DETECT_SM_LIST_THRESHOLD) == NULL) {
+                goto error;
+            }
         }
 
     } else if (id == 0 && gid > 0) {
@@ -452,19 +428,14 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
                 de->new_action = parsed_new_action;
                 de->timeout = parsed_timeout;
 
-                sm = SigMatchAlloc();
-                if (sm == NULL) {
-                    SCLogError("Error allocating SigMatch");
-                    goto error;
-                }
-
+                uint16_t smtype = DETECT_THRESHOLD;
                 if (parsed_type == TYPE_RATE)
-                    sm->type = DETECT_DETECTION_FILTER;
-                else
-                    sm->type = DETECT_THRESHOLD;
-                sm->ctx = (void *)de;
+                    smtype = DETECT_DETECTION_FILTER;
 
-                SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_THRESHOLD);
+                if (SigMatchAppendSMToList(de_ctx, s, smtype, (SigMatchCtx *)de,
+                            DETECT_SM_LIST_THRESHOLD) == NULL) {
+                    goto error;
+                }
             }
         }
     } else if (id > 0 && gid == 0) {
@@ -525,19 +496,14 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
             de->new_action = parsed_new_action;
             de->timeout = parsed_timeout;
 
-            sm = SigMatchAlloc();
-            if (sm == NULL) {
-                SCLogError("Error allocating SigMatch");
-                goto error;
-            }
-
+            uint16_t smtype = DETECT_THRESHOLD;
             if (parsed_type == TYPE_RATE)
-                sm->type = DETECT_DETECTION_FILTER;
-            else
-                sm->type = DETECT_THRESHOLD;
-            sm->ctx = (void *)de;
+                smtype = DETECT_DETECTION_FILTER;
 
-            SigMatchAppendSMToList(s, sm, DETECT_SM_LIST_THRESHOLD);
+            if (SigMatchAppendSMToList(
+                        de_ctx, s, smtype, (SigMatchCtx *)de, DETECT_SM_LIST_THRESHOLD) == NULL) {
+                goto error;
+            }
         }
     }
 end:

From 193e0ea1a9aab2f735399ee771340337ca5cfcb1 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Sat, 29 Jul 2023 10:03:39 -0400
Subject: [PATCH 158/462] memory/alloc: Use SCCalloc instead of malloc/memset

---
 src/alert-debuglog.c                    |  6 ++--
 src/alert-fastlog.c                     |  3 +-
 src/alert-syslog.c                      |  7 ++---
 src/app-layer-detect-proto.c            | 17 +++++-------
 src/app-layer-enip-common.c             |  2 --
 src/app-layer-enip.c                    |  5 +---
 src/app-layer-events.c                  |  3 +-
 src/app-layer-ftp.c                     |  3 +-
 src/app-layer-htp.c                     |  3 +-
 src/app-layer-parser.c                  |  3 +-
 src/app-layer-smtp.c                    |  9 ++----
 src/app-layer-ssl.c                     |  3 +-
 src/app-layer.c                         |  3 +-
 src/decode.c                            |  3 +-
 src/defrag-hash.c                       |  4 +--
 src/detect-byte-extract.c               |  3 +-
 src/detect-csum.c                       | 21 +++++---------
 src/detect-detection-filter.c           |  4 +--
 src/detect-engine-build.c               |  3 +-
 src/detect-engine-iponly.c              | 12 +++-----
 src/detect-engine-siggroup.c            | 21 ++++----------
 src/detect-engine-sigorder.c            |  6 ++--
 src/detect-engine-state.c               |  6 ++--
 src/detect-engine-tag.c                 |  3 +-
 src/detect-engine.c                     | 21 +++++---------
 src/detect-fast-pattern.c               |  6 ++--
 src/detect-file-hash-common.c           |  4 +--
 src/detect-filestore.c                  |  3 +-
 src/detect-flowvar.c                    |  6 ++--
 src/detect-fragbits.c                   |  4 +--
 src/detect-geoip.c                      |  4 +--
 src/detect-lua.c                        |  7 ++---
 src/detect-parse.c                      |  9 ++----
 src/detect-ssh-proto-version.c          |  3 +-
 src/detect-tcp-flags.c                  |  3 +-
 src/detect-threshold.c                  |  4 +--
 src/flow-util.c                         |  3 +-
 src/host.c                              |  4 +--
 src/ippair.c                            |  4 +--
 src/log-httplog.c                       |  6 ++--
 src/log-pcap.c                          |  6 ++--
 src/log-stats.c                         |  6 ++--
 src/log-tcp-data.c                      |  6 ++--
 src/log-tlslog.c                        |  4 +--
 src/log-tlsstore.c                      |  3 +-
 src/output-file.c                       |  8 ++----
 src/output-filedata.c                   |  9 ++----
 src/output-filestore.c                  |  3 +-
 src/output-flow.c                       | 11 +++-----
 src/output-json-alert.c                 |  3 +-
 src/output-json-dns.c                   |  3 +-
 src/output-json-frame.c                 |  3 +-
 src/output-json-http2.c                 |  2 +-
 src/output-json-pgsql.c                 |  2 +-
 src/output-lua.c                        |  6 ++--
 src/output-packet.c                     | 11 +++-----
 src/output-stats.c                      | 11 +++-----
 src/output-streaming.c                  | 11 +++-----
 src/output-tx.c                         | 11 +++-----
 src/reputation.c                        | 10 ++-----
 src/runmode-unix-socket.c               |  3 +-
 src/source-af-packet.c                  |  6 ++--
 src/source-af-xdp.c                     |  3 +-
 src/source-erf-dag.c                    |  4 +--
 src/source-erf-file.c                   |  3 +-
 src/source-ipfw.c                       |  4 +--
 src/source-nflog.c                      |  3 +-
 src/source-pcap-file-directory-helper.c |  3 +-
 src/source-pcap-file.c                  |  9 ++----
 src/source-pfring.c                     |  3 +-
 src/stream-tcp-reassemble.c             |  4 +--
 src/stream-tcp.c                        |  3 +-
 src/suricata.c                          |  6 ++--
 src/tm-threads.c                        |  6 ++--
 src/tmqh-flow.c                         |  6 ++--
 src/util-bloomfilter-counting.c         |  6 ++--
 src/util-bloomfilter.c                  |  6 ++--
 src/util-buffer.c                       |  3 +-
 src/util-classification-config.c        |  3 +-
 src/util-debug-filters.c                | 26 ++++++-----------
 src/util-decode-mime.c                  | 25 ++++++-----------
 src/util-file.c                         |  6 ++--
 src/util-fmemopen.c                     |  3 +-
 src/util-hash.c                         | 11 +++-----
 src/util-hashlist.c                     |  9 ++----
 src/util-hyperscan.c                    |  3 +-
 src/util-mpm-ac-bs.c                    | 30 ++++++--------------
 src/util-mpm-ac-ks.c                    | 19 ++++---------
 src/util-mpm-ac.c                       | 16 ++++-------
 src/util-mpm-hs.c                       | 37 ++++++++-----------------
 src/util-mpm.c                          |  6 ++--
 src/util-pool.c                         |  7 ++---
 src/util-profiling-keywords.c           | 17 ++++--------
 src/util-profiling-prefilter.c          | 10 ++-----
 src/util-profiling-rules.c              | 11 ++------
 src/util-radix-tree.c                   | 18 ++++--------
 src/util-reference-config.c             |  3 +-
 src/util-rohash.c                       |  9 ++----
 src/util-runmodes.c                     |  3 +-
 src/util-spm-bm.c                       | 12 +++-----
 src/util-spm-hs.c                       | 12 +++-----
 src/util-storage.c                      | 15 ++++------
 src/util-threshold-config.c             |  9 ++----
 103 files changed, 252 insertions(+), 522 deletions(-)

diff --git a/src/alert-debuglog.c b/src/alert-debuglog.c
index e0a0802051e1..aaba84cc6ea6 100644
--- a/src/alert-debuglog.c
+++ b/src/alert-debuglog.c
@@ -374,10 +374,9 @@ static TmEcode AlertDebugLogDecoderEvent(ThreadVars *tv, const Packet *p, void *
 
 static TmEcode AlertDebugLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    AlertDebugLogThread *aft = SCMalloc(sizeof(AlertDebugLogThread));
+    AlertDebugLogThread *aft = SCCalloc(1, sizeof(AlertDebugLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(AlertDebugLogThread));
 
     if(initdata == NULL)
     {
@@ -447,11 +446,10 @@ static OutputInitResult AlertDebugLogInitCtx(ConfNode *conf)
         goto error;
     }
 
-    OutputCtx *output_ctx = SCMalloc(sizeof(OutputCtx));
+    OutputCtx *output_ctx = SCCalloc(1, sizeof(OutputCtx));
     if (unlikely(output_ctx == NULL))
         goto error;
 
-    memset(output_ctx, 0x00, sizeof(OutputCtx));
     output_ctx->data = file_ctx;
     output_ctx->DeInit = AlertDebugLogDeInitCtx;
 
diff --git a/src/alert-fastlog.c b/src/alert-fastlog.c
index 8cd4a3c58aa8..7b4a22a85954 100644
--- a/src/alert-fastlog.c
+++ b/src/alert-fastlog.c
@@ -188,10 +188,9 @@ int AlertFastLogger(ThreadVars *tv, void *data, const Packet *p)
 
 TmEcode AlertFastLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    AlertFastLogThread *aft = SCMalloc(sizeof(AlertFastLogThread));
+    AlertFastLogThread *aft = SCCalloc(1, sizeof(AlertFastLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(AlertFastLogThread));
     if(initdata == NULL)
     {
         SCLogDebug("Error getting context for AlertFastLog.  \"initdata\" argument NULL");
diff --git a/src/alert-syslog.c b/src/alert-syslog.c
index df0be1a94a63..fd1742adb01f 100644
--- a/src/alert-syslog.c
+++ b/src/alert-syslog.c
@@ -121,13 +121,12 @@ static OutputInitResult AlertSyslogInitCtx(ConfNode *conf)
 
     openlog(ident, LOG_PID|LOG_NDELAY, facility);
 
-    OutputCtx *output_ctx = SCMalloc(sizeof(OutputCtx));
+    OutputCtx *output_ctx = SCCalloc(1, sizeof(OutputCtx));
     if (unlikely(output_ctx == NULL)) {
         SCLogDebug("could not create new OutputCtx");
         LogFileFreeCtx(logfile_ctx);
         return result;
     }
-    memset(output_ctx, 0x00, sizeof(OutputCtx));
 
     output_ctx->data = logfile_ctx;
     output_ctx->DeInit = AlertSyslogDeInitCtx;
@@ -155,12 +154,10 @@ static TmEcode AlertSyslogThreadInit(ThreadVars *t, const void *initdata, void *
         return TM_ECODE_FAILED;
     }
 
-    AlertSyslogThread *ast = SCMalloc(sizeof(AlertSyslogThread));
+    AlertSyslogThread *ast = SCCalloc(1, sizeof(AlertSyslogThread));
     if (unlikely(ast == NULL))
         return TM_ECODE_FAILED;
 
-    memset(ast, 0, sizeof(AlertSyslogThread));
-
     /** Use the Output Context (file pointer and mutex) */
     ast->file_ctx = ((OutputCtx *)initdata)->data;
 
diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index 77f3c648c0da..cb31b4d6b969 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -693,11 +693,11 @@ static AppLayerProtoDetectProbingParserElement *AppLayerProtoDetectProbingParser
 {
     SCEnter();
 
-    AppLayerProtoDetectProbingParserElement *p = SCMalloc(sizeof(AppLayerProtoDetectProbingParserElement));
+    AppLayerProtoDetectProbingParserElement *p =
+            SCCalloc(1, sizeof(AppLayerProtoDetectProbingParserElement));
     if (unlikely(p == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(p, 0, sizeof(AppLayerProtoDetectProbingParserElement));
 
     SCReturnPtr(p, "AppLayerProtoDetectProbingParserElement");
 }
@@ -714,11 +714,11 @@ static AppLayerProtoDetectProbingParserPort *AppLayerProtoDetectProbingParserPor
 {
     SCEnter();
 
-    AppLayerProtoDetectProbingParserPort *p = SCMalloc(sizeof(AppLayerProtoDetectProbingParserPort));
+    AppLayerProtoDetectProbingParserPort *p =
+            SCCalloc(1, sizeof(AppLayerProtoDetectProbingParserPort));
     if (unlikely(p == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(p, 0, sizeof(AppLayerProtoDetectProbingParserPort));
 
     SCReturnPtr(p, "AppLayerProtoDetectProbingParserPort");
 }
@@ -752,11 +752,10 @@ static AppLayerProtoDetectProbingParser *AppLayerProtoDetectProbingParserAlloc(v
 {
     SCEnter();
 
-    AppLayerProtoDetectProbingParser *p = SCMalloc(sizeof(AppLayerProtoDetectProbingParser));
+    AppLayerProtoDetectProbingParser *p = SCCalloc(1, sizeof(AppLayerProtoDetectProbingParser));
     if (unlikely(p == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(p, 0, sizeof(AppLayerProtoDetectProbingParser));
 
     SCReturnPtr(p, "AppLayerProtoDetectProbingParser");
 }
@@ -1269,10 +1268,9 @@ static int AppLayerProtoDetectPMMapSignatures(AppLayerProtoDetectPMCtx *ctx)
     int mpm_ret;
     SigIntId id = 0;
 
-    ctx->map = SCMalloc(ctx->max_sig_id * sizeof(AppLayerProtoDetectPMSignature *));
+    ctx->map = SCCalloc(1, ctx->max_sig_id * sizeof(AppLayerProtoDetectPMSignature *));
     if (ctx->map == NULL)
         goto error;
-    memset(ctx->map, 0, ctx->max_sig_id * sizeof(AppLayerProtoDetectPMSignature *));
 
     /* add an array indexed by rule id to look up the sig */
     for (s = ctx->head; s != NULL; ) {
@@ -1985,10 +1983,9 @@ AppLayerProtoDetectThreadCtx *AppLayerProtoDetectGetCtxThread(void)
         }
     }
 
-    alpd_tctx = SCMalloc(sizeof(*alpd_tctx));
+    alpd_tctx = SCCalloc(1, sizeof(*alpd_tctx));
     if (alpd_tctx == NULL)
         goto error;
-    memset(alpd_tctx, 0, sizeof(*alpd_tctx));
 
     /* Get the max pat id for all the mpm ctxs. */
     if (PmqSetup(&alpd_tctx->pmq) < 0)
diff --git a/src/app-layer-enip-common.c b/src/app-layer-enip-common.c
index 305eb8312b0a..0608080e21f1 100644
--- a/src/app-layer-enip-common.c
+++ b/src/app-layer-enip-common.c
@@ -140,8 +140,6 @@ static CIPServiceEntry *CIPServiceAlloc(ENIPTransaction *tx)
     if (unlikely(svc == NULL))
         return NULL;
 
-    memset(svc, 0x00, sizeof(CIPServiceEntry));
-
     TAILQ_INIT(&svc->segment_list);
     TAILQ_INIT(&svc->attrib_list);
 
diff --git a/src/app-layer-enip.c b/src/app-layer-enip.c
index 94c707c144b8..f059774b6da5 100644
--- a/src/app-layer-enip.c
+++ b/src/app-layer-enip.c
@@ -144,12 +144,10 @@ static int ENIPStateGetEventInfoById(int event_id, const char **event_name,
 static void *ENIPStateAlloc(void *orig_state, AppProto proto_orig)
 {
     SCLogDebug("ENIPStateAlloc");
-    void *s = SCMalloc(sizeof(ENIPState));
+    void *s = SCCalloc(1, sizeof(ENIPState));
     if (unlikely(s == NULL))
         return NULL;
 
-    memset(s, 0, sizeof(ENIPState));
-
     ENIPState *enip_state = (ENIPState *) s;
 
     TAILQ_INIT(&enip_state->tx_list);
@@ -242,7 +240,6 @@ static ENIPTransaction *ENIPTransactionAlloc(ENIPState *state)
     state->curr = tx;
     state->transaction_max++;
 
-    memset(tx, 0x00, sizeof(ENIPTransaction));
     TAILQ_INIT(&tx->service_list);
 
     tx->enip  = state;
diff --git a/src/app-layer-events.c b/src/app-layer-events.c
index e3d381135085..be5ee99ac290 100644
--- a/src/app-layer-events.c
+++ b/src/app-layer-events.c
@@ -91,11 +91,10 @@ int AppLayerGetPktEventInfo(const char *event_name, int *event_id)
 void AppLayerDecoderEventsSetEventRaw(AppLayerDecoderEvents **sevents, uint8_t event)
 {
     if (*sevents == NULL) {
-        AppLayerDecoderEvents *new_devents = SCMalloc(sizeof(AppLayerDecoderEvents));
+        AppLayerDecoderEvents *new_devents = SCCalloc(1, sizeof(AppLayerDecoderEvents));
         if (new_devents == NULL)
             return;
 
-        memset(new_devents, 0, sizeof(AppLayerDecoderEvents));
         *sevents = new_devents;
 
     }
diff --git a/src/app-layer-ftp.c b/src/app-layer-ftp.c
index f46a4a967e24..8925d6dd6a13 100644
--- a/src/app-layer-ftp.c
+++ b/src/app-layer-ftp.c
@@ -1216,11 +1216,10 @@ static AppLayerGetFileState FTPDataStateGetTxFiles(void *_state, void *tx, uint8
 
 static void FTPSetMpmState(void)
 {
-    ftp_mpm_ctx = SCMalloc(sizeof(MpmCtx));
+    ftp_mpm_ctx = SCCalloc(1, sizeof(MpmCtx));
     if (unlikely(ftp_mpm_ctx == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(ftp_mpm_ctx, 0, sizeof(MpmCtx));
     MpmInitCtx(ftp_mpm_ctx, FTP_MPM);
 
     uint32_t i = 0;
diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index 000fc88bbd0c..5d48611812c1 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -3011,10 +3011,9 @@ void HTPConfigure(void)
         SCLogDebug("LIBHTP server %s", s->name);
 
         HTPCfgRec *nextrec = cfglist.next;
-        HTPCfgRec *htprec = SCMalloc(sizeof(HTPCfgRec));
+        HTPCfgRec *htprec = SCCalloc(1, sizeof(HTPCfgRec));
         if (NULL == htprec)
             exit(EXIT_FAILURE);
-        memset(htprec, 0x00, sizeof(*htprec));
 
         cfglist.next = htprec;
 
diff --git a/src/app-layer-parser.c b/src/app-layer-parser.c
index 572e15f628cc..1f6066471757 100644
--- a/src/app-layer-parser.c
+++ b/src/app-layer-parser.c
@@ -238,10 +238,9 @@ AppLayerParserState *AppLayerParserStateAlloc(void)
 {
     SCEnter();
 
-    AppLayerParserState *pstate = (AppLayerParserState *)SCMalloc(sizeof(*pstate));
+    AppLayerParserState *pstate = (AppLayerParserState *)SCCalloc(1, sizeof(*pstate));
     if (pstate == NULL)
         goto end;
-    memset(pstate, 0, sizeof(*pstate));
 
  end:
     SCReturnPtr(pstate, "AppLayerParserState");
diff --git a/src/app-layer-smtp.c b/src/app-layer-smtp.c
index 7b921324ae8e..a4d94a94ded2 100644
--- a/src/app-layer-smtp.c
+++ b/src/app-layer-smtp.c
@@ -1521,10 +1521,9 @@ static AppLayerResult SMTPParseServerRecord(Flow *f, void *alstate, AppLayerPars
  */
 void *SMTPStateAlloc(void *orig_state, AppProto proto_orig)
 {
-    SMTPState *smtp_state = SCMalloc(sizeof(SMTPState));
+    SMTPState *smtp_state = SCCalloc(1, sizeof(SMTPState));
     if (unlikely(smtp_state == NULL))
         return NULL;
-    memset(smtp_state, 0, sizeof(SMTPState));
 
     smtp_state->cmds = SCMalloc(sizeof(uint8_t) *
                                 SMTP_COMMAND_BUFFER_STEPS);
@@ -1541,10 +1540,9 @@ void *SMTPStateAlloc(void *orig_state, AppProto proto_orig)
 
 static SMTPString *SMTPStringAlloc(void)
 {
-    SMTPString *smtp_string = SCMalloc(sizeof(SMTPString));
+    SMTPString *smtp_string = SCCalloc(1, sizeof(SMTPString));
     if (unlikely(smtp_string == NULL))
         return NULL;
-    memset(smtp_string, 0, sizeof(SMTPString));
 
     return smtp_string;
 }
@@ -1656,11 +1654,10 @@ static void SMTPStateFree(void *p)
 
 static void SMTPSetMpmState(void)
 {
-    smtp_mpm_ctx = SCMalloc(sizeof(MpmCtx));
+    smtp_mpm_ctx = SCCalloc(1, sizeof(MpmCtx));
     if (unlikely(smtp_mpm_ctx == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(smtp_mpm_ctx, 0, sizeof(MpmCtx));
     MpmInitCtx(smtp_mpm_ctx, SMTP_MPM);
 
     uint32_t i = 0;
diff --git a/src/app-layer-ssl.c b/src/app-layer-ssl.c
index 302225f1903d..cb094f3801ab 100644
--- a/src/app-layer-ssl.c
+++ b/src/app-layer-ssl.c
@@ -2646,10 +2646,9 @@ static AppLayerResult SSLParseServerRecord(Flow *f, void *alstate, AppLayerParse
  */
 static void *SSLStateAlloc(void *orig_state, AppProto proto_orig)
 {
-    SSLState *ssl_state = SCMalloc(sizeof(SSLState));
+    SSLState *ssl_state = SCCalloc(1, sizeof(SSLState));
     if (unlikely(ssl_state == NULL))
         return NULL;
-    memset(ssl_state, 0, sizeof(SSLState));
     ssl_state->client_connp.cert_log_flag = 0;
     ssl_state->server_connp.cert_log_flag = 0;
     memset(ssl_state->client_connp.random, 0, TLS_RANDOM_LEN);
diff --git a/src/app-layer.c b/src/app-layer.c
index b031afce8ac8..3625e87e9ed6 100644
--- a/src/app-layer.c
+++ b/src/app-layer.c
@@ -992,10 +992,9 @@ AppLayerThreadCtx *AppLayerGetCtxThread(ThreadVars *tv)
 {
     SCEnter();
 
-    AppLayerThreadCtx *app_tctx = SCMalloc(sizeof(*app_tctx));
+    AppLayerThreadCtx *app_tctx = SCCalloc(1, sizeof(*app_tctx));
     if (app_tctx == NULL)
         goto error;
-    memset(app_tctx, 0, sizeof(*app_tctx));
 
     if ((app_tctx->alpd_tctx = AppLayerProtoDetectGetCtxThread()) == NULL)
         goto error;
diff --git a/src/decode.c b/src/decode.c
index 5cdeeead6b96..d302c7654675 100644
--- a/src/decode.c
+++ b/src/decode.c
@@ -689,9 +689,8 @@ DecodeThreadVars *DecodeThreadVarsAlloc(ThreadVars *tv)
 {
     DecodeThreadVars *dtv = NULL;
 
-    if ( (dtv = SCMalloc(sizeof(DecodeThreadVars))) == NULL)
+    if ((dtv = SCCalloc(1, sizeof(DecodeThreadVars))) == NULL)
         return NULL;
-    memset(dtv, 0, sizeof(DecodeThreadVars));
 
     dtv->app_tctx = AppLayerGetCtxThread(tv);
 
diff --git a/src/defrag-hash.c b/src/defrag-hash.c
index 2f19ce28ee13..eb754d6eface 100644
--- a/src/defrag-hash.c
+++ b/src/defrag-hash.c
@@ -93,12 +93,10 @@ static DefragTracker *DefragTrackerAlloc(void)
 
     (void) SC_ATOMIC_ADD(defrag_memuse, sizeof(DefragTracker));
 
-    DefragTracker *dt = SCMalloc(sizeof(DefragTracker));
+    DefragTracker *dt = SCCalloc(1, sizeof(DefragTracker));
     if (unlikely(dt == NULL))
         goto error;
 
-    memset(dt, 0x00, sizeof(DefragTracker));
-
     SCMutexInit(&dt->lock, NULL);
     SC_ATOMIC_INIT(dt->use_cnt);
     return dt;
diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index 5c69e4442df7..cf9b24348e5e 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -228,10 +228,9 @@ static inline DetectByteExtractData *DetectByteExtractParse(DetectEngineCtx *de_
         goto error;
     }
 
-    bed = SCMalloc(sizeof(DetectByteExtractData));
+    bed = SCCalloc(1, sizeof(DetectByteExtractData));
     if (unlikely(bed == NULL))
         goto error;
-    memset(bed, 0, sizeof(DetectByteExtractData));
 
     /* no of bytes to extract */
     char nbytes_str[64] = "";
diff --git a/src/detect-csum.c b/src/detect-csum.c
index 8947725f4d26..ba2088a61e99 100644
--- a/src/detect-csum.c
+++ b/src/detect-csum.c
@@ -277,9 +277,8 @@ static int DetectIPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -368,9 +367,8 @@ static int DetectTCPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -459,9 +457,8 @@ static int DetectTCPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -550,9 +547,8 @@ static int DetectUDPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -641,9 +637,8 @@ static int DetectUDPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const cha
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -730,9 +725,8 @@ static int DetectICMPV4CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const ch
 
     //printf("DetectCsumSetup: \'%s\'\n", csum_str);
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
@@ -822,9 +816,8 @@ static int DetectICMPV6CsumSetup(DetectEngineCtx *de_ctx, Signature *s, const ch
 {
     DetectCsumData *cd = NULL;
 
-    if ( (cd = SCMalloc(sizeof(DetectCsumData))) == NULL)
+    if ((cd = SCCalloc(1, sizeof(DetectCsumData))) == NULL)
         goto error;
-    memset(cd, 0, sizeof(DetectCsumData));
 
     if (DetectCsumParseArg(csum_str, cd) == 0)
         goto error;
diff --git a/src/detect-detection-filter.c b/src/detect-detection-filter.c
index b55d663b68eb..cbd1898a31a4 100644
--- a/src/detect-detection-filter.c
+++ b/src/detect-detection-filter.c
@@ -139,12 +139,10 @@ static DetectThresholdData *DetectDetectionFilterParse(const char *rawstr)
         goto error;
     }
 
-    df = SCMalloc(sizeof(DetectThresholdData));
+    df = SCCalloc(1, sizeof(DetectThresholdData));
     if (unlikely(df == NULL))
         goto error;
 
-    memset(df, 0, sizeof(DetectThresholdData));
-
     df->type = TYPE_DETECTION;
 
     for (i = 0; i < (ret - 1); i++) {
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index af59884cced1..f632bd8e5bbf 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1383,10 +1383,9 @@ int SigAddressPrepareStage1(DetectEngineCtx *de_ctx)
 
     de_ctx->sig_array_len = DetectEngineGetMaxSigId(de_ctx);
     de_ctx->sig_array_size = (de_ctx->sig_array_len * sizeof(Signature *));
-    de_ctx->sig_array = (Signature **)SCMalloc(de_ctx->sig_array_size);
+    de_ctx->sig_array = (Signature **)SCCalloc(1, de_ctx->sig_array_size);
     if (de_ctx->sig_array == NULL)
         goto error;
-    memset(de_ctx->sig_array,0,de_ctx->sig_array_size);
 
     SCLogDebug("signature lookup array: %" PRIu32 " sigs, %" PRIu32 " bytes",
                de_ctx->sig_array_len, de_ctx->sig_array_size);
diff --git a/src/detect-engine-iponly.c b/src/detect-engine-iponly.c
index 03b464982178..63261ee716d5 100644
--- a/src/detect-engine-iponly.c
+++ b/src/detect-engine-iponly.c
@@ -74,10 +74,9 @@ static IPOnlyCIDRItem *IPOnlyCIDRItemNew(void)
     SCEnter();
     IPOnlyCIDRItem *item = NULL;
 
-    item = SCMalloc(sizeof(IPOnlyCIDRItem));
+    item = SCCalloc(1, sizeof(IPOnlyCIDRItem));
     if (unlikely(item == NULL))
         SCReturnPtr(NULL, "IPOnlyCIDRItem");
-    memset(item, 0, sizeof(IPOnlyCIDRItem));
 
     SCReturnPtr(item, "IPOnlyCIDRItem");
 }
@@ -550,19 +549,17 @@ static void SigNumArrayPrint(void *tmp)
 static SigNumArray *SigNumArrayNew(DetectEngineCtx *de_ctx,
                             DetectEngineIPOnlyCtx *io_ctx)
 {
-    SigNumArray *new = SCMalloc(sizeof(SigNumArray));
+    SigNumArray *new = SCCalloc(1, sizeof(SigNumArray));
 
     if (unlikely(new == NULL)) {
         FatalError("Fatal error encountered in SigNumArrayNew. Exiting...");
     }
-    memset(new, 0, sizeof(SigNumArray));
 
-    new->array = SCMalloc(io_ctx->max_idx / 8 + 1);
+    new->array = SCCalloc(1, io_ctx->max_idx / 8 + 1);
     if (new->array == NULL) {
        exit(EXIT_FAILURE);
     }
 
-    memset(new->array, 0, io_ctx->max_idx / 8 + 1);
     new->size = io_ctx->max_idx / 8 + 1;
 
     SCLogDebug("max idx= %u", io_ctx->max_idx);
@@ -580,13 +577,12 @@ static SigNumArray *SigNumArrayNew(DetectEngineCtx *de_ctx,
  */
 static SigNumArray *SigNumArrayCopy(SigNumArray *orig)
 {
-    SigNumArray *new = SCMalloc(sizeof(SigNumArray));
+    SigNumArray *new = SCCalloc(1, sizeof(SigNumArray));
 
     if (unlikely(new == NULL)) {
         FatalError("Fatal error encountered in SigNumArrayCopy. Exiting...");
     }
 
-    memset(new, 0, sizeof(SigNumArray));
     new->size = orig->size;
 
     new->array = SCMalloc(orig->size);
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index 67af1c115cf4..36e3872c0452 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -86,19 +86,15 @@ void SigGroupHeadInitDataFree(SigGroupHeadInitData *sghid)
 
 static SigGroupHeadInitData *SigGroupHeadInitDataAlloc(uint32_t size)
 {
-    SigGroupHeadInitData *sghid = SCMalloc(sizeof(SigGroupHeadInitData));
+    SigGroupHeadInitData *sghid = SCCalloc(1, sizeof(SigGroupHeadInitData));
     if (unlikely(sghid == NULL))
         return NULL;
 
-    memset(sghid, 0x00, sizeof(SigGroupHeadInitData));
-
     /* initialize the signature bitarray */
     sghid->sig_size = size;
-    if ( (sghid->sig_array = SCMalloc(sghid->sig_size)) == NULL)
+    if ((sghid->sig_array = SCCalloc(1, sghid->sig_size)) == NULL)
         goto error;
 
-    memset(sghid->sig_array, 0, sghid->sig_size);
-
     return sghid;
 error:
     SigGroupHeadInitDataFree(sghid);
@@ -139,10 +135,9 @@ void SigGroupHeadStore(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
  */
 static SigGroupHead *SigGroupHeadAlloc(const DetectEngineCtx *de_ctx, uint32_t size)
 {
-    SigGroupHead *sgh = SCMalloc(sizeof(SigGroupHead));
+    SigGroupHead *sgh = SCCalloc(1, sizeof(SigGroupHead));
     if (unlikely(sgh == NULL))
         return NULL;
-    memset(sgh, 0, sizeof(SigGroupHead));
 
     sgh->init = SigGroupHeadInitDataAlloc(size);
     if (sgh->init == NULL)
@@ -498,12 +493,10 @@ int SigGroupHeadBuildMatchArray(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
 
     BUG_ON(sgh->init->match_array != NULL);
 
-    sgh->init->match_array = SCMalloc(sgh->init->sig_cnt * sizeof(Signature *));
+    sgh->init->match_array = SCCalloc(1, sgh->init->sig_cnt * sizeof(Signature *));
     if (sgh->init->match_array == NULL)
         return -1;
 
-    memset(sgh->init->match_array, 0, sgh->init->sig_cnt * sizeof(Signature *));
-
     for (sig = 0; sig < max_idx + 1; sig++) {
         if (!(sgh->init->sig_array[(sig / 8)] & (1 << (sig % 8))) )
             continue;
@@ -679,15 +672,13 @@ int SigGroupHeadBuildNonPrefilterArray(DetectEngineCtx *de_ctx, SigGroupHead *sg
     }
 
     if (non_pf > 0) {
-        sgh->non_pf_other_store_array = SCMalloc(non_pf * sizeof(SignatureNonPrefilterStore));
+        sgh->non_pf_other_store_array = SCCalloc(1, non_pf * sizeof(SignatureNonPrefilterStore));
         BUG_ON(sgh->non_pf_other_store_array == NULL);
-        memset(sgh->non_pf_other_store_array, 0, non_pf * sizeof(SignatureNonPrefilterStore));
     }
 
     if (non_pf_syn > 0) {
-        sgh->non_pf_syn_store_array = SCMalloc(non_pf_syn * sizeof(SignatureNonPrefilterStore));
+        sgh->non_pf_syn_store_array = SCCalloc(1, non_pf_syn * sizeof(SignatureNonPrefilterStore));
         BUG_ON(sgh->non_pf_syn_store_array == NULL);
-        memset(sgh->non_pf_syn_store_array, 0, non_pf_syn * sizeof(SignatureNonPrefilterStore));
     }
 
     for (sig = 0; sig < sgh->init->sig_cnt; sig++) {
diff --git a/src/detect-engine-sigorder.c b/src/detect-engine-sigorder.c
index bb342e385afe..ea51e191ecee 100644
--- a/src/detect-engine-sigorder.c
+++ b/src/detect-engine-sigorder.c
@@ -98,10 +98,9 @@ static void SCSigRegisterSignatureOrderingFunc(DetectEngineCtx *de_ctx,
         curr = curr->next;
     }
 
-    if ( (temp = SCMalloc(sizeof(SCSigOrderFunc))) == NULL) {
+    if ((temp = SCCalloc(1, sizeof(SCSigOrderFunc))) == NULL) {
         FatalError("Fatal error encountered in SCSigRegisterSignatureOrderingFunc. Exiting...");
     }
-    memset(temp, 0, sizeof(SCSigOrderFunc));
 
     temp->SWCompare = SWCompare;
 
@@ -708,9 +707,8 @@ static inline SCSigSignatureWrapper *SCSigAllocSignatureWrapper(Signature *sig)
 {
     SCSigSignatureWrapper *sw = NULL;
 
-    if ( (sw = SCMalloc(sizeof(SCSigSignatureWrapper))) == NULL)
+    if ((sw = SCCalloc(1, sizeof(SCSigSignatureWrapper))) == NULL)
         return NULL;
-    memset(sw, 0, sizeof(SCSigSignatureWrapper));
 
     sw->sig = sig;
 
diff --git a/src/detect-engine-state.c b/src/detect-engine-state.c
index 6fd7f96e58be..74f87ff938f1 100644
--- a/src/detect-engine-state.c
+++ b/src/detect-engine-state.c
@@ -84,10 +84,9 @@ static inline int StateIsValid(uint16_t alproto, void *alstate)
 
 static DeStateStore *DeStateStoreAlloc(void)
 {
-    DeStateStore *d = SCMalloc(sizeof(DeStateStore));
+    DeStateStore *d = SCCalloc(1, sizeof(DeStateStore));
     if (unlikely(d == NULL))
         return NULL;
-    memset(d, 0, sizeof(DeStateStore));
 
     return d;
 }
@@ -163,10 +162,9 @@ static void DeStateSignatureAppend(DetectEngineState *state,
 
 DetectEngineState *DetectEngineStateAlloc(void)
 {
-    DetectEngineState *d = SCMalloc(sizeof(DetectEngineState));
+    DetectEngineState *d = SCCalloc(1, sizeof(DetectEngineState));
     if (unlikely(d == NULL))
         return NULL;
-    memset(d, 0, sizeof(DetectEngineState));
 
     return d;
 }
diff --git a/src/detect-engine-tag.c b/src/detect-engine-tag.c
index e6a4134048e4..21610264ef5b 100644
--- a/src/detect-engine-tag.c
+++ b/src/detect-engine-tag.c
@@ -91,11 +91,10 @@ int TagHostHasTag(Host *host)
 
 static DetectTagDataEntry *DetectTagDataCopy(DetectTagDataEntry *dtd)
 {
-    DetectTagDataEntry *tde = SCMalloc(sizeof(DetectTagDataEntry));
+    DetectTagDataEntry *tde = SCCalloc(1, sizeof(DetectTagDataEntry));
     if (unlikely(tde == NULL)) {
         return NULL;
     }
-    memset(tde, 0, sizeof(DetectTagDataEntry));
 
     tde->sid = dtd->sid;
     tde->gid = dtd->gid;
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 25e76445edd9..678031fa44df 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -248,11 +248,11 @@ void DetectAppLayerInspectEngineRegister2(const char *name,
         direction = 1;
     }
 
-    DetectEngineAppInspectionEngine *new_engine = SCMalloc(sizeof(DetectEngineAppInspectionEngine));
+    DetectEngineAppInspectionEngine *new_engine =
+            SCCalloc(1, sizeof(DetectEngineAppInspectionEngine));
     if (unlikely(new_engine == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(new_engine, 0, sizeof(*new_engine));
     new_engine->alproto = alproto;
     new_engine->dir = direction;
     new_engine->sm_list = (uint16_t)sm_list;
@@ -2469,11 +2469,10 @@ static int DetectEngineReloadThreads(DetectEngineCtx *new_de_ctx)
 static DetectEngineCtx *DetectEngineCtxInitReal(
         enum DetectEngineType type, const char *prefix, uint32_t tenant_id)
 {
-    DetectEngineCtx *de_ctx = SCMalloc(sizeof(DetectEngineCtx));
+    DetectEngineCtx *de_ctx = SCCalloc(1, sizeof(DetectEngineCtx));
     if (unlikely(de_ctx == NULL))
         goto error;
 
-    memset(de_ctx,0,sizeof(DetectEngineCtx));
     memset(&de_ctx->sig_stat, 0, sizeof(SigFileLoaderStat));
     TAILQ_INIT(&de_ctx->sig_stat.failed_sigs);
     de_ctx->sigerror = NULL;
@@ -3047,14 +3046,12 @@ static int DetectEngineThreadCtxInitKeywords(DetectEngineCtx *de_ctx, DetectEngi
 {
     if (de_ctx->keyword_id > 0) {
         // coverity[suspicious_sizeof : FALSE]
-        det_ctx->keyword_ctxs_array = SCMalloc(de_ctx->keyword_id * sizeof(void *));
+        det_ctx->keyword_ctxs_array = SCCalloc(1, de_ctx->keyword_id * sizeof(void *));
         if (det_ctx->keyword_ctxs_array == NULL) {
             SCLogError("setting up thread local detect ctx");
             return TM_ECODE_FAILED;
         }
 
-        memset(det_ctx->keyword_ctxs_array, 0x00, de_ctx->keyword_id * sizeof(void *));
-
         det_ctx->keyword_ctxs_size = de_ctx->keyword_id;
 
         HashListTableBucket *hb = HashListTableGetListHead(de_ctx->keyword_hash);
@@ -3229,12 +3226,10 @@ static TmEcode ThreadCtxDoInit (DetectEngineCtx *de_ctx, DetectEngineThreadCtx *
     /* DeState */
     if (de_ctx->sig_array_len > 0) {
         det_ctx->match_array_len = de_ctx->sig_array_len;
-        det_ctx->match_array = SCMalloc(det_ctx->match_array_len * sizeof(Signature *));
+        det_ctx->match_array = SCCalloc(1, det_ctx->match_array_len * sizeof(Signature *));
         if (det_ctx->match_array == NULL) {
             return TM_ECODE_FAILED;
         }
-        memset(det_ctx->match_array, 0,
-               det_ctx->match_array_len * sizeof(Signature *));
 
         RuleMatchCandidateTxArrayInit(det_ctx, de_ctx->sig_array_len);
     }
@@ -3316,10 +3311,9 @@ static TmEcode ThreadCtxDoInit (DetectEngineCtx *de_ctx, DetectEngineThreadCtx *
  */
 TmEcode DetectEngineThreadCtxInit(ThreadVars *tv, void *initdata, void **data)
 {
-    DetectEngineThreadCtx *det_ctx = SCMalloc(sizeof(DetectEngineThreadCtx));
+    DetectEngineThreadCtx *det_ctx = SCCalloc(1, sizeof(DetectEngineThreadCtx));
     if (unlikely(det_ctx == NULL))
         return TM_ECODE_FAILED;
-    memset(det_ctx, 0, sizeof(DetectEngineThreadCtx));
 
     det_ctx->tv = tv;
     det_ctx->de_ctx = DetectEngineGetCurrent();
@@ -3382,10 +3376,9 @@ TmEcode DetectEngineThreadCtxInit(ThreadVars *tv, void *initdata, void **data)
 DetectEngineThreadCtx *DetectEngineThreadCtxInitForReload(
         ThreadVars *tv, DetectEngineCtx *new_de_ctx, int mt)
 {
-    DetectEngineThreadCtx *det_ctx = SCMalloc(sizeof(DetectEngineThreadCtx));
+    DetectEngineThreadCtx *det_ctx = SCCalloc(1, sizeof(DetectEngineThreadCtx));
     if (unlikely(det_ctx == NULL))
         return NULL;
-    memset(det_ctx, 0, sizeof(DetectEngineThreadCtx));
 
     det_ctx->tenant_id = new_de_ctx->tenant_id;
     det_ctx->tv = tv;
diff --git a/src/detect-fast-pattern.c b/src/detect-fast-pattern.c
index b82f3274d709..6748186727c9 100644
--- a/src/detect-fast-pattern.c
+++ b/src/detect-fast-pattern.c
@@ -95,10 +95,9 @@ static void Add(SCFPSupportSMList **list, const int list_id, const int priority)
     }
 
     if (*list == NULL) {
-        SCFPSupportSMList *new = SCMalloc(sizeof(SCFPSupportSMList));
+        SCFPSupportSMList *new = SCCalloc(1, sizeof(SCFPSupportSMList));
         if (unlikely(new == NULL))
             exit(EXIT_FAILURE);
-        memset(new, 0, sizeof(SCFPSupportSMList));
         new->list_id = list_id;
         new->priority = priority;
 
@@ -106,10 +105,9 @@ static void Add(SCFPSupportSMList **list, const int list_id, const int priority)
         return;
     }
 
-    SCFPSupportSMList *new = SCMalloc(sizeof(SCFPSupportSMList));
+    SCFPSupportSMList *new = SCCalloc(1, sizeof(SCFPSupportSMList));
     if (unlikely(new == NULL))
         exit(EXIT_FAILURE);
-    memset(new, 0, sizeof(SCFPSupportSMList));
     new->list_id = list_id;
     new->priority = priority;
     if (ip == NULL) {
diff --git a/src/detect-file-hash-common.c b/src/detect-file-hash-common.c
index b028bff74bf1..f81ce4be29ea 100644
--- a/src/detect-file-hash-common.c
+++ b/src/detect-file-hash-common.c
@@ -201,12 +201,10 @@ static DetectFileHashData *DetectFileHashParse (const DetectEngineCtx *de_ctx,
     char *rule_filename = NULL;
 
     /* We have a correct hash algorithm option */
-    filehash = SCMalloc(sizeof(DetectFileHashData));
+    filehash = SCCalloc(1, sizeof(DetectFileHashData));
     if (unlikely(filehash == NULL))
         goto error;
 
-    memset(filehash, 0x00, sizeof(DetectFileHashData));
-
     if (strlen(str) && str[0] == '!') {
         filehash->negated = 1;
         str++;
diff --git a/src/detect-filestore.c b/src/detect-filestore.c
index 4efa59209967..5e22e4c6cde5 100644
--- a/src/detect-filestore.c
+++ b/src/detect-filestore.c
@@ -401,10 +401,9 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
             }
         }
 
-        fd = SCMalloc(sizeof(DetectFilestoreData));
+        fd = SCCalloc(1, sizeof(DetectFilestoreData));
         if (unlikely(fd == NULL))
             goto error;
-        memset(fd, 0x00, sizeof(DetectFilestoreData));
 
         if (args[0] != NULL) {
             SCLogDebug("first arg %s", args[0]);
diff --git a/src/detect-flowvar.c b/src/detect-flowvar.c
index 38c8dc062919..c923be5d0b77 100644
--- a/src/detect-flowvar.c
+++ b/src/detect-flowvar.c
@@ -163,10 +163,9 @@ static int DetectFlowvarSetup (DetectEngineCtx *de_ctx, Signature *s, const char
     if (res == -1)
         goto error;
 
-    fd = SCMalloc(sizeof(DetectFlowvarData));
+    fd = SCCalloc(1, sizeof(DetectFlowvarData));
     if (unlikely(fd == NULL))
         goto error;
-    memset(fd, 0x00, sizeof(*fd));
 
     fd->content = SCMalloc(contentlen);
     if (unlikely(fd->content == NULL))
@@ -261,10 +260,9 @@ int DetectFlowvarPostMatchSetup(DetectEngineCtx *de_ctx, Signature *s, uint32_t
 {
     DetectFlowvarData *fv = NULL;
 
-    fv = SCMalloc(sizeof(DetectFlowvarData));
+    fv = SCCalloc(1, sizeof(DetectFlowvarData));
     if (unlikely(fv == NULL))
         goto error;
-    memset(fv, 0x00, sizeof(*fv));
 
     /* we only need the idx */
     fv->idx = idx;
diff --git a/src/detect-fragbits.c b/src/detect-fragbits.c
index a9657641424c..e8e8b78057bb 100644
--- a/src/detect-fragbits.c
+++ b/src/detect-fragbits.c
@@ -198,12 +198,10 @@ static DetectFragBitsData *DetectFragBitsParse (const char *rawstr)
         goto error;
     }
 
-    de = SCMalloc(sizeof(DetectFragBitsData));
+    de = SCCalloc(1, sizeof(DetectFragBitsData));
     if (unlikely(de == NULL))
         goto error;
 
-    memset(de,0,sizeof(DetectFragBitsData));
-
     /** First parse args[0] */
 
     if (args[0] && strlen(args[0])) {
diff --git a/src/detect-geoip.c b/src/detect-geoip.c
index e31e9fd518b4..9198b9e45129 100644
--- a/src/detect-geoip.c
+++ b/src/detect-geoip.c
@@ -305,12 +305,10 @@ static DetectGeoipData *DetectGeoipDataParse (DetectEngineCtx *de_ctx, const cha
         goto error;
 
     /* We have a correct geoip options string */
-    geoipdata = SCMalloc(sizeof(DetectGeoipData));
+    geoipdata = SCCalloc(1, sizeof(DetectGeoipData));
     if (unlikely(geoipdata == NULL))
         goto error;
 
-    memset(geoipdata, 0x00, sizeof(DetectGeoipData));
-
     /* Parse the geoip option string */
     while (pos <= slen)
     {
diff --git a/src/detect-lua.c b/src/detect-lua.c
index 4f66fa7395ab..0ea74452739e 100644
--- a/src/detect-lua.c
+++ b/src/detect-lua.c
@@ -596,12 +596,11 @@ static void *DetectLuaThreadInit(void *data)
     DetectLuaData *lua = (DetectLuaData *)data;
     BUG_ON(lua == NULL);
 
-    DetectLuaThreadData *t = SCMalloc(sizeof(DetectLuaThreadData));
+    DetectLuaThreadData *t = SCCalloc(1, sizeof(DetectLuaThreadData));
     if (unlikely(t == NULL)) {
         SCLogError("couldn't alloc ctx memory");
         return NULL;
     }
-    memset(t, 0x00, sizeof(DetectLuaThreadData));
 
     t->alproto = lua->alproto;
     t->flags = lua->flags;
@@ -681,12 +680,10 @@ static DetectLuaData *DetectLuaParse (DetectEngineCtx *de_ctx, const char *str)
     DetectLuaData *lua = NULL;
 
     /* We have a correct lua option */
-    lua = SCMalloc(sizeof(DetectLuaData));
+    lua = SCCalloc(1, sizeof(DetectLuaData));
     if (unlikely(lua == NULL))
         goto error;
 
-    memset(lua, 0x00, sizeof(DetectLuaData));
-
     if (strlen(str) && str[0] == '!') {
         lua->negated = 1;
         str++;
diff --git a/src/detect-parse.c b/src/detect-parse.c
index ba3c17d789ae..e1ac5f74b5a4 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -334,11 +334,10 @@ int DetectEngineContentModifierBufferSetup(DetectEngineCtx *de_ctx,
 
 SigMatch *SigMatchAlloc(void)
 {
-    SigMatch *sm = SCMalloc(sizeof(SigMatch));
+    SigMatch *sm = SCCalloc(1, sizeof(SigMatch));
     if (unlikely(sm == NULL))
         return NULL;
 
-    memset(sm, 0, sizeof(SigMatch));
     sm->prev = NULL;
     sm->next = NULL;
     return sm;
@@ -1514,10 +1513,9 @@ int SignatureInitDataBufferCheckExpand(Signature *s)
 
 Signature *SigAlloc (void)
 {
-    Signature *sig = SCMalloc(sizeof(Signature));
+    Signature *sig = SCCalloc(1, sizeof(Signature));
     if (unlikely(sig == NULL))
         return NULL;
-    memset(sig, 0, sizeof(Signature));
 
     sig->init_data = SCCalloc(1, sizeof(SignatureInitData));
     if (sig->init_data == NULL) {
@@ -2463,11 +2461,10 @@ static inline int DetectEngineSignatureIsDuplicate(DetectEngineCtx *de_ctx,
     SigDuplWrapper *sw = NULL;
 
     /* used for making a duplicate_sig_hash_table entry */
-    sw = SCMalloc(sizeof(SigDuplWrapper));
+    sw = SCCalloc(1, sizeof(SigDuplWrapper));
     if (unlikely(sw == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(sw, 0, sizeof(SigDuplWrapper));
     sw->s = sig;
 
     /* check if we have a duplicate entry for this signature */
diff --git a/src/detect-ssh-proto-version.c b/src/detect-ssh-proto-version.c
index 1ca99e620287..6a852f89ef13 100644
--- a/src/detect-ssh-proto-version.c
+++ b/src/detect-ssh-proto-version.c
@@ -179,12 +179,11 @@ static DetectSshVersionData *DetectSshVersionParse (DetectEngineCtx *de_ctx, con
         }
 
         /* We have a correct id option */
-        ssh = SCMalloc(sizeof(DetectSshVersionData));
+        ssh = SCCalloc(1, sizeof(DetectSshVersionData));
         if (unlikely(ssh == NULL)) {
             pcre2_substring_free((PCRE2_UCHAR *)str_ptr);
             goto error;
         }
-        memset(ssh, 0x00, sizeof(DetectSshVersionData));
 
         /* If we expect a protocol version 2 or 1.99 (considered 2, we
          * will compare it with both strings) */
diff --git a/src/detect-tcp-flags.c b/src/detect-tcp-flags.c
index 04caed0209a2..5809a5dce983 100644
--- a/src/detect-tcp-flags.c
+++ b/src/detect-tcp-flags.c
@@ -218,10 +218,9 @@ static DetectFlagsData *DetectFlagsParse (const char *rawstr)
         goto error;
     }
 
-    de = SCMalloc(sizeof(DetectFlagsData));
+    de = SCCalloc(1, sizeof(DetectFlagsData));
     if (unlikely(de == NULL))
         goto error;
-    memset(de, 0, sizeof(DetectFlagsData));
     de->ignored_flags = 0xff;
 
     /** First parse args1 */
diff --git a/src/detect-threshold.c b/src/detect-threshold.c
index 768447204267..98eb3ce8dc03 100644
--- a/src/detect-threshold.c
+++ b/src/detect-threshold.c
@@ -154,12 +154,10 @@ static DetectThresholdData *DetectThresholdParse(const char *rawstr)
         goto error;
     }
 
-    de = SCMalloc(sizeof(DetectThresholdData));
+    de = SCCalloc(1, sizeof(DetectThresholdData));
     if (unlikely(de == NULL))
         goto error;
 
-    memset(de,0,sizeof(DetectThresholdData));
-
     for (i = 0; i < (ret - 1); i++) {
 
         res = pcre2_substring_get_bynumber(match, i + 1, (PCRE2_UCHAR8 **)&str_ptr, &pcre2_len);
diff --git a/src/flow-util.c b/src/flow-util.c
index dc6a7103a6bd..672abc23d2ba 100644
--- a/src/flow-util.c
+++ b/src/flow-util.c
@@ -62,12 +62,11 @@ Flow *FlowAlloc(void)
 
     (void) SC_ATOMIC_ADD(flow_memuse, size);
 
-    f = SCMalloc(size);
+    f = SCCalloc(1, size);
     if (unlikely(f == NULL)) {
         (void)SC_ATOMIC_SUB(flow_memuse, size);
         return NULL;
     }
-    memset(f, 0, size);
 
     /* coverity[missing_lock] */
     FLOW_INITIALIZE(f);
diff --git a/src/host.c b/src/host.c
index d19d321ca739..7a5305ac18cc 100644
--- a/src/host.c
+++ b/src/host.c
@@ -115,12 +115,10 @@ Host *HostAlloc(void)
     }
     (void) SC_ATOMIC_ADD(host_memuse, g_host_size);
 
-    Host *h = SCMalloc(g_host_size);
+    Host *h = SCCalloc(1, g_host_size);
     if (unlikely(h == NULL))
         goto error;
 
-    memset(h, 0x00, g_host_size);
-
     SCMutexInit(&h->m, NULL);
     SC_ATOMIC_INIT(h->use_cnt);
     return h;
diff --git a/src/ippair.c b/src/ippair.c
index 110eb9f6c62b..81362f63f4c9 100644
--- a/src/ippair.c
+++ b/src/ippair.c
@@ -114,12 +114,10 @@ IPPair *IPPairAlloc(void)
 
     (void) SC_ATOMIC_ADD(ippair_memuse, g_ippair_size);
 
-    IPPair *h = SCMalloc(g_ippair_size);
+    IPPair *h = SCCalloc(1, g_ippair_size);
     if (unlikely(h == NULL))
         goto error;
 
-    memset(h, 0x00, g_ippair_size);
-
     SCMutexInit(&h->m, NULL);
     SC_ATOMIC_INIT(h->use_cnt);
     return h;
diff --git a/src/log-httplog.c b/src/log-httplog.c
index 1e45053abb2c..68fa62e95b7f 100644
--- a/src/log-httplog.c
+++ b/src/log-httplog.c
@@ -503,10 +503,9 @@ int LogHttpLogger(ThreadVars *tv, void *thread_data, const Packet *p, Flow *f, v
 
 TmEcode LogHttpLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    LogHttpLogThread *aft = SCMalloc(sizeof(LogHttpLogThread));
+    LogHttpLogThread *aft = SCCalloc(1, sizeof(LogHttpLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(LogHttpLogThread));
 
     if(initdata == NULL)
     {
@@ -561,12 +560,11 @@ OutputInitResult LogHttpLogInitCtx(ConfNode *conf)
         return result;
     }
 
-    LogHttpFileCtx *httplog_ctx = SCMalloc(sizeof(LogHttpFileCtx));
+    LogHttpFileCtx *httplog_ctx = SCCalloc(1, sizeof(LogHttpFileCtx));
     if (unlikely(httplog_ctx == NULL)) {
         LogFileFreeCtx(file_ctx);
         return result;
     }
-    memset(httplog_ctx, 0x00, sizeof(LogHttpFileCtx));
 
     httplog_ctx->file_ctx = file_ctx;
 
diff --git a/src/log-pcap.c b/src/log-pcap.c
index f8c8bf716562..e237b7ad9e99 100644
--- a/src/log-pcap.c
+++ b/src/log-pcap.c
@@ -1349,11 +1349,10 @@ static OutputInitResult PcapLogInitCtx(ConfNode *conf)
     int en;
     PCRE2_SIZE eo = 0;
 
-    PcapLogData *pl = SCMalloc(sizeof(PcapLogData));
+    PcapLogData *pl = SCCalloc(1, sizeof(PcapLogData));
     if (unlikely(pl == NULL)) {
         FatalError("Failed to allocate Memory for PcapLogData");
     }
-    memset(pl, 0, sizeof(PcapLogData));
 
     pl->h = SCMalloc(sizeof(*pl->h));
     if (pl->h == NULL) {
@@ -1737,11 +1736,10 @@ static int PcapLogOpenFileCtx(PcapLogData *pl)
     SCTime_t ts = TimeGet();
 
     /* Place to store the name of our PCAP file */
-    PcapFileName *pf = SCMalloc(sizeof(PcapFileName));
+    PcapFileName *pf = SCCalloc(1, sizeof(PcapFileName));
     if (unlikely(pf == NULL)) {
         return -1;
     }
-    memset(pf, 0, sizeof(PcapFileName));
 
     if (pl->mode == LOGMODE_SGUIL) {
         struct tm local_tm;
diff --git a/src/log-stats.c b/src/log-stats.c
index e694eb8d8d56..69669e9c2257 100644
--- a/src/log-stats.c
+++ b/src/log-stats.c
@@ -165,10 +165,9 @@ static int LogStatsLogger(ThreadVars *tv, void *thread_data, const StatsTable *s
 
 TmEcode LogStatsLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    LogStatsLogThread *aft = SCMalloc(sizeof(LogStatsLogThread));
+    LogStatsLogThread *aft = SCCalloc(1, sizeof(LogStatsLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(LogStatsLogThread));
 
     if(initdata == NULL)
     {
@@ -223,12 +222,11 @@ static OutputInitResult LogStatsLogInitCtx(ConfNode *conf)
         return result;
     }
 
-    LogStatsFileCtx *statslog_ctx = SCMalloc(sizeof(LogStatsFileCtx));
+    LogStatsFileCtx *statslog_ctx = SCCalloc(1, sizeof(LogStatsFileCtx));
     if (unlikely(statslog_ctx == NULL)) {
         LogFileFreeCtx(file_ctx);
         return result;
     }
-    memset(statslog_ctx, 0x00, sizeof(LogStatsFileCtx));
 
     statslog_ctx->flags = LOG_STATS_TOTALS;
 
diff --git a/src/log-tcp-data.c b/src/log-tcp-data.c
index 538cfc91e12b..9c67497c16c6 100644
--- a/src/log-tcp-data.c
+++ b/src/log-tcp-data.c
@@ -165,10 +165,9 @@ int LogTcpDataLogger(ThreadVars *tv, void *thread_data, const Flow *f,
 
 TmEcode LogTcpDataLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    LogTcpDataLogThread *aft = SCMalloc(sizeof(LogTcpDataLogThread));
+    LogTcpDataLogThread *aft = SCCalloc(1, sizeof(LogTcpDataLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(LogTcpDataLogThread));
 
     if(initdata == NULL)
     {
@@ -222,12 +221,11 @@ OutputInitResult LogTcpDataLogInitCtx(ConfNode *conf)
         return result;
     }
 
-    LogTcpDataFileCtx *tcpdatalog_ctx = SCMalloc(sizeof(LogTcpDataFileCtx));
+    LogTcpDataFileCtx *tcpdatalog_ctx = SCCalloc(1, sizeof(LogTcpDataFileCtx));
     if (unlikely(tcpdatalog_ctx == NULL)) {
         LogFileFreeCtx(file_ctx);
         return result;
     }
-    memset(tcpdatalog_ctx, 0x00, sizeof(LogTcpDataFileCtx));
 
     tcpdatalog_ctx->file_ctx = file_ctx;
 
diff --git a/src/log-tlslog.c b/src/log-tlslog.c
index dc32d3d814d6..6217c5fe9813 100644
--- a/src/log-tlslog.c
+++ b/src/log-tlslog.c
@@ -141,12 +141,10 @@ int TLSGetIPInformations(const Packet *p, char* srcip, size_t srcip_len,
 static TmEcode LogTlsLogThreadInit(ThreadVars *t, const void *initdata,
                                    void **data)
 {
-    LogTlsLogThread *aft = SCMalloc(sizeof(LogTlsLogThread));
+    LogTlsLogThread *aft = SCCalloc(1, sizeof(LogTlsLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
 
-    memset(aft, 0, sizeof(LogTlsLogThread));
-
     if (initdata == NULL) {
         SCLogDebug("Error getting context for TLSLog. \"initdata\" argument NULL");
         SCFree(aft);
diff --git a/src/log-tlsstore.c b/src/log-tlsstore.c
index 969044553673..50e6c6e5c481 100644
--- a/src/log-tlsstore.c
+++ b/src/log-tlsstore.c
@@ -275,10 +275,9 @@ static int LogTlsStoreLogger(ThreadVars *tv, void *thread_data, const Packet *p,
 
 static TmEcode LogTlsStoreLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    LogTlsStoreLogThread *aft = SCMalloc(sizeof(LogTlsStoreLogThread));
+    LogTlsStoreLogThread *aft = SCCalloc(1, sizeof(LogTlsStoreLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(LogTlsStoreLogThread));
 
     if (initdata == NULL) {
         SCLogDebug("Error getting context for LogTLSStore. \"initdata\" argument NULL");
diff --git a/src/output-file.c b/src/output-file.c
index 317e1139a56a..ff8b83739443 100644
--- a/src/output-file.c
+++ b/src/output-file.c
@@ -59,10 +59,9 @@ int OutputRegisterFileLogger(LoggerId id, const char *name, FileLogger LogFunc,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputFileLogger *op = SCMalloc(sizeof(*op));
+    OutputFileLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->output_ctx = output_ctx;
@@ -188,9 +187,8 @@ TmEcode OutputFileLogThreadInit(ThreadVars *tv, OutputFileLoggerThreadData **dat
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-/* todo */      BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                /* todo */ BUG_ON(ts == NULL);
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-filedata.c b/src/output-filedata.c
index a9890769a7b3..daaab1bcc48d 100644
--- a/src/output-filedata.c
+++ b/src/output-filedata.c
@@ -57,10 +57,9 @@ int OutputRegisterFiledataLogger(LoggerId id, const char *name,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputFiledataLogger *op = SCMalloc(sizeof(*op));
+    OutputFiledataLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->output_ctx = output_ctx;
@@ -204,10 +203,9 @@ void OutputFiledataLogFfc(ThreadVars *tv, OutputFiledataLoggerThreadData *td, Pa
  *  loggers */
 TmEcode OutputFiledataLogThreadInit(ThreadVars *tv, OutputFiledataLoggerThreadData **data)
 {
-    OutputFiledataLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputFiledataLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
     *data = td;
 
 #ifdef HAVE_MAGIC
@@ -225,9 +223,8 @@ TmEcode OutputFiledataLogThreadInit(ThreadVars *tv, OutputFiledataLoggerThreadDa
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
                 /* todo */ BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-filestore.c b/src/output-filestore.c
index dcf4c1aea502..607fe292ffc1 100644
--- a/src/output-filestore.c
+++ b/src/output-filestore.c
@@ -263,10 +263,9 @@ static int OutputFilestoreLogger(ThreadVars *tv, void *thread_data, const Packet
 static TmEcode OutputFilestoreLogThreadInit(ThreadVars *t, const void *initdata,
         void **data)
 {
-    OutputFilestoreLogThread *aft = SCMalloc(sizeof(OutputFilestoreLogThread));
+    OutputFilestoreLogThread *aft = SCCalloc(1, sizeof(OutputFilestoreLogThread));
     if (unlikely(aft == NULL))
         return TM_ECODE_FAILED;
-    memset(aft, 0, sizeof(OutputFilestoreLogThread));
 
     if (initdata == NULL) {
         SCLogDebug("Error getting context for LogFileStore. \"initdata\" argument NULL");
diff --git a/src/output-flow.c b/src/output-flow.c
index fa60f3c579b8..5231a3667942 100644
--- a/src/output-flow.c
+++ b/src/output-flow.c
@@ -55,10 +55,9 @@ int OutputRegisterFlowLogger(const char *name, FlowLogger LogFunc,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputFlowLogger *op = SCMalloc(sizeof(*op));
+    OutputFlowLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->output_ctx = output_ctx;
@@ -123,10 +122,9 @@ TmEcode OutputFlowLog(ThreadVars *tv, void *thread_data, Flow *f)
  *  loggers */
 TmEcode OutputFlowLogThreadInit(ThreadVars *tv, void *initdata, void **data)
 {
-    OutputFlowLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputFlowLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
 
     *data = (void *)td;
 
@@ -137,9 +135,8 @@ TmEcode OutputFlowLogThreadInit(ThreadVars *tv, void *initdata, void **data)
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-/* todo */      BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                /* todo */ BUG_ON(ts == NULL);
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-json-alert.c b/src/output-json-alert.c
index ad9d236f3090..c7acfe64d140 100644
--- a/src/output-json-alert.c
+++ b/src/output-json-alert.c
@@ -941,11 +941,10 @@ static OutputInitResult JsonAlertLogInitCtxSub(ConfNode *conf, OutputCtx *parent
     if (unlikely(output_ctx == NULL))
         return result;
 
-    json_output_ctx = SCMalloc(sizeof(AlertJsonOutputCtx));
+    json_output_ctx = SCCalloc(1, sizeof(AlertJsonOutputCtx));
     if (unlikely(json_output_ctx == NULL)) {
         goto error;
     }
-    memset(json_output_ctx, 0, sizeof(AlertJsonOutputCtx));
 
     json_output_ctx->file_ctx = ajt->file_ctx;
     json_output_ctx->eve_ctx = ajt;
diff --git a/src/output-json-dns.c b/src/output-json-dns.c
index b27c67feb240..27aa55d8e305 100644
--- a/src/output-json-dns.c
+++ b/src/output-json-dns.c
@@ -568,11 +568,10 @@ static OutputInitResult JsonDnsLogInitCtxSub(ConfNode *conf, OutputCtx *parent_c
 
     OutputJsonCtx *ojc = parent_ctx->data;
 
-    LogDnsFileCtx *dnslog_ctx = SCMalloc(sizeof(LogDnsFileCtx));
+    LogDnsFileCtx *dnslog_ctx = SCCalloc(1, sizeof(LogDnsFileCtx));
     if (unlikely(dnslog_ctx == NULL)) {
         return result;
     }
-    memset(dnslog_ctx, 0x00, sizeof(LogDnsFileCtx));
 
     dnslog_ctx->eve_ctx = ojc;
 
diff --git a/src/output-json-frame.c b/src/output-json-frame.c
index b7aaabc1dea9..665010a6e44a 100644
--- a/src/output-json-frame.c
+++ b/src/output-json-frame.c
@@ -477,11 +477,10 @@ static OutputInitResult JsonFrameLogInitCtxSub(ConfNode *conf, OutputCtx *parent
     if (unlikely(output_ctx == NULL))
         return result;
 
-    json_output_ctx = SCMalloc(sizeof(FrameJsonOutputCtx));
+    json_output_ctx = SCCalloc(1, sizeof(FrameJsonOutputCtx));
     if (unlikely(json_output_ctx == NULL)) {
         goto error;
     }
-    memset(json_output_ctx, 0, sizeof(FrameJsonOutputCtx));
 
     json_output_ctx->file_ctx = ajt->file_ctx;
     json_output_ctx->eve_ctx = ajt;
diff --git a/src/output-json-http2.c b/src/output-json-http2.c
index 7165ae8f6302..cb096f37a043 100644
--- a/src/output-json-http2.c
+++ b/src/output-json-http2.c
@@ -138,7 +138,7 @@ static OutputInitResult OutputHttp2LogInitSub(ConfNode *conf, OutputCtx *parent_
     OutputInitResult result = { NULL, false };
     OutputJsonCtx *ojc = parent_ctx->data;
 
-    OutputHttp2Ctx *http2_ctx = SCMalloc(sizeof(OutputHttp2Ctx));
+    OutputHttp2Ctx *http2_ctx = SCCalloc(1, sizeof(OutputHttp2Ctx));
     if (unlikely(http2_ctx == NULL))
         return result;
 
diff --git a/src/output-json-pgsql.c b/src/output-json-pgsql.c
index d586a48f291a..43eb95709844 100644
--- a/src/output-json-pgsql.c
+++ b/src/output-json-pgsql.c
@@ -114,7 +114,7 @@ static OutputInitResult OutputPgsqlLogInitSub(ConfNode *conf, OutputCtx *parent_
     OutputInitResult result = { NULL, false };
     OutputJsonCtx *ojc = parent_ctx->data;
 
-    OutputPgsqlCtx *pgsql_ctx = SCMalloc(sizeof(OutputPgsqlCtx));
+    OutputPgsqlCtx *pgsql_ctx = SCCalloc(1, sizeof(OutputPgsqlCtx));
     if (unlikely(pgsql_ctx == NULL))
         return result;
 
diff --git a/src/output-lua.c b/src/output-lua.c
index 28ba3e9f91b7..776cf51b9c9b 100644
--- a/src/output-lua.c
+++ b/src/output-lua.c
@@ -606,10 +606,9 @@ static OutputInitResult OutputLuaLogInitSub(ConfNode *conf, OutputCtx *parent_ct
     if (conf == NULL)
         return result;
 
-    LogLuaCtx *lua_ctx = SCMalloc(sizeof(LogLuaCtx));
+    LogLuaCtx *lua_ctx = SCCalloc(1, sizeof(LogLuaCtx));
     if (unlikely(lua_ctx == NULL))
         return result;
-    memset(lua_ctx, 0x00, sizeof(*lua_ctx));
 
     OutputCtx *output_ctx = SCCalloc(1, sizeof(OutputCtx));
     if (unlikely(output_ctx == NULL)) {
@@ -842,10 +841,9 @@ static void OutputLuaLogDoDeinit(LogLuaCtx *lua_ctx)
  */
 static TmEcode LuaLogThreadInit(ThreadVars *t, const void *initdata, void **data)
 {
-    LogLuaThreadCtx *td = SCMalloc(sizeof(*td));
+    LogLuaThreadCtx *td = SCCalloc(1, sizeof(*td));
     if (unlikely(td == NULL))
         return TM_ECODE_FAILED;
-    memset(td, 0, sizeof(*td));
 
     if (initdata == NULL) {
         SCLogDebug("Error getting context for LuaLog. \"initdata\" argument NULL");
diff --git a/src/output-packet.c b/src/output-packet.c
index d42d1033cade..98ccf7b6b081 100644
--- a/src/output-packet.c
+++ b/src/output-packet.c
@@ -58,10 +58,9 @@ int OutputRegisterPacketLogger(LoggerId logger_id, const char *name,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputPacketLogger *op = SCMalloc(sizeof(*op));
+    OutputPacketLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->ConditionFunc = ConditionFunc;
@@ -126,10 +125,9 @@ static TmEcode OutputPacketLog(ThreadVars *tv, Packet *p, void *thread_data)
  *  loggers */
 static TmEcode OutputPacketLogThreadInit(ThreadVars *tv, const void *initdata, void **data)
 {
-    OutputPacketLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputPacketLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
 
     *data = (void *)td;
 
@@ -140,9 +138,8 @@ static TmEcode OutputPacketLogThreadInit(ThreadVars *tv, const void *initdata, v
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-/* todo */      BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                /* todo */ BUG_ON(ts == NULL);
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-stats.c b/src/output-stats.c
index 839606f2bde9..b59432bac4a2 100644
--- a/src/output-stats.c
+++ b/src/output-stats.c
@@ -54,10 +54,9 @@ int OutputRegisterStatsLogger(const char *name, StatsLogger LogFunc,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputStatsLogger *op = SCMalloc(sizeof(*op));
+    OutputStatsLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->output_ctx = output_ctx;
@@ -112,10 +111,9 @@ TmEcode OutputStatsLog(ThreadVars *tv, void *thread_data, StatsTable *st)
  *  loggers */
 static TmEcode OutputStatsLogThreadInit(ThreadVars *tv, const void *initdata, void **data)
 {
-    OutputStatsLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputStatsLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
 
     *data = (void *)td;
 
@@ -126,9 +124,8 @@ static TmEcode OutputStatsLogThreadInit(ThreadVars *tv, const void *initdata, vo
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-/* todo */      BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                /* todo */ BUG_ON(ts == NULL);
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-streaming.c b/src/output-streaming.c
index 4aca9546d4a7..75ee211022f6 100644
--- a/src/output-streaming.c
+++ b/src/output-streaming.c
@@ -67,10 +67,9 @@ int OutputRegisterStreamingLogger(LoggerId id, const char *name,
     ThreadDeinitFunc ThreadDeinit,
     ThreadExitPrintStatsFunc ThreadExitPrintStats)
 {
-    OutputStreamingLogger *op = SCMalloc(sizeof(*op));
+    OutputStreamingLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->LogFunc = LogFunc;
     op->output_ctx = output_ctx;
@@ -365,10 +364,9 @@ static TmEcode OutputStreamingLog(ThreadVars *tv, Packet *p, void *thread_data)
  *  This will run the thread init functions for the individual registered
  *  loggers */
 static TmEcode OutputStreamingLogThreadInit(ThreadVars *tv, const void *initdata, void **data) {
-    OutputStreamingLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputStreamingLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
 
     *data = (void *)td;
 
@@ -379,9 +377,8 @@ static TmEcode OutputStreamingLogThreadInit(ThreadVars *tv, const void *initdata
         if (logger->ThreadInit) {
             void *retptr = NULL;
             if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-/* todo */      BUG_ON(ts == NULL);
-                memset(ts, 0x00, sizeof(*ts));
+                OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                /* todo */ BUG_ON(ts == NULL);
 
                 /* store thread handle */
                 ts->thread_data = retptr;
diff --git a/src/output-tx.c b/src/output-tx.c
index 8eb6a842a656..cf9a1bd11dae 100644
--- a/src/output-tx.c
+++ b/src/output-tx.c
@@ -77,10 +77,9 @@ int OutputRegisterTxLogger(LoggerId id, const char *name, AppProto alproto,
                 "%s logger not enabled: protocol %s is disabled", name, AppProtoToString(alproto));
         return -1;
     }
-    OutputTxLogger *op = SCMalloc(sizeof(*op));
+    OutputTxLogger *op = SCCalloc(1, sizeof(*op));
     if (op == NULL)
         return -1;
-    memset(op, 0x00, sizeof(*op));
 
     op->alproto = alproto;
     op->LogFunc = LogFunc;
@@ -540,10 +539,9 @@ static TmEcode OutputTxLog(ThreadVars *tv, Packet *p, void *thread_data)
  *  loggers */
 static TmEcode OutputTxLogThreadInit(ThreadVars *tv, const void *_initdata, void **data)
 {
-    OutputTxLoggerThreadData *td = SCMalloc(sizeof(*td));
+    OutputTxLoggerThreadData *td = SCCalloc(1, sizeof(*td));
     if (td == NULL)
         return TM_ECODE_FAILED;
-    memset(td, 0x00, sizeof(*td));
 
     *data = (void *)td;
     SCLogDebug("OutputTxLogThreadInit happy (*data %p)", *data);
@@ -554,9 +552,8 @@ static TmEcode OutputTxLogThreadInit(ThreadVars *tv, const void *_initdata, void
             if (logger->ThreadInit) {
                 void *retptr = NULL;
                 if (logger->ThreadInit(tv, (void *)logger->output_ctx, &retptr) == TM_ECODE_OK) {
-                    OutputLoggerThreadStore *ts = SCMalloc(sizeof(*ts));
-    /* todo */      BUG_ON(ts == NULL);
-                    memset(ts, 0x00, sizeof(*ts));
+                    OutputLoggerThreadStore *ts = SCCalloc(1, sizeof(*ts));
+                    /* todo */ BUG_ON(ts == NULL);
 
                     /* store thread handle */
                     ts->thread_data = retptr;
diff --git a/src/reputation.c b/src/reputation.c
index 75f3ba0c3fa4..17f43ca37178 100644
--- a/src/reputation.c
+++ b/src/reputation.c
@@ -80,10 +80,9 @@ static void SRepCIDRFreeUserData(void *data)
 static void SRepCIDRAddNetblock(SRepCIDRTree *cidr_ctx, char *ip, int cat, uint8_t value)
 {
     SReputation *user_data = NULL;
-    if ((user_data = SCMalloc(sizeof(SReputation))) == NULL) {
+    if ((user_data = SCCalloc(1, sizeof(SReputation))) == NULL) {
         FatalError("Error allocating memory. Exiting");
     }
-    memset(user_data, 0x00, sizeof(SReputation));
 
     user_data->version = SRepGetVersion();
     user_data->rep[cat] = value;
@@ -487,10 +486,8 @@ int SRepLoadFileFromFD(SRepCIDRTree *cidr_ctx, FILE *fp)
                 //SCLogInfo("host %p", h);
 
                 if (h->iprep == NULL) {
-                    h->iprep = SCMalloc(sizeof(SReputation));
+                    h->iprep = SCCalloc(1, sizeof(SReputation));
                     if (h->iprep != NULL) {
-                        memset(h->iprep, 0x00, sizeof(SReputation));
-
                         HostIncrUsecnt(h);
                     }
                 }
@@ -589,10 +586,9 @@ int SRepInit(DetectEngineCtx *de_ctx)
     int init = 0;
     int i = 0;
 
-    de_ctx->srepCIDR_ctx = (SRepCIDRTree *)SCMalloc(sizeof(SRepCIDRTree));
+    de_ctx->srepCIDR_ctx = (SRepCIDRTree *)SCCalloc(1, sizeof(SRepCIDRTree));
     if (de_ctx->srepCIDR_ctx == NULL)
         exit(EXIT_FAILURE);
-    memset(de_ctx->srepCIDR_ctx, 0, sizeof(SRepCIDRTree));
     SRepCIDRTree *cidr_ctx = de_ctx->srepCIDR_ctx;
 
     for (i = 0; i < SREP_MAX_CATS; i++) {
diff --git a/src/runmode-unix-socket.c b/src/runmode-unix-socket.c
index e695cb8dfbd6..099d56cbda2d 100644
--- a/src/runmode-unix-socket.c
+++ b/src/runmode-unix-socket.c
@@ -272,12 +272,11 @@ static TmEcode UnixListAddFile(PcapCommand *this, const char *filename, const ch
     PcapFiles *cfile = NULL;
     if (filename == NULL || this == NULL)
         return TM_ECODE_FAILED;
-    cfile = SCMalloc(sizeof(PcapFiles));
+    cfile = SCCalloc(1, sizeof(PcapFiles));
     if (unlikely(cfile == NULL)) {
         SCLogError("Unable to allocate new file");
         return TM_ECODE_FAILED;
     }
-    memset(cfile, 0, sizeof(PcapFiles));
 
     cfile->filename = SCStrdup(filename);
     if (unlikely(cfile->filename == NULL)) {
diff --git a/src/source-af-packet.c b/src/source-af-packet.c
index 6112cb9d8869..3c783a149029 100644
--- a/src/source-af-packet.c
+++ b/src/source-af-packet.c
@@ -490,13 +490,12 @@ TmEcode AFPPeersListCheck(void)
 static TmEcode AFPPeersListAdd(AFPThreadVars *ptv)
 {
     SCEnter();
-    AFPPeer *peer = SCMalloc(sizeof(AFPPeer));
+    AFPPeer *peer = SCCalloc(1, sizeof(AFPPeer));
     AFPPeer *pitem;
 
     if (unlikely(peer == NULL)) {
         SCReturnInt(TM_ECODE_FAILED);
     }
-    memset(peer, 0, sizeof(AFPPeer));
     SC_ATOMIC_INIT(peer->socket);
     SC_ATOMIC_INIT(peer->sock_usage);
     SC_ATOMIC_INIT(peer->if_idx);
@@ -2497,12 +2496,11 @@ TmEcode ReceiveAFPThreadInit(ThreadVars *tv, const void *initdata, void **data)
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    AFPThreadVars *ptv = SCMalloc(sizeof(AFPThreadVars));
+    AFPThreadVars *ptv = SCCalloc(1, sizeof(AFPThreadVars));
     if (unlikely(ptv == NULL)) {
         afpconfig->DerefFunc(afpconfig);
         SCReturnInt(TM_ECODE_FAILED);
     }
-    memset(ptv, 0, sizeof(AFPThreadVars));
 
     ptv->tv = tv;
 
diff --git a/src/source-af-xdp.c b/src/source-af-xdp.c
index 8f4b6071bd75..17fa0efeec4f 100644
--- a/src/source-af-xdp.c
+++ b/src/source-af-xdp.c
@@ -606,12 +606,11 @@ static TmEcode ReceiveAFXDPThreadInit(ThreadVars *tv, const void *initdata, void
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    AFXDPThreadVars *ptv = SCMalloc(sizeof(AFXDPThreadVars));
+    AFXDPThreadVars *ptv = SCCalloc(1, sizeof(AFXDPThreadVars));
     if (unlikely(ptv == NULL)) {
         afxdpconfig->DerefFunc(afxdpconfig);
         SCReturnInt(TM_ECODE_FAILED);
     }
-    memset(ptv, 0, sizeof(AFXDPThreadVars));
 
     ptv->tv = tv;
 
diff --git a/src/source-erf-dag.c b/src/source-erf-dag.c
index b1a8286360cc..e3c820dc4c08 100644
--- a/src/source-erf-dag.c
+++ b/src/source-erf-dag.c
@@ -186,13 +186,11 @@ ReceiveErfDagThreadInit(ThreadVars *tv, void *initdata, void **data)
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    ErfDagThreadVars *ewtn = SCMalloc(sizeof(ErfDagThreadVars));
+    ErfDagThreadVars *ewtn = SCMClloc(1, sizeof(ErfDagThreadVars));
     if (unlikely(ewtn == NULL)) {
         FatalError("Failed to allocate memory for ERF DAG thread vars.");
     }
 
-    memset(ewtn, 0, sizeof(*ewtn));
-
     /* dag_parse_name will return a DAG device name and stream number
      * to open for this thread.
      */
diff --git a/src/source-erf-file.c b/src/source-erf-file.c
index fcbc304d369b..4803f8b3e28f 100644
--- a/src/source-erf-file.c
+++ b/src/source-erf-file.c
@@ -233,13 +233,12 @@ ReceiveErfFileThreadInit(ThreadVars *tv, const void *initdata, void **data)
         exit(EXIT_FAILURE);
     }
 
-    ErfFileThreadVars *etv = SCMalloc(sizeof(ErfFileThreadVars));
+    ErfFileThreadVars *etv = SCCalloc(1, sizeof(ErfFileThreadVars));
     if (unlikely(etv == NULL)) {
         SCLogError("Failed to allocate memory for ERF file thread vars.");
         fclose(erf);
         SCReturnInt(TM_ECODE_FAILED);
     }
-    memset(etv, 0, sizeof(*etv));
     etv->erf = erf;
     etv->tv = tv;
     *data = (void *)etv;
diff --git a/src/source-ipfw.c b/src/source-ipfw.c
index 75bd738fee02..6d0f67c11572 100644
--- a/src/source-ipfw.c
+++ b/src/source-ipfw.c
@@ -650,10 +650,8 @@ TmEcode VerdictIPFWThreadInit(ThreadVars *tv, const void *initdata, void **data)
     SCEnter();
 
     /* Setup Thread vars */
-    if ( (ptv = SCMalloc(sizeof(IPFWThreadVars))) == NULL)
+    if ((ptv = SCCalloc(1, sizeof(IPFWThreadVars))) == NULL)
         SCReturnInt(TM_ECODE_FAILED);
-    memset(ptv, 0, sizeof(IPFWThreadVars));
-
 
     *data = (void *)ptv;
 
diff --git a/src/source-nflog.c b/src/source-nflog.c
index da544e7c6a70..f7d3616c621d 100644
--- a/src/source-nflog.c
+++ b/src/source-nflog.c
@@ -219,12 +219,11 @@ TmEcode ReceiveNFLOGThreadInit(ThreadVars *tv, const void *initdata, void **data
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    NFLOGThreadVars *ntv = SCMalloc(sizeof(NFLOGThreadVars));
+    NFLOGThreadVars *ntv = SCCalloc(1, sizeof(NFLOGThreadVars));
     if (unlikely(ntv == NULL)) {
         nflconfig->DerefFunc(nflconfig);
         SCReturnInt(TM_ECODE_FAILED);
     }
-    memset(ntv, 0, sizeof(NFLOGThreadVars));
 
     ntv->tv = tv;
     ntv->group = nflconfig->group;
diff --git a/src/source-pcap-file-directory-helper.c b/src/source-pcap-file-directory-helper.c
index 17abd03a888a..59c2116f99df 100644
--- a/src/source-pcap-file-directory-helper.c
+++ b/src/source-pcap-file-directory-helper.c
@@ -414,12 +414,11 @@ TmEcode PcapDirectoryDispatchForTimeRange(PcapFileDirectoryVars *pv,
             } else {
                 SCLogDebug("Processing file %s", current_file->filename);
 
-                PcapFileFileVars *pftv = SCMalloc(sizeof(PcapFileFileVars));
+                PcapFileFileVars *pftv = SCCalloc(1, sizeof(PcapFileFileVars));
                 if (unlikely(pftv == NULL)) {
                     SCLogError("Failed to allocate PcapFileFileVars");
                     SCReturnInt(TM_ECODE_FAILED);
                 }
-                memset(pftv, 0, sizeof(PcapFileFileVars));
 
                 pftv->filename = SCStrdup(current_file->filename);
                 if (unlikely(pftv->filename == NULL)) {
diff --git a/src/source-pcap-file.c b/src/source-pcap-file.c
index 547722a32543..e54a607d4875 100644
--- a/src/source-pcap-file.c
+++ b/src/source-pcap-file.c
@@ -205,11 +205,10 @@ TmEcode ReceivePcapFileThreadInit(ThreadVars *tv, const void *initdata, void **d
         SCReturnInt(TM_ECODE_OK);
     }
 
-    PcapFileThreadVars *ptv = SCMalloc(sizeof(PcapFileThreadVars));
+    PcapFileThreadVars *ptv = SCCalloc(1, sizeof(PcapFileThreadVars));
     if (unlikely(ptv == NULL)) {
         SCReturnInt(TM_ECODE_OK);
     }
-    memset(ptv, 0, sizeof(PcapFileThreadVars));
     memset(&ptv->shared.last_processed, 0, sizeof(struct timespec));
 
     intmax_t tenant = 0;
@@ -250,13 +249,12 @@ TmEcode ReceivePcapFileThreadInit(ThreadVars *tv, const void *initdata, void **d
 
     if(directory == NULL) {
         SCLogDebug("argument %s was a file", (char *)initdata);
-        PcapFileFileVars *pv = SCMalloc(sizeof(PcapFileFileVars));
+        PcapFileFileVars *pv = SCCalloc(1, sizeof(PcapFileFileVars));
         if (unlikely(pv == NULL)) {
             SCLogError("Failed to allocate file vars");
             CleanupPcapFileThreadVars(ptv);
             SCReturnInt(TM_ECODE_OK);
         }
-        memset(pv, 0, sizeof(PcapFileFileVars));
 
         pv->filename = SCStrdup((char *)initdata);
         if (unlikely(pv->filename == NULL)) {
@@ -279,14 +277,13 @@ TmEcode ReceivePcapFileThreadInit(ThreadVars *tv, const void *initdata, void **d
         }
     } else {
         SCLogInfo("Argument %s was a directory", (char *)initdata);
-        PcapFileDirectoryVars *pv = SCMalloc(sizeof(PcapFileDirectoryVars));
+        PcapFileDirectoryVars *pv = SCCalloc(1, sizeof(PcapFileDirectoryVars));
         if (unlikely(pv == NULL)) {
             SCLogError("Failed to allocate directory vars");
             closedir(directory);
             CleanupPcapFileThreadVars(ptv);
             SCReturnInt(TM_ECODE_OK);
         }
-        memset(pv, 0, sizeof(PcapFileDirectoryVars));
 
         pv->filename = SCStrdup((char*)initdata);
         if (unlikely(pv->filename == NULL)) {
diff --git a/src/source-pfring.c b/src/source-pfring.c
index 96da94eff533..40a42723d6a7 100644
--- a/src/source-pfring.c
+++ b/src/source-pfring.c
@@ -497,12 +497,11 @@ TmEcode ReceivePfringThreadInit(ThreadVars *tv, const void *initdata, void **dat
     if (pfconf == NULL)
         return TM_ECODE_FAILED;
 
-    PfringThreadVars *ptv = SCMalloc(sizeof(PfringThreadVars));
+    PfringThreadVars *ptv = SCCalloc(1, sizeof(PfringThreadVars));
     if (unlikely(ptv == NULL)) {
         pfconf->DerefFunc(pfconf);
         return TM_ECODE_FAILED;
     }
-    memset(ptv, 0, sizeof(PfringThreadVars));
 
     ptv->tv = tv;
     ptv->threads = 1;
diff --git a/src/stream-tcp-reassemble.c b/src/stream-tcp-reassemble.c
index 06992da791a5..9d705f1f5277 100644
--- a/src/stream-tcp-reassemble.c
+++ b/src/stream-tcp-reassemble.c
@@ -561,12 +561,10 @@ void StreamTcpReassembleFree(bool quiet)
 TcpReassemblyThreadCtx *StreamTcpReassembleInitThreadCtx(ThreadVars *tv)
 {
     SCEnter();
-    TcpReassemblyThreadCtx *ra_ctx = SCMalloc(sizeof(TcpReassemblyThreadCtx));
+    TcpReassemblyThreadCtx *ra_ctx = SCCalloc(1, sizeof(TcpReassemblyThreadCtx));
     if (unlikely(ra_ctx == NULL))
         return NULL;
 
-    memset(ra_ctx, 0x00, sizeof(TcpReassemblyThreadCtx));
-
     ra_ctx->app_tctx = AppLayerGetCtxThread(tv);
 
     SCMutexLock(&segment_thread_pool_mutex);
diff --git a/src/stream-tcp.c b/src/stream-tcp.c
index d41110aac5c7..b77423161800 100644
--- a/src/stream-tcp.c
+++ b/src/stream-tcp.c
@@ -5750,10 +5750,9 @@ TmEcode StreamTcp (ThreadVars *tv, Packet *p, void *data, PacketQueueNoLock *pq)
 TmEcode StreamTcpThreadInit(ThreadVars *tv, void *initdata, void **data)
 {
     SCEnter();
-    StreamTcpThread *stt = SCMalloc(sizeof(StreamTcpThread));
+    StreamTcpThread *stt = SCCalloc(1, sizeof(StreamTcpThread));
     if (unlikely(stt == NULL))
         SCReturnInt(TM_ECODE_FAILED);
-    memset(stt, 0, sizeof(StreamTcpThread));
     stt->ssn_pool_id = -1;
     StreamTcpThreadCacheEnable();
 
diff --git a/src/suricata.c b/src/suricata.c
index fd069e6e5ddd..ffa970ae7297 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -471,10 +471,9 @@ static int SetBpfString(int argc, char *argv[])
     if (bpf_len == 0)
         return TM_ECODE_OK;
 
-    bpf_filter = SCMalloc(bpf_len);
+    bpf_filter = SCCalloc(1, bpf_len);
     if (unlikely(bpf_filter == NULL))
         return TM_ECODE_FAILED;
-    memset(bpf_filter, 0x00, bpf_len);
 
     tmpindex = optind;
     while(argv[tmpindex] != NULL) {
@@ -519,12 +518,11 @@ static void SetBpfStringFromFile(char *filename)
     }
     bpf_len = st.st_size + 1;
 
-    bpf_filter = SCMalloc(bpf_len);
+    bpf_filter = SCCalloc(1, bpf_len);
     if (unlikely(bpf_filter == NULL)) {
         SCLogError("Failed to allocate buffer for bpf filter in file %s", filename);
         exit(EXIT_FAILURE);
     }
-    memset(bpf_filter, 0x00, bpf_len);
 
     nm = fread(bpf_filter, 1, bpf_len - 1, fp);
     if ((ferror(fp) != 0) || (nm != (bpf_len - 1))) {
diff --git a/src/tm-threads.c b/src/tm-threads.c
index b173cb84f442..10ef45da278f 100644
--- a/src/tm-threads.c
+++ b/src/tm-threads.c
@@ -641,10 +641,9 @@ static TmEcode TmThreadSetSlots(ThreadVars *tv, const char *name, void *(*fn_p)(
  */
 void TmSlotSetFuncAppend(ThreadVars *tv, TmModule *tm, const void *data)
 {
-    TmSlot *slot = SCMalloc(sizeof(TmSlot));
+    TmSlot *slot = SCCalloc(1, sizeof(TmSlot));
     if (unlikely(slot == NULL))
         return;
-    memset(slot, 0, sizeof(TmSlot));
     SC_ATOMIC_INITPTR(slot->slot_data);
     slot->SlotThreadInit = tm->ThreadInit;
     slot->slot_initdata = data;
@@ -916,10 +915,9 @@ ThreadVars *TmThreadCreate(const char *name, const char *inq_name, const char *i
     SCLogDebug("creating thread \"%s\"...", name);
 
     /* XXX create separate function for this: allocate a thread container */
-    tv = SCMalloc(sizeof(ThreadVars));
+    tv = SCCalloc(1, sizeof(ThreadVars));
     if (unlikely(tv == NULL))
         goto error;
-    memset(tv, 0, sizeof(ThreadVars));
 
     SC_ATOMIC_INIT(tv->flags);
     SCMutexInit(&tv->perf_public_ctx.m, NULL);
diff --git a/src/tmqh-flow.c b/src/tmqh-flow.c
index 4d4f8b9a8bed..e11d05d71373 100644
--- a/src/tmqh-flow.c
+++ b/src/tmqh-flow.c
@@ -133,11 +133,10 @@ static int StoreQueueId(TmqhFlowCtx *ctx, char *name)
 
     if (ctx->queues == NULL) {
         ctx->size = 1;
-        ctx->queues = SCMalloc(ctx->size * sizeof(TmqhFlowMode));
+        ctx->queues = SCCalloc(1, ctx->size * sizeof(TmqhFlowMode));
         if (ctx->queues == NULL) {
             return -1;
         }
-        memset(ctx->queues, 0, ctx->size * sizeof(TmqhFlowMode));
     } else {
         ctx->size++;
         ptmp = SCRealloc(ctx->queues, ctx->size * sizeof(TmqhFlowMode));
@@ -172,10 +171,9 @@ void *TmqhOutputFlowSetupCtx(const char *queue_str)
 
     SCLogDebug("queue_str %s", queue_str);
 
-    TmqhFlowCtx *ctx = SCMalloc(sizeof(TmqhFlowCtx));
+    TmqhFlowCtx *ctx = SCCalloc(1, sizeof(TmqhFlowCtx));
     if (unlikely(ctx == NULL))
         return NULL;
-    memset(ctx,0x00,sizeof(TmqhFlowCtx));
 
     char *str = SCStrdup(queue_str);
     if (unlikely(str == NULL)) {
diff --git a/src/util-bloomfilter-counting.c b/src/util-bloomfilter-counting.c
index cbf99e5bae48..6fc9cadc7deb 100644
--- a/src/util-bloomfilter-counting.c
+++ b/src/util-bloomfilter-counting.c
@@ -48,20 +48,18 @@ BloomFilterCounting *BloomFilterCountingInit(uint32_t size, uint8_t type, uint8_
     }
 
     /* setup the filter */
-    bf = SCMalloc(sizeof(BloomFilterCounting));
+    bf = SCCalloc(1, sizeof(BloomFilterCounting));
     if (unlikely(bf == NULL))
         goto error;
-    memset(bf,0,sizeof(BloomFilterCounting));
     bf->type = type; /* size of the type: 1, 2, 4 */
     bf->array_size = size;
     bf->hash_iterations = iter;
     bf->Hash = Hash;
 
     /* setup the bitarray */
-    bf->array = SCMalloc(bf->array_size * bf->type);
+    bf->array = SCCalloc(1, bf->array_size * bf->type);
     if (bf->array == NULL)
         goto error;
-    memset(bf->array,0,bf->array_size * bf->type);
 
     return bf;
 
diff --git a/src/util-bloomfilter.c b/src/util-bloomfilter.c
index ae0b0453222a..5d2549c6a9a4 100644
--- a/src/util-bloomfilter.c
+++ b/src/util-bloomfilter.c
@@ -40,19 +40,17 @@ BloomFilter *BloomFilterInit(uint32_t size, uint8_t iter,
     }
 
     /* setup the filter */
-    bf = SCMalloc(sizeof(BloomFilter));
+    bf = SCCalloc(1, sizeof(BloomFilter));
     if (unlikely(bf == NULL))
         goto error;
-    memset(bf,0,sizeof(BloomFilter));
     bf->bitarray_size = size;
     bf->hash_iterations = iter;
     bf->Hash = Hash;
 
     /* setup the bitarray */
-    bf->bitarray = SCMalloc((bf->bitarray_size/8)+1);
+    bf->bitarray = SCCalloc(1, (bf->bitarray_size / 8) + 1);
     if (bf->bitarray == NULL)
         goto error;
-    memset(bf->bitarray,0,(bf->bitarray_size/8)+1);
 
     return bf;
 
diff --git a/src/util-buffer.c b/src/util-buffer.c
index 2dd94f6eac3b..be7aee4046c8 100644
--- a/src/util-buffer.c
+++ b/src/util-buffer.c
@@ -42,12 +42,11 @@ MemBuffer *MemBufferCreateNew(uint32_t size)
 
     uint32_t total_size = size + sizeof(MemBuffer);
 
-    MemBuffer *buffer = SCMalloc(total_size);
+    MemBuffer *buffer = SCCalloc(1, total_size);
     if (unlikely(buffer == NULL)) {
         sc_errno = SC_ENOMEM;
         return NULL;
     }
-    memset(buffer, 0, total_size);
 
     buffer->size = size;
     buffer->buffer = (uint8_t *)buffer + sizeof(MemBuffer);
diff --git a/src/util-classification-config.c b/src/util-classification-config.c
index 9d7ed05bde32..5bcc0960c887 100644
--- a/src/util-classification-config.c
+++ b/src/util-classification-config.c
@@ -396,9 +396,8 @@ static SCClassConfClasstype *SCClassConfAllocClasstype(uint16_t classtype_id,
     if (classtype == NULL)
         return NULL;
 
-    if ( (ct = SCMalloc(sizeof(SCClassConfClasstype))) == NULL)
+    if ((ct = SCCalloc(1, sizeof(SCClassConfClasstype))) == NULL)
         return NULL;
-    memset(ct, 0, sizeof(SCClassConfClasstype));
 
     if ((ct->classtype = SCClassConfStringToLowercase(classtype)) == NULL) {
         SCClassConfDeAllocClasstype(ct);
diff --git a/src/util-debug-filters.c b/src/util-debug-filters.c
index fe6c4d5e9aa7..fd66375cb9fc 100644
--- a/src/util-debug-filters.c
+++ b/src/util-debug-filters.c
@@ -579,11 +579,10 @@ int SCLogCheckFDFilterEntry(const char *function)
         return 1;
     }
 
-    if ( (thread_list_temp = SCMalloc(sizeof(SCLogFDFilterThreadList))) == NULL) {
+    if ((thread_list_temp = SCCalloc(1, sizeof(SCLogFDFilterThreadList))) == NULL) {
         SCMutexUnlock(&sc_log_fd_filters_tl_m);
         return 0;
     }
-    memset(thread_list_temp, 0, sizeof(SCLogFDFilterThreadList));
 
     thread_list_temp->t = self;
     thread_list_temp->entered++;
@@ -694,11 +693,10 @@ int SCLogAddFDFilter(const char *function)
         curr = curr->next;
     }
 
-    if ( (temp = SCMalloc(sizeof(SCLogFDFilter))) == NULL) {
-        printf("Error Allocating memory (SCMalloc)\n");
+    if ((temp = SCCalloc(1, sizeof(SCLogFDFilter))) == NULL) {
+        printf("Error Allocating memory (SCCalloc)\n");
         exit(EXIT_FAILURE);
     }
-    memset(temp, 0, sizeof(SCLogFDFilter));
 
     if ( (temp->func = SCStrdup(function)) == NULL) {
         printf("Error Allocating memory (SCStrdup)\n");
@@ -864,30 +862,27 @@ void SCLogAddToFGFFileList(SCLogFGFilterFile *fgf_file,
     SCLogFGFilterFunc *fgf_func_temp = NULL;
     SCLogFGFilterLine *fgf_line_temp = NULL;
 
-    if ( (fgf_file_temp = SCMalloc(sizeof(SCLogFGFilterFile))) == NULL) {
+    if ((fgf_file_temp = SCCalloc(1, sizeof(SCLogFGFilterFile))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFFileList. Exiting...");
     }
-    memset(fgf_file_temp, 0, sizeof(SCLogFGFilterFile));
 
     if ( file != NULL && (fgf_file_temp->file = SCStrdup(file)) == NULL) {
         printf("Error Allocating memory\n");
         exit(EXIT_FAILURE);
     }
 
-    if ( (fgf_func_temp = SCMalloc(sizeof(SCLogFGFilterFunc))) == NULL) {
+    if ((fgf_func_temp = SCCalloc(1, sizeof(SCLogFGFilterFunc))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFFileList. Exiting...");
     }
-    memset(fgf_func_temp, 0, sizeof(SCLogFGFilterFunc));
 
     if ( function != NULL && (fgf_func_temp->func = SCStrdup(function)) == NULL) {
         printf("Error Allocating memory\n");
         exit(EXIT_FAILURE);
     }
 
-    if ( (fgf_line_temp = SCMalloc(sizeof(SCLogFGFilterLine))) == NULL) {
+    if ((fgf_line_temp = SCCalloc(1, sizeof(SCLogFGFilterLine))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFFileList. Exiting...");
     }
-    memset(fgf_line_temp, 0, sizeof(SCLogFGFilterLine));
 
     fgf_line_temp->line = line;
 
@@ -925,20 +920,18 @@ void SCLogAddToFGFFuncList(SCLogFGFilterFile *fgf_file,
     SCLogFGFilterFunc *fgf_func_temp = NULL;
     SCLogFGFilterLine *fgf_line_temp = NULL;
 
-    if ( (fgf_func_temp = SCMalloc(sizeof(SCLogFGFilterFunc))) == NULL) {
+    if ((fgf_func_temp = SCCalloc(1, sizeof(SCLogFGFilterFunc))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFFuncList. Exiting...");
     }
-    memset(fgf_func_temp, 0, sizeof(SCLogFGFilterFunc));
 
     if ( function != NULL && (fgf_func_temp->func = SCStrdup(function)) == NULL) {
         printf("Error Allocating memory\n");
         exit(EXIT_FAILURE);
     }
 
-    if ( (fgf_line_temp = SCMalloc(sizeof(SCLogFGFilterLine))) == NULL) {
+    if ((fgf_line_temp = SCCalloc(1, sizeof(SCLogFGFilterLine))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFFuncList. Exiting...");
     }
-    memset(fgf_line_temp, 0, sizeof(SCLogFGFilterLine));
 
     fgf_line_temp->line = line;
 
@@ -972,10 +965,9 @@ void SCLogAddToFGFLineList(SCLogFGFilterFunc *fgf_func,
 {
     SCLogFGFilterLine *fgf_line_temp = NULL;
 
-    if ( (fgf_line_temp = SCMalloc(sizeof(SCLogFGFilterLine))) == NULL) {
+    if ((fgf_line_temp = SCCalloc(1, sizeof(SCLogFGFilterLine))) == NULL) {
         FatalError("Fatal error encountered in SCLogAddToFGFLineList. Exiting...");
     }
-    memset(fgf_line_temp, 0, sizeof(SCLogFGFilterLine));
 
     fgf_line_temp->line = line;
 
diff --git a/src/util-decode-mime.c b/src/util-decode-mime.c
index 5e7a8d5713f4..d9941cd986b9 100644
--- a/src/util-decode-mime.c
+++ b/src/util-decode-mime.c
@@ -266,11 +266,10 @@ void MimeDecFreeUrl(MimeDecUrl *url)
  */
 MimeDecField * MimeDecAddField(MimeDecEntity *entity)
 {
-    MimeDecField *node = SCMalloc(sizeof(MimeDecField));
+    MimeDecField *node = SCCalloc(1, sizeof(MimeDecField));
     if (unlikely(node == NULL)) {
         return NULL;
     }
-    memset(node, 0x00, sizeof(MimeDecField));
 
     /* If list is empty, then set as head of list */
     if (entity->field_list == NULL) {
@@ -351,11 +350,10 @@ MimeDecField * MimeDecFindField(const MimeDecEntity *entity, const char *name) {
  */
 static MimeDecUrl * MimeDecAddUrl(MimeDecEntity *entity, uint8_t *url, uint32_t url_len, uint8_t flags)
 {
-    MimeDecUrl *node = SCMalloc(sizeof(MimeDecUrl));
+    MimeDecUrl *node = SCCalloc(1, sizeof(MimeDecUrl));
     if (unlikely(node == NULL)) {
         return NULL;
     }
-    memset(node, 0x00, sizeof(MimeDecUrl));
 
     node->url = url;
     node->url_len = url_len;
@@ -384,11 +382,10 @@ static MimeDecUrl * MimeDecAddUrl(MimeDecEntity *entity, uint8_t *url, uint32_t
  */
 MimeDecEntity * MimeDecAddEntity(MimeDecEntity *parent)
 {
-    MimeDecEntity *node = SCMalloc(sizeof(MimeDecEntity));
+    MimeDecEntity *node = SCCalloc(1, sizeof(MimeDecEntity));
     if (unlikely(node == NULL)) {
         return NULL;
     }
-    memset(node, 0x00, sizeof(MimeDecEntity));
 
     /* If parent is NULL then just return the new pointer */
     if (parent != NULL) {
@@ -464,7 +461,7 @@ static MimeDecStackNode * PushStack(MimeDecStack *stack)
     /* Attempt to pull from free nodes list */
     MimeDecStackNode *node = stack->free_nodes;
     if (node == NULL) {
-        node = SCMalloc(sizeof(MimeDecStackNode));
+        node = SCCalloc(1, sizeof(MimeDecStackNode));
         if (unlikely(node == NULL)) {
             return NULL;
         }
@@ -472,8 +469,8 @@ static MimeDecStackNode * PushStack(MimeDecStack *stack)
         /* Move free nodes pointer over */
         stack->free_nodes = stack->free_nodes->next;
         stack->free_nodes_cnt--;
+        memset(node, 0x00, sizeof(MimeDecStackNode));
     }
-    memset(node, 0x00, sizeof(MimeDecStackNode));
 
     /* Push to top of stack */
     node->next = stack->top;
@@ -561,11 +558,10 @@ static void FreeMimeDecStack(MimeDecStack *stack)
  */
 static DataValue * AddDataValue(DataValue *dv)
 {
-    DataValue *curr, *node = SCMalloc(sizeof(DataValue));
+    DataValue *curr, *node = SCCalloc(1, sizeof(DataValue));
     if (unlikely(node == NULL)) {
         return NULL;
     }
-    memset(node, 0x00, sizeof(DataValue));
 
     if (dv != NULL) {
         curr = dv;
@@ -2412,26 +2408,23 @@ MimeDecParseState * MimeDecInitParser(void *data,
     MimeDecParseState *state;
     MimeDecEntity *mimeMsg;
 
-    state = SCMalloc(sizeof(MimeDecParseState));
+    state = SCCalloc(1, sizeof(MimeDecParseState));
     if (unlikely(state == NULL)) {
         return NULL;
     }
-    memset(state, 0x00, sizeof(MimeDecParseState));
 
-    state->stack = SCMalloc(sizeof(MimeDecStack));
+    state->stack = SCCalloc(1, sizeof(MimeDecStack));
     if (unlikely(state->stack == NULL)) {
         SCFree(state);
         return NULL;
     }
-    memset(state->stack, 0x00, sizeof(MimeDecStack));
 
-    mimeMsg = SCMalloc(sizeof(MimeDecEntity));
+    mimeMsg = SCCalloc(1, sizeof(MimeDecEntity));
     if (unlikely(mimeMsg == NULL)) {
         SCFree(state->stack);
         SCFree(state);
         return NULL;
     }
-    memset(mimeMsg, 0x00, sizeof(MimeDecEntity));
     mimeMsg->ctnt_flags |= CTNT_IS_MSG;
 
     /* Init state */
diff --git a/src/util-file.c b/src/util-file.c
index 0449a2edae6c..3221d116870d 100644
--- a/src/util-file.c
+++ b/src/util-file.c
@@ -493,12 +493,11 @@ static void FilePrune(FileContainer *ffc, const StreamingBufferConfig *cfg)
  */
 FileContainer *FileContainerAlloc(void)
 {
-    FileContainer *new = SCMalloc(sizeof(FileContainer));
+    FileContainer *new = SCCalloc(1, sizeof(FileContainer));
     if (unlikely(new == NULL)) {
         SCLogError("Error allocating mem");
         return NULL;
     }
-    memset(new, 0, sizeof(FileContainer));
     new->head = new->tail = NULL;
     return new;
 }
@@ -554,12 +553,11 @@ void FileContainerFree(FileContainer *ffc, const StreamingBufferConfig *cfg)
  */
 static File *FileAlloc(const uint8_t *name, uint16_t name_len)
 {
-    File *new = SCMalloc(sizeof(File));
+    File *new = SCCalloc(1, sizeof(File));
     if (unlikely(new == NULL)) {
         SCLogError("Error allocating mem");
         return NULL;
     }
-    memset(new, 0, sizeof(File));
 
     new->name = SCMalloc(name_len);
     if (new->name == NULL) {
diff --git a/src/util-fmemopen.c b/src/util-fmemopen.c
index e40cf8ff39c9..412b9783ef3a 100644
--- a/src/util-fmemopen.c
+++ b/src/util-fmemopen.c
@@ -183,11 +183,10 @@ static int CloseFn(void *handler)
  */
 FILE *SCFmemopen(void *buf, size_t size, const char *mode)
 {
-    SCFmem *mem = (SCFmem *) SCMalloc(sizeof(SCFmem));
+    SCFmem *mem = (SCFmem *)SCCalloc(1, sizeof(SCFmem));
     if (mem == NULL)
         return NULL;
 
-    memset(mem, 0, sizeof(SCFmem));
     mem->size = size, mem->buffer = buf;
 
     return funopen(mem, ReadFn, WriteFn, SeekFn, CloseFn);
diff --git a/src/util-hash.c b/src/util-hash.c
index d94d46f943fb..a81882d52ac1 100644
--- a/src/util-hash.c
+++ b/src/util-hash.c
@@ -46,10 +46,9 @@ HashTable* HashTableInit(uint32_t size, uint32_t (*Hash)(struct HashTable_ *, vo
     }
 
     /* setup the filter */
-    ht = SCMalloc(sizeof(HashTable));
+    ht = SCCalloc(1, sizeof(HashTable));
     if (unlikely(ht == NULL))
-    goto error;
-    memset(ht,0,sizeof(HashTable));
+        goto error;
     ht->array_size = size;
     ht->Hash = Hash;
     ht->Free = Free;
@@ -60,10 +59,9 @@ HashTable* HashTableInit(uint32_t size, uint32_t (*Hash)(struct HashTable_ *, vo
         ht->Compare = HashTableDefaultCompare;
 
     /* setup the bitarray */
-    ht->array = SCMalloc(ht->array_size * sizeof(HashTableBucket *));
+    ht->array = SCCalloc(1, ht->array_size * sizeof(HashTableBucket *));
     if (ht->array == NULL)
         goto error;
-    memset(ht->array,0,ht->array_size * sizeof(HashTableBucket *));
 
     return ht;
 
@@ -118,10 +116,9 @@ int HashTableAdd(HashTable *ht, void *data, uint16_t datalen)
 
     uint32_t hash = ht->Hash(ht, data, datalen);
 
-    HashTableBucket *hb = SCMalloc(sizeof(HashTableBucket));
+    HashTableBucket *hb = SCCalloc(1, sizeof(HashTableBucket));
     if (unlikely(hb == NULL))
         goto error;
-    memset(hb, 0, sizeof(HashTableBucket));
     hb->data = data;
     hb->size = datalen;
     hb->next = NULL;
diff --git a/src/util-hashlist.c b/src/util-hashlist.c
index 1a6df14fe348..88f8144b3c37 100644
--- a/src/util-hashlist.c
+++ b/src/util-hashlist.c
@@ -50,12 +50,11 @@ HashListTable *HashListTableInit(uint32_t size,
     }
 
     /* setup the filter */
-    ht = SCMalloc(sizeof(HashListTable));
+    ht = SCCalloc(1, sizeof(HashListTable));
     if (unlikely(ht == NULL)) {
         sc_errno = SC_ENOMEM;
         goto error;
     }
-    memset(ht,0,sizeof(HashListTable));
     ht->array_size = size;
     ht->Hash = Hash;
     ht->Free = Free;
@@ -66,12 +65,11 @@ HashListTable *HashListTableInit(uint32_t size,
         ht->Compare = HashListTableDefaultCompare;
 
     /* setup the bitarray */
-    ht->array = SCMalloc(ht->array_size * sizeof(HashListTableBucket *));
+    ht->array = SCCalloc(1, ht->array_size * sizeof(HashListTableBucket *));
     if (ht->array == NULL) {
         sc_errno = SC_ENOMEM;
         goto error;
     }
-    memset(ht->array,0,ht->array_size * sizeof(HashListTableBucket *));
 
     ht->listhead = NULL;
     ht->listtail = NULL;
@@ -130,10 +128,9 @@ int HashListTableAdd(HashListTable *ht, void *data, uint16_t datalen)
 
     SCLogDebug("ht %p hash %"PRIu32"", ht, hash);
 
-    HashListTableBucket *hb = SCMalloc(sizeof(HashListTableBucket));
+    HashListTableBucket *hb = SCCalloc(1, sizeof(HashListTableBucket));
     if (unlikely(hb == NULL))
         goto error;
-    memset(hb, 0, sizeof(HashListTableBucket));
     hb->data = data;
     hb->size = datalen;
     hb->bucknext = NULL;
diff --git a/src/util-hyperscan.c b/src/util-hyperscan.c
index c5886d6691ca..c49012b6b202 100644
--- a/src/util-hyperscan.c
+++ b/src/util-hyperscan.c
@@ -43,11 +43,10 @@ char *HSRenderPattern(const uint8_t *pat, uint16_t pat_len)
         return NULL;
     }
     const size_t hex_len = (pat_len * 4) + 1;
-    char *str = SCMalloc(hex_len);
+    char *str = SCCalloc(1, hex_len);
     if (str == NULL) {
         return NULL;
     }
-    memset(str, 0, hex_len);
     char *sp = str;
     for (uint16_t i = 0; i < pat_len; i++) {
         snprintf(sp, 5, "\\x%02x", pat[i]);
diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c
index 304894b71250..aebcd778a6b7 100644
--- a/src/util-mpm-ac-bs.c
+++ b/src/util-mpm-ac-bs.c
@@ -420,11 +420,10 @@ static inline void SCACBSCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* allot space for the failure table.  A failure entry in the table for
      * every state(SCACBSCtx->state_count) */
-    ctx->failure_table = SCMalloc(ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
-    memset(ctx->failure_table, 0, ctx->state_count * sizeof(int32_t));
 
     /* add the failure transitions for the 0th state, and add every non-fail
      * transition from the 0th state to the queue for further processing
@@ -672,23 +671,20 @@ static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx)
          * but by avoiding it, we save a lot of time on handling alignment */
         size += (ctx->state_count * sizeof(SC_AC_BS_STATE_TYPE_U16) +
                  256 * sizeof(SC_AC_BS_STATE_TYPE_U16) * 1);
-        ctx->state_table_mod = SCMalloc(size);
+        ctx->state_table_mod = SCCalloc(1, size);
         if (ctx->state_table_mod == NULL) {
             FatalError("Error allocating memory");
         }
-        memset(ctx->state_table_mod, 0, size);
 
         mpm_ctx->memory_cnt++;
         mpm_ctx->memory_size += size;
 
         /* buffer to hold pointers in the buffer, so that a state can use it
          * directly to access its state data */
-        ctx->state_table_mod_pointers = SCMalloc(ctx->state_count * sizeof(uint8_t *));
+        ctx->state_table_mod_pointers = SCCalloc(1, ctx->state_count * sizeof(uint8_t *));
         if (ctx->state_table_mod_pointers == NULL) {
             FatalError("Error allocating memory");
         }
-        memset(ctx->state_table_mod_pointers, 0,
-               ctx->state_count * sizeof(uint8_t *));
 
         SC_AC_BS_STATE_TYPE_U16 temp_states[256];
         uint16_t *curr_loc = (uint16_t *)ctx->state_table_mod;
@@ -744,23 +740,20 @@ static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx)
          * but by avoiding it, we save a lot of time on handling alignment */
         size += (ctx->state_count * sizeof(SC_AC_BS_STATE_TYPE_U32) +
                  256 * sizeof(SC_AC_BS_STATE_TYPE_U32) * 1);
-        ctx->state_table_mod = SCMalloc(size);
+        ctx->state_table_mod = SCCalloc(1, size);
         if (ctx->state_table_mod == NULL) {
             FatalError("Error allocating memory");
         }
-        memset(ctx->state_table_mod, 0, size);
 
         mpm_ctx->memory_cnt++;
         mpm_ctx->memory_size += size;
 
         /* buffer to hold pointers in the buffer, so that a state can use it
          * directly to access its state data */
-        ctx->state_table_mod_pointers = SCMalloc(ctx->state_count * sizeof(uint8_t *));
+        ctx->state_table_mod_pointers = SCCalloc(1, ctx->state_count * sizeof(uint8_t *));
         if (ctx->state_table_mod_pointers == NULL) {
             FatalError("Error allocating memory");
         }
-        memset(ctx->state_table_mod_pointers, 0,
-               ctx->state_count * sizeof(uint8_t *));
 
         SC_AC_BS_STATE_TYPE_U32 temp_states[256];
         uint32_t *curr_loc = (uint32_t *)ctx->state_table_mod;
@@ -868,11 +861,9 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCMalloc(mpm_ctx->pattern_cnt *
-                                           sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
-    memset(ctx->parray, 0, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += (mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
 
@@ -896,11 +887,10 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
     ctx->single_state_size = sizeof(int32_t) * 256;
 
     /* handle no case patterns */
-    ctx->pid_pat_list = SCMalloc((mpm_ctx->max_pat_id + 1)* sizeof(SCACBSPatternList));
+    ctx->pid_pat_list = SCCalloc(1, (mpm_ctx->max_pat_id + 1) * sizeof(SCACBSPatternList));
     if (ctx->pid_pat_list == NULL) {
         FatalError("Error allocating memory");
     }
-    memset(ctx->pid_pat_list, 0, (mpm_ctx->max_pat_id + 1) * sizeof(SCACBSPatternList));
 
     for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
         if (!(ctx->parray[i]->flags & MPM_PATTERN_FLAG_NOCASE)) {
@@ -950,21 +940,19 @@ void SCACBSInitCtx(MpmCtx *mpm_ctx)
     if (mpm_ctx->ctx != NULL)
         return;
 
-    mpm_ctx->ctx = SCMalloc(sizeof(SCACBSCtx));
+    mpm_ctx->ctx = SCCalloc(1, sizeof(SCACBSCtx));
     if (mpm_ctx->ctx == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->ctx, 0, sizeof(SCACBSCtx));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCACBSCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCMalloc(sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->init_hash, 0, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
 
     /* get conf values for AC from our yaml file.  We have no conf values for
      * now.  We will certainly need this, as we develop the algo */
diff --git a/src/util-mpm-ac-ks.c b/src/util-mpm-ac-ks.c
index 5f47cf495da5..9b2c799eef11 100644
--- a/src/util-mpm-ac-ks.c
+++ b/src/util-mpm-ac-ks.c
@@ -508,11 +508,10 @@ static void SCACTileCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* Allocate space for the failure table.  A failure entry in the table for
      * every state(SCACTileCtx->state_count) */
-    ctx->failure_table = SCMalloc(ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
-    memset(ctx->failure_table, 0, ctx->state_count * sizeof(int32_t));
 
     /* Add the failure transitions for the 0th state, and add every non-fail
      * transition from the 0th state to the queue for further processing
@@ -709,11 +708,10 @@ static void SCACTileClubOutputStatePresenceWithDeltaTable(MpmCtx *mpm_ctx)
 
     /* Allocate next-state table. */
     int size = ctx->state_count * ctx->bytes_per_state * ctx->alphabet_storage;
-    void *state_table = SCMalloc(size);
+    void *state_table = SCCalloc(1, size);
     if (unlikely(state_table == NULL)) {
         FatalError("Error allocating memory");
     }
-    memset(state_table, 0, size);
     ctx->state_table = state_table;
 
     mpm_ctx->memory_cnt++;
@@ -876,11 +874,9 @@ int SCACTilePreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCMalloc(mpm_ctx->pattern_cnt *
-                                               sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
-    memset(ctx->parray, 0, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
 
     /* populate it with the patterns in the hash */
     uint32_t i = 0, p = 0;
@@ -969,11 +965,10 @@ void SCACTileInitCtx(MpmCtx *mpm_ctx)
         return;
 
     /* Search Context */
-    mpm_ctx->ctx = SCMalloc(sizeof(SCACTileSearchCtx));
+    mpm_ctx->ctx = SCCalloc(1, sizeof(SCACTileSearchCtx));
     if (mpm_ctx->ctx == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->ctx, 0, sizeof(SCACTileSearchCtx));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCACTileSearchCtx);
@@ -981,21 +976,19 @@ void SCACTileInitCtx(MpmCtx *mpm_ctx)
     SCACTileSearchCtx *search_ctx = (SCACTileSearchCtx *)mpm_ctx->ctx;
 
     /* MPM Creation context */
-    search_ctx->init_ctx = SCMalloc(sizeof(SCACTileCtx));
+    search_ctx->init_ctx = SCCalloc(1, sizeof(SCACTileCtx));
     if (search_ctx->init_ctx == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(search_ctx->init_ctx, 0, sizeof(SCACTileCtx));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCACTileCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCMalloc(sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->init_hash, 0, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
 
     /* get conf values for AC from our yaml file.  We have no conf values for
      * now.  We will certainly need this, as we develop the algo */
diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index 6d1d44b30d14..cb663b8aba9e 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -476,11 +476,10 @@ static inline void SCACCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* allot space for the failure table.  A failure entry in the table for
      * every state(SCACCtx->state_count) */
-    ctx->failure_table = SCMalloc(ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
-    memset(ctx->failure_table, 0, ctx->state_count * sizeof(int32_t));
 
     /* add the failure transitions for the 0th state, and add every non-fail
      * transition from the 0th state to the queue for further processing
@@ -737,11 +736,9 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCMalloc(mpm_ctx->pattern_cnt *
-                                           sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
-    memset(ctx->parray, 0, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += (mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
 
@@ -765,11 +762,10 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx)
     ctx->single_state_size = sizeof(int32_t) * 256;
 
     /* handle no case patterns */
-    ctx->pid_pat_list = SCMalloc((mpm_ctx->max_pat_id + 1)* sizeof(SCACPatternList));
+    ctx->pid_pat_list = SCCalloc(1, (mpm_ctx->max_pat_id + 1) * sizeof(SCACPatternList));
     if (ctx->pid_pat_list == NULL) {
         FatalError("Error allocating memory");
     }
-    memset(ctx->pid_pat_list, 0, (mpm_ctx->max_pat_id + 1) * sizeof(SCACPatternList));
 
     for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
         if (!(ctx->parray[i]->flags & MPM_PATTERN_FLAG_NOCASE)) {
@@ -826,21 +822,19 @@ void SCACInitCtx(MpmCtx *mpm_ctx)
     if (mpm_ctx->ctx != NULL)
         return;
 
-    mpm_ctx->ctx = SCMalloc(sizeof(SCACCtx));
+    mpm_ctx->ctx = SCCalloc(1, sizeof(SCACCtx));
     if (mpm_ctx->ctx == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->ctx, 0, sizeof(SCACCtx));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCACCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCMalloc(sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->init_hash, 0, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
 
     /* get conf values for AC from our yaml file.  We have no conf values for
      * now.  We will certainly need this, as we develop the algo */
diff --git a/src/util-mpm-hs.c b/src/util-mpm-hs.c
index a6fb82372cf7..7f26570981c1 100644
--- a/src/util-mpm-hs.c
+++ b/src/util-mpm-hs.c
@@ -178,11 +178,10 @@ static inline SCHSPattern *SCHSInitHashLookup(SCHSCtx *ctx, uint8_t *pat,
  */
 static inline SCHSPattern *SCHSAllocPattern(MpmCtx *mpm_ctx)
 {
-    SCHSPattern *p = SCMalloc(sizeof(SCHSPattern));
+    SCHSPattern *p = SCCalloc(1, sizeof(SCHSPattern));
     if (unlikely(p == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(p, 0, sizeof(SCHSPattern));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCHSPattern);
@@ -380,37 +379,32 @@ typedef struct SCHSCompileData_ {
 
 static SCHSCompileData *SCHSAllocCompileData(unsigned int pattern_cnt)
 {
-    SCHSCompileData *cd = SCMalloc(pattern_cnt * sizeof(SCHSCompileData));
+    SCHSCompileData *cd = SCCalloc(1, pattern_cnt * sizeof(SCHSCompileData));
     if (cd == NULL) {
         goto error;
     }
-    memset(cd, 0, pattern_cnt * sizeof(SCHSCompileData));
 
     cd->pattern_cnt = pattern_cnt;
 
-    cd->ids = SCMalloc(pattern_cnt * sizeof(unsigned int));
+    cd->ids = SCCalloc(1, pattern_cnt * sizeof(unsigned int));
     if (cd->ids == NULL) {
         goto error;
     }
-    memset(cd->ids, 0, pattern_cnt * sizeof(unsigned int));
 
-    cd->flags = SCMalloc(pattern_cnt * sizeof(unsigned int));
+    cd->flags = SCCalloc(1, pattern_cnt * sizeof(unsigned int));
     if (cd->flags == NULL) {
         goto error;
     }
-    memset(cd->flags, 0, pattern_cnt * sizeof(unsigned int));
 
-    cd->expressions = SCMalloc(pattern_cnt * sizeof(char *));
+    cd->expressions = SCCalloc(1, pattern_cnt * sizeof(char *));
     if (cd->expressions == NULL) {
         goto error;
     }
-    memset(cd->expressions, 0, pattern_cnt * sizeof(char *));
 
-    cd->ext = SCMalloc(pattern_cnt * sizeof(hs_expr_ext_t *));
+    cd->ext = SCCalloc(1, pattern_cnt * sizeof(hs_expr_ext_t *));
     if (cd->ext == NULL) {
         goto error;
     }
-    memset(cd->ext, 0, pattern_cnt * sizeof(hs_expr_ext_t *));
 
     return cd;
 
@@ -556,23 +550,20 @@ static void PatternDatabaseTableFree(void *data)
 
 static PatternDatabase *PatternDatabaseAlloc(uint32_t pattern_cnt)
 {
-    PatternDatabase *pd = SCMalloc(sizeof(PatternDatabase));
+    PatternDatabase *pd = SCCalloc(1, sizeof(PatternDatabase));
     if (pd == NULL) {
         return NULL;
     }
-    memset(pd, 0, sizeof(PatternDatabase));
     pd->pattern_cnt = pattern_cnt;
     pd->ref_cnt = 0;
     pd->hs_db = NULL;
 
     /* alloc the pattern array */
-    pd->parray =
-        (SCHSPattern **)SCMalloc(pd->pattern_cnt * sizeof(SCHSPattern *));
+    pd->parray = (SCHSPattern **)SCCalloc(1, pd->pattern_cnt * sizeof(SCHSPattern *));
     if (pd->parray == NULL) {
         SCFree(pd);
         return NULL;
     }
-    memset(pd->parray, 0, pd->pattern_cnt * sizeof(SCHSPattern *));
 
     return pd;
 }
@@ -667,12 +658,11 @@ int SCHSPreparePatterns(MpmCtx *mpm_ctx)
         cd->expressions[i] = HSRenderPattern(p->original_pat, p->len);
 
         if (p->flags & (MPM_PATTERN_FLAG_OFFSET | MPM_PATTERN_FLAG_DEPTH)) {
-            cd->ext[i] = SCMalloc(sizeof(hs_expr_ext_t));
+            cd->ext[i] = SCCalloc(1, sizeof(hs_expr_ext_t));
             if (cd->ext[i] == NULL) {
                 SCMutexUnlock(&g_db_table_mutex);
                 goto error;
             }
-            memset(cd->ext[i], 0, sizeof(hs_expr_ext_t));
 
             if (p->flags & MPM_PATTERN_FLAG_OFFSET) {
                 cd->ext[i]->flags |= HS_EXT_FLAG_MIN_OFFSET;
@@ -756,13 +746,12 @@ void SCHSInitThreadCtx(MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx)
 {
     memset(mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
 
-    SCHSThreadCtx *ctx = SCMalloc(sizeof(SCHSThreadCtx));
+    SCHSThreadCtx *ctx = SCCalloc(1, sizeof(SCHSThreadCtx));
     if (ctx == NULL) {
         exit(EXIT_FAILURE);
     }
     mpm_thread_ctx->ctx = ctx;
 
-    memset(ctx, 0, sizeof(SCHSThreadCtx));
     mpm_thread_ctx->memory_cnt++;
     mpm_thread_ctx->memory_size += sizeof(SCHSThreadCtx);
 
@@ -807,22 +796,20 @@ void SCHSInitCtx(MpmCtx *mpm_ctx)
     if (mpm_ctx->ctx != NULL)
         return;
 
-    mpm_ctx->ctx = SCMalloc(sizeof(SCHSCtx));
+    mpm_ctx->ctx = SCCalloc(1, sizeof(SCHSCtx));
     if (mpm_ctx->ctx == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(mpm_ctx->ctx, 0, sizeof(SCHSCtx));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(SCHSCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
     SCHSCtx *ctx = (SCHSCtx *)mpm_ctx->ctx;
-    ctx->init_hash = SCMalloc(sizeof(SCHSPattern *) * INIT_HASH_SIZE);
+    ctx->init_hash = SCCalloc(1, sizeof(SCHSPattern *) * INIT_HASH_SIZE);
     if (ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
-    memset(ctx->init_hash, 0, sizeof(SCHSPattern *) * INIT_HASH_SIZE);
 }
 
 /**
diff --git a/src/util-mpm.c b/src/util-mpm.c
index 0bacc9330b1c..0638a8876c53 100644
--- a/src/util-mpm.c
+++ b/src/util-mpm.c
@@ -133,11 +133,10 @@ int32_t MpmFactoryIsMpmCtxAvailable(const DetectEngineCtx *de_ctx, const MpmCtx
 MpmCtx *MpmFactoryGetMpmCtxForProfile(const DetectEngineCtx *de_ctx, int32_t id, int direction)
 {
     if (id == MPM_CTX_FACTORY_UNIQUE_CONTEXT) {
-        MpmCtx *mpm_ctx = SCMalloc(sizeof(MpmCtx));
+        MpmCtx *mpm_ctx = SCCalloc(1, sizeof(MpmCtx));
         if (unlikely(mpm_ctx == NULL)) {
             FatalError("Error allocating memory");
         }
-        memset(mpm_ctx, 0, sizeof(MpmCtx));
         return mpm_ctx;
     } else if (id < -1) {
         SCLogError("Invalid argument - %d\n", id);
@@ -339,11 +338,10 @@ static inline MpmPattern *MpmInitHashLookup(MpmCtx *ctx,
  */
 static inline MpmPattern *MpmAllocPattern(MpmCtx *mpm_ctx)
 {
-    MpmPattern *p = SCMalloc(sizeof(MpmPattern));
+    MpmPattern *p = SCCalloc(1, sizeof(MpmPattern));
     if (unlikely(p == NULL)) {
         exit(EXIT_FAILURE);
     }
-    memset(p, 0, sizeof(MpmPattern));
 
     mpm_ctx->memory_cnt++;
     mpm_ctx->memory_size += sizeof(MpmPattern);
diff --git a/src/util-pool.c b/src/util-pool.c
index 348c679741ce..bb9ff520c965 100644
--- a/src/util-pool.c
+++ b/src/util-pool.c
@@ -107,14 +107,12 @@ Pool *PoolInit(uint32_t size, uint32_t prealloc_size, uint32_t elt_size,
     }
 
     /* setup the filter */
-    p = SCMalloc(sizeof(Pool));
+    p = SCCalloc(1, sizeof(Pool));
     if (unlikely(p == NULL)) {
         sc_errno = SC_ENOMEM;
         goto error;
     }
 
-    memset(p,0,sizeof(Pool));
-
     p->max_buckets = size;
     p->preallocated = prealloc_size;
     p->elt_size = elt_size;
@@ -158,12 +156,11 @@ Pool *PoolInit(uint32_t size, uint32_t prealloc_size, uint32_t elt_size,
     /* prealloc the buckets and requeue them to the alloc list */
     for (u32 = 0; u32 < prealloc_size; u32++) {
         if (size == 0) { /* unlimited */
-            PoolBucket *pb = SCMalloc(sizeof(PoolBucket));
+            PoolBucket *pb = SCCalloc(1, sizeof(PoolBucket));
             if (unlikely(pb == NULL)) {
                 sc_errno = SC_ENOMEM;
                 goto error;
             }
-            memset(pb, 0, sizeof(PoolBucket));
 
             if (p->Alloc) {
                 pb->data = p->Alloc();
diff --git a/src/util-profiling-keywords.c b/src/util-profiling-keywords.c
index b21eb433e798..bd7cda526b39 100644
--- a/src/util-profiling-keywords.c
+++ b/src/util-profiling-keywords.c
@@ -241,9 +241,8 @@ SCProfilingKeywordUpdateCounter(DetectEngineThreadCtx *det_ctx, int id, uint64_t
 
 static SCProfileKeywordDetectCtx *SCProfilingKeywordInitCtx(void)
 {
-    SCProfileKeywordDetectCtx *ctx = SCMalloc(sizeof(SCProfileKeywordDetectCtx));
+    SCProfileKeywordDetectCtx *ctx = SCCalloc(1, sizeof(SCProfileKeywordDetectCtx));
     if (ctx != NULL) {
-        memset(ctx, 0x00, sizeof(SCProfileKeywordDetectCtx));
 
         if (pthread_mutex_init(&ctx->data_m, NULL) != 0) {
             FatalError("Failed to initialize hash table mutex.");
@@ -284,9 +283,8 @@ void SCProfilingKeywordThreadSetup(SCProfileKeywordDetectCtx *ctx, DetectEngineT
     if (ctx == NULL)
         return;
 
-    SCProfileKeywordData *a = SCMalloc(sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+    SCProfileKeywordData *a = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
     if (a != NULL) {
-        memset(a, 0x00, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
         det_ctx->keyword_perf_data = a;
     }
 
@@ -296,12 +294,10 @@ void SCProfilingKeywordThreadSetup(SCProfileKeywordDetectCtx *ctx, DetectEngineT
 
     int i;
     for (i = 0; i < nlists; i++) {
-        SCProfileKeywordData *b = SCMalloc(sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+        SCProfileKeywordData *b = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
         if (b != NULL) {
-            memset(b, 0x00, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
             det_ctx->keyword_perf_data_per_list[i] = b;
         }
-
     }
 }
 
@@ -373,9 +369,8 @@ SCProfilingKeywordInitCounters(DetectEngineCtx *de_ctx)
     de_ctx->profile_keyword_ctx = SCProfilingKeywordInitCtx();
     BUG_ON(de_ctx->profile_keyword_ctx == NULL);
 
-    de_ctx->profile_keyword_ctx->data = SCMalloc(sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+    de_ctx->profile_keyword_ctx->data = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
     BUG_ON(de_ctx->profile_keyword_ctx->data == NULL);
-    memset(de_ctx->profile_keyword_ctx->data, 0x00, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
 
     de_ctx->profile_keyword_ctx_per_list = SCCalloc(nlists, sizeof(SCProfileKeywordDetectCtx *));
     BUG_ON(de_ctx->profile_keyword_ctx_per_list == NULL);
@@ -384,9 +379,9 @@ SCProfilingKeywordInitCounters(DetectEngineCtx *de_ctx)
     for (i = 0; i < nlists; i++) {
         de_ctx->profile_keyword_ctx_per_list[i] = SCProfilingKeywordInitCtx();
         BUG_ON(de_ctx->profile_keyword_ctx_per_list[i] == NULL);
-        de_ctx->profile_keyword_ctx_per_list[i]->data = SCMalloc(sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+        de_ctx->profile_keyword_ctx_per_list[i]->data =
+                SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
         BUG_ON(de_ctx->profile_keyword_ctx_per_list[i]->data == NULL);
-        memset(de_ctx->profile_keyword_ctx_per_list[i]->data, 0x00, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
     }
 
     SCLogPerf("Registered %"PRIu32" keyword profiling counters.", DETECT_TBLSIZE);
diff --git a/src/util-profiling-prefilter.c b/src/util-profiling-prefilter.c
index 280d5144e220..958846ae68c6 100644
--- a/src/util-profiling-prefilter.c
+++ b/src/util-profiling-prefilter.c
@@ -210,10 +210,8 @@ void SCProfilingPrefilterUpdateCounter(DetectEngineThreadCtx *det_ctx, int id, u
 
 static SCProfilePrefilterDetectCtx *SCProfilingPrefilterInitCtx(void)
 {
-    SCProfilePrefilterDetectCtx *ctx = SCMalloc(sizeof(SCProfilePrefilterDetectCtx));
+    SCProfilePrefilterDetectCtx *ctx = SCCalloc(1, sizeof(SCProfilePrefilterDetectCtx));
     if (ctx != NULL) {
-        memset(ctx, 0x00, sizeof(SCProfilePrefilterDetectCtx));
-
         if (pthread_mutex_init(&ctx->data_m, NULL) != 0) {
             FatalError("Failed to initialize hash table mutex.");
         }
@@ -248,9 +246,8 @@ void SCProfilingPrefilterThreadSetup(SCProfilePrefilterDetectCtx *ctx, DetectEng
 
     const uint32_t size = det_ctx->de_ctx->prefilter_id;
 
-    SCProfilePrefilterData *a = SCMalloc(sizeof(SCProfilePrefilterData) * size);
+    SCProfilePrefilterData *a = SCCalloc(1, sizeof(SCProfilePrefilterData) * size);
     if (a != NULL) {
-        memset(a, 0x00, sizeof(SCProfilePrefilterData) * size);
         det_ctx->prefilter_perf_data = a;
     }
 }
@@ -310,9 +307,8 @@ SCProfilingPrefilterInitCounters(DetectEngineCtx *de_ctx)
     BUG_ON(de_ctx->profile_prefilter_ctx == NULL);
     de_ctx->profile_prefilter_ctx->size = size;
 
-    de_ctx->profile_prefilter_ctx->data = SCMalloc(sizeof(SCProfilePrefilterData) * size);
+    de_ctx->profile_prefilter_ctx->data = SCCalloc(1, sizeof(SCProfilePrefilterData) * size);
     BUG_ON(de_ctx->profile_prefilter_ctx->data == NULL);
-    memset(de_ctx->profile_prefilter_ctx->data, 0x00, sizeof(SCProfilePrefilterData) * size);
 
     HashListTableBucket *hb = HashListTableGetListHead(de_ctx->prefilter_hash_table);
     for ( ; hb != NULL; hb = HashListTableGetListNext(hb)) {
diff --git a/src/util-profiling-rules.c b/src/util-profiling-rules.c
index 7a14c93b7731..0397b8a0c0c6 100644
--- a/src/util-profiling-rules.c
+++ b/src/util-profiling-rules.c
@@ -562,10 +562,8 @@ SCProfilingRuleUpdateCounter(DetectEngineThreadCtx *det_ctx, uint16_t id, uint64
 
 static SCProfileDetectCtx *SCProfilingRuleInitCtx(void)
 {
-    SCProfileDetectCtx *ctx = SCMalloc(sizeof(SCProfileDetectCtx));
+    SCProfileDetectCtx *ctx = SCCalloc(1, sizeof(SCProfileDetectCtx));
     if (ctx != NULL) {
-        memset(ctx, 0x00, sizeof(SCProfileDetectCtx));
-
         if (pthread_mutex_init(&ctx->data_m, NULL) != 0) {
             FatalError("Failed to initialize hash table mutex.");
         }
@@ -590,10 +588,8 @@ void SCProfilingRuleThreadSetup(SCProfileDetectCtx *ctx, DetectEngineThreadCtx *
     if (ctx == NULL|| ctx->size == 0)
         return;
 
-    SCProfileData *a = SCMalloc(sizeof(SCProfileData) * ctx->size);
+    SCProfileData *a = SCCalloc(1, sizeof(SCProfileData) * ctx->size);
     if (a != NULL) {
-        memset(a, 0x00, sizeof(SCProfileData) * ctx->size);
-
         det_ctx->rule_perf_data = a;
         det_ctx->rule_perf_data_size = ctx->size;
     }
@@ -669,9 +665,8 @@ SCProfilingRuleInitCounters(DetectEngineCtx *de_ctx)
     }
 
     if (count > 0) {
-        de_ctx->profile_ctx->data = SCMalloc(sizeof(SCProfileData) * de_ctx->profile_ctx->size);
+        de_ctx->profile_ctx->data = SCCalloc(1, sizeof(SCProfileData) * de_ctx->profile_ctx->size);
         BUG_ON(de_ctx->profile_ctx->data == NULL);
-        memset(de_ctx->profile_ctx->data, 0x00, sizeof(SCProfileData) * de_ctx->profile_ctx->size);
 
         sig = de_ctx->sig_list;
         while (sig != NULL) {
diff --git a/src/util-radix-tree.c b/src/util-radix-tree.c
index 97c85602d8a0..861d1256f409 100644
--- a/src/util-radix-tree.c
+++ b/src/util-radix-tree.c
@@ -47,14 +47,12 @@
  */
 static SCRadixUserData *SCRadixAllocSCRadixUserData(uint8_t netmask, void *user)
 {
-    SCRadixUserData *user_data = SCMalloc(sizeof(SCRadixUserData));
+    SCRadixUserData *user_data = SCCalloc(1, sizeof(SCRadixUserData));
     if (unlikely(user_data == NULL)) {
         SCLogError("Error allocating memory");
         return NULL;
     }
 
-    memset(user_data, 0, sizeof(SCRadixUserData));
-
     user_data->netmask = netmask;
     user_data->user = user;
 
@@ -143,16 +141,12 @@ static SCRadixPrefix *SCRadixCreatePrefix(uint8_t *key_stream,
         return NULL;
     }
 
-    if ( (prefix = SCMalloc(sizeof(SCRadixPrefix))) == NULL)
+    if ((prefix = SCCalloc(1, sizeof(SCRadixPrefix))) == NULL)
         goto error;
 
-    memset(prefix, 0, sizeof(SCRadixPrefix));
-
-    if ( (prefix->stream = SCMalloc(key_bitlen / 8)) == NULL)
+    if ((prefix->stream = SCCalloc(1, key_bitlen / 8)) == NULL)
         goto error;
 
-    memset(prefix->stream, 0, key_bitlen / 8);
-
     memcpy(prefix->stream, key_stream, key_bitlen / 8);
     prefix->bitlen = key_bitlen;
 
@@ -382,11 +376,10 @@ static inline SCRadixNode *SCRadixCreateNode(void)
 {
     SCRadixNode *node = NULL;
 
-    if ( (node = SCMalloc(sizeof(SCRadixNode))) == NULL) {
+    if ((node = SCCalloc(1, sizeof(SCRadixNode))) == NULL) {
         SCLogError("Fatal error encountered in SCRadixCreateNode. Mem not allocated...");
         return NULL;
     }
-    memset(node, 0, sizeof(SCRadixNode));
 
     return node;
 }
@@ -425,10 +418,9 @@ SCRadixTree *SCRadixCreateRadixTree(void (*Free)(void*), void (*PrintData)(void*
 {
     SCRadixTree *tree = NULL;
 
-    if ( (tree = SCMalloc(sizeof(SCRadixTree))) == NULL) {
+    if ((tree = SCCalloc(1, sizeof(SCRadixTree))) == NULL) {
         FatalError("Fatal error encountered in SCRadixCreateRadixTree. Exiting...");
     }
-    memset(tree, 0, sizeof(SCRadixTree));
 
     tree->Free = Free;
     tree->PrintData = PrintData;
diff --git a/src/util-reference-config.c b/src/util-reference-config.c
index 0e5c51ea141e..89cc1d23881c 100644
--- a/src/util-reference-config.c
+++ b/src/util-reference-config.c
@@ -364,10 +364,9 @@ SCRConfReference *SCRConfAllocSCRConfReference(const char *system,
         return NULL;
     }
 
-    if ((ref = SCMalloc(sizeof(SCRConfReference))) == NULL) {
+    if ((ref = SCCalloc(1, sizeof(SCRConfReference))) == NULL) {
         return NULL;
     }
-    memset(ref, 0, sizeof(SCRConfReference));
 
     if ((ref->system = SCRConfStringToLowercase(system)) == NULL) {
         SCFree(ref);
diff --git a/src/util-rohash.c b/src/util-rohash.c
index e57a74dd244b..53437430291b 100644
--- a/src/util-rohash.c
+++ b/src/util-rohash.c
@@ -74,12 +74,11 @@ ROHashTable *ROHashInit(uint8_t hash_bits, uint16_t item_size)
 
     uint32_t size = hashsize(hash_bits) * sizeof(ROHashTableOffsets);
 
-    ROHashTable *table = SCMalloc(sizeof(ROHashTable) + size);
+    ROHashTable *table = SCCalloc(1, sizeof(ROHashTable) + size);
     if (unlikely(table == NULL)) {
         SCLogError("failed to alloc memory");
         return NULL;
     }
-    memset(table, 0, sizeof(ROHashTable) + size);
 
     table->items = 0;
     table->item_size = item_size;
@@ -161,9 +160,8 @@ int ROHashInitQueueValue(ROHashTable *table, void *value, uint16_t size)
         return 0;
     }
 
-    ROHashTableItem *item = SCMalloc(sizeof(ROHashTableItem) + table->item_size);
+    ROHashTableItem *item = SCCalloc(1, sizeof(ROHashTableItem) + table->item_size);
     if (item != NULL) {
-        memset(item, 0x00, sizeof(ROHashTableItem));
         memcpy((void *)item + sizeof(ROHashTableItem), value, table->item_size);
         TAILQ_INSERT_TAIL(&table->head, item, next);
         return 1;
@@ -208,12 +206,11 @@ int ROHashInitFinalize(ROHashTable *table)
 
     /* get the data block */
     uint32_t newsize = table->items * table->item_size;
-    table->data = SCMalloc(newsize);
+    table->data = SCCalloc(1, newsize);
     if (table->data == NULL) {
         SCLogError("failed to alloc memory");
         return 0;
     }
-    memset(table->data, 0x00, newsize);
 
     /* calc offsets into the block per hash value */
     uint32_t total = 0;
diff --git a/src/util-runmodes.c b/src/util-runmodes.c
index ccd1ce3aa96c..f78e857abfc6 100644
--- a/src/util-runmodes.c
+++ b/src/util-runmodes.c
@@ -62,12 +62,11 @@ char *RunmodeAutoFpCreatePickupQueuesString(int n)
     size_t queues_size = n * 13;
     char qname[TM_QUEUE_NAME_MAX];
 
-    char *queues = SCMalloc(queues_size);
+    char *queues = SCCalloc(1, queues_size);
     if (unlikely(queues == NULL)) {
         SCLogError("failed to alloc queues buffer: %s", strerror(errno));
         return NULL;
     }
-    memset(queues, 0x00, queues_size);
 
     for (int thread = 0; thread < n; thread++) {
         if (strlen(queues) > 0)
diff --git a/src/util-spm-bm.c b/src/util-spm-bm.c
index 29dbf4a3ecd6..449c6b62962e 100644
--- a/src/util-spm-bm.c
+++ b/src/util-spm-bm.c
@@ -391,21 +391,19 @@ typedef struct SpmBmCtx_ {
 static SpmCtx *BMInitCtx(const uint8_t *needle, uint16_t needle_len, int nocase,
                          SpmGlobalThreadCtx *global_thread_ctx)
 {
-    SpmCtx *ctx = SCMalloc(sizeof(SpmCtx));
+    SpmCtx *ctx = SCCalloc(1, sizeof(SpmCtx));
     if (ctx == NULL) {
         SCLogDebug("Unable to alloc SpmCtx.");
         return NULL;
     }
-    memset(ctx, 0, sizeof(*ctx));
     ctx->matcher = SPM_BM;
 
-    SpmBmCtx *sctx = SCMalloc(sizeof(SpmBmCtx));
+    SpmBmCtx *sctx = SCCalloc(1, sizeof(SpmBmCtx));
     if (sctx == NULL) {
         SCLogDebug("Unable to alloc SpmBmCtx.");
         SCFree(ctx);
         return NULL;
     }
-    memset(sctx, 0, sizeof(*sctx));
 
     sctx->needle = SCMalloc(needle_len);
     if (sctx->needle == NULL) {
@@ -463,12 +461,11 @@ static uint8_t *BMScan(const SpmCtx *ctx, SpmThreadCtx *thread_ctx,
 
 static SpmGlobalThreadCtx *BMInitGlobalThreadCtx(void)
 {
-    SpmGlobalThreadCtx *global_thread_ctx = SCMalloc(sizeof(SpmGlobalThreadCtx));
+    SpmGlobalThreadCtx *global_thread_ctx = SCCalloc(1, sizeof(SpmGlobalThreadCtx));
     if (global_thread_ctx == NULL) {
         SCLogDebug("Unable to alloc SpmThreadCtx.");
         return NULL;
     }
-    memset(global_thread_ctx, 0, sizeof(*global_thread_ctx));
     global_thread_ctx->matcher = SPM_BM;
     return global_thread_ctx;
 }
@@ -490,12 +487,11 @@ static void BMDestroyThreadCtx(SpmThreadCtx *thread_ctx)
 }
 
 static SpmThreadCtx *BMMakeThreadCtx(const SpmGlobalThreadCtx *global_thread_ctx) {
-    SpmThreadCtx *thread_ctx = SCMalloc(sizeof(SpmThreadCtx));
+    SpmThreadCtx *thread_ctx = SCCalloc(1, sizeof(SpmThreadCtx));
     if (thread_ctx == NULL) {
         SCLogDebug("Unable to alloc SpmThreadCtx.");
         return NULL;
     }
-    memset(thread_ctx, 0, sizeof(*thread_ctx));
     thread_ctx->matcher = SPM_BM;
     return thread_ctx;
 }
diff --git a/src/util-spm-hs.c b/src/util-spm-hs.c
index cfcb8acd52a9..d58de651d943 100644
--- a/src/util-spm-hs.c
+++ b/src/util-spm-hs.c
@@ -108,15 +108,14 @@ static int HSBuildDatabase(const uint8_t *needle, uint16_t needle_len,
 static SpmCtx *HSInitCtx(const uint8_t *needle, uint16_t needle_len, int nocase,
                          SpmGlobalThreadCtx *global_thread_ctx)
 {
-    SpmCtx *ctx = SCMalloc(sizeof(SpmCtx));
+    SpmCtx *ctx = SCCalloc(1, sizeof(SpmCtx));
     if (ctx == NULL) {
         SCLogDebug("Unable to alloc SpmCtx.");
         return NULL;
     }
-    memset(ctx, 0, sizeof(SpmCtx));
     ctx->matcher = SPM_HS;
 
-    SpmHsCtx *sctx = SCMalloc(sizeof(SpmHsCtx));
+    SpmHsCtx *sctx = SCCalloc(1, sizeof(SpmHsCtx));
     if (sctx == NULL) {
         SCLogDebug("Unable to alloc SpmHsCtx.");
         SCFree(ctx);
@@ -124,7 +123,6 @@ static SpmCtx *HSInitCtx(const uint8_t *needle, uint16_t needle_len, int nocase,
     }
     ctx->ctx = sctx;
 
-    memset(sctx, 0, sizeof(SpmHsCtx));
     if (HSBuildDatabase(needle, needle_len, nocase, sctx,
                         global_thread_ctx) != 0) {
         SCLogDebug("HSBuildDatabase failed.");
@@ -168,12 +166,11 @@ static uint8_t *HSScan(const SpmCtx *ctx, SpmThreadCtx *thread_ctx,
 
 static SpmGlobalThreadCtx *HSInitGlobalThreadCtx(void)
 {
-    SpmGlobalThreadCtx *global_thread_ctx = SCMalloc(sizeof(SpmGlobalThreadCtx));
+    SpmGlobalThreadCtx *global_thread_ctx = SCCalloc(1, sizeof(SpmGlobalThreadCtx));
     if (global_thread_ctx == NULL) {
         SCLogDebug("Unable to alloc SpmGlobalThreadCtx.");
         return NULL;
     }
-    memset(global_thread_ctx, 0, sizeof(*global_thread_ctx));
     global_thread_ctx->matcher = SPM_HS;
 
     /* We store scratch in the HS-specific ctx. This will be initialized as
@@ -203,12 +200,11 @@ static void HSDestroyThreadCtx(SpmThreadCtx *thread_ctx)
 
 static SpmThreadCtx *HSMakeThreadCtx(const SpmGlobalThreadCtx *global_thread_ctx)
 {
-    SpmThreadCtx *thread_ctx = SCMalloc(sizeof(SpmThreadCtx));
+    SpmThreadCtx *thread_ctx = SCCalloc(1, sizeof(SpmThreadCtx));
     if (thread_ctx == NULL) {
         SCLogDebug("Unable to alloc SpmThreadCtx.");
         return NULL;
     }
-    memset(thread_ctx, 0, sizeof(*thread_ctx));
     thread_ctx->matcher = SPM_HS;
 
     if (global_thread_ctx->ctx != NULL) {
diff --git a/src/util-storage.c b/src/util-storage.c
index a0108d03a66a..1394f205231a 100644
--- a/src/util-storage.c
+++ b/src/util-storage.c
@@ -118,12 +118,10 @@ int StorageRegister(const StorageEnum type, const char *name, const unsigned int
         list = list->next;
     }
 
-    StorageList *entry = SCMalloc(sizeof(StorageList));
+    StorageList *entry = SCCalloc(1, sizeof(StorageList));
     if (unlikely(entry == NULL))
         return -1;
 
-    memset(entry, 0x00, sizeof(StorageList));
-
     entry->map.type = type;
     entry->map.name = name;
     entry->map.size = size;
@@ -151,18 +149,16 @@ int StorageFinalize(void)
     if (count == 0)
         return 0;
 
-    storage_map = SCMalloc(sizeof(StorageMapping *) * STORAGE_MAX);
+    storage_map = SCCalloc(1, sizeof(StorageMapping *) * STORAGE_MAX);
     if (unlikely(storage_map == NULL)) {
         return -1;
     }
-    memset(storage_map, 0x00, sizeof(StorageMapping *) * STORAGE_MAX);
 
     for (i = 0; i < STORAGE_MAX; i++) {
         if (storage_max_id[i] > 0) {
-            storage_map[i] = SCMalloc(sizeof(StorageMapping) * storage_max_id[i]);
+            storage_map[i] = SCCalloc(1, sizeof(StorageMapping) * storage_max_id[i]);
             if (storage_map[i] == NULL)
                 return -1;
-            memset(storage_map[i], 0x00, sizeof(StorageMapping) * storage_max_id[i]);
         }
     }
 
@@ -266,10 +262,9 @@ void *StorageAllocById(Storage **storage, StorageEnum type, int id)
     Storage *store = *storage;
     if (store == NULL) {
         // coverity[suspicious_sizeof : FALSE]
-        store = SCMalloc(sizeof(void *) * storage_max_id[type]);
+        store = SCCalloc(1, sizeof(void *) * storage_max_id[type]);
         if (unlikely(store == NULL))
-        return NULL;
-        memset(store, 0x00, sizeof(void *) * storage_max_id[type]);
+            return NULL;
     }
     SCLogDebug("store %p", store);
 
diff --git a/src/util-threshold-config.c b/src/util-threshold-config.c
index b093467b398a..70cc41a73e91 100644
--- a/src/util-threshold-config.c
+++ b/src/util-threshold-config.c
@@ -380,10 +380,9 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
                 continue;
             }
 
-            de = SCMalloc(sizeof(DetectThresholdData));
+            de = SCCalloc(1, sizeof(DetectThresholdData));
             if (unlikely(de == NULL))
                 goto error;
-            memset(de,0,sizeof(DetectThresholdData));
 
             de->type = parsed_type;
             de->track = parsed_track;
@@ -416,10 +415,9 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
                     continue;
                 }
 
-                de = SCMalloc(sizeof(DetectThresholdData));
+                de = SCCalloc(1, sizeof(DetectThresholdData));
                 if (unlikely(de == NULL))
                     goto error;
-                memset(de,0,sizeof(DetectThresholdData));
 
                 de->type = parsed_type;
                 de->track = parsed_track;
@@ -484,10 +482,9 @@ static int SetupThresholdRule(DetectEngineCtx *de_ctx, uint32_t id, uint32_t gid
                 }
             }
 
-            de = SCMalloc(sizeof(DetectThresholdData));
+            de = SCCalloc(1, sizeof(DetectThresholdData));
             if (unlikely(de == NULL))
                 goto error;
-            memset(de,0,sizeof(DetectThresholdData));
 
             de->type = parsed_type;
             de->track = parsed_track;

From ec1482cf4811b74715c3ce4bd35dc1c70680f01d Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 21 Nov 2023 08:55:28 -0500
Subject: [PATCH 159/462] calloc: Use nmemb with SCCalloc

This commit modifies calls to SCCalloc that had a member count of 1 and
a size count calculated as: element_count * sizeof(element).
---
 src/app-layer-detect-proto.c    |  2 +-
 src/detect-engine-build.c       |  2 +-
 src/detect-engine-siggroup.c    |  6 +++---
 src/detect-engine.c             |  4 ++--
 src/util-bloomfilter-counting.c |  2 +-
 src/util-hash.c                 |  2 +-
 src/util-hashlist.c             |  2 +-
 src/util-mpm-ac-bs.c            | 12 ++++++------
 src/util-mpm-ac-ks.c            |  6 +++---
 src/util-mpm-ac.c               |  8 ++++----
 src/util-mpm-hs.c               | 14 +++++++-------
 src/util-profiling-keywords.c   |  8 ++++----
 src/util-profiling-rules.c      |  4 ++--
 src/util-storage.c              |  6 +++---
 14 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index cb31b4d6b969..690950d34e72 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -1268,7 +1268,7 @@ static int AppLayerProtoDetectPMMapSignatures(AppLayerProtoDetectPMCtx *ctx)
     int mpm_ret;
     SigIntId id = 0;
 
-    ctx->map = SCCalloc(1, ctx->max_sig_id * sizeof(AppLayerProtoDetectPMSignature *));
+    ctx->map = SCCalloc(ctx->max_sig_id, sizeof(AppLayerProtoDetectPMSignature *));
     if (ctx->map == NULL)
         goto error;
 
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index f632bd8e5bbf..8c01104c48de 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1383,7 +1383,7 @@ int SigAddressPrepareStage1(DetectEngineCtx *de_ctx)
 
     de_ctx->sig_array_len = DetectEngineGetMaxSigId(de_ctx);
     de_ctx->sig_array_size = (de_ctx->sig_array_len * sizeof(Signature *));
-    de_ctx->sig_array = (Signature **)SCCalloc(1, de_ctx->sig_array_size);
+    de_ctx->sig_array = (Signature **)SCCalloc(de_ctx->sig_array_len, sizeof(Signature *));
     if (de_ctx->sig_array == NULL)
         goto error;
 
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index 36e3872c0452..36df347a503c 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -493,7 +493,7 @@ int SigGroupHeadBuildMatchArray(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
 
     BUG_ON(sgh->init->match_array != NULL);
 
-    sgh->init->match_array = SCCalloc(1, sgh->init->sig_cnt * sizeof(Signature *));
+    sgh->init->match_array = SCCalloc(sgh->init->sig_cnt, sizeof(Signature *));
     if (sgh->init->match_array == NULL)
         return -1;
 
@@ -672,12 +672,12 @@ int SigGroupHeadBuildNonPrefilterArray(DetectEngineCtx *de_ctx, SigGroupHead *sg
     }
 
     if (non_pf > 0) {
-        sgh->non_pf_other_store_array = SCCalloc(1, non_pf * sizeof(SignatureNonPrefilterStore));
+        sgh->non_pf_other_store_array = SCCalloc(non_pf, sizeof(SignatureNonPrefilterStore));
         BUG_ON(sgh->non_pf_other_store_array == NULL);
     }
 
     if (non_pf_syn > 0) {
-        sgh->non_pf_syn_store_array = SCCalloc(1, non_pf_syn * sizeof(SignatureNonPrefilterStore));
+        sgh->non_pf_syn_store_array = SCCalloc(non_pf_syn, sizeof(SignatureNonPrefilterStore));
         BUG_ON(sgh->non_pf_syn_store_array == NULL);
     }
 
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 678031fa44df..4cf145df6e2b 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -3046,7 +3046,7 @@ static int DetectEngineThreadCtxInitKeywords(DetectEngineCtx *de_ctx, DetectEngi
 {
     if (de_ctx->keyword_id > 0) {
         // coverity[suspicious_sizeof : FALSE]
-        det_ctx->keyword_ctxs_array = SCCalloc(1, de_ctx->keyword_id * sizeof(void *));
+        det_ctx->keyword_ctxs_array = SCCalloc(de_ctx->keyword_id, sizeof(void *));
         if (det_ctx->keyword_ctxs_array == NULL) {
             SCLogError("setting up thread local detect ctx");
             return TM_ECODE_FAILED;
@@ -3226,7 +3226,7 @@ static TmEcode ThreadCtxDoInit (DetectEngineCtx *de_ctx, DetectEngineThreadCtx *
     /* DeState */
     if (de_ctx->sig_array_len > 0) {
         det_ctx->match_array_len = de_ctx->sig_array_len;
-        det_ctx->match_array = SCCalloc(1, det_ctx->match_array_len * sizeof(Signature *));
+        det_ctx->match_array = SCCalloc(det_ctx->match_array_len, sizeof(Signature *));
         if (det_ctx->match_array == NULL) {
             return TM_ECODE_FAILED;
         }
diff --git a/src/util-bloomfilter-counting.c b/src/util-bloomfilter-counting.c
index 6fc9cadc7deb..620b507dfa99 100644
--- a/src/util-bloomfilter-counting.c
+++ b/src/util-bloomfilter-counting.c
@@ -57,7 +57,7 @@ BloomFilterCounting *BloomFilterCountingInit(uint32_t size, uint8_t type, uint8_
     bf->Hash = Hash;
 
     /* setup the bitarray */
-    bf->array = SCCalloc(1, bf->array_size * bf->type);
+    bf->array = SCCalloc(bf->array_size, bf->type);
     if (bf->array == NULL)
         goto error;
 
diff --git a/src/util-hash.c b/src/util-hash.c
index a81882d52ac1..412a46fa7eb9 100644
--- a/src/util-hash.c
+++ b/src/util-hash.c
@@ -59,7 +59,7 @@ HashTable* HashTableInit(uint32_t size, uint32_t (*Hash)(struct HashTable_ *, vo
         ht->Compare = HashTableDefaultCompare;
 
     /* setup the bitarray */
-    ht->array = SCCalloc(1, ht->array_size * sizeof(HashTableBucket *));
+    ht->array = SCCalloc(ht->array_size, sizeof(HashTableBucket *));
     if (ht->array == NULL)
         goto error;
 
diff --git a/src/util-hashlist.c b/src/util-hashlist.c
index 88f8144b3c37..e4b62a613d6e 100644
--- a/src/util-hashlist.c
+++ b/src/util-hashlist.c
@@ -65,7 +65,7 @@ HashListTable *HashListTableInit(uint32_t size,
         ht->Compare = HashListTableDefaultCompare;
 
     /* setup the bitarray */
-    ht->array = SCCalloc(1, ht->array_size * sizeof(HashListTableBucket *));
+    ht->array = SCCalloc(ht->array_size, sizeof(HashListTableBucket *));
     if (ht->array == NULL) {
         sc_errno = SC_ENOMEM;
         goto error;
diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c
index aebcd778a6b7..72d2065ca7c1 100644
--- a/src/util-mpm-ac-bs.c
+++ b/src/util-mpm-ac-bs.c
@@ -420,7 +420,7 @@ static inline void SCACBSCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* allot space for the failure table.  A failure entry in the table for
      * every state(SCACBSCtx->state_count) */
-    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(ctx->state_count, sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
@@ -681,7 +681,7 @@ static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx)
 
         /* buffer to hold pointers in the buffer, so that a state can use it
          * directly to access its state data */
-        ctx->state_table_mod_pointers = SCCalloc(1, ctx->state_count * sizeof(uint8_t *));
+        ctx->state_table_mod_pointers = SCCalloc(ctx->state_count, sizeof(uint8_t *));
         if (ctx->state_table_mod_pointers == NULL) {
             FatalError("Error allocating memory");
         }
@@ -750,7 +750,7 @@ static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx)
 
         /* buffer to hold pointers in the buffer, so that a state can use it
          * directly to access its state data */
-        ctx->state_table_mod_pointers = SCCalloc(1, ctx->state_count * sizeof(uint8_t *));
+        ctx->state_table_mod_pointers = SCCalloc(ctx->state_count, sizeof(uint8_t *));
         if (ctx->state_table_mod_pointers == NULL) {
             FatalError("Error allocating memory");
         }
@@ -861,7 +861,7 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(mpm_ctx->pattern_cnt, sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
     mpm_ctx->memory_cnt++;
@@ -887,7 +887,7 @@ int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
     ctx->single_state_size = sizeof(int32_t) * 256;
 
     /* handle no case patterns */
-    ctx->pid_pat_list = SCCalloc(1, (mpm_ctx->max_pat_id + 1) * sizeof(SCACBSPatternList));
+    ctx->pid_pat_list = SCCalloc((mpm_ctx->max_pat_id + 1), sizeof(SCACBSPatternList));
     if (ctx->pid_pat_list == NULL) {
         FatalError("Error allocating memory");
     }
@@ -949,7 +949,7 @@ void SCACBSInitCtx(MpmCtx *mpm_ctx)
     mpm_ctx->memory_size += sizeof(SCACBSCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(MPM_INIT_HASH_SIZE, sizeof(MpmPattern *));
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
diff --git a/src/util-mpm-ac-ks.c b/src/util-mpm-ac-ks.c
index 9b2c799eef11..b2f3ebc1afed 100644
--- a/src/util-mpm-ac-ks.c
+++ b/src/util-mpm-ac-ks.c
@@ -508,7 +508,7 @@ static void SCACTileCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* Allocate space for the failure table.  A failure entry in the table for
      * every state(SCACTileCtx->state_count) */
-    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(ctx->state_count, sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
@@ -874,7 +874,7 @@ int SCACTilePreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(mpm_ctx->pattern_cnt, sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
 
@@ -985,7 +985,7 @@ void SCACTileInitCtx(MpmCtx *mpm_ctx)
     mpm_ctx->memory_size += sizeof(SCACTileCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(MPM_INIT_HASH_SIZE, sizeof(MpmPattern *));
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index cb663b8aba9e..22347d6fef23 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -476,7 +476,7 @@ static inline void SCACCreateFailureTable(MpmCtx *mpm_ctx)
 
     /* allot space for the failure table.  A failure entry in the table for
      * every state(SCACCtx->state_count) */
-    ctx->failure_table = SCCalloc(1, ctx->state_count * sizeof(int32_t));
+    ctx->failure_table = SCCalloc(ctx->state_count, sizeof(int32_t));
     if (ctx->failure_table == NULL) {
         FatalError("Error allocating memory");
     }
@@ -736,7 +736,7 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx)
     }
 
     /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCCalloc(1, mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
+    ctx->parray = (MpmPattern **)SCCalloc(mpm_ctx->pattern_cnt, sizeof(MpmPattern *));
     if (ctx->parray == NULL)
         goto error;
     mpm_ctx->memory_cnt++;
@@ -762,7 +762,7 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx)
     ctx->single_state_size = sizeof(int32_t) * 256;
 
     /* handle no case patterns */
-    ctx->pid_pat_list = SCCalloc(1, (mpm_ctx->max_pat_id + 1) * sizeof(SCACPatternList));
+    ctx->pid_pat_list = SCCalloc((mpm_ctx->max_pat_id + 1), sizeof(SCACPatternList));
     if (ctx->pid_pat_list == NULL) {
         FatalError("Error allocating memory");
     }
@@ -831,7 +831,7 @@ void SCACInitCtx(MpmCtx *mpm_ctx)
     mpm_ctx->memory_size += sizeof(SCACCtx);
 
     /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCCalloc(1, sizeof(MpmPattern *) * MPM_INIT_HASH_SIZE);
+    mpm_ctx->init_hash = SCCalloc(MPM_INIT_HASH_SIZE, sizeof(MpmPattern *));
     if (mpm_ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
diff --git a/src/util-mpm-hs.c b/src/util-mpm-hs.c
index 7f26570981c1..a3b896abde93 100644
--- a/src/util-mpm-hs.c
+++ b/src/util-mpm-hs.c
@@ -379,29 +379,29 @@ typedef struct SCHSCompileData_ {
 
 static SCHSCompileData *SCHSAllocCompileData(unsigned int pattern_cnt)
 {
-    SCHSCompileData *cd = SCCalloc(1, pattern_cnt * sizeof(SCHSCompileData));
+    SCHSCompileData *cd = SCCalloc(pattern_cnt, sizeof(SCHSCompileData));
     if (cd == NULL) {
         goto error;
     }
 
     cd->pattern_cnt = pattern_cnt;
 
-    cd->ids = SCCalloc(1, pattern_cnt * sizeof(unsigned int));
+    cd->ids = SCCalloc(pattern_cnt, sizeof(unsigned int));
     if (cd->ids == NULL) {
         goto error;
     }
 
-    cd->flags = SCCalloc(1, pattern_cnt * sizeof(unsigned int));
+    cd->flags = SCCalloc(pattern_cnt, sizeof(unsigned int));
     if (cd->flags == NULL) {
         goto error;
     }
 
-    cd->expressions = SCCalloc(1, pattern_cnt * sizeof(char *));
+    cd->expressions = SCCalloc(pattern_cnt, sizeof(char *));
     if (cd->expressions == NULL) {
         goto error;
     }
 
-    cd->ext = SCCalloc(1, pattern_cnt * sizeof(hs_expr_ext_t *));
+    cd->ext = SCCalloc(pattern_cnt, sizeof(hs_expr_ext_t *));
     if (cd->ext == NULL) {
         goto error;
     }
@@ -559,7 +559,7 @@ static PatternDatabase *PatternDatabaseAlloc(uint32_t pattern_cnt)
     pd->hs_db = NULL;
 
     /* alloc the pattern array */
-    pd->parray = (SCHSPattern **)SCCalloc(1, pd->pattern_cnt * sizeof(SCHSPattern *));
+    pd->parray = (SCHSPattern **)SCCalloc(pd->pattern_cnt, sizeof(SCHSPattern *));
     if (pd->parray == NULL) {
         SCFree(pd);
         return NULL;
@@ -806,7 +806,7 @@ void SCHSInitCtx(MpmCtx *mpm_ctx)
 
     /* initialize the hash we use to speed up pattern insertions */
     SCHSCtx *ctx = (SCHSCtx *)mpm_ctx->ctx;
-    ctx->init_hash = SCCalloc(1, sizeof(SCHSPattern *) * INIT_HASH_SIZE);
+    ctx->init_hash = SCCalloc(INIT_HASH_SIZE, sizeof(SCHSPattern *));
     if (ctx->init_hash == NULL) {
         exit(EXIT_FAILURE);
     }
diff --git a/src/util-profiling-keywords.c b/src/util-profiling-keywords.c
index bd7cda526b39..c0620a751bcd 100644
--- a/src/util-profiling-keywords.c
+++ b/src/util-profiling-keywords.c
@@ -283,7 +283,7 @@ void SCProfilingKeywordThreadSetup(SCProfileKeywordDetectCtx *ctx, DetectEngineT
     if (ctx == NULL)
         return;
 
-    SCProfileKeywordData *a = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+    SCProfileKeywordData *a = SCCalloc(DETECT_TBLSIZE, sizeof(SCProfileKeywordData));
     if (a != NULL) {
         det_ctx->keyword_perf_data = a;
     }
@@ -294,7 +294,7 @@ void SCProfilingKeywordThreadSetup(SCProfileKeywordDetectCtx *ctx, DetectEngineT
 
     int i;
     for (i = 0; i < nlists; i++) {
-        SCProfileKeywordData *b = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+        SCProfileKeywordData *b = SCCalloc(DETECT_TBLSIZE, sizeof(SCProfileKeywordData));
         if (b != NULL) {
             det_ctx->keyword_perf_data_per_list[i] = b;
         }
@@ -369,7 +369,7 @@ SCProfilingKeywordInitCounters(DetectEngineCtx *de_ctx)
     de_ctx->profile_keyword_ctx = SCProfilingKeywordInitCtx();
     BUG_ON(de_ctx->profile_keyword_ctx == NULL);
 
-    de_ctx->profile_keyword_ctx->data = SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+    de_ctx->profile_keyword_ctx->data = SCCalloc(DETECT_TBLSIZE, sizeof(SCProfileKeywordData));
     BUG_ON(de_ctx->profile_keyword_ctx->data == NULL);
 
     de_ctx->profile_keyword_ctx_per_list = SCCalloc(nlists, sizeof(SCProfileKeywordDetectCtx *));
@@ -380,7 +380,7 @@ SCProfilingKeywordInitCounters(DetectEngineCtx *de_ctx)
         de_ctx->profile_keyword_ctx_per_list[i] = SCProfilingKeywordInitCtx();
         BUG_ON(de_ctx->profile_keyword_ctx_per_list[i] == NULL);
         de_ctx->profile_keyword_ctx_per_list[i]->data =
-                SCCalloc(1, sizeof(SCProfileKeywordData) * DETECT_TBLSIZE);
+                SCCalloc(DETECT_TBLSIZE, sizeof(SCProfileKeywordData));
         BUG_ON(de_ctx->profile_keyword_ctx_per_list[i]->data == NULL);
     }
 
diff --git a/src/util-profiling-rules.c b/src/util-profiling-rules.c
index 0397b8a0c0c6..8262f71f4c8a 100644
--- a/src/util-profiling-rules.c
+++ b/src/util-profiling-rules.c
@@ -588,7 +588,7 @@ void SCProfilingRuleThreadSetup(SCProfileDetectCtx *ctx, DetectEngineThreadCtx *
     if (ctx == NULL|| ctx->size == 0)
         return;
 
-    SCProfileData *a = SCCalloc(1, sizeof(SCProfileData) * ctx->size);
+    SCProfileData *a = SCCalloc(ctx->size, sizeof(SCProfileData));
     if (a != NULL) {
         det_ctx->rule_perf_data = a;
         det_ctx->rule_perf_data_size = ctx->size;
@@ -665,7 +665,7 @@ SCProfilingRuleInitCounters(DetectEngineCtx *de_ctx)
     }
 
     if (count > 0) {
-        de_ctx->profile_ctx->data = SCCalloc(1, sizeof(SCProfileData) * de_ctx->profile_ctx->size);
+        de_ctx->profile_ctx->data = SCCalloc(de_ctx->profile_ctx->size, sizeof(SCProfileData));
         BUG_ON(de_ctx->profile_ctx->data == NULL);
 
         sig = de_ctx->sig_list;
diff --git a/src/util-storage.c b/src/util-storage.c
index 1394f205231a..52b819d0e680 100644
--- a/src/util-storage.c
+++ b/src/util-storage.c
@@ -149,14 +149,14 @@ int StorageFinalize(void)
     if (count == 0)
         return 0;
 
-    storage_map = SCCalloc(1, sizeof(StorageMapping *) * STORAGE_MAX);
+    storage_map = SCCalloc(STORAGE_MAX, sizeof(StorageMapping *));
     if (unlikely(storage_map == NULL)) {
         return -1;
     }
 
     for (i = 0; i < STORAGE_MAX; i++) {
         if (storage_max_id[i] > 0) {
-            storage_map[i] = SCCalloc(1, sizeof(StorageMapping) * storage_max_id[i]);
+            storage_map[i] = SCCalloc(storage_max_id[i], sizeof(StorageMapping));
             if (storage_map[i] == NULL)
                 return -1;
         }
@@ -262,7 +262,7 @@ void *StorageAllocById(Storage **storage, StorageEnum type, int id)
     Storage *store = *storage;
     if (store == NULL) {
         // coverity[suspicious_sizeof : FALSE]
-        store = SCCalloc(1, sizeof(void *) * storage_max_id[type]);
+        store = SCCalloc(storage_max_id[type], sizeof(void *));
         if (unlikely(store == NULL))
             return NULL;
     }

From edc89ce7915824b825d7d67e691b3b3add688427 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 13 Sep 2023 07:01:53 +0200
Subject: [PATCH 160/462] packetpool: dynamic return threshold

Problem:

In pcap autofp mode, there is one threads reading packets (RX). These packets
are then passed on to worker threads. When these workers are done with a
packet, they return packets to the pcap reader threads packet pool, which is
the owner of the packets. Since this requires expensive synchronization between
threads, there is logic in place to batch this operation.

When the reader thread depletes its pool, it notifies the other threads that
it is starving and that a sync needs to happen asap. Then the reader enters
a wait state. During this time no new packets are read.

However, there is a problem with this approach. When the reader encountered
an empty pool, it would set an atomic flag that it needed a sync. The first
worker to return a packet to the pool would then set this flag, sync, and
unset the flag. This forced sync could result in just a single packet being
synchronized, or several. So if unlucky, the reader would just get a single
packet before hitting the same condition again.

Solution:

This patch updates the logic to use a new approach. Instead of using a
binary flag approach where the behavior only changes when the reader is
already starved, it uses a dynamic sync threshold that is controlled by
the reader. The reader keeps a running count of packets it its pool,
and calculates the percentage of available packets. This percentage is
then used to set the sync threshold.

When the pool is starved, it sets the threshold to 1 (sync for each packet).
After each successful get/sync the threshold is adjusted.
---
 src/tmqh-packetpool.c | 46 +++++++++++++++++++++++++++++++++++--------
 src/tmqh-packetpool.h |  8 +++++++-
 2 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/src/tmqh-packetpool.c b/src/tmqh-packetpool.c
index 85946517953a..c302bde18732 100644
--- a/src/tmqh-packetpool.c
+++ b/src/tmqh-packetpool.c
@@ -35,6 +35,8 @@
 #include "util-validate.h"
 #include "action-globals.h"
 
+extern uint16_t max_pending_packets;
+
 /* Number of freed packet to save for one pool before freeing them. */
 #define MAX_PENDING_RETURN_PACKETS 32
 static uint32_t max_pending_return_packets = MAX_PENDING_RETURN_PACKETS;
@@ -66,15 +68,27 @@ static int PacketPoolIsEmpty(PktPool *pool)
     return 1;
 }
 
+static void UpdateReturnThreshold(PktPool *pool)
+{
+    const float perc = (float)pool->cnt / (float)max_pending_packets;
+    uint32_t threshold = (uint32_t)(perc * (float)max_pending_return_packets);
+    if (threshold != SC_ATOMIC_GET(pool->return_stack.return_threshold)) {
+        SC_ATOMIC_SET(pool->return_stack.return_threshold, threshold);
+    }
+}
+
 void PacketPoolWait(void)
 {
     PktPool *my_pool = GetThreadPacketPool();
 
     if (PacketPoolIsEmpty(my_pool)) {
+        SC_ATOMIC_SET(my_pool->return_stack.return_threshold, 1);
+
         SCMutexLock(&my_pool->return_stack.mutex);
-        SC_ATOMIC_ADD(my_pool->return_stack.sync_now, 1);
         SCCondWait(&my_pool->return_stack.cond, &my_pool->return_stack.mutex);
         SCMutexUnlock(&my_pool->return_stack.mutex);
+
+        UpdateReturnThreshold(my_pool);
     }
 
     while(PacketPoolIsEmpty(my_pool))
@@ -98,6 +112,8 @@ static void PacketPoolGetReturnedPackets(PktPool *pool)
     /* Move all the packets from the locked return stack to the local stack. */
     pool->head = pool->return_stack.head;
     pool->return_stack.head = NULL;
+    pool->cnt += pool->return_stack.cnt;
+    pool->return_stack.cnt = 0;
     SCMutexUnlock(&pool->return_stack.mutex);
 }
 
@@ -119,8 +135,14 @@ Packet *PacketPoolGetPacket(void)
         /* Stack is not empty. */
         Packet *p = pool->head;
         pool->head = p->next;
+        pool->cnt--;
         p->pool = pool;
         PacketReinit(p);
+
+        UpdateReturnThreshold(pool);
+        SCLogDebug("pp: %0.2f cnt:%u max:%d threshold:%u",
+                ((float)pool->cnt / (float)max_pending_packets) * (float)100, pool->cnt,
+                max_pending_packets, SC_ATOMIC_GET(pool->return_stack.return_threshold));
         return p;
     }
 
@@ -135,8 +157,14 @@ Packet *PacketPoolGetPacket(void)
         /* Stack is not empty. */
         Packet *p = pool->head;
         pool->head = p->next;
+        pool->cnt--;
         p->pool = pool;
         PacketReinit(p);
+
+        UpdateReturnThreshold(pool);
+        SCLogDebug("pp: %0.2f cnt:%u max:%d threshold:%u",
+                ((float)pool->cnt / (float)max_pending_packets) * (float)100, pool->cnt,
+                max_pending_packets, SC_ATOMIC_GET(pool->return_stack.return_threshold));
         return p;
     }
 
@@ -170,6 +198,7 @@ void PacketPoolReturnPacket(Packet *p)
         /* Push back onto this thread's own stack, so no locking. */
         p->next = my_pool->head;
         my_pool->head = p;
+        my_pool->cnt++;
     } else {
         PktPool *pending_pool = my_pool->pending_pool;
         if (pending_pool == NULL || pending_pool == pool) {
@@ -187,12 +216,13 @@ void PacketPoolReturnPacket(Packet *p)
                 my_pool->pending_count++;
             }
 
-            if (SC_ATOMIC_GET(pool->return_stack.sync_now) || my_pool->pending_count > max_pending_return_packets) {
+            const uint32_t threshold = SC_ATOMIC_GET(pool->return_stack.return_threshold);
+            if (my_pool->pending_count >= threshold) {
                 /* Return the entire list of pending packets. */
                 SCMutexLock(&pool->return_stack.mutex);
                 my_pool->pending_tail->next = pool->return_stack.head;
                 pool->return_stack.head = my_pool->pending_head;
-                SC_ATOMIC_RESET(pool->return_stack.sync_now);
+                pool->return_stack.cnt += my_pool->pending_count;
                 SCCondSignal(&pool->return_stack.cond);
                 SCMutexUnlock(&pool->return_stack.mutex);
                 /* Clear the list of pending packets to return. */
@@ -206,7 +236,7 @@ void PacketPoolReturnPacket(Packet *p)
             SCMutexLock(&pool->return_stack.mutex);
             p->next = pool->return_stack.head;
             pool->return_stack.head = p;
-            SC_ATOMIC_RESET(pool->return_stack.sync_now);
+            pool->return_stack.cnt++;
             SCMutexUnlock(&pool->return_stack.mutex);
             SCCondSignal(&pool->return_stack.cond);
         }
@@ -225,13 +255,12 @@ void PacketPoolInitEmpty(void)
 
     SCMutexInit(&my_pool->return_stack.mutex, NULL);
     SCCondInit(&my_pool->return_stack.cond, NULL);
-    SC_ATOMIC_INIT(my_pool->return_stack.sync_now);
+    SC_ATOMIC_INIT(my_pool->return_stack.return_threshold);
+    SC_ATOMIC_SET(my_pool->return_stack.return_threshold, 32);
 }
 
 void PacketPoolInit(void)
 {
-    extern uint16_t max_pending_packets;
-
     PktPool *my_pool = GetThreadPacketPool();
 
 #ifdef DEBUG_VALIDATION
@@ -242,7 +271,8 @@ void PacketPoolInit(void)
 
     SCMutexInit(&my_pool->return_stack.mutex, NULL);
     SCCondInit(&my_pool->return_stack.cond, NULL);
-    SC_ATOMIC_INIT(my_pool->return_stack.sync_now);
+    SC_ATOMIC_INIT(my_pool->return_stack.return_threshold);
+    SC_ATOMIC_SET(my_pool->return_stack.return_threshold, 32);
 
     /* pre allocate packets */
     SCLogDebug("preallocating packets... packet size %" PRIuMAX "",
diff --git a/src/tmqh-packetpool.h b/src/tmqh-packetpool.h
index 2e9672d4458c..74074f953378 100644
--- a/src/tmqh-packetpool.h
+++ b/src/tmqh-packetpool.h
@@ -32,7 +32,11 @@ typedef struct PktPoolLockedStack_{
     /* linked list of free packets. */
     SCMutex mutex;
     SCCondT cond;
-    SC_ATOMIC_DECLARE(int, sync_now);
+    /** number of packets in needed to trigger a sync during
+     *  the return to pool logic. Updated by pool owner based
+     *  on how full the pool is. */
+    SC_ATOMIC_DECLARE(uint32_t, return_threshold);
+    uint32_t cnt;
     Packet *head;
 } __attribute__((aligned(CLS))) PktPoolLockedStack;
 
@@ -41,6 +45,8 @@ typedef struct PktPool_ {
      * No mutex is needed.
      */
     Packet *head;
+    uint32_t cnt;
+
     /* Packets waiting (pending) to be returned to the given Packet
      * Pool. Accumulate packets for the same pool until a threshold is
      * reached, then return them all at once.  Keep the head and tail

From 41c0526fdc6c5ab51e73553c992aa44234fce2bf Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 16 Nov 2023 15:29:53 +0100
Subject: [PATCH 161/462] packetpool: signal condition within lock

Completes: dc40a139acb3 ("packetpool: signal waiter within lock")
---
 src/tmqh-packetpool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/tmqh-packetpool.c b/src/tmqh-packetpool.c
index c302bde18732..f71274b4502c 100644
--- a/src/tmqh-packetpool.c
+++ b/src/tmqh-packetpool.c
@@ -237,8 +237,8 @@ void PacketPoolReturnPacket(Packet *p)
             p->next = pool->return_stack.head;
             pool->return_stack.head = p;
             pool->return_stack.cnt++;
-            SCMutexUnlock(&pool->return_stack.mutex);
             SCCondSignal(&pool->return_stack.cond);
+            SCMutexUnlock(&pool->return_stack.mutex);
         }
     }
 }

From 3a79984e5ea81f8fd376caaee19ed69f68b805a3 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 21 Nov 2023 21:31:10 +0100
Subject: [PATCH 162/462] detect/xbits: fix coverity warning

CID 1554237 and CID 1554233

Basically make the code easier to reason with for coverity
without changing the behavior which was fine.
---
 src/detect-xbits.c | 23 +++++++++--------------
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/src/detect-xbits.c b/src/detect-xbits.c
index 92b86ba9da0b..a3f67dbfc9b7 100644
--- a/src/detect-xbits.c
+++ b/src/detect-xbits.c
@@ -340,44 +340,39 @@ int DetectXbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
     int result = DetectXbitParse(de_ctx, rawstr, &cd);
     if (result < 0) {
         return -1;
-    /* noalert doesn't use a cd/sm struct. It flags the sig. We're done. */
-    } else if (result == 0 && cd == NULL) {
+    } else if (cd == NULL) {
+        /* noalert doesn't use a cd/sm struct. It flags the sig. We're done. */
         s->flags |= SIG_FLAG_NOALERT;
         return 0;
     }
 
     /* Okay so far so good, lets get this into a SigMatch
      * and put it in the Signature. */
-
     switch (cd->cmd) {
         /* case DETECT_XBITS_CMD_NOALERT can't happen here */
-
         case DETECT_XBITS_CMD_ISNOTSET:
         case DETECT_XBITS_CMD_ISSET:
             /* checks, so packet list */
             if (SigMatchAppendSMToList(
                         de_ctx, s, DETECT_XBITS, (SigMatchCtx *)cd, DETECT_SM_LIST_MATCH) == NULL) {
-                goto error;
+                SCFree(cd);
+                return -1;
             }
             break;
 
-        case DETECT_XBITS_CMD_SET:
-        case DETECT_XBITS_CMD_UNSET:
-        case DETECT_XBITS_CMD_TOGGLE:
+        // all other cases
+        // DETECT_XBITS_CMD_SET, DETECT_XBITS_CMD_UNSET, DETECT_XBITS_CMD_TOGGLE:
+        default:
             /* modifiers, only run when entire sig has matched */
             if (SigMatchAppendSMToList(de_ctx, s, DETECT_XBITS, (SigMatchCtx *)cd,
                         DETECT_SM_LIST_POSTMATCH) == NULL) {
-                goto error;
+                SCFree(cd);
+                return -1;
             }
             break;
     }
 
     return 0;
-
-error:
-    if (cd != NULL)
-        SCFree(cd);
-    return -1;
 }
 
 static void DetectXbitFree (DetectEngineCtx *de_ctx, void *ptr)

From 1f7e61ef61b6a3cc6e69d2f20992f4e386453981 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 22 Nov 2023 08:17:45 +0100
Subject: [PATCH 163/462] host: fix minor coverity warning

CID 1554240:  Data race undermines locking  (LOCK_EVASION)
---
 src/host.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/src/host.c b/src/host.c
index 7a5305ac18cc..37c0f4dad479 100644
--- a/src/host.c
+++ b/src/host.c
@@ -340,14 +340,11 @@ void HostShutdown(void)
  */
 void HostCleanup(void)
 {
-    Host *h;
-    uint32_t u;
-
     if (host_hash != NULL) {
-        for (u = 0; u < host_config.hash_size; u++) {
-            h = host_hash[u].head;
+        for (uint32_t u = 0; u < host_config.hash_size; u++) {
             HostHashRow *hb = &host_hash[u];
             HRLOCK_LOCK(hb);
+            Host *h = host_hash[u].head;
             while (h) {
                 if ((SC_ATOMIC_GET(h->use_cnt) > 0) && (h->iprep != NULL)) {
                     /* iprep is attached to host only clear local storage */

From 487b78fb3da85cf449a2e18dbf5f13488ca8b860 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 22 Nov 2023 08:21:23 +0100
Subject: [PATCH 164/462] ippair: fix minor coverity warning

CID 1554232:  Data race undermines locking  (LOCK_EVASION)

Ticket #6565.
---
 src/ippair.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/src/ippair.c b/src/ippair.c
index 81362f63f4c9..b06f3d164414 100644
--- a/src/ippair.c
+++ b/src/ippair.c
@@ -338,14 +338,11 @@ void IPPairShutdown(void)
  */
 void IPPairCleanup(void)
 {
-    IPPair *h;
-    uint32_t u;
-
     if (ippair_hash != NULL) {
-        for (u = 0; u < ippair_config.hash_size; u++) {
-            h = ippair_hash[u].head;
+        for (uint32_t u = 0; u < ippair_config.hash_size; u++) {
             IPPairHashRow *hb = &ippair_hash[u];
             HRLOCK_LOCK(hb);
+            IPPair *h = ippair_hash[u].head;
             while (h) {
                 if ((SC_ATOMIC_GET(h->use_cnt) > 0)) {
                     /* iprep is attached to ippair only clear local storage */

From 3107a4953d087b80d54cfecf2f5d489e7b35ee09 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 22 Nov 2023 09:03:09 +0100
Subject: [PATCH 165/462] flow: fix condition signalling

Signal threads while holding lock. This should make the signalling
more reliable.

From PTHREAD_COND(3):

"Unlocking the mutex and suspending on the condition variable is done
 atomically. Thus, if all threads always acquire the mutex before
 signaling the condition, this guarantees that the condition cannot be
 signaled (and thus ignored) between the time a thread locks the
 mutex and the time it waits on the condition variable."

Ticket: #6569.
---
 src/flow-manager.c | 28 ++++++++++++++++++----------
 src/flow-manager.h |  9 ++-------
 2 files changed, 20 insertions(+), 17 deletions(-)

diff --git a/src/flow-manager.c b/src/flow-manager.c
index e5e1aa270276..bcc1498c8bd8 100644
--- a/src/flow-manager.c
+++ b/src/flow-manager.c
@@ -84,10 +84,24 @@ SC_ATOMIC_DECLARE(uint32_t, flowrec_cnt);
 SC_ATOMIC_DECLARE(uint32_t, flowrec_busy);
 SC_ATOMIC_EXTERN(unsigned int, flow_flags);
 
-SCCtrlCondT flow_manager_ctrl_cond;
-SCCtrlMutex flow_manager_ctrl_mutex;
-SCCtrlCondT flow_recycler_ctrl_cond;
-SCCtrlMutex flow_recycler_ctrl_mutex;
+static SCCtrlCondT flow_manager_ctrl_cond = PTHREAD_COND_INITIALIZER;
+static SCCtrlMutex flow_manager_ctrl_mutex = PTHREAD_MUTEX_INITIALIZER;
+static SCCtrlCondT flow_recycler_ctrl_cond = PTHREAD_COND_INITIALIZER;
+static SCCtrlMutex flow_recycler_ctrl_mutex = PTHREAD_MUTEX_INITIALIZER;
+
+void FlowWakeupFlowManagerThread(void)
+{
+    SCCtrlMutexLock(&flow_manager_ctrl_mutex);
+    SCCtrlCondSignal(&flow_manager_ctrl_cond);
+    SCCtrlMutexUnlock(&flow_manager_ctrl_mutex);
+}
+
+void FlowWakeupFlowRecyclerThread(void)
+{
+    SCCtrlMutexLock(&flow_recycler_ctrl_mutex);
+    SCCtrlCondSignal(&flow_recycler_ctrl_cond);
+    SCCtrlMutexUnlock(&flow_recycler_ctrl_mutex);
+}
 
 void FlowTimeoutsInit(void)
 {
@@ -942,9 +956,6 @@ void FlowManagerThreadSpawn(void)
     }
     flowmgr_number = (uint32_t)setting;
 
-    SCCtrlCondInit(&flow_manager_ctrl_cond, NULL);
-    SCCtrlMutexInit(&flow_manager_ctrl_mutex, NULL);
-
     SCLogConfig("using %u flow manager threads", flowmgr_number);
     StatsRegisterGlobalCounter("flow.memuse", FlowGetMemuse);
 
@@ -1148,9 +1159,6 @@ void FlowRecyclerThreadSpawn(void)
     }
     flowrec_number = (uint32_t)setting;
 
-    SCCtrlCondInit(&flow_recycler_ctrl_cond, NULL);
-    SCCtrlMutexInit(&flow_recycler_ctrl_mutex, NULL);
-
     SCLogConfig("using %u flow recycler threads", flowrec_number);
 
     for (uint32_t u = 0; u < flowrec_number; u++) {
diff --git a/src/flow-manager.h b/src/flow-manager.h
index 157358d170f5..7cdd017000aa 100644
--- a/src/flow-manager.h
+++ b/src/flow-manager.h
@@ -24,13 +24,8 @@
 #ifndef __FLOW_MANAGER_H__
 #define __FLOW_MANAGER_H__
 
-/** flow manager scheduling condition */
-extern SCCtrlCondT flow_manager_ctrl_cond;
-extern SCCtrlMutex flow_manager_ctrl_mutex;
-#define FlowWakeupFlowManagerThread() SCCtrlCondSignal(&flow_manager_ctrl_cond)
-extern SCCtrlCondT flow_recycler_ctrl_cond;
-extern SCCtrlMutex flow_recycler_ctrl_mutex;
-#define FlowWakeupFlowRecyclerThread() SCCtrlCondSignal(&flow_recycler_ctrl_cond)
+void FlowWakeupFlowManagerThread(void);
+void FlowWakeupFlowRecyclerThread(void);
 
 #define FlowTimeoutsReset() FlowTimeoutsInit()
 void FlowTimeoutsInit(void);

From 74bc27ecd5a6b5d4b862c25397486ce5e82d44e0 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 22 Nov 2023 09:31:09 +0100
Subject: [PATCH 166/462] stats: turn sync macros into functions

---
 src/counters.c | 12 ++++++++++++
 src/counters.h | 12 ++----------
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/src/counters.c b/src/counters.c
index e3ed5244cdfe..56332430d8a2 100644
--- a/src/counters.c
+++ b/src/counters.c
@@ -453,6 +453,18 @@ static void *StatsMgmtThread(void *arg)
     return NULL;
 }
 
+void StatsSyncCounters(ThreadVars *tv)
+{
+    StatsUpdateCounterArray(&tv->perf_private_ctx, &tv->perf_public_ctx);
+}
+
+void StatsSyncCountersIfSignalled(ThreadVars *tv)
+{
+    if (tv->perf_public_ctx.perf_flag == 1) {
+        StatsUpdateCounterArray(&tv->perf_private_ctx, &tv->perf_public_ctx);
+    }
+}
+
 /**
  * \brief Wake up thread.  This thread wakes up every TTS(time to sleep) seconds
  *        and sets the flag for every ThreadVars' StatsPublicThreadContext
diff --git a/src/counters.h b/src/counters.h
index b1505e750a7d..100fec94f7e6 100644
--- a/src/counters.h
+++ b/src/counters.h
@@ -135,16 +135,8 @@ uint64_t StatsGetLocalCounterValue(struct ThreadVars_ *, uint16_t);
 int StatsSetupPrivate(struct ThreadVars_ *);
 void StatsThreadCleanup(struct ThreadVars_ *);
 
-#define StatsSyncCounters(tv) \
-    StatsUpdateCounterArray(&(tv)->perf_private_ctx, &(tv)->perf_public_ctx);  \
-
-#define StatsSyncCountersIfSignalled(tv)                                       \
-    do {                                                                        \
-        if ((tv)->perf_public_ctx.perf_flag == 1) {                             \
-            StatsUpdateCounterArray(&(tv)->perf_private_ctx,                   \
-                                     &(tv)->perf_public_ctx);                   \
-        }                                                                       \
-    } while (0)
+void StatsSyncCounters(struct ThreadVars_ *tv);
+void StatsSyncCountersIfSignalled(struct ThreadVars_ *tv);
 
 #ifdef BUILD_UNIX_SOCKET
 TmEcode StatsOutputCounterSocket(json_t *cmd,

From d005fff7b9552de6203995b774ad64678357b381 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 22 Nov 2023 09:31:38 +0100
Subject: [PATCH 167/462] stats: improve sync signalling

Make syncs more reliable by using a atomic "sync now" variable and
signalling the conditions under lock.

Ticket: #6569.
---
 src/counters.c | 16 +++++++---------
 src/counters.h |  2 +-
 2 files changed, 8 insertions(+), 10 deletions(-)

diff --git a/src/counters.c b/src/counters.c
index 56332430d8a2..790f416ba050 100644
--- a/src/counters.c
+++ b/src/counters.c
@@ -130,7 +130,7 @@ static void StatsPublicThreadContextCleanup(StatsPublicThreadContext *t)
     SCMutexLock(&t->m);
     StatsReleaseCounters(t->head);
     t->head = NULL;
-    t->perf_flag = 0;
+    SC_ATOMIC_SET(t->sync_now, false);
     t->curr_id = 0;
     SCMutexUnlock(&t->m);
     SCMutexDestroy(&t->m);
@@ -460,7 +460,7 @@ void StatsSyncCounters(ThreadVars *tv)
 
 void StatsSyncCountersIfSignalled(ThreadVars *tv)
 {
-    if (tv->perf_public_ctx.perf_flag == 1) {
+    if (SC_ATOMIC_GET(tv->perf_public_ctx.sync_now) == true) {
         StatsUpdateCounterArray(&tv->perf_private_ctx, &tv->perf_public_ctx);
     }
 }
@@ -521,13 +521,13 @@ static void *StatsWakeupThread(void *arg)
                 continue;
             }
 
-            /* assuming the assignment of an int to be atomic, and even if it's
-             * not, it should be okay */
-            tv->perf_public_ctx.perf_flag = 1;
+            SC_ATOMIC_SET(tv->perf_public_ctx.sync_now, true);
 
             if (tv->inq != NULL) {
                 PacketQueue *q = tv->inq->pq;
+                SCMutexLock(&q->mutex_q);
                 SCCondSignal(&q->cond_q);
+                SCMutexUnlock(&q->mutex_q);
             }
 
             tv = tv->next;
@@ -541,9 +541,7 @@ static void *StatsWakeupThread(void *arg)
                 continue;
             }
 
-            /* assuming the assignment of an int to be atomic, and even if it's
-             * not, it should be okay */
-            tv->perf_public_ctx.perf_flag = 1;
+            SC_ATOMIC_SET(tv->perf_public_ctx.sync_now, true);
 
             tv = tv->next;
         }
@@ -1256,7 +1254,7 @@ int StatsUpdateCounterArray(StatsPrivateThreadContext *pca, StatsPublicThreadCon
     }
     SCMutexUnlock(&pctx->m);
 
-    pctx->perf_flag = 0;
+    SC_ATOMIC_SET(pctx->sync_now, false);
     return 1;
 }
 
diff --git a/src/counters.h b/src/counters.h
index 100fec94f7e6..b3ffddbd569f 100644
--- a/src/counters.h
+++ b/src/counters.h
@@ -63,7 +63,7 @@ typedef struct StatsCounter_ {
  */
 typedef struct StatsPublicThreadContext_ {
     /* flag set by the wakeup thread, to inform the client threads to sync */
-    uint32_t perf_flag;
+    SC_ATOMIC_DECLARE(bool, sync_now);
 
     /* pointer to the head of a list of counters assigned under this context */
     StatsCounter *head;

From 5954a914572c78b668b0be1ac958c906a7e3acc9 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 27 Nov 2023 11:10:50 +0100
Subject: [PATCH 168/462] detect/filestore: fix memory leak on sig parsing

Ticket: 6574

Introduced by commit c272a646c5ae739d18901776cc5a940afd3d3d38
---
 src/detect-filestore.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/detect-filestore.c b/src/detect-filestore.c
index 5e22e4c6cde5..07bbd91ff199 100644
--- a/src/detect-filestore.c
+++ b/src/detect-filestore.c
@@ -455,6 +455,7 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
 
     if (SigMatchAppendSMToList(
                 de_ctx, s, DETECT_FILESTORE, (SigMatchCtx *)fd, g_file_match_list_id) == NULL) {
+        DetectFilestoreFree(de_ctx, fd);
         goto error;
     }
     s->filestore_ctx = fd;

From 9c3ab36afc1f7f81d280f9d53b4acbe6866b56fd Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 27 Nov 2023 13:40:44 +0100
Subject: [PATCH 169/462] source: fix resource leak

CID: 1426081
---
 src/source-pcap-file.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/source-pcap-file.c b/src/source-pcap-file.c
index e54a607d4875..c4f97bc0c4c1 100644
--- a/src/source-pcap-file.c
+++ b/src/source-pcap-file.c
@@ -288,6 +288,7 @@ TmEcode ReceivePcapFileThreadInit(ThreadVars *tv, const void *initdata, void **d
         pv->filename = SCStrdup((char*)initdata);
         if (unlikely(pv->filename == NULL)) {
             SCLogError("Failed to allocate filename");
+            closedir(directory);
             CleanupPcapFileDirectoryVars(pv);
             CleanupPcapFileThreadVars(ptv);
             SCReturnInt(TM_ECODE_OK);
@@ -309,6 +310,7 @@ TmEcode ReceivePcapFileThreadInit(ThreadVars *tv, const void *initdata, void **d
         if (pv->should_recurse == true && pv->should_loop == true) {
             SCLogError("Error, --pcap-file-continuous and --pcap-file-recursive "
                        "cannot be used together.");
+            closedir(directory);
             CleanupPcapFileDirectoryVars(pv);
             CleanupPcapFileThreadVars(ptv);
             SCReturnInt(TM_ECODE_FAILED);

From 69f61c00281d28009270275cc8f53ac10e2fb513 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:17:57 +0100
Subject: [PATCH 170/462] storage: remove unused code

Only used in a unittest; removed that as well.
---
 src/util-storage.c | 138 ---------------------------------------------
 src/util-storage.h |   3 -
 2 files changed, 141 deletions(-)

diff --git a/src/util-storage.c b/src/util-storage.c
index 52b819d0e680..aec5c1d10fc6 100644
--- a/src/util-storage.c
+++ b/src/util-storage.c
@@ -251,36 +251,6 @@ void *StorageAllocByIdPrealloc(Storage *storage, StorageEnum type, int id)
     return storage[id];
 }
 
-void *StorageAllocById(Storage **storage, StorageEnum type, int id)
-{
-#ifdef DEBUG
-    BUG_ON(!storage_registration_closed);
-#endif
-    SCLogDebug("storage %p id %d", storage, id);
-
-    StorageMapping *map = &storage_map[type][id];
-    Storage *store = *storage;
-    if (store == NULL) {
-        // coverity[suspicious_sizeof : FALSE]
-        store = SCCalloc(storage_max_id[type], sizeof(void *));
-        if (unlikely(store == NULL))
-            return NULL;
-    }
-    SCLogDebug("store %p", store);
-
-    if (store[id] == NULL && map->Alloc != NULL) {
-        store[id] = map->Alloc(map->size);
-        if (store[id] == NULL) {
-            SCFree(store);
-            *storage = NULL;
-            return NULL;
-        }
-    }
-
-    *storage = store;
-    return store[id];
-}
-
 void StorageFreeById(Storage *storage, StorageEnum type, int id)
 {
 #ifdef DEBUG
@@ -326,32 +296,6 @@ void StorageFreeAll(Storage *storage, StorageEnum type)
     }
 }
 
-void StorageFree(Storage **storage, StorageEnum type)
-{
-    if (*storage == NULL)
-        return;
-
-#ifdef DEBUG
-    BUG_ON(!storage_registration_closed);
-#endif
-#ifdef UNITTESTS
-    if (storage_map == NULL)
-        return;
-#endif
-
-    Storage *store = *storage;
-    int i;
-    for (i = 0; i < storage_max_id[type]; i++) {
-        if (store[i] != NULL) {
-            StorageMapping *map = &storage_map[type][i];
-            map->Free(store[i]);
-            store[i] = NULL;
-        }
-    }
-    SCFree(*storage);
-    *storage = NULL;
-}
-
 #ifdef UNITTESTS
 
 static void *StorageTestAlloc(unsigned int size)
@@ -389,87 +333,6 @@ static int StorageTest01(void)
     return 0;
 }
 
-struct StorageTest02Data {
-    int abc;
-};
-
-static void *StorageTest02Init(unsigned int size)
-{
-    struct StorageTest02Data *data = (struct StorageTest02Data *)SCMalloc(size);
-    if (data != NULL)
-        data->abc = 1234;
-    return (void *)data;
-}
-
-static int StorageTest02(void)
-{
-    struct StorageTest02Data *test = NULL;
-
-    StorageInit();
-
-    int id1 = StorageRegister(STORAGE_HOST, "test", 4, StorageTest02Init, StorageTestFree);
-    if (id1 < 0) {
-        printf("StorageRegister failed (2): ");
-        goto error;
-    }
-    int id2 = StorageRegister(STORAGE_HOST, "test2", 4, StorageTest02Init, StorageTestFree);
-    if (id2 < 0) {
-        printf("StorageRegister failed (2): ");
-        goto error;
-    }
-
-    if (StorageFinalize() < 0) {
-        printf("StorageFinalize failed: ");
-        goto error;
-    }
-
-    Storage *storage = NULL;
-    void *data = StorageAllocById(&storage, STORAGE_HOST, id1);
-    if (data == NULL) {
-        printf("StorageAllocById failed, data == NULL, storage %p: ", storage);
-        goto error;
-    }
-    test = (struct StorageTest02Data *)data;
-    if (test->abc != 1234) {
-        printf("setup failed, test->abc != 1234, but %d (1):", test->abc);
-        goto error;
-    }
-    test->abc = 4321;
-
-    data = StorageAllocById(&storage, STORAGE_HOST, id2);
-    if (data == NULL) {
-        printf("StorageAllocById failed, data == NULL, storage %p: ", storage);
-        goto error;
-    }
-    test = (struct StorageTest02Data *)data;
-    if (test->abc != 1234) {
-        printf("setup failed, test->abc != 1234, but %d (2):", test->abc);
-        goto error;
-    }
-
-    data = StorageGetById(storage, STORAGE_HOST, id1);
-    if (data == NULL) {
-        printf("StorageAllocById failed, data == NULL, storage %p: ", storage);
-        goto error;
-    }
-    test = (struct StorageTest02Data *)data;
-    if (test->abc != 4321) {
-        printf("setup failed, test->abc != 4321, but %d (3):", test->abc);
-        goto error;
-    }
-
-    //StorageFreeById(storage, STORAGE_HOST, id1);
-    //StorageFreeById(storage, STORAGE_HOST, id2);
-
-    StorageFree(&storage, STORAGE_HOST);
-
-    StorageCleanup();
-    return 1;
-error:
-    StorageCleanup();
-    return 0;
-}
-
 static int StorageTest03(void)
 {
     StorageInit();
@@ -541,7 +404,6 @@ static int StorageTest03(void)
 void StorageRegisterTests(void)
 {
     UtRegisterTest("StorageTest01", StorageTest01);
-    UtRegisterTest("StorageTest02", StorageTest02);
     UtRegisterTest("StorageTest03", StorageTest03);
 }
 #endif
diff --git a/src/util-storage.h b/src/util-storage.h
index 984bcf8c20bc..6942ec2f2b82 100644
--- a/src/util-storage.h
+++ b/src/util-storage.h
@@ -66,11 +66,8 @@ int StorageSetById(Storage *storage, const StorageEnum type, const int id, void
 /** \brief AllocById func for prealloc'd base storage (storage ptrs are part
  *         of another memory block) */
 void *StorageAllocByIdPrealloc(Storage *storage, StorageEnum type, int id);
-/** \brief AllocById func for when we manage the Storage ptr itself */
-void *StorageAllocById(Storage **storage, const StorageEnum type, const int id);
 void StorageFreeById(Storage *storage, const StorageEnum type, const int id);
 void StorageFreeAll(Storage *storage, const StorageEnum type);
-void StorageFree(Storage **storage, const StorageEnum type);
 
 void StorageRegisterTests(void);
 #endif

From c1b920d0565a145833f2e555788ee4dd0fed35d8 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:28:17 +0100
Subject: [PATCH 171/462] storage: use proper type instead of void ptr

---
 src/util-storage.c | 24 ++++++++++++------------
 src/util-storage.h |  4 +++-
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/src/util-storage.c b/src/util-storage.c
index aec5c1d10fc6..493b139ee85f 100644
--- a/src/util-storage.c
+++ b/src/util-storage.c
@@ -218,7 +218,7 @@ void *StorageGetById(const Storage *storage, const StorageEnum type, const int i
     SCLogDebug("storage %p id %d", storage, id);
     if (storage == NULL)
         return NULL;
-    return storage[id];
+    return storage[id].ptr;
 }
 
 int StorageSetById(Storage *storage, const StorageEnum type, const int id, void *ptr)
@@ -229,7 +229,7 @@ int StorageSetById(Storage *storage, const StorageEnum type, const int id, void
     SCLogDebug("storage %p id %d", storage, id);
     if (storage == NULL)
         return -1;
-    storage[id] = ptr;
+    storage[id].ptr = ptr;
     return 0;
 }
 
@@ -241,14 +241,14 @@ void *StorageAllocByIdPrealloc(Storage *storage, StorageEnum type, int id)
     SCLogDebug("storage %p id %d", storage, id);
 
     StorageMapping *map = &storage_map[type][id];
-    if (storage[id] == NULL && map->Alloc != NULL) {
-        storage[id] = map->Alloc(map->size);
-        if (storage[id] == NULL) {
+    if (storage[id].ptr == NULL && map->Alloc != NULL) {
+        storage[id].ptr = map->Alloc(map->size);
+        if (storage[id].ptr == NULL) {
             return NULL;
         }
     }
 
-    return storage[id];
+    return storage[id].ptr;
 }
 
 void StorageFreeById(Storage *storage, StorageEnum type, int id)
@@ -265,10 +265,10 @@ void StorageFreeById(Storage *storage, StorageEnum type, int id)
     Storage *store = storage;
     if (store != NULL) {
         SCLogDebug("store %p", store);
-        if (store[id] != NULL) {
+        if (store[id].ptr != NULL) {
             StorageMapping *map = &storage_map[type][id];
-            map->Free(store[id]);
-            store[id] = NULL;
+            map->Free(store[id].ptr);
+            store[id].ptr = NULL;
         }
     }
 }
@@ -288,10 +288,10 @@ void StorageFreeAll(Storage *storage, StorageEnum type)
     Storage *store = storage;
     int i;
     for (i = 0; i < storage_max_id[type]; i++) {
-        if (store[i] != NULL) {
+        if (store[i].ptr != NULL) {
             StorageMapping *map = &storage_map[type][i];
-            map->Free(store[i]);
-            store[i] = NULL;
+            map->Free(store[i].ptr);
+            store[i].ptr = NULL;
         }
     }
 }
diff --git a/src/util-storage.h b/src/util-storage.h
index 6942ec2f2b82..6866874f7deb 100644
--- a/src/util-storage.h
+++ b/src/util-storage.h
@@ -36,7 +36,9 @@ typedef enum StorageEnum_ {
 } StorageEnum;
 
 /** void ptr array for now */
-typedef void* Storage;
+typedef struct Storage {
+    void *ptr;
+} Storage;
 
 void StorageInit(void);
 void StorageCleanup(void);

From d405efd3f67704209186c5def93846610ff91726 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:28:37 +0100
Subject: [PATCH 172/462] flow/storage: use flex array instead of calculated
 ptr

---
 src/flow-storage.c | 10 +++++-----
 src/flow.h         |  4 ++++
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/src/flow-storage.c b/src/flow-storage.c
index 1ce9c07948d6..53b67ba63048 100644
--- a/src/flow-storage.c
+++ b/src/flow-storage.c
@@ -39,28 +39,28 @@ unsigned int FlowStorageSize(void)
 
 void *FlowGetStorageById(const Flow *f, FlowStorageId id)
 {
-    return StorageGetById((Storage *)((void *)f + sizeof(Flow)), STORAGE_FLOW, id.id);
+    return StorageGetById(f->storage, STORAGE_FLOW, id.id);
 }
 
 int FlowSetStorageById(Flow *f, FlowStorageId id, void *ptr)
 {
-    return StorageSetById((Storage *)((void *)f + sizeof(Flow)), STORAGE_FLOW, id.id, ptr);
+    return StorageSetById(f->storage, STORAGE_FLOW, id.id, ptr);
 }
 
 void *FlowAllocStorageById(Flow *f, FlowStorageId id)
 {
-    return StorageAllocByIdPrealloc((Storage *)((void *)f + sizeof(Flow)), STORAGE_FLOW, id.id);
+    return StorageAllocByIdPrealloc(f->storage, STORAGE_FLOW, id.id);
 }
 
 void FlowFreeStorageById(Flow *f, FlowStorageId id)
 {
-    StorageFreeById((Storage *)((void *)f + sizeof(Flow)), STORAGE_FLOW, id.id);
+    StorageFreeById(f->storage, STORAGE_FLOW, id.id);
 }
 
 void FlowFreeStorage(Flow *f)
 {
     if (FlowStorageSize() > 0)
-        StorageFreeAll((Storage *)((void *)f + sizeof(Flow)), STORAGE_FLOW);
+        StorageFreeAll(f->storage, STORAGE_FLOW);
 }
 
 FlowStorageId FlowStorageRegister(const char *name, const unsigned int size,
diff --git a/src/flow.h b/src/flow.h
index 0a730e0ea3b8..c7b5867ea896 100644
--- a/src/flow.h
+++ b/src/flow.h
@@ -325,6 +325,8 @@ typedef unsigned short FlowStateType;
 /** Local Thread ID */
 typedef uint16_t FlowThreadId;
 
+#include "util-storage.h"
+
 /**
  *  \brief Flow data structure.
  *
@@ -489,6 +491,8 @@ typedef struct Flow_
     uint32_t tosrcpktcnt;
     uint64_t todstbytecnt;
     uint64_t tosrcbytecnt;
+
+    Storage storage[];
 } Flow;
 
 enum FlowState {

From 3d3a62dfe6c9f0789fea4d3bcfc9ed4940bca270 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:35:54 +0100
Subject: [PATCH 173/462] host/storage: use flex array for host storage

---
 src/host-storage.c | 10 +++++-----
 src/host.h         |  6 +++---
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/host-storage.c b/src/host-storage.c
index 72261de99d52..234c67112edf 100644
--- a/src/host-storage.c
+++ b/src/host-storage.c
@@ -74,7 +74,7 @@ HostStorageId HostStorageRegister(const char *name, const unsigned int size,
 
 int HostSetStorageById(Host *h, HostStorageId id, void *ptr)
 {
-    return StorageSetById((Storage *)((void *)h + sizeof(Host)), STORAGE_HOST, id.id, ptr);
+    return StorageSetById(h->storage, STORAGE_HOST, id.id, ptr);
 }
 
 /**
@@ -87,7 +87,7 @@ int HostSetStorageById(Host *h, HostStorageId id, void *ptr)
 
 void *HostGetStorageById(Host *h, HostStorageId id)
 {
-    return StorageGetById((Storage *)((void *)h + sizeof(Host)), STORAGE_HOST, id.id);
+    return StorageGetById(h->storage, STORAGE_HOST, id.id);
 }
 
 /**
@@ -98,18 +98,18 @@ void *HostGetStorageById(Host *h, HostStorageId id)
 
 void *HostAllocStorageById(Host *h, HostStorageId id)
 {
-    return StorageAllocByIdPrealloc((Storage *)((void *)h + sizeof(Host)), STORAGE_HOST, id.id);
+    return StorageAllocByIdPrealloc(h->storage, STORAGE_HOST, id.id);
 }
 
 void HostFreeStorageById(Host *h, HostStorageId id)
 {
-    StorageFreeById((Storage *)((void *)h + sizeof(Host)), STORAGE_HOST, id.id);
+    StorageFreeById(h->storage, STORAGE_HOST, id.id);
 }
 
 void HostFreeStorage(Host *h)
 {
     if (HostStorageSize() > 0)
-        StorageFreeAll((Storage *)((void *)h + sizeof(Host)), STORAGE_HOST);
+        StorageFreeAll(h->storage, STORAGE_HOST);
 }
 
 
diff --git a/src/host.h b/src/host.h
index 2c6a037edf7d..f4f248b5eec6 100644
--- a/src/host.h
+++ b/src/host.h
@@ -68,9 +68,6 @@ typedef struct Host_ {
     /** pointers to iprep storage */
     void *iprep;
 
-    /** storage api handle */
-    Storage *storage;
-
     /** hash pointers, protected by hash row mutex/spin */
     struct Host_ *hnext;
     struct Host_ *hprev;
@@ -78,6 +75,9 @@ typedef struct Host_ {
     /** list pointers, protected by host-queue mutex/spin */
     struct Host_ *lnext;
     struct Host_ *lprev;
+
+    /** storage api handle */
+    Storage storage[];
 } Host;
 
 typedef struct HostHashRow_ {

From 11d73e284c6ab3099919fdb0ed167679ee2bc3fc Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:38:16 +0100
Subject: [PATCH 174/462] ippair/storage: use flex array instead of calculated
 ptr

---
 src/ippair-storage.c | 10 +++++-----
 src/ippair.h         |  6 +++---
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/ippair-storage.c b/src/ippair-storage.c
index 0d1fd4af99d0..a0e65c75df39 100644
--- a/src/ippair-storage.c
+++ b/src/ippair-storage.c
@@ -34,28 +34,28 @@ unsigned int IPPairStorageSize(void)
 
 void *IPPairGetStorageById(IPPair *h, IPPairStorageId id)
 {
-    return StorageGetById((Storage *)((void *)h + sizeof(IPPair)), STORAGE_IPPAIR, id.id);
+    return StorageGetById(h->storage, STORAGE_IPPAIR, id.id);
 }
 
 int IPPairSetStorageById(IPPair *h, IPPairStorageId id, void *ptr)
 {
-    return StorageSetById((Storage *)((void *)h + sizeof(IPPair)), STORAGE_IPPAIR, id.id, ptr);
+    return StorageSetById(h->storage, STORAGE_IPPAIR, id.id, ptr);
 }
 
 void *IPPairAllocStorageById(IPPair *h, IPPairStorageId id)
 {
-    return StorageAllocByIdPrealloc((Storage *)((void *)h + sizeof(IPPair)), STORAGE_IPPAIR, id.id);
+    return StorageAllocByIdPrealloc(h->storage, STORAGE_IPPAIR, id.id);
 }
 
 void IPPairFreeStorageById(IPPair *h, IPPairStorageId id)
 {
-    StorageFreeById((Storage *)((void *)h + sizeof(IPPair)), STORAGE_IPPAIR, id.id);
+    StorageFreeById(h->storage, STORAGE_IPPAIR, id.id);
 }
 
 void IPPairFreeStorage(IPPair *h)
 {
     if (IPPairStorageSize() > 0)
-        StorageFreeAll((Storage *)((void *)h + sizeof(IPPair)), STORAGE_IPPAIR);
+        StorageFreeAll(h->storage, STORAGE_IPPAIR);
 }
 
 IPPairStorageId IPPairStorageRegister(const char *name, const unsigned int size,
diff --git a/src/ippair.h b/src/ippair.h
index 82f1c094af9a..3eef45ad8fcc 100644
--- a/src/ippair.h
+++ b/src/ippair.h
@@ -65,9 +65,6 @@ typedef struct IPPair_ {
     /** use cnt, reference counter */
     SC_ATOMIC_DECLARE(unsigned int, use_cnt);
 
-    /** storage api handle */
-    Storage *storage;
-
     /** hash pointers, protected by hash row mutex/spin */
     struct IPPair_ *hnext;
     struct IPPair_ *hprev;
@@ -75,6 +72,9 @@ typedef struct IPPair_ {
     /** list pointers, protected by ippair-queue mutex/spin */
     struct IPPair_ *lnext;
     struct IPPair_ *lprev;
+
+    /** storage api handle as a flex array member, so must stay last */
+    Storage storage[];
 } IPPair;
 
 typedef struct IPPairHashRow_ {

From f10233fecf8a7fd18a6e7ceac8d6d54e35f53f20 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 24 Nov 2023 19:41:47 +0100
Subject: [PATCH 175/462] device/storage: use flex array instead of calculated
 ptr

---
 src/device-storage.c | 11 +++++------
 src/util-device.h    |  3 +++
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/src/device-storage.c b/src/device-storage.c
index f424888ecfcc..d4314c021ea5 100644
--- a/src/device-storage.c
+++ b/src/device-storage.c
@@ -75,7 +75,7 @@ LiveDevStorageId LiveDevStorageRegister(const char *name, const unsigned int siz
 
 int LiveDevSetStorageById(LiveDevice *d, LiveDevStorageId id, void *ptr)
 {
-    return StorageSetById((Storage *)((void *)d + sizeof(LiveDevice)), STORAGE_DEVICE, id.id, ptr);
+    return StorageSetById(d->storage, STORAGE_DEVICE, id.id, ptr);
 }
 
 /**
@@ -88,7 +88,7 @@ int LiveDevSetStorageById(LiveDevice *d, LiveDevStorageId id, void *ptr)
 
 void *LiveDevGetStorageById(LiveDevice *d, LiveDevStorageId id)
 {
-    return StorageGetById((Storage *)((void *)d + sizeof(LiveDevice)), STORAGE_DEVICE, id.id);
+    return StorageGetById(d->storage, STORAGE_DEVICE, id.id);
 }
 
 /**
@@ -99,19 +99,18 @@ void *LiveDevGetStorageById(LiveDevice *d, LiveDevStorageId id)
 
 void *LiveDevAllocStorageById(LiveDevice *d, LiveDevStorageId id)
 {
-    return StorageAllocByIdPrealloc(
-            (Storage *)((void *)d + sizeof(LiveDevice)), STORAGE_DEVICE, id.id);
+    return StorageAllocByIdPrealloc(d->storage, STORAGE_DEVICE, id.id);
 }
 
 void LiveDevFreeStorageById(LiveDevice *d, LiveDevStorageId id)
 {
-    StorageFreeById((Storage *)((void *)d + sizeof(LiveDevice)), STORAGE_DEVICE, id.id);
+    StorageFreeById(d->storage, STORAGE_DEVICE, id.id);
 }
 
 void LiveDevFreeStorage(LiveDevice *d)
 {
     if (LiveDevStorageSize() > 0)
-        StorageFreeAll((Storage *)((void *)d + sizeof(LiveDevice)), STORAGE_DEVICE);
+        StorageFreeAll(d->storage, STORAGE_DEVICE);
 }
 
 
diff --git a/src/util-device.h b/src/util-device.h
index 0f756b78ca3a..d5a2d46caf08 100644
--- a/src/util-device.h
+++ b/src/util-device.h
@@ -23,6 +23,7 @@
 #endif /* HAVE_DPDK */
 
 #include "queue.h"
+#include "util-storage.h"
 
 #define OFFLOAD_FLAG_SG     (1<<0)
 #define OFFLOAD_FLAG_TSO    (1<<1)
@@ -66,6 +67,8 @@ typedef struct LiveDevice_ {
     // DPDK resources that needs to be cleaned after workers are stopped and devices closed
     DPDKDeviceResources dpdk_vars;
 #endif
+    /** storage handle as a flex array member */
+    Storage storage[];
 } LiveDevice;
 
 typedef struct LiveDeviceName_ {

From bc7508e4df00efec53dc200044f6d66b8f94bdfa Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 25 Nov 2023 07:43:16 +0100
Subject: [PATCH 176/462] log-pcap: constify PcapWrite args

General cleanup, but also needed for packet changes.
---
 src/log-pcap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/log-pcap.c b/src/log-pcap.c
index e237b7ad9e99..6039e057b8a4 100644
--- a/src/log-pcap.c
+++ b/src/log-pcap.c
@@ -498,7 +498,7 @@ static void PcapLogUnlock(PcapLogData *pl)
 }
 
 static inline int PcapWrite(
-        PcapLogData *pl, PcapLogCompressionData *comp, uint8_t *data, size_t len)
+        PcapLogData *pl, PcapLogCompressionData *comp, const uint8_t *data, const size_t len)
 {
     struct timeval current_dump;
     gettimeofday(&current_dump, NULL);

From 9ae2cd0c597b9cc6756fc86a502206efa9441ec9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 25 Nov 2023 07:43:34 +0100
Subject: [PATCH 177/462] packet: access packet data through flex array

---
 src/decode.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/decode.h b/src/decode.h
index dedfbb09efd0..7478134ceb5e 100644
--- a/src/decode.h
+++ b/src/decode.h
@@ -217,8 +217,8 @@ typedef struct Address_ {
 #define GET_TCP_DST_PORT(p)  ((p)->dp)
 
 #define GET_PKT_LEN(p) ((p)->pktlen)
-#define GET_PKT_DATA(p) ((((p)->ext_pkt) == NULL ) ? (uint8_t *)((p) + 1) : (p)->ext_pkt)
-#define GET_PKT_DIRECT_DATA(p) (uint8_t *)((p) + 1)
+#define GET_PKT_DATA(p)            ((((p)->ext_pkt) == NULL) ? GET_PKT_DIRECT_DATA(p) : (p)->ext_pkt)
+#define GET_PKT_DIRECT_DATA(p)     (p)->pkt_data
 #define GET_PKT_DIRECT_MAX_SIZE(p) (default_packet_size)
 
 #define SET_PKT_LEN(p, len) do { \
@@ -651,6 +651,11 @@ typedef struct Packet_
          */
         SCSpinlock tunnel_lock;
     } persistent;
+
+    /** flex array accessor to allocated packet data. Size of the additional
+     *  data is `default_packet_size`. If this is insufficient,
+     *  Packet::ext_pkt will be used instead. */
+    uint8_t pkt_data[];
 } Packet;
 
 /** highest mtu of the interfaces we monitor */

From 4ccc8293b1037e1b1fec5d30098964e7e09acfa3 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 25 Nov 2023 08:33:53 +0100
Subject: [PATCH 178/462] packet: minor macro cleanups

---
 src/decode.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/decode.h b/src/decode.h
index 7478134ceb5e..6392f3361e58 100644
--- a/src/decode.h
+++ b/src/decode.h
@@ -216,8 +216,8 @@ typedef struct Address_ {
 #define GET_TCP_SRC_PORT(p)  ((p)->sp)
 #define GET_TCP_DST_PORT(p)  ((p)->dp)
 
-#define GET_PKT_LEN(p) ((p)->pktlen)
-#define GET_PKT_DATA(p)            ((((p)->ext_pkt) == NULL) ? GET_PKT_DIRECT_DATA(p) : (p)->ext_pkt)
+#define GET_PKT_LEN(p)             (p)->pktlen
+#define GET_PKT_DATA(p)            (((p)->ext_pkt == NULL) ? GET_PKT_DIRECT_DATA(p) : (p)->ext_pkt)
 #define GET_PKT_DIRECT_DATA(p)     (p)->pkt_data
 #define GET_PKT_DIRECT_MAX_SIZE(p) (default_packet_size)
 

From ee66a7246fc2db41c53d1934e2f515dabf725bf4 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 23 Sep 2023 07:43:54 +0200
Subject: [PATCH 179/462] detect-engine: minor content inspection cleanup

---
 src/detect-engine.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 4cf145df6e2b..c078b824d033 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2257,10 +2257,6 @@ int DetectEngineInspectPktBufferGeneric(
         return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
     }
 
-    const uint32_t data_len = buffer->inspect_len;
-    const uint8_t *data = buffer->inspect;
-    const uint64_t offset = 0;
-
     uint8_t ci_flags = DETECT_CI_FLAGS_START|DETECT_CI_FLAGS_END;
     ci_flags |= buffer->flags;
 
@@ -2270,11 +2266,9 @@ int DetectEngineInspectPktBufferGeneric(
 
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    int r = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx,
-                                          s, engine->smd,
-                                          p, p->flow,
-                                          (uint8_t *)data, data_len, offset, ci_flags,
-                                          DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER);
+    int r = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p, p->flow,
+            buffer->inspect, buffer->inspect_len, 0, ci_flags,
+            DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER);
     if (r == 1) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {

From 368adf4599e896a2b1d21bfbdfd389ec10213bcd Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 23 Sep 2023 07:15:33 +0200
Subject: [PATCH 180/462] detect/file-data: simplify content inspect loop

---
 src/detect-file-data.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)

diff --git a/src/detect-file-data.c b/src/detect-file-data.c
index ec1069678f99..c57687027652 100644
--- a/src/detect-file-data.c
+++ b/src/detect-file-data.c
@@ -401,7 +401,6 @@ uint8_t DetectEngineInspectFiledata(DetectEngineCtx *de_ctx, DetectEngineThreadC
         return DETECT_ENGINE_INSPECT_SIG_CANT_MATCH_FILES;
     }
 
-    bool match = false;
     int local_file_id = 0;
     File *file = ffc->head;
     for (; file != NULL; file = file->next) {
@@ -418,22 +417,16 @@ uint8_t DetectEngineInspectFiledata(DetectEngineCtx *de_ctx, DetectEngineThreadC
         det_ctx->buffer_offset = 0;
         det_ctx->discontinue_matching = 0;
         det_ctx->inspection_recursion_counter = 0;
-        match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, ciflags,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset, ciflags,
+                DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
-            break;
+            return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_file_id++;
     }
 
-    if (match)
-        return DETECT_ENGINE_INSPECT_SIG_MATCH;
-    else
-        return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
+    return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
 }
 
 /** \brief Filedata Filedata Mpm prefilter callback

From 7f42506760d6c8345b2f565e535aad276c2441e7 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Sep 2023 21:08:29 +0200
Subject: [PATCH 181/462] detect: reimplement discontinue matching logic

Previously various steps in the content inspection logic would use
a variable in the DetectEngineThreadCtx to flag that matching should
be discontinued.

This patch reimplements this logic by using a new return code instead.

Split content inspection into public and private version, so that
common initialization can be done in a single place.

Update the callsites.
---
 src/detect-base64-data.c               |  8 +--
 src/detect-dns-query.c                 | 15 ++---
 src/detect-engine-content-inspection.c | 86 +++++++++++++++-----------
 src/detect-engine-content-inspection.h |  8 ++-
 src/detect-engine-frame.c              | 15 ++---
 src/detect-engine-payload.c            | 44 ++++---------
 src/detect-engine.c                    | 24 +++----
 src/detect-file-data.c                 |  3 -
 src/detect-filemagic.c                 | 14 ++---
 src/detect-filename.c                  | 14 ++---
 src/detect-http-client-body.c          | 11 ++--
 src/detect-http-header.c               | 31 +++-------
 src/detect-http2.c                     | 15 ++---
 src/detect-ike-vendor.c                |  8 +--
 src/detect-krb5-cname.c                | 16 ++---
 src/detect-krb5-sname.c                | 15 ++---
 src/detect-mqtt-subscribe-topic.c      | 15 ++---
 src/detect-mqtt-unsubscribe-topic.c    | 15 ++---
 src/detect-quic-cyu-hash.c             |  8 +--
 src/detect-quic-cyu-string.c           |  8 +--
 src/detect-template-rust-buffer.c      |  9 ++-
 src/detect-tls-certs.c                 | 14 ++---
 src/detect.h                           |  2 -
 23 files changed, 146 insertions(+), 252 deletions(-)

diff --git a/src/detect-base64-data.c b/src/detect-base64-data.c
index 4c892a919c26..09d89113d675 100644
--- a/src/detect-base64-data.c
+++ b/src/detect-base64-data.c
@@ -65,10 +65,10 @@ int DetectBase64DataDoMatch(DetectEngineCtx *de_ctx,
     DetectEngineThreadCtx *det_ctx, const Signature *s, Flow *f)
 {
     if (det_ctx->base64_decoded_len) {
-        return DetectEngineContentInspection(de_ctx, det_ctx, s,
-            s->sm_arrays[DETECT_SM_LIST_BASE64_DATA], NULL, f, det_ctx->base64_decoded,
-            det_ctx->base64_decoded_len, 0, DETECT_CI_FLAGS_SINGLE,
-            DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        return DetectEngineContentInspectionInternal(de_ctx, det_ctx, s,
+                s->sm_arrays[DETECT_SM_LIST_BASE64_DATA], NULL, f, det_ctx->base64_decoded,
+                det_ctx->base64_decoded_len, 0, DETECT_CI_FLAGS_SINGLE,
+                DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
     }
 
     return 0;
diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index fd2c7450853e..d2dbe8e99021 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -114,17 +114,10 @@ static uint8_t DetectEngineInspectDnsQuery(DetectEngineCtx *de_ctx, DetectEngine
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 8c5feb61a226..2ac2319c9d53 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -100,10 +100,11 @@
  *                        buffer inspection modes or dce inspection mode.
  * \param flags           DETECT_CI_FLAG_*
  *
+ *  \retval -1 no match and give up (discontinue matching)
  *  \retval 0 no match
  *  \retval 1 match
  */
-uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         uint32_t buffer_len, uint32_t stream_start_offset, uint8_t flags, uint8_t inspection_mode)
 {
@@ -113,9 +114,8 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
     det_ctx->inspection_recursion_counter++;
 
     if (det_ctx->inspection_recursion_counter == de_ctx->inspection_recursion_limit) {
-        det_ctx->discontinue_matching = 1;
         KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
-        SCReturnInt(0);
+        SCReturnInt(-1);
     }
 
     // we want the ability to match on bsize: 0
@@ -303,7 +303,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
                 if (!(cd->flags & DETECT_CONTENT_NEGATED)) {
                     if ((cd->flags & (DETECT_CONTENT_DISTANCE | DETECT_CONTENT_WITHIN)) == 0) {
                         /* independent match from previous matches, so failure is fatal */
-                        det_ctx->discontinue_matching = 1;
+                        goto no_match_discontinue;
                     }
 
                     goto no_match;
@@ -328,8 +328,9 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
                             goto match;
                         }
                     }
-                    if (DETECT_CONTENT_IS_SINGLE(cd))
-                        det_ctx->discontinue_matching = 1;
+                    if (DETECT_CONTENT_IS_SINGLE(cd)) {
+                        goto no_match_discontinue;
+                    }
                     goto no_match;
                 } else {
                     SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32 "", cd->id,
@@ -360,23 +361,21 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
                         /* see if the next buffer keywords match. If not, we will
                          * search for another occurrence of this content and see
                          * if the others match then until we run out of matches */
-                        uint8_t r = DetectEngineContentInspection(de_ctx, det_ctx, s, smd + 1, p, f,
-                                buffer, buffer_len, stream_start_offset, flags, inspection_mode);
+                        int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1,
+                                p, f, buffer, buffer_len, stream_start_offset, flags,
+                                inspection_mode);
                         if (r == 1) {
                             SCReturnInt(1);
-                        }
-                        SCLogDebug("no match for 'next sm'");
-
-                        if (det_ctx->discontinue_matching) {
+                        } else if (r == -1) {
                             SCLogDebug("'next sm' said to discontinue this right now");
-                            goto no_match;
+                            SCReturnInt(-1);
                         }
+                        SCLogDebug("no match for 'next sm'");
 
                         /* no match and no reason to look for another instance */
                         if ((cd->flags & DETECT_CONTENT_WITHIN_NEXT) == 0) {
                             SCLogDebug("'next sm' does not depend on me, so we can give up");
-                            det_ctx->discontinue_matching = 1;
-                            goto no_match;
+                            SCReturnInt(-1);
                         }
 
                         SCLogDebug("'next sm' depends on me %p, lets see what we can do (flags %u)",
@@ -441,12 +440,10 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         DetectPcreData *pe = (DetectPcreData *)smd->ctx;
         uint32_t prev_buffer_offset = det_ctx->buffer_offset;
         uint32_t prev_offset = 0;
-        int r = 0;
 
         det_ctx->pcre_match_start_offset = 0;
         do {
-            r = DetectPcrePayloadMatch(det_ctx, s, smd, p, f,
-                                       buffer, buffer_len);
+            int r = DetectPcrePayloadMatch(det_ctx, s, smd, p, f, buffer, buffer_len);
             if (r == 0) {
                 goto no_match;
             }
@@ -463,16 +460,14 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
             /* see if the next payload keywords match. If not, we will
              * search for another occurrence of this pcre and see
              * if the others match, until we run out of matches */
-            r = DetectEngineContentInspection(de_ctx, det_ctx, s, smd+1,
-                    p, f, buffer, buffer_len, stream_start_offset, flags,
-                    inspection_mode);
+            r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1, p, f, buffer,
+                    buffer_len, stream_start_offset, flags, inspection_mode);
             if (r == 1) {
                 SCReturnInt(1);
+            } else if (r == -1) {
+                SCReturnInt(-1);
             }
 
-            if (det_ctx->discontinue_matching)
-                goto no_match;
-
             det_ctx->buffer_offset = prev_buffer_offset;
             det_ctx->pcre_match_start_offset = prev_offset;
         } while (1);
@@ -611,9 +606,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         const uint64_t data_size = buffer_len + stream_start_offset;
         int r = DetectBsizeMatch(smd->ctx, data_size, eof);
         if (r < 0) {
-            det_ctx->discontinue_matching = 1;
-            goto no_match;
-
+            goto no_match_discontinue;
         } else if (r == 0) {
             goto no_match;
         }
@@ -627,8 +620,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         if (r == 1) {
             goto match;
         }
-        det_ctx->discontinue_matching = 1;
-        goto no_match;
+        goto no_match_discontinue;
 
     } else if (smd->type == DETECT_DATAREP) {
 
@@ -638,8 +630,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         if (r == 1) {
             goto match;
         }
-        det_ctx->discontinue_matching = 1;
-        goto no_match;
+        goto no_match_discontinue;
 
     } else if (smd->type == DETECT_AL_URILEN) {
         SCLogDebug("inspecting uri len");
@@ -655,10 +646,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         if (r == 1) {
             goto match;
         }
-
-        det_ctx->discontinue_matching = 1;
-
-        goto no_match;
+        goto no_match_discontinue;
 #ifdef HAVE_LUA
     }
     else if (smd->type == DETECT_LUA) {
@@ -677,7 +665,7 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
         if (DetectBase64DecodeDoMatch(det_ctx, s, smd, buffer, buffer_len)) {
             if (s->sm_arrays[DETECT_SM_LIST_BASE64_DATA] != NULL) {
                 KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-                if (DetectBase64DataDoMatch(de_ctx, det_ctx, s, f)) {
+                if (DetectBase64DataDoMatch(de_ctx, det_ctx, s, f) == 1) {
                     /* Base64 is a terminal list. */
                     goto final_match;
                 }
@@ -694,12 +682,16 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
     KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
     SCReturnInt(0);
 
+no_match_discontinue:
+    KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
+    SCReturnInt(-1);
+
 match:
     /* this sigmatch matched, inspect the next one. If it was the last,
      * the buffer portion of the signature matched. */
     if (!smd->is_last) {
         KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-        uint8_t r = DetectEngineContentInspection(de_ctx, det_ctx, s, smd + 1, p, f, buffer,
+        int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1, p, f, buffer,
                 buffer_len, stream_start_offset, flags, inspection_mode);
         SCReturnInt(r);
     }
@@ -708,6 +700,26 @@ uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThrea
     SCReturnInt(1);
 }
 
+/** \brief wrapper around DetectEngineContentInspectionInternal to return true/false only
+ *
+ *  \param smd sigmatches to evaluate
+ */
+bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
+        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
+        const uint8_t inspection_mode)
+{
+    det_ctx->buffer_offset = 0;
+    det_ctx->inspection_recursion_counter = 0;
+
+    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd, p, f, buffer, buffer_len,
+            stream_start_offset, flags, inspection_mode);
+    if (r == 1)
+        return true;
+    else
+        return false;
+}
+
 #ifdef UNITTESTS
 #include "tests/detect-engine-content-inspection.c"
 #endif
diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index ae1e8ed5bfa1..188ebef2d881 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -46,7 +46,13 @@ enum {
  *  inspection function contains both start and end of the data. */
 #define DETECT_CI_FLAGS_SINGLE  (DETECT_CI_FLAGS_START|DETECT_CI_FLAGS_END)
 
-uint8_t DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+/* "internal" returns 1 match, 0 no match, -1 can't match */
+int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
+        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
+        const uint8_t inspection_mode);
+/* implicit "public" just returns true match, false no match */
+bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         uint32_t buffer_len, uint32_t stream_start_offset, uint8_t flags, uint8_t inspection_mode);
 
diff --git a/src/detect-engine-frame.c b/src/detect-engine-frame.c
index 722263d45390..0ed70757d599 100644
--- a/src/detect-engine-frame.c
+++ b/src/detect-engine-frame.c
@@ -311,10 +311,10 @@ static int DetectFrameInspectUdp(DetectEngineThreadCtx *det_ctx,
 
     // PrintRawDataFp(stdout, data, data_len);
 
-    int r = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p, p->flow,
-            (uint8_t *)data, data_len, 0, buffer->flags,
+    const bool match = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p,
+            p->flow, (uint8_t *)data, data_len, 0, buffer->flags,
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME);
-    if (r == 1) {
+    if (match) {
         SCLogDebug("match!");
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {
@@ -457,9 +457,6 @@ static int FrameStreamDataInspectFunc(
     const uint8_t *data = buffer->inspect;
     const uint64_t data_offset = buffer->inspect_offset;
     DetectEngineThreadCtx *det_ctx = fsd->det_ctx;
-    det_ctx->discontinue_matching = 0;
-    det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
 
     const DetectEngineFrameInspectionEngine *engine = fsd->inspect_engine;
     const Signature *s = fsd->s;
@@ -481,10 +478,10 @@ static int FrameStreamDataInspectFunc(
 #endif
     BUG_ON(fsd->frame->len > 0 && (int64_t)data_len > fsd->frame->len);
 
-    int r = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p, p->flow,
-            (uint8_t *)data, data_len, data_offset, buffer->flags,
+    const bool match = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p,
+            p->flow, (uint8_t *)data, data_len, data_offset, buffer->flags,
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME);
-    if (r == 1) {
+    if (match) {
         SCLogDebug("DETECT_ENGINE_INSPECT_SIG_MATCH");
         fsd->inspect_result = DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {
diff --git a/src/detect-engine-payload.c b/src/detect-engine-payload.c
index ef92e68629f8..7da3c3b81f93 100644
--- a/src/detect-engine-payload.c
+++ b/src/detect-engine-payload.c
@@ -153,7 +153,6 @@ uint8_t DetectEngineInspectPacketPayload(DetectEngineCtx *de_ctx, DetectEngineTh
         const Signature *s, Flow *f, Packet *p)
 {
     SCEnter();
-    int r = 0;
 
     if (s->sm_arrays[DETECT_SM_LIST_PMATCH] == NULL) {
         SCReturnInt(0);
@@ -162,16 +161,12 @@ uint8_t DetectEngineInspectPacketPayload(DetectEngineCtx *de_ctx, DetectEngineTh
     det_ctx->payload_persig_cnt++;
     det_ctx->payload_persig_size += p->payload_len;
 #endif
-    det_ctx->buffer_offset = 0;
-    det_ctx->discontinue_matching = 0;
-    det_ctx->inspection_recursion_counter = 0;
     det_ctx->replist = NULL;
 
-    r = DetectEngineContentInspection(de_ctx, det_ctx,
-            s, s->sm_arrays[DETECT_SM_LIST_PMATCH],
-            p, f, p->payload, p->payload_len, 0,
+    const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s,
+            s->sm_arrays[DETECT_SM_LIST_PMATCH], p, f, p->payload, p->payload_len, 0,
             DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);
-    if (r == 1) {
+    if (match) {
         SCReturnInt(1);
     }
     SCReturnInt(0);
@@ -195,7 +190,6 @@ static uint8_t DetectEngineInspectStreamUDPPayload(DetectEngineCtx *de_ctx,
         Packet *p)
 {
     SCEnter();
-    int r = 0;
 
     if (smd == NULL) {
         SCReturnInt(0);
@@ -204,15 +198,12 @@ static uint8_t DetectEngineInspectStreamUDPPayload(DetectEngineCtx *de_ctx,
     det_ctx->payload_persig_cnt++;
     det_ctx->payload_persig_size += p->payload_len;
 #endif
-    det_ctx->buffer_offset = 0;
-    det_ctx->discontinue_matching = 0;
-    det_ctx->inspection_recursion_counter = 0;
     det_ctx->replist = NULL;
 
-    r = DetectEngineContentInspection(de_ctx, det_ctx, s, smd,
-            p, f, p->payload, p->payload_len, 0, DETECT_CI_FLAGS_SINGLE,
-            DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);
-    if (r == 1) {
+    const bool match =
+            DetectEngineContentInspection(de_ctx, det_ctx, s, smd, p, f, p->payload, p->payload_len,
+                    0, DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);
+    if (match) {
         SCReturnInt(1);
     }
     SCReturnInt(0);
@@ -229,21 +220,17 @@ static int StreamContentInspectFunc(
         void *cb_data, const uint8_t *data, const uint32_t data_len, const uint64_t _offset)
 {
     SCEnter();
-    int r = 0;
     struct StreamContentInspectData *smd = cb_data;
 #ifdef DEBUG
     smd->det_ctx->stream_persig_cnt++;
     smd->det_ctx->stream_persig_size += data_len;
 #endif
-    smd->det_ctx->buffer_offset = 0;
-    smd->det_ctx->discontinue_matching = 0;
-    smd->det_ctx->inspection_recursion_counter = 0;
 
-    r = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx,
-            smd->s, smd->s->sm_arrays[DETECT_SM_LIST_PMATCH],
-            NULL, smd->f, (uint8_t *)data, data_len, 0, 0, //TODO
+    const bool match = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx, smd->s,
+            smd->s->sm_arrays[DETECT_SM_LIST_PMATCH], NULL, smd->f, (uint8_t *)data, data_len, 0,
+            0, // TODO
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM);
-    if (r == 1) {
+    if (match) {
         SCReturnInt(1);
     }
 
@@ -288,21 +275,16 @@ static int StreamContentInspectEngineFunc(
         void *cb_data, const uint8_t *data, const uint32_t data_len, const uint64_t _offset)
 {
     SCEnter();
-    int r = 0;
     struct StreamContentInspectEngineData *smd = cb_data;
 #ifdef DEBUG
     smd->det_ctx->stream_persig_cnt++;
     smd->det_ctx->stream_persig_size += data_len;
 #endif
-    smd->det_ctx->buffer_offset = 0;
-    smd->det_ctx->discontinue_matching = 0;
-    smd->det_ctx->inspection_recursion_counter = 0;
 
-    r = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx,
-            smd->s, smd->smd,
+    const bool match = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx, smd->s, smd->smd,
             NULL, smd->f, (uint8_t *)data, data_len, 0, 0, // TODO
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM);
-    if (r == 1) {
+    if (match) {
         SCReturnInt(1);
     }
 
diff --git a/src/detect-engine.c b/src/detect-engine.c
index c078b824d033..c4f630699a70 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2204,18 +2204,12 @@ uint8_t DetectEngineInspectBufferGeneric(DetectEngineCtx *de_ctx, DetectEngineTh
     ci_flags |= (offset == 0 ? DETECT_CI_FLAGS_START : 0);
     ci_flags |= buffer->flags;
 
-    det_ctx->discontinue_matching = 0;
-    det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
-
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    int r = DetectEngineContentInspection(de_ctx, det_ctx,
-                                          s, engine->smd,
-                                          NULL, f,
-                                          (uint8_t *)data, data_len, offset, ci_flags,
-                                          DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-    if (r == 1) {
+    const bool match =
+            DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, (uint8_t *)data,
+                    data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+    if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {
         return eof ? DETECT_ENGINE_INSPECT_SIG_CANT_MATCH :
@@ -2260,16 +2254,12 @@ int DetectEngineInspectPktBufferGeneric(
     uint8_t ci_flags = DETECT_CI_FLAGS_START|DETECT_CI_FLAGS_END;
     ci_flags |= buffer->flags;
 
-    det_ctx->discontinue_matching = 0;
-    det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
-
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    int r = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p, p->flow,
-            buffer->inspect, buffer->inspect_len, 0, ci_flags,
+    const bool match = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p,
+            p->flow, buffer->inspect, buffer->inspect_len, 0, ci_flags,
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER);
-    if (r == 1) {
+    if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {
         return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
diff --git a/src/detect-file-data.c b/src/detect-file-data.c
index c57687027652..f31715adda01 100644
--- a/src/detect-file-data.c
+++ b/src/detect-file-data.c
@@ -414,9 +414,6 @@ uint8_t DetectEngineInspectFiledata(DetectEngineCtx *de_ctx, DetectEngineThreadC
         if (buffer->inspect_offset == 0)
             ciflags |= DETECT_CI_FLAGS_START;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset, ciflags,
                 DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
diff --git a/src/detect-filemagic.c b/src/detect-filemagic.c
index d816b8c53dfe..7ade159fb52d 100644
--- a/src/detect-filemagic.c
+++ b/src/detect-filemagic.c
@@ -320,16 +320,10 @@ static uint8_t DetectEngineInspectFilemagic(DetectEngineCtx *de_ctx, DetectEngin
         if (buffer == NULL)
             continue;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-        int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         } else {
             r = DETECT_ENGINE_INSPECT_SIG_CANT_MATCH_FILES;
diff --git a/src/detect-filename.c b/src/detect-filename.c
index 5eb446af5134..88e580862452 100644
--- a/src/detect-filename.c
+++ b/src/detect-filename.c
@@ -257,16 +257,10 @@ static uint8_t DetectEngineInspectFilename(DetectEngineCtx *de_ctx, DetectEngine
         if (buffer == NULL)
             continue;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-        int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         } else {
             r = DETECT_ENGINE_INSPECT_SIG_CANT_MATCH_FILES;
diff --git a/src/detect-http-client-body.c b/src/detect-http-client-body.c
index 32c407a00aeb..1d3d7a87cc88 100644
--- a/src/detect-http-client-body.c
+++ b/src/detect-http-client-body.c
@@ -323,15 +323,12 @@ static uint8_t DetectEngineInspectBufferHttpBody(DetectEngineCtx *de_ctx,
     ci_flags |= (offset == 0 ? DETECT_CI_FLAGS_START : 0);
     ci_flags |= buffer->flags;
 
-    det_ctx->discontinue_matching = 0;
-    det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
-
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    int r = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, (uint8_t *)data,
-            data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-    if (r == 1) {
+    const bool match =
+            DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, (uint8_t *)data,
+                    data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+    if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     }
 
diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index 9d4b187a9f25..a4596c4085f2 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -199,17 +199,12 @@ static uint8_t DetectEngineInspectBufferHttpHeader(DetectEngineCtx *de_ctx,
     const uint8_t *data = buffer->inspect;
     const uint64_t offset = buffer->inspect_offset;
 
-    det_ctx->discontinue_matching = 0;
-    det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
-
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    int r = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-            NULL, f, (uint8_t *)data, data_len, offset,
-            DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-    SCLogDebug("r = %d", r);
-    if (r == 1) {
+    const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+            (uint8_t *)data, data_len, offset, DETECT_CI_FLAGS_SINGLE,
+            DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+    if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     }
 end:
@@ -546,18 +541,13 @@ static uint8_t DetectEngineInspectHttp2Header(DetectEngineCtx *de_ctx,
         };
         InspectionBuffer *buffer =
                 GetHttp2HeaderData(det_ctx, flags, transforms, f, &cbdata, engine->sm_list);
-
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
@@ -698,18 +688,13 @@ static uint8_t DetectEngineInspectHttp1Header(DetectEngineCtx *de_ctx,
         };
         InspectionBuffer *buffer =
                 GetHttp1HeaderData(det_ctx, flags, transforms, f, &cbdata, engine->sm_list);
-
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-http2.c b/src/detect-http2.c
index 9991b85ad9ac..a1ede963825e 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -702,17 +702,10 @@ static uint8_t DetectEngineInspectHttp2HeaderName(DetectEngineCtx *de_ctx,
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-ike-vendor.c b/src/detect-ike-vendor.c
index 1af41bac23d8..f5c5b94f35d5 100644
--- a/src/detect-ike-vendor.c
+++ b/src/detect-ike-vendor.c
@@ -155,14 +155,10 @@ static uint8_t DetectEngineInspectIkeVendor(DetectEngineCtx *de_ctx, DetectEngin
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-krb5-cname.c b/src/detect-krb5-cname.c
index 632df0ea5dd8..8664f2bc2877 100644
--- a/src/detect-krb5-cname.c
+++ b/src/detect-krb5-cname.c
@@ -100,21 +100,13 @@ static uint8_t DetectEngineInspectKrb5CName(DetectEngineCtx *de_ctx, DetectEngin
         struct Krb5PrincipalNameDataArgs cbdata = { local_id, txv, };
         InspectionBuffer *buffer =
                 GetKrb5CNameData(det_ctx, transforms, f, &cbdata, engine->sm_list);
-
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index 19d3c6716116..1e4ae24a4bd1 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -104,17 +104,10 @@ static uint8_t DetectEngineInspectKrb5SName(DetectEngineCtx *de_ctx, DetectEngin
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-mqtt-subscribe-topic.c b/src/detect-mqtt-subscribe-topic.c
index 258dc0b4cf6d..9eaf39d3029c 100644
--- a/src/detect-mqtt-subscribe-topic.c
+++ b/src/detect-mqtt-subscribe-topic.c
@@ -107,17 +107,10 @@ static uint8_t DetectEngineInspectMQTTSubscribeTopic(DetectEngineCtx *de_ctx,
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-mqtt-unsubscribe-topic.c b/src/detect-mqtt-unsubscribe-topic.c
index 2c1cb02c4234..268d72bc8789 100644
--- a/src/detect-mqtt-unsubscribe-topic.c
+++ b/src/detect-mqtt-unsubscribe-topic.c
@@ -107,17 +107,10 @@ static uint8_t DetectEngineInspectMQTTUnsubscribeTopic(DetectEngineCtx *de_ctx,
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f,
-                                              (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-quic-cyu-hash.c b/src/detect-quic-cyu-hash.c
index a475a23f1e5e..88197a5e382a 100644
--- a/src/detect-quic-cyu-hash.c
+++ b/src/detect-quic-cyu-hash.c
@@ -106,14 +106,10 @@ static uint8_t DetectEngineInspectQuicHash(DetectEngineCtx *de_ctx, DetectEngine
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-quic-cyu-string.c b/src/detect-quic-cyu-string.c
index 53775d0ffc20..9290fa41233c 100644
--- a/src/detect-quic-cyu-string.c
+++ b/src/detect-quic-cyu-string.c
@@ -104,14 +104,10 @@ static uint8_t DetectEngineInspectQuicString(DetectEngineCtx *de_ctx,
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
         local_id++;
diff --git a/src/detect-template-rust-buffer.c b/src/detect-template-rust-buffer.c
index 86fc282712ba..f1c8c97bb278 100644
--- a/src/detect-template-rust-buffer.c
+++ b/src/detect-template-rust-buffer.c
@@ -91,7 +91,7 @@ static uint8_t DetectEngineInspectTemplateRustBuffer(DetectEngineCtx *de_ctx,
         DetectEngineThreadCtx *det_ctx, const struct DetectEngineAppInspectionEngine_ *engine,
         const Signature *s, Flow *f, uint8_t flags, void *alstate, void *txv, uint64_t tx_id)
 {
-    uint8_t ret = 0;
+    uint8_t ret = DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
     const uint8_t *data = NULL;
     uint32_t data_len = 0;
 
@@ -102,12 +102,15 @@ static uint8_t DetectEngineInspectTemplateRustBuffer(DetectEngineCtx *de_ctx,
     }
 
     if (data != NULL) {
-        ret = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
                 (uint8_t *)data, data_len, 0, DETECT_CI_FLAGS_SINGLE,
                 DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
+            ret = DETECT_ENGINE_INSPECT_SIG_MATCH;
+        }
     }
 
-    SCLogNotice("Returning %d.", ret);
+    SCLogNotice("Returning %u.", ret);
     return ret;
 }
 
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index e994c9e2b0e8..9ff185c494d6 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -194,16 +194,10 @@ static uint8_t DetectEngineInspectTlsCerts(DetectEngineCtx *de_ctx, DetectEngine
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        det_ctx->buffer_offset = 0;
-        det_ctx->discontinue_matching = 0;
-        det_ctx->inspection_recursion_counter = 0;
-
-        const int match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd,
-                                              NULL, f, (uint8_t *)buffer->inspect,
-                                              buffer->inspect_len,
-                                              buffer->inspect_offset, DETECT_CI_FLAGS_SINGLE,
-                                              DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-        if (match == 1) {
+        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
+                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }
 
diff --git a/src/detect.h b/src/detect.h
index a3cd161fa654..cdc098368fc4 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1141,8 +1141,6 @@ typedef struct DetectEngineThreadCtx_ {
         uint32_t *to_clear_queue;
     } multi_inspect;
 
-    /* used to discontinue any more matching */
-    uint16_t discontinue_matching;
     uint16_t flags; /**< DETECT_ENGINE_THREAD_CTX_* flags */
 
     /* true if tx_id is set */

From cc0adaaf4a747fbefa5e8d5afec73760cbb8b2f9 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 30 Nov 2023 11:46:14 -0600
Subject: [PATCH 182/462] userguide: remove old css files

In our conf.py we reference some ReadTheDocs stylesheets that appear to
be old and break formatting of some items like bulletted lists.

Bug: #6589
---
 doc/userguide/conf.py | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

diff --git a/doc/userguide/conf.py b/doc/userguide/conf.py
index cf87f19c311e..d043a288cd3f 100644
--- a/doc/userguide/conf.py
+++ b/doc/userguide/conf.py
@@ -137,20 +137,15 @@
         html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
     except:
         html_theme = 'default'
-    def setup(app):
-        if hasattr(app, 'add_css_file'):
-            app.add_css_file('css/suricata.css')
-        else:
-            app.add_stylesheet('css/suricata.css')
 else:
     html_theme = 'sphinx_rtd_theme'
-    html_context = {
-        'css_files': [
-            'https://media.readthedocs.org/css/sphinx_rtd_theme.css',
-            'https://media.readthedocs.org/css/readthedocs-doc-embed.css',
-            '_static/css/suricata.css',
-        ],
-    }
+
+# Add in our own stylesheet.
+def setup(app):
+    if hasattr(app, 'add_css_file'):
+        app.add_css_file('css/suricata.css')
+    else:
+        app.add_stylesheet('css/suricata.css')
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

From ee7c1400289bc35699cab9fa3fe52c8b7c70005c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 12:19:59 +0100
Subject: [PATCH 183/462] detect: minor cleanup

MPM_HS does not need a guard.
---
 src/detect-engine.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index c4f630699a70..b197369f100a 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2728,10 +2728,7 @@ static int DetectEngineCtxLoadConf(DetectEngineCtx *de_ctx)
         /* for now, since we still haven't implemented any intelligence into
          * understanding the patterns and distributing mpm_ctx across sgh */
         if (de_ctx->mpm_matcher == MPM_AC || de_ctx->mpm_matcher == MPM_AC_KS ||
-#ifdef BUILD_HYPERSCAN
-            de_ctx->mpm_matcher == MPM_HS ||
-#endif
-            de_ctx->mpm_matcher == MPM_AC_BS) {
+                de_ctx->mpm_matcher == MPM_HS || de_ctx->mpm_matcher == MPM_AC_BS) {
             de_ctx->sgh_mpm_ctx_cnf = ENGINE_SGH_MPM_FACTORY_CONTEXT_SINGLE;
         } else {
             de_ctx->sgh_mpm_ctx_cnf = ENGINE_SGH_MPM_FACTORY_CONTEXT_FULL;

From 6a73b3c90b8c083a908de97dd890440f2052e679 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 12:01:19 +0100
Subject: [PATCH 184/462] mpm: remove ac-bs implementation

Ticket: #6586.
---
 src/Makefile.am      |    2 -
 src/detect-engine.c  |    2 +-
 src/util-mpm-ac-bs.c | 2373 ------------------------------------------
 src/util-mpm-ac-bs.h |   74 --
 src/util-mpm.c       |    2 -
 src/util-mpm.h       |    1 -
 6 files changed, 1 insertion(+), 2453 deletions(-)
 delete mode 100644 src/util-mpm-ac-bs.c
 delete mode 100644 src/util-mpm-ac-bs.h

diff --git a/src/Makefile.am b/src/Makefile.am
index 21e1dfe5fbeb..4695c2d35f51 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -578,7 +578,6 @@ noinst_HEADERS = \
 	util-mem.h \
 	util-memrchr.h \
 	util-misc.h \
-	util-mpm-ac-bs.h \
 	util-mpm-ac.h \
 	util-mpm-ac-ks.h \
 	util-mpm.h \
@@ -1177,7 +1176,6 @@ libsuricata_c_a_SOURCES = \
 	util-memcmp.c \
 	util-memrchr.c \
 	util-misc.c \
-	util-mpm-ac-bs.c \
 	util-mpm-ac.c \
 	util-mpm-ac-ks.c \
 	util-mpm-ac-ks-small.c \
diff --git a/src/detect-engine.c b/src/detect-engine.c
index b197369f100a..a4ce2126544d 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2728,7 +2728,7 @@ static int DetectEngineCtxLoadConf(DetectEngineCtx *de_ctx)
         /* for now, since we still haven't implemented any intelligence into
          * understanding the patterns and distributing mpm_ctx across sgh */
         if (de_ctx->mpm_matcher == MPM_AC || de_ctx->mpm_matcher == MPM_AC_KS ||
-                de_ctx->mpm_matcher == MPM_HS || de_ctx->mpm_matcher == MPM_AC_BS) {
+                de_ctx->mpm_matcher == MPM_HS) {
             de_ctx->sgh_mpm_ctx_cnf = ENGINE_SGH_MPM_FACTORY_CONTEXT_SINGLE;
         } else {
             de_ctx->sgh_mpm_ctx_cnf = ENGINE_SGH_MPM_FACTORY_CONTEXT_FULL;
diff --git a/src/util-mpm-ac-bs.c b/src/util-mpm-ac-bs.c
deleted file mode 100644
index 72d2065ca7c1..000000000000
--- a/src/util-mpm-ac-bs.c
+++ /dev/null
@@ -1,2373 +0,0 @@
-/* Copyright (C) 2007-2014 Open Information Security Foundation
- *
- * You can copy, redistribute or modify this Program under the terms of
- * the GNU General Public License version 2 as published by the Free
- * Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * version 2 along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
- * 02110-1301, USA.
- */
-
-/**
- * \file
- *
- * \author Anoop Saldanha <anoopsaldanha@gmail.com>
- *
- *         First iteration of aho-corasick MPM from -
- *
- *         Efficient String Matching: An Aid to Bibliographic Search
- *         Alfred V. Aho and Margaret J. Corasick
- *
- *         - Uses the delta table for calculating transitions, instead of having
- *           separate goto and failure transitions.
- *         - If we cross 2 ** 16 states, we use 4 bytes in the transition table
- *           to hold each state, otherwise we use 2 bytes.
- *         - This version of the MPM is heavy on memory, but it performs well.
- *           If you can fit the ruleset with this mpm on your box without hitting
- *           swap, this is the MPM to go for.
- *
- * \todo - Do a proper analysis of our existing MPMs and suggest a good one based
- *         on the pattern distribution and the expected traffic(say http).
- *       - Tried out loop unrolling without any perf increase.  Need to dig deeper.
- *       - Irrespective of whether we cross 2 ** 16 states or not,shift to using
- *         uint32_t for state type, so that we can integrate it's status as a
- *         final state or not in the topmost byte.  We are already doing it if
- *         state_count is > 2 ** 16.
- *       - Test case-sensitive patterns if they have any ascii chars.  If they
- *         don't treat them as nocase.
- *       - Carry out other optimizations we are working on.  hashes, compression.
- */
-
-#include "suricata-common.h"
-#include "suricata.h"
-
-#include "detect.h"
-#include "detect-parse.h"
-#include "detect-engine.h"
-#include "detect-engine-build.h"
-#include "util-mpm-ac-bs.h"
-
-#include "conf.h"
-#include "util-debug.h"
-#include "util-unittest.h"
-#include "util-unittest-helper.h"
-#include "util-memcmp.h"
-#include "util-memcpy.h"
-#include "util-validate.h"
-
-void SCACBSInitCtx(MpmCtx *);
-void SCACBSDestroyCtx(MpmCtx *);
-int SCACBSAddPatternCI(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
-                       uint32_t, SigIntId, uint8_t);
-int SCACBSAddPatternCS(MpmCtx *, uint8_t *, uint16_t, uint16_t, uint16_t,
-                       uint32_t, SigIntId, uint8_t);
-int SCACBSPreparePatterns(MpmCtx *mpm_ctx);
-uint32_t SCACBSSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
-                      PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen);
-void SCACBSPrintInfo(MpmCtx *mpm_ctx);
-void SCACBSRegisterTests(void);
-
-/* a placeholder to denote a failure transition in the goto table */
-#define SC_AC_BS_FAIL (-1)
-
-#define STATE_QUEUE_CONTAINER_SIZE 65536
-
-/**
- * \brief Helper structure used by AC during state table creation
- */
-typedef struct StateQueue_ {
-    int32_t store[STATE_QUEUE_CONTAINER_SIZE];
-    int top;
-    int bot;
-} StateQueue;
-
-/**
- * \brief Register the aho-corasick mpm.
- */
-void MpmACBSRegister(void)
-{
-    mpm_table[MPM_AC_BS].name = "ac-bs";
-    mpm_table[MPM_AC_BS].InitCtx = SCACBSInitCtx;
-    mpm_table[MPM_AC_BS].DestroyCtx = SCACBSDestroyCtx;
-    mpm_table[MPM_AC_BS].AddPattern = SCACBSAddPatternCS;
-    mpm_table[MPM_AC_BS].AddPatternNocase = SCACBSAddPatternCI;
-    mpm_table[MPM_AC_BS].Prepare = SCACBSPreparePatterns;
-    mpm_table[MPM_AC_BS].Search = SCACBSSearch;
-    mpm_table[MPM_AC_BS].PrintCtx = SCACBSPrintInfo;
-    mpm_table[MPM_AC_BS].RegisterUnittests = SCACBSRegisterTests;
-}
-
-/**
- * \internal
- * \brief Initialize the AC context with user specified conf parameters.  We
- *        aren't retrieving anything for AC conf now, but we will certainly
- *        need it, when we customize AC.
- */
-static void SCACBSGetConfig(void)
-{
-    //ConfNode *ac_conf;
-    //const char *hash_val = NULL;
-
-    //ConfNode *pm = ConfGetNode("pattern-matcher");
-
-    return;
-}
-
-/**
- * \internal
- * \brief Initialize a new state in the goto and output tables.
- *
- * \param mpm_ctx Pointer to the mpm context.
- *
- * \retval The state id, of the newly created state.
- */
-static inline int SCACBSInitNewState(MpmCtx *mpm_ctx)
-{
-    void *ptmp;
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int ascii_code = 0;
-    int size = 0;
-
-    /* reallocate space in the goto table to include a new state */
-    size = (ctx->state_count + 1) * ctx->single_state_size;
-    ptmp = SCRealloc(ctx->goto_table, size);
-    if (ptmp == NULL) {
-        SCFree(ctx->goto_table);
-        ctx->goto_table = NULL;
-        FatalError("Error allocating memory");
-    }
-    ctx->goto_table = ptmp;
-
-    /* set all transitions for the newly assigned state as FAIL transitions */
-    for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-        ctx->goto_table[ctx->state_count][ascii_code] = SC_AC_BS_FAIL;
-    }
-
-    /* reallocate space in the output table for the new state */
-    size = (ctx->state_count + 1) * sizeof(SCACBSOutputTable);
-    ptmp = SCRealloc(ctx->output_table, size);
-    if (ptmp == NULL) {
-        SCFree(ctx->output_table);
-        ctx->output_table = NULL;
-        FatalError("Error allocating memory");
-    }
-    ctx->output_table = ptmp;
-
-    memset(ctx->output_table + ctx->state_count, 0, sizeof(SCACBSOutputTable));
-
-    /* \todo using it temporarily now during dev, since I have restricted
-     *       state var in SCACBSCtx->state_table to uint16_t. */
-    //if (ctx->state_count > 65536) {
-    //    printf("state count exceeded\n");
-    //    exit(EXIT_FAILURE);
-    //}
-
-    return ctx->state_count++;
-}
-
-/**
- * \internal
- * \brief Adds a pid to the output table for a state.
- *
- * \param state   The state to whose output table we should add the pid.
- * \param pid     The pattern id to add.
- * \param mpm_ctx Pointer to the mpm context.
- */
-static void SCACBSSetOutputState(int32_t state, uint32_t pid, MpmCtx *mpm_ctx)
-{
-    void *ptmp;
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    SCACBSOutputTable *output_state = &ctx->output_table[state];
-    uint32_t i = 0;
-
-    for (i = 0; i < output_state->no_of_entries; i++) {
-        if (output_state->pids[i] == pid)
-            return;
-    }
-
-    output_state->no_of_entries++;
-    ptmp = SCRealloc(output_state->pids,
-                     output_state->no_of_entries * sizeof(uint32_t));
-    if (ptmp == NULL) {
-        SCFree(output_state->pids);
-        output_state->pids = NULL;
-        FatalError("Error allocating memory");
-    }
-    output_state->pids = ptmp;
-
-    output_state->pids[output_state->no_of_entries - 1] = pid;
-
-    return;
-}
-
-/**
- * \brief Helper function used by SCACBSCreateGotoTable.  Adds a pattern to the
- *        goto table.
- *
- * \param pattern     Pointer to the pattern.
- * \param pattern_len Pattern length.
- * \param pid         The pattern id, that corresponds to this pattern.  We
- *                    need it to updated the output table for this pattern.
- * \param mpm_ctx     Pointer to the mpm context.
- */
-static inline void SCACBSEnter(uint8_t *pattern, uint16_t pattern_len, uint32_t pid,
-                               MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int32_t state = 0;
-    int32_t newstate = 0;
-    int i = 0;
-    int p = 0;
-
-    /* walk down the trie till we have a match for the pattern prefix */
-    state = 0;
-    for (i = 0; i < pattern_len; i++) {
-        if (ctx->goto_table[state][pattern[i]] != SC_AC_BS_FAIL) {
-            state = ctx->goto_table[state][pattern[i]];
-        } else {
-            break;
-        }
-    }
-
-    /* add the non-matching pattern suffix to the trie, from the last state
-     * we left off */
-    for (p = i; p < pattern_len; p++) {
-        newstate = SCACBSInitNewState(mpm_ctx);
-        ctx->goto_table[state][pattern[p]] = newstate;
-        state = newstate;
-    }
-
-    /* add this pattern id, to the output table of the last state, where the
-     * pattern ends in the trie */
-    SCACBSSetOutputState(state, pid, mpm_ctx);
-
-    return;
-}
-
-/**
- * \internal
- * \brief Create the goto table.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSCreateGotoTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    uint32_t i = 0;
-
-    /* add each pattern to create the goto table */
-    for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
-        SCACBSEnter(ctx->parray[i]->ci, ctx->parray[i]->len,
-                  ctx->parray[i]->id, mpm_ctx);
-    }
-
-    int ascii_code = 0;
-    for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-        if (ctx->goto_table[0][ascii_code] == SC_AC_BS_FAIL) {
-            ctx->goto_table[0][ascii_code] = 0;
-        }
-    }
-
-    return;
-}
-
-static inline int SCACBSStateQueueIsEmpty(StateQueue *q)
-{
-    if (q->top == q->bot)
-        return 1;
-    else
-        return 0;
-}
-
-static inline void SCACBSEnqueue(StateQueue *q, int32_t state)
-{
-    int i = 0;
-
-    /*if we already have this */
-    for (i = q->bot; i < q->top; i++) {
-        if (q->store[i] == state)
-            return;
-    }
-
-    q->store[q->top++] = state;
-
-    if (q->top == STATE_QUEUE_CONTAINER_SIZE)
-        q->top = 0;
-
-    if (q->top == q->bot) {
-        FatalError("Just ran out of space in the queue.  "
-                   "Fatal Error.  Exiting.  Please file a bug report on this");
-    }
-
-    return;
-}
-
-static inline int32_t SCACBSDequeue(StateQueue *q)
-{
-    if (q->bot == STATE_QUEUE_CONTAINER_SIZE)
-        q->bot = 0;
-
-    if (q->bot == q->top) {
-        FatalError("StateQueue behaving weirdly.  "
-                   "Fatal Error.  Exiting.  Please file a bug report on this");
-    }
-
-    return q->store[q->bot++];
-}
-
-/*
-#define SCACBSStateQueueIsEmpty(q) (((q)->top == (q)->bot) ? 1 : 0)
-
-#define SCACBSEnqueue(q, state) do { \
-                                  int i = 0; \
-                                             \
-                                  for (i = (q)->bot; i < (q)->top; i++) { \
-                                      if ((q)->store[i] == state)       \
-                                      return; \
-                                  } \
-                                    \
-                                  (q)->store[(q)->top++] = state;   \
-                                                                \
-                                  if ((q)->top == STATE_QUEUE_CONTAINER_SIZE) \
-                                      (q)->top = 0;                     \
-                                                                        \
-                                  if ((q)->top == (q)->bot) {           \
-                                  FatalError("Just ran out of space in the queue.  " \
-                                                "Fatal Error.  Exiting.  Please file a bug report on
-this"); \
-                                  }                                     \
-                              } while (0)
-
-#define SCACBSDequeue(q) ( (((q)->bot == STATE_QUEUE_CONTAINER_SIZE)? ((q)->bot = 0): 0), \
-                         (((q)->bot == (q)->top) ?                      \
-                          (printf("StateQueue behaving "                \
-                                         "weirdly.  Fatal Error.  Exiting.  Please " \
-                                         "file a bug report on this"), \
-                           exit(EXIT_FAILURE)) : 0), \
-                         (q)->store[(q)->bot++])     \
-*/
-
-/**
- * \internal
- * \brief Club the output data from 2 states and store it in the 1st state.
- *        dst_state_data = {dst_state_data} UNION {src_state_data}
- *
- * \param dst_state First state(also the destination) for the union operation.
- * \param src_state Second state for the union operation.
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSClubOutputStates(int32_t dst_state, int32_t src_state,
-                                        MpmCtx *mpm_ctx)
-{
-    void *ptmp;
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    uint32_t i = 0;
-    uint32_t j = 0;
-
-    SCACBSOutputTable *output_dst_state = &ctx->output_table[dst_state];
-    SCACBSOutputTable *output_src_state = &ctx->output_table[src_state];
-
-    for (i = 0; i < output_src_state->no_of_entries; i++) {
-        for (j = 0; j < output_dst_state->no_of_entries; j++) {
-            if (output_src_state->pids[i] == output_dst_state->pids[j]) {
-                break;
-            }
-        }
-        if (j == output_dst_state->no_of_entries) {
-            output_dst_state->no_of_entries++;
-
-            ptmp = SCRealloc(output_dst_state->pids,
-                             (output_dst_state->no_of_entries * sizeof(uint32_t)));
-            if (ptmp == NULL) {
-                SCFree(output_dst_state->pids);
-                output_dst_state->pids = NULL;
-                FatalError("Error allocating memory");
-            }
-            else {
-                output_dst_state->pids = ptmp;
-            }
-
-            output_dst_state->pids[output_dst_state->no_of_entries - 1] =
-                output_src_state->pids[i];
-        }
-    }
-
-    return;
-}
-
-/**
- * \internal
- * \brief Create the failure table.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSCreateFailureTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int ascii_code = 0;
-    int32_t state = 0;
-    int32_t r_state = 0;
-
-    StateQueue q;
-    memset(&q, 0, sizeof(StateQueue));
-
-    /* allot space for the failure table.  A failure entry in the table for
-     * every state(SCACBSCtx->state_count) */
-    ctx->failure_table = SCCalloc(ctx->state_count, sizeof(int32_t));
-    if (ctx->failure_table == NULL) {
-        FatalError("Error allocating memory");
-    }
-
-    /* add the failure transitions for the 0th state, and add every non-fail
-     * transition from the 0th state to the queue for further processing
-     * of failure states */
-    for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-        int32_t temp_state = ctx->goto_table[0][ascii_code];
-        if (temp_state != 0) {
-            SCACBSEnqueue(&q, temp_state);
-            ctx->failure_table[temp_state] = 0;
-        }
-    }
-
-    while (!SCACBSStateQueueIsEmpty(&q)) {
-        /* pick up every state from the queue and add failure transitions */
-        r_state = SCACBSDequeue(&q);
-        for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-            int32_t temp_state = ctx->goto_table[r_state][ascii_code];
-            if (temp_state == SC_AC_BS_FAIL)
-                continue;
-            SCACBSEnqueue(&q, temp_state);
-            state = ctx->failure_table[r_state];
-
-            while(ctx->goto_table[state][ascii_code] == SC_AC_BS_FAIL)
-                state = ctx->failure_table[state];
-            ctx->failure_table[temp_state] = ctx->goto_table[state][ascii_code];
-            SCACBSClubOutputStates(temp_state, ctx->failure_table[temp_state],
-                                 mpm_ctx);
-        }
-    }
-
-    return;
-}
-
-/**
- * \internal
- * \brief Create the delta table.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSCreateDeltaTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int ascii_code = 0;
-    int32_t r_state = 0;
-
-    if (ctx->state_count < 32767) {
-        ctx->state_table_u16 = SCCalloc(ctx->state_count, sizeof(*ctx->state_table_u16));
-        if (ctx->state_table_u16 == NULL) {
-            FatalError("Error allocating memory");
-        }
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size += (ctx->state_count * sizeof(*ctx->state_table_u16));
-
-        StateQueue q;
-        memset(&q, 0, sizeof(StateQueue));
-
-        for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-            DEBUG_VALIDATE_BUG_ON(ctx->goto_table[0][ascii_code] > UINT16_MAX);
-            SC_AC_BS_STATE_TYPE_U16 temp_state = (uint16_t)ctx->goto_table[0][ascii_code];
-            ctx->state_table_u16[0][ascii_code] = temp_state;
-            if (temp_state != 0)
-                SCACBSEnqueue(&q, temp_state);
-        }
-
-        while (!SCACBSStateQueueIsEmpty(&q)) {
-            r_state = SCACBSDequeue(&q);
-
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                int32_t temp_state = ctx->goto_table[r_state][ascii_code];
-                if (temp_state != SC_AC_BS_FAIL) {
-                    SCACBSEnqueue(&q, temp_state);
-                    DEBUG_VALIDATE_BUG_ON(temp_state > UINT16_MAX);
-                    ctx->state_table_u16[r_state][ascii_code] = (uint16_t)temp_state;
-                } else {
-                    ctx->state_table_u16[r_state][ascii_code] =
-                        ctx->state_table_u16[ctx->failure_table[r_state]][ascii_code];
-                }
-            }
-        }
-    } else {
-        /* create space for the state table.  We could have used the existing goto
-         * table, but since we have it set to hold 32 bit state values, we will create
-         * a new state table here of type SC_AC_BS_STATE_TYPE(current set to uint16_t) */
-        ctx->state_table_u32 = SCCalloc(ctx->state_count, sizeof(*ctx->state_table_u32));
-        if (ctx->state_table_u32 == NULL) {
-            FatalError("Error allocating memory");
-        }
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size += (ctx->state_count * sizeof(*ctx->state_table_u32));
-
-        StateQueue q;
-        memset(&q, 0, sizeof(StateQueue));
-
-        for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-            SC_AC_BS_STATE_TYPE_U32 temp_state = ctx->goto_table[0][ascii_code];
-            ctx->state_table_u32[0][ascii_code] = temp_state;
-            if (temp_state != 0)
-                SCACBSEnqueue(&q, temp_state);
-        }
-
-        while (!SCACBSStateQueueIsEmpty(&q)) {
-            r_state = SCACBSDequeue(&q);
-
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                int32_t temp_state = ctx->goto_table[r_state][ascii_code];
-                if (temp_state != SC_AC_BS_FAIL) {
-                    SCACBSEnqueue(&q, temp_state);
-                    ctx->state_table_u32[r_state][ascii_code] = temp_state;
-                } else {
-                    ctx->state_table_u32[r_state][ascii_code] =
-                        ctx->state_table_u32[ctx->failure_table[r_state]][ascii_code];
-                }
-            }
-        }
-    }
-
-    return;
-}
-
-static inline void SCACBSClubOutputStatePresenceWithDeltaTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int ascii_code = 0;
-    uint32_t state = 0;
-    uint32_t temp_state = 0;
-
-    if (ctx->state_count < 32767) {
-        for (state = 0; state < ctx->state_count; state++) {
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                temp_state = ctx->state_table_u16[state & 0x7FFF][ascii_code];
-                if (ctx->output_table[temp_state & 0x7FFF].no_of_entries != 0)
-                    ctx->state_table_u16[state & 0x7FFF][ascii_code] |= (1 << 15);
-            }
-        }
-    } else {
-        for (state = 0; state < ctx->state_count; state++) {
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                temp_state = ctx->state_table_u32[state & 0x00FFFFFF][ascii_code];
-                if (ctx->output_table[temp_state & 0x00FFFFFF].no_of_entries != 0)
-                    ctx->state_table_u32[state & 0x00FFFFFF][ascii_code] |= (1 << 24);
-            }
-        }
-    }
-
-    return;
-}
-
-static inline void SCACBSInsertCaseSensitiveEntriesForPatterns(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    uint32_t state = 0;
-    uint32_t k = 0;
-
-    for (state = 0; state < ctx->state_count; state++) {
-        if (ctx->output_table[state].no_of_entries == 0)
-            continue;
-
-        for (k = 0; k < ctx->output_table[state].no_of_entries; k++) {
-            if (ctx->pid_pat_list[ctx->output_table[state].pids[k]].cs != NULL) {
-                ctx->output_table[state].pids[k] &= 0x0000FFFF;
-                ctx->output_table[state].pids[k] |= 1 << 16;
-            }
-        }
-    }
-
-    return;
-}
-
-#if 0
-static void SCACBSPrintDeltaTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    int i = 0, j = 0;
-
-    printf("##############Delta Table##############\n");
-    for (i = 0; i < ctx->state_count; i++) {
-        printf("%d: \n", i);
-        for (j = 0; j < 256; j++) {
-            if (SCACBSGetDelta(i, j, mpm_ctx) != 0) {
-                printf("  %c -> %d\n", j, SCACBSGetDelta(i, j, mpm_ctx));
-            }
-        }
-    }
-
-    return;
-}
-#endif
-
-static inline int SCACBSZeroTransitionPresent(SCACBSCtx *ctx, uint32_t state)
-{
-    if (state == 0)
-        return 1;
-
-    if (ctx->state_count < 32767) {
-        int ascii;
-        for (ascii = 0; ascii < 256; ascii++) {
-            if ((ctx->state_table_u16[0][ascii] & 0x7fff) == (state & 0x7fff)) {
-                return 1;
-            }
-        }
-
-        return 0;
-    } else {
-        int ascii;
-        for (ascii = 0; ascii < 256; ascii++) {
-            if ((ctx->state_table_u32[0][ascii] & 0x00FFFFFF) ==
-                (state & 0x00FFFFFF)) {
-                return 1;
-            }
-        }
-
-        return 0;
-    }
-}
-
-/**
- * \internal
- * \brief Creates a new goto table structure(throw out all the failure
- *        transitions), to hold the existing goto table.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSCreateModDeltaTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-
-    if (ctx->state_count < 32767) {
-        int size = 0;
-        uint32_t state;
-
-        for (state = 1; state < ctx->state_count; state++) {
-            int ascii_code;
-            int k = 0;
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                uint32_t temp_state = ctx->state_table_u16[state][ascii_code];
-                if (SCACBSZeroTransitionPresent(ctx, temp_state))
-                    continue;
-                k++;
-            }
-            size += sizeof(uint16_t) * k * 2;
-        }
-
-        /* Let us use uint16_t for all.  That way we don//'t have to worry about
-         * alignment.  Technically 8 bits is all we need to store ascii codes,
-         * but by avoiding it, we save a lot of time on handling alignment */
-        size += (ctx->state_count * sizeof(SC_AC_BS_STATE_TYPE_U16) +
-                 256 * sizeof(SC_AC_BS_STATE_TYPE_U16) * 1);
-        ctx->state_table_mod = SCCalloc(1, size);
-        if (ctx->state_table_mod == NULL) {
-            FatalError("Error allocating memory");
-        }
-
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size += size;
-
-        /* buffer to hold pointers in the buffer, so that a state can use it
-         * directly to access its state data */
-        ctx->state_table_mod_pointers = SCCalloc(ctx->state_count, sizeof(uint8_t *));
-        if (ctx->state_table_mod_pointers == NULL) {
-            FatalError("Error allocating memory");
-        }
-
-        SC_AC_BS_STATE_TYPE_U16 temp_states[256];
-        uint16_t *curr_loc = (uint16_t *)ctx->state_table_mod;
-        uint16_t *no_of_entries = NULL;
-        uint16_t *ascii_codes = NULL;
-        uint16_t ascii_code = 0;
-        uint16_t k = 0;
-        for (state = 0; state < ctx->state_count; state++) {
-            /* store the starting location in the buffer for this state */
-            ctx->state_table_mod_pointers[state] = (uint8_t *)curr_loc;
-            no_of_entries = curr_loc++;
-            ascii_codes = curr_loc;
-            k = 0;
-            /* store all states that have non 0 transitions in the temp buffer */
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                uint32_t temp_state = ctx->state_table_u16[state][ascii_code];
-                if (state != 0 && SCACBSZeroTransitionPresent(ctx, temp_state))
-                    continue;
-
-                ascii_codes[k] = ascii_code;
-                temp_states[k] = ctx->state_table_u16[state][ascii_code];
-                k++;
-            }
-            /* if we have any non 0 transitions from our previous for search,
-             * store the ascii codes as well the corresponding states */
-            if (k > 0) {
-                no_of_entries[0] = k;
-                if (state != 0)
-                    curr_loc += k;
-                memcpy(curr_loc, temp_states, k * sizeof(SC_AC_BS_STATE_TYPE_U16));
-                curr_loc += k;
-            }
-        }
-
-        /* > 33766 */
-    } else {
-        int size = 0;
-        uint32_t state;
-        for (state = 1; state < ctx->state_count; state++) {
-            int ascii_code;
-            int k = 0;
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                uint32_t temp_state = ctx->state_table_u32[state][ascii_code];
-                if (SCACBSZeroTransitionPresent(ctx, temp_state))
-                    continue;
-                k++;
-            }
-            size += sizeof(uint32_t) * k * 2;
-        }
-
-        /* Let us use uint32_t for all.  That way we don//'t have to worry about
-         * alignment.  Technically 8 bits is all we need to store ascii codes,
-         * but by avoiding it, we save a lot of time on handling alignment */
-        size += (ctx->state_count * sizeof(SC_AC_BS_STATE_TYPE_U32) +
-                 256 * sizeof(SC_AC_BS_STATE_TYPE_U32) * 1);
-        ctx->state_table_mod = SCCalloc(1, size);
-        if (ctx->state_table_mod == NULL) {
-            FatalError("Error allocating memory");
-        }
-
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size += size;
-
-        /* buffer to hold pointers in the buffer, so that a state can use it
-         * directly to access its state data */
-        ctx->state_table_mod_pointers = SCCalloc(ctx->state_count, sizeof(uint8_t *));
-        if (ctx->state_table_mod_pointers == NULL) {
-            FatalError("Error allocating memory");
-        }
-
-        SC_AC_BS_STATE_TYPE_U32 temp_states[256];
-        uint32_t *curr_loc = (uint32_t *)ctx->state_table_mod;
-        uint32_t *no_of_entries = NULL;
-        uint32_t *ascii_codes = NULL;
-        uint32_t ascii_code = 0;
-        uint32_t k = 0;
-        for (state = 0; state < ctx->state_count; state++) {
-            /* store the starting location in the buffer for this state */
-            ctx->state_table_mod_pointers[state] = (uint8_t *)curr_loc;
-            no_of_entries = curr_loc++;
-            ascii_codes = curr_loc;
-            k = 0;
-            /* store all states that have non 0 transitions in the temp buffer */
-            for (ascii_code = 0; ascii_code < 256; ascii_code++) {
-                uint32_t temp_state = ctx->state_table_u32[state][ascii_code];
-                if (state != 0 && SCACBSZeroTransitionPresent(ctx, temp_state))
-                    continue;
-
-                ascii_codes[k] = ascii_code;
-                temp_states[k] = ctx->state_table_u32[state][ascii_code];
-                k++;
-            }
-            /* if we have any non 0 transitions from our previous for search,
-             * store the ascii codes as well the corresponding states */
-            if (k > 0) {
-                no_of_entries[0] = k;
-                if (state != 0)
-                    curr_loc += k;
-                memcpy(curr_loc, temp_states, k * sizeof(SC_AC_BS_STATE_TYPE_U32));
-                curr_loc += k;
-            }
-        }
-
-    }
-
-    return;
-}
-
-/**
- * \brief Process the patterns and prepare the state table.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-static inline void SCACBSPrepareStateTable(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-
-    /* create the 0th state in the goto table and output_table */
-    SCACBSInitNewState(mpm_ctx);
-
-    /* create the goto table */
-    SCACBSCreateGotoTable(mpm_ctx);
-    /* create the failure table */
-    SCACBSCreateFailureTable(mpm_ctx);
-    /* create the final state(delta) table */
-    SCACBSCreateDeltaTable(mpm_ctx);
-    /* club the output state presence with delta transition entries */
-    SCACBSClubOutputStatePresenceWithDeltaTable(mpm_ctx);
-    /* create the modified table */
-    SCACBSCreateModDeltaTable(mpm_ctx);
-
-    /* club nocase entries */
-    SCACBSInsertCaseSensitiveEntriesForPatterns(mpm_ctx);
-
-//    int state = 0;
-//    for (state = 0; state < ctx->state_count; state++) {
-//        int i = 0;
-//        for (i = 0; i < 256; i++) {
-//            if (ctx->state_table_u16[state][i] != 0) {
-//                printf("%d-%d-%d\n", state, i, ctx->state_table_u16[state][i] & 0x7fff) ;
-//            }
-//        }
-//    }
-
-#if 0
-    SCACBSPrintDeltaTable(mpm_ctx);
-#endif
-
-    /* we don't need these anymore */
-    SCFree(ctx->goto_table);
-    ctx->goto_table = NULL;
-    SCFree(ctx->failure_table);
-    ctx->failure_table = NULL;
-    SCFree(ctx->state_table_u16);
-    ctx->state_table_u16 = NULL;
-    SCFree(ctx->state_table_u32);
-    ctx->state_table_u32 = NULL;
-
-    return;
-}
-
-/**
- * \brief Process the patterns added to the mpm, and create the internal tables.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-int SCACBSPreparePatterns(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-
-    if (mpm_ctx->pattern_cnt == 0 || mpm_ctx->init_hash == NULL) {
-        SCLogDebug("no patterns supplied to this mpm_ctx");
-        return 0;
-    }
-
-    /* alloc the pattern array */
-    ctx->parray = (MpmPattern **)SCCalloc(mpm_ctx->pattern_cnt, sizeof(MpmPattern *));
-    if (ctx->parray == NULL)
-        goto error;
-    mpm_ctx->memory_cnt++;
-    mpm_ctx->memory_size += (mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
-
-    /* populate it with the patterns in the hash */
-    uint32_t i = 0, p = 0;
-    for (i = 0; i < MPM_INIT_HASH_SIZE; i++) {
-        MpmPattern *node = mpm_ctx->init_hash[i], *nnode = NULL;
-        while(node != NULL) {
-            nnode = node->next;
-            node->next = NULL;
-            ctx->parray[p++] = node;
-            node = nnode;
-        }
-    }
-
-    /* we no longer need the hash, so free it's memory */
-    SCFree(mpm_ctx->init_hash);
-    mpm_ctx->init_hash = NULL;
-
-    /* the memory consumed by a single state in our goto table */
-    ctx->single_state_size = sizeof(int32_t) * 256;
-
-    /* handle no case patterns */
-    ctx->pid_pat_list = SCCalloc((mpm_ctx->max_pat_id + 1), sizeof(SCACBSPatternList));
-    if (ctx->pid_pat_list == NULL) {
-        FatalError("Error allocating memory");
-    }
-
-    for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
-        if (!(ctx->parray[i]->flags & MPM_PATTERN_FLAG_NOCASE)) {
-            ctx->pid_pat_list[ctx->parray[i]->id].cs = SCMalloc(ctx->parray[i]->len);
-            if (ctx->pid_pat_list[ctx->parray[i]->id].cs == NULL) {
-                FatalError("Error allocating memory");
-            }
-            memcpy(ctx->pid_pat_list[ctx->parray[i]->id].cs,
-                   ctx->parray[i]->original_pat, ctx->parray[i]->len);
-            ctx->pid_pat_list[ctx->parray[i]->id].patlen = ctx->parray[i]->len;
-        }
-
-        /* ACPatternList now owns this memory */
-        ctx->pid_pat_list[ctx->parray[i]->id].sids_size = ctx->parray[i]->sids_size;
-        ctx->pid_pat_list[ctx->parray[i]->id].sids = ctx->parray[i]->sids;
-    }
-
-    /* prepare the state table required by AC */
-    SCACBSPrepareStateTable(mpm_ctx);
-
-    /* free all the stored patterns.  Should save us a good 100-200 mbs */
-    for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
-        if (ctx->parray[i] != NULL) {
-            MpmFreePattern(mpm_ctx, ctx->parray[i]);
-        }
-    }
-    SCFree(ctx->parray);
-    ctx->parray = NULL;
-    mpm_ctx->memory_cnt--;
-    mpm_ctx->memory_size -= (mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
-
-    ctx->pattern_id_bitarray_size = (mpm_ctx->max_pat_id / 8) + 1;
-
-    return 0;
-
-error:
-    return -1;
-}
-
-/**
- * \brief Initialize the AC context.
- *
- * \param mpm_ctx       Mpm context.
- */
-void SCACBSInitCtx(MpmCtx *mpm_ctx)
-{
-    if (mpm_ctx->ctx != NULL)
-        return;
-
-    mpm_ctx->ctx = SCCalloc(1, sizeof(SCACBSCtx));
-    if (mpm_ctx->ctx == NULL) {
-        exit(EXIT_FAILURE);
-    }
-
-    mpm_ctx->memory_cnt++;
-    mpm_ctx->memory_size += sizeof(SCACBSCtx);
-
-    /* initialize the hash we use to speed up pattern insertions */
-    mpm_ctx->init_hash = SCCalloc(MPM_INIT_HASH_SIZE, sizeof(MpmPattern *));
-    if (mpm_ctx->init_hash == NULL) {
-        exit(EXIT_FAILURE);
-    }
-
-    /* get conf values for AC from our yaml file.  We have no conf values for
-     * now.  We will certainly need this, as we develop the algo */
-    SCACBSGetConfig();
-
-    SCReturn;
-}
-
-/**
- * \brief Destroy the mpm context.
- *
- * \param mpm_ctx Pointer to the mpm context.
- */
-void SCACBSDestroyCtx(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    if (ctx == NULL)
-        return;
-
-    if (mpm_ctx->init_hash != NULL) {
-        SCFree(mpm_ctx->init_hash);
-        mpm_ctx->init_hash = NULL;
-        mpm_ctx->memory_cnt--;
-        mpm_ctx->memory_size -= (MPM_INIT_HASH_SIZE * sizeof(MpmPattern *));
-    }
-
-    if (ctx->parray != NULL) {
-        uint32_t i;
-        for (i = 0; i < mpm_ctx->pattern_cnt; i++) {
-            if (ctx->parray[i] != NULL) {
-                MpmFreePattern(mpm_ctx, ctx->parray[i]);
-            }
-        }
-
-        SCFree(ctx->parray);
-        ctx->parray = NULL;
-        mpm_ctx->memory_cnt--;
-        mpm_ctx->memory_size -= (mpm_ctx->pattern_cnt * sizeof(MpmPattern *));
-    }
-
-    if (ctx->state_table_u16 != NULL) {
-        SCFree(ctx->state_table_u16);
-        ctx->state_table_u16 = NULL;
-
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size -= (ctx->state_count *
-                                 sizeof(SC_AC_BS_STATE_TYPE_U16) * 256);
-    } else if (ctx->state_table_u32 != NULL) {
-        SCFree(ctx->state_table_u32);
-        ctx->state_table_u32 = NULL;
-
-        mpm_ctx->memory_cnt++;
-        mpm_ctx->memory_size -= (ctx->state_count *
-                                 sizeof(SC_AC_BS_STATE_TYPE_U32) * 256);
-    }
-
-    if (ctx->output_table != NULL) {
-        uint32_t state_count;
-        for (state_count = 0; state_count < ctx->state_count; state_count++) {
-            if (ctx->output_table[state_count].pids != NULL) {
-                SCFree(ctx->output_table[state_count].pids);
-            }
-        }
-        SCFree(ctx->output_table);
-    }
-
-    if (ctx->pid_pat_list != NULL) {
-        uint32_t i;
-        for (i = 0; i < (mpm_ctx->max_pat_id + 1); i++) {
-            if (ctx->pid_pat_list[i].cs != NULL)
-                SCFree(ctx->pid_pat_list[i].cs);
-            if (ctx->pid_pat_list[i].sids != NULL)
-                SCFree(ctx->pid_pat_list[i].sids);
-        }
-        SCFree(ctx->pid_pat_list);
-    }
-
-    if (ctx->state_table_mod != NULL) {
-        SCFree(ctx->state_table_mod);
-        ctx->state_table_mod = NULL;
-    }
-
-    if (ctx->state_table_mod_pointers != NULL) {
-        SCFree(ctx->state_table_mod_pointers);
-        ctx->state_table_mod_pointers = NULL;
-    }
-
-    SCFree(mpm_ctx->ctx);
-    mpm_ctx->memory_cnt--;
-    mpm_ctx->memory_size -= sizeof(SCACBSCtx);
-
-    return;
-}
-
-/**
- * \brief The aho corasick search function.
- *
- * \param mpm_ctx        Pointer to the mpm context.
- * \param mpm_thread_ctx Pointer to the mpm thread context.
- * \param pmq            Pointer to the Pattern Matcher Queue to hold
- *                       search matches.
- * \param buf            Buffer to be searched.
- * \param buflen         Buffer length.
- *
- * \retval matches Match count.
- */
-uint32_t SCACBSSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
-                      PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen)
-{
-    const SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-    uint32_t i = 0;
-    int matches = 0;
-    uint8_t buf_local;
-
-    /* \todo tried loop unrolling with register var, with no perf increase.  Need
-     * to dig deeper */
-    /* \todo Change it for stateful MPM.  Supply the state using mpm_thread_ctx */
-    SCACBSPatternList *pid_pat_list = ctx->pid_pat_list;
-
-    uint8_t bitarray[ctx->pattern_id_bitarray_size];
-    memset(bitarray, 0, ctx->pattern_id_bitarray_size);
-
-    if (ctx->state_count < 32767) {
-        register SC_AC_BS_STATE_TYPE_U16 state = 0;
-        uint16_t no_of_entries;
-        uint16_t *ascii_codes;
-        uint16_t **state_table_mod_pointers = (uint16_t **)ctx->state_table_mod_pointers;
-        uint16_t *zero_state = state_table_mod_pointers[0] + 1;
-
-        for (i = 0; i < buflen; i++) {
-            if (state == 0) {
-                state = zero_state[u8_tolower(buf[i])];
-            } else {
-                no_of_entries = *(state_table_mod_pointers[state & 0x7FFF]);
-                if (no_of_entries == 1) {
-                    ascii_codes = state_table_mod_pointers[state & 0x7FFF] + 1;
-                    buf_local = u8_tolower(buf[i]);
-                    if (buf_local == ascii_codes[0]) {
-                        state = *(ascii_codes + no_of_entries);
-                    } else {
-                        state = zero_state[buf_local];
-                    }
-                } else {
-                    if (no_of_entries == 0) {
-                        state = zero_state[u8_tolower(buf[i])];
-                        goto match_u16;
-                    }
-                    buf_local = u8_tolower(buf[i]);
-                    ascii_codes = state_table_mod_pointers[state & 0x7FFF] + 1;
-                    int low = 0;
-                    int high = no_of_entries;
-                    int mid;
-                    while (low <= high) {
-                        mid = (low + high) / 2;
-                        if (ascii_codes[mid] == buf_local) {
-                            state = ((ascii_codes + no_of_entries))[mid];
-                            goto match_u16;
-                        } else if (ascii_codes[mid] < buf_local) {
-                            low = mid + 1;
-                        } else {
-                            high = mid - 1;
-                        }
-                    } /* while */
-                    state = zero_state[buf_local];
-                } /* else - if (no_of_entires == 1) */
-            }
-
-        match_u16:
-            if (state & 0x8000) {
-                uint32_t nentries = ctx->output_table[state & 0x7FFF].no_of_entries;
-                uint32_t *pids = ctx->output_table[state & 0x7FFF].pids;
-                uint32_t k;
-                for (k = 0; k < nentries; k++) {
-                    if (pids[k] & 0xFFFF0000) {
-                        uint32_t lower_pid = pids[k] & 0x0000FFFF;
-                        if (SCMemcmp(pid_pat_list[lower_pid].cs,
-                                     buf + i - pid_pat_list[lower_pid].patlen + 1,
-                                     pid_pat_list[lower_pid].patlen) != 0) {
-                            /* inside loop */
-                            continue;
-                        }
-                        if (bitarray[(lower_pid) / 8] & (1 << ((lower_pid) % 8))) {
-                            ;
-                        } else {
-                            bitarray[(lower_pid) / 8] |= (1 << ((lower_pid) % 8));
-                            PrefilterAddSids(pmq, pid_pat_list[lower_pid].sids,
-                                    pid_pat_list[lower_pid].sids_size);
-                        }
-                        matches++;
-                    } else {
-                        if (bitarray[pids[k] / 8] & (1 << (pids[k] % 8))) {
-                            ;
-                        } else {
-                            bitarray[pids[k] / 8] |= (1 << (pids[k] % 8));
-                            PrefilterAddSids(pmq, pid_pat_list[pids[k]].sids,
-                                    pid_pat_list[pids[k]].sids_size);
-                        }
-                        matches++;
-                    }
-                    //loop1:
-                    //;
-                }
-            }
-        } /* for (i = 0; i < buflen; i++) */
-
-    } else {
-        register SC_AC_BS_STATE_TYPE_U32 state = 0;
-        uint32_t no_of_entries;
-        uint32_t *ascii_codes;
-        uint32_t **state_table_mod_pointers = (uint32_t **)ctx->state_table_mod_pointers;
-        uint32_t *zero_state = state_table_mod_pointers[0] + 1;
-
-        for (i = 0; i < buflen; i++) {
-            if (state == 0) {
-                state = zero_state[u8_tolower(buf[i])];
-            } else {
-                no_of_entries = *(state_table_mod_pointers[state & 0x00FFFFFF]);
-                if (no_of_entries == 1) {
-                    ascii_codes = state_table_mod_pointers[state & 0x00FFFFFF] + 1;
-                    buf_local = u8_tolower(buf[i]);
-                    if (buf_local == ascii_codes[0]) {
-                        state = *(ascii_codes + no_of_entries);
-                    } else {
-                        state = zero_state[buf_local];
-                    }
-                } else {
-                    if (no_of_entries == 0) {
-                        state = zero_state[u8_tolower(buf[i])];
-                        goto match_u32;
-                    }
-                    buf_local = u8_tolower(buf[i]);
-                    ascii_codes = state_table_mod_pointers[state & 0x00FFFFFF] + 1;
-                    int low = 0;
-                    int high = no_of_entries;
-                    int mid;
-                    while (low <= high) {
-                        mid = (low + high) / 2;
-                        if (ascii_codes[mid] == buf_local) {
-                            state = ((ascii_codes + no_of_entries))[mid];
-                            goto match_u32;
-                        } else if (ascii_codes[mid] < buf_local) {
-                            low = mid + 1;
-                        } else {
-                            high = mid - 1;
-                        }
-                    } /* while */
-                    state = zero_state[buf_local];
-                } /* else - if (no_of_entires == 1) */
-            }
-
-        match_u32:
-            if (state & 0xFF000000) {
-                uint32_t nentries = ctx->output_table[state & 0x00FFFFFF].no_of_entries;
-                uint32_t *pids = ctx->output_table[state & 0x00FFFFFF].pids;
-                uint32_t k;
-                for (k = 0; k < nentries; k++) {
-                    if (pids[k] & 0xFFFF0000) {
-                        uint32_t lower_pid = pids[k] & 0x0000FFFF;
-                        if (SCMemcmp(pid_pat_list[lower_pid].cs,
-                                     buf + i - pid_pat_list[lower_pid].patlen + 1,
-                                     pid_pat_list[lower_pid].patlen) != 0) {
-                            /* inside loop */
-                            continue;
-                        }
-                        if (bitarray[(lower_pid) / 8] & (1 << ((lower_pid) % 8))) {
-                            ;
-                        } else {
-                            bitarray[(lower_pid) / 8] |= (1 << ((lower_pid) % 8));
-                            PrefilterAddSids(pmq, pid_pat_list[lower_pid].sids,
-                                    pid_pat_list[lower_pid].sids_size);
-                        }
-                        matches++;
-                    } else {
-                        if (bitarray[pids[k] / 8] & (1 << (pids[k] % 8))) {
-                            ;
-                        } else {
-                            bitarray[pids[k] / 8] |= (1 << (pids[k] % 8));
-                            PrefilterAddSids(pmq, pid_pat_list[pids[k]].sids,
-                                    pid_pat_list[pids[k]].sids_size);
-                        }
-                        matches++;
-                    }
-                    //loop1:
-                    //;
-                }
-            }
-        } /* for (i = 0; i < buflen; i++) */
-    }
-
-    return matches;
-}
-
-/**
- * \brief Add a case insensitive pattern.  Although we have different calls for
- *        adding case sensitive and insensitive patterns, we make a single call
- *        for either case.  No special treatment for either case.
- *
- * \param mpm_ctx Pointer to the mpm context.
- * \param pat     The pattern to add.
- * \param patnen  The pattern length.
- * \param offset  Ignored.
- * \param depth   Ignored.
- * \param pid     The pattern id.
- * \param sid     Ignored.
- * \param flags   Flags associated with this pattern.
- *
- * \retval  0 On success.
- * \retval -1 On failure.
- */
-int SCACBSAddPatternCI(MpmCtx *mpm_ctx, uint8_t *pat, uint16_t patlen,
-                     uint16_t offset, uint16_t depth, uint32_t pid,
-                     SigIntId sid, uint8_t flags)
-{
-    flags |= MPM_PATTERN_FLAG_NOCASE;
-    return MpmAddPattern(mpm_ctx, pat, patlen, offset, depth, pid, sid, flags);
-}
-
-/**
- * \brief Add a case sensitive pattern.  Although we have different calls for
- *        adding case sensitive and insensitive patterns, we make a single call
- *        for either case.  No special treatment for either case.
- *
- * \param mpm_ctx Pointer to the mpm context.
- * \param pat     The pattern to add.
- * \param patnen  The pattern length.
- * \param offset  Ignored.
- * \param depth   Ignored.
- * \param pid     The pattern id.
- * \param sid     Ignored.
- * \param flags   Flags associated with this pattern.
- *
- * \retval  0 On success.
- * \retval -1 On failure.
- */
-int SCACBSAddPatternCS(MpmCtx *mpm_ctx, uint8_t *pat, uint16_t patlen,
-                     uint16_t offset, uint16_t depth, uint32_t pid,
-                     SigIntId sid, uint8_t flags)
-{
-    return MpmAddPattern(mpm_ctx, pat, patlen, offset, depth, pid, sid, flags);
-}
-
-void SCACBSPrintInfo(MpmCtx *mpm_ctx)
-{
-    SCACBSCtx *ctx = (SCACBSCtx *)mpm_ctx->ctx;
-
-    printf("MPM AC Information:\n");
-    printf("Memory allocs:   %" PRIu32 "\n", mpm_ctx->memory_cnt);
-    printf("Memory alloced:  %" PRIu32 "\n", mpm_ctx->memory_size);
-    printf(" Sizeof:\n");
-    printf("  MpmCtx         %" PRIuMAX "\n", (uintmax_t)sizeof(MpmCtx));
-    printf("  SCACBSCtx:         %" PRIuMAX "\n", (uintmax_t)sizeof(SCACBSCtx));
-    printf("  MpmPattern      %" PRIuMAX "\n", (uintmax_t)sizeof(MpmPattern));
-    printf("  MpmPattern     %" PRIuMAX "\n", (uintmax_t)sizeof(MpmPattern));
-    printf("Unique Patterns: %" PRIu32 "\n", mpm_ctx->pattern_cnt);
-    printf("Smallest:        %" PRIu32 "\n", mpm_ctx->minlen);
-    printf("Largest:         %" PRIu32 "\n", mpm_ctx->maxlen);
-    printf("Total states in the state table:    %" PRIu32 "\n", ctx->state_count);
-    printf("\n");
-
-    return;
-}
-
-/*************************************Unittests********************************/
-
-#ifdef UNITTESTS
-#include "detect-engine-alert.h"
-
-static int SCACBSTest01(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghjiklmnopqrstuvwxyz";
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest02(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abce", 4, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghjiklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 0)
-        result = 1;
-    else
-        printf("0 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest03(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcde", 4, 0, 0, 1, 0, 0);
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"fghj", 4, 0, 0, 2, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghjiklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 3)
-        result = 1;
-    else
-        printf("3 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest04(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdegh", 6, 0, 0, 1, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"fghjxyz", 7, 0, 0, 2, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghjiklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest05(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"fghJikl", 7, 0, 0, 2, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghjiklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 3)
-        result = 1;
-    else
-        printf("3 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest06(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcd";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest07(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* should match 30 times */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"A", 1, 0, 0, 0, 0, 0);
-    /* should match 29 times */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 1, 0, 0);
-    /* should match 28 times */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAA", 3, 0, 0, 2, 0, 0);
-    /* 26 */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAAAA", 5, 0, 0, 3, 0, 0);
-    /* 21 */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAAAAAAAAA", 10, 0, 0, 4, 0, 0);
-    /* 1 */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
-                     30, 0, 0, 5, 0, 0);
-    PmqSetup(&pmq);
-    /* total matches: 135 */
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 135)
-        result = 1;
-    else
-        printf("135 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest08(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)"a", 1);
-
-    if (cnt == 0)
-        result = 1;
-    else
-        printf("0 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest09(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ab", 2, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)"ab", 2);
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest10(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcdefgh", 8, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "01234567890123456789012345678901234567890123456789"
-                "01234567890123456789012345678901234567890123456789"
-                "abcdefgh"
-                "01234567890123456789012345678901234567890123456789"
-                "01234567890123456789012345678901234567890123456789";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest11(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"he", 2, 0, 0, 1, 0, 0) == -1)
-        goto end;
-    if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"she", 3, 0, 0, 2, 0, 0) == -1)
-        goto end;
-    if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"his", 3, 0, 0, 3, 0, 0) == -1)
-        goto end;
-    if (MpmAddPatternCS(&mpm_ctx, (uint8_t *)"hers", 4, 0, 0, 4, 0, 0) == -1)
-        goto end;
-    PmqSetup(&pmq);
-
-    if (SCACBSPreparePatterns(&mpm_ctx) == -1)
-        goto end;
-
-    result = 1;
-
-    const char *buf = "he";
-    result &= (SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq, (uint8_t *)buf,
-                          strlen(buf)) == 1);
-    buf = "she";
-    result &= (SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq, (uint8_t *)buf,
-                          strlen(buf)) == 2);
-    buf = "his";
-    result &= (SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq, (uint8_t *)buf,
-                          strlen(buf)) == 1);
-    buf = "hers";
-    result &= (SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq, (uint8_t *)buf,
-                          strlen(buf)) == 2);
-
- end:
-     SCACBSDestroyCtx(&mpm_ctx);
-     PmqFree(&pmq);
-     return result;
-}
-
-static int SCACBSTest12(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"wxyz", 4, 0, 0, 0, 0, 0);
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"vwxyz", 5, 0, 0, 1, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 2)
-        result = 1;
-    else
-        printf("2 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest13(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcdefghijklmnopqrstuvwxyzABCD";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyzABCD";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest14(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDE";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyzABCDE";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest15(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcdefghijklmnopqrstuvwxyzABCDEF";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyzABCDEF";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest16(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcdefghijklmnopqrstuvwxyzABC";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyzABC";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest17(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcdefghijklmnopqrstuvwxyzAB";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyzAB";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest18(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    const char pat[] = "abcde"
-                       "fghij"
-                       "klmno"
-                       "pqrst"
-                       "uvwxy"
-                       "z";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcde""fghij""klmno""pqrst""uvwxy""z";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest19(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 */
-    const char pat[] = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest20(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 */
-    const char pat[] = "AAAAA"
-                       "AAAAA"
-                       "AAAAA"
-                       "AAAAA"
-                       "AAAAA"
-                       "AAAAA"
-                       "AA";
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)pat, sizeof(pat) - 1, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "AAAAA""AAAAA""AAAAA""AAAAA""AAAAA""AAAAA""AA";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest21(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                              (uint8_t *)"AA", 2);
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest22(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcd", 4, 0, 0, 0, 0, 0);
-    /* 1 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcde", 5, 0, 0, 1, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefghijklmnopqrstuvwxyz";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                              (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 2)
-        result = 1;
-    else
-        printf("2 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest23(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                              (uint8_t *)"aa", 2);
-
-    if (cnt == 0)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest24(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 1 */
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"AA", 2, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                              (uint8_t *)"aa", 2);
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("1 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest25(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"ABCD", 4, 0, 0, 0, 0, 0);
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"bCdEfG", 6, 0, 0, 1, 0, 0);
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"fghiJkl", 7, 0, 0, 2, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 3)
-        result = 1;
-    else
-        printf("3 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest26(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCI(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 0, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"Works", 5, 0, 0, 1, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "works";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 1)
-        result = 1;
-    else
-        printf("3 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest27(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 0 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"ONE", 3, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "tone";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 0)
-        result = 1;
-    else
-        printf("0 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest28(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    /* 0 match */
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"one", 3, 0, 0, 0, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "tONE";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                               (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 0)
-        result = 1;
-    else
-        printf("0 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest29(void)
-{
-    int result = 0;
-    MpmCtx mpm_ctx;
-    MpmThreadCtx mpm_thread_ctx;
-    PrefilterRuleStore pmq;
-
-    memset(&mpm_ctx, 0x00, sizeof(MpmCtx));
-    memset(&mpm_thread_ctx, 0, sizeof(MpmThreadCtx));
-    MpmInitCtx(&mpm_ctx, MPM_AC_BS);
-
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"abcde", 5, 0, 0, 0, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"bcdef", 5, 0, 0, 1, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"cdefg", 5, 0, 0, 3, 0, 0);
-    MpmAddPatternCS(&mpm_ctx, (uint8_t *)"defgh", 5, 0, 0, 4, 0, 0);
-    PmqSetup(&pmq);
-
-    SCACBSPreparePatterns(&mpm_ctx);
-
-    const char *buf = "abcdefgh";
-    uint32_t cnt = SCACBSSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
-                                (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 4)
-        result = 1;
-    else
-        printf("3 != %" PRIu32 " ",cnt);
-
-    SCACBSDestroyCtx(&mpm_ctx);
-    PmqFree(&pmq);
-    return result;
-}
-
-static int SCACBSTest30(void)
-{
-    uint8_t buf[] = "onetwothreefourfivesixseveneightnine";
-    uint16_t buflen = sizeof(buf) - 1;
-    Packet *p = NULL;
-    ThreadVars th_v;
-    DetectEngineThreadCtx *det_ctx = NULL;
-    int result = 0;
-
-    memset(&th_v, 0, sizeof(th_v));
-    p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
-
-    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-    de_ctx->mpm_matcher = MPM_AC_BS;
-
-    de_ctx->flags |= DE_QUIET;
-
-    de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
-                               "(content:\"onetwothreefourfivesixseveneightnine\"; sid:1;)");
-    if (de_ctx->sig_list == NULL)
-        goto end;
-    de_ctx->sig_list->next = SigInit(de_ctx, "alert tcp any any -> any any "
-                               "(content:\"onetwothreefourfivesixseveneightnine\"; fast_pattern:3,3; sid:2;)");
-    if (de_ctx->sig_list->next == NULL)
-        goto end;
-
-    SigGroupBuild(de_ctx);
-    DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
-
-    SigMatchSignatures(&th_v, de_ctx, det_ctx, p);
-    if (PacketAlertCheck(p, 1) != 1) {
-        printf("if (PacketAlertCheck(p, 1) != 1) failure\n");
-        goto end;
-    }
-    if (PacketAlertCheck(p, 2) != 1) {
-        printf("if (PacketAlertCheck(p, 1) != 2) failure\n");
-        goto end;
-    }
-
-    result = 1;
-end:
-    if (de_ctx != NULL) {
-        SigGroupCleanup(de_ctx);
-        SigCleanSignatures(de_ctx);
-
-        DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
-        DetectEngineCtxFree(de_ctx);
-    }
-
-    UTHFreePackets(&p, 1);
-    return result;
-}
-
-#endif /* UNITTESTS */
-
-void SCACBSRegisterTests(void)
-{
-
-#ifdef UNITTESTS
-    UtRegisterTest("SCACBSTest01", SCACBSTest01);
-    UtRegisterTest("SCACBSTest02", SCACBSTest02);
-    UtRegisterTest("SCACBSTest03", SCACBSTest03);
-    UtRegisterTest("SCACBSTest04", SCACBSTest04);
-    UtRegisterTest("SCACBSTest05", SCACBSTest05);
-    UtRegisterTest("SCACBSTest06", SCACBSTest06);
-    UtRegisterTest("SCACBSTest07", SCACBSTest07);
-    UtRegisterTest("SCACBSTest08", SCACBSTest08);
-    UtRegisterTest("SCACBSTest09", SCACBSTest09);
-    UtRegisterTest("SCACBSTest10", SCACBSTest10);
-    UtRegisterTest("SCACBSTest11", SCACBSTest11);
-    UtRegisterTest("SCACBSTest12", SCACBSTest12);
-    UtRegisterTest("SCACBSTest13", SCACBSTest13);
-    UtRegisterTest("SCACBSTest14", SCACBSTest14);
-    UtRegisterTest("SCACBSTest15", SCACBSTest15);
-    UtRegisterTest("SCACBSTest16", SCACBSTest16);
-    UtRegisterTest("SCACBSTest17", SCACBSTest17);
-    UtRegisterTest("SCACBSTest18", SCACBSTest18);
-    UtRegisterTest("SCACBSTest19", SCACBSTest19);
-    UtRegisterTest("SCACBSTest20", SCACBSTest20);
-    UtRegisterTest("SCACBSTest21", SCACBSTest21);
-    UtRegisterTest("SCACBSTest22", SCACBSTest22);
-    UtRegisterTest("SCACBSTest23", SCACBSTest23);
-    UtRegisterTest("SCACBSTest24", SCACBSTest24);
-    UtRegisterTest("SCACBSTest25", SCACBSTest25);
-    UtRegisterTest("SCACBSTest26", SCACBSTest26);
-    UtRegisterTest("SCACBSTest27", SCACBSTest27);
-    UtRegisterTest("SCACBSTest28", SCACBSTest28);
-    UtRegisterTest("SCACBSTest29", SCACBSTest29);
-    UtRegisterTest("SCACBSTest30", SCACBSTest30);
-#endif
-
-    return;
-}
diff --git a/src/util-mpm-ac-bs.h b/src/util-mpm-ac-bs.h
deleted file mode 100644
index d1135b1cfea6..000000000000
--- a/src/util-mpm-ac-bs.h
+++ /dev/null
@@ -1,74 +0,0 @@
-/* Copyright (C) 2007-2014 Open Information Security Foundation
- *
- * You can copy, redistribute or modify this Program under the terms of
- * the GNU General Public License version 2 as published by the Free
- * Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * version 2 along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
- * 02110-1301, USA.
- */
-
-/**
- * \file
- *
- * \author Anoop Saldanha <anoopsaldanha@gmail.com>
- *
- */
-
-#include "util-mpm.h"
-
-#define SC_AC_BS_STATE_TYPE_U16 uint16_t
-#define SC_AC_BS_STATE_TYPE_U32 uint32_t
-
-typedef struct SCACBSPatternList_ {
-    uint8_t *cs;
-    uint16_t patlen;
-
-    /* sid(s) for this pattern */
-    uint32_t sids_size;
-    SigIntId *sids;
-} SCACBSPatternList;
-
-typedef struct SCACBSOutputTable_ {
-    /* list of pattern sids */
-    uint32_t *pids;
-    /* no of entries we have in pids */
-    uint32_t no_of_entries;
-} SCACBSOutputTable;
-
-typedef struct SCACBSCtx_ {
-    /* pattern arrays.  We need this only during the goto table creation phase */
-    MpmPattern **parray;
-
-    /* no of states used by ac */
-    uint32_t state_count;
-
-    uint32_t pattern_id_bitarray_size;
-
-    /* the all important memory hungry state_table */
-    SC_AC_BS_STATE_TYPE_U16 (*state_table_u16)[256];
-    /* the all important memory hungry state_table */
-    SC_AC_BS_STATE_TYPE_U32 (*state_table_u32)[256];
-    /* the modified goto_table */
-    uint8_t *state_table_mod;
-    uint8_t **state_table_mod_pointers;
-
-    /* goto_table, failure table and output table.  Needed to create state_table.
-     * Will be freed, once we have created the state_table */
-    int32_t (*goto_table)[256];
-    int32_t *failure_table;
-    SCACBSOutputTable *output_table;
-    SCACBSPatternList *pid_pat_list;
-
-    /* the size of each state */
-    uint16_t single_state_size;
-} SCACBSCtx;
-
-void MpmACBSRegister(void);
diff --git a/src/util-mpm.c b/src/util-mpm.c
index 0638a8876c53..47a19b2c8e88 100644
--- a/src/util-mpm.c
+++ b/src/util-mpm.c
@@ -29,7 +29,6 @@
 
 /* include pattern matchers */
 #include "util-mpm-ac.h"
-#include "util-mpm-ac-bs.h"
 #include "util-mpm-ac-ks.h"
 #include "util-mpm-hs.h"
 #include "util-hashlist.h"
@@ -229,7 +228,6 @@ void MpmTableSetup(void)
     mpm_default_matcher = DEFAULT_MPM;
 
     MpmACRegister();
-    MpmACBSRegister();
     MpmACTileRegister();
 #ifdef BUILD_HYPERSCAN
     #ifdef HAVE_HS_VALID_PLATFORM
diff --git a/src/util-mpm.h b/src/util-mpm.h
index 87eec5e793a9..67988efd0314 100644
--- a/src/util-mpm.h
+++ b/src/util-mpm.h
@@ -34,7 +34,6 @@ enum {
 
     /* aho-corasick */
     MPM_AC,
-    MPM_AC_BS,
     MPM_AC_KS,
     MPM_HS,
     /* table size */

From 83630ebb9cee36612a280f387b693db4b347d604 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 12:08:20 +0100
Subject: [PATCH 185/462] mpm/ac: return only unique match count

Bring implementation in line with Hyperscan, which only counts unique matches.

Update test to reflect the new behavior.
---
 src/util-mpm-ac.c | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index 22347d6fef23..6d0fc050b99a 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -930,7 +930,7 @@ void SCACDestroyCtx(MpmCtx *mpm_ctx)
  * \param buf            Buffer to be searched.
  * \param buflen         Buffer length.
  *
- * \retval matches Match count.
+ * \retval matches Match count: counts unique matches per pattern.
  */
 uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                     PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen)
@@ -975,8 +975,8 @@ uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         } else {
                             bitarray[(lower_pid) / 8] |= (1 << ((lower_pid) % 8));
                             PrefilterAddSids(pmq, pat->sids, pat->sids_size);
+                            matches++;
                         }
-                        matches++;
                     } else {
                         const SCACPatternList *pat = &pid_pat_list[pids[k]];
                         const int offset = i - pat->patlen + 1;
@@ -989,8 +989,8 @@ uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         } else {
                             bitarray[pids[k] / 8] |= (1 << (pids[k] % 8));
                             PrefilterAddSids(pmq, pat->sids, pat->sids_size);
+                            matches++;
                         }
-                        matches++;
                     }
                     //loop1:
                     //;
@@ -1026,8 +1026,8 @@ uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         } else {
                             bitarray[(lower_pid) / 8] |= (1 << ((lower_pid) % 8));
                             PrefilterAddSids(pmq, pat->sids, pat->sids_size);
+                            matches++;
                         }
-                        matches++;
                     } else {
                         const SCACPatternList *pat = &pid_pat_list[pids[k]];
                         const int offset = i - pat->patlen + 1;
@@ -1040,8 +1040,8 @@ uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         } else {
                             bitarray[pids[k] / 8] |= (1 << (pids[k] % 8));
                             PrefilterAddSids(pmq, pat->sids, pat->sids_size);
+                            matches++;
                         }
-                        matches++;
                     }
                     //loop1:
                     //;
@@ -1343,7 +1343,6 @@ static int SCACTest06(void)
 
 static int SCACTest07(void)
 {
-    int result = 0;
     MpmCtx mpm_ctx;
     MpmThreadCtx mpm_thread_ctx;
     PrefilterRuleStore pmq;
@@ -1366,22 +1365,18 @@ static int SCACTest07(void)
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
                      30, 0, 0, 5, 0, 0);
     PmqSetup(&pmq);
-    /* total matches: 135 */
+    /* total matches: 135: unique matches: 6 */
 
     SCACPreparePatterns(&mpm_ctx);
 
     const char *buf = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
     uint32_t cnt = SCACSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
                                (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 135)
-        result = 1;
-    else
-        printf("135 != %" PRIu32 " ",cnt);
+    FAIL_IF_NOT(cnt == 6);
 
     SCACDestroyCtx(&mpm_ctx);
     PmqFree(&pmq);
-    return result;
+    PASS;
 }
 
 static int SCACTest08(void)

From 74ef5fc3d1a3abed60d80eeba3252886441c25eb Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 12:16:41 +0100
Subject: [PATCH 186/462] mpm/ac-ks: return only unique match count

Bring implementation in line with Hyperscan, which only counts unique matches.

Update test to reflect the new behavior.
---
 src/util-mpm-ac-ks.c | 18 +++---------------
 1 file changed, 3 insertions(+), 15 deletions(-)

diff --git a/src/util-mpm-ac-ks.c b/src/util-mpm-ac-ks.c
index b2f3ebc1afed..465b66918b62 100644
--- a/src/util-mpm-ac-ks.c
+++ b/src/util-mpm-ac-ks.c
@@ -1126,13 +1126,6 @@ static int CheckMatch(const SCACTileSearchCtx *ctx, PrefilterRuleStore *pmq,
         MpmPatternIndex pindex = patterns[k] & 0x0FFFFFFF;
         if (mpm_bitarray[pindex / 8] & (1 << (pindex % 8))) {
             /* Pattern already seen by this MPM. */
-            /* NOTE: This is faster then rechecking if it is a case-sensitive match
-             * since we know this pattern has already been seen, but incrementing
-             * matches here could over report matches. For example if the case-sensitive
-             * pattern is "Foo" and the string is "Foo bar foo", matches would be reported
-             * as 2, when it should really be 1, since "foo" is not a true match.
-             */
-            matches++;
             continue;
         }
         const SCACTilePatternList *pat = &pattern_list[pindex];
@@ -1613,7 +1606,6 @@ static int SCACTileTest06(void)
 
 static int SCACTileTest07(void)
 {
-    int result = 0;
     MpmCtx mpm_ctx;
     MpmThreadCtx mpm_thread_ctx;
     PrefilterRuleStore pmq;
@@ -1636,22 +1628,18 @@ static int SCACTileTest07(void)
     MpmAddPatternCS(&mpm_ctx, (uint8_t *)"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
                      30, 0, 0, 5, 0, 0);
     PmqSetup(&pmq);
-    /* total matches: 135 */
+    /* total matches: 135: 6 unique */
 
     SCACTilePreparePatterns(&mpm_ctx);
 
     const char *buf = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
     uint32_t cnt = SCACTileSearch(&mpm_ctx, &mpm_thread_ctx, &pmq,
                                   (uint8_t *)buf, strlen(buf));
-
-    if (cnt == 135)
-        result = 1;
-    else
-        printf("135 != %" PRIu32 " ",cnt);
+    FAIL_IF_NOT(cnt == 6);
 
     SCACTileDestroyCtx(&mpm_ctx);
     PmqFree(&pmq);
-    return result;
+    PASS;
 }
 
 static int SCACTileTest08(void)

From 2be36c0f0c51d5be13abb9596d600dfd98dc54e5 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 12:22:18 +0100
Subject: [PATCH 187/462] mpm: document Search callback return value

---
 src/util-mpm.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/util-mpm.h b/src/util-mpm.h
index 67988efd0314..d3ac12bdec89 100644
--- a/src/util-mpm.h
+++ b/src/util-mpm.h
@@ -159,6 +159,7 @@ typedef struct MpmTableElmt_ {
     int  (*AddPattern)(struct MpmCtx_ *, uint8_t *, uint16_t, uint16_t, uint16_t, uint32_t, SigIntId, uint8_t);
     int  (*AddPatternNocase)(struct MpmCtx_ *, uint8_t *, uint16_t, uint16_t, uint16_t, uint32_t, SigIntId, uint8_t);
     int  (*Prepare)(struct MpmCtx_ *);
+    /** \retval cnt number of patterns that matches: once per pattern max. */
     uint32_t (*Search)(const struct MpmCtx_ *, struct MpmThreadCtx_ *, PrefilterRuleStore *, const uint8_t *, uint32_t);
     void (*PrintCtx)(struct MpmCtx_ *);
     void (*PrintThreadCtx)(struct MpmThreadCtx_ *);

From 0b21b543a26116eb6b5e6cfa7b22fa8b828e3967 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 14:17:19 +0100
Subject: [PATCH 188/462] mpm/ac-bs: add warning if still used

Fall back to default matcher.

Ticket #6586.
---
 src/detect-engine-mpm.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/detect-engine-mpm.c b/src/detect-engine-mpm.c
index 48c4da115a45..849930a7a9cf 100644
--- a/src/detect-engine-mpm.c
+++ b/src/detect-engine-mpm.c
@@ -854,6 +854,9 @@ uint8_t PatternMatchDefaultMatcher(void)
 #endif
             if (strcmp("auto", mpm_algo) == 0) {
                 goto done;
+            } else if (strcmp("ac-bs", mpm_algo) == 0) {
+                SCLogWarning("mpm-algo \"ac-bs\" has been removed. See ticket #6586.");
+                goto done;
             }
             for (uint8_t u = 0; u < MPM_TABLE_SIZE; u++) {
                 if (mpm_table[u].name == NULL)

From 14c452e4c6af559c1203fedecaa0cc6008908e03 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 27 Nov 2023 09:52:15 +0100
Subject: [PATCH 189/462] mpm: cleanup pattern free function

Avoid redundant pointer checks; instead check once.
---
 src/util-mpm.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/src/util-mpm.c b/src/util-mpm.c
index 47a19b2c8e88..0ccb78897c09 100644
--- a/src/util-mpm.c
+++ b/src/util-mpm.c
@@ -356,30 +356,30 @@ static inline MpmPattern *MpmAllocPattern(MpmCtx *mpm_ctx)
  */
 void MpmFreePattern(MpmCtx *mpm_ctx, MpmPattern *p)
 {
-    if (p != NULL && p->cs != NULL && p->cs != p->ci) {
+    if (p == NULL)
+        return;
+
+    if (p->cs != NULL && p->cs != p->ci) {
         SCFree(p->cs);
         mpm_ctx->memory_cnt--;
         mpm_ctx->memory_size -= p->len;
     }
 
-    if (p != NULL && p->ci != NULL) {
+    if (p->ci != NULL) {
         SCFree(p->ci);
         mpm_ctx->memory_cnt--;
         mpm_ctx->memory_size -= p->len;
     }
 
-    if (p != NULL && p->original_pat != NULL) {
+    if (p->original_pat != NULL) {
         SCFree(p->original_pat);
         mpm_ctx->memory_cnt--;
         mpm_ctx->memory_size -= p->len;
     }
 
-    if (p != NULL) {
-        SCFree(p);
-        mpm_ctx->memory_cnt--;
-        mpm_ctx->memory_size -= sizeof(MpmPattern);
-    }
-    return;
+    SCFree(p);
+    mpm_ctx->memory_cnt--;
+    mpm_ctx->memory_size -= sizeof(MpmPattern);
 }
 
 static inline uint32_t MpmInitHash(MpmPattern *p)

From 3b3b0ed30a306860a22afdb9544199f8fbedb00c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 27 Nov 2023 10:05:55 +0100
Subject: [PATCH 190/462] mpm: free sids in MpmFreePattern as well

---
 src/util-mpm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/util-mpm.c b/src/util-mpm.c
index 0ccb78897c09..23ff23082885 100644
--- a/src/util-mpm.c
+++ b/src/util-mpm.c
@@ -377,6 +377,10 @@ void MpmFreePattern(MpmCtx *mpm_ctx, MpmPattern *p)
         mpm_ctx->memory_size -= p->len;
     }
 
+    if (p->sids != NULL) {
+        SCFree(p->sids);
+    }
+
     SCFree(p);
     mpm_ctx->memory_cnt--;
     mpm_ctx->memory_size -= sizeof(MpmPattern);

From 99c616389eabd88b5ff2fc58ef2c2d0718e9bbc0 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 27 Nov 2023 11:41:58 +0100
Subject: [PATCH 191/462] util/prefilter: constify sids

---
 src/util-prefilter.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/util-prefilter.h b/src/util-prefilter.h
index b956143114ab..67c24d6c9de7 100644
--- a/src/util-prefilter.h
+++ b/src/util-prefilter.h
@@ -58,8 +58,8 @@ int PrefilterAddSidsResize(PrefilterRuleStore *pmq, uint32_t new_size);
  *  \param sids_size number of Signature IDs in sids array.
  *
  */
-static inline void
-PrefilterAddSids(PrefilterRuleStore *pmq, SigIntId *sids, uint32_t sids_size)
+static inline void PrefilterAddSids(
+        PrefilterRuleStore *pmq, const SigIntId *sids, uint32_t sids_size)
 {
     if (sids_size == 0)
         return;

From 132fe57ac6befd46b72b63b602e57aa5a6ce5c9f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 28 Nov 2023 14:59:19 +0100
Subject: [PATCH 192/462] rust: add copyright header to common.rs

---
 rust/src/common.rs | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/rust/src/common.rs b/rust/src/common.rs
index 1d10bbe443d0..c874cc58b87d 100644
--- a/rust/src/common.rs
+++ b/rust/src/common.rs
@@ -1,3 +1,20 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
 //! Utility library module for commonly used strings, hexadecimals and other elements.
 
 use super::build_slice;

From 23d15259f5d24f4668359f0a3f13197c30e21dd2 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 7 Oct 2023 07:01:20 +0200
Subject: [PATCH 193/462] util/print: minor code cleanups

---
 src/util-print.c | 32 ++++++++++----------------------
 1 file changed, 10 insertions(+), 22 deletions(-)

diff --git a/src/util-print.c b/src/util-print.c
index ef69efe4b1ed..be87af1ccfda 100644
--- a/src/util-print.c
+++ b/src/util-print.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2010 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -43,9 +43,7 @@
  */
 void PrintBufferRawLineHex(char *nbuf, int *offset, int max_size, const uint8_t *buf, uint32_t buflen)
 {
-    uint32_t u = 0;
-
-    for (u = 0; u < buflen; u++) {
+    for (uint32_t u = 0; u < buflen; u++) {
         PrintBufferData(nbuf, offset, max_size, "%02X ", buf[u]);
     }
 }
@@ -63,9 +61,7 @@ void PrintBufferRawLineHex(char *nbuf, int *offset, int max_size, const uint8_t
 void PrintRawLineHexBuf(char *retbuf, uint32_t retbuflen, const uint8_t *buf, uint32_t buflen)
 {
     uint32_t offset = 0;
-    uint32_t u = 0;
-
-    for (u = 0; u < buflen; u++) {
+    for (uint32_t u = 0; u < buflen; u++) {
         PrintBufferData(retbuf, &offset, retbuflen, "%02X ", buf[u]);
     }
 }
@@ -75,9 +71,8 @@ void PrintRawJsonFp(FILE *fp, uint8_t *buf, uint32_t buflen)
 #define BUFFER_LENGTH 2048
     char nbuf[BUFFER_LENGTH] = "";
     uint32_t offset = 0;
-    uint32_t u = 0;
 
-    for (u = 0; u < buflen; u++) {
+    for (uint32_t u = 0; u < buflen; u++) {
         if (buf[u] == '\\' || buf[u] == '/' || buf[u] == '\"') {
             PrintBufferData(nbuf, &offset, BUFFER_LENGTH,
                              "\\%c", buf[u]);
@@ -97,9 +92,8 @@ void PrintRawUriFp(FILE *fp, uint8_t *buf, uint32_t buflen)
 #define BUFFER_LENGTH 2048
     char nbuf[BUFFER_LENGTH] = "";
     uint32_t offset = 0;
-    uint32_t u = 0;
 
-    for (u = 0; u < buflen; u++) {
+    for (uint32_t u = 0; u < buflen; u++) {
         if (isprint(buf[u]) && buf[u] != '\"') {
             if (buf[u] == '\\') {
                 PrintBufferData(nbuf, &offset, BUFFER_LENGTH,
@@ -120,9 +114,7 @@ void PrintRawUriFp(FILE *fp, uint8_t *buf, uint32_t buflen)
 void PrintRawUriBuf(char *retbuf, uint32_t *offset, uint32_t retbuflen,
                     uint8_t *buf, uint32_t buflen)
 {
-    uint32_t u = 0;
-
-    for (u = 0; u < buflen; u++) {
+    for (uint32_t u = 0; u < buflen; u++) {
         if (isprint(buf[u]) && buf[u] != '\"') {
             if (buf[u] == '\\') {
                 PrintBufferData(retbuf, offset, retbuflen,
@@ -143,13 +135,12 @@ void PrintRawUriBuf(char *retbuf, uint32_t *offset, uint32_t retbuflen,
 void PrintRawDataFp(FILE *fp, const uint8_t *buf, uint32_t buflen)
 {
     int ch = 0;
-    uint32_t u = 0;
 
     if (buf == NULL) {
         fprintf(fp, " (null)\n");
         return;
     }
-    for (u = 0; u < buflen; u+=16) {
+    for (uint32_t u = 0; u < buflen; u += 16) {
         fprintf(fp ," %04X  ", u);
         for (ch = 0; (u+ch) < buflen && ch < 16; ch++) {
              fprintf(fp, "%02X ", (uint8_t)buf[u+ch]);
@@ -182,9 +173,8 @@ void PrintRawDataToBuffer(uint8_t *dst_buf, uint32_t *dst_buf_offset_ptr, uint32
                           const uint8_t *src_buf, uint32_t src_buf_len)
 {
     int ch = 0;
-    uint32_t u = 0;
 
-    for (u = 0; u < src_buf_len; u+=16) {
+    for (uint32_t u = 0; u < src_buf_len; u += 16) {
         PrintBufferData((char *)dst_buf, dst_buf_offset_ptr, dst_buf_size,
                         " %04X  ", u);
         for (ch = 0; (u + ch) < src_buf_len && ch < 16; ch++) {
@@ -230,8 +220,7 @@ void PrintRawDataToBuffer(uint8_t *dst_buf, uint32_t *dst_buf_offset_ptr, uint32
 void PrintStringsToBuffer(uint8_t *dst_buf, uint32_t *dst_buf_offset_ptr, uint32_t dst_buf_size,
                           const uint8_t *src_buf, const uint32_t src_buf_len)
 {
-    uint32_t ch = 0;
-    for (ch = 0; ch < src_buf_len && *dst_buf_offset_ptr < dst_buf_size;
+    for (uint32_t ch = 0; ch < src_buf_len && *dst_buf_offset_ptr < dst_buf_size;
             ch++, (*dst_buf_offset_ptr)++) {
         if (isprint((uint8_t)src_buf[ch]) || src_buf[ch] == '\n' || src_buf[ch] == '\r') {
             dst_buf[*dst_buf_offset_ptr] = src_buf[ch];
@@ -250,7 +239,6 @@ void PrintStringsToBuffer(uint8_t *dst_buf, uint32_t *dst_buf_offset_ptr, uint32
 
 static const char *PrintInetIPv6(const void *src, char *dst, socklen_t size)
 {
-    int i;
     char s_part[6];
     uint16_t x[8];
     memcpy(&x, src, 16);
@@ -261,7 +249,7 @@ static const char *PrintInetIPv6(const void *src, char *dst, socklen_t size)
         return NULL;
     }
     memset(dst, 0, size);
-    for(i = 0; i < 8; i++) {
+    for (int i = 0; i < 8; i++) {
         snprintf(s_part, sizeof(s_part), "%04x:", htons(x[i]));
         strlcat(dst, s_part, size);
     }

From 529e02686b02b4569f852cf2b05ea89adbfa42b9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 20:40:42 +0200
Subject: [PATCH 194/462] detect/http.uri: modernize unittest

---
 src/tests/detect-http-uri.c | 89 +++++++------------------------------
 1 file changed, 15 insertions(+), 74 deletions(-)

diff --git a/src/tests/detect-http-uri.c b/src/tests/detect-http-uri.c
index f167be46cf75..8c60d430588e 100644
--- a/src/tests/detect-http-uri.c
+++ b/src/tests/detect-http-uri.c
@@ -2561,23 +2561,22 @@ static int UriTestSig26(void)
  */
 static int UriTestSig27(void)
 {
-    int result = 0;
     uint8_t *http_buf = (uint8_t *)"POST /we_need_to_fix_this_and_yes_fix_this_now HTTP/1.0\r\n"
         "User-Agent: Mozilla/1.0\r\n";
     uint32_t http_buf_len = strlen((char *)http_buf);
     Flow f;
     TcpSession ssn;
-    HtpState *http_state = NULL;
-    Packet *p = NULL;
     ThreadVars tv;
     DetectEngineThreadCtx *det_ctx = NULL;
     AppLayerParserThreadCtx *alp_tctx = AppLayerParserThreadCtxAlloc();
+    FAIL_IF_NULL(alp_tctx);
 
     memset(&tv, 0, sizeof(ThreadVars));
     memset(&f, 0, sizeof(Flow));
     memset(&ssn, 0, sizeof(TcpSession));
 
-    p = UTHBuildPacket(http_buf, http_buf_len, IPPROTO_TCP);
+    Packet *p = UTHBuildPacket(http_buf, http_buf_len, IPPROTO_TCP);
+    FAIL_IF_NULL(p);
 
     FLOW_INITIALIZE(&f);
     f.protoctx = (void *)&ssn;
@@ -2593,95 +2592,37 @@ static int UriTestSig27(void)
     StreamTcpInitConfig(true);
 
     DetectEngineCtx *de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL) {
-        goto end;
-    }
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
 
-    de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
-                               "(msg:\"test multiple relative uricontents\"; "
-                               "uricontent:\"fix_this\"; isdataat:!10,relative; sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        goto end;
-    }
+    Signature *s = DetectEngineAppendSig(de_ctx,
+            "alert tcp any any -> any any ("
+            "uricontent:\"fix_this\"; isdataat:!10,relative; sid:1;)");
+    FAIL_IF_NULL(s);
 
     SigGroupBuild(de_ctx);
     DetectEngineThreadCtxInit(&tv, (void *)de_ctx, (void *)&det_ctx);
 
     int r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, http_buf, http_buf_len);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: ", r);
-        goto end;
-    }
-
-    http_state = f.alstate;
-    if (http_state == NULL) {
-        printf("no http state: ");
-        goto end;
-    }
+    FAIL_IF_NOT(r == 0);
+    FAIL_IF_NULL(f.alstate);
 
     /* do detect */
     SigMatchSignatures(&tv, de_ctx, det_ctx, p);
 
-    if (!PacketAlertCheck(p, 1)) {
-        printf("sig 1 didn't alert, but it should have: ");
-        goto end;
-    }
-
-    result = 1;
+    FAIL_IF_NOT(PacketAlertCheck(p, 1));
 
-end:
-    if (alp_tctx != NULL)
-        AppLayerParserThreadCtxFree(alp_tctx);
-    if (det_ctx != NULL)
-        DetectEngineThreadCtxDeinit(&tv, det_ctx);
-    if (de_ctx != NULL)
-        SigGroupCleanup(de_ctx);
-    if (de_ctx != NULL)
-        DetectEngineCtxFree(de_ctx);
+    AppLayerParserThreadCtxFree(alp_tctx);
+    DetectEngineThreadCtxDeinit(&tv, det_ctx);
+    DetectEngineCtxFree(de_ctx);
 
     StreamTcpFreeConfig(true);
     FLOW_DESTROY(&f);
     UTHFreePacket(p);
-    return result;
+    PASS;
 }
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
 static int UriTestSig28(void)
 {
     int result = 0;

From b69f4cb5cf964114103812fa8370530d2260fbeb Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 5 Oct 2023 10:25:16 +0200
Subject: [PATCH 195/462] detect/pcre: match data is const at match time

---
 src/detect-pcre.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect-pcre.c b/src/detect-pcre.c
index 913d782f4afe..848f6b9680fe 100644
--- a/src/detect-pcre.c
+++ b/src/detect-pcre.c
@@ -79,7 +79,7 @@ static int pcre2_use_jit = 1;
 
 /* \brief Helper function for using pcre2_match with/without JIT
  */
-static inline int DetectPcreExec(DetectEngineThreadCtx *det_ctx, DetectPcreData *pd,
+static inline int DetectPcreExec(DetectEngineThreadCtx *det_ctx, const DetectPcreData *pd,
         const char *str, const size_t strlen, int start_offset, int options,
         pcre2_match_data *match)
 {
@@ -182,7 +182,7 @@ int DetectPcrePayloadMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     uint32_t len = 0;
     PCRE2_SIZE capture_len = 0;
 
-    DetectPcreData *pe = (DetectPcreData *)smd->ctx;
+    const DetectPcreData *pe = (const DetectPcreData *)smd->ctx;
 
     if (pe->flags & DETECT_PCRE_RELATIVE) {
         ptr = payload + det_ctx->buffer_offset;

From 6cf0e4d604bc30451b84eba3bc222e8fe6696fc6 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 26 Sep 2023 15:55:09 +0200
Subject: [PATCH 196/462] detect/content-inspect: keyword context as const

---
 src/detect-engine-content-inspection.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 2ac2319c9d53..4569cd713981 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -437,7 +437,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
     } else if (smd->type == DETECT_PCRE) {
         SCLogDebug("inspecting pcre");
-        DetectPcreData *pe = (DetectPcreData *)smd->ctx;
+        const DetectPcreData *pe = (const DetectPcreData *)smd->ctx;
         uint32_t prev_buffer_offset = det_ctx->buffer_offset;
         uint32_t prev_offset = 0;
 
@@ -473,7 +473,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
         } while (1);
 
     } else if (smd->type == DETECT_BYTETEST) {
-        DetectBytetestData *btd = (DetectBytetestData *)smd->ctx;
+        const DetectBytetestData *btd = (const DetectBytetestData *)smd->ctx;
         uint16_t btflags = btd->flags;
         int32_t offset = btd->offset;
         uint64_t value = btd->value;
@@ -505,7 +505,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
         goto match;
 
     } else if (smd->type == DETECT_BYTEJUMP) {
-        DetectBytejumpData *bjd = (DetectBytejumpData *)smd->ctx;
+        const DetectBytejumpData *bjd = (const DetectBytejumpData *)smd->ctx;
         uint16_t bjflags = bjd->flags;
         int32_t offset = bjd->offset;
         int32_t nbytes;
@@ -538,7 +538,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
     } else if (smd->type == DETECT_BYTE_EXTRACT) {
 
-        DetectByteExtractData *bed = (DetectByteExtractData *)smd->ctx;
+        const DetectByteExtractData *bed = (const DetectByteExtractData *)smd->ctx;
         uint8_t endian = bed->endian;
 
         /* if we have dce enabled we will have to use the endianness
@@ -602,7 +602,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
     } else if (smd->type == DETECT_BSIZE) {
 
-        bool eof = (flags & DETECT_CI_FLAGS_END);
+        const bool eof = (flags & DETECT_CI_FLAGS_END);
         const uint64_t data_size = buffer_len + stream_start_offset;
         int r = DetectBsizeMatch(smd->ctx, data_size, eof);
         if (r < 0) {
@@ -635,8 +635,8 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
     } else if (smd->type == DETECT_AL_URILEN) {
         SCLogDebug("inspecting uri len");
 
-        int r = 0;
-        DetectUrilenData *urilend = (DetectUrilenData *) smd->ctx;
+        int r;
+        const DetectUrilenData *urilend = (const DetectUrilenData *)smd->ctx;
         if (buffer_len > UINT16_MAX) {
             r = DetectU16Match(UINT16_MAX, &urilend->du16);
         } else {

From fa450c0531ca4df0a319901826b257ec5ecfd543 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 5 Oct 2023 15:33:05 +0200
Subject: [PATCH 197/462] detect: modernize unittest

---
 src/tests/detect.c | 94 +++++++++++++---------------------------------
 1 file changed, 26 insertions(+), 68 deletions(-)

diff --git a/src/tests/detect.c b/src/tests/detect.c
index 37dc2cfe9bf7..08bf9997536b 100644
--- a/src/tests/detect.c
+++ b/src/tests/detect.c
@@ -3174,12 +3174,8 @@ static int SigTest38(void)
 
 static int SigTest39(void)
 {
-    Packet *p1 = PacketGetFromAlloc();
-    if (unlikely(p1 == NULL))
-        return 0;
     ThreadVars th_v;
     DetectEngineThreadCtx *det_ctx = NULL;
-    int result = 1;
     uint8_t raw_eth[] = {
         0x00, 0x00, 0x03, 0x04, 0x00, 0x06, 0x00,
         0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -3216,23 +3212,13 @@ static int SigTest39(void)
 
     memset(&th_v, 0, sizeof(ThreadVars));
 
+    Packet *p1 = PacketGetFromAlloc();
+    FAIL_IF_NULL(p1);
     /* Copy raw data into packet */
-    if (PacketCopyData(p1, raw_eth, ethlen) == -1) {
-        SCFree(p1);
-        return 1;
-    }
-    if (PacketCopyDataOffset(p1, ethlen, raw_ipv4, ipv4len) == -1) {
-        SCFree(p1);
-        return 1;
-    }
-    if (PacketCopyDataOffset(p1, ethlen + ipv4len, raw_tcp, tcplen) == -1) {
-        SCFree(p1);
-        return 1;
-    }
-    if (PacketCopyDataOffset(p1, ethlen + ipv4len + tcplen, buf, buflen) == -1) {
-        SCFree(p1);
-        return 1;
-    }
+    FAIL_IF(PacketCopyData(p1, raw_eth, ethlen) == -1);
+    FAIL_IF(PacketCopyDataOffset(p1, ethlen, raw_ipv4, ipv4len) == -1);
+    FAIL_IF(PacketCopyDataOffset(p1, ethlen + ipv4len, raw_tcp, tcplen) == -1);
+    FAIL_IF(PacketCopyDataOffset(p1, ethlen + ipv4len + tcplen, buf, buflen) == -1);
     SET_PKT_LEN(p1, ethlen + ipv4len + tcplen + buflen);
 
     PACKET_RESET_CHECKSUMS(p1);
@@ -3246,64 +3232,36 @@ static int SigTest39(void)
     p1->proto = IPPROTO_TCP;
 
     DetectEngineCtx *de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL) {
-        goto end;
-    }
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
 
-    de_ctx->sig_list = SigInit(de_ctx,
-                               "alert tcp any any -> any any "
-                               "(content:\"LEN1|20|\"; "
-                               "byte_test:4,=,8,0; "
-                               "byte_jump:4,0; "
-                               "byte_test:6,=,0x4c454e312038,0,relative; "
-                               "msg:\"byte_jump keyword check(1)\"; sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        result &= 0;
-        goto end;
-    }
-    // XXX TODO
-    de_ctx->sig_list->next = SigInit(de_ctx,
-                               "alert tcp any any -> any any "
-                               "(content:\"LEN1|20|\"; "
-                               "byte_test:4,=,8,4,relative,string,dec; "
-                               "byte_jump:4,4,relative,string,dec,post_offset 2; "
-                               "byte_test:4,=,0x4c454e32,0,relative; "
-                               "msg:\"byte_jump keyword check(2)\"; sid:2;)");
-    if (de_ctx->sig_list->next == NULL) {
-        result &= 0;
-        goto end;
-    }
-
+    Signature *s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any "
+                                                 "(content:\"LEN1|20|\"; "
+                                                 "byte_test:4,=,8,0; "
+                                                 "byte_jump:4,0; "
+                                                 "byte_test:6,=,0x4c454e312038,0,relative; "
+                                                 "msg:\"byte_jump keyword check(1)\"; sid:1;)");
+    FAIL_IF_NULL(s);
+    s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any "
+                                      "(content:\"LEN1|20|\"; "
+                                      "byte_test:4,=,8,4,relative,string,dec; "
+                                      "byte_jump:4,4,relative,string,dec,post_offset 2; "
+                                      "byte_test:4,=,0x4c454e32,0,relative; "
+                                      "msg:\"byte_jump keyword check(2)\"; sid:2;)");
+    FAIL_IF_NULL(s);
     SigGroupBuild(de_ctx);
     DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
 
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);
-    if (PacketAlertCheck(p1, 1)) {
-        result = 1;
-    } else {
-        result = 0;
-        printf("sid 1 didn't alert, but should have: ");
-        goto cleanup;
-    }
-    if (PacketAlertCheck(p1, 2)) {
-        result = 1;
-    } else {
-        result = 0;
-        printf("sid 2 didn't alert, but should have: ");
-        goto cleanup;
-    }
 
-cleanup:
-    SigGroupCleanup(de_ctx);
-    SigCleanSignatures(de_ctx);
-    if (det_ctx != NULL)
-        DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
+    FAIL_IF_NOT(PacketAlertCheck(p1, 1));
+    FAIL_IF_NOT(PacketAlertCheck(p1, 2));
+
+    DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
     DetectEngineCtxFree(de_ctx);
 
-end:
     SCFree(p1);
-    return result;
+    PASS;
 }
 
 /**

From 474a89e09868fe7262b0b5e1ff201ff36ea2bb1f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 5 Oct 2023 16:24:15 +0200
Subject: [PATCH 198/462] detect/file.data: modernize test

---
 src/tests/detect-http-server-body.c | 79 +++++++++--------------------
 1 file changed, 25 insertions(+), 54 deletions(-)

diff --git a/src/tests/detect-http-server-body.c b/src/tests/detect-http-server-body.c
index 29340fb4aa77..89180fe56b98 100644
--- a/src/tests/detect-http-server-body.c
+++ b/src/tests/detect-http-server-body.c
@@ -3150,7 +3150,6 @@ static int DetectEngineHttpServerBodyFileDataTest03(void)
     Packet *p1 = NULL;
     Packet *p2 = NULL;
     ThreadVars th_v;
-    DetectEngineCtx *de_ctx = NULL;
     DetectEngineThreadCtx *det_ctx = NULL;
     HtpState *http_state = NULL;
     Flow f;
@@ -3167,7 +3166,6 @@ static int DetectEngineHttpServerBodyFileDataTest03(void)
         "\r\n"
         "XYZ_klm_1234abcd_XYZ_klm_5678abcd";
     uint32_t http_len2 = sizeof(http_buf2) - 1;
-    int result = 0;
     AppLayerParserThreadCtx *alp_tctx = AppLayerParserThreadCtxAlloc();
 
     memset(&th_v, 0, sizeof(th_v));
@@ -3194,82 +3192,55 @@ static int DetectEngineHttpServerBodyFileDataTest03(void)
 
     StreamTcpInitConfig(true);
 
-    de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-
+    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
 
-    if (!(DetectEngineAppendSig(de_ctx, "alert http any any -> any any "
-                               "(msg:\"match on 1st\"; "
-                               "file_data; content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; distance:4; byte_test:4,=,1234,-8,relative,string;"
-                               "sid:1;)")))
-        goto end;
-    if (!(DetectEngineAppendSig(de_ctx, "alert http any any -> any any "
-                               "(msg:\"match on 2nd\"; "
-                               "file_data; content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; distance:4; byte_test:4,=,5678,-8,relative,string;"
-                               "sid:2;)")))
-        goto end;
+    Signature *s = DetectEngineAppendSig(de_ctx,
+            "alert http any any -> any any "
+            "(msg:\"match on 1st\"; "
+            "file_data; content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; "
+            "distance:4; byte_test:4,=,1234,-8,relative,string;"
+            "sid:1;)");
+    FAIL_IF_NULL(s);
+    s = DetectEngineAppendSig(de_ctx,
+            "alert http any any -> any any "
+            "(msg:\"match on 2nd\"; "
+            "file_data; content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; "
+            "distance:4; byte_test:4,=,5678,-8,relative,string;"
+            "sid:2;)");
+    FAIL_IF_NULL(s);
 
     SigGroupBuild(de_ctx);
     DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
 
     int r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, http_buf1, http_len1);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: ", r);
-        result = 0;
-        goto end;
-    }
-
+    FAIL_IF(r != 0);
     http_state = f.alstate;
-    if (http_state == NULL) {
-        printf("no http state: \n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(http_state);
 
     /* do detect */
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);
-
-    if (PacketAlertCheck(p1, 1)) {
-        printf("sid 1 matched but shouldn't have: ");
-        goto end;
-    }
+    FAIL_IF(PacketAlertCheck(p1, 1));
 
     r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOCLIENT, http_buf2, http_len2);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: \n", r);
-        result = 0;
-        goto end;
-    }
+    FAIL_IF(r != 0);
 
     /* do detect */
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p2);
 
-    if (!PacketAlertCheck(p2, 1)) {
-        printf("sid 1 did not match but should have: ");
-        goto end;
-    }
-    if (!PacketAlertCheck(p2, 2)) {
-        printf("sid 2 did not match but should have: ");
-        goto end;
-    }
-
-    result = 1;
-
-end:
-    if (alp_tctx != NULL)
-        AppLayerParserThreadCtxFree(alp_tctx);
-    if (de_ctx != NULL)
-        DetectEngineCtxFree(de_ctx);
+    FAIL_IF_NOT(PacketAlertCheck(p2, 1));
+    FAIL_IF_NOT(PacketAlertCheck(p2, 2));
 
+    AppLayerParserThreadCtxFree(alp_tctx);
+    DetectEngineCtxFree(de_ctx);
     StreamTcpFreeConfig(true);
     FLOW_DESTROY(&f);
     UTHFreePackets(&p1, 1);
     UTHFreePackets(&p2, 1);
-    return result;
+    PASS;
 }
 
 static int DetectEngineHttpServerBodyFileDataTest04(void)

From 83c4de4cee9e54ba5bd01338b6a348f6a3e32c86 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 5 Oct 2023 15:46:18 +0200
Subject: [PATCH 199/462] detect/bytejump: fix debug messages

Remove newlines.
---
 src/detect-bytejump.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 37c01ed8c30b..3e7ae4f5e000 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -222,14 +222,14 @@ bool DetectBytejumpDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
     /* Calculate the jump location */
     if (flags & DETECT_BYTEJUMP_BEGIN) {
         jumpptr = payload + (int64_t)val;
-        SCLogDebug("NEWVAL: payload %p + %" PRIi64 " = %p\n", payload, (int64_t)val, jumpptr + val);
+        SCLogDebug("NEWVAL: payload %p + %" PRIi64 " = %p", payload, (int64_t)val, jumpptr + val);
     } else if (flags & DETECT_BYTEJUMP_END) {
         jumpptr = payload + payload_len + (int64_t)val;
         SCLogDebug(
                 "NEWVAL: payload %p + %" PRIu32 " + %" PRIi64, payload, payload_len, (int64_t)val);
     } else {
         jumpptr = ptr + (int64_t)val + extbytes;
-        SCLogDebug("NEWVAL: ptr %p + %" PRIi64 " = %p\n", ptr, val, jumpptr);
+        SCLogDebug("NEWVAL: ptr %p + %" PRIi64 " = %p", ptr, val, jumpptr);
     }
 
     /* Validate that the jump location is still in the packet

From 65560ad8ca1e8a6aeb927382f3af9cc304690e7c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 5 Oct 2023 16:15:28 +0200
Subject: [PATCH 200/462] detect/content: test cleanup

---
 src/detect-content.c | 40 ++++++++--------------------------------
 1 file changed, 8 insertions(+), 32 deletions(-)

diff --git a/src/detect-content.c b/src/detect-content.c
index 5f34ffd13df1..c7a240d7e01d 100644
--- a/src/detect-content.c
+++ b/src/detect-content.c
@@ -1163,16 +1163,12 @@ static int DetectContentParseTest08 (void)
 static int DetectContentLongPatternMatchTest(uint8_t *raw_eth_pkt, uint16_t pktsize, const char *sig,
                       uint32_t sid)
 {
-    int result = 0;
-
     Packet *p = PacketGetFromAlloc();
-    if (unlikely(p == NULL))
-        return 0;
+    FAIL_IF_NULL(p);
     DecodeThreadVars dtv;
 
     ThreadVars th_v;
     DetectEngineThreadCtx *det_ctx = NULL;
-
     memset(&dtv, 0, sizeof(DecodeThreadVars));
     memset(&th_v, 0, sizeof(th_v));
 
@@ -1180,26 +1176,17 @@ static int DetectContentLongPatternMatchTest(uint8_t *raw_eth_pkt, uint16_t pkts
     DecodeEthernet(&th_v, &dtv, p, raw_eth_pkt, pktsize);
 
     DetectEngineCtx *de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL) {
-        goto end;
-    }
-
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
 
-    de_ctx->sig_list = SigInit(de_ctx, sig);
-    if (de_ctx->sig_list == NULL) {
-        goto end;
-    }
-    de_ctx->sig_list->next = NULL;
+    Signature *s = DetectEngineAppendSig(de_ctx, sig);
+    FAIL_IF_NULL(s);
 
     if (de_ctx->sig_list->init_data->smlists_tail[DETECT_SM_LIST_PMATCH]->type == DETECT_CONTENT) {
         DetectContentData *co = (DetectContentData *)de_ctx->sig_list->init_data
                                         ->smlists_tail[DETECT_SM_LIST_PMATCH]
                                         ->ctx;
-        if (co->flags & DETECT_CONTENT_RELATIVE_NEXT) {
-            printf("relative next flag set on final match which is content: ");
-            goto end;
-        }
+        FAIL_IF(co->flags & DETECT_CONTENT_RELATIVE_NEXT);
     }
 
     SCLogDebug("---DetectContentLongPatternMatchTest---");
@@ -1209,23 +1196,12 @@ static int DetectContentLongPatternMatchTest(uint8_t *raw_eth_pkt, uint16_t pkts
     DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
 
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p);
-    if (PacketAlertCheck(p, sid) != 1) {
-        goto end;
-    }
+    int result = PacketAlertCheck(p, sid);
 
-    result = 1;
-end:
-    if (de_ctx != NULL)
-    {
-        SigGroupCleanup(de_ctx);
-        SigCleanSignatures(de_ctx);
-        if (det_ctx != NULL)
-            DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
-        DetectEngineCtxFree(de_ctx);
-    }
+    DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
+    DetectEngineCtxFree(de_ctx);
     PacketRecycle(p);
     FlowShutdown();
-
     SCFree(p);
     return result;
 }

From 15b545d16fd827f0ac4a5b48bf88f371ecc8e22f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 12 Oct 2023 15:31:35 +0200
Subject: [PATCH 201/462] detect: improve explanation of offset tracking

---
 src/detect.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/detect.h b/src/detect.h
index cdc098368fc4..e30bac279a06 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1103,10 +1103,11 @@ typedef struct DetectEngineThreadCtx_ {
 
     uint64_t raw_stream_progress;
 
-    /** offset into the payload of the last match by:
-     *  content, pcre, etc */
+    /** offset into the payload of the end of the last match by: content, pcre, etc */
     uint32_t buffer_offset;
-    /* used by pcre match function alone */
+
+    /** used by pcre match function alone: normally in sync with buffer_offset, but
+     *  points to 1 byte after the start of the last pcre match if a pcre match happened. */
     uint32_t pcre_match_start_offset;
 
     /* counter for the filestore array below -- up here for cache reasons. */

From 643f25280bdf01e44566bda23c328f43b6fe6911 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 27 Sep 2023 12:23:23 +0200
Subject: [PATCH 202/462] detect/app-layer-events: constify arguments; minor
 cleanups

---
 src/app-layer-events.h       | 7 +++----
 src/detect-app-layer-event.c | 8 ++++----
 2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/src/app-layer-events.h b/src/app-layer-events.h
index 83cb0d9ba471..8b0dc8276d61 100644
--- a/src/app-layer-events.h
+++ b/src/app-layer-events.h
@@ -59,15 +59,14 @@ int AppLayerGetEventInfoById(int event_id, const char **event_name,
                              AppLayerEventType *event_type);
 void AppLayerDecoderEventsSetEventRaw(AppLayerDecoderEvents **sevents, uint8_t event);
 
-static inline int AppLayerDecoderEventsIsEventSet(AppLayerDecoderEvents *devents,
-                                                  uint8_t event)
+static inline int AppLayerDecoderEventsIsEventSet(
+        const AppLayerDecoderEvents *devents, uint8_t event)
 {
     if (devents == NULL)
         return 0;
 
-    int i;
     int cnt = devents->cnt;
-    for (i = 0; i < cnt; i++) {
+    for (int i = 0; i < cnt; i++) {
         if (devents->events[i] == event)
             return 1;
     }
diff --git a/src/detect-app-layer-event.c b/src/detect-app-layer-event.c
index d5eb0117218e..9c323359b577 100644
--- a/src/detect-app-layer-event.c
+++ b/src/detect-app-layer-event.c
@@ -92,14 +92,14 @@ static uint8_t DetectEngineAptEventInspect(DetectEngineCtx *de_ctx, DetectEngine
 {
     int r = 0;
     const AppProto alproto = f->alproto;
-    AppLayerDecoderEvents *decoder_events =
-        AppLayerParserGetEventsByTx(f->proto, alproto, tx);
+    const AppLayerDecoderEvents *decoder_events =
+            AppLayerParserGetEventsByTx(f->proto, alproto, tx);
     if (decoder_events == NULL) {
         goto end;
     }
-    SigMatchData *smd = engine->smd;
+    const SigMatchData *smd = engine->smd;
     while (1) {
-        DetectAppLayerEventData *aled = (DetectAppLayerEventData *)smd->ctx;
+        const DetectAppLayerEventData *aled = (const DetectAppLayerEventData *)smd->ctx;
         KEYWORD_PROFILING_START;
 
         if (AppLayerDecoderEventsIsEventSet(decoder_events, aled->event_id)) {

From 8ba7f23c9bbb7de823df04643cb1acf66db76831 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Sep 2023 10:14:11 +0200
Subject: [PATCH 203/462] detect/content: use const pointer where possible

---
 src/detect-engine-content-inspection.c | 3 +--
 src/detect-replace.c                   | 5 ++---
 src/detect-replace.h                   | 3 ++-
 src/detect.h                           | 2 +-
 4 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 4569cd713981..242ad03fa120 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -126,8 +126,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
     /* \todo unify this which is phase 2 of payload inspection unification */
     if (smd->type == DETECT_CONTENT) {
-
-        DetectContentData *cd = (DetectContentData *)smd->ctx;
+        const DetectContentData *cd = (const DetectContentData *)smd->ctx;
         SCLogDebug("inspecting content %"PRIu32" buffer_len %"PRIu32, cd->id, buffer_len);
 
         /* we might have already have this content matched by the mpm.
diff --git a/src/detect-replace.c b/src/detect-replace.c
index 147c3e94d8b1..e70a2e1677a4 100644
--- a/src/detect-replace.c
+++ b/src/detect-replace.c
@@ -176,9 +176,8 @@ int DetectReplaceSetup(DetectEngineCtx *de_ctx, Signature *s, const char *replac
  * earlier changes. Thus the highest priority modifications should be
  * applied last.
  */
-DetectReplaceList *DetectReplaceAddToList(DetectReplaceList *replist,
-                                          uint8_t *found,
-                                          DetectContentData *cd)
+DetectReplaceList *DetectReplaceAddToList(
+        DetectReplaceList *replist, uint8_t *found, const DetectContentData *cd)
 {
     DetectReplaceList *newlist;
 
diff --git a/src/detect-replace.h b/src/detect-replace.h
index c444188c0948..4944420527e8 100644
--- a/src/detect-replace.h
+++ b/src/detect-replace.h
@@ -26,7 +26,8 @@
 
 #include "detect-content.h"
 
-DetectReplaceList * DetectReplaceAddToList(DetectReplaceList *replist, uint8_t *found, DetectContentData *cd);
+DetectReplaceList *DetectReplaceAddToList(
+        DetectReplaceList *replist, uint8_t *found, const DetectContentData *cd);
 
 /* Internal functions are only called via the inline functions below. */
 void DetectReplaceExecuteInternal(Packet *p, DetectReplaceList *replist);
diff --git a/src/detect.h b/src/detect.h
index e30bac279a06..53ca2b0931f6 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -721,7 +721,7 @@ typedef struct DetectPatternTracker {
 } DetectPatternTracker;
 
 typedef struct DetectReplaceList_ {
-    struct DetectContentData_ *cd;
+    const struct DetectContentData_ *cd;
     uint8_t *found;
     struct DetectReplaceList_ *next;
 } DetectReplaceList;

From 3d7e0927bfb27c8e114032516f55297b56563d2a Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Sep 2023 09:48:56 +0200
Subject: [PATCH 204/462] detect/content: minor code/comment cleanups

---
 src/detect-engine-content-inspection.c | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 242ad03fa120..90e55c86f450 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2017 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -68,16 +68,6 @@
 /**
  * \brief Run the actual payload match functions
  *
- * The following keywords are inspected:
- * - content, including all the http and dce modified contents
- * - isdataat
- * - pcre
- * - bytejump
- * - bytetest
- * - byte_extract
- * - urilen
- * -
- *
  * All keywords are evaluated against the buffer with buffer_len.
  *
  * For accounting the last match in relative matching the
@@ -124,7 +114,6 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
         SCReturnInt(0);
     }
 
-    /* \todo unify this which is phase 2 of payload inspection unification */
     if (smd->type == DETECT_CONTENT) {
         const DetectContentData *cd = (const DetectContentData *)smd->ctx;
         SCLogDebug("inspecting content %"PRIu32" buffer_len %"PRIu32, cd->id, buffer_len);

From a3ac3e69d613cf8b49c6f4e2e796642ea58bfb6d Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Sep 2023 10:14:23 +0200
Subject: [PATCH 205/462] detect/replace: minor code cleanup

---
 src/detect-replace.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/src/detect-replace.c b/src/detect-replace.c
index e70a2e1677a4..a5d0387457e5 100644
--- a/src/detect-replace.c
+++ b/src/detect-replace.c
@@ -179,24 +179,20 @@ int DetectReplaceSetup(DetectEngineCtx *de_ctx, Signature *s, const char *replac
 DetectReplaceList *DetectReplaceAddToList(
         DetectReplaceList *replist, uint8_t *found, const DetectContentData *cd)
 {
-    DetectReplaceList *newlist;
-
     if (cd->content_len != cd->replace_len)
         return NULL;
     SCLogDebug("replace: Adding match");
 
-    newlist = SCMalloc(sizeof(DetectReplaceList));
+    DetectReplaceList *newlist = SCMalloc(sizeof(DetectReplaceList));
     if (unlikely(newlist == NULL))
         return replist;
     newlist->found = found;
     newlist->cd = cd;
     /* Push new value onto the front of the list. */
     newlist->next = replist;
-
     return newlist;
 }
 
-
 void DetectReplaceExecuteInternal(Packet *p, DetectReplaceList *replist)
 {
     DetectReplaceList *tlist = NULL;

From 9639da32b7f8cd99d9d2429e82c742f31804b068 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 24 Sep 2023 07:26:24 +0200
Subject: [PATCH 206/462] detect/content-inspect: minor code cleanups

---
 src/detect-engine-content-inspection.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 90e55c86f450..1ec78fb550bb 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -141,10 +141,8 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
         do {
             if ((cd->flags & DETECT_CONTENT_DISTANCE) ||
                 (cd->flags & DETECT_CONTENT_WITHIN)) {
-                SCLogDebug("det_ctx->buffer_offset %"PRIu32, det_ctx->buffer_offset);
-
+                SCLogDebug("det_ctx->buffer_offset %" PRIu32, det_ctx->buffer_offset);
                 offset = prev_buffer_offset;
-                depth = buffer_len;
 
                 int distance = cd->distance;
                 if (cd->flags & DETECT_CONTENT_DISTANCE) {
@@ -266,7 +264,6 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
             const uint8_t *sbuffer = buffer + offset;
             uint32_t sbuffer_len = depth - offset;
-            uint32_t match_offset = 0;
             SCLogDebug("sbuffer_len %" PRIu32 " depth: %" PRIu32 ", buffer_len: %" PRIu32,
                     sbuffer_len, depth, buffer_len);
 #ifdef DEBUG
@@ -299,7 +296,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
                     goto match;
                 }
             } else {
-                match_offset = (uint32_t)((found - buffer) + cd->content_len);
+                uint32_t match_offset = (uint32_t)((found - buffer) + cd->content_len);
                 if (cd->flags & DETECT_CONTENT_NEGATED) {
                     SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32
                                ", but negated so no match",

From 2a4fd85d1d2ab27234e1e4fecdd33afc00af0b87 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 1 Dec 2023 09:47:13 +0100
Subject: [PATCH 207/462] flow/timeout: use const TcpSession; cleanup
 prototypes

---
 src/flow-timeout.c | 13 +++----------
 src/flow-timeout.h |  3 +++
 src/flow-worker.c  |  4 ++--
 3 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/src/flow-timeout.c b/src/flow-timeout.c
index 6a9b707c2186..90a97fa66688 100644
--- a/src/flow-timeout.c
+++ b/src/flow-timeout.c
@@ -76,10 +76,8 @@
  *                  packets need to force reassembly, in which case we just
  *                  set dummy ack/seq values.
  */
-static inline Packet *FlowForceReassemblyPseudoPacketSetup(Packet *p,
-                                                           int direction,
-                                                           Flow *f,
-                                                           TcpSession *ssn)
+static inline Packet *FlowForceReassemblyPseudoPacketSetup(
+        Packet *p, int direction, Flow *f, const TcpSession *ssn)
 {
     const int orig_dir = direction;
     p->tenant_id = f->tenant_id;
@@ -265,12 +263,7 @@ static inline Packet *FlowForceReassemblyPseudoPacketSetup(Packet *p,
     return NULL;
 }
 
-Packet *FlowForceReassemblyPseudoPacketGet(int direction,
-                                                         Flow *f,
-                                                         TcpSession *ssn);
-Packet *FlowForceReassemblyPseudoPacketGet(int direction,
-                                                         Flow *f,
-                                                         TcpSession *ssn)
+Packet *FlowForceReassemblyPseudoPacketGet(int direction, Flow *f, const TcpSession *ssn)
 {
     PacketPoolWait();
     Packet *p = PacketPoolGetPacket();
diff --git a/src/flow-timeout.h b/src/flow-timeout.h
index 7426d59d60c6..882f4e63b201 100644
--- a/src/flow-timeout.h
+++ b/src/flow-timeout.h
@@ -24,8 +24,11 @@
 #ifndef __FLOW_TIMEOUT_H__
 #define __FLOW_TIMEOUT_H__
 
+#include "stream-tcp-private.h"
+
 void FlowForceReassemblyForFlow(Flow *f);
 int FlowForceReassemblyNeedReassembly(Flow *f);
 void FlowForceReassembly(void);
+Packet *FlowForceReassemblyPseudoPacketGet(int direction, Flow *f, const TcpSession *ssn);
 
 #endif /* __FLOW_TIMEOUT_H__ */
diff --git a/src/flow-worker.c b/src/flow-worker.c
index a20e053c59c9..3baa8ad7cbc5 100644
--- a/src/flow-worker.c
+++ b/src/flow-worker.c
@@ -96,8 +96,8 @@ typedef struct FlowWorkerThreadData_ {
 
 } FlowWorkerThreadData;
 
-static void FlowWorkerFlowTimeout(ThreadVars *tv, Packet *p, FlowWorkerThreadData *fw, void *detect_thread);
-Packet *FlowForceReassemblyPseudoPacketGet(int direction, Flow *f, TcpSession *ssn);
+static void FlowWorkerFlowTimeout(
+        ThreadVars *tv, Packet *p, FlowWorkerThreadData *fw, void *detect_thread);
 
 /**
  * \internal

From ea4503c3e3be351eddc1205a8aec01c8c301fdfa Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 11:58:03 +0200
Subject: [PATCH 208/462] flow/timeout: use single packet for timeout handling

In the FlowFinish logic, one or two pseudo packets are used to finish flow
handling. In the case of 2 (one per direction), the logic first set up the
2 packets, then it would process them one by one. This lead to poor cache
locality.

This patch processes the first packet entirely first, followed by the second
packet.
---
 src/flow-worker.c | 58 +++++++++++++++++++++++------------------------
 1 file changed, 28 insertions(+), 30 deletions(-)

diff --git a/src/flow-worker.c b/src/flow-worker.c
index 3baa8ad7cbc5..5c03689a4573 100644
--- a/src/flow-worker.c
+++ b/src/flow-worker.c
@@ -111,7 +111,6 @@ static void FlowWorkerFlowTimeout(
  */
 static int FlowFinish(ThreadVars *tv, Flow *f, FlowWorkerThreadData *fw, void *detect_thread)
 {
-    Packet *p1 = NULL, *p2 = NULL;
     const int server = f->ffr_tc;
     const int client = f->ffr_ts;
 
@@ -128,47 +127,46 @@ static int FlowFinish(ThreadVars *tv, Flow *f, FlowWorkerThreadData *fw, void *d
 
     /* insert a pseudo packet in the toserver direction */
     if (client == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
-        p1 = FlowForceReassemblyPseudoPacketGet(0, f, ssn);
-        if (p1 == NULL) {
+        Packet *p = FlowForceReassemblyPseudoPacketGet(0, f, ssn);
+        if (unlikely(p == NULL)) {
             return 0;
         }
-        PKT_SET_SRC(p1, PKT_SRC_FFR);
+        PKT_SET_SRC(p, PKT_SRC_FFR);
+        if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NONE) {
+            p->flowflags |= FLOW_PKT_LAST_PSEUDO;
+        }
+        FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
+        PacketPoolReturnPacket(p);
 
         if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
-            p2 = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
-            if (p2 == NULL) {
-                FlowDeReference(&p1->flow);
-                TmqhOutputPacketpool(NULL, p1);
+            p = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
+            if (unlikely(p == NULL)) {
                 return 0;
             }
-            PKT_SET_SRC(p2, PKT_SRC_FFR);
-            p2->flowflags |= FLOW_PKT_LAST_PSEUDO;
-        } else {
-            p1->flowflags |= FLOW_PKT_LAST_PSEUDO;
+            PKT_SET_SRC(p, PKT_SRC_FFR);
+            p->flowflags |= FLOW_PKT_LAST_PSEUDO;
+            FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
+            PacketPoolReturnPacket(p);
+            f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
+            return 2;
         }
+        f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
+        return 1;
+
     } else {
         if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
-            p1 = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
-            if (p1 == NULL) {
-                return 0;
+            Packet *p = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
+            if (likely(p != NULL)) {
+                PKT_SET_SRC(p, PKT_SRC_FFR);
+                p->flowflags |= FLOW_PKT_LAST_PSEUDO;
+                FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
+                PacketPoolReturnPacket(p);
+                f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
+                return 1;
             }
-            PKT_SET_SRC(p1, PKT_SRC_FFR);
-            p1->flowflags |= FLOW_PKT_LAST_PSEUDO;
-        } else {
-            /* impossible */
-            BUG_ON(1);
         }
     }
-    f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
-
-    FlowWorkerFlowTimeout(tv, p1, fw, detect_thread);
-    PacketPoolReturnPacket(p1);
-    if (p2) {
-        FlowWorkerFlowTimeout(tv, p2, fw, detect_thread);
-        PacketPoolReturnPacket(p2);
-        return 2;
-    }
-    return 1;
+    return 0;
 }
 
 extern uint32_t flow_spare_pool_block_size;

From 13cc49388523ba19d3b5b1e72063ca7f0f444be6 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 1 Dec 2023 09:47:57 +0100
Subject: [PATCH 209/462] flow/timeout: clean up flow finish code

---
 src/flow-worker.c | 58 ++++++++++++++++-------------------------------
 1 file changed, 20 insertions(+), 38 deletions(-)

diff --git a/src/flow-worker.c b/src/flow-worker.c
index 5c03689a4573..6980570d3ce1 100644
--- a/src/flow-worker.c
+++ b/src/flow-worker.c
@@ -113,60 +113,42 @@ static int FlowFinish(ThreadVars *tv, Flow *f, FlowWorkerThreadData *fw, void *d
 {
     const int server = f->ffr_tc;
     const int client = f->ffr_ts;
+    int cnt = 0;
 
     /* Get the tcp session for the flow */
-    TcpSession *ssn = (TcpSession *)f->protoctx;
-
-    /* The packets we use are based on what segments in what direction are
-     * unprocessed.
-     * p1 if we have client segments for reassembly purpose only.  If we
-     * have no server segments p2 can be a toserver packet with dummy
-     * seq/ack, and if we have server segments p2 has to carry out reassembly
-     * for server segment as well, in which case we will also need a p3 in the
-     * toclient which is now dummy since all we need it for is detection */
+    const TcpSession *ssn = (TcpSession *)f->protoctx;
 
     /* insert a pseudo packet in the toserver direction */
     if (client == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
         Packet *p = FlowForceReassemblyPseudoPacketGet(0, f, ssn);
-        if (unlikely(p == NULL)) {
-            return 0;
-        }
-        PKT_SET_SRC(p, PKT_SRC_FFR);
-        if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NONE) {
-            p->flowflags |= FLOW_PKT_LAST_PSEUDO;
+        if (p != NULL) {
+            PKT_SET_SRC(p, PKT_SRC_FFR);
+            if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NONE) {
+                p->flowflags |= FLOW_PKT_LAST_PSEUDO;
+            }
+            FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
+            PacketPoolReturnPacket(p);
+            cnt++;
         }
-        FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
-        PacketPoolReturnPacket(p);
+    }
 
-        if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
-            p = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
-            if (unlikely(p == NULL)) {
-                return 0;
-            }
+    /* handle toclient */
+    if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
+        Packet *p = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
+        if (p != NULL) {
             PKT_SET_SRC(p, PKT_SRC_FFR);
             p->flowflags |= FLOW_PKT_LAST_PSEUDO;
             FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
             PacketPoolReturnPacket(p);
             f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
-            return 2;
+            cnt++;
         }
-        f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
-        return 1;
+    }
 
-    } else {
-        if (server == STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION) {
-            Packet *p = FlowForceReassemblyPseudoPacketGet(1, f, ssn);
-            if (likely(p != NULL)) {
-                PKT_SET_SRC(p, PKT_SRC_FFR);
-                p->flowflags |= FLOW_PKT_LAST_PSEUDO;
-                FlowWorkerFlowTimeout(tv, p, fw, detect_thread);
-                PacketPoolReturnPacket(p);
-                f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
-                return 1;
-            }
-        }
+    if (cnt > 0) {
+        f->flags |= FLOW_TIMEOUT_REASSEMBLY_DONE;
     }
-    return 0;
+    return cnt;
 }
 
 extern uint32_t flow_spare_pool_block_size;

From d3ccff58228807998d7d6b73d0b2b78aa697a560 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 20:14:29 +0200
Subject: [PATCH 210/462] detect/asn1: handle in PMATCH

Since the asn1 keyword is processing payload data, move the handling of
the keyword into the PMATCH with content inspection.

Use u32 as buffer length in the Rust FFI
---
 rust/src/asn1/mod.rs                   |  2 +-
 src/detect-asn1.c                      | 41 +++++---------------------
 src/detect-asn1.h                      |  3 ++
 src/detect-engine-content-inspection.c |  8 +++++
 4 files changed, 20 insertions(+), 34 deletions(-)

diff --git a/rust/src/asn1/mod.rs b/rust/src/asn1/mod.rs
index 4b77b0ca28d5..cf382cf6077c 100644
--- a/rust/src/asn1/mod.rs
+++ b/rust/src/asn1/mod.rs
@@ -218,7 +218,7 @@ fn asn1_decode<'a>(
 /// pointer must be freed using `rs_asn1_free`
 #[no_mangle]
 pub unsafe extern "C" fn rs_asn1_decode(
-    input: *const u8, input_len: u16, buffer_offset: u32, ad_ptr: *const DetectAsn1Data,
+    input: *const u8, input_len: u32, buffer_offset: u32, ad_ptr: *const DetectAsn1Data,
 ) -> *mut Asn1<'static> {
     if input.is_null() || input_len == 0 || ad_ptr.is_null() {
         return std::ptr::null_mut();
diff --git a/src/detect-asn1.c b/src/detect-asn1.c
index 5b3a3a2229b2..c70bf8921fd3 100644
--- a/src/detect-asn1.c
+++ b/src/detect-asn1.c
@@ -36,8 +36,6 @@
 #include "util-byte.h"
 #include "util-debug.h"
 
-static int DetectAsn1Match(DetectEngineThreadCtx *, Packet *,
-                     const Signature *, const SigMatchCtx *);
 static int DetectAsn1Setup (DetectEngineCtx *, Signature *, const char *);
 #ifdef UNITTESTS
 static void DetectAsn1RegisterTests(void);
@@ -50,7 +48,6 @@ static void DetectAsn1Free(DetectEngineCtx *, void *);
 void DetectAsn1Register(void)
 {
     sigmatch_table[DETECT_ASN1].name = "asn1";
-    sigmatch_table[DETECT_ASN1].Match = DetectAsn1Match;
     sigmatch_table[DETECT_ASN1].Setup = DetectAsn1Setup;
     sigmatch_table[DETECT_ASN1].Free  = DetectAsn1Free;
 #ifdef UNITTESTS
@@ -58,37 +55,14 @@ void DetectAsn1Register(void)
 #endif
 }
 
-/**
- * \brief This function will decode the asn1 data and inspect the resulting
- *        nodes to detect if any of the specified checks match this data
- *
- * \param det_ctx pointer to the detect engine thread context
- * \param p pointer to the current packet
- * \param s pointer to the signature
- * \param ctx pointer to the sigmatch that we will cast into `DetectAsn1Data`
- *
- * \retval 1 match
- * \retval 0 no match
- */
-static int DetectAsn1Match(DetectEngineThreadCtx *det_ctx, Packet *p,
-                    const Signature *s, const SigMatchCtx *ctx)
+bool DetectAsn1Match(const SigMatchData *smd, const uint8_t *buffer, const uint32_t buffer_len,
+        const uint32_t offset)
 {
-    uint8_t ret = 0;
-
-    if (p->payload_len == 0) {
-        /* No error, parser done, no data in bounds to decode */
-        return 0;
-    }
-
-    const DetectAsn1Data *ad = (const DetectAsn1Data *)ctx;
-
-    Asn1 *asn1 = rs_asn1_decode(p->payload, p->payload_len, det_ctx->buffer_offset, ad);
-
-    ret = rs_asn1_checks(asn1, ad);
-
+    const DetectAsn1Data *ad = (const DetectAsn1Data *)smd->ctx;
+    Asn1 *asn1 = rs_asn1_decode(buffer, buffer_len, offset, ad);
+    uint8_t ret = rs_asn1_checks(asn1, ad);
     rs_asn1_free(asn1);
-
-    return ret;
+    return ret == 1;
 }
 
 /**
@@ -127,12 +101,13 @@ static int DetectAsn1Setup(DetectEngineCtx *de_ctx, Signature *s, const char *as
     if (ad == NULL)
         return -1;
 
-    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ASN1, (SigMatchCtx *)ad, DETECT_SM_LIST_MATCH) ==
+    if (SigMatchAppendSMToList(de_ctx, s, DETECT_ASN1, (SigMatchCtx *)ad, DETECT_SM_LIST_PMATCH) ==
             NULL) {
         DetectAsn1Free(de_ctx, ad);
         return -1;
     }
 
+    s->flags |= SIG_FLAG_REQUIRE_PACKET;
     return 0;
 }
 
diff --git a/src/detect-asn1.h b/src/detect-asn1.h
index a7b67340aa27..8c81ddcb305b 100644
--- a/src/detect-asn1.h
+++ b/src/detect-asn1.h
@@ -26,4 +26,7 @@
 /* prototypes */
 void DetectAsn1Register (void);
 
+bool DetectAsn1Match(const SigMatchData *smd, const uint8_t *buffer, const uint32_t buffer_len,
+        const uint32_t offset);
+
 #endif /* __DETECT_ASN1_H__ */
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 1ec78fb550bb..0070494380c2 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -31,6 +31,7 @@
 #include "detect.h"
 #include "detect-engine.h"
 #include "detect-parse.h"
+#include "detect-asn1.h"
 #include "detect-content.h"
 #include "detect-pcre.h"
 #include "detect-isdataat.h"
@@ -656,6 +657,13 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
                 }
             }
         }
+    } else if (smd->type == DETECT_ASN1) {
+        if (!DetectAsn1Match(smd, buffer, buffer_len, det_ctx->buffer_offset)) {
+            SCLogDebug("asn1 no_match");
+            goto no_match;
+        }
+        SCLogDebug("asn1 match");
+        goto match;
     } else {
         SCLogDebug("sm->type %u", smd->type);
 #ifdef DEBUG

From 1f78a4fcd8cfd831b37916335af8a3f7518d3bff Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Wed, 1 Nov 2023 16:01:35 +0530
Subject: [PATCH 211/462] detect-engine: use only the exact match fn

DetectFlagsSignatureNeedsSynPackets checks if TCP SYN flag is set among
other flags.
DetectFlagsSignatureNeedsSynOnlyPackets checks if only TCP SYN flag is
set and no other flag.

Since DetectFlagsSignatureNeedsSynOnlyPackets also already checks for
TCP SYN flag, it does not need to be used in combination with
DetectFlagsSignatureNeedsSynPackets as this fn seems to be the superset
of the former.
---
 src/detect-engine-build.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 8c01104c48de..05f93169d142 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1141,8 +1141,7 @@ static int RuleSetWhitelist(Signature *s)
             SCLogDebug("Rule %u No MPM. Payload inspecting. Whitelisting SGH's.", s->id);
             wl = 55;
 
-        } else if (DetectFlagsSignatureNeedsSynPackets(s) &&
-                   DetectFlagsSignatureNeedsSynOnlyPackets(s)) {
+        } else if (DetectFlagsSignatureNeedsSynOnlyPackets(s)) {
             SCLogDebug("Rule %u Needs SYN, so inspected often. Whitelisting SGH's.", s->id);
             wl = 33;
         }
@@ -1189,12 +1188,10 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
 
         /* see if we want to exclude directionless sigs that really care only for
          * to_server syn scans/floods */
-        if ((direction == SIG_FLAG_TOCLIENT) &&
-             DetectFlagsSignatureNeedsSynPackets(s) &&
-             DetectFlagsSignatureNeedsSynOnlyPackets(s) &&
-            ((s->flags & (SIG_FLAG_TOSERVER|SIG_FLAG_TOCLIENT)) == (SIG_FLAG_TOSERVER|SIG_FLAG_TOCLIENT)) &&
-            (!(s->dp->port == 0 && s->dp->port2 == 65535)))
-        {
+        if ((direction == SIG_FLAG_TOCLIENT) && DetectFlagsSignatureNeedsSynOnlyPackets(s) &&
+                ((s->flags & (SIG_FLAG_TOSERVER | SIG_FLAG_TOCLIENT)) ==
+                        (SIG_FLAG_TOSERVER | SIG_FLAG_TOCLIENT)) &&
+                (!(s->dp->port == 0 && s->dp->port2 == 65535))) {
             SCLogWarning("rule %u: SYN-only to port(s) %u:%u "
                          "w/o direction specified, disabling for toclient direction",
                     s->id, s->dp->port, s->dp->port2);

From 2b73a17bb04b04f5166a7a9cb4947ff1ea5592c5 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 16 Nov 2023 13:41:39 +0530
Subject: [PATCH 212/462] detect: rename whitelist to score

The term "whitelist" is actually used to store a list of DetectPort type
items for tcp and udp in detect.h. Using the same term for also keeping
the score that affects the grouping of rules is confusing. So, rename
the variable to "score".
---
 doc/userguide/upgrade.rst    |  4 ++++
 src/detect-engine-build.c    | 12 ++++++------
 src/detect-engine-siggroup.c |  4 ++--
 src/detect.h                 |  4 ++--
 4 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/doc/userguide/upgrade.rst b/doc/userguide/upgrade.rst
index 991e55ae75c1..e35a596a407b 100644
--- a/doc/userguide/upgrade.rst
+++ b/doc/userguide/upgrade.rst
@@ -34,6 +34,10 @@ also check all the new features that have been added but are not covered by
 this guide. Those features are either not enabled by default or require
 dedicated new configuration.
 
+Upgrading 7.0 to 8.0
+--------------------
+.. note:: ``stats.whitelist`` has been renamed to ``stats.score`` in ``eve.json``
+
 Upgrading 6.0 to 7.0
 --------------------
 
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 05f93169d142..e9711eddaba2 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -877,7 +877,7 @@ static json_t *RulesGroupPrintSghStats(const DetectEngineCtx *de_ctx, const SigG
     }
     json_object_set_new(js, "stats", stats);
 
-    json_object_set_new(js, "whitelist", json_integer(sgh->init->whitelist));
+    json_object_set_new(js, "score", json_integer(sgh->init->score));
 
     return js;
 }
@@ -1147,7 +1147,7 @@ static int RuleSetWhitelist(Signature *s)
         }
     }
 
-    s->init_data->whitelist = wl;
+    s->init_data->score = wl;
     return wl;
 }
 
@@ -1198,7 +1198,7 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
             goto next;
         }
 
-        int wl = s->init_data->whitelist;
+        int wl = s->init_data->score;
         while (p) {
             int pwl = PortIsWhitelisted(de_ctx, p, ipproto) ? 111 : 0;
             pwl = MAX(wl,pwl);
@@ -1206,12 +1206,12 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
             DetectPort *lookup = DetectPortHashLookup(de_ctx, p);
             if (lookup) {
                 SigGroupHeadAppendSig(de_ctx, &lookup->sh, s);
-                lookup->sh->init->whitelist = MAX(lookup->sh->init->whitelist, pwl);
+                lookup->sh->init->score = MAX(lookup->sh->init->score, pwl);
             } else {
                 DetectPort *tmp2 = DetectPortCopySingle(de_ctx, p);
                 BUG_ON(tmp2 == NULL);
                 SigGroupHeadAppendSig(de_ctx, &tmp2->sh, s);
-                tmp2->sh->init->whitelist = pwl;
+                tmp2->sh->init->score = pwl;
                 DetectPortHashAdd(de_ctx, tmp2);
             }
 
@@ -1519,7 +1519,7 @@ int SigAddressPrepareStage1(DetectEngineCtx *de_ctx)
 
 static int PortGroupWhitelist(const DetectPort *a)
 {
-    return a->sh->init->whitelist;
+    return a->sh->init->score;
 }
 
 int CreateGroupedPortListCmpCnt(DetectPort *a, DetectPort *b)
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index 36df347a503c..b063fda8a614 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -402,8 +402,8 @@ int SigGroupHeadCopySigs(DetectEngineCtx *de_ctx, SigGroupHead *src, SigGroupHea
     for (idx = 0; idx < src->init->sig_size; idx++)
         (*dst)->init->sig_array[idx] = (*dst)->init->sig_array[idx] | src->init->sig_array[idx];
 
-    if (src->init->whitelist)
-        (*dst)->init->whitelist = MAX((*dst)->init->whitelist, src->init->whitelist);
+    if (src->init->score)
+        (*dst)->init->score = MAX((*dst)->init->score, src->init->score);
 
     return 0;
 
diff --git a/src/detect.h b/src/detect.h
index 53ca2b0931f6..90ac0d7206a6 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -561,7 +561,7 @@ typedef struct SignatureInitData_ {
     /** score to influence rule grouping. A higher value leads to a higher
      *  likelihood of a rulegroup with this sig ending up as a contained
      *  group. */
-    int whitelist;
+    int score;
 
     /** address settings for this signature */
     const DetectAddressHead *src, *dst;
@@ -1413,7 +1413,7 @@ typedef struct SigGroupHeadInitData_ {
 
     uint8_t protos[256];    /**< proto(s) this sgh is for */
     uint32_t direction;     /**< set to SIG_FLAG_TOSERVER, SIG_FLAG_TOCLIENT or both */
-    int whitelist;          /**< try to make this group a unique one */
+    int score;              /**< try to make this group a unique one */
 
     MpmCtx **app_mpms;
     MpmCtx **pkt_mpms;

From 4a00ae6076df94a5096e467ec85896c9d34c6488 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 16 Nov 2023 13:48:06 +0530
Subject: [PATCH 213/462] detect/engine: fix whitelisted port range check

So far, the condition for checking if the whitelisted port was in the
port range of "a" said

a->port >= w->port && a->port2 <= w->port

But, if a->port <= a->port2, this condition could only be true when
a->port == w->port == a->port2. However, the motivation for this fn was
to be able to find if the whitelisted port for a carrier proto already
was in the range of the given protocol and calculate a score for the
port accordingly.
Fix the range check such that a->port <= w->port <= a->port2.
---
 src/detect-engine-build.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index e9711eddaba2..676aa030cc30 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1101,8 +1101,9 @@ static int PortIsWhitelisted(const DetectEngineCtx *de_ctx,
         w = de_ctx->udp_whitelist;
 
     while (w) {
-        if (a->port >= w->port && a->port2 <= w->port) {
-            SCLogDebug("port group %u:%u whitelisted -> %d", a->port, a->port2, w->port);
+        /* Make sure the whitelist port falls in the port range of a */
+        DEBUG_VALIDATE_BUG_ON(a->port > a->port2);
+        if (w->port >= a->port && w->port <= a->port2) {
             return 1;
         }
         w = w->next;

From 6076b9e2f0e2e9423dafc6e667ee2de93e31bc79 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 20 Nov 2023 17:12:54 +0530
Subject: [PATCH 214/462] detect: use proper names for whitelist score criteria

---
 src/detect-engine-build.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 676aa030cc30..c13264f88e33 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -43,6 +43,13 @@
 #include "util-var-name.h"
 #include "util-conf.h"
 
+/* Magic numbers to make the rules of a certain order fall in the same group */
+#define DETECT_PGSCORE_RULE_PORT_WHITELISTED 111 /* Rule port group contains a whitelisted port */
+#define DETECT_PGSCORE_RULE_MPM_FAST_PATTERN 99  /* Rule contains an MPM fast pattern */
+#define DETECT_PGSCORE_RULE_MPM_NEGATED      77  /* Rule contains a negated MPM */
+#define DETECT_PGSCORE_RULE_NO_MPM           55  /* Rule does not contain MPM */
+#define DETECT_PGSCORE_RULE_SYN_ONLY         33  /* Rule needs SYN check */
+
 void SigCleanSignatures(DetectEngineCtx *de_ctx)
 {
     if (de_ctx == NULL)
@@ -1129,22 +1136,22 @@ static int RuleSetWhitelist(Signature *s)
         /* pure pcre, bytetest, etc rules */
         if (RuleInspectsPayloadHasNoMpm(s)) {
             SCLogDebug("Rule %u MPM has 1 byte fast_pattern. Whitelisting SGH's.", s->id);
-            wl = 99;
+            wl = DETECT_PGSCORE_RULE_MPM_FAST_PATTERN;
 
         } else if (RuleMpmIsNegated(s)) {
             SCLogDebug("Rule %u MPM is negated. Whitelisting SGH's.", s->id);
-            wl = 77;
+            wl = DETECT_PGSCORE_RULE_MPM_NEGATED;
 
             /* one byte pattern in packet/stream payloads */
         } else if (s->init_data->mpm_sm != NULL &&
                    s->init_data->mpm_sm_list == DETECT_SM_LIST_PMATCH &&
                    RuleGetMpmPatternSize(s) == 1) {
             SCLogDebug("Rule %u No MPM. Payload inspecting. Whitelisting SGH's.", s->id);
-            wl = 55;
+            wl = DETECT_PGSCORE_RULE_NO_MPM;
 
         } else if (DetectFlagsSignatureNeedsSynOnlyPackets(s)) {
             SCLogDebug("Rule %u Needs SYN, so inspected often. Whitelisting SGH's.", s->id);
-            wl = 33;
+            wl = DETECT_PGSCORE_RULE_SYN_ONLY;
         }
     }
 
@@ -1201,7 +1208,8 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
 
         int wl = s->init_data->score;
         while (p) {
-            int pwl = PortIsWhitelisted(de_ctx, p, ipproto) ? 111 : 0;
+            int pwl = PortIsWhitelisted(de_ctx, p, ipproto) ? DETECT_PGSCORE_RULE_PORT_WHITELISTED
+                                                            : 0;
             pwl = MAX(wl,pwl);
 
             DetectPort *lookup = DetectPortHashLookup(de_ctx, p);

From 945ec4bc0a795c44f569ed68551446426a91b00a Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 20 Nov 2023 17:14:13 +0530
Subject: [PATCH 215/462] detect: remove redundant null setting

de_ctx->dport_hash_table is already set to NULL in the fn
DetectPortHashFree which is called right before this setting.
Remove the redundant setting.
---
 src/detect-engine-build.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index c13264f88e33..e6804b57b37d 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1244,7 +1244,6 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
         BUG_ON(r == -1);
     }
     DetectPortHashFree(de_ctx);
-    de_ctx->dport_hash_table = NULL;
 
     SCLogDebug("rules analyzed");
 

From d8a887e518bde0a146bad4062f668bd30e4839c8 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 20 Nov 2023 18:00:37 +0530
Subject: [PATCH 216/462] detect/engine: defensive check and comment update

---
 src/detect-engine-build.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index e6804b57b37d..42b439945b80 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1264,6 +1264,7 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
     DetectPort *iter;
     for (iter = list ; iter != NULL; iter = iter->next) {
         BUG_ON (iter->sh == NULL);
+        DEBUG_VALIDATE_BUG_ON(own + ref != cnt);
         cnt++;
 
         SigGroupHead *lookup_sgh = SigGroupHeadHashLookup(de_ctx, iter->sh);
@@ -1579,7 +1580,7 @@ int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, Detect
     uint32_t groups = 0;
     DetectPort *list;
 
-    /* insert the addresses into the tmplist, where it will
+    /* insert the ports into the tmplist, where it will
      * be sorted descending on 'cnt' and on whether a group
      * is whitelisted. */
 

From 8960a86f4fcc2e306fbf4de9a2b4550a7e8a35d3 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Tue, 21 Nov 2023 13:53:09 +0530
Subject: [PATCH 217/462] detect/port: remove BUG_ON in favor of PORT_ER

Either the BUG_ON condition would hit or PORT_ER. Prefer to return error
in case of an error as the fn expects that.
---
 src/detect-engine-port.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/src/detect-engine-port.c b/src/detect-engine-port.c
index 04d2a49d557f..86a367d376cd 100644
--- a/src/detect-engine-port.c
+++ b/src/detect-engine-port.c
@@ -594,9 +594,6 @@ int DetectPortCmp(DetectPort *a, DetectPort *b)
     } else if (a_port1 > b_port2) {
         //SCLogDebug("PORT_GT");
         return PORT_GT;
-    } else {
-        /* should be unreachable */
-        BUG_ON(1);
     }
 
     return PORT_ER;

From 77eb85e2243cc395b3076faaaa07fb9191fd744d Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Tue, 21 Nov 2023 14:20:49 +0530
Subject: [PATCH 218/462] detect: remove misleading comment

The comment seems to have come from the enum for addresses where IPv4
and IPv6 matters.
---
 src/detect.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect.h b/src/detect.h
index 90ac0d7206a6..3861b603d801 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -193,7 +193,7 @@ typedef struct DetectMatchAddressIPv6_ {
 
 /* a is ... than b */
 enum {
-    PORT_ER = -1, /* error e.g. compare ipv4 and ipv6 */
+    PORT_ER = -1, /* error */
     PORT_LT,      /* smaller              [aaa] [bbb] */
     PORT_LE,      /* smaller with overlap [aa[bab]bb] */
     PORT_EQ,      /* exactly equal        [abababab]  */

From c1bf955326a3fcd5ad9d53f315dd2612b32cfcc9 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 4 Dec 2023 20:06:57 +0530
Subject: [PATCH 219/462] detect-engine: use ports only after edge case
 handling

Also, add comments to clarify what's happening in the code.
---
 src/detect-engine-build.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 42b439945b80..33b8ca206b22 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1176,8 +1176,10 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
         /* IP Only rules are handled separately */
         if (s->type == SIG_TYPE_IPONLY)
             goto next;
+        /* Protocol does not match the Signature protocol and is neither IP or pkthdr */
         if (!(s->proto.proto[ipproto / 8] & (1<<(ipproto % 8)) || (s->proto.flags & DETECT_PROTO_ANY)))
             goto next;
+        /* Direction does not match Signature direction */
         if (direction == SIG_FLAG_TOSERVER) {
             if (!(s->flags & SIG_FLAG_TOSERVER))
                 goto next;
@@ -1186,14 +1188,6 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
                 goto next;
         }
 
-        DetectPort *p = NULL;
-        if (direction == SIG_FLAG_TOSERVER)
-            p = s->dp;
-        else if (direction == SIG_FLAG_TOCLIENT)
-            p = s->sp;
-        else
-            BUG_ON(1);
-
         /* see if we want to exclude directionless sigs that really care only for
          * to_server syn scans/floods */
         if ((direction == SIG_FLAG_TOCLIENT) && DetectFlagsSignatureNeedsSynOnlyPackets(s) &&
@@ -1206,6 +1200,14 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
             goto next;
         }
 
+        DetectPort *p = NULL;
+        if (direction == SIG_FLAG_TOSERVER)
+            p = s->dp;
+        else if (direction == SIG_FLAG_TOCLIENT)
+            p = s->sp;
+        else
+            BUG_ON(1);
+
         int wl = s->init_data->score;
         while (p) {
             int pwl = PortIsWhitelisted(de_ctx, p, ipproto) ? DETECT_PGSCORE_RULE_PORT_WHITELISTED

From 1ac5d97259994f73e64533366d74e40f9fe22694 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 27 Nov 2023 17:22:19 -0300
Subject: [PATCH 220/462] pgsql: add unknonwn frontend message type

We had unkonwn message type for the backend, but not the frontend
messages. It's important to better identify those to improve pgsql
probing functions.

Related to
Bug #6080
---
 rust/src/pgsql/logger.rs |  7 +++++++
 rust/src/pgsql/parser.rs | 14 +++++++++++++-
 rust/src/pgsql/pgsql.rs  |  5 +++++
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index 03b1ad0f4677..57a3e526709c 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -102,6 +102,13 @@ fn log_request(req: &PgsqlFEMessage, flags: u32) -> Result<JsonBuilder, JsonErro
         }) => {
             js.set_string("message", req.to_str())?;
         }
+        PgsqlFEMessage::UnknownMessageType(RegularPacket {
+            identifier: _,
+            length: _,
+            payload: _,
+        }) => {
+            // We don't want to log these, for now. Cf redmine: #6576
+        }
     }
     js.close()?;
     Ok(js)
diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index ae07d5d5a078..27ea3217e88c 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -320,6 +320,7 @@ pub enum PgsqlFEMessage {
     SASLResponse(RegularPacket),
     SimpleQuery(RegularPacket),
     Terminate(TerminationMessage),
+    UnknownMessageType(RegularPacket),
 }
 
 impl PgsqlFEMessage {
@@ -332,6 +333,7 @@ impl PgsqlFEMessage {
             PgsqlFEMessage::SASLResponse(_) => "sasl_response",
             PgsqlFEMessage::SimpleQuery(_) => "simple_query",
             PgsqlFEMessage::Terminate(_) => "termination_message",
+            PgsqlFEMessage::UnknownMessageType(_) => "unknown_message_type",
         }
     }
 }
@@ -673,7 +675,17 @@ pub fn parse_request(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
         b'\0' => pgsql_parse_startup_packet(i)?,
         b'Q' => parse_simple_query(i)?,
         b'X' => parse_terminate_message(i)?,
-        _ => return Err(Err::Error(make_error(i, ErrorKind::Switch))),
+        _ => {
+            let (i, identifier) = be_u8(i)?;
+            let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+            let (i, payload) = take(length - PGSQL_LENGTH_FIELD)(i)?;
+            let unknown = PgsqlFEMessage::UnknownMessageType (RegularPacket{
+                identifier,
+                length,
+                payload: payload.to_vec(),
+            });
+            (i, unknown)
+        }
     };
     Ok((i, message))
 }
diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index f5fbebc8f950..fa19785ff9e1 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -284,6 +284,11 @@ impl PgsqlState {
                 SCLogDebug!("Match: Terminate message");
                 Some(PgsqlStateProgress::ConnectionTerminated)
             }
+            PgsqlFEMessage::UnknownMessageType(_) => {
+                SCLogDebug!("Match: Unknown message type");
+                // Not changing state when we don't know the message
+                None
+            }
         }
     }
 

From 4f85d061926e870a48aeaf13bdbb4666ad7fc07e Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 23 Nov 2023 08:14:24 -0300
Subject: [PATCH 221/462] pgsql: fix probing functions

Some non-pgsql traffic seen by Suricata is mistankenly identified as
pgsql, as the probing function is too generic. Now, if the parser sees
an unknown message type, even if it looks like pgsql, it will fail.

Bug #6080
---
 rust/src/pgsql/pgsql.rs | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index fa19785ff9e1..94e26d84815c 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -151,7 +151,7 @@ impl Default for PgsqlState {
         Self::new()
     }
 }
-    
+
 impl PgsqlState {
     pub fn new() -> Self {
         Self {
@@ -563,8 +563,20 @@ pub unsafe extern "C" fn rs_pgsql_probing_parser_ts(
     if input_len >= 1 && !input.is_null() {
 
         let slice: &[u8] = build_slice!(input, input_len as usize);
-        if probe_ts(slice) {
-            return ALPROTO_PGSQL;
+
+        match parser::parse_request(slice) {
+            Ok((_, request)) => {
+                if let PgsqlFEMessage::UnknownMessageType(_) = request {
+                    return ALPROTO_FAILED;
+                }
+                return ALPROTO_PGSQL;
+            }
+            Err(Err::Incomplete(_)) => {
+                return ALPROTO_UNKNOWN;
+            }
+            Err(_e) => {
+                return ALPROTO_FAILED;
+            }
         }
     }
     return ALPROTO_UNKNOWN;
@@ -584,7 +596,10 @@ pub unsafe extern "C" fn rs_pgsql_probing_parser_tc(
         }
 
         match parser::pgsql_parse_response(slice) {
-            Ok((_, _response)) => {
+            Ok((_, response)) => {
+                if let PgsqlBEMessage::UnknownMessageType(_) = response {
+                    return ALPROTO_FAILED;
+                }
                 return ALPROTO_PGSQL;
             }
             Err(Err::Incomplete(_)) => {

From afd6e4dc414708f12c46b251d2ca8df1afd3f66b Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 27 Nov 2023 17:16:52 -0300
Subject: [PATCH 222/462] pgsql: don't log unknown message type

---
 rust/src/pgsql/logger.rs | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index 57a3e526709c..b17986dec85b 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -184,12 +184,10 @@ fn log_response(res: &PgsqlBEMessage, jb: &mut JsonBuilder) -> Result<(), JsonEr
         }
         PgsqlBEMessage::UnknownMessageType(RegularPacket {
             identifier: _,
-            length,
-            payload,
+            length: _,
+            payload: _,
         }) => {
-            // jb.set_string_from_bytes("identifier", identifier.to_vec())?;
-            jb.set_uint("length", (*length).into())?;
-            jb.set_string_from_bytes("payload", payload)?;
+            // We don't want to log these, for now. Cf redmine: #6576
         }
         PgsqlBEMessage::AuthenticationOk(_)
         | PgsqlBEMessage::AuthenticationCleartextPassword(_)

From 53d29f652ac0fe3ce342140d88492b077b0f2c01 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 27 Nov 2023 17:17:11 -0300
Subject: [PATCH 223/462] pgsql: remove unused error handling call

---
 rust/src/pgsql/parser.rs | 1 -
 1 file changed, 1 deletion(-)

diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 27ea3217e88c..bed3682bb43d 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -1056,7 +1056,6 @@ pub fn pgsql_parse_response(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
                 b'T' => parse_row_description(i)?,
                 b'A' => parse_notification_response(i)?,
                 b'D' => parse_consolidated_data_row(i)?,
-                // _ => return Err(Err::Error(make_error(i, ErrorKind::Switch))),
                 _ => {
                     let (i, payload) = rest(i)?;
                     let unknown = PgsqlBEMessage::UnknownMessageType (RegularPacket{

From 9aeeac532eadc57c399b17d364d7337c5c69bd8e Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 27 Nov 2023 17:01:33 -0300
Subject: [PATCH 224/462] pgsql: remove probe_ts function

With the changes in the probing_ts function, this other one could become
obsolete. Remove it, and directly call `parser::parse_request` when
checking for gaps, instead.
---
 rust/src/pgsql/pgsql.rs | 41 +----------------------------------------
 1 file changed, 1 insertion(+), 40 deletions(-)

diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index 94e26d84815c..8b9b12c4d694 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -318,7 +318,7 @@ impl PgsqlState {
 
         // If there was gap, check we can sync up again.
         if self.request_gap {
-            if !probe_ts(input) {
+            if parser::parse_request(input).is_ok() {
                 // The parser now needs to decide what to do as we are not in sync.
                 // For now, we'll just try again next time.
                 SCLogDebug!("Suricata interprets there's a gap in the request");
@@ -532,14 +532,6 @@ impl PgsqlState {
     }
 }
 
-/// Probe for a valid PostgreSQL request
-///
-/// PGSQL messages don't have a header per se, so we parse the slice for an ok()
-fn probe_ts(input: &[u8]) -> bool {
-    SCLogDebug!("We are in probe_ts");
-    parser::parse_request(input).is_ok()
-}
-
 /// Probe for a valid PostgreSQL response
 ///
 /// Currently, for parser usage only. We have a bit more logic in the function
@@ -801,37 +793,6 @@ pub unsafe extern "C" fn rs_pgsql_register_parser() {
 mod test {
     use super::*;
 
-    #[test]
-    fn test_request_probe() {
-        // An SSL Request
-        let buf: &[u8] = &[0x00, 0x00, 0x00, 0x08, 0x04, 0xd2, 0x16, 0x2f];
-        assert!(probe_ts(buf));
-
-        // incomplete messages, probe must return false
-        assert!(!probe_ts(&buf[0..6]));
-        assert!(!probe_ts(&buf[0..3]));
-
-        // length is wrong (7), probe must return false
-        let buf: &[u8] = &[0x00, 0x00, 0x00, 0x07, 0x04, 0xd2, 0x16, 0x2f];
-        assert!(!probe_ts(buf));
-
-        // A valid startup message/request
-        let buf: &[u8] = &[
-            0x00, 0x00, 0x00, 0x26, 0x00, 0x03, 0x00, 0x00, 0x75, 0x73, 0x65, 0x72, 0x00, 0x6f,
-            0x72, 0x79, 0x78, 0x00, 0x64, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x00, 0x6d,
-            0x61, 0x69, 0x6c, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x00, 0x00,
-        ];
-        assert!(probe_ts(buf));
-
-        // A non valid startup message/request (length is shorter by one. Would `exact!` help?)
-        let buf: &[u8] = &[
-            0x00, 0x00, 0x00, 0x25, 0x00, 0x03, 0x00, 0x00, 0x75, 0x73, 0x65, 0x72, 0x00, 0x6f,
-            0x72, 0x79, 0x78, 0x00, 0x64, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x00, 0x6d,
-            0x61, 0x69, 0x6c, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x00, 0x00,
-        ];
-        assert!(!probe_ts(buf));
-    }
-
     #[test]
     fn test_response_probe() {
         /* Authentication Request MD5 password salt value f211a3ed */

From 64d12aacc8f462318ee7cc9b06dffaca84548d4d Mon Sep 17 00:00:00 2001
From: Vincent Li <vincent.mc.li@gmail.com>
Date: Wed, 2 Aug 2023 20:31:54 +0000
Subject: [PATCH 225/462] ebpf: Update eBPF map to BTF defined map

legacy map definition is removed from libbpf1.0+.
update the legacy map definition to BTF defined map.

Distros with < libbpf1.0 (0.5, 0.6, 0.7, 0.8) bpf_helpers.h
support BTF map definition, this change does not break
old libbpf and support new libpbf1.0+.

Bug: #6250

Signed-off-by: Vincent Li <vincent.mc.li@gmail.com>
Co-authored-by: Victor Julien <vjulien@oisf.net>
---
 .github/workflows/builds.yml |   3 +-
 configure.ac                 |  17 ++
 ebpf/Makefile.am             |   4 +-
 ebpf/bpf_helpers.h           | 365 -----------------------------------
 ebpf/bypass_filter.c         |  27 +--
 ebpf/filter.c                |  15 +-
 ebpf/lb.c                    |   3 +-
 ebpf/llvm_bpfload.h          |   7 +
 ebpf/vlan_filter.c           |   2 +-
 ebpf/xdp_filter.c            | 122 ++++++------
 ebpf/xdp_lb.c                |  43 +++--
 11 files changed, 135 insertions(+), 473 deletions(-)
 delete mode 100644 ebpf/bpf_helpers.h
 create mode 100644 ebpf/llvm_bpfload.h

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 93708415294a..aa6d7771840d 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -2430,7 +2430,6 @@ jobs:
                 zlib1g \
                 zlib1g-dev \
                 clang \
-                libbpf-dev \
                 libelf-dev
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain $RUST_VERSION_KNOWN -y
@@ -2448,7 +2447,7 @@ jobs:
           cp prep/cbindgen $HOME/.cargo/bin
           chmod 755 $HOME/.cargo/bin/cbindgen
       - run: ./autogen.sh
-      - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests --enable-fuzztargets --enable-ebpf --enable-ebpf-build
+      - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests --enable-fuzztargets
       - run: make -j2
       - run: make check
       - run: tar xf prep/suricata-verify.tar.gz
diff --git a/configure.ac b/configure.ac
index c2bed717616c..3acab5b3acfd 100644
--- a/configure.ac
+++ b/configure.ac
@@ -482,6 +482,23 @@
                     AC_SUBST(LLC)
                   ],
                   [AC_MSG_ERROR([clang needed to build ebpf files])])
+            AC_MSG_CHECKING([libbpf has bpf/bpf_helpers.h])
+            AC_COMPILE_IFELSE(
+                [AC_LANG_PROGRAM(
+                    [
+                        #include <stddef.h>
+                        #include <linux/bpf.h>
+                        #include <bpf/bpf_helpers.h>
+                    ],
+                    [
+                    ])],
+                [HAVE_BPF="yes"],
+                [HAVE_BPF="no"])
+            if test "$HAVE_BPF" = "no"; then
+                AC_MSG_ERROR([libbpf include bpf/bpf_helpers.h not found])
+            else
+                AC_MSG_RESULT([ok])
+            fi
           ])
 
   # enable debug output
diff --git a/ebpf/Makefile.am b/ebpf/Makefile.am
index ad32efd34a0b..450bd19ff49d 100644
--- a/ebpf/Makefile.am
+++ b/ebpf/Makefile.am
@@ -1,5 +1,5 @@
 EXTRA_DIST= include bypass_filter.c filter.c lb.c vlan_filter.c xdp_filter.c \
-	    xdp_lb.c bpf_helpers.h hash_func01.h
+	    xdp_lb.c hash_func01.h
 
 if BUILD_EBPF
 
@@ -18,7 +18,7 @@ all: $(BPF_TARGETS)
 
 $(BPF_TARGETS): %.bpf: %.c
 #      From C-code to LLVM-IR format suffix .ll (clang -S -emit-llvm)
-	${CLANG} -Wall $(BPF_CFLAGS) -O2 \
+	${CLANG} -Wall $(BPF_CFLAGS) -O2 -g \
 		-I/usr/include/$(build_cpu)-$(build_os)/ \
 		-D__KERNEL__ -D__ASM_SYSREG_H \
 		-target bpf -S -emit-llvm $< -o ${@:.bpf=.ll}
diff --git a/ebpf/bpf_helpers.h b/ebpf/bpf_helpers.h
deleted file mode 100644
index 6c77cf7bedce..000000000000
--- a/ebpf/bpf_helpers.h
+++ /dev/null
@@ -1,365 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __BPF_HELPERS_H
-#define __BPF_HELPERS_H
-
-/* helper macro to place programs, maps, license in
- * different sections in elf_bpf file. Section names
- * are interpreted by elf_bpf loader
- */
-#define SEC(NAME) __attribute__((section(NAME), used))
-
-/* helper functions called from eBPF programs written in C */
-static void *(*bpf_map_lookup_elem)(void *map, void *key) =
-	(void *) BPF_FUNC_map_lookup_elem;
-static int (*bpf_map_update_elem)(void *map, void *key, void *value,
-				  unsigned long long flags) =
-	(void *) BPF_FUNC_map_update_elem;
-static int (*bpf_map_delete_elem)(void *map, void *key) =
-	(void *) BPF_FUNC_map_delete_elem;
-static int (*bpf_map_push_elem)(void *map, void *value,
-				unsigned long long flags) =
-	(void *) BPF_FUNC_map_push_elem;
-static int (*bpf_map_pop_elem)(void *map, void *value) =
-	(void *) BPF_FUNC_map_pop_elem;
-static int (*bpf_map_peek_elem)(void *map, void *value) =
-	(void *) BPF_FUNC_map_peek_elem;
-static int (*bpf_probe_read)(void *dst, int size, void *unsafe_ptr) =
-	(void *) BPF_FUNC_probe_read;
-static unsigned long long (*bpf_ktime_get_ns)(void) =
-	(void *) BPF_FUNC_ktime_get_ns;
-static int (*bpf_trace_printk)(const char *fmt, int fmt_size, ...) =
-	(void *) BPF_FUNC_trace_printk;
-static void (*bpf_tail_call)(void *ctx, void *map, int index) =
-	(void *) BPF_FUNC_tail_call;
-static unsigned long long (*bpf_get_smp_processor_id)(void) =
-	(void *) BPF_FUNC_get_smp_processor_id;
-static unsigned long long (*bpf_get_current_pid_tgid)(void) =
-	(void *) BPF_FUNC_get_current_pid_tgid;
-static unsigned long long (*bpf_get_current_uid_gid)(void) =
-	(void *) BPF_FUNC_get_current_uid_gid;
-static int (*bpf_get_current_comm)(void *buf, int buf_size) =
-	(void *) BPF_FUNC_get_current_comm;
-static unsigned long long (*bpf_perf_event_read)(void *map,
-						 unsigned long long flags) =
-	(void *) BPF_FUNC_perf_event_read;
-static int (*bpf_clone_redirect)(void *ctx, int ifindex, int flags) =
-	(void *) BPF_FUNC_clone_redirect;
-static int (*bpf_redirect)(int ifindex, int flags) =
-	(void *) BPF_FUNC_redirect;
-static int (*bpf_redirect_map)(void *map, int key, int flags) =
-	(void *) BPF_FUNC_redirect_map;
-static int (*bpf_perf_event_output)(void *ctx, void *map,
-				    unsigned long long flags, void *data,
-				    int size) =
-	(void *) BPF_FUNC_perf_event_output;
-static int (*bpf_get_stackid)(void *ctx, void *map, int flags) =
-	(void *) BPF_FUNC_get_stackid;
-static int (*bpf_probe_write_user)(void *dst, void *src, int size) =
-	(void *) BPF_FUNC_probe_write_user;
-static int (*bpf_current_task_under_cgroup)(void *map, int index) =
-	(void *) BPF_FUNC_current_task_under_cgroup;
-static int (*bpf_skb_get_tunnel_key)(void *ctx, void *key, int size, int flags) =
-	(void *) BPF_FUNC_skb_get_tunnel_key;
-static int (*bpf_skb_set_tunnel_key)(void *ctx, void *key, int size, int flags) =
-	(void *) BPF_FUNC_skb_set_tunnel_key;
-static int (*bpf_skb_get_tunnel_opt)(void *ctx, void *md, int size) =
-	(void *) BPF_FUNC_skb_get_tunnel_opt;
-static int (*bpf_skb_set_tunnel_opt)(void *ctx, void *md, int size) =
-	(void *) BPF_FUNC_skb_set_tunnel_opt;
-static unsigned long long (*bpf_get_prandom_u32)(void) =
-	(void *) BPF_FUNC_get_prandom_u32;
-static int (*bpf_xdp_adjust_head)(void *ctx, int offset) =
-	(void *) BPF_FUNC_xdp_adjust_head;
-static int (*bpf_xdp_adjust_meta)(void *ctx, int offset) =
-	(void *) BPF_FUNC_xdp_adjust_meta;
-static int (*bpf_get_socket_cookie)(void *ctx) =
-	(void *) BPF_FUNC_get_socket_cookie;
-static int (*bpf_setsockopt)(void *ctx, int level, int optname, void *optval,
-			     int optlen) =
-	(void *) BPF_FUNC_setsockopt;
-static int (*bpf_getsockopt)(void *ctx, int level, int optname, void *optval,
-			     int optlen) =
-	(void *) BPF_FUNC_getsockopt;
-static int (*bpf_sock_ops_cb_flags_set)(void *ctx, int flags) =
-	(void *) BPF_FUNC_sock_ops_cb_flags_set;
-static int (*bpf_sk_redirect_map)(void *ctx, void *map, int key, int flags) =
-	(void *) BPF_FUNC_sk_redirect_map;
-static int (*bpf_sk_redirect_hash)(void *ctx, void *map, void *key, int flags) =
-	(void *) BPF_FUNC_sk_redirect_hash;
-static int (*bpf_sock_map_update)(void *map, void *key, void *value,
-				  unsigned long long flags) =
-	(void *) BPF_FUNC_sock_map_update;
-static int (*bpf_sock_hash_update)(void *map, void *key, void *value,
-				   unsigned long long flags) =
-	(void *) BPF_FUNC_sock_hash_update;
-static int (*bpf_perf_event_read_value)(void *map, unsigned long long flags,
-					void *buf, unsigned int buf_size) =
-	(void *) BPF_FUNC_perf_event_read_value;
-static int (*bpf_perf_prog_read_value)(void *ctx, void *buf,
-				       unsigned int buf_size) =
-	(void *) BPF_FUNC_perf_prog_read_value;
-static int (*bpf_override_return)(void *ctx, unsigned long rc) =
-	(void *) BPF_FUNC_override_return;
-static int (*bpf_msg_redirect_map)(void *ctx, void *map, int key, int flags) =
-	(void *) BPF_FUNC_msg_redirect_map;
-static int (*bpf_msg_redirect_hash)(void *ctx,
-				    void *map, void *key, int flags) =
-	(void *) BPF_FUNC_msg_redirect_hash;
-static int (*bpf_msg_apply_bytes)(void *ctx, int len) =
-	(void *) BPF_FUNC_msg_apply_bytes;
-static int (*bpf_msg_cork_bytes)(void *ctx, int len) =
-	(void *) BPF_FUNC_msg_cork_bytes;
-static int (*bpf_msg_pull_data)(void *ctx, int start, int end, int flags) =
-	(void *) BPF_FUNC_msg_pull_data;
-static int (*bpf_msg_push_data)(void *ctx, int start, int end, int flags) =
-	(void *) BPF_FUNC_msg_push_data;
-static int (*bpf_msg_pop_data)(void *ctx, int start, int cut, int flags) =
-	(void *) BPF_FUNC_msg_pop_data;
-static int (*bpf_bind)(void *ctx, void *addr, int addr_len) =
-	(void *) BPF_FUNC_bind;
-static int (*bpf_xdp_adjust_tail)(void *ctx, int offset) =
-	(void *) BPF_FUNC_xdp_adjust_tail;
-static int (*bpf_skb_get_xfrm_state)(void *ctx, int index, void *state,
-				     int size, int flags) =
-	(void *) BPF_FUNC_skb_get_xfrm_state;
-static int (*bpf_sk_select_reuseport)(void *ctx, void *map, void *key, __u32 flags) =
-	(void *) BPF_FUNC_sk_select_reuseport;
-static int (*bpf_get_stack)(void *ctx, void *buf, int size, int flags) =
-	(void *) BPF_FUNC_get_stack;
-static int (*bpf_fib_lookup)(void *ctx, struct bpf_fib_lookup *params,
-			     int plen, __u32 flags) =
-	(void *) BPF_FUNC_fib_lookup;
-static int (*bpf_lwt_push_encap)(void *ctx, unsigned int type, void *hdr,
-				 unsigned int len) =
-	(void *) BPF_FUNC_lwt_push_encap;
-static int (*bpf_lwt_seg6_store_bytes)(void *ctx, unsigned int offset,
-				       void *from, unsigned int len) =
-	(void *) BPF_FUNC_lwt_seg6_store_bytes;
-static int (*bpf_lwt_seg6_action)(void *ctx, unsigned int action, void *param,
-				  unsigned int param_len) =
-	(void *) BPF_FUNC_lwt_seg6_action;
-static int (*bpf_lwt_seg6_adjust_srh)(void *ctx, unsigned int offset,
-				      unsigned int len) =
-	(void *) BPF_FUNC_lwt_seg6_adjust_srh;
-static int (*bpf_rc_repeat)(void *ctx) =
-	(void *) BPF_FUNC_rc_repeat;
-static int (*bpf_rc_keydown)(void *ctx, unsigned int protocol,
-			     unsigned long long scancode, unsigned int toggle) =
-	(void *) BPF_FUNC_rc_keydown;
-static unsigned long long (*bpf_get_current_cgroup_id)(void) =
-	(void *) BPF_FUNC_get_current_cgroup_id;
-static void *(*bpf_get_local_storage)(void *map, unsigned long long flags) =
-	(void *) BPF_FUNC_get_local_storage;
-static unsigned long long (*bpf_skb_cgroup_id)(void *ctx) =
-	(void *) BPF_FUNC_skb_cgroup_id;
-static unsigned long long (*bpf_skb_ancestor_cgroup_id)(void *ctx, int level) =
-	(void *) BPF_FUNC_skb_ancestor_cgroup_id;
-static struct bpf_sock *(*bpf_sk_lookup_tcp)(void *ctx,
-					     struct bpf_sock_tuple *tuple,
-					     int size, unsigned long long netns_id,
-					     unsigned long long flags) =
-	(void *) BPF_FUNC_sk_lookup_tcp;
-static struct bpf_sock *(*bpf_sk_lookup_udp)(void *ctx,
-					     struct bpf_sock_tuple *tuple,
-					     int size, unsigned long long netns_id,
-					     unsigned long long flags) =
-	(void *) BPF_FUNC_sk_lookup_udp;
-static int (*bpf_sk_release)(struct bpf_sock *sk) =
-	(void *) BPF_FUNC_sk_release;
-static int (*bpf_skb_vlan_push)(void *ctx, __be16 vlan_proto, __u16 vlan_tci) =
-	(void *) BPF_FUNC_skb_vlan_push;
-static int (*bpf_skb_vlan_pop)(void *ctx) =
-	(void *) BPF_FUNC_skb_vlan_pop;
-static int (*bpf_rc_pointer_rel)(void *ctx, int rel_x, int rel_y) =
-	(void *) BPF_FUNC_rc_pointer_rel;
-
-/* llvm builtin functions that eBPF C program may use to
- * emit BPF_LD_ABS and BPF_LD_IND instructions
- */
-struct sk_buff;
-unsigned long long load_byte(void *skb,
-			     unsigned long long off) asm("llvm.bpf.load.byte");
-unsigned long long load_half(void *skb,
-			     unsigned long long off) asm("llvm.bpf.load.half");
-unsigned long long load_word(void *skb,
-			     unsigned long long off) asm("llvm.bpf.load.word");
-
-/* a helper structure used by eBPF C program
- * to describe map attributes to elf_bpf loader
- */
-struct bpf_map_def {
-	unsigned int type;
-	unsigned int key_size;
-	unsigned int value_size;
-	unsigned int max_entries;
-	unsigned int map_flags;
-	unsigned int inner_map_idx;
-	unsigned int numa_node;
-};
-
-#define BPF_ANNOTATE_KV_PAIR(name, type_key, type_val)		\
-	struct ____btf_map_##name {				\
-		type_key key;					\
-		type_val value;					\
-	};							\
-	struct ____btf_map_##name				\
-	__attribute__ ((section(".maps." #name), used))		\
-		____btf_map_##name = { }
-
-static int (*bpf_skb_load_bytes)(void *ctx, int off, void *to, int len) =
-	(void *) BPF_FUNC_skb_load_bytes;
-static int (*bpf_skb_load_bytes_relative)(void *ctx, int off, void *to, int len, __u32 start_header) =
-	(void *) BPF_FUNC_skb_load_bytes_relative;
-static int (*bpf_skb_store_bytes)(void *ctx, int off, void *from, int len, int flags) =
-	(void *) BPF_FUNC_skb_store_bytes;
-static int (*bpf_l3_csum_replace)(void *ctx, int off, int from, int to, int flags) =
-	(void *) BPF_FUNC_l3_csum_replace;
-static int (*bpf_l4_csum_replace)(void *ctx, int off, int from, int to, int flags) =
-	(void *) BPF_FUNC_l4_csum_replace;
-static int (*bpf_csum_diff)(void *from, int from_size, void *to, int to_size, int seed) =
-	(void *) BPF_FUNC_csum_diff;
-static int (*bpf_skb_under_cgroup)(void *ctx, void *map, int index) =
-	(void *) BPF_FUNC_skb_under_cgroup;
-static int (*bpf_skb_change_head)(void *, int len, int flags) =
-	(void *) BPF_FUNC_skb_change_head;
-static int (*bpf_skb_pull_data)(void *, int len) =
-	(void *) BPF_FUNC_skb_pull_data;
-
-/* Scan the ARCH passed in from ARCH env variable (see Makefile) */
-#if defined(__TARGET_ARCH_x86)
-	#define bpf_target_x86
-	#define bpf_target_defined
-#elif defined(__TARGET_ARCH_s930x)
-	#define bpf_target_s930x
-	#define bpf_target_defined
-#elif defined(__TARGET_ARCH_arm64)
-	#define bpf_target_arm64
-	#define bpf_target_defined
-#elif defined(__TARGET_ARCH_mips)
-	#define bpf_target_mips
-	#define bpf_target_defined
-#elif defined(__TARGET_ARCH_powerpc)
-	#define bpf_target_powerpc
-	#define bpf_target_defined
-#elif defined(__TARGET_ARCH_sparc)
-	#define bpf_target_sparc
-	#define bpf_target_defined
-#else
-	#undef bpf_target_defined
-#endif
-
-/* Fall back to what the compiler says */
-#ifndef bpf_target_defined
-#if defined(__x86_64__)
-	#define bpf_target_x86
-#elif defined(__s390x__)
-	#define bpf_target_s930x
-#elif defined(__aarch64__)
-	#define bpf_target_arm64
-#elif defined(__mips__)
-	#define bpf_target_mips
-#elif defined(__powerpc__)
-	#define bpf_target_powerpc
-#elif defined(__sparc__)
-	#define bpf_target_sparc
-#endif
-#endif
-
-#if defined(bpf_target_x86)
-
-#define PT_REGS_PARM1(x) ((x)->di)
-#define PT_REGS_PARM2(x) ((x)->si)
-#define PT_REGS_PARM3(x) ((x)->dx)
-#define PT_REGS_PARM4(x) ((x)->cx)
-#define PT_REGS_PARM5(x) ((x)->r8)
-#define PT_REGS_RET(x) ((x)->sp)
-#define PT_REGS_FP(x) ((x)->bp)
-#define PT_REGS_RC(x) ((x)->ax)
-#define PT_REGS_SP(x) ((x)->sp)
-#define PT_REGS_IP(x) ((x)->ip)
-
-#elif defined(bpf_target_s390x)
-
-#define PT_REGS_PARM1(x) ((x)->gprs[2])
-#define PT_REGS_PARM2(x) ((x)->gprs[3])
-#define PT_REGS_PARM3(x) ((x)->gprs[4])
-#define PT_REGS_PARM4(x) ((x)->gprs[5])
-#define PT_REGS_PARM5(x) ((x)->gprs[6])
-#define PT_REGS_RET(x) ((x)->gprs[14])
-#define PT_REGS_FP(x) ((x)->gprs[11]) /* Works only with CONFIG_FRAME_POINTER */
-#define PT_REGS_RC(x) ((x)->gprs[2])
-#define PT_REGS_SP(x) ((x)->gprs[15])
-#define PT_REGS_IP(x) ((x)->psw.addr)
-
-#elif defined(bpf_target_arm64)
-
-#define PT_REGS_PARM1(x) ((x)->regs[0])
-#define PT_REGS_PARM2(x) ((x)->regs[1])
-#define PT_REGS_PARM3(x) ((x)->regs[2])
-#define PT_REGS_PARM4(x) ((x)->regs[3])
-#define PT_REGS_PARM5(x) ((x)->regs[4])
-#define PT_REGS_RET(x) ((x)->regs[30])
-#define PT_REGS_FP(x) ((x)->regs[29]) /* Works only with CONFIG_FRAME_POINTER */
-#define PT_REGS_RC(x) ((x)->regs[0])
-#define PT_REGS_SP(x) ((x)->sp)
-#define PT_REGS_IP(x) ((x)->pc)
-
-#elif defined(bpf_target_mips)
-
-#define PT_REGS_PARM1(x) ((x)->regs[4])
-#define PT_REGS_PARM2(x) ((x)->regs[5])
-#define PT_REGS_PARM3(x) ((x)->regs[6])
-#define PT_REGS_PARM4(x) ((x)->regs[7])
-#define PT_REGS_PARM5(x) ((x)->regs[8])
-#define PT_REGS_RET(x) ((x)->regs[31])
-#define PT_REGS_FP(x) ((x)->regs[30]) /* Works only with CONFIG_FRAME_POINTER */
-#define PT_REGS_RC(x) ((x)->regs[1])
-#define PT_REGS_SP(x) ((x)->regs[29])
-#define PT_REGS_IP(x) ((x)->cp0_epc)
-
-#elif defined(bpf_target_powerpc)
-
-#define PT_REGS_PARM1(x) ((x)->gpr[3])
-#define PT_REGS_PARM2(x) ((x)->gpr[4])
-#define PT_REGS_PARM3(x) ((x)->gpr[5])
-#define PT_REGS_PARM4(x) ((x)->gpr[6])
-#define PT_REGS_PARM5(x) ((x)->gpr[7])
-#define PT_REGS_RC(x) ((x)->gpr[3])
-#define PT_REGS_SP(x) ((x)->sp)
-#define PT_REGS_IP(x) ((x)->nip)
-
-#elif defined(bpf_target_sparc)
-
-#define PT_REGS_PARM1(x) ((x)->u_regs[UREG_I0])
-#define PT_REGS_PARM2(x) ((x)->u_regs[UREG_I1])
-#define PT_REGS_PARM3(x) ((x)->u_regs[UREG_I2])
-#define PT_REGS_PARM4(x) ((x)->u_regs[UREG_I3])
-#define PT_REGS_PARM5(x) ((x)->u_regs[UREG_I4])
-#define PT_REGS_RET(x) ((x)->u_regs[UREG_I7])
-#define PT_REGS_RC(x) ((x)->u_regs[UREG_I0])
-#define PT_REGS_SP(x) ((x)->u_regs[UREG_FP])
-
-/* Should this also be a bpf_target check for the sparc case? */
-#if defined(__arch64__)
-#define PT_REGS_IP(x) ((x)->tpc)
-#else
-#define PT_REGS_IP(x) ((x)->pc)
-#endif
-
-#endif
-
-#ifdef bpf_target_powerpc
-#define BPF_KPROBE_READ_RET_IP(ip, ctx)		({ (ip) = (ctx)->link; })
-#define BPF_KRETPROBE_READ_RET_IP		BPF_KPROBE_READ_RET_IP
-#elif bpf_target_sparc
-#define BPF_KPROBE_READ_RET_IP(ip, ctx)		({ (ip) = PT_REGS_RET(ctx); })
-#define BPF_KRETPROBE_READ_RET_IP		BPF_KPROBE_READ_RET_IP
-#else
-#define BPF_KPROBE_READ_RET_IP(ip, ctx)		({				\
-		bpf_probe_read(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
-#define BPF_KRETPROBE_READ_RET_IP(ip, ctx)	({				\
-		bpf_probe_read(&(ip), sizeof(ip),				\
-				(void *)(PT_REGS_FP(ctx) + sizeof(ip))); })
-#endif
-
-#endif
diff --git a/ebpf/bypass_filter.c b/ebpf/bypass_filter.c
index eda9650edcf5..9dc79f38381b 100644
--- a/ebpf/bypass_filter.c
+++ b/ebpf/bypass_filter.c
@@ -25,7 +25,8 @@
 #include <linux/ipv6.h>
 #include <linux/filter.h>
 
-#include "bpf_helpers.h"
+#include <bpf/bpf_helpers.h>
+#include "llvm_bpfload.h"
 
 /* vlan tracking: set it to 0 if you don't use VLAN for flow tracking */
 #define VLAN_TRACKING    1
@@ -61,19 +62,19 @@ struct pair {
     __u64 bytes;
 };
 
-struct bpf_map_def SEC("maps") flow_table_v4 = {
-    .type = BPF_MAP_TYPE_PERCPU_HASH,
-    .key_size = sizeof(struct flowv4_keys),
-    .value_size = sizeof(struct pair),
-    .max_entries = 32768,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
+    __type(key, struct flowv4_keys);
+    __type(value, struct pair);
+    __uint(max_entries, 32768);
+} flow_table_v4 SEC(".maps");
 
-struct bpf_map_def SEC("maps") flow_table_v6 = {
-    .type = BPF_MAP_TYPE_PERCPU_HASH,
-    .key_size = sizeof(struct flowv6_keys),
-    .value_size = sizeof(struct pair),
-    .max_entries = 32768,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
+    __type(key, struct flowv6_keys);
+    __type(value, struct pair);
+    __uint(max_entries, 32768);
+} flow_table_v6 SEC(".maps");
 
 struct vlan_hdr {
     __u16	h_vlan_TCI;
diff --git a/ebpf/filter.c b/ebpf/filter.c
index 38aeb701a3b4..ce6eb6006294 100644
--- a/ebpf/filter.c
+++ b/ebpf/filter.c
@@ -25,18 +25,19 @@
 #include <linux/ipv6.h>
 #include <linux/filter.h>
 
-#include "bpf_helpers.h"
+#include <bpf/bpf_helpers.h>
+#include "llvm_bpfload.h"
 
 #define DEBUG 0
 
 #define LINUX_VERSION_CODE 263682
 
-struct bpf_map_def SEC("maps") ipv4_drop = {
-    .type = BPF_MAP_TYPE_PERCPU_HASH,
-    .key_size = sizeof(__u32),
-    .value_size = sizeof(__u32),
-    .max_entries = 32768,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, 32768);
+} ipv4_drop SEC(".maps");
 
 struct vlan_hdr {
     __u16   h_vlan_TCI;
diff --git a/ebpf/lb.c b/ebpf/lb.c
index cd4e6bec9a4f..e95a409f5e8b 100644
--- a/ebpf/lb.c
+++ b/ebpf/lb.c
@@ -25,7 +25,8 @@
 #include <linux/ipv6.h>
 #include <linux/filter.h>
 
-#include "bpf_helpers.h"
+#include <bpf/bpf_helpers.h>
+#include "llvm_bpfload.h"
 
 #define LINUX_VERSION_CODE 263682
 
diff --git a/ebpf/llvm_bpfload.h b/ebpf/llvm_bpfload.h
new file mode 100644
index 000000000000..f7667aed681c
--- /dev/null
+++ b/ebpf/llvm_bpfload.h
@@ -0,0 +1,7 @@
+/* llvm builtin functions that eBPF C program may use to
+ * emit BPF_LD_ABS and BPF_LD_IND instructions
+ */
+struct sk_buff;
+unsigned long long load_byte(void *skb, unsigned long long off) asm("llvm.bpf.load.byte");
+unsigned long long load_half(void *skb, unsigned long long off) asm("llvm.bpf.load.half");
+unsigned long long load_word(void *skb, unsigned long long off) asm("llvm.bpf.load.word");
diff --git a/ebpf/vlan_filter.c b/ebpf/vlan_filter.c
index d797b94bfbd5..0615dbc23ac1 100644
--- a/ebpf/vlan_filter.c
+++ b/ebpf/vlan_filter.c
@@ -18,7 +18,7 @@
 #include <stddef.h>
 #include <linux/bpf.h>
 
-#include "bpf_helpers.h"
+#include <bpf/bpf_helpers.h>
 
 #define LINUX_VERSION_CODE 263682
 
diff --git a/ebpf/xdp_filter.c b/ebpf/xdp_filter.c
index 9ef2d92f7bfb..6c5e8b644aec 100644
--- a/ebpf/xdp_filter.c
+++ b/ebpf/xdp_filter.c
@@ -27,7 +27,8 @@
 #include <linux/ipv6.h>
 #include <linux/tcp.h>
 #include <linux/udp.h>
-#include "bpf_helpers.h"
+
+#include <bpf/bpf_helpers.h>
 
 #include "hash_func01.h"
 
@@ -94,97 +95,96 @@ struct pair {
     __u64 bytes;
 };
 
-struct bpf_map_def SEC("maps") flow_table_v4 = {
+struct {
 #if USE_PERCPU_HASH
-    .type = BPF_MAP_TYPE_PERCPU_HASH,
+    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
 #else
-    .type = BPF_MAP_TYPE_HASH,
+    __uint(type, BPF_MAP_TYPE_HASH);
 #endif
-    .key_size = sizeof(struct flowv4_keys),
-    .value_size = sizeof(struct pair),
-    .max_entries = 32768,
-};
+    __type(key, struct flowv4_keys);
+    __type(value, struct pair);
+    __uint(max_entries, 32768);
+} flow_table_v4 SEC(".maps");
 
-struct bpf_map_def SEC("maps") flow_table_v6 = {
+struct {
 #if USE_PERCPU_HASH
-    .type = BPF_MAP_TYPE_PERCPU_HASH,
+    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
 #else
-    .type = BPF_MAP_TYPE_HASH,
+    __uint(type, BPF_MAP_TYPE_HASH);
 #endif
-    .key_size = sizeof(struct flowv6_keys),
-    .value_size = sizeof(struct pair),
-    .max_entries = 32768,
-};
-
+    __type(key, struct flowv6_keys);
+    __type(value, struct pair);
+    __uint(max_entries, 32768);
+} flow_table_v6 SEC(".maps");
 
 #if ENCRYPTED_TLS_BYPASS
-struct bpf_map_def SEC("maps") tls_bypass_count = {
+struct {
 #if USE_PERCPU_HASH
-    .type		= BPF_MAP_TYPE_PERCPU_ARRAY,
+    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
 #else
-    .type		= BPF_MAP_TYPE_ARRAY,
+    __uint(type, BPF_MAP_TYPE_ARRAY);
 #endif
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u64),
-    .max_entries	= 1,
-};
+    __type(key, __u32);
+    __type(value, __u64);
+    __uint(max_entries, 1);
+} tls_bypass_count SEC(".maps");
 #endif
 
 #if BUILD_CPUMAP
 /* Special map type that can XDP_REDIRECT frames to another CPU */
-struct bpf_map_def SEC("maps") cpu_map = {
-    .type		= BPF_MAP_TYPE_CPUMAP,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= CPUMAP_MAX_CPUS,
-};
-
-struct bpf_map_def SEC("maps") cpus_available = {
-    .type		= BPF_MAP_TYPE_ARRAY,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= CPUMAP_MAX_CPUS,
-};
-
-struct bpf_map_def SEC("maps") cpus_count = {
-    .type		= BPF_MAP_TYPE_ARRAY,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= 1,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_CPUMAP);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, CPUMAP_MAX_CPUS);
+} cpu_map SEC(".maps");
+
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, CPUMAP_MAX_CPUS);
+} cpus_available SEC(".maps");
+
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, 1);
+} cpus_count SEC(".maps");
 #endif
 
 #if GOT_TX_PEER
 /* Map has only one element as we don't handle any sort of
  * routing for now. Key value set by user space is 0 and
  * value is the peer interface. */
-struct bpf_map_def SEC("maps") tx_peer = {
-    .type = BPF_MAP_TYPE_DEVMAP,
-    .key_size = sizeof(int),
-    .value_size = sizeof(int),
-    .max_entries = 1,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_DEVMAP);
+    __type(key, int);
+    __type(value, int);
+    __uint(max_entries, 1);
+} tx_peer SEC(".maps");
 
 /* single entry to indicate if we have peer, key value
  * set in user space is 0. It is only used to see if
  * a interface has a peer we need to send the information to */
-struct bpf_map_def SEC("maps") tx_peer_int = {
-    .type = BPF_MAP_TYPE_ARRAY,
-    .key_size = sizeof(int),
-    .value_size = sizeof(int),
-    .max_entries = 1,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, int);
+    __type(value, int);
+    __uint(max_entries, 1);
+} tx_peer_int SEC(".maps");
 #endif
 
 #define USE_GLOBAL_BYPASS   0
 #if USE_GLOBAL_BYPASS
 /* single entry to indicate if global bypass switch is on */
-struct bpf_map_def SEC("maps") global_bypass = {
-    .type = BPF_MAP_TYPE_ARRAY,
-    .key_size = sizeof(char),
-    .value_size = sizeof(char),
-    .max_entries = 1,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, char);
+    __type(value, char);
+    __uint(max_entries, 1);
+} global_bypass SEC(".maps");
 #endif
 
 
diff --git a/ebpf/xdp_lb.c b/ebpf/xdp_lb.c
index a88724ea2885..87846b12062d 100644
--- a/ebpf/xdp_lb.c
+++ b/ebpf/xdp_lb.c
@@ -31,7 +31,8 @@
 #include <linux/ipv6.h>
 #include <linux/tcp.h>
 #include <linux/udp.h>
-#include "bpf_helpers.h"
+
+#include <bpf/bpf_helpers.h>
 
 #include "hash_func01.h"
 
@@ -49,26 +50,26 @@ struct vlan_hdr {
 };
 
 /* Special map type that can XDP_REDIRECT frames to another CPU */
-struct bpf_map_def SEC("maps") cpu_map = {
-    .type		= BPF_MAP_TYPE_CPUMAP,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= CPUMAP_MAX_CPUS,
-};
-
-struct bpf_map_def SEC("maps") cpus_available = {
-    .type		= BPF_MAP_TYPE_ARRAY,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= CPUMAP_MAX_CPUS,
-};
-
-struct bpf_map_def SEC("maps") cpus_count = {
-    .type		= BPF_MAP_TYPE_ARRAY,
-    .key_size	= sizeof(__u32),
-    .value_size	= sizeof(__u32),
-    .max_entries	= 1,
-};
+struct {
+    __uint(type, BPF_MAP_TYPE_CPUMAP);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, CPUMAP_MAX_CPUS);
+} cpu_map SEC(".maps");
+
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, CPUMAP_MAX_CPUS);
+} cpus_available SEC(".maps");
+
+struct {
+    __uint(type, BPF_MAP_TYPE_ARRAY);
+    __type(key, __u32);
+    __type(value, __u32);
+    __uint(max_entries, 1);
+} cpus_count SEC(".maps");
 
 static int __always_inline hash_ipv4(void *data, void *data_end)
 {

From bbc17b1c7dbb0cb56fb3c03411ed9c61bb2d86a0 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 20:26:29 +0000
Subject: [PATCH 226/462] doc: add file.name information to http keyword doc

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/http-keywords.rst | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/doc/userguide/rules/http-keywords.rst b/doc/userguide/rules/http-keywords.rst
index 0c0f652ad397..ba0d7621f339 100644
--- a/doc/userguide/rules/http-keywords.rst
+++ b/doc/userguide/rules/http-keywords.rst
@@ -44,6 +44,7 @@ http.accept_enc                http_accept_enc (*)      Request
 http.referer                   http_referer (*)         Request
 http.connection                http_connection (*)      Both
 file.data                      file_data (*)            Both
+file.name                      filename (*)             Request
 http.content_type              http_content_type (*)    Both
 http.content_len               http_content_len (*)     Both
 http.start                     http_start (*)           Both
@@ -670,7 +671,6 @@ Example::
     alert http any any -> any any (flow:to_client; \
             http.location; content:"http://www.google.com"; sid:1;)
 
-
 http.host and http.host.raw
 ---------------------------
 
@@ -843,4 +843,16 @@ Notes
 Multiple Buffer Matching
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-``file.data`` supports multiple buffer matching, see :doc:`multi-buffer-matching`.
\ No newline at end of file
+``file.data`` supports multiple buffer matching, see :doc:`multi-buffer-matching`.
+
+file.name
+---------
+
+The ``file.name`` keyword can be used at the HTTP application level.
+
+Example::
+
+  alert http any any -> any any (msg:"http layer file.name keyword usage"; \
+  file.name; content:"picture.jpg"; classtype:bad-unknown; sid:1; rev:1;)
+
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From bb1f7575d388234afcf2dd85f67f3da9b8b40dea Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 20:44:36 +0000
Subject: [PATCH 227/462] doc: add file.name information to ftp keyword doc

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/ftp-keywords.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/doc/userguide/rules/ftp-keywords.rst b/doc/userguide/rules/ftp-keywords.rst
index 068b14e33cc7..0d25f6031213 100644
--- a/doc/userguide/rules/ftp-keywords.rst
+++ b/doc/userguide/rules/ftp-keywords.rst
@@ -29,3 +29,15 @@ Detect FTP bounce attacks.
 Syntax::
 
   ftpbounce
+
+file.name
+---------
+
+The ``file.name`` keyword can be used at the FTP application level.
+
+Example::
+
+alert ftp-data any any -> any any (msg:"ftp layer file.name keyword usage"; \
+file.name; content:"file.txt"; classtype:bad-unknown; sid:1; rev:1;)
+
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From e4077b880365cf447b94dd10781a7754ea778949 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 20:51:52 +0000
Subject: [PATCH 228/462] doc: update ftp keyword doc example rule format

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/ftp-keywords.rst | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/doc/userguide/rules/ftp-keywords.rst b/doc/userguide/rules/ftp-keywords.rst
index 0d25f6031213..d93451684486 100644
--- a/doc/userguide/rules/ftp-keywords.rst
+++ b/doc/userguide/rules/ftp-keywords.rst
@@ -1,6 +1,8 @@
 FTP/FTP-DATA Keywords
 =====================
 
+.. role:: example-rule-options
+
 ftpdata_command
 ---------------
 
@@ -12,14 +14,13 @@ Syntax::
 
   ftpdata_command:(retr|stor)
 
-Examples::
-
-  ftpdata_command:retr
-  ftpdata_command:stor
+Signature Example:
 
-Signature example::
+.. container:: example-rule
 
- alert ftp-data any any -> any any (msg:"FTP store password"; filestore; filename:"password"; ftpdata_command:stor; sid:3; rev:1;)
+  alert ftp-data any any -> any any (msg:"FTP store password"; \
+  filestore; filename:"password"; \
+  :example-rule-options:`ftpdata_command:stor;` sid:3; rev:1;)
 
 ftpbounce
 ---------
@@ -35,9 +36,12 @@ file.name
 
 The ``file.name`` keyword can be used at the FTP application level.
 
-Example::
+Signature Example:
+
+.. container:: example-rule
 
-alert ftp-data any any -> any any (msg:"ftp layer file.name keyword usage"; \
-file.name; content:"file.txt"; classtype:bad-unknown; sid:1; rev:1;)
+  alert ftp-data any any -> any any (msg:"FTP file.name usage"; \
+  :example-rule-options:`file.name; content:"file.txt";` \
+  classtype:bad-unknown; sid:1; rev:1;)
 
 For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From 327ba7397a42c20e21c006760363c8130b8cfef5 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 20:58:21 +0000
Subject: [PATCH 229/462] doc: add file.name information to smb keyword doc

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/smb-keywords.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/doc/userguide/rules/smb-keywords.rst b/doc/userguide/rules/smb-keywords.rst
index 02cf190bc794..13133354403d 100644
--- a/doc/userguide/rules/smb-keywords.rst
+++ b/doc/userguide/rules/smb-keywords.rst
@@ -1,6 +1,8 @@
 SMB Keywords
 ==============
 
+.. role:: example-rule-options
+
 SMB keywords used in both SMB1 and SMB2 protocols.
 
 smb.named_pipe
@@ -58,3 +60,18 @@ Examples::
 ``smb.ntlmssp_domain`` is a 'sticky buffer'.
 
 ``smb.ntlmssp_domain`` can be used as ``fast_pattern``.
+
+file.name
+---------
+
+The ``file.name`` keyword can be used at the SMB application level. 
+
+Signature Example:
+
+.. container:: example-rule
+
+  alert smb any any -> any any (msg:"SMB file.name usage"; \
+  :example-rule-options:`file.name; content:"file.txt";` \
+  classtype:bad-unknown; sid:1; rev:1;)
+
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From 9d1ad0187e0cda5db6be413bbfb016c78a1e8d8d Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 21:42:10 +0000
Subject: [PATCH 230/462] doc: add file.name information to nfs keyword doc

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/index.rst        |  1 +
 doc/userguide/rules/nfs-keywords.rst | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 doc/userguide/rules/nfs-keywords.rst

diff --git a/doc/userguide/rules/index.rst b/doc/userguide/rules/index.rst
index 76266b334581..2715da79ac7d 100644
--- a/doc/userguide/rules/index.rst
+++ b/doc/userguide/rules/index.rst
@@ -33,6 +33,7 @@ Suricata Rules
    ike-keywords
    http2-keywords
    quic-keywords
+   nfs-keywords
    app-layer
    xbits
    thresholding
diff --git a/doc/userguide/rules/nfs-keywords.rst b/doc/userguide/rules/nfs-keywords.rst
new file mode 100644
index 000000000000..22c3304f63e8
--- /dev/null
+++ b/doc/userguide/rules/nfs-keywords.rst
@@ -0,0 +1,19 @@
+NFS Keywords
+============
+
+.. role:: example-rule-options
+
+file.name
+---------
+
+The ``file.name`` keyword can be used at the NFS application level. 
+
+Signature Example:
+
+.. container:: example-rule
+
+  alert nfs any any -> any any (msg:"NFS file.name usage"; \
+  :example-rule-options:`file.name; content:"file.txt";` \
+  classtype:bad-unknown; sid:1; rev:1;)
+
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From fc81c99b587af6201f57948929e2de13172078c9 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 20 Sep 2023 21:46:04 +0000
Subject: [PATCH 231/462] doc: add file.name information to smtp keyword doc

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/index.rst         |  1 +
 doc/userguide/rules/smtp-keywords.rst | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 doc/userguide/rules/smtp-keywords.rst

diff --git a/doc/userguide/rules/index.rst b/doc/userguide/rules/index.rst
index 2715da79ac7d..e174c6787bc5 100644
--- a/doc/userguide/rules/index.rst
+++ b/doc/userguide/rules/index.rst
@@ -34,6 +34,7 @@ Suricata Rules
    http2-keywords
    quic-keywords
    nfs-keywords
+   smtp-keywords
    app-layer
    xbits
    thresholding
diff --git a/doc/userguide/rules/smtp-keywords.rst b/doc/userguide/rules/smtp-keywords.rst
new file mode 100644
index 000000000000..ec91f6fc0c1e
--- /dev/null
+++ b/doc/userguide/rules/smtp-keywords.rst
@@ -0,0 +1,19 @@
+SMTP Keywords
+=============
+
+.. role:: example-rule-options
+
+file.name
+---------
+
+The ``file.name`` keyword can be used at the SMTP application level. 
+
+Signature Example:
+
+.. container:: example-rule
+
+  alert smtp any any -> any any (msg:"SMTP file.name usage"; \
+  :example-rule-options:`file.name; content:"winmail.dat";` \
+  classtype:bad-unknown; sid:1; rev:1;)
+
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file

From bdec2d8ea80cf69a8d1ce9c4505452f17f6058fe Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Tue, 5 Dec 2023 10:56:28 -0300
Subject: [PATCH 232/462] pgsql: don't log password msg if password disabled

If the logging of the password is disabled, there isn't much point in
logging the password message itself.
---
 rust/src/pgsql/logger.rs | 2 --
 1 file changed, 2 deletions(-)

diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index b17986dec85b..4a6f24880252 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -78,8 +78,6 @@ fn log_request(req: &PgsqlFEMessage, flags: u32) -> Result<JsonBuilder, JsonErro
         }) => {
             if flags & PGSQL_LOG_PASSWORDS != 0 {
                 js.set_string_from_bytes("password", payload)?;
-            } else {
-                js.set_string(req.to_str(), "password log disabled")?;
             }
         }
         PgsqlFEMessage::SASLResponse(RegularPacket {

From d3095ac0589228a4c63a466f5ff3b99bab98bf25 Mon Sep 17 00:00:00 2001
From: Ilya Bakhtin <ilya.bakhtin@gmail.com>
Date: Fri, 29 Sep 2023 16:23:04 +0200
Subject: [PATCH 233/462] util/time: Prevent usecs overflow

This commit takes care of original seconds value and prevents the
useconds field from overflowing pas its maximum value.

Issue: 6372
---
 src/util-time.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/src/util-time.h b/src/util-time.h
index 9bbd8798dd17..479f9a2b352f 100644
--- a/src/util-time.h
+++ b/src/util-time.h
@@ -57,7 +57,11 @@ typedef struct {
 #define SCTIME_SECS(t)           ((uint64_t)(t).secs)
 #define SCTIME_MSECS(t)          (SCTIME_SECS(t) * 1000 + SCTIME_USECS(t) / 1000)
 #define SCTIME_ADD_SECS(ts, s)   SCTIME_FROM_SECS((ts).secs + (s))
-#define SCTIME_ADD_USECS(ts, us) SCTIME_FROM_USECS((ts).usecs + (us))
+#define SCTIME_ADD_USECS(ts, us)                                                                   \
+    (SCTime_t)                                                                                     \
+    {                                                                                              \
+        .secs = (ts).secs + ((ts).usecs + (us)) / 1000000, .usecs = ((ts).usecs + (us)) % 1000000  \
+    }
 #define SCTIME_FROM_SECS(s)                                                                        \
     (SCTime_t)                                                                                     \
     {                                                                                              \

From 417806ca35e643d4120552ac569bba7484295e70 Mon Sep 17 00:00:00 2001
From: Ilya Bakhtin <ilya.bakhtin@gmail.com>
Date: Fri, 29 Sep 2023 16:56:48 +0200
Subject: [PATCH 234/462] napatech: Fix packet timestamps

Initialize both seconds and useconds of packet timestamp from napatech
timestamp format.

This commit uses updated macro definitions from util-utime.h to avoid
zero seconds value.

Issue: 6372
---
 src/source-napatech.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/source-napatech.c b/src/source-napatech.c
index 071d9ae68416..54575d431e21 100644
--- a/src/source-napatech.c
+++ b/src/source-napatech.c
@@ -931,19 +931,19 @@ TmEcode NapatechPacketLoop(ThreadVars *tv, void *data, void *slot)
          */
         switch (NT_NET_GET_PKT_TIMESTAMP_TYPE(packet_buffer)) {
             case NT_TIMESTAMP_TYPE_NATIVE_UNIX:
-                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts / 100000000),
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_SECS(pkt_ts / 100000000),
                         ((pkt_ts % 100000000) / 100) + ((pkt_ts % 100) > 50 ? 1 : 0));
                 break;
             case NT_TIMESTAMP_TYPE_PCAP:
-                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts >> 32), pkt_ts & 0xFFFFFFFF);
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_SECS(pkt_ts >> 32), pkt_ts & 0xFFFFFFFF);
                 break;
             case NT_TIMESTAMP_TYPE_PCAP_NANOTIME:
-                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS(pkt_ts >> 32),
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_SECS(pkt_ts >> 32),
                         ((pkt_ts & 0xFFFFFFFF) / 1000) + ((pkt_ts % 1000) > 500 ? 1 : 0));
                 break;
             case NT_TIMESTAMP_TYPE_NATIVE_NDIS:
                 /* number of seconds between 1/1/1601 and 1/1/1970 */
-                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_USECS((pkt_ts / 100000000) - 11644473600),
+                p->ts = SCTIME_ADD_USECS(SCTIME_FROM_SECS((pkt_ts / 100000000) - 11644473600),
                         ((pkt_ts % 100000000) / 100) + ((pkt_ts % 100) > 50 ? 1 : 0));
                 break;
             default:

From 0850e3d137e553e22fd8f598d276794ddd098c41 Mon Sep 17 00:00:00 2001
From: Stephen Donnelly <stephen.donnelly@endace.com>
Date: Wed, 29 Nov 2023 10:23:45 +1300
Subject: [PATCH 235/462] util/time: Improve usecs handling in time macros

Fix SCTIME_ADD_SECS zeroing subsecond part

When adding s seconds to SCtime_t ts, don't zero out the ts.usecs field.

Issue: 6584

Fix SCTIME_FROM_TIMESPEC garbage microseconds part

When converting nanosecond to microseconds divide by 1000 instead
of multiplying by 1000.

Issue: 6585
---
 src/util-time.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/util-time.h b/src/util-time.h
index 479f9a2b352f..b0f7207b3c86 100644
--- a/src/util-time.h
+++ b/src/util-time.h
@@ -56,12 +56,16 @@ typedef struct {
 #define SCTIME_USECS(t)          ((uint64_t)(t).usecs)
 #define SCTIME_SECS(t)           ((uint64_t)(t).secs)
 #define SCTIME_MSECS(t)          (SCTIME_SECS(t) * 1000 + SCTIME_USECS(t) / 1000)
-#define SCTIME_ADD_SECS(ts, s)   SCTIME_FROM_SECS((ts).secs + (s))
 #define SCTIME_ADD_USECS(ts, us)                                                                   \
     (SCTime_t)                                                                                     \
     {                                                                                              \
         .secs = (ts).secs + ((ts).usecs + (us)) / 1000000, .usecs = ((ts).usecs + (us)) % 1000000  \
     }
+#define SCTIME_ADD_SECS(ts, s)                                                                     \
+    (SCTime_t)                                                                                     \
+    {                                                                                              \
+        .secs = (ts).secs + (s), .usecs = (ts).usecs                                               \
+    }
 #define SCTIME_FROM_SECS(s)                                                                        \
     (SCTime_t)                                                                                     \
     {                                                                                              \
@@ -87,7 +91,7 @@ typedef struct {
 #define SCTIME_FROM_TIMESPEC(ts)                                                                   \
     (SCTime_t)                                                                                     \
     {                                                                                              \
-        .secs = (ts)->tv_sec, .usecs = (ts)->tv_nsec * 1000                                        \
+        .secs = (ts)->tv_sec, .usecs = (ts)->tv_nsec / 1000                                        \
     }
 
 #define SCTIME_TO_TIMEVAL(tv, t)                                                                   \

From a7c9028f01caa69fb2c61fdb4c1af79a7b02d116 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 09:09:07 +0200
Subject: [PATCH 236/462] detect/content-inspect: reduce scope of variables

---
 src/detect-engine-content-inspection.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 0070494380c2..170892044d2a 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -133,13 +133,12 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
         /* search for our pattern, checking the matches recursively.
          * if we match we look for the next SigMatch as well */
-        const uint8_t *found = NULL;
-        uint32_t offset = 0;
-        uint32_t depth = buffer_len;
         uint32_t prev_offset = 0; /**< used in recursive searching */
         uint32_t prev_buffer_offset = det_ctx->buffer_offset;
 
         do {
+            uint32_t depth = buffer_len;
+            uint32_t offset = 0;
             if ((cd->flags & DETECT_CONTENT_DISTANCE) ||
                 (cd->flags & DETECT_CONTENT_WITHIN)) {
                 SCLogDebug("det_ctx->buffer_offset %" PRIu32, det_ctx->buffer_offset);
@@ -270,6 +269,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 #ifdef DEBUG
             BUG_ON(sbuffer_len > buffer_len);
 #endif
+            const uint8_t *found;
             if (cd->flags & DETECT_CONTENT_ENDS_WITH && depth < buffer_len) {
                 SCLogDebug("depth < buffer_len while DETECT_CONTENT_ENDS_WITH is set. Can't possibly match.");
                 found = NULL;

From 0ba4b297df88f9753cd391f2c5958603c95ff12e Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 09:09:33 +0200
Subject: [PATCH 237/462] detect/content-inspect: pass const to inspect func

---
 src/detect-engine-content-inspection.c | 3 ++-
 src/detect-engine-content-inspection.h | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 170892044d2a..1fa632025017 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -97,7 +97,8 @@
  */
 int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
-        uint32_t buffer_len, uint32_t stream_start_offset, uint8_t flags, uint8_t inspection_mode)
+        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
+        const uint8_t inspection_mode)
 {
     SCEnter();
     KEYWORD_PROFILING_START;
diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index 188ebef2d881..1d04a48b1855 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -54,7 +54,8 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 /* implicit "public" just returns true match, false no match */
 bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
-        uint32_t buffer_len, uint32_t stream_start_offset, uint8_t flags, uint8_t inspection_mode);
+        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
+        const uint8_t inspection_mode);
 
 void DetectEngineContentInspectionRegisterTests(void);
 

From b1fa9755035a5ebfc2d67ea69d8436fb52d8f693 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 23 Sep 2023 09:26:12 +0200
Subject: [PATCH 238/462] detect/content-inspect: remove const casting

---
 src/detect-dns-query.c              | 2 +-
 src/detect-engine-frame.c           | 9 ++-------
 src/detect-engine-payload.c         | 4 ++--
 src/detect-engine.c                 | 5 ++---
 src/detect-file-data.c              | 2 +-
 src/detect-filemagic.c              | 2 +-
 src/detect-filename.c               | 2 +-
 src/detect-http-client-body.c       | 5 ++---
 src/detect-http-header.c            | 2 +-
 src/detect-http2.c                  | 2 +-
 src/detect-ike-vendor.c             | 2 +-
 src/detect-krb5-cname.c             | 2 +-
 src/detect-krb5-sname.c             | 2 +-
 src/detect-mqtt-subscribe-topic.c   | 2 +-
 src/detect-mqtt-unsubscribe-topic.c | 2 +-
 src/detect-quic-cyu-hash.c          | 2 +-
 src/detect-quic-cyu-string.c        | 2 +-
 src/detect-template-rust-buffer.c   | 2 +-
 src/detect-tls-certs.c              | 2 +-
 19 files changed, 23 insertions(+), 30 deletions(-)

diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index d2dbe8e99021..a0bf46f3867d 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -115,7 +115,7 @@ static uint8_t DetectEngineInspectDnsQuery(DetectEngineCtx *de_ctx, DetectEngine
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-engine-frame.c b/src/detect-engine-frame.c
index 0ed70757d599..71ac9d3d1674 100644
--- a/src/detect-engine-frame.c
+++ b/src/detect-engine-frame.c
@@ -306,13 +306,8 @@ static int DetectFrameInspectUdp(DetectEngineThreadCtx *det_ctx,
     if (buffer->inspect == NULL)
         return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
 
-    const uint32_t data_len = buffer->inspect_len;
-    const uint8_t *data = buffer->inspect;
-
-    // PrintRawDataFp(stdout, data, data_len);
-
     const bool match = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p,
-            p->flow, (uint8_t *)data, data_len, 0, buffer->flags,
+            p->flow, buffer->inspect, buffer->inspect_len, 0, buffer->flags,
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME);
     if (match) {
         SCLogDebug("match!");
@@ -479,7 +474,7 @@ static int FrameStreamDataInspectFunc(
     BUG_ON(fsd->frame->len > 0 && (int64_t)data_len > fsd->frame->len);
 
     const bool match = DetectEngineContentInspection(det_ctx->de_ctx, det_ctx, s, engine->smd, p,
-            p->flow, (uint8_t *)data, data_len, data_offset, buffer->flags,
+            p->flow, data, data_len, data_offset, buffer->flags,
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME);
     if (match) {
         SCLogDebug("DETECT_ENGINE_INSPECT_SIG_MATCH");
diff --git a/src/detect-engine-payload.c b/src/detect-engine-payload.c
index 7da3c3b81f93..d051303ddbb1 100644
--- a/src/detect-engine-payload.c
+++ b/src/detect-engine-payload.c
@@ -227,7 +227,7 @@ static int StreamContentInspectFunc(
 #endif
 
     const bool match = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx, smd->s,
-            smd->s->sm_arrays[DETECT_SM_LIST_PMATCH], NULL, smd->f, (uint8_t *)data, data_len, 0,
+            smd->s->sm_arrays[DETECT_SM_LIST_PMATCH], NULL, smd->f, data, data_len, 0,
             0, // TODO
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM);
     if (match) {
@@ -282,7 +282,7 @@ static int StreamContentInspectEngineFunc(
 #endif
 
     const bool match = DetectEngineContentInspection(smd->de_ctx, smd->det_ctx, smd->s, smd->smd,
-            NULL, smd->f, (uint8_t *)data, data_len, 0, 0, // TODO
+            NULL, smd->f, data, data_len, 0, 0, // TODO
             DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM);
     if (match) {
         SCReturnInt(1);
diff --git a/src/detect-engine.c b/src/detect-engine.c
index a4ce2126544d..58aee1bfc078 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2206,9 +2206,8 @@ uint8_t DetectEngineInspectBufferGeneric(DetectEngineCtx *de_ctx, DetectEngineTh
 
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    const bool match =
-            DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, (uint8_t *)data,
-                    data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+    const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, data,
+            data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
     if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     } else {
diff --git a/src/detect-file-data.c b/src/detect-file-data.c
index f31715adda01..533fc8441d93 100644
--- a/src/detect-file-data.c
+++ b/src/detect-file-data.c
@@ -415,7 +415,7 @@ uint8_t DetectEngineInspectFiledata(DetectEngineCtx *de_ctx, DetectEngineThreadC
             ciflags |= DETECT_CI_FLAGS_START;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset, ciflags,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset, ciflags,
                 DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-filemagic.c b/src/detect-filemagic.c
index 7ade159fb52d..b7a737e6c7b7 100644
--- a/src/detect-filemagic.c
+++ b/src/detect-filemagic.c
@@ -321,7 +321,7 @@ static uint8_t DetectEngineInspectFilemagic(DetectEngineCtx *de_ctx, DetectEngin
             continue;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-filename.c b/src/detect-filename.c
index 88e580862452..10646f019f1a 100644
--- a/src/detect-filename.c
+++ b/src/detect-filename.c
@@ -258,7 +258,7 @@ static uint8_t DetectEngineInspectFilename(DetectEngineCtx *de_ctx, DetectEngine
             continue;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-http-client-body.c b/src/detect-http-client-body.c
index 1d3d7a87cc88..9e14b1b49fad 100644
--- a/src/detect-http-client-body.c
+++ b/src/detect-http-client-body.c
@@ -325,9 +325,8 @@ static uint8_t DetectEngineInspectBufferHttpBody(DetectEngineCtx *de_ctx,
 
     /* Inspect all the uricontents fetched on each
      * transaction at the app layer */
-    const bool match =
-            DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, (uint8_t *)data,
-                    data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+    const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f, data,
+            data_len, offset, ci_flags, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
     if (match) {
         return DETECT_ENGINE_INSPECT_SIG_MATCH;
     }
diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index a4596c4085f2..91e17d886379 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -545,7 +545,7 @@ static uint8_t DetectEngineInspectHttp2Header(DetectEngineCtx *de_ctx,
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-http2.c b/src/detect-http2.c
index a1ede963825e..a65115a14cea 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -703,7 +703,7 @@ static uint8_t DetectEngineInspectHttp2HeaderName(DetectEngineCtx *de_ctx,
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-ike-vendor.c b/src/detect-ike-vendor.c
index f5c5b94f35d5..3b84da26660b 100644
--- a/src/detect-ike-vendor.c
+++ b/src/detect-ike-vendor.c
@@ -156,7 +156,7 @@ static uint8_t DetectEngineInspectIkeVendor(DetectEngineCtx *de_ctx, DetectEngin
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-krb5-cname.c b/src/detect-krb5-cname.c
index 8664f2bc2877..d509116ee73b 100644
--- a/src/detect-krb5-cname.c
+++ b/src/detect-krb5-cname.c
@@ -104,7 +104,7 @@ static uint8_t DetectEngineInspectKrb5CName(DetectEngineCtx *de_ctx, DetectEngin
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index 1e4ae24a4bd1..9fbe550b02f9 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -105,7 +105,7 @@ static uint8_t DetectEngineInspectKrb5SName(DetectEngineCtx *de_ctx, DetectEngin
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-mqtt-subscribe-topic.c b/src/detect-mqtt-subscribe-topic.c
index 9eaf39d3029c..9e0058785ec1 100644
--- a/src/detect-mqtt-subscribe-topic.c
+++ b/src/detect-mqtt-subscribe-topic.c
@@ -108,7 +108,7 @@ static uint8_t DetectEngineInspectMQTTSubscribeTopic(DetectEngineCtx *de_ctx,
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-mqtt-unsubscribe-topic.c b/src/detect-mqtt-unsubscribe-topic.c
index 268d72bc8789..297142dd83da 100644
--- a/src/detect-mqtt-unsubscribe-topic.c
+++ b/src/detect-mqtt-unsubscribe-topic.c
@@ -108,7 +108,7 @@ static uint8_t DetectEngineInspectMQTTUnsubscribeTopic(DetectEngineCtx *de_ctx,
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-quic-cyu-hash.c b/src/detect-quic-cyu-hash.c
index 88197a5e382a..246f36a41efe 100644
--- a/src/detect-quic-cyu-hash.c
+++ b/src/detect-quic-cyu-hash.c
@@ -107,7 +107,7 @@ static uint8_t DetectEngineInspectQuicHash(DetectEngineCtx *de_ctx, DetectEngine
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-quic-cyu-string.c b/src/detect-quic-cyu-string.c
index 9290fa41233c..dfec432c2049 100644
--- a/src/detect-quic-cyu-string.c
+++ b/src/detect-quic-cyu-string.c
@@ -105,7 +105,7 @@ static uint8_t DetectEngineInspectQuicString(DetectEngineCtx *de_ctx,
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-template-rust-buffer.c b/src/detect-template-rust-buffer.c
index f1c8c97bb278..3f016f4b164b 100644
--- a/src/detect-template-rust-buffer.c
+++ b/src/detect-template-rust-buffer.c
@@ -103,7 +103,7 @@ static uint8_t DetectEngineInspectTemplateRustBuffer(DetectEngineCtx *de_ctx,
 
     if (data != NULL) {
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)data, data_len, 0, DETECT_CI_FLAGS_SINGLE,
+                data, data_len, 0, DETECT_CI_FLAGS_SINGLE,
                 DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             ret = DETECT_ENGINE_INSPECT_SIG_MATCH;
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index 9ff185c494d6..7310461ea235 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -195,7 +195,7 @@ static uint8_t DetectEngineInspectTlsCerts(DetectEngineCtx *de_ctx, DetectEngine
             break;
 
         const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                (uint8_t *)buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
+                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
                 DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;

From 1f265d9d79e1aebb93d9f43da57bcdb88f3741f9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 23 Sep 2023 09:32:14 +0200
Subject: [PATCH 239/462] detect/content-inspect: assist branch prediction

Hitting the recursion limit should be rare.
---
 src/detect-engine-content-inspection.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 1fa632025017..6c13b443488d 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -105,7 +105,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 
     det_ctx->inspection_recursion_counter++;
 
-    if (det_ctx->inspection_recursion_counter == de_ctx->inspection_recursion_limit) {
+    if (unlikely(det_ctx->inspection_recursion_counter == de_ctx->inspection_recursion_limit)) {
         KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
         SCReturnInt(-1);
     }

From b357532e7e69e59028ea4500df19a10be5a74273 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 2 Dec 2023 09:41:02 +0100
Subject: [PATCH 240/462] detect/content-inspect: switch type of enum

---
 src/detect-engine-content-inspection.c | 4 ++--
 src/detect-engine-content-inspection.h | 7 ++++---
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 6c13b443488d..d5ffc135af67 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -98,7 +98,7 @@
 int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
-        const uint8_t inspection_mode)
+        const enum DetectContentInspectionType inspection_mode)
 {
     SCEnter();
     KEYWORD_PROFILING_START;
@@ -701,7 +701,7 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
 bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
-        const uint8_t inspection_mode)
+        const enum DetectContentInspectionType inspection_mode)
 {
     det_ctx->buffer_offset = 0;
     det_ctx->inspection_recursion_counter = 0;
diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index 1d04a48b1855..06c5407f5a67 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -28,7 +28,7 @@
 /** indication to content engine what type of data
  *  we're inspecting
  */
-enum {
+enum DetectContentInspectionType {
     DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD = 0, /* enables 'replace' logic */
     DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER,
     DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM,
@@ -50,12 +50,13 @@ enum {
 int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
-        const uint8_t inspection_mode);
+        const enum DetectContentInspectionType inspection_mode);
+
 /* implicit "public" just returns true match, false no match */
 bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
-        const uint8_t inspection_mode);
+        const enum DetectContentInspectionType inspection_mode);
 
 void DetectEngineContentInspectionRegisterTests(void);
 

From d73cce478c4071cbe796428532955d87afb92d40 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 2 Dec 2023 09:44:06 +0100
Subject: [PATCH 241/462] detect/content-inspect: add entry for
 InspectionBuffer

This is a convinience addition to abstract away the internals of
the InspectionBuffer in keyword specific detection code.
---
 src/detect-engine-content-inspection.c | 19 +++++++++++++++++++
 src/detect-engine-content-inspection.h | 14 ++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index d5ffc135af67..09d838378fea 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -714,6 +714,25 @@ bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCt
         return false;
 }
 
+/** \brief wrapper around DetectEngineContentInspectionInternal to return true/false only
+ *
+ *  \param smd sigmatches to evaluate
+ */
+bool DetectEngineContentInspectionBuffer(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const InspectionBuffer *b,
+        const enum DetectContentInspectionType inspection_mode)
+{
+    det_ctx->buffer_offset = 0;
+    det_ctx->inspection_recursion_counter = 0;
+
+    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd, p, f, b->inspect,
+            b->inspect_len, b->inspect_offset, b->flags, inspection_mode);
+    if (r == 1)
+        return true;
+    else
+        return false;
+}
+
 #ifdef UNITTESTS
 #include "tests/detect-engine-content-inspection.c"
 #endif
diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index 06c5407f5a67..4e362dad8d96 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -58,6 +58,20 @@ bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCt
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
         const enum DetectContentInspectionType inspection_mode);
 
+/** \brief content inspect entry for inspection buffers
+ *  \param de_ctx detection engine
+ *  \param det_ctx detect engine thread ctx
+ *  \param s signature being inspected
+ *  \param smd array of content inspection matches
+ *  \param p packet
+ *  \param f flow
+ *  \param b inspection buffer to inspect
+ *  \param inspection_mode inspection mode to use
+ *  \retval bool true if smd matched the buffer b, false otherwise */
+bool DetectEngineContentInspectionBuffer(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const InspectionBuffer *b,
+        const enum DetectContentInspectionType inspection_mode);
+
 void DetectEngineContentInspectionRegisterTests(void);
 
 #endif /* __DETECT_ENGINE_CONTENT_INSPECTION_H__ */

From c9ab95cbe2a1a6c4348ac7212c01df6afc2a3e88 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 2 Dec 2023 09:44:58 +0100
Subject: [PATCH 242/462] detect/dns.query: use new content inspect entry

---
 src/detect-dns-query.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index a0bf46f3867d..43e1595e491a 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2013-2018 Open Information Security Foundation
+/* Copyright (C) 2013-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -92,6 +92,7 @@ static InspectionBuffer *DnsQueryGetData(DetectEngineThreadCtx *det_ctx,
         return NULL;
     }
     InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    buffer->flags = DETECT_CI_FLAGS_SINGLE;
 
     SCReturnPtr(buffer, "InspectionBuffer");
 }
@@ -114,9 +115,8 @@ static uint8_t DetectEngineInspectDnsQuery(DetectEngineCtx *de_ctx, DetectEngine
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
-                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        const bool match = DetectEngineContentInspectionBuffer(de_ctx, det_ctx, s, engine->smd,
+                NULL, f, buffer, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }

From 6a01f40d40895388b2eace39b1e01c40ef7f66ab Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 2 Dec 2023 09:45:15 +0100
Subject: [PATCH 243/462] detect/krb5.sname: use new content inspect entry

---
 src/detect-krb5-sname.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index 9fbe550b02f9..dae5c46e5215 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -81,6 +81,7 @@ static InspectionBuffer *GetKrb5SNameData(DetectEngineThreadCtx *det_ctx,
     }
 
     InspectionBufferSetupMulti(buffer, transforms, b, b_len);
+    buffer->flags = DETECT_CI_FLAGS_SINGLE;
 
     SCReturnPtr(buffer, "InspectionBuffer");
 }
@@ -100,13 +101,11 @@ static uint8_t DetectEngineInspectKrb5SName(DetectEngineCtx *de_ctx, DetectEngin
         struct Krb5PrincipalNameDataArgs cbdata = { local_id, txv, };
         InspectionBuffer *buffer =
                 GetKrb5SNameData(det_ctx, transforms, f, &cbdata, engine->sm_list);
-
         if (buffer == NULL || buffer->inspect == NULL)
             break;
 
-        const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s, engine->smd, NULL, f,
-                buffer->inspect, buffer->inspect_len, buffer->inspect_offset,
-                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        const bool match = DetectEngineContentInspectionBuffer(de_ctx, det_ctx, s, engine->smd,
+                NULL, f, buffer, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
         if (match) {
             return DETECT_ENGINE_INSPECT_SIG_MATCH;
         }

From e9b33c48f02e539874e01bd9adbbd3fcc3f031c7 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 23 Sep 2023 13:01:05 +0200
Subject: [PATCH 244/462] detect/base64: move content inspection logic

Integrate with rest of content inspect code.
---
 src/detect-base64-data.c               | 13 -------------
 src/detect-base64-data.h               |  2 --
 src/detect-engine-content-inspection.c | 14 ++++++++++----
 3 files changed, 10 insertions(+), 19 deletions(-)

diff --git a/src/detect-base64-data.c b/src/detect-base64-data.c
index 09d89113d675..770061350a86 100644
--- a/src/detect-base64-data.c
+++ b/src/detect-base64-data.c
@@ -61,19 +61,6 @@ static int DetectBase64DataSetup(DetectEngineCtx *de_ctx, Signature *s,
     return 0;
 }
 
-int DetectBase64DataDoMatch(DetectEngineCtx *de_ctx,
-    DetectEngineThreadCtx *det_ctx, const Signature *s, Flow *f)
-{
-    if (det_ctx->base64_decoded_len) {
-        return DetectEngineContentInspectionInternal(de_ctx, det_ctx, s,
-                s->sm_arrays[DETECT_SM_LIST_BASE64_DATA], NULL, f, det_ctx->base64_decoded,
-                det_ctx->base64_decoded_len, 0, DETECT_CI_FLAGS_SINGLE,
-                DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
-    }
-
-    return 0;
-}
-
 #ifdef UNITTESTS
 
 static int g_file_data_buffer_id = 0;
diff --git a/src/detect-base64-data.h b/src/detect-base64-data.h
index 38bb93fc0691..4b7d54d04ebe 100644
--- a/src/detect-base64-data.h
+++ b/src/detect-base64-data.h
@@ -19,7 +19,5 @@
 #define __DETECT_BASE64_DATA_H__
 
 void DetectBase64DataRegister(void);
-int DetectBase64DataDoMatch(DetectEngineCtx *, DetectEngineThreadCtx *,
-        const Signature *, Flow *);
 
 #endif /* __DETECT_BASE64_DATA_H__ */
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 09d838378fea..19e7fe1096f9 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -651,10 +651,16 @@ int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineT
     } else if (smd->type == DETECT_BASE64_DECODE) {
         if (DetectBase64DecodeDoMatch(det_ctx, s, smd, buffer, buffer_len)) {
             if (s->sm_arrays[DETECT_SM_LIST_BASE64_DATA] != NULL) {
-                KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-                if (DetectBase64DataDoMatch(de_ctx, det_ctx, s, f) == 1) {
-                    /* Base64 is a terminal list. */
-                    goto final_match;
+                if (det_ctx->base64_decoded_len) {
+                    KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
+                    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s,
+                            s->sm_arrays[DETECT_SM_LIST_BASE64_DATA], NULL, f,
+                            det_ctx->base64_decoded, det_ctx->base64_decoded_len, 0,
+                            DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+                    if (r == 1) {
+                        /* Base64 is a terminal list. */
+                        goto final_match;
+                    }
                 }
             }
         }

From 97f78e1b496adb2b00c35e73292c764221f814ee Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 24 Sep 2023 06:56:57 +0200
Subject: [PATCH 245/462] detect/content-inspect: reduce scope of internal func

---
 src/detect-engine-content-inspection.c | 7 ++++---
 src/detect-engine-content-inspection.h | 6 ------
 2 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 19e7fe1096f9..3a3b786e4670 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -95,9 +95,10 @@
  *  \retval 0 no match
  *  \retval 1 match
  */
-int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
-        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
-        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
+static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
+        DetectEngineThreadCtx *det_ctx, const Signature *s, const SigMatchData *smd, Packet *p,
+        Flow *f, const uint8_t *buffer, const uint32_t buffer_len,
+        const uint32_t stream_start_offset, const uint8_t flags,
         const enum DetectContentInspectionType inspection_mode)
 {
     SCEnter();
diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index 4e362dad8d96..21ba40c96a0d 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -46,12 +46,6 @@ enum DetectContentInspectionType {
  *  inspection function contains both start and end of the data. */
 #define DETECT_CI_FLAGS_SINGLE  (DETECT_CI_FLAGS_START|DETECT_CI_FLAGS_END)
 
-/* "internal" returns 1 match, 0 no match, -1 can't match */
-int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
-        const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,
-        const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
-        const enum DetectContentInspectionType inspection_mode);
-
 /* implicit "public" just returns true match, false no match */
 bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const uint8_t *buffer,

From 4cce7ba48b5055eb48ec641ca8e980eb3d7e4b62 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 24 Sep 2023 06:51:33 +0200
Subject: [PATCH 246/462] detect/content-inspect: localize recursion counting

Use stack local var instead of DetectEngineThreadCtx member. Instead
setup a stack local struct that both counts and holds the limit. Make sure
the limit is a const so we can avoid rereading it.

This is part of an effort to reduce the size of the DetectEngineThreadCtx
structure and reduce the number of memory writes to it. Additionally, it
is part of an effect to reduce the number of places where detection
tracks various forms of state.
---
 src/detect-engine-content-inspection.c       | 51 +++++++++++++-------
 src/detect-tls-sni.c                         |  2 +-
 src/detect.h                                 |  3 --
 src/tests/detect-engine-content-inspection.c | 43 +++++++++--------
 4 files changed, 57 insertions(+), 42 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 3a3b786e4670..868a26d3ce12 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -66,6 +66,17 @@
 #include "util-lua.h"
 #endif
 
+#ifdef UNITTESTS
+thread_local uint32_t ut_inspection_recursion_counter = 0;
+#endif
+
+struct DetectEngineContentInspectionCtx {
+    struct {
+        uint32_t count;
+        const uint32_t limit;
+    } recursion;
+};
+
 /**
  * \brief Run the actual payload match functions
  *
@@ -74,7 +85,6 @@
  * For accounting the last match in relative matching the
  * det_ctx->buffer_offset int is used.
  *
- * \param de_ctx          Detection engine context
  * \param det_ctx         Detection engine thread context
  * \param s               Signature to inspect
  * \param sm              SigMatch to inspect
@@ -95,18 +105,17 @@
  *  \retval 0 no match
  *  \retval 1 match
  */
-static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
-        DetectEngineThreadCtx *det_ctx, const Signature *s, const SigMatchData *smd, Packet *p,
-        Flow *f, const uint8_t *buffer, const uint32_t buffer_len,
+static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
+        struct DetectEngineContentInspectionCtx *ctx, const Signature *s, const SigMatchData *smd,
+        Packet *p, Flow *f, const uint8_t *buffer, const uint32_t buffer_len,
         const uint32_t stream_start_offset, const uint8_t flags,
         const enum DetectContentInspectionType inspection_mode)
 {
     SCEnter();
     KEYWORD_PROFILING_START;
 
-    det_ctx->inspection_recursion_counter++;
-
-    if (unlikely(det_ctx->inspection_recursion_counter == de_ctx->inspection_recursion_limit)) {
+    ctx->recursion.count++;
+    if (unlikely(ctx->recursion.count == ctx->recursion.limit)) {
         KEYWORD_PROFILING_END(det_ctx, smd->type, 0);
         SCReturnInt(-1);
     }
@@ -349,9 +358,8 @@ static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
                         /* see if the next buffer keywords match. If not, we will
                          * search for another occurrence of this content and see
                          * if the others match then until we run out of matches */
-                        int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1,
-                                p, f, buffer, buffer_len, stream_start_offset, flags,
-                                inspection_mode);
+                        int r = DetectEngineContentInspectionInternal(det_ctx, ctx, s, smd + 1, p,
+                                f, buffer, buffer_len, stream_start_offset, flags, inspection_mode);
                         if (r == 1) {
                             SCReturnInt(1);
                         } else if (r == -1) {
@@ -448,7 +456,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
             /* see if the next payload keywords match. If not, we will
              * search for another occurrence of this pcre and see
              * if the others match, until we run out of matches */
-            r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1, p, f, buffer,
+            r = DetectEngineContentInspectionInternal(det_ctx, ctx, s, smd + 1, p, f, buffer,
                     buffer_len, stream_start_offset, flags, inspection_mode);
             if (r == 1) {
                 SCReturnInt(1);
@@ -654,7 +662,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
             if (s->sm_arrays[DETECT_SM_LIST_BASE64_DATA] != NULL) {
                 if (det_ctx->base64_decoded_len) {
                     KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-                    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s,
+                    int r = DetectEngineContentInspectionInternal(det_ctx, ctx, s,
                             s->sm_arrays[DETECT_SM_LIST_BASE64_DATA], NULL, f,
                             det_ctx->base64_decoded, det_ctx->base64_decoded_len, 0,
                             DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
@@ -692,7 +700,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineCtx *de_ctx,
      * the buffer portion of the signature matched. */
     if (!smd->is_last) {
         KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-        int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd + 1, p, f, buffer,
+        int r = DetectEngineContentInspectionInternal(det_ctx, ctx, s, smd + 1, p, f, buffer,
                 buffer_len, stream_start_offset, flags, inspection_mode);
         SCReturnInt(r);
     }
@@ -710,11 +718,15 @@ bool DetectEngineContentInspection(DetectEngineCtx *de_ctx, DetectEngineThreadCt
         const uint32_t buffer_len, const uint32_t stream_start_offset, const uint8_t flags,
         const enum DetectContentInspectionType inspection_mode)
 {
+    struct DetectEngineContentInspectionCtx ctx = { .recursion.count = 0,
+        .recursion.limit = de_ctx->inspection_recursion_limit };
     det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
 
-    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd, p, f, buffer, buffer_len,
+    int r = DetectEngineContentInspectionInternal(det_ctx, &ctx, s, smd, p, f, buffer, buffer_len,
             stream_start_offset, flags, inspection_mode);
+#ifdef UNITTESTS
+    ut_inspection_recursion_counter = ctx.recursion.count;
+#endif
     if (r == 1)
         return true;
     else
@@ -729,11 +741,16 @@ bool DetectEngineContentInspectionBuffer(DetectEngineCtx *de_ctx, DetectEngineTh
         const Signature *s, const SigMatchData *smd, Packet *p, Flow *f, const InspectionBuffer *b,
         const enum DetectContentInspectionType inspection_mode)
 {
+    struct DetectEngineContentInspectionCtx ctx = { .recursion.count = 0,
+        .recursion.limit = de_ctx->inspection_recursion_limit };
+
     det_ctx->buffer_offset = 0;
-    det_ctx->inspection_recursion_counter = 0;
 
-    int r = DetectEngineContentInspectionInternal(de_ctx, det_ctx, s, smd, p, f, b->inspect,
+    int r = DetectEngineContentInspectionInternal(det_ctx, &ctx, s, smd, p, f, b->inspect,
             b->inspect_len, b->inspect_offset, b->flags, inspection_mode);
+#ifdef UNITTESTS
+    ut_inspection_recursion_counter = ctx.recursion.count;
+#endif
     if (r == 1)
         return true;
     else
diff --git a/src/detect-tls-sni.c b/src/detect-tls-sni.c
index 69b066e8e979..6ac644f1de3a 100644
--- a/src/detect-tls-sni.c
+++ b/src/detect-tls-sni.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2016 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
diff --git a/src/detect.h b/src/detect.h
index 3861b603d801..9230f501d697 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1159,9 +1159,6 @@ typedef struct DetectEngineThreadCtx_ {
 
     SC_ATOMIC_DECLARE(int, so_far_used_by_detect);
 
-    /* holds the current recursion depth on content inspection */
-    int inspection_recursion_counter;
-
     /** array of signature pointers we're going to inspect in the detection
      *  loop. */
     Signature **match_array;
diff --git a/src/tests/detect-engine-content-inspection.c b/src/tests/detect-engine-content-inspection.c
index ee1b605f2c0d..65a4f578b824 100644
--- a/src/tests/detect-engine-content-inspection.c
+++ b/src/tests/detect-engine-content-inspection.c
@@ -29,33 +29,34 @@
 #include "../detect.h"
 #include "detect-engine-build.h"
 
+extern thread_local uint32_t ut_inspection_recursion_counter;
+
 #define TEST_HEADER                                     \
     ThreadVars tv;                                      \
     memset(&tv, 0, sizeof(tv));                         \
     Flow f;                                             \
     memset(&f, 0, sizeof(f));
 
-#define TEST_RUN(buf, buflen, sig, match, steps)                                            \
-{                                                                                           \
-    DetectEngineCtx *de_ctx = DetectEngineCtxInit();                                        \
-    FAIL_IF_NULL(de_ctx);                                                                   \
-    DetectEngineThreadCtx *det_ctx = NULL;                                                  \
-    char rule[2048];                                                                        \
-    snprintf(rule, sizeof(rule), "alert tcp any any -> any any (%s sid:1; rev:1;)", (sig)); \
-    Signature *s = DetectEngineAppendSig(de_ctx, rule);                                     \
-    FAIL_IF_NULL(s);                                                                        \
-    SigGroupBuild(de_ctx);                                                                  \
-    DetectEngineThreadCtxInit(&tv, (void *)de_ctx, (void *)&det_ctx);                       \
-    FAIL_IF_NULL(det_ctx);                                                                  \
-    int r = DetectEngineContentInspection(de_ctx, det_ctx,                                  \
-                s, s->sm_arrays[DETECT_SM_LIST_PMATCH], NULL, &f,                           \
-                (uint8_t *)(buf), (buflen), 0, DETECT_CI_FLAGS_SINGLE,                      \
-                DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);                             \
-    FAIL_IF_NOT(r == (match));                                                              \
-    FAIL_IF_NOT(det_ctx->inspection_recursion_counter == (steps));                          \
-    DetectEngineThreadCtxDeinit(&tv, det_ctx);                                              \
-    DetectEngineCtxFree(de_ctx);                                                            \
-}
+#define TEST_RUN(buf, buflen, sig, match, steps)                                                   \
+    {                                                                                              \
+        DetectEngineCtx *de_ctx = DetectEngineCtxInit();                                           \
+        FAIL_IF_NULL(de_ctx);                                                                      \
+        DetectEngineThreadCtx *det_ctx = NULL;                                                     \
+        char rule[2048];                                                                           \
+        snprintf(rule, sizeof(rule), "alert tcp any any -> any any (%s sid:1; rev:1;)", (sig));    \
+        Signature *s = DetectEngineAppendSig(de_ctx, rule);                                        \
+        FAIL_IF_NULL(s);                                                                           \
+        SigGroupBuild(de_ctx);                                                                     \
+        DetectEngineThreadCtxInit(&tv, (void *)de_ctx, (void *)&det_ctx);                          \
+        FAIL_IF_NULL(det_ctx);                                                                     \
+        int r = DetectEngineContentInspection(de_ctx, det_ctx, s,                                  \
+                s->sm_arrays[DETECT_SM_LIST_PMATCH], NULL, &f, (uint8_t *)(buf), (buflen), 0,      \
+                DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);            \
+        FAIL_IF_NOT(r == (match));                                                                 \
+        FAIL_IF_NOT(ut_inspection_recursion_counter == (steps));                                   \
+        DetectEngineThreadCtxDeinit(&tv, det_ctx);                                                 \
+        DetectEngineCtxFree(de_ctx);                                                               \
+    }
 #define TEST_FOOTER     \
     PASS
 

From c19d11ff0d7841c58b00f541bd8c25d8585ae85d Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 24 Sep 2023 07:42:37 +0200
Subject: [PATCH 247/462] detect/content-inspect: flatten branches

Flatten else branches after terminating ifs.
---
 src/detect-engine-content-inspection.c | 138 ++++++++++++-------------
 1 file changed, 68 insertions(+), 70 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 868a26d3ce12..ac8a39226ab9 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -307,83 +307,81 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
                 } else {
                     goto match;
                 }
-            } else {
-                uint32_t match_offset = (uint32_t)((found - buffer) + cd->content_len);
-                if (cd->flags & DETECT_CONTENT_NEGATED) {
-                    SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32
-                               ", but negated so no match",
-                            cd->id, match_offset);
-                    /* don't bother carrying recursive matches now, for preceding
-                     * relative keywords */
-
-                    /* found a match but not at the end of the buffer */
-                    if (cd->flags & DETECT_CONTENT_ENDS_WITH) {
-                        if (sbuffer_len != match_offset) {
-                            SCLogDebug("content \"%s\" %" PRIu32 " matched at offset %" PRIu32
-                                       ", but not at end of buffer so match",
-                                    cd->content, cd->id, match_offset);
-                            goto match;
-                        }
-                    }
-                    if (DETECT_CONTENT_IS_SINGLE(cd)) {
-                        goto no_match_discontinue;
+            }
+
+            uint32_t match_offset = (uint32_t)((found - buffer) + cd->content_len);
+            if (cd->flags & DETECT_CONTENT_NEGATED) {
+                SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32
+                           ", but negated so no match",
+                        cd->id, match_offset);
+                /* don't bother carrying recursive matches now, for preceding
+                 * relative keywords */
+
+                /* found a match but not at the end of the buffer */
+                if (cd->flags & DETECT_CONTENT_ENDS_WITH) {
+                    if (sbuffer_len != match_offset) {
+                        SCLogDebug("content \"%s\" %" PRIu32 " matched at offset %" PRIu32
+                                   ", but not at end of buffer so match",
+                                cd->content, cd->id, match_offset);
+                        goto match;
                     }
-                    goto no_match;
-                } else {
-                    SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32 "", cd->id,
-                            match_offset);
-                    det_ctx->buffer_offset = match_offset;
-
-                    if ((cd->flags & DETECT_CONTENT_ENDS_WITH) == 0 || match_offset == buffer_len) {
-                        /* Match branch, add replace to the list if needed */
-                        if (cd->flags & DETECT_CONTENT_REPLACE) {
-                            if (inspection_mode == DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD) {
-                                /* we will need to replace content if match is confirmed
-                                 * cast to non-const as replace writes to it. */
-                                det_ctx->replist = DetectReplaceAddToList(
-                                        det_ctx->replist, (uint8_t *)found, cd);
-                            } else {
-                                SCLogWarning("Can't modify payload without packet");
-                            }
-                        }
+                }
+                if (DETECT_CONTENT_IS_SINGLE(cd)) {
+                    goto no_match_discontinue;
+                }
+                goto no_match;
+            }
 
-                        /* if this is the last match we're done */
-                        if (smd->is_last) {
-                            goto match;
-                        }
+            SCLogDebug("content %" PRIu32 " matched at offset %" PRIu32 "", cd->id, match_offset);
+            det_ctx->buffer_offset = match_offset;
+
+            if ((cd->flags & DETECT_CONTENT_ENDS_WITH) == 0 || match_offset == buffer_len) {
+                /* Match branch, add replace to the list if needed */
+                if (cd->flags & DETECT_CONTENT_REPLACE) {
+                    if (inspection_mode == DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD) {
+                        /* we will need to replace content if match is confirmed
+                         * cast to non-const as replace writes to it. */
+                        det_ctx->replist =
+                                DetectReplaceAddToList(det_ctx->replist, (uint8_t *)found, cd);
+                    } else {
+                        SCLogWarning("Can't modify payload without packet");
+                    }
+                }
 
-                        SCLogDebug("content %" PRIu32, cd->id);
-                        KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
-
-                        /* see if the next buffer keywords match. If not, we will
-                         * search for another occurrence of this content and see
-                         * if the others match then until we run out of matches */
-                        int r = DetectEngineContentInspectionInternal(det_ctx, ctx, s, smd + 1, p,
-                                f, buffer, buffer_len, stream_start_offset, flags, inspection_mode);
-                        if (r == 1) {
-                            SCReturnInt(1);
-                        } else if (r == -1) {
-                            SCLogDebug("'next sm' said to discontinue this right now");
-                            SCReturnInt(-1);
-                        }
-                        SCLogDebug("no match for 'next sm'");
+                /* if this is the last match we're done */
+                if (smd->is_last) {
+                    goto match;
+                }
 
-                        /* no match and no reason to look for another instance */
-                        if ((cd->flags & DETECT_CONTENT_WITHIN_NEXT) == 0) {
-                            SCLogDebug("'next sm' does not depend on me, so we can give up");
-                            SCReturnInt(-1);
-                        }
+                SCLogDebug("content %" PRIu32, cd->id);
+                KEYWORD_PROFILING_END(det_ctx, smd->type, 1);
+
+                /* see if the next buffer keywords match. If not, we will
+                 * search for another occurrence of this content and see
+                 * if the others match then until we run out of matches */
+                int r = DetectEngineContentInspectionInternal(det_ctx, ctx, s, smd + 1, p, f,
+                        buffer, buffer_len, stream_start_offset, flags, inspection_mode);
+                if (r == 1) {
+                    SCReturnInt(1);
+                } else if (r == -1) {
+                    SCLogDebug("'next sm' said to discontinue this right now");
+                    SCReturnInt(-1);
+                }
+                SCLogDebug("no match for 'next sm'");
 
-                        SCLogDebug("'next sm' depends on me %p, lets see what we can do (flags %u)",
-                                cd, cd->flags);
-                    }
-                    /* set the previous match offset to the start of this match + 1 */
-                    prev_offset = (match_offset - (cd->content_len - 1));
-                    SCLogDebug("trying to see if there is another match after prev_offset %" PRIu32,
-                            prev_offset);
+                /* no match and no reason to look for another instance */
+                if ((cd->flags & DETECT_CONTENT_WITHIN_NEXT) == 0) {
+                    SCLogDebug("'next sm' does not depend on me, so we can give up");
+                    SCReturnInt(-1);
                 }
-            }
 
+                SCLogDebug("'next sm' depends on me %p, lets see what we can do (flags %u)", cd,
+                        cd->flags);
+            }
+            /* set the previous match offset to the start of this match + 1 */
+            prev_offset = (match_offset - (cd->content_len - 1));
+            SCLogDebug("trying to see if there is another match after prev_offset %" PRIu32,
+                    prev_offset);
         } while(1);
 
     } else if (smd->type == DETECT_ISDATAAT) {

From 6db02563f1bf2e0e98852189f021e9db01f2305f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 10:16:27 +0200
Subject: [PATCH 248/462] detect/isdataat: optimize recursion mismatches

Since recursive content matching goes through the buffer from left to
right, it is possible to bail early when isdataat is part of the
recursive checking. If `isdataat:50,relative` fails for offset 10, it
will surely also fail for offset 20. So break inspection in such cases.

The exception is for dynamic isdataat, where the value is determined
by a prior byte_extract that may be updated during the recursion.
---
 src/detect-engine-content-inspection.c       | 15 +++++++++++++--
 src/tests/detect-engine-content-inspection.c |  9 +++++++++
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index ac8a39226ab9..76baead03513 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -408,23 +408,34 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
                 SCLogDebug("det_ctx->buffer_offset + dataat %"PRIu32" > %"PRIu32, det_ctx->buffer_offset + dataat, buffer_len);
                 if (id->flags & ISDATAAT_NEGATED)
                     goto match;
+                if ((id->flags & ISDATAAT_OFFSET_VAR) == 0) {
+                    goto no_match_discontinue;
+                }
                 goto no_match;
             } else {
                 SCLogDebug("relative isdataat match");
-                if (id->flags & ISDATAAT_NEGATED)
+                if (id->flags & ISDATAAT_NEGATED) {
                     goto no_match;
+                }
                 goto match;
             }
         } else {
             if (dataat < buffer_len) {
                 SCLogDebug("absolute isdataat match");
-                if (id->flags & ISDATAAT_NEGATED)
+                if (id->flags & ISDATAAT_NEGATED) {
+                    if ((id->flags & ISDATAAT_OFFSET_VAR) == 0) {
+                        goto no_match_discontinue;
+                    }
                     goto no_match;
+                }
                 goto match;
             } else {
                 SCLogDebug("absolute isdataat mismatch, id->isdataat %"PRIu32", buffer_len %"PRIu32"", dataat, buffer_len);
                 if (id->flags & ISDATAAT_NEGATED)
                     goto match;
+                if ((id->flags & ISDATAAT_OFFSET_VAR) == 0) {
+                    goto no_match_discontinue;
+                }
                 goto no_match;
             }
         }
diff --git a/src/tests/detect-engine-content-inspection.c b/src/tests/detect-engine-content-inspection.c
index 65a4f578b824..c4e1a3bdff68 100644
--- a/src/tests/detect-engine-content-inspection.c
+++ b/src/tests/detect-engine-content-inspection.c
@@ -143,6 +143,10 @@ static int DetectEngineContentInspectionTest06(void) {
     // 6 steps: (1) a, (2) 1st b, (3) c not found, (4) 2nd b, (5) c found, isdataat
     TEST_RUN("ababc", 5, "content:\"a\"; content:\"b\"; distance:0; within:1; content:\"c\"; distance:0; within:1; isdataat:!1,relative;", true, 5);
     TEST_RUN("ababc", 5, "content:\"a\"; content:\"b\"; distance:0; within:1; content:\"c\"; distance:0; within:1; isdataat:1,relative;", false, 6);
+    TEST_RUN("abcabc", 6,
+            "content:\"a\"; content:\"b\"; distance:0; within:1; content:\"c\"; distance:0; "
+            "within:1; isdataat:10,relative;",
+            false, 4);
 
     TEST_RUN("ababcabc", 8, "content:\"a\"; content:\"b\"; distance:0; within:1; content:\"c\"; distance:0; within:1; isdataat:!1,relative;", true, 7);
     TEST_RUN("ababcabc", 8, "content:\"a\"; content:\"b\"; distance:0; within:1; content:\"c\"; distance:0; within:1; isdataat:1,relative;", true, 6);
@@ -228,6 +232,11 @@ static int DetectEngineContentInspectionTest10(void) {
     TEST_RUN("x9x9abcdefghi", 13, "content:\"x\"; byte_extract:1,0,data_size,string,relative; isdataat:data_size,relative;", true, 3);
     TEST_RUN("x9x9abcdefgh", 12, "content:\"x\"; byte_extract:1,0,data_size,string,relative; isdataat:!data_size,relative;", true, 5);
     TEST_RUN("x9x9abcdefgh", 12, "content:\"x\"; depth:1; byte_extract:1,0,data_size,string,relative; isdataat:!data_size,relative;", false, 3);
+    /* first isdataat should fail, second succeed */
+    TEST_RUN("x9x5abcdef", 10,
+            "content:\"x\"; byte_extract:1,0,data_size,string,relative; "
+            "isdataat:data_size,relative;",
+            true, 5);
     /* check for super high extracted values */
     TEST_RUN("100000000abcdefghi", 18, "byte_extract:0,0,data_size,string; isdataat:data_size;", false, 2);
     TEST_RUN("100000000abcdefghi", 18, "byte_extract:0,0,data_size,string; isdataat:!data_size;", true, 2);

From e2fbcf9654c3687d22ec4e6f2bb9dd9c0a0b3f90 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 25 Sep 2023 10:53:22 +0200
Subject: [PATCH 249/462] detect/payload: remove unneeded pointer reset

DetectEngineThreadCtx::replist is managed elsewhere.
---
 src/detect-engine-payload.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/src/detect-engine-payload.c b/src/detect-engine-payload.c
index d051303ddbb1..7fc4c0e161d4 100644
--- a/src/detect-engine-payload.c
+++ b/src/detect-engine-payload.c
@@ -161,8 +161,6 @@ uint8_t DetectEngineInspectPacketPayload(DetectEngineCtx *de_ctx, DetectEngineTh
     det_ctx->payload_persig_cnt++;
     det_ctx->payload_persig_size += p->payload_len;
 #endif
-    det_ctx->replist = NULL;
-
     const bool match = DetectEngineContentInspection(de_ctx, det_ctx, s,
             s->sm_arrays[DETECT_SM_LIST_PMATCH], p, f, p->payload, p->payload_len, 0,
             DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);
@@ -198,8 +196,6 @@ static uint8_t DetectEngineInspectStreamUDPPayload(DetectEngineCtx *de_ctx,
     det_ctx->payload_persig_cnt++;
     det_ctx->payload_persig_size += p->payload_len;
 #endif
-    det_ctx->replist = NULL;
-
     const bool match =
             DetectEngineContentInspection(de_ctx, det_ctx, s, smd, p, f, p->payload, p->payload_len,
                     0, DETECT_CI_FLAGS_SINGLE, DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD);

From 53591702aadd0e38fc582cd05571ac2fc313568e Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 26 Sep 2023 09:34:09 +0200
Subject: [PATCH 250/462] detect/bytemath: pass match ctx directly

Adjust includes to enable this.
---
 src/detect-byte.c                      |  3 ++-
 src/detect-bytemath.c                  | 13 +++++++------
 src/detect-bytemath.h                  |  2 +-
 src/detect-engine-content-inspection.c | 13 ++++++-------
 src/detect-engine-register.c           |  2 ++
 5 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/src/detect-byte.c b/src/detect-byte.c
index 19fdc472229e..dd29734a3f3f 100644
--- a/src/detect-byte.c
+++ b/src/detect-byte.c
@@ -22,10 +22,11 @@
  */
 
 #include "suricata-common.h"
+#include "rust.h"
 #include "detect-byte.h"
 #include "detect-byte-extract.h"
 #include "detect-bytemath.h"
-#include "rust.h"
+
 /**
  * \brief Used to retrieve args from BM.
  *
diff --git a/src/detect-bytemath.c b/src/detect-bytemath.c
index a2880216cffa..3e848f21a5bb 100644
--- a/src/detect-bytemath.c
+++ b/src/detect-bytemath.c
@@ -29,21 +29,23 @@
 #include "threads.h"
 #include "decode.h"
 
+#include "app-layer-parser.h"
+#include "app-layer-protos.h"
+
 #include "detect.h"
 #include "detect-parse.h"
 #include "detect-engine.h"
 #include "detect-engine-mpm.h"
 #include "detect-engine-state.h"
 #include "detect-engine-build.h"
+
+#include "rust-bindings.h"
+
 #include "detect-content.h"
 #include "detect-pcre.h"
 #include "detect-byte.h"
 #include "detect-bytemath.h"
 
-#include "app-layer-parser.h"
-#include "app-layer-protos.h"
-#include "rust-bindings.h"
-
 #include "flow.h"
 #include "flow-var.h"
 #include "flow-util.h"
@@ -82,11 +84,10 @@ static inline bool DetectByteMathValidateNbytesOnly(const DetectByteMathData *da
            (((data->flags & DETECT_BYTEMATH_FLAG_STRING) && nbytes <= 10) || (nbytes <= 4));
 }
 
-int DetectByteMathDoMatch(DetectEngineThreadCtx *det_ctx, const SigMatchData *smd,
+int DetectByteMathDoMatch(DetectEngineThreadCtx *det_ctx, const DetectByteMathData *data,
         const Signature *s, const uint8_t *payload, uint16_t payload_len, uint8_t nbytes,
         uint64_t rvalue, uint64_t *value, uint8_t endian)
 {
-    const DetectByteMathData *data = (DetectByteMathData *)smd->ctx;
     if (payload_len == 0) {
         return 0;
     }
diff --git a/src/detect-bytemath.h b/src/detect-bytemath.h
index 672f799ca4f5..4fbc9ae5ce15 100644
--- a/src/detect-bytemath.h
+++ b/src/detect-bytemath.h
@@ -27,7 +27,7 @@
 void DetectBytemathRegister(void);
 
 SigMatch *DetectByteMathRetrieveSMVar(const char *, const Signature *);
-int DetectByteMathDoMatch(DetectEngineThreadCtx *, const SigMatchData *, const Signature *,
+int DetectByteMathDoMatch(DetectEngineThreadCtx *, const DetectByteMathData *, const Signature *,
         const uint8_t *, uint16_t, uint8_t, uint64_t, uint64_t *, uint8_t);
 
 #endif /* __DETECT_BYTEMATH_H__ */
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 76baead03513..5d6ad2be5629 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -25,12 +25,14 @@
 
 #include "suricata-common.h"
 #include "suricata.h"
-
 #include "decode.h"
 
 #include "detect.h"
 #include "detect-engine.h"
 #include "detect-parse.h"
+
+#include "rust.h"
+
 #include "detect-asn1.h"
 #include "detect-content.h"
 #include "detect-pcre.h"
@@ -60,8 +62,6 @@
 #include "util-unittest-helper.h"
 #include "util-profiling.h"
 
-#include "rust.h"
-
 #ifdef HAVE_LUA
 #include "util-lua.h"
 #endif
@@ -569,17 +569,16 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
 
     } else if (smd->type == DETECT_BYTEMATH) {
 
-        DetectByteMathData *bmd = (DetectByteMathData *)smd->ctx;
+        const DetectByteMathData *bmd = (const DetectByteMathData *)smd->ctx;
         uint8_t endian = bmd->endian;
 
         /* if we have dce enabled we will have to use the endianness
          * specified by the dce header */
         if ((bmd->flags & DETECT_BYTEMATH_FLAG_ENDIAN) && endian == (int)EndianDCE &&
                 flags & (DETECT_CI_FLAGS_DCE_LE | DETECT_CI_FLAGS_DCE_BE)) {
-
             /* enable the endianness flag temporarily.  once we are done
              * processing we reset the flags to the original value*/
-            endian |= (uint8_t)((flags & DETECT_CI_FLAGS_DCE_LE) ? LittleEndian : BigEndian);
+            endian = (uint8_t)((flags & DETECT_CI_FLAGS_DCE_LE) ? LittleEndian : BigEndian);
         }
         uint64_t rvalue;
         if (bmd->flags & DETECT_BYTEMATH_FLAG_RVALUE_VAR) {
@@ -596,7 +595,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
         }
 
         DEBUG_VALIDATE_BUG_ON(buffer_len > UINT16_MAX);
-        if (DetectByteMathDoMatch(det_ctx, smd, s, buffer, (uint16_t)buffer_len, nbytes, rvalue,
+        if (DetectByteMathDoMatch(det_ctx, bmd, s, buffer, (uint16_t)buffer_len, nbytes, rvalue,
                     &det_ctx->byte_values[bmd->local_id], endian) != 1) {
             goto no_match;
         }
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index 0f459eccb67b..b27f5a511f9c 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -43,6 +43,8 @@
 #include "detect-engine-threshold.h"
 #include "detect-engine-prefilter.h"
 
+#include "rust.h"
+
 #include "detect-engine-payload.h"
 #include "detect-engine-dcepayload.h"
 #include "detect-dns-opcode.h"

From 0014077a369328f111a6600b412b753866e4b1e4 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 26 Sep 2023 10:10:14 +0200
Subject: [PATCH 251/462] detect: optimize struct layout

Move reference count to top of DetectEngineThreadCtx, to move it to the
same cache line as the other members that are checked first in Detect().
---
 src/detect.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect.h b/src/detect.h
index 9230f501d697..2b634e761701 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1082,6 +1082,8 @@ typedef struct DetectEngineThreadCtx_ {
      *        on this being the first member */
     uint32_t tenant_id;
 
+    SC_ATOMIC_DECLARE(int, so_far_used_by_detect);
+
     /* the thread to which this detection engine thread belongs */
     ThreadVars *tv;
 
@@ -1157,8 +1159,6 @@ typedef struct DetectEngineThreadCtx_ {
     uint16_t alert_queue_capacity;
     PacketAlert *alert_queue;
 
-    SC_ATOMIC_DECLARE(int, so_far_used_by_detect);
-
     /** array of signature pointers we're going to inspect in the detection
      *  loop. */
     Signature **match_array;

From 06c809573bdae364bc18d5f411037376ce391970 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 26 Sep 2023 10:10:52 +0200
Subject: [PATCH 252/462] detect/content-inspect: optimize struct layout

Move members used by DetectEngineContentInspection() to the same cache line.
---
 src/detect.h | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/src/detect.h b/src/detect.h
index 2b634e761701..36e4e2ba3b25 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1112,6 +1112,17 @@ typedef struct DetectEngineThreadCtx_ {
      *  points to 1 byte after the start of the last pcre match if a pcre match happened. */
     uint32_t pcre_match_start_offset;
 
+    /** SPM thread context used for scanning. This has been cloned from the
+     * prototype held by DetectEngineCtx. */
+    SpmThreadCtx *spm_thread_ctx;
+
+    /* byte_* values */
+    uint64_t *byte_values;
+
+    uint8_t *base64_decoded;
+    int base64_decoded_len;
+    int base64_decoded_len_max;
+
     /* counter for the filestore array below -- up here for cache reasons. */
     uint16_t filestore_cnt;
 
@@ -1177,13 +1188,6 @@ typedef struct DetectEngineThreadCtx_ {
     MpmThreadCtx mtc; /**< thread ctx for the mpm */
     PrefilterRuleStore pmq;
 
-    /** SPM thread context used for scanning. This has been cloned from the
-     * prototype held by DetectEngineCtx. */
-    SpmThreadCtx *spm_thread_ctx;
-
-    /* byte_* values */
-    uint64_t *byte_values;
-
     /* string to replace */
     DetectReplaceList *replist;
     /* vars to store in post match function */
@@ -1205,10 +1209,6 @@ typedef struct DetectEngineThreadCtx_ {
     int global_keyword_ctxs_size;
     void **global_keyword_ctxs_array;
 
-    uint8_t *base64_decoded;
-    int base64_decoded_len;
-    int base64_decoded_len_max;
-
     AppLayerDecoderEvents *decoder_events;
     uint16_t events;
 

From 332c2ea470b9497e0c68b1e8732bce0c6e54cc76 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 6 Dec 2023 17:32:15 +0100
Subject: [PATCH 253/462] detect/content-inspect: improve header docs

---
 src/detect-engine-content-inspection.h | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/src/detect-engine-content-inspection.h b/src/detect-engine-content-inspection.h
index 21ba40c96a0d..ec785165df1a 100644
--- a/src/detect-engine-content-inspection.h
+++ b/src/detect-engine-content-inspection.h
@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2021 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -30,13 +30,15 @@
  */
 enum DetectContentInspectionType {
     DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD = 0, /* enables 'replace' logic */
-    DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER,
-    DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM,
-    DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME,
-    DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE,
+    DETECT_ENGINE_CONTENT_INSPECTION_MODE_HEADER,      /* indicates a header is being inspected */
+    DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM,      /* enables "stream" inspection logic */
+    DETECT_ENGINE_CONTENT_INSPECTION_MODE_FRAME,       /* enables "frame" inspection logic */
+    DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE, /* enables "state" - used for buffers coming from
+                                                    the app-layer state. */
 };
 
-#define DETECT_CI_FLAGS_START   BIT_U8(0)   /**< unused, reserved for future use */
+#define DETECT_CI_FLAGS_START                                                                      \
+    BIT_U8(0) /**< indication that current buffer is the start of the data */
 #define DETECT_CI_FLAGS_END     BIT_U8(1)   /**< indication that current buffer
                                              *   is the end of the data */
 #define DETECT_CI_FLAGS_DCE_LE  BIT_U8(2)   /**< DCERPC record in little endian */

From c82d93490c4b41f3047f78aa34c4e135ddbf2c79 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 7 Dec 2023 10:07:24 +0100
Subject: [PATCH 254/462] github/action: fix Debian 12 intermittent failures

Parallel builds caused issues during `cargo vendor`. So do just a single
thread build.

 make[4]: Entering directory '/__w/suricata/suricata/rust'
cbindgen --config /__w/suricata/suricata/rust/cbindgen.toml \
	--quiet --output /__w/suricata/suricata/rust/dist/rust-bindings.h
CARGO_HOME="/github/home/.cargo"  /usr/bin/cargo vendor
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
ERROR: Couldn't execute `cargo metadata` with manifest "/__w/suricata/suricata/rust/Cargo.toml": Metadata(Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: "    Blocking waiting for file lock on package cache\n    Blocking waiting for file lock on package cache\nerror: failed to download `adler v1.0.2`\n\nCaused by:\n  unable to get packages from source\n\nCaused by:\n  failed to parse manifest at `/github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/adler-1.0.2/Cargo.toml`\n\nCaused by:\n  no targets specified in the manifest\n  either src/lib.rs, src/main.rs, a [lib] section, or [[bin]] section must be present\n" })
ERROR: Couldn't generate bindings for /__w/suricata/suricata/rust.
make[4]: *** [Makefile:597: dist/rust-bindings.h] Error 1
make[4]: *** Waiting for unfinished jobs....
---
 .github/workflows/builds.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index aa6d7771840d..eb02c0c21775 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -2130,7 +2130,8 @@ jobs:
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests
       - run: make -j2
       - run: make check
-      - run: make -j2 distcheck
+        # -j2 caused random failures during cargo vendor
+      - run: make distcheck
         env:
           DISTCHECK_CONFIGURE_FLAGS: "--enable-unittests --enable-debug --enable-lua --enable-geoip --enable-profiling --enable-profiling-locks --enable-dpdk"
       - run: test -e doc/userguide/suricata.1

From b9540df5ad4752455e1af76c7a95f5f036c4ef3d Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 7 Dec 2023 17:18:24 +0530
Subject: [PATCH 255/462] doc: clarify IP-only with iprep

---
 doc/userguide/rules/ip-reputation-rules.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/userguide/rules/ip-reputation-rules.rst b/doc/userguide/rules/ip-reputation-rules.rst
index f0b5f18d4f86..beeaa57860fa 100644
--- a/doc/userguide/rules/ip-reputation-rules.rst
+++ b/doc/userguide/rules/ip-reputation-rules.rst
@@ -32,10 +32,10 @@ Example:
 
 This rule will alert when a system in $HOME_NET acts as a client while communicating with any IP in the CnC category that has a reputation score set to greater than 30.
 
-IP-only
-~~~~~~~
+Compatibility with IP-only
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The "iprep" keyword is compatible to "IP-only" rules. This means that a rule like:
+The "iprep" keyword is compatible with "IP-only" rules. This means that a rule like:
 
 ::
 

From bd41b3100504d9608e1bd5037103b0e548c803af Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 7 Dec 2023 12:53:51 +0530
Subject: [PATCH 256/462] detect: rename SigAddressPrepare fns to SigPrepare

There is nothing Address specific going on in the preparations.
Stage 1: Preprocessing happens. Sigs classified as IP Only, Masks
applied, content specific limits applied, etc and sig array built.
Stage 2: Sigs grouped by IPOnly, ports and protocols.
Stage 3: Decoder Events SGH built.
Stage 4: File flags set, sig grouping done per prefilter, etc.
---
 src/detect-content.c         |  2 +-
 src/detect-engine-build.c    | 16 ++++++++--------
 src/detect-engine-build.h    |  8 ++++----
 src/detect-engine-siggroup.c | 10 +++++-----
 4 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/src/detect-content.c b/src/detect-content.c
index c7a240d7e01d..ebe28a1b04fa 100644
--- a/src/detect-content.c
+++ b/src/detect-content.c
@@ -811,7 +811,7 @@ static bool TestLastContent(const Signature *s, uint16_t o, uint16_t d)
         snprintf(rule, sizeof(rule), "alert tcp any any -> any any (%s sid:1; rev:1;)", (sig));    \
         Signature *s = DetectEngineAppendSig(de_ctx, rule);                                        \
         FAIL_IF_NULL(s);                                                                           \
-        SigAddressPrepareStage1(de_ctx);                                                           \
+        SigPrepareStage1(de_ctx);                                                                  \
         bool res = TestLastContent(s, (o), (d));                                                   \
         FAIL_IF(res == false);                                                                     \
         DetectEngineCtxFree(de_ctx);                                                               \
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 33b8ca206b22..be1eb2c53f09 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1377,7 +1377,7 @@ void SignatureSetType(DetectEngineCtx *de_ctx, Signature *s)
  * \retval  0 on success
  * \retval -1 on failure
  */
-int SigAddressPrepareStage1(DetectEngineCtx *de_ctx)
+int SigPrepareStage1(DetectEngineCtx *de_ctx)
 {
     uint32_t cnt_iponly = 0;
     uint32_t cnt_payload = 0;
@@ -1720,7 +1720,7 @@ static void DetectEngineAddDecoderEventSig(DetectEngineCtx *de_ctx, Signature *s
  * \retval  0 On success
  * \retval -1 On failure
  */
-int SigAddressPrepareStage2(DetectEngineCtx *de_ctx)
+int SigPrepareStage2(DetectEngineCtx *de_ctx)
 {
     SCLogDebug("building signature grouping structure, stage 2: "
             "building source address lists...");
@@ -1760,7 +1760,7 @@ static void DetectEngineBuildDecoderEventSgh(DetectEngineCtx *de_ctx)
     SigGroupHeadBuildMatchArray(de_ctx, de_ctx->decoder_event_sgh, max_idx);
 }
 
-int SigAddressPrepareStage3(DetectEngineCtx *de_ctx)
+int SigPrepareStage3(DetectEngineCtx *de_ctx)
 {
     /* prepare the decoder event sgh */
     DetectEngineBuildDecoderEventSgh(de_ctx);
@@ -1841,7 +1841,7 @@ static void DbgPrintSigs2(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
 #endif
 
 /** \brief finalize preparing sgh's */
-int SigAddressPrepareStage4(DetectEngineCtx *de_ctx)
+int SigPrepareStage4(DetectEngineCtx *de_ctx)
 {
     SCEnter();
 
@@ -2002,18 +2002,18 @@ int SigGroupBuild(DetectEngineCtx *de_ctx)
 
     SigInitStandardMpmFactoryContexts(de_ctx);
 
-    if (SigAddressPrepareStage1(de_ctx) != 0) {
+    if (SigPrepareStage1(de_ctx) != 0) {
         FatalError("initializing the detection engine failed");
     }
 
-    if (SigAddressPrepareStage2(de_ctx) != 0) {
+    if (SigPrepareStage2(de_ctx) != 0) {
         FatalError("initializing the detection engine failed");
     }
 
-    if (SigAddressPrepareStage3(de_ctx) != 0) {
+    if (SigPrepareStage3(de_ctx) != 0) {
         FatalError("initializing the detection engine failed");
     }
-    if (SigAddressPrepareStage4(de_ctx) != 0) {
+    if (SigPrepareStage4(de_ctx) != 0) {
         FatalError("initializing the detection engine failed");
     }
 
diff --git a/src/detect-engine-build.h b/src/detect-engine-build.h
index 2c9c48792856..513845477ec6 100644
--- a/src/detect-engine-build.h
+++ b/src/detect-engine-build.h
@@ -29,10 +29,10 @@ int SignatureIsFileSha256Inspecting(const Signature *s);
 int SignatureIsFilesizeInspecting(const Signature *);
 void SignatureSetType(DetectEngineCtx *de_ctx, Signature *s);
 
-int SigAddressPrepareStage1(DetectEngineCtx *de_ctx);
-int SigAddressPrepareStage2(DetectEngineCtx *de_ctx);
-int SigAddressPrepareStage3(DetectEngineCtx *de_ctx);
-int SigAddressPrepareStage4(DetectEngineCtx *de_ctx);
+int SigPrepareStage1(DetectEngineCtx *de_ctx);
+int SigPrepareStage2(DetectEngineCtx *de_ctx);
+int SigPrepareStage3(DetectEngineCtx *de_ctx);
+int SigPrepareStage4(DetectEngineCtx *de_ctx);
 int SigAddressCleanupStage1(DetectEngineCtx *de_ctx);
 
 void SigCleanSignatures(DetectEngineCtx *);
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index b063fda8a614..dfb3c10895e8 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -766,7 +766,7 @@ int SigGroupHeadContainsSigId(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
 
 #ifdef UNITTESTS
 
-int SigAddressPrepareStage1(DetectEngineCtx *);
+int SigPrepareStage1(DetectEngineCtx *);
 
 /**
  * \test Check if a SigGroupHead hash table is properly allocated and
@@ -823,7 +823,7 @@ static int SigGroupHeadTest02(void)
                                       "content:\"test2\"; content:\"test3\"; sid:5;)");
     FAIL_IF_NULL(s);
 
-    SigAddressPrepareStage1(de_ctx);
+    SigPrepareStage1(de_ctx);
 
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list);
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list->next->next);
@@ -883,7 +883,7 @@ static int SigGroupHeadTest03(void)
                                       "content:\"test2\"; content:\"test3\"; sid:5;)");
     FAIL_IF_NULL(s);
 
-    SigAddressPrepareStage1(de_ctx);
+    SigPrepareStage1(de_ctx);
 
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list);
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list->next->next);
@@ -951,7 +951,7 @@ static int SigGroupHeadTest04(void)
                                       "content:\"test2\"; content:\"test3\"; sid:5;)");
     FAIL_IF_NULL(s);
 
-    SigAddressPrepareStage1(de_ctx);
+    SigPrepareStage1(de_ctx);
 
     SigGroupHeadAppendSig(de_ctx, &src_sh, de_ctx->sig_list);
     SigGroupHeadAppendSig(de_ctx, &src_sh, de_ctx->sig_list->next->next);
@@ -1021,7 +1021,7 @@ static int SigGroupHeadTest05(void)
                                       "content:\"test2\"; content:\"test3\"; sid:5;)");
     FAIL_IF_NULL(s);
 
-    SigAddressPrepareStage1(de_ctx);
+    SigPrepareStage1(de_ctx);
 
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list);
     SigGroupHeadAppendSig(de_ctx, &sh, de_ctx->sig_list->next->next);

From 47c9a14543dff9c57a2797752691617f7ddaadb2 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 7 Dec 2023 14:22:41 +0530
Subject: [PATCH 257/462] detect-engine: use bool return type

---
 src/detect-engine-build.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index be1eb2c53f09..71c2983fc421 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -598,11 +598,11 @@ static void SigInitStandardMpmFactoryContexts(DetectEngineCtx *de_ctx)
 }
 
 /** \brief Pure-PCRE or bytetest rule */
-static int RuleInspectsPayloadHasNoMpm(const Signature *s)
+static bool RuleInspectsPayloadHasNoMpm(const Signature *s)
 {
     if (s->init_data->mpm_sm == NULL && s->init_data->smlists[DETECT_SM_LIST_PMATCH] != NULL)
-        return 1;
-    return 0;
+        return true;
+    return false;
 }
 
 static int RuleGetMpmPatternSize(const Signature *s)
@@ -618,17 +618,17 @@ static int RuleGetMpmPatternSize(const Signature *s)
     return (int)cd->content_len;
 }
 
-static int RuleMpmIsNegated(const Signature *s)
+static bool RuleMpmIsNegated(const Signature *s)
 {
     if (s->init_data->mpm_sm == NULL)
-        return 0;
+        return false;
     int mpm_list = s->init_data->mpm_sm_list;
     if (mpm_list < 0)
-        return 0;
+        return false;
     const DetectContentData *cd = (const DetectContentData *)s->init_data->mpm_sm->ctx;
     if (cd == NULL)
-        return 0;
-    return (cd->flags & DETECT_CONTENT_NEGATED);
+        return false;
+    return (cd->flags & DETECT_CONTENT_NEGATED) ? true : false;
 }
 
 static json_t *RulesGroupPrintSghStats(const DetectEngineCtx *de_ctx, const SigGroupHead *sgh,

From 34858808c112c064cd9799894f46bcba9389291d Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 7 Dec 2023 14:27:01 +0530
Subject: [PATCH 258/462] detect-engine: use flag SIG_FLAG_MPM_NEG

The flag SIG_FLAG_MPM_NEG is set before whitelisting the rules. Make it
better by checking for the flag in the beginning and return immediately.
---
 src/detect-engine-build.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 71c2983fc421..70611df46f5c 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -620,6 +620,8 @@ static int RuleGetMpmPatternSize(const Signature *s)
 
 static bool RuleMpmIsNegated(const Signature *s)
 {
+    if (s->flags & SIG_FLAG_MPM_NEG)
+        return true;
     if (s->init_data->mpm_sm == NULL)
         return false;
     int mpm_list = s->init_data->mpm_sm_list;

From 75471dd69b78d0915819b978e937483dca8b4a04 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Sat, 9 Dec 2023 12:49:31 +0530
Subject: [PATCH 259/462] detect/flowbits: remove DETECT_FLOWBITS_CMD_NOALERT

DETECT_FLOWBITS_CMD_NOALERT is misleading as it gives an impression that
noalert is a flowbit specific command that'll be used and dealt with at
some point but as soon as noalert is found in the rule lang, signature
flag for noalert is set and control is returned. It never gets added to
cmd of the flowbits object.
---
 src/detect-flowbits.c | 13 +++++--------
 src/detect-flowbits.h |  3 +--
 2 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/src/detect-flowbits.c b/src/detect-flowbits.c
index b04c271dc548..dce56625ec16 100644
--- a/src/detect-flowbits.c
+++ b/src/detect-flowbits.c
@@ -285,7 +285,10 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
     }
 
     if (strcmp(fb_cmd_str,"noalert") == 0) {
-        fb_cmd = DETECT_FLOWBITS_CMD_NOALERT;
+        if (strlen(fb_name) != 0)
+            goto error;
+        s->flags |= SIG_FLAG_NOALERT;
+        return 0;
     } else if (strcmp(fb_cmd_str,"isset") == 0) {
         fb_cmd = DETECT_FLOWBITS_CMD_ISSET;
     } else if (strcmp(fb_cmd_str,"isnotset") == 0) {
@@ -302,11 +305,6 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
     }
 
     switch (fb_cmd) {
-        case DETECT_FLOWBITS_CMD_NOALERT:
-            if (strlen(fb_name) != 0)
-                goto error;
-            s->flags |= SIG_FLAG_NOALERT;
-            return 0;
         case DETECT_FLOWBITS_CMD_ISNOTSET:
         case DETECT_FLOWBITS_CMD_ISSET:
         case DETECT_FLOWBITS_CMD_SET:
@@ -340,8 +338,7 @@ int DetectFlowbitSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
      * and put it in the Signature. */
 
     switch (fb_cmd) {
-        /* case DETECT_FLOWBITS_CMD_NOALERT can't happen here */
-
+        /* noalert can't happen here */
         case DETECT_FLOWBITS_CMD_ISNOTSET:
         case DETECT_FLOWBITS_CMD_ISSET:
             /* checks, so packet list */
diff --git a/src/detect-flowbits.h b/src/detect-flowbits.h
index 5ecd6cf87296..5e382de0a7a6 100644
--- a/src/detect-flowbits.h
+++ b/src/detect-flowbits.h
@@ -30,8 +30,7 @@
 #define DETECT_FLOWBITS_CMD_UNSET    2
 #define DETECT_FLOWBITS_CMD_ISNOTSET 3
 #define DETECT_FLOWBITS_CMD_ISSET    4
-#define DETECT_FLOWBITS_CMD_NOALERT  5
-#define DETECT_FLOWBITS_CMD_MAX      6
+#define DETECT_FLOWBITS_CMD_MAX      5
 
 typedef struct DetectFlowbitsData_ {
     uint32_t idx;

From 1b5e04bee3c8bb0469b53af0059ec86ce15f9b9c Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Sun, 19 Nov 2023 20:28:28 +0100
Subject: [PATCH 260/462] http2: do not have leading space for response line

Ticket: 6547
---
 rust/src/http2/detect.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rust/src/http2/detect.rs b/rust/src/http2/detect.rs
index 1c595a0cb0f8..53258b3aa7d8 100644
--- a/rust/src/http2/detect.rs
+++ b/rust/src/http2/detect.rs
@@ -545,7 +545,7 @@ fn http2_tx_get_resp_line(tx: &mut HTTP2Transaction) {
         } else {
             &empty
         };
-    resp_line.extend(b" HTTP/2 ");
+    resp_line.extend(b"HTTP/2 ");
     resp_line.extend(status);
     resp_line.extend(b"\r\n");
     tx.resp_line.extend(resp_line)

From 8b2fd434fc04ae84317d44d0a450af3ead9a157f Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Sat, 25 Nov 2023 09:20:43 -0500
Subject: [PATCH 261/462] cppcheck/detect: Address cppcheck memory leak

Issue: 6527

Ensure that the `map->string` memory isn't leaked following an error
return from `HashListTableAdd`
---
 src/detect-engine-address.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/src/detect-engine-address.c b/src/detect-engine-address.c
index 7819969e6ef3..c30ebd6878dc 100644
--- a/src/detect-engine-address.c
+++ b/src/detect-engine-address.c
@@ -1362,23 +1362,28 @@ void DetectAddressMapFree(DetectEngineCtx *de_ctx)
     return;
 }
 
-static int DetectAddressMapAdd(DetectEngineCtx *de_ctx, const char *string,
-                        DetectAddressHead *address, bool contains_negation)
+static bool DetectAddressMapAdd(DetectEngineCtx *de_ctx, const char *string,
+        DetectAddressHead *address, bool contains_negation)
 {
     DetectAddressMap *map = SCCalloc(1, sizeof(*map));
     if (map == NULL)
-        return -1;
+        return false;
 
     map->string = SCStrdup(string);
     if (map->string == NULL) {
         SCFree(map);
-        return -1;
+        return false;
     }
     map->address = address;
     map->contains_negation = contains_negation;
 
-    BUG_ON(HashListTableAdd(de_ctx->address_table, (void *)map, 0) != 0);
-    return 0;
+    if (HashListTableAdd(de_ctx->address_table, (void *)map, 0) != 0) {
+        SCFree(map->string);
+        SCFree(map);
+        return false;
+    }
+
+    return true;
 }
 
 static const DetectAddressMap *DetectAddressMapLookup(DetectEngineCtx *de_ctx,
@@ -1471,8 +1476,11 @@ const DetectAddressHead *DetectParseAddress(DetectEngineCtx *de_ctx,
         *contains_negation = false;
     }
 
-    DetectAddressMapAdd((DetectEngineCtx *)de_ctx, string, head,
-            *contains_negation);
+    if (!DetectAddressMapAdd((DetectEngineCtx *)de_ctx, string, head, *contains_negation)) {
+        DetectAddressHeadFree(head);
+        return NULL;
+    }
+
     return head;
 }
 

From 40e3514e7a6c89a786ebf17469404524fb0d2d52 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Sat, 25 Nov 2023 09:22:19 -0500
Subject: [PATCH 262/462] cppcheck: Address cpcheck report of an FP

Issue: 6527

Address the FP raised by cppcheck -- note that although the code
corectly checks to ensure that `to_shift != &sb->reqion`, the logic was
detected as a FP. Rework the code to eliminate the FP.
---
 src/util-streaming-buffer.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/src/util-streaming-buffer.c b/src/util-streaming-buffer.c
index 7608b5082109..6ff4f438a40a 100644
--- a/src/util-streaming-buffer.c
+++ b/src/util-streaming-buffer.c
@@ -842,16 +842,11 @@ static inline void StreamingBufferSlideToOffsetWithRegions(
             r = next;
         }
         SCLogDebug("to_shift %p", to_shift);
-    } else {
-        to_shift = &sb->region;
-        SCLogDebug("shift start region %p", to_shift);
-    }
 
-    // this region is main, or will xfer its buffer to main
-    if (to_shift) {
-        SCLogDebug("main: offset %" PRIu64 " buf %p size %u offset %u", to_shift->stream_offset,
-                to_shift->buf, to_shift->buf_size, to_shift->buf_offset);
-        if (to_shift != &sb->region) {
+        // this region is main, or will xfer its buffer to main
+        if (to_shift && to_shift != &sb->region) {
+            SCLogDebug("main: offset %" PRIu64 " buf %p size %u offset %u", to_shift->stream_offset,
+                    to_shift->buf, to_shift->buf_size, to_shift->buf_offset);
             DEBUG_VALIDATE_BUG_ON(sb->region.buf != NULL);
 
             sb->region.buf = to_shift->buf;
@@ -860,12 +855,20 @@ static inline void StreamingBufferSlideToOffsetWithRegions(
             sb->region.buf_size = to_shift->buf_size;
             sb->region.next = to_shift->next;
 
+            BUG_ON(to_shift == &sb->region);
             FREE(cfg, to_shift, sizeof(*to_shift));
             to_shift = &sb->region;
             sb->regions--;
             DEBUG_VALIDATE_BUG_ON(sb->regions == 0);
         }
 
+    } else {
+        to_shift = &sb->region;
+        SCLogDebug("shift start region %p", to_shift);
+    }
+
+    // this region is main, or will xfer its buffer to main
+    if (to_shift) {
         // Do the shift. If new region is exactly at the slide offset we can skip this.
         DEBUG_VALIDATE_BUG_ON(to_shift->stream_offset > slide_offset);
         const uint32_t s = slide_offset - to_shift->stream_offset;

From 5ebae1e8ed85134d2aaf246a80fad7a4380a1229 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 6 Dec 2023 10:09:03 -0600
Subject: [PATCH 263/462] clang-format.sh: prefer clang-format-14

Add clang-format-14 as the preferred version, this is the default on
Ubuntu 22.04.
---
 scripts/clang-format.sh | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/clang-format.sh b/scripts/clang-format.sh
index fe16db07af41..fc69e49dbf31 100755
--- a/scripts/clang-format.sh
+++ b/scripts/clang-format.sh
@@ -560,9 +560,13 @@ SetTopLevelDir
 
 RequireProgram GIT git
 # ubuntu uses clang-format-{version} name for newer versions. fedora not.
-RequireProgram GIT_CLANG_FORMAT git-clang-format-11 git-clang-format-10 git-clang-format-9 git-clang-format
+RequireProgram GIT_CLANG_FORMAT git-clang-format-14 git-clang-format-11 git-clang-format-10 git-clang-format-9 git-clang-format
 GIT_CLANG_FORMAT_BINARY=clang-format
-if [[ $GIT_CLANG_FORMAT =~ .*git-clang-format-11$ ]]; then
+if [[ $GIT_CLANG_FORMAT =~ .*git-clang-format-14$ ]]; then
+    # default binary is clang-format, specify the correct version.
+    # Alternative: git config clangformat.binary "clang-format-14"
+    GIT_CLANG_FORMAT_BINARY="clang-format-14"
+elif [[ $GIT_CLANG_FORMAT =~ .*git-clang-format-11$ ]]; then
     # default binary is clang-format, specify the correct version.
     # Alternative: git config clangformat.binary "clang-format-11"
     GIT_CLANG_FORMAT_BINARY="clang-format-11"

From 93071501b5a233b0499dc4c5f00ba9be34eb2c52 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 6 Dec 2023 10:09:47 -0600
Subject: [PATCH 264/462] github-ci/formatting: update to Ubuntu 22.04

Update the formatting CI job to Ubuntu 22.04 to get a newer version of
clang-format, in this case clang-format-14.
---
 .github/workflows/formatting.yml | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

diff --git a/.github/workflows/formatting.yml b/.github/workflows/formatting.yml
index ca7b018e36df..03a8e81169f5 100644
--- a/.github/workflows/formatting.yml
+++ b/.github/workflows/formatting.yml
@@ -21,9 +21,9 @@ jobs:
 
   # Checking for correct formatting of branch for C code changes
   check-formatting:
-    name: Formatting Check (clang 9)
-    runs-on: ubuntu-20.04
-    container: ubuntu:20.04
+    name: Formatting Check (clang 14)
+    runs-on: ubuntu-22.04
+    container: ubuntu:22.04
     continue-on-error: false
     steps:
 
@@ -43,6 +43,8 @@ jobs:
                 autoconf \
                 automake \
                 cargo \
+                cbindgen \
+                clang-format-14 \
                 git \
                 libtool \
                 libpcap-dev \
@@ -58,21 +60,14 @@ jobs:
                 libnfnetlink0 \
                 libhiredis-dev \
                 libjansson-dev \
-                libpython2.7 \
                 make \
-                python \
                 rustc \
+                python-is-python3 \
+                python3 \
                 software-properties-common \
                 wget \
                 zlib1g \
                 zlib1g-dev
-      - name: Install packages for clang-format 9
-        run: |
-          # no need to install full clang
-          apt-get install -y clang-format-9
-      - name: Install cbindgen
-        run: cargo install --force --debug --version 0.24.3 cbindgen
-      - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       # Checking out the branch is not as simple as "checking out".
       #
       # In case master has any new commits since we branched off, github will

From 3456dea276c209b5bf0f95259a42f89d121ada32 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 7 Dec 2023 10:27:41 +0100
Subject: [PATCH 265/462] doc/userguide: update guidance on 5 to 6 upgrading

TCP memory use can be higher than expected in certain configs.

Ticket: #6552.
---
 doc/userguide/upgrade.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/doc/userguide/upgrade.rst b/doc/userguide/upgrade.rst
index e35a596a407b..fd6e27501047 100644
--- a/doc/userguide/upgrade.rst
+++ b/doc/userguide/upgrade.rst
@@ -179,6 +179,12 @@ Removals
   if this behavior is still required. See :ref:`multiple-eve-instances`.
 - Unified2 has been removed. See :ref:`unified2-removed`.
 
+Performance
+~~~~~~~~~~~
+- In YAML files w/o a `flow-timeouts.tcp.closed` setting, the default went from 0 to 10 seconds.
+  This may lead to higher than expected TCP memory use:
+  https://redmine.openinfosecfoundation.org/issues/6552
+
 Upgrading 4.1 to 5.0
 --------------------
 

From 879db3dbc3e93912c784375c85d88404a9371f31 Mon Sep 17 00:00:00 2001
From: Stephen Donnelly <stephen.donnelly@endace.com>
Date: Mon, 11 Dec 2023 15:32:06 +1300
Subject: [PATCH 266/462] endace: Fix source-dag timestamps

Bug: #6618.

Fix Endace ERF to SCTime_t timestamp conversion

Fix typo preventing compilation with --enable-dag
---
 src/source-erf-dag.c  | 10 +++-------
 src/source-erf-file.c |  7 +------
 2 files changed, 4 insertions(+), 13 deletions(-)

diff --git a/src/source-erf-dag.c b/src/source-erf-dag.c
index e3c820dc4c08..b7ea14d83239 100644
--- a/src/source-erf-dag.c
+++ b/src/source-erf-dag.c
@@ -186,7 +186,7 @@ ReceiveErfDagThreadInit(ThreadVars *tv, void *initdata, void **data)
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    ErfDagThreadVars *ewtn = SCMClloc(1, sizeof(ErfDagThreadVars));
+    ErfDagThreadVars *ewtn = SCCalloc(1, sizeof(ErfDagThreadVars));
     if (unlikely(ewtn == NULL)) {
         FatalError("Failed to allocate memory for ERF DAG thread vars.");
     }
@@ -506,17 +506,13 @@ ProcessErfDagRecord(ErfDagThreadVars *ewtn, char *prec)
         SCReturnInt(TM_ECODE_FAILED);
     }
 
-    /* Convert ERF time to timeval - from libpcap. */
+    /* Convert ERF time to SCTime_t */
     uint64_t ts = dr->ts;
     p->ts = SCTIME_FROM_SECS(ts >> 32);
     ts = (ts & 0xffffffffULL) * 1000000;
     ts += 0x80000000; /* rounding */
     uint64_t usecs = ts >> 32;
-    if (usecs >= 1000000) {
-        usecs -= 1000000;
-        p->ts += SCTIME_FROM_SECS(1);
-    }
-    p->ts += SCTIME_FROM_USECS(usecs);
+    p->ts = SCTIME_ADD_USECS(p->ts, usecs);
 
     StatsIncr(ewtn->tv, ewtn->packets);
     ewtn->bytes += wlen;
diff --git a/src/source-erf-file.c b/src/source-erf-file.c
index 4803f8b3e28f..f3102cebf3e7 100644
--- a/src/source-erf-file.c
+++ b/src/source-erf-file.c
@@ -195,17 +195,12 @@ static inline TmEcode ReadErfRecord(ThreadVars *tv, Packet *p, void *data)
     GET_PKT_LEN(p) = wlen;
     p->datalink = LINKTYPE_ETHERNET;
 
-    /* Convert ERF time to timeval - from libpcap. */
+    /* Convert ERF time to SCTime_t */
     uint64_t ts = dr.ts;
     p->ts = SCTIME_FROM_SECS(ts >> 32);
     ts = (ts & 0xffffffffULL) * 1000000;
     ts += 0x80000000; /* rounding */
     uint64_t usecs = (ts >> 32);
-    if (usecs >= 1000000) {
-        usecs -= 1000000;
-        p->ts = SCTIME_ADD_SECS(p->ts, 1);
-        usecs++;
-    }
     p->ts = SCTIME_ADD_USECS(p->ts, usecs);
 
     etv->pkts++;

From 774f05d83d3fb39d8a60a147cb259614a2477854 Mon Sep 17 00:00:00 2001
From: Hadiqa Alamdar Bukhari <hadiqaalamdar@gmail.com>
Date: Wed, 13 Dec 2023 16:15:50 +0500
Subject: [PATCH 267/462] detect/analyzer: add details to flowbits keyword

Task #6309
---
 src/detect-engine-analyzer.c | 42 ++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/src/detect-engine-analyzer.c b/src/detect-engine-analyzer.c
index a37afabb0f00..0eda31b2fc4a 100644
--- a/src/detect-engine-analyzer.c
+++ b/src/detect-engine-analyzer.c
@@ -45,6 +45,8 @@
 #include "util-time.h"
 #include "util-validate.h"
 #include "util-conf.h"
+#include "detect-flowbits.h"
+#include "util-var-name.h"
 
 static int rule_warnings_only = 0;
 
@@ -861,6 +863,46 @@ static void DumpMatches(RuleAnalyzer *ctx, JsonBuilder *js, const SigMatchData *
                 jb_close(js);
                 break;
             }
+            case DETECT_FLOWBITS: {
+                const DetectFlowbitsData *cd = (const DetectFlowbitsData *)smd->ctx;
+
+                jb_open_object(js, "flowbits");
+                switch (cd->cmd) {
+                    case DETECT_FLOWBITS_CMD_ISSET:
+                        jb_set_string(js, "cmd", "isset");
+                        break;
+                    case DETECT_FLOWBITS_CMD_ISNOTSET:
+                        jb_set_string(js, "cmd", "isnotset");
+                        break;
+                    case DETECT_FLOWBITS_CMD_SET:
+                        jb_set_string(js, "cmd", "set");
+                        break;
+                    case DETECT_FLOWBITS_CMD_UNSET:
+                        jb_set_string(js, "cmd", "unset");
+                        break;
+                    case DETECT_FLOWBITS_CMD_TOGGLE:
+                        jb_set_string(js, "cmd", "toggle");
+                        break;
+                }
+                bool is_or = false;
+                jb_open_array(js, "names");
+                if (cd->or_list_size == 0) {
+                    jb_append_string(js, VarNameStoreSetupLookup(cd->idx, VAR_TYPE_FLOW_BIT));
+                } else if (cd->or_list_size > 0) {
+                    is_or = true;
+                    for (uint8_t i = 0; i < cd->or_list_size; i++) {
+                        const char *varname =
+                                VarNameStoreSetupLookup(cd->or_list[i], VAR_TYPE_FLOW_BIT);
+                        jb_append_string(js, varname);
+                    }
+                }
+                jb_close(js); // array
+                if (is_or) {
+                    jb_set_string(js, "operator", "or");
+                }
+                jb_close(js); // object
+                break;
+            }
         }
         jb_close(js);
 

From 50be0988395e47cea2ed969081d17ccab12784d7 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 23 Oct 2023 15:05:43 -0600
Subject: [PATCH 268/462] detect: rename DetectAppLayerMpmRegister2 to
 DetectAppLayerMpmRegister

The old DetectAppLayerMpmRegister has not been around since 4.1.x.
Rename the v2 of this function to a versionless function as there is no
documentation referring to what the 2 means.
---
 .../extending/app-layer/transactions.rst      |  6 +++---
 src/detect-dce-stub-data.c                    | 20 ++++++++-----------
 src/detect-dnp3.c                             | 10 ++++------
 src/detect-dns-query.c                        |  5 ++---
 src/detect-engine-mpm.c                       |  2 +-
 src/detect-engine-mpm.h                       |  2 +-
 src/detect-http-client-body.c                 |  4 ++--
 src/detect-http-cookie.c                      |  8 ++++----
 src/detect-http-header-names.c                |  8 ++++----
 src/detect-http-header.c                      | 16 +++++++--------
 src/detect-http-headers-stub.h                |  8 ++++----
 src/detect-http-host.c                        |  8 ++++----
 src/detect-http-method.c                      |  4 ++--
 src/detect-http-protocol.c                    |  8 ++++----
 src/detect-http-raw-header.c                  |  8 ++++----
 src/detect-http-request-line.c                |  4 ++--
 src/detect-http-response-line.c               |  4 ++--
 src/detect-http-start.c                       |  4 ++--
 src/detect-http-stat-code.c                   |  4 ++--
 src/detect-http-stat-msg.c                    |  4 ++--
 src/detect-http-ua.c                          |  4 ++--
 src/detect-http-uri.c                         |  8 ++++----
 src/detect-http2.c                            | 10 ++++------
 src/detect-ike-key-exchange-payload.c         |  4 ++--
 src/detect-ike-nonce-payload.c                |  4 ++--
 src/detect-ike-spi.c                          |  4 ++--
 src/detect-ike-vendor.c                       |  2 +-
 src/detect-krb5-cname.c                       |  5 ++---
 src/detect-krb5-sname.c                       |  5 ++---
 src/detect-mqtt-connect-clientid.c            |  5 ++---
 src/detect-mqtt-connect-password.c            |  5 ++---
 src/detect-mqtt-connect-protocol-string.c     |  2 +-
 src/detect-mqtt-connect-username.c            |  5 ++---
 src/detect-mqtt-connect-willmessage.c         |  5 ++---
 src/detect-mqtt-connect-willtopic.c           |  5 ++---
 src/detect-mqtt-publish-message.c             |  5 ++---
 src/detect-mqtt-publish-topic.c               |  5 ++---
 src/detect-mqtt-subscribe-topic.c             |  5 ++---
 src/detect-mqtt-unsubscribe-topic.c           |  5 ++---
 src/detect-parse.c                            | 10 ++++------
 src/detect-quic-cyu-hash.c                    |  2 +-
 src/detect-quic-cyu-string.c                  |  2 +-
 src/detect-quic-sni.c                         |  2 +-
 src/detect-quic-ua.c                          |  2 +-
 src/detect-quic-version.c                     |  4 ++--
 src/detect-rfb-name.c                         |  5 ++---
 src/detect-sip-method.c                       |  5 ++---
 src/detect-sip-protocol.c                     | 10 ++++------
 src/detect-sip-request-line.c                 |  5 ++---
 src/detect-sip-response-line.c                |  5 ++---
 src/detect-sip-stat-code.c                    |  5 ++---
 src/detect-sip-stat-msg.c                     |  5 ++---
 src/detect-sip-uri.c                          |  5 ++---
 src/detect-smb-ntlmssp.c                      |  4 ++--
 src/detect-smb-share.c                        | 10 ++++------
 src/detect-snmp-community.c                   |  8 ++++----
 src/detect-snmp-usm.c                         |  4 ++--
 src/detect-ssh-hassh-server-string.c          |  6 ++----
 src/detect-ssh-hassh-server.c                 |  5 ++---
 src/detect-ssh-hassh-string.c                 |  6 ++----
 src/detect-ssh-hassh.c                        | 11 ++++------
 src/detect-ssh-proto.c                        | 16 ++++++---------
 src/detect-ssh-software.c                     | 15 ++++++--------
 src/detect-tls-cert-fingerprint.c             |  7 +++----
 src/detect-tls-cert-issuer.c                  |  7 +++----
 src/detect-tls-cert-serial.c                  |  7 +++----
 src/detect-tls-cert-subject.c                 |  9 ++++-----
 src/detect-tls-certs.c                        |  9 ++++-----
 src/detect-tls-ja3-hash.c                     |  6 +++---
 src/detect-tls-ja3-string.c                   |  6 +++---
 src/detect-tls-ja3s-hash.c                    |  6 +++---
 src/detect-tls-ja3s-string.c                  |  6 +++---
 src/detect-tls-random.c                       | 16 +++++++--------
 src/detect-tls-sni.c                          |  4 ++--
 74 files changed, 206 insertions(+), 259 deletions(-)

diff --git a/doc/userguide/devguide/extending/app-layer/transactions.rst b/doc/userguide/devguide/extending/app-layer/transactions.rst
index 357bdcd76d73..1a7e4ca46443 100644
--- a/doc/userguide/devguide/extending/app-layer/transactions.rst
+++ b/doc/userguide/devguide/extending/app-layer/transactions.rst
@@ -68,7 +68,7 @@ Rule Matching
 
 Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns.
 
-As seen in ``DetectAppLayerMpmRegister2`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
+As seen in ``DetectAppLayerMpmRegister`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
 ``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions - in the last
 example, for ``FTPDATA``,
 the existence of a transaction implies that a file is being transferred. Hence the ``0`` value.
@@ -80,10 +80,10 @@ the existence of a transaction implies that a file is being transferred. Hence t
     {
         .
         .
-        DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOSERVER, 2,
+        DetectAppLayerMpmRegister("file_data", SIG_FLAG_TOSERVER, 2,
                 PrefilterMpmFiledataRegister, NULL,
                 ALPROTO_HTTP2, HTTP2StateDataClient);
-        DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOCLIENT, 2,
+        DetectAppLayerMpmRegister("file_data", SIG_FLAG_TOCLIENT, 2,
                 PrefilterMpmFiledataRegister, NULL,
                 ALPROTO_HTTP2, HTTP2StateDataServer);
         .
diff --git a/src/detect-dce-stub-data.c b/src/detect-dce-stub-data.c
index 50d0387b0758..ec7f0f620f37 100644
--- a/src/detect-dce-stub-data.c
+++ b/src/detect-dce-stub-data.c
@@ -129,31 +129,27 @@ void DetectDceStubDataRegister(void)
             ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric,
             GetSMBData);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetSMBData,
-            ALPROTO_SMB, 0);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetSMBData, ALPROTO_SMB, 0);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SMB, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric,
             GetSMBData);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetSMBData,
-            ALPROTO_SMB, 0);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetSMBData, ALPROTO_SMB, 0);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_DCERPC, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric,
             GetDCEData);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetDCEData,
-            ALPROTO_DCERPC, 0);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetDCEData, ALPROTO_DCERPC, 0);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_DCERPC, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric,
             GetDCEData);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetDCEData,
-            ALPROTO_DCERPC, 0);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetDCEData, ALPROTO_DCERPC, 0);
 
     g_dce_stub_data_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
 }
diff --git a/src/detect-dnp3.c b/src/detect-dnp3.c
index 6d92596c1d73..596e1fa13b24 100644
--- a/src/detect-dnp3.c
+++ b/src/detect-dnp3.c
@@ -533,17 +533,15 @@ static void DetectDNP3DataRegister(void)
             ALPROTO_DNP3, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric,
             GetDNP3Data);
-    DetectAppLayerMpmRegister2("dnp3_data", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetDNP3Data,
-            ALPROTO_DNP3, 0);
+    DetectAppLayerMpmRegister("dnp3_data", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetDNP3Data, ALPROTO_DNP3, 0);
 
     DetectAppLayerInspectEngineRegister2("dnp3_data",
             ALPROTO_DNP3, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric,
             GetDNP3Data);
-    DetectAppLayerMpmRegister2("dnp3_data", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetDNP3Data,
-            ALPROTO_DNP3, 0);
+    DetectAppLayerMpmRegister("dnp3_data", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetDNP3Data, ALPROTO_DNP3, 0);
 
     g_dnp3_data_buffer_id = DetectBufferTypeGetByName("dnp3_data");
     SCReturn;
diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index 43e1595e491a..b9231fd85b6e 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -203,9 +203,8 @@ void DetectDnsQueryRegister (void)
     sigmatch_table[DETECT_AL_DNS_QUERY].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_DNS_QUERY].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2("dns_query", SIG_FLAG_TOSERVER, 2,
-            PrefilterMpmDnsQueryRegister, NULL,
-            ALPROTO_DNS, 1);
+    DetectAppLayerMpmRegister(
+            "dns_query", SIG_FLAG_TOSERVER, 2, PrefilterMpmDnsQueryRegister, NULL, ALPROTO_DNS, 1);
 
     DetectAppLayerInspectEngineRegister2("dns_query",
             ALPROTO_DNS, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-engine-mpm.c b/src/detect-engine-mpm.c
index 849930a7a9cf..6a637ffc6431 100644
--- a/src/detect-engine-mpm.c
+++ b/src/detect-engine-mpm.c
@@ -86,7 +86,7 @@ static int g_mpm_list_cnt[DETECT_BUFFER_MPM_TYPE_SIZE] = { 0, 0, 0 };
  *
  *  \note to be used at start up / registration only. Errors are fatal.
  */
-void DetectAppLayerMpmRegister2(const char *name, int direction, int priority,
+void DetectAppLayerMpmRegister(const char *name, int direction, int priority,
         PrefilterRegisterFunc PrefilterRegister, InspectionBufferGetDataPtr GetData,
         AppProto alproto, int tx_min_progress)
 {
diff --git a/src/detect-engine-mpm.h b/src/detect-engine-mpm.h
index adb40297190f..b05f86e43eb1 100644
--- a/src/detect-engine-mpm.h
+++ b/src/detect-engine-mpm.h
@@ -90,7 +90,7 @@ typedef int (*PrefilterRegisterFunc)(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
  *  \note direction must be set to either toserver or toclient.
  *        If both are needed, register the keyword twice.
  */
-void DetectAppLayerMpmRegister2(const char *name, int direction, int priority,
+void DetectAppLayerMpmRegister(const char *name, int direction, int priority,
         PrefilterRegisterFunc PrefilterRegister, InspectionBufferGetDataPtr GetData,
         AppProto alproto, int tx_min_progress);
 void DetectAppLayerMpmRegisterByParentId(
diff --git a/src/detect-http-client-body.c b/src/detect-http-client-body.c
index 9e14b1b49fad..49024221e47b 100644
--- a/src/detect-http-client-body.c
+++ b/src/detect-http-client-body.c
@@ -106,12 +106,12 @@ void DetectHttpClientBodyRegister(void)
     DetectAppLayerInspectEngineRegister2("http_client_body", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_BODY, DetectEngineInspectBufferHttpBody, NULL);
 
-    DetectAppLayerMpmRegister2("http_client_body", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_client_body", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttpRequestBodyRegister, NULL, ALPROTO_HTTP1, HTP_REQUEST_BODY);
 
     DetectAppLayerInspectEngineRegister2("http_client_body", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectFiledata, NULL);
-    DetectAppLayerMpmRegister2("http_client_body", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_client_body", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmFiledataRegister, NULL, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_client_body",
diff --git a/src/detect-http-cookie.c b/src/detect-http-cookie.c
index e2754138fd44..eb6e8e01eb55 100644
--- a/src/detect-http-cookie.c
+++ b/src/detect-http-cookie.c
@@ -111,9 +111,9 @@ void DetectHttpCookieRegister(void)
     DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetResponseData);
 
-    DetectAppLayerMpmRegister2("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRequestData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
-    DetectAppLayerMpmRegister2("http_cookie", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetResponseData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
     DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
@@ -121,9 +121,9 @@ void DetectHttpCookieRegister(void)
     DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetResponseData2);
 
-    DetectAppLayerMpmRegister2("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRequestData2, ALPROTO_HTTP2, HTTP2StateDataClient);
-    DetectAppLayerMpmRegister2("http_cookie", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetResponseData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_cookie",
diff --git a/src/detect-http-header-names.c b/src/detect-http-header-names.c
index 58989a1825df..8f65726e4eef 100644
--- a/src/detect-http-header-names.c
+++ b/src/detect-http-header-names.c
@@ -219,9 +219,9 @@ void DetectHttpHeaderNamesRegister(void)
     sigmatch_table[DETECT_AL_HTTP_HEADER_NAMES].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
     /* http1 */
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_RESPONSE_HEADERS);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
@@ -230,9 +230,9 @@ void DetectHttpHeaderNamesRegister(void)
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferGeneric, GetBuffer1ForTX);
 
     /* http2 */
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataClient);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index 91e17d886379..c6de07a4d48d 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -431,24 +431,24 @@ void DetectHttpHeaderRegister(void)
 
     DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferHttpHeader, NULL);
-    DetectAppLayerMpmRegister2("http_header", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttpHeaderRequestRegister, NULL, ALPROTO_HTTP1,
             0); /* not used, registered twice: HEADERS/TRAILER */
 
     DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferHttpHeader, NULL);
-    DetectAppLayerMpmRegister2("http_header", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttpHeaderResponseRegister, NULL, ALPROTO_HTTP1,
             0); /* not used, registered twice: HEADERS/TRAILER */
 
     DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
-    DetectAppLayerMpmRegister2("http_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
-    DetectAppLayerMpmRegister2("http_header", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_header",
@@ -724,11 +724,11 @@ void DetectHttpRequestHeaderRegister(void)
     sigmatch_table[DETECT_HTTP_REQUEST_HEADER].flags |=
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2("http_request_header", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_request_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttp2HeaderRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
     DetectAppLayerInspectEngineRegister2("http_request_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateOpen, DetectEngineInspectHttp2Header, NULL);
-    DetectAppLayerMpmRegister2("http_request_header", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_request_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttp1HeaderRegister, NULL, ALPROTO_HTTP1, 0);
     DetectAppLayerInspectEngineRegister2("http_request_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectHttp1Header, NULL);
@@ -759,11 +759,11 @@ void DetectHttpResponseHeaderRegister(void)
     sigmatch_table[DETECT_HTTP_RESPONSE_HEADER].flags |=
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2("http_response_header", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_response_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttp2HeaderRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
     DetectAppLayerInspectEngineRegister2("http_response_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateOpen, DetectEngineInspectHttp2Header, NULL);
-    DetectAppLayerMpmRegister2("http_response_header", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_response_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttp1HeaderRegister, NULL, ALPROTO_HTTP1, 0);
     DetectAppLayerInspectEngineRegister2("http_response_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectHttp1Header, NULL);
diff --git a/src/detect-http-headers-stub.h b/src/detect-http-headers-stub.h
index 3a036d62209e..1f5d166063c2 100644
--- a/src/detect-http-headers-stub.h
+++ b/src/detect-http-headers-stub.h
@@ -186,15 +186,15 @@ static void DetectHttpHeadersRegisterStub(void)
     sigmatch_table[KEYWORD_ID].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
 #ifdef KEYWORD_TOSERVER
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRequestData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRequestData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 #endif
 #ifdef KEYWORD_TOCLIENT
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetResponseData, ALPROTO_HTTP1, HTP_RESPONSE_HEADERS);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetResponseData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 #endif
 #ifdef KEYWORD_TOSERVER
diff --git a/src/detect-http-host.c b/src/detect-http-host.c
index 6f32044a112c..df9c594d31c2 100644
--- a/src/detect-http-host.c
+++ b/src/detect-http-host.c
@@ -108,13 +108,13 @@ void DetectHttpHHRegister(void)
     DetectAppLayerInspectEngineRegister2("http_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
     DetectAppLayerInspectEngineRegister2("http_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeRegisterValidateCallback("http_host",
@@ -143,13 +143,13 @@ void DetectHttpHHRegister(void)
     DetectAppLayerInspectEngineRegister2("http_raw_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetRawData);
 
-    DetectAppLayerMpmRegister2("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRawData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
     DetectAppLayerInspectEngineRegister2("http_raw_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetRawData2);
 
-    DetectAppLayerMpmRegister2("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRawData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_raw_host",
diff --git a/src/detect-http-method.c b/src/detect-http-method.c
index 0ce246359ce9..ab2982238c97 100644
--- a/src/detect-http-method.c
+++ b/src/detect-http-method.c
@@ -100,13 +100,13 @@ void DetectHttpMethodRegister(void)
     DetectAppLayerInspectEngineRegister2("http_method", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_method", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_method",
diff --git a/src/detect-http-protocol.c b/src/detect-http-protocol.c
index 9dc3455d2149..f771735c6e69 100644
--- a/src/detect-http-protocol.c
+++ b/src/detect-http-protocol.c
@@ -140,9 +140,9 @@ void DetectHttpProtocolRegister(void)
     sigmatch_table[DETECT_AL_HTTP_PROTOCOL].Setup = DetectHttpProtocolSetup;
     sigmatch_table[DETECT_AL_HTTP_PROTOCOL].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
@@ -151,11 +151,11 @@ void DetectHttpProtocolRegister(void)
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME,
diff --git a/src/detect-http-raw-header.c b/src/detect-http-raw-header.c
index 1494f02d22d7..590a9d1c140a 100644
--- a/src/detect-http-raw-header.c
+++ b/src/detect-http-raw-header.c
@@ -100,10 +100,10 @@ void DetectHttpRawHeaderRegister(void)
     DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS + 1, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_raw_header", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttpHeaderRawRequestRegister, NULL, ALPROTO_HTTP1,
             0); /* progress handled in register */
-    DetectAppLayerMpmRegister2("http_raw_header", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttpHeaderRawResponseRegister, NULL, ALPROTO_HTTP1,
             0); /* progress handled in register */
 
@@ -112,9 +112,9 @@ void DetectHttpRawHeaderRegister(void)
     DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_raw_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
-    DetectAppLayerMpmRegister2("http_raw_header", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_raw_header",
diff --git a/src/detect-http-request-line.c b/src/detect-http-request-line.c
index 89d38cbd0a8a..2c56c72003e6 100644
--- a/src/detect-http-request-line.c
+++ b/src/detect-http-request-line.c
@@ -112,12 +112,12 @@ void DetectHttpRequestLineRegister(void)
     DetectAppLayerInspectEngineRegister2("http_request_line", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_request_line", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_request_line", SIG_FLAG_TOSERVER, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_request_line", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerMpmRegister2("http_request_line", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("http_request_line", SIG_FLAG_TOSERVER, 2,
             PrefilterGenericMpmRegister, GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_request_line",
diff --git a/src/detect-http-response-line.c b/src/detect-http-response-line.c
index 8758644681c7..9b1b9ed23adc 100644
--- a/src/detect-http-response-line.c
+++ b/src/detect-http-response-line.c
@@ -111,12 +111,12 @@ void DetectHttpResponseLineRegister(void)
     DetectAppLayerInspectEngineRegister2("http_response_line", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_response_line", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_response_line", SIG_FLAG_TOCLIENT, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_response_line", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerMpmRegister2("http_response_line", SIG_FLAG_TOCLIENT, 2,
+    DetectAppLayerMpmRegister("http_response_line", SIG_FLAG_TOCLIENT, 2,
             PrefilterGenericMpmRegister, GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_response_line",
diff --git a/src/detect-http-start.c b/src/detect-http-start.c
index fed1abc96256..7433c6e4cde2 100644
--- a/src/detect-http-start.c
+++ b/src/detect-http-start.c
@@ -188,9 +188,9 @@ void DetectHttpStartRegister(void)
     sigmatch_table[DETECT_AL_HTTP_START].Setup = DetectHttpStartSetup;
     sigmatch_table[DETECT_AL_HTTP_START].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_RESPONSE_HEADERS);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
diff --git a/src/detect-http-stat-code.c b/src/detect-http-stat-code.c
index 1e7087a318b3..15d8b25af611 100644
--- a/src/detect-http-stat-code.c
+++ b/src/detect-http-stat-code.c
@@ -101,13 +101,13 @@ void DetectHttpStatCodeRegister (void)
     DetectAppLayerInspectEngineRegister2("http_stat_code", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_stat_code", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_stat_code",
diff --git a/src/detect-http-stat-msg.c b/src/detect-http-stat-msg.c
index 6be7de64f756..403b87a97025 100644
--- a/src/detect-http-stat-msg.c
+++ b/src/detect-http-stat-msg.c
@@ -111,12 +111,12 @@ void DetectHttpStatMsgRegister (void)
     DetectAppLayerInspectEngineRegister2("http_stat_msg", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_stat_msg", SIG_FLAG_TOCLIENT, 3, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_stat_msg", SIG_FLAG_TOCLIENT, 3, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_stat_msg", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerMpmRegister2("http_stat_msg", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_stat_msg", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 
     DetectBufferTypeSetDescriptionByName("http_stat_msg",
diff --git a/src/detect-http-ua.c b/src/detect-http-ua.c
index 7138cf93fea4..7840478d602f 100644
--- a/src/detect-http-ua.c
+++ b/src/detect-http-ua.c
@@ -101,13 +101,13 @@ void DetectHttpUARegister(void)
     DetectAppLayerInspectEngineRegister2("http_user_agent", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
     DetectAppLayerInspectEngineRegister2("http_user_agent", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_user_agent",
diff --git a/src/detect-http-uri.c b/src/detect-http-uri.c
index cc43023a783a..f7aa2a58205a 100644
--- a/src/detect-http-uri.c
+++ b/src/detect-http-uri.c
@@ -110,13 +110,13 @@ void DetectHttpUriRegister (void)
     DetectAppLayerInspectEngineRegister2("http_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
     DetectAppLayerInspectEngineRegister2("http_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_uri",
@@ -148,14 +148,14 @@ void DetectHttpUriRegister (void)
     DetectAppLayerInspectEngineRegister2("http_raw_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetRawData);
 
-    DetectAppLayerMpmRegister2("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRawData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
     // no difference between raw and decoded uri for HTTP2
     DetectAppLayerInspectEngineRegister2("http_raw_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
-    DetectAppLayerMpmRegister2("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
 
     DetectBufferTypeSetDescriptionByName("http_raw_uri",
diff --git a/src/detect-http2.c b/src/detect-http2.c
index a65115a14cea..8143f172d21e 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -177,15 +177,13 @@ void DetectHttp2Register(void)
     sigmatch_table[DETECT_HTTP2_HEADERNAME].Setup = DetectHTTP2headerNameSetup;
     sigmatch_table[DETECT_HTTP2_HEADERNAME].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2("http2_header_name", SIG_FLAG_TOCLIENT, 2,
-                               PrefilterMpmHttp2HeaderNameRegister, NULL,
-                               ALPROTO_HTTP2, HTTP2StateOpen);
+    DetectAppLayerMpmRegister("http2_header_name", SIG_FLAG_TOCLIENT, 2,
+            PrefilterMpmHttp2HeaderNameRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
     DetectAppLayerInspectEngineRegister2("http2_header_name",
                                          ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateOpen,
                                          DetectEngineInspectHttp2HeaderName, NULL);
-    DetectAppLayerMpmRegister2("http2_header_name", SIG_FLAG_TOSERVER, 2,
-                               PrefilterMpmHttp2HeaderNameRegister, NULL,
-                               ALPROTO_HTTP2, HTTP2StateOpen);
+    DetectAppLayerMpmRegister("http2_header_name", SIG_FLAG_TOSERVER, 2,
+            PrefilterMpmHttp2HeaderNameRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
     DetectAppLayerInspectEngineRegister2("http2_header_name",
                                          ALPROTO_HTTP2, SIG_FLAG_TOSERVER, HTTP2StateOpen,
                                          DetectEngineInspectHttp2HeaderName, NULL);
diff --git a/src/detect-ike-key-exchange-payload.c b/src/detect-ike-key-exchange-payload.c
index 813e5bf080cd..03121e8d1a47 100644
--- a/src/detect-ike-key-exchange-payload.c
+++ b/src/detect-ike-key-exchange-payload.c
@@ -103,13 +103,13 @@ void DetectIkeKeyExchangeRegister(void)
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOSERVER,
             1, DetectEngineInspectBufferGeneric, GetKeyExchangeData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerMpmRegister(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOSERVER, 1,
             PrefilterGenericMpmRegister, GetKeyExchangeData, ALPROTO_IKE, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOCLIENT,
             1, DetectEngineInspectBufferGeneric, GetKeyExchangeData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerMpmRegister(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOCLIENT, 1,
             PrefilterGenericMpmRegister, GetKeyExchangeData, ALPROTO_IKE, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME_KEY_EXCHANGE, BUFFER_DESC_KEY_EXCHANGE);
diff --git a/src/detect-ike-nonce-payload.c b/src/detect-ike-nonce-payload.c
index a6b73cdf8487..6ee5ab7e72b9 100644
--- a/src/detect-ike-nonce-payload.c
+++ b/src/detect-ike-nonce-payload.c
@@ -102,13 +102,13 @@ void DetectIkeNonceRegister(void)
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetNonceData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_NONCE, SIG_FLAG_TOSERVER, 1, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME_NONCE, SIG_FLAG_TOSERVER, 1, PrefilterGenericMpmRegister,
             GetNonceData, ALPROTO_IKE, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetNonceData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_NONCE, SIG_FLAG_TOCLIENT, 1, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME_NONCE, SIG_FLAG_TOCLIENT, 1, PrefilterGenericMpmRegister,
             GetNonceData, ALPROTO_IKE, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME_NONCE, BUFFER_DESC_NONCE);
diff --git a/src/detect-ike-spi.c b/src/detect-ike-spi.c
index 5514d0202cb5..94009a4e72d6 100644
--- a/src/detect-ike-spi.c
+++ b/src/detect-ike-spi.c
@@ -141,7 +141,7 @@ void DetectIkeSpiRegister(void)
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_INITIATOR, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetInitiatorData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_INITIATOR, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerMpmRegister(BUFFER_NAME_INITIATOR, SIG_FLAG_TOSERVER, 1,
             PrefilterGenericMpmRegister, GetInitiatorData, ALPROTO_IKE, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME_INITIATOR, BUFFER_DESC_INITIATOR);
@@ -161,7 +161,7 @@ void DetectIkeSpiRegister(void)
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME_RESPONDER, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetResponderData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME_RESPONDER, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerMpmRegister(BUFFER_NAME_RESPONDER, SIG_FLAG_TOCLIENT, 1,
             PrefilterGenericMpmRegister, GetResponderData, ALPROTO_IKE, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME_RESPONDER, BUFFER_DESC_RESPONDER);
diff --git a/src/detect-ike-vendor.c b/src/detect-ike-vendor.c
index 3b84da26660b..ab26de9c0280 100644
--- a/src/detect-ike-vendor.c
+++ b/src/detect-ike-vendor.c
@@ -178,7 +178,7 @@ void DetectIkeVendorRegister(void)
     sigmatch_table[DETECT_AL_IKE_VENDOR].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_IKE_VENDOR].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerMpmRegister2("ike.vendor", SIG_FLAG_TOSERVER, 1, PrefilterMpmIkeVendorRegister,
+    DetectAppLayerMpmRegister("ike.vendor", SIG_FLAG_TOSERVER, 1, PrefilterMpmIkeVendorRegister,
             NULL, ALPROTO_IKE, 1);
 
     DetectAppLayerInspectEngineRegister2(
diff --git a/src/detect-krb5-cname.c b/src/detect-krb5-cname.c
index d509116ee73b..c6c00370f91b 100644
--- a/src/detect-krb5-cname.c
+++ b/src/detect-krb5-cname.c
@@ -187,9 +187,8 @@ void DetectKrb5CNameRegister(void)
     sigmatch_table[DETECT_AL_KRB5_CNAME].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
     sigmatch_table[DETECT_AL_KRB5_CNAME].desc = "sticky buffer to match on Kerberos 5 client name";
 
-    DetectAppLayerMpmRegister2("krb5_cname", SIG_FLAG_TOCLIENT, 2,
-            PrefilterMpmKrb5CNameRegister, NULL,
-            ALPROTO_KRB5, 1);
+    DetectAppLayerMpmRegister("krb5_cname", SIG_FLAG_TOCLIENT, 2, PrefilterMpmKrb5CNameRegister,
+            NULL, ALPROTO_KRB5, 1);
 
     DetectAppLayerInspectEngineRegister2("krb5_cname",
             ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index dae5c46e5215..17d8eee027dc 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -187,9 +187,8 @@ void DetectKrb5SNameRegister(void)
     sigmatch_table[DETECT_AL_KRB5_SNAME].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
     sigmatch_table[DETECT_AL_KRB5_SNAME].desc = "sticky buffer to match on Kerberos 5 server name";
 
-    DetectAppLayerMpmRegister2("krb5_sname", SIG_FLAG_TOCLIENT, 2,
-            PrefilterMpmKrb5SNameRegister, NULL,
-            ALPROTO_KRB5, 1);
+    DetectAppLayerMpmRegister("krb5_sname", SIG_FLAG_TOCLIENT, 2, PrefilterMpmKrb5SNameRegister,
+            NULL, ALPROTO_KRB5, 1);
 
     DetectAppLayerInspectEngineRegister2("krb5_sname",
             ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
diff --git a/src/detect-mqtt-connect-clientid.c b/src/detect-mqtt-connect-clientid.c
index 1acebf9943bc..10788441bff5 100644
--- a/src/detect-mqtt-connect-clientid.c
+++ b/src/detect-mqtt-connect-clientid.c
@@ -82,9 +82,8 @@ void DetectMQTTConnectClientIDRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-connect-password.c b/src/detect-mqtt-connect-password.c
index c08390748fe0..e337e449007f 100644
--- a/src/detect-mqtt-connect-password.c
+++ b/src/detect-mqtt-connect-password.c
@@ -82,9 +82,8 @@ void DetectMQTTConnectPasswordRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-connect-protocol-string.c b/src/detect-mqtt-connect-protocol-string.c
index 421b293845b7..333ba988f9fa 100644
--- a/src/detect-mqtt-connect-protocol-string.c
+++ b/src/detect-mqtt-connect-protocol-string.c
@@ -83,7 +83,7 @@ void DetectMQTTConnectProtocolStringRegister(void)
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
diff --git a/src/detect-mqtt-connect-username.c b/src/detect-mqtt-connect-username.c
index dbc772d22058..c3b2093da45e 100644
--- a/src/detect-mqtt-connect-username.c
+++ b/src/detect-mqtt-connect-username.c
@@ -82,9 +82,8 @@ void DetectMQTTConnectUsernameRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-connect-willmessage.c b/src/detect-mqtt-connect-willmessage.c
index 48d851d3209e..2ee26c1feffd 100644
--- a/src/detect-mqtt-connect-willmessage.c
+++ b/src/detect-mqtt-connect-willmessage.c
@@ -82,9 +82,8 @@ void DetectMQTTConnectWillMessageRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-connect-willtopic.c b/src/detect-mqtt-connect-willtopic.c
index da3d2640dd96..0dee68a9a686 100644
--- a/src/detect-mqtt-connect-willtopic.c
+++ b/src/detect-mqtt-connect-willtopic.c
@@ -82,9 +82,8 @@ void DetectMQTTConnectWillTopicRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-publish-message.c b/src/detect-mqtt-publish-message.c
index 32f3bd6460ad..6ab85667c3b4 100644
--- a/src/detect-mqtt-publish-message.c
+++ b/src/detect-mqtt-publish-message.c
@@ -82,9 +82,8 @@ void DetectMQTTPublishMessageRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-publish-topic.c b/src/detect-mqtt-publish-topic.c
index c03a47b5eda7..c25d277e3c29 100644
--- a/src/detect-mqtt-publish-topic.c
+++ b/src/detect-mqtt-publish-topic.c
@@ -82,9 +82,8 @@ void DetectMQTTPublishTopicRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_MQTT,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_MQTT, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-mqtt-subscribe-topic.c b/src/detect-mqtt-subscribe-topic.c
index 9e0058785ec1..6ff9d9104e12 100644
--- a/src/detect-mqtt-subscribe-topic.c
+++ b/src/detect-mqtt-subscribe-topic.c
@@ -203,9 +203,8 @@ void DetectMQTTSubscribeTopicRegister (void)
                 subscribe_topic_match_limit);
     }
 
-    DetectAppLayerMpmRegister2("mqtt.subscribe.topic", SIG_FLAG_TOSERVER, 1,
-            PrefilterMpmMQTTSubscribeTopicRegister, NULL,
-            ALPROTO_MQTT, 1);
+    DetectAppLayerMpmRegister("mqtt.subscribe.topic", SIG_FLAG_TOSERVER, 1,
+            PrefilterMpmMQTTSubscribeTopicRegister, NULL, ALPROTO_MQTT, 1);
 
     DetectAppLayerInspectEngineRegister2("mqtt.subscribe.topic",
             ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-mqtt-unsubscribe-topic.c b/src/detect-mqtt-unsubscribe-topic.c
index 297142dd83da..2d4de3a1f69a 100644
--- a/src/detect-mqtt-unsubscribe-topic.c
+++ b/src/detect-mqtt-unsubscribe-topic.c
@@ -203,9 +203,8 @@ void DetectMQTTUnsubscribeTopicRegister (void)
                 unsubscribe_topic_match_limit);
     }
 
-    DetectAppLayerMpmRegister2("mqtt.unsubscribe.topic", SIG_FLAG_TOSERVER, 1,
-            PrefilterMpmMQTTUnsubscribeTopicRegister, NULL,
-            ALPROTO_MQTT, 1);
+    DetectAppLayerMpmRegister("mqtt.unsubscribe.topic", SIG_FLAG_TOSERVER, 1,
+            PrefilterMpmMQTTUnsubscribeTopicRegister, NULL, ALPROTO_MQTT, 1);
 
     DetectAppLayerInspectEngineRegister2("mqtt.unsubscribe.topic",
             ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-parse.c b/src/detect-parse.c
index e1ac5f74b5a4..ba4a4db54cff 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -107,17 +107,15 @@ void DetectFileRegisterFileProtocols(DetectFileHandlerTableElmt *reg)
                                 : al_protocols[i].direction;
 
         if (direction & SIG_FLAG_TOCLIENT) {
-            DetectAppLayerMpmRegister2(reg->name, SIG_FLAG_TOCLIENT, reg->priority,
-                    reg->PrefilterFn, reg->GetData, al_protocols[i].al_proto,
-                    al_protocols[i].to_client_progress);
+            DetectAppLayerMpmRegister(reg->name, SIG_FLAG_TOCLIENT, reg->priority, reg->PrefilterFn,
+                    reg->GetData, al_protocols[i].al_proto, al_protocols[i].to_client_progress);
             DetectAppLayerInspectEngineRegister2(reg->name, al_protocols[i].al_proto,
                     SIG_FLAG_TOCLIENT, al_protocols[i].to_client_progress, reg->Callback,
                     reg->GetData);
         }
         if (direction & SIG_FLAG_TOSERVER) {
-            DetectAppLayerMpmRegister2(reg->name, SIG_FLAG_TOSERVER, reg->priority,
-                    reg->PrefilterFn, reg->GetData, al_protocols[i].al_proto,
-                    al_protocols[i].to_server_progress);
+            DetectAppLayerMpmRegister(reg->name, SIG_FLAG_TOSERVER, reg->priority, reg->PrefilterFn,
+                    reg->GetData, al_protocols[i].al_proto, al_protocols[i].to_server_progress);
             DetectAppLayerInspectEngineRegister2(reg->name, al_protocols[i].al_proto,
                     SIG_FLAG_TOSERVER, al_protocols[i].to_server_progress, reg->Callback,
                     reg->GetData);
diff --git a/src/detect-quic-cyu-hash.c b/src/detect-quic-cyu-hash.c
index 246f36a41efe..520538feb8ff 100644
--- a/src/detect-quic-cyu-hash.c
+++ b/src/detect-quic-cyu-hash.c
@@ -230,7 +230,7 @@ void DetectQuicCyuHashRegister(void)
     sigmatch_table[DETECT_AL_QUIC_CYU_HASH].RegisterTests = DetectQuicCyuHashRegisterTests;
 #endif
 
-    DetectAppLayerMpmRegister2(
+    DetectAppLayerMpmRegister(
             BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterMpmQuicHashRegister, NULL, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2(
diff --git a/src/detect-quic-cyu-string.c b/src/detect-quic-cyu-string.c
index dfec432c2049..1dafaea0d09c 100644
--- a/src/detect-quic-cyu-string.c
+++ b/src/detect-quic-cyu-string.c
@@ -183,7 +183,7 @@ void DetectQuicCyuStringRegister(void)
     sigmatch_table[DETECT_AL_QUIC_CYU_STRING].RegisterTests = DetectQuicCyuStringRegisterTests;
 #endif
 
-    DetectAppLayerMpmRegister2(
+    DetectAppLayerMpmRegister(
             BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterMpmListIdRegister, NULL, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2(
diff --git a/src/detect-quic-sni.c b/src/detect-quic-sni.c
index 722f50d04697..647308084087 100644
--- a/src/detect-quic-sni.c
+++ b/src/detect-quic-sni.c
@@ -80,7 +80,7 @@ void DetectQuicSniRegister(void)
     sigmatch_table[DETECT_AL_QUIC_SNI].RegisterTests = DetectQuicSniRegisterTests;
 #endif
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetSniData, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-quic-ua.c b/src/detect-quic-ua.c
index c491d05b06a2..f101ec9577a6 100644
--- a/src/detect-quic-ua.c
+++ b/src/detect-quic-ua.c
@@ -80,7 +80,7 @@ void DetectQuicUaRegister(void)
     sigmatch_table[DETECT_AL_QUIC_UA].RegisterTests = DetectQuicUaRegisterTests;
 #endif
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetUaData, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-quic-version.c b/src/detect-quic-version.c
index fcd99545aad5..ef4d3a602711 100644
--- a/src/detect-quic-version.c
+++ b/src/detect-quic-version.c
@@ -80,9 +80,9 @@ void DetectQuicVersionRegister(void)
     sigmatch_table[DETECT_AL_QUIC_VERSION].RegisterTests = DetectQuicVersionRegisterTests;
 #endif
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetVersionData, ALPROTO_QUIC, 1);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetVersionData, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-rfb-name.c b/src/detect-rfb-name.c
index 5e8251d51a5a..965532952bfb 100644
--- a/src/detect-rfb-name.c
+++ b/src/detect-rfb-name.c
@@ -100,9 +100,8 @@ void DetectRfbNameRegister(void)
             SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 1,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_RFB,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 1, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_RFB, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-method.c b/src/detect-sip-method.c
index fccc8a73f9fc..60160616f0da 100644
--- a/src/detect-sip-method.c
+++ b/src/detect-sip-method.c
@@ -138,9 +138,8 @@ void DetectSipMethodRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-protocol.c b/src/detect-sip-protocol.c
index 41fdcac538b3..3feb6f6e24ad 100644
--- a/src/detect-sip-protocol.c
+++ b/src/detect-sip-protocol.c
@@ -100,12 +100,10 @@ void DetectSipProtocolRegister(void)
     sigmatch_table[DETECT_AL_SIP_PROTOCOL].Setup = DetectSipProtocolSetup;
     sigmatch_table[DETECT_AL_SIP_PROTOCOL].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SIP, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetData);
diff --git a/src/detect-sip-request-line.c b/src/detect-sip-request-line.c
index 9d9f4c9c5fe5..ac5e9276ef6d 100644
--- a/src/detect-sip-request-line.c
+++ b/src/detect-sip-request-line.c
@@ -104,9 +104,8 @@ void DetectSipRequestLineRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-	        1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-response-line.c b/src/detect-sip-response-line.c
index 99061f951d5a..9929eb3644ac 100644
--- a/src/detect-sip-response-line.c
+++ b/src/detect-sip-response-line.c
@@ -104,9 +104,8 @@ void DetectSipResponseLineRegister(void)
             SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-stat-code.c b/src/detect-sip-stat-code.c
index 9b663c971e8e..eeb427dd1326 100644
--- a/src/detect-sip-stat-code.c
+++ b/src/detect-sip-stat-code.c
@@ -107,9 +107,8 @@ void DetectSipStatCodeRegister (void)
             SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 4,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-stat-msg.c b/src/detect-sip-stat-msg.c
index a9b9247a5d70..583654803c3d 100644
--- a/src/detect-sip-stat-msg.c
+++ b/src/detect-sip-stat-msg.c
@@ -107,9 +107,8 @@ void DetectSipStatMsgRegister (void)
             SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 3,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 3, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-sip-uri.c b/src/detect-sip-uri.c
index 1a000fdb543a..5c568e8c04a8 100644
--- a/src/detect-sip-uri.c
+++ b/src/detect-sip-uri.c
@@ -116,9 +116,8 @@ void DetectSipUriRegister(void)
             SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SIP,
-            1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SIP, 1);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-smb-ntlmssp.c b/src/detect-smb-ntlmssp.c
index a88b89c6f473..558488b5069a 100644
--- a/src/detect-smb-ntlmssp.c
+++ b/src/detect-smb-ntlmssp.c
@@ -81,7 +81,7 @@ void DetectSmbNtlmsspUserRegister(void)
     sigmatch_table[KEYWORD_ID].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
     sigmatch_table[KEYWORD_ID].desc = "sticky buffer to match on SMB ntlmssp user in session setup";
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetNtlmsspUserData, ALPROTO_SMB, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
@@ -139,7 +139,7 @@ void DetectSmbNtlmsspDomainRegister(void)
     sigmatch_table[KEYWORD_ID].desc =
             "sticky buffer to match on SMB ntlmssp domain in session setup";
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetNtlmsspDomainData, ALPROTO_SMB, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
diff --git a/src/detect-smb-share.c b/src/detect-smb-share.c
index 8d4d145fad8c..7d90e5622d1c 100644
--- a/src/detect-smb-share.c
+++ b/src/detect-smb-share.c
@@ -83,9 +83,8 @@ void DetectSmbNamedPipeRegister(void)
     sigmatch_table[KEYWORD_ID].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
     sigmatch_table[KEYWORD_ID].desc = "sticky buffer to match on SMB named pipe in tree connect";
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetNamedPipeData,
-            ALPROTO_SMB, 1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetNamedPipeData, ALPROTO_SMB, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
@@ -146,9 +145,8 @@ void DetectSmbShareRegister(void)
     sigmatch_table[KEYWORD_ID].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
     sigmatch_table[KEYWORD_ID].desc = "sticky buffer to match on SMB share name in tree connect";
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetShareData,
-            ALPROTO_SMB, 1);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetShareData, ALPROTO_SMB, 1);
 
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
diff --git a/src/detect-snmp-community.c b/src/detect-snmp-community.c
index 93e7d21671ab..1205f2e1a3dc 100644
--- a/src/detect-snmp-community.c
+++ b/src/detect-snmp-community.c
@@ -65,13 +65,13 @@ void DetectSNMPCommunityRegister(void)
     DetectAppLayerInspectEngineRegister2("snmp.community",
             ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerMpmRegister2("snmp.community", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SNMP, 0);
+    DetectAppLayerMpmRegister("snmp.community", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SNMP, 0);
     DetectAppLayerInspectEngineRegister2("snmp.community",
             ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerMpmRegister2("snmp.community", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_SNMP, 0);
+    DetectAppLayerMpmRegister("snmp.community", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_SNMP, 0);
 
     DetectBufferTypeSetDescriptionByName("snmp.community", "SNMP Community identifier");
 
diff --git a/src/detect-snmp-usm.c b/src/detect-snmp-usm.c
index 2e03fca16b94..153ba94d8519 100644
--- a/src/detect-snmp-usm.c
+++ b/src/detect-snmp-usm.c
@@ -68,11 +68,11 @@ void DetectSNMPUsmRegister(void)
     /* register inspect engines */
     DetectAppLayerInspectEngineRegister2("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerMpmRegister2("snmp.usm", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("snmp.usm", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
     DetectAppLayerInspectEngineRegister2("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerMpmRegister2("snmp.usm", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("snmp.usm", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
 
     DetectBufferTypeSetDescriptionByName("snmp.usm", "SNMP USM");
diff --git a/src/detect-ssh-hassh-server-string.c b/src/detect-ssh-hassh-server-string.c
index 27b0e0cb7595..c38301de0d28 100644
--- a/src/detect-ssh-hassh-server-string.c
+++ b/src/detect-ssh-hassh-server-string.c
@@ -129,10 +129,8 @@ void DetectSshHasshServerStringRegister(void)
     sigmatch_table[DETECT_AL_SSH_HASSH_SERVER_STRING].Setup = DetectSshHasshServerStringSetup;
     sigmatch_table[DETECT_AL_SSH_HASSH_SERVER_STRING].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, 
-            PrefilterGenericMpmRegister, GetSshData,
-            ALPROTO_SSH, SshStateBannerDone);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
             SIG_FLAG_TOCLIENT, SshStateBannerDone, 
             DetectEngineInspectBufferGeneric, GetSshData);
diff --git a/src/detect-ssh-hassh-server.c b/src/detect-ssh-hassh-server.c
index 295284108f10..fe225bd2fcef 100644
--- a/src/detect-ssh-hassh-server.c
+++ b/src/detect-ssh-hassh-server.c
@@ -197,9 +197,8 @@ void DetectSshHasshServerRegister(void)
     sigmatch_table[DETECT_AL_SSH_HASSH_SERVER].Setup = DetectSshHasshServerSetup;
     sigmatch_table[DETECT_AL_SSH_HASSH_SERVER].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, 
-            PrefilterGenericMpmRegister, GetSshData, 
-            ALPROTO_SSH, SshStateBannerDone);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
             SIG_FLAG_TOCLIENT, SshStateBannerDone, 
             DetectEngineInspectBufferGeneric, GetSshData);
diff --git a/src/detect-ssh-hassh-string.c b/src/detect-ssh-hassh-string.c
index e639e64b134f..af98c21bf291 100644
--- a/src/detect-ssh-hassh-string.c
+++ b/src/detect-ssh-hassh-string.c
@@ -129,10 +129,8 @@ void DetectSshHasshStringRegister(void)
     sigmatch_table[DETECT_AL_SSH_HASSH_STRING].Setup = DetectSshHasshStringSetup;
     sigmatch_table[DETECT_AL_SSH_HASSH_STRING].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, 
-            PrefilterGenericMpmRegister, GetSshData, 
-            ALPROTO_SSH, SshStateBannerDone);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
             SIG_FLAG_TOSERVER, SshStateBannerDone, 
             DetectEngineInspectBufferGeneric, GetSshData);
diff --git a/src/detect-ssh-hassh.c b/src/detect-ssh-hassh.c
index b410a5ffee84..4704b95a658e 100644
--- a/src/detect-ssh-hassh.c
+++ b/src/detect-ssh-hassh.c
@@ -199,13 +199,10 @@ void DetectSshHasshRegister(void)
     sigmatch_table[DETECT_AL_SSH_HASSH].Setup = DetectSshHasshSetup;
     sigmatch_table[DETECT_AL_SSH_HASSH].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, 
-            PrefilterGenericMpmRegister, GetSshData, 
-            ALPROTO_SSH, SshStateBannerDone),
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
-            SIG_FLAG_TOSERVER, SshStateBannerDone, 
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone),
+            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+                    SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
     g_ssh_hassh_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-ssh-proto.c b/src/detect-ssh-proto.c
index a979190de1a9..e56f846c8416 100644
--- a/src/detect-ssh-proto.c
+++ b/src/detect-ssh-proto.c
@@ -101,17 +101,13 @@ void DetectSshProtocolRegister(void)
     sigmatch_table[DETECT_AL_SSH_PROTOCOL].Setup = DetectSshProtocolSetup;
     sigmatch_table[DETECT_AL_SSH_PROTOCOL].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone),
+            DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
+                    PrefilterGenericMpmRegister, GetSshData, ALPROTO_SSH, SshStateBannerDone),
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetSshData,
-			ALPROTO_SSH, SshStateBannerDone),
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetSshData,
-			ALPROTO_SSH, SshStateBannerDone),
-
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SSH, SIG_FLAG_TOSERVER, SshStateBannerDone,
-            DetectEngineInspectBufferGeneric, GetSshData);
+            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+                    SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SSH, SIG_FLAG_TOCLIENT, SshStateBannerDone,
             DetectEngineInspectBufferGeneric, GetSshData);
diff --git a/src/detect-ssh-software.c b/src/detect-ssh-software.c
index cd11c5c20904..2b0e3d47d1cc 100644
--- a/src/detect-ssh-software.c
+++ b/src/detect-ssh-software.c
@@ -102,16 +102,13 @@ void DetectSshSoftwareRegister(void)
     sigmatch_table[DETECT_AL_SSH_SOFTWARE].Setup = DetectSshSoftwareSetup;
     sigmatch_table[DETECT_AL_SSH_SOFTWARE].flags |= SIGMATCH_INFO_STICKY_BUFFER | SIGMATCH_NOOPT;
 
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetSshData,
-			ALPROTO_SSH, SshStateBannerDone),
-    DetectAppLayerMpmRegister2(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetSshData,
-			ALPROTO_SSH, SshStateBannerDone),
+    DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetSshData, ALPROTO_SSH, SshStateBannerDone),
+            DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
+                    PrefilterGenericMpmRegister, GetSshData, ALPROTO_SSH, SshStateBannerDone),
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SSH, SIG_FLAG_TOSERVER, SshStateBannerDone,
-            DetectEngineInspectBufferGeneric, GetSshData);
+            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+                    SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
     DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
             ALPROTO_SSH, SIG_FLAG_TOCLIENT, SshStateBannerDone,
             DetectEngineInspectBufferGeneric, GetSshData);
diff --git a/src/detect-tls-cert-fingerprint.c b/src/detect-tls-cert-fingerprint.c
index 98ba46143db4..354171113f04 100644
--- a/src/detect-tls-cert-fingerprint.c
+++ b/src/detect-tls-cert-fingerprint.c
@@ -87,14 +87,13 @@ void DetectTlsFingerprintRegister(void)
             SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_fingerprint", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS,
-            TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.cert_fingerprint", SIG_FLAG_TOCLIENT, 2,
+            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectAppLayerInspectEngineRegister2("tls.cert_fingerprint", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_fingerprint", SIG_FLAG_TOSERVER, 2,
+    DetectAppLayerMpmRegister("tls.cert_fingerprint", SIG_FLAG_TOSERVER, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectBufferTypeSetDescriptionByName("tls.cert_fingerprint",
diff --git a/src/detect-tls-cert-issuer.c b/src/detect-tls-cert-issuer.c
index 9146f8d0f40b..fd8f1bcbc0ed 100644
--- a/src/detect-tls-cert-issuer.c
+++ b/src/detect-tls-cert-issuer.c
@@ -82,16 +82,15 @@ void DetectTlsIssuerRegister(void)
     DetectAppLayerInspectEngineRegister2("tls.cert_issuer", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_issuer", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.cert_issuer", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectAppLayerInspectEngineRegister2("tls.cert_issuer", ALPROTO_TLS,
             SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_issuer", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS,
-            TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.cert_issuer", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectBufferTypeSetDescriptionByName("tls.cert_issuer",
             "TLS certificate issuer");
diff --git a/src/detect-tls-cert-serial.c b/src/detect-tls-cert-serial.c
index 19c86be80e24..b1fd15d537e2 100644
--- a/src/detect-tls-cert-serial.c
+++ b/src/detect-tls-cert-serial.c
@@ -87,14 +87,13 @@ void DetectTlsSerialRegister(void)
             SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_serial", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS,
-            TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.cert_serial", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectAppLayerInspectEngineRegister2("tls.cert_serial", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_serial", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.cert_serial", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectBufferTypeSetDescriptionByName("tls.cert_serial",
diff --git a/src/detect-tls-cert-subject.c b/src/detect-tls-cert-subject.c
index 9ec7fb96fb1f..d4ceacfb1a95 100644
--- a/src/detect-tls-cert-subject.c
+++ b/src/detect-tls-cert-subject.c
@@ -82,15 +82,14 @@ void DetectTlsSubjectRegister(void)
     DetectAppLayerInspectEngineRegister2("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_subject", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.cert_subject", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectAppLayerInspectEngineRegister2("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.cert_subject", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS,
-            TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.cert_subject", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectBufferTypeSupportsMultiInstance("tls.cert_subject");
 
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index 7310461ea235..ad9d0fef0e5e 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -97,15 +97,14 @@ void DetectTlsCertsRegister(void)
             SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
             DetectEngineInspectTlsCerts, NULL);
 
-    DetectAppLayerMpmRegister2("tls.certs", SIG_FLAG_TOCLIENT, 2,
-            PrefilterMpmTlsCertsRegister, NULL, ALPROTO_TLS,
-            TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.certs", SIG_FLAG_TOCLIENT, 2, PrefilterMpmTlsCertsRegister, NULL,
+            ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectAppLayerInspectEngineRegister2("tls.certs", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectTlsCerts, NULL);
 
-    DetectAppLayerMpmRegister2("tls.certs", SIG_FLAG_TOSERVER, 2, PrefilterMpmTlsCertsRegister,
-            NULL, ALPROTO_TLS, TLS_STATE_CERT_READY);
+    DetectAppLayerMpmRegister("tls.certs", SIG_FLAG_TOSERVER, 2, PrefilterMpmTlsCertsRegister, NULL,
+            ALPROTO_TLS, TLS_STATE_CERT_READY);
 
     DetectBufferTypeSetDescriptionByName("tls.certs", "TLS certificate");
 
diff --git a/src/detect-tls-ja3-hash.c b/src/detect-tls-ja3-hash.c
index 7660fde4c2a0..2b8b5ff8912b 100644
--- a/src/detect-tls-ja3-hash.c
+++ b/src/detect-tls-ja3-hash.c
@@ -83,10 +83,10 @@ void DetectTlsJa3HashRegister(void)
     DetectAppLayerInspectEngineRegister2("ja3.hash", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("ja3.hash", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister(
+            "ja3.hash", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
 
-    DetectAppLayerMpmRegister2("ja3.hash", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("ja3.hash", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetHash, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2("ja3.hash", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-tls-ja3-string.c b/src/detect-tls-ja3-string.c
index 87a61bfd8738..920e6f4a163c 100644
--- a/src/detect-tls-ja3-string.c
+++ b/src/detect-tls-ja3-string.c
@@ -79,10 +79,10 @@ void DetectTlsJa3StringRegister(void)
     DetectAppLayerInspectEngineRegister2("ja3.string", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("ja3.string", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister("ja3.string", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, 0);
 
-    DetectAppLayerMpmRegister2("ja3.string", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("ja3.string", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetString, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2("ja3.string", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
diff --git a/src/detect-tls-ja3s-hash.c b/src/detect-tls-ja3s-hash.c
index 583566012d08..9d7429b202f7 100644
--- a/src/detect-tls-ja3s-hash.c
+++ b/src/detect-tls-ja3s-hash.c
@@ -82,10 +82,10 @@ void DetectTlsJa3SHashRegister(void)
     DetectAppLayerInspectEngineRegister2("ja3s.hash", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("ja3s.hash", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister("ja3s.hash", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, 0);
 
-    DetectAppLayerMpmRegister2("ja3s.hash", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("ja3s.hash", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetHash, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2("ja3s.hash", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
diff --git a/src/detect-tls-ja3s-string.c b/src/detect-tls-ja3s-string.c
index 0f7f7d61d067..0c4f1ba262fc 100644
--- a/src/detect-tls-ja3s-string.c
+++ b/src/detect-tls-ja3s-string.c
@@ -79,10 +79,10 @@ void DetectTlsJa3SStringRegister(void)
     DetectAppLayerInspectEngineRegister2("ja3s.string", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("ja3s.string", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister("ja3s.string", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetData, ALPROTO_TLS, 0);
 
-    DetectAppLayerMpmRegister2("ja3s.string", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("ja3s.string", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetString, ALPROTO_QUIC, 1);
 
     DetectAppLayerInspectEngineRegister2("ja3s.string", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
diff --git a/src/detect-tls-random.c b/src/detect-tls-random.c
index fc4369ab1861..6bce53a732f4 100644
--- a/src/detect-tls-random.c
+++ b/src/detect-tls-random.c
@@ -64,13 +64,13 @@ void DetectTlsRandomTimeRegister(void)
     /* Register engine for Server random */
     DetectAppLayerInspectEngineRegister2("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomTimeData);
-    DetectAppLayerMpmRegister2("tls.random_time", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.random_time", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRandomTimeData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
     DetectAppLayerInspectEngineRegister2("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomTimeData);
-    DetectAppLayerMpmRegister2("tls.random_time", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.random_time", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetRandomTimeData, ALPROTO_TLS, 0);
 
     DetectBufferTypeSetDescriptionByName("tls.random_time", "TLS Random Time");
@@ -92,14 +92,14 @@ void DetectTlsRandomBytesRegister(void)
     /* Register engine for Server random */
     DetectAppLayerInspectEngineRegister2("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomBytesData);
-    DetectAppLayerMpmRegister2("tls.random_bytes", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetRandomBytesData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister("tls.random_bytes", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+            GetRandomBytesData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
     DetectAppLayerInspectEngineRegister2("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomBytesData);
-    DetectAppLayerMpmRegister2("tls.random_bytes", SIG_FLAG_TOCLIENT, 2,
-            PrefilterGenericMpmRegister, GetRandomBytesData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister("tls.random_bytes", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+            GetRandomBytesData, ALPROTO_TLS, 0);
 
     DetectBufferTypeSetDescriptionByName("tls.random_bytes", "TLS Random Bytes");
 
@@ -124,13 +124,13 @@ void DetectTlsRandomRegister(void)
     /* Register engine for Server random */
     DetectAppLayerInspectEngineRegister2("tls.random", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomData);
-    DetectAppLayerMpmRegister2("tls.random", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.random", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRandomData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
     DetectAppLayerInspectEngineRegister2("tls.random", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomData);
-    DetectAppLayerMpmRegister2("tls.random", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
+    DetectAppLayerMpmRegister("tls.random", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetRandomData, ALPROTO_TLS, 0);
 
     DetectBufferTypeSetDescriptionByName("tls.random", "TLS Random");
diff --git a/src/detect-tls-sni.c b/src/detect-tls-sni.c
index 6ac644f1de3a..702d16081733 100644
--- a/src/detect-tls-sni.c
+++ b/src/detect-tls-sni.c
@@ -76,8 +76,8 @@ void DetectTlsSniRegister(void)
     DetectAppLayerInspectEngineRegister2("tls.sni", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerMpmRegister2("tls.sni", SIG_FLAG_TOSERVER, 2,
-            PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
+    DetectAppLayerMpmRegister(
+            "tls.sni", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, 0);
 
     DetectBufferTypeSetDescriptionByName("tls.sni",
             "TLS Server Name Indication (SNI) extension");

From b11bb1c4129728a6b8054238f221e8077029f455 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 23 Oct 2023 16:24:23 -0600
Subject: [PATCH 269/462] detect: rename DetectAppLayerInspectEngineRegister2

Rename DetectAppLayerInspectEngineRegister2 to
DetectAppLayerInspectEngineRegister as there is no other variant of
this function, and the versioning with lack of supporting
documentation can lead to confusion.
---
 .../extending/app-layer/transactions.rst      |  6 ++---
 src/detect-app-layer-event.c                  |  4 ++--
 src/detect-cipservice.c                       |  8 +++----
 src/detect-dce-iface.c                        |  8 +++----
 src/detect-dce-stub-data.c                    | 24 +++++++------------
 src/detect-dhcp-leasetime.c                   |  4 ++--
 src/detect-dhcp-rebinding-time.c              |  4 ++--
 src/detect-dhcp-renewal-time.c                |  4 ++--
 src/detect-dnp3.c                             | 16 +++++--------
 src/detect-dns-opcode.c                       |  4 ++--
 src/detect-dns-query.c                        |  9 ++++---
 src/detect-engine.c                           |  6 ++---
 src/detect-engine.h                           |  6 ++---
 src/detect-ftpbounce.c                        |  2 +-
 src/detect-ftpdata.c                          |  4 ++--
 src/detect-http-client-body.c                 |  4 ++--
 src/detect-http-cookie.c                      |  8 +++----
 src/detect-http-header-names.c                |  8 +++----
 src/detect-http-header.c                      | 16 ++++++-------
 src/detect-http-headers-stub.h                |  8 +++----
 src/detect-http-host.c                        |  8 +++----
 src/detect-http-method.c                      |  4 ++--
 src/detect-http-protocol.c                    |  8 +++----
 src/detect-http-raw-header.c                  |  8 +++----
 src/detect-http-request-line.c                |  4 ++--
 src/detect-http-response-line.c               |  4 ++--
 src/detect-http-start.c                       |  4 ++--
 src/detect-http-stat-code.c                   |  4 ++--
 src/detect-http-stat-msg.c                    |  4 ++--
 src/detect-http-ua.c                          |  4 ++--
 src/detect-http-uri.c                         |  8 +++----
 src/detect-http2.c                            | 14 +++++------
 src/detect-ike-chosen-sa.c                    |  2 +-
 src/detect-ike-exch-type.c                    |  4 ++--
 src/detect-ike-key-exchange-payload-length.c  |  4 ++--
 src/detect-ike-key-exchange-payload.c         |  8 +++----
 src/detect-ike-nonce-payload-length.c         |  4 ++--
 src/detect-ike-nonce-payload.c                |  4 ++--
 src/detect-ike-spi.c                          |  4 ++--
 src/detect-ike-vendor.c                       |  2 +-
 src/detect-krb5-cname.c                       |  5 ++--
 src/detect-krb5-errcode.c                     |  4 ++--
 src/detect-krb5-msgtype.c                     |  4 ++--
 src/detect-krb5-sname.c                       |  5 ++--
 src/detect-krb5-ticket-encryption.c           |  2 +-
 src/detect-lua.c                              |  4 ++--
 src/detect-modbus.c                           |  2 +-
 src/detect-mqtt-connack-sessionpresent.c      |  2 +-
 src/detect-mqtt-connect-clientid.c            |  3 +--
 src/detect-mqtt-connect-flags.c               |  2 +-
 src/detect-mqtt-connect-password.c            |  3 +--
 src/detect-mqtt-connect-protocol-string.c     |  2 +-
 src/detect-mqtt-connect-username.c            |  3 +--
 src/detect-mqtt-connect-willmessage.c         |  3 +--
 src/detect-mqtt-connect-willtopic.c           |  3 +--
 src/detect-mqtt-flags.c                       |  2 +-
 src/detect-mqtt-protocol-version.c            |  4 ++--
 src/detect-mqtt-publish-message.c             |  3 +--
 src/detect-mqtt-publish-topic.c               |  3 +--
 src/detect-mqtt-qos.c                         |  2 +-
 src/detect-mqtt-reason-code.c                 |  2 +-
 src/detect-mqtt-subscribe-topic.c             |  3 +--
 src/detect-mqtt-type.c                        |  2 +-
 src/detect-mqtt-unsubscribe-topic.c           |  5 ++--
 src/detect-nfs-procedure.c                    |  2 +-
 src/detect-nfs-version.c                      |  2 +-
 src/detect-parse.c                            |  4 ++--
 src/detect-quic-cyu-hash.c                    |  2 +-
 src/detect-quic-cyu-string.c                  |  2 +-
 src/detect-quic-sni.c                         |  2 +-
 src/detect-quic-ua.c                          |  2 +-
 src/detect-quic-version.c                     |  4 ++--
 src/detect-rfb-name.c                         |  3 +--
 src/detect-rfb-secresult.c                    |  2 +-
 src/detect-rfb-sectype.c                      |  2 +-
 src/detect-sip-method.c                       |  3 +--
 src/detect-sip-protocol.c                     |  6 ++---
 src/detect-sip-request-line.c                 |  3 +--
 src/detect-sip-response-line.c                |  3 +--
 src/detect-sip-stat-code.c                    |  3 +--
 src/detect-sip-stat-msg.c                     |  3 +--
 src/detect-sip-uri.c                          |  3 +--
 src/detect-smb-ntlmssp.c                      |  4 ++--
 src/detect-smb-share.c                        |  6 ++---
 src/detect-snmp-community.c                   |  6 ++---
 src/detect-snmp-pdu_type.c                    |  4 ++--
 src/detect-snmp-usm.c                         |  4 ++--
 src/detect-snmp-version.c                     |  4 ++--
 src/detect-ssh-hassh-server-string.c          |  5 ++--
 src/detect-ssh-hassh-server.c                 |  5 ++--
 src/detect-ssh-hassh-string.c                 |  5 ++--
 src/detect-ssh-hassh.c                        |  2 +-
 src/detect-ssh-proto.c                        |  7 +++---
 src/detect-ssh-software-version.c             |  4 ++--
 src/detect-ssh-software.c                     |  7 +++---
 src/detect-ssl-state.c                        |  4 ++--
 src/detect-template-rust-buffer.c             |  4 ++--
 src/detect-tls-cert-fingerprint.c             |  7 +++---
 src/detect-tls-cert-issuer.c                  |  7 +++---
 src/detect-tls-cert-serial.c                  |  7 +++---
 src/detect-tls-cert-subject.c                 |  4 ++--
 src/detect-tls-cert-validity.c                |  2 +-
 src/detect-tls-certs.c                        |  9 ++++---
 src/detect-tls-ja3-hash.c                     |  4 ++--
 src/detect-tls-ja3-string.c                   |  4 ++--
 src/detect-tls-ja3s-hash.c                    |  4 ++--
 src/detect-tls-ja3s-string.c                  |  4 ++--
 src/detect-tls-random.c                       | 12 +++++-----
 src/detect-tls-sni.c                          |  2 +-
 src/detect-tls.c                              |  4 ++--
 110 files changed, 236 insertions(+), 288 deletions(-)

diff --git a/doc/userguide/devguide/extending/app-layer/transactions.rst b/doc/userguide/devguide/extending/app-layer/transactions.rst
index 1a7e4ca46443..1105aad97128 100644
--- a/doc/userguide/devguide/extending/app-layer/transactions.rst
+++ b/doc/userguide/devguide/extending/app-layer/transactions.rst
@@ -68,7 +68,7 @@ Rule Matching
 
 Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns.
 
-As seen in ``DetectAppLayerMpmRegister`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
+As seen in ``DetectAppLayerMpmRegister`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister``, which expects ``int tx_min_progress``, for instance. In the code snippet,
 ``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions - in the last
 example, for ``FTPDATA``,
 the existence of a transaction implies that a file is being transferred. Hence the ``0`` value.
@@ -88,10 +88,10 @@ the existence of a transaction implies that a file is being transferred. Hence t
                 ALPROTO_HTTP2, HTTP2StateDataServer);
         .
         .
-        DetectAppLayerInspectEngineRegister2("file_data",
+        DetectAppLayerInspectEngineRegister("file_data",
             ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateDataServer,
             DetectEngineInspectFiledata, NULL);
-        DetectAppLayerInspectEngineRegister2(
+        DetectAppLayerInspectEngineRegister(
                 "file_data", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0, DetectEngineInspectFiledata, NULL);
         .
         .
diff --git a/src/detect-app-layer-event.c b/src/detect-app-layer-event.c
index 9c323359b577..07ba7dcb4750 100644
--- a/src/detect-app-layer-event.c
+++ b/src/detect-app-layer-event.c
@@ -78,9 +78,9 @@ void DetectAppLayerEventRegister(void)
     sigmatch_table[DETECT_AL_APP_LAYER_EVENT].Setup = DetectAppLayerEventSetup;
     sigmatch_table[DETECT_AL_APP_LAYER_EVENT].Free = DetectAppLayerEventFree;
 
-    DetectAppLayerInspectEngineRegister2("app-layer-events", ALPROTO_UNKNOWN, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("app-layer-events", ALPROTO_UNKNOWN, SIG_FLAG_TOSERVER, 0,
             DetectEngineAptEventInspect, NULL);
-    DetectAppLayerInspectEngineRegister2("app-layer-events", ALPROTO_UNKNOWN, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("app-layer-events", ALPROTO_UNKNOWN, SIG_FLAG_TOCLIENT, 0,
             DetectEngineAptEventInspect, NULL);
 
     g_applayer_events_list_id = DetectBufferTypeGetByName("app-layer-events");
diff --git a/src/detect-cipservice.c b/src/detect-cipservice.c
index 494e1e17520f..a7f1f0333c40 100644
--- a/src/detect-cipservice.c
+++ b/src/detect-cipservice.c
@@ -63,9 +63,9 @@ void DetectCipServiceRegister(void)
     sigmatch_table[DETECT_CIPSERVICE].RegisterTests
             = DetectCipServiceRegisterTests;
 #endif
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "cip", ALPROTO_ENIP, SIG_FLAG_TOSERVER, 0, DetectEngineInspectCIP, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "cip", ALPROTO_ENIP, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectCIP, NULL);
 
     g_cip_buffer_id = DetectBufferTypeGetByName("cip");
@@ -309,9 +309,9 @@ void DetectEnipCommandRegister(void)
     sigmatch_table[DETECT_ENIPCOMMAND].RegisterTests
             = DetectEnipCommandRegisterTests;
 #endif
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "enip", ALPROTO_ENIP, SIG_FLAG_TOSERVER, 0, DetectEngineInspectENIP, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "enip", ALPROTO_ENIP, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectENIP, NULL);
 
     g_enip_buffer_id = DetectBufferTypeGetByName("enip");
diff --git a/src/detect-dce-iface.c b/src/detect-dce-iface.c
index a85248e0afc7..178d3e3f921b 100644
--- a/src/detect-dce-iface.c
+++ b/src/detect-dce-iface.c
@@ -80,14 +80,14 @@ void DetectDceIfaceRegister(void)
 
     g_dce_generic_list_id = DetectBufferTypeRegister("dce_generic");
 
-    DetectAppLayerInspectEngineRegister2("dce_generic", ALPROTO_DCERPC, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("dce_generic", ALPROTO_DCERPC, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dce_generic", ALPROTO_SMB, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("dce_generic", ALPROTO_DCERPC, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("dce_generic", ALPROTO_DCERPC, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dce_generic", ALPROTO_SMB, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
 }
 
diff --git a/src/detect-dce-stub-data.c b/src/detect-dce-stub-data.c
index ec7f0f620f37..5d919e084e64 100644
--- a/src/detect-dce-stub-data.c
+++ b/src/detect-dce-stub-data.c
@@ -125,29 +125,21 @@ void DetectDceStubDataRegister(void)
 #endif
     sigmatch_table[DETECT_DCE_STUB_DATA].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
-            DetectEngineInspectBufferGeneric,
-            GetSMBData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
+            DetectEngineInspectBufferGeneric, GetSMBData);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetSMBData, ALPROTO_SMB, 0);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SMB, SIG_FLAG_TOCLIENT, 0,
-            DetectEngineInspectBufferGeneric,
-            GetSMBData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOCLIENT, 0,
+            DetectEngineInspectBufferGeneric, GetSMBData);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetSMBData, ALPROTO_SMB, 0);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_DCERPC, SIG_FLAG_TOSERVER, 0,
-            DetectEngineInspectBufferGeneric,
-            GetDCEData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_DCERPC, SIG_FLAG_TOSERVER, 0,
+            DetectEngineInspectBufferGeneric, GetDCEData);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetDCEData, ALPROTO_DCERPC, 0);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_DCERPC, SIG_FLAG_TOCLIENT, 0,
-            DetectEngineInspectBufferGeneric,
-            GetDCEData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_DCERPC, SIG_FLAG_TOCLIENT, 0,
+            DetectEngineInspectBufferGeneric, GetDCEData);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetDCEData, ALPROTO_DCERPC, 0);
 
diff --git a/src/detect-dhcp-leasetime.c b/src/detect-dhcp-leasetime.c
index fea0d108fd58..f86d645dc633 100644
--- a/src/detect-dhcp-leasetime.c
+++ b/src/detect-dhcp-leasetime.c
@@ -117,10 +117,10 @@ void DetectDHCPLeaseTimeRegister(void)
     sigmatch_table[DETECT_AL_DHCP_LEASETIME].Setup = DetectDHCPLeaseTimeSetup;
     sigmatch_table[DETECT_AL_DHCP_LEASETIME].Free = DetectDHCPLeaseTimeFree;
 
-    DetectAppLayerInspectEngineRegister2("dhcp.leasetime", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.leasetime", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("dhcp.leasetime", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.leasetime", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     g_buffer_id = DetectBufferTypeGetByName("dhcp.leasetime");
diff --git a/src/detect-dhcp-rebinding-time.c b/src/detect-dhcp-rebinding-time.c
index 8d546376a394..737d332841b7 100644
--- a/src/detect-dhcp-rebinding-time.c
+++ b/src/detect-dhcp-rebinding-time.c
@@ -118,10 +118,10 @@ void DetectDHCPRebindingTimeRegister(void)
     sigmatch_table[DETECT_AL_DHCP_REBINDING_TIME].Setup = DetectDHCPRebindingTimeSetup;
     sigmatch_table[DETECT_AL_DHCP_REBINDING_TIME].Free = DetectDHCPRebindingTimeFree;
 
-    DetectAppLayerInspectEngineRegister2("dhcp.rebinding-time", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.rebinding-time", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("dhcp.rebinding-time", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.rebinding-time", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     g_buffer_id = DetectBufferTypeGetByName("dhcp.rebinding-time");
diff --git a/src/detect-dhcp-renewal-time.c b/src/detect-dhcp-renewal-time.c
index 20ee763d9b90..d991fa1d2a4f 100644
--- a/src/detect-dhcp-renewal-time.c
+++ b/src/detect-dhcp-renewal-time.c
@@ -117,10 +117,10 @@ void DetectDHCPRenewalTimeRegister(void)
     sigmatch_table[DETECT_AL_DHCP_RENEWAL_TIME].Setup = DetectDHCPRenewalTimeSetup;
     sigmatch_table[DETECT_AL_DHCP_RENEWAL_TIME].Free = DetectDHCPRenewalTimeFree;
 
-    DetectAppLayerInspectEngineRegister2("dhcp.renewal-time", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.renewal-time", ALPROTO_DHCP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("dhcp.renewal-time", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("dhcp.renewal-time", ALPROTO_DHCP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     g_buffer_id = DetectBufferTypeGetByName("dhcp.renewal-time");
diff --git a/src/detect-dnp3.c b/src/detect-dnp3.c
index 596e1fa13b24..807c189611d8 100644
--- a/src/detect-dnp3.c
+++ b/src/detect-dnp3.c
@@ -529,17 +529,13 @@ static void DetectDNP3DataRegister(void)
     sigmatch_table[DETECT_AL_DNP3DATA].Setup         = DetectDNP3DataSetup;
     sigmatch_table[DETECT_AL_DNP3DATA].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("dnp3_data",
-            ALPROTO_DNP3, SIG_FLAG_TOSERVER, 0,
-            DetectEngineInspectBufferGeneric,
-            GetDNP3Data);
+    DetectAppLayerInspectEngineRegister("dnp3_data", ALPROTO_DNP3, SIG_FLAG_TOSERVER, 0,
+            DetectEngineInspectBufferGeneric, GetDNP3Data);
     DetectAppLayerMpmRegister("dnp3_data", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetDNP3Data, ALPROTO_DNP3, 0);
 
-    DetectAppLayerInspectEngineRegister2("dnp3_data",
-            ALPROTO_DNP3, SIG_FLAG_TOCLIENT, 0,
-            DetectEngineInspectBufferGeneric,
-            GetDNP3Data);
+    DetectAppLayerInspectEngineRegister("dnp3_data", ALPROTO_DNP3, SIG_FLAG_TOCLIENT, 0,
+            DetectEngineInspectBufferGeneric, GetDNP3Data);
     DetectAppLayerMpmRegister("dnp3_data", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetDNP3Data, ALPROTO_DNP3, 0);
 
@@ -556,9 +552,9 @@ void DetectDNP3Register(void)
     DetectDNP3ObjRegister();
 
     /* Register the list of func, ind and obj. */
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dnp3", ALPROTO_DNP3, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dnp3", ALPROTO_DNP3, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
 
     g_dnp3_match_buffer_id = DetectBufferTypeRegister("dnp3");
diff --git a/src/detect-dns-opcode.c b/src/detect-dns-opcode.c
index 853b01f0097d..4baee19b8cd3 100644
--- a/src/detect-dns-opcode.c
+++ b/src/detect-dns-opcode.c
@@ -79,10 +79,10 @@ void DetectDnsOpcodeRegister(void)
     sigmatch_table[DETECT_AL_DNS_OPCODE].AppLayerTxMatch =
         DetectDnsOpcodeMatch;
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dns.opcode", ALPROTO_DNS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dns.opcode", ALPROTO_DNS, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
 
     dns_opcode_list_id = DetectBufferTypeGetByName("dns.opcode");
diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index b9231fd85b6e..ebdc7088e78e 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -206,9 +206,8 @@ void DetectDnsQueryRegister (void)
     DetectAppLayerMpmRegister(
             "dns_query", SIG_FLAG_TOSERVER, 2, PrefilterMpmDnsQueryRegister, NULL, ALPROTO_DNS, 1);
 
-    DetectAppLayerInspectEngineRegister2("dns_query",
-            ALPROTO_DNS, SIG_FLAG_TOSERVER, 1,
-            DetectEngineInspectDnsQuery, NULL);
+    DetectAppLayerInspectEngineRegister(
+            "dns_query", ALPROTO_DNS, SIG_FLAG_TOSERVER, 1, DetectEngineInspectDnsQuery, NULL);
 
     DetectBufferTypeSetDescriptionByName("dns_query",
             "dns request query");
@@ -218,9 +217,9 @@ void DetectDnsQueryRegister (void)
 
 #ifdef HAVE_LUA
     /* register these generic engines from here for now */
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "dns_request", ALPROTO_DNS, SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2("dns_response", ALPROTO_DNS, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister("dns_response", ALPROTO_DNS, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectGenericList, NULL);
 
     DetectBufferTypeSetDescriptionByName("dns_request",
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 58aee1bfc078..22166127f44f 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -213,10 +213,8 @@ void DetectFrameInspectEngineRegister(const char *name, int dir,
 /** \brief register inspect engine at start up time
  *
  *  \note errors are fatal */
-void DetectAppLayerInspectEngineRegister2(const char *name,
-        AppProto alproto, uint32_t dir, int progress,
-        InspectEngineFuncPtr2 Callback2,
-        InspectionBufferGetDataPtr GetData)
+void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uint32_t dir,
+        int progress, InspectEngineFuncPtr2 Callback2, InspectionBufferGetDataPtr GetData)
 {
     BUG_ON(progress >= 48);
 
diff --git a/src/detect-engine.h b/src/detect-engine.h
index 02e784ee973c..a5bfc329a94e 100644
--- a/src/detect-engine.h
+++ b/src/detect-engine.h
@@ -161,10 +161,8 @@ int DetectEngineInspectPktBufferGeneric(
  * \param progress Minimal progress value for inspect engine to run
  * \param Callback The engine callback.
  */
-void DetectAppLayerInspectEngineRegister2(const char *name,
-        AppProto alproto, uint32_t dir, int progress,
-        InspectEngineFuncPtr2 Callback2,
-        InspectionBufferGetDataPtr GetData);
+void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uint32_t dir,
+        int progress, InspectEngineFuncPtr2 Callback2, InspectionBufferGetDataPtr GetData);
 
 void DetectPktInspectEngineRegister(const char *name,
         InspectionBufferGetPktDataPtr GetPktData,
diff --git a/src/detect-ftpbounce.c b/src/detect-ftpbounce.c
index 79b0f1b579e2..db54c71182cd 100644
--- a/src/detect-ftpbounce.c
+++ b/src/detect-ftpbounce.c
@@ -69,7 +69,7 @@ void DetectFtpbounceRegister(void)
 
     g_ftp_request_list_id = DetectBufferTypeRegister("ftp_request");
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "ftp_request", ALPROTO_FTP, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 }
 
diff --git a/src/detect-ftpdata.c b/src/detect-ftpdata.c
index ce9e5c3c211c..ef502c3bdd8d 100644
--- a/src/detect-ftpdata.c
+++ b/src/detect-ftpdata.c
@@ -73,10 +73,10 @@ void DetectFtpdataRegister(void) {
 #ifdef UNITTESTS
     sigmatch_table[DETECT_FTPDATA].RegisterTests = DetectFtpdataRegisterTests;
 #endif
-    DetectAppLayerInspectEngineRegister2("ftpdata_command", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("ftpdata_command", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("ftpdata_command", ALPROTO_FTPDATA, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("ftpdata_command", ALPROTO_FTPDATA, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
     g_ftpdata_buffer_id = DetectBufferTypeGetByName("ftpdata_command");
 
diff --git a/src/detect-http-client-body.c b/src/detect-http-client-body.c
index 49024221e47b..5e5604ea594d 100644
--- a/src/detect-http-client-body.c
+++ b/src/detect-http-client-body.c
@@ -103,13 +103,13 @@ void DetectHttpClientBodyRegister(void)
     sigmatch_table[DETECT_HTTP_REQUEST_BODY].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_HTTP_REQUEST_BODY].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_client_body", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_client_body", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_BODY, DetectEngineInspectBufferHttpBody, NULL);
 
     DetectAppLayerMpmRegister("http_client_body", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttpRequestBodyRegister, NULL, ALPROTO_HTTP1, HTP_REQUEST_BODY);
 
-    DetectAppLayerInspectEngineRegister2("http_client_body", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_client_body", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectFiledata, NULL);
     DetectAppLayerMpmRegister("http_client_body", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmFiledataRegister, NULL, ALPROTO_HTTP2, HTTP2StateDataClient);
diff --git a/src/detect-http-cookie.c b/src/detect-http-cookie.c
index eb6e8e01eb55..b10b8fa81e4d 100644
--- a/src/detect-http-cookie.c
+++ b/src/detect-http-cookie.c
@@ -106,9 +106,9 @@ void DetectHttpCookieRegister(void)
     sigmatch_table[DETECT_HTTP_COOKIE].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_HTTP_COOKIE].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_cookie", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetRequestData);
-    DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_cookie", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetResponseData);
 
     DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
@@ -116,9 +116,9 @@ void DetectHttpCookieRegister(void)
     DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetResponseData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetRequestData2);
-    DetectAppLayerInspectEngineRegister2("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_cookie", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetResponseData2);
 
     DetectAppLayerMpmRegister("http_cookie", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-header-names.c b/src/detect-http-header-names.c
index 8f65726e4eef..66bc73d44c80 100644
--- a/src/detect-http-header-names.c
+++ b/src/detect-http-header-names.c
@@ -224,9 +224,9 @@ void DetectHttpHeaderNamesRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_RESPONSE_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetBuffer1ForTX);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferGeneric, GetBuffer1ForTX);
 
     /* http2 */
@@ -235,9 +235,9 @@ void DetectHttpHeaderNamesRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataServer);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME,
diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index c6de07a4d48d..93942c06fa7a 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -429,24 +429,24 @@ void DetectHttpHeaderRegister(void)
     sigmatch_table[DETECT_HTTP_HEADER].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_HTTP_HEADER].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferHttpHeader, NULL);
     DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttpHeaderRequestRegister, NULL, ALPROTO_HTTP1,
             0); /* not used, registered twice: HEADERS/TRAILER */
 
-    DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferHttpHeader, NULL);
     DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttpHeaderResponseRegister, NULL, ALPROTO_HTTP1,
             0); /* not used, registered twice: HEADERS/TRAILER */
 
-    DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
     DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataClient);
 
-    DetectAppLayerInspectEngineRegister2("http_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetBuffer2ForTX);
     DetectAppLayerMpmRegister("http_header", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer2ForTX, ALPROTO_HTTP2, HTTP2StateDataServer);
@@ -726,11 +726,11 @@ void DetectHttpRequestHeaderRegister(void)
 
     DetectAppLayerMpmRegister("http_request_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttp2HeaderRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
-    DetectAppLayerInspectEngineRegister2("http_request_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_request_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateOpen, DetectEngineInspectHttp2Header, NULL);
     DetectAppLayerMpmRegister("http_request_header", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttp1HeaderRegister, NULL, ALPROTO_HTTP1, 0);
-    DetectAppLayerInspectEngineRegister2("http_request_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_request_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectHttp1Header, NULL);
 
     DetectBufferTypeSetDescriptionByName("http_request_header", "HTTP header name and value");
@@ -761,11 +761,11 @@ void DetectHttpResponseHeaderRegister(void)
 
     DetectAppLayerMpmRegister("http_response_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttp2HeaderRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
-    DetectAppLayerInspectEngineRegister2("http_response_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_response_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateOpen, DetectEngineInspectHttp2Header, NULL);
     DetectAppLayerMpmRegister("http_response_header", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttp1HeaderRegister, NULL, ALPROTO_HTTP1, 0);
-    DetectAppLayerInspectEngineRegister2("http_response_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_response_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectHttp1Header, NULL);
 
     DetectBufferTypeSetDescriptionByName("http_response_header", "HTTP header name and value");
diff --git a/src/detect-http-headers-stub.h b/src/detect-http-headers-stub.h
index 1f5d166063c2..82d5f543d7a9 100644
--- a/src/detect-http-headers-stub.h
+++ b/src/detect-http-headers-stub.h
@@ -198,15 +198,15 @@ static void DetectHttpHeadersRegisterStub(void)
             GetResponseData2, ALPROTO_HTTP2, HTTP2StateDataServer);
 #endif
 #ifdef KEYWORD_TOSERVER
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetRequestData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetRequestData2);
 #endif
 #ifdef KEYWORD_TOCLIENT
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferGeneric, GetResponseData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetResponseData2);
 #endif
 
diff --git a/src/detect-http-host.c b/src/detect-http-host.c
index df9c594d31c2..fe36a261e6cc 100644
--- a/src/detect-http-host.c
+++ b/src/detect-http-host.c
@@ -105,13 +105,13 @@ void DetectHttpHHRegister(void)
     sigmatch_table[DETECT_HTTP_HOST].Setup = DetectHttpHostSetup;
     sigmatch_table[DETECT_HTTP_HOST].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2("http_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
@@ -140,13 +140,13 @@ void DetectHttpHHRegister(void)
     sigmatch_table[DETECT_HTTP_HOST_RAW].Setup = DetectHttpHostRawSetupSticky;
     sigmatch_table[DETECT_HTTP_HOST_RAW].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_raw_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_host", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetRawData);
 
     DetectAppLayerMpmRegister("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRawData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2("http_raw_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_host", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetRawData2);
 
     DetectAppLayerMpmRegister("http_raw_host", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-method.c b/src/detect-http-method.c
index ab2982238c97..8d08f0369e90 100644
--- a/src/detect-http-method.c
+++ b/src/detect-http-method.c
@@ -97,13 +97,13 @@ void DetectHttpMethodRegister(void)
     sigmatch_table[DETECT_HTTP_METHOD].Setup = DetectHttpMethodSetupSticky;
     sigmatch_table[DETECT_HTTP_METHOD].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_method", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_method", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_method", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_method", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_method", SIG_FLAG_TOSERVER, 4, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-protocol.c b/src/detect-http-protocol.c
index f771735c6e69..ce81c5eb9804 100644
--- a/src/detect-http-protocol.c
+++ b/src/detect-http-protocol.c
@@ -144,16 +144,16 @@ void DetectHttpProtocolRegister(void)
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
diff --git a/src/detect-http-raw-header.c b/src/detect-http-raw-header.c
index 590a9d1c140a..0bb834b7726a 100644
--- a/src/detect-http-raw-header.c
+++ b/src/detect-http-raw-header.c
@@ -95,9 +95,9 @@ void DetectHttpRawHeaderRegister(void)
     sigmatch_table[DETECT_HTTP_RAW_HEADER].Setup = DetectHttpRawHeaderSetupSticky;
     sigmatch_table[DETECT_HTTP_RAW_HEADER].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_header", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS + 1, DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_raw_header", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS + 1, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOSERVER, 2,
@@ -107,9 +107,9 @@ void DetectHttpRawHeaderRegister(void)
             PrefilterMpmHttpHeaderRawResponseRegister, NULL, ALPROTO_HTTP1,
             0); /* progress handled in register */
 
-    DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_header", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
-    DetectAppLayerInspectEngineRegister2("http_raw_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_raw_header", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_raw_header", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-request-line.c b/src/detect-http-request-line.c
index 2c56c72003e6..886e643a3eda 100644
--- a/src/detect-http-request-line.c
+++ b/src/detect-http-request-line.c
@@ -109,13 +109,13 @@ void DetectHttpRequestLineRegister(void)
 #endif
     sigmatch_table[DETECT_AL_HTTP_REQUEST_LINE].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_request_line", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_request_line", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_request_line", SIG_FLAG_TOSERVER, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_request_line", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_request_line", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
     DetectAppLayerMpmRegister("http_request_line", SIG_FLAG_TOSERVER, 2,
             PrefilterGenericMpmRegister, GetData2, ALPROTO_HTTP2, HTTP2StateDataClient);
diff --git a/src/detect-http-response-line.c b/src/detect-http-response-line.c
index 9b1b9ed23adc..69ee8c2709ab 100644
--- a/src/detect-http-response-line.c
+++ b/src/detect-http-response-line.c
@@ -108,13 +108,13 @@ void DetectHttpResponseLineRegister(void)
 #endif
     sigmatch_table[DETECT_AL_HTTP_RESPONSE_LINE].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_response_line", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_response_line", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_response_line", SIG_FLAG_TOCLIENT, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_response_line", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_response_line", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
     DetectAppLayerMpmRegister("http_response_line", SIG_FLAG_TOCLIENT, 2,
             PrefilterGenericMpmRegister, GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
diff --git a/src/detect-http-start.c b/src/detect-http-start.c
index 7433c6e4cde2..e88ac3cdf68f 100644
--- a/src/detect-http-start.c
+++ b/src/detect-http-start.c
@@ -193,9 +193,9 @@ void DetectHttpStartRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetBuffer1ForTX, ALPROTO_HTTP1, HTP_RESPONSE_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetBuffer1ForTX);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_HEADERS, DetectEngineInspectBufferGeneric, GetBuffer1ForTX);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME,
diff --git a/src/detect-http-stat-code.c b/src/detect-http-stat-code.c
index 15d8b25af611..37dfb2efbdcc 100644
--- a/src/detect-http-stat-code.c
+++ b/src/detect-http-stat-code.c
@@ -98,13 +98,13 @@ void DetectHttpStatCodeRegister (void)
     sigmatch_table[DETECT_HTTP_STAT_CODE].Setup = DetectHttpStatCodeSetupSticky;
     sigmatch_table[DETECT_HTTP_STAT_CODE].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_stat_code", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_stat_code", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_stat_code", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_stat_code", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_stat_code", SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-stat-msg.c b/src/detect-http-stat-msg.c
index 403b87a97025..b1a485d7a933 100644
--- a/src/detect-http-stat-msg.c
+++ b/src/detect-http-stat-msg.c
@@ -108,13 +108,13 @@ void DetectHttpStatMsgRegister (void)
     sigmatch_table[DETECT_HTTP_STAT_MSG].Setup = DetectHttpStatMsgSetupSticky;
     sigmatch_table[DETECT_HTTP_STAT_MSG].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_stat_msg", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_stat_msg", ALPROTO_HTTP1, SIG_FLAG_TOCLIENT,
             HTP_RESPONSE_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_stat_msg", SIG_FLAG_TOCLIENT, 3, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_RESPONSE_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_stat_msg", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("http_stat_msg", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
             HTTP2StateDataServer, DetectEngineInspectBufferGeneric, GetData2);
     DetectAppLayerMpmRegister("http_stat_msg", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData2, ALPROTO_HTTP2, HTTP2StateDataServer);
diff --git a/src/detect-http-ua.c b/src/detect-http-ua.c
index 7840478d602f..8babd9adcb50 100644
--- a/src/detect-http-ua.c
+++ b/src/detect-http-ua.c
@@ -98,13 +98,13 @@ void DetectHttpUARegister(void)
     sigmatch_table[DETECT_HTTP_UA].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_HTTP_UA].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_user_agent", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_user_agent", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_HEADERS, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_HEADERS);
 
-    DetectAppLayerInspectEngineRegister2("http_user_agent", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_user_agent", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_user_agent", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-http-uri.c b/src/detect-http-uri.c
index f7aa2a58205a..12c6f8788549 100644
--- a/src/detect-http-uri.c
+++ b/src/detect-http-uri.c
@@ -107,13 +107,13 @@ void DetectHttpUriRegister (void)
     sigmatch_table[DETECT_HTTP_URI].Setup = DetectHttpUriSetupSticky;
     sigmatch_table[DETECT_HTTP_URI].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
-    DetectAppLayerInspectEngineRegister2("http_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
@@ -145,14 +145,14 @@ void DetectHttpUriRegister (void)
     sigmatch_table[DETECT_HTTP_URI_RAW].Setup = DetectHttpRawUriSetupSticky;
     sigmatch_table[DETECT_HTTP_URI_RAW].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("http_raw_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_uri", ALPROTO_HTTP1, SIG_FLAG_TOSERVER,
             HTP_REQUEST_LINE, DetectEngineInspectBufferGeneric, GetRawData);
 
     DetectAppLayerMpmRegister("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRawData, ALPROTO_HTTP1, HTP_REQUEST_LINE);
 
     // no difference between raw and decoded uri for HTTP2
-    DetectAppLayerInspectEngineRegister2("http_raw_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("http_raw_uri", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
             HTTP2StateDataClient, DetectEngineInspectBufferGeneric, GetData2);
 
     DetectAppLayerMpmRegister("http_raw_uri", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-http2.c b/src/detect-http2.c
index 8143f172d21e..25c77e11a2a2 100644
--- a/src/detect-http2.c
+++ b/src/detect-http2.c
@@ -179,22 +179,20 @@ void DetectHttp2Register(void)
 
     DetectAppLayerMpmRegister("http2_header_name", SIG_FLAG_TOCLIENT, 2,
             PrefilterMpmHttp2HeaderNameRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
-    DetectAppLayerInspectEngineRegister2("http2_header_name",
-                                         ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateOpen,
-                                         DetectEngineInspectHttp2HeaderName, NULL);
+    DetectAppLayerInspectEngineRegister("http2_header_name", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT,
+            HTTP2StateOpen, DetectEngineInspectHttp2HeaderName, NULL);
     DetectAppLayerMpmRegister("http2_header_name", SIG_FLAG_TOSERVER, 2,
             PrefilterMpmHttp2HeaderNameRegister, NULL, ALPROTO_HTTP2, HTTP2StateOpen);
-    DetectAppLayerInspectEngineRegister2("http2_header_name",
-                                         ALPROTO_HTTP2, SIG_FLAG_TOSERVER, HTTP2StateOpen,
-                                         DetectEngineInspectHttp2HeaderName, NULL);
+    DetectAppLayerInspectEngineRegister("http2_header_name", ALPROTO_HTTP2, SIG_FLAG_TOSERVER,
+            HTTP2StateOpen, DetectEngineInspectHttp2HeaderName, NULL);
     DetectBufferTypeSupportsMultiInstance("http2_header_name");
     DetectBufferTypeSetDescriptionByName("http2_header_name",
                                          "HTTP2 header name");
     g_http2_header_name_buffer_id = DetectBufferTypeGetByName("http2_header_name");
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "http2", ALPROTO_HTTP2, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "http2", ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
 
     g_http2_match_buffer_id = DetectBufferTypeRegister("http2");
diff --git a/src/detect-ike-chosen-sa.c b/src/detect-ike-chosen-sa.c
index 0ae8d400cba6..4564b5c33755 100644
--- a/src/detect-ike-chosen-sa.c
+++ b/src/detect-ike-chosen-sa.c
@@ -77,7 +77,7 @@ void DetectIkeChosenSaRegister(void)
 #endif
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("ike.chosen_sa_attribute", ALPROTO_IKE, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("ike.chosen_sa_attribute", ALPROTO_IKE, SIG_FLAG_TOCLIENT,
             1, DetectEngineInspectGenericList, NULL);
 
     g_ike_chosen_sa_buffer_id = DetectBufferTypeGetByName("ike.chosen_sa_attribute");
diff --git a/src/detect-ike-exch-type.c b/src/detect-ike-exch-type.c
index 38d4218d7faa..f77b6200a08c 100644
--- a/src/detect-ike-exch-type.c
+++ b/src/detect-ike-exch-type.c
@@ -57,10 +57,10 @@ void DetectIkeExchTypeRegister(void)
     sigmatch_table[DETECT_AL_IKE_EXCH_TYPE].Setup = DetectIkeExchTypeSetup;
     sigmatch_table[DETECT_AL_IKE_EXCH_TYPE].Free = DetectIkeExchTypeFree;
 
-    DetectAppLayerInspectEngineRegister2("ike.exchtype", ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("ike.exchtype", ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("ike.exchtype", ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister("ike.exchtype", ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectGenericList, NULL);
 
     g_ike_exch_type_buffer_id = DetectBufferTypeGetByName("ike.exchtype");
diff --git a/src/detect-ike-key-exchange-payload-length.c b/src/detect-ike-key-exchange-payload-length.c
index 4caad8038717..c698ca649193 100644
--- a/src/detect-ike-key-exchange-payload-length.c
+++ b/src/detect-ike-key-exchange-payload-length.c
@@ -61,10 +61,10 @@ void DetectIkeKeyExchangePayloadLengthRegister(void)
     sigmatch_table[DETECT_AL_IKE_KEY_EXCHANGE_PAYLOAD_LENGTH].Free =
             DetectIkeKeyExchangePayloadLengthFree;
 
-    DetectAppLayerInspectEngineRegister2("ike.key_exchange_payload_length", ALPROTO_IKE,
+    DetectAppLayerInspectEngineRegister("ike.key_exchange_payload_length", ALPROTO_IKE,
             SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("ike.key_exchange_payload_length", ALPROTO_IKE,
+    DetectAppLayerInspectEngineRegister("ike.key_exchange_payload_length", ALPROTO_IKE,
             SIG_FLAG_TOCLIENT, 1, DetectEngineInspectGenericList, NULL);
 
     g_ike_key_exch_payload_length_buffer_id =
diff --git a/src/detect-ike-key-exchange-payload.c b/src/detect-ike-key-exchange-payload.c
index 03121e8d1a47..9d83fba33dec 100644
--- a/src/detect-ike-key-exchange-payload.c
+++ b/src/detect-ike-key-exchange-payload.c
@@ -100,14 +100,14 @@ void DetectIkeKeyExchangeRegister(void)
     sigmatch_table[DETECT_AL_IKE_KEY_EXCHANGE].flags |=
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOSERVER,
-            1, DetectEngineInspectBufferGeneric, GetKeyExchangeData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
+            DetectEngineInspectBufferGeneric, GetKeyExchangeData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOSERVER, 1,
             PrefilterGenericMpmRegister, GetKeyExchangeData, ALPROTO_IKE, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOCLIENT,
-            1, DetectEngineInspectBufferGeneric, GetKeyExchangeData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_KEY_EXCHANGE, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
+            DetectEngineInspectBufferGeneric, GetKeyExchangeData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_KEY_EXCHANGE, SIG_FLAG_TOCLIENT, 1,
             PrefilterGenericMpmRegister, GetKeyExchangeData, ALPROTO_IKE, 1);
diff --git a/src/detect-ike-nonce-payload-length.c b/src/detect-ike-nonce-payload-length.c
index fbb3a903366a..f76fdb70d496 100644
--- a/src/detect-ike-nonce-payload-length.c
+++ b/src/detect-ike-nonce-payload-length.c
@@ -57,10 +57,10 @@ void DetectIkeNoncePayloadLengthRegister(void)
     sigmatch_table[DETECT_AL_IKE_NONCE_PAYLOAD_LENGTH].Setup = DetectIkeNoncePayloadLengthSetup;
     sigmatch_table[DETECT_AL_IKE_NONCE_PAYLOAD_LENGTH].Free = DetectIkeNoncePayloadLengthFree;
 
-    DetectAppLayerInspectEngineRegister2("ike.nonce_payload_length", ALPROTO_IKE, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("ike.nonce_payload_length", ALPROTO_IKE, SIG_FLAG_TOSERVER,
             1, DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("ike.nonce_payload_length", ALPROTO_IKE, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("ike.nonce_payload_length", ALPROTO_IKE, SIG_FLAG_TOCLIENT,
             1, DetectEngineInspectGenericList, NULL);
 
     g_ike_nonce_payload_length_buffer_id = DetectBufferTypeGetByName("ike.nonce_payload_length");
diff --git a/src/detect-ike-nonce-payload.c b/src/detect-ike-nonce-payload.c
index 6ee5ab7e72b9..a2c4ac6f9a2a 100644
--- a/src/detect-ike-nonce-payload.c
+++ b/src/detect-ike-nonce-payload.c
@@ -99,13 +99,13 @@ void DetectIkeNonceRegister(void)
     sigmatch_table[DETECT_AL_IKE_NONCE].Setup = DetectNonceSetup;
     sigmatch_table[DETECT_AL_IKE_NONCE].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetNonceData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_NONCE, SIG_FLAG_TOSERVER, 1, PrefilterGenericMpmRegister,
             GetNonceData, ALPROTO_IKE, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_NONCE, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetNonceData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_NONCE, SIG_FLAG_TOCLIENT, 1, PrefilterGenericMpmRegister,
diff --git a/src/detect-ike-spi.c b/src/detect-ike-spi.c
index 94009a4e72d6..9f310b8f580a 100644
--- a/src/detect-ike-spi.c
+++ b/src/detect-ike-spi.c
@@ -138,7 +138,7 @@ void DetectIkeSpiRegister(void)
     sigmatch_table[DETECT_AL_IKE_SPI_INITIATOR].flags |=
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_INITIATOR, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_INITIATOR, ALPROTO_IKE, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetInitiatorData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_INITIATOR, SIG_FLAG_TOSERVER, 1,
@@ -158,7 +158,7 @@ void DetectIkeSpiRegister(void)
     sigmatch_table[DETECT_AL_IKE_SPI_RESPONDER].flags |=
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME_RESPONDER, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME_RESPONDER, ALPROTO_IKE, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetResponderData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME_RESPONDER, SIG_FLAG_TOCLIENT, 1,
diff --git a/src/detect-ike-vendor.c b/src/detect-ike-vendor.c
index ab26de9c0280..5baf24c875e1 100644
--- a/src/detect-ike-vendor.c
+++ b/src/detect-ike-vendor.c
@@ -181,7 +181,7 @@ void DetectIkeVendorRegister(void)
     DetectAppLayerMpmRegister("ike.vendor", SIG_FLAG_TOSERVER, 1, PrefilterMpmIkeVendorRegister,
             NULL, ALPROTO_IKE, 1);
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "ike.vendor", ALPROTO_IKE, SIG_FLAG_TOSERVER, 1, DetectEngineInspectIkeVendor, NULL);
 
     g_ike_vendor_buffer_id = DetectBufferTypeGetByName("ike.vendor");
diff --git a/src/detect-krb5-cname.c b/src/detect-krb5-cname.c
index c6c00370f91b..a85b7aa8f800 100644
--- a/src/detect-krb5-cname.c
+++ b/src/detect-krb5-cname.c
@@ -190,9 +190,8 @@ void DetectKrb5CNameRegister(void)
     DetectAppLayerMpmRegister("krb5_cname", SIG_FLAG_TOCLIENT, 2, PrefilterMpmKrb5CNameRegister,
             NULL, ALPROTO_KRB5, 1);
 
-    DetectAppLayerInspectEngineRegister2("krb5_cname",
-            ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
-            DetectEngineInspectKrb5CName, NULL);
+    DetectAppLayerInspectEngineRegister(
+            "krb5_cname", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectKrb5CName, NULL);
 
     DetectBufferTypeSetDescriptionByName("krb5_cname",
             "Kerberos 5 ticket client name");
diff --git a/src/detect-krb5-errcode.c b/src/detect-krb5-errcode.c
index f9d22cbede5d..b422bdc38e7f 100644
--- a/src/detect-krb5-errcode.c
+++ b/src/detect-krb5-errcode.c
@@ -69,10 +69,10 @@ void DetectKrb5ErrCodeRegister(void)
     sigmatch_table[DETECT_AL_KRB5_ERRCODE].RegisterTests = DetectKrb5ErrCodeRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2("krb5_err_code", ALPROTO_KRB5, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("krb5_err_code", ALPROTO_KRB5, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("krb5_err_code", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("krb5_err_code", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     /* set up the PCRE for keyword parsing */
diff --git a/src/detect-krb5-msgtype.c b/src/detect-krb5-msgtype.c
index 4e2ae85848ed..9220a2929f1b 100644
--- a/src/detect-krb5-msgtype.c
+++ b/src/detect-krb5-msgtype.c
@@ -69,10 +69,10 @@ void DetectKrb5MsgTypeRegister(void)
     sigmatch_table[DETECT_AL_KRB5_MSGTYPE].RegisterTests = DetectKrb5MsgTypeRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2("krb5_msg_type", ALPROTO_KRB5, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("krb5_msg_type", ALPROTO_KRB5, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("krb5_msg_type", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("krb5_msg_type", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     /* set up the PCRE for keyword parsing */
diff --git a/src/detect-krb5-sname.c b/src/detect-krb5-sname.c
index 17d8eee027dc..f5670bd5544c 100644
--- a/src/detect-krb5-sname.c
+++ b/src/detect-krb5-sname.c
@@ -190,9 +190,8 @@ void DetectKrb5SNameRegister(void)
     DetectAppLayerMpmRegister("krb5_sname", SIG_FLAG_TOCLIENT, 2, PrefilterMpmKrb5SNameRegister,
             NULL, ALPROTO_KRB5, 1);
 
-    DetectAppLayerInspectEngineRegister2("krb5_sname",
-            ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0,
-            DetectEngineInspectKrb5SName, NULL);
+    DetectAppLayerInspectEngineRegister(
+            "krb5_sname", ALPROTO_KRB5, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectKrb5SName, NULL);
 
     DetectBufferTypeSetDescriptionByName("krb5_sname",
             "Kerberos 5 ticket server name");
diff --git a/src/detect-krb5-ticket-encryption.c b/src/detect-krb5-ticket-encryption.c
index e3550084ffb6..6cdacd19a835 100644
--- a/src/detect-krb5-ticket-encryption.c
+++ b/src/detect-krb5-ticket-encryption.c
@@ -78,7 +78,7 @@ void DetectKrb5TicketEncryptionRegister(void)
     sigmatch_table[DETECT_AL_KRB5_TICKET_ENCRYPTION].Free = DetectKrb5TicketEncryptionFree;
 
     // Tickets are only from server to client
-    DetectAppLayerInspectEngineRegister2("krb5_ticket_encryption", ALPROTO_KRB5, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("krb5_ticket_encryption", ALPROTO_KRB5, SIG_FLAG_TOCLIENT,
             0, DetectEngineInspectGenericList, NULL);
 
     g_krb5_ticket_encryption_list_id = DetectBufferTypeRegister("krb5_ticket_encryption");
diff --git a/src/detect-lua.c b/src/detect-lua.c
index 0ea74452739e..93f4a687f87d 100644
--- a/src/detect-lua.c
+++ b/src/detect-lua.c
@@ -123,9 +123,9 @@ void DetectLuaRegister(void)
 #endif
     g_smtp_generic_list_id = DetectBufferTypeRegister("smtp_generic");
 
-    DetectAppLayerInspectEngineRegister2("smtp_generic", ALPROTO_SMTP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("smtp_generic", ALPROTO_SMTP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2("smtp_generic", ALPROTO_SMTP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("smtp_generic", ALPROTO_SMTP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     SCLogDebug("registering lua rule option");
diff --git a/src/detect-modbus.c b/src/detect-modbus.c
index f4e6d4fd03ff..e9e64c42fa57 100644
--- a/src/detect-modbus.c
+++ b/src/detect-modbus.c
@@ -127,7 +127,7 @@ void DetectModbusRegister(void)
     sigmatch_table[DETECT_AL_MODBUS].Free = DetectModbusFree;
     sigmatch_table[DETECT_AL_MODBUS].AppLayerTxMatch = DetectModbusMatch;
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "modbus", ALPROTO_MODBUS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 
     g_modbus_buffer_id = DetectBufferTypeGetByName("modbus");
diff --git a/src/detect-mqtt-connack-sessionpresent.c b/src/detect-mqtt-connack-sessionpresent.c
index 4b29158b1f89..4bb10eacacd2 100644
--- a/src/detect-mqtt-connack-sessionpresent.c
+++ b/src/detect-mqtt-connack-sessionpresent.c
@@ -62,7 +62,7 @@ void DetectMQTTConnackSessionPresentRegister (void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("mqtt.connack.session_present", ALPROTO_MQTT,
+    DetectAppLayerInspectEngineRegister("mqtt.connack.session_present", ALPROTO_MQTT,
             SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
     mqtt_connack_session_present_id = DetectBufferTypeGetByName("mqtt.connack.session_present");
diff --git a/src/detect-mqtt-connect-clientid.c b/src/detect-mqtt-connect-clientid.c
index 10788441bff5..c3bc31474342 100644
--- a/src/detect-mqtt-connect-clientid.c
+++ b/src/detect-mqtt-connect-clientid.c
@@ -78,8 +78,7 @@ void DetectMQTTConnectClientIDRegister(void)
     sigmatch_table[DETECT_AL_MQTT_CONNECT_CLIENTID].Setup = DetectMQTTConnectClientIDSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_CLIENTID].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-connect-flags.c b/src/detect-mqtt-connect-flags.c
index ce543ecdaa41..fa44af9ee042 100644
--- a/src/detect-mqtt-connect-flags.c
+++ b/src/detect-mqtt-connect-flags.c
@@ -70,7 +70,7 @@ void DetectMQTTConnectFlagsRegister (void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("mqtt.connect.flags", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("mqtt.connect.flags", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectGenericList, NULL);
 
     mqtt_connect_flags_id = DetectBufferTypeGetByName("mqtt.connect.flags");
diff --git a/src/detect-mqtt-connect-password.c b/src/detect-mqtt-connect-password.c
index e337e449007f..57ec1ba24ff9 100644
--- a/src/detect-mqtt-connect-password.c
+++ b/src/detect-mqtt-connect-password.c
@@ -78,8 +78,7 @@ void DetectMQTTConnectPasswordRegister(void)
     sigmatch_table[DETECT_AL_MQTT_CONNECT_PASSWORD].Setup = DetectMQTTConnectPasswordSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_PASSWORD].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-connect-protocol-string.c b/src/detect-mqtt-connect-protocol-string.c
index 333ba988f9fa..254cc3e2d24d 100644
--- a/src/detect-mqtt-connect-protocol-string.c
+++ b/src/detect-mqtt-connect-protocol-string.c
@@ -80,7 +80,7 @@ void DetectMQTTConnectProtocolStringRegister(void)
             DetectMQTTConnectProtocolStringSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_PROTOCOL_STRING].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-connect-username.c b/src/detect-mqtt-connect-username.c
index c3b2093da45e..607a35685493 100644
--- a/src/detect-mqtt-connect-username.c
+++ b/src/detect-mqtt-connect-username.c
@@ -78,8 +78,7 @@ void DetectMQTTConnectUsernameRegister(void)
     sigmatch_table[DETECT_AL_MQTT_CONNECT_USERNAME].Setup = DetectMQTTConnectUsernameSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_USERNAME].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-connect-willmessage.c b/src/detect-mqtt-connect-willmessage.c
index 2ee26c1feffd..8ff68a6594e3 100644
--- a/src/detect-mqtt-connect-willmessage.c
+++ b/src/detect-mqtt-connect-willmessage.c
@@ -78,8 +78,7 @@ void DetectMQTTConnectWillMessageRegister(void)
     sigmatch_table[DETECT_AL_MQTT_CONNECT_WILLMESSAGE].Setup = DetectMQTTConnectWillMessageSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_WILLMESSAGE].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-connect-willtopic.c b/src/detect-mqtt-connect-willtopic.c
index 0dee68a9a686..55efe93122eb 100644
--- a/src/detect-mqtt-connect-willtopic.c
+++ b/src/detect-mqtt-connect-willtopic.c
@@ -78,8 +78,7 @@ void DetectMQTTConnectWillTopicRegister(void)
     sigmatch_table[DETECT_AL_MQTT_CONNECT_WILLTOPIC].Setup = DetectMQTTConnectWillTopicSetup;
     sigmatch_table[DETECT_AL_MQTT_CONNECT_WILLTOPIC].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-flags.c b/src/detect-mqtt-flags.c
index d0614061416d..ad2fed2a0ab3 100644
--- a/src/detect-mqtt-flags.c
+++ b/src/detect-mqtt-flags.c
@@ -66,7 +66,7 @@ void DetectMQTTFlagsRegister (void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "mqtt.flags", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
     mqtt_flags_id = DetectBufferTypeGetByName("mqtt.flags");
diff --git a/src/detect-mqtt-protocol-version.c b/src/detect-mqtt-protocol-version.c
index 6ba183d75c8a..8368d49580e3 100644
--- a/src/detect-mqtt-protocol-version.c
+++ b/src/detect-mqtt-protocol-version.c
@@ -59,8 +59,8 @@ void DetectMQTTProtocolVersionRegister (void)
     sigmatch_table[DETECT_AL_MQTT_PROTOCOL_VERSION].RegisterTests = MQTTProtocolVersionRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2("mqtt.protocol_version", ALPROTO_MQTT, SIG_FLAG_TOSERVER,
-            1, DetectEngineInspectGenericList, NULL);
+    DetectAppLayerInspectEngineRegister("mqtt.protocol_version", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
+            DetectEngineInspectGenericList, NULL);
 
     mqtt_protocol_version_id = DetectBufferTypeGetByName("mqtt.protocol_version");
 }
diff --git a/src/detect-mqtt-publish-message.c b/src/detect-mqtt-publish-message.c
index 6ab85667c3b4..02595737271c 100644
--- a/src/detect-mqtt-publish-message.c
+++ b/src/detect-mqtt-publish-message.c
@@ -78,8 +78,7 @@ void DetectMQTTPublishMessageRegister(void)
     sigmatch_table[DETECT_AL_MQTT_PUBLISH_MESSAGE].Setup = DetectMQTTPublishMessageSetup;
     sigmatch_table[DETECT_AL_MQTT_PUBLISH_MESSAGE].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-publish-topic.c b/src/detect-mqtt-publish-topic.c
index c25d277e3c29..6538857e8c2b 100644
--- a/src/detect-mqtt-publish-topic.c
+++ b/src/detect-mqtt-publish-topic.c
@@ -78,8 +78,7 @@ void DetectMQTTPublishTopicRegister(void)
     sigmatch_table[DETECT_AL_MQTT_PUBLISH_TOPIC].Setup = DetectMQTTPublishTopicSetup;
     sigmatch_table[DETECT_AL_MQTT_PUBLISH_TOPIC].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_MQTT,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_MQTT, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-mqtt-qos.c b/src/detect-mqtt-qos.c
index a00eaee185a2..a94530a2957b 100644
--- a/src/detect-mqtt-qos.c
+++ b/src/detect-mqtt-qos.c
@@ -58,7 +58,7 @@ void DetectMQTTQosRegister (void)
     sigmatch_table[DETECT_AL_MQTT_QOS].RegisterTests = MQTTQosRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "mqtt.qos", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
     mqtt_qos_id = DetectBufferTypeGetByName("mqtt.qos");
diff --git a/src/detect-mqtt-reason-code.c b/src/detect-mqtt-reason-code.c
index e6ecba44cc26..1c60b371919c 100644
--- a/src/detect-mqtt-reason-code.c
+++ b/src/detect-mqtt-reason-code.c
@@ -64,7 +64,7 @@ void DetectMQTTReasonCodeRegister (void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("mqtt.reason_code", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("mqtt.reason_code", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectGenericList, NULL);
 
     mqtt_reason_code_id = DetectBufferTypeGetByName("mqtt.reason_code");
diff --git a/src/detect-mqtt-subscribe-topic.c b/src/detect-mqtt-subscribe-topic.c
index 6ff9d9104e12..0dbcedb48f72 100644
--- a/src/detect-mqtt-subscribe-topic.c
+++ b/src/detect-mqtt-subscribe-topic.c
@@ -206,8 +206,7 @@ void DetectMQTTSubscribeTopicRegister (void)
     DetectAppLayerMpmRegister("mqtt.subscribe.topic", SIG_FLAG_TOSERVER, 1,
             PrefilterMpmMQTTSubscribeTopicRegister, NULL, ALPROTO_MQTT, 1);
 
-    DetectAppLayerInspectEngineRegister2("mqtt.subscribe.topic",
-            ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("mqtt.subscribe.topic", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectMQTTSubscribeTopic, NULL);
 
     DetectBufferTypeSetDescriptionByName("mqtt.subscribe.topic",
diff --git a/src/detect-mqtt-type.c b/src/detect-mqtt-type.c
index 5e23a509ca7c..60b5386de838 100644
--- a/src/detect-mqtt-type.c
+++ b/src/detect-mqtt-type.c
@@ -57,7 +57,7 @@ void DetectMQTTTypeRegister (void)
     sigmatch_table[DETECT_AL_MQTT_TYPE].RegisterTests = MQTTTypeRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "mqtt.type", ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
     mqtt_type_id = DetectBufferTypeGetByName("mqtt.type");
diff --git a/src/detect-mqtt-unsubscribe-topic.c b/src/detect-mqtt-unsubscribe-topic.c
index 2d4de3a1f69a..27ae75a6c3e1 100644
--- a/src/detect-mqtt-unsubscribe-topic.c
+++ b/src/detect-mqtt-unsubscribe-topic.c
@@ -206,9 +206,8 @@ void DetectMQTTUnsubscribeTopicRegister (void)
     DetectAppLayerMpmRegister("mqtt.unsubscribe.topic", SIG_FLAG_TOSERVER, 1,
             PrefilterMpmMQTTUnsubscribeTopicRegister, NULL, ALPROTO_MQTT, 1);
 
-    DetectAppLayerInspectEngineRegister2("mqtt.unsubscribe.topic",
-            ALPROTO_MQTT, SIG_FLAG_TOSERVER, 1,
-            DetectEngineInspectMQTTUnsubscribeTopic, NULL);
+    DetectAppLayerInspectEngineRegister("mqtt.unsubscribe.topic", ALPROTO_MQTT, SIG_FLAG_TOSERVER,
+            1, DetectEngineInspectMQTTUnsubscribeTopic, NULL);
 
     DetectBufferTypeSetDescriptionByName("mqtt.unsubscribe.topic",
             "unsubscribe topic query");
diff --git a/src/detect-nfs-procedure.c b/src/detect-nfs-procedure.c
index 24c1563df18f..3afa327bec9d 100644
--- a/src/detect-nfs-procedure.c
+++ b/src/detect-nfs-procedure.c
@@ -74,7 +74,7 @@ void DetectNfsProcedureRegister (void)
     sigmatch_table[DETECT_AL_NFS_PROCEDURE].RegisterTests = DetectNfsProcedureRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "nfs_request", ALPROTO_NFS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 
     g_nfs_request_buffer_id = DetectBufferTypeGetByName("nfs_request");
diff --git a/src/detect-nfs-version.c b/src/detect-nfs-version.c
index 99c88149a73e..6ed20a2288bf 100644
--- a/src/detect-nfs-version.c
+++ b/src/detect-nfs-version.c
@@ -69,7 +69,7 @@ void DetectNfsVersionRegister (void)
     sigmatch_table[DETECT_AL_NFS_VERSION].Setup = DetectNfsVersionSetup;
     sigmatch_table[DETECT_AL_NFS_VERSION].Free = DetectNfsVersionFree;
     // unit tests were the same as DetectNfsProcedureRegisterTests
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "nfs_request", ALPROTO_NFS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
 
     g_nfs_request_buffer_id = DetectBufferTypeGetByName("nfs_request");
diff --git a/src/detect-parse.c b/src/detect-parse.c
index ba4a4db54cff..83c22aa8adac 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -109,14 +109,14 @@ void DetectFileRegisterFileProtocols(DetectFileHandlerTableElmt *reg)
         if (direction & SIG_FLAG_TOCLIENT) {
             DetectAppLayerMpmRegister(reg->name, SIG_FLAG_TOCLIENT, reg->priority, reg->PrefilterFn,
                     reg->GetData, al_protocols[i].al_proto, al_protocols[i].to_client_progress);
-            DetectAppLayerInspectEngineRegister2(reg->name, al_protocols[i].al_proto,
+            DetectAppLayerInspectEngineRegister(reg->name, al_protocols[i].al_proto,
                     SIG_FLAG_TOCLIENT, al_protocols[i].to_client_progress, reg->Callback,
                     reg->GetData);
         }
         if (direction & SIG_FLAG_TOSERVER) {
             DetectAppLayerMpmRegister(reg->name, SIG_FLAG_TOSERVER, reg->priority, reg->PrefilterFn,
                     reg->GetData, al_protocols[i].al_proto, al_protocols[i].to_server_progress);
-            DetectAppLayerInspectEngineRegister2(reg->name, al_protocols[i].al_proto,
+            DetectAppLayerInspectEngineRegister(reg->name, al_protocols[i].al_proto,
                     SIG_FLAG_TOSERVER, al_protocols[i].to_server_progress, reg->Callback,
                     reg->GetData);
         }
diff --git a/src/detect-quic-cyu-hash.c b/src/detect-quic-cyu-hash.c
index 520538feb8ff..309e722cfc0f 100644
--- a/src/detect-quic-cyu-hash.c
+++ b/src/detect-quic-cyu-hash.c
@@ -233,7 +233,7 @@ void DetectQuicCyuHashRegister(void)
     DetectAppLayerMpmRegister(
             BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterMpmQuicHashRegister, NULL, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 0, DetectEngineInspectQuicHash, NULL);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
diff --git a/src/detect-quic-cyu-string.c b/src/detect-quic-cyu-string.c
index 1dafaea0d09c..3cc846d6a4e9 100644
--- a/src/detect-quic-cyu-string.c
+++ b/src/detect-quic-cyu-string.c
@@ -186,7 +186,7 @@ void DetectQuicCyuStringRegister(void)
     DetectAppLayerMpmRegister(
             BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterMpmListIdRegister, NULL, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 0, DetectEngineInspectQuicString, NULL);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
diff --git a/src/detect-quic-sni.c b/src/detect-quic-sni.c
index 647308084087..4515baa6a7ec 100644
--- a/src/detect-quic-sni.c
+++ b/src/detect-quic-sni.c
@@ -83,7 +83,7 @@ void DetectQuicSniRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetSniData, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetSniData);
 
     quic_sni_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-quic-ua.c b/src/detect-quic-ua.c
index f101ec9577a6..4f4e9fd7d2e7 100644
--- a/src/detect-quic-ua.c
+++ b/src/detect-quic-ua.c
@@ -83,7 +83,7 @@ void DetectQuicUaRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetUaData, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetUaData);
 
     quic_ua_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-quic-version.c b/src/detect-quic-version.c
index ef4d3a602711..58257d143ba4 100644
--- a/src/detect-quic-version.c
+++ b/src/detect-quic-version.c
@@ -85,9 +85,9 @@ void DetectQuicVersionRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetVersionData, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetVersionData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetVersionData);
 
     quic_version_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-rfb-name.c b/src/detect-rfb-name.c
index 965532952bfb..222223a44999 100644
--- a/src/detect-rfb-name.c
+++ b/src/detect-rfb-name.c
@@ -96,8 +96,7 @@ void DetectRfbNameRegister(void)
     sigmatch_table[DETECT_AL_RFB_NAME].Setup = DetectRfbNameSetup;
     sigmatch_table[DETECT_AL_RFB_NAME].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_RFB,
-            SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_RFB, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 1, PrefilterGenericMpmRegister,
diff --git a/src/detect-rfb-secresult.c b/src/detect-rfb-secresult.c
index 403c16d08aa5..946886ca230a 100644
--- a/src/detect-rfb-secresult.c
+++ b/src/detect-rfb-secresult.c
@@ -67,7 +67,7 @@ void DetectRfbSecresultRegister (void)
 #endif
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("rfb.secresult", ALPROTO_RFB, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister("rfb.secresult", ALPROTO_RFB, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectGenericList, NULL);
 
     rfb_secresult_id = DetectBufferTypeGetByName("rfb.secresult");
diff --git a/src/detect-rfb-sectype.c b/src/detect-rfb-sectype.c
index d942a4503a49..c9afa8b46c0f 100644
--- a/src/detect-rfb-sectype.c
+++ b/src/detect-rfb-sectype.c
@@ -54,7 +54,7 @@ void DetectRfbSectypeRegister (void)
     sigmatch_table[DETECT_AL_RFB_SECTYPE].Setup = DetectRfbSectypeSetup;
     sigmatch_table[DETECT_AL_RFB_SECTYPE].Free = DetectRfbSectypeFree;
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "rfb.sectype", ALPROTO_RFB, SIG_FLAG_TOSERVER, 1, DetectEngineInspectGenericList, NULL);
 
     g_rfb_sectype_buffer_id = DetectBufferTypeGetByName("rfb.sectype");
diff --git a/src/detect-sip-method.c b/src/detect-sip-method.c
index 60160616f0da..d4ee89ad193b 100644
--- a/src/detect-sip-method.c
+++ b/src/detect-sip-method.c
@@ -134,8 +134,7 @@ void DetectSipMethodRegister(void)
     sigmatch_table[DETECT_AL_SIP_METHOD].Setup = DetectSipMethodSetup;
     sigmatch_table[DETECT_AL_SIP_METHOD].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-sip-protocol.c b/src/detect-sip-protocol.c
index 3feb6f6e24ad..6adf74452988 100644
--- a/src/detect-sip-protocol.c
+++ b/src/detect-sip-protocol.c
@@ -104,11 +104,9 @@ void DetectSipProtocolRegister(void)
             GetData, ALPROTO_SIP, 1);
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SIP, 1);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SIP, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, GetData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SIP, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
diff --git a/src/detect-sip-request-line.c b/src/detect-sip-request-line.c
index ac5e9276ef6d..5852f7fbe843 100644
--- a/src/detect-sip-request-line.c
+++ b/src/detect-sip-request-line.c
@@ -100,8 +100,7 @@ void DetectSipRequestLineRegister(void)
     sigmatch_table[DETECT_AL_SIP_REQUEST_LINE].Setup = DetectSipRequestLineSetup;
     sigmatch_table[DETECT_AL_SIP_REQUEST_LINE].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-sip-response-line.c b/src/detect-sip-response-line.c
index 9929eb3644ac..12be766dfeb3 100644
--- a/src/detect-sip-response-line.c
+++ b/src/detect-sip-response-line.c
@@ -100,8 +100,7 @@ void DetectSipResponseLineRegister(void)
     sigmatch_table[DETECT_AL_SIP_RESPONSE_LINE].Setup = DetectSipResponseLineSetup;
     sigmatch_table[DETECT_AL_SIP_RESPONSE_LINE].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-sip-stat-code.c b/src/detect-sip-stat-code.c
index eeb427dd1326..883872b169f3 100644
--- a/src/detect-sip-stat-code.c
+++ b/src/detect-sip-stat-code.c
@@ -103,8 +103,7 @@ void DetectSipStatCodeRegister (void)
     sigmatch_table[DETECT_AL_SIP_STAT_CODE].Setup = DetectSipStatCodeSetup;
     sigmatch_table[DETECT_AL_SIP_STAT_CODE].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 4, PrefilterGenericMpmRegister,
diff --git a/src/detect-sip-stat-msg.c b/src/detect-sip-stat-msg.c
index 583654803c3d..bda224b3e169 100644
--- a/src/detect-sip-stat-msg.c
+++ b/src/detect-sip-stat-msg.c
@@ -103,8 +103,7 @@ void DetectSipStatMsgRegister (void)
     sigmatch_table[DETECT_AL_SIP_STAT_MSG].Setup = DetectSipStatMsgSetup;
     sigmatch_table[DETECT_AL_SIP_STAT_MSG].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 3, PrefilterGenericMpmRegister,
diff --git a/src/detect-sip-uri.c b/src/detect-sip-uri.c
index 5c568e8c04a8..f71627e035e1 100644
--- a/src/detect-sip-uri.c
+++ b/src/detect-sip-uri.c
@@ -112,8 +112,7 @@ void DetectSipUriRegister(void)
     sigmatch_table[DETECT_AL_SIP_URI].Setup = DetectSipUriSetup;
     sigmatch_table[DETECT_AL_SIP_URI].flags |= SIGMATCH_NOOPT;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SIP,
-            SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SIP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-smb-ntlmssp.c b/src/detect-smb-ntlmssp.c
index 558488b5069a..aa53269309cf 100644
--- a/src/detect-smb-ntlmssp.c
+++ b/src/detect-smb-ntlmssp.c
@@ -84,7 +84,7 @@ void DetectSmbNtlmsspUserRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetNtlmsspUserData, ALPROTO_SMB, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetNtlmsspUserData);
 
     g_smb_nltmssp_user_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
@@ -142,7 +142,7 @@ void DetectSmbNtlmsspDomainRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetNtlmsspDomainData, ALPROTO_SMB, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetNtlmsspDomainData);
 
     g_smb_nltmssp_domain_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-smb-share.c b/src/detect-smb-share.c
index 7d90e5622d1c..018d8ceefd79 100644
--- a/src/detect-smb-share.c
+++ b/src/detect-smb-share.c
@@ -86,8 +86,7 @@ void DetectSmbNamedPipeRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetNamedPipeData, ALPROTO_SMB, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetNamedPipeData);
 
     g_smb_named_pipe_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
@@ -148,8 +147,7 @@ void DetectSmbShareRegister(void)
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetShareData, ALPROTO_SMB, 1);
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetShareData);
 
     g_smb_share_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-snmp-community.c b/src/detect-snmp-community.c
index 1205f2e1a3dc..f1dd740e3d53 100644
--- a/src/detect-snmp-community.c
+++ b/src/detect-snmp-community.c
@@ -62,13 +62,11 @@ void DetectSNMPCommunityRegister(void)
     sigmatch_table[DETECT_AL_SNMP_COMMUNITY].flags |= SIGMATCH_NOOPT|SIGMATCH_INFO_STICKY_BUFFER;
 
     /* register inspect engines */
-    DetectAppLayerInspectEngineRegister2("snmp.community",
-            ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("snmp.community", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
     DetectAppLayerMpmRegister("snmp.community", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
-    DetectAppLayerInspectEngineRegister2("snmp.community",
-            ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("snmp.community", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
     DetectAppLayerMpmRegister("snmp.community", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
diff --git a/src/detect-snmp-pdu_type.c b/src/detect-snmp-pdu_type.c
index 243d6c323be8..331abce7f355 100644
--- a/src/detect-snmp-pdu_type.c
+++ b/src/detect-snmp-pdu_type.c
@@ -68,10 +68,10 @@ void DetectSNMPPduTypeRegister(void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("snmp.pdu_type", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("snmp.pdu_type", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("snmp.pdu_type", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("snmp.pdu_type", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     g_snmp_pdu_type_buffer_id = DetectBufferTypeGetByName("snmp.pdu_type");
diff --git a/src/detect-snmp-usm.c b/src/detect-snmp-usm.c
index 153ba94d8519..fd1a814d164d 100644
--- a/src/detect-snmp-usm.c
+++ b/src/detect-snmp-usm.c
@@ -66,11 +66,11 @@ void DetectSNMPUsmRegister(void)
     sigmatch_table[DETECT_AL_SNMP_USM].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
     /* register inspect engines */
-    DetectAppLayerInspectEngineRegister2("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
     DetectAppLayerMpmRegister("snmp.usm", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
-    DetectAppLayerInspectEngineRegister2("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("snmp.usm", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
     DetectAppLayerMpmRegister("snmp.usm", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_SNMP, 0);
diff --git a/src/detect-snmp-version.c b/src/detect-snmp-version.c
index 64029659381e..cde70f251024 100644
--- a/src/detect-snmp-version.c
+++ b/src/detect-snmp-version.c
@@ -60,10 +60,10 @@ void DetectSNMPVersionRegister (void)
     sigmatch_table[DETECT_AL_SNMP_VERSION].RegisterTests = DetectSNMPVersionRegisterTests;
 #endif
 
-    DetectAppLayerInspectEngineRegister2("snmp.version", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("snmp.version", ALPROTO_SNMP, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("snmp.version", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("snmp.version", ALPROTO_SNMP, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectGenericList, NULL);
 
     g_snmp_version_buffer_id = DetectBufferTypeGetByName("snmp.version");
diff --git a/src/detect-ssh-hassh-server-string.c b/src/detect-ssh-hassh-server-string.c
index c38301de0d28..f62c72e79c79 100644
--- a/src/detect-ssh-hassh-server-string.c
+++ b/src/detect-ssh-hassh-server-string.c
@@ -131,9 +131,8 @@ void DetectSshHasshServerStringRegister(void)
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetSshData, ALPROTO_SSH, SshStateBannerDone);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
-            SIG_FLAG_TOCLIENT, SshStateBannerDone, 
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOCLIENT,
+            SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-ssh-hassh-server.c b/src/detect-ssh-hassh-server.c
index fe225bd2fcef..98f7d3dc2e2f 100644
--- a/src/detect-ssh-hassh-server.c
+++ b/src/detect-ssh-hassh-server.c
@@ -199,9 +199,8 @@ void DetectSshHasshServerRegister(void)
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetSshData, ALPROTO_SSH, SshStateBannerDone);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
-            SIG_FLAG_TOCLIENT, SshStateBannerDone, 
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOCLIENT,
+            SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
     g_ssh_hassh_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-ssh-hassh-string.c b/src/detect-ssh-hassh-string.c
index af98c21bf291..ad29b90ee764 100644
--- a/src/detect-ssh-hassh-string.c
+++ b/src/detect-ssh-hassh-string.c
@@ -131,9 +131,8 @@ void DetectSshHasshStringRegister(void)
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetSshData, ALPROTO_SSH, SshStateBannerDone);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, 
-            SIG_FLAG_TOSERVER, SshStateBannerDone, 
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+            SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-ssh-hassh.c b/src/detect-ssh-hassh.c
index 4704b95a658e..377aa9d2c433 100644
--- a/src/detect-ssh-hassh.c
+++ b/src/detect-ssh-hassh.c
@@ -201,7 +201,7 @@ void DetectSshHasshRegister(void)
 
     DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetSshData, ALPROTO_SSH, SshStateBannerDone),
-            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+            DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
                     SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-ssh-proto.c b/src/detect-ssh-proto.c
index e56f846c8416..19807511e757 100644
--- a/src/detect-ssh-proto.c
+++ b/src/detect-ssh-proto.c
@@ -106,11 +106,10 @@ void DetectSshProtocolRegister(void)
             DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
                     PrefilterGenericMpmRegister, GetSshData, ALPROTO_SSH, SshStateBannerDone),
 
-            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+            DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
                     SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SSH, SIG_FLAG_TOCLIENT, SshStateBannerDone,
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOCLIENT,
+            SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-ssh-software-version.c b/src/detect-ssh-software-version.c
index c2ba4ba888ef..60602a4e02cb 100644
--- a/src/detect-ssh-software-version.c
+++ b/src/detect-ssh-software-version.c
@@ -98,9 +98,9 @@ void DetectSshSoftwareVersionRegister(void)
 
     g_ssh_banner_list_id = DetectBufferTypeRegister("ssh_banner");
 
-    DetectAppLayerInspectEngineRegister2("ssh_banner", ALPROTO_SSH, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("ssh_banner", ALPROTO_SSH, SIG_FLAG_TOSERVER,
             SshStateBannerDone, DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2("ssh_banner", ALPROTO_SSH, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("ssh_banner", ALPROTO_SSH, SIG_FLAG_TOCLIENT,
             SshStateBannerDone, DetectEngineInspectGenericList, NULL);
 }
 
diff --git a/src/detect-ssh-software.c b/src/detect-ssh-software.c
index 2b0e3d47d1cc..0a8d5aab0d97 100644
--- a/src/detect-ssh-software.c
+++ b/src/detect-ssh-software.c
@@ -107,11 +107,10 @@ void DetectSshSoftwareRegister(void)
             DetectAppLayerMpmRegister(BUFFER_NAME, SIG_FLAG_TOCLIENT, 2,
                     PrefilterGenericMpmRegister, GetSshData, ALPROTO_SSH, SshStateBannerDone),
 
-            DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
+            DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOSERVER,
                     SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME,
-            ALPROTO_SSH, SIG_FLAG_TOCLIENT, SshStateBannerDone,
-            DetectEngineInspectBufferGeneric, GetSshData);
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_SSH, SIG_FLAG_TOCLIENT,
+            SshStateBannerDone, DetectEngineInspectBufferGeneric, GetSshData);
 
     DetectBufferTypeSetDescriptionByName(BUFFER_NAME, BUFFER_DESC);
 
diff --git a/src/detect-ssl-state.c b/src/detect-ssl-state.c
index fd60f045a4c3..888aeaef4d6e 100644
--- a/src/detect-ssl-state.c
+++ b/src/detect-ssl-state.c
@@ -89,9 +89,9 @@ void DetectSslStateRegister(void)
     DetectBufferTypeSetDescriptionByName("tls_generic",
             "generic ssl/tls inspection");
 
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "tls_generic", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
-    DetectAppLayerInspectEngineRegister2(
+    DetectAppLayerInspectEngineRegister(
             "tls_generic", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
 }
 
diff --git a/src/detect-template-rust-buffer.c b/src/detect-template-rust-buffer.c
index 3f016f4b164b..6f9ef9b8dd96 100644
--- a/src/detect-template-rust-buffer.c
+++ b/src/detect-template-rust-buffer.c
@@ -67,9 +67,9 @@ void DetectTemplateRustBufferRegister(void)
     sigmatch_table[DETECT_AL_TEMPLATE_BUFFER].flags |= SIGMATCH_NOOPT;
 
     /* register inspect engines */
-    DetectAppLayerInspectEngineRegister2("template_buffer", ALPROTO_TEMPLATE, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("template_buffer", ALPROTO_TEMPLATE, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectTemplateRustBuffer, NULL);
-    DetectAppLayerInspectEngineRegister2("template_buffer", ALPROTO_TEMPLATE, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("template_buffer", ALPROTO_TEMPLATE, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectTemplateRustBuffer, NULL);
 
     g_template_rust_id = DetectBufferTypeGetByName("template_buffer");
diff --git a/src/detect-tls-cert-fingerprint.c b/src/detect-tls-cert-fingerprint.c
index 354171113f04..9fec32151dd6 100644
--- a/src/detect-tls-cert-fingerprint.c
+++ b/src/detect-tls-cert-fingerprint.c
@@ -83,14 +83,13 @@ void DetectTlsFingerprintRegister(void)
     sigmatch_table[DETECT_AL_TLS_CERT_FINGERPRINT].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_CERT_FINGERPRINT].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_fingerprint", ALPROTO_TLS,
-            SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
-            DetectEngineInspectBufferGeneric, GetData);
+    DetectAppLayerInspectEngineRegister("tls.cert_fingerprint", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+            TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_fingerprint", SIG_FLAG_TOCLIENT, 2,
             PrefilterGenericMpmRegister, GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_fingerprint", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls.cert_fingerprint", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_fingerprint", SIG_FLAG_TOSERVER, 2,
diff --git a/src/detect-tls-cert-issuer.c b/src/detect-tls-cert-issuer.c
index fd8f1bcbc0ed..49bada4cdf6d 100644
--- a/src/detect-tls-cert-issuer.c
+++ b/src/detect-tls-cert-issuer.c
@@ -79,15 +79,14 @@ void DetectTlsIssuerRegister(void)
     sigmatch_table[DETECT_AL_TLS_CERT_ISSUER].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_CERT_ISSUER].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_issuer", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls.cert_issuer", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_issuer", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_issuer", ALPROTO_TLS,
-            SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
-            DetectEngineInspectBufferGeneric, GetData);
+    DetectAppLayerInspectEngineRegister("tls.cert_issuer", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+            TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_issuer", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
diff --git a/src/detect-tls-cert-serial.c b/src/detect-tls-cert-serial.c
index b1fd15d537e2..0ac7bfdd20cc 100644
--- a/src/detect-tls-cert-serial.c
+++ b/src/detect-tls-cert-serial.c
@@ -83,14 +83,13 @@ void DetectTlsSerialRegister(void)
     sigmatch_table[DETECT_AL_TLS_CERT_SERIAL].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_CERT_SERIAL].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_serial", ALPROTO_TLS,
-            SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
-            DetectEngineInspectBufferGeneric, GetData);
+    DetectAppLayerInspectEngineRegister("tls.cert_serial", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+            TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_serial", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_serial", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls.cert_serial", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_serial", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-tls-cert-subject.c b/src/detect-tls-cert-subject.c
index d4ceacfb1a95..e0dcde30a830 100644
--- a/src/detect-tls-cert-subject.c
+++ b/src/detect-tls-cert-subject.c
@@ -79,13 +79,13 @@ void DetectTlsSubjectRegister(void)
     sigmatch_table[DETECT_AL_TLS_CERT_SUBJECT].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_CERT_SUBJECT].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_subject", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetData, ALPROTO_TLS, TLS_STATE_CERT_READY);
 
-    DetectAppLayerInspectEngineRegister2("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("tls.cert_subject", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
             TLS_STATE_CERT_READY, DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("tls.cert_subject", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
diff --git a/src/detect-tls-cert-validity.c b/src/detect-tls-cert-validity.c
index 3720d287db5c..7ccf8df6740a 100644
--- a/src/detect-tls-cert-validity.c
+++ b/src/detect-tls-cert-validity.c
@@ -123,7 +123,7 @@ void DetectTlsValidityRegister (void)
 
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex);
 
-    DetectAppLayerInspectEngineRegister2("tls_validity", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("tls_validity", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
             TLS_STATE_CERT_READY, DetectEngineInspectGenericList, NULL);
 
     g_tls_validity_buffer_id = DetectBufferTypeGetByName("tls_validity");
diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index ad9d0fef0e5e..a082c345df4d 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -93,14 +93,13 @@ void DetectTlsCertsRegister(void)
     sigmatch_table[DETECT_AL_TLS_CERTS].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_CERTS].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.certs", ALPROTO_TLS,
-            SIG_FLAG_TOCLIENT, TLS_STATE_CERT_READY,
-            DetectEngineInspectTlsCerts, NULL);
+    DetectAppLayerInspectEngineRegister("tls.certs", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+            TLS_STATE_CERT_READY, DetectEngineInspectTlsCerts, NULL);
 
     DetectAppLayerMpmRegister("tls.certs", SIG_FLAG_TOCLIENT, 2, PrefilterMpmTlsCertsRegister, NULL,
             ALPROTO_TLS, TLS_STATE_CERT_READY);
 
-    DetectAppLayerInspectEngineRegister2("tls.certs", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls.certs", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectTlsCerts, NULL);
 
     DetectAppLayerMpmRegister("tls.certs", SIG_FLAG_TOSERVER, 2, PrefilterMpmTlsCertsRegister, NULL,
@@ -351,7 +350,7 @@ void DetectTlsCertChainLenRegister(void)
     sigmatch_table[KEYWORD_ID].Setup = DetectTLSCertChainLenSetup;
     sigmatch_table[KEYWORD_ID].Free = DetectTLSCertChainLenFree;
 
-    DetectAppLayerInspectEngineRegister2(BUFFER_NAME, ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister(BUFFER_NAME, ALPROTO_TLS, SIG_FLAG_TOCLIENT,
             TLS_STATE_CERT_READY, DetectEngineInspectGenericList, NULL);
 
     g_tls_cert_buffer_id = DetectBufferTypeGetByName(BUFFER_NAME);
diff --git a/src/detect-tls-ja3-hash.c b/src/detect-tls-ja3-hash.c
index 2b8b5ff8912b..0cfe18d66e65 100644
--- a/src/detect-tls-ja3-hash.c
+++ b/src/detect-tls-ja3-hash.c
@@ -80,7 +80,7 @@ void DetectTlsJa3HashRegister(void)
     sigmatch_table[DETECT_AL_TLS_JA3_HASH].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_JA3_HASH].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("ja3.hash", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("ja3.hash", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(
@@ -89,7 +89,7 @@ void DetectTlsJa3HashRegister(void)
     DetectAppLayerMpmRegister("ja3.hash", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetHash, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2("ja3.hash", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("ja3.hash", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, Ja3DetectGetHash);
 
     DetectBufferTypeSetDescriptionByName("ja3.hash", "TLS JA3 hash");
diff --git a/src/detect-tls-ja3-string.c b/src/detect-tls-ja3-string.c
index 920e6f4a163c..6c2fbc6ad975 100644
--- a/src/detect-tls-ja3-string.c
+++ b/src/detect-tls-ja3-string.c
@@ -76,7 +76,7 @@ void DetectTlsJa3StringRegister(void)
     sigmatch_table[DETECT_AL_TLS_JA3_STRING].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_JA3_STRING].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("ja3.string", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("ja3.string", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("ja3.string", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
@@ -85,7 +85,7 @@ void DetectTlsJa3StringRegister(void)
     DetectAppLayerMpmRegister("ja3.string", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetString, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2("ja3.string", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
+    DetectAppLayerInspectEngineRegister("ja3.string", ALPROTO_QUIC, SIG_FLAG_TOSERVER, 1,
             DetectEngineInspectBufferGeneric, Ja3DetectGetString);
 
     DetectBufferTypeSetDescriptionByName("ja3.string", "TLS JA3 string");
diff --git a/src/detect-tls-ja3s-hash.c b/src/detect-tls-ja3s-hash.c
index 9d7429b202f7..a1a334a4f16b 100644
--- a/src/detect-tls-ja3s-hash.c
+++ b/src/detect-tls-ja3s-hash.c
@@ -79,7 +79,7 @@ void DetectTlsJa3SHashRegister(void)
     sigmatch_table[DETECT_AL_TLS_JA3S_HASH].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_JA3S_HASH].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("ja3s.hash", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("ja3s.hash", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("ja3s.hash", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
@@ -88,7 +88,7 @@ void DetectTlsJa3SHashRegister(void)
     DetectAppLayerMpmRegister("ja3s.hash", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetHash, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2("ja3s.hash", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister("ja3s.hash", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, Ja3DetectGetHash);
 
     DetectBufferTypeSetDescriptionByName("ja3s.hash", "TLS JA3S hash");
diff --git a/src/detect-tls-ja3s-string.c b/src/detect-tls-ja3s-string.c
index 0c4f1ba262fc..32117df68442 100644
--- a/src/detect-tls-ja3s-string.c
+++ b/src/detect-tls-ja3s-string.c
@@ -76,7 +76,7 @@ void DetectTlsJa3SStringRegister(void)
     sigmatch_table[DETECT_AL_TLS_JA3S_STRING].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_JA3S_STRING].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("ja3s.string", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("ja3s.string", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister("ja3s.string", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
@@ -85,7 +85,7 @@ void DetectTlsJa3SStringRegister(void)
     DetectAppLayerMpmRegister("ja3s.string", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             Ja3DetectGetString, ALPROTO_QUIC, 1);
 
-    DetectAppLayerInspectEngineRegister2("ja3s.string", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
+    DetectAppLayerInspectEngineRegister("ja3s.string", ALPROTO_QUIC, SIG_FLAG_TOCLIENT, 1,
             DetectEngineInspectBufferGeneric, Ja3DetectGetString);
 
     DetectBufferTypeSetDescriptionByName("ja3s.string", "TLS JA3S string");
diff --git a/src/detect-tls-random.c b/src/detect-tls-random.c
index 6bce53a732f4..b8af73490a32 100644
--- a/src/detect-tls-random.c
+++ b/src/detect-tls-random.c
@@ -62,13 +62,13 @@ void DetectTlsRandomTimeRegister(void)
     sigmatch_table[DETECT_AL_TLS_RANDOM_TIME].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
     /* Register engine for Server random */
-    DetectAppLayerInspectEngineRegister2("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomTimeData);
     DetectAppLayerMpmRegister("tls.random_time", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRandomTimeData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
-    DetectAppLayerInspectEngineRegister2("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("tls.random_time", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomTimeData);
     DetectAppLayerMpmRegister("tls.random_time", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetRandomTimeData, ALPROTO_TLS, 0);
@@ -90,13 +90,13 @@ void DetectTlsRandomBytesRegister(void)
             SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
     /* Register engine for Server random */
-    DetectAppLayerInspectEngineRegister2("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomBytesData);
     DetectAppLayerMpmRegister("tls.random_bytes", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRandomBytesData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
-    DetectAppLayerInspectEngineRegister2("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("tls.random_bytes", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomBytesData);
     DetectAppLayerMpmRegister("tls.random_bytes", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetRandomBytesData, ALPROTO_TLS, 0);
@@ -122,13 +122,13 @@ void DetectTlsRandomRegister(void)
     sigmatch_table[DETECT_AL_TLS_RANDOM].flags |= SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
 
     /* Register engine for Server random */
-    DetectAppLayerInspectEngineRegister2("tls.random", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("tls.random", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetRandomData);
     DetectAppLayerMpmRegister("tls.random", SIG_FLAG_TOSERVER, 2, PrefilterGenericMpmRegister,
             GetRandomData, ALPROTO_TLS, 0);
 
     /* Register engine for Client random */
-    DetectAppLayerInspectEngineRegister2("tls.random", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
+    DetectAppLayerInspectEngineRegister("tls.random", ALPROTO_TLS, SIG_FLAG_TOCLIENT, 0,
             DetectEngineInspectBufferGeneric, GetRandomData);
     DetectAppLayerMpmRegister("tls.random", SIG_FLAG_TOCLIENT, 2, PrefilterGenericMpmRegister,
             GetRandomData, ALPROTO_TLS, 0);
diff --git a/src/detect-tls-sni.c b/src/detect-tls-sni.c
index 702d16081733..ce8a068a4717 100644
--- a/src/detect-tls-sni.c
+++ b/src/detect-tls-sni.c
@@ -73,7 +73,7 @@ void DetectTlsSniRegister(void)
     sigmatch_table[DETECT_AL_TLS_SNI].flags |= SIGMATCH_NOOPT;
     sigmatch_table[DETECT_AL_TLS_SNI].flags |= SIGMATCH_INFO_STICKY_BUFFER;
 
-    DetectAppLayerInspectEngineRegister2("tls.sni", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
+    DetectAppLayerInspectEngineRegister("tls.sni", ALPROTO_TLS, SIG_FLAG_TOSERVER, 0,
             DetectEngineInspectBufferGeneric, GetData);
 
     DetectAppLayerMpmRegister(
diff --git a/src/detect-tls.c b/src/detect-tls.c
index 8a9c98fac795..da530f6abf39 100644
--- a/src/detect-tls.c
+++ b/src/detect-tls.c
@@ -141,10 +141,10 @@ void DetectTlsRegister (void)
     g_tls_cert_list_id = DetectBufferTypeRegister("tls_cert");
     g_tls_cert_fingerprint_list_id = DetectBufferTypeRegister("tls.cert_fingerprint");
 
-    DetectAppLayerInspectEngineRegister2("tls_cert", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
+    DetectAppLayerInspectEngineRegister("tls_cert", ALPROTO_TLS, SIG_FLAG_TOCLIENT,
             TLS_STATE_CERT_READY, DetectEngineInspectGenericList, NULL);
 
-    DetectAppLayerInspectEngineRegister2("tls_cert", ALPROTO_TLS, SIG_FLAG_TOSERVER,
+    DetectAppLayerInspectEngineRegister("tls_cert", ALPROTO_TLS, SIG_FLAG_TOSERVER,
             TLS_STATE_CERT_READY, DetectEngineInspectGenericList, NULL);
 }
 

From 66ff23f9bfdfaf45b6fc069d7cb648e4dc30c6e8 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Sun, 12 Nov 2023 14:15:11 +0100
Subject: [PATCH 270/462] detect: rename InspectEngineFuncPtr2 to
 InspectEngineFuncPtr

Version 1 of the API no longer exists.
---
 src/detect-engine.c | 15 ++++++---------
 src/detect-engine.h |  2 +-
 src/detect-parse.h  |  2 +-
 src/detect.h        |  4 ++--
 4 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/src/detect-engine.c b/src/detect-engine.c
index 22166127f44f..3e1ce93cf671 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -214,7 +214,7 @@ void DetectFrameInspectEngineRegister(const char *name, int dir,
  *
  *  \note errors are fatal */
 void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uint32_t dir,
-        int progress, InspectEngineFuncPtr2 Callback2, InspectionBufferGetDataPtr GetData)
+        int progress, InspectEngineFuncPtr Callback, InspectionBufferGetDataPtr GetData)
 {
     BUG_ON(progress >= 48);
 
@@ -225,15 +225,12 @@ void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uin
     }
     SCLogDebug("name %s id %d", name, sm_list);
 
-    if ((alproto >= ALPROTO_FAILED) ||
-        (!(dir == SIG_FLAG_TOSERVER || dir == SIG_FLAG_TOCLIENT)) ||
-        (sm_list < DETECT_SM_LIST_MATCH) || (sm_list >= SHRT_MAX) ||
-        (progress < 0 || progress >= SHRT_MAX) ||
-        (Callback2 == NULL))
-    {
+    if ((alproto >= ALPROTO_FAILED) || (!(dir == SIG_FLAG_TOSERVER || dir == SIG_FLAG_TOCLIENT)) ||
+            (sm_list < DETECT_SM_LIST_MATCH) || (sm_list >= SHRT_MAX) ||
+            (progress < 0 || progress >= SHRT_MAX) || (Callback == NULL)) {
         SCLogError("Invalid arguments");
         BUG_ON(1);
-    } else if (Callback2 == DetectEngineInspectBufferGeneric && GetData == NULL) {
+    } else if (Callback == DetectEngineInspectBufferGeneric && GetData == NULL) {
         SCLogError("Invalid arguments: must register "
                    "GetData with DetectEngineInspectBufferGeneric");
         BUG_ON(1);
@@ -256,7 +253,7 @@ void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uin
     new_engine->sm_list = (uint16_t)sm_list;
     new_engine->sm_list_base = (uint16_t)sm_list;
     new_engine->progress = (int16_t)progress;
-    new_engine->v2.Callback = Callback2;
+    new_engine->v2.Callback = Callback;
     new_engine->v2.GetData = GetData;
 
     if (g_app_inspect_engines == NULL) {
diff --git a/src/detect-engine.h b/src/detect-engine.h
index a5bfc329a94e..c0b694eb017f 100644
--- a/src/detect-engine.h
+++ b/src/detect-engine.h
@@ -162,7 +162,7 @@ int DetectEngineInspectPktBufferGeneric(
  * \param Callback The engine callback.
  */
 void DetectAppLayerInspectEngineRegister(const char *name, AppProto alproto, uint32_t dir,
-        int progress, InspectEngineFuncPtr2 Callback2, InspectionBufferGetDataPtr GetData);
+        int progress, InspectEngineFuncPtr Callback2, InspectionBufferGetDataPtr GetData);
 
 void DetectPktInspectEngineRegister(const char *name,
         InspectionBufferGetPktDataPtr GetPktData,
diff --git a/src/detect-parse.h b/src/detect-parse.h
index 2eecd286f631..990180141058 100644
--- a/src/detect-parse.h
+++ b/src/detect-parse.h
@@ -33,7 +33,7 @@ typedef struct DetectFileHandlerTableElmt_ {
     const char *name;
     int priority;
     PrefilterRegisterFunc PrefilterFn;
-    InspectEngineFuncPtr2 Callback;
+    InspectEngineFuncPtr Callback;
     InspectionBufferGetDataPtr GetData;
     int al_protocols[MAX_DETECT_ALPROTO_CNT];
     int tx_progress;
diff --git a/src/detect.h b/src/detect.h
index 36e4e2ba3b25..8278290a2992 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -413,7 +413,7 @@ typedef InspectionBuffer *(*InspectionBufferGetDataPtr)(
         void *txv, const int list_id);
 struct DetectEngineAppInspectionEngine_;
 
-typedef uint8_t (*InspectEngineFuncPtr2)(struct DetectEngineCtx_ *de_ctx,
+typedef uint8_t (*InspectEngineFuncPtr)(struct DetectEngineCtx_ *de_ctx,
         struct DetectEngineThreadCtx_ *det_ctx,
         const struct DetectEngineAppInspectionEngine_ *engine, const struct Signature_ *s, Flow *f,
         uint8_t flags, void *alstate, void *txv, uint64_t tx_id);
@@ -430,7 +430,7 @@ typedef struct DetectEngineAppInspectionEngine_ {
 
     struct {
         InspectionBufferGetDataPtr GetData;
-        InspectEngineFuncPtr2 Callback;
+        InspectEngineFuncPtr Callback;
         /** pointer to the transforms in the 'DetectBuffer entry for this list */
         const DetectEngineTransforms *transforms;
     } v2;

From 4620776a30b9c87d0d33ac9fa57a3418be87bba7 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 16 Nov 2023 09:43:33 -0600
Subject: [PATCH 271/462] rustfmt: replace deprecated fn_args_layout with
 fn_params_layout

---
 rust/rustfmt.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rust/rustfmt.toml b/rust/rustfmt.toml
index 848158b48e0f..064b795a873b 100644
--- a/rust/rustfmt.toml
+++ b/rust/rustfmt.toml
@@ -1,4 +1,4 @@
 # Rust format configuration file. If empty, then this is a message that
 # we expect the default formatting rules to be used.
 
-fn_args_layout = "compressed"
+fn_params_layout = "compressed"

From e2d7a7f8770b4ef8856d8a8eae30e4351feb4242 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 16 Nov 2023 09:44:07 -0600
Subject: [PATCH 272/462] dns: rustfmt with latest stable

---
 rust/src/dns/detect.rs | 64 ++++++++++++++++++------------------------
 rust/src/dns/dns.rs    |  6 ++--
 rust/src/dns/log.rs    | 15 ++++++----
 3 files changed, 41 insertions(+), 44 deletions(-)

diff --git a/rust/src/dns/detect.rs b/rust/src/dns/detect.rs
index 268a409eac8d..5d9d945be0ce 100644
--- a/rust/src/dns/detect.rs
+++ b/rust/src/dns/detect.rs
@@ -156,44 +156,36 @@ mod test {
 
     #[test]
     fn test_match_opcode() {
-        assert!(
-            match_opcode(
-                &DetectDnsOpcode {
-                    negate: false,
-                    opcode: 0,
-                },
-                0b0000_0000_0000_0000,
-            )
-        );
+        assert!(match_opcode(
+            &DetectDnsOpcode {
+                negate: false,
+                opcode: 0,
+            },
+            0b0000_0000_0000_0000,
+        ));
 
-        assert!(
-            !match_opcode(
-                &DetectDnsOpcode {
-                    negate: true,
-                    opcode: 0,
-                },
-                0b0000_0000_0000_0000,
-            )
-        );
+        assert!(!match_opcode(
+            &DetectDnsOpcode {
+                negate: true,
+                opcode: 0,
+            },
+            0b0000_0000_0000_0000,
+        ));
 
-        assert!(
-            match_opcode(
-                &DetectDnsOpcode {
-                    negate: false,
-                    opcode: 4,
-                },
-                0b0010_0000_0000_0000,
-            )
-        );
+        assert!(match_opcode(
+            &DetectDnsOpcode {
+                negate: false,
+                opcode: 4,
+            },
+            0b0010_0000_0000_0000,
+        ));
 
-        assert!(
-            !match_opcode(
-                &DetectDnsOpcode {
-                    negate: true,
-                    opcode: 4,
-                },
-                0b0010_0000_0000_0000,
-            )
-        );
+        assert!(!match_opcode(
+            &DetectDnsOpcode {
+                negate: true,
+                opcode: 4,
+            },
+            0b0010_0000_0000_0000,
+        ));
     }
 }
diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index 382c76ae59b5..c93547c15126 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -250,10 +250,10 @@ impl Transaction for DNSTransaction {
 
 impl DNSTransaction {
     pub fn new(direction: Direction) -> Self {
-	Self {
-	    tx_data: AppLayerTxData::for_direction(direction),
+        Self {
+            tx_data: AppLayerTxData::for_direction(direction),
             ..Default::default()
-	}
+        }
     }
 
     /// Get the DNS transactions ID (not the internal tracking ID).
diff --git a/rust/src/dns/log.rs b/rust/src/dns/log.rs
index 5212b1a0da7c..1bece89a5ae6 100644
--- a/rust/src/dns/log.rs
+++ b/rust/src/dns/log.rs
@@ -524,7 +524,8 @@ fn dns_log_json_answer(
                 match &answer.data {
                     DNSRData::A(addr) | DNSRData::AAAA(addr) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
+                            answer_types
+                                .insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_string(&dns_print_addr(addr))?;
@@ -537,7 +538,8 @@ fn dns_log_json_answer(
                     | DNSRData::NULL(bytes)
                     | DNSRData::PTR(bytes) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
+                            answer_types
+                                .insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_string_from_bytes(bytes)?;
@@ -545,7 +547,8 @@ fn dns_log_json_answer(
                     }
                     DNSRData::SOA(soa) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
+                            answer_types
+                                .insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_soa(soa)?)?;
@@ -553,7 +556,8 @@ fn dns_log_json_answer(
                     }
                     DNSRData::SSHFP(sshfp) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
+                            answer_types
+                                .insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_sshfp(sshfp)?)?;
@@ -561,7 +565,8 @@ fn dns_log_json_answer(
                     }
                     DNSRData::SRV(srv) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
+                            answer_types
+                                .insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_srv(srv)?)?;

From 9464d0b14a8d1b39652210626d6a8852a7474372 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 15 Nov 2023 15:58:36 -0600
Subject: [PATCH 273/462] dns: consolidate DNSRequest and DNSResponse to
 DNSMessage

DNS request and response messages follow the same format so there is
no reason not to use the same data structure for each. While its
unlikely to see fields like answers in a request, the message format
does not disallow them, so it might be interesting data to have the
ability to log.
---
 rust/src/dns/dns.rs    | 27 ++++---------
 rust/src/dns/log.rs    |  2 +-
 rust/src/dns/parser.rs | 88 ++++++++++++++++++------------------------
 3 files changed, 46 insertions(+), 71 deletions(-)

diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index c93547c15126..aa0042f0b259 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -221,13 +221,7 @@ pub struct DNSAnswerEntry {
 }
 
 #[derive(Debug)]
-pub struct DNSRequest {
-    pub header: DNSHeader,
-    pub queries: Vec<DNSQueryEntry>,
-}
-
-#[derive(Debug)]
-pub struct DNSResponse {
+pub struct DNSMessage {
     pub header: DNSHeader,
     pub queries: Vec<DNSQueryEntry>,
     pub answers: Vec<DNSAnswerEntry>,
@@ -237,8 +231,8 @@ pub struct DNSResponse {
 #[derive(Debug, Default)]
 pub struct DNSTransaction {
     pub id: u64,
-    pub request: Option<DNSRequest>,
-    pub response: Option<DNSResponse>,
+    pub request: Option<DNSMessage>,
+    pub response: Option<DNSMessage>,
     pub tx_data: AppLayerTxData,
 }
 
@@ -402,7 +396,7 @@ impl DNSState {
             return !is_tcp;
         };
 
-        match parser::dns_parse_request_body(body, input, header) {
+        match parser::dns_parse_body(body, input, header) {
             Ok((_, request)) => {
                 if request.header.flags & 0x8000 != 0 {
                     SCLogDebug!("DNS message is not a request");
@@ -474,7 +468,7 @@ impl DNSState {
             return !is_tcp;
         };
 
-        match parser::dns_parse_response_body(body, input, header) {
+        match parser::dns_parse_body(body, input, header) {
             Ok((_, response)) => {
                 SCLogDebug!("Response header flags: {}", response.header.flags);
 
@@ -702,14 +696,9 @@ fn probe(input: &[u8], dlen: usize) -> (bool, bool, bool) {
         }
     }
 
-    match parser::dns_parse_request(input) {
-        Ok((_, request)) => {
-            return probe_header_validity(&request.header, dlen);
-        }
-        Err(Err::Incomplete(_)) => match parser::dns_parse_header(input) {
-            Ok((_, header)) => {
-                return probe_header_validity(&header, dlen);
-            }
+    match parser::dns_parse_header(input) {
+        Ok((body, header)) => match parser::dns_parse_body(body, input, header) {
+            Ok((_, request)) => probe_header_validity(&request.header, dlen),
             Err(Err::Incomplete(_)) => (false, false, true),
             Err(_) => (false, false, false),
         },
diff --git a/rust/src/dns/log.rs b/rust/src/dns/log.rs
index 1bece89a5ae6..4c0d4fc065b4 100644
--- a/rust/src/dns/log.rs
+++ b/rust/src/dns/log.rs
@@ -476,7 +476,7 @@ fn dns_log_json_answer_detail(answer: &DNSAnswerEntry) -> Result<JsonBuilder, Js
 }
 
 fn dns_log_json_answer(
-    js: &mut JsonBuilder, response: &DNSResponse, flags: u64,
+    js: &mut JsonBuilder, response: &DNSMessage, flags: u64,
 ) -> Result<(), JsonError> {
     let header = &response.header;
 
diff --git a/rust/src/dns/parser.rs b/rust/src/dns/parser.rs
index a1d97a53fd02..f7f9fd0d6e8c 100644
--- a/rust/src/dns/parser.rs
+++ b/rust/src/dns/parser.rs
@@ -24,27 +24,6 @@ use nom7::multi::{count, length_data, many_m_n};
 use nom7::number::streaming::{be_u16, be_u32, be_u8};
 use nom7::{error_position, Err, IResult};
 
-// Parse a DNS header.
-pub fn dns_parse_header(i: &[u8]) -> IResult<&[u8], DNSHeader> {
-    let (i, tx_id) = be_u16(i)?;
-    let (i, flags) = be_u16(i)?;
-    let (i, questions) = be_u16(i)?;
-    let (i, answer_rr) = be_u16(i)?;
-    let (i, authority_rr) = be_u16(i)?;
-    let (i, additional_rr) = be_u16(i)?;
-    Ok((
-        i,
-        DNSHeader {
-            tx_id,
-            flags,
-            questions,
-            answer_rr,
-            authority_rr,
-            additional_rr,
-        },
-    ))
-}
-
 /// Parse a DNS name.
 ///
 /// Parameters:
@@ -191,23 +170,6 @@ fn dns_parse_answer<'a>(
     return Ok((input, answers));
 }
 
-pub fn dns_parse_response_body<'a>(
-    i: &'a [u8], message: &'a [u8], header: DNSHeader,
-) -> IResult<&'a [u8], DNSResponse> {
-    let (i, queries) = count(|b| dns_parse_query(b, message), header.questions as usize)(i)?;
-    let (i, answers) = dns_parse_answer(i, message, header.answer_rr as usize)?;
-    let (i, authorities) = dns_parse_answer(i, message, header.authority_rr as usize)?;
-    Ok((
-        i,
-        DNSResponse {
-            header,
-            queries,
-            answers,
-            authorities,
-        },
-    ))
-}
-
 /// Parse a single DNS query.
 ///
 /// Arguments are suitable for using with call!:
@@ -343,19 +305,42 @@ pub fn dns_parse_rdata<'a>(
     }
 }
 
-/// Parse a DNS request.
-pub fn dns_parse_request(input: &[u8]) -> IResult<&[u8], DNSRequest> {
-    let i = input;
-    let (i, header) = dns_parse_header(i)?;
-    dns_parse_request_body(i, input, header)
+// Parse a DNS header.
+pub fn dns_parse_header(i: &[u8]) -> IResult<&[u8], DNSHeader> {
+    let (i, tx_id) = be_u16(i)?;
+    let (i, flags) = be_u16(i)?;
+    let (i, questions) = be_u16(i)?;
+    let (i, answer_rr) = be_u16(i)?;
+    let (i, authority_rr) = be_u16(i)?;
+    let (i, additional_rr) = be_u16(i)?;
+    Ok((
+        i,
+        DNSHeader {
+            tx_id,
+            flags,
+            questions,
+            answer_rr,
+            authority_rr,
+            additional_rr,
+        },
+    ))
 }
 
-pub fn dns_parse_request_body<'a>(
-    input: &'a [u8], message: &'a [u8], header: DNSHeader,
-) -> IResult<&'a [u8], DNSRequest> {
-    let i = input;
+pub fn dns_parse_body<'a>(
+    i: &'a [u8], message: &'a [u8], header: DNSHeader,
+) -> IResult<&'a [u8], DNSMessage> {
     let (i, queries) = count(|b| dns_parse_query(b, message), header.questions as usize)(i)?;
-    Ok((i, DNSRequest { header, queries }))
+    let (i, answers) = dns_parse_answer(i, message, header.answer_rr as usize)?;
+    let (i, authorities) = dns_parse_answer(i, message, header.authority_rr as usize)?;
+    Ok((
+        i,
+        DNSMessage {
+            header,
+            queries,
+            answers,
+            authorities,
+        },
+    ))
 }
 
 #[cfg(test)]
@@ -490,7 +475,8 @@ mod tests {
             0x00, 0x00, 0x00, /* ... */
         ];
 
-        let res = dns_parse_request(pkt);
+        let (body, header) = dns_parse_header(pkt).unwrap();
+        let res = dns_parse_body(body, pkt, header);
         match res {
             Ok((rem, request)) => {
                 // For now we have some remainder data as there is an
@@ -523,10 +509,10 @@ mod tests {
     }
 
     /// Parse a DNS response.
-    fn dns_parse_response(message: &[u8]) -> IResult<&[u8], DNSResponse> {
+    fn dns_parse_response(message: &[u8]) -> IResult<&[u8], DNSMessage> {
         let i = message;
         let (i, header) = dns_parse_header(i)?;
-        dns_parse_response_body(i, message, header)
+        dns_parse_body(i, message, header)
     }
 
     #[test]

From 5f99abb0cb096c3cf6e829fa43a3dd150012c6a4 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 23 Oct 2023 15:28:40 -0600
Subject: [PATCH 274/462] dns: add dns.answer.name keyword

This sticky buffer will allow content matching on the answer names.
While ansers typically only occur in DNS responses, we allow the buffer
to be used in request context as well as the request message format
allows it.

Feature: #6496
---
 rust/src/dns/dns.rs          |  25 +++++
 src/Makefile.am              |   2 +
 src/detect-dns-answer-name.c | 172 +++++++++++++++++++++++++++++++++++
 src/detect-dns-answer-name.h |  23 +++++
 src/detect-engine-register.c |   2 +
 src/detect-engine-register.h |   1 +
 6 files changed, 225 insertions(+)
 create mode 100644 src/detect-dns-answer-name.c
 create mode 100644 src/detect-dns-answer-name.h

diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index aa0042f0b259..8933c680db56 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -870,6 +870,31 @@ pub unsafe extern "C" fn rs_dns_tx_get_query_name(
     return 0;
 }
 
+/// Get the DNS response answer name and index i.
+#[no_mangle]
+pub unsafe extern "C" fn SCDnsTxGetAnswerName(
+    tx: &mut DNSTransaction, to_client: bool, i: u32, buf: *mut *const u8, len: *mut u32,
+) -> bool {
+    let answers = if to_client {
+        tx.response.as_ref().map(|response| &response.answers)
+    } else {
+        tx.request.as_ref().map(|request| &request.answers)
+    };
+    let index = i as usize;
+
+    if let Some(answers) = answers {
+        if let Some(answer) = answers.get(index) {
+            if !answer.name.is_empty() {
+                *buf = answer.name.as_ptr();
+                *len = answer.name.len() as u32;
+                return true;
+            }
+        }
+    }
+
+    false
+}
+
 /// Get the DNS transaction ID of a transaction.
 //
 /// extern uint16_t rs_dns_tx_get_tx_id(RSDNSTransaction *);
diff --git a/src/Makefile.am b/src/Makefile.am
index 4695c2d35f51..25bec98e2e7b 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -121,6 +121,7 @@ noinst_HEADERS = \
 	detect-detection-filter.h \
 	detect-distance.h \
 	detect-dnp3.h \
+	detect-dns-answer-name.h \
 	detect-dns-opcode.h \
 	detect-dns-query.h \
 	detect-dsize.h \
@@ -736,6 +737,7 @@ libsuricata_c_a_SOURCES = \
 	detect-detection-filter.c \
 	detect-distance.c \
 	detect-dnp3.c \
+	detect-dns-answer-name.c \
 	detect-dns-opcode.c \
 	detect-dns-query.c \
 	detect-dsize.c \
diff --git a/src/detect-dns-answer-name.c b/src/detect-dns-answer-name.c
new file mode 100644
index 000000000000..bc64f55fbf49
--- /dev/null
+++ b/src/detect-dns-answer-name.c
@@ -0,0 +1,172 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * Detect keyword for DNS answer name: dns.answer.name
+ */
+
+#include "detect.h"
+#include "detect-parse.h"
+#include "detect-engine.h"
+#include "detect-engine-prefilter.h"
+#include "detect-engine-content-inspection.h"
+#include "detect-dns-answer-name.h"
+#include "util-profiling.h"
+#include "rust.h"
+
+typedef struct PrefilterMpm {
+    int list_id;
+    const MpmCtx *mpm_ctx;
+    const DetectEngineTransforms *transforms;
+} PrefilterMpm;
+
+static int detect_buffer_id = 0;
+
+static int DetectSetup(DetectEngineCtx *de_ctx, Signature *s, const char *str)
+{
+    if (DetectBufferSetActiveList(de_ctx, s, detect_buffer_id) < 0) {
+        return -1;
+    }
+    if (DetectSignatureSetAppProto(s, ALPROTO_DNS) < 0) {
+        return -1;
+    }
+
+    return 0;
+}
+
+static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx, uint8_t flags,
+        const DetectEngineTransforms *transforms, void *txv, uint32_t index, int list_id)
+{
+    InspectionBuffer *buffer = InspectionBufferMultipleForListGet(det_ctx, list_id, index);
+    if (buffer == NULL) {
+        return NULL;
+    }
+    if (buffer->initialized) {
+        return buffer;
+    }
+
+    bool to_client = (flags & STREAM_TOSERVER) == 0;
+    const uint8_t *data = NULL;
+    uint32_t data_len = 0;
+
+    if (!SCDnsTxGetAnswerName(txv, to_client, index, &data, &data_len)) {
+        InspectionBufferSetupMultiEmpty(buffer);
+        return NULL;
+    }
+    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    buffer->flags = DETECT_CI_FLAGS_SINGLE;
+    return buffer;
+}
+
+static uint8_t DetectEngineInspectCb(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const struct DetectEngineAppInspectionEngine_ *engine, const Signature *s, Flow *f,
+        uint8_t flags, void *alstate, void *txv, uint64_t tx_id)
+{
+    const DetectEngineTransforms *transforms = NULL;
+    if (!engine->mpm) {
+        transforms = engine->v2.transforms;
+    }
+
+    for (uint32_t i = 0;; i++) {
+        InspectionBuffer *buffer = GetBuffer(det_ctx, flags, transforms, txv, i, engine->sm_list);
+        if (buffer == NULL || buffer->inspect == NULL) {
+            break;
+        }
+
+        const bool match = DetectEngineContentInspectionBuffer(de_ctx, det_ctx, s, engine->smd,
+                NULL, f, buffer, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
+            return DETECT_ENGINE_INSPECT_SIG_MATCH;
+        }
+    }
+
+    return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
+}
+
+static void PrefilterTx(DetectEngineThreadCtx *det_ctx, const void *pectx, Packet *p, Flow *f,
+        void *txv, const uint64_t idx, const AppLayerTxData *_txd, const uint8_t flags)
+{
+    SCEnter();
+
+    const PrefilterMpm *ctx = (const PrefilterMpm *)pectx;
+    const MpmCtx *mpm_ctx = ctx->mpm_ctx;
+    const int list_id = ctx->list_id;
+
+    for (uint32_t i = 0;; i++) {
+        InspectionBuffer *buffer = GetBuffer(det_ctx, flags, ctx->transforms, txv, i, list_id);
+        if (buffer == NULL) {
+            break;
+        }
+
+        if (buffer->inspect_len >= mpm_ctx->minlen) {
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+            PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
+        }
+    }
+}
+
+static void PrefilterMpmFree(void *ptr)
+{
+    SCFree(ptr);
+}
+
+static int PrefilterMpmRegister(DetectEngineCtx *de_ctx, SigGroupHead *sgh, MpmCtx *mpm_ctx,
+        const DetectBufferMpmRegistry *mpm_reg, int list_id)
+{
+    PrefilterMpm *pectx = SCCalloc(1, sizeof(*pectx));
+    if (pectx == NULL) {
+        return -1;
+    }
+    pectx->list_id = list_id;
+    pectx->mpm_ctx = mpm_ctx;
+    pectx->transforms = &mpm_reg->transforms;
+
+    return PrefilterAppendTxEngine(de_ctx, sgh, PrefilterTx, mpm_reg->app_v2.alproto,
+            mpm_reg->app_v2.tx_min_progress, pectx, PrefilterMpmFree, mpm_reg->pname);
+}
+
+void DetectDnsAnswerNameRegister(void)
+{
+    static const char *keyword = "dns.answer.name";
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].name = keyword;
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].desc = "DNS answer name sticky buffer";
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].url = "/rules/dns-keywords.html#dns-answer-name";
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].Setup = DetectSetup;
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].flags |= SIGMATCH_NOOPT;
+    sigmatch_table[DETECT_AL_DNS_ANSWER_NAME].flags |= SIGMATCH_INFO_STICKY_BUFFER;
+
+    /* Register in the TO_SERVER direction, even though this is not
+       normal, it could be provided as part of a request. */
+    DetectAppLayerInspectEngineRegister(
+            keyword, ALPROTO_DNS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectCb, NULL);
+    DetectAppLayerMpmRegister(
+            keyword, SIG_FLAG_TOSERVER, 2, PrefilterMpmRegister, NULL, ALPROTO_DNS, 1);
+
+    /* Register in the TO_CLIENT direction. */
+    DetectAppLayerInspectEngineRegister(
+            keyword, ALPROTO_DNS, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectCb, NULL);
+    DetectAppLayerMpmRegister(
+            keyword, SIG_FLAG_TOCLIENT, 2, PrefilterMpmRegister, NULL, ALPROTO_DNS, 1);
+
+    DetectBufferTypeSetDescriptionByName(keyword, "dns answer name");
+    DetectBufferTypeSupportsMultiInstance(keyword);
+
+    detect_buffer_id = DetectBufferTypeGetByName(keyword);
+}
diff --git a/src/detect-dns-answer-name.h b/src/detect-dns-answer-name.h
new file mode 100644
index 000000000000..4f84b4894c16
--- /dev/null
+++ b/src/detect-dns-answer-name.h
@@ -0,0 +1,23 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#ifndef __DETECT_DNS_ANSWER_NAME_H__
+#define __DETECT_DNS_ANSWER_NAME_H__
+
+void DetectDnsAnswerNameRegister(void);
+
+#endif /* __DETECT_DNS_ANSWER_NAME_H__ */
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index b27f5a511f9c..258bbef3f543 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -49,6 +49,7 @@
 #include "detect-engine-dcepayload.h"
 #include "detect-dns-opcode.h"
 #include "detect-dns-query.h"
+#include "detect-dns-answer-name.h"
 #include "detect-tls-sni.h"
 #include "detect-tls-certs.h"
 #include "detect-tls-cert-fingerprint.h"
@@ -517,6 +518,7 @@ void SigTableSetup(void)
 
     DetectDnsQueryRegister();
     DetectDnsOpcodeRegister();
+    DetectDnsAnswerNameRegister();
     DetectModbusRegister();
     DetectCipServiceRegister();
     DetectEnipCommandRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 273aa10d7c9b..0a529954e953 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -227,6 +227,7 @@ enum DetectKeywordId {
 
     DETECT_AL_DNS_QUERY,
     DETECT_AL_DNS_OPCODE,
+    DETECT_AL_DNS_ANSWER_NAME,
     DETECT_AL_TLS_SNI,
     DETECT_AL_TLS_CERTS,
     DETECT_AL_TLS_CERT_ISSUER,

From 482325e28b39ac35b364698eb9d6b512038ed777 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 14 Nov 2023 17:01:49 -0600
Subject: [PATCH 275/462] dns: add dns.query.name sticky buffer

This buffer is much like dns.query_name but allows for detection in both
directions.

Feature: #6497
---
 rust/src/dns/dns.rs          |  25 +++++
 src/Makefile.am              |   2 +
 src/detect-dns-query-name.c  | 171 +++++++++++++++++++++++++++++++++++
 src/detect-dns-query-name.h  |  23 +++++
 src/detect-engine-register.c |   2 +
 src/detect-engine-register.h |   1 +
 6 files changed, 224 insertions(+)
 create mode 100644 src/detect-dns-query-name.c
 create mode 100644 src/detect-dns-query-name.h

diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index 8933c680db56..e8558828d5d0 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -870,6 +870,31 @@ pub unsafe extern "C" fn rs_dns_tx_get_query_name(
     return 0;
 }
 
+/// Get the DNS query name at index i.
+#[no_mangle]
+pub unsafe extern "C" fn SCDnsTxGetQueryName(
+    tx: &mut DNSTransaction, to_client: bool, i: u32, buf: *mut *const u8, len: *mut u32,
+) -> bool {
+    let queries = if to_client {
+        tx.response.as_ref().map(|response| &response.queries)
+    } else {
+        tx.request.as_ref().map(|request| &request.queries)
+    };
+    let index = i as usize;
+
+    if let Some(queries) = queries {
+        if let Some(query) = queries.get(index) {
+            if !query.name.is_empty() {
+                *buf = query.name.as_ptr();
+                *len = query.name.len() as u32;
+                return true;
+            }
+        }
+    }
+
+    false
+}
+
 /// Get the DNS response answer name and index i.
 #[no_mangle]
 pub unsafe extern "C" fn SCDnsTxGetAnswerName(
diff --git a/src/Makefile.am b/src/Makefile.am
index 25bec98e2e7b..6d115ac48ae5 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -124,6 +124,7 @@ noinst_HEADERS = \
 	detect-dns-answer-name.h \
 	detect-dns-opcode.h \
 	detect-dns-query.h \
+	detect-dns-query-name.h \
 	detect-dsize.h \
 	detect-engine-address.h \
 	detect-engine-address-ipv4.h \
@@ -740,6 +741,7 @@ libsuricata_c_a_SOURCES = \
 	detect-dns-answer-name.c \
 	detect-dns-opcode.c \
 	detect-dns-query.c \
+	detect-dns-query-name.c \
 	detect-dsize.c \
 	detect-engine-address.c \
 	detect-engine-address-ipv4.c \
diff --git a/src/detect-dns-query-name.c b/src/detect-dns-query-name.c
new file mode 100644
index 000000000000..a3983bf575cd
--- /dev/null
+++ b/src/detect-dns-query-name.c
@@ -0,0 +1,171 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * Detect keyword for DNS query names: dns.query.name
+ */
+
+#include "detect.h"
+#include "detect-parse.h"
+#include "detect-engine.h"
+#include "detect-engine-prefilter.h"
+#include "detect-engine-content-inspection.h"
+#include "detect-dns-query-name.h"
+#include "util-profiling.h"
+#include "rust.h"
+
+typedef struct PrefilterMpm {
+    int list_id;
+    const MpmCtx *mpm_ctx;
+    const DetectEngineTransforms *transforms;
+} PrefilterMpm;
+
+static int detect_buffer_id = 0;
+
+static int DetectSetup(DetectEngineCtx *de_ctx, Signature *s, const char *str)
+{
+    if (DetectBufferSetActiveList(de_ctx, s, detect_buffer_id) < 0) {
+        return -1;
+    }
+    if (DetectSignatureSetAppProto(s, ALPROTO_DNS) < 0) {
+        return -1;
+    }
+
+    return 0;
+}
+
+static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx, const uint8_t flags,
+        const DetectEngineTransforms *transforms, void *txv, uint32_t index, int list_id)
+{
+    InspectionBuffer *buffer = InspectionBufferMultipleForListGet(det_ctx, list_id, index);
+    if (buffer == NULL) {
+        return NULL;
+    }
+    if (buffer->initialized) {
+        return buffer;
+    }
+
+    bool to_client = (flags & STREAM_TOSERVER) == 0;
+    const uint8_t *data = NULL;
+    uint32_t data_len = 0;
+
+    if (!SCDnsTxGetQueryName(txv, to_client, index, &data, &data_len)) {
+        InspectionBufferSetupMultiEmpty(buffer);
+        return NULL;
+    }
+    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    buffer->flags = DETECT_CI_FLAGS_SINGLE;
+    return buffer;
+}
+
+static uint8_t DetectEngineInspectCb(DetectEngineCtx *de_ctx, DetectEngineThreadCtx *det_ctx,
+        const struct DetectEngineAppInspectionEngine_ *engine, const Signature *s, Flow *f,
+        uint8_t flags, void *alstate, void *txv, uint64_t tx_id)
+{
+    const DetectEngineTransforms *transforms = NULL;
+    if (!engine->mpm) {
+        transforms = engine->v2.transforms;
+    }
+
+    for (uint32_t i = 0;; i++) {
+        InspectionBuffer *buffer = GetBuffer(det_ctx, flags, transforms, txv, i, engine->sm_list);
+        if (buffer == NULL || buffer->inspect == NULL) {
+            break;
+        }
+
+        const bool match = DetectEngineContentInspectionBuffer(de_ctx, det_ctx, s, engine->smd,
+                NULL, f, buffer, DETECT_ENGINE_CONTENT_INSPECTION_MODE_STATE);
+        if (match) {
+            return DETECT_ENGINE_INSPECT_SIG_MATCH;
+        }
+    }
+
+    return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
+}
+
+static void PrefilterTx(DetectEngineThreadCtx *det_ctx, const void *pectx, Packet *p, Flow *f,
+        void *txv, const uint64_t idx, const AppLayerTxData *_txd, const uint8_t flags)
+{
+    SCEnter();
+
+    const PrefilterMpm *ctx = (const PrefilterMpm *)pectx;
+    const MpmCtx *mpm_ctx = ctx->mpm_ctx;
+    const int list_id = ctx->list_id;
+
+    for (uint32_t i = 0;; i++) {
+        InspectionBuffer *buffer = GetBuffer(det_ctx, flags, ctx->transforms, txv, i, list_id);
+        if (buffer == NULL) {
+            break;
+        }
+
+        if (buffer->inspect_len >= mpm_ctx->minlen) {
+            (void)mpm_table[mpm_ctx->mpm_type].Search(
+                    mpm_ctx, &det_ctx->mtc, &det_ctx->pmq, buffer->inspect, buffer->inspect_len);
+            PREFILTER_PROFILING_ADD_BYTES(det_ctx, buffer->inspect_len);
+        }
+    }
+}
+
+static void PrefilterMpmFree(void *ptr)
+{
+    SCFree(ptr);
+}
+
+static int PrefilterMpmRegister(DetectEngineCtx *de_ctx, SigGroupHead *sgh, MpmCtx *mpm_ctx,
+        const DetectBufferMpmRegistry *mpm_reg, int list_id)
+{
+    PrefilterMpm *pectx = SCCalloc(1, sizeof(*pectx));
+    if (pectx == NULL) {
+        return -1;
+    }
+    pectx->list_id = list_id;
+    pectx->mpm_ctx = mpm_ctx;
+    pectx->transforms = &mpm_reg->transforms;
+
+    return PrefilterAppendTxEngine(de_ctx, sgh, PrefilterTx, mpm_reg->app_v2.alproto,
+            mpm_reg->app_v2.tx_min_progress, pectx, PrefilterMpmFree, mpm_reg->pname);
+}
+
+void DetectDnsQueryNameRegister(void)
+{
+    static const char *keyword = "dns.query.name";
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].name = keyword;
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].desc = "DNS query name sticky buffer";
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].url = "/rules/dns-keywords.html#dns-query-name";
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].Setup = DetectSetup;
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].flags |= SIGMATCH_NOOPT;
+    sigmatch_table[DETECT_AL_DNS_QUERY_NAME].flags |= SIGMATCH_INFO_STICKY_BUFFER;
+
+    /* Register in both directions as the query is usually echoed back
+       in the response. */
+    DetectAppLayerInspectEngineRegister(
+            keyword, ALPROTO_DNS, SIG_FLAG_TOSERVER, 0, DetectEngineInspectCb, NULL);
+    DetectAppLayerMpmRegister(
+            keyword, SIG_FLAG_TOSERVER, 2, PrefilterMpmRegister, NULL, ALPROTO_DNS, 1);
+
+    DetectAppLayerInspectEngineRegister(
+            keyword, ALPROTO_DNS, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectCb, NULL);
+    DetectAppLayerMpmRegister(
+            keyword, SIG_FLAG_TOCLIENT, 2, PrefilterMpmRegister, NULL, ALPROTO_DNS, 1);
+
+    DetectBufferTypeSetDescriptionByName(keyword, "dns query name");
+    DetectBufferTypeSupportsMultiInstance(keyword);
+
+    detect_buffer_id = DetectBufferTypeGetByName(keyword);
+}
diff --git a/src/detect-dns-query-name.h b/src/detect-dns-query-name.h
new file mode 100644
index 000000000000..b1d7db99e8c5
--- /dev/null
+++ b/src/detect-dns-query-name.h
@@ -0,0 +1,23 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#ifndef __DETECT_DNS_QUERY_NAME_H__
+#define __DETECT_DNS_QUERY_NAME_H__
+
+void DetectDnsQueryNameRegister(void);
+
+#endif /* __DETECT_DNS_QUERY_NAME_H__ */
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index 258bbef3f543..a97da4617197 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -50,6 +50,7 @@
 #include "detect-dns-opcode.h"
 #include "detect-dns-query.h"
 #include "detect-dns-answer-name.h"
+#include "detect-dns-query-name.h"
 #include "detect-tls-sni.h"
 #include "detect-tls-certs.h"
 #include "detect-tls-cert-fingerprint.h"
@@ -519,6 +520,7 @@ void SigTableSetup(void)
     DetectDnsQueryRegister();
     DetectDnsOpcodeRegister();
     DetectDnsAnswerNameRegister();
+    DetectDnsQueryNameRegister();
     DetectModbusRegister();
     DetectCipServiceRegister();
     DetectEnipCommandRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 0a529954e953..2e4a330788ed 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -228,6 +228,7 @@ enum DetectKeywordId {
     DETECT_AL_DNS_QUERY,
     DETECT_AL_DNS_OPCODE,
     DETECT_AL_DNS_ANSWER_NAME,
+    DETECT_AL_DNS_QUERY_NAME,
     DETECT_AL_TLS_SNI,
     DETECT_AL_TLS_CERTS,
     DETECT_AL_TLS_CERT_ISSUER,

From f91122e0e8ca02850368eac46ad5b2b516ac9cb8 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 15 Nov 2023 12:31:12 -0600
Subject: [PATCH 276/462] dns: replace usage of rs_dns_tx_get_query_name with
 SCDnsTxGetQueryName

SCDnsTxGetQueryName was introduced to allow for getting the query name
in responses as well as requests, so covers the functionality of
rs_dns_tx_get_query_name.
---
 rust/src/dns/dns.rs    | 17 -----------------
 src/detect-dns-query.c |  2 +-
 2 files changed, 1 insertion(+), 18 deletions(-)

diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index e8558828d5d0..57f66c0f73df 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -853,23 +853,6 @@ pub unsafe extern "C" fn rs_dns_state_get_tx_data(
 
 export_state_data_get!(rs_dns_get_state_data, DNSState);
 
-#[no_mangle]
-pub unsafe extern "C" fn rs_dns_tx_get_query_name(
-    tx: &mut DNSTransaction, i: u32, buf: *mut *const u8, len: *mut u32,
-) -> u8 {
-    if let Some(request) = &tx.request {
-        if (i as usize) < request.queries.len() {
-            let query = &request.queries[i as usize];
-            if !query.name.is_empty() {
-                *len = query.name.len() as u32;
-                *buf = query.name.as_ptr();
-                return 1;
-            }
-        }
-    }
-    return 0;
-}
-
 /// Get the DNS query name at index i.
 #[no_mangle]
 pub unsafe extern "C" fn SCDnsTxGetQueryName(
diff --git a/src/detect-dns-query.c b/src/detect-dns-query.c
index ebdc7088e78e..3225f126f2df 100644
--- a/src/detect-dns-query.c
+++ b/src/detect-dns-query.c
@@ -87,7 +87,7 @@ static InspectionBuffer *DnsQueryGetData(DetectEngineThreadCtx *det_ctx,
 
     const uint8_t *data;
     uint32_t data_len;
-    if (rs_dns_tx_get_query_name(cbdata->txv, cbdata->local_id, &data, &data_len) == 0) {
+    if (SCDnsTxGetQueryName(cbdata->txv, false, cbdata->local_id, &data, &data_len) == 0) {
         InspectionBufferSetupMultiEmpty(buffer);
         return NULL;
     }

From c1a8dbcb72584b10819fd0d07d28c4c59a39e1cd Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 15 Nov 2023 12:11:51 -0600
Subject: [PATCH 277/462] doc/userguide: document dns.query.name,
 dns.answer.name

With some other minor cleanups in the DNS keyword section.
---
 doc/userguide/rules/dns-keywords.rst | 63 +++++++++++++++++++++-------
 1 file changed, 49 insertions(+), 14 deletions(-)

diff --git a/doc/userguide/rules/dns-keywords.rst b/doc/userguide/rules/dns-keywords.rst
index e62a25d40bed..a514ae25195b 100644
--- a/doc/userguide/rules/dns-keywords.rst
+++ b/doc/userguide/rules/dns-keywords.rst
@@ -1,10 +1,27 @@
 DNS Keywords
 ============
 
-There are some more content modifiers (If you are unfamiliar with
-content modifiers, please visit the page :doc:`payload-keywords` These
-ones make sure the signature checks a specific part of the
-network-traffic.
+Suricata supports sticky buffers as well as keywords for efficiently
+matching on specific fields in DNS messages.
+
+Note that sticky buffers are expected to be followed by one or more
+:doc:`payload-keywords`.
+
+dns.answer.name
+---------------
+
+``dns.answer.name`` is a sticky buffer that is used to look at the
+name field in DNS answer resource records.
+
+``dns.answer.name`` will look at both requests and responses, so
+``flow`` is recommended to confine to a specific direction.
+
+The buffer being matched on contains the complete re-assembled
+resource name, for example "www.suricata.io".
+
+``dns.answer.name`` supports :doc:`multi-buffer-matching`.
+
+``dns.answer.name`` was introduced in Suricata 8.0.0.
 
 dns.opcode
 ----------
@@ -32,20 +49,26 @@ Match on DNS requests where the **opcode** is NOT 0::
 dns.query
 ---------
 
-With **dns.query** the DNS request queries are inspected. The dns.query
-keyword works a bit different from the normal content modifiers. When
-used in a rule all contents following it are affected by it.  Example:
+``dns.query`` is a sticky buffer that is used to inspect DNS query
+names in DNS request messages. Example::
 
-  alert dns any any -> any any (msg:"Test dns.query option";
-  dns.query; content:"google"; nocase; sid:1;)
+  alert dns any any -> any any (msg:"Test dns.query option"; dns.query; content:"google"; nocase; sid:1;)
+
+Being a sticky buffer, payload keywords such as content are to be used after ``dns.query``:
 
 .. image:: dns-keywords/dns_query.png
 
-The **dns.query** keyword affects all following contents, until pkt_data
-is used or it reaches the end of the rule.
+The ``dns.query`` keyword affects all following contents, until
+pkt_data is used or it reaches the end of the rule.
 
 .. note:: **dns.query** is equivalent to the older **dns_query**.
 
+.. note:: **dns.query** will only match on DNS request messages, to
+          also match on DNS response message, see
+          `dns.query.name`_.
+
+``dns.query.name`` supports :doc:`multi-buffer-matching`.
+
 Normalized Buffer
 ~~~~~~~~~~~~~~~~~
 
@@ -68,7 +91,19 @@ DNS query on the wire (snippet)::
 
     mail.google.com
 
-Multiple Buffer Matching
-~~~~~~~~~~~~~~~~~~~~~~~~
+dns.query.name
+---------------
+
+``dns.query.name`` is a sticky buffer that is used to look at the name
+field in DNS query (question) resource records. It is nearly identical
+to ``dns.query`` but supports both DNS requests and responses.
+
+``dns.query.name`` will look at both requests and responses, so
+``flow`` is recommended to confine to a specific direction.
+
+The buffer being matched on contains the complete re-assembled
+resource name, for example "www.suricata.io".
+
+``dns.query.name`` supports :doc:`multi-buffer-matching`.
 
-``dns.query`` supports multiple buffer matching, see :doc:`multi-buffer-matching`.
\ No newline at end of file
+``dns.query.name`` was introduced in Suricata 8.0.0.

From 97744b7ea7b5405783a70e93e257e32233f1d666 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Sun, 12 Nov 2023 14:03:25 +0100
Subject: [PATCH 278/462] output-json-alert: remove un-needed includes

---
 src/output-json-alert.c | 14 --------------
 1 file changed, 14 deletions(-)

diff --git a/src/output-json-alert.c b/src/output-json-alert.c
index c7acfe64d140..5f511b962d29 100644
--- a/src/output-json-alert.c
+++ b/src/output-json-alert.c
@@ -31,30 +31,20 @@
 #include "conf.h"
 
 #include "stream.h"
-#include "threads.h"
-#include "tm-threads.h"
 #include "threadvars.h"
 #include "util-debug.h"
 
 #include "util-logopenfile.h"
 #include "util-misc.h"
 #include "util-time.h"
-#include "util-unittest.h"
-#include "util-unittest-helper.h"
 
-#include "detect-parse.h"
 #include "detect-engine.h"
-#include "detect-engine-mpm.h"
-#include "detect-reference.h"
 #include "detect-metadata.h"
 #include "app-layer-parser.h"
 #include "app-layer-dnp3.h"
-#include "app-layer-htp.h"
 #include "app-layer-htp-xff.h"
 #include "app-layer-ftp.h"
 #include "app-layer-frames.h"
-#include "util-classification-config.h"
-#include "util-syslog.h"
 #include "log-pcap.h"
 
 #include "output.h"
@@ -64,7 +54,6 @@
 #include "output-json-dns.h"
 #include "output-json-http.h"
 #include "output-json-tls.h"
-#include "output-json-ssh.h"
 #include "rust.h"
 #include "output-json-smtp.h"
 #include "output-json-email-common.h"
@@ -79,10 +68,7 @@
 #include "output-json-frame.h"
 #include "output-json-quic.h"
 
-#include "util-byte.h"
-#include "util-privs.h"
 #include "util-print.h"
-#include "util-proto-name.h"
 #include "util-optimize.h"
 #include "util-buffer.h"
 #include "util-validate.h"

From 7d95c4c017cd1fe97b4a85895f308d590cd07c11 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Sun, 12 Nov 2023 08:46:07 -0500
Subject: [PATCH 279/462] output-json-dns: remove un-needed includes

---
 src/output-json-dns.c | 13 -------------
 1 file changed, 13 deletions(-)

diff --git a/src/output-json-dns.c b/src/output-json-dns.c
index 27aa55d8e305..3c7009b501ca 100644
--- a/src/output-json-dns.c
+++ b/src/output-json-dns.c
@@ -24,27 +24,14 @@
  */
 
 #include "suricata-common.h"
-#include "detect.h"
-#include "pkt-var.h"
 #include "conf.h"
 
-#include "threads.h"
 #include "threadvars.h"
-#include "tm-threads.h"
-
-#include "util-print.h"
-#include "util-unittest.h"
 
 #include "util-debug.h"
 #include "util-mem.h"
 #include "app-layer-parser.h"
 #include "output.h"
-#include "app-layer.h"
-#include "util-privs.h"
-#include "util-buffer.h"
-#include "util-proto-name.h"
-#include "util-logopenfile.h"
-#include "util-time.h"
 
 #include "output-json.h"
 #include "output-json-dns.h"

From 0ab32be355e41779eee1b3e32420c284a5f20317 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 28 Sep 2023 14:41:20 +0200
Subject: [PATCH 280/462] eve/stream: add sb main region size; segment count

Gives more detail about memory use.
---
 src/output-eve-stream.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/src/output-eve-stream.c b/src/output-eve-stream.c
index 919505dce70d..51296c657a1a 100644
--- a/src/output-eve-stream.c
+++ b/src/output-eve-stream.c
@@ -259,6 +259,11 @@ void EveAddFlowTcpFlags(const TcpSession *ssn, const char *name, JsonBuilder *jb
     jb_close(jb);
 }
 
+static void LogStreamSB(const StreamingBuffer *sb, JsonBuilder *js)
+{
+    jb_set_uint(js, "sb_region_size", sb->region.buf_size);
+}
+
 static void LogStream(const TcpStream *stream, JsonBuilder *js)
 {
     jb_set_uint(js, "isn", stream->isn);
@@ -273,6 +278,15 @@ static void LogStream(const TcpStream *stream, JsonBuilder *js)
     jb_set_uint(js, "wscale", stream->wscale);
 
     EveAddFlowTcpStreamFlags(stream, "flags", js);
+
+    TcpSegment *s;
+    uint32_t segs = 0;
+    RB_FOREACH(s, TCPSEG, (struct TCPSEG *)&stream->seg_tree)
+    {
+        segs++;
+    }
+    jb_set_uint(js, "seg_cnt", segs);
+    LogStreamSB(&stream->sb, js);
 }
 
 /**

From b8440a0917548061049df19e4d42200df67cb478 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 8 Dec 2023 10:30:25 +0100
Subject: [PATCH 281/462] jsonbuilder: add set_int for signed ints

Bug: #6615
---
 rust/src/jsonbuilder.rs | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/rust/src/jsonbuilder.rs b/rust/src/jsonbuilder.rs
index 82be09953c70..9ff623429509 100644
--- a/rust/src/jsonbuilder.rs
+++ b/rust/src/jsonbuilder.rs
@@ -597,6 +597,27 @@ impl JsonBuilder {
         Ok(self)
     }
 
+    /// Set a key and a signed integer type on an object.
+    pub fn set_int(&mut self, key: &str, val: i64) -> Result<&mut Self, JsonError> {
+        match self.current_state() {
+            State::ObjectNth => {
+                self.push(',')?;
+            }
+            State::ObjectFirst => {
+                self.set_state(State::ObjectNth);
+            }
+            _ => {
+                debug_validate_fail!("invalid state");
+                return Err(JsonError::InvalidState);
+            }
+        }
+        self.push('"')?;
+        self.push_str(key)?;
+        self.push_str("\":")?;
+        self.push_str(&val.to_string())?;
+        Ok(self)
+    }
+
     pub fn set_float(&mut self, key: &str, val: f64) -> Result<&mut Self, JsonError> {
         match self.current_state() {
             State::ObjectNth => {
@@ -940,6 +961,14 @@ pub unsafe extern "C" fn jb_set_uint(js: &mut JsonBuilder, key: *const c_char, v
     return false;
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn jb_set_int(js: &mut JsonBuilder, key: *const c_char, val: i64) -> bool {
+    if let Ok(key) = CStr::from_ptr(key).to_str() {
+        return js.set_int(key, val).is_ok();
+    }
+    return false;
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn jb_set_float(js: &mut JsonBuilder, key: *const c_char, val: f64) -> bool {
     if let Ok(key) = CStr::from_ptr(key).to_str() {

From de5b8ae0b45636352e641cff7eeb4f1fc89c8129 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 8 Dec 2023 10:31:21 +0100
Subject: [PATCH 282/462] detect/analyzer: print int keyword values correctly

To avoid negative values to be misrepresented.

Bug: #6615.
---
 src/detect-engine-analyzer.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/detect-engine-analyzer.c b/src/detect-engine-analyzer.c
index 0eda31b2fc4a..b936ba1e0c38 100644
--- a/src/detect-engine-analyzer.c
+++ b/src/detect-engine-analyzer.c
@@ -703,10 +703,10 @@ static void DumpContent(JsonBuilder *js, const DetectContentData *cd)
         jb_set_uint(js, "depth", cd->depth);
     }
     if (cd->flags & DETECT_CONTENT_DISTANCE) {
-        jb_set_uint(js, "distance", cd->distance);
+        jb_set_int(js, "distance", cd->distance);
     }
     if (cd->flags & DETECT_CONTENT_WITHIN) {
-        jb_set_uint(js, "within", cd->within);
+        jb_set_int(js, "within", cd->within);
     }
     jb_set_bool(js, "fast_pattern", cd->flags & DETECT_CONTENT_FAST_PATTERN);
     jb_set_bool(js, "relative_next", cd->flags & DETECT_CONTENT_RELATIVE_NEXT);
@@ -779,9 +779,9 @@ static void DumpMatches(RuleAnalyzer *ctx, JsonBuilder *js, const SigMatchData *
 
                 jb_open_object(js, "byte_jump");
                 jb_set_uint(js, "nbytes", cd->nbytes);
-                jb_set_uint(js, "offset", cd->offset);
+                jb_set_int(js, "offset", cd->offset);
                 jb_set_uint(js, "multiplier", cd->multiplier);
-                jb_set_uint(js, "post_offset", cd->post_offset);
+                jb_set_int(js, "post_offset", cd->post_offset);
                 switch (cd->base) {
                     case DETECT_BYTEJUMP_BASE_UNSET:
                         jb_set_string(js, "base", "unset");
@@ -824,7 +824,7 @@ static void DumpMatches(RuleAnalyzer *ctx, JsonBuilder *js, const SigMatchData *
 
                 jb_open_object(js, "byte_test");
                 jb_set_uint(js, "nbytes", cd->nbytes);
-                jb_set_uint(js, "offset", cd->offset);
+                jb_set_int(js, "offset", cd->offset);
                 switch (cd->base) {
                     case DETECT_BYTETEST_BASE_UNSET:
                         jb_set_string(js, "base", "unset");

From 101452056d559b4e5eab27c2d9fb651d2e32c412 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 8 Dec 2023 09:38:38 +0100
Subject: [PATCH 283/462] detect/bytejump: don't reuse content flag

To avoid future problems with overlapping flag values, give bytejump
its own DETECT_BYTEJUMP_OFFSET_VAR flag.

The values are currently not overlapping, so this patch should have
no side effects.
---
 src/detect-byte-extract.c              | 21 +++++++--------------
 src/detect-bytejump.c                  |  2 +-
 src/detect-bytejump.h                  |  1 +
 src/detect-engine-content-inspection.c |  2 +-
 4 files changed, 10 insertions(+), 16 deletions(-)

diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index cf9b24348e5e..09289da6d217 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -3513,8 +3513,7 @@ static int DetectByteExtractTest53(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 0) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
         printf("three failed\n");
         result = 0;
         goto end;
@@ -3618,8 +3617,7 @@ static int DetectByteExtractTest54(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 0) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
         printf("three failed\n");
         result = 0;
         goto end;
@@ -3631,8 +3629,7 @@ static int DetectByteExtractTest54(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 1) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 1) {
         printf("four failed\n");
         result = 0;
         goto end;
@@ -4165,8 +4162,7 @@ static int DetectByteExtractTest58(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 0) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
         printf("three failed\n");
         result = 0;
         goto end;
@@ -4178,8 +4174,7 @@ static int DetectByteExtractTest58(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 1) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 1) {
         printf("four failed\n");
         result = 0;
         goto end;
@@ -4298,8 +4293,7 @@ static int DetectByteExtractTest59(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 0) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
         printf("three failed\n");
         result = 0;
         goto end;
@@ -4311,8 +4305,7 @@ static int DetectByteExtractTest59(void)
         goto end;
     }
     bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_CONTENT_OFFSET_VAR ||
-        bjd->offset != 1) {
+    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 1) {
         printf("four failed\n");
         result = 0;
         goto end;
diff --git a/src/detect-bytejump.c b/src/detect-bytejump.c
index 3e7ae4f5e000..e04d8a7a94fb 100644
--- a/src/detect-bytejump.c
+++ b/src/detect-bytejump.c
@@ -563,7 +563,7 @@ static int DetectBytejumpSetup(DetectEngineCtx *de_ctx, Signature *s, const char
             goto error;
         }
         data->offset = index;
-        data->flags |= DETECT_CONTENT_OFFSET_VAR;
+        data->flags |= DETECT_BYTEJUMP_OFFSET_VAR;
         SCFree(offset);
         offset = NULL;
     }
diff --git a/src/detect-bytejump.h b/src/detect-bytejump.h
index f8ee530b3864..15f610344320 100644
--- a/src/detect-bytejump.h
+++ b/src/detect-bytejump.h
@@ -41,6 +41,7 @@
 #define DETECT_BYTEJUMP_OFFSET_BE BIT_U16(7) /**< "byte extract" enabled */
 #define DETECT_BYTEJUMP_END       BIT_U16(8) /**< "from_end" jump */
 #define DETECT_BYTEJUMP_NBYTES_VAR BIT_U16(9) /**< nbytes string*/
+#define DETECT_BYTEJUMP_OFFSET_VAR BIT_U16(10) /**< byte extract value enabled */
 
 typedef struct DetectBytejumpData_ {
     uint8_t nbytes;                   /**< Number of bytes to compare */
diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 5d6ad2be5629..81f2e30f31a6 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -515,7 +515,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
         int32_t offset = bjd->offset;
         int32_t nbytes;
 
-        if (bjflags & DETECT_CONTENT_OFFSET_VAR) {
+        if (bjflags & DETECT_BYTEJUMP_OFFSET_VAR) {
             offset = det_ctx->byte_values[offset];
         }
 

From fb497bfa7e275474e880727aa232aac0f09db1e9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 8 Dec 2023 12:45:06 +0100
Subject: [PATCH 284/462] detect/bytejump: test cleanup

Just one used during debugging.
---
 src/detect-byte-extract.c | 115 +++++++++++---------------------------
 1 file changed, 33 insertions(+), 82 deletions(-)

diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index 09289da6d217..214b21877d96 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -3438,98 +3438,49 @@ static int DetectByteExtractTest52(void)
 
 static int DetectByteExtractTest53(void)
 {
-    DetectEngineCtx *de_ctx = NULL;
-    int result = 0;
-    Signature *s = NULL;
-    SigMatch *sm = NULL;
-    DetectContentData *cd = NULL;
-    DetectByteExtractData *bed = NULL;
-    DetectBytejumpData *bjd = NULL;
-
-    de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-
+    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
-    s = de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
-                                   "(msg:\"Testing bytejump_body\"; "
-                                   "content:\"one\"; "
-                                   "byte_extract:4,0,two,string,hex; "
-                                   "byte_jump: 2,two; "
-                                   "sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        result = 0;
-        goto end;
-    }
 
-    if (s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH] == NULL) {
-        result = 0;
-        goto end;
-    }
+    Signature *s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any ("
+                                                 "content:\"one\"; "
+                                                 "byte_extract:4,0,two,string,hex; "
+                                                 "byte_jump: 2,two; "
+                                                 "sid:1;)");
+    FAIL_IF_NULL(s);
+    FAIL_IF_NULL(s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH]);
 
-    sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
-    if (sm->type != DETECT_CONTENT) {
-        result = 0;
-        goto end;
-    }
-    cd = (DetectContentData *)sm->ctx;
-    if (cd->flags & DETECT_CONTENT_RAWBYTES ||
-        strncmp((char *)cd->content, "one", cd->content_len) != 0 ||
-        cd->flags & DETECT_CONTENT_NOCASE ||
-        cd->flags & DETECT_CONTENT_WITHIN ||
-        cd->flags & DETECT_CONTENT_DISTANCE ||
-        cd->flags & DETECT_CONTENT_FAST_PATTERN ||
-        cd->flags & DETECT_CONTENT_RELATIVE_NEXT ||
-        cd->flags & DETECT_CONTENT_NEGATED ) {
-        printf("one failed\n");
-        result = 0;
-        goto end;
-    }
+    SigMatch *sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
+    FAIL_IF(sm->type != DETECT_CONTENT);
+    DetectContentData *cd = (DetectContentData *)sm->ctx;
+    FAIL_IF(cd->flags != 0);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTE_EXTRACT) {
-        result = 0;
-        goto end;
-    }
-    bed = (DetectByteExtractData *)sm->ctx;
-    if (bed->nbytes != 4 ||
-        bed->offset != 0 ||
-        strcmp(bed->name, "two") != 0 ||
-        bed->flags != DETECT_BYTE_EXTRACT_FLAG_STRING ||
-        bed->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE ||
-        bed->base != DETECT_BYTE_EXTRACT_BASE_HEX ||
-        bed->align_value != 0 ||
-        bed->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT) {
-        goto end;
-    }
-    if (bed->local_id != 0) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTE_EXTRACT);
+    DetectByteExtractData *bed = (DetectByteExtractData *)sm->ctx;
+
+    FAIL_IF(bed->nbytes != 4);
+    FAIL_IF(bed->offset != 0);
+    FAIL_IF(strcmp(bed->name, "two") != 0);
+    FAIL_IF(bed->flags != DETECT_BYTE_EXTRACT_FLAG_STRING);
+    FAIL_IF(bed->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE);
+    FAIL_IF(bed->base != DETECT_BYTE_EXTRACT_BASE_HEX);
+    FAIL_IF(bed->align_value != 0);
+    FAIL_IF(bed->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT);
+    FAIL_IF(bed->local_id != 0);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTEJUMP) {
-        result = 0;
-        goto end;
-    }
-    bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
-        printf("three failed\n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTEJUMP);
+    DetectBytejumpData *bjd = (DetectBytejumpData *)sm->ctx;
 
-    if (sm->next != NULL)
-        goto end;
-
-    result = 1;
+    FAIL_IF(bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR);
+    FAIL_IF(bjd->offset != 0);
 
- end:
-    SigGroupCleanup(de_ctx);
-    SigCleanSignatures(de_ctx);
+    FAIL_IF_NOT_NULL(sm->next);
     DetectEngineCtxFree(de_ctx);
-
-    return result;
+    PASS;
 }
 
 static int DetectByteExtractTest54(void)

From 83ed2c3b97925d390c2a57fdc8eea52f7d3d2e4c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 7 Dec 2023 17:07:04 +0100
Subject: [PATCH 285/462] detect/bytemath: bump length to uint32_t

This puts the logic in line with the other payload inspection
functions.
---
 src/detect-bytemath.c | 2 +-
 src/detect-bytemath.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect-bytemath.c b/src/detect-bytemath.c
index 3e848f21a5bb..8909085390e4 100644
--- a/src/detect-bytemath.c
+++ b/src/detect-bytemath.c
@@ -85,7 +85,7 @@ static inline bool DetectByteMathValidateNbytesOnly(const DetectByteMathData *da
 }
 
 int DetectByteMathDoMatch(DetectEngineThreadCtx *det_ctx, const DetectByteMathData *data,
-        const Signature *s, const uint8_t *payload, uint16_t payload_len, uint8_t nbytes,
+        const Signature *s, const uint8_t *payload, const uint32_t payload_len, uint8_t nbytes,
         uint64_t rvalue, uint64_t *value, uint8_t endian)
 {
     if (payload_len == 0) {
diff --git a/src/detect-bytemath.h b/src/detect-bytemath.h
index 4fbc9ae5ce15..c18a3e82dcab 100644
--- a/src/detect-bytemath.h
+++ b/src/detect-bytemath.h
@@ -28,6 +28,6 @@ void DetectBytemathRegister(void);
 
 SigMatch *DetectByteMathRetrieveSMVar(const char *, const Signature *);
 int DetectByteMathDoMatch(DetectEngineThreadCtx *, const DetectByteMathData *, const Signature *,
-        const uint8_t *, uint16_t, uint8_t, uint64_t, uint64_t *, uint8_t);
+        const uint8_t *, const uint32_t, uint8_t, uint64_t, uint64_t *, uint8_t);
 
 #endif /* __DETECT_BYTEMATH_H__ */

From 804a40e0366a0f9a14aba1ae53ae3d057040dee7 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 12 Dec 2023 14:54:41 +0100
Subject: [PATCH 286/462] detect/byte_extract: modernize tests

---
 src/detect-byte-extract.c | 278 +++++++++++++-------------------------
 1 file changed, 91 insertions(+), 187 deletions(-)

diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index 214b21877d96..379d3dce015f 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -1575,90 +1575,47 @@ static int DetectByteExtractTest35(void)
 
 static int DetectByteExtractTest36(void)
 {
-    DetectEngineCtx *de_ctx = NULL;
-    int result = 0;
-    Signature *s = NULL;
-    SigMatch *sm = NULL;
-    DetectContentData *cd = NULL;
-    DetectBytejumpData *bjd = NULL;
-    DetectByteExtractData *bed = NULL;
-
-    de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-
+    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
-    s = de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
-                                   "(msg:\"Testing bytejump_body\"; "
-                                   "content:\"one\"; byte_jump:1,13; "
-                                   "byte_extract:4,0,two,relative,string,hex; "
-                                   "sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        result = 0;
-        goto end;
-    }
 
-    if (s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH] == NULL) {
-        result = 0;
-        goto end;
-    }
+    Signature *s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any ("
+                                                 "content:\"one\"; byte_jump:1,13; "
+                                                 "byte_extract:4,0,two,relative,string,hex; "
+                                                 "sid:1;)");
+    FAIL_IF_NULL(s);
+    FAIL_IF_NULL(s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH]);
 
-    sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
-    if (sm->type != DETECT_CONTENT) {
-        result = 0;
-        goto end;
-    }
-    cd = (DetectContentData *)sm->ctx;
-    if (cd->flags & DETECT_CONTENT_RAWBYTES ||
-        strncmp((char *)cd->content, "one", cd->content_len) != 0 ||
-        cd->flags & DETECT_CONTENT_NOCASE ||
-        cd->flags & DETECT_CONTENT_WITHIN ||
-        cd->flags & DETECT_CONTENT_DISTANCE ||
-        cd->flags & DETECT_CONTENT_FAST_PATTERN ||
-        cd->flags & DETECT_CONTENT_RELATIVE_NEXT ||
-        cd->flags & DETECT_CONTENT_NEGATED ) {
-        printf("one failed\n");
-        result = 0;
-        goto end;
-    }
+    SigMatch *sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
+    FAIL_IF(sm->type != DETECT_CONTENT);
+    DetectContentData *cd = (DetectContentData *)sm->ctx;
+    FAIL_IF(cd->flags & DETECT_CONTENT_RAWBYTES);
+    FAIL_IF(strncmp((char *)cd->content, "one", cd->content_len) != 0);
+    FAIL_IF(cd->flags & DETECT_CONTENT_NOCASE);
+    FAIL_IF(cd->flags & DETECT_CONTENT_WITHIN);
+    FAIL_IF(cd->flags & DETECT_CONTENT_DISTANCE);
+    FAIL_IF(cd->flags & DETECT_CONTENT_FAST_PATTERN);
+    FAIL_IF(cd->flags & DETECT_CONTENT_RELATIVE_NEXT);
+    FAIL_IF(cd->flags & DETECT_CONTENT_NEGATED);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTEJUMP) {
-        result = 0;
-        goto end;
-    }
-    bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != 0) {
-        result = 0;
-        goto end;
-    }
-
+    FAIL_IF(sm->type != DETECT_BYTEJUMP);
+    DetectBytejumpData *bjd = (DetectBytejumpData *)sm->ctx;
+    FAIL_IF(bjd->flags != 0);
     sm = sm->next;
-    if (sm->type != DETECT_BYTE_EXTRACT) {
-        result = 0;
-        goto end;
-    }
-    bed = (DetectByteExtractData *)sm->ctx;
-    if (bed->nbytes != 4 ||
-        bed->offset != 0 ||
-        strcmp(bed->name, "two") != 0 ||
-        bed->flags != (DETECT_BYTE_EXTRACT_FLAG_RELATIVE |
-                       DETECT_BYTE_EXTRACT_FLAG_STRING) ||
-        bed->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE ||
-        bed->base != DETECT_BYTE_EXTRACT_BASE_HEX ||
-        bed->align_value != 0 ||
-        bed->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT) {
-        goto end;
-    }
-
-    result = 1;
+    FAIL_IF(sm->type != DETECT_BYTE_EXTRACT);
+    DetectByteExtractData *bed = (DetectByteExtractData *)sm->ctx;
+    FAIL_IF(bed->nbytes != 4);
+    FAIL_IF(bed->offset != 0);
+    FAIL_IF(strcmp(bed->name, "two") != 0);
+    FAIL_IF(bed->flags != (DETECT_BYTE_EXTRACT_FLAG_RELATIVE | DETECT_BYTE_EXTRACT_FLAG_STRING));
+    FAIL_IF(bed->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE);
+    FAIL_IF(bed->base != DETECT_BYTE_EXTRACT_BASE_HEX);
+    FAIL_IF(bed->align_value != 0);
+    FAIL_IF(bed->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT);
 
- end:
-    SigGroupCleanup(de_ctx);
-    SigCleanSignatures(de_ctx);
     DetectEngineCtxFree(de_ctx);
-
-    return result;
+    PASS;
 }
 
 static int DetectByteExtractTest37(void)
@@ -4159,134 +4116,81 @@ static int DetectByteExtractTest58(void)
 
 static int DetectByteExtractTest59(void)
 {
-    DetectEngineCtx *de_ctx = NULL;
-    int result = 0;
-    Signature *s = NULL;
-    SigMatch *sm = NULL;
-    DetectContentData *cd = NULL;
-    DetectByteExtractData *bed1 = NULL;
-    DetectBytejumpData *bjd = NULL;
-    DetectIsdataatData *isdd = NULL;
-
-    de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-
+    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
-    s = de_ctx->sig_list = SigInit(de_ctx, "alert tcp any any -> any any "
-                                   "(msg:\"Testing bytejump_body\"; "
-                                   "content:\"one\"; "
-                                   "byte_extract:4,0,two,string,hex; "
-                                   "byte_extract:4,0,three,string,hex; "
-                                   "byte_jump: 2,two; "
-                                   "byte_jump: 3,three; "
-                                   "isdataat: three,relative; "
-                                   "sid:1;)");
-    if (de_ctx->sig_list == NULL) {
-        result = 0;
-        goto end;
-    }
 
-    if (s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH] == NULL) {
-        result = 0;
-        goto end;
-    }
+    Signature *s = DetectEngineAppendSig(de_ctx, "alert tcp any any -> any any ("
+                                                 "content:\"one\"; "
+                                                 "byte_extract:4,0,two,string,hex; "
+                                                 "byte_extract:4,0,three,string,hex; "
+                                                 "byte_jump: 2,two; "
+                                                 "byte_jump: 3,three; "
+                                                 "isdataat: three,relative; "
+                                                 "sid:1;)");
+    FAIL_IF_NULL(s);
 
-    sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
-    if (sm->type != DETECT_CONTENT) {
-        result = 0;
-        goto end;
-    }
-    cd = (DetectContentData *)sm->ctx;
-    if (cd->flags & DETECT_CONTENT_RAWBYTES ||
-        strncmp((char *)cd->content, "one", cd->content_len) != 0 ||
-        cd->flags & DETECT_CONTENT_NOCASE ||
-        cd->flags & DETECT_CONTENT_WITHIN ||
-        cd->flags & DETECT_CONTENT_DISTANCE ||
-        cd->flags & DETECT_CONTENT_FAST_PATTERN ||
-        cd->flags & DETECT_CONTENT_RELATIVE_NEXT ||
-        cd->flags & DETECT_CONTENT_NEGATED ) {
-        printf("one failed\n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(s->init_data->smlists_tail[DETECT_SM_LIST_PMATCH]);
+    SigMatch *sm = s->init_data->smlists[DETECT_SM_LIST_PMATCH];
+    FAIL_IF(sm->type != DETECT_CONTENT);
 
-    sm = sm->next;
-    if (sm->type != DETECT_BYTE_EXTRACT) {
-        result = 0;
-        goto end;
-    }
-    bed1 = (DetectByteExtractData *)sm->ctx;
-    if (bed1->nbytes != 4 ||
-        bed1->offset != 0 ||
-        strcmp(bed1->name, "two") != 0 ||
-        bed1->flags != DETECT_BYTE_EXTRACT_FLAG_STRING ||
-        bed1->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE ||
-        bed1->base != DETECT_BYTE_EXTRACT_BASE_HEX ||
-        bed1->align_value != 0 ||
-        bed1->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT) {
-        goto end;
-    }
-    if (bed1->local_id != 0) {
-        result = 0;
-        goto end;
-    }
+    DetectContentData *cd = (DetectContentData *)sm->ctx;
+    FAIL_IF(cd->flags & DETECT_CONTENT_RAWBYTES);
+    FAIL_IF(strncmp((char *)cd->content, "one", cd->content_len) != 0);
+    FAIL_IF(cd->flags & DETECT_CONTENT_NOCASE);
+    FAIL_IF(cd->flags & DETECT_CONTENT_WITHIN);
+    FAIL_IF(cd->flags & DETECT_CONTENT_DISTANCE);
+    FAIL_IF(cd->flags & DETECT_CONTENT_FAST_PATTERN);
+    FAIL_IF(cd->flags & DETECT_CONTENT_RELATIVE_NEXT);
+    FAIL_IF(cd->flags & DETECT_CONTENT_NEGATED);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTE_EXTRACT) {
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTE_EXTRACT);
+
+    DetectByteExtractData *bed1 = (DetectByteExtractData *)sm->ctx;
+    FAIL_IF(bed1->nbytes != 4);
+    FAIL_IF(bed1->offset != 0);
+    FAIL_IF(strcmp(bed1->name, "two") != 0);
+    FAIL_IF(bed1->flags != DETECT_BYTE_EXTRACT_FLAG_STRING);
+    FAIL_IF(bed1->endian != DETECT_BYTE_EXTRACT_ENDIAN_NONE);
+    FAIL_IF(bed1->base != DETECT_BYTE_EXTRACT_BASE_HEX);
+    FAIL_IF(bed1->align_value != 0);
+    FAIL_IF(bed1->multiplier_value != DETECT_BYTE_EXTRACT_MULTIPLIER_DEFAULT);
+
+    FAIL_IF(bed1->local_id != 0);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTEJUMP) {
-        result = 0;
-        goto end;
-    }
-    bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 0) {
-        printf("three failed\n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTE_EXTRACT);
 
     sm = sm->next;
-    if (sm->type != DETECT_BYTEJUMP) {
-        result = 0;
-        goto end;
-    }
-    bjd = (DetectBytejumpData *)sm->ctx;
-    if (bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR || bjd->offset != 1) {
-        printf("four failed\n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTEJUMP);
+
+    DetectBytejumpData *bjd = (DetectBytejumpData *)sm->ctx;
+    FAIL_IF(bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR);
+    FAIL_IF(bjd->offset != 0);
 
     sm = sm->next;
-    if (sm->type != DETECT_ISDATAAT) {
-        result = 0;
-        goto end;
-    }
-    isdd = (DetectIsdataatData *)sm->ctx;
-    if (isdd->flags != (ISDATAAT_OFFSET_VAR |
-                        ISDATAAT_RELATIVE) ||
-        isdd->dataat != 1) {
-        printf("isdataat failed\n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_BYTEJUMP);
 
-    if (sm->next != NULL)
-        goto end;
+    bjd = (DetectBytejumpData *)sm->ctx;
+    FAIL_IF(bjd->flags != DETECT_BYTEJUMP_OFFSET_VAR);
+    FAIL_IF(bjd->offset != 1);
 
-    result = 1;
+    sm = sm->next;
+    FAIL_IF_NULL(sm);
+    FAIL_IF(sm->type != DETECT_ISDATAAT);
+    DetectIsdataatData *isdd = (DetectIsdataatData *)sm->ctx;
+    FAIL_IF(isdd->flags != (ISDATAAT_OFFSET_VAR | ISDATAAT_RELATIVE));
+    FAIL_IF(isdd->dataat != 1);
 
- end:
-    SigGroupCleanup(de_ctx);
-    SigCleanSignatures(de_ctx);
+    FAIL_IF(sm->next != NULL);
     DetectEngineCtxFree(de_ctx);
 
-    return result;
+    PASS;
 }
 
 static int DetectByteExtractTest60(void)

From 3ba8e2d3ea616ab3efdca6391bff3aa334bea94a Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 13 Dec 2023 08:45:06 +0100
Subject: [PATCH 287/462] detect/byte: remove unneeded SIG_FLAG_APPLAYER sets

Flag will be set during list(s) setup if needed.
---
 src/detect-byte-extract.c | 3 ---
 src/detect-bytemath.c     | 3 ---
 2 files changed, 6 deletions(-)

diff --git a/src/detect-byte-extract.c b/src/detect-byte-extract.c
index 379d3dce015f..117cce597534 100644
--- a/src/detect-byte-extract.c
+++ b/src/detect-byte-extract.c
@@ -563,7 +563,6 @@ static int DetectByteExtractSetup(DetectEngineCtx *de_ctx, Signature *s, const c
 
         if (DetectSignatureSetAppProto(s, ALPROTO_DCERPC) < 0)
             goto error;
-        s->flags |= SIG_FLAG_APPLAYER;
 
     } else if (data->flags & DETECT_BYTE_EXTRACT_FLAG_RELATIVE) {
         prev_pm = DetectGetLastSMFromLists(s,
@@ -576,8 +575,6 @@ static int DetectByteExtractSetup(DetectEngineCtx *de_ctx, Signature *s, const c
             sm_list = SigMatchListSMBelongsTo(s, prev_pm);
             if (sm_list < 0)
                 goto error;
-            if (sm_list != DETECT_SM_LIST_PMATCH)
-                s->flags |= SIG_FLAG_APPLAYER;
         }
 
     } else {
diff --git a/src/detect-bytemath.c b/src/detect-bytemath.c
index 8909085390e4..49c2989f7dc9 100644
--- a/src/detect-bytemath.c
+++ b/src/detect-bytemath.c
@@ -325,7 +325,6 @@ static int DetectByteMathSetup(DetectEngineCtx *de_ctx, Signature *s, const char
 
         if (DetectSignatureSetAppProto(s, ALPROTO_DCERPC) < 0)
             goto error;
-        s->flags |= SIG_FLAG_APPLAYER;
 
     } else if (data->flags & DETECT_BYTEMATH_FLAG_RELATIVE) {
         prev_pm = DetectGetLastSMFromLists(s, DETECT_CONTENT, DETECT_PCRE,
@@ -338,8 +337,6 @@ static int DetectByteMathSetup(DetectEngineCtx *de_ctx, Signature *s, const char
             sm_list = SigMatchListSMBelongsTo(s, prev_pm);
             if (sm_list < 0)
                 goto error;
-            if (sm_list != DETECT_SM_LIST_PMATCH)
-                s->flags |= SIG_FLAG_APPLAYER;
         }
 
     } else {

From 7fa8bbfe43f396215238e7d8a2b7ce94a22560bc Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 30 Nov 2023 18:55:13 -0300
Subject: [PATCH 288/462] pgsql: extract length validation into function

This is called so many times that it seems to make sense that we use a
function for this.
---
 rust/src/pgsql/parser.rs | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index bed3682bb43d..1cfa19da17b1 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -37,6 +37,10 @@ pub const PGSQL_DUMMY_PROTO_MAJOR: u16 = 1234; // 0x04d2
 pub const PGSQL_DUMMY_PROTO_MINOR_SSL: u16 = 5679; //0x162f
 pub const _PGSQL_DUMMY_PROTO_MINOR_GSSAPI: u16 = 5680; // 0x1630
 
+fn parse_length(i: &[u8]) -> IResult<&[u8], u32> {
+    verify(be_u32, |&x| x >= PGSQL_LENGTH_FIELD)(i)
+}
+
 #[derive(Debug, PartialEq, Eq)]
 pub enum PgsqlParameters {
     // startup parameters
@@ -564,7 +568,7 @@ fn parse_sasl_initial_response_payload(i: &[u8]) -> IResult<&[u8], (SASLAuthenti
 
 pub fn parse_sasl_initial_response(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'p')(i)?;
-    let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, payload) = map_parser(take(length - PGSQL_LENGTH_FIELD), parse_sasl_initial_response_payload)(i)?;
     Ok((i, PgsqlFEMessage::SASLInitialResponse(
                 SASLInitialResponsePacket {
@@ -578,7 +582,7 @@ pub fn parse_sasl_initial_response(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
 
 pub fn parse_sasl_response(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'p')(i)?;
-    let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, payload) = take(length - PGSQL_LENGTH_FIELD)(i)?;
     let resp = PgsqlFEMessage::SASLResponse(
         RegularPacket {
@@ -638,7 +642,7 @@ pub fn pgsql_parse_startup_packet(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
 // Password can be encrypted or in cleartext
 pub fn parse_password_message(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'p')(i)?;
-    let (i, length) = verify(be_u32, |&x| x >= PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, password) = map_parser(
         take(length - PGSQL_LENGTH_FIELD),
         take_until1("\x00")
@@ -653,7 +657,7 @@ pub fn parse_password_message(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
 
 fn parse_simple_query(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'Q')(i)?;
-    let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, query) = map_parser(take(length - PGSQL_LENGTH_FIELD), take_until1("\x00"))(i)?;
     Ok((i, PgsqlFEMessage::SimpleQuery(RegularPacket {
         identifier,
@@ -664,7 +668,7 @@ fn parse_simple_query(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
 
 fn parse_terminate_message(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'X')(i)?;
-    let (i, length) = verify(be_u32, |&x| x == PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     Ok((i, PgsqlFEMessage::Terminate(TerminationMessage { identifier, length })))
 }
 
@@ -772,7 +776,7 @@ fn pgsql_parse_authentication_message<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pgsq
 
 fn parse_parameter_status_message(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'S')(i)?;
-    let (i, length) = verify(be_u32, |&x| x >= PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, param) = map_parser(take(length - PGSQL_LENGTH_FIELD), pgsql_parse_generic_parameter)(i)?;
     Ok((i, PgsqlBEMessage::ParameterStatus(ParameterStatusMessage {
         identifier,
@@ -803,7 +807,7 @@ fn parse_backend_key_data_message(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
 
 fn parse_command_complete(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'C')(i)?;
-    let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+    let (i, length) = parse_length(i)?;
     let (i, payload) = map_parser(take(length - PGSQL_LENGTH_FIELD), take_until("\x00"))(i)?;
     Ok((i, PgsqlBEMessage::CommandComplete(RegularPacket {
         identifier,

From 7dcc2e7a713de20af0dbc1e935ec791cba2e6f95 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 11 Dec 2023 17:26:31 -0300
Subject: [PATCH 289/462] doc/eve-format: break pgsql section to char limit

---
 doc/userguide/output/eve/eve-json-format.rst | 67 ++++++++++++++------
 1 file changed, 47 insertions(+), 20 deletions(-)

diff --git a/doc/userguide/output/eve/eve-json-format.rst b/doc/userguide/output/eve/eve-json-format.rst
index 0c7bc867aeb6..f33f205e0107 100644
--- a/doc/userguide/output/eve/eve-json-format.rst
+++ b/doc/userguide/output/eve/eve-json-format.rst
@@ -2432,13 +2432,17 @@ Example of HTTP2 logging, of a request and response:
 Event type: PGSQL
 -----------------
 
-PGSQL eve-logs reflect the bidirectional nature of the protocol transactions. Each PGSQL event lists at most one
-"Request" message field and one or more "Response" messages.
+PGSQL eve-logs reflect the bidirectional nature of the protocol transactions.
+Each PGSQL event lists at most one "Request" message field and one or more
+"Response" messages.
 
-The PGSQL parser merges individual messages into one EVE output item if they belong to the same transaction. In such cases, the source and destination information (IP/port) reflect the direction of the initial request, but contain messages from both sides.
+The PGSQL parser merges individual messages into one EVE output item if they
+belong to the same transaction. In such cases, the source and destination
+information (IP/port) reflect the direction of the initial request, but contain
+messages from both sides.
 
-
-Example of ``pgsql`` event for a SimpleQuery transaction complete with request with a ``SELECT`` statement and its response::
+Example of ``pgsql`` event for a SimpleQuery transaction complete with request
+with a ``SELECT`` statement and its response::
 
   {
     "timestamp": "2021-11-24T16:56:24.403417+0000",
@@ -2464,51 +2468,74 @@ Example of ``pgsql`` event for a SimpleQuery transaction complete with request w
     }
   }
 
-While on the wire PGSQL messages follow basically two types (startup messages and regular messages), those may have different subfields and/or meanings, based on the message type. Messages are logged based on their type and relevant fields.
+While on the wire PGSQL messages follow basically two types (startup messages
+and regular messages), those may have different subfields and/or meanings, based
+on the message type. Messages are logged based on their type and relevant fields.
 
-We list a few possible message types and what they mean in Suricata. For more details on message types and formats as well as what each message and field mean for PGSQL, check  `PostgreSQL's official documentation <https://www.postgresql.org/docs/14/protocol-message-formats.html>`_.
+We list a few possible message types and what they mean in Suricata. For more
+details on message types and formats as well as what each message and field mean
+for PGSQL, check `PostgreSQL's official documentation <https://www.postgresql.org
+/docs/14/protocol-message-formats.html>`_.
 
 Fields
 ~~~~~~
 
 * "tx_id": internal transaction id.
-* "request":  each PGSQL transaction may have up to one request message. The possible messages will be described in another section.
-* "response": even when there are several "Response" messages, there is one ``response`` field that summarizes all responses for that transaction. The possible messages will be described in another section.
+* "request":  each PGSQL transaction may have up to one request message. The
+  possible messages will be described in another section.
+* "response": even when there are several "Response" messages, there is one
+  ``response`` field that summarizes all responses for that transaction. The
+  possible messages will be described in another section.
 
 Request Messages
 ~~~~~~~~~~~~~~~~
 
 Some of the possible request messages are:
 
-* "startup_message": message sent by a frontend/client process to start a new PostgreSQL connection
-* "password_message": if password output for PGSQL is enabled in suricata.yaml, carries the password sent during Authentication phase
-* "simple_query": issued SQL command during simple query subprotocol. PostgreSQL identifies specific sets of commands that change the set of expected messages to be exchanged as subprotocols.
-* "message": frontend responses which do not have meaningful payloads are logged like this, where the field value is the message type
+* "startup_message": message sent by a frontend/client process to start a new
+  PostgreSQL connection
+* "password_message": if password output for PGSQL is enabled in suricata.yaml,
+  carries the password sent during Authentication phase
+* "simple_query": issued SQL command during simple query subprotocol. PostgreSQL
+  identifies specific sets of commands that change the set of expected messages
+  to be exchanged as subprotocols.
+* "message": frontend responses which do not have meaningful payloads are logged
+  like this, where the field value is the message type
 
-There are several different authentication messages possible, based on selected authentication method. (e.g. the SASL authentication will have a set of authentication messages different from when ``md5`` authentication is chosen).
+There are several different authentication messages possible, based on selected
+authentication method. (e.g. the SASL authentication will have a set of
+authentication messages different from when ``md5`` authentication is chosen).
 
 Response Messages
 ~~~~~~~~~~~~~~~~~
 
 Some of the possible request messages are:
 
-* "authentication_sasl_final": final SCRAM ``server-final-message``, as explained at https://www.postgresql.org/docs/14/sasl-authentication.html#SASL-SCRAM-SHA-256
-* "message": Backend responses which do not have meaningful payloads are logged like this, where the field value is the message type
+* "authentication_sasl_final": final SCRAM ``server-final-message``, as explained
+  at https://www.postgresql.org/docs/14/sasl-authentication.html#SASL-SCRAM-SHA-256
+* "message": Backend responses which do not have meaningful payloads are logged
+  like this, where the field value is the message type
 * "error_response"
 * "notice_response"
 * "notification_response"
 * "authentication_md5_password": a string with the ``md5`` salt value
 * "parameter_status": logged as an array
 * "backend_key_data"
-* "data_rows": integer. When one or many ``DataRow`` messages are parsed, the total returned rows
-* "data_size": in bytes. When one or many ``DataRow`` messages are parsed, the total size in bytes of the data returned
+* "data_rows": integer. When one or many ``DataRow`` messages are parsed, the
+  total returned rows
+* "data_size": in bytes. When one or many ``DataRow`` messages are parsed, the
+  total size in bytes of the data returned
 * "command_completed": string. Informs the command just completed by the backend
-* "ssl_accepted": bool. With this event, the initial PGSQL SSL Handshake negotiation is complete in terms of tracking and logging. The session will be upgraded to use TLS encryption
+* "ssl_accepted": bool. With this event, the initial PGSQL SSL Handshake
+  negotiation is complete in terms of tracking and logging. The session will be
+  upgraded to use TLS encryption
 
 Examples
 ~~~~~~~~
 
-The two ``pgsql`` events in this example represent a rejected ``SSL handshake`` and a following connection request where the authentication method indicated by the backend was ``md5``::
+The two ``pgsql`` events in this example represent a rejected ``SSL handshake``
+and a following connection request where the authentication method indicated by
+the backend was ``md5``::
 
   {
     "timestamp": "2021-11-24T16:56:19.435242+0000",

From 30ac77ce65ece4a84ebd5cd64e3f741cd9b0c1bf Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 11 Dec 2023 17:10:13 -0300
Subject: [PATCH 290/462] pgsql: add cancel request message

A CanceldRequest can occur after any query request, and is sent over a
new connection, leading to a new flow. It won't take any reply, but, if
processed by the backend, will lead to an ErrorResponse.

Task #6577
---
 doc/userguide/output/eve/eve-json-format.rst | 96 ++++++++++++++++++++
 etc/schema.json                              |  6 ++
 rust/src/pgsql/logger.rs                     |  8 ++
 rust/src/pgsql/parser.rs                     | 66 ++++++++++++--
 rust/src/pgsql/pgsql.rs                      |  3 +
 5 files changed, 171 insertions(+), 8 deletions(-)

diff --git a/doc/userguide/output/eve/eve-json-format.rst b/doc/userguide/output/eve/eve-json-format.rst
index f33f205e0107..3bb7894dc4b4 100644
--- a/doc/userguide/output/eve/eve-json-format.rst
+++ b/doc/userguide/output/eve/eve-json-format.rst
@@ -2501,6 +2501,11 @@ Some of the possible request messages are:
   to be exchanged as subprotocols.
 * "message": frontend responses which do not have meaningful payloads are logged
   like this, where the field value is the message type
+* ``"message": "cancel_request"``: sent after a query, when the frontend
+  attempts to cancel said query. This message is sent over a different port,
+  thus bring shown as a different flow. It has no direct answer from the
+  backend, but if successful will lead to an ``ErrorResponse`` in the
+  transaction where the query was sent.
 
 There are several different authentication messages possible, based on selected
 authentication method. (e.g. the SASL authentication will have a set of
@@ -2590,6 +2595,97 @@ the backend was ``md5``::
     }
   }
 
+``AuthenticationOk``: a response indicating that the connection was successfully
+established.::
+
+  {
+    "pgsql": {
+      "tx_id": 3,
+      "response": {
+        "message": "authentication_ok",
+        "parameter_status": [
+          {
+            "application_name": "psql"
+          },
+          {
+            "client_encoding": "UTF8"
+          },
+          {
+            "date_style": "ISO, MDY"
+          },
+          {
+            "integer_datetimes": "on"
+          },
+          {
+            "interval_style": "postgres"
+          },
+          {
+            "is_superuser": "on"
+          },
+          {
+            "server_encoding": "UTF8"
+          },
+          {
+            "server_version": "13.6 (Debian 13.6-1.pgdg110+1)"
+          },
+          {
+            "session_authorization": "rules"
+          },
+          {
+            "standard_conforming_strings": "on"
+          },
+          {
+            "time_zone": "Etc/UTC"
+          }
+        ],
+        "process_id": 28954,
+        "secret_key": 889887985
+      }
+    }
+  }
+
+.. note::
+   In Suricata, the ``AuthenticationOk`` message is also where the backend's
+   ``process_id`` and ``secret_key`` are logged. These must be sent by the
+   frontend when it issues a ``CancelRequest`` message (seen below).
+
+A ``CancelRequest`` message::
+
+   {
+      "timestamp": "2023-12-07T15:46:56.971150+0000",
+      "flow_id": 775771889500133,
+      "event_type": "pgsql",
+      "src_ip": "100.88.2.140",
+      "src_port": 39706,
+      "dest_ip": "100.96.199.113",
+      "dest_port": 5432,
+      "proto": "TCP",
+      "pkt_src": "stream (flow timeout)",
+      "pgsql": {
+        "tx_id": 1,
+        "request": {
+          "message": "cancel_request",
+          "process_id": 28954,
+          "secret_key": 889887985
+        }
+      }
+   }
+
+.. note::
+   As the ``CancelRequest`` message is sent over a new connection, the way to
+   correlate it with the proper frontend/flow from which it originates is by
+   querying on ``process_id`` and ``secret_key`` seen in the
+   ``AuthenticationOk`` event.
+
+References:
+  * `PostgreSQL protocol - Canceling Requests in Progress`_
+  * `PostgreSQL message format - BackendKeyData`_
+
+.. _PostgreSQL protocol - Canceling Requests in Progress: https://www.postgresql
+   .org/docs/current/protocol-flow.html#PROTOCOL-FLOW-CANCELING-REQUESTS
+.. _PostgreSQL message format - BackendKeyData: https://www.postgresql.org/docs
+   /current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-BACKENDKEYDATA
+
 
 Event type: IKE
 ---------------
diff --git a/etc/schema.json b/etc/schema.json
index c194017ddf6f..008c5a8c00f7 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -2805,6 +2805,9 @@
                         "password_message": {
                             "type": "string"
                         },
+                        "process_id": {
+                            "type": "integer"
+                        },
                         "protocol_version": {
                             "type": "string"
                         },
@@ -2817,6 +2820,9 @@
                         "sasl_response": {
                             "type": "string"
                         },
+                        "secret_key": {
+                            "type": "integer"
+                        },
                         "simple_query": {
                             "type": "string"
                         },
diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index 4a6f24880252..51f6cb60993b 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -94,6 +94,14 @@ fn log_request(req: &PgsqlFEMessage, flags: u32) -> Result<JsonBuilder, JsonErro
         }) => {
             js.set_string_from_bytes(req.to_str(), payload)?;
         }
+        PgsqlFEMessage::CancelRequest(CancelRequestMessage {
+            pid,
+            backend_key,
+        }) => {
+            js.set_string("message", "cancel_request")?;
+            js.set_uint("process_id", (*pid).into())?;
+            js.set_uint("secret_key", (*backend_key).into())?;
+        }
         PgsqlFEMessage::Terminate(TerminationMessage {
             identifier: _,
             length: _,
diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 1cfa19da17b1..345f30a79872 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -34,6 +34,7 @@ use nom7::{Err, IResult};
 pub const PGSQL_LENGTH_FIELD: u32 = 4;
 
 pub const PGSQL_DUMMY_PROTO_MAJOR: u16 = 1234; // 0x04d2
+pub const PGSQL_DUMMY_PROTO_CANCEL_REQUEST: u16 = 5678; // 0x162e
 pub const PGSQL_DUMMY_PROTO_MINOR_SSL: u16 = 5679; //0x162f
 pub const _PGSQL_DUMMY_PROTO_MINOR_GSSAPI: u16 = 5680; // 0x1630
 
@@ -315,6 +316,12 @@ pub struct TerminationMessage {
     pub length: u32,
 }
 
+#[derive(Debug, PartialEq, Eq)]
+pub struct CancelRequestMessage {
+    pub pid: u32,
+    pub backend_key: u32,
+}
+
 #[derive(Debug, PartialEq, Eq)]
 pub enum PgsqlFEMessage {
     SSLRequest(DummyStartupPacket),
@@ -323,6 +330,7 @@ pub enum PgsqlFEMessage {
     SASLInitialResponse(SASLInitialResponsePacket),
     SASLResponse(RegularPacket),
     SimpleQuery(RegularPacket),
+    CancelRequest(CancelRequestMessage),
     Terminate(TerminationMessage),
     UnknownMessageType(RegularPacket),
 }
@@ -336,6 +344,7 @@ impl PgsqlFEMessage {
             PgsqlFEMessage::SASLInitialResponse(_) => "sasl_initial_response",
             PgsqlFEMessage::SASLResponse(_) => "sasl_response",
             PgsqlFEMessage::SimpleQuery(_) => "simple_query",
+            PgsqlFEMessage::CancelRequest(_) => "cancel_request",
             PgsqlFEMessage::Terminate(_) => "termination_message",
             PgsqlFEMessage::UnknownMessageType(_) => "unknown_message_type",
         }
@@ -611,16 +620,20 @@ pub fn pgsql_parse_startup_packet(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
             },
             PGSQL_DUMMY_PROTO_MAJOR => {
                 let (b, proto_major) = be_u16(b)?;
-                let (b, proto_minor) = all_consuming(be_u16)(b)?;
-                let _message = match proto_minor {
-                    PGSQL_DUMMY_PROTO_MINOR_SSL => (len, proto_major, proto_minor),
+                let (b, proto_minor) = be_u16(b)?;
+                let (b, message) = match proto_minor {
+                    PGSQL_DUMMY_PROTO_CANCEL_REQUEST => {
+                        parse_cancel_request(b)?
+                    },
+                    PGSQL_DUMMY_PROTO_MINOR_SSL => (b, PgsqlFEMessage::SSLRequest(DummyStartupPacket{
+                        length: len,
+                        proto_major,
+                        proto_minor
+                    })),
                     _ => return Err(Err::Error(make_error(b, ErrorKind::Switch))),
                 };
 
-                (b, PgsqlFEMessage::SSLRequest(DummyStartupPacket{
-                    length: len,
-                    proto_major,
-                    proto_minor}))
+                (b, message)
             }
             _ => return Err(Err::Error(make_error(b, ErrorKind::Switch))),
         };
@@ -666,6 +679,15 @@ fn parse_simple_query(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     })))
 }
 
+fn parse_cancel_request(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
+    let (i, pid) = be_u32(i)?;
+    let (i, backend_key) = be_u32(i)?;
+    Ok((i, PgsqlFEMessage::CancelRequest(CancelRequestMessage {
+        pid,
+        backend_key,
+    })))
+}
+
 fn parse_terminate_message(i: &[u8]) -> IResult<&[u8], PgsqlFEMessage> {
     let (i, identifier) = verify(be_u8, |&x| x == b'X')(i)?;
     let (i, length) = parse_length(i)?;
@@ -1262,9 +1284,37 @@ mod tests {
         let result = parse_request(&buf[0..3]);
         assert!(result.is_err());
 
-        // TODO add other messages
     }
 
+    #[test]
+    fn test_cancel_request_message() {
+        // A cancel request message
+        let buf: &[u8] = &[
+            0x00, 0x00, 0x00, 0x10, // length: 16 (fixed)
+            0x04, 0xd2, 0x16, 0x2e, // 1234.5678 - identifies a cancel request
+            0x00, 0x00, 0x76, 0x31, // PID: 30257
+            0x23, 0x84, 0xf7, 0x2d]; // Backend key: 595916589
+        let result = parse_cancel_request(buf);
+        assert!(result.is_ok());
+
+        let result = parse_cancel_request(&buf[0..3]);
+        assert!(result.is_err());
+
+        let result = pgsql_parse_startup_packet(buf);
+        assert!(result.is_ok());
+
+        let fail_result = pgsql_parse_startup_packet(&buf[0..3]);
+        assert!(fail_result.is_err());
+
+        let result = parse_request(buf);
+        assert!(result.is_ok());
+
+        let fail_result = parse_request(&buf[0..3]);
+        assert!(fail_result.is_err());
+    }
+
+
+
     #[test]
     fn test_parse_error_response_code() {
         let buf: &[u8] = &[0x43, 0x32, 0x38, 0x30, 0x30, 0x30, 0x00];
diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index 8b9b12c4d694..8ca7f4de6a61 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -117,6 +117,7 @@ pub enum PgsqlStateProgress {
     DataRowReceived,
     CommandCompletedReceived,
     ErrorMessageReceived,
+    CancelRequestReceived,
     ConnectionTerminated,
     #[cfg(test)]
     UnknownState,
@@ -229,6 +230,7 @@ impl PgsqlState {
             || self.state_progress == PgsqlStateProgress::SimpleQueryReceived
             || self.state_progress == PgsqlStateProgress::SSLRequestReceived
             || self.state_progress == PgsqlStateProgress::ConnectionTerminated
+            || self.state_progress == PgsqlStateProgress::CancelRequestReceived
         {
             let tx = self.new_tx();
             self.transactions.push_back(tx);
@@ -280,6 +282,7 @@ impl PgsqlState {
 
                 // Important to keep in mind that: "In simple Query mode, the format of retrieved values is always text, except when the given command is a FETCH from a cursor declared with the BINARY option. In that case, the retrieved values are in binary format. The format codes given in the RowDescription message tell which format is being used." (from pgsql official documentation)
             }
+            PgsqlFEMessage::CancelRequest(_) => Some(PgsqlStateProgress::CancelRequestReceived),
             PgsqlFEMessage::Terminate(_) => {
                 SCLogDebug!("Match: Terminate message");
                 Some(PgsqlStateProgress::ConnectionTerminated)

From bba3d4fc6375b62b774ad584a953442da9f4cbde Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 11 Dec 2023 17:55:37 -0300
Subject: [PATCH 291/462] userguide/eve: explain pgsql requests & responses

Add a more visible explanation of that requests, responses, frontend and
and backend are, in Pgsql context, to avoid having to repeat that over
different portions of the docs.
---
 doc/userguide/output/eve/eve-json-format.rst | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/doc/userguide/output/eve/eve-json-format.rst b/doc/userguide/output/eve/eve-json-format.rst
index 3bb7894dc4b4..3184426b30cc 100644
--- a/doc/userguide/output/eve/eve-json-format.rst
+++ b/doc/userguide/output/eve/eve-json-format.rst
@@ -2490,22 +2490,22 @@ Fields
 Request Messages
 ~~~~~~~~~~~~~~~~
 
-Some of the possible request messages are:
+Requests are sent by the frontend (client), which would be the source of a pgsql
+flow. Some of the possible request messages are:
 
-* "startup_message": message sent by a frontend/client process to start a new
-  PostgreSQL connection
+* "startup_message": message sent to start a new PostgreSQL connection
 * "password_message": if password output for PGSQL is enabled in suricata.yaml,
   carries the password sent during Authentication phase
 * "simple_query": issued SQL command during simple query subprotocol. PostgreSQL
   identifies specific sets of commands that change the set of expected messages
   to be exchanged as subprotocols.
-* "message": frontend responses which do not have meaningful payloads are logged
-  like this, where the field value is the message type
 * ``"message": "cancel_request"``: sent after a query, when the frontend
   attempts to cancel said query. This message is sent over a different port,
   thus bring shown as a different flow. It has no direct answer from the
   backend, but if successful will lead to an ``ErrorResponse`` in the
   transaction where the query was sent.
+* "message": requests which do not have meaningful payloads are logged like this,
+  where the field value is the message type
 
 There are several different authentication messages possible, based on selected
 authentication method. (e.g. the SASL authentication will have a set of
@@ -2514,7 +2514,8 @@ authentication messages different from when ``md5`` authentication is chosen).
 Response Messages
 ~~~~~~~~~~~~~~~~~
 
-Some of the possible request messages are:
+Responses are sent by the backend (server), which would be the destination of a
+pgsql flow. Some of the possible request messages are:
 
 * "authentication_sasl_final": final SCRAM ``server-final-message``, as explained
   at https://www.postgresql.org/docs/14/sasl-authentication.html#SASL-SCRAM-SHA-256

From 467c3f2c641d50703723a8fd8dffec1f05e5af22 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Mon, 11 Dec 2023 17:10:54 -0300
Subject: [PATCH 292/462] schema: apply clang formatting changes

---
 etc/schema.json | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/etc/schema.json b/etc/schema.json
index 008c5a8c00f7..ed8adcf49fa0 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -561,7 +561,7 @@
                 "renewal_time": {
                     "type": "integer"
                 },
-                "requested_ip":{
+                "requested_ip": {
                     "type": "string"
                 },
                 "subnet_mask": {
@@ -570,7 +570,7 @@
                 "type": {
                     "type": "string"
                 },
-                "vendor_class_identifier":{
+                "vendor_class_identifier": {
                     "type": "string"
                 },
                 "dns_servers": {
@@ -2977,7 +2977,8 @@
             "optional": true,
             "properties": {
                 "cyu": {
-                    "description": "ja3-like fingerprint for versions of QUIC before standardization",
+                    "description":
+                            "ja3-like fingerprint for versions of QUIC before standardization",
                     "type": "array",
                     "minItems": 1,
                     "items": {

From 4bcdc79ed8366053947a25cd0f86ceba33e5495d Mon Sep 17 00:00:00 2001
From: Philippe Antoine <contact@catenacyber.fr>
Date: Mon, 20 Nov 2023 14:54:45 +0100
Subject: [PATCH 293/462] stats: always use tcp/udp prefix

Even when on detection-only mode.
So that we always have enip_tcp and enip_udp in stats
and never just `enip`.

Ticket: 6304
---
 src/app-layer.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/app-layer.c b/src/app-layer.c
index 3625e87e9ed6..102319042bcc 100644
--- a/src/app-layer.c
+++ b/src/app-layer.c
@@ -1061,16 +1061,18 @@ void AppLayerSetupCounters(void)
     for (uint8_t p = 0; p < IPPROTOS_MAX; p++) {
         const uint8_t ipproto = ipprotos[p];
         const uint8_t ipproto_map = FlowGetProtoMapping(ipproto);
-        const uint8_t other_ipproto = ipproto == IPPROTO_TCP ? IPPROTO_UDP : IPPROTO_TCP;
         const char *ipproto_suffix = (ipproto == IPPROTO_TCP) ? "_tcp" : "_udp";
+        uint8_t ipprotos_all[256 / 8];
 
         for (AppProto alproto = 0; alproto < ALPROTO_MAX; alproto++) {
             if (alprotos[alproto] == 1) {
                 const char *tx_str = "app_layer.tx.";
                 const char *alproto_str = AppLayerGetProtoName(alproto);
 
-                if (AppLayerParserProtoIsRegistered(ipproto, alproto) &&
-                        AppLayerParserProtoIsRegistered(other_ipproto, alproto)) {
+                memset(ipprotos_all, 0, sizeof(ipprotos_all));
+                AppLayerProtoDetectSupportedIpprotos(alproto, ipprotos_all);
+                if ((ipprotos_all[IPPROTO_TCP / 8] & (1 << (IPPROTO_TCP % 8))) &&
+                        (ipprotos_all[IPPROTO_UDP / 8] & (1 << (IPPROTO_UDP % 8)))) {
                     snprintf(applayer_counter_names[ipproto_map][alproto].name,
                             sizeof(applayer_counter_names[ipproto_map][alproto].name),
                             "%s%s%s", str, alproto_str, ipproto_suffix);

From f714678d72146fac30e2bbfbb8a0df1a1689d13f Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 11 Sep 2023 09:51:24 +0200
Subject: [PATCH 294/462] schema: adds missing modbus field

./stats/app_layer/error/modbus
---
 etc/schema.json | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index ed8adcf49fa0..76d88ec4c8fb 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -3783,6 +3783,9 @@
                                 "krb5_udp": {
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
+                                "modbus": {
+                                    "$ref": "#/$defs/stats_applayer_error"
+                                },
                                 "mqtt": {
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },

From 3103505cb0fa87f18b63434a94c1b3814f5b8003 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 14 Dec 2023 11:31:37 +0100
Subject: [PATCH 295/462] stats: incr app-proto flow counter for detection-only

Ticket: 6633
---
 src/app-layer.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/src/app-layer.c b/src/app-layer.c
index 102319042bcc..e159b932afc3 100644
--- a/src/app-layer.c
+++ b/src/app-layer.c
@@ -510,6 +510,20 @@ static int TCPProtoDetect(ThreadVars *tv, TcpReassemblyThreadCtx *ra_ctx,
         if (r != 1) {
             StreamTcpUpdateAppLayerProgress(ssn, direction, data_len);
         }
+        if (r == 0) {
+            if (*alproto_otherdir == ALPROTO_UNKNOWN) {
+                TcpStream *opposing_stream;
+                if (*stream == &ssn->client) {
+                    opposing_stream = &ssn->server;
+                } else {
+                    opposing_stream = &ssn->client;
+                }
+                if (StreamTcpIsSetStreamFlagAppProtoDetectionCompleted(opposing_stream)) {
+                    // can happen in detection-only
+                    AppLayerIncFlowCounter(tv, f);
+                }
+            }
+        }
         if (r < 0) {
             goto parser_error;
         }

From 1afb485dfa253f4b409fa1acf0b7790cf1d2f09b Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Fri, 15 Dec 2023 13:57:01 -0300
Subject: [PATCH 296/462] pgsql: remove unused msg field

The `ConsolidatedDataRow` struct had a `length` field that wasn't truly
used.

Related to
Bug #6389
---
 rust/src/pgsql/logger.rs | 1 -
 rust/src/pgsql/parser.rs | 2 --
 rust/src/pgsql/pgsql.rs  | 1 -
 3 files changed, 4 deletions(-)

diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index 51f6cb60993b..d54b97b3e1a1 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -234,7 +234,6 @@ fn log_response(res: &PgsqlBEMessage, jb: &mut JsonBuilder) -> Result<(), JsonEr
         }
         PgsqlBEMessage::ConsolidatedDataRow(ConsolidatedDataRowPacket {
             identifier: _,
-            length: _,
             row_cnt,
             data_size,
         }) => {
diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 345f30a79872..3b8afcabf306 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -210,7 +210,6 @@ pub struct BackendKeyDataMessage {
 #[derive(Debug, PartialEq, Eq)]
 pub struct ConsolidatedDataRowPacket {
     pub identifier: u8,
-    pub length: u32,
     pub row_cnt: u16,
     pub data_size: u64,
 }
@@ -924,7 +923,6 @@ pub fn parse_consolidated_data_row(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
     Ok((i, PgsqlBEMessage::ConsolidatedDataRow(
                 ConsolidatedDataRowPacket {
                     identifier,
-                    length,
                     row_cnt: 1,
                     data_size: add_up_data_size(rows),
                 }
diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index 8ca7f4de6a61..d2d0a02f88da 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -487,7 +487,6 @@ impl PgsqlState {
                             let dummy_resp =
                                 PgsqlBEMessage::ConsolidatedDataRow(ConsolidatedDataRowPacket {
                                     identifier: b'D',
-                                    length: tx.get_row_cnt() as u32, // TODO this is ugly. We can probably get rid of `length` field altogether...
                                     row_cnt: tx.get_row_cnt(),
                                     data_size: tx.data_size, // total byte count of all data_row messages combined
                                 });

From 15ed51f9b87011025615245d89152da9c567f49b Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 14 Dec 2023 18:19:41 -0600
Subject: [PATCH 297/462] feature: provide a Rust binding to the feature API

As the feature module is not available for Rust unit tests, a mock
version is also provided.
---
 rust/src/feature.rs | 60 +++++++++++++++++++++++++++++++++++++++++++++
 rust/src/lib.rs     |  1 +
 2 files changed, 61 insertions(+)
 create mode 100644 rust/src/feature.rs

diff --git a/rust/src/feature.rs b/rust/src/feature.rs
new file mode 100644
index 000000000000..abd09669af11
--- /dev/null
+++ b/rust/src/feature.rs
@@ -0,0 +1,60 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+//! Rust bindings to the "feature" API.
+//!
+//! As this feature module is a binding to a Suricata C module it is
+//! not available to Rust unit tests. Instead when running Rust unit
+//! tests and "mock" version is provided that will return true for any
+//! feature starting with "true" and false for any other feature name.
+
+#[cfg(test)]
+mod mock {
+    /// Check for a feature returning true if found.
+    ///
+    /// This a "mock" variant of `requires` that will return true for
+    /// any feature starting with string `true`, and false for
+    /// anything else.
+    pub fn requires(feature: &str) -> bool {
+        return feature.starts_with("true");
+    }
+}
+
+#[cfg(not(test))]
+mod real {
+    use std::ffi::CString;
+    use std::os::raw::c_char;
+
+    extern "C" {
+        fn RequiresFeature(feature: *const c_char) -> bool;
+    }
+
+    /// Check for a feature returning true if found.
+    pub fn requires(feature: &str) -> bool {
+        if let Ok(feature) = CString::new(feature) {
+            unsafe { RequiresFeature(feature.as_ptr()) }
+        } else {
+            false
+        }
+    }
+}
+
+#[cfg(not(test))]
+pub use real::*;
+
+#[cfg(test)]
+pub use mock::*;
diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index da2859637783..84b82bde19f7 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -116,3 +116,4 @@ pub mod plugin;
 pub mod lzma;
 pub mod util;
 pub mod ffi;
+pub mod feature;

From 5d5b0509a543f2b6f09cc81acf0248a361b03aa1 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 28 Nov 2023 15:35:09 -0600
Subject: [PATCH 298/462] requires: add requires keyword

Add a new rule keyword "requires" that allows a rule to require specific
Suricata versions and/or Suricata features to be enabled.

Example:

  requires: feature geoip, version >= 7.0.0, version < 8;
  requires: version >= 7.0.3 < 8
  requires: version >= 7.0.3 < 8 | >= 8.0.3

Feature: #5972

Co-authored-by: Philippe Antoine <pantoine@oisf.net>
---
 doc/userguide/rules/meta.rst |  47 ++
 rust/src/detect/mod.rs       |   1 +
 rust/src/detect/requires.rs  | 805 +++++++++++++++++++++++++++++++++++
 rust/src/log.rs              |   9 +-
 src/Makefile.am              |   2 +
 src/detect-engine-loader.c   |  50 ++-
 src/detect-engine-register.c |   2 +
 src/detect-engine-register.h |   2 +
 src/detect-engine.c          |   4 +
 src/detect-parse.c           |  11 +-
 src/detect-requires.c        |  50 +++
 src/detect-requires.h        |  23 +
 src/detect.h                 |  10 +
 13 files changed, 997 insertions(+), 19 deletions(-)
 create mode 100644 rust/src/detect/requires.rs
 create mode 100644 src/detect-requires.c
 create mode 100644 src/detect-requires.h

diff --git a/doc/userguide/rules/meta.rst b/doc/userguide/rules/meta.rst
index 06e5040e73a5..0e888add697a 100644
--- a/doc/userguide/rules/meta.rst
+++ b/doc/userguide/rules/meta.rst
@@ -211,3 +211,50 @@ The format is::
 If the value is src_ip then the source IP in the generated event (src_ip
 field in JSON) is the target of the attack. If target is set to dest_ip
 then the target is the destination IP in the generated event.
+
+requires
+--------
+
+The ``requires`` keyword allows a rule to require specific Suricata
+features to be enabled, or the Suricata version to match an
+expression. Rules that do not meet the requirements will by ignored,
+and Suricata will not treat them as errors.
+
+When parsing rules, the parser attempts to process the ``requires``
+keywords before others. This allows it to occur after keywords that
+may only be present in specific versions of Suricata, as specified by
+the ``requires`` statement. However, the keywords preceding it must
+still adhere to the basic known formats of Suricata rules.
+
+The format is::
+
+   requires: feature geoip, version >= 7.0.0
+
+To require multiple features, the feature sub-keyword must be
+specified multiple times::
+
+   requires: feature geoip, feature lua
+
+Alternatively, *and* expressions may be expressed like::
+
+   requires: version >= 7.0.4 < 8
+
+and *or* expressions may expressed with ``|`` like::
+
+   requires: version >= 7.0.4 < 8 | >= 8.0.3
+
+to express that a rules requires version 7.0.4 or greater, but less
+than 8, **OR** greater than or equal to 8.0.3. Which could be useful
+if a keyword wasn't added until 7.0.4 and the 8.0.3 patch releases, as
+it would not exist in 8.0.1.
+
+This can be extended to multiple release branches::
+
+   requires: version >= 7.0.10 < 8 | >= 8.0.5 < 9 | >= 9.0.3
+
+If no *minor* or *patch* version component is provided, it will
+default to 0.
+
+The ``version`` may only be specified once, if specified more than
+once the rule will log an error and not be loaded.
+
diff --git a/rust/src/detect/mod.rs b/rust/src/detect/mod.rs
index 41c7ff2455bd..d33c9ae7fabf 100644
--- a/rust/src/detect/mod.rs
+++ b/rust/src/detect/mod.rs
@@ -24,3 +24,4 @@ pub mod parser;
 pub mod stream_size;
 pub mod uint;
 pub mod uri;
+pub mod requires;
diff --git a/rust/src/detect/requires.rs b/rust/src/detect/requires.rs
new file mode 100644
index 000000000000..e9e1acac5087
--- /dev/null
+++ b/rust/src/detect/requires.rs
@@ -0,0 +1,805 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+use std::collections::{HashSet, VecDeque};
+use std::{cmp::Ordering, ffi::CStr};
+
+// std::ffi::{c_char, c_int} is recommended these days, but requires
+// Rust 1.64.0.
+use std::os::raw::{c_char, c_int};
+
+use nom7::bytes::complete::take_while;
+use nom7::combinator::map;
+use nom7::multi::{many1, separated_list1};
+use nom7::sequence::tuple;
+use nom7::{
+    branch::alt,
+    bytes::complete::{tag, take_till},
+    character::complete::{char, multispace0},
+    combinator::map_res,
+    sequence::preceded,
+    IResult,
+};
+
+#[derive(Debug, Eq, PartialEq)]
+enum RequiresError {
+    /// Suricata is greater than the required version.
+    VersionGt,
+
+    /// Suricata is less than the required version.
+    VersionLt(SuricataVersion),
+
+    /// The running Suricata is missing a required feature.
+    MissingFeature(String),
+
+    /// The Suricata version, of Suricata itself is bad and failed to parse.
+    BadSuricataVersion,
+
+    /// The requires expression is bad and failed to parse.
+    BadRequires,
+
+    /// MultipleVersions
+    MultipleVersions,
+
+    /// Passed in requirements not a valid UTF-8 string.
+    Utf8Error,
+}
+
+impl RequiresError {
+    /// Return a pointer to a C compatible constant error message.
+    const fn c_errmsg(&self) -> *const c_char {
+        let msg = match self {
+            Self::VersionGt => "Suricata version greater than required\0",
+            Self::VersionLt(_) => "Suricata version less than required\0",
+            Self::MissingFeature(_) => "Suricata missing a required feature\0",
+            Self::BadSuricataVersion => "Failed to parse running Suricata version\0",
+            Self::BadRequires => "Failed to parse requires expression\0",
+            Self::MultipleVersions => "Version may only be specified once\0",
+            Self::Utf8Error => "Requires expression is not valid UTF-8\0",
+        };
+        msg.as_ptr() as *const c_char
+    }
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+enum VersionCompareOp {
+    Gt,
+    Gte,
+    Lt,
+    Lte,
+}
+
+#[derive(Debug, Clone, Eq, PartialEq)]
+struct SuricataVersion {
+    major: u8,
+    minor: u8,
+    patch: u8,
+}
+
+impl PartialOrd for SuricataVersion {
+    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for SuricataVersion {
+    fn cmp(&self, other: &Self) -> Ordering {
+        match self.major.cmp(&other.major) {
+            Ordering::Equal => match self.minor.cmp(&other.minor) {
+                Ordering::Equal => self.patch.cmp(&other.patch),
+                other => other,
+            },
+            other => other,
+        }
+    }
+}
+
+impl std::fmt::Display for SuricataVersion {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}.{}.{}", self.major, self.minor, self.patch)
+    }
+}
+
+impl SuricataVersion {
+    fn new(major: u8, minor: u8, patch: u8) -> Self {
+        Self {
+            major,
+            minor,
+            patch,
+        }
+    }
+}
+
+/// Parse a version expression.
+///
+/// Parse into a version expression into a nested array, for example:
+///
+///    version: >= 7.0.3 < 8 | >= 8.0.3
+///
+/// would result in something like:
+///
+/// [
+///     [{op: gte, version: 7.0.3}, {op:lt, version: 8}],
+///     [{op: gte, version: 8.0.3}],
+/// ]
+fn parse_version_expression(input: &str) -> IResult<&str, Vec<Vec<RuleRequireVersion>>> {
+    let sep = preceded(multispace0, tag("|"));
+    let inner_parser = many1(tuple((parse_op, parse_version)));
+    let (input, versions) = separated_list1(sep, inner_parser)(input)?;
+
+    let versions = versions
+        .into_iter()
+        .map(|versions| {
+            versions
+                .into_iter()
+                .map(|(op, version)| RuleRequireVersion { op, version })
+                .collect()
+        })
+        .collect();
+
+    Ok((input, versions))
+}
+
+#[derive(Debug, Eq, PartialEq)]
+struct RuleRequireVersion {
+    pub op: VersionCompareOp,
+    pub version: SuricataVersion,
+}
+
+#[derive(Debug, Default, Eq, PartialEq)]
+struct Requires {
+    pub features: Vec<String>,
+
+    /// The version expression.
+    ///
+    /// - All of the inner most must evaluate to true.
+    /// - To pass, any of the outer must be true.
+    pub version: Vec<Vec<RuleRequireVersion>>,
+}
+
+fn parse_op(input: &str) -> IResult<&str, VersionCompareOp> {
+    preceded(
+        multispace0,
+        alt((
+            map(tag(">="), |_| VersionCompareOp::Gte),
+            map(tag(">"), |_| VersionCompareOp::Gt),
+            map(tag("<="), |_| VersionCompareOp::Lte),
+            map(tag("<"), |_| VersionCompareOp::Lt),
+        )),
+    )(input)
+}
+
+/// Parse the next part of the version.
+///
+/// That is all chars up to eof, or the next '.' or '-'.
+fn parse_next_version_part(input: &str) -> IResult<&str, u8> {
+    map_res(
+        take_till(|c| c == '.' || c == '-' || c == ' '),
+        |s: &str| s.parse::<u8>(),
+    )(input)
+}
+
+/// Parse a version string into a SuricataVersion.
+fn parse_version(input: &str) -> IResult<&str, SuricataVersion> {
+    let (input, major) = preceded(multispace0, parse_next_version_part)(input)?;
+    let (input, minor) = if input.is_empty() || input.starts_with(' ') {
+        (input, 0)
+    } else {
+        preceded(char('.'), parse_next_version_part)(input)?
+    };
+    let (input, patch) = if input.is_empty() || input.starts_with(' ') {
+        (input, 0)
+    } else {
+        preceded(char('.'), parse_next_version_part)(input)?
+    };
+
+    Ok((input, SuricataVersion::new(major, minor, patch)))
+}
+
+fn parse_key_value(input: &str) -> IResult<&str, (&str, &str)> {
+    // Parse the keyword, any sequence of characters, numbers or "-" or "_".
+    let (input, key) = preceded(
+        multispace0,
+        take_while(|c: char| c.is_alphanumeric() || c == '-' || c == '_'),
+    )(input)?;
+    let (input, value) = preceded(multispace0, take_till(|c: char| c == ','))(input)?;
+    Ok((input, (key, value)))
+}
+
+fn parse_requires(mut input: &str) -> Result<Requires, RequiresError> {
+    let mut requires = Requires::default();
+
+    while !input.is_empty() {
+        let (rest, (keyword, value)) =
+            parse_key_value(input).map_err(|_| RequiresError::BadRequires)?;
+        match keyword {
+            "feature" => {
+                requires.features.push(value.trim().to_string());
+            }
+            "version" => {
+                if !requires.version.is_empty() {
+                    return Err(RequiresError::MultipleVersions);
+                }
+                let (_, versions) =
+                    parse_version_expression(value).map_err(|_| RequiresError::BadRequires)?;
+                requires.version = versions;
+            }
+            _ => {
+                // Unknown keyword, allow by warn in case we extend
+                // this in the future.
+                SCLogWarning!("Unknown requires keyword: {}", keyword);
+            }
+        }
+
+        // No consume any remaining ',' or whitespace.
+        input = rest.trim_start_matches(|c: char| c == ',' || c.is_whitespace());
+    }
+    Ok(requires)
+}
+
+fn parse_suricata_version(version: &CStr) -> Result<SuricataVersion, *const c_char> {
+    let version = version
+        .to_str()
+        .map_err(|_| RequiresError::BadSuricataVersion.c_errmsg())?;
+    let (_, version) =
+        parse_version(version).map_err(|_| RequiresError::BadSuricataVersion.c_errmsg())?;
+    Ok(version)
+}
+
+fn check_version(
+    version: &RuleRequireVersion, suricata_version: &SuricataVersion,
+) -> Result<(), RequiresError> {
+    match version.op {
+        VersionCompareOp::Gt => {
+            if suricata_version <= &version.version {
+                return Err(RequiresError::VersionLt(version.version.clone()));
+            }
+        }
+        VersionCompareOp::Gte => {
+            if suricata_version < &version.version {
+                return Err(RequiresError::VersionLt(version.version.clone()));
+            }
+        }
+        VersionCompareOp::Lt => {
+            if suricata_version >= &version.version {
+                return Err(RequiresError::VersionGt);
+            }
+        }
+        VersionCompareOp::Lte => {
+            if suricata_version > &version.version {
+                return Err(RequiresError::VersionGt);
+            }
+        }
+    }
+    Ok(())
+}
+
+fn check_requires(
+    requires: &Requires, suricata_version: &SuricataVersion,
+) -> Result<(), RequiresError> {
+    if !requires.version.is_empty() {
+        let mut errs = VecDeque::new();
+        let mut ok = 0;
+        for or_versions in &requires.version {
+            let mut err = None;
+            for version in or_versions {
+                if let Err(_err) = check_version(version, suricata_version) {
+                    err = Some(_err);
+                    break;
+                }
+            }
+            if let Some(err) = err {
+                errs.push_back(err);
+            } else {
+                ok += 1;
+            }
+        }
+        if ok == 0 {
+            return Err(errs.pop_front().unwrap());
+        }
+    }
+
+    for feature in &requires.features {
+        if !crate::feature::requires(feature) {
+            return Err(RequiresError::MissingFeature(feature.to_string()));
+        }
+    }
+
+    Ok(())
+}
+
+/// Status object to hold required features and the latest version of
+/// Suricata required.
+///
+/// Full qualified name as it is exposed to C.
+#[derive(Debug, Default)]
+pub struct SCDetectRequiresStatus {
+    min_version: Option<SuricataVersion>,
+    features: HashSet<String>,
+
+    /// Number of rules that didn't meet a feature.
+    feature_count: u64,
+
+    /// Number of rules where the Suricata version wasn't new enough.
+    lt_count: u64,
+
+    /// Number of rules where the Suricata version was too new.
+    gt_count: u64,
+}
+
+#[no_mangle]
+pub extern "C" fn SCDetectRequiresStatusNew() -> *mut SCDetectRequiresStatus {
+    Box::into_raw(Box::default())
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn SCDetectRequiresStatusFree(status: *mut SCDetectRequiresStatus) {
+    if !status.is_null() {
+        std::mem::drop(Box::from_raw(status));
+    }
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn SCDetectRequiresStatusLog(
+    status: &mut SCDetectRequiresStatus, suricata_version: *const c_char, tenant_id: u32,
+) {
+    let suricata_version = CStr::from_ptr(suricata_version)
+        .to_str()
+        .unwrap_or("<unknown>");
+
+    let mut parts = vec![];
+    if status.lt_count > 0 {
+        let min_version = status
+            .min_version
+            .as_ref()
+            .map(|v| v.to_string())
+            .unwrap_or_else(|| "<unknown>".to_string());
+        let msg = format!(
+            "{} {} skipped because the running Suricata version {} is less than {}",
+            status.lt_count,
+            if status.lt_count > 1 {
+                "rules were"
+            } else {
+                "rule was"
+            },
+            suricata_version,
+            &min_version
+        );
+        parts.push(msg);
+    }
+    if status.gt_count > 0 {
+        let msg = format!(
+            "{} {} for an older version Suricata",
+            status.gt_count,
+            if status.gt_count > 1 {
+                "rules were skipped as they are"
+            } else {
+                "rule was skipped as it is"
+            }
+        );
+        parts.push(msg);
+    }
+    if status.feature_count > 0 {
+        let features = status
+            .features
+            .iter()
+            .map(|f| f.to_string())
+            .collect::<Vec<String>>()
+            .join(", ");
+        let msg = format!(
+            "{}{} {} skipped because the running Suricata version does not have feature{}: [{}]",
+            if tenant_id > 0 {
+                format!("tenant id: {}  ", tenant_id)
+            } else {
+                String::new()
+            },
+            status.feature_count,
+            if status.feature_count > 1 {
+                "rules were"
+            } else {
+                "rule was"
+            },
+            if status.feature_count > 1 { "s" } else { "" },
+            &features
+        );
+        parts.push(msg);
+    }
+
+    let msg = parts.join("; ");
+
+    if status.lt_count > 0 {
+        SCLogNotice!("{}", &msg);
+    } else if status.gt_count > 0 || status.feature_count > 0 {
+        SCLogInfo!("{}", &msg);
+    }
+}
+
+/// Parse a "requires" rule option.
+///
+/// Return values:
+///   *  0 - OK, rule should continue loading
+///   * -1 - Error parsing the requires content
+///   * -4 - Requirements not met, don't continue loading the rule, this
+///          value is chosen so it can be passed back to the options parser
+///          as its treated as a non-fatal silent error.
+#[no_mangle]
+pub unsafe extern "C" fn SCDetectCheckRequires(
+    requires: *const c_char, suricata_version_string: *const c_char, errstr: *mut *const c_char,
+    status: &mut SCDetectRequiresStatus,
+) -> c_int {
+    // First parse the running Suricata version.
+    let suricata_version = match parse_suricata_version(CStr::from_ptr(suricata_version_string)) {
+        Ok(version) => version,
+        Err(err) => {
+            *errstr = err;
+            return -1;
+        }
+    };
+
+    let requires = match CStr::from_ptr(requires)
+        .to_str()
+        .map_err(|_| RequiresError::Utf8Error)
+        .and_then(parse_requires)
+    {
+        Ok(requires) => requires,
+        Err(err) => {
+            *errstr = err.c_errmsg();
+            return -1;
+        }
+    };
+
+    match check_requires(&requires, &suricata_version) {
+        Ok(()) => 0,
+        Err(err) => {
+            match &err {
+                RequiresError::VersionLt(version) => {
+                    if let Some(min_version) = &status.min_version {
+                        if version > min_version {
+                            status.min_version = Some(version.clone());
+                        }
+                    } else {
+                        status.min_version = Some(version.clone());
+                    }
+                    status.lt_count += 1;
+                }
+                RequiresError::MissingFeature(feature) => {
+                    status.features.insert(feature.to_string());
+                    status.feature_count += 1;
+                }
+                RequiresError::VersionGt => {
+                    status.gt_count += 1;
+                }
+                _ => {}
+            }
+            *errstr = err.c_errmsg();
+            return -4;
+        }
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn test_suricata_version() {
+        // 7.1.1 < 7.1.2
+        assert!(SuricataVersion::new(7, 1, 1) < SuricataVersion::new(7, 1, 2));
+
+        // 7.1.1 <= 7.1.2
+        assert!(SuricataVersion::new(7, 1, 1) <= SuricataVersion::new(7, 1, 2));
+
+        // 7.1.1 <= 7.1.1
+        assert!(SuricataVersion::new(7, 1, 1) <= SuricataVersion::new(7, 1, 1));
+
+        // NOT 7.1.1 < 7.1.1
+        assert!(SuricataVersion::new(7, 1, 1) >= SuricataVersion::new(7, 1, 1));
+
+        // 7.3.1 < 7.22.1
+        assert!(SuricataVersion::new(7, 3, 1) < SuricataVersion::new(7, 22, 1));
+
+        // 7.22.1 >= 7.3.4
+        assert!(SuricataVersion::new(7, 22, 1) >= SuricataVersion::new(7, 3, 4));
+    }
+
+    #[test]
+    fn test_parse_op() {
+        assert_eq!(parse_op(">").unwrap().1, VersionCompareOp::Gt);
+        assert_eq!(parse_op(">=").unwrap().1, VersionCompareOp::Gte);
+        assert_eq!(parse_op("<").unwrap().1, VersionCompareOp::Lt);
+        assert_eq!(parse_op("<=").unwrap().1, VersionCompareOp::Lte);
+
+        assert!(parse_op("=").is_err());
+    }
+
+    #[test]
+    fn test_parse_version() {
+        assert_eq!(
+            parse_version("7").unwrap().1,
+            SuricataVersion {
+                major: 7,
+                minor: 0,
+                patch: 0,
+            }
+        );
+
+        assert_eq!(
+            parse_version("7.1").unwrap().1,
+            SuricataVersion {
+                major: 7,
+                minor: 1,
+                patch: 0,
+            }
+        );
+
+        assert_eq!(
+            parse_version("7.1.2").unwrap().1,
+            SuricataVersion {
+                major: 7,
+                minor: 1,
+                patch: 2,
+            }
+        );
+
+        // Suricata pre-releases will have a suffix starting with a
+        // '-', so make sure we accept those versions as well.
+        assert_eq!(
+            parse_version("8.0.0-dev").unwrap().1,
+            SuricataVersion {
+                major: 8,
+                minor: 0,
+                patch: 0,
+            }
+        );
+
+        assert!(parse_version("7.1.2a").is_err());
+        assert!(parse_version("a").is_err());
+        assert!(parse_version("777").is_err());
+        assert!(parse_version("product-1").is_err());
+    }
+
+    #[test]
+    fn test_parse_requires() {
+        let requires = parse_requires("  feature geoip").unwrap();
+        assert_eq!(&requires.features[0], "geoip");
+
+        let requires = parse_requires("  feature geoip,    feature    lua  ").unwrap();
+        assert_eq!(&requires.features[0], "geoip");
+        assert_eq!(&requires.features[1], "lua");
+
+        let requires = parse_requires("version >=7").unwrap();
+        assert_eq!(
+            requires,
+            Requires {
+                features: vec![],
+                version: vec![vec![RuleRequireVersion {
+                    op: VersionCompareOp::Gte,
+                    version: SuricataVersion {
+                        major: 7,
+                        minor: 0,
+                        patch: 0,
+                    }
+                }]],
+            }
+        );
+
+        let requires = parse_requires("version >= 7.1").unwrap();
+        assert_eq!(
+            requires,
+            Requires {
+                features: vec![],
+                version: vec![vec![RuleRequireVersion {
+                    op: VersionCompareOp::Gte,
+                    version: SuricataVersion {
+                        major: 7,
+                        minor: 1,
+                        patch: 0,
+                    }
+                }]],
+            }
+        );
+
+        let requires = parse_requires("feature output::file-store, version >= 7.1.2").unwrap();
+        assert_eq!(
+            requires,
+            Requires {
+                features: vec!["output::file-store".to_string()],
+                version: vec![vec![RuleRequireVersion {
+                    op: VersionCompareOp::Gte,
+                    version: SuricataVersion {
+                        major: 7,
+                        minor: 1,
+                        patch: 2,
+                    }
+                }]],
+            }
+        );
+
+        let requires = parse_requires("feature geoip, version >= 7.1.2 < 8").unwrap();
+        assert_eq!(
+            requires,
+            Requires {
+                features: vec!["geoip".to_string()],
+                version: vec![vec![
+                    RuleRequireVersion {
+                        op: VersionCompareOp::Gte,
+                        version: SuricataVersion {
+                            major: 7,
+                            minor: 1,
+                            patch: 2,
+                        },
+                    },
+                    RuleRequireVersion {
+                        op: VersionCompareOp::Lt,
+                        version: SuricataVersion {
+                            major: 8,
+                            minor: 0,
+                            patch: 0,
+                        }
+                    }
+                ]],
+            }
+        );
+    }
+
+    #[test]
+    fn test_check_requires() {
+        // Have 7.0.4, require >= 8.
+        let suricata_version = SuricataVersion::new(7, 0, 4);
+        let requires = parse_requires("version >= 8").unwrap();
+        assert_eq!(
+            check_requires(&requires, &suricata_version),
+            Err(RequiresError::VersionLt(SuricataVersion {
+                major: 8,
+                minor: 0,
+                patch: 0,
+            })),
+        );
+
+        // Have 7.0.4, require 7.0.3.
+        let suricata_version = SuricataVersion::new(7, 0, 4);
+        let requires = parse_requires("version >= 7.0.3").unwrap();
+        assert_eq!(check_requires(&requires, &suricata_version), Ok(()));
+
+        // Have 8.0.0, require >= 7.0.0 and < 8.0
+        let suricata_version = SuricataVersion::new(8, 0, 0);
+        let requires = parse_requires("version >= 7.0.0 < 8").unwrap();
+        assert_eq!(
+            check_requires(&requires, &suricata_version),
+            Err(RequiresError::VersionGt)
+        );
+
+        // Have 8.0.0, require >= 7.0.0 and < 9.0
+        let suricata_version = SuricataVersion::new(8, 0, 0);
+        let requires = parse_requires("version >= 7.0.0 < 9").unwrap();
+        assert_eq!(check_requires(&requires, &suricata_version), Ok(()));
+
+        // Require feature foobar.
+        let suricata_version = SuricataVersion::new(8, 0, 0);
+        let requires = parse_requires("feature foobar").unwrap();
+        assert_eq!(
+            check_requires(&requires, &suricata_version),
+            Err(RequiresError::MissingFeature("foobar".to_string()))
+        );
+
+        // Require feature foobar, but this time we have the feature.
+        let suricata_version = SuricataVersion::new(8, 0, 0);
+        let requires = parse_requires("feature true_foobar").unwrap();
+        assert_eq!(check_requires(&requires, &suricata_version), Ok(()));
+
+        let suricata_version = SuricataVersion::new(8, 0, 1);
+        let requires = parse_requires("version >= 7.0.3 < 8").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_err());
+
+        let suricata_version = SuricataVersion::new(7, 0, 1);
+        let requires = parse_requires("version >= 7.0.3 < 8").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_err());
+
+        let suricata_version = SuricataVersion::new(7, 0, 3);
+        let requires = parse_requires("version >= 7.0.3 < 8").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_ok());
+
+        let suricata_version = SuricataVersion::new(8, 0, 3);
+        let requires = parse_requires("version >= 7.0.3 < 8 | >= 8.0.3").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_ok());
+
+        let suricata_version = SuricataVersion::new(8, 0, 2);
+        let requires = parse_requires("version >= 7.0.3 < 8 | >= 8.0.3").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_err());
+
+        let suricata_version = SuricataVersion::new(7, 0, 2);
+        let requires = parse_requires("version >= 7.0.3 < 8 | >= 8.0.3").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_err());
+
+        let suricata_version = SuricataVersion::new(7, 0, 3);
+        let requires = parse_requires("version >= 7.0.3 < 8 | >= 8.0.3").unwrap();
+        assert!(check_requires(&requires, &suricata_version).is_ok());
+
+        // Example of something that requires a fix/feature that was
+        // implemented in 7.0.5, 8.0.4, 9.0.3.
+        let requires = parse_requires("version >= 7.0.5 < 8 | >= 8.0.4 < 9 | >= 9.0.3").unwrap();
+        assert!(check_requires(&requires, &SuricataVersion::new(6, 0, 0)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(7, 0, 4)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(7, 0, 5)).is_ok());
+        assert!(check_requires(&requires, &SuricataVersion::new(8, 0, 3)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(8, 0, 4)).is_ok());
+        assert!(check_requires(&requires, &SuricataVersion::new(9, 0, 2)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(9, 0, 3)).is_ok());
+        assert!(check_requires(&requires, &SuricataVersion::new(10, 0, 0)).is_ok());
+
+        let requires = parse_requires("version >= 8 < 9").unwrap();
+        assert!(check_requires(&requires, &SuricataVersion::new(6, 0, 0)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(7, 0, 0)).is_err());
+        assert!(check_requires(&requires, &SuricataVersion::new(8, 0, 0)).is_ok());
+        assert!(check_requires(&requires, &SuricataVersion::new(9, 0, 0)).is_err());
+
+        // Unknown keyword.
+        let requires = parse_requires("feature lua, foo bar, version >= 7.0.3").unwrap();
+        assert_eq!(
+            requires,
+            Requires {
+                features: vec!["lua".to_string()],
+                version: vec![vec![RuleRequireVersion {
+                    op: VersionCompareOp::Gte,
+                    version: SuricataVersion {
+                        major: 7,
+                        minor: 0,
+                        patch: 3,
+                    }
+                }]],
+            }
+        );
+    }
+
+    #[test]
+    fn test_parse_version_expression() {
+        let version_str = ">= 7.0.3 < 8 | >= 8.0.3";
+        let (rest, versions) = parse_version_expression(version_str).unwrap();
+        assert!(rest.is_empty());
+        assert_eq!(
+            versions,
+            vec![
+                vec![
+                    RuleRequireVersion {
+                        op: VersionCompareOp::Gte,
+                        version: SuricataVersion {
+                            major: 7,
+                            minor: 0,
+                            patch: 3,
+                        }
+                    },
+                    RuleRequireVersion {
+                        op: VersionCompareOp::Lt,
+                        version: SuricataVersion {
+                            major: 8,
+                            minor: 0,
+                            patch: 0,
+                        }
+                    },
+                ],
+                vec![RuleRequireVersion {
+                    op: VersionCompareOp::Gte,
+                    version: SuricataVersion {
+                        major: 8,
+                        minor: 0,
+                        patch: 3,
+                    }
+                },],
+            ]
+        );
+    }
+}
diff --git a/rust/src/log.rs b/rust/src/log.rs
index 744169a97039..7bf0be8a97c6 100644
--- a/rust/src/log.rs
+++ b/rust/src/log.rs
@@ -29,7 +29,7 @@ pub enum Level {
     NotSet = -1,
     _None = 0,
     Error,
-    _Warning,
+    Warning,
     Notice,
     Info,
     _Perf,
@@ -115,6 +115,13 @@ macro_rules!SCLogError {
     };
 }
 
+#[macro_export]
+macro_rules!SCLogWarning {
+    ($($arg:tt)*) => {
+        $crate::do_log!($crate::log::Level::Warning, $($arg)*);
+    };
+}
+
 #[macro_export]
 macro_rules!SCLogNotice {
     ($($arg:tt)*) => {
diff --git a/src/Makefile.am b/src/Makefile.am
index 6d115ac48ae5..9cb4f815531c 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -280,6 +280,7 @@ noinst_HEADERS = \
 	detect-rawbytes.h \
 	detect-reference.h \
 	detect-replace.h \
+	detect-requires.h \
 	detect-rev.h \
 	detect-rfb-name.h \
 	detect-rfb-secresult.h \
@@ -895,6 +896,7 @@ libsuricata_c_a_SOURCES = \
 	detect-rawbytes.c \
 	detect-reference.c \
 	detect-replace.c \
+	detect-requires.c \
 	detect-rev.c \
 	detect-rfb-name.c \
 	detect-rfb-secresult.c \
diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 40919568503a..9c3e3e8b533c 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -44,6 +44,8 @@
 #include "util-threshold-config.h"
 #include "util-path.h"
 
+#include "rust.h"
+
 #ifdef HAVE_GLOB_H
 #include <glob.h>
 #endif
@@ -109,11 +111,11 @@ char *DetectLoadCompleteSigPath(const DetectEngineCtx *de_ctx, const char *sig_f
  *  \param badsigs_tot Will store number of invalid signatures in the file
  *  \retval 0 on success, -1 on error
  */
-static int DetectLoadSigFile(DetectEngineCtx *de_ctx, char *sig_file,
-        int *goodsigs, int *badsigs)
+static int DetectLoadSigFile(
+        DetectEngineCtx *de_ctx, char *sig_file, int *goodsigs, int *badsigs, int *skippedsigs)
 {
     Signature *sig = NULL;
-    int good = 0, bad = 0;
+    int good = 0, bad = 0, skipped = 0;
     char line[DETECT_MAX_RULE_SIZE] = "";
     size_t offset = 0;
     int lineno = 0, multiline = 0;
@@ -196,6 +198,12 @@ static int DetectLoadSigFile(DetectEngineCtx *de_ctx, char *sig_file,
             if (!de_ctx->sigerror_ok) {
                 bad++;
             }
+            if (de_ctx->sigerror_requires) {
+                SCLogInfo("Skipping signature due to missing requirements: %s from file %s at line "
+                          "%" PRId32,
+                        line, sig_file, lineno - multiline);
+                skipped++;
+            }
         }
         multiline = 0;
     }
@@ -203,6 +211,7 @@ static int DetectLoadSigFile(DetectEngineCtx *de_ctx, char *sig_file,
 
     *goodsigs = good;
     *badsigs = bad;
+    *skippedsigs = skipped;
     return 0;
 }
 
@@ -212,8 +221,8 @@ static int DetectLoadSigFile(DetectEngineCtx *de_ctx, char *sig_file,
  *  \param sig_file Filename (or pattern) holding signatures
  *  \retval -1 on error
  */
-static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern,
-        SigFileLoaderStat *st, int *good_sigs, int *bad_sigs)
+static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern, SigFileLoaderStat *st,
+        int *good_sigs, int *bad_sigs, int *skipped_sigs)
 {
     int r = 0;
 
@@ -250,7 +259,7 @@ static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern,
         } else {
             SCLogConfig("Loading rule file: %s", fname);
         }
-        r = DetectLoadSigFile(de_ctx, fname, good_sigs, bad_sigs);
+        r = DetectLoadSigFile(de_ctx, fname, good_sigs, bad_sigs, skipped_sigs);
         if (r < 0) {
             ++(st->bad_files);
         }
@@ -259,6 +268,7 @@ static int ProcessSigFiles(DetectEngineCtx *de_ctx, char *pattern,
 
         st->good_sigs_total += *good_sigs;
         st->bad_sigs_total += *bad_sigs;
+        st->skipped_sigs_total += *skipped_sigs;
 
 #ifdef HAVE_GLOB_H
     }
@@ -286,6 +296,7 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, bool sig_file_exc
     char varname[128] = "rule-files";
     int good_sigs = 0;
     int bad_sigs = 0;
+    int skipped_sigs = 0;
 
     if (strlen(de_ctx->config_prefix) > 0) {
         snprintf(varname, sizeof(varname), "%s.rule-files",
@@ -307,8 +318,9 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, bool sig_file_exc
             else {
                 TAILQ_FOREACH(file, &rule_files->head, next) {
                     sfile = DetectLoadCompleteSigPath(de_ctx, file->val);
-                    good_sigs = bad_sigs = 0;
-                    ret = ProcessSigFiles(de_ctx, sfile, sig_stat, &good_sigs, &bad_sigs);
+                    good_sigs = bad_sigs = skipped_sigs = 0;
+                    ret = ProcessSigFiles(
+                            de_ctx, sfile, sig_stat, &good_sigs, &bad_sigs, &skipped_sigs);
                     SCFree(sfile);
 
                     if (de_ctx->failure_fatal && ret != 0) {
@@ -327,7 +339,7 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, bool sig_file_exc
 
     /* If a Signature file is specified from command-line, parse it too */
     if (sig_file != NULL) {
-        ret = ProcessSigFiles(de_ctx, sig_file, sig_stat, &good_sigs, &bad_sigs);
+        ret = ProcessSigFiles(de_ctx, sig_file, sig_stat, &good_sigs, &bad_sigs, &skipped_sigs);
 
         if (ret != 0) {
             if (de_ctx->failure_fatal) {
@@ -351,15 +363,23 @@ int SigLoadSignatures(DetectEngineCtx *de_ctx, char *sig_file, bool sig_file_exc
         }
     } else {
         /* we report the total of files and rules successfully loaded and failed */
-        if (strlen(de_ctx->config_prefix) > 0)
+        if (strlen(de_ctx->config_prefix) > 0) {
             SCLogInfo("tenant id %d:  %" PRId32 " rule files processed. %" PRId32
-                      " rules successfully loaded, %" PRId32 " rules failed",
+                      " rules successfully loaded, %" PRId32 " rules failed, %" PRId32
+                      " rules skipped",
                     de_ctx->tenant_id, sig_stat->total_files, sig_stat->good_sigs_total,
-                    sig_stat->bad_sigs_total);
-        else
+                    sig_stat->bad_sigs_total, sig_stat->skipped_sigs_total);
+        } else {
             SCLogInfo("%" PRId32 " rule files processed. %" PRId32
-                      " rules successfully loaded, %" PRId32 " rules failed",
-                    sig_stat->total_files, sig_stat->good_sigs_total, sig_stat->bad_sigs_total);
+                      " rules successfully loaded, %" PRId32 " rules failed, %" PRId32
+                      " rules skipped",
+                    sig_stat->total_files, sig_stat->good_sigs_total, sig_stat->bad_sigs_total,
+                    sig_stat->skipped_sigs_total);
+        }
+        if (de_ctx->requirements != NULL && sig_stat->skipped_sigs_total > 0) {
+            SCDetectRequiresStatusLog(de_ctx->requirements, PROG_VER,
+                    strlen(de_ctx->config_prefix) > 0 ? de_ctx->tenant_id : 0);
+        }
     }
 
     if ((sig_stat->bad_sigs_total || sig_stat->bad_files) && de_ctx->failure_fatal) {
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index a97da4617197..9f37e0945544 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -119,6 +119,7 @@
 #include "detect-flow.h"
 #include "detect-flow-age.h"
 #include "detect-flow-pkts.h"
+#include "detect-requires.h"
 #include "detect-tcp-window.h"
 #include "detect-ftpbounce.h"
 #include "detect-isdataat.h"
@@ -575,6 +576,7 @@ void SigTableSetup(void)
     DetectFlowPktsToServerRegister();
     DetectFlowBytesToClientRegister();
     DetectFlowBytesToServerRegister();
+    DetectRequiresRegister();
     DetectWindowRegister();
     DetectRpcRegister();
     DetectFtpbounceRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 2e4a330788ed..9dd01f5fd487 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -115,6 +115,8 @@ enum DetectKeywordId {
     DETECT_FLOW_BYTES_TO_CLIENT,
     DETECT_FLOW_BYTES_TO_SERVER,
 
+    DETECT_REQUIRES,
+
     DETECT_AL_TLS_VERSION,
     DETECT_AL_TLS_SUBJECT,
     DETECT_AL_TLS_ISSUERDN,
diff --git a/src/detect-engine.c b/src/detect-engine.c
index 3e1ce93cf671..13e09be71889 100644
--- a/src/detect-engine.c
+++ b/src/detect-engine.c
@@ -2652,6 +2652,10 @@ void DetectEngineCtxFree(DetectEngineCtx *de_ctx)
         SCFree(de_ctx->tenant_path);
     }
 
+    if (de_ctx->requirements) {
+        SCDetectRequiresStatusFree(de_ctx->requirements);
+    }
+
     SCFree(de_ctx);
     //DetectAddressGroupPrintMemory();
     //DetectSigGroupPrintMemory();
diff --git a/src/detect-parse.c b/src/detect-parse.c
index 83c22aa8adac..b94d54dd6cd0 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -2146,12 +2146,17 @@ static Signature *SigInitHelper(DetectEngineCtx *de_ctx, const char *sigstr,
     sig->gid = 1;
 
     int ret = SigParse(de_ctx, sig, sigstr, dir, &parser);
-    if (ret == -3) {
+    if (ret == -4) {
+        /* Rule requirements not met. */
         de_ctx->sigerror_silent = true;
         de_ctx->sigerror_ok = true;
+        de_ctx->sigerror_requires = true;
         goto error;
-    }
-    else if (ret == -2) {
+    } else if (ret == -3) {
+        de_ctx->sigerror_silent = true;
+        de_ctx->sigerror_ok = true;
+        goto error;
+    } else if (ret == -2) {
         de_ctx->sigerror_silent = true;
         goto error;
     } else if (ret < 0) {
diff --git a/src/detect-requires.c b/src/detect-requires.c
new file mode 100644
index 000000000000..4d7f916b3b82
--- /dev/null
+++ b/src/detect-requires.c
@@ -0,0 +1,50 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#include "detect-requires.h"
+#include "suricata-common.h"
+#include "detect-engine.h"
+#include "rust.h"
+
+static int DetectRequiresSetup(DetectEngineCtx *de_ctx, Signature *s, const char *rawstr)
+{
+    if (de_ctx->requirements == NULL) {
+        de_ctx->requirements = (void *)SCDetectRequiresStatusNew();
+        BUG_ON(de_ctx->requirements == NULL);
+    }
+
+    const char *errmsg = NULL;
+    int res = SCDetectCheckRequires(rawstr, PROG_VER, &errmsg, de_ctx->requirements);
+    if (res == -1) {
+        // The requires expression is bad, log an error.
+        SCLogError("%s: %s", errmsg, rawstr);
+        de_ctx->sigerror = errmsg;
+    } else if (res < -1) {
+        // This Suricata instance didn't meet the requirements.
+        SCLogInfo("Suricata did not meet the rule requirements: %s: %s", errmsg, rawstr);
+        return -4;
+    }
+    return res;
+}
+
+void DetectRequiresRegister(void)
+{
+    sigmatch_table[DETECT_REQUIRES].name = "requires";
+    sigmatch_table[DETECT_REQUIRES].desc = "require Suricata version or features";
+    sigmatch_table[DETECT_REQUIRES].url = "/rules/meta-keywords.html#requires";
+    sigmatch_table[DETECT_REQUIRES].Setup = DetectRequiresSetup;
+}
diff --git a/src/detect-requires.h b/src/detect-requires.h
new file mode 100644
index 000000000000..70f1dc43b814
--- /dev/null
+++ b/src/detect-requires.h
@@ -0,0 +1,23 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#ifndef __DETECT_REQUIRES_H__
+#define __DETECT_REQUIRES_H__
+
+void DetectRequiresRegister(void);
+
+#endif /* __DETECT_REQUIRES_H__ */
diff --git a/src/detect.h b/src/detect.h
index 8278290a2992..5d55720758ea 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -53,6 +53,9 @@
 struct SCSigOrderFunc_;
 struct SCSigSignatureWrapper_;
 
+/* Forward declarations for structures from Rust. */
+typedef struct SCDetectRequiresStatus SCDetectRequiresStatus;
+
 enum SignatureType {
     SIG_TYPE_NOT_SET = 0,
     SIG_TYPE_IPONLY,      // rule is handled by IPONLY engine
@@ -797,6 +800,7 @@ typedef struct SigFileLoaderStat_ {
     int total_files;
     int good_sigs_total;
     int bad_sigs_total;
+    int skipped_sigs_total;
 } SigFileLoaderStat;
 
 typedef struct DetectEngineThreadKeywordCtxItem_ {
@@ -925,6 +929,9 @@ typedef struct DetectEngineCtx_ {
     bool sigerror_silent;
     bool sigerror_ok;
 
+    /** The rule errored out due to missing requirements. */
+    bool sigerror_requires;
+
     bool filedata_config_initialized;
 
     /* specify the configuration for mpm context factory */
@@ -1032,6 +1039,9 @@ typedef struct DetectEngineCtx_ {
 
     /* path to the tenant yaml for this engine */
     char *tenant_path;
+
+    /* Track rule requirements for reporting after loading rules. */
+    SCDetectRequiresStatus *requirements;
 } DetectEngineCtx;
 
 /* Engine groups profiles (low, medium, high, custom) */

From 435c03172ed7ebaa117765760e75bdfd38c7fca0 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 29 Nov 2023 10:54:54 -0600
Subject: [PATCH 299/462] requires: pre-scan rule for requires expressions

Add a "pre-scan" rule parse that will check for requires statement. It
will return a special error code (-4) if the requires fails due to
missing requirements.

Syntactic errors will also abort parsing here.

Feature: #5972
---
 src/detect-parse.c | 62 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 47 insertions(+), 15 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index b94d54dd6cd0..493bee10ea9f 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -845,7 +845,8 @@ int SigMatchListSMBelongsTo(const Signature *s, const SigMatch *key_sm)
     return -1;
 }
 
-static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr, char *output, size_t output_size)
+static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr, char *output,
+        size_t output_size, bool requires)
 {
     SigTableElmt *st = NULL;
     char *optname = NULL;
@@ -899,6 +900,12 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
     }
     optname = optstr;
 
+    if (requires) {
+        if (strcmp(optname, "requires")) {
+            goto finish;
+        }
+    }
+
     /* Call option parsing */
     st = SigTableGet(optname);
     if (st == NULL || st->Setup == NULL) {
@@ -1038,6 +1045,7 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
     }
     s->init_data->negated = false;
 
+finish:
     if (strlen(optend) > 0) {
         strlcpy(output, optend, output_size);
         return 1;
@@ -1323,9 +1331,11 @@ static inline int SigParseList(char **input, char *output,
 /**
  *  \internal
  *  \brief split a signature string into a few blocks for further parsing
+ *
+ *  \param scan_only just scan, don't validate
  */
-static int SigParseBasics(DetectEngineCtx *de_ctx,
-        Signature *s, const char *sigstr, SignatureParser *parser, uint8_t addrs_direction)
+static int SigParseBasics(DetectEngineCtx *de_ctx, Signature *s, const char *sigstr,
+        SignatureParser *parser, uint8_t addrs_direction, bool scan_only)
 {
     char *index, dup[DETECT_MAX_RULE_SIZE];
 
@@ -1370,6 +1380,10 @@ static int SigParseBasics(DetectEngineCtx *de_ctx,
     }
     strlcpy(parser->opts, index, sizeof(parser->opts));
 
+    if (scan_only) {
+        return 0;
+    }
+
     /* Parse Action */
     if (SigParseAction(s, parser->action) < 0)
         goto error;
@@ -1431,12 +1445,13 @@ static inline bool CheckAscii(const char *str)
  *  \param s memory structure to store the signature in
  *  \param sigstr the raw signature as a null terminated string
  *  \param addrs_direction direction (for bi-directional sigs)
+ *  \param require only scan rule for requires
  *
  *  \param -1 parse error
  *  \param 0 ok
  */
-static int SigParse(DetectEngineCtx *de_ctx, Signature *s,
-        const char *sigstr, uint8_t addrs_direction, SignatureParser *parser)
+static int SigParse(DetectEngineCtx *de_ctx, Signature *s, const char *sigstr,
+        uint8_t addrs_direction, SignatureParser *parser, bool requires)
 {
     SCEnter();
 
@@ -1450,12 +1465,7 @@ static int SigParse(DetectEngineCtx *de_ctx, Signature *s,
         SCReturnInt(-1);
     }
 
-    s->sig_str = SCStrdup(sigstr);
-    if (unlikely(s->sig_str == NULL)) {
-        SCReturnInt(-1);
-    }
-
-    int ret = SigParseBasics(de_ctx, s, sigstr, parser, addrs_direction);
+    int ret = SigParseBasics(de_ctx, s, sigstr, parser, addrs_direction, requires);
     if (ret < 0) {
         SCLogDebug("SigParseBasics failed");
         SCReturnInt(-1);
@@ -1467,21 +1477,27 @@ static int SigParse(DetectEngineCtx *de_ctx, Signature *s,
         char input[buffer_size];
         char output[buffer_size];
         memset(input, 0x00, buffer_size);
-        memcpy(input, parser->opts, strlen(parser->opts)+1);
+        memcpy(input, parser->opts, strlen(parser->opts) + 1);
 
         /* loop the option parsing. Each run processes one option
          * and returns the rest of the option string through the
          * output variable. */
         do {
             memset(output, 0x00, buffer_size);
-            ret = SigParseOptions(de_ctx, s, input, output, buffer_size);
+            ret = SigParseOptions(de_ctx, s, input, output, buffer_size, requires);
             if (ret == 1) {
                 memcpy(input, output, buffer_size);
             }
 
         } while (ret == 1);
+
+        if (ret < 0) {
+            /* Suricata didn't meet the rule requirements, skip. */
+            goto end;
+        }
     }
 
+end:
     DetectIPProtoRemoveAllSMs(de_ctx, s);
 
     SCReturnInt(ret);
@@ -2142,17 +2158,33 @@ static Signature *SigInitHelper(DetectEngineCtx *de_ctx, const char *sigstr,
     if (sig == NULL)
         goto error;
 
+    sig->sig_str = SCStrdup(sigstr);
+    if (unlikely(sig->sig_str == NULL)) {
+        goto error;
+    }
+
     /* default gid to 1 */
     sig->gid = 1;
 
-    int ret = SigParse(de_ctx, sig, sigstr, dir, &parser);
+    /* We do a first parse of the rule in a requires, or scan-only
+     * mode. Syntactic errors will be picked up here, but the only
+     * part of the rule that is validated completely is the "requires"
+     * keyword. */
+    int ret = SigParse(de_ctx, sig, sigstr, dir, &parser, true);
     if (ret == -4) {
         /* Rule requirements not met. */
         de_ctx->sigerror_silent = true;
         de_ctx->sigerror_ok = true;
         de_ctx->sigerror_requires = true;
         goto error;
-    } else if (ret == -3) {
+    } else if (ret < 0) {
+        goto error;
+    }
+
+    /* Now completely parse the rule. */
+    ret = SigParse(de_ctx, sig, sigstr, dir, &parser, false);
+    BUG_ON(ret == -4);
+    if (ret == -3) {
         de_ctx->sigerror_silent = true;
         de_ctx->sigerror_ok = true;
         goto error;

From 71bbba9248e696f0fd2e912ad9631052b3788775 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 29 Nov 2023 12:57:23 -0600
Subject: [PATCH 300/462] detect-parse: parse sid in pre-scan

During the pre-scan for "requires", also parse the SID if possible. If
the rule fails high level parsing (syntax), the SID will not be
parsed.

But every keyword other than "sid" and "requires" should expect to be
provided with a parsed sid.
---
 src/detect-parse.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index 493bee10ea9f..45f188df1167 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -900,10 +900,11 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
     }
     optname = optstr;
 
-    if (requires) {
-        if (strcmp(optname, "requires")) {
-            goto finish;
-        }
+    /* Check for options that are only to be processed during the
+     * first "requires" pass. */
+    bool requires_only = strcmp(optname, "requires") == 0 || strcmp(optname, "sid") == 0;
+    if ((requires && !requires_only) || (!requires && requires_only)) {
+        goto finish;
     }
 
     /* Call option parsing */
@@ -2137,10 +2138,7 @@ static int SigValidate(DetectEngineCtx *de_ctx, Signature *s)
             AppLayerHtpNeedFileInspection();
         }
     }
-    if (s->id == 0) {
-        SCLogError("Signature missing required value \"sid\".");
-        SCReturnInt(0);
-    }
+
     SCReturnInt(1);
 }
 
@@ -2181,6 +2179,12 @@ static Signature *SigInitHelper(DetectEngineCtx *de_ctx, const char *sigstr,
         goto error;
     }
 
+    /* Check for a SID before continuuing. */
+    if (sig->id == 0) {
+        SCLogError("Signature missing required value \"sid\".");
+        goto error;
+    }
+
     /* Now completely parse the rule. */
     ret = SigParse(de_ctx, sig, sigstr, dir, &parser, false);
     BUG_ON(ret == -4);

From 5cc872fa1a328a38939f72f0f21b4e9a46900dee Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 14 Dec 2023 12:32:59 -0600
Subject: [PATCH 301/462] rust.h: don't include util-file.h, not needed

---
 src/rust.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/rust.h b/src/rust.h
index 12c90e67f30d..8ab174f9ec78 100644
--- a/src/rust.h
+++ b/src/rust.h
@@ -18,8 +18,6 @@
 #ifndef __RUST_H__
 #define __RUST_H__
 
-#include "util-file.h"
-
 // hack for include orders cf SCSha256
 typedef struct HttpRangeContainerBlock HttpRangeContainerBlock;
 #include "rust-context.h"

From b453eea1502cc5455f383c4fe82c6011fee93495 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 19 Dec 2023 12:13:23 -0600
Subject: [PATCH 302/462] stats: add rules skipped

Rule skipped is a count of the number of rules that are skipped due to
missing requirements.

Feature: #6637
---
 etc/schema.json         | 3 +++
 src/output-json-stats.c | 1 +
 2 files changed, 4 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index 76d88ec4c8fb..0756acd00800 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -4871,6 +4871,9 @@
                                     },
                                     "rules_failed": {
                                         "type": "integer"
+                                    },
+                                    "rules_skipped": {
+                                        "type": "integer"
                                     }
                                 },
                                 "additionalProperties": false
diff --git a/src/output-json-stats.c b/src/output-json-stats.c
index 7bfcfc58cad2..718298e48592 100644
--- a/src/output-json-stats.c
+++ b/src/output-json-stats.c
@@ -98,6 +98,7 @@ static json_t *EngineStats2Json(const DetectEngineCtx *de_ctx,
                             json_integer(sig_stat->good_sigs_total));
         json_object_set_new(jdata, "rules_failed",
                             json_integer(sig_stat->bad_sigs_total));
+        json_object_set_new(jdata, "rules_skipped", json_integer(sig_stat->skipped_sigs_total));
     }
 
     return jdata;

From f12e0266967a526e0d7ad0ef9cb8fecb126f3889 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Thu, 21 Dec 2023 09:41:29 -0500
Subject: [PATCH 303/462] mqtt: Move conf code to rust

Issue: 6387

This commit moves the configuration logic to Rust.
---
 rust/src/mqtt/mqtt.rs   | 19 +++++++-----
 rust/src/mqtt/parser.rs |  4 +--
 src/Makefile.am         |  2 --
 src/app-layer-mqtt.c    | 64 -----------------------------------------
 src/app-layer-mqtt.h    | 30 -------------------
 src/app-layer-parser.c  |  3 +-
 src/output-json-mqtt.c  |  1 -
 7 files changed, 15 insertions(+), 108 deletions(-)
 delete mode 100644 src/app-layer-mqtt.c
 delete mode 100644 src/app-layer-mqtt.h

diff --git a/rust/src/mqtt/mqtt.rs b/rust/src/mqtt/mqtt.rs
index f1c37d83c881..7f60e2a757cd 100644
--- a/rust/src/mqtt/mqtt.rs
+++ b/rust/src/mqtt/mqtt.rs
@@ -1,4 +1,4 @@
-/* Copyright (C) 2020-2022 Open Information Security Foundation
+/* Copyright (C) 2020-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -21,7 +21,7 @@ use super::mqtt_message::*;
 use super::parser::*;
 use crate::applayer::*;
 use crate::applayer::{self, LoggerFlags};
-use crate::conf::conf_get;
+use crate::conf::{conf_get, get_memval};
 use crate::core::*;
 use crate::frames::*;
 use nom7::Err;
@@ -112,7 +112,7 @@ pub struct MQTTState {
     connected: bool,
     skip_request: usize,
     skip_response: usize,
-    max_msg_len: usize,
+    max_msg_len: u32,
     tx_index_completed: usize,
 }
 
@@ -142,7 +142,7 @@ impl MQTTState {
             connected: false,
             skip_request: 0,
             skip_response: 0,
-            max_msg_len: unsafe { MAX_MSG_LEN as usize },
+            max_msg_len: unsafe { MAX_MSG_LEN},
             tx_index_completed: 0,
         }
     }
@@ -778,10 +778,8 @@ export_tx_data_get!(rs_mqtt_get_tx_data, MQTTTransaction);
 export_state_data_get!(rs_mqtt_get_state_data, MQTTState);
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_mqtt_register_parser(cfg_max_msg_len: u32) {
+pub unsafe extern "C" fn SCMqttRegisterParser() {
     let default_port = CString::new("[1883]").unwrap();
-    let max_msg_len = &mut MAX_MSG_LEN;
-    *max_msg_len = cfg_max_msg_len;
     let parser = RustParser {
         name: PARSER_NAME.as_ptr() as *const std::os::raw::c_char,
         default_port: default_port.as_ptr(),
@@ -830,6 +828,13 @@ pub unsafe extern "C" fn rs_mqtt_register_parser(cfg_max_msg_len: u32) {
                 SCLogError!("Invalid value for mqtt.max-tx");
             }
         }
+        if let Some(val) = conf_get("app-layer.protocols.mqtt.max-msg-length") {
+            if let Ok(v) = get_memval(val) {
+                MAX_MSG_LEN = v as u32;
+            } else {
+                SCLogError!("Invalid value for mqtt.max-msg-length: {}", val);
+            }
+        }
     } else {
         SCLogDebug!("Protocol detector and parser disabled for MQTT.");
     }
diff --git a/rust/src/mqtt/parser.rs b/rust/src/mqtt/parser.rs
index 8b1c8c542aba..9b576e54a5c0 100644
--- a/rust/src/mqtt/parser.rs
+++ b/rust/src/mqtt/parser.rs
@@ -634,7 +634,7 @@ fn parse_remaining_message<'a>(
 pub fn parse_message(
     input: &[u8],
     protocol_version: u8,
-    max_msg_size: usize,
+    max_msg_size: u32,
 ) -> IResult<&[u8], MQTTMessage> {
     // Parse the fixed header first. This is identical across versions and can
     // be between 2 and 5 bytes long.
@@ -652,7 +652,7 @@ pub fn parse_message(
             // limit, we return a special truncation message type, containing
             // no parsed metadata but just the skipped length and the message
             // type.
-            if len > max_msg_size {
+            if len > max_msg_size as usize {
                 let msg = MQTTMessage {
                     header,
                     op: MQTTOperation::TRUNCATED(MQTTTruncatedData {
diff --git a/src/Makefile.am b/src/Makefile.am
index 9cb4f815531c..133ed47cd1e8 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -36,7 +36,6 @@ noinst_HEADERS = \
 	app-layer-krb5.h \
 	app-layer-modbus.h \
 	app-layer-quic.h \
-	app-layer-mqtt.h \
 	app-layer-nfs-tcp.h \
 	app-layer-nfs-udp.h \
 	app-layer-ntp.h \
@@ -655,7 +654,6 @@ libsuricata_c_a_SOURCES = \
 	app-layer-krb5.c \
 	app-layer-modbus.c \
 	app-layer-quic.c \
-	app-layer-mqtt.c \
 	app-layer-nfs-tcp.c \
 	app-layer-nfs-udp.c \
 	app-layer-ntp.c \
diff --git a/src/app-layer-mqtt.c b/src/app-layer-mqtt.c
deleted file mode 100644
index 96b4cc27afcc..000000000000
--- a/src/app-layer-mqtt.c
+++ /dev/null
@@ -1,64 +0,0 @@
-/* Copyright (C) 2020 Open Information Security Foundation
- *
- * You can copy, redistribute or modify this Program under the terms of
- * the GNU General Public License version 2 as published by the Free
- * Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * version 2 along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
- * 02110-1301, USA.
- */
-
-/**
- * \file
- *
- * \author Sascha Steinbiss <sascha@steinbiss.name>
- */
-
-#include "suricata-common.h"
-#include "stream.h"
-#include "conf.h"
-
-#include "util-misc.h"
-#include "util-unittest.h"
-
-#include "app-layer-detect-proto.h"
-#include "app-layer-parser.h"
-
-#include "app-layer-mqtt.h"
-#include "rust.h"
-
-void RegisterMQTTParsers(void)
-{
-    SCLogDebug("Registering Rust mqtt parser.");
-    uint32_t max_msg_len = 1048576; /* default: 1MB */
-
-    if (AppLayerParserConfParserEnabled("tcp", "mqtt")) {
-        ConfNode *p = ConfGetNode("app-layer.protocols.mqtt.max-msg-length");
-        if (p != NULL) {
-            uint32_t value;
-            if (ParseSizeStringU32(p->val, &value) < 0) {
-                SCLogError("invalid value for max-msg-length: %s", p->val);
-            } else {
-                max_msg_len = value;
-            }
-        }
-        rs_mqtt_register_parser(max_msg_len);
-    }
-#ifdef UNITTESTS
-    AppLayerParserRegisterProtocolUnittests(IPPROTO_TCP, ALPROTO_MQTT,
-        MQTTParserRegisterTests);
-#endif
-}
-
-void MQTTParserRegisterTests(void)
-{
-#ifdef UNITTESTS
-#endif
-}
diff --git a/src/app-layer-mqtt.h b/src/app-layer-mqtt.h
deleted file mode 100644
index b55720ec8d73..000000000000
--- a/src/app-layer-mqtt.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/* Copyright (C) 2020 Open Information Security Foundation
- *
- * You can copy, redistribute or modify this Program under the terms of
- * the GNU General Public License version 2 as published by the Free
- * Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * version 2 along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
- * 02110-1301, USA.
- */
-
-/**
- * \file
- *
- * \author Sascha Steinbiss <sascha@steinbiss.name>
- */
-
-#ifndef __APP_LAYER_MQTT_H__
-#define __APP_LAYER_MQTT_H__
-
-void RegisterMQTTParsers(void);
-void MQTTParserRegisterTests(void);
-
-#endif /* __APP_LAYER_MQTT_H__ */
diff --git a/src/app-layer-parser.c b/src/app-layer-parser.c
index 1f6066471757..96fc607fd257 100644
--- a/src/app-layer-parser.c
+++ b/src/app-layer-parser.c
@@ -57,7 +57,6 @@
 #include "app-layer-krb5.h"
 #include "app-layer-sip.h"
 #include "app-layer-rfb.h"
-#include "app-layer-mqtt.h"
 #include "app-layer-snmp.h"
 #include "app-layer-quic.h"
 #include "app-layer-rdp.h"
@@ -1766,7 +1765,7 @@ void AppLayerParserRegisterProtocolParsers(void)
     RegisterQuicParsers();
     rs_template_register_parser();
     RegisterRFBParsers();
-    RegisterMQTTParsers();
+    SCMqttRegisterParser();
     rs_pgsql_register_parser();
     RegisterRdpParsers();
     RegisterHTTP2Parsers();
diff --git a/src/output-json-mqtt.c b/src/output-json-mqtt.c
index 2f600343e20d..b743229f36e3 100644
--- a/src/output-json-mqtt.c
+++ b/src/output-json-mqtt.c
@@ -41,7 +41,6 @@
 #include "app-layer.h"
 #include "app-layer-parser.h"
 
-#include "app-layer-mqtt.h"
 #include "output-json-mqtt.h"
 #include "rust.h"
 

From 08eb67f74c5de65c1ba98345405288d3c5b59050 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 22 Nov 2023 14:03:27 -0300
Subject: [PATCH 304/462] devguide: make 'contributing' a chapter

This could be justified from a semantic point of view, and also can help
in bringing more attention to where this information is, as it is less
hidden, now.

Also add Dev Guide as one of our resources in our Readme.
---
 .github/CONTRIBUTING.md                                      | 2 +-
 .github/PULL_REQUEST_TEMPLATE.md                             | 2 +-
 README.md                                                    | 5 +++--
 doc/userguide/devguide/codebase/index.rst                    | 1 -
 .../{codebase => }/contributing/code-submission-process.rst  | 0
 .../{codebase => }/contributing/contribution-process.rst     | 0
 .../{codebase => }/contributing/github-pr-workflow.rst       | 0
 doc/userguide/devguide/{codebase => }/contributing/index.rst | 0
 doc/userguide/devguide/index.rst                             | 1 +
 doc/userguide/support-status.rst                             | 2 +-
 10 files changed, 7 insertions(+), 6 deletions(-)
 rename doc/userguide/devguide/{codebase => }/contributing/code-submission-process.rst (100%)
 rename doc/userguide/devguide/{codebase => }/contributing/contribution-process.rst (100%)
 rename doc/userguide/devguide/{codebase => }/contributing/github-pr-workflow.rst (100%)
 rename doc/userguide/devguide/{codebase => }/contributing/index.rst (100%)

diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index eaa19b74f9a8..f3b58981fdf0 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -2,7 +2,7 @@ Contributing to Suricata
 ========================
 
 We're happily taking patches and other contributions. The process is documented at
-[Contribution Process](https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html). Please have a look at this document before submitting.
+[Contribution Process](https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html). Please have a look at this document before submitting.
 
 Contribution Agreement
 ----------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 15977b06f540..b62c865f9667 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,7 +1,7 @@
 Make sure these boxes are signed before submitting your Pull Request -- thank you.
 
 - [ ] I have read the contributing guide lines at
-   https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html
+   https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html
 - [ ] I have signed the Open Information Security Foundation contribution agreement at
    https://suricata.io/about/contribution-agreement/ (note: this is only required once)
 - [ ] I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)
diff --git a/README.md b/README.md
index 2b1a213bcbcb..657549be2619 100644
--- a/README.md
+++ b/README.md
@@ -13,6 +13,7 @@ developed by the [OISF](https://oisf.net) and the Suricata community.
 - [Home Page](https://suricata.io)
 - [Bug Tracker](https://redmine.openinfosecfoundation.org/projects/suricata)
 - [User Guide](https://docs.suricata.io)
+- [Dev Guide](https://docs.suricata.io/en/latest/devguide/index.html)
 - [Installation Guide](https://docs.suricata.io/en/latest/install.html)
 - [User Support Forum](https://forum.suricata.io)
 
@@ -20,7 +21,7 @@ developed by the [OISF](https://oisf.net) and the Suricata community.
 
 We're happily taking patches and other contributions. Please see our
 [Contribution
-Process](https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html)
+Process](https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html)
 for how to get started.
 
 Suricata is a complex piece of software dealing with mostly untrusted
@@ -105,7 +106,7 @@ change, it will probably go into the next major version.
 __Q: Why was my PR closed?__
 
 A: As documented in the [Suricata GitHub
-workflow](https://docs.suricata.io/en/latest/devguide/codebase/contributing/github-pr-workflow.html),
+workflow](https://docs.suricata.io/en/latest/devguide/contributing/github-pr-workflow.html),
 we expect a new pull request for every change.
 
 Normally, the team (or community) will give feedback on a pull request
diff --git a/doc/userguide/devguide/codebase/index.rst b/doc/userguide/devguide/codebase/index.rst
index f6cb95592cc7..8f0e493f2057 100644
--- a/doc/userguide/devguide/codebase/index.rst
+++ b/doc/userguide/devguide/codebase/index.rst
@@ -4,7 +4,6 @@ Working with the Codebase
 .. toctree::
    :maxdepth: 2
 
-   contributing/index.rst
    installation-from-git
    code-style
    fuzz-testing
diff --git a/doc/userguide/devguide/codebase/contributing/code-submission-process.rst b/doc/userguide/devguide/contributing/code-submission-process.rst
similarity index 100%
rename from doc/userguide/devguide/codebase/contributing/code-submission-process.rst
rename to doc/userguide/devguide/contributing/code-submission-process.rst
diff --git a/doc/userguide/devguide/codebase/contributing/contribution-process.rst b/doc/userguide/devguide/contributing/contribution-process.rst
similarity index 100%
rename from doc/userguide/devguide/codebase/contributing/contribution-process.rst
rename to doc/userguide/devguide/contributing/contribution-process.rst
diff --git a/doc/userguide/devguide/codebase/contributing/github-pr-workflow.rst b/doc/userguide/devguide/contributing/github-pr-workflow.rst
similarity index 100%
rename from doc/userguide/devguide/codebase/contributing/github-pr-workflow.rst
rename to doc/userguide/devguide/contributing/github-pr-workflow.rst
diff --git a/doc/userguide/devguide/codebase/contributing/index.rst b/doc/userguide/devguide/contributing/index.rst
similarity index 100%
rename from doc/userguide/devguide/codebase/contributing/index.rst
rename to doc/userguide/devguide/contributing/index.rst
diff --git a/doc/userguide/devguide/index.rst b/doc/userguide/devguide/index.rst
index b6b5fbc1c7d3..26a83497114f 100644
--- a/doc/userguide/devguide/index.rst
+++ b/doc/userguide/devguide/index.rst
@@ -5,5 +5,6 @@ Suricata Developer Guide
    :maxdepth: 2
 
    codebase/index.rst
+   contributing/index.rst
    internals/index.rst
    extending/index.rst
diff --git a/doc/userguide/support-status.rst b/doc/userguide/support-status.rst
index 8a870d209a96..49dc31ca7d74 100644
--- a/doc/userguide/support-status.rst
+++ b/doc/userguide/support-status.rst
@@ -77,7 +77,7 @@ support is done by the core team. If someone wants to help maintain
 and support such a feature, we recommend talking to the core team
 before spending a lot of time on it.
 
-Please see :doc:`devguide/codebase/contributing/contribution-process`
+Please see :doc:`devguide/contributing/contribution-process`
 for more information if you wish to contribute.
 
 Distributions

From 71e4ca81ef2ae1bcb0ed6a3d3fc90b8b7e559363 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 22 Nov 2023 12:52:59 -0300
Subject: [PATCH 305/462] devguide: reorganize pr-workflow section

This section seemed to aim both at PR reviewers and PR authors at the
same time, even though some info is probably of low value for
contributors.

Created new section for PR reviewers and maintainers, and kept the info
for PR authors separated. Also highlighted information on requested
changes and stale PRs.
---
 .../contributing/code-submission-process.rst  | 17 ++++---
 .../contributing/github-pr-workflow.rst       | 49 +++++++++++++------
 2 files changed, 43 insertions(+), 23 deletions(-)

diff --git a/doc/userguide/devguide/contributing/code-submission-process.rst b/doc/userguide/devguide/contributing/code-submission-process.rst
index 22bf16046a40..99ac93d5e201 100644
--- a/doc/userguide/devguide/contributing/code-submission-process.rst
+++ b/doc/userguide/devguide/contributing/code-submission-process.rst
@@ -9,16 +9,17 @@ Commits
 #. Commits need to be logically separated. Don't fix unrelated things in one commit.
 #. Don't add unnecessary commits, if commit 2 fixes commit 1 merge them together (squash)
 #. Commits need to have proper messages, explaining anything that is non-trivial
-#. Commits should not at the same time change, rename and/or move code. Use separate commits
-   for each of this, e.g, a commit to rename files, then a commit to change the code.
+#. Commits should not, at the same time, change, rename and/or move code. Use separate commits
+     for each of this, e.g, a commit to rename files, then a commit to change the code.
 #. Documentation updates should be in their own commit (not mixed with code commits)
-#. Commit messages need to be properly formatted:
-    * Meaningful and short (50 chars max) subject line followed by an empty line
-    * Naming convention: prefix message with sub-system ("rule parsing: fixing foobar"). If
-      you're not sure what to use, look at past commits to the file(s) in your PR.
-    * Description, wrapped at ~72 characters
+#. Commit messages need to be properly formatted (check the example further
+     below in this section):
+      * Meaningful and short (50 chars max) subject line followed by an empty line
+      * Naming convention: prefix message with sub-system (**"rule parsing: fixing foobar"**). If
+        you're not sure what to use, look at past commits to the file(s) in your PR.
+      * Description, wrapped at ~72 characters
 #. Commits should be individually compilable, starting with the oldest commit. Make sure that
-   each commit can be built if it and the preceding commits in the PR are used.
+     each commit can be built if it and the preceding commits in the PR are used.
 #. Commits should be authored with the format: "FirstName LastName <name@example.com>"
 
 Information that needs to be part of a commit (if applicable):
diff --git a/doc/userguide/devguide/contributing/github-pr-workflow.rst b/doc/userguide/devguide/contributing/github-pr-workflow.rst
index 618c966c43c2..51ed0568c05d 100644
--- a/doc/userguide/devguide/contributing/github-pr-workflow.rst
+++ b/doc/userguide/devguide/contributing/github-pr-workflow.rst
@@ -4,12 +4,12 @@ GitHub Pull Request Workflow
 Draft Pull Requests
 ~~~~~~~~~~~~~~~~~~~
 
-A Pull Request (PR) should be marked as `draft` if it is not intended to be merged as is,
+A Pull Request (PR) should be marked as *draft* if it is not intended to be merged as is,
 but is waiting for some sort of feedback.
 The author of the PR should be explicit with what kind of feedback is expected
 (CI/QA run, discussion on the code, etc...)
 
-GitHub filter is ``is:pr is:open draft:true sort:updated-asc``
+The GitHub filter is ``is:pr is:open draft:true sort:updated-asc``.
 
 A draft may be closed if it has not been updated in two months.
 
@@ -22,25 +22,44 @@ When a Pull Request is intended to be merged as is, the workflow is the followin
     (and eventually request changes if CI finds anything)
  3. get merged and closed
 
-A newly created PR should match the filter
-``is:pr is:open draft:false review:none sort:updated-asc no:assignee``
+ Once submitted, we aim at providing a first PR review within two weeks and a
+ month.
+
+ If either code, documentation wording or commit messages need re-work, the
+ reviewer will set the PR state to *changes requested*.
+
+.. note:: It is expected that the author will create a new PR with a new version
+   of the patch as described in :ref:`Pull Requests Criteria <pull-requests-criteria>`.
+   A PR may be closed as stale if it has not been updated in two months after
+   changes were requested.
+
+A PR may be labeled *decision-required* if the reviewer thinks the team needs
+more time to analyze the best approach to a proposed solution or discussion
+raised by the PR.
+
+Once in approved state, the PRs are in the responsibility of the maintainer, along
+with the next branches/PRs.
+
+Reviewers and Maintainers
+-------------------------
+
+A newly created PR should match the filter::
+
+    is:pr is:open draft:false review:none sort:updated-asc no:assignee
+
 The whole team is responsible to assign a PR to someone precise within 2 weeks.
 
-When someone gets assigned a PR, the PR should get a review status within 2 weeks:
+When someone gets assigned a PR, it should get a review status within 2 weeks:
 either changes requested, approved, or assigned to someone else if more
 expertise is needed.
 
-GitHub filter for changes-requested PRs is ``is:pr is:open draft:false sort:
-updated-asc review:changes-requested``
+The GitHub filter for changes-requested PRs is::
 
-Such a PR may be closed if it has not been updated in two months.
-It is expected that the author creates a new PR with a new version of the patch
-as described in :ref:`Pull Requests Criteria <pull-requests-criteria>`.
+    is:pr is:open draft:false sort: updated-asc review:changes-requested
 
-Command to get approved PRs is ``gh pr list --json number,reviewDecision --search
-"state:open type:pr -review:none" | jq '.[] | select(.reviewDecision=="")'``
+The command to get approved PRs is::
 
-Web UI filter does not work cf https://github.com/orgs/community/discussions/55826
+    gh pr list --json number,reviewDecision --search "state:open type:pr -review:none" | jq '.[] | select(.reviewDecision=="")'
+
+An approved PR should match the filter: ``is:open is:pr review:approved``.
 
-Once in approved state, the PRs are in the responsibility of the merger, along
-with the next branches/PRs.

From de8bffd244012cef15b9f608a86fce93c434150d Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Fri, 24 Nov 2023 08:29:06 -0300
Subject: [PATCH 306/462] devguide: doc from behavior changes needs ticket #

If a commit introduces code that changes Suricata behavior, the related
documentation changes should go in a separate commit, but refer to the
same ticket number.
This reduces the chances of said changes being lost if there are backports
while still keeping the backporting process a bit less bulky, for each
commit.

Related to
Task #6568
---
 .../devguide/contributing/code-submission-process.rst    | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/doc/userguide/devguide/contributing/code-submission-process.rst b/doc/userguide/devguide/contributing/code-submission-process.rst
index 99ac93d5e201..e6a0f2fd33c8 100644
--- a/doc/userguide/devguide/contributing/code-submission-process.rst
+++ b/doc/userguide/devguide/contributing/code-submission-process.rst
@@ -9,9 +9,12 @@ Commits
 #. Commits need to be logically separated. Don't fix unrelated things in one commit.
 #. Don't add unnecessary commits, if commit 2 fixes commit 1 merge them together (squash)
 #. Commits need to have proper messages, explaining anything that is non-trivial
-#. Commits should not, at the same time, change, rename and/or move code. Use separate commits
-     for each of this, e.g, a commit to rename files, then a commit to change the code.
-#. Documentation updates should be in their own commit (not mixed with code commits)
+#. Commits should not, at the same time, change, rename and/or move code. Use
+   separate commits for each of this, e.g, a commit to rename files, then a commit
+   to change the code.
+#. If your code changes or adds new behavior, add the related documentation
+   updates in their own commit, but make sure to add the same ticket number to
+   both commit messages.
 #. Commit messages need to be properly formatted (check the example further
      below in this section):
       * Meaningful and short (50 chars max) subject line followed by an empty line

From 9fbdfd219c428ea20ccd8e3f8c48590d6ec126c1 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 22 Nov 2023 14:56:56 -0300
Subject: [PATCH 307/462] devguide: add chapter with backports guide

Task #6568
---
 .../devguide/contributing/backports-guide.rst | 128 ++++++++++++++++++
 doc/userguide/devguide/contributing/index.rst |   1 +
 2 files changed, 129 insertions(+)
 create mode 100644 doc/userguide/devguide/contributing/backports-guide.rst

diff --git a/doc/userguide/devguide/contributing/backports-guide.rst b/doc/userguide/devguide/contributing/backports-guide.rst
new file mode 100644
index 000000000000..4d719a6c1d55
--- /dev/null
+++ b/doc/userguide/devguide/contributing/backports-guide.rst
@@ -0,0 +1,128 @@
+========================
+Suricata Backports Guide
+========================
+
+This document describes the processes used to backport content to current stable
+Suricata releases. Most often, this means security and/or bug fixes;
+however, in some cases, features may be backported to previous Suricata releases.
+
+There are multiple versions of Suricata at any given time:
+    * Master
+    * Major stable release
+    * Old stable release
+
+For example, at the moment, there are 3 releases based on these Suricata branches:
+    * master: 8.0.0-dev, current development branch
+    * main-7.0.x: major stable release (note we're changing our naming conventions)
+    * master-6.0.x: old stable release
+
+For Suricata's release cadence and *end of life* policies, please check
+https://suricata.io/our-story/eol-policy/.
+
+The next sections discuss when and what to backport, and some guidelines when
+doing so.
+
+What should be backported?
+--------------------------
+
+Usually, when the team creates a ticket, we'll add the *Needs backport* related
+labels, so necessary backporting tickets will be automatically created. If you
+are working on a ticket that doesn't have such labels, nor backporting tasks
+associated, it probably doesn't need backporting. If you understand that the
+issue should be backported, please let us know in the ticket or related PR. But
+sometimes we'll miss those.
+
+The general principle used to determine what will be backported is:
+    * security fixes (please see our `Security Policy <https://github.com/OISF/suricata/blob/master/SECURITY.md>`_)
+    * bug fixes
+    * in some cases, new features are backported if there are sufficient reasons to
+      backport a new feature.
+
+.. Note:: Exceptions
+
+    There can be cases where backports may be "missed" -- some issues may not be
+    labeled as needing backports and some PRs may be merged without an issue.
+
+    This guide may be insufficient for some situations. When in doubt, please reach
+    out to the team on the backport ticket or PR.
+
+Selection overview
+------------------
+
+All items considered for backports should be reviewed with the following:
+    * risk estimate: will the change introduce new bugs? Consider the scope and
+      items affected by the change.
+    * behavioral change: how much will the behavior of the system be changed by the
+      backport. For example, a small change to decode additional encapsulation
+      protocols may result in more traffic being presented to Suricata.
+    * default settings: if the issue alters behavior, can it be made optional, and
+      at what cost?
+
+Creating backport tickets -- new issues
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Redmine: for security and bug fixes, when creating a new Redmine issue,
+label the Redmine issue with "Needs backport to x.0", where x.0 is a supported
+Suricata release, e.g, 7.0.x.
+
+Creating backports tickets -- existing issues/PRs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We want to minimize the occurrence of "missed backports" -- that is, work that
+should be backported but wasn't. Sometimes this happens when there is no Redmine
+issue, or the Redmine issue wasn't labeled as needing a backport.
+
+Therefore, we will be periodically reviewing:
+    * Redmine issues without backport labels, including recently closed issues, to
+      see which require backport labels.
+    * PRs without associated Redmine issues. Those requiring backports should be
+      labeled with *needs backport*.
+
+Then, also periodically, we will create backport issues from those items
+identified in the previous steps. When doing so, we will evaluate what are the
+relevant target backport releases. Some issues reported against master or the
+current Suricata release may not apply to older releases.
+
+Git Backport Workflow
+---------------------
+
+If you are working on a task that needs to be backported, only start the
+backporting process once the PR for master has been merged. Then:
+
+    * *Identify the commit(s) needed* for the backport. Start with the PR that merged
+      the commits into master and select only the commits from the issue being
+      backported.
+    * *Bring each commit into the new branch,* one at a time -- starting with the
+      oldest commit. Use ``git cherry-pick -x commit-hash``, where ``commit-hash``
+      is the hash to the commit already in master or main-7.0x that is being
+      backported, as it maintains the linkage with said cherry-picked commit.
+    * *Resolve conflicts:* Some of the cherry-picked commits may contain merge
+      conflicts. If the conflicts are small, include the corrections in the
+      cherry-picked commit.
+    * *Add additional commits*, if any are needed (e.g., to adjust cherry-picked code
+      to old behavior).
+
+.. Note:: Commit hashes
+
+   We have a CI check that ensures the validity of the cherry-pick line.
+
+.. Note:: Exceptions
+
+   Sometimes, the fix for master will not work for the stable or old releases.
+   In such cases, the backporting process won't be through cherry-picking, but
+   through actually implementing a fix for the specific version.
+
+Create a PR:
+~~~~~~~~~~~~
+
+Please indicate in the title that this is a backport PR, with something like
+*(7.0.x-backport)*, and add the related milestone label.
+
+In the PR description, indicate the backport ticket.
+
+QA
+--
+
+Add suricata-verify PRs when needed. Some existing suricata-verify tests may require
+version specification changes.
+
diff --git a/doc/userguide/devguide/contributing/index.rst b/doc/userguide/devguide/contributing/index.rst
index e0d29125bbdd..9b1baeaab139 100644
--- a/doc/userguide/devguide/contributing/index.rst
+++ b/doc/userguide/devguide/contributing/index.rst
@@ -7,3 +7,4 @@ Contributing
     contribution-process
     code-submission-process
     github-pr-workflow
+    backports-guide

From d15877b2c0f322864fee6f077a9d5af6333bc1f3 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Fri, 24 Nov 2023 10:07:05 -0300
Subject: [PATCH 308/462] devguide: update branches, refer to backports guide

Update the list of active branches to include 7 renaming and new master,
link to backports document.
---
 .../devguide/contributing/contribution-process.rst | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/doc/userguide/devguide/contributing/contribution-process.rst b/doc/userguide/devguide/contributing/contribution-process.rst
index 198ff142afc2..7903e3d11a3f 100644
--- a/doc/userguide/devguide/contributing/contribution-process.rst
+++ b/doc/userguide/devguide/contributing/contribution-process.rst
@@ -151,18 +151,20 @@ it, so everyone knows that work is still open and waiting to be done.
 What branch to work on
 ======================
 
-There are 2 or 3 active branches:
+There are usually 2 or 3 active branches:
 
-    * master-x.x.x (e.g. master-6.x.y)
+    * master-x.x.x (e.g. master-6.0.x)
+    * main-x.x.x (e.g. main-7.0.x)
     * master
 
-The former is the stable branch. The latter the development branch.
+The ones with version numbers are stable branches. **master** is the development branch.
 
-The stable branch should only be worked on for important bug fixes. Those are
-mainly expected from more experienced contributors.
+The stable branch should only be worked on for important bug fixes or other
+needed :doc:`backports<backports-guide>`. Those are mainly expected from more
+experienced contributors.
 
 Development of new features or large scale redesign is done in the development
-branch. New development and new contributors should work with ``master`` except
+branch. New development and new contributors should work with *master* except
 in very special cases - which should and would be discussed with us first.
 
 If in doubt, please reach out to us via :ref:`Redmine, Discord or

From fc2acf8cb06a665d04db243e829f6ee3f3cc3114 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Fri, 24 Nov 2023 11:46:41 -0300
Subject: [PATCH 309/462] devguide: fix main channels list

Sphinx and RtD sometimes render lists in weird ways. The communication
channels list barely looked like one, at all...
---
 .../devguide/contributing/contribution-process.rst          | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/userguide/devguide/contributing/contribution-process.rst b/doc/userguide/devguide/contributing/contribution-process.rst
index 7903e3d11a3f..eb8e53d820d0 100644
--- a/doc/userguide/devguide/contributing/contribution-process.rst
+++ b/doc/userguide/devguide/contributing/contribution-process.rst
@@ -37,10 +37,10 @@ optimizations, and/or ask for help, it is important to communicate.
 
 These are our main channels:
 
-- `Suricata's issue tracker <https://redmine.openinfosecfoundation.org/
+* `Suricata's issue tracker <https://redmine.openinfosecfoundation.org/
   projects/suricata/issues>`_
-- `Suricata's forum <https://forum.suricata.io/c/developers/8>`_
-- `Suricata's Discord server <https://discord.com/invite/t3rV2x7MrG>`_
+* `Suricata's forum <https://forum.suricata.io/c/developers/8>`_
+* `Suricata's Discord server <https://discord.com/invite/t3rV2x7MrG>`_
 
 
 .. _claim-ticket:

From 673d13d44555843883915eb691b58a195ad021e8 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 4 Jan 2024 09:51:33 +0100
Subject: [PATCH 310/462] rust: allow clippy::items_after_test_module

As clippy began to complain about jsonbuilder.rs
---
 rust/src/lib.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index 84b82bde19f7..15e21c4057d1 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -27,6 +27,9 @@
 #![allow(clippy::let_and_return)]
 #![allow(clippy::uninlined_format_args)]
 
+// We find this is beyond what the linter should flag.
+#![allow(clippy::items_after_test_module)]
+
 // We find this makes sense at time.
 #![allow(clippy::module_inception)]
 

From d3218385e942a4a01f3938f0cd7050f5dc7c9a6f Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 4 Jan 2024 11:42:55 +0100
Subject: [PATCH 311/462] detect: case-insensitive comparison for requires

Ticket: 6656
---
 src/detect-parse.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index 45f188df1167..bf4684045898 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -902,7 +902,7 @@ static int SigParseOptions(DetectEngineCtx *de_ctx, Signature *s, char *optstr,
 
     /* Check for options that are only to be processed during the
      * first "requires" pass. */
-    bool requires_only = strcmp(optname, "requires") == 0 || strcmp(optname, "sid") == 0;
+    bool requires_only = strcasecmp(optname, "requires") == 0 || strcasecmp(optname, "sid") == 0;
     if ((requires && !requires_only) || (!requires && requires_only)) {
         goto finish;
     }

From a37fa627100b2aa28e0cd351246083c726bcf984 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 30 Nov 2023 12:16:27 -0300
Subject: [PATCH 312/462] devguide: explain example-rule container usage

Have these options documented, so that whoever writes rule-related
documentation can easily know what they could use to make the doc look
better.
---
 .../contributing/contribution-process.rst     | 63 ++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/devguide/contributing/contribution-process.rst b/doc/userguide/devguide/contributing/contribution-process.rst
index eb8e53d820d0..e3058d716525 100644
--- a/doc/userguide/devguide/contributing/contribution-process.rst
+++ b/doc/userguide/devguide/contributing/contribution-process.rst
@@ -199,7 +199,7 @@ Documentation Style
 For documenting *code*, please follow Rust documentation and/or Doxygen
 guidelines, according to what your contribution is using (Rust or C).
 
-If you are writing or updating *documentation pages*, please:
+When writing or updating *documentation pages*, please:
 
 * wrap up lines at 79 (80 at most) characters;
 * when adding diagrams or images, we prefer alternatives that can be generated
@@ -208,6 +208,67 @@ If you are writing or updating *documentation pages*, please:
   /docs.suricata.io/en/latest/#suricata-user-guide>`_ and can also be
   built to pdf, so it is important that it looks good in such formats.
 
+Rule examples
+-------------
+
+.. role:: example-rule-action
+.. role:: example-rule-header
+.. role:: example-rule-options
+.. role:: example-rule-emphasis
+
+For rule documentation, we have a special container::
+
+    example-rule
+
+This will present the rule in a box with an easier to read font size, and also
+allows highlighting specific elements in the signature, as the names indicate
+- action, header, options, or emphasize custom portions:
+
+    - example-rule-action
+    - example-rule-header
+    - example-rule-options
+    - example-rule-emphasis
+
+When using these, indicate the portion to be highlighted by surrounding it with
+` . Before using them, one has to invoke the specific role, like so::
+
+    .. role:: example-rule-role
+
+It is only necessary to invoke the role once per document. One can see these
+being invoked in our introduction to the rule language (see `Rules intro
+<https://raw.githubusercontent.com/OISF/suricata/master/doc/userguide/rules/intro.rst>`_).
+
+A rule example like::
+
+    .. container:: example-rule
+
+    :example-rule-action:`alert` :example-rule-header:`http $HOME_NET any ->
+    $EXTERNAL_NET any` :example-rule-options:`(msg:"HTTP GET Request Containing
+    Rule in URI"; flow:established,to_server; http.method; content:"GET"; http.uri;
+    content:"rule"; fast_pattern; classtype:bad-unknown; sid:123; rev:1;)`
+
+Results in:
+
+.. container:: example-rule
+
+    :example-rule-action:`alert` :example-rule-header:`http $HOME_NET any ->
+    $EXTERNAL_NET any`  :example-rule-options:`(msg:"HTTP GET Request Containing
+    Rule in URI"; flow:established,to_server; http.method; content:"GET"; http.uri;
+    content:"rule"; fast_pattern; classtype:bad-unknown; sid:123; rev:1;)`
+
+Example - emphasis::
+
+    .. container:: example-rule
+
+    alert ssh any any -> any any (msg:"match SSH protocol version";
+    :example-rule-emphasis:`ssh.proto;` content:"2.0"; sid:1000010;)
+
+Renders as:
+
+.. container:: example-rule
+
+    alert ssh any any -> any any (msg:"match SSH protocol version";
+    :example-rule-emphasis:`ssh.proto;` content:"2.0"; sid:1000010;)
 
 Commit History matters
 ======================

From 4933b817aacc649edc52409426500a9ec271ccc6 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 6 Dec 2023 20:32:50 +0100
Subject: [PATCH 313/462] doc: fix byte_test examples

As this keyword has 4 mandatory arguments, and some examples
had only three...

Ticket: 6629
---
 doc/userguide/rules/payload-keywords.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/doc/userguide/rules/payload-keywords.rst b/doc/userguide/rules/payload-keywords.rst
index 412f7b4fe0e4..9a609a217f04 100644
--- a/doc/userguide/rules/payload-keywords.rst
+++ b/doc/userguide/rules/payload-keywords.rst
@@ -412,23 +412,23 @@ Example::
 
   alert tcp any any -> any any \
 	 (msg:"Byte_Test Example - Num = Value"; \
-	 content:"|00 01 00 02|"; byte_test:2,=,0x01;)
+	 content:"|00 01 00 02|"; byte_test:2,=,0x01,0;)
 
   alert tcp any any -> any any \
 	 (msg:"Byte_Test Example - Num = Value relative to content"; \
-	 content:"|00 01 00 02|"; byte_test:2,=,0x03,relative;)
+	 content:"|00 01 00 02|"; byte_test:2,=,0x03,2,relative;)
 
   alert tcp any any -> any any \
 	 (msg:"Byte_Test Example - Num != Value"; content:"|00 01 00 02|"; \
-	 byte_test:2,!=,0x06;)
+	 byte_test:2,!=,0x06,0;)
 
   alert tcp any any -> any any \ 
          (msg:"Byte_Test Example - Detect Large Values"; content:"|00 01 00 02|"; \
-         byte_test:2,>,1000,relative;)
+         byte_test:2,>,1000,1,relative;)
 
   alert tcp any any -> any any \
 	 (msg:"Byte_Test Example - Lowest bit is set"; \
-	 content:"|00 01 00 02|"; byte_test:2,&,0x01,relative;)
+	 content:"|00 01 00 02|"; byte_test:2,&,0x01,12,relative;)
 
   alert tcp any any -> any any (msg:"Byte_Test Example - Compare to String"; \
  	 content:"foobar"; byte_test:4,=,1337,1,relative,string,dec;)

From 8d3de85edda97b0c585253803da3faeccc257942 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 11 Oct 2023 17:24:45 -0300
Subject: [PATCH 314/462] pgsql: fix u16 overflow in query data_row

Found by oss-fuzz with quadfuzz.

Cf https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63113

According to PostgreSQL documentation the maximum number of rows can be
the maximum of tuples that can fit onto max u32 pages - 4,294,967,295 (cf
https://www.postgresql.org/docs/current/limits.html). Some rough
calculations for that indicate that this could go over max u32, so
updating the data_row data type to u64.

Bug #6389
---
 rust/src/pgsql/logger.rs | 2 +-
 rust/src/pgsql/parser.rs | 2 +-
 rust/src/pgsql/pgsql.rs  | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/rust/src/pgsql/logger.rs b/rust/src/pgsql/logger.rs
index d54b97b3e1a1..934b549a1671 100644
--- a/rust/src/pgsql/logger.rs
+++ b/rust/src/pgsql/logger.rs
@@ -237,7 +237,7 @@ fn log_response(res: &PgsqlBEMessage, jb: &mut JsonBuilder) -> Result<(), JsonEr
             row_cnt,
             data_size,
         }) => {
-            jb.set_uint("data_rows", (*row_cnt).into())?;
+            jb.set_uint("data_rows", *row_cnt)?;
             jb.set_uint("data_size", *data_size)?;
         }
         PgsqlBEMessage::NotificationResponse(NotificationResponse {
diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 3b8afcabf306..97a16b57384f 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -210,7 +210,7 @@ pub struct BackendKeyDataMessage {
 #[derive(Debug, PartialEq, Eq)]
 pub struct ConsolidatedDataRowPacket {
     pub identifier: u8,
-    pub row_cnt: u16,
+    pub row_cnt: u64,
     pub data_size: u64,
 }
 
diff --git a/rust/src/pgsql/pgsql.rs b/rust/src/pgsql/pgsql.rs
index d2d0a02f88da..5c46008c379c 100644
--- a/rust/src/pgsql/pgsql.rs
+++ b/rust/src/pgsql/pgsql.rs
@@ -50,7 +50,7 @@ pub struct PgsqlTransaction {
     pub request: Option<PgsqlFEMessage>,
     pub responses: Vec<PgsqlBEMessage>,
 
-    pub data_row_cnt: u16,
+    pub data_row_cnt: u64,
     pub data_size: u64,
 
     tx_data: AppLayerTxData,
@@ -82,10 +82,10 @@ impl PgsqlTransaction {
     }
 
     pub fn incr_row_cnt(&mut self) {
-        self.data_row_cnt += 1;
+        self.data_row_cnt = self.data_row_cnt.saturating_add(1);
     }
 
-    pub fn get_row_cnt(&self) -> u16 {
+    pub fn get_row_cnt(&self) -> u64 {
         self.data_row_cnt
     }
 

From bcb2b50cfc34430e0e91dea781c90d2259ef8f0d Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 16 Dec 2023 16:45:00 +0100
Subject: [PATCH 315/462] detect/profiling: improve pcap reading performance

When reading a pcap, packet time can move much faster than wall
clock time. This would trigger many more profile syncs than before.

As the sync is using a lock to synchronize with other threads, this
is an expensive operation.

Bug: #6619.

Fixes: b591813b8690 ("profiling/rules: reduce sync logic scope")
---
 src/detect.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index 5cb4e6bfbc44..d671a3866fa5 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -1797,9 +1797,11 @@ TmEcode Detect(ThreadVars *tv, Packet *p, void *data)
 
 #ifdef PROFILE_RULES
     /* aggregate statistics */
-    if (SCTIME_SECS(p->ts) != det_ctx->rule_perf_last_sync) {
+    struct timeval ts;
+    gettimeofday(&ts, NULL);
+    if (ts.tv_sec != det_ctx->rule_perf_last_sync) {
         SCProfilingRuleThreatAggregate(det_ctx);
-        det_ctx->rule_perf_last_sync = SCTIME_SECS(p->ts);
+        det_ctx->rule_perf_last_sync = ts.tv_sec;
     }
 #endif
 

From e3f2b3418aa3aa33e4007fe5e3a7db59b9279b8c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 17 Dec 2023 18:02:35 +0100
Subject: [PATCH 316/462] detect/content-inspect: use of replace keyword is
 rare

Hint compiler about this.
---
 src/detect-engine-content-inspection.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index 81f2e30f31a6..b204d1edaa7e 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -337,7 +337,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
 
             if ((cd->flags & DETECT_CONTENT_ENDS_WITH) == 0 || match_offset == buffer_len) {
                 /* Match branch, add replace to the list if needed */
-                if (cd->flags & DETECT_CONTENT_REPLACE) {
+                if (unlikely(cd->flags & DETECT_CONTENT_REPLACE)) {
                     if (inspection_mode == DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD) {
                         /* we will need to replace content if match is confirmed
                          * cast to non-const as replace writes to it. */

From 9dc35fbd00e178a4951e76b78ddf5878d4acdea5 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 17 Dec 2023 21:55:42 +0100
Subject: [PATCH 317/462] detect/pcre: put commonly used members on first cache
 line

---
 src/detect-pcre.h | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/src/detect-pcre.h b/src/detect-pcre.h
index 79fd1af74ae2..3ac583369764 100644
--- a/src/detect-pcre.h
+++ b/src/detect-pcre.h
@@ -41,15 +41,14 @@
 #define SC_MATCH_LIMIT_RECURSION_DEFAULT 1500
 
 typedef struct DetectPcreData_ {
-    /* pcre options */
     DetectParseRegex parse_regex;
+    int thread_ctx_id;
 
     int opts;
     uint16_t flags;
     uint8_t idx;
     uint8_t captypes[DETECT_PCRE_CAPTURE_MAX];
     uint32_t capids[DETECT_PCRE_CAPTURE_MAX];
-    int thread_ctx_id;
 } DetectPcreData;
 
 /* prototypes */

From 18dfa69364b0cdea4959fa50bbd1063de0cb0575 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 20 Dec 2023 09:33:20 +0100
Subject: [PATCH 318/462] detect/pcre: remove unused opts field

---
 src/detect-pcre.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/detect-pcre.h b/src/detect-pcre.h
index 3ac583369764..2d78dd0047fc 100644
--- a/src/detect-pcre.h
+++ b/src/detect-pcre.h
@@ -44,7 +44,6 @@ typedef struct DetectPcreData_ {
     DetectParseRegex parse_regex;
     int thread_ctx_id;
 
-    int opts;
     uint16_t flags;
     uint8_t idx;
     uint8_t captypes[DETECT_PCRE_CAPTURE_MAX];

From eca6639a82a06e29f0aec47d98a3aa4153c32cac Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 20 Dec 2023 09:42:53 +0100
Subject: [PATCH 319/462] detect/pcre: localize match limit option parsing

No need to put it into a per ctx flag.
---
 src/detect-pcre.c | 6 ++++--
 src/detect-pcre.h | 1 -
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/detect-pcre.c b/src/detect-pcre.c
index 848f6b9680fe..0c22e8d8eafb 100644
--- a/src/detect-pcre.c
+++ b/src/detect-pcre.c
@@ -360,6 +360,8 @@ static DetectPcreData *DetectPcreParse (DetectEngineCtx *de_ctx,
     int check_host_header = 0;
     char op_str[64] = "";
 
+    bool apply_match_limit = false;
+
     int cut_capture = 0;
     char *fcap = strstr(regexstr, "flow:");
     char *pcap = strstr(regexstr, "pkt:");
@@ -472,7 +474,7 @@ static DetectPcreData *DetectPcreParse (DetectEngineCtx *de_ctx,
                     break;
 
                 case 'O':
-                    pd->flags |= DETECT_PCRE_MATCH_LIMIT;
+                    apply_match_limit = true;
                     break;
 
                 case 'B': /* snort's option */
@@ -678,7 +680,7 @@ static DetectPcreData *DetectPcreParse (DetectEngineCtx *de_ctx,
     }
     pd->parse_regex.match = pcre2_match_data_create_from_pattern(pd->parse_regex.regex, NULL);
 
-    if (pd->flags & DETECT_PCRE_MATCH_LIMIT) {
+    if (apply_match_limit) {
         if (pcre_match_limit >= -1) {
             pcre2_set_match_limit(pd->parse_regex.context, pcre_match_limit);
         }
diff --git a/src/detect-pcre.h b/src/detect-pcre.h
index 2d78dd0047fc..9ac14ab90aa4 100644
--- a/src/detect-pcre.h
+++ b/src/detect-pcre.h
@@ -31,7 +31,6 @@
 #define DETECT_PCRE_RAWBYTES            0x00002
 #define DETECT_PCRE_CASELESS            0x00004
 
-#define DETECT_PCRE_MATCH_LIMIT         0x00020
 #define DETECT_PCRE_RELATIVE_NEXT       0x00040
 #define DETECT_PCRE_NEGATE              0x00080
 

From 3e8db9768753dc8a4c0c3c4b6f1839bdedf54a41 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 18 Dec 2023 08:04:52 +0100
Subject: [PATCH 320/462] detect/bytemath: fix u32 buffer size logic

Remove u16 cast. Remove debug assert for u16 size.

In 83ed2c3b97925d390c2a57fdc8eea52f7d3d2e4c the input was changed to
u32
---
 src/detect-engine-content-inspection.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/src/detect-engine-content-inspection.c b/src/detect-engine-content-inspection.c
index b204d1edaa7e..a4fa5c72b7d7 100644
--- a/src/detect-engine-content-inspection.c
+++ b/src/detect-engine-content-inspection.c
@@ -594,8 +594,7 @@ static int DetectEngineContentInspectionInternal(DetectEngineThreadCtx *det_ctx,
             nbytes = bmd->nbytes;
         }
 
-        DEBUG_VALIDATE_BUG_ON(buffer_len > UINT16_MAX);
-        if (DetectByteMathDoMatch(det_ctx, bmd, s, buffer, (uint16_t)buffer_len, nbytes, rvalue,
+        if (DetectByteMathDoMatch(det_ctx, bmd, s, buffer, buffer_len, nbytes, rvalue,
                     &det_ctx->byte_values[bmd->local_id], endian) != 1) {
             goto no_match;
         }

From f2e9c258c47ed232499c93885b155c9a66644e50 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 18 Dec 2023 11:29:01 +0100
Subject: [PATCH 321/462] detect/pcre: remove unused match member

pcre2_match_data is created per thread when needed.
---
 src/detect-parse.c | 7 -------
 src/detect-parse.h | 1 -
 src/detect-pcre.c  | 1 -
 3 files changed, 9 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index bf4684045898..c259fd36f943 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -2710,9 +2710,6 @@ void DetectParseFreeRegex(DetectParseRegex *r)
     if (r->context) {
         pcre2_match_context_free(r->context);
     }
-    if (r->match) {
-        pcre2_match_data_free(r->match);
-    }
 }
 
 void DetectParseFreeRegexes(void)
@@ -2738,7 +2735,6 @@ void DetectParseRegexAddToFreeList(DetectParseRegex *detect_parse)
         FatalError("failed to alloc memory for pcre free list");
     }
     r->regex = detect_parse->regex;
-    r->match = detect_parse->match;
     r->next = g_detect_parse_regex_list;
     g_detect_parse_regex_list = r;
 }
@@ -2758,8 +2754,6 @@ bool DetectSetupParseRegexesOpts(const char *parse_str, DetectParseRegex *detect
                 parse_str, en, errbuffer);
         return false;
     }
-    detect_parse->match = pcre2_match_data_create_from_pattern(detect_parse->regex, NULL);
-
     DetectParseRegexAddToFreeList(detect_parse);
 
     return true;
@@ -2785,7 +2779,6 @@ DetectParseRegex *DetectSetupPCRE2(const char *parse_str, int opts)
         SCFree(detect_parse);
         return NULL;
     }
-    detect_parse->match = pcre2_match_data_create_from_pattern(detect_parse->regex, NULL);
 
     detect_parse->next = g_detect_parse_regex_list;
     g_detect_parse_regex_list = detect_parse;
diff --git a/src/detect-parse.h b/src/detect-parse.h
index 990180141058..900771a3a9e6 100644
--- a/src/detect-parse.h
+++ b/src/detect-parse.h
@@ -62,7 +62,6 @@ enum {
 typedef struct DetectParseRegex {
     pcre2_code *regex;
     pcre2_match_context *context;
-    pcre2_match_data *match;
     struct DetectParseRegex *next;
 } DetectParseRegex;
 
diff --git a/src/detect-pcre.c b/src/detect-pcre.c
index 0c22e8d8eafb..6d20dd08bb5d 100644
--- a/src/detect-pcre.c
+++ b/src/detect-pcre.c
@@ -678,7 +678,6 @@ static DetectPcreData *DetectPcreParse (DetectEngineCtx *de_ctx,
         SCLogError("pcre2 could not create match context");
         goto error;
     }
-    pd->parse_regex.match = pcre2_match_data_create_from_pattern(pd->parse_regex.regex, NULL);
 
     if (apply_match_limit) {
         if (pcre_match_limit >= -1) {

From fd75aca1a1fbc4c877f6bcd66240cf777ab7b5bb Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 17 Dec 2023 19:54:43 +0100
Subject: [PATCH 322/462] detect/bytetest: remove unused Match function

All matching is done as part of content inspection.
---
 src/detect-bytetest.c | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/src/detect-bytetest.c b/src/detect-bytetest.c
index 481eb51136db..e637c5999ce6 100644
--- a/src/detect-bytetest.c
+++ b/src/detect-bytetest.c
@@ -68,8 +68,6 @@
 
 static DetectParseRegex parse_regex;
 
-static int DetectBytetestMatch(DetectEngineThreadCtx *det_ctx,
-                        Packet *p, const Signature *s, const SigMatchCtx *ctx);
 static int DetectBytetestSetup(DetectEngineCtx *de_ctx, Signature *s, const char *optstr);
 static void DetectBytetestFree(DetectEngineCtx *, void *ptr);
 #ifdef UNITTESTS
@@ -81,7 +79,6 @@ void DetectBytetestRegister (void)
     sigmatch_table[DETECT_BYTETEST].name = "byte_test";
     sigmatch_table[DETECT_BYTETEST].desc = "extract <num of bytes> and perform an operation selected with <operator> against the value in <test value> at a particular <offset>";
     sigmatch_table[DETECT_BYTETEST].url = "/rules/payload-keywords.html#byte-test";
-    sigmatch_table[DETECT_BYTETEST].Match = DetectBytetestMatch;
     sigmatch_table[DETECT_BYTETEST].Setup = DetectBytetestSetup;
     sigmatch_table[DETECT_BYTETEST].Free  = DetectBytetestFree;
 #ifdef UNITTESTS
@@ -313,13 +310,6 @@ int DetectBytetestDoMatch(DetectEngineThreadCtx *det_ctx, const Signature *s,
 
 }
 
-static int DetectBytetestMatch(DetectEngineThreadCtx *det_ctx,
-                        Packet *p, const Signature *s, const SigMatchCtx *ctx)
-{
-    return DetectBytetestDoMatch(det_ctx, s, ctx, p->payload, p->payload_len,
-            ((DetectBytetestData *)ctx)->flags, 0, 0, 0);
-}
-
 static DetectBytetestData *DetectBytetestParse(
         const char *optstr, char **value, char **offset, char **nbytes_str)
 {

From 222dcf776e58f314ce454f202f8bf76466f88597 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 19 Dec 2023 07:52:45 +0100
Subject: [PATCH 323/462] detect/content-inspect: add negation tests

Test mixing of negation, endswith and depth.
---
 src/tests/detect-engine-content-inspection.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/tests/detect-engine-content-inspection.c b/src/tests/detect-engine-content-inspection.c
index c4e1a3bdff68..4a72402d35aa 100644
--- a/src/tests/detect-engine-content-inspection.c
+++ b/src/tests/detect-engine-content-inspection.c
@@ -202,6 +202,12 @@ static int DetectEngineContentInspectionTest08(void) {
     TEST_RUN("abcdefghy", 9,
             "content:\"a\"; content:!\"x\"; content:!\"c\"; distance:2; within:1; ", true, 3);
 
+    TEST_RUN("aaabbbccc", 9, "content:\"ccc\"; endswith; content:!\"bccc\"; endswith; ", false, 2);
+    TEST_RUN("aaabbbccc", 9, "content:\"ccc\"; endswith; content:!\"accc\"; endswith; ", true, 2);
+    TEST_RUN("aaabbbccc", 9, "content:\"ccc\"; endswith; content:!\"bccc\"; endswith; depth:4; ",
+            true, 2);
+    TEST_RUN("aaabbbccc", 9, "content:\"ccc\"; endswith; content:!\"bccc\"; endswith; depth:9; ",
+            false, 2);
     TEST_FOOTER;
 }
 

From bd66504a436ef87d88260badb9efdd24b8169327 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 17 Dec 2023 10:24:56 +0100
Subject: [PATCH 324/462] detect: implement --qa-skip-prefilter

Option meant for testing performance of rule engine w/o prefilter
optimizations.
---
 src/detect-engine-build.c | 6 +++---
 src/detect-engine-mpm.c   | 5 +++++
 src/suricata.c            | 5 +++++
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 70611df46f5c..b9d0c9a4380b 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1371,6 +1371,7 @@ void SignatureSetType(DetectEngineCtx *de_ctx, Signature *s)
     }
 }
 
+extern int g_skip_prefilter;
 /**
  * \brief Preprocess signature, classify ip-only, etc, build sig array
  *
@@ -1459,9 +1460,8 @@ int SigPrepareStage1(DetectEngineCtx *de_ctx)
         RuleSetWhitelist(s);
 
         /* if keyword engines are enabled in the config, handle them here */
-        if (de_ctx->prefilter_setting == DETECT_PREFILTER_AUTO &&
-                !(s->flags & SIG_FLAG_PREFILTER))
-        {
+        if (!g_skip_prefilter && de_ctx->prefilter_setting == DETECT_PREFILTER_AUTO &&
+                !(s->flags & SIG_FLAG_PREFILTER)) {
             int prefilter_list = DETECT_TBLSIZE;
 
             // TODO buffers?
diff --git a/src/detect-engine-mpm.c b/src/detect-engine-mpm.c
index 6a637ffc6431..84f9a6c8feab 100644
--- a/src/detect-engine-mpm.c
+++ b/src/detect-engine-mpm.c
@@ -1066,8 +1066,13 @@ static SigMatch *GetMpmForList(const Signature *s, SigMatch *list, SigMatch *mpm
     return mpm_sm;
 }
 
+int g_skip_prefilter = 0;
+
 void RetrieveFPForSig(const DetectEngineCtx *de_ctx, Signature *s)
 {
+    if (g_skip_prefilter)
+        return;
+
     if (s->init_data->mpm_sm != NULL)
         return;
 
diff --git a/src/suricata.c b/src/suricata.c
index ffa970ae7297..257cb7bc1df6 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -1302,6 +1302,8 @@ static bool IsLogDirectoryWritable(const char* str)
     return false;
 }
 
+extern int g_skip_prefilter;
+
 static TmEcode ParseCommandLine(int argc, char** argv, SCInstance *suri)
 {
     int opt;
@@ -1396,6 +1398,9 @@ static TmEcode ParseCommandLine(int argc, char** argv, SCInstance *suri)
         {"simulate-packet-tcp-ssn-memcap", required_argument, 0, 0},
         {"simulate-packet-defrag-memcap", required_argument, 0, 0},
         {"simulate-alert-queue-realloc-failure", 0, 0, 0},
+
+        {"qa-skip-prefilter", 0, &g_skip_prefilter, 1 },
+
         {"include", required_argument, 0, 0},
 
         {NULL, 0, NULL, 0}

From 4558c5c515866d4271470b9425455136125f8a5f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 19 Dec 2023 14:41:21 +0100
Subject: [PATCH 325/462] detect/http_server_body: modernize test

---
 src/tests/detect-http-server-body.c | 76 ++++++-----------------------
 1 file changed, 16 insertions(+), 60 deletions(-)

diff --git a/src/tests/detect-http-server-body.c b/src/tests/detect-http-server-body.c
index 89180fe56b98..5f102ee9bb9e 100644
--- a/src/tests/detect-http-server-body.c
+++ b/src/tests/detect-http-server-body.c
@@ -1989,11 +1989,9 @@ libhtp:\n\
     Packet *p1 = NULL;
     Packet *p2 = NULL;
     ThreadVars th_v;
-    DetectEngineCtx *de_ctx = NULL;
     DetectEngineThreadCtx *det_ctx = NULL;
     HtpState *http_state = NULL;
     Flow f;
-    int result = 0;
     uint8_t http_buf1[] =
         "GET /index.html HTTP/1.0\r\n"
         "Host: www.openinfosecfoundation.org\r\n"
@@ -2036,94 +2034,52 @@ libhtp:\n\
 
     StreamTcpInitConfig(true);
 
-    de_ctx = DetectEngineCtxInit();
-    if (de_ctx == NULL)
-        goto end;
-
+    DetectEngineCtx *de_ctx = DetectEngineCtxInit();
+    FAIL_IF_NULL(de_ctx);
     de_ctx->flags |= DE_QUIET;
 
-    de_ctx->sig_list = SigInit(de_ctx,"alert http any any -> any any "
-                               "(msg:\"http server body test\"; "
-                               "content:\"890\"; within:3; http_server_body; "
-                               "sid:1;)");
-    if (de_ctx->sig_list == NULL)
-        goto end;
+    Signature *s = DetectEngineAppendSig(de_ctx, "alert http any any -> any any ("
+                                                 "content:\"890\"; within:3; http_server_body; "
+                                                 "sid:1;)");
+    FAIL_IF_NULL(s);
 
     SigGroupBuild(de_ctx);
     DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
 
     int r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, http_buf1, http_len1);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: ", r);
-        result = 0;
-        goto end;
-    }
+    FAIL_IF(r != 0);
 
     http_state = f.alstate;
-    if (http_state == NULL) {
-        printf("no http state: \n");
-        result = 0;
-        goto end;
-    }
+    FAIL_IF_NULL(http_state);
 
-    /* do detect */
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);
-
-    if (PacketAlertCheck(p1, 1)) {
-        printf("sid 1 matched but shouldn't have\n");
-        goto end;
-    }
+    FAIL_IF(PacketAlertCheck(p1, 1));
 
     r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOCLIENT, http_buf2, http_len2);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: \n", r);
-        result = 0;
-        goto end;
-    }
+    FAIL_IF(r != 0);
 
-    /* do detect */
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p2);
-
-    if (PacketAlertCheck(p2, 1)) {
-        printf("sid 1 matched but shouldn't have\n");
-        goto end;
-    }
+    FAIL_IF(PacketAlertCheck(p2, 1));
 
     r = AppLayerParserParse(
             NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOCLIENT, http_buf3, http_len3);
-    if (r != 0) {
-        printf("toserver chunk 1 returned %" PRId32 ", expected 0: \n", r);
-        result = 0;
-        goto end;
-    }
+    FAIL_IF(r != 0);
 
-    /* do detect */
     SigMatchSignatures(&th_v, de_ctx, det_ctx, p2);
+    FAIL_IF(PacketAlertCheck(p2, 1));
 
-    if (PacketAlertCheck(p2, 1)) {
-        printf("sid 1 matched but shouldn't have\n");
-        goto end;
-    }
-
-    result = 1;
-
-end:
-    if (alp_tctx != NULL)
-        AppLayerParserThreadCtxFree(alp_tctx);
+    AppLayerParserThreadCtxFree(alp_tctx);
     HTPFreeConfig();
     HtpConfigRestoreBackup();
     ConfRestoreContextBackup();
-
-    if (de_ctx != NULL)
-        DetectEngineCtxFree(de_ctx);
-
+    DetectEngineCtxFree(de_ctx);
     StreamTcpFreeConfig(true);
     FLOW_DESTROY(&f);
     UTHFreePackets(&p1, 1);
     UTHFreePackets(&p2, 1);
-    return result;
+    PASS;
 }
 
 static int DetectEngineHttpServerBodyTest17(void)

From ea5cf44fc2d4053325319673db0bc94020c64ad1 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 19 Dec 2023 11:11:10 +0100
Subject: [PATCH 326/462] mpm: remove unused flags field

---
 src/util-mpm.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/util-mpm.h b/src/util-mpm.h
index d3ac12bdec89..fe07bebceb79 100644
--- a/src/util-mpm.h
+++ b/src/util-mpm.h
@@ -164,7 +164,6 @@ typedef struct MpmTableElmt_ {
     void (*PrintCtx)(struct MpmCtx_ *);
     void (*PrintThreadCtx)(struct MpmThreadCtx_ *);
     void (*RegisterUnittests)(void);
-    uint8_t flags;
 } MpmTableElmt;
 
 extern MpmTableElmt mpm_table[MPM_TABLE_SIZE];

From 4a6a3dc296a13ac1d5422d8bace2f41805c0a8df Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 19 Dec 2023 11:31:48 +0100
Subject: [PATCH 327/462] mpm: UNITTESTS guard for RegisterUnittests func

---
 src/util-mpm-ac-ks.c | 12 ++++++------
 src/util-mpm-ac.c    | 15 ++++++---------
 src/util-mpm-hs.c    | 17 +++++++----------
 src/util-mpm.h       |  2 ++
 4 files changed, 21 insertions(+), 25 deletions(-)

diff --git a/src/util-mpm-ac-ks.c b/src/util-mpm-ac-ks.c
index 465b66918b62..df36452be3f3 100644
--- a/src/util-mpm-ac-ks.c
+++ b/src/util-mpm-ac-ks.c
@@ -95,7 +95,9 @@ uint32_t SCACTileSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                         PrefilterRuleStore *pmq, const uint8_t *buf,
                         uint32_t buflen);
 void SCACTilePrintInfo(MpmCtx *mpm_ctx);
-void SCACTileRegisterTests(void);
+#ifdef UNITTESTS
+static void SCACTileRegisterTests(void);
+#endif
 
 uint32_t SCACTileSearchLarge(const SCACTileSearchCtx *ctx, MpmThreadCtx *mpm_thread_ctx,
                              PrefilterRuleStore *pmq,
@@ -1403,7 +1405,9 @@ void MpmACTileRegister(void)
     mpm_table[MPM_AC_KS].Prepare = SCACTilePreparePatterns;
     mpm_table[MPM_AC_KS].Search = SCACTileSearch;
     mpm_table[MPM_AC_KS].PrintCtx = SCACTilePrintInfo;
+#ifdef UNITTESTS
     mpm_table[MPM_AC_KS].RegisterUnittests = SCACTileRegisterTests;
+#endif
 }
 
 
@@ -2384,12 +2388,8 @@ static int SCACTileTest29(void)
     return result;
 }
 
-#endif /* UNITTESTS */
-
 void SCACTileRegisterTests(void)
 {
-
-#ifdef UNITTESTS
     UtRegisterTest("SCACTileTest01", SCACTileTest01);
     UtRegisterTest("SCACTileTest02", SCACTileTest02);
     UtRegisterTest("SCACTileTest03", SCACTileTest03);
@@ -2419,8 +2419,8 @@ void SCACTileRegisterTests(void)
     UtRegisterTest("SCACTileTest27", SCACTileTest27);
     UtRegisterTest("SCACTileTest28", SCACTileTest28);
     UtRegisterTest("SCACTileTest29", SCACTileTest29);
-#endif
 }
+#endif
 
 #else /* we're big endian */
 
diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index 6d0fc050b99a..94de6000b2f7 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -72,7 +72,9 @@ int SCACPreparePatterns(MpmCtx *mpm_ctx);
 uint32_t SCACSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                     PrefilterRuleStore *pmq, const uint8_t *buf, uint32_t buflen);
 void SCACPrintInfo(MpmCtx *mpm_ctx);
-void SCACRegisterTests(void);
+#ifdef UNITTESTS
+static void SCACRegisterTests(void);
+#endif
 
 /* a placeholder to denote a failure transition in the goto table */
 #define SC_AC_FAIL (-1)
@@ -1139,8 +1141,9 @@ void MpmACRegister(void)
     mpm_table[MPM_AC].Prepare = SCACPreparePatterns;
     mpm_table[MPM_AC].Search = SCACSearch;
     mpm_table[MPM_AC].PrintCtx = SCACPrintInfo;
+#ifdef UNITTESTS
     mpm_table[MPM_AC].RegisterUnittests = SCACRegisterTests;
-
+#endif
     return;
 }
 
@@ -2121,12 +2124,8 @@ static int SCACTest29(void)
     return result;
 }
 
-#endif /* UNITTESTS */
-
 void SCACRegisterTests(void)
 {
-
-#ifdef UNITTESTS
     UtRegisterTest("SCACTest01", SCACTest01);
     UtRegisterTest("SCACTest02", SCACTest02);
     UtRegisterTest("SCACTest03", SCACTest03);
@@ -2156,7 +2155,5 @@ void SCACRegisterTests(void)
     UtRegisterTest("SCACTest27", SCACTest27);
     UtRegisterTest("SCACTest28", SCACTest28);
     UtRegisterTest("SCACTest29", SCACTest29);
-#endif
-
-    return;
 }
+#endif /* UNITTESTS */
diff --git a/src/util-mpm-hs.c b/src/util-mpm-hs.c
index a3b896abde93..f1594434ff8e 100644
--- a/src/util-mpm-hs.c
+++ b/src/util-mpm-hs.c
@@ -60,7 +60,9 @@ uint32_t SCHSSearch(const MpmCtx *mpm_ctx, MpmThreadCtx *mpm_thread_ctx,
                     PrefilterRuleStore *pmq, const uint8_t *buf, const uint32_t buflen);
 void SCHSPrintInfo(MpmCtx *mpm_ctx);
 void SCHSPrintSearchStats(MpmThreadCtx *mpm_thread_ctx);
-void SCHSRegisterTests(void);
+#ifdef UNITTESTS
+static void SCHSRegisterTests(void);
+#endif
 
 /* size of the hash table used to speed up pattern insertions initially */
 #define INIT_HASH_SIZE 65536
@@ -1049,8 +1051,9 @@ void MpmHSRegister(void)
     mpm_table[MPM_HS].Search = SCHSSearch;
     mpm_table[MPM_HS].PrintCtx = SCHSPrintInfo;
     mpm_table[MPM_HS].PrintThreadCtx = SCHSPrintSearchStats;
+#ifdef UNITTESTS
     mpm_table[MPM_HS].RegisterUnittests = SCHSRegisterTests;
-
+#endif
     /* Set Hyperscan memory allocators */
     SCHSSetAllocators();
 }
@@ -2132,11 +2135,8 @@ static int SCHSTest29(void)
     return result;
 }
 
-#endif /* UNITTESTS */
-
-void SCHSRegisterTests(void)
+static void SCHSRegisterTests(void)
 {
-#ifdef UNITTESTS
     UtRegisterTest("SCHSTest01", SCHSTest01);
     UtRegisterTest("SCHSTest02", SCHSTest02);
     UtRegisterTest("SCHSTest03", SCHSTest03);
@@ -2166,9 +2166,6 @@ void SCHSRegisterTests(void)
     UtRegisterTest("SCHSTest27", SCHSTest27);
     UtRegisterTest("SCHSTest28", SCHSTest28);
     UtRegisterTest("SCHSTest29", SCHSTest29);
-#endif
-
-    return;
 }
-
+#endif /* UNITTESTS */
 #endif /* BUILD_HYPERSCAN */
diff --git a/src/util-mpm.h b/src/util-mpm.h
index fe07bebceb79..96cbeadb2129 100644
--- a/src/util-mpm.h
+++ b/src/util-mpm.h
@@ -163,7 +163,9 @@ typedef struct MpmTableElmt_ {
     uint32_t (*Search)(const struct MpmCtx_ *, struct MpmThreadCtx_ *, PrefilterRuleStore *, const uint8_t *, uint32_t);
     void (*PrintCtx)(struct MpmCtx_ *);
     void (*PrintThreadCtx)(struct MpmThreadCtx_ *);
+#ifdef UNITTESTS
     void (*RegisterUnittests)(void);
+#endif
 } MpmTableElmt;
 
 extern MpmTableElmt mpm_table[MPM_TABLE_SIZE];

From 0172c01dc2e94f1df9ada3a4e34af93fce6120a9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 20 Dec 2023 14:38:01 +0100
Subject: [PATCH 328/462] spm/bm: minor code cleanups; constification

---
 src/util-spm-bm.c | 48 ++++++++++++-----------------------------------
 src/util-spm-bm.h |  6 ++++--
 2 files changed, 16 insertions(+), 38 deletions(-)

diff --git a/src/util-spm-bm.c b/src/util-spm-bm.c
index 449c6b62962e..27583ef89344 100644
--- a/src/util-spm-bm.c
+++ b/src/util-spm-bm.c
@@ -297,36 +297,23 @@ static void PreBmGsNocase(const uint8_t *x, uint16_t m, uint16_t *bmGs)
  *
  * \retval ptr to start of the match; NULL if no match
  */
-uint8_t *BoyerMoore(const uint8_t *x, uint16_t m, const uint8_t *y, uint32_t n, BmCtx *bm_ctx)
+uint8_t *BoyerMoore(
+        const uint8_t *x, const uint16_t m, const uint8_t *y, const uint32_t n, const BmCtx *bm_ctx)
 {
-    uint16_t *bmGs = bm_ctx->bmGs;
-    uint16_t *bmBc = bm_ctx->bmBc;
+    const uint16_t *bmGs = bm_ctx->bmGs;
+    const uint16_t *bmBc = bm_ctx->bmBc;
 
     int i, j, m1, m2;
-    int32_t int_n;
-#if 0
-    printf("\nBad:\n");
-    for (i=0;i<ALPHABET_SIZE;i++)
-        printf("%c,%d ", i, bmBc[i]);
-
-    printf("\ngood:\n");
-    for (i=0;i<m;i++)
-        printf("%c, %d ", x[i],bmBc[i]);
-    printf("\n");
-#endif
     // force casting to int32_t (if possible)
-    int_n = unlikely(n > INT32_MAX) ? INT32_MAX : n;
+    const int32_t int_n = unlikely(n > INT32_MAX) ? INT32_MAX : n;
     j = 0;
     while (j <= int_n - m ) {
         for (i = m - 1; i >= 0 && x[i] == y[i + j]; --i);
 
         if (i < 0) {
             return (uint8_t *)(y + j);
-            //j += bmGs[0];
         } else {
-//          printf("%c", y[i+j]);
-            j += (m1 = bmGs[i]) > (m2 = bmBc[y[i + j]] - m + 1 + i)? m1: m2;
-//          printf("%d, %d\n", m1, m2);
+            j += (m1 = bmGs[i]) > (m2 = bmBc[y[i + j]] - m + 1 + i) ? m1 : m2;
         }
     }
     return NULL;
@@ -348,24 +335,14 @@ uint8_t *BoyerMoore(const uint8_t *x, uint16_t m, const uint8_t *y, uint32_t n,
  *
  * \retval ptr to start of the match; NULL if no match
  */
-uint8_t *BoyerMooreNocase(const uint8_t *x, uint16_t m, const uint8_t *y, uint32_t n, BmCtx *bm_ctx)
+uint8_t *BoyerMooreNocase(
+        const uint8_t *x, const uint16_t m, const uint8_t *y, const uint32_t n, const BmCtx *bm_ctx)
 {
-    uint16_t *bmGs = bm_ctx->bmGs;
-    uint16_t *bmBc = bm_ctx->bmBc;
+    const uint16_t *bmGs = bm_ctx->bmGs;
+    const uint16_t *bmBc = bm_ctx->bmBc;
     int i, j, m1, m2;
-    int32_t int_n;
-#if 0
-    printf("\nBad:\n");
-    for (i=0;i<ALPHABET_SIZE;i++)
-        printf("%c,%d ", i, bmBc[i]);
-
-    printf("\ngood:\n");
-    for (i=0;i<m;i++)
-        printf("%c, %d ", x[i],bmBc[i]);
-    printf("\n");
-#endif
     // force casting to int32_t (if possible)
-    int_n = unlikely(n > INT32_MAX) ? INT32_MAX : n;
+    const int32_t int_n = unlikely(n > INT32_MAX) ? INT32_MAX : n;
     j = 0;
     while (j <= int_n - m ) {
          /* x is stored in lowercase. */
@@ -374,8 +351,7 @@ uint8_t *BoyerMooreNocase(const uint8_t *x, uint16_t m, const uint8_t *y, uint32
         if (i < 0) {
             return (uint8_t *)(y + j);
         } else {
-            j += (m1 = bmGs[i]) > (m2 = bmBc[y[i + j]] - m + 1 + i)?
-                m1: m2;
+            j += (m1 = bmGs[i]) > (m2 = bmBc[y[i + j]] - m + 1 + i) ? m1 : m2;
         }
     }
     return NULL;
diff --git a/src/util-spm-bm.h b/src/util-spm-bm.h
index 3c5e59ce03fb..6c0afbc3d746 100644
--- a/src/util-spm-bm.h
+++ b/src/util-spm-bm.h
@@ -41,8 +41,10 @@ BmCtx *BoyerMooreCtxInit(const uint8_t *needle, uint16_t needle_len);
 BmCtx *BoyerMooreNocaseCtxInit(uint8_t *needle, uint16_t needle_len);
 
 void BoyerMooreCtxToNocase(BmCtx *, uint8_t *, uint16_t);
-uint8_t *BoyerMoore(const uint8_t *x, uint16_t m, const uint8_t *y, uint32_t n, BmCtx *bm_ctx);
-uint8_t *BoyerMooreNocase(const uint8_t *x, uint16_t m, const uint8_t *y, uint32_t n, BmCtx *bm_ctx);
+uint8_t *BoyerMoore(const uint8_t *x, const uint16_t m, const uint8_t *y, const uint32_t n,
+        const BmCtx *bm_ctx);
+uint8_t *BoyerMooreNocase(const uint8_t *x, const uint16_t m, const uint8_t *y, const uint32_t n,
+        const BmCtx *bm_ctx);
 void BoyerMooreCtxDeInit(BmCtx *);
 
 void SpmBMRegister(void);

From 18eafb622f644eab6316b48b9cc2ff023037f369 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 6 Oct 2023 12:52:54 +0200
Subject: [PATCH 329/462] detect/content-inspect: add more tests

---
 src/tests/detect-engine-content-inspection.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/tests/detect-engine-content-inspection.c b/src/tests/detect-engine-content-inspection.c
index 4a72402d35aa..4430422a83da 100644
--- a/src/tests/detect-engine-content-inspection.c
+++ b/src/tests/detect-engine-content-inspection.c
@@ -289,6 +289,20 @@ static int DetectEngineContentInspectionTest13(void) {
     TEST_FOOTER;
 }
 
+static int DetectEngineContentInspectionTest14(void)
+{
+    TEST_HEADER;
+    TEST_RUN("XYZ_klm_1234abcd_XYZ_klm_5678abcd", 33,
+            "content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; distance:4; "
+            "byte_test:4,=,1234,-8,relative,string;",
+            true, 4);
+    TEST_RUN("XYZ_klm_1234abcd_XYZ_klm_5678abcd", 33,
+            "content:\"XYZ\"; content:\"_klm_\"; distance:0; content:\"abcd\"; distance:4; "
+            "byte_test:4,=,5678,-8,relative,string;",
+            true, 5);
+    TEST_FOOTER;
+}
+
 void DetectEngineContentInspectionRegisterTests(void)
 {
     UtRegisterTest("DetectEngineContentInspectionTest01",
@@ -317,6 +331,8 @@ void DetectEngineContentInspectionRegisterTests(void)
                    DetectEngineContentInspectionTest12);
     UtRegisterTest("DetectEngineContentInspectionTest13 mix startswith/endswith",
                    DetectEngineContentInspectionTest13);
+    UtRegisterTest("DetectEngineContentInspectionTest14 byte_test negative offset",
+            DetectEngineContentInspectionTest14);
 }
 
 #undef TEST_HEADER

From 2911656d6c11e8ef64a55ce64216382cd22151fe Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 20 Dec 2023 21:40:47 +0100
Subject: [PATCH 330/462] detect/content: fix offset for negative distance

Fix offset calculation on sigs with negative distance. Can lead to FN
in certain cases.

Bug: #6661.
---
 src/detect-content.c                         | 19 +++++++++++++++----
 src/tests/detect-engine-content-inspection.c | 13 +++++++++++++
 2 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/src/detect-content.c b/src/detect-content.c
index ebe28a1b04fa..944172702d92 100644
--- a/src/detect-content.c
+++ b/src/detect-content.c
@@ -571,10 +571,21 @@ static void PropagateLimits(Signature *s, SigMatch *sm_head)
                 SCLogDebug("stored: offset %u depth %u offset_plus_pat %u "
                            "has_active_depth_chain %s",
                         offset, depth, offset_plus_pat, has_active_depth_chain ? "true" : "false");
-                if (cd->flags & DETECT_CONTENT_DISTANCE && cd->distance >= 0) {
-                    VALIDATE((uint32_t)offset_plus_pat + cd->distance <= UINT16_MAX);
-                    offset = cd->offset = (uint16_t)(offset_plus_pat + cd->distance);
-                    SCLogDebug("updated content to have offset %u", cd->offset);
+                if (cd->flags & DETECT_CONTENT_DISTANCE) {
+                    if (cd->distance >= 0) {
+                        VALIDATE((uint32_t)offset_plus_pat + cd->distance <= UINT16_MAX);
+                        offset = cd->offset = (uint16_t)(offset_plus_pat + cd->distance);
+                        SCLogDebug("distance %d: updated content to have offset %u", cd->distance,
+                                cd->offset);
+                    } else {
+                        if (abs(cd->distance) > offset_plus_pat)
+                            offset = cd->offset = 0;
+                        else
+                            offset = cd->offset = (uint16_t)(offset_plus_pat + cd->distance);
+                        offset_plus_pat = offset + cd->content_len;
+                        SCLogDebug("distance %d: updated content to have offset %u", cd->distance,
+                                cd->offset);
+                    }
                 }
                 if (has_active_depth_chain) {
                     if (offset_plus_pat && cd->flags & DETECT_CONTENT_WITHIN && cd->within >= 0) {
diff --git a/src/tests/detect-engine-content-inspection.c b/src/tests/detect-engine-content-inspection.c
index 4430422a83da..780ae6d4c614 100644
--- a/src/tests/detect-engine-content-inspection.c
+++ b/src/tests/detect-engine-content-inspection.c
@@ -303,6 +303,17 @@ static int DetectEngineContentInspectionTest14(void)
     TEST_FOOTER;
 }
 
+/** \brief negative distance */
+static int DetectEngineContentInspectionTest17(void)
+{
+    TEST_HEADER;
+    TEST_RUN("aaabbbcccdddee", 14,
+            "content:\"aaa\"; content:\"ee\"; within:2; distance:9; content:\"bbb\"; within:3; "
+            "distance:-11; content:\"ccc\"; within:3; distance:0;",
+            true, 4);
+    TEST_FOOTER;
+}
+
 void DetectEngineContentInspectionRegisterTests(void)
 {
     UtRegisterTest("DetectEngineContentInspectionTest01",
@@ -333,6 +344,8 @@ void DetectEngineContentInspectionRegisterTests(void)
                    DetectEngineContentInspectionTest13);
     UtRegisterTest("DetectEngineContentInspectionTest14 byte_test negative offset",
             DetectEngineContentInspectionTest14);
+    UtRegisterTest("DetectEngineContentInspectionTest17 negative distance",
+            DetectEngineContentInspectionTest17);
 }
 
 #undef TEST_HEADER

From 2b3ec34de80816b06c26495d331a863ae3567b96 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Thu, 21 Dec 2023 11:35:22 +0100
Subject: [PATCH 331/462] detect: use do { } while loop for app engine loop

---
 src/detect.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index d671a3866fa5..494886c22e41 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -1097,7 +1097,7 @@ static bool DetectRunTxInspectRule(ThreadVars *tv,
     }
 
     const DetectEngineAppInspectionEngine *engine = s->app_inspect;
-    while (engine != NULL) { // TODO could be do {} while as s->app_inspect cannot be null
+    do {
         TRACE_SID_TXS(s->id, tx, "engine %p inspect_flags %x", engine, inspect_flags);
         if (!(inspect_flags & BIT_U32(engine->id)) &&
                 direction == engine->dir)
@@ -1178,7 +1178,7 @@ static bool DetectRunTxInspectRule(ThreadVars *tv,
             break;
         }
         engine = engine->next;
-    }
+    } while (engine != NULL);
     TRACE_SID_TXS(s->id, tx, "inspect_flags %x, total_matches %u, engine %p",
             inspect_flags, total_matches, engine);
 

From 4f0f7b196976e7f3291085f541f0906062f4aec4 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Dec 2023 08:31:22 +0100
Subject: [PATCH 332/462] detect/dsize: minor code cleanup

---
 src/detect-dsize.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect-dsize.c b/src/detect-dsize.c
index bf095cd4fda5..93bd08a9074d 100644
--- a/src/detect-dsize.c
+++ b/src/detect-dsize.c
@@ -243,7 +243,7 @@ int SigParseGetMaxDsize(const Signature *s)
 void SigParseSetDsizePair(Signature *s)
 {
     if (s->flags & SIG_FLAG_DSIZE && s->init_data->dsize_sm != NULL) {
-        DetectU16Data *dd = (DetectU16Data *)s->init_data->dsize_sm->ctx;
+        const DetectU16Data *dd = (const DetectU16Data *)s->init_data->dsize_sm->ctx;
 
         uint16_t low = 0;
         uint16_t high = 65535;

From e06d2c402ae86eb74e1f651ba5d2930112fa7f4c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Dec 2023 08:43:46 +0100
Subject: [PATCH 333/462] detect/content: limits prop comment cleanup

---
 src/detect-content.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/detect-content.c b/src/detect-content.c
index 944172702d92..2f9884000523 100644
--- a/src/detect-content.c
+++ b/src/detect-content.c
@@ -383,9 +383,9 @@ void DetectContentFree(DetectEngineCtx *de_ctx, void *ptr)
     SCReturn;
 }
 
-/*
+/**
  *  \brief Determine the size needed to accommodate the content
- *  elements of a signature
+ *         elements of a signature
  *  \param s signature to get dsize value from
  *  \param max_size Maximum buffer/data size allowed.
  *  \param list signature match list.

From 88cc999184f60987791346e69c3d7cd3ac238862 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 22 Dec 2023 10:51:24 +0100
Subject: [PATCH 334/462] detect/bsize: constify keyword args during size check

---
 src/detect-bsize.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/detect-bsize.c b/src/detect-bsize.c
index f69e20851839..e95c803b5f3d 100644
--- a/src/detect-bsize.c
+++ b/src/detect-bsize.c
@@ -40,7 +40,7 @@
 /*prototypes*/
 static int DetectBsizeSetup (DetectEngineCtx *, Signature *, const char *);
 static void DetectBsizeFree (DetectEngineCtx *, void *);
-static int SigParseGetMaxBsize(DetectU64Data *bsz);
+static int SigParseGetMaxBsize(const DetectU64Data *bsz);
 #ifdef UNITTESTS
 static void DetectBsizeRegisterTests (void);
 #endif
@@ -48,10 +48,10 @@ static void DetectBsizeRegisterTests (void);
 bool DetectBsizeValidateContentCallback(Signature *s, const SignatureInitDataBuffer *b)
 {
     int bsize = -1;
-    DetectU64Data *bsz;
+    const DetectU64Data *bsz;
     for (const SigMatch *sm = b->head; sm != NULL; sm = sm->next) {
         if (sm->type == DETECT_BSIZE) {
-            bsz = (DetectU64Data *)sm->ctx;
+            bsz = (const DetectU64Data *)sm->ctx;
             bsize = SigParseGetMaxBsize(bsz);
             break;
         }
@@ -171,7 +171,7 @@ static DetectU64Data *DetectBsizeParse(const char *str)
     return DetectU64Parse(str);
 }
 
-static int SigParseGetMaxBsize(DetectU64Data *bsz)
+static int SigParseGetMaxBsize(const DetectU64Data *bsz)
 {
     switch (bsz->mode) {
         case DETECT_UINT_LT:

From fd4ca53eb7110a2ab0f24d08902d5c2a04f00a8c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 27 Dec 2023 17:01:18 +0100
Subject: [PATCH 335/462] app-layer: micro optimization for AppProtoEquals

Add most common condition first.
---
 src/app-layer-protos.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/src/app-layer-protos.h b/src/app-layer-protos.h
index dd372550cbf5..e2efbec4d4b1 100644
--- a/src/app-layer-protos.h
+++ b/src/app-layer-protos.h
@@ -87,14 +87,16 @@ static inline bool AppProtoIsValid(AppProto a)
 // whether a signature AppProto matches a flow (or signature) AppProto
 static inline bool AppProtoEquals(AppProto sigproto, AppProto alproto)
 {
+    if (sigproto == alproto) {
+        return true;
+    }
     switch (sigproto) {
         case ALPROTO_HTTP:
-            return (alproto == ALPROTO_HTTP1) || (alproto == ALPROTO_HTTP2) ||
-                   (alproto == ALPROTO_HTTP);
+            return (alproto == ALPROTO_HTTP1) || (alproto == ALPROTO_HTTP2);
         case ALPROTO_DCERPC:
-            return (alproto == ALPROTO_DCERPC || alproto == ALPROTO_SMB);
+            return (alproto == ALPROTO_SMB);
     }
-    return (sigproto == alproto);
+    return false;
 }
 
 /**

From e4550bee0a85fc145c18df92f740789c9bd297d2 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 27 Dec 2023 17:01:42 +0100
Subject: [PATCH 336/462] detect: minor cleanup for rule group get function

---
 src/detect.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index 494886c22e41..d602a19be90f 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -202,8 +202,6 @@ const SigGroupHead *SigMatchSignaturesGetSgh(const DetectEngineCtx *de_ctx,
         const Packet *p)
 {
     SCEnter();
-
-    int f;
     SigGroupHead *sgh = NULL;
 
     /* if the packet proto is 0 (not set), we're inspecting it against
@@ -218,33 +216,30 @@ const SigGroupHead *SigMatchSignaturesGetSgh(const DetectEngineCtx *de_ctx,
     }
 
     /* select the flow_gh */
-    if (p->flowflags & FLOW_PKT_TOCLIENT)
-        f = 0;
-    else
-        f = 1;
+    const int dir = (p->flowflags & FLOW_PKT_TOCLIENT) == 0;
 
     int proto = IP_GET_IPPROTO(p);
     if (proto == IPPROTO_TCP) {
-        DetectPort *list = de_ctx->flow_gh[f].tcp;
-        SCLogDebug("tcp toserver %p, tcp toclient %p: going to use %p",
-                de_ctx->flow_gh[1].tcp, de_ctx->flow_gh[0].tcp, de_ctx->flow_gh[f].tcp);
-        uint16_t port = f ? p->dp : p->sp;
+        DetectPort *list = de_ctx->flow_gh[dir].tcp;
+        SCLogDebug("tcp toserver %p, tcp toclient %p: going to use %p", de_ctx->flow_gh[1].tcp,
+                de_ctx->flow_gh[0].tcp, de_ctx->flow_gh[dir].tcp);
+        const uint16_t port = dir ? p->dp : p->sp;
         SCLogDebug("tcp port %u -> %u:%u", port, p->sp, p->dp);
         DetectPort *sghport = DetectPortLookupGroup(list, port);
         if (sghport != NULL)
             sgh = sghport->sh;
-        SCLogDebug("TCP list %p, port %u, direction %s, sghport %p, sgh %p",
-                list, port, f ? "toserver" : "toclient", sghport, sgh);
+        SCLogDebug("TCP list %p, port %u, direction %s, sghport %p, sgh %p", list, port,
+                dir ? "toserver" : "toclient", sghport, sgh);
     } else if (proto == IPPROTO_UDP) {
-        DetectPort *list = de_ctx->flow_gh[f].udp;
-        uint16_t port = f ? p->dp : p->sp;
+        DetectPort *list = de_ctx->flow_gh[dir].udp;
+        uint16_t port = dir ? p->dp : p->sp;
         DetectPort *sghport = DetectPortLookupGroup(list, port);
         if (sghport != NULL)
             sgh = sghport->sh;
-        SCLogDebug("UDP list %p, port %u, direction %s, sghport %p, sgh %p",
-                list, port, f ? "toserver" : "toclient", sghport, sgh);
+        SCLogDebug("UDP list %p, port %u, direction %s, sghport %p, sgh %p", list, port,
+                dir ? "toserver" : "toclient", sghport, sgh);
     } else {
-        sgh = de_ctx->flow_gh[f].sgh[proto];
+        sgh = de_ctx->flow_gh[dir].sgh[proto];
     }
 
     SCReturnPtr(sgh, "SigGroupHead");

From 91f153fb1dbecbd53bf465e353ce794ffa9a710a Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 27 Dec 2023 17:12:33 +0100
Subject: [PATCH 337/462] detect: constify flow flags in tx rule inspect

---
 src/detect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect.c b/src/detect.c
index d602a19be90f..7aff83240973 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -1062,7 +1062,7 @@ static bool DetectRunTxInspectRule(ThreadVars *tv,
         RuleMatchCandidateTx *can,
         DetectRunScratchpad *scratch)
 {
-    uint8_t flow_flags = in_flow_flags;
+    const uint8_t flow_flags = in_flow_flags;
     const int direction = (flow_flags & STREAM_TOSERVER) ? 0 : 1;
     uint32_t inspect_flags = stored_flags ? *stored_flags : 0;
     int total_matches = 0;

From db2484276eb46b32083bfed79a56da609030dff3 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 27 Dec 2023 19:08:29 +0100
Subject: [PATCH 338/462] detect: shrink sgh to have all runtime members on one
 cache line

---
 src/detect.h | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/src/detect.h b/src/detect.h
index 5d55720758ea..c640ff3b6923 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1314,14 +1314,14 @@ enum {
     DETECT_EVENT_TOO_MANY_BUFFERS,
 };
 
-#define SIG_GROUP_HEAD_HAVERAWSTREAM    BIT_U32(0)
+#define SIG_GROUP_HEAD_HAVERAWSTREAM BIT_U16(0)
 #ifdef HAVE_MAGIC
-#define SIG_GROUP_HEAD_HAVEFILEMAGIC    BIT_U32(20)
+#define SIG_GROUP_HEAD_HAVEFILEMAGIC BIT_U16(1)
 #endif
-#define SIG_GROUP_HEAD_HAVEFILEMD5      BIT_U32(21)
-#define SIG_GROUP_HEAD_HAVEFILESIZE     BIT_U32(22)
-#define SIG_GROUP_HEAD_HAVEFILESHA1     BIT_U32(23)
-#define SIG_GROUP_HEAD_HAVEFILESHA256   BIT_U32(24)
+#define SIG_GROUP_HEAD_HAVEFILEMD5    BIT_U16(2)
+#define SIG_GROUP_HEAD_HAVEFILESIZE   BIT_U16(3)
+#define SIG_GROUP_HEAD_HAVEFILESHA1   BIT_U16(4)
+#define SIG_GROUP_HEAD_HAVEFILESHA256 BIT_U16(5)
 
 enum MpmBuiltinBuffers {
     MPMB_TCP_PKT_TS,
@@ -1443,9 +1443,15 @@ typedef struct SigGroupHeadInitData_ {
 
 /** \brief Container for matching data for a signature group */
 typedef struct SigGroupHead_ {
-    uint32_t flags;
+    uint16_t flags;
     /* coccinelle: SigGroupHead:flags:SIG_GROUP_HEAD_ */
 
+    /** the number of signatures in this sgh that have the filestore keyword
+     *  set. */
+    uint16_t filestore_cnt;
+
+    uint32_t id; /**< unique id used to index sgh_array for stats */
+
     /* non prefilter list excluding SYN rules */
     uint32_t non_pf_other_store_cnt;
     uint32_t non_pf_syn_store_cnt;
@@ -1453,12 +1459,6 @@ typedef struct SigGroupHead_ {
     /* non mpm list including SYN rules */
     SignatureNonPrefilterStore *non_pf_syn_store_array; // size is non_mpm_syn_store_cnt * sizeof(SignatureNonPrefilterStore)
 
-    /** the number of signatures in this sgh that have the filestore keyword
-     *  set. */
-    uint16_t filestore_cnt;
-
-    uint32_t id; /**< unique id used to index sgh_array for stats */
-
     PrefilterEngine *pkt_engines;
     PrefilterEngine *payload_engines;
     PrefilterEngine *tx_engines;

From 11bf60aa3a8476323403ec06c8259c4e7e182855 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 29 Dec 2023 09:22:14 +0100
Subject: [PATCH 339/462] detect/mpm: minor cleanup

---
 src/detect-engine-mpm.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/detect-engine-mpm.c b/src/detect-engine-mpm.c
index 84f9a6c8feab..02e63c5f896b 100644
--- a/src/detect-engine-mpm.c
+++ b/src/detect-engine-mpm.c
@@ -944,10 +944,8 @@ uint32_t PatternStrength(uint8_t *pat, uint16_t patlen)
     return s;
 }
 
-static void PopulateMpmHelperAddPattern(MpmCtx *mpm_ctx,
-                                        const DetectContentData *cd,
-                                        const Signature *s, uint8_t flags,
-                                        int chop)
+static void PopulateMpmHelperAddPattern(MpmCtx *mpm_ctx, const DetectContentData *cd,
+        const Signature *s, const uint8_t flags, const int chop)
 {
     uint16_t pat_offset = cd->offset;
     uint16_t pat_depth = cd->depth;

From aad403d87dc85266b9537eb744c93d90919b63f2 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 29 Dec 2023 10:59:43 +0100
Subject: [PATCH 340/462] mpm/hs: improve pointer hygene

---
 src/util-mpm-hs.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/util-mpm-hs.c b/src/util-mpm-hs.c
index f1594434ff8e..5c241991842e 100644
--- a/src/util-mpm-hs.c
+++ b/src/util-mpm-hs.c
@@ -873,6 +873,7 @@ void SCHSDestroyCtx(MpmCtx *mpm_ctx)
     SCMutexUnlock(&g_db_table_mutex);
 
     SCFree(mpm_ctx->ctx);
+    mpm_ctx->ctx = NULL;
     mpm_ctx->memory_cnt--;
     mpm_ctx->memory_size -= sizeof(SCHSCtx);
 }

From 96aee6434f1b36669c5e781891cc0a385a601c84 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 29 Dec 2023 18:27:17 +0100
Subject: [PATCH 341/462] mpm/ac: pointer hygene

---
 src/util-mpm-ac.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/util-mpm-ac.c b/src/util-mpm-ac.c
index 94de6000b2f7..05d777397c45 100644
--- a/src/util-mpm-ac.c
+++ b/src/util-mpm-ac.c
@@ -916,6 +916,7 @@ void SCACDestroyCtx(MpmCtx *mpm_ctx)
     }
 
     SCFree(mpm_ctx->ctx);
+    mpm_ctx->ctx = NULL;
     mpm_ctx->memory_cnt--;
     mpm_ctx->memory_size -= sizeof(SCACCtx);
 

From 7b2d6b68946c7cceba8994d2874147afff76a309 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 31 Dec 2023 10:17:38 +0100
Subject: [PATCH 342/462] detect/address: minor cleanup

---
 src/detect-parse.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index c259fd36f943..a656d570cd3f 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -1797,8 +1797,8 @@ static void SigBuildAddressMatchArray(Signature *s)
     /* source addresses */
     uint16_t cnt = 0;
     uint16_t idx = 0;
-    DetectAddress *da = s->init_data->src->ipv4_head;
-    for ( ; da != NULL; da = da->next) {
+
+    for (const DetectAddress *da = s->init_data->src->ipv4_head; da != NULL; da = da->next) {
         cnt++;
     }
     if (cnt > 0) {
@@ -1807,7 +1807,7 @@ static void SigBuildAddressMatchArray(Signature *s)
             exit(EXIT_FAILURE);
         }
 
-        for (da = s->init_data->src->ipv4_head; da != NULL; da = da->next) {
+        for (const DetectAddress *da = s->init_data->src->ipv4_head; da != NULL; da = da->next) {
             s->addr_src_match4[idx].ip = SCNtohl(da->ip.addr_data32[0]);
             s->addr_src_match4[idx].ip2 = SCNtohl(da->ip2.addr_data32[0]);
             idx++;
@@ -1818,8 +1818,7 @@ static void SigBuildAddressMatchArray(Signature *s)
     /* destination addresses */
     cnt = 0;
     idx = 0;
-    da = s->init_data->dst->ipv4_head;
-    for ( ; da != NULL; da = da->next) {
+    for (const DetectAddress *da = s->init_data->dst->ipv4_head; da != NULL; da = da->next) {
         cnt++;
     }
     if (cnt > 0) {
@@ -1828,7 +1827,7 @@ static void SigBuildAddressMatchArray(Signature *s)
             exit(EXIT_FAILURE);
         }
 
-        for (da = s->init_data->dst->ipv4_head; da != NULL; da = da->next) {
+        for (const DetectAddress *da = s->init_data->dst->ipv4_head; da != NULL; da = da->next) {
             s->addr_dst_match4[idx].ip = SCNtohl(da->ip.addr_data32[0]);
             s->addr_dst_match4[idx].ip2 = SCNtohl(da->ip2.addr_data32[0]);
             idx++;
@@ -1839,8 +1838,7 @@ static void SigBuildAddressMatchArray(Signature *s)
     /* source addresses IPv6 */
     cnt = 0;
     idx = 0;
-    da = s->init_data->src->ipv6_head;
-    for ( ; da != NULL; da = da->next) {
+    for (const DetectAddress *da = s->init_data->src->ipv6_head; da != NULL; da = da->next) {
         cnt++;
     }
     if (cnt > 0) {
@@ -1849,7 +1847,7 @@ static void SigBuildAddressMatchArray(Signature *s)
             exit(EXIT_FAILURE);
         }
 
-        for (da = s->init_data->src->ipv6_head; da != NULL; da = da->next) {
+        for (const DetectAddress *da = s->init_data->src->ipv6_head; da != NULL; da = da->next) {
             s->addr_src_match6[idx].ip[0] = SCNtohl(da->ip.addr_data32[0]);
             s->addr_src_match6[idx].ip[1] = SCNtohl(da->ip.addr_data32[1]);
             s->addr_src_match6[idx].ip[2] = SCNtohl(da->ip.addr_data32[2]);
@@ -1866,8 +1864,7 @@ static void SigBuildAddressMatchArray(Signature *s)
     /* destination addresses IPv6 */
     cnt = 0;
     idx = 0;
-    da = s->init_data->dst->ipv6_head;
-    for ( ; da != NULL; da = da->next) {
+    for (const DetectAddress *da = s->init_data->dst->ipv6_head; da != NULL; da = da->next) {
         cnt++;
     }
     if (cnt > 0) {
@@ -1876,7 +1873,7 @@ static void SigBuildAddressMatchArray(Signature *s)
             exit(EXIT_FAILURE);
         }
 
-        for (da = s->init_data->dst->ipv6_head; da != NULL; da = da->next) {
+        for (const DetectAddress *da = s->init_data->dst->ipv6_head; da != NULL; da = da->next) {
             s->addr_dst_match6[idx].ip[0] = SCNtohl(da->ip.addr_data32[0]);
             s->addr_dst_match6[idx].ip[1] = SCNtohl(da->ip.addr_data32[1]);
             s->addr_dst_match6[idx].ip[2] = SCNtohl(da->ip.addr_data32[2]);

From 5c6089f93f4342fcef73ff1665c74376d6ed9952 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 31 Dec 2023 10:22:13 +0100
Subject: [PATCH 343/462] detect/address: refactor match array building

---
 src/detect-parse.c | 147 +++++++++++++++++++--------------------------
 1 file changed, 63 insertions(+), 84 deletions(-)

diff --git a/src/detect-parse.c b/src/detect-parse.c
index a656d570cd3f..31df3d0aaed3 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -1786,106 +1786,85 @@ int DetectSignatureSetAppProto(Signature *s, AppProto alproto)
     return 0;
 }
 
-/**
- *  \internal
- *  \brief build address match array for cache efficient matching
- *
- *  \param s the signature
- */
-static void SigBuildAddressMatchArray(Signature *s)
+static DetectMatchAddressIPv4 *SigBuildAddressMatchArrayIPv4(
+        const DetectAddress *head, uint16_t *match4_cnt)
 {
-    /* source addresses */
     uint16_t cnt = 0;
-    uint16_t idx = 0;
 
-    for (const DetectAddress *da = s->init_data->src->ipv4_head; da != NULL; da = da->next) {
+    for (const DetectAddress *da = head; da != NULL; da = da->next) {
         cnt++;
     }
-    if (cnt > 0) {
-        s->addr_src_match4 = SCMalloc(cnt * sizeof(DetectMatchAddressIPv4));
-        if (s->addr_src_match4 == NULL) {
-            exit(EXIT_FAILURE);
-        }
-
-        for (const DetectAddress *da = s->init_data->src->ipv4_head; da != NULL; da = da->next) {
-            s->addr_src_match4[idx].ip = SCNtohl(da->ip.addr_data32[0]);
-            s->addr_src_match4[idx].ip2 = SCNtohl(da->ip2.addr_data32[0]);
-            idx++;
-        }
-        s->addr_src_match4_cnt = cnt;
+    if (cnt == 0) {
+        return NULL;
     }
-
-    /* destination addresses */
-    cnt = 0;
-    idx = 0;
-    for (const DetectAddress *da = s->init_data->dst->ipv4_head; da != NULL; da = da->next) {
-        cnt++;
+    DetectMatchAddressIPv4 *addr_match4 = SCCalloc(cnt, sizeof(DetectMatchAddressIPv4));
+    if (addr_match4 == NULL) {
+        exit(EXIT_FAILURE);
     }
-    if (cnt > 0) {
-        s->addr_dst_match4 = SCMalloc(cnt * sizeof(DetectMatchAddressIPv4));
-        if (s->addr_dst_match4 == NULL) {
-            exit(EXIT_FAILURE);
-        }
 
-        for (const DetectAddress *da = s->init_data->dst->ipv4_head; da != NULL; da = da->next) {
-            s->addr_dst_match4[idx].ip = SCNtohl(da->ip.addr_data32[0]);
-            s->addr_dst_match4[idx].ip2 = SCNtohl(da->ip2.addr_data32[0]);
-            idx++;
-        }
-        s->addr_dst_match4_cnt = cnt;
+    uint16_t idx = 0;
+    for (const DetectAddress *da = head; da != NULL; da = da->next) {
+        addr_match4[idx].ip = SCNtohl(da->ip.addr_data32[0]);
+        addr_match4[idx].ip2 = SCNtohl(da->ip2.addr_data32[0]);
+        idx++;
     }
+    *match4_cnt = cnt;
+    return addr_match4;
+}
 
-    /* source addresses IPv6 */
-    cnt = 0;
-    idx = 0;
-    for (const DetectAddress *da = s->init_data->src->ipv6_head; da != NULL; da = da->next) {
+static DetectMatchAddressIPv6 *SigBuildAddressMatchArrayIPv6(
+        const DetectAddress *head, uint16_t *match6_cnt)
+{
+    uint16_t cnt = 0;
+    for (const DetectAddress *da = head; da != NULL; da = da->next) {
         cnt++;
     }
-    if (cnt > 0) {
-        s->addr_src_match6 = SCMalloc(cnt * sizeof(DetectMatchAddressIPv6));
-        if (s->addr_src_match6 == NULL) {
-            exit(EXIT_FAILURE);
-        }
-
-        for (const DetectAddress *da = s->init_data->src->ipv6_head; da != NULL; da = da->next) {
-            s->addr_src_match6[idx].ip[0] = SCNtohl(da->ip.addr_data32[0]);
-            s->addr_src_match6[idx].ip[1] = SCNtohl(da->ip.addr_data32[1]);
-            s->addr_src_match6[idx].ip[2] = SCNtohl(da->ip.addr_data32[2]);
-            s->addr_src_match6[idx].ip[3] = SCNtohl(da->ip.addr_data32[3]);
-            s->addr_src_match6[idx].ip2[0] = SCNtohl(da->ip2.addr_data32[0]);
-            s->addr_src_match6[idx].ip2[1] = SCNtohl(da->ip2.addr_data32[1]);
-            s->addr_src_match6[idx].ip2[2] = SCNtohl(da->ip2.addr_data32[2]);
-            s->addr_src_match6[idx].ip2[3] = SCNtohl(da->ip2.addr_data32[3]);
-            idx++;
-        }
-        s->addr_src_match6_cnt = cnt;
+    if (cnt == 0) {
+        return NULL;
     }
 
-    /* destination addresses IPv6 */
-    cnt = 0;
-    idx = 0;
-    for (const DetectAddress *da = s->init_data->dst->ipv6_head; da != NULL; da = da->next) {
-        cnt++;
+    DetectMatchAddressIPv6 *addr_match6 = SCCalloc(cnt, sizeof(DetectMatchAddressIPv6));
+    if (addr_match6 == NULL) {
+        exit(EXIT_FAILURE);
     }
-    if (cnt > 0) {
-        s->addr_dst_match6 = SCMalloc(cnt * sizeof(DetectMatchAddressIPv6));
-        if (s->addr_dst_match6 == NULL) {
-            exit(EXIT_FAILURE);
-        }
 
-        for (const DetectAddress *da = s->init_data->dst->ipv6_head; da != NULL; da = da->next) {
-            s->addr_dst_match6[idx].ip[0] = SCNtohl(da->ip.addr_data32[0]);
-            s->addr_dst_match6[idx].ip[1] = SCNtohl(da->ip.addr_data32[1]);
-            s->addr_dst_match6[idx].ip[2] = SCNtohl(da->ip.addr_data32[2]);
-            s->addr_dst_match6[idx].ip[3] = SCNtohl(da->ip.addr_data32[3]);
-            s->addr_dst_match6[idx].ip2[0] = SCNtohl(da->ip2.addr_data32[0]);
-            s->addr_dst_match6[idx].ip2[1] = SCNtohl(da->ip2.addr_data32[1]);
-            s->addr_dst_match6[idx].ip2[2] = SCNtohl(da->ip2.addr_data32[2]);
-            s->addr_dst_match6[idx].ip2[3] = SCNtohl(da->ip2.addr_data32[3]);
-            idx++;
-        }
-        s->addr_dst_match6_cnt = cnt;
-    }
+    uint16_t idx = 0;
+    for (const DetectAddress *da = head; da != NULL; da = da->next) {
+        addr_match6[idx].ip[0] = SCNtohl(da->ip.addr_data32[0]);
+        addr_match6[idx].ip[1] = SCNtohl(da->ip.addr_data32[1]);
+        addr_match6[idx].ip[2] = SCNtohl(da->ip.addr_data32[2]);
+        addr_match6[idx].ip[3] = SCNtohl(da->ip.addr_data32[3]);
+        addr_match6[idx].ip2[0] = SCNtohl(da->ip2.addr_data32[0]);
+        addr_match6[idx].ip2[1] = SCNtohl(da->ip2.addr_data32[1]);
+        addr_match6[idx].ip2[2] = SCNtohl(da->ip2.addr_data32[2]);
+        addr_match6[idx].ip2[3] = SCNtohl(da->ip2.addr_data32[3]);
+        idx++;
+    }
+    *match6_cnt = cnt;
+    return addr_match6;
+}
+
+/**
+ *  \internal
+ *  \brief build address match array for cache efficient matching
+ *
+ *  \param s the signature
+ */
+static void SigBuildAddressMatchArray(Signature *s)
+{
+    /* source addresses */
+    s->addr_src_match4 =
+            SigBuildAddressMatchArrayIPv4(s->init_data->src->ipv4_head, &s->addr_src_match4_cnt);
+    /* destination addresses */
+    s->addr_dst_match4 =
+            SigBuildAddressMatchArrayIPv4(s->init_data->dst->ipv4_head, &s->addr_dst_match4_cnt);
+
+    /* source addresses IPv6 */
+    s->addr_src_match6 =
+            SigBuildAddressMatchArrayIPv6(s->init_data->src->ipv6_head, &s->addr_src_match6_cnt);
+    /* destination addresses IPv6 */
+    s->addr_dst_match6 =
+            SigBuildAddressMatchArrayIPv6(s->init_data->dst->ipv6_head, &s->addr_dst_match6_cnt);
 }
 
 static int SigMatchListLen(SigMatch *sm)

From 72841be050714d90b5bc34de7ac91917a9f571b5 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 3 Jan 2024 10:44:09 +0100
Subject: [PATCH 344/462] detect/rule-header: minor code cleanups

---
 src/detect.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index 7aff83240973..73ff82a8277b 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -587,10 +587,9 @@ static inline int DetectRunInspectRuleHeader(
     if ((p->flags & PKT_HAS_FLOW) && (sflags & SIG_FLAG_REQUIRE_FLOWVAR)) {
         DEBUG_VALIDATE_BUG_ON(f == NULL);
 
-        int m  = f->flowvar ? 1 : 0;
-
         /* no flowvars? skip this sig */
-        if (m == 0) {
+        const bool fv = f->flowvar != NULL;
+        if (fv == false) {
             SCLogDebug("skipping sig as the flow has no flowvars and sig "
                     "has SIG_FLAG_REQUIRE_FLOWVAR flag set.");
             return 0;
@@ -616,7 +615,7 @@ static inline int DetectRunInspectRuleHeader(
         if (!(sflags & SIG_FLAG_DP_ANY)) {
             if (p->flags & PKT_IS_FRAGMENT)
                 return 0;
-            DetectPort *dport = DetectPortLookupGroup(s->dp,p->dp);
+            const DetectPort *dport = DetectPortLookupGroup(s->dp, p->dp);
             if (dport == NULL) {
                 SCLogDebug("dport didn't match.");
                 return 0;
@@ -625,7 +624,7 @@ static inline int DetectRunInspectRuleHeader(
         if (!(sflags & SIG_FLAG_SP_ANY)) {
             if (p->flags & PKT_IS_FRAGMENT)
                 return 0;
-            DetectPort *sport = DetectPortLookupGroup(s->sp,p->sp);
+            const DetectPort *sport = DetectPortLookupGroup(s->sp, p->sp);
             if (sport == NULL) {
                 SCLogDebug("sport didn't match.");
                 return 0;

From 44a8bf463eb1f5585ec317c5aa5f535c2f9a6e0a Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 3 Jan 2024 10:50:04 +0100
Subject: [PATCH 345/462] detect/rule-header: use bool type

Update frame prototype as well, to match already returned true/false values.
---
 src/detect-engine-frame.c |  2 +-
 src/detect-engine-frame.h |  2 +-
 src/detect.c              | 50 +++++++++++++++++++--------------------
 3 files changed, 26 insertions(+), 28 deletions(-)

diff --git a/src/detect-engine-frame.c b/src/detect-engine-frame.c
index 71ac9d3d1674..fd3163d59732 100644
--- a/src/detect-engine-frame.c
+++ b/src/detect-engine-frame.c
@@ -224,7 +224,7 @@ int PrefilterGenericMpmFrameRegister(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
     return r;
 }
 
-int DetectRunFrameInspectRule(ThreadVars *tv, DetectEngineThreadCtx *det_ctx, const Signature *s,
+bool DetectRunFrameInspectRule(ThreadVars *tv, DetectEngineThreadCtx *det_ctx, const Signature *s,
         Flow *f, Packet *p, const Frames *frames, const Frame *frame)
 {
     BUG_ON(s->frame_inspect == NULL);
diff --git a/src/detect-engine-frame.h b/src/detect-engine-frame.h
index 8ec927f9432f..a529e55c4d00 100644
--- a/src/detect-engine-frame.h
+++ b/src/detect-engine-frame.h
@@ -26,7 +26,7 @@
 
 void DetectRunPrefilterFrame(DetectEngineThreadCtx *det_ctx, const SigGroupHead *sgh, Packet *p,
         const Frames *frames, const Frame *frame, const AppProto alproto);
-int DetectRunFrameInspectRule(ThreadVars *tv, DetectEngineThreadCtx *det_ctx, const Signature *s,
+bool DetectRunFrameInspectRule(ThreadVars *tv, DetectEngineThreadCtx *det_ctx, const Signature *s,
         Flow *f, Packet *p, const Frames *frames, const Frame *frame);
 
 int PrefilterGenericMpmFrameRegister(DetectEngineCtx *de_ctx, SigGroupHead *sgh, MpmCtx *mpm_ctx,
diff --git a/src/detect.c b/src/detect.c
index 73ff82a8277b..dca6fe9f651d 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -573,13 +573,11 @@ static void DetectRunInspectIPOnly(ThreadVars *tv, const DetectEngineCtx *de_ctx
     }
 }
 
-/* returns 0 if no match, 1 if match */
-static inline int DetectRunInspectRuleHeader(
-    const Packet *p,
-    const Flow *f,
-    const Signature *s,
-    const uint32_t sflags,
-    const uint8_t s_proto_flags)
+/** \internal
+ *  \brief inspect the rule header: protocol, ports, etc
+ *  \retval bool false if no match, true if match */
+static inline bool DetectRunInspectRuleHeader(const Packet *p, const Flow *f, const Signature *s,
+        const uint32_t sflags, const uint8_t s_proto_flags)
 {
     /* check if this signature has a requirement for flowvars of some type
      * and if so, if we actually have any in the flow. If not, the sig
@@ -592,71 +590,71 @@ static inline int DetectRunInspectRuleHeader(
         if (fv == false) {
             SCLogDebug("skipping sig as the flow has no flowvars and sig "
                     "has SIG_FLAG_REQUIRE_FLOWVAR flag set.");
-            return 0;
+            return false;
         }
     }
 
     if ((s_proto_flags & DETECT_PROTO_IPV4) && !PKT_IS_IPV4(p)) {
         SCLogDebug("ip version didn't match");
-        return 0;
+        return false;
     }
     if ((s_proto_flags & DETECT_PROTO_IPV6) && !PKT_IS_IPV6(p)) {
         SCLogDebug("ip version didn't match");
-        return 0;
+        return false;
     }
 
     if (DetectProtoContainsProto(&s->proto, IP_GET_IPPROTO(p)) == 0) {
         SCLogDebug("proto didn't match");
-        return 0;
+        return false;
     }
 
     /* check the source & dst port in the sig */
     if (p->proto == IPPROTO_TCP || p->proto == IPPROTO_UDP || p->proto == IPPROTO_SCTP) {
         if (!(sflags & SIG_FLAG_DP_ANY)) {
             if (p->flags & PKT_IS_FRAGMENT)
-                return 0;
+                return false;
             const DetectPort *dport = DetectPortLookupGroup(s->dp, p->dp);
             if (dport == NULL) {
                 SCLogDebug("dport didn't match.");
-                return 0;
+                return false;
             }
         }
         if (!(sflags & SIG_FLAG_SP_ANY)) {
             if (p->flags & PKT_IS_FRAGMENT)
-                return 0;
+                return false;
             const DetectPort *sport = DetectPortLookupGroup(s->sp, p->sp);
             if (sport == NULL) {
                 SCLogDebug("sport didn't match.");
-                return 0;
+                return false;
             }
         }
     } else if ((sflags & (SIG_FLAG_DP_ANY|SIG_FLAG_SP_ANY)) != (SIG_FLAG_DP_ANY|SIG_FLAG_SP_ANY)) {
         SCLogDebug("port-less protocol and sig needs ports");
-        return 0;
+        return false;
     }
 
     /* check the destination address */
     if (!(sflags & SIG_FLAG_DST_ANY)) {
         if (PKT_IS_IPV4(p)) {
             if (DetectAddressMatchIPv4(s->addr_dst_match4, s->addr_dst_match4_cnt, &p->dst) == 0)
-                return 0;
+                return false;
         } else if (PKT_IS_IPV6(p)) {
             if (DetectAddressMatchIPv6(s->addr_dst_match6, s->addr_dst_match6_cnt, &p->dst) == 0)
-                return 0;
+                return false;
         }
     }
     /* check the source address */
     if (!(sflags & SIG_FLAG_SRC_ANY)) {
         if (PKT_IS_IPV4(p)) {
             if (DetectAddressMatchIPv4(s->addr_src_match4, s->addr_src_match4_cnt, &p->src) == 0)
-                return 0;
+                return false;
         } else if (PKT_IS_IPV6(p)) {
             if (DetectAddressMatchIPv6(s->addr_src_match6, s->addr_src_match6_cnt, &p->src) == 0)
-                return 0;
+                return false;
         }
     }
 
-    return 1;
+    return true;
 }
 
 /** \internal
@@ -783,7 +781,7 @@ static inline void DetectRulePacketRules(
             }
         }
 
-        if (DetectRunInspectRuleHeader(p, pflow, s, sflags, s_proto_flags) == 0) {
+        if (DetectRunInspectRuleHeader(p, pflow, s, sflags, s_proto_flags) == false) {
             goto next;
         }
 
@@ -1075,7 +1073,7 @@ static bool DetectRunTxInspectRule(ThreadVars *tv,
     /* for a new inspection we inspect pkt header and packet matches */
     if (likely(stored_flags == NULL)) {
         TRACE_SID_TXS(s->id, tx, "first inspect, run packet matches");
-        if (DetectRunInspectRuleHeader(p, f, s, s->flags, s->proto.flags) == 0) {
+        if (DetectRunInspectRuleHeader(p, f, s, s->flags, s->proto.flags) == false) {
             TRACE_SID_TXS(s->id, tx, "DetectRunInspectRuleHeader() no match");
             return false;
         }
@@ -1637,10 +1635,10 @@ static void DetectRunFrames(ThreadVars *tv, DetectEngineCtx *de_ctx, DetectEngin
 
             /* call individual rule inspection */
             RULE_PROFILING_START(p);
-            int r = DetectRunInspectRuleHeader(p, f, s, s->flags, s->proto.flags);
-            if (r == 1) {
+            bool r = DetectRunInspectRuleHeader(p, f, s, s->flags, s->proto.flags);
+            if (r == true) {
                 r = DetectRunFrameInspectRule(tv, det_ctx, s, f, p, frames, frame);
-                if (r == 1) {
+                if (r == true) {
                     /* match */
                     DetectRunPostMatch(tv, det_ctx, p, s);
 

From 3b8ed937d723d553c0ff486b4ba28a2b14beed06 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 31 Dec 2023 08:17:24 +0100
Subject: [PATCH 346/462] detect: remove DCERPC mask logic

Added nothing over alproto check already in place.
---
 src/detect-engine-analyzer.c |  3 ---
 src/detect-engine-build.c    | 32 --------------------------------
 src/detect.h                 |  3 +--
 3 files changed, 1 insertion(+), 37 deletions(-)

diff --git a/src/detect-engine-analyzer.c b/src/detect-engine-analyzer.c
index b936ba1e0c38..43fd6d84c595 100644
--- a/src/detect-engine-analyzer.c
+++ b/src/detect-engine-analyzer.c
@@ -949,9 +949,6 @@ void EngineAnalysisRules2(const DetectEngineCtx *de_ctx, const Signature *s)
     if (s->mask & SIG_MASK_REQUIRE_FLAGS_UNUSUAL) {
         jb_append_string(ctx.js, "tcp_flags_unusual");
     }
-    if (s->mask & SIG_MASK_REQUIRE_DCERPC) {
-        jb_append_string(ctx.js, "dcerpc");
-    }
     if (s->mask & SIG_MASK_REQUIRE_ENGINE_EVENT) {
         jb_append_string(ctx.js, "engine_event");
     }
diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index b9d0c9a4380b..54323bce097b 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -434,44 +434,12 @@ PacketCreateMask(Packet *p, SignatureMask *mask, AppProto alproto,
         SCLogDebug("packet has flow");
         (*mask) |= SIG_MASK_REQUIRE_FLOW;
     }
-
-    if (alproto == ALPROTO_SMB || alproto == ALPROTO_DCERPC) {
-        SCLogDebug("packet will be inspected for DCERPC");
-        (*mask) |= SIG_MASK_REQUIRE_DCERPC;
-    }
-}
-
-static int g_dce_generic_list_id = -1;
-static int g_dce_stub_data_buffer_id = -1;
-
-static bool SignatureNeedsDCERPCMask(const Signature *s)
-{
-    if (g_dce_generic_list_id == -1) {
-        g_dce_generic_list_id = DetectBufferTypeGetByName("dce_generic");
-        SCLogDebug("g_dce_generic_list_id %d", g_dce_generic_list_id);
-    }
-    if (g_dce_stub_data_buffer_id == -1) {
-        g_dce_stub_data_buffer_id = DetectBufferTypeGetByName("dce_stub_data");
-        SCLogDebug("g_dce_stub_data_buffer_id %d", g_dce_stub_data_buffer_id);
-    }
-
-    if (DetectBufferIsPresent(s, g_dce_generic_list_id) ||
-            DetectBufferIsPresent(s, g_dce_stub_data_buffer_id)) {
-        return true;
-    }
-
-    return false;
 }
 
 static int SignatureCreateMask(Signature *s)
 {
     SCEnter();
 
-    if (SignatureNeedsDCERPCMask(s)) {
-        s->mask |= SIG_MASK_REQUIRE_DCERPC;
-        SCLogDebug("sig requires DCERPC");
-    }
-
     if (s->init_data->smlists[DETECT_SM_LIST_PMATCH] != NULL) {
         s->mask |= SIG_MASK_REQUIRE_PAYLOAD;
         SCLogDebug("sig requires payload");
diff --git a/src/detect.h b/src/detect.h
index c640ff3b6923..181ae2292306 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -298,8 +298,7 @@ typedef struct DetectPort_ {
 #define SIG_MASK_REQUIRE_FLAGS_INITDEINIT   BIT_U8(2)    /* SYN, FIN, RST */
 #define SIG_MASK_REQUIRE_FLAGS_UNUSUAL      BIT_U8(3)    /* URG, ECN, CWR */
 #define SIG_MASK_REQUIRE_NO_PAYLOAD         BIT_U8(4)
-#define SIG_MASK_REQUIRE_DCERPC             BIT_U8(5)    /* require either SMB+DCE or raw DCE */
-// vacancy
+// vacancy 2x
 #define SIG_MASK_REQUIRE_ENGINE_EVENT       BIT_U8(7)
 
 /* for now a uint8_t is enough */

From 75c1b7fb10ba808f4cb3961a3c7b9374c9906196 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 3 Jan 2024 12:09:59 +0100
Subject: [PATCH 347/462] detect: consolidate per rule group file loops

Don't loop multiple times over the per group sig array.
---
 src/detect-engine-build.c    |   5 +-
 src/detect-engine-siggroup.c | 101 ++++-------------------------------
 src/detect-engine-siggroup.h |   6 +--
 3 files changed, 13 insertions(+), 99 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 54323bce097b..6a1c53113601 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1825,10 +1825,7 @@ int SigPrepareStage4(DetectEngineCtx *de_ctx)
 
         SCLogDebug("sgh %p", sgh);
 
-        SigGroupHeadSetFilemagicFlag(de_ctx, sgh);
-        SigGroupHeadSetFileHashFlag(de_ctx, sgh);
-        SigGroupHeadSetFilesizeFlag(de_ctx, sgh);
-        SigGroupHeadSetFilestoreCount(de_ctx, sgh);
+        SigGroupHeadSetupFiles(de_ctx, sgh);
         SCLogDebug("filestore count %u", sgh->filestore_cnt);
 
         PrefilterSetupRuleGroup(de_ctx, sgh);
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index dfb3c10895e8..4fadaac5c013 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -513,122 +513,41 @@ int SigGroupHeadBuildMatchArray(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
 }
 
 /**
- *  \brief Set the need magic flag in the sgh.
- *
- *  \param de_ctx detection engine ctx for the signatures
- *  \param sgh sig group head to set the flag in
- */
-void SigGroupHeadSetFilemagicFlag(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
-{
-#ifdef HAVE_MAGIC
-    Signature *s = NULL;
-    uint32_t sig = 0;
-
-    if (sgh == NULL)
-        return;
-
-    for (sig = 0; sig < sgh->init->sig_cnt; sig++) {
-        s = sgh->init->match_array[sig];
-        if (s == NULL)
-            continue;
-
-        if (SignatureIsFilemagicInspecting(s)) {
-            sgh->flags |= SIG_GROUP_HEAD_HAVEFILEMAGIC;
-            break;
-        }
-    }
-#endif
-    return;
-}
-
-/**
- *  \brief Set the need size flag in the sgh.
+ *  \brief Set the need hash flag in the sgh.
  *
  *  \param de_ctx detection engine ctx for the signatures
- *  \param sgh sig group head to set the flag in
+ *  \param sgh sig group head to update
  */
-void SigGroupHeadSetFilesizeFlag(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
+void SigGroupHeadSetupFiles(const DetectEngineCtx *de_ctx, SigGroupHead *sgh)
 {
-    Signature *s = NULL;
-    uint32_t sig = 0;
-
     if (sgh == NULL)
         return;
 
-    for (sig = 0; sig < sgh->init->sig_cnt; sig++) {
-        s = sgh->init->match_array[sig];
+    for (uint32_t sig = 0; sig < sgh->init->sig_cnt; sig++) {
+        const Signature *s = sgh->init->match_array[sig];
         if (s == NULL)
             continue;
 
         if (SignatureIsFilesizeInspecting(s)) {
             sgh->flags |= SIG_GROUP_HEAD_HAVEFILESIZE;
-            break;
         }
-    }
-
-    return;
-}
-
-/**
- *  \brief Set the need hash flag in the sgh.
- *
- *  \param de_ctx detection engine ctx for the signatures
- *  \param sgh sig group head to set the flag in
- */
-void SigGroupHeadSetFileHashFlag(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
-{
-    Signature *s = NULL;
-    uint32_t sig = 0;
-
-    if (sgh == NULL)
-        return;
-
-    for (sig = 0; sig < sgh->init->sig_cnt; sig++) {
-        s = sgh->init->match_array[sig];
-        if (s == NULL)
-            continue;
-
         if (SignatureIsFileMd5Inspecting(s)) {
             sgh->flags |= SIG_GROUP_HEAD_HAVEFILEMD5;
             SCLogDebug("sgh %p has filemd5", sgh);
-            break;
         }
-
         if (SignatureIsFileSha1Inspecting(s)) {
             sgh->flags |= SIG_GROUP_HEAD_HAVEFILESHA1;
             SCLogDebug("sgh %p has filesha1", sgh);
-            break;
         }
-
         if (SignatureIsFileSha256Inspecting(s)) {
             sgh->flags |= SIG_GROUP_HEAD_HAVEFILESHA256;
             SCLogDebug("sgh %p has filesha256", sgh);
-            break;
         }
-    }
-
-    return;
-}
-
-/**
- *  \brief Set the filestore_cnt in the sgh.
- *
- *  \param de_ctx detection engine ctx for the signatures
- *  \param sgh sig group head to set the counter in
- */
-void SigGroupHeadSetFilestoreCount(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
-{
-    Signature *s = NULL;
-    uint32_t sig = 0;
-
-    if (sgh == NULL)
-        return;
-
-    for (sig = 0; sig < sgh->init->sig_cnt; sig++) {
-        s = sgh->init->match_array[sig];
-        if (s == NULL)
-            continue;
-
+#ifdef HAVE_MAGIC
+        if (SignatureIsFilemagicInspecting(s)) {
+            sgh->flags |= SIG_GROUP_HEAD_HAVEFILEMAGIC;
+        }
+#endif
         if (SignatureIsFilestoring(s)) {
             sgh->filestore_cnt++;
         }
diff --git a/src/detect-engine-siggroup.h b/src/detect-engine-siggroup.h
index d4c9e93c6771..45887edbfb00 100644
--- a/src/detect-engine-siggroup.h
+++ b/src/detect-engine-siggroup.h
@@ -53,10 +53,8 @@ void SigGroupHeadRegisterTests(void);
 void SigGroupHeadPrintSigs(DetectEngineCtx *de_ctx, SigGroupHead *sgh);
 
 void SigGroupHeadStore(DetectEngineCtx *, SigGroupHead *);
-void SigGroupHeadSetFilemagicFlag(DetectEngineCtx *, SigGroupHead *);
-void SigGroupHeadSetFilestoreCount(DetectEngineCtx *, SigGroupHead *);
-void SigGroupHeadSetFileHashFlag(DetectEngineCtx *, SigGroupHead *);
-void SigGroupHeadSetFilesizeFlag(DetectEngineCtx *, SigGroupHead *);
+
+void SigGroupHeadSetupFiles(const DetectEngineCtx *de_ctx, SigGroupHead *sgh);
 
 int SigGroupHeadBuildNonPrefilterArray(DetectEngineCtx *de_ctx, SigGroupHead *sgh);
 

From 609cac58b71c856598ad25610bd7463458cedad0 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Wed, 3 Jan 2024 12:16:25 +0100
Subject: [PATCH 348/462] flow: minor optimization

Most of the time FlowGetFlowFromHash will succeed.
---
 src/flow.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/src/flow.c b/src/flow.c
index 9783b7883b0b..a7f4788150ed 100644
--- a/src/flow.c
+++ b/src/flow.c
@@ -536,12 +536,10 @@ void FlowHandlePacket(ThreadVars *tv, FlowLookupStruct *fls, Packet *p)
      * a new flow if necessary. If we get NULL, we're out of flow memory.
      * The returned flow is locked. */
     Flow *f = FlowGetFlowFromHash(tv, fls, p, &p->flow);
-    if (f == NULL)
-        return;
-
-    /* set the flow in the packet */
-    p->flags |= PKT_HAS_FLOW;
-    return;
+    if (f != NULL) {
+        /* set the flow in the packet */
+        p->flags |= PKT_HAS_FLOW;
+    }
 }
 
 /** \brief initialize the configuration

From f5565f42e7ae993963d7135271813ff30780bc96 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 8 Jan 2024 07:43:04 +0100
Subject: [PATCH 349/462] eve/email: improve logging binary data

Use jb_append_string_from_bytes() as it works better than
BytesToString+jb_append_string when logging binary data.

Bug: #6664.
---
 src/output-json-email-common.c | 48 ++++++++++------------------------
 1 file changed, 14 insertions(+), 34 deletions(-)

diff --git a/src/output-json-email-common.c b/src/output-json-email-common.c
index 31d855758579..19afe78b7ce6 100644
--- a/src/output-json-email-common.c
+++ b/src/output-json-email-common.c
@@ -149,9 +149,7 @@ static int JsonEmailAddToJsonArray(const uint8_t *val, size_t len, void *data)
 
     if (ajs == NULL)
         return 0;
-    char *value = BytesToString((uint8_t *)val, len);
-    jb_append_string(ajs, value);
-    SCFree(value);
+    jb_append_string_from_bytes(ajs, val, (uint32_t)len);
     return 1;
 }
 
@@ -193,12 +191,8 @@ static void EveEmailLogJSONCustom(OutputJsonEmailCtx *email_ctx, JsonBuilder *js
             } else {
                 field = MimeDecFindField(entity, email_fields[f].email_field);
                 if (field != NULL) {
-                    char *s = BytesToString((uint8_t *)field->value,
-                            (size_t)field->value_len);
-                    if (likely(s != NULL)) {
-                        jb_set_string(js, email_fields[f].config_field, s);
-                        SCFree(s);
-                    }
+                    jb_set_string_from_bytes(
+                            js, email_fields[f].config_field, field->value, field->value_len);
                 }
             }
 
@@ -295,19 +289,14 @@ static bool EveEmailLogJsonData(const Flow *f, void *state, void *vtx, uint64_t
             bool has_ipv4_url = false;
             bool has_exe_url = false;
             for (url = entity->url_list; url != NULL; url = url->next) {
-                char *s = BytesToString((uint8_t *)url->url,
-                                        (size_t)url->url_len);
-                if (s != NULL) {
-                    jb_append_string(js_url, s);
-                    if (url->url_flags & URL_IS_EXE)
-                        has_exe_url = true;
-                    if (url->url_flags & URL_IS_IP6)
-                        has_ipv6_url = true;
-                    if (url->url_flags & URL_IS_IP4)
-                        has_ipv6_url = true;
-                    SCFree(s);
-                    url_cnt += 1;
-                }
+                jb_append_string_from_bytes(js_url, url->url, url->url_len);
+                if (url->url_flags & URL_IS_EXE)
+                    has_exe_url = true;
+                if (url->url_flags & URL_IS_IP6)
+                    has_ipv6_url = true;
+                if (url->url_flags & URL_IS_IP4)
+                    has_ipv6_url = true;
+                url_cnt += 1;
             }
             jb_set_bool(sjs, "has_ipv6_url", has_ipv6_url);
             jb_set_bool(sjs, "has_ipv4_url", has_ipv4_url);
@@ -315,23 +304,14 @@ static bool EveEmailLogJsonData(const Flow *f, void *state, void *vtx, uint64_t
         }
         for (entity = entity->child; entity != NULL; entity = entity->next) {
             if (entity->ctnt_flags & CTNT_IS_ATTACHMENT) {
-
-                char *s = BytesToString((uint8_t *)entity->filename,
-                                        (size_t)entity->filename_len);
-                jb_append_string(js_attach, s);
-                SCFree(s);
+                jb_append_string_from_bytes(js_attach, entity->filename, entity->filename_len);
                 attach_cnt += 1;
             }
             if (entity->url_list != NULL) {
                 MimeDecUrl *url;
                 for (url = entity->url_list; url != NULL; url = url->next) {
-                    char *s = BytesToString((uint8_t *)url->url,
-                                            (size_t)url->url_len);
-                    if (s != NULL) {
-                        jb_append_string(js_url, s);
-                        SCFree(s);
-                        url_cnt += 1;
-                    }
+                    jb_append_string_from_bytes(js_url, url->url, url->url_len);
+                    url_cnt += 1;
                 }
             }
         }

From 9a14d7a723f55ee1c2509673be27c6d98e62f7b9 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 8 Jan 2024 09:02:15 +0100
Subject: [PATCH 350/462] eve/http: use numeric status code by default

To avoid costly string operations.
---
 src/output-json-http.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/output-json-http.c b/src/output-json-http.c
index 5f44e955573d..e49eb5e8a74e 100644
--- a/src/output-json-http.c
+++ b/src/output-json-http.c
@@ -288,8 +288,13 @@ static void EveHttpLogJSONExtended(JsonBuilder *js, htp_tx_t *tx)
                 js, "protocol", bstr_ptr(tx->request_protocol), bstr_len(tx->request_protocol));
     }
 
-    /* response status */
-    if (tx->response_status != NULL) {
+    /* response status: from libhtp:
+     * "Response status code, available only if we were able to parse it, HTP_STATUS_INVALID
+     *  otherwise. HTP_STATUS_UNKNOWN until parsing is attempted" .*/
+    const int resp = tx->response_status_number;
+    if (resp > 0) {
+        jb_set_uint(js, "status", (uint32_t)resp);
+    } else if (tx->response_status != NULL) {
         const size_t status_size = bstr_len(tx->response_status) * 2 + 1;
         char status_string[status_size];
         BytesToStringBuffer(bstr_ptr(tx->response_status), bstr_len(tx->response_status),

From 1dcf69b211ac1ef438951aadc5d706705abf1b6c Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Mon, 8 Jan 2024 09:02:46 +0100
Subject: [PATCH 351/462] eve/http: add location header independent of status
 availability

---
 src/output-json-http.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/output-json-http.c b/src/output-json-http.c
index e49eb5e8a74e..abbeaf07edb6 100644
--- a/src/output-json-http.c
+++ b/src/output-json-http.c
@@ -301,12 +301,12 @@ static void EveHttpLogJSONExtended(JsonBuilder *js, htp_tx_t *tx)
                 status_string, status_size);
         unsigned int val = strtoul(status_string, NULL, 10);
         jb_set_uint(js, "status", val);
+    }
 
-        htp_header_t *h_location = htp_table_get_c(tx->response_headers, "location");
-        if (h_location != NULL) {
-            jb_set_string_from_bytes(
-                    js, "redirect", bstr_ptr(h_location->value), bstr_len(h_location->value));
-        }
+    htp_header_t *h_location = htp_table_get_c(tx->response_headers, "location");
+    if (h_location != NULL) {
+        jb_set_string_from_bytes(
+                js, "redirect", bstr_ptr(h_location->value), bstr_len(h_location->value));
     }
 
     /* length */

From adf5e6da7bdf81d65ccfeb115e6bc50e7031a0ca Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 12 Dec 2023 09:34:04 +0100
Subject: [PATCH 352/462] detect: strip_pseudo_headers transform

Ticket: 6546
---
 doc/userguide/rules/transforms.rst          |  14 +++
 src/Makefile.am                             |   2 +
 src/detect-engine-register.c                |   2 +
 src/detect-engine-register.h                |   1 +
 src/detect-transform-strip-pseudo-headers.c | 100 ++++++++++++++++++++
 src/detect-transform-strip-pseudo-headers.h |  30 ++++++
 6 files changed, 149 insertions(+)
 create mode 100644 src/detect-transform-strip-pseudo-headers.c
 create mode 100644 src/detect-transform-strip-pseudo-headers.h

diff --git a/doc/userguide/rules/transforms.rst b/doc/userguide/rules/transforms.rst
index 0067ace1de8b..f730f0d2dc71 100644
--- a/doc/userguide/rules/transforms.rst
+++ b/doc/userguide/rules/transforms.rst
@@ -174,3 +174,17 @@ Example::
 
     alert http any any -> any any (msg:"HTTP authorization"; http.header_names; \
         header_lowercase; content:"authorization:"; sid:1;)
+
+strip_pseudo_headers
+--------------------
+
+This transform is meant for HTTP/1 HTTP/2 header names normalization.
+It strips HTTP2 pseudo-headers (names and values).
+
+The implementation just strips every line beginning by ``:``.
+
+This example alerts for both HTTP/1 and HTTP/2 with only a user agent
+Example::
+
+    alert http any any -> any any (msg:"HTTP ua only"; http.header_names; \
+       bsize:16; content:"|0d 0a|User-Agent|0d 0a 0d 0a|"; nocase; sid:1;)
diff --git a/src/Makefile.am b/src/Makefile.am
index 133ed47cd1e8..2af8b1d44cd8 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -349,6 +349,7 @@ noinst_HEADERS = \
 	detect-transform-pcrexform.h \
 	detect-transform-sha1.h \
 	detect-transform-sha256.h \
+	detect-transform-strip-pseudo-headers.h \
 	detect-transform-strip-whitespace.h \
 	detect-transform-urldecode.h \
 	detect-transform-xor.h \
@@ -964,6 +965,7 @@ libsuricata_c_a_SOURCES = \
 	detect-transform-pcrexform.c \
 	detect-transform-sha1.c \
 	detect-transform-sha256.c \
+	detect-transform-strip-pseudo-headers.c \
 	detect-transform-strip-whitespace.c \
 	detect-transform-urldecode.c \
 	detect-transform-xor.c \
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index 9f37e0945544..218b0d7f0cb1 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -244,6 +244,7 @@
 
 #include "detect-transform-compress-whitespace.h"
 #include "detect-transform-strip-whitespace.h"
+#include "detect-transform-strip-pseudo-headers.h"
 #include "detect-transform-md5.h"
 #include "detect-transform-sha1.h"
 #include "detect-transform-sha256.h"
@@ -706,6 +707,7 @@ void SigTableSetup(void)
 
     DetectTransformCompressWhitespaceRegister();
     DetectTransformStripWhitespaceRegister();
+    DetectTransformStripPseudoHeadersRegister();
     DetectTransformMd5Register();
     DetectTransformSha1Register();
     DetectTransformSha256Register();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 9dd01f5fd487..8d4c7dfad3c1 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -323,6 +323,7 @@ enum DetectKeywordId {
 
     DETECT_TRANSFORM_COMPRESS_WHITESPACE,
     DETECT_TRANSFORM_STRIP_WHITESPACE,
+    DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS,
     DETECT_TRANSFORM_MD5,
     DETECT_TRANSFORM_SHA1,
     DETECT_TRANSFORM_SHA256,
diff --git a/src/detect-transform-strip-pseudo-headers.c b/src/detect-transform-strip-pseudo-headers.c
new file mode 100644
index 000000000000..450900d46037
--- /dev/null
+++ b/src/detect-transform-strip-pseudo-headers.c
@@ -0,0 +1,100 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Philippe Antoine <contact@catenacyber.fr>
+ *
+ * Implements the strip_pseudo_headers transform keyword with option support
+ */
+
+#include "suricata-common.h"
+#include "detect.h"
+#include "detect-engine.h"
+#include "detect-parse.h"
+#include "detect-transform-strip-pseudo-headers.h"
+
+/**
+ *  \internal
+ *  \brief Apply the strip_pseudo_headers keyword to the last pattern match
+ *  \param det_ctx detection engine ctx
+ *  \param s signature
+ *  \param optstr options string
+ *  \retval 0 ok
+ *  \retval -1 failure
+ */
+static int DetectTransformStripPseudoHeadersSetup(
+        DetectEngineCtx *de_ctx, Signature *s, const char *optstr)
+{
+    SCEnter();
+    int r = DetectSignatureAddTransform(s, DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS, NULL);
+    SCReturnInt(r);
+}
+
+static void DetectTransformStripPseudoHeaders(InspectionBuffer *buffer, void *options)
+{
+    const uint8_t *input = buffer->inspect;
+    const uint32_t input_len = buffer->inspect_len;
+    if (input_len == 0) {
+        return;
+    }
+    uint8_t output[input_len];
+
+    bool new_line = true;
+    bool pseudo = false;
+    uint32_t j = 0;
+    for (uint32_t i = 0; i < input_len; i++) {
+        if (new_line) {
+            if (input[i] == ':') {
+                pseudo = true;
+            }
+            if (input[i] != '\r' && input[i] != '\n') {
+                new_line = false;
+            }
+        } else {
+            if (input[i] == '\n') {
+                new_line = true;
+                if (!pseudo) {
+                    output[j] = input[i];
+                    j++;
+                }
+                pseudo = false;
+                continue;
+            }
+        }
+        if (!pseudo) {
+            output[j] = input[i];
+            j++;
+        }
+    }
+    InspectionBufferCopy(buffer, output, j);
+}
+
+void DetectTransformStripPseudoHeadersRegister(void)
+{
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].name = "strip_pseudo_headers";
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].desc =
+            "modify buffer via stripping pseudo headers";
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].url =
+            "/rules/transforms.html#strip_pseudo_headers";
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].Transform =
+            DetectTransformStripPseudoHeaders;
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].Setup =
+            DetectTransformStripPseudoHeadersSetup;
+    sigmatch_table[DETECT_TRANSFORM_STRIP_PSEUDO_HEADERS].flags |= SIGMATCH_NOOPT;
+}
diff --git a/src/detect-transform-strip-pseudo-headers.h b/src/detect-transform-strip-pseudo-headers.h
new file mode 100644
index 000000000000..c2016d438f04
--- /dev/null
+++ b/src/detect-transform-strip-pseudo-headers.h
@@ -0,0 +1,30 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Philippe Antoine <contact@catenacyber.fr>
+ */
+
+#ifndef __DETECT_TRANSFORM_STRIP_PSEUDOHEADERS_H__
+#define __DETECT_TRANSFORM_STRIP_PSEUDOHEADERS_H__
+
+/* prototypes */
+void DetectTransformStripPseudoHeadersRegister(void);
+
+#endif /* __DETECT_TRANSFORM_STRIP_PSEUDOHEADERS_H__ */

From b239e88c9303d83a3074ed69c4989bd136bbf8b4 Mon Sep 17 00:00:00 2001
From: Gleb Smirnoff <glebius@FreeBSD.org>
Date: Thu, 11 Jan 2024 13:35:21 -0800
Subject: [PATCH 353/462] ipfw: close(2) instead shutdown(2) of the divert(4)
 socket

The shutdown(2) syscall would always return ENOTCONN for FreeBSD 11,
FreeBSD 12, FreeBSD 13 and FreeBSD 14.  It could do some action on the
socket in the kernel in FreeBSD 10 and before, did not test.
---
 src/source-ipfw.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/src/source-ipfw.c b/src/source-ipfw.c
index 6d0f67c11572..4bdb7724598a 100644
--- a/src/source-ipfw.c
+++ b/src/source-ipfw.c
@@ -412,8 +412,7 @@ TmEcode ReceiveIPFWThreadDeinit(ThreadVars *tv, void *data)
 
     SCEnter();
 
-    /* Attempt to shut the socket down...close instead? */
-    if (shutdown(nq->fd, SHUT_RD) < 0) {
+    if (close(nq->fd) < 0) {
         SCLogWarning("Unable to disable ipfw socket: %s", strerror(errno));
         SCReturnInt(TM_ECODE_FAILED);
     }

From c49463c86f4c54f3bd322eaead07ce9a2c0b1d56 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 4 Jan 2024 11:48:32 +0100
Subject: [PATCH 354/462] rust: fix assertions_on_constants for assert!(true)

Which will be optimized away by the compiler
---
 rust/src/tftp/tftp.rs | 20 ++++----------------
 1 file changed, 4 insertions(+), 16 deletions(-)

diff --git a/rust/src/tftp/tftp.rs b/rust/src/tftp/tftp.rs
index 1b093fed256d..8797e4ea1372 100644
--- a/rust/src/tftp/tftp.rs
+++ b/rust/src/tftp/tftp.rs
@@ -225,14 +225,8 @@ mod test {
             tx_data: AppLayerTxData::new(),
         };
 
-        match parse_tftp_request(&READ_REQUEST[..]) {
-            Some(txp) => {
-                assert_eq!(tx, txp);
-            }
-            None => {
-                assert!(true);
-            }
-        }
+        let txp = parse_tftp_request(&READ_REQUEST[..]).unwrap();
+        assert_eq!(tx, txp);
     }
 
     #[test]
@@ -245,14 +239,8 @@ mod test {
             tx_data: AppLayerTxData::new(),
         };
 
-        match parse_tftp_request(&WRITE_REQUEST[..]) {
-            Some(txp) => {
-                assert_eq!(tx, txp);
-            }
-            None => {
-                assert!(true, "fadfasd");
-            }
-        }
+        let txp = parse_tftp_request(&WRITE_REQUEST[..]).unwrap();
+        assert_eq!(tx, txp);
     }
 
     // Invalid request: filename not terminated

From a8199bf2ca16e8394b6bf5c41ba1bafe88f6ff53 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 4 Jan 2024 12:48:12 +0100
Subject: [PATCH 355/462] rust: fix assertions_on_constants for assert!(false)

using panic! instead with a string message
---
 rust/src/dcerpc/dcerpc_udp.rs |  29 +--
 rust/src/detect/byte_math.rs  | 264 ++++++----------------
 rust/src/detect/uint.rs       |  29 +--
 rust/src/dhcp/parser.rs       |  62 +++---
 rust/src/dns/parser.rs        | 408 +++++++++++++++-------------------
 rust/src/sip/parser.rs        |  30 +--
 6 files changed, 302 insertions(+), 520 deletions(-)

diff --git a/rust/src/dcerpc/dcerpc_udp.rs b/rust/src/dcerpc/dcerpc_udp.rs
index 83707bddcb21..d34c3e480b36 100644
--- a/rust/src/dcerpc/dcerpc_udp.rs
+++ b/rust/src/dcerpc/dcerpc_udp.rs
@@ -410,12 +410,7 @@ mod tests {
             0x1c, 0x7d, 0xcf, 0x11,
         ];
 
-        match parser::parse_dcerpc_udp_header(request) {
-            Ok((_rem, _header)) => {
-                { assert!(false); }
-            }
-            _ => {}
-        }
+        assert!(parser::parse_dcerpc_udp_header(request).is_err());
     }
 
     #[test]
@@ -428,13 +423,9 @@ mod tests {
             0x79, 0xbe, 0x01, 0x34, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
             0xff, 0xff, 0xff, 0xff, 0x68, 0x00, 0x00, 0x00, 0x0a, 0x00,
         ];
-        match parser::parse_dcerpc_udp_header(request) {
-            Ok((rem, header)) => {
-                assert_eq!(4, header.rpc_vers);
-                assert_eq!(80, request.len() - rem.len());
-            }
-            _ => { assert!(false); }
-        }
+        let (rem, header) = parser::parse_dcerpc_udp_header(request).unwrap();
+        assert_eq!(4, header.rpc_vers);
+        assert_eq!(80, request.len() - rem.len());
     }
 
     #[test]
@@ -447,14 +438,10 @@ mod tests {
             0x79, 0xbe, 0x01, 0x34, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
             0xff, 0xff, 0xff, 0xff, 0x68, 0x00, 0x00, 0x00, 0x0a, 0x00,
         ];
-        match parser::parse_dcerpc_udp_header(request) {
-            Ok((rem, header)) => {
-                assert_eq!(4, header.rpc_vers);
-                assert_eq!(80, request.len() - rem.len());
-                assert_eq!(0, rem.len());
-            }
-            _ => { assert!(false); }
-        }
+        let (rem, header) = parser::parse_dcerpc_udp_header(request).unwrap();
+        assert_eq!(4, header.rpc_vers);
+        assert_eq!(80, request.len() - rem.len());
+        assert_eq!(0, rem.len());
     }
 
     #[test]
diff --git a/rust/src/detect/byte_math.rs b/rust/src/detect/byte_math.rs
index 80bd3d5ee178..0586a92e5df8 100644
--- a/rust/src/detect/byte_math.rs
+++ b/rust/src/detect/byte_math.rs
@@ -517,14 +517,8 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath(args) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath(args).unwrap();
+        assert_eq!(val, bmd);
     }
 
     #[test]
@@ -623,52 +617,26 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, string dec",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
 
         bmd.flags = DETECT_BYTEMATH_FLAG_RVALUE_VAR;
         bmd.base = BASE_DEFAULT;
-        match parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
 
         bmd.flags = DETECT_BYTEMATH_FLAG_RVALUE_VAR | DETECT_BYTEMATH_FLAG_STRING;
         bmd.base = ByteMathBase::BaseHex;
-        match parse_bytemath(
-            "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, string hex",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, string hex").unwrap();
+        assert_eq!(val, bmd);
 
         bmd.base = ByteMathBase::BaseOct;
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, string oct",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
     }
 
     #[test]
@@ -774,40 +742,25 @@ mod tests {
         };
 
         bmd.bitmask_val = 0x12345678;
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, bitmask 0x12345678",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.bitmask_val = 0xffff1234;
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, bitmask ffff1234",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.bitmask_val = 0xffff1234;
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, bitmask 0Xffff1234",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
+
     }
     #[test]
     fn test_parser_endian_valid() {
@@ -824,49 +777,29 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, endian big",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.endian = ByteMathEndian::LittleEndian;
-        match parse_bytemath(
+        let (_, val) = parse_bytemath(
             "bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, endian little",
-        ) {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        ).unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.endian = ByteMathEndian::EndianDCE;
-        match parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, dce") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo, dce").unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.endian = DETECT_BYTEMATH_ENDIAN_DEFAULT;
         bmd.flags = DETECT_BYTEMATH_FLAG_RVALUE_VAR;
-        match parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
     }
 
     #[test]
@@ -920,61 +853,31 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper +, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.oper = ByteMathOperator::Subtraction;
-        match parse_bytemath("bytes 4, offset 3933, oper -, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper -, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.oper = ByteMathOperator::Multiplication;
-        match parse_bytemath("bytes 4, offset 3933, oper *, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper *, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
         bmd.oper = ByteMathOperator::Division;
-        match parse_bytemath("bytes 4, offset 3933, oper /, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper /, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
         bmd.oper = ByteMathOperator::RightShift;
-        match parse_bytemath("bytes 4, offset 3933, oper >>, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper >>, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
         bmd.oper = ByteMathOperator::LeftShift;
-        match parse_bytemath("bytes 4, offset 3933, oper <<, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 3933, oper <<, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
     }
 
     #[test]
@@ -1013,33 +916,18 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath("bytes 4, offset 47303, oper *, rvalue 4294967295      , result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 47303, oper *, rvalue 4294967295      , result foo").unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.rvalue = 1;
-        match parse_bytemath("bytes 4, offset 47303, oper *, rvalue 1, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 47303, oper *, rvalue 1, result foo").unwrap();
+        assert_eq!(val, bmd);
+
         bmd.rvalue = 0;
-        match parse_bytemath("bytes 4, offset 47303, oper *, rvalue 0, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 47303, oper *, rvalue 0, result foo").unwrap();
+        assert_eq!(val, bmd);
+
     }
 
     #[test]
@@ -1064,24 +952,14 @@ mod tests {
             ..Default::default()
         };
 
-        match parse_bytemath("bytes 4, offset -65535, oper *, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset -65535, oper *, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
 
         bmd.offset = 65535;
-        match parse_bytemath("bytes 4, offset 65535, oper *, rvalue myrvalue, result foo") {
-            Ok((_, val)) => {
-                assert_eq!(val, bmd);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = parse_bytemath("bytes 4, offset 65535, oper *, rvalue myrvalue, result foo").unwrap();
+        assert_eq!(val, bmd);
+
     }
 
     #[test]
diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index 3d6a5baab0ca..8c758e3a5d69 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -409,27 +409,12 @@ mod tests {
 
     #[test]
     fn test_parse_uint_unit() {
-        match detect_parse_uint::<u64>(" 2kb") {
-            Ok((_, val)) => {
-                assert_eq!(val.arg1, 2048);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
-        match detect_parse_uint::<u8>("2kb") {
-            Ok((_, _val)) => {
-                assert!(false);
-            }
-            Err(_) => {}
-        }
-        match detect_parse_uint::<u32>("3MB") {
-            Ok((_, val)) => {
-                assert_eq!(val.arg1, 3 * 1024 * 1024);
-            }
-            Err(_) => {
-                assert!(false);
-            }
-        }
+        let (_, val) = detect_parse_uint::<u64>(" 2kb").unwrap();
+        assert_eq!(val.arg1, 2048);
+
+        assert!(detect_parse_uint::<u8>("2kb").is_err());
+
+        let (_, val) = detect_parse_uint::<u32>("3MB").unwrap();
+        assert_eq!(val.arg1, 3 * 1024 * 1024);
     }
 }
diff --git a/rust/src/dhcp/parser.rs b/rust/src/dhcp/parser.rs
index 48acccff2c71..698462f301a1 100644
--- a/rust/src/dhcp/parser.rs
+++ b/rust/src/dhcp/parser.rs
@@ -242,42 +242,36 @@ mod tests {
         let pcap = include_bytes!("discover.pcap");
         let payload = &pcap[24 + 16 + 42..];
 
-        match dhcp_parse(payload) {
-            Ok((_rem, message)) => {
-                let header = message.header;
-                assert_eq!(header.opcode, BOOTP_REQUEST);
-                assert_eq!(header.htype, 1);
-                assert_eq!(header.hlen, 6);
-                assert_eq!(header.hops, 0);
-                assert_eq!(header.txid, 0x00003d1d);
-                assert_eq!(header.seconds, 0);
-                assert_eq!(header.flags, 0);
-                assert_eq!(header.clientip, &[0, 0, 0, 0]);
-                assert_eq!(header.yourip, &[0, 0, 0, 0]);
-                assert_eq!(header.serverip, &[0, 0, 0, 0]);
-                assert_eq!(header.giaddr, &[0, 0, 0, 0]);
-                assert_eq!(
-                    &header.clienthw[..(header.hlen as usize)],
-                    &[0x00, 0x0b, 0x82, 0x01, 0xfc, 0x42]
-                );
-                assert!(header.servername.iter().all(|&x| x == 0));
-                assert!(header.bootfilename.iter().all(|&x| x == 0));
-                assert_eq!(header.magic, &[0x63, 0x82, 0x53, 0x63]);
+        let (_rem, message) = dhcp_parse(payload).unwrap();
+        let header = message.header;
+        assert_eq!(header.opcode, BOOTP_REQUEST);
+        assert_eq!(header.htype, 1);
+        assert_eq!(header.hlen, 6);
+        assert_eq!(header.hops, 0);
+        assert_eq!(header.txid, 0x00003d1d);
+        assert_eq!(header.seconds, 0);
+        assert_eq!(header.flags, 0);
+        assert_eq!(header.clientip, &[0, 0, 0, 0]);
+        assert_eq!(header.yourip, &[0, 0, 0, 0]);
+        assert_eq!(header.serverip, &[0, 0, 0, 0]);
+        assert_eq!(header.giaddr, &[0, 0, 0, 0]);
+        assert_eq!(
+            &header.clienthw[..(header.hlen as usize)],
+            &[0x00, 0x0b, 0x82, 0x01, 0xfc, 0x42]
+        );
+        assert!(header.servername.iter().all(|&x| x == 0));
+        assert!(header.bootfilename.iter().all(|&x| x == 0));
+        assert_eq!(header.magic, &[0x63, 0x82, 0x53, 0x63]);
 
-                assert!(!message.malformed_options);
-                assert!(!message.truncated_options);
+        assert!(!message.malformed_options);
+        assert!(!message.truncated_options);
 
-                assert_eq!(message.options.len(), 5);
-                assert_eq!(message.options[0].code, DHCP_OPT_TYPE);
-                assert_eq!(message.options[1].code, DHCP_OPT_CLIENT_ID);
-                assert_eq!(message.options[2].code, DHCP_OPT_REQUESTED_IP);
-                assert_eq!(message.options[3].code, DHCP_OPT_PARAMETER_LIST);
-                assert_eq!(message.options[4].code, DHCP_OPT_END);
-            }
-            _ => {
-                assert!(false);
-            }
-        }
+        assert_eq!(message.options.len(), 5);
+        assert_eq!(message.options[0].code, DHCP_OPT_TYPE);
+        assert_eq!(message.options[1].code, DHCP_OPT_CLIENT_ID);
+        assert_eq!(message.options[2].code, DHCP_OPT_REQUESTED_IP);
+        assert_eq!(message.options[3].code, DHCP_OPT_PARAMETER_LIST);
+        assert_eq!(message.options[4].code, DHCP_OPT_END);
     }
 
     #[test]
diff --git a/rust/src/dns/parser.rs b/rust/src/dns/parser.rs
index f7f9fd0d6e8c..3d9938abf8f2 100644
--- a/rust/src/dns/parser.rs
+++ b/rust/src/dns/parser.rs
@@ -477,35 +477,29 @@ mod tests {
 
         let (body, header) = dns_parse_header(pkt).unwrap();
         let res = dns_parse_body(body, pkt, header);
-        match res {
-            Ok((rem, request)) => {
-                // For now we have some remainder data as there is an
-                // additional record type we don't parse yet.
-                assert!(!rem.is_empty());
-
-                assert_eq!(
-                    request.header,
-                    DNSHeader {
-                        tx_id: 0x8d32,
-                        flags: 0x0120,
-                        questions: 1,
-                        answer_rr: 0,
-                        authority_rr: 0,
-                        additional_rr: 1,
-                    }
-                );
-
-                assert_eq!(request.queries.len(), 1);
+        let (rem, request) = res.unwrap();
+        // For now we have some remainder data as there is an
+        // additional record type we don't parse yet.
+        assert!(!rem.is_empty());
 
-                let query = &request.queries[0];
-                assert_eq!(query.name, "www.suricata-ids.org".as_bytes().to_vec());
-                assert_eq!(query.rrtype, 1);
-                assert_eq!(query.rrclass, 1);
-            }
-            _ => {
-                assert!(false);
+        assert_eq!(
+            request.header,
+            DNSHeader {
+                tx_id: 0x8d32,
+                flags: 0x0120,
+                questions: 1,
+                answer_rr: 0,
+                authority_rr: 0,
+                additional_rr: 1,
             }
-        }
+        );
+
+        assert_eq!(request.queries.len(), 1);
+
+        let query = &request.queries[0];
+        assert_eq!(query.name, "www.suricata-ids.org".as_bytes().to_vec());
+        assert_eq!(query.rrtype, 1);
+        assert_eq!(query.rrclass, 1);
     }
 
     /// Parse a DNS response.
@@ -534,64 +528,57 @@ mod tests {
             0x00, 0x04, 0xc0, 0x00, 0x4e, 0x19, /* ....N. */
         ];
 
-        let res = dns_parse_response(pkt);
-        match res {
-            Ok((rem, response)) => {
-                // The response should be full parsed.
-                assert_eq!(rem.len(), 0);
-
-                assert_eq!(
-                    response.header,
-                    DNSHeader {
-                        tx_id: 0x8d32,
-                        flags: 0x81a0,
-                        questions: 1,
-                        answer_rr: 3,
-                        authority_rr: 0,
-                        additional_rr: 0,
-                    }
-                );
-
-                assert_eq!(response.answers.len(), 3);
-
-                let answer1 = &response.answers[0];
-                assert_eq!(answer1.name, "www.suricata-ids.org".as_bytes().to_vec());
-                assert_eq!(answer1.rrtype, 5);
-                assert_eq!(answer1.rrclass, 1);
-                assert_eq!(answer1.ttl, 3544);
-                assert_eq!(
-                    answer1.data,
-                    DNSRData::CNAME("suricata-ids.org".as_bytes().to_vec())
-                );
-
-                let answer2 = &response.answers[1];
-                assert_eq!(
-                    answer2,
-                    &DNSAnswerEntry {
-                        name: "suricata-ids.org".as_bytes().to_vec(),
-                        rrtype: 1,
-                        rrclass: 1,
-                        ttl: 244,
-                        data: DNSRData::A([192, 0, 78, 24].to_vec()),
-                    }
-                );
-
-                let answer3 = &response.answers[2];
-                assert_eq!(
-                    answer3,
-                    &DNSAnswerEntry {
-                        name: "suricata-ids.org".as_bytes().to_vec(),
-                        rrtype: 1,
-                        rrclass: 1,
-                        ttl: 244,
-                        data: DNSRData::A([192, 0, 78, 25].to_vec()),
-                    }
-                )
+        let (rem, response) = dns_parse_response(pkt).unwrap();
+        // The response should be full parsed.
+        assert_eq!(rem.len(), 0);
+
+        assert_eq!(
+            response.header,
+            DNSHeader {
+                tx_id: 0x8d32,
+                flags: 0x81a0,
+                questions: 1,
+                answer_rr: 3,
+                authority_rr: 0,
+                additional_rr: 0,
+            }
+        );
+
+        assert_eq!(response.answers.len(), 3);
+
+        let answer1 = &response.answers[0];
+        assert_eq!(answer1.name, "www.suricata-ids.org".as_bytes().to_vec());
+        assert_eq!(answer1.rrtype, 5);
+        assert_eq!(answer1.rrclass, 1);
+        assert_eq!(answer1.ttl, 3544);
+        assert_eq!(
+            answer1.data,
+            DNSRData::CNAME("suricata-ids.org".as_bytes().to_vec())
+        );
+
+        let answer2 = &response.answers[1];
+        assert_eq!(
+            answer2,
+            &DNSAnswerEntry {
+                name: "suricata-ids.org".as_bytes().to_vec(),
+                rrtype: 1,
+                rrclass: 1,
+                ttl: 244,
+                data: DNSRData::A([192, 0, 78, 24].to_vec()),
             }
-            _ => {
-                assert!(false);
+        );
+
+        let answer3 = &response.answers[2];
+        assert_eq!(
+            answer3,
+            &DNSAnswerEntry {
+                name: "suricata-ids.org".as_bytes().to_vec(),
+                rrtype: 1,
+                rrclass: 1,
+                ttl: 244,
+                data: DNSRData::A([192, 0, 78, 25].to_vec()),
             }
-        }
+        )
     }
 
     #[test]
@@ -617,49 +604,42 @@ mod tests {
             0x00, 0x00, 0x00, 0x00, /* .... */
         ];
 
-        let res = dns_parse_response(pkt);
-        match res {
-            Ok((rem, response)) => {
-                // For now we have some remainder data as there is an
-                // additional record type we don't parse yet.
-                assert!(!rem.is_empty());
-
-                assert_eq!(
-                    response.header,
-                    DNSHeader {
-                        tx_id: 0x8295,
-                        flags: 0x8183,
-                        questions: 1,
-                        answer_rr: 0,
-                        authority_rr: 1,
-                        additional_rr: 1,
-                    }
-                );
-
-                assert_eq!(response.authorities.len(), 1);
-
-                let authority = &response.authorities[0];
-                assert_eq!(authority.name, "oisf.net".as_bytes().to_vec());
-                assert_eq!(authority.rrtype, 6);
-                assert_eq!(authority.rrclass, 1);
-                assert_eq!(authority.ttl, 899);
-                assert_eq!(
-                    authority.data,
-                    DNSRData::SOA(DNSRDataSOA {
-                        mname: "ns-110.awsdns-13.com".as_bytes().to_vec(),
-                        rname: "awsdns-hostmaster.amazon.com".as_bytes().to_vec(),
-                        serial: 1,
-                        refresh: 7200,
-                        retry: 900,
-                        expire: 1209600,
-                        minimum: 86400,
-                    })
-                );
-            }
-            _ => {
-                assert!(false);
+        let (rem, response) = dns_parse_response(pkt).unwrap();
+        // For now we have some remainder data as there is an
+        // additional record type we don't parse yet.
+        assert!(!rem.is_empty());
+
+        assert_eq!(
+            response.header,
+            DNSHeader {
+                tx_id: 0x8295,
+                flags: 0x8183,
+                questions: 1,
+                answer_rr: 0,
+                authority_rr: 1,
+                additional_rr: 1,
             }
-        }
+        );
+
+        assert_eq!(response.authorities.len(), 1);
+
+        let authority = &response.authorities[0];
+        assert_eq!(authority.name, "oisf.net".as_bytes().to_vec());
+        assert_eq!(authority.rrtype, 6);
+        assert_eq!(authority.rrclass, 1);
+        assert_eq!(authority.ttl, 899);
+        assert_eq!(
+            authority.data,
+            DNSRData::SOA(DNSRDataSOA {
+                mname: "ns-110.awsdns-13.com".as_bytes().to_vec(),
+                rname: "awsdns-hostmaster.amazon.com".as_bytes().to_vec(),
+                serial: 1,
+                refresh: 7200,
+                retry: 900,
+                expire: 1209600,
+                minimum: 86400,
+            })
+        );
     }
 
     #[test]
@@ -678,49 +658,42 @@ mod tests {
             0x44, 0x03, 0xc5, 0xe9, 0x01, /* D.... */
         ];
 
-        let res = dns_parse_response(pkt);
-        match res {
-            Ok((rem, response)) => {
-                // The response should be fully parsed.
-                assert_eq!(rem.len(), 0);
-
-                assert_eq!(
-                    response.header,
-                    DNSHeader {
-                        tx_id: 0x12b0,
-                        flags: 0x8400,
-                        questions: 1,
-                        answer_rr: 1,
-                        authority_rr: 0,
-                        additional_rr: 0,
-                    }
-                );
-
-                assert_eq!(response.queries.len(), 1);
-                let query = &response.queries[0];
-                assert_eq!(query.name, "vaaaakardli.pirate.sea".as_bytes().to_vec());
-                assert_eq!(query.rrtype, DNS_RECORD_TYPE_NULL);
-                assert_eq!(query.rrclass, 1);
-
-                assert_eq!(response.answers.len(), 1);
-
-                let answer = &response.answers[0];
-                assert_eq!(answer.name, "vaaaakardli.pirate.sea".as_bytes().to_vec());
-                assert_eq!(answer.rrtype, DNS_RECORD_TYPE_NULL);
-                assert_eq!(answer.rrclass, 1);
-                assert_eq!(answer.ttl, 0);
-                assert_eq!(
-                    answer.data,
-                    DNSRData::NULL(vec![
-                        0x56, 0x41, 0x43, 0x4b, /* VACK */
-                        0x44, 0x03, 0xc5, 0xe9, 0x01, /* D.... */
-                    ])
-                );
-            }
-            _ => {
-                assert!(false);
+        let (rem, response) = dns_parse_response(pkt).unwrap();
+        // The response should be fully parsed.
+        assert_eq!(rem.len(), 0);
+
+        assert_eq!(
+            response.header,
+            DNSHeader {
+                tx_id: 0x12b0,
+                flags: 0x8400,
+                questions: 1,
+                answer_rr: 1,
+                authority_rr: 0,
+                additional_rr: 0,
             }
-        }
+        );
+
+        assert_eq!(response.queries.len(), 1);
+        let query = &response.queries[0];
+        assert_eq!(query.name, "vaaaakardli.pirate.sea".as_bytes().to_vec());
+        assert_eq!(query.rrtype, DNS_RECORD_TYPE_NULL);
+        assert_eq!(query.rrclass, 1);
+
+        assert_eq!(response.answers.len(), 1);
+
+        let answer = &response.answers[0];
+        assert_eq!(answer.name, "vaaaakardli.pirate.sea".as_bytes().to_vec());
+        assert_eq!(answer.rrtype, DNS_RECORD_TYPE_NULL);
+        assert_eq!(answer.rrclass, 1);
+        assert_eq!(answer.ttl, 0);
+        assert_eq!(
+            answer.data,
+            DNSRData::NULL(vec![
+                0x56, 0x41, 0x43, 0x4b, /* VACK */
+                0x44, 0x03, 0xc5, 0xe9, 0x01, /* D.... */
+            ])
+        );
     }
 
     #[test]
@@ -734,26 +707,16 @@ mod tests {
             0x9a, 0xbc, 0xde, 0xf6, 0x78, 0x90,
         ];
 
-        let res = dns_parse_rdata_sshfp(data);
-        match res {
-            Ok((rem, rdata)) => {
-                // The data should be fully parsed.
-                assert_eq!(rem.len(), 0);
-
-                match rdata {
-                    DNSRData::SSHFP(sshfp) => {
-                        assert_eq!(sshfp.algo, 2);
-                        assert_eq!(sshfp.fp_type, 1);
-                        assert_eq!(sshfp.fingerprint, &data[2..]);
-                    }
-                    _ => {
-                        assert!(false);
-                    }
-                }
-            }
-            _ => {
-                assert!(false);
-            }
+        let (rem, rdata) = dns_parse_rdata_sshfp(data).unwrap();
+        // The data should be fully parsed.
+        assert_eq!(rem.len(), 0);
+
+        if let DNSRData::SSHFP(sshfp) = rdata {
+            assert_eq!(sshfp.algo, 2);
+            assert_eq!(sshfp.fp_type, 1);
+            assert_eq!(sshfp.fingerprint, &data[2..]);
+        } else {
+            panic!("Expected DNSRData::SSHFP");
         }
     }
 
@@ -790,48 +753,35 @@ mod tests {
             0x67, 0x6c, 0x65, 0x03, 0x63, 0x6f, 0x6d, 0x00,
         ];
 
-        let res = dns_parse_response(pkt);
-        match res {
-            Ok((rem, response)) => {
-                // The data should be fully parsed.
-                assert_eq!(rem.len(), 0);
-
-                assert_eq!(response.answers.len(), 2);
-
-                let answer1 = &response.answers[0];
-                match &answer1.data {
-                    DNSRData::SRV(srv) => {
-                        assert_eq!(srv.priority, 20);
-                        assert_eq!(srv.weight, 1);
-                        assert_eq!(srv.port, 5060);
-                        assert_eq!(
-                            srv.target,
-                            "sip-anycast-2.voice.google.com".as_bytes().to_vec()
-                        );
-                    }
-                    _ => {
-                        assert!(false);
-                    }
-                }
-                let answer2 = &response.answers[1];
-                match &answer2.data {
-                    DNSRData::SRV(srv) => {
-                        assert_eq!(srv.priority, 10);
-                        assert_eq!(srv.weight, 1);
-                        assert_eq!(srv.port, 5060);
-                        assert_eq!(
-                            srv.target,
-                            "sip-anycast-1.voice.google.com".as_bytes().to_vec()
-                        );
-                    }
-                    _ => {
-                        assert!(false);
-                    }
-                }
-            }
-            _ => {
-                assert!(false);
-            }
+        let (rem, response) = dns_parse_response(pkt).unwrap();
+        // The data should be fully parsed.
+        assert_eq!(rem.len(), 0);
+
+        assert_eq!(response.answers.len(), 2);
+
+        let answer1 = &response.answers[0];
+        if let DNSRData::SRV(srv) = &answer1.data {
+            assert_eq!(srv.priority, 20);
+            assert_eq!(srv.weight, 1);
+            assert_eq!(srv.port, 5060);
+            assert_eq!(
+                srv.target,
+                "sip-anycast-2.voice.google.com".as_bytes().to_vec()
+            );
+        } else {
+            panic!("Expected DNSRData::SRV");
+        }
+        let answer2 = &response.answers[1];
+        if let DNSRData::SRV(srv) = &answer2.data {
+            assert_eq!(srv.priority, 10);
+            assert_eq!(srv.weight, 1);
+            assert_eq!(srv.port, 5060);
+            assert_eq!(
+                srv.target,
+                "sip-anycast-1.voice.google.com".as_bytes().to_vec()
+            );
+        } else {
+            panic!("Expected DNSRData::SRV");
         }
     }
 }
diff --git a/rust/src/sip/parser.rs b/rust/src/sip/parser.rs
index a34bc2615e53..cd98c252eb8b 100644
--- a/rust/src/sip/parser.rs
+++ b/rust/src/sip/parser.rs
@@ -275,17 +275,11 @@ mod tests {
                           \r\n"
             .as_bytes();
 
-        match sip_parse_request(buf) {
-            Ok((_, req)) => {
-                assert_eq!(req.method, "REGISTER");
-                assert_eq!(req.path, "sip:sip.cybercity.dk");
-                assert_eq!(req.version, "SIP/2.0");
-                assert_eq!(req.headers["Content-Length"], "0");
-            }
-            _ => {
-                assert!(false);
-            }
-        }
+        let (_, req) = sip_parse_request(buf).unwrap();
+        assert_eq!(req.method, "REGISTER");
+        assert_eq!(req.path, "sip:sip.cybercity.dk");
+        assert_eq!(req.version, "SIP/2.0");
+        assert_eq!(req.headers["Content-Length"], "0");
     }
 
     #[test]
@@ -311,15 +305,9 @@ mod tests {
                           \r\n"
             .as_bytes();
 
-        match sip_parse_response(buf) {
-            Ok((_, resp)) => {
-                assert_eq!(resp.version, "SIP/2.0");
-                assert_eq!(resp.code, "401");
-                assert_eq!(resp.reason, "Unauthorized");
-            }
-            _ => {
-                assert!(false);
-            }
-        }
+        let (_, resp) = sip_parse_response(buf).unwrap();
+        assert_eq!(resp.version, "SIP/2.0");
+        assert_eq!(resp.code, "401");
+        assert_eq!(resp.reason, "Unauthorized");
     }
 }

From 85329f5351dddf91b581b15bbbb1dd9ce435d4d6 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 5 Jan 2024 10:22:39 +0100
Subject: [PATCH 356/462] rust: fix zero_prefixed_literal

warning: this is a decimal constant
   --> src/mqtt/parser.rs:888:19
    |
888 |             0x00, 06, /* Topic Length: 6 */
    |                   ^^
    |
---
 rust/src/mqtt/parser.rs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/rust/src/mqtt/parser.rs b/rust/src/mqtt/parser.rs
index 9b576e54a5c0..eb1de176f44f 100644
--- a/rust/src/mqtt/parser.rs
+++ b/rust/src/mqtt/parser.rs
@@ -885,7 +885,7 @@ mod tests {
     #[test]
     fn test_parse_publish() {
         let buf = [
-            0x00, 06, /* Topic Length: 6 */
+            0x00, 0x06, /* Topic Length: 6 */
             0x74, 0x6f, 0x70, 0x69, 0x63, 0x58, /* Topic: topicX */
             0x00, 0x01, /* Message Identifier: 1 */
             0x00, /* Properties 6 */
@@ -914,7 +914,7 @@ mod tests {
     #[test]
     fn test_parse_msgidonly_v3() {
         let buf = [
-            0x00, 01, /* Message Identifier: 1 */
+            0x00, 0x01, /* Message Identifier: 1 */
             0x74, 0x6f, 0x70, 0x69, 0x63, 0x58, 0x00, 0x61, 0x75, 0x74, 0x6f, 0x2d, 0x42, 0x34,
             0x33, 0x45, 0x38, 0x30,
         ];
@@ -939,7 +939,7 @@ mod tests {
     #[test]
     fn test_parse_msgidonly_v5() {
         let buf = [
-            0x00, 01,   /* Message Identifier: 1 */
+            0x00, 0x01,   /* Message Identifier: 1 */
             0x00, /* Reason Code: 0 */
             0x00, /* Properties */
             0x00, 0x61, 0x75, 0x74, 0x6f, 0x2d, 0x42, 0x34, 0x33, 0x45, 0x38, 0x30,

From 9a84681bd9df7b033cf8dac79680677e673c746e Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 5 Jan 2024 10:27:34 +0100
Subject: [PATCH 357/462] rust: fix vec_init_then_push

warning: calls to `push` immediately after creation
    --> src/pgsql/parser.rs:1179:9
     |
1179 | /         let mut database_param: Vec<PgsqlParameter> = Vec::new();
1180 | |         database_param.push(database);
     | |______________________________________^
help: consider using the `vec![]` macro: `let database_param: Vec<PgsqlParameter> = vec![..];`
---
 rust/src/pgsql/parser.rs | 8 ++------
 rust/src/rdp/parser.rs   | 6 +-----
 2 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 97a16b57384f..792fb23a130b 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -1176,8 +1176,7 @@ mod tests {
             name: PgsqlParameters::Database,
             value: br#"mailstore"#.to_vec(),
         };
-        let mut database_param: Vec<PgsqlParameter> = Vec::new();
-        database_param.push(database);
+        let database_param: Vec<PgsqlParameter> = vec![database];
         let params = PgsqlStartupParameters {
             user,
             optional_params: Some(database_param),
@@ -2229,10 +2228,7 @@ mod tests {
             format_code: 0,
         };
 
-        let mut fields_vec = Vec::<RowField>::new();
-        fields_vec.push(field1);
-        fields_vec.push(field2);
-        fields_vec.push(field3);
+        let fields_vec = vec![field1, field2, field3];
 
         let ok_res = PgsqlBEMessage::RowDescription(RowDescriptionMessage {
             identifier: b'T',
diff --git a/rust/src/rdp/parser.rs b/rust/src/rdp/parser.rs
index a8004e290b96..cc9b9cc2429d 100644
--- a/rust/src/rdp/parser.rs
+++ b/rust/src/rdp/parser.rs
@@ -1197,11 +1197,7 @@ mod tests_core_49350 {
             typ: 0xc002,
             data: BYTES[0x16c..0x16c + 0x8].to_vec(),
         }));
-        let mut channels = Vec::new();
-        channels.push(String::from("rdpdr"));
-        channels.push(String::from("rdpsnd"));
-        channels.push(String::from("drdynvc"));
-        channels.push(String::from("cliprdr"));
+        let channels = vec![String::from("rdpdr"), String::from("rdpsnd"), String::from("drdynvc"), String::from("cliprdr")];
         children.push(McsConnectRequestChild::CsNet(CsNet { channels }));
         let t123_tpkt: T123Tpkt = T123Tpkt {
             child: T123TpktChild::Data(X223Data {

From b141eb9f1146bfe44bb9c4db136e27eee55e81a8 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 5 Jan 2024 10:30:33 +0100
Subject: [PATCH 358/462] rust: fix single_match

warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
   --> src/http2/parser.rs:882:17
    |
882 | /                 match ctx.value {
883 | |                     Some(_) => {
884 | |                         panic!("Unexpected value");
885 | |                     }
886 | |                     None => {}
887 | |                 }
    | |_________________^
---
 rust/src/http2/parser.rs |  7 +------
 rust/src/ssh/parser.rs   | 14 ++++----------
 2 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/rust/src/http2/parser.rs b/rust/src/http2/parser.rs
index adabeb28c6e4..64799460aa9a 100644
--- a/rust/src/http2/parser.rs
+++ b/rust/src/http2/parser.rs
@@ -879,12 +879,7 @@ mod tests {
         match r {
             Ok((rem, ctx)) => {
                 assert_eq!(ctx.id, HTTP2SettingsId::EnablePush);
-                match ctx.value {
-                    Some(_) => {
-                        panic!("Unexpected value");
-                    }
-                    None => {}
-                }
+                assert!(ctx.value.is_none());
                 assert_eq!(rem.len(), 0);
             }
             Err(e) => {
diff --git a/rust/src/ssh/parser.rs b/rust/src/ssh/parser.rs
index bfad8c005a9d..0fbd17a412e9 100644
--- a/rust/src/ssh/parser.rs
+++ b/rust/src/ssh/parser.rs
@@ -552,11 +552,8 @@ mod tests {
         ,0x00 ,0x00 ,0x00 ,0x00 ,0x00 ,0x00 ,0x00 ,0x00];
         let mut hassh_string: Vec<u8> = vec!();
         let mut hassh: Vec<u8> = vec!();
-        match ssh_parse_key_exchange(&client_key_exchange){
-            Ok((_, key_exchange)) => { 
-                key_exchange.generate_hassh(&mut hassh_string, &mut hassh, &true); 
-            }
-            Err(_) => { }
+        if let Ok((_, key_exchange)) = ssh_parse_key_exchange(&client_key_exchange){
+            key_exchange.generate_hassh(&mut hassh_string, &mut hassh, &true);
         }
 
         assert_eq!(hassh_string, "curve25519-sha256,curve25519-sha256@libssh.org,\
@@ -643,11 +640,8 @@ mod tests {
         0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00];
         let mut hassh_server_string: Vec<u8> = vec!();
         let mut hassh_server: Vec<u8> = vec!();
-        match ssh_parse_key_exchange(&server_key_exchange){
-            Ok((_, key_exchange)) => { 
-                key_exchange.generate_hassh(&mut hassh_server_string, &mut hassh_server, &true);
-            }
-            Err(_) => { }
+        if let Ok((_, key_exchange)) = ssh_parse_key_exchange(&server_key_exchange){
+            key_exchange.generate_hassh(&mut hassh_server_string, &mut hassh_server, &true);
         }
         assert_eq!(hassh_server, "b12d2871a1189eff20364cf5333619ee".as_bytes().to_vec());
     }

From 259cdf169e20b580643117b8fa227e470a0377f8 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Sat, 6 Jan 2024 22:15:11 +0100
Subject: [PATCH 359/462] rust: fix single_binding

error: this match could be written as a `let` statement
   --> src/nfs/nfs3_records.rs:747:9
    |
747 | /         match result {
748 | |             (r, request) => {
749 | |                 assert_eq!(r.len(), 0);
750 | |                 assert_eq!(request.handle, expected_handle);
751 | |                 assert_eq!(request.name_vec, br#"bln"#);
752 | |             }
753 | |         }
    | |_________^
---
 rust/src/nfs/nfs2_records.rs |  68 +++++-------
 rust/src/nfs/nfs3_records.rs | 200 +++++++++++++----------------------
 2 files changed, 96 insertions(+), 172 deletions(-)

diff --git a/rust/src/nfs/nfs2_records.rs b/rust/src/nfs/nfs2_records.rs
index d8fe84f49d8f..ceb55b992c69 100644
--- a/rust/src/nfs/nfs2_records.rs
+++ b/rust/src/nfs/nfs2_records.rs
@@ -113,13 +113,9 @@ mod tests {
             0x00, 0x00, 0xb2, 0x5a, 0x00, 0x00, 0x00, 0x29
         ];
 
-        let result = parse_nfs2_handle(buf).unwrap();
-        match result {
-            (r, res) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(res.value, buf);
-            }
-        }
+        let (r, res) = parse_nfs2_handle(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(res.value, buf);
     }
 
     #[test]
@@ -136,14 +132,10 @@ mod tests {
         let (_, handle) = parse_nfs2_handle(buf).unwrap();
         assert_eq!(handle.value, &buf[..32]);
 
-        let result = parse_nfs2_request_lookup(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, handle);
-                assert_eq!(request.name_vec, b"am".to_vec());
-            }
-        }
+        let (r, request) = parse_nfs2_request_lookup(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, handle);
+        assert_eq!(request.name_vec, b"am".to_vec());
     }
 
     #[test]
@@ -162,14 +154,10 @@ mod tests {
         let (_, handle) = parse_nfs2_handle(buf).unwrap();
         assert_eq!(handle.value, &buf[..32]);
 
-        let result = parse_nfs2_request_read(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 4);
-                assert_eq!(request.handle, handle);
-                assert_eq!(request.offset, 0);
-            }
-        }
+        let (r, request) = parse_nfs2_request_read(buf).unwrap();
+        assert_eq!(r.len(), 4);
+        assert_eq!(request.handle, handle);
+        assert_eq!(request.offset, 0);
     }
 
     #[test]
@@ -192,19 +180,15 @@ mod tests {
             0x00, /*_data_padding*/
         ];
 
-        let result = parse_nfs2_reply_read(buf).unwrap();
-        match result {
-            (r, response) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(response.status, 0);
-                assert_eq!(response.attr_follows, 1);
-                assert_eq!(response.attr_blob.len(), 68);
-                assert_eq!(response.count, response.data_len);
-                assert!(!response.eof);
-                assert_eq!(response.data_len, 11);
-                assert_eq!(response.data, &buf[76..87]);
-            }
-        }
+        let (r, response) = parse_nfs2_reply_read(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(response.status, 0);
+        assert_eq!(response.attr_follows, 1);
+        assert_eq!(response.attr_blob.len(), 68);
+        assert_eq!(response.count, response.data_len);
+        assert!(!response.eof);
+        assert_eq!(response.data_len, 11);
+        assert_eq!(response.data, &buf[76..87]);
     }
 
     #[test]
@@ -223,13 +207,9 @@ mod tests {
             0x00, 0x08, 0x16, 0x50
         ];
 
-        let result = parse_nfs2_attribs(buf).unwrap();
-        match result {
-            (r, res) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(res.atype, 1);
-                assert_eq!(res.asize, 0);
-            }
-        }
+        let (r, res) = parse_nfs2_attribs(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(res.atype, 1);
+        assert_eq!(res.asize, 0);
     }
 }
diff --git a/rust/src/nfs/nfs3_records.rs b/rust/src/nfs/nfs3_records.rs
index 952b367db827..daaa27e1ead7 100644
--- a/rust/src/nfs/nfs3_records.rs
+++ b/rust/src/nfs/nfs3_records.rs
@@ -480,17 +480,13 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_create(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.name_len, 1);
-                assert_eq!(request.create_mode, 0);
-                assert_eq!(request.verifier.len(), 44);
-                assert_eq!(request.name_vec, br#"h"#.to_vec());
-            }
-        }
+        let (r, request) = parse_nfs3_request_create(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.name_len, 1);
+        assert_eq!(request.create_mode, 0);
+        assert_eq!(request.verifier.len(), 44);
+        assert_eq!(request.name_vec, br#"h"#.to_vec());
     }
 
     #[test]
@@ -511,15 +507,11 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_remove(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.name_len, 1);
-                assert_eq!(request.name_vec, br#"h"#.to_vec());
-            }
-        }
+        let (r, request) = parse_nfs3_request_remove(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.name_len, 1);
+        assert_eq!(request.name_vec, br#"h"#.to_vec());
     }
 
     #[test]
@@ -540,14 +532,10 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_rmdir(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.name_vec, br#"d"#.to_vec());
-            }
-        }
+        let (r, request) = parse_nfs3_request_rmdir(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.name_vec, br#"d"#.to_vec());
     }
 
     #[test]
@@ -573,13 +561,9 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_mkdir(buf).unwrap();
-        match result {
-            (_r, request) => {
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.name_vec, br#"d"#.to_vec());
-            }
-        }
+        let (_r, request) = parse_nfs3_request_mkdir(buf).unwrap();
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.name_vec, br#"d"#.to_vec());
     }
 
     #[test]
@@ -611,18 +595,14 @@ mod tests {
         let (_, expected_from_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
         let (_, expected_to_handle) = parse_nfs3_handle(&buf[44..80]).unwrap();
 
-        let result = parse_nfs3_request_rename(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
+        let (r, request) = parse_nfs3_request_rename(buf).unwrap();
+        assert_eq!(r.len(), 0);
 
-                assert_eq!(request.from_handle, expected_from_handle);
-                assert_eq!(request.from_name_vec, br#"a"#.to_vec());
+        assert_eq!(request.from_handle, expected_from_handle);
+        assert_eq!(request.from_name_vec, br#"a"#.to_vec());
 
-                assert_eq!(request.to_handle, expected_to_handle);
-                assert_eq!(request.to_name_vec, br#"am"#.to_vec());
-            }
-        }
+        assert_eq!(request.to_handle, expected_to_handle);
+        assert_eq!(request.to_name_vec, br#"am"#.to_vec());
     }
 
     #[test]
@@ -638,13 +618,9 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(buf).unwrap();
 
-        let result = parse_nfs3_request_getattr(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-            }
-        }
+        let (r, request) = parse_nfs3_request_getattr(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
     }
 
     #[test]
@@ -662,14 +638,10 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_access(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.check_access, 12);
-            }
-        }
+        let (r, request) = parse_nfs3_request_access(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.check_access, 12);
     }
 
     #[test]
@@ -689,13 +661,9 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_commit(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-            }
-        }
+        let (r, request) = parse_nfs3_request_commit(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
     }
 
     #[test]
@@ -714,14 +682,10 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_read(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.offset, 0);
-            }
-        }
+        let (r, request) = parse_nfs3_request_read(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.offset, 0);
     }
 
     #[test]
@@ -743,14 +707,10 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_lookup(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.name_vec, br#"bln"#);
-            }
-        }
+        let (r, request) = parse_nfs3_request_lookup(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.name_vec, br#"bln"#);
     }
 
     #[test]
@@ -831,14 +791,10 @@ mod tests {
         let (_, entry0) = parse_nfs3_response_readdirplus_entry(entry0_buf).unwrap();
         let (_, entry1) = parse_nfs3_response_readdirplus_entry(entry1_buf).unwrap();
 
-        let response = many0_nfs3_response_readdirplus_entries(data_buf).unwrap();
-        match response {
-            (r, entries) => {
-                assert_eq!(r.len(), 4);
-                assert_eq!(entries[0], Nfs3ResponseReaddirplusEntry { entry: Some(entry0) });
-                assert_eq!(entries[1], Nfs3ResponseReaddirplusEntry { entry: Some(entry1) });
-            }
-        }
+        let (r, entries) = many0_nfs3_response_readdirplus_entries(data_buf).unwrap();
+        assert_eq!(r.len(), 4);
+        assert_eq!(entries[0], Nfs3ResponseReaddirplusEntry { entry: Some(entry0) });
+        assert_eq!(entries[1], Nfs3ResponseReaddirplusEntry { entry: Some(entry1) });
     }
 
     #[test]
@@ -904,18 +860,14 @@ mod tests {
         assert_eq!(expected_handle.len, 36);
         assert_eq!(expected_handle.value, &buf[4..40]);
 
-        let result = parse_nfs3_request_readdirplus(buf).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.cookie, 0);
-                assert_eq!(request.verifier, "\0\0\0\0\0\0\0\0".as_bytes());
-                assert_eq!(request.verifier.len(), 8);
-                assert_eq!(request.dircount, 512);
-                assert_eq!(request.maxcount, 4096);
-            }
-        }
+        let (r, request) = parse_nfs3_request_readdirplus(buf).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.cookie, 0);
+        assert_eq!(request.verifier, "\0\0\0\0\0\0\0\0".as_bytes());
+        assert_eq!(request.verifier.len(), 8);
+        assert_eq!(request.dircount, 512);
+        assert_eq!(request.maxcount, 4096);
     }
 
     #[test]
@@ -942,18 +894,14 @@ mod tests {
 
         let (_, expected_handle) = parse_nfs3_handle(&buf[..36]).unwrap();
 
-        let result = parse_nfs3_request_write(buf, true).unwrap();
-        match result {
-            (r, request) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(request.handle, expected_handle);
-                assert_eq!(request.offset, 0);
-                assert_eq!(request.count, 17);
-                assert_eq!(request.stable, 1);
-                assert_eq!(request.file_len, 17);
-                assert_eq!(request.file_data, "hallo\nthe b file\n".as_bytes());
-            }
-        }
+        let (r, request) = parse_nfs3_request_write(buf, true).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(request.handle, expected_handle);
+        assert_eq!(request.offset, 0);
+        assert_eq!(request.count, 17);
+        assert_eq!(request.stable, 1);
+        assert_eq!(request.file_len, 17);
+        assert_eq!(request.file_data, "hallo\nthe b file\n".as_bytes());
     }
 
     #[test]
@@ -983,18 +931,14 @@ mod tests {
             0x00, /*_data_padding*/
         ];
 
-        let result = parse_nfs3_reply_read(buf, true).unwrap();
-        match result {
-            (r, reply) => {
-                assert_eq!(r.len(), 0);
-                assert_eq!(reply.status, 0);
-                assert_eq!(reply.attr_follows, 1);
-                assert_eq!(reply.attr_blob.len(), 84);
-                assert_eq!(reply.count, 11);
-                assert!(reply.eof);
-                assert_eq!(reply.data_len, 11);
-                assert_eq!(reply.data, "the b file\n".as_bytes());
-            }
-        }
+        let (r, reply) = parse_nfs3_reply_read(buf, true).unwrap();
+        assert_eq!(r.len(), 0);
+        assert_eq!(reply.status, 0);
+        assert_eq!(reply.attr_follows, 1);
+        assert_eq!(reply.attr_blob.len(), 84);
+        assert_eq!(reply.count, 11);
+        assert!(reply.eof);
+        assert_eq!(reply.data_len, 11);
+        assert_eq!(reply.data, "the b file\n".as_bytes());
     }
 }

From bedd48596f16f375b7b77d031687f1225376c1e2 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Sat, 6 Jan 2024 22:16:24 +0100
Subject: [PATCH 360/462] ci: run clippy on test code as well

---
 .github/workflows/rust.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index b25c1bab5872..42c135657294 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -83,5 +83,5 @@ jobs:
               echo "::error ::Clippy --fix made changes, please fix"
               exit 1
           fi
-      - run: cargo clippy --all-features
+      - run: cargo clippy --all-features --all-targets
         working-directory: rust

From 6896a93d87c40719d34d79d2d39c1dd28fe17f05 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 5 Jan 2024 10:18:46 +0100
Subject: [PATCH 361/462] rust: update test_case crate

fixes unused_unit

warning: unneeded unit expression
   --> src/bittorrent_dht/parser.rs:590:5
    |
590 | /     #[test_case(
591 | |         b"",
592 | |         "Error: discovered Dict but expected EOF" ;
593 | |         "test parse bittorrent dht packet err 1"
594 | |     )]
    | |______^
---
 rust/Cargo.lock.in | 32 ++++++++++++++++++++++++++------
 rust/Cargo.toml.in |  2 +-
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/rust/Cargo.lock.in b/rust/Cargo.lock.in
index da0d6307e3b6..b3013aafe412 100644
--- a/rust/Cargo.lock.in
+++ b/rust/Cargo.lock.in
@@ -945,7 +945,7 @@ checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"
 
 [[package]]
 name = "suricata"
-version = "7.0.3-dev"
+version = "8.0.0-dev"
 dependencies = [
  "aes",
  "aes-gcm",
@@ -991,7 +991,7 @@ dependencies = [
 
 [[package]]
 name = "suricata-derive"
-version = "7.0.3-dev"
+version = "8.0.0-dev"
 dependencies = [
  "proc-macro-crate",
  "proc-macro2 1.0.69",
@@ -1046,15 +1046,35 @@ dependencies = [
 
 [[package]]
 name = "test-case"
-version = "1.1.0"
+version = "3.3.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "956044ef122917dde830c19dec5f76d0670329fde4104836d62ebcb14f4865f1"
+checksum = "eb2550dd13afcd286853192af8601920d959b14c401fcece38071d53bf0768a8"
+dependencies = [
+ "test-case-macros",
+]
+
+[[package]]
+name = "test-case-core"
+version = "3.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "adcb7fd841cd518e279be3d5a3eb0636409487998a4aff22f3de87b81e88384f"
 dependencies = [
  "cfg-if",
  "proc-macro2 1.0.69",
  "quote 1.0.33",
- "syn 1.0.109",
- "version_check",
+ "syn 2.0.39",
+]
+
+[[package]]
+name = "test-case-macros"
+version = "3.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c89e72a01ed4c579669add59014b9a524d609c0c88c6a585ce37485879f6ffb"
+dependencies = [
+ "proc-macro2 1.0.69",
+ "quote 1.0.33",
+ "syn 2.0.39",
+ "test-case-core",
 ]
 
 [[package]]
diff --git a/rust/Cargo.toml.in b/rust/Cargo.toml.in
index 0bac7e159e0a..1e135510e9e3 100644
--- a/rust/Cargo.toml.in
+++ b/rust/Cargo.toml.in
@@ -70,5 +70,5 @@ time = "=0.3.13"
 suricata-derive = { path = "./derive" }
 
 [dev-dependencies]
-test-case = "~1.1.0"
+test-case = "~3.3.1"
 hex = "~0.4.3"

From f80d26db0b62d4a547290a1cf536754f79927a3b Mon Sep 17 00:00:00 2001
From: Vincent Li <vincent.mc.li@gmail.com>
Date: Fri, 4 Aug 2023 17:49:35 +0000
Subject: [PATCH 362/462] flow-bypass: Set bypass thread to running state

When running Suricata in XDP bypass mode (bypass: yes),

Suricata started up with error:
Error: threads: thread "FB" failed to start in time: flags 0003

"FB" thread does not transition from THV_INIT_DONE to THV_RUNNING.

Set "FB" thread THV_RUNNING state in BypassedFlowManager().

Bug: #6254

Signed-off-by: Vincent Li <vincent.mc.li@gmail.com>
---
 src/flow-bypass.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/flow-bypass.c b/src/flow-bypass.c
index 8dbb5ab17d74..10ecf91f494e 100644
--- a/src/flow-bypass.c
+++ b/src/flow-bypass.c
@@ -93,7 +93,14 @@ static TmEcode BypassedFlowManager(ThreadVars *th_v, void *thread_data)
     if (!found)
         return TM_ECODE_OK;
 
+    TmThreadsSetFlag(th_v, THV_RUNNING);
+
     while (1) {
+        if (TmThreadsCheckFlag(th_v, THV_PAUSE)) {
+            TmThreadsSetFlag(th_v, THV_PAUSED);
+            TmThreadTestThreadUnPaused(th_v);
+            TmThreadsUnsetFlag(th_v, THV_PAUSED);
+        }
         SCLogDebug("Dumping the table");
         gettimeofday(&tv, NULL);
         TIMEVAL_TO_TIMESPEC(&tv, &curtime);

From 90ae3a223feff4f31d87ef1cad3e91262ed4f2ad Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 9 Jan 2024 11:40:48 -0600
Subject: [PATCH 363/462] eve/schema: allow authorities in dns.answers in alert

Factor out dns.authorities to a definition.
---
 etc/schema.json | 102 +++++++++++++++++++++++++-----------------------
 1 file changed, 54 insertions(+), 48 deletions(-)

diff --git a/etc/schema.json b/etc/schema.json
index 0756acd00800..b06dbd4b29fe 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -1040,53 +1040,7 @@
                     }
                 },
                 "authorities": {
-                    "type": "array",
-                    "minItems": 1,
-                    "items": {
-                        "type": "object",
-                        "properties": {
-                            "rdata": {
-                                "type": "string"
-                            },
-                            "rrname": {
-                                "type": "string"
-                            },
-                            "rrtype": {
-                                "type": "string"
-                            },
-                            "ttl": {
-                                "type": "integer"
-                            },
-                            "soa": {
-                                "type": "object",
-                                "properties": {
-                                    "expire": {
-                                        "type": "integer"
-                                    },
-                                    "minimum": {
-                                        "type": "integer"
-                                    },
-                                    "mname": {
-                                        "type": "string"
-                                    },
-                                    "refresh": {
-                                        "type": "integer"
-                                    },
-                                    "retry": {
-                                        "type": "integer"
-                                    },
-                                    "rname": {
-                                        "type": "string"
-                                    },
-                                    "serial": {
-                                        "type": "integer"
-                                    }
-                                },
-                                "additionalProperties": false
-                            }
-                        },
-                        "additionalProperties": false
-                    }
+                    "$ref": "#/$defs/dns.authorities"
                 },
                 "query": {
                     "type": "array",
@@ -1156,7 +1110,10 @@
                         "opcode": {
                             "description": "DNS opcode as an integer",
                             "type": "integer"
-                        }
+                        },
+                        "authorities": {
+                            "$ref": "#/$defs/dns.authorities"
+			}
                     },
                     "additionalProperties": false
                 },
@@ -5504,6 +5461,55 @@
         }
     },
     "$defs": {
+        "dns.authorities": {
+            "type": "array",
+            "minItems": 1,
+            "items": {
+                "type": "object",
+                "properties": {
+                    "rdata": {
+                        "type": "string"
+                    },
+                    "rrname": {
+                        "type": "string"
+                    },
+                    "rrtype": {
+                        "type": "string"
+                    },
+                    "ttl": {
+                        "type": "integer"
+                    },
+                    "soa": {
+                        "type": "object",
+                        "properties": {
+                            "expire": {
+                                "type": "integer"
+                            },
+                            "minimum": {
+                                "type": "integer"
+                            },
+                            "mname": {
+                                "type": "string"
+                            },
+                            "refresh": {
+                                "type": "integer"
+                            },
+                            "retry": {
+                                "type": "integer"
+                            },
+                            "rname": {
+                                "type": "string"
+                            },
+                            "serial": {
+                                "type": "integer"
+                            }
+                        },
+                        "additionalProperties": false
+                    }
+                },
+                "additionalProperties": false
+            }
+        },
         "stats_applayer_error": {
             "type": "object",
             "properties": {

From ff609f5dc369e29bef5670986addc8aba0cc1917 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 16 Jan 2024 08:30:25 -0500
Subject: [PATCH 364/462] conf/log: Remove sguil mode

Issue: 6347
---
 suricata.yaml.in | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/suricata.yaml.in b/suricata.yaml.in
index 630399126dbe..749c94359d4b 100644
--- a/suricata.yaml.in
+++ b/suricata.yaml.in
@@ -358,8 +358,8 @@ outputs:
       enabled: no
       #certs-log-dir: certs # directory to store the certificates files
 
-  # Packet log... log packets in pcap format. 3 modes of operation: "normal"
-  # "multi" and "sguil".
+  # Packet log... log packets in pcap format. 2 modes of operation: "normal"
+  # and "multi".
   #
   # In normal mode a pcap file "filename" is created in the default-log-dir,
   # or as specified by "dir".
@@ -379,11 +379,6 @@ outputs:
   # So the size limit when using 8 threads with 1000mb files and 2000 files
   # is: 8*1000*2000 ~ 16TiB.
   #
-  # In Sguil mode "dir" indicates the base directory. In this base dir the
-  # pcaps are created in the directory structure Sguil expects:
-  #
-  # $sguil-base-dir/YYYY-MM-DD/$filename.<timestamp>
-  #
   # By default all packets are logged except:
   # - TCP streams beyond stream.reassembly.depth
   # - encrypted streams after the key exchange
@@ -401,8 +396,7 @@ outputs:
       max-files: 2000
 
       # Compression algorithm for pcap files. Possible values: none, lz4.
-      # Enabling compression is incompatible with the sguil mode. Note also
-      # that on Windows, enabling compression will *increase* disk I/O.
+      # Note also that on Windows, enabling compression will *increase* disk I/O.
       compression: none
 
       # Further options for lz4 compression. The compression level can be set
@@ -411,10 +405,10 @@ outputs:
       #lz4-checksum: no
       #lz4-level: 0
 
-      mode: normal # normal, multi or sguil.
+      mode: normal # normal or multi
 
       # Directory to place pcap files. If not provided the default log
-      # directory will be used. Required for "sguil" mode.
+      # directory will be used.
       #dir: /nsm_data/
 
       #ts-format: usec # sec or usec second format (default) is filename.sec usec is filename.sec.usec

From 9101878e115e095821d3811c658cf4d7599d733f Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 16 Jan 2024 09:08:44 -0500
Subject: [PATCH 365/462] log/pcap: Remove sguil mode

Issue: 6347

Remove sguil-mode pcap logging capability.
---
 src/log-pcap.c | 112 ++++++-------------------------------------------
 1 file changed, 12 insertions(+), 100 deletions(-)

diff --git a/src/log-pcap.c b/src/log-pcap.c
index 6039e057b8a4..43b93b44defb 100644
--- a/src/log-pcap.c
+++ b/src/log-pcap.c
@@ -63,8 +63,7 @@
 #define DEFAULT_FILE_LIMIT              0
 
 #define LOGMODE_NORMAL                  0
-#define LOGMODE_SGUIL                   1
-#define LOGMODE_MULTI                   2
+#define LOGMODE_MULTI                   1
 
 typedef enum LogModeConditionalType_ {
     LOGMODE_COND_ALL,
@@ -145,7 +144,7 @@ typedef struct PcapLogData_ {
     uint64_t pkt_cnt;		    /**< total number of packets */
     struct pcap_pkthdr *h;      /**< pcap header struct */
     char *filename;             /**< current filename */
-    int mode;                   /**< normal or sguil */
+    int mode;                   /**< normal or multi */
     int prev_day;               /**< last day, for finding out when */
     uint64_t size_current;      /**< file current size */
     uint64_t size_limit;        /**< file size limit */
@@ -341,7 +340,6 @@ static void PcapFileNameFree(PcapFileName *pf)
 static int PcapLogRotateFile(ThreadVars *t, PcapLogData *pl)
 {
     PcapFileName *pf;
-    PcapFileName *pfnext;
 
     PCAPLOG_PROFILE_START;
 
@@ -360,25 +358,6 @@ static int PcapLogRotateFile(ThreadVars *t, PcapLogData *pl)
             //           pf->filename, strerror( errno ));
         }
 
-        /* Remove directory if Sguil mode and no files left in sguil dir */
-        if (pl->mode == LOGMODE_SGUIL) {
-            pfnext = TAILQ_NEXT(pf,next);
-
-            if (strcmp(pf->dirname, pfnext->dirname) == 0) {
-                SCLogDebug("Current entry dir %s and next entry %s "
-                        "are equal: not removing dir",
-                        pf->dirname, pfnext->dirname);
-            } else {
-                SCLogDebug("current entry %s and %s are "
-                        "not equal: removing dir",
-                        pf->dirname, pfnext->dirname);
-
-                if (remove(pf->dirname) != 0) {
-                    SCLogWarning("failed to remove sguil log %s: %s", pf->dirname, strerror(errno));
-                }
-            }
-        }
-
         TAILQ_REMOVE(&pl->pcap_file_list, pf, next);
         PcapFileNameFree(pf);
         pl->file_cnt--;
@@ -629,15 +608,6 @@ static int PcapLog (ThreadVars *t, void *thread_data, const Packet *p)
         SCLogDebug("Opening PCAP log file %s", pl->filename);
     }
 
-    if (pl->mode == LOGMODE_SGUIL) {
-        struct tm local_tm;
-        struct tm *tms = SCLocalTime(SCTIME_SECS(p->ts), &local_tm);
-        if (tms->tm_mday != pl->prev_day) {
-            rotate = 1;
-            pl->prev_day = tms->tm_mday;
-        }
-    }
-
     PcapLogCompressionData *comp = &pl->compression;
     if (comp->format == PCAP_LOG_COMPRESSION_FORMAT_NONE) {
         if ((pl->size_current + len) > pl->size_limit || rotate) {
@@ -1426,35 +1396,23 @@ static OutputInitResult PcapLogInitCtx(ConfNode *conf)
         const char *s_mode = NULL;
         s_mode = ConfNodeLookupChildValue(conf, "mode");
         if (s_mode != NULL) {
-            if (strcasecmp(s_mode, "sguil") == 0) {
-                pl->mode = LOGMODE_SGUIL;
-            } else if (strcasecmp(s_mode, "multi") == 0) {
+            if (strcasecmp(s_mode, "multi") == 0) {
                 pl->mode = LOGMODE_MULTI;
             } else if (strcasecmp(s_mode, "normal") != 0) {
-                SCLogError("log-pcap: invalid mode \"%s\". Valid options: \"normal\", "
-                           "\"sguil\", or \"multi\" mode ",
+                FatalError("log-pcap: invalid mode \"%s\". Valid options: \"normal\""
+                           "or \"multi\" mode ",
                         s_mode);
-                exit(EXIT_FAILURE);
             }
         }
 
         const char *s_dir = NULL;
         s_dir = ConfNodeLookupChildValue(conf, "dir");
         if (s_dir == NULL) {
-            s_dir = ConfNodeLookupChildValue(conf, "sguil-base-dir");
-        }
-        if (s_dir == NULL) {
-            if (pl->mode == LOGMODE_SGUIL) {
-                FatalError("log-pcap \"sguil\" mode requires \"sguil-base-dir\" "
-                           "option to be set.");
-            } else {
-                const char *log_dir = NULL;
-                log_dir = ConfigGetLogDirectory();
+            const char *log_dir = NULL;
+            log_dir = ConfigGetLogDirectory();
 
-                strlcpy(pl->dir,
-                    log_dir, sizeof(pl->dir));
-                    SCLogInfo("Using log dir %s", pl->dir);
-            }
+            strlcpy(pl->dir, log_dir, sizeof(pl->dir));
+            SCLogInfo("Using log dir %s", pl->dir);
         } else {
             if (PathIsAbsolute(s_dir)) {
                 strlcpy(pl->dir,
@@ -1469,10 +1427,9 @@ static OutputInitResult PcapLogInitCtx(ConfNode *conf)
 
             struct stat stat_buf;
             if (stat(pl->dir, &stat_buf) != 0) {
-                SCLogError("The sguil-base-dir directory \"%s\" "
+                FatalError("The dir directory \"%s\" "
                            "supplied doesn't exist. Shutting down the engine",
                         pl->dir);
-                exit(EXIT_FAILURE);
             }
             SCLogInfo("Using log dir %s", pl->dir);
         }
@@ -1491,13 +1448,6 @@ static OutputInitResult PcapLogInitCtx(ConfNode *conf)
             comp->pcap_buf_wrapper = NULL;
         } else if (strcmp(compression_str, "lz4") == 0) {
 #ifdef HAVE_LIBLZ4
-            if (pl->mode == LOGMODE_SGUIL) {
-                SCLogError("Compressed pcap "
-                           "logs are not possible in sguil mode");
-                SCFree(pl->h);
-                SCFree(pl);
-                return result;
-            }
             pl->compression.format = PCAP_LOG_COMPRESSION_FORMAT_LZ4;
 
             /* Use SCFmemopen so we can make pcap_dump write to a buffer. */
@@ -1606,8 +1556,7 @@ static OutputInitResult PcapLogInitCtx(ConfNode *conf)
     if (ParseFilename(pl, filename) != 0)
         exit(EXIT_FAILURE);
 
-    SCLogInfo("using %s logging", pl->mode == LOGMODE_SGUIL ?
-              "Sguil compatible" : (pl->mode == LOGMODE_MULTI ? "multi" : "normal"));
+    SCLogInfo("using %s logging", (pl->mode == LOGMODE_MULTI ? "multi" : "normal"));
 
     uint32_t max_file_limit = DEFAULT_FILE_LIMIT;
     if (conf != NULL) {
@@ -1741,44 +1690,7 @@ static int PcapLogOpenFileCtx(PcapLogData *pl)
         return -1;
     }
 
-    if (pl->mode == LOGMODE_SGUIL) {
-        struct tm local_tm;
-        struct tm *tms = SCLocalTime(SCTIME_SECS(ts), &local_tm);
-
-        char dirname[32], dirfull[PATH_MAX] = "";
-
-        snprintf(dirname, sizeof(dirname), "%04d-%02d-%02d",
-                tms->tm_year + 1900, tms->tm_mon + 1, tms->tm_mday);
-
-        /* create the filename to use */
-        int ret = snprintf(dirfull, sizeof(dirfull), "%s/%s", pl->dir, dirname);
-        if (ret < 0 || (size_t)ret >= sizeof(dirfull)) {
-            SCLogError("failed to construct path");
-            goto error;
-        }
-
-        /* if mkdir fails file open will fail, so deal with errors there */
-        (void)SCMkDir(dirfull, 0700);
-
-        if ((pf->dirname = SCStrdup(dirfull)) == NULL) {
-            SCLogError("Error allocating memory for "
-                       "directory name");
-            goto error;
-        }
-
-        int written;
-        if (pl->timestamp_format == TS_FORMAT_SEC) {
-            written = snprintf(filename, PATH_MAX, "%s/%s.%" PRIu32 "%s", dirfull, pl->prefix,
-                    (uint32_t)SCTIME_SECS(ts), pl->suffix);
-        } else {
-            written = snprintf(filename, PATH_MAX, "%s/%s.%" PRIu32 ".%" PRIu32 "%s", dirfull,
-                    pl->prefix, (uint32_t)SCTIME_SECS(ts), (uint32_t)SCTIME_USECS(ts), pl->suffix);
-        }
-        if (written == PATH_MAX) {
-            SCLogError("log-pcap path overflow");
-            goto error;
-        }
-    } else if (pl->mode == LOGMODE_NORMAL) {
+    if (pl->mode == LOGMODE_NORMAL) {
         int ret;
         /* create the filename to use */
         if (pl->timestamp_format == TS_FORMAT_SEC) {

From 58f882db943e18077aa7fb87719b6539f85eca60 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 16 Jan 2024 09:11:36 -0500
Subject: [PATCH 366/462] doc/pcap-log: Remove squil documentation

Issue: 6347
---
 doc/userguide/configuration/suricata-yaml.rst | 25 +++++--------------
 1 file changed, 6 insertions(+), 19 deletions(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index 0b39705d896b..6d85f874f2d3 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -457,8 +457,8 @@ look at all packets whenever you want.  In the normal mode a pcap file
 is created in the default-log-dir. It can also be created elsewhere if
 a absolute path is set in the yaml-file.
 
-The file that is saved in example the default -log-dir
-/var/log/suricata, can be be opened with every program which supports
+The file that is saved in example the ``default-log-dir``
+`/var/log/suricata`, can be be opened with every program which supports
 the pcap file format. This can be Wireshark, TCPdump, Suricata, Snort
 and many others.
 
@@ -466,25 +466,13 @@ The pcap-log option can be enabled and disabled.
 
 There is a size limit for the pcap-log file that can be set. The
 default limit is 32 MB. If the log-file reaches this limit, the file
-will be rotated and a new one will be created. The pcap-log option
-has an extra functionality for "Sguil":http://sguil.sourceforge.net/
-that can be enabled in the 'mode' option. In the sguil mode the
-"sguil_base_dir" indicates the base directory. In this base dir the
-pcaps are created in a Sguil-specific directory structure that is
-based on the day:
-
-::
-
-  $sguil_base_dir/YYYY-MM-DD/$filename.<timestamp>
-
-If you would like to use Suricata with Sguil, do not forget to enable
-(and if necessary modify) the base dir in the suricata.yaml file.
+will be rotated and a new one will be created.
 Remember that in the 'normal' mode, the file will be saved in
 default-log-dir or in the absolute path (if set).
 
 The pcap files can be compressed before being written to disk by setting
-the compression option to lz4. This option is incompatible with sguil
-mode. Note: On Windows, this option increases disk I/O instead of
+the compression option to lz4.
+Note: On Windows, this option increases disk I/O instead of
 reducing it. When using lz4 compression, you can enable checksums using
 the lz4-checksum option, and you can set the compression level lz4-level
 to a value between 0 and 16, where higher levels result in higher
@@ -514,8 +502,7 @@ the alert.
       # Limit in MB.
       limit: 32
 
-      mode: sguil # "normal" (default) or sguil.
-      sguil_base_dir: /nsm_data/
+      mode: normal # "normal" or multi
       conditional: alerts
 
 Verbose Alerts Log (alert-debug.log)

From 995f5fc8c5fa925e8ac1540f8f4bc2e68df3908a Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 16 Jan 2024 10:36:07 -0500
Subject: [PATCH 367/462] htp/swf: Remove flash deprecation notice

Issue: 6605

Flash decompression will remain so the deprecation notice is not needed.
---
 src/app-layer-htp.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index 5d48611812c1..1e8c0b8ea689 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -2905,8 +2905,6 @@ static void HTPConfigParseParameters(HTPCfgRec *cfg_prec, ConfNode *s,
                 if (strcasecmp("enabled", pval->name) == 0) {
                     if (ConfValIsTrue(pval->val)) {
                         cfg_prec->swf_decompression_enabled = 1;
-                        SCLogWarning("Flash decompression is deprecated and will be removed in "
-                                     "Suricata 8; see ticket #6179");
                     } else if (ConfValIsFalse(pval->val)) {
                         cfg_prec->swf_decompression_enabled = 0;
                     } else {

From 778820ba06d35ce1f736b3165f1ae7f4f498845b Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Tue, 16 Jan 2024 14:09:57 +0530
Subject: [PATCH 368/462] detect/alert: remove unnecessary else

---
 src/detect-engine-alert.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/src/detect-engine-alert.c b/src/detect-engine-alert.c
index f9cbed1564c5..f9f12f9b786e 100644
--- a/src/detect-engine-alert.c
+++ b/src/detect-engine-alert.c
@@ -319,8 +319,7 @@ static int AlertQueueSortHelper(const void *a, const void *b)
     const PacketAlert *pa1 = b;
     if (pa1->num == pa0->num)
         return pa0->tx_id < pa1->tx_id ? 1 : -1;
-    else
-        return pa0->num > pa1->num ? 1 : -1;
+    return pa0->num > pa1->num ? 1 : -1;
 }
 
 /** \internal

From 588af05565e2f52f5b1f758312e18c1567dbb993 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Tue, 16 Jan 2024 14:10:59 +0530
Subject: [PATCH 369/462] detect: remove unneeded size in DetectEngineCtx

sig_array_size can easily be calculated with length and is only used at
one place for debugging purposes. Remove it from the DetectEngineCtx
struct to avoid making it unnecessarily heavy.
---
 src/detect-engine-build.c | 4 ----
 src/detect.h              | 1 -
 2 files changed, 5 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 6a1c53113601..604164b21c15 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1361,14 +1361,10 @@ int SigPrepareStage1(DetectEngineCtx *de_ctx)
     }
 
     de_ctx->sig_array_len = DetectEngineGetMaxSigId(de_ctx);
-    de_ctx->sig_array_size = (de_ctx->sig_array_len * sizeof(Signature *));
     de_ctx->sig_array = (Signature **)SCCalloc(de_ctx->sig_array_len, sizeof(Signature *));
     if (de_ctx->sig_array == NULL)
         goto error;
 
-    SCLogDebug("signature lookup array: %" PRIu32 " sigs, %" PRIu32 " bytes",
-               de_ctx->sig_array_len, de_ctx->sig_array_size);
-
     /* now for every rule add the source group */
     for (Signature *s = de_ctx->sig_list; s != NULL; s = s->next) {
         de_ctx->sig_array[s->num] = s;
diff --git a/src/detect.h b/src/detect.h
index 181ae2292306..9b7b58794e6f 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -850,7 +850,6 @@ typedef struct DetectEngineCtx_ {
     SRepCIDRTree *srepCIDR_ctx;
 
     Signature **sig_array;
-    uint32_t sig_array_size; /* size in bytes */
     uint32_t sig_array_len;  /* size in array members */
 
     uint32_t signum;

From 26b81ca00733987f447a819de3f00a3ef52ba256 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Tue, 16 Jan 2024 14:11:34 +0530
Subject: [PATCH 370/462] detect: make SigMatch.is_last bool

It is used like bool so much so that nothing needs to be changed even
after changing its type.
---
 src/detect.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect.h b/src/detect.h
index 9b7b58794e6f..0fc5d21fb1d8 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -354,7 +354,7 @@ typedef struct SigMatch_ {
 /** \brief Data needed for Match() */
 typedef struct SigMatchData_ {
     uint16_t type;   /**< match type */
-    uint8_t is_last; /**< Last element of the list */
+    bool is_last;    /**< Last element of the list */
     SigMatchCtx *ctx; /**< plugin specific data */
 } SigMatchData;
 

From c28cc93e23e829dac34ca0170df754ab1063c482 Mon Sep 17 00:00:00 2001
From: Stephen Donnelly <stephen.donnelly@endace.com>
Date: Wed, 10 Jan 2024 16:39:34 +1300
Subject: [PATCH 371/462] source/erf-dag: compiler warnings

Bug: #6667.

Fix compiler warnings for function pointer parameters missing const with --enable-dag
---
 src/source-erf-dag.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/src/source-erf-dag.c b/src/source-erf-dag.c
index b7ea14d83239..ebac89e2d8b9 100644
--- a/src/source-erf-dag.c
+++ b/src/source-erf-dag.c
@@ -118,10 +118,10 @@ static inline TmEcode ProcessErfDagRecords(ErfDagThreadVars *ewtn, uint8_t *top,
     uint32_t *pkts_read);
 static inline TmEcode ProcessErfDagRecord(ErfDagThreadVars *ewtn, char *prec);
 TmEcode ReceiveErfDagLoop(ThreadVars *, void *data, void *slot);
-TmEcode ReceiveErfDagThreadInit(ThreadVars *, void *, void **);
+TmEcode ReceiveErfDagThreadInit(ThreadVars *, const void *, void **);
 void ReceiveErfDagThreadExitStats(ThreadVars *, void *);
 TmEcode ReceiveErfDagThreadDeinit(ThreadVars *, void *);
-TmEcode DecodeErfDagThreadInit(ThreadVars *, void *, void **);
+TmEcode DecodeErfDagThreadInit(ThreadVars *, const void *, void **);
 TmEcode DecodeErfDagThreadDeinit(ThreadVars *tv, void *data);
 TmEcode DecodeErfDag(ThreadVars *, Packet *, void *);
 void ReceiveErfDagCloseStream(int dagfd, int stream);
@@ -175,8 +175,7 @@ TmModuleDecodeErfDagRegister(void)
  * \param data      data pointer gets populated with
  *
  */
-TmEcode
-ReceiveErfDagThreadInit(ThreadVars *tv, void *initdata, void **data)
+TmEcode ReceiveErfDagThreadInit(ThreadVars *tv, const void *initdata, void **data)
 {
     SCEnter();
     int stream_count = 0;
@@ -196,14 +195,14 @@ ReceiveErfDagThreadInit(ThreadVars *tv, void *initdata, void **data)
      */
     if (dag_parse_name(initdata, ewtn->dagname, DAGNAME_BUFSIZE,
             &ewtn->dagstream) < 0) {
-        SCLogError("Failed to parse DAG interface: %s", (char *)initdata);
+        SCLogError("Failed to parse DAG interface: %s", (const char *)initdata);
         SCFree(ewtn);
         exit(EXIT_FAILURE);
     }
 
     ewtn->livedev = LiveGetDevice(initdata);
     if (ewtn->livedev == NULL) {
-        SCLogError("Unable to get %s live device", (char *)initdata);
+        SCLogError("Unable to get %s live device", (const char *)initdata);
         SCFree(ewtn);
         SCReturnInt(TM_ECODE_FAILED);
     }
@@ -612,8 +611,7 @@ DecodeErfDag(ThreadVars *tv, Packet *p, void *data)
     SCReturnInt(TM_ECODE_OK);
 }
 
-TmEcode
-DecodeErfDagThreadInit(ThreadVars *tv, void *initdata, void **data)
+TmEcode DecodeErfDagThreadInit(ThreadVars *tv, const void *initdata, void **data)
 {
     SCEnter();
     DecodeThreadVars *dtv = NULL;

From ca6f7c2d002cdad8eb3630860b5023560626aee0 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Mon, 30 Oct 2023 12:33:57 +0100
Subject: [PATCH 372/462] dpdk: rework hugepage hints to use per-numa
 information

Previous integration of hugepage analysis only fetched data
from /proc/meminfo. However this proved to be often
deceiving mainly for providing only global information and
not taking into account different hugepage sizes (e.g. 1GB
hugepages) and different NUMA nodes.

Ticket: #6419
---
 doc/userguide/capture-hardware/dpdk.rst |  51 +++
 src/Makefile.am                         |   2 +
 src/suricata.c                          |   8 +-
 src/util-dpdk.c                         |  99 ------
 src/util-dpdk.h                         |   1 -
 src/util-hugepages.c                    | 411 ++++++++++++++++++++++++
 src/util-hugepages.h                    |  53 +++
 7 files changed, 524 insertions(+), 101 deletions(-)
 create mode 100644 src/util-hugepages.c
 create mode 100644 src/util-hugepages.h

diff --git a/doc/userguide/capture-hardware/dpdk.rst b/doc/userguide/capture-hardware/dpdk.rst
index 91ae1c876ca9..1b9ecae763a4 100644
--- a/doc/userguide/capture-hardware/dpdk.rst
+++ b/doc/userguide/capture-hardware/dpdk.rst
@@ -15,6 +15,57 @@ learn more about the basic setup for DPDK.
 The following sections contain examples of how to set up DPDK and Suricata for
 more obscure use-cases.
 
+Hugepage analysis
+-----------------
+
+Suricata can analyse utilized hugepages on the system. This can be particularly 
+beneficial when there's a potential overallocation of hugepages. 
+The hugepage analysis is designed to examine the hugepages in use and 
+provide recommendations on an adequate number of hugepages. This then ensures 
+Suricata operates optimally while leaving sufficient memory for other 
+applications on the system. The analysis works by comparing snapshots of the
+hugepages before and after Suricata is initialized. After the initialization,
+no more hugepages are allocated by Suricata.
+The hugepage analysis can be seen in the Perf log level and is printed out 
+during the Suricata start. It is only printed when Suricata detects some 
+disrepancies in the system related to hugepage allocation.
+
+It's recommended to perform this analysis from a "clean" state - 
+that is a state when all your hugepages are free. It is especially recommended 
+when no other hugepage-dependent applications are running on your system.
+This can be checked in one of two ways:
+
+.. code-block:: 
+
+  # global check
+  cat /proc/meminfo
+
+  HugePages_Total:    1024
+  HugePages_Free:     1024
+
+  # per-numa check depends on NUMA node ID, hugepage size, 
+  # and nr_hugepages/free_hugepages - e.g.:
+  cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages
+
+After the termination of Suricata and other hugepage-related applications, 
+if the count of free hugepages is not equal with the total number of hugepages, 
+it indicates some hugepages were not freed completely.
+This can be fixed by removing DPDK-related files from the hugepage-mounted 
+directory (filesystem). 
+It's important to exercise caution while removing hugepages, especially when 
+other hugepage-dependent applications are in operation, as this action will 
+disrupt their memory functionality.
+Removing the DPDK files from the hugepage directory can often be done as:
+
+.. code-block:: bash
+
+  sudo rm -rf /dev/hugepages/rtemap_*
+
+  # To check where hugepages are mounted:
+  dpdk-hugepages.py -s
+  # or 
+  mount | grep huge
+
 Bond interface
 --------------
 
diff --git a/src/Makefile.am b/src/Makefile.am
index 2af8b1d44cd8..ed1a78c6f71b 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -555,6 +555,7 @@ noinst_HEADERS = \
 	util-hash-string.h \
 	util-host-info.h \
 	util-host-os-info.h \
+	util-hugepages.h \
 	util-hyperscan.h \
 	util-ioctl.h \
 	util-ip.h \
@@ -1157,6 +1158,7 @@ libsuricata_c_a_SOURCES = \
 	util-hash-string.c \
 	util-host-info.c \
 	util-host-os-info.c \
+	util-hugepages.c \
 	util-hyperscan.c \
 	util-ioctl.c \
 	util-ip.c \
diff --git a/src/suricata.c b/src/suricata.c
index 257cb7bc1df6..126d02f900f0 100644
--- a/src/suricata.c
+++ b/src/suricata.c
@@ -127,6 +127,7 @@
 #include "util-ebpf.h"
 #include "util-exception-policy.h"
 #include "util-host-os-info.h"
+#include "util-hugepages.h"
 #include "util-ioctl.h"
 #include "util-landlock.h"
 #include "util-luajit.h"
@@ -2973,6 +2974,7 @@ int SuricataMain(int argc, char **argv)
         goto out;
     }
 
+    SystemHugepageSnapshot *prerun_snap = SystemHugepageSnapshotCreate();
     SCSetStartTime(&suricata);
     RunModeDispatch(suricata.run_mode, suricata.runmode_custom_mode,
             suricata.capture_plugin_name, suricata.capture_plugin_args);
@@ -3031,7 +3033,11 @@ int SuricataMain(int argc, char **argv)
 
     PostRunStartedDetectSetup(&suricata);
 
-    DPDKEvaluateHugepages();
+    SystemHugepageSnapshot *postrun_snap = SystemHugepageSnapshotCreate();
+    if (run_mode == RUNMODE_DPDK) // only DPDK uses hpages at the moment
+        SystemHugepageEvaluateHugepages(prerun_snap, postrun_snap);
+    SystemHugepageSnapshotDestroy(prerun_snap);
+    SystemHugepageSnapshotDestroy(postrun_snap);
 
     SCPledge();
     SuricataMainLoop(&suricata);
diff --git a/src/util-dpdk.c b/src/util-dpdk.c
index 089aa45674ae..13329a81d13a 100644
--- a/src/util-dpdk.c
+++ b/src/util-dpdk.c
@@ -66,106 +66,7 @@ void DPDKFreeDevice(LiveDevice *ldev)
 #endif
 }
 
-static FILE *HugepagesMeminfoOpen(void)
-{
-    FILE *fp = fopen("/proc/meminfo", "r");
-    if (fp == NULL) {
-        SCLogInfo("Can't analyze hugepage usage: failed to open /proc/meminfo");
-    }
-    return fp;
-}
-
-static void HugepagesMeminfoClose(FILE *fp)
-{
-    if (fp) {
-        fclose(fp);
-    }
-}
-
-/**
- * Parsing values of meminfo
- *
- * \param fp Opened file pointer for reading of file /proc/meminfo at beginning
- * \param keyword Entry to look for e.g. "HugePages_Free:"
- * \return n Value of the entry
- * \return -1 On error
- *
- */
-static int32_t MemInfoParseValue(FILE *fp, const char *keyword)
-{
-    char path[256], value_str[64];
-    int32_t value = -1;
-
-    while (fscanf(fp, "%255s", path) != EOF) {
-        if (strcmp(path, keyword) == 0) {
-            if (fscanf(fp, "%63s", value_str) == EOF) {
-                SCLogDebug("%s: not followed by any number", keyword);
-                break;
-            }
-
-            if (StringParseInt32(&value, 10, 23, value_str) < 0) {
-                SCLogDebug("Failed to convert %s from /proc/meminfo", keyword);
-                value = -1;
-            }
-            break;
-        }
-    }
-    return value;
-}
-
-static void MemInfoEvaluateHugepages(FILE *fp)
-{
-    int32_t free_hugepages = MemInfoParseValue(fp, "HugePages_Free:");
-    if (free_hugepages < 0) {
-        SCLogInfo("HugePages_Free information not found in /proc/meminfo");
-        return;
-    }
-
-    rewind(fp);
-
-    int32_t total_hugepages = MemInfoParseValue(fp, "HugePages_Total:");
-    if (total_hugepages < 0) {
-        SCLogInfo("HugePages_Total information not found in /proc/meminfo");
-        return;
-    } else if (total_hugepages == 0) {
-        SCLogInfo("HugePages_Total equals to zero");
-        return;
-    }
-
-    float free_hugepages_ratio = (float)free_hugepages / (float)total_hugepages;
-    if (free_hugepages_ratio > 0.5) {
-        SCLogInfo("%" PRIu32 " of %" PRIu32
-                  " of hugepages are free - number of hugepages can be lowered to e.g. %.0lf",
-                free_hugepages, total_hugepages, ceil((total_hugepages - free_hugepages) * 1.15));
-    }
-}
-
-static void MemInfoWith(void (*callback)(FILE *))
-{
-    FILE *fp = HugepagesMeminfoOpen();
-    if (fp) {
-        callback(fp);
-        HugepagesMeminfoClose(fp);
-    }
-}
-
-void DPDKEvaluateHugepages(void)
-{
-    if (run_mode != RUNMODE_DPDK)
-        return;
-
-#ifdef HAVE_DPDK
-    if (rte_eal_has_hugepages() == 0) { // hugepages disabled
-        SCLogPerf("Hugepages not enabled - enabling hugepages can improve performance");
-        return;
-    }
-#endif
-
-    MemInfoWith(MemInfoEvaluateHugepages);
-}
-
 #ifdef HAVE_DPDK
-
 /**
  * Retrieves name of the port from port id
  * Not thread-safe
diff --git a/src/util-dpdk.h b/src/util-dpdk.h
index a94f46225217..1fb3532f5d4d 100644
--- a/src/util-dpdk.h
+++ b/src/util-dpdk.h
@@ -121,7 +121,6 @@ void DPDKCleanupEAL(void);
 
 void DPDKCloseDevice(LiveDevice *ldev);
 void DPDKFreeDevice(LiveDevice *ldev);
-void DPDKEvaluateHugepages(void);
 
 #ifdef HAVE_DPDK
 const char *DPDKGetPortNameByPortID(uint16_t pid);
diff --git a/src/util-hugepages.c b/src/util-hugepages.c
new file mode 100644
index 000000000000..2af74c3e4c6e
--- /dev/null
+++ b/src/util-hugepages.c
@@ -0,0 +1,411 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Lukas Sismis <lsismis@oisf.net>
+ */
+
+#include "suricata.h"
+#include "util-debug.h"
+#include "util-hugepages.h"
+
+static uint16_t SystemHugepageSizesCntPerNodeGet(uint16_t node_index);
+static uint16_t SystemNodeCountGet(void);
+static void SystemHugepagePerNodeGetHugepageSizes(
+        uint16_t node_index, uint16_t hp_sizes_cnt, uint32_t *hp_sizes);
+static HugepageInfo *SystemHugepageHugepageInfoCreate(uint16_t hp_size_cnt);
+static int16_t SystemHugepagePerNodeGetHugepageInfo(uint16_t node_index, NodeInfo *node);
+static void SystemHugepageHugepageInfoDestroy(HugepageInfo *h);
+static void SystemHugepageNodeInfoDestroy(NodeInfo *n);
+static void SystemHugepageNodeInfoDump(NodeInfo *n);
+static void SystemHugepageSnapshotDump(SystemHugepageSnapshot *s);
+
+static bool SystemHugepageSupported(void)
+{
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+    return true;
+#else
+    return false;
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+}
+
+// block of all hugepage-specific internal functions
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+
+/**
+ * \brief Linux-specific function to detect number of NUMA nodes on the system
+ * \returns number of NUMA nodes, 0 on error
+ */
+static uint16_t SystemNodeCountGetLinux(void)
+{
+    char dir_path[] = "/sys/devices/system/node/";
+    DIR *dir = opendir(dir_path);
+    if (dir == NULL) {
+        SCLogError("unable to open %s", dir_path);
+        return 0;
+    }
+
+    uint16_t count = 0;
+    struct dirent *entry;
+    while ((entry = readdir(dir)) != NULL) {
+        char d_name[] = "node";
+        if (entry->d_type == DT_DIR && strncmp(entry->d_name, d_name, strlen(d_name)) == 0)
+            count++;
+    }
+    closedir(dir);
+    return count;
+}
+
+/**
+ * \brief Linux-specific function to detect number of unique hugepage sizes
+ * \param[in] node_index index of the NUMA node
+ * \returns number of hugepage sizes, 0 on error
+ */
+static uint16_t SystemHugepageSizesCntPerNodeGetLinux(uint16_t node_index)
+{
+    char dir_path[256];
+    snprintf(dir_path, sizeof(dir_path), "/sys/devices/system/node/node%d/hugepages/", node_index);
+    DIR *dir = opendir(dir_path);
+    if (dir == NULL) {
+        SCLogError("unable to open %s", dir_path);
+        return 0;
+    }
+
+    uint16_t count = 0;
+    struct dirent *entry;
+    while ((entry = readdir(dir)) != NULL) {
+        char d_name[] = "hugepages-";
+        if (entry->d_type == DT_DIR && strncmp(entry->d_name, d_name, strlen(d_name)) == 0)
+            count++;
+    }
+    closedir(dir);
+    return count;
+}
+
+/**
+ * \brief Linux-specific function to detect unique hugepage sizes
+ * \note Arrays `hugepages` and `hp_sizes` are expected to have the same size
+ * \param[in] node_index index of the NUMA node
+ * \param[in] hp_sizes_cnt number of the unique hugepage sizes
+ * \param[out] hp_sizes a pointer to the array of hugepage sizes
+ */
+static void SystemHugepagePerNodeGetHugepageSizesLinux(
+        uint16_t node_index, uint16_t hp_sizes_cnt, uint32_t *hp_sizes)
+{
+    char dir_path[256];
+    snprintf(dir_path, sizeof(dir_path), "/sys/devices/system/node/node%d/hugepages/", node_index);
+    DIR *dir = opendir(dir_path);
+    if (dir == NULL) {
+        SCLogError("unable to open %s", dir_path);
+        return;
+    }
+    uint16_t index = 0;
+    struct dirent *entry;
+    while ((entry = readdir(dir)) != NULL) {
+        if (entry->d_type == DT_DIR && strncmp(entry->d_name, "hugepages-", 10) == 0) {
+            sscanf(entry->d_name, "hugepages-%ukB", &(hp_sizes[index]));
+            index++;
+        }
+    }
+    closedir(dir);
+}
+
+/**
+ * \brief Linux-specific function to detect number of unique hugepage sizes
+ * \note Arrays `hugepages` and `hp_sizes` are expected to have the same size
+ * \param[out] hugepages a pointer to the array of hugepage info structures
+ * \param[in] hp_sizes a pointer to the array of hugepage sizes
+ * \param[in] hp_sizes_cnt number of hugepage sizes
+ * \param[in] node_index index of the NUMA node
+ * \returns 0 on success, negative number on error
+ */
+static int16_t SystemHugepagePerNodeGetHugepageInfoLinux(
+        HugepageInfo *hugepages, uint32_t *hp_sizes, uint16_t hp_sizes_cnt, uint16_t node_index)
+{
+    for (int16_t i = 0; i < hp_sizes_cnt; i++) {
+        hugepages[i].size_kb = hp_sizes[i];
+        char path[256];
+        snprintf(path, sizeof(path),
+                "/sys/devices/system/node/node%hu/hugepages/hugepages-%ukB/nr_hugepages",
+                node_index, hp_sizes[i]);
+        FILE *f = fopen(path, "r");
+        if (!f) {
+            SCLogError("unable to open %s", path);
+            return -SC_EEXIST;
+        }
+        if (fscanf(f, "%hu", &hugepages[i].allocated) != 1) {
+            SCLogError("failed to read the total number of allocated hugepages (%ukB) on node %hu",
+                    hp_sizes[i], node_index);
+            fclose(f);
+            return -SC_EINVAL;
+        }
+        fclose(f);
+
+        snprintf(path, sizeof(path),
+                "/sys/devices/system/node/node%hu/hugepages/hugepages-%ukB/free_hugepages",
+                node_index, hp_sizes[i]);
+        f = fopen(path, "r");
+        if (!f) {
+            SCLogError("unable to open %s", path);
+            return -SC_EEXIST;
+        }
+        if (fscanf(f, "%hu", &hugepages[i].free) != 1) {
+            SCLogError("failed to read the total number of free hugepages (%ukB) on node %hu",
+                    hp_sizes[i], node_index);
+            fclose(f);
+            return -SC_EINVAL;
+        }
+        fclose(f);
+    }
+
+    return 0;
+}
+
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+
+/**
+ * \brief The function gathers information about hugepages on a given node
+ * \param[in] node_index index of the NUMA node
+ * \param[out] node a pointer to the structure to hold hugepage info
+ * \returns 0 on success, negative number on error
+ */
+static int16_t SystemHugepagePerNodeGetHugepageInfo(uint16_t node_index, NodeInfo *node)
+{
+    uint16_t hp_sizes_cnt = SystemHugepageSizesCntPerNodeGet(node_index);
+    if (hp_sizes_cnt == 0) {
+        SCLogError("hugepages not found for node %d", node_index);
+        return -SC_EEXIST;
+    }
+    uint32_t *hp_sizes = SCCalloc(hp_sizes_cnt, sizeof(*hp_sizes));
+    if (hp_sizes == NULL) {
+        FatalError("failed to allocate memory for hugepage info");
+    }
+    SystemHugepagePerNodeGetHugepageSizes(node_index, hp_sizes_cnt, hp_sizes);
+
+    node->hugepages = SystemHugepageHugepageInfoCreate(hp_sizes_cnt);
+    node->num_hugepage_sizes = hp_sizes_cnt;
+
+    int16_t ret = 0;
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+    ret = SystemHugepagePerNodeGetHugepageInfoLinux(
+            node->hugepages, hp_sizes, node->num_hugepage_sizes, node_index);
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+
+    SCFree(hp_sizes);
+    return ret;
+}
+
+/**
+ * \brief The function detects number of NUMA nodes on the system
+ * \returns 0 if detection is unsuccessful, otherwise number of detected nodes
+ */
+static uint16_t SystemNodeCountGet(void)
+{
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+    return SystemNodeCountGetLinux();
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+    return 0;
+}
+
+/**
+ * \brief The function detects the number of unique hugepage sizes
+ * \returns 0 if detection is unsuccessful, otherwise number of hugepage sizes
+ */
+static uint16_t SystemHugepageSizesCntPerNodeGet(uint16_t node_index)
+{
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+    return SystemHugepageSizesCntPerNodeGetLinux(node_index);
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+    return 0;
+}
+
+/**
+ * \brief The function fills an array with unique hugepage sizes
+ * \note Arrays `hugepages` and `hp_sizes` are expected to have the same size
+ * \param[in] node_index index of the NUMA node
+ * \param[in] hp_sizes_cnt number of hugepage sizes
+ * \param[out] hp_sizes a pointer to the array of hugepage sizes
+ */
+static void SystemHugepagePerNodeGetHugepageSizes(
+        uint16_t node_index, uint16_t hp_sizes_cnt, uint32_t *hp_sizes)
+{
+#if !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun
+    return SystemHugepagePerNodeGetHugepageSizesLinux(node_index, hp_sizes_cnt, hp_sizes);
+#endif /* !defined __CYGWIN__ && !defined OS_WIN32 && !defined __OpenBSD__ && !defined sun */
+}
+
+static HugepageInfo *SystemHugepageHugepageInfoCreate(uint16_t hp_size_cnt)
+{
+    HugepageInfo *h = SCCalloc(hp_size_cnt, sizeof(*h));
+    if (h == NULL) {
+        FatalError("failed to allocate hugepage info array");
+    }
+    return h;
+}
+
+static void SystemHugepageHugepageInfoDestroy(HugepageInfo *h)
+{
+    if (h != NULL)
+        SCFree(h);
+}
+
+static void SystemHugepageNodeInfoDestroy(NodeInfo *n)
+{
+    if (n == NULL)
+        return;
+
+    SystemHugepageHugepageInfoDestroy(n->hugepages);
+}
+
+static void SystemHugepageNodeInfoDump(NodeInfo *n)
+{
+    if (n == NULL)
+        return;
+
+    for (uint16_t i = 0; i < n->num_hugepage_sizes; i++) {
+        SCLogDebug("Hugepage size - %dkB - allocated: %d free: %d", n->hugepages[i].size_kb,
+                n->hugepages[i].allocated, n->hugepages[i].free);
+    }
+}
+
+/**
+ * \brief The function prints out the hugepage snapshot
+ * \param[in] s a pointer to the snapshot
+ */
+static void SystemHugepageSnapshotDump(SystemHugepageSnapshot *s)
+{
+    if (s == NULL)
+        return;
+
+    for (uint16_t i = 0; i < s->num_nodes; i++) {
+        SCLogDebug("NUMA Node %d", i);
+        SystemHugepageNodeInfoDump(&(s->nodes[i]));
+    }
+}
+
+void SystemHugepageSnapshotDestroy(SystemHugepageSnapshot *s)
+{
+    if (s == NULL)
+        return;
+
+    for (uint16_t i = 0; i < s->num_nodes; i++) {
+        SystemHugepageNodeInfoDestroy(&(s->nodes[i]));
+    }
+    SCFree(s->nodes);
+    SCFree(s);
+}
+
+/**
+ * \brief The function creates a snapshot of the system's hugepage usage
+ *        per NUMA node and per hugepage size.
+ *        The snapshot is used to evaluate the system's hugepage usage after
+ *        initialization of Suricata.
+ * \returns a pointer to the snapshot, NULL on error
+ */
+SystemHugepageSnapshot *SystemHugepageSnapshotCreate(void)
+{
+    if (!SystemHugepageSupported())
+        return NULL;
+
+    uint16_t node_cnt = SystemNodeCountGet();
+    if (node_cnt == 0) {
+        SCLogError("failed to obtain number of NUMA nodes in the system");
+        return NULL;
+    }
+    NodeInfo *nodes = SCCalloc(node_cnt, sizeof(*nodes));
+    if (nodes == NULL) {
+        FatalError("failed to allocate memory for NUMA node info");
+    }
+
+    SystemHugepageSnapshot *s = SCCalloc(1, sizeof(*s));
+    if (s == NULL) {
+        SCFree(nodes);
+        FatalError("failed to allocate memory for NUMA node snapshot");
+    }
+    s->num_nodes = node_cnt;
+    s->nodes = nodes;
+
+    for (uint16_t i = 0; i < s->num_nodes; i++) {
+        int16_t ret = SystemHugepagePerNodeGetHugepageInfo(i, &s->nodes[i]);
+        if (ret != 0) {
+            SystemHugepageSnapshotDestroy(s);
+            return NULL;
+        }
+    }
+
+    return s;
+}
+
+/**
+ * \brief The function compares two hugepage snapshots and prints out
+ *        recommendations for hugepage configuration
+ * \param[in] pre_s  a pointer to the snapshot taken before Suricata initialization
+ * \param[in] post_s a pointer to the snapshot taken after Suricata initialization
+ */
+void SystemHugepageEvaluateHugepages(SystemHugepageSnapshot *pre_s, SystemHugepageSnapshot *post_s)
+{
+    if (!SystemHugepageSupported() || pre_s == NULL || post_s == NULL)
+        return;
+
+    SCLogDebug("Hugepages before initialization");
+    SystemHugepageSnapshotDump(pre_s);
+
+    SCLogDebug("Hugepages after initialization");
+    SystemHugepageSnapshotDump(post_s);
+
+    if (pre_s->num_nodes != post_s->num_nodes)
+        FatalError("Number of NUMA nodes changed during hugepage evaluation");
+
+    for (int32_t i = 0; i < post_s->num_nodes; i++) {
+        if (pre_s->nodes[i].num_hugepage_sizes != post_s->nodes[i].num_hugepage_sizes)
+            FatalError("Number of NUMA node hugepage sizes changed during hugepage evaluation");
+
+        for (int32_t j = 0; j < post_s->nodes->num_hugepage_sizes; j++) {
+            HugepageInfo *prerun_hp = &pre_s->nodes[i].hugepages[j];
+            HugepageInfo *postrun_hp = &post_s->nodes[i].hugepages[j];
+
+            if (prerun_hp->free == 0) {
+                continue; // this HP size on this node has no HPs allocated
+            } else if (prerun_hp->free < postrun_hp->free) {
+                SCLogWarning(
+                        "Hugepage usage decreased while it should only increase/stay the same");
+            } else if (prerun_hp->free > 0 && prerun_hp->free == postrun_hp->free) {
+                SCLogPerf("Hugepages on NUMA node %u are unused and can be deallocated", i);
+            } else { // assumes this is an active NUMA node because at least some hugepages were
+                     // used
+                // speculative hint only for 2048kB pages as e.g. 1 GB pages can leave a lot of room
+                // for additional allocations
+                if (postrun_hp->size_kb == 2048 && postrun_hp->free == 0) {
+                    SCLogPerf("all %ukB hugepages used on NUMA node %d - consider increasing to "
+                              "prevent memory allocation from other NUMA nodes",
+                            postrun_hp->size_kb, i);
+                }
+
+                float free_hugepages_ratio = (float)postrun_hp->free / (float)prerun_hp->free;
+                if (free_hugepages_ratio > 0.5) {
+                    int32_t used_hps = prerun_hp->free - postrun_hp->free;
+                    SCLogPerf("Hugepages on NUMA node %u can be set to %.0lf (only using %u/%u "
+                              "%ukB hugepages)",
+                            i, ceil((prerun_hp->free - postrun_hp->free) * 1.15), used_hps,
+                            prerun_hp->free, postrun_hp->size_kb);
+                }
+            }
+        }
+    }
+}
diff --git a/src/util-hugepages.h b/src/util-hugepages.h
new file mode 100644
index 000000000000..8946eae6adec
--- /dev/null
+++ b/src/util-hugepages.h
@@ -0,0 +1,53 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Lukas Sismis <lsismis@oisf.net>
+ */
+
+#ifndef UTIL_HUGEPAGES_H
+#define UTIL_HUGEPAGES_H
+
+typedef struct {
+    uint32_t size_kb;
+    uint16_t allocated;
+    uint16_t free;
+} HugepageInfo;
+
+// Structure to hold information about individual NUMA nodes in the system and
+// and their respective allocated hugepages
+// So for e.g. NUMA node 0 there can be 2 hugepage_size - 2 MB and 1 GB
+// Each hugepage size will then have a record of number of allocated/free hpages
+typedef struct {
+    uint16_t num_hugepage_sizes;
+    HugepageInfo *hugepages;
+} NodeInfo;
+
+// Structure to hold information about all hugepage sizes residing on all NUMA
+// nodes in the system
+typedef struct {
+    uint16_t num_nodes;
+    NodeInfo *nodes;
+} SystemHugepageSnapshot;
+
+SystemHugepageSnapshot *SystemHugepageSnapshotCreate(void);
+void SystemHugepageSnapshotDestroy(SystemHugepageSnapshot *s);
+void SystemHugepageEvaluateHugepages(SystemHugepageSnapshot *pre_s, SystemHugepageSnapshot *post_s);
+
+#endif /* UTIL_HUGEPAGES_H */

From 2a2898053c120fa59cb792623b10c2660d8e8cde Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Wed, 27 Sep 2023 14:38:36 +0200
Subject: [PATCH 373/462] dpdk: add interrupt (power-saving) mode

When the packet load is low, Suricata can run in interrupt
mode. This more resembles the classic approach of processing
packets - CPU cores run low and only fetch packets
on interrupt.

Ticket: #5839
---
 doc/userguide/capture-hardware/dpdk.rst | 38 +++++++++++++
 src/runmode-dpdk.c                      | 27 +++++++++
 src/runmode-dpdk.h                      |  1 +
 src/source-dpdk.c                       | 73 ++++++++++++++++++++++++-
 src/source-dpdk.h                       |  1 +
 suricata.yaml.in                        |  1 +
 6 files changed, 140 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/capture-hardware/dpdk.rst b/doc/userguide/capture-hardware/dpdk.rst
index 1b9ecae763a4..6be7278b8cbb 100644
--- a/doc/userguide/capture-hardware/dpdk.rst
+++ b/doc/userguide/capture-hardware/dpdk.rst
@@ -146,3 +146,41 @@ management and worker CPU set.
         - worker-cpu-set:
             cpu: [ 2,4,6,8 ]
     ...
+
+Interrupt (power-saving) mode
+-----------------------------
+
+The DPDK is traditionally recognized for its polling mode operation. 
+In this mode, CPU cores are continuously querying for packets from 
+the Network Interface Card (NIC). While this approach offers benefits like 
+reduced latency and improved performance, it might not be the most efficient 
+in scenarios with sporadic or low traffic. 
+The constant polling can lead to unnecessary CPU consumption. 
+To address this, DPDK offers an `interrupt` mode.
+
+The obvious advantage that interrupt mode brings is power efficiency. 
+So far in our tests, we haven't observed a decrease in performance. Suricata's
+performance has actually seen a slight improvement.
+The (IPS runmode) users should be aware that interrupts can 
+introduce non-deterministic latency. However, the latency should never be 
+higher than in other (e.g. AF_PACKET/AF_XDP/...) capture methods. 
+
+Interrupt mode in DPDK can be configured on a per-interface basis. 
+This allows for a hybrid setup where some workers operate in polling mode, 
+while others utilize the interrupt mode. 
+The configuration for the interrupt mode can be found and modified in the 
+DPDK section of the suricata.yaml file.
+
+Below is a sample configuration that demonstrates how to enable the interrupt mode for a specific interface:
+
+::
+
+  ...
+  dpdk:
+      eal-params:
+        proc-type: primary
+
+      interfaces:
+        - interface: 0000:3b:00.0
+          interrupt-mode: true
+          threads: 4
diff --git a/src/runmode-dpdk.c b/src/runmode-dpdk.c
index 2cdf5cb32505..8a7643b250e6 100644
--- a/src/runmode-dpdk.c
+++ b/src/runmode-dpdk.c
@@ -111,6 +111,7 @@ static void *ParseDpdkConfigAndConfigureDevice(const char *iface);
 static void DPDKDerefConfig(void *conf);
 
 #define DPDK_CONFIG_DEFAULT_THREADS                     "auto"
+#define DPDK_CONFIG_DEFAULT_INTERRUPT_MODE              false
 #define DPDK_CONFIG_DEFAULT_MEMPOOL_SIZE                65535
 #define DPDK_CONFIG_DEFAULT_MEMPOOL_CACHE_SIZE          "auto"
 #define DPDK_CONFIG_DEFAULT_RX_DESCRIPTORS              1024
@@ -126,6 +127,7 @@ static void DPDKDerefConfig(void *conf);
 
 DPDKIfaceConfigAttributes dpdk_yaml = {
     .threads = "threads",
+    .irq_mode = "interrupt-mode",
     .promisc = "promisc",
     .multicast = "multicast",
     .checksum_checks = "checksum-checks",
@@ -434,6 +436,15 @@ static int ConfigSetThreads(DPDKIfaceConfig *iconf, const char *entry_str)
     SCReturnInt(0);
 }
 
+static bool ConfigSetInterruptMode(DPDKIfaceConfig *iconf, bool enable)
+{
+    SCEnter();
+    if (enable)
+        iconf->flags |= DPDK_IRQ_MODE;
+
+    SCReturnBool(true);
+}
+
 static int ConfigSetRxQueues(DPDKIfaceConfig *iconf, uint16_t nb_queues)
 {
     SCEnter();
@@ -695,6 +706,17 @@ static int ConfigLoad(DPDKIfaceConfig *iconf, const char *iface)
     if (retval < 0)
         SCReturnInt(retval);
 
+    bool irq_enable;
+    retval = ConfGetChildValueBoolWithDefault(if_root, if_default, dpdk_yaml.irq_mode, &entry_bool);
+    if (retval != 1) {
+        irq_enable = DPDK_CONFIG_DEFAULT_INTERRUPT_MODE;
+    } else {
+        irq_enable = entry_bool ? true : false;
+    }
+    retval = ConfigSetInterruptMode(iconf, irq_enable);
+    if (retval != true)
+        SCReturnInt(-EINVAL);
+
     // currently only mapping "1 thread == 1 RX (and 1 TX queue in IPS mode)" is supported
     retval = ConfigSetRxQueues(iconf, (uint16_t)iconf->threads);
     if (retval < 0)
@@ -1106,6 +1128,11 @@ static void DeviceInitPortConf(const DPDKIfaceConfig *iconf,
             },
     };
 
+    SCLogConfig("%s: interrupt mode is %s", iconf->iface,
+            iconf->flags & DPDK_IRQ_MODE ? "enabled" : "disabled");
+    if (iconf->flags & DPDK_IRQ_MODE)
+        port_conf->intr_conf.rxq = 1;
+
     // configure RX offloads
     if (dev_info->rx_offload_capa & RTE_ETH_RX_OFFLOAD_RSS_HASH) {
         if (iconf->nb_rx_queues > 1) {
diff --git a/src/runmode-dpdk.h b/src/runmode-dpdk.h
index a00327ba9e24..152c1d687893 100644
--- a/src/runmode-dpdk.h
+++ b/src/runmode-dpdk.h
@@ -25,6 +25,7 @@
 
 typedef struct DPDKIfaceConfigAttributes_ {
     const char *threads;
+    const char *irq_mode;
     const char *promisc;
     const char *multicast;
     const char *checksum_checks;
diff --git a/src/source-dpdk.c b/src/source-dpdk.c
index 54503e212271..480bb6c63676 100644
--- a/src/source-dpdk.c
+++ b/src/source-dpdk.c
@@ -93,6 +93,13 @@ TmEcode NoDPDKSupportExit(ThreadVars *tv, const void *initdata, void **data)
 
 #define BURST_SIZE 32
 static struct timeval machine_start_time = { 0, 0 };
+// interrupt mode constants
+#define MIN_ZERO_POLL_COUNT          10U
+#define MIN_ZERO_POLL_COUNT_TO_SLEEP 10U
+#define MINIMUM_SLEEP_TIME_US        1U
+#define STANDARD_SLEEP_TIME_US       100U
+#define MAX_EPOLL_TIMEOUT_MS         500U
+static rte_spinlock_t intr_lock[RTE_MAX_ETHPORTS];
 
 /**
  * \brief Structure to hold thread specific variables.
@@ -104,6 +111,7 @@ typedef struct DPDKThreadVars_ {
     TmSlot *slot;
     LiveDevice *livedev;
     ChecksumValidationMode checksum_mode;
+    bool intr_enabled;
     /* references to packet and drop counters */
     uint16_t capture_dpdk_packets;
     uint16_t capture_dpdk_rx_errs;
@@ -142,6 +150,40 @@ static uint64_t CyclesToSeconds(uint64_t cycles);
 static void DPDKFreeMbufArray(struct rte_mbuf **mbuf_array, uint16_t mbuf_cnt, uint16_t offset);
 static uint64_t DPDKGetSeconds(void);
 
+static bool InterruptsRXEnable(uint16_t port_id, uint16_t queue_id)
+{
+    uint32_t event_data = port_id << UINT16_WIDTH | queue_id;
+    int32_t ret = rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD,
+            RTE_INTR_EVENT_ADD, (void *)((uintptr_t)event_data));
+
+    if (ret != 0) {
+        SCLogError("%s-Q%d: failed to enable interrupt mode: %s", DPDKGetPortNameByPortID(port_id),
+                queue_id, rte_strerror(-ret));
+        return false;
+    }
+    return true;
+}
+
+static inline uint32_t InterruptsSleepHeuristic(uint32_t no_pkt_polls_count)
+{
+    if (no_pkt_polls_count < MIN_ZERO_POLL_COUNT_TO_SLEEP)
+        return MINIMUM_SLEEP_TIME_US;
+
+    return STANDARD_SLEEP_TIME_US;
+}
+
+static inline void InterruptsTurnOnOff(uint16_t port_id, uint16_t queue_id, bool on)
+{
+    rte_spinlock_lock(&(intr_lock[port_id]));
+
+    if (on)
+        rte_eth_dev_rx_intr_enable(port_id, queue_id);
+    else
+        rte_eth_dev_rx_intr_disable(port_id, queue_id);
+
+    rte_spinlock_unlock(&(intr_lock[port_id]));
+}
+
 static void DPDKFreeMbufArray(struct rte_mbuf **mbuf_array, uint16_t mbuf_cnt, uint16_t offset)
 {
     for (int i = offset; i < mbuf_cnt; i++) {
@@ -377,6 +419,11 @@ static TmEcode ReceiveDPDKLoop(ThreadVars *tv, void *data, void *slot)
 
     rte_eth_stats_reset(ptv->port_id);
     rte_eth_xstats_reset(ptv->port_id);
+
+    uint32_t pwd_zero_rx_packet_polls_count = 0;
+    if (ptv->intr_enabled && !InterruptsRXEnable(ptv->port_id, ptv->queue_id))
+        SCReturnInt(TM_ECODE_FAILED);
+
     while (1) {
         if (unlikely(suricata_ctl_flags != 0)) {
             SCLogDebug("Stopping Suricata!");
@@ -398,7 +445,27 @@ static TmEcode ReceiveDPDKLoop(ThreadVars *tv, void *data, void *slot)
                 TmThreadsCaptureHandleTimeout(tv, NULL);
                 last_timeout_msec = msecs;
             }
-            continue;
+
+            if (!ptv->intr_enabled)
+                continue;
+
+            pwd_zero_rx_packet_polls_count++;
+            if (pwd_zero_rx_packet_polls_count <= MIN_ZERO_POLL_COUNT)
+                continue;
+
+            uint32_t pwd_idle_hint = InterruptsSleepHeuristic(pwd_zero_rx_packet_polls_count);
+
+            if (pwd_idle_hint < STANDARD_SLEEP_TIME_US) {
+                rte_delay_us(pwd_idle_hint);
+            } else {
+                InterruptsTurnOnOff(ptv->port_id, ptv->queue_id, true);
+                struct rte_epoll_event event;
+                rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, 1, MAX_EPOLL_TIMEOUT_MS);
+                InterruptsTurnOnOff(ptv->port_id, ptv->queue_id, false);
+                continue;
+            }
+        } else if (ptv->intr_enabled && pwd_zero_rx_packet_polls_count) {
+            pwd_zero_rx_packet_polls_count = 0;
         }
 
         ptv->pkts += (uint64_t)nb_rx;
@@ -522,6 +589,7 @@ static TmEcode ReceiveDPDKThreadInit(ThreadVars *tv, const void *initdata, void
     ptv->checksum_mode = dpdk_config->checksum_mode;
 
     ptv->threads = dpdk_config->threads;
+    ptv->intr_enabled = (dpdk_config->flags & DPDK_IRQ_MODE) ? true : false;
     ptv->port_id = dpdk_config->port_id;
     ptv->out_port_id = dpdk_config->out_port_id;
     ptv->port_socket_id = dpdk_config->socket_id;
@@ -569,6 +637,9 @@ static TmEcode ReceiveDPDKThreadInit(ThreadVars *tv, const void *initdata, void
                     "%s: unable to determine NIC's NUMA node, degraded performance can be expected",
                     dpdk_config->iface);
         }
+        if (ptv->intr_enabled) {
+            rte_spinlock_init(&intr_lock[ptv->port_id]);
+        }
     }
 
     *data = (void *)ptv;
diff --git a/src/source-dpdk.h b/src/source-dpdk.h
index 3fdb63cb35d9..b962d866d4bd 100644
--- a/src/source-dpdk.h
+++ b/src/source-dpdk.h
@@ -38,6 +38,7 @@ typedef enum { DPDK_COPY_MODE_NONE, DPDK_COPY_MODE_TAP, DPDK_COPY_MODE_IPS } Dpd
 // General flags
 #define DPDK_PROMISC   (1 << 0) /**< Promiscuous mode */
 #define DPDK_MULTICAST (1 << 1) /**< Enable multicast packets */
+#define DPDK_IRQ_MODE  (1 << 2) /**< Interrupt mode */
 // Offloads
 #define DPDK_RX_CHECKSUM_OFFLOAD (1 << 4) /**< Enable chsum offload */
 
diff --git a/suricata.yaml.in b/suricata.yaml.in
index 749c94359d4b..412ab1aea850 100644
--- a/suricata.yaml.in
+++ b/suricata.yaml.in
@@ -747,6 +747,7 @@ dpdk:
       # - auto takes all cores
       # in IPS mode it is required to specify the number of cores and the numbers on both interfaces must match
       threads: auto
+      # interrupt-mode: false # true to switch to interrupt mode 
       promisc: true # promiscuous mode - capture all packets
       multicast: true # enables also detection on multicast packets
       checksum-checks: true # if Suricata should validate checksums

From 6e4cc79b398d91e243bd1574cbbcbbf7cd021c5d Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Mon, 11 Dec 2023 01:47:55 +0100
Subject: [PATCH 374/462] doc: remove references to prehistoric versions

Remove references that are mentioning Suricata 3 or less
As a note - only one Suricata 4 reference found:
(suricata-yaml.rst:"In 4.1.x")
Fast pattern selection criteria can be internally found by inspecting
SupportFastPatternForSigMatchList and SigTableSetup functions.

Ticket: #6570
---
 doc/userguide/configuration/suricata-yaml.rst | 17 +++--
 doc/userguide/output/custom-http-logging.rst  |  2 -
 .../rules/fast-pattern-explained.rst          | 74 ++-----------------
 3 files changed, 17 insertions(+), 76 deletions(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index 6d85f874f2d3..ba103a1f38de 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -643,9 +643,9 @@ For setting the option sgh-mpm-context, you can choose from auto, full
 or single. The default setting is 'auto', meaning Suricata selects
 full or single based on the algorithm you use. 'Full' means that every
 group has its own MPM-context, and 'single' that all groups share one
-MPM-context. The two algorithms ac and ac-gfbs are new in 1.03. These
-algorithms use a single MPM-context if the Sgh-MPM-context setting is
-'auto'. The rest of the algorithms use full in that case.
+MPM-context. The algorithm "ac" uses a single MPM-context if the 
+Sgh-MPM-context setting is 'auto'. The rest of the algorithms use full 
+in that case.
 
 The inspection-recursion-limit option has to mitigate that possible
 bugs in Suricata cause big problems. Often Suricata has to deal with
@@ -1287,7 +1287,7 @@ the default behavior).
 
 Each supported protocol has a dedicated subsection under ``protocols``.
 
-Asn1_max_frames (new in 1.0.3 and 1.1)
+Asn1_max_frames
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Asn1 (`Abstract Syntax One
@@ -1860,14 +1860,15 @@ Default Log Format
 ~~~~~~~~~~~~~~~~~~
 
 A logging line exists of two parts. First it displays meta information
-(thread id, date etc.), and finally the actual log message. Example:
+(Log-level, Suricata module), and finally the actual log message. Example:
 
 ::
 
-  [27708] 15/10/2010 -- 11:40:07 - (suricata.c:425) <Info> (main) – This is Suricata version 1.0.2
+  i: suricata: This is Suricata version 7.0.2 RELEASE running in USER mode
 
-(Here the part until the – is the meta info, "This is Suricata 1.0.2"
-is the actual message.)
+(Here the part until the second `:` is the meta info, 
+"This is Suricata version 7.0.2 RELEASE running in USER mode" is the actual 
+message.)
 
 It is possible to determine which information will be displayed in
 this line and (the manner how it will be displayed) in which format it
diff --git a/doc/userguide/output/custom-http-logging.rst b/doc/userguide/output/custom-http-logging.rst
index f7d21adac5d8..e4ab4076436d 100644
--- a/doc/userguide/output/custom-http-logging.rst
+++ b/doc/userguide/output/custom-http-logging.rst
@@ -1,8 +1,6 @@
 Custom http logging
 ===================
 
-As of Suricata 1.3.1 you can enable a custom http logging option.
-
 In your Suricata.yaml, find the http-log section and edit as follows:
 
 
diff --git a/doc/userguide/rules/fast-pattern-explained.rst b/doc/userguide/rules/fast-pattern-explained.rst
index 5ee45e3e293c..88f0f3b33173 100644
--- a/doc/userguide/rules/fast-pattern-explained.rst
+++ b/doc/userguide/rules/fast-pattern-explained.rst
@@ -17,25 +17,23 @@ The fast_pattern selection criteria are as follows:
 
 #. Suricata first identifies all content matches that have the highest
    "priority" that are used in the signature.  The priority is based
-   off of the buffer being matched on and generally 'http_*' buffers
-   have a higher priority (lower number is higher priority).  See
-   :ref:`Appendix B <fast-pattern-explained-appendix-b>` for details
-   on which buffers have what priority.
+   off of the buffer being matched on and generally application layer buffers
+   have a higher priority (lower number is higher priority). The buffer
+   `http_method` is an exception and has lower priority than the general 
+   `content` buffer.
 #. Within the content matches identified in step 1 (the highest
    priority content matches), the longest (in terms of character/byte
    length) content match is used as the fast pattern match.
 #. If multiple content matches have the same highest priority and
    qualify for the longest length, the one with the highest
    character/byte diversity score ("Pattern Strength") is used as the
-   fast pattern match.  See :ref:`Appendix C
-   <fast-pattern-explained-appendix-c>` for details on the algorithm
+   fast pattern match.  See :ref:`Appendix A
+   <fast-pattern-explained-appendix-a>` for details on the algorithm
    used to determine Pattern Strength.
 #. If multiple content matches have the same highest priority, qualify
    for the longest length, and the same highest Pattern Strength, the
    buffer ("list_id") that was *registered last* is used as the fast
-   pattern match.  See :ref:`Appendix B
-   <fast-pattern-explained-appendix-b>` for the registration order of
-   the different buffers/lists.
+   pattern match.
 #. If multiple content matches have the same highest priority, qualify
    for the longest length, the same highest Pattern Strength, and have
    the same list_id (i.e. are looking in the same buffer), then the
@@ -52,63 +50,7 @@ Appendices
 
 .. _fast-pattern-explained-appendix-a:
 
-Appendix A - Buffers, list_id values, and Registration Order for Suricata 1.3.4
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This should be pretty much the same for Suricata 1.1.x - 1.4.x.
-
-======= ============================== ======================== ==================
-list_id Content Modifier Keyword       Buffer Name              Registration Order
-======= ============================== ======================== ==================
-1       <none> (regular content match) DETECT_SM_LIST_PMATCH    1 (first)
-2       http_uri                       DETECT_SM_LIST_UMATCH    2
-6       http_client_body               DETECT_SM_LIST_HCBDMATCH 3
-7       http_server_body               DETECT_SM_LIST_HSBDMATCH 4
-8       http_header                    DETECT_SM_LIST_HHDMATCH  5
-9       http_raw_header                DETECT_SM_LIST_HRHDMATCH 6
-10      http_method                    DETECT_SM_LIST_HMDMATCH  7
-11      http_cookie                    DETECT_SM_LIST_HCDMATCH  8
-12      http_raw_uri                   DETECT_SM_LIST_HRUDMATCH 9
-13      http_stat_msg                  DETECT_SM_LIST_HSMDMATCH 10
-14      http_stat_code                 DETECT_SM_LIST_HSCDMATCH 11
-15      http_user_agent                DETECT_SM_LIST_HUADMATCH 12 (last)
-======= ============================== ======================== ==================
-
-Note: registration order doesn't matter when it comes to determining the fast pattern match for Suricata 1.3.4 but list_id value does.
-
-.. _fast-pattern-explained-appendix-b:
-
-Appendix B - Buffers, list_id values, Priorities, and Registration Order for Suricata 2.0.7
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This should be pretty much the same for Suricata 2.0.x.
-
-========================================== ================== ============================== ============================= =======
-Priority (lower number is higher priority) Registration Order Content Modifier Keyword       Buffer Name                   list_id
-========================================== ================== ============================== ============================= =======
-3                                          11                 <none> (regular content match) DETECT_SM_LIST_PMATCH         1
-3                                          12                 http_method                    DETECT_SM_LIST_HMDMATCH       12
-3                                          13                 http_stat_code                 DETECT_SM_LIST_HSCDMATCH      9
-3                                          14                 http_stat_msg                  DETECT_SM_LIST_HSMDMATCH      8
-2                                          1 (first)          http_client_body               DETECT_SM_LIST_HCBDMATCH      4
-2                                          2                  http_server_body               DETECT_SM_LIST_HSBDMATCH      5
-2                                          3                  http_header                    DETECT_SM_LIST_HHDMATCH       6
-2                                          4                  http_raw_header                DETECT_SM_LIST_HRHDMATCH      7
-2                                          5                  http_uri                       DETECT_SM_LIST_UMATCH         2
-2                                          6                  http_raw_uri                   DETECT_SM_LIST_HRUDMATCH      3
-2                                          7                  http_host                      DETECT_SM_LIST_HHHDMATCH      10
-2                                          8                  http_raw_host                  DETECT_SM_LIST_HRHHDMATCH     11
-2                                          9                  http_cookie                    DETECT_SM_LIST_HCDMATCH       13
-2                                          10                 http_user_agent                DETECT_SM_LIST_HUADMATCH      14
-2                                          15 (last)          dns_query                      DETECT_SM_LIST_DNSQUERY_MATCH 20
-========================================== ================== ============================== ============================= =======
-
-Note: list_id value doesn't matter when it comes to determining the
-fast pattern match for Suricata 2.0.7 but registration order does.
-
-.. _fast-pattern-explained-appendix-c:
-
-Appendix C - Pattern Strength Algorithm
+Appendix A - Pattern Strength Algorithm
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 From detect-engine-mpm.c. Basically the Pattern Strength "score"

From df6444822eb7c094498d3986052ab1e33f4c8983 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Thu, 18 Jan 2024 17:24:33 -0300
Subject: [PATCH 375/462] userguide: clarify midstream exception policy

The description of behavior when midstream is enabled and exception
policy is set to ignore wasn't descriptive enough.

Fix typos.
---
 doc/userguide/configuration/exception-policies.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/userguide/configuration/exception-policies.rst b/doc/userguide/configuration/exception-policies.rst
index 5944f529428a..aae2acff5d46 100644
--- a/doc/userguide/configuration/exception-policies.rst
+++ b/doc/userguide/configuration/exception-policies.rst
@@ -46,7 +46,7 @@ Auto
 ''''
 
 **In IPS mode**, the default behavior for most of the exception policies is to
-fail close. This means droping the flow, or the packet, when the flow action is
+fail close. This means dropping the flow, or the packet, when the flow action is
 not supported. The default policy for the midstream exception will be ignore if
 midstream flows are accepted.
 
@@ -131,7 +131,7 @@ midstream pick-ups enabled or not and the various exception policy values:
      - Midstream pick-up sessions ENABLED (stream.midstream=true)
      - Midstream pick-up sessions DISABLED (stream.midstream=false)
    * - Ignore
-     - Session tracket and parsed.
+     - Session tracked and parsed, inspect and log app-layer traffic, do detection.
      - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
    * - Drop-flow
      - Not valid.*
@@ -169,7 +169,7 @@ whole flow.
      - Midstream pick-up sessions ENABLED (stream.midstream=true)
      - Midstream pick-up sessions DISABLED (stream.midstream=false)
    * - Ignore
-     - Session tracket and parsed.
+     - Session tracked and parsed, inspect and log app-layer traffic, do detection.
      - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
    * - Drop-flow
      - Not valid.*

From 415722dab26127da1d0c019ececb5edb63e194db Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eloy=20P=C3=A9rez=20Gonz=C3=A1lez?=
 <zer1t0ps@protonmail.com>
Date: Thu, 28 Apr 2022 12:57:51 +0200
Subject: [PATCH 376/462] smb: add smb.version keyword

Ticket: #5075

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 rust/src/smb/detect.rs       |  70 ++++++++++++++++
 src/Makefile.am              |   2 +
 src/detect-engine-register.c |   3 +
 src/detect-engine-register.h |   1 +
 src/detect-smb-version.c     | 155 +++++++++++++++++++++++++++++++++++
 src/detect-smb-version.h     |  25 ++++++
 6 files changed, 256 insertions(+)
 create mode 100644 src/detect-smb-version.c
 create mode 100644 src/detect-smb-version.h

diff --git a/rust/src/smb/detect.rs b/rust/src/smb/detect.rs
index c85a6f59ce33..3445274bbcaf 100644
--- a/rust/src/smb/detect.rs
+++ b/rust/src/smb/detect.rs
@@ -21,6 +21,9 @@ use crate::smb::smb::*;
 use crate::dcerpc::detect::{DCEIfaceData, DCEOpnumData, DETECT_DCE_OPNUM_RANGE_UNINITIALIZED};
 use crate::dcerpc::dcerpc::DCERPC_TYPE_REQUEST;
 use crate::detect::uint::detect_match_uint;
+use std::ffi::CStr;
+use std::os::raw::{c_char, c_void};
+use crate::smb::smb::SMBTransaction;
 
 #[no_mangle]
 pub unsafe extern "C" fn rs_smb_tx_get_share(tx: &mut SMBTransaction,
@@ -192,3 +195,70 @@ pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_domain(tx: &mut SMBTransaction,
     *buffer_len = 0;
     return 0;
 }
+
+#[no_mangle]
+pub unsafe extern "C" fn rs_smb_version_match(
+    tx: &mut SMBTransaction, version_data: &mut u8,
+) -> u8 {
+
+    let version = tx.vercmd.get_version();
+    if version == *version_data {
+        return 1;
+    }
+
+    return 0;
+}
+
+
+#[no_mangle]
+pub unsafe extern "C" fn rs_smb_version_parse(carg: *const c_char) -> *mut c_void {
+    if carg.is_null() {
+        return std::ptr::null_mut();
+    }
+
+    if let Ok(arg) = CStr::from_ptr(carg).to_str() {
+        if let Ok(detect) = parse_version_data(arg) {
+            return Box::into_raw(Box::new(detect)) as *mut _;
+        }
+    }
+
+    return std::ptr::null_mut();
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn rs_smb_version_free(ptr: *mut c_void) {
+    if ptr != std::ptr::null_mut() {
+        std::mem::drop(Box::from_raw(ptr as *mut u8));
+    }
+}
+
+fn parse_version_data(arg: &str) -> Result<u8, ()> {
+    let arg = arg.trim();
+    let version = u8::from_str_radix(&arg, 10).map_err(|_| ())?;
+
+    if version != 1 && version != 2 {
+        return Err(());
+    }
+
+    return Ok(version);
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_parse_cmd_data() {
+        assert_eq!(Err(()), parse_version_data("0"));
+        assert_eq!(1u8, parse_version_data("1").unwrap());
+        assert_eq!(2u8, parse_version_data("2").unwrap());
+        assert_eq!(Err(()), parse_version_data("3"));
+    }
+
+    #[test]
+    fn test_parse_cmd_data_with_spaces() {
+        assert_eq!(1u8, parse_version_data(" 1").unwrap());
+        assert_eq!(2u8, parse_version_data(" 2 ").unwrap());
+    }
+
+}
diff --git a/src/Makefile.am b/src/Makefile.am
index ed1a78c6f71b..b8c28dcf6372 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -296,6 +296,7 @@ noinst_HEADERS = \
 	detect-sip-uri.h \
 	detect-smb-ntlmssp.h \
 	detect-smb-share.h \
+	detect-smb-version.h \
 	detect-snmp-community.h \
 	detect-snmp-pdu_type.h \
 	detect-snmp-usm.h \
@@ -913,6 +914,7 @@ libsuricata_c_a_SOURCES = \
 	detect-sip-uri.c \
 	detect-smb-ntlmssp.c \
 	detect-smb-share.c \
+	detect-smb-version.c \
 	detect-snmp-community.c \
 	detect-snmp-pdu_type.c \
 	detect-snmp-usm.c \
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index 218b0d7f0cb1..4ed42f312c4d 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -80,6 +80,7 @@
 #include "detect-config.h"
 
 #include "detect-smb-share.h"
+#include "detect-smb-version.h"
 
 #include "detect-base64-decode.h"
 #include "detect-base64-data.h"
@@ -618,6 +619,8 @@ void SigTableSetup(void)
     DetectSmbShareRegister();
     DetectSmbNtlmsspUserRegister();
     DetectSmbNtlmsspDomainRegister();
+    DetectSmbVersionRegister();
+
     DetectTlsRegister();
     DetectTlsValidityRegister();
     DetectTlsVersionRegister();
diff --git a/src/detect-engine-register.h b/src/detect-engine-register.h
index 8d4c7dfad3c1..eff9c0ed2572 100644
--- a/src/detect-engine-register.h
+++ b/src/detect-engine-register.h
@@ -203,6 +203,7 @@ enum DetectKeywordId {
     DETECT_SMB_SHARE,
     DETECT_SMB_NTLMSSP_USER,
     DETECT_SMB_NTLMSSP_DOMAIN,
+    DETECT_SMB_VERSION,
 
     DETECT_ASN1,
 
diff --git a/src/detect-smb-version.c b/src/detect-smb-version.c
new file mode 100644
index 000000000000..c8a423038689
--- /dev/null
+++ b/src/detect-smb-version.c
@@ -0,0 +1,155 @@
+/* Copyright (C) 2022-2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+/**
+ * \file
+ *
+ * \author Eloy Pérez
+ * \author Jason Taylor
+ *
+ * Implements the smb.version keyword
+ */
+
+#include "suricata-common.h"
+
+#include "detect.h"
+#include "detect-parse.h"
+
+#include "detect-engine.h"
+#include "detect-engine-mpm.h"
+#include "detect-engine-state.h"
+#include "detect-engine-prefilter.h"
+#include "detect-engine-content-inspection.h"
+
+#include "detect-smb-version.h"
+#include "rust.h"
+
+#define BUFFER_NAME  "smb_version"
+#define KEYWORD_NAME "smb.version"
+#define KEYWORD_ID   DETECT_SMB_VERSION
+
+static int g_smb_version_list_id = 0;
+
+static void DetectSmbVersionFree(DetectEngineCtx *de_ctx, void *ptr)
+{
+
+    SCLogDebug("smb_version: DetectSmbVersionFree");
+    rs_smb_version_free(ptr);
+}
+
+/**
+ * \brief Creates a SigMatch for the "smb.version" keyword being sent as argument,
+ *        and appends it to the rs_smb_version_match Signature(s).
+ *
+ * \param de_ctx Pointer to the detection engine context.
+ * \param s      Pointer to signature for the current Signature being parsed
+ *               from the rules.
+ * \param arg    Pointer to the string holding the keyword value.
+ *
+ * \retval 0 on success, -1 on failure
+ */
+
+static int DetectSmbVersionSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg)
+{
+    SCLogDebug("smb_version: DetectSmbVersionSetup");
+
+    if (DetectSignatureSetAppProto(s, ALPROTO_SMB) < 0)
+        return -1;
+
+    if (arg == NULL) {
+        SCLogError("Error parsing smb.version option in signature, it needs a value");
+        return -1;
+    }
+
+    if (DetectGetLastSMFromLists(s, DETECT_SMB_VERSION, -1)) {
+        SCLogError("Can't use 2 or more smb.version declarations in "
+                   "the same sig. Invalidating signature.");
+        return -1;
+    }
+
+    void *dod = rs_smb_version_parse(arg);
+
+    if (dod == NULL) {
+        SCLogError("Error parsing smb.version option in signature");
+        return -1;
+    }
+
+    if (SigMatchAppendSMToList(
+                de_ctx, s, DETECT_SMB_VERSION, (SigMatchCtx *)dod, g_smb_version_list_id) == NULL) {
+        DetectSmbVersionFree(de_ctx, dod);
+        return -1;
+    }
+
+    return 0;
+}
+
+/**
+ * \brief App layer match function for the "smb.version" keyword.
+ *
+ * \param t       Pointer to the ThreadVars instance.
+ * \param det_ctx Pointer to the DetectEngineThreadCtx.
+ * \param f       Pointer to the flow.
+ * \param flags   Pointer to the flags indicating the flow direction.
+ * \param state   Pointer to the app layer state data.
+ * \param s       Pointer to the Signature instance.
+ * \param m       Pointer to the SigMatch.
+ *
+ * \retval 1 On Match.
+ * \retval 0 On no match.
+ */
+
+static int DetectSmbVersionMatchRust(DetectEngineThreadCtx *det_ctx, Flow *f, uint8_t flags,
+        void *state, void *txv, const Signature *s, const SigMatchCtx *m)
+{
+
+    SCLogDebug("smb_version: DetectSmbVersionMatchRust");
+
+    int matchvalue = rs_smb_version_match(txv, (void *)m);
+
+    if (matchvalue != 1) {
+        SCLogDebug("rs_smb_version_match: didn't match");
+        SCReturnInt(0);
+    } else {
+        SCLogDebug("rs_smb_version_match: matched!");
+        return matchvalue;
+    }
+}
+
+/**
+ * \brief Registers the keyword handlers for the "smb_version" keyword.
+ */
+
+void DetectSmbVersionRegister(void)
+{
+    sigmatch_table[DETECT_SMB_VERSION].name = KEYWORD_NAME;
+    sigmatch_table[DETECT_SMB_VERSION].Setup = DetectSmbVersionSetup;
+    sigmatch_table[DETECT_SMB_VERSION].Match = NULL;
+    sigmatch_table[DETECT_SMB_VERSION].AppLayerTxMatch = DetectSmbVersionMatchRust;
+    sigmatch_table[DETECT_SMB_VERSION].Free = DetectSmbVersionFree;
+    sigmatch_table[DETECT_SMB_VERSION].desc = "smb keyword to match on SMB version";
+    sigmatch_table[DETECT_FLOW_AGE].url = "/rules/smb-keywords.html#smb-version";
+
+    DetectAppLayerInspectEngineRegister(
+            BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOSERVER, 0, DetectEngineInspectGenericList, NULL);
+
+    DetectAppLayerInspectEngineRegister(
+            BUFFER_NAME, ALPROTO_SMB, SIG_FLAG_TOCLIENT, 0, DetectEngineInspectGenericList, NULL);
+
+    g_smb_version_list_id = DetectBufferTypeRegister(BUFFER_NAME);
+
+    SCLogDebug("registering " BUFFER_NAME " rule option");
+}
\ No newline at end of file
diff --git a/src/detect-smb-version.h b/src/detect-smb-version.h
new file mode 100644
index 000000000000..11bdb1feae26
--- /dev/null
+++ b/src/detect-smb-version.h
@@ -0,0 +1,25 @@
+/* Copyright (C) 2022-2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#ifndef __DETECT_SMB_VERSION_H__
+#define __DETECT_SMB_VERSION_H__
+
+/** \brief registers the keyword into the engine. Called from
+ *         detect.c::SigTableSetup() */
+void DetectSmbVersionRegister(void);
+
+#endif /* __DETECT_SMB_VERSION_H__ */
\ No newline at end of file

From a4901a1f700361c95accf47781fa667316e98050 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eloy=20P=C3=A9rez=20Gonz=C3=A1lez?=
 <zer1t0ps@protonmail.com>
Date: Thu, 28 Apr 2022 12:59:03 +0200
Subject: [PATCH 377/462] smb: add smb.keyword documentation

---
 doc/userguide/rules/smb-keywords.rst | 43 +++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/rules/smb-keywords.rst b/doc/userguide/rules/smb-keywords.rst
index 13133354403d..228b834fc254 100644
--- a/doc/userguide/rules/smb-keywords.rst
+++ b/doc/userguide/rules/smb-keywords.rst
@@ -61,6 +61,47 @@ Examples::
 
 ``smb.ntlmssp_domain`` can be used as ``fast_pattern``.
 
+smb.version
+--------------
+
+Used to match the SMB version, that can be 1 or 2.
+
+Example signatures::
+
+  alert smb any any -> any any (msg: "SMB1 version rule"; smb.version: 1; sid: 44;)
+  alert smb any any -> any any (msg: "SMB2 version rule"; smb.version: 2; sid: 45;)
+
+Matching in transition from SMBv1 to SMBv2
+********************************************
+
+In the initial negotiation protocol request, a client supporting SMBv1 and SMBv2 can send an initial SMBv1 request and receive an SMBv2 response from server, indicating that SMBv2 will be used.
+
+This first SMBv2 response made by the server will match as SMBv1, since the entire transaction will be considered a SMBv1 transaction.
+
+Does `smb.version` match SMBv3?
+****************************************
+
+Yes, it will match SMBv3 messages using `smb.version: 2;`, which will match SMBv2 and SMBv3, since they use the same version identifier in the SMB header.
+
+This keyword will use the Protocol ID specified in SMB header to determine the version. Here is a summary of the Protocol ID codes:
+
+- 0xffSMB is SMB1 `header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb/3c0848a6-efe9-47c2-b57a-f7e8217150b9>`_
+- 0xfeSMB is SMB2 `normal header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/5cd64522-60b3-4f3e-a157-fe66f1228052>`_ (can be `sync <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/fb188936-5050-48d3-b350-dc43059638a4>`_ or `async <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/ea4560b7-90da-4803-82b5-344754b92a79>`_)
+- 0xfdSMB is SMB2 `transform header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_. This is only valid for the SMB 3.x dialect family.
+- 0xfcSMB is SMB2 `transform compression header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_ (can be `chained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/aa880fe8-ebed-4409-a474-ec6e0ca0dbcb>`_ or `unchained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/793db6bb-25b4-4469-be49-a8d7045ba3a6>`_). These ones requires the use of 3.1.1 dialect.
+
+The Protocol ID in header distinguishes only SMB1 and SMB2 since they are totally different protocols with total different message formats, types and implementation.
+
+On the other hand SMB3 is more an extension for SMB2. When using SMB2 we can select one of the following dialects for the conversation between client and server:
+
+- 2.0.2
+- 2.1
+- 3.0
+- 3.0.2
+- 3.1.1
+
+We say we are using SMB3 when we select a 3.x dialect for the conversation, so you can use SMB3.0, SMB3.0.2 or SMB3.1.1. The higher you choose, the more capabilities you have, but the message syntax and message command number remains the same.
+
 file.name
 ---------
 
@@ -74,4 +115,4 @@ Signature Example:
   :example-rule-options:`file.name; content:"file.txt";` \
   classtype:bad-unknown; sid:1; rev:1;)
 
-For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.
\ No newline at end of file
+For additional information on the ``file.name`` keyword, see :doc:`file-keywords`.

From bfc0790d8700b158062932ec0c53c582df192df1 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 30 Aug 2023 18:44:07 +0000
Subject: [PATCH 378/462] rust: fix rustfmt warnings for smb detect

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 rust/src/smb/detect.rs | 82 ++++++++++++++++++------------------------
 1 file changed, 34 insertions(+), 48 deletions(-)

diff --git a/rust/src/smb/detect.rs b/rust/src/smb/detect.rs
index 3445274bbcaf..4bc80c919ede 100644
--- a/rust/src/smb/detect.rs
+++ b/rust/src/smb/detect.rs
@@ -15,22 +15,20 @@
  * 02110-1301, USA.
  */
 
-use std::ptr;
 use crate::core::*;
-use crate::smb::smb::*;
-use crate::dcerpc::detect::{DCEIfaceData, DCEOpnumData, DETECT_DCE_OPNUM_RANGE_UNINITIALIZED};
 use crate::dcerpc::dcerpc::DCERPC_TYPE_REQUEST;
+use crate::dcerpc::detect::{DCEIfaceData, DCEOpnumData, DETECT_DCE_OPNUM_RANGE_UNINITIALIZED};
 use crate::detect::uint::detect_match_uint;
+use crate::smb::smb::SMBTransaction;
+use crate::smb::smb::*;
 use std::ffi::CStr;
 use std::os::raw::{c_char, c_void};
-use crate::smb::smb::SMBTransaction;
+use std::ptr;
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_smb_tx_get_share(tx: &mut SMBTransaction,
-                                            buffer: *mut *const u8,
-                                            buffer_len: *mut u32)
-                                            -> u8
-{
+pub unsafe extern "C" fn rs_smb_tx_get_share(
+    tx: &mut SMBTransaction, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
     if let Some(SMBTransactionTypeData::TREECONNECT(ref x)) = tx.type_data {
         SCLogDebug!("is_pipe {}", x.is_pipe);
         if !x.is_pipe {
@@ -46,11 +44,9 @@ pub unsafe extern "C" fn rs_smb_tx_get_share(tx: &mut SMBTransaction,
 }
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_smb_tx_get_named_pipe(tx: &mut SMBTransaction,
-                                            buffer: *mut *const u8,
-                                            buffer_len: *mut u32)
-                                            -> u8
-{
+pub unsafe extern "C" fn rs_smb_tx_get_named_pipe(
+    tx: &mut SMBTransaction, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
     if let Some(SMBTransactionTypeData::TREECONNECT(ref x)) = tx.type_data {
         SCLogDebug!("is_pipe {}", x.is_pipe);
         if x.is_pipe {
@@ -66,12 +62,9 @@ pub unsafe extern "C" fn rs_smb_tx_get_named_pipe(tx: &mut SMBTransaction,
 }
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_smb_tx_get_stub_data(tx: &mut SMBTransaction,
-                                            direction: u8,
-                                            buffer: *mut *const u8,
-                                            buffer_len: *mut u32)
-                                            -> u8
-{
+pub unsafe extern "C" fn rs_smb_tx_get_stub_data(
+    tx: &mut SMBTransaction, direction: u8, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
     if let Some(SMBTransactionTypeData::DCERPC(ref x)) = tx.type_data {
         let vref = if direction == Direction::ToServer as u8 {
             &x.stub_data_ts
@@ -91,10 +84,9 @@ pub unsafe extern "C" fn rs_smb_tx_get_stub_data(tx: &mut SMBTransaction,
 }
 
 #[no_mangle]
-pub extern "C" fn rs_smb_tx_match_dce_opnum(tx: &mut SMBTransaction,
-                                          dce_data: &mut DCEOpnumData)
-                                            -> u8
-{
+pub extern "C" fn rs_smb_tx_match_dce_opnum(
+    tx: &mut SMBTransaction, dce_data: &mut DCEOpnumData,
+) -> u8 {
     SCLogDebug!("rs_smb_tx_get_dce_opnum: start");
     if let Some(SMBTransactionTypeData::DCERPC(ref x)) = tx.type_data {
         if x.req_cmd == DCERPC_TYPE_REQUEST {
@@ -118,17 +110,13 @@ pub extern "C" fn rs_smb_tx_match_dce_opnum(tx: &mut SMBTransaction,
  *                     dce_opnum and dce_stub_data)
  * - only match on approved ifaces (so ack_result == 0) */
 #[no_mangle]
-pub extern "C" fn rs_smb_tx_get_dce_iface(state: &mut SMBState,
-                                            tx: &mut SMBTransaction,
-                                            dce_data: &mut DCEIfaceData)
-                                            -> u8
-{
+pub extern "C" fn rs_smb_tx_get_dce_iface(
+    state: &mut SMBState, tx: &mut SMBTransaction, dce_data: &mut DCEIfaceData,
+) -> u8 {
     let if_uuid = dce_data.if_uuid.as_slice();
     let is_dcerpc_request = match tx.type_data {
-        Some(SMBTransactionTypeData::DCERPC(ref x)) => {
-            x.req_cmd == DCERPC_TYPE_REQUEST
-        },
-        _ => { false },
+        Some(SMBTransactionTypeData::DCERPC(ref x)) => x.req_cmd == DCERPC_TYPE_REQUEST,
+        _ => false,
     };
     if !is_dcerpc_request {
         return 0;
@@ -137,13 +125,18 @@ pub extern "C" fn rs_smb_tx_get_dce_iface(state: &mut SMBState,
         Some(ref x) => x,
         _ => {
             return 0;
-        },
+        }
     };
 
     SCLogDebug!("looking for UUID {:?}", if_uuid);
 
     for i in ifaces {
-        SCLogDebug!("stored UUID {:?} acked {} ack_result {}", i, i.acked, i.ack_result);
+        SCLogDebug!(
+            "stored UUID {:?} acked {} ack_result {}",
+            i,
+            i.acked,
+            i.ack_result
+        );
 
         if i.acked && i.ack_result == 0 && i.uuid == if_uuid {
             if let Some(x) = &dce_data.du16 {
@@ -159,11 +152,9 @@ pub extern "C" fn rs_smb_tx_get_dce_iface(state: &mut SMBState,
 }
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_user(tx: &mut SMBTransaction,
-                                            buffer: *mut *const u8,
-                                            buffer_len: *mut u32)
-                                            -> u8
-{
+pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_user(
+    tx: &mut SMBTransaction, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
     if let Some(SMBTransactionTypeData::SESSIONSETUP(ref x)) = tx.type_data {
         if let Some(ref ntlmssp) = x.ntlmssp {
             *buffer = ntlmssp.user.as_ptr();
@@ -178,11 +169,9 @@ pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_user(tx: &mut SMBTransaction,
 }
 
 #[no_mangle]
-pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_domain(tx: &mut SMBTransaction,
-                                            buffer: *mut *const u8,
-                                            buffer_len: *mut u32)
-                                            -> u8
-{
+pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_domain(
+    tx: &mut SMBTransaction, buffer: *mut *const u8, buffer_len: *mut u32,
+) -> u8 {
     if let Some(SMBTransactionTypeData::SESSIONSETUP(ref x)) = tx.type_data {
         if let Some(ref ntlmssp) = x.ntlmssp {
             *buffer = ntlmssp.domain.as_ptr();
@@ -200,7 +189,6 @@ pub unsafe extern "C" fn rs_smb_tx_get_ntlmssp_domain(tx: &mut SMBTransaction,
 pub unsafe extern "C" fn rs_smb_version_match(
     tx: &mut SMBTransaction, version_data: &mut u8,
 ) -> u8 {
-
     let version = tx.vercmd.get_version();
     if version == *version_data {
         return 1;
@@ -209,7 +197,6 @@ pub unsafe extern "C" fn rs_smb_version_match(
     return 0;
 }
 
-
 #[no_mangle]
 pub unsafe extern "C" fn rs_smb_version_parse(carg: *const c_char) -> *mut c_void {
     if carg.is_null() {
@@ -260,5 +247,4 @@ mod tests {
         assert_eq!(1u8, parse_version_data(" 1").unwrap());
         assert_eq!(2u8, parse_version_data(" 2 ").unwrap());
     }
-
 }

From 3cb7112aa5392371848a97ebd2a9b8ffd524fa8c Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 30 Aug 2023 18:50:19 +0000
Subject: [PATCH 379/462] detect: update smb.version keyword

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/rules/smb-keywords.rst | 44 +++++++++++++++++-----------
 rust/src/smb/detect.rs               | 22 +++++++-------
 src/detect-engine-register.c         |  1 -
 3 files changed, 38 insertions(+), 29 deletions(-)

diff --git a/doc/userguide/rules/smb-keywords.rst b/doc/userguide/rules/smb-keywords.rst
index 228b834fc254..5bf5c6c75c80 100644
--- a/doc/userguide/rules/smb-keywords.rst
+++ b/doc/userguide/rules/smb-keywords.rst
@@ -61,38 +61,46 @@ Examples::
 
 ``smb.ntlmssp_domain`` can be used as ``fast_pattern``.
 
+
 smb.version
---------------
+------------
 
-Used to match the SMB version, that can be 1 or 2.
+Keyword to match on the SMB version seen in an SMB transaction.
 
-Example signatures::
+Signature Example:
+
+.. container:: example-rule
+
+  alert smb $HOME_NET any -> any any (msg:"SMBv1 version rule"; \
+  :example-rule-options:`smb.version:1;` sid:1;)
+  
+  alert smb $HOME_NET any -> any any (msg:"SMBv2 version rule"; \
+  :example-rule-options:`smb.version:2;` sid:2;)
 
-  alert smb any any -> any any (msg: "SMB1 version rule"; smb.version: 1; sid: 44;)
-  alert smb any any -> any any (msg: "SMB2 version rule"; smb.version: 2; sid: 45;)
 
 Matching in transition from SMBv1 to SMBv2
-********************************************
+******************************************
 
-In the initial negotiation protocol request, a client supporting SMBv1 and SMBv2 can send an initial SMBv1 request and receive an SMBv2 response from server, indicating that SMBv2 will be used.
+In the initial protocol negotiation request, a client supporting SMBv1 and SMBv2 can send an initial SMBv1 request and receive a SMBv2 response from server, indicating that SMBv2 will be used.
 
 This first SMBv2 response made by the server will match as SMBv1, since the entire transaction will be considered a SMBv1 transaction.
 
-Does `smb.version` match SMBv3?
-****************************************
+Will ``smb.version`` match SMBv3 traffic?
+*****************************************
 
-Yes, it will match SMBv3 messages using `smb.version: 2;`, which will match SMBv2 and SMBv3, since they use the same version identifier in the SMB header.
+Yes, it will match SMBv3 messages using `smb.version:2;`, which will match SMBv2 and SMBv3, since they use the same version identifier in the SMB header.
 
 This keyword will use the Protocol ID specified in SMB header to determine the version. Here is a summary of the Protocol ID codes:
 
-- 0xffSMB is SMB1 `header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb/3c0848a6-efe9-47c2-b57a-f7e8217150b9>`_
-- 0xfeSMB is SMB2 `normal header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/5cd64522-60b3-4f3e-a157-fe66f1228052>`_ (can be `sync <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/fb188936-5050-48d3-b350-dc43059638a4>`_ or `async <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/ea4560b7-90da-4803-82b5-344754b92a79>`_)
-- 0xfdSMB is SMB2 `transform header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_. This is only valid for the SMB 3.x dialect family.
-- 0xfcSMB is SMB2 `transform compression header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_ (can be `chained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/aa880fe8-ebed-4409-a474-ec6e0ca0dbcb>`_ or `unchained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/793db6bb-25b4-4469-be49-a8d7045ba3a6>`_). These ones requires the use of 3.1.1 dialect.
+- 0xffSMB is SMBv1 `header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb/3c0848a6-efe9-47c2-b57a-f7e8217150b9>`_
+- 0xfeSMB is SMBv2 `normal header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/5cd64522-60b3-4f3e-a157-fe66f1228052>`_ (can be `sync <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/fb188936-5050-48d3-b350-dc43059638a4>`_ or `async <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/ea4560b7-90da-4803-82b5-344754b92a79>`_)
+- 0xfdSMB is SMBv2 `transform header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_. This is only valid for the SMB 3.x dialect family.
+- 0xfcSMB is SMBv2 `transform compression header <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d6ce2327-a4c9-4793-be66-7b5bad2175fa>`_ (can be `chained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/aa880fe8-ebed-4409-a474-ec6e0ca0dbcb>`_ or `unchained <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/793db6bb-25b4-4469-be49-a8d7045ba3a6>`_). These ones require the use of the 3.1.1 dialect.
 
-The Protocol ID in header distinguishes only SMB1 and SMB2 since they are totally different protocols with total different message formats, types and implementation.
+The Protocol ID in the header distinguishes only SMBv1 and SMBv2 since they are completely different protocols with entirely
+different message formats, types and implementations.
 
-On the other hand SMB3 is more an extension for SMB2. When using SMB2 we can select one of the following dialects for the conversation between client and server:
+On the other hand, SMBv3 is more like an extension of SMBv2. When using SMBv2 we can select one of the following dialects for the conversation between client and server:
 
 - 2.0.2
 - 2.1
@@ -100,7 +108,9 @@ On the other hand SMB3 is more an extension for SMB2. When using SMB2 we can sel
 - 3.0.2
 - 3.1.1
 
-We say we are using SMB3 when we select a 3.x dialect for the conversation, so you can use SMB3.0, SMB3.0.2 or SMB3.1.1. The higher you choose, the more capabilities you have, but the message syntax and message command number remains the same.
+We say we are using SMBv3 when we select a 3.x dialect for the conversation, so you can use SMB 3.0, SMB 3.0.2 or SMB 3.1.1. The higher you choose, the more capabilities you have, but the message syntax and message command number remains the same.
+
+SMB version and dialect are separate components. In the case of SMBv3 for instance, the SMB version will be 2 but the dialect will be 3.x. Dialect specification is not available currently via keyword.
 
 file.name
 ---------
diff --git a/rust/src/smb/detect.rs b/rust/src/smb/detect.rs
index 4bc80c919ede..dd525ec825f3 100644
--- a/rust/src/smb/detect.rs
+++ b/rust/src/smb/detect.rs
@@ -1,4 +1,4 @@
-/* Copyright (C) 2017 Open Information Security Foundation
+/* Copyright (C) 2017-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -19,7 +19,6 @@ use crate::core::*;
 use crate::dcerpc::dcerpc::DCERPC_TYPE_REQUEST;
 use crate::dcerpc::detect::{DCEIfaceData, DCEOpnumData, DETECT_DCE_OPNUM_RANGE_UNINITIALIZED};
 use crate::detect::uint::detect_match_uint;
-use crate::smb::smb::SMBTransaction;
 use crate::smb::smb::*;
 use std::ffi::CStr;
 use std::os::raw::{c_char, c_void};
@@ -190,6 +189,7 @@ pub unsafe extern "C" fn rs_smb_version_match(
     tx: &mut SMBTransaction, version_data: &mut u8,
 ) -> u8 {
     let version = tx.vercmd.get_version();
+    SCLogDebug!("smb_version: version returned: {}", version);
     if version == *version_data {
         return 1;
     }
@@ -212,16 +212,11 @@ pub unsafe extern "C" fn rs_smb_version_parse(carg: *const c_char) -> *mut c_voi
     return std::ptr::null_mut();
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn rs_smb_version_free(ptr: *mut c_void) {
-    if ptr != std::ptr::null_mut() {
-        std::mem::drop(Box::from_raw(ptr as *mut u8));
-    }
-}
-
 fn parse_version_data(arg: &str) -> Result<u8, ()> {
     let arg = arg.trim();
-    let version = u8::from_str_radix(&arg, 10).map_err(|_| ())?;
+    let version: u8 = arg.parse().map_err(|_| ())?;
+
+    SCLogDebug!("smb_version: sig parse arg: {} version: {}", arg, version);
 
     if version != 1 && version != 2 {
         return Err(());
@@ -230,6 +225,11 @@ fn parse_version_data(arg: &str) -> Result<u8, ()> {
     return Ok(version);
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn rs_smb_version_free(ptr: *mut c_void) {
+    std::mem::drop(Box::from_raw(ptr as *mut u8));
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -247,4 +247,4 @@ mod tests {
         assert_eq!(1u8, parse_version_data(" 1").unwrap());
         assert_eq!(2u8, parse_version_data(" 2 ").unwrap());
     }
-}
+}
\ No newline at end of file
diff --git a/src/detect-engine-register.c b/src/detect-engine-register.c
index 4ed42f312c4d..595bffcc5737 100644
--- a/src/detect-engine-register.c
+++ b/src/detect-engine-register.c
@@ -620,7 +620,6 @@ void SigTableSetup(void)
     DetectSmbNtlmsspUserRegister();
     DetectSmbNtlmsspDomainRegister();
     DetectSmbVersionRegister();
-
     DetectTlsRegister();
     DetectTlsValidityRegister();
     DetectTlsVersionRegister();

From de3cbe4c90fb7fb6d54b6876ab515f791145271a Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 24 Jan 2024 09:02:19 -0600
Subject: [PATCH 380/462] detect/requires: reset sigerror flags for each rule

"sigerror_ok" and "sigerror_requires" were not being reset after each
rule which could lead to a rule load error being incorrectly tracked
as skipped rather than failed.

Also initialize "skippedsigs" to 0 along with "goodsigs" and
"badsigs", while not directly related to this issue, could also throw
off some stats.

Ticket: #6710
---
 src/detect-engine-loader.c | 1 +
 src/detect-parse.c         | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 9c3e3e8b533c..1f5363c6f80c 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -122,6 +122,7 @@ static int DetectLoadSigFile(
 
     (*goodsigs) = 0;
     (*badsigs) = 0;
+    (*skippedsigs) = 0;
 
     FILE *fp = fopen(sig_file, "r");
     if (fp == NULL) {
diff --git a/src/detect-parse.c b/src/detect-parse.c
index 31df3d0aaed3..e1c073efd6a3 100644
--- a/src/detect-parse.c
+++ b/src/detect-parse.c
@@ -2315,7 +2315,9 @@ Signature *SigInit(DetectEngineCtx *de_ctx, const char *sigstr)
     SCEnter();
 
     uint32_t oldsignum = de_ctx->signum;
+    de_ctx->sigerror_ok = false;
     de_ctx->sigerror_silent = false;
+    de_ctx->sigerror_requires = false;
 
     Signature *sig;
 

From 8bf8131c31088f501f4d151f8748a817f89f7c1f Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 24 Jan 2024 09:55:37 -0600
Subject: [PATCH 381/462] doc: note what version "requires" was added in

---
 doc/userguide/rules/meta.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/userguide/rules/meta.rst b/doc/userguide/rules/meta.rst
index 0e888add697a..1ceb5fe834e0 100644
--- a/doc/userguide/rules/meta.rst
+++ b/doc/userguide/rules/meta.rst
@@ -258,3 +258,4 @@ default to 0.
 The ``version`` may only be specified once, if specified more than
 once the rule will log an error and not be loaded.
 
+The ``requires`` keyword was introduced in Suricata 7.0.3 and 8.0.0.

From c3b3c11e30e1afdbde5c196a0692ee5e2c379cb1 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Wed, 24 Jan 2024 14:41:47 -0600
Subject: [PATCH 382/462] requirements: use libhtp 0.5.x

Move to libhtp to the 0.5.x branch instead of 0.5.45.
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 289c0223144a..0c29e60313c8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,5 +3,5 @@
 # Format:
 #
 #   name {repo} {branch|tag}
-libhtp https://github.com/OISF/libhtp 0.5.45
+libhtp https://github.com/OISF/libhtp 0.5.x
 suricata-update https://github.com/OISF/suricata-update 1.3.0

From 9240ae250cc369306803740279df2ab3eca6b54a Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 7 Sep 2023 14:33:04 +0200
Subject: [PATCH 383/462] detect: avoids case of useless detection on txs

When a TCP flow packet has not led to app-layer updates,
it is useless to run DetectRunTx, as there cannot be new
matches.

This happens for instance, when one side sends in a row multiple
packets which are not acked (and thus not parsed in IDS mode).

Doing so requires to move up the call to
AppLayerParserSetTransactionInspectId
so that it is run the same times DetectRunTx is run, and not in the
case where the transaction was not updated.

Ticket: 6299
---
 src/detect.c               | 19 +++++++++++--------
 src/util-unittest-helper.c |  1 +
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index dca6fe9f651d..cbecdda2a563 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -152,6 +152,12 @@ static void DetectRun(ThreadVars *th_v,
                 DetectRunFrames(th_v, de_ctx, det_ctx, p, pflow, &scratch);
                 // PACKET_PROFILING_DETECT_END(p, PROF_DETECT_TX);
             }
+            // no update to transactions
+            if (!PKT_IS_PSEUDOPKT(p) && p->app_update_direction == 0 &&
+                    ((PKT_IS_TOSERVER(p) && (p->flow->flags & FLOW_TS_APP_UPDATED) == 0) ||
+                            (PKT_IS_TOCLIENT(p) && (p->flow->flags & FLOW_TC_APP_UPDATED) == 0))) {
+                goto end;
+            }
         } else if (p->proto == IPPROTO_UDP) {
             DetectRunFrames(th_v, de_ctx, det_ctx, p, pflow, &scratch);
         }
@@ -159,6 +165,11 @@ static void DetectRun(ThreadVars *th_v,
         PACKET_PROFILING_DETECT_START(p, PROF_DETECT_TX);
         DetectRunTx(th_v, de_ctx, det_ctx, p, pflow, &scratch);
         PACKET_PROFILING_DETECT_END(p, PROF_DETECT_TX);
+        /* see if we need to increment the inspect_id and reset the de_state */
+        PACKET_PROFILING_DETECT_START(p, PROF_DETECT_TX_UPDATE);
+        AppLayerParserSetTransactionInspectId(
+                pflow, pflow->alparser, pflow->alstate, scratch.flow_flags, (scratch.sgh == NULL));
+        PACKET_PROFILING_DETECT_END(p, PROF_DETECT_TX_UPDATE);
     }
 
 end:
@@ -911,14 +922,6 @@ static inline void DetectRunPostRules(
     Flow * const pflow,
     DetectRunScratchpad *scratch)
 {
-    /* see if we need to increment the inspect_id and reset the de_state */
-    if (pflow && pflow->alstate) {
-        PACKET_PROFILING_DETECT_START(p, PROF_DETECT_TX_UPDATE);
-        AppLayerParserSetTransactionInspectId(pflow, pflow->alparser, pflow->alstate,
-                scratch->flow_flags, (scratch->sgh == NULL));
-        PACKET_PROFILING_DETECT_END(p, PROF_DETECT_TX_UPDATE);
-    }
-
     /* so now let's iterate the alerts and remove the ones after a pass rule
      * matched (if any). This is done inside PacketAlertFinalize() */
     /* PR: installed "tag" keywords are handled after the threshold inspection */
diff --git a/src/util-unittest-helper.c b/src/util-unittest-helper.c
index 48d2a045c19b..743ed1f43e8c 100644
--- a/src/util-unittest-helper.c
+++ b/src/util-unittest-helper.c
@@ -316,6 +316,7 @@ Packet *UTHBuildPacketReal(uint8_t *payload, uint16_t payload_len,
     }
     SET_PKT_LEN(p, hdr_offset + payload_len);
     p->payload = GET_PKT_DATA(p)+hdr_offset;
+    p->app_update_direction = UPDATE_DIR_BOTH;
 
     return p;
 

From 5bb8800588e7b4a09e1770f049cd88be71e2d30b Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 18 Sep 2023 13:27:47 +0200
Subject: [PATCH 384/462] detect: merge sorted lists instead of qsort

Ticket: #6299

Simply because it is faster (just linear).

This is for merging match_array into tx_candidates
---
 src/detect.c | 95 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 78 insertions(+), 17 deletions(-)

diff --git a/src/detect.c b/src/detect.c
index cbecdda2a563..659f96441b29 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -1296,6 +1296,81 @@ static inline void StoreDetectFlags(DetectTransaction *tx, const uint8_t flow_fl
     }
 }
 
+// Merge 'state' rules from the regular prefilter
+// updates array_idx on the way
+static inline void RuleMatchCandidateMergeStateRules(
+        DetectEngineThreadCtx *det_ctx, uint32_t *array_idx)
+{
+    // Now, we will merge 2 sorted lists :
+    // the one in det_ctx->tx_candidates
+    // and the one in det_ctx->match_array
+    // For match_array, we take only the relevant elements where s->app_inspect != NULL
+
+    // Basically, we iterate at the same time over the 2 lists
+    // comparing and taking an element from either.
+
+    // Trick is to do so in place in det_ctx->tx_candidates,
+    // so as to minimize the number of moves in det_ctx->tx_candidates.
+    // For this, the algorithm traverses the lists in reverse order.
+    // Otherwise, if the first element of match_array was to be put before
+    // all tx_candidates, we would need to shift all tx_candidates
+
+    // Retain the number of elements sorted in tx_candidates before merge
+    uint32_t j = *array_idx;
+    // First loop only counting the number of elements to add
+    for (uint32_t i = 0; i < det_ctx->match_array_cnt; i++) {
+        const Signature *s = det_ctx->match_array[i];
+        if (s->app_inspect != NULL) {
+            (*array_idx)++;
+        }
+    }
+    // Future number of elements in tx_candidates after merge
+    uint32_t k = *array_idx;
+
+    if (k == j) {
+        // no new element from match_array to merge in tx_candidates
+        return;
+    }
+
+    // variable i is for all elements of match_array (even not relevant ones)
+    // variable j is for elements of tx_candidates before merge
+    // variable k is for elements of tx_candidates after merge
+    for (uint32_t i = det_ctx->match_array_cnt; i > 0;) {
+        const Signature *s = det_ctx->match_array[i - 1];
+        if (s->app_inspect == NULL) {
+            // no relevant element, get the next one from match_array
+            i--;
+            continue;
+        }
+        // we have one element from match_array to merge in tx_candidates
+        k--;
+        if (j > 0) {
+            // j > 0 means there is still at least one element in tx_candidates to merge
+            const RuleMatchCandidateTx *s0 = &det_ctx->tx_candidates[j - 1];
+            if (s->num <= s0->id) {
+                // get next element from previous tx_candidates
+                j--;
+                // take the element from tx_candidates before merge
+                det_ctx->tx_candidates[k].s = det_ctx->tx_candidates[j].s;
+                det_ctx->tx_candidates[k].id = det_ctx->tx_candidates[j].id;
+                det_ctx->tx_candidates[k].flags = det_ctx->tx_candidates[j].flags;
+                det_ctx->tx_candidates[k].stream_reset = det_ctx->tx_candidates[j].stream_reset;
+                continue;
+            }
+        } // otherwise
+        // get next element from match_array
+        i--;
+        // take the element from match_array
+        det_ctx->tx_candidates[k].s = s;
+        det_ctx->tx_candidates[k].id = s->num;
+        det_ctx->tx_candidates[k].flags = NULL;
+        det_ctx->tx_candidates[k].stream_reset = 0;
+    }
+    // Even if k > 0 or j > 0, the loop is over. (Note that j == k now)
+    // The remaining elements in tx_candidates up to k were already sorted
+    // and come before any other element later in the list
+}
+
 static void DetectRunTx(ThreadVars *tv,
                     DetectEngineCtx *de_ctx,
                     DetectEngineThreadCtx *det_ctx,
@@ -1369,24 +1444,10 @@ static void DetectRunTx(ThreadVars *tv,
         }
 
         /* merge 'state' rules from the regular prefilter */
+#ifdef PROFILING
         uint32_t x = array_idx;
-        for (uint32_t i = 0; i < det_ctx->match_array_cnt; i++) {
-            const Signature *s = det_ctx->match_array[i];
-            if (s->app_inspect != NULL) {
-                const SigIntId id = s->num;
-                det_ctx->tx_candidates[array_idx].s = s;
-                det_ctx->tx_candidates[array_idx].id = id;
-                det_ctx->tx_candidates[array_idx].flags = NULL;
-                det_ctx->tx_candidates[array_idx].stream_reset = 0;
-                array_idx++;
-
-                SCLogDebug("%p/%"PRIu64" rule %u (%u) added from 'match' list",
-                        tx.tx_ptr, tx.tx_id, s->id, id);
-            }
-        }
-        do_sort = (array_idx > x); // sort if match added anything
-        SCLogDebug("%p/%" PRIu64 " rules added from 'match' list: %u", tx.tx_ptr, tx.tx_id,
-                array_idx - x);
+#endif
+        RuleMatchCandidateMergeStateRules(det_ctx, &array_idx);
 
         /* merge stored state into results */
         if (tx.de_state != NULL) {

From 2fb50598f23b112f14ec15330e11c40b74caa35f Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 18 Sep 2023 13:29:08 +0200
Subject: [PATCH 385/462] detect: do not store state without flags

If flags are zero, there is nothing to store and remember.

Stored signatures will be reused on a later packet, and
qsorted (which may be expensive), with newer matches candidates.

Avoiding to store, leads to avoid the call to qsort.
---
 src/detect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/detect.c b/src/detect.c
index 659f96441b29..72e40eaff3fd 100644
--- a/src/detect.c
+++ b/src/detect.c
@@ -1218,7 +1218,7 @@ static bool DetectRunTxInspectRule(ThreadVars *tv,
         } else if ((inspect_flags & DE_STATE_FLAG_FULL_INSPECT) == 0 && mpm_in_progress) {
             TRACE_SID_TXS(s->id, tx, "no need to store no-match sig, "
                     "mpm will revisit it");
-        } else {
+        } else if (inspect_flags != 0 || file_no_match != 0) {
             TRACE_SID_TXS(s->id, tx, "storing state: flags %08x", inspect_flags);
             DetectRunStoreStateTx(scratch->sgh, f, tx->tx_ptr, tx->tx_id, s,
                     inspect_flags, flow_flags, file_no_match);

From 89936b6530690c6d03869b2ad8b82f9f84776f94 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 17 Oct 2023 10:26:57 +0200
Subject: [PATCH 386/462] mqtt: fix logic when setting event

Especially sets transactions to complete when we get a response
without having seen the request, so that the transactions
end up getting cleaned (instead of living/leaking in the state).

Also try to set the event on the relevant transaction, instead
of creating a new transaction just for the purpose of having
the event.

Ticket: #6299
---
 rust/src/mqtt/mqtt.rs | 141 ++++++++++++++++--------------------------
 1 file changed, 55 insertions(+), 86 deletions(-)

diff --git a/rust/src/mqtt/mqtt.rs b/rust/src/mqtt/mqtt.rs
index 7f60e2a757cd..fbf03e19af69 100644
--- a/rust/src/mqtt/mqtt.rs
+++ b/rust/src/mqtt/mqtt.rs
@@ -183,11 +183,11 @@ impl MQTTState {
     }
 
     fn new_tx(&mut self, msg: MQTTMessage, toclient: bool) -> MQTTTransaction {
-	let direction = if toclient {
-	    Direction::ToClient
-	} else {
-	    Direction::ToServer
-	};
+        let direction = if toclient {
+            Direction::ToClient
+        } else {
+            Direction::ToServer
+        };
         let mut tx = MQTTTransaction::new(msg, direction);
         self.tx_id += 1;
         tx.tx_id = self.tx_id;
@@ -217,104 +217,82 @@ impl MQTTState {
         match msg.op {
             MQTTOperation::CONNECT(ref conn) => {
                 self.protocol_version = conn.protocol_version;
+                let mut tx = self.new_tx(msg, toclient);
+                tx.pkt_id = Some(MQTT_CONNECT_PKT_ID);
                 if self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::DoubleConnect);
-                    self.transactions.push_back(tx);
-                } else {
-                    let mut tx = self.new_tx(msg, toclient);
-                    tx.pkt_id = Some(MQTT_CONNECT_PKT_ID);
-                    self.transactions.push_back(tx);
                 }
+                self.transactions.push_back(tx);
             }
             MQTTOperation::PUBLISH(ref publish) => {
-                if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
-                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
-                }
-                match msg.header.qos_level {
+                let qos = msg.header.qos_level;
+                let pkt_id = publish.message_id;
+                let mut tx = self.new_tx(msg, toclient);
+                match qos {
                     0 => {
                         // with QOS level 0, we do not need to wait for a
                         // response
-                        let mut tx = self.new_tx(msg, toclient);
                         tx.complete = true;
-                        self.transactions.push_back(tx);
                     }
                     1..=2 => {
-                        if let Some(pkt_id) = publish.message_id {
-                            let mut tx = self.new_tx(msg, toclient);
+                        if let Some(pkt_id) = pkt_id {
                             tx.pkt_id = Some(pkt_id as u32);
-                            self.transactions.push_back(tx);
                         } else {
-                            let mut tx = self.new_tx(msg, toclient);
                             MQTTState::set_event(&mut tx, MQTTEvent::MissingMsgId);
-                            self.transactions.push_back(tx);
                         }
                     }
                     _ => {
-                        let mut tx = self.new_tx(msg, toclient);
                         MQTTState::set_event(&mut tx, MQTTEvent::InvalidQosLevel);
-                        self.transactions.push_back(tx);
                     }
                 }
-            }
-            MQTTOperation::SUBSCRIBE(ref subscribe) => {
                 if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
                 }
+                self.transactions.push_back(tx);
+            }
+            MQTTOperation::SUBSCRIBE(ref subscribe) => {
                 let pkt_id = subscribe.message_id as u32;
-                match msg.header.qos_level {
+                let qos = msg.header.qos_level;
+                let mut tx = self.new_tx(msg, toclient);
+                match qos {
                     0 => {
                         // with QOS level 0, we do not need to wait for a
                         // response
-                        let mut tx = self.new_tx(msg, toclient);
                         tx.complete = true;
-                        self.transactions.push_back(tx);
                     }
                     1..=2 => {
-                        let mut tx = self.new_tx(msg, toclient);
                         tx.pkt_id = Some(pkt_id);
-                        self.transactions.push_back(tx);
                     }
                     _ => {
-                        let mut tx = self.new_tx(msg, toclient);
                         MQTTState::set_event(&mut tx, MQTTEvent::InvalidQosLevel);
-                        self.transactions.push_back(tx);
                     }
                 }
-            }
-            MQTTOperation::UNSUBSCRIBE(ref unsubscribe) => {
                 if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
                 }
+                self.transactions.push_back(tx);
+            }
+            MQTTOperation::UNSUBSCRIBE(ref unsubscribe) => {
                 let pkt_id = unsubscribe.message_id as u32;
-                match msg.header.qos_level {
+                let qos = msg.header.qos_level;
+                let mut tx = self.new_tx(msg, toclient);
+                match qos {
                     0 => {
                         // with QOS level 0, we do not need to wait for a
                         // response
-                        let mut tx = self.new_tx(msg, toclient);
                         tx.complete = true;
-                        self.transactions.push_back(tx);
                     }
                     1..=2 => {
-                        let mut tx = self.new_tx(msg, toclient);
                         tx.pkt_id = Some(pkt_id);
-                        self.transactions.push_back(tx);
                     }
                     _ => {
-                        let mut tx = self.new_tx(msg, toclient);
                         MQTTState::set_event(&mut tx, MQTTEvent::InvalidQosLevel);
-                        self.transactions.push_back(tx);
                     }
                 }
+                if !self.connected {
+                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
+                }
+                self.transactions.push_back(tx);
             }
             MQTTOperation::CONNACK(ref _connack) => {
                 if let Some(tx) = self.get_tx_by_pkt_id(MQTT_CONNECT_PKT_ID) {
@@ -325,31 +303,24 @@ impl MQTTState {
                 } else {
                     let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::MissingConnect);
+                    tx.complete = true;
                     self.transactions.push_back(tx);
                 }
             }
             MQTTOperation::PUBREC(ref v) | MQTTOperation::PUBREL(ref v) => {
-                if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
-                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
-                }
                 if let Some(tx) = self.get_tx_by_pkt_id(v.message_id as u32) {
                     tx.msg.push(msg);
                 } else {
                     let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::MissingPublish);
+                    if !self.connected {
+                        MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
+                    }
+                    tx.complete = true;
                     self.transactions.push_back(tx);
                 }
             }
             MQTTOperation::PUBACK(ref v) | MQTTOperation::PUBCOMP(ref v) => {
-                if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
-                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
-                }
                 if let Some(tx) = self.get_tx_by_pkt_id(v.message_id as u32) {
                     tx.msg.push(msg);
                     tx.complete = true;
@@ -357,16 +328,14 @@ impl MQTTState {
                 } else {
                     let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::MissingPublish);
+                    if !self.connected {
+                        MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
+                    }
+                    tx.complete = true;
                     self.transactions.push_back(tx);
                 }
             }
             MQTTOperation::SUBACK(ref suback) => {
-                if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
-                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
-                }
                 if let Some(tx) = self.get_tx_by_pkt_id(suback.message_id as u32) {
                     tx.msg.push(msg);
                     tx.complete = true;
@@ -374,16 +343,14 @@ impl MQTTState {
                 } else {
                     let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::MissingSubscribe);
+                    if !self.connected {
+                        MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
+                    }
+                    tx.complete = true;
                     self.transactions.push_back(tx);
                 }
             }
             MQTTOperation::UNSUBACK(ref unsuback) => {
-                if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
-                    MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
-                }
                 if let Some(tx) = self.get_tx_by_pkt_id(unsuback.message_id as u32) {
                     tx.msg.push(msg);
                     tx.complete = true;
@@ -391,6 +358,10 @@ impl MQTTState {
                 } else {
                     let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::MissingUnsubscribe);
+                    if !self.connected {
+                        MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
+                    }
+                    tx.complete = true;
                     self.transactions.push_back(tx);
                 }
             }
@@ -406,25 +377,19 @@ impl MQTTState {
                 self.transactions.push_back(tx);
             }
             MQTTOperation::AUTH(_) | MQTTOperation::DISCONNECT(_) => {
+                let mut tx = self.new_tx(msg, toclient);
+                tx.complete = true;
                 if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
                 }
-                let mut tx = self.new_tx(msg, toclient);
-                tx.complete = true;
                 self.transactions.push_back(tx);
             }
             MQTTOperation::PINGREQ | MQTTOperation::PINGRESP => {
+                let mut tx = self.new_tx(msg, toclient);
+                tx.complete = true;
                 if !self.connected {
-                    let mut tx = self.new_tx(msg, toclient);
                     MQTTState::set_event(&mut tx, MQTTEvent::UnintroducedMessage);
-                    self.transactions.push_back(tx);
-                    return;
                 }
-                let mut tx = self.new_tx(msg, toclient);
-                tx.complete = true;
                 self.transactions.push_back(tx);
             }
         }
@@ -608,7 +573,11 @@ impl MQTTState {
     }
 
     fn set_event_notx(&mut self, event: MQTTEvent, toclient: bool) {
-        let mut tx = MQTTTransaction::new_empty(if toclient { Direction::ToClient } else { Direction::ToServer });
+        let mut tx = MQTTTransaction::new_empty(if toclient {
+            Direction::ToClient
+        } else {
+            Direction::ToServer
+        });
         self.tx_id += 1;
         tx.tx_id = self.tx_id;
         if toclient {

From 38db51b8789e9f930ff093f592b5763edb190f43 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 16 Jan 2024 11:47:30 +0100
Subject: [PATCH 387/462] rust: make cargo clippy clean

Fixing single_match and manual_find intertwined with SCLogDebug
---
 rust/src/dcerpc/dcerpc_udp.rs | 13 +++---
 rust/src/dns/dns.rs           | 10 +---
 rust/src/nfs/nfs.rs           | 34 +++-----------
 rust/src/smb/auth.rs          | 29 ++++++------
 rust/src/smb/smb.rs           | 86 ++++++++++++++++-------------------
 rust/src/smb/smb1.rs          | 23 ++++------
 rust/src/smb/smb1_session.rs  | 13 ++----
 rust/src/smb/smb2_session.rs  | 13 ++----
 8 files changed, 88 insertions(+), 133 deletions(-)

diff --git a/rust/src/dcerpc/dcerpc_udp.rs b/rust/src/dcerpc/dcerpc_udp.rs
index d34c3e480b36..b17cc8d1fa8b 100644
--- a/rust/src/dcerpc/dcerpc_udp.rs
+++ b/rust/src/dcerpc/dcerpc_udp.rs
@@ -141,13 +141,12 @@ impl DCERPCUDPState {
     }
 
     fn find_incomplete_tx(&mut self, hdr: &DCERPCHdrUdp) -> Option<&mut DCERPCTransaction> {
-        for tx in &mut self.transactions {
-            if tx.seqnum == hdr.seqnum && tx.activityuuid == hdr.activityuuid && ((hdr.pkt_type == DCERPC_TYPE_REQUEST && !tx.req_done) || (hdr.pkt_type == DCERPC_TYPE_RESPONSE && !tx.resp_done)) {
-                SCLogDebug!("found tx id {}, last tx_id {}, {} {}", tx.id, self.tx_id, tx.seqnum, tx.activityuuid[0]);
-                return Some(tx);
-            }
-        }
-        None
+        return self.transactions.iter_mut().find(|tx| {
+            tx.seqnum == hdr.seqnum
+                && tx.activityuuid == hdr.activityuuid
+                && ((hdr.pkt_type == DCERPC_TYPE_REQUEST && !tx.req_done)
+                    || (hdr.pkt_type == DCERPC_TYPE_RESPONSE && !tx.resp_done))
+        });
     }
 
     pub fn handle_fragment_data(&mut self, hdr: &DCERPCHdrUdp, input: &[u8]) -> bool {
diff --git a/rust/src/dns/dns.rs b/rust/src/dns/dns.rs
index 57f66c0f73df..d4f71d268273 100644
--- a/rust/src/dns/dns.rs
+++ b/rust/src/dns/dns.rs
@@ -358,15 +358,7 @@ impl DNSState {
     }
 
     pub fn get_tx(&mut self, tx_id: u64) -> Option<&DNSTransaction> {
-        SCLogDebug!("get_tx: tx_id={}", tx_id);
-        for tx in &mut self.transactions {
-            if tx.id == tx_id + 1 {
-                SCLogDebug!("Found DNS TX with ID {}", tx_id);
-                return Some(tx);
-            }
-        }
-        SCLogDebug!("Failed to find DNS TX with ID {}", tx_id);
-        return None;
+        return self.transactions.iter().find(|&tx| tx.id == tx_id + 1);
     }
 
     /// Set an event. The event is set on the most recent transaction.
diff --git a/rust/src/nfs/nfs.rs b/rust/src/nfs/nfs.rs
index dfb5e0e72445..e14b0114eac9 100644
--- a/rust/src/nfs/nfs.rs
+++ b/rust/src/nfs/nfs.rs
@@ -462,27 +462,11 @@ impl NFSState {
     }
 
     pub fn get_tx_by_id(&mut self, tx_id: u64) -> Option<&NFSTransaction> {
-        SCLogDebug!("get_tx_by_id: tx_id={}", tx_id);
-        for tx in &mut self.transactions {
-            if tx.id == tx_id + 1 {
-                SCLogDebug!("Found NFS TX with ID {}", tx_id);
-                return Some(tx);
-            }
-        }
-        SCLogDebug!("Failed to find NFS TX with ID {}", tx_id);
-        return None;
+        return self.transactions.iter().find(|&tx| tx.id == tx_id + 1);
     }
 
     pub fn get_tx_by_xid(&mut self, tx_xid: u32) -> Option<&mut NFSTransaction> {
-        SCLogDebug!("get_tx_by_xid: tx_xid={}", tx_xid);
-        for tx in &mut self.transactions {
-            if !tx.is_file_tx && tx.xid == tx_xid {
-                SCLogDebug!("Found NFS TX with ID {} XID {:04X}", tx.id, tx.xid);
-                return Some(tx);
-            }
-        }
-        SCLogDebug!("Failed to find NFS TX with XID {:04X}", tx_xid);
-        return None;
+        return self.transactions.iter_mut().find(|tx| !tx.is_file_tx && tx.xid == tx_xid);
     }
 
     /// Set an event. The event is set on the most recent transaction.
@@ -685,15 +669,11 @@ impl NFSState {
     }
 
     pub fn xidmap_handle2name(&mut self, xidmap: &mut NFSRequestXidMap) {
-        match self.namemap.get(&xidmap.file_handle) {
-            Some(n) => {
-                SCLogDebug!("xidmap_handle2name: name {:?}", n);
-                xidmap.file_name = n.to_vec();
-            },
-            _ => {
-                SCLogDebug!("xidmap_handle2name: object {:?} not found",
-                        xidmap.file_handle);
-            },
+        if let Some(n) = self.namemap.get(&xidmap.file_handle) {
+            SCLogDebug!("xidmap_handle2name: name {:?}", n);
+            xidmap.file_name = n.to_vec();
+        } else {
+            SCLogDebug!("xidmap_handle2name: object {:?} not found", xidmap.file_handle);
         }
     }
 
diff --git a/rust/src/smb/auth.rs b/rust/src/smb/auth.rs
index c5d20bba6e29..46f22bf7cffd 100644
--- a/rust/src/smb/auth.rs
+++ b/rust/src/smb/auth.rs
@@ -105,21 +105,20 @@ fn parse_secblob_spnego(blob: &[u8]) -> Option<SpnegoRequest>
             BerObjectContent::Sequence(ref seq) => {
                 for se in seq {
                     SCLogDebug!("SEQ {:?}", se);
-                    match se.content {
-                        BerObjectContent::OID(ref oid) => {
-                            SCLogDebug!("OID {:?}", oid);
-                            match oid.to_string().as_str() {
-                                "1.2.840.48018.1.2.2" => { SCLogDebug!("Microsoft Kerberos 5"); },
-                                "1.2.840.113554.1.2.2" => { SCLogDebug!("Kerberos 5"); have_kerberos = true; },
-                                "1.2.840.113554.1.2.2.1" => { SCLogDebug!("krb5-name"); },
-                                "1.2.840.113554.1.2.2.2" => { SCLogDebug!("krb5-principal"); },
-                                "1.2.840.113554.1.2.2.3" => { SCLogDebug!("krb5-user-to-user-mech"); },
-                                "1.3.6.1.4.1.311.2.2.10" => { SCLogDebug!("NTLMSSP"); have_ntlmssp = true; },
-                                "1.3.6.1.4.1.311.2.2.30" => { SCLogDebug!("NegoEx"); },
-                                _ => { SCLogDebug!("unexpected OID {:?}", oid); },
-                            }
-                        },
-                        _ => { SCLogDebug!("expected OID, got {:?}", se); },
+                    if let BerObjectContent::OID(ref oid) = se.content {
+                        SCLogDebug!("OID {:?}", oid);
+                        match oid.to_string().as_str() {
+                            "1.2.840.48018.1.2.2" => { SCLogDebug!("Microsoft Kerberos 5"); },
+                            "1.2.840.113554.1.2.2" => { SCLogDebug!("Kerberos 5"); have_kerberos = true; },
+                            "1.2.840.113554.1.2.2.1" => { SCLogDebug!("krb5-name"); },
+                            "1.2.840.113554.1.2.2.2" => { SCLogDebug!("krb5-principal"); },
+                            "1.2.840.113554.1.2.2.3" => { SCLogDebug!("krb5-user-to-user-mech"); },
+                            "1.3.6.1.4.1.311.2.2.10" => { SCLogDebug!("NTLMSSP"); have_ntlmssp = true; },
+                            "1.3.6.1.4.1.311.2.2.30" => { SCLogDebug!("NegoEx"); },
+                            _ => { SCLogDebug!("unexpected OID {:?}", oid); },
+                        }
+                    } else {
+                        SCLogDebug!("expected OID, got {:?}", se);
                     }
                 }
             },
diff --git a/rust/src/smb/smb.rs b/rust/src/smb/smb.rs
index d6b0a565c060..a34621746349 100644
--- a/rust/src/smb/smb.rs
+++ b/rust/src/smb/smb.rs
@@ -2025,57 +2025,51 @@ fn smb_probe_tcp_midstream(direction: Direction, slice: &[u8], rdir: *mut u8, be
     } else {
         search_smb_record(slice)
     };
-    match r {
-        Ok((_, data)) => {
-            SCLogDebug!("smb found");
-            match parse_smb_version(data) {
-                Ok((_, ref smb)) => {
-                    SCLogDebug!("SMB {:?}", smb);
-                    if smb.version == 0xff_u8 { // SMB1
-                        SCLogDebug!("SMBv1 record");
-                        if let Ok((_, ref smb_record)) = parse_smb_record(data) {
-                            if smb_record.flags & 0x80 != 0 {
-                                SCLogDebug!("RESPONSE {:02x}", smb_record.flags);
-                                if direction == Direction::ToServer {
-                                    unsafe { *rdir = Direction::ToClient as u8; }
-                                }
-                            } else {
-                                SCLogDebug!("REQUEST {:02x}", smb_record.flags);
-                                if direction == Direction::ToClient {
-                                    unsafe { *rdir = Direction::ToServer as u8; }
-                                }
-                            }
-                            return 1;
+    if let Ok((_, data)) = r {
+        SCLogDebug!("smb found");
+        if let Ok((_, ref smb)) = parse_smb_version(data) {
+            SCLogDebug!("SMB {:?}", smb);
+            if smb.version == 0xff_u8 { // SMB1
+                SCLogDebug!("SMBv1 record");
+                if let Ok((_, ref smb_record)) = parse_smb_record(data) {
+                    if smb_record.flags & 0x80 != 0 {
+                        SCLogDebug!("RESPONSE {:02x}", smb_record.flags);
+                        if direction == Direction::ToServer {
+                            unsafe { *rdir = Direction::ToClient as u8; }
                         }
-                    } else if smb.version == 0xfe_u8 { // SMB2
-                        SCLogDebug!("SMB2 record");
-                        if let Ok((_, ref smb_record)) = parse_smb2_record_direction(data) {
-                            if direction == Direction::ToServer {
-                                SCLogDebug!("direction Direction::ToServer smb_record {:?}", smb_record);
-                                if !smb_record.request {
-                                    unsafe { *rdir = Direction::ToClient as u8; }
-                                }
-                            } else {
-                                SCLogDebug!("direction Direction::ToClient smb_record {:?}", smb_record);
-                                if smb_record.request {
-                                    unsafe { *rdir = Direction::ToServer as u8; }
-                                }
-                            }
+                    } else {
+                        SCLogDebug!("REQUEST {:02x}", smb_record.flags);
+                        if direction == Direction::ToClient {
+                            unsafe { *rdir = Direction::ToServer as u8; }
                         }
                     }
-                    else if smb.version == 0xfd_u8 { // SMB3 transform
-                        SCLogDebug!("SMB3 record");
-                    }
                     return 1;
-                },
-                    _ => {
-                        SCLogDebug!("smb not found in {:?}", slice);
-                    },
+                }
+            } else if smb.version == 0xfe_u8 { // SMB2
+                SCLogDebug!("SMB2 record");
+                if let Ok((_, ref smb_record)) = parse_smb2_record_direction(data) {
+                    if direction == Direction::ToServer {
+                        SCLogDebug!("direction Direction::ToServer smb_record {:?}", smb_record);
+                        if !smb_record.request {
+                            unsafe { *rdir = Direction::ToClient as u8; }
+                        }
+                    } else {
+                        SCLogDebug!("direction Direction::ToClient smb_record {:?}", smb_record);
+                        if smb_record.request {
+                            unsafe { *rdir = Direction::ToServer as u8; }
+                        }
+                    }
+                }
+            }
+            else if smb.version == 0xfd_u8 { // SMB3 transform
+                SCLogDebug!("SMB3 record");
             }
-        },
-        _ => {
-            SCLogDebug!("no dice");
-        },
+            return 1;
+        } else {
+            SCLogDebug!("smb not found in {:?}", slice);
+        }
+    } else {
+        SCLogDebug!("no dice");
     }
     return 0;
 }
diff --git a/rust/src/smb/smb1.rs b/rust/src/smb/smb1.rs
index 9d7d47e27c85..a353c5539ccd 100644
--- a/rust/src/smb/smb1.rs
+++ b/rust/src/smb/smb1.rs
@@ -725,19 +725,16 @@ fn smb1_response_record_one(state: &mut SMBState, r: &SmbRecord, command: u8, an
                         SCLogDebug!("Create AndX {:?}", cr);
 
                         let guid_key = SMBCommonHdr::from1(r, SMBHDR_TYPE_FILENAME);
-                        match state.ssn2vec_map.remove(&guid_key) {
-                            Some(mut p) => {
-                                p.retain(|&i|i != 0x00);
-
-                                let mut fid = cr.fid.to_vec();
-                                fid.extend_from_slice(&u32_as_bytes(r.ssn_id));
-                                SCLogDebug!("SMB1_COMMAND_NT_CREATE_ANDX fid {:?}", fid);
-                                SCLogDebug!("fid {:?} name {:?}", fid, p);
-                                state.guid2name_map.insert(fid, p);
-                            },
-                            _ => {
-                                SCLogDebug!("SMBv1 response: GUID NOT FOUND");
-                            },
+                        if let Some(mut p) = state.ssn2vec_map.remove(&guid_key) {
+                            p.retain(|&i|i != 0x00);
+
+                            let mut fid = cr.fid.to_vec();
+                            fid.extend_from_slice(&u32_as_bytes(r.ssn_id));
+                            SCLogDebug!("SMB1_COMMAND_NT_CREATE_ANDX fid {:?}", fid);
+                            SCLogDebug!("fid {:?} name {:?}", fid, p);
+                            state.guid2name_map.insert(fid, p);
+                        } else {
+                            SCLogDebug!("SMBv1 response: GUID NOT FOUND");
                         }
 
                         let tx_hdr = SMBCommonHdr::from1(r, SMBHDR_TYPE_GENERICTX);
diff --git a/rust/src/smb/smb1_session.rs b/rust/src/smb/smb1_session.rs
index c39c7ce98fc9..80f23c8a64e9 100644
--- a/rust/src/smb/smb1_session.rs
+++ b/rust/src/smb/smb1_session.rs
@@ -187,16 +187,13 @@ pub fn smb1_session_setup_response(state: &mut SMBState, r: &SmbRecord, andx_off
     };
     // otherwise try match with ssn id 0 (e.g. NTLMSSP_NEGOTIATE)
     if !found {
-        match state.get_sessionsetup_tx(
+        if let Some(tx) = state.get_sessionsetup_tx(
                 SMBCommonHdr::new(SMBHDR_TYPE_HEADER, 0, 0, r.multiplex_id as u64))
         {
-            Some(tx) => {
-                smb1_session_setup_update_tx(tx, r, andx_offset);
-                SCLogDebug!("smb1_session_setup_response: tx {:?}", tx);
-            },
-            None => {
-                SCLogDebug!("smb1_session_setup_response: tx not found for {:?}", r);
-            },
+            smb1_session_setup_update_tx(tx, r, andx_offset);
+            SCLogDebug!("smb1_session_setup_response: tx {:?}", tx);
+        } else {
+            SCLogDebug!("smb1_session_setup_response: tx not found for {:?}", r);
         }
     }
 }
diff --git a/rust/src/smb/smb2_session.rs b/rust/src/smb/smb2_session.rs
index 93cc99cdd4c6..31a34165e4f2 100644
--- a/rust/src/smb/smb2_session.rs
+++ b/rust/src/smb/smb2_session.rs
@@ -70,16 +70,13 @@ pub fn smb2_session_setup_response(state: &mut SMBState, r: &Smb2Record)
     };
     // otherwise try match with ssn id 0 (e.g. NTLMSSP_NEGOTIATE)
     if !found {
-        match state.get_sessionsetup_tx(
+        if let Some(tx) = state.get_sessionsetup_tx(
                 SMBCommonHdr::new(SMBHDR_TYPE_HEADER, 0, 0, r.message_id))
         {
-            Some(tx) => {
-                smb2_session_setup_update_tx(tx, r);
-                SCLogDebug!("smb2_session_setup_response: tx {:?}", tx);
-            },
-            None => {
-                SCLogDebug!("smb2_session_setup_response: tx not found for {:?}", r);
-            },
+            smb2_session_setup_update_tx(tx, r);
+            SCLogDebug!("smb2_session_setup_response: tx {:?}", tx);
+        } else {
+            SCLogDebug!("smb2_session_setup_response: tx not found for {:?}", r);
         }
     }
 }

From d73ccd0f5203298ef75a3567dddec8e35b7c658b Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 16 Jan 2024 11:50:10 +0100
Subject: [PATCH 388/462] ci: run clippy without all features

---
 .github/workflows/rust.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 42c135657294..e6e8460733a5 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -85,3 +85,6 @@ jobs:
           fi
       - run: cargo clippy --all-features --all-targets
         working-directory: rust
+      # especially without debug feature
+      - run: cargo clippy
+        working-directory: rust

From 3b65a2bb61a0b338444730564930d0de3a2f063e Mon Sep 17 00:00:00 2001
From: Philippe Antoine <contact@catenacyber.fr>
Date: Fri, 1 Dec 2023 10:51:39 +0100
Subject: [PATCH 389/462] detect: integer keywords now support hexadecimal

So that we can write enip.revision: 0x203

Ticket: 6645
---
 rust/src/detect/uint.rs | 41 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)

diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index 8c758e3a5d69..f52c876460c6 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -17,7 +17,7 @@
 
 use nom7::branch::alt;
 use nom7::bytes::complete::{is_a, tag, tag_no_case, take_while};
-use nom7::character::complete::digit1;
+use nom7::character::complete::{digit1, hex_digit1};
 use nom7::combinator::{all_consuming, map_opt, opt, value, verify};
 use nom7::error::{make_error, ErrorKind};
 use nom7::Err;
@@ -73,8 +73,25 @@ pub fn detect_parse_uint_unit(i: &str) -> IResult<&str, u64> {
     return Ok((i, unit));
 }
 
+pub fn detect_parse_uint_value_hex<T: DetectIntType>(i: &str) -> IResult<&str, T> {
+    let (i, _) = tag("0x")(i)?;
+    let (i, arg1s) = hex_digit1(i)?;
+    match T::from_str_radix(arg1s, 16) {
+        Ok(arg1) => Ok((i, arg1)),
+        _ => Err(Err::Error(make_error(i, ErrorKind::Verify))),
+    }
+}
+
+pub fn detect_parse_uint_value<T: DetectIntType>(i: &str) -> IResult<&str, T> {
+    let (i, arg1) = alt((
+        detect_parse_uint_value_hex,
+        map_opt(digit1, |s: &str| s.parse::<T>().ok()),
+    ))(i)?;
+    Ok((i, arg1))
+}
+
 pub fn detect_parse_uint_with_unit<T: DetectIntType>(i: &str) -> IResult<&str, T> {
-    let (i, arg1) = map_opt(digit1, |s: &str| s.parse::<T>().ok())(i)?;
+    let (i, arg1) = detect_parse_uint_value::<T>(i)?;
     let (i, unit) = opt(detect_parse_uint_unit)(i)?;
     if arg1 >= T::one() {
         if let Some(u) = unit {
@@ -107,11 +124,11 @@ pub fn detect_parse_uint_start_equal<T: DetectIntType>(
 pub fn detect_parse_uint_start_interval<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
-    let (i, arg1) = map_opt(digit1, |s: &str| s.parse::<T>().ok())(i)?;
+    let (i, arg1) = detect_parse_uint_value(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
     let (i, _) = alt((tag("-"), tag("<>")))(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
-    let (i, arg2) = verify(map_opt(digit1, |s: &str| s.parse::<T>().ok()), |x| {
+    let (i, arg2) = verify(detect_parse_uint_value, |x| {
         x > &arg1 && *x - arg1 > T::one()
     })(i)?;
     Ok((
@@ -127,13 +144,13 @@ pub fn detect_parse_uint_start_interval<T: DetectIntType>(
 fn detect_parse_uint_start_interval_inclusive<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
-    let (i, arg1) = verify(map_opt(digit1, |s: &str| s.parse::<T>().ok()), |x| {
+    let (i, arg1) = verify(detect_parse_uint_value::<T>, |x| {
         *x > T::min_value()
     })(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
     let (i, _) = alt((tag("-"), tag("<>")))(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
-    let (i, arg2) = verify(map_opt(digit1, |s: &str| s.parse::<T>().ok()), |x| {
+    let (i, arg2) = verify(detect_parse_uint_value::<T>, |x| {
         *x > arg1 && *x < T::max_value()
     })(i)?;
     Ok((
@@ -162,7 +179,7 @@ pub fn detect_parse_uint_mode(i: &str) -> IResult<&str, DetectUintMode> {
 fn detect_parse_uint_start_symbol<T: DetectIntType>(i: &str) -> IResult<&str, DetectUintData<T>> {
     let (i, mode) = detect_parse_uint_mode(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
-    let (i, arg1) = map_opt(digit1, |s: &str| s.parse::<T>().ok())(i)?;
+    let (i, arg1) = detect_parse_uint_value(i)?;
 
     match mode {
         DetectUintMode::DetectUintModeNe => {}
@@ -407,6 +424,16 @@ pub unsafe extern "C" fn rs_detect_u16_free(ctx: &mut DetectUintData<u16>) {
 mod tests {
     use super::*;
 
+    #[test]
+    fn test_parse_uint_hex() {
+        let (_, val) = detect_parse_uint::<u64>("0x100").unwrap();
+        assert_eq!(val.arg1, 0x100);
+        let (_, val) = detect_parse_uint::<u8>("0xFF").unwrap();
+        assert_eq!(val.arg1, 255);
+        let (_, val) = detect_parse_uint::<u8>("0xff").unwrap();
+        assert_eq!(val.arg1, 255);
+    }
+
     #[test]
     fn test_parse_uint_unit() {
         let (_, val) = detect_parse_uint::<u64>(" 2kb").unwrap();

From 06c5dd3133077b5f79927b987e62e07fc173d2fb Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 30 Nov 2023 15:18:20 +0100
Subject: [PATCH 390/462] detect: integer keywords now accept negated ranges

Ticket: 6646
---
 rust/src/detect/uint.rs | 36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index f52c876460c6..0d813bfd0883 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -17,7 +17,7 @@
 
 use nom7::branch::alt;
 use nom7::bytes::complete::{is_a, tag, tag_no_case, take_while};
-use nom7::character::complete::{digit1, hex_digit1};
+use nom7::character::complete::{char, digit1, hex_digit1};
 use nom7::combinator::{all_consuming, map_opt, opt, value, verify};
 use nom7::error::{make_error, ErrorKind};
 use nom7::Err;
@@ -35,6 +35,7 @@ pub enum DetectUintMode {
     DetectUintModeGte,
     DetectUintModeRange,
     DetectUintModeNe,
+    DetectUintModeNegRg,
 }
 
 #[derive(Debug)]
@@ -124,6 +125,7 @@ pub fn detect_parse_uint_start_equal<T: DetectIntType>(
 pub fn detect_parse_uint_start_interval<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
+    let (i, neg) = opt(char('!'))(i)?;
     let (i, arg1) = detect_parse_uint_value(i)?;
     let (i, _) = opt(is_a(" "))(i)?;
     let (i, _) = alt((tag("-"), tag("<>")))(i)?;
@@ -131,12 +133,17 @@ pub fn detect_parse_uint_start_interval<T: DetectIntType>(
     let (i, arg2) = verify(detect_parse_uint_value, |x| {
         x > &arg1 && *x - arg1 > T::one()
     })(i)?;
+    let mode = if neg.is_some() {
+        DetectUintMode::DetectUintModeNegRg
+    } else {
+        DetectUintMode::DetectUintModeRange
+    };
     Ok((
         i,
         DetectUintData {
             arg1,
             arg2,
-            mode: DetectUintMode::DetectUintModeRange,
+            mode,
         },
     ))
 }
@@ -144,6 +151,7 @@ pub fn detect_parse_uint_start_interval<T: DetectIntType>(
 fn detect_parse_uint_start_interval_inclusive<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
+    let (i, neg) = opt(char('!'))(i)?;
     let (i, arg1) = verify(detect_parse_uint_value::<T>, |x| {
         *x > T::min_value()
     })(i)?;
@@ -153,12 +161,17 @@ fn detect_parse_uint_start_interval_inclusive<T: DetectIntType>(
     let (i, arg2) = verify(detect_parse_uint_value::<T>, |x| {
         *x > arg1 && *x < T::max_value()
     })(i)?;
+    let mode = if neg.is_some() {
+        DetectUintMode::DetectUintModeNegRg
+    } else {
+        DetectUintMode::DetectUintModeRange
+    };
     Ok((
         i,
         DetectUintData {
             arg1: arg1 - T::one(),
             arg2: arg2 + T::one(),
-            mode: DetectUintMode::DetectUintModeRange,
+            mode,
         },
     ))
 }
@@ -255,6 +268,11 @@ pub fn detect_match_uint<T: DetectIntType>(x: &DetectUintData<T>, val: T) -> boo
                 return true;
             }
         }
+        DetectUintMode::DetectUintModeNegRg => {
+            if val <= x.arg1 || val >= x.arg2 {
+                return true;
+            }
+        }
     }
     return false;
 }
@@ -434,6 +452,18 @@ mod tests {
         assert_eq!(val.arg1, 255);
     }
 
+    #[test]
+    fn test_parse_uint_negated_range() {
+        let (_, val) = detect_parse_uint::<u8>("!1-6").unwrap();
+        assert_eq!(val.arg1, 1);
+        assert_eq!(val.arg2, 6);
+        assert_eq!(val.mode, DetectUintMode::DetectUintModeNegRg);
+        assert!(detect_match_uint(&val, 1));
+        assert!(!detect_match_uint(&val, 2));
+        assert!(!detect_match_uint(&val, 5));
+        assert!(detect_match_uint(&val, 6));
+    }
+
     #[test]
     fn test_parse_uint_unit() {
         let (_, val) = detect_parse_uint::<u64>(" 2kb").unwrap();

From 370ac0541984791978f7e92db2d90f9e9ade6ec5 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 22 Dec 2023 11:59:35 +0100
Subject: [PATCH 391/462] detect/integer: rust derive for enumerations

Ticket: 6647

Allows keywords using integers to use strings in signature
parsing based on a rust enumeration with a derive.
---
 rust/derive/src/lib.rs        | 16 ++++++
 rust/derive/src/stringenum.rs | 96 +++++++++++++++++++++++++++++++++++
 rust/src/detect/mod.rs        | 43 ++++++++++++++++
 rust/src/detect/uint.rs       | 45 ++++++++++++++++
 4 files changed, 200 insertions(+)
 create mode 100644 rust/derive/src/stringenum.rs

diff --git a/rust/derive/src/lib.rs b/rust/derive/src/lib.rs
index a2b7a6ad0442..a36f19390c0c 100644
--- a/rust/derive/src/lib.rs
+++ b/rust/derive/src/lib.rs
@@ -23,6 +23,7 @@ use proc_macro::TokenStream;
 
 mod applayerevent;
 mod applayerframetype;
+mod stringenum;
 
 /// The `AppLayerEvent` derive macro generates a `AppLayerEvent` trait
 /// implementation for enums that define AppLayerEvents.
@@ -50,3 +51,18 @@ pub fn derive_app_layer_event(input: TokenStream) -> TokenStream {
 pub fn derive_app_layer_frame_type(input: TokenStream) -> TokenStream {
     applayerframetype::derive_app_layer_frame_type(input)
 }
+
+#[proc_macro_derive(EnumStringU8, attributes(name))]
+pub fn derive_enum_string_u8(input: TokenStream) -> TokenStream {
+    stringenum::derive_enum_string::<u8>(input, "u8")
+}
+
+#[proc_macro_derive(EnumStringU16, attributes(name))]
+pub fn derive_enum_string_u16(input: TokenStream) -> TokenStream {
+    stringenum::derive_enum_string::<u16>(input, "u16")
+}
+
+#[proc_macro_derive(EnumStringU32, attributes(name))]
+pub fn derive_enum_string_u32(input: TokenStream) -> TokenStream {
+    stringenum::derive_enum_string::<u32>(input, "u32")
+}
diff --git a/rust/derive/src/stringenum.rs b/rust/derive/src/stringenum.rs
new file mode 100644
index 000000000000..5344b934cde6
--- /dev/null
+++ b/rust/derive/src/stringenum.rs
@@ -0,0 +1,96 @@
+/* Copyright (C) 2023 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+extern crate proc_macro;
+use super::applayerevent::transform_name;
+use proc_macro::TokenStream;
+use quote::quote;
+use syn::{self, parse_macro_input, DeriveInput};
+use std::str::FromStr;
+
+pub fn derive_enum_string<T: std::str::FromStr + quote::ToTokens>(input: TokenStream, ustr: &str) -> TokenStream where <T as FromStr>::Err: std::fmt::Display {
+    let input = parse_macro_input!(input as DeriveInput);
+    let name = input.ident;
+    let mut values = Vec::new();
+    let mut names = Vec::new();
+    let mut fields = Vec::new();
+
+    if let syn::Data::Enum(ref data) = input.data {
+        for v in (&data.variants).into_iter() {
+            if let Some((_, val)) = &v.discriminant {
+                let fname = transform_name(&v.ident.to_string());
+                names.push(fname);
+                fields.push(v.ident.clone());
+                if let syn::Expr::Lit(l) = val {
+                    if let syn::Lit::Int(li) = &l.lit {
+                        if let Ok(value) = li.base10_parse::<T>() {
+                            values.push(value);
+                        } else {
+                            panic!("EnumString requires explicit {}", ustr);
+                        }
+                    } else {
+                        panic!("EnumString requires explicit literal integer");
+                    }
+                } else {
+                    panic!("EnumString requires explicit literal");
+                }
+            } else {
+                panic!("EnumString requires explicit values");
+            }
+        }
+    } else {
+        panic!("EnumString can only be derived for enums");
+    }
+
+    let is_suricata = std::env::var("CARGO_PKG_NAME").map(|var| var == "suricata").unwrap_or(false);
+    let crate_id = if is_suricata {
+        syn::Ident::new("crate", proc_macro2::Span::call_site())
+    } else {
+        syn::Ident::new("suricata", proc_macro2::Span::call_site())
+    };
+
+    let utype_str = syn::Ident::new(ustr, proc_macro2::Span::call_site());
+
+    let expanded = quote! {
+        impl #crate_id::detect::EnumString<#utype_str> for #name {
+            fn from_u(v: #utype_str) -> Option<Self> {
+                match v {
+                    #( #values => Some(#name::#fields) ,)*
+                    _ => None,
+                }
+            }
+            fn into_u(self) -> #utype_str {
+                match self {
+                    #( #name::#fields => #values ,)*
+                }
+            }
+            fn to_str(&self) -> &'static str {
+                match *self {
+                    #( #name::#fields => #names ,)*
+                }
+            }
+            fn from_str(s: &str) -> Option<Self> {
+                match s {
+                    #( #names => Some(#name::#fields) ,)*
+                    _ => None
+                }
+            }
+        }
+    };
+
+    proc_macro::TokenStream::from(expanded)
+}
diff --git a/rust/src/detect/mod.rs b/rust/src/detect/mod.rs
index d33c9ae7fabf..cad086f161b5 100644
--- a/rust/src/detect/mod.rs
+++ b/rust/src/detect/mod.rs
@@ -25,3 +25,46 @@ pub mod stream_size;
 pub mod uint;
 pub mod uri;
 pub mod requires;
+
+/// EnumString trait that will be implemented on enums that
+/// derive StringEnum.
+pub trait EnumString<T> {
+    /// Return the enum variant of the given numeric value.
+    fn from_u(v: T) -> Option<Self> where Self: Sized;
+
+    /// Convert the enum variant to the numeric value.
+    fn into_u(self) -> T;
+
+    /// Return the string for logging the enum value.
+    fn to_str(&self) -> &'static str;
+
+    /// Get an enum variant from parsing a string.
+    fn from_str(s: &str) -> Option<Self> where Self: Sized;
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use suricata_derive::EnumStringU8;
+
+    #[derive(Clone, Debug, PartialEq, EnumStringU8)]
+    #[repr(u8)]
+    pub enum TestEnum {
+        Zero = 0,
+        BestValueEver = 42,
+    }
+
+    #[test]
+    fn test_enum_string_u8() {
+        assert_eq!(TestEnum::from_u(0), Some(TestEnum::Zero));
+        assert_eq!(TestEnum::from_u(1), None);
+        assert_eq!(TestEnum::from_u(42), Some(TestEnum::BestValueEver));
+        assert_eq!(TestEnum::Zero.into_u(), 0);
+        assert_eq!(TestEnum::BestValueEver.into_u(), 42);
+        assert_eq!(TestEnum::Zero.to_str(), "zero");
+        assert_eq!(TestEnum::BestValueEver.to_str(), "best_value_ever");
+        assert_eq!(TestEnum::from_str("zero"), Some(TestEnum::Zero));
+        assert_eq!(TestEnum::from_str("nope"), None);
+        assert_eq!(TestEnum::from_str("best_value_ever"), Some(TestEnum::BestValueEver));
+    }
+}
diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index 0d813bfd0883..fd6079a53635 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -23,6 +23,8 @@ use nom7::error::{make_error, ErrorKind};
 use nom7::Err;
 use nom7::IResult;
 
+use super::EnumString;
+
 use std::ffi::CStr;
 
 #[derive(PartialEq, Eq, Clone, Debug)]
@@ -46,6 +48,29 @@ pub struct DetectUintData<T> {
     pub mode: DetectUintMode,
 }
 
+/// Parses a string for detection with integers, using enumeration strings
+///
+/// Needs to specify T1 the integer type (like u8)
+/// And the Enumeration for the stringer.
+/// Will try to parse numerical value first, as any integer detection keyword
+/// And if this fails, will resort to using the enumeration strings.
+///
+/// Returns Some DetectUintData on success, None on failure
+pub fn detect_parse_uint_enum<T1: DetectIntType, T2: EnumString<T1>>(s: &str) -> Option<DetectUintData<T1>> {
+    if let Ok((_, ctx)) = detect_parse_uint::<T1>(s) {
+        return Some(ctx);
+    }
+    if let Some(enum_val) = T2::from_str(s) {
+        let ctx = DetectUintData::<T1> {
+            arg1: enum_val.into_u(),
+            arg2: T1::min_value(),
+            mode: DetectUintMode::DetectUintModeEqual,
+        };
+        return Some(ctx);
+    }
+    return None;
+}
+
 pub trait DetectIntType:
     std::str::FromStr
     + std::cmp::PartialOrd
@@ -442,6 +467,26 @@ pub unsafe extern "C" fn rs_detect_u16_free(ctx: &mut DetectUintData<u16>) {
 mod tests {
     use super::*;
 
+    use suricata_derive::EnumStringU8;
+
+    #[derive(Clone, Debug, PartialEq, EnumStringU8)]
+    #[repr(u8)]
+    pub enum TestEnum {
+        Zero = 0,
+        BestValueEver = 42,
+    }
+
+    #[test]
+    fn test_detect_parse_uint_enum() {
+        let ctx = detect_parse_uint_enum::<u8, TestEnum>("best_value_ever").unwrap();
+        assert_eq!(ctx.arg1, 42);
+        assert_eq!(ctx.mode, DetectUintMode::DetectUintModeEqual);
+
+        let ctx = detect_parse_uint_enum::<u8, TestEnum>(">1").unwrap();
+        assert_eq!(ctx.arg1, 1);
+        assert_eq!(ctx.mode, DetectUintMode::DetectUintModeGt);
+    }
+
     #[test]
     fn test_parse_uint_hex() {
         let (_, val) = detect_parse_uint::<u64>("0x100").unwrap();

From d05f3ac791c25f789040e677cfa09c7e3fad7d60 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Sat, 30 Dec 2023 21:46:54 +0100
Subject: [PATCH 392/462] detect: integer keywords now accept bitmasks

Ticket: 6648

Like &0x40=0x40 to test for a specific bit set
---
 rust/src/detect/uint.rs | 62 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index fd6079a53635..7ce86d57f3e2 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -38,6 +38,8 @@ pub enum DetectUintMode {
     DetectUintModeRange,
     DetectUintModeNe,
     DetectUintModeNegRg,
+    DetectUintModeBitmask,
+    DetectUintModeNegBitmask,
 }
 
 #[derive(Debug)]
@@ -173,6 +175,37 @@ pub fn detect_parse_uint_start_interval<T: DetectIntType>(
     ))
 }
 
+pub fn detect_parse_uint_bitmask<T: DetectIntType>(
+    i: &str,
+) -> IResult<&str, DetectUintData<T>> {
+    let (i, _) = opt(is_a(" "))(i)?;
+    let (i, _) = tag("&")(i)?;
+    let (i, _) = opt(is_a(" "))(i)?;
+    let (i, arg1) = detect_parse_uint_value(i)?;
+    let (i, _) = opt(is_a(" "))(i)?;
+    let (i, neg) = opt(tag("!"))(i)?;
+    let (i, _) = tag("=")(i)?;
+    let (i, _) = opt(is_a(" "))(i)?;
+    let (i, arg2) = detect_parse_uint_value(i)?;
+    if arg2 & arg1 != arg2 {
+        // could never match
+        return Err(Err::Error(make_error(i, ErrorKind::Verify)));
+    }
+    let mode = if neg.is_none() {
+        DetectUintMode::DetectUintModeBitmask
+    } else {
+        DetectUintMode::DetectUintModeNegBitmask
+    };
+    Ok((
+        i,
+        DetectUintData {
+            arg1,
+            arg2,
+            mode,
+        },
+    ))
+}
+
 fn detect_parse_uint_start_interval_inclusive<T: DetectIntType>(
     i: &str,
 ) -> IResult<&str, DetectUintData<T>> {
@@ -298,6 +331,16 @@ pub fn detect_match_uint<T: DetectIntType>(x: &DetectUintData<T>, val: T) -> boo
                 return true;
             }
         }
+        DetectUintMode::DetectUintModeBitmask => {
+            if val & x.arg1 == x.arg2 {
+                return true;
+            }
+        }
+        DetectUintMode::DetectUintModeNegBitmask => {
+            if val & x.arg1 != x.arg2 {
+                return true;
+            }
+        }
     }
     return false;
 }
@@ -305,6 +348,7 @@ pub fn detect_match_uint<T: DetectIntType>(x: &DetectUintData<T>, val: T) -> boo
 pub fn detect_parse_uint_notending<T: DetectIntType>(i: &str) -> IResult<&str, DetectUintData<T>> {
     let (i, _) = opt(is_a(" "))(i)?;
     let (i, uint) = alt((
+        detect_parse_uint_bitmask,
         detect_parse_uint_start_interval,
         detect_parse_uint_start_equal,
         detect_parse_uint_start_symbol,
@@ -487,6 +531,24 @@ mod tests {
         assert_eq!(ctx.mode, DetectUintMode::DetectUintModeGt);
     }
 
+    #[test]
+    fn test_parse_uint_bitmask() {
+        let (_, val) = detect_parse_uint::<u64>("&0x40!=0").unwrap();
+        assert_eq!(val.arg1, 0x40);
+        assert_eq!(val.arg2, 0);
+        assert_eq!(val.mode, DetectUintMode::DetectUintModeNegBitmask);
+        assert!(!detect_match_uint(&val, 0xBF));
+        assert!(detect_match_uint(&val, 0x40));
+        let (_, val) = detect_parse_uint::<u64>("&0xc0=0x80").unwrap();
+        assert_eq!(val.arg1, 0xc0);
+        assert_eq!(val.arg2, 0x80);
+        assert_eq!(val.mode, DetectUintMode::DetectUintModeBitmask);
+        assert!(detect_match_uint(&val, 0x80));
+        assert!(!detect_match_uint(&val, 0x40));
+        assert!(!detect_match_uint(&val, 0xc0));
+        // could never match
+        assert!(detect_parse_uint::<u64>("&0xc0=12").is_err());
+    }
     #[test]
     fn test_parse_uint_hex() {
         let (_, val) = detect_parse_uint::<u64>("0x100").unwrap();

From b8bc2c7e0fab56af649c962b57c316570a4adb0e Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 4 Jan 2024 11:00:51 +0100
Subject: [PATCH 393/462] doc: integer keywords

Ticket: 6628

Document the generic detection capabilities for integer keywords.
and make every integer keyword pointing to this section.
---
 doc/userguide/rules/dhcp-keywords.rst    |  6 ++
 doc/userguide/rules/file-keywords.rst    |  2 +
 doc/userguide/rules/flow-keywords.rst    | 10 +++
 doc/userguide/rules/header-keywords.rst  | 10 +++
 doc/userguide/rules/http-keywords.rst    |  2 +
 doc/userguide/rules/http2-keywords.rst   |  6 ++
 doc/userguide/rules/ike-keywords.rst     |  6 ++
 doc/userguide/rules/index.rst            |  1 +
 doc/userguide/rules/integer-keywords.rst | 77 ++++++++++++++++++++++++
 doc/userguide/rules/mqtt-keywords.rst    |  2 +
 doc/userguide/rules/payload-keywords.rst |  4 ++
 doc/userguide/rules/rfb-keywords.rst     |  2 +
 doc/userguide/rules/tls-keywords.rst     |  2 +
 13 files changed, 130 insertions(+)
 create mode 100644 doc/userguide/rules/integer-keywords.rst

diff --git a/doc/userguide/rules/dhcp-keywords.rst b/doc/userguide/rules/dhcp-keywords.rst
index 05675a947e73..cc4ca1001e83 100644
--- a/doc/userguide/rules/dhcp-keywords.rst
+++ b/doc/userguide/rules/dhcp-keywords.rst
@@ -6,6 +6,8 @@ dhcp.leasetime
 
 DHCP lease time (integer).
 
+dhcp.leasetime uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  dhcp.leasetime:[op]<number>
@@ -25,6 +27,8 @@ dhcp.rebinding_time
 
 DHCP rebinding time (integer).
 
+dhcp.rebinding_time uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  dhcp.rebinding_time:[op]<number>
@@ -44,6 +48,8 @@ dhcp.renewal_time
 
 DHCP renewal time (integer).
 
+dhcp.renewal_time uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  dhcp.renewal_time:[op]<number>
diff --git a/doc/userguide/rules/file-keywords.rst b/doc/userguide/rules/file-keywords.rst
index c708ee746c0d..91637576852a 100644
--- a/doc/userguide/rules/file-keywords.rst
+++ b/doc/userguide/rules/file-keywords.rst
@@ -244,6 +244,8 @@ filesize
 
 Match on the size of the file as it is being transferred.
 
+filesize uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
   filesize:<value>;
diff --git a/doc/userguide/rules/flow-keywords.rst b/doc/userguide/rules/flow-keywords.rst
index 6d451ce82aab..41b3f8d514e0 100644
--- a/doc/userguide/rules/flow-keywords.rst
+++ b/doc/userguide/rules/flow-keywords.rst
@@ -292,6 +292,8 @@ flow.age
 Flow age in seconds (integer)
 This keyword does not wait for the end of the flow, but will be checked at each packet.
 
+flow.age uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  flow.age: [op]<number>
@@ -314,6 +316,8 @@ flow.pkts_toclient
 Flow number of packets to client (integer)
 This keyword does not wait for the end of the flow, but will be checked at each packet.
 
+flow.pkts_toclient uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  flow.pkts_toclient: [op]<number>
@@ -334,6 +338,8 @@ flow.pkts_toserver
 Flow number of packets to server (integer)
 This keyword does not wait for the end of the flow, but will be checked at each packet.
 
+flow.pkts_toserver uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  flow.pkts_toserver: [op]<number>
@@ -354,6 +360,8 @@ flow.bytes_toclient
 Flow number of bytes to client (integer)
 This keyword does not wait for the end of the flow, but will be checked at each packet.
 
+flow.bytes_toclient uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  flow.bytes_toclient: [op]<number>
@@ -374,6 +382,8 @@ flow.bytes_toserver
 Flow number of bytes to server (integer)
 This keyword does not wait for the end of the flow, but will be checked at each packet.
 
+flow.bytes_toserver uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 Syntax::
 
  flow.bytes_toserver: [op]<number>
diff --git a/doc/userguide/rules/header-keywords.rst b/doc/userguide/rules/header-keywords.rst
index 36d1437647f3..e28b14e28315 100644
--- a/doc/userguide/rules/header-keywords.rst
+++ b/doc/userguide/rules/header-keywords.rst
@@ -15,6 +15,8 @@ For example::
 
   ttl:10;
 
+ttl uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 At the end of the ttl keyword you can enter the value on which you
 want to match. The Time-to-live value determines the maximal amount
 of time a packet can be in the Internet-system. If this field is set
@@ -431,6 +433,8 @@ tcp.mss
 Match on the TCP MSS option value. Will not match if the option is not
 present.
 
+tcp.mss uses an :ref:`unsigned 16-bit integer <rules-integer-keywords>`.
+
 The format of the keyword::
 
   tcp.mss:<min>-<max>;
@@ -506,6 +510,8 @@ messages. The different messages are distinct by different names, but
 more important by numeric values. For more information see the table
 with message-types and codes.
 
+itype uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 The format of the itype keyword::
 
   itype:min<>max;
@@ -565,6 +571,8 @@ code of a ICMP message clarifies the message. Together with the
 ICMP-type it indicates with what kind of problem you are dealing with.
 A code has a different purpose with every ICMP-type.
 
+icode uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 The format of the icode keyword::
 
   icode:min<>max;
@@ -719,6 +727,8 @@ icmpv6.mtu
 Match on the ICMPv6 MTU optional value. Will not match if the MTU is not
 present.
 
+icmpv6.mtu uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 The format of the keyword::
 
   icmpv6.mtu:<min>-<max>;
diff --git a/doc/userguide/rules/http-keywords.rst b/doc/userguide/rules/http-keywords.rst
index ba0d7621f339..04f4093ddbb1 100644
--- a/doc/userguide/rules/http-keywords.rst
+++ b/doc/userguide/rules/http-keywords.rst
@@ -237,6 +237,8 @@ The ``urilen`` keyword is used to match on the length of the request
 URI. It is possible to use the ``<`` and ``>`` operators, which
 indicate respectively *smaller than* and *larger than*.
 
+urilen uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 The format of ``urilen`` is::
 
   urilen:3;
diff --git a/doc/userguide/rules/http2-keywords.rst b/doc/userguide/rules/http2-keywords.rst
index 1ad83554c6ef..c4761151bcb8 100644
--- a/doc/userguide/rules/http2-keywords.rst
+++ b/doc/userguide/rules/http2-keywords.rst
@@ -31,6 +31,8 @@ http2.priority
 
 Match on the value of the HTTP2 priority field present in a PRIORITY or HEADERS frame.
 
+http2.priority uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
@@ -49,6 +51,8 @@ http2.window
 
 Match on the value of the HTTP2 value field present in a WINDOWUPDATE frame.
 
+http2.window uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
@@ -68,6 +72,8 @@ Match on the size of the HTTP2 Dynamic Headers Table.
 More information on the protocol can be found here:
 `<https://tools.ietf.org/html/rfc7541#section-6.3>`_
 
+http2.size_update uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
diff --git a/doc/userguide/rules/ike-keywords.rst b/doc/userguide/rules/ike-keywords.rst
index e0d9557bc306..38d78954de74 100644
--- a/doc/userguide/rules/ike-keywords.rst
+++ b/doc/userguide/rules/ike-keywords.rst
@@ -61,6 +61,8 @@ ike.exchtype
 
 Match on the value of the Exchange Type.
 
+ike.exchtype uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
@@ -106,6 +108,8 @@ ike.key_exchange_payload_length
 
 Match against the length of the public key exchange payload (e.g. Diffie-Hellman) of the server or client.
 
+ike.key_exchange_payload_length uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
@@ -138,6 +142,8 @@ ike.nonce_payload_length
 
 Match against the length of the nonce of the server or client.
 
+ike.nonce_payload_length uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
diff --git a/doc/userguide/rules/index.rst b/doc/userguide/rules/index.rst
index e174c6787bc5..2450f4486be9 100644
--- a/doc/userguide/rules/index.rst
+++ b/doc/userguide/rules/index.rst
@@ -7,6 +7,7 @@ Suricata Rules
    meta
    header-keywords
    payload-keywords
+   integer-keywords
    transforms
    prefilter-keywords
    flow-keywords
diff --git a/doc/userguide/rules/integer-keywords.rst b/doc/userguide/rules/integer-keywords.rst
new file mode 100644
index 000000000000..c70b8b5f1860
--- /dev/null
+++ b/doc/userguide/rules/integer-keywords.rst
@@ -0,0 +1,77 @@
+.. _rules-integer-keywords:
+
+Integer Keywords
+================
+
+Many keywords will match on an integer value on the network traffic.
+These are unsigned integers that can be 8, 16, 32 or 64 bits.
+
+Simple example::
+
+    bsize:integer value;
+
+The integer value can be written as base-10 like ``100`` or as 
+an hexadecimal value like ``0x64``.
+
+The most direct example is to match for equality, but there are
+different modes.
+
+Comparison modes
+----------------
+
+Integers can be matched for
+* Equality
+* Inequality
+* Greater than
+* Less than
+* Range
+* Negated range
+* Bitmask
+* Negated Bitmask
+
+.. note::
+
+    Comparisons are strict by default. Ranges are thus exclusive.
+    That means a range between 1 and 4 will match 2 and 3, but neither 1 nor 4.
+    Negated range !1-4 will match for 1 or below and for 4 or above.
+
+Examples::
+
+    bsize:19; # equality
+    bsize:=0x13; # equality
+    bsize:!0x14; # inequality
+    bsize:!=20; # inequality
+    bsize:>21; # greater than
+    bsize:>=21; # greater than or equal
+    bsize:<22; # lesser than
+    bsize:<=22; # lesser than or equal
+    bsize:19-22; # range between value1 and value2
+    bsize:!19-22; # negated range between value1 and value2
+    bsize:&0xc0=0x80; # bitmask mask is compared to value for equality
+    bsize:&0xc0!=0; # bitmask mask is compared to value for inequality
+
+Enumerations
+------------
+
+Some integers on the wire represent an enumeration, that is, some values
+have a string/meaning associated to it.
+Rules can be written using one of these strings to check for equality.
+This is meant to make rules more human-readable and equivalent for matching.
+
+Examples::
+
+    websocket.opcode:text;
+    websocket.opcode:1; # behaves the same
+
+Bitmasks
+--------
+
+Some integers on the wire represent multiple bits.
+Some of these bits have a string/meaning associated to it.
+Rules can be written using a list (comma-separated) of these strings,
+where each item can be negated.
+
+Examples::
+
+    websocket.flags:fin,!comp;
+    websocket.flags:&0xc0=0x80; # behaves the same
diff --git a/doc/userguide/rules/mqtt-keywords.rst b/doc/userguide/rules/mqtt-keywords.rst
index 058a17b7ffde..36133b20840b 100644
--- a/doc/userguide/rules/mqtt-keywords.rst
+++ b/doc/userguide/rules/mqtt-keywords.rst
@@ -8,6 +8,8 @@ mqtt.protocol_version
 
 Match on the value of the MQTT protocol version field in the fixed header.
 
+mqtt.protocol_version uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 The format of the keyword::
 
   mqtt.protocol_version:<min>-<max>;
diff --git a/doc/userguide/rules/payload-keywords.rst b/doc/userguide/rules/payload-keywords.rst
index 9a609a217f04..bc2bc42d0829 100644
--- a/doc/userguide/rules/payload-keywords.rst
+++ b/doc/userguide/rules/payload-keywords.rst
@@ -280,6 +280,8 @@ bsize
 With the ``bsize`` keyword, you can match on the length of the buffer. This adds
 precision to the content match, previously this could have been done with ``isdataat``.
 
+bsize uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
+
 An optional operator can be specified; if no operator is present, the operator will
 default to '='. When a relational operator is used, e.g., '<', '>' or '<>' (range),
 the bsize value will be compared using the relational operator. Ranges are inclusive.
@@ -336,6 +338,8 @@ This may be convenient in detecting buffer overflows.
 
 dsize cannot be used when using app/streamlayer protocol keywords (i.e. http.uri)
 
+dsize uses an :ref:`unsigned 16-bit integer <rules-integer-keywords>`.
+
 Format::
 
   dsize:[<>!]number; || dsize:min<>max;
diff --git a/doc/userguide/rules/rfb-keywords.rst b/doc/userguide/rules/rfb-keywords.rst
index 628b3d85c563..1715143daa7b 100644
--- a/doc/userguide/rules/rfb-keywords.rst
+++ b/doc/userguide/rules/rfb-keywords.rst
@@ -36,6 +36,8 @@ rfb.sectype
 
 Match on the value of the RFB security type field, e.g. ``2`` for VNC challenge-response authentication, ``0`` for no authentication, and ``30`` for Apple's custom Remote Desktop authentication.
 
+rfb.sectype uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 This keyword takes a numeric argument after a colon and supports additional qualifiers, such as:
 
 * ``>`` (greater than)
diff --git a/doc/userguide/rules/tls-keywords.rst b/doc/userguide/rules/tls-keywords.rst
index dc28c97cd583..a6d1bd6dbec8 100644
--- a/doc/userguide/rules/tls-keywords.rst
+++ b/doc/userguide/rules/tls-keywords.rst
@@ -284,6 +284,8 @@ tls.cert_chain_len
 
 Matches on the TLS certificate chain length.
 
+tls.cert_chain_len uses an :ref:`unsigned 32-bit integer <rules-integer-keywords>`.
+
 tls.cert_chain_len supports `<, >, <>, !` and using an exact value.
 
 Example::

From 244a35d539989c00eb5cbac86e7355227cd9da03 Mon Sep 17 00:00:00 2001
From: Juliana Fajardini <jufajardini@oisf.net>
Date: Wed, 24 Jan 2024 12:00:41 -0300
Subject: [PATCH 394/462] userguide: fix explanation about bsize ranges

Our code handles Uint ranges as exclusive, but for bsize, our
documentation stated that they're inclusive.

Cf. from uint.rs:

    DetectUintMode::DetectUintModeRange => {
        if val > x.arg1 && val < x.arg2 {
            return true;
        }
    }

Task #6708
---
 doc/userguide/rules/payload-keywords.rst | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/rules/payload-keywords.rst b/doc/userguide/rules/payload-keywords.rst
index bc2bc42d0829..aa7d5c007332 100644
--- a/doc/userguide/rules/payload-keywords.rst
+++ b/doc/userguide/rules/payload-keywords.rst
@@ -284,7 +284,7 @@ bsize uses an :ref:`unsigned 64-bit integer <rules-integer-keywords>`.
 
 An optional operator can be specified; if no operator is present, the operator will
 default to '='. When a relational operator is used, e.g., '<', '>' or '<>' (range),
-the bsize value will be compared using the relational operator. Ranges are inclusive.
+the bsize value will be compared using the relational operator. Ranges are exclusive.
 
 If one or more ``content`` keywords precedes ``bsize``, each occurrence of ``content``
 will be inspected and an error will be raised if the content length and the bsize
@@ -327,6 +327,9 @@ Examples of ``bsize`` in a rule:
 
    alert dns any any -> any any (msg:"test bsize rule"; dns.query; content:"middle"; bsize:6<>15; sid:126; rev:1;)
 
+To emphasize how range works: in the example above, a match will occur if
+``bsize`` is greater than 6 and less than 15.
+
 dsize
 -----
 

From 8fc0faf5c2622cd0685e07fbfd42ccc89aa08266 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 15 Jan 2024 12:39:34 +0530
Subject: [PATCH 395/462] util/streaming-buffer: remove unneeded fn param

StreamingBuffer is not required to find the intersecting regions, so,
don't pass it as a param to the fn.
---
 src/util-streaming-buffer.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/util-streaming-buffer.c b/src/util-streaming-buffer.c
index 6ff4f438a40a..b396ef04c765 100644
--- a/src/util-streaming-buffer.c
+++ b/src/util-streaming-buffer.c
@@ -123,7 +123,7 @@ StreamingBufferBlock *SBB_RB_FIND_INCLUSIVE(struct SBB *head, StreamingBufferBlo
  *  \brief does data region intersect with list region 'r'
  *  Takes the max gap into account.
  */
-static inline bool RegionsIntersect(const StreamingBuffer *sb, const StreamingBufferConfig *cfg,
+static inline bool RegionsIntersect(const StreamingBufferConfig *cfg,
         const StreamingBufferRegion *r, const uint64_t offset, const uint64_t re)
 {
     /* create the data range for the region, adding the max gap */
@@ -158,7 +158,7 @@ static StreamingBufferRegion *FindFirstRegionForOffset(const StreamingBuffer *sb
 
     StreamingBufferRegion *p = NULL;
     for (; r != NULL; r = r->next) {
-        if (RegionsIntersect(sb, cfg, r, offset, data_re) == true) {
+        if (RegionsIntersect(cfg, r, offset, data_re) == true) {
             *prev = p;
             return r;
         }
@@ -182,7 +182,7 @@ static StreamingBufferRegion *FindLargestRegionForOffset(const StreamingBuffer *
         SCLogDebug("checking: %p/%" PRIu64 "/%" PRIu64 ", offset %" PRIu64 "/%" PRIu64, r,
                 r->stream_offset, reg_re, offset, data_re);
 #endif
-        if (!RegionsIntersect(sb, cfg, r, offset, data_re))
+        if (!RegionsIntersect(cfg, r, offset, data_re))
             return candidate;
 
         if (r->buf_size > candidate->buf_size) {
@@ -200,7 +200,7 @@ static StreamingBufferRegion *FindRightEdge(const StreamingBuffer *sb,
     const uint64_t data_re = offset + len;
     StreamingBufferRegion *candidate = r;
     for (; r != NULL; r = r->next) {
-        if (!RegionsIntersect(sb, cfg, r, offset, data_re)) {
+        if (!RegionsIntersect(cfg, r, offset, data_re)) {
             SCLogDebug(
                     "r %p is out of scope: %" PRIu64 "/%u/%" PRIu64, r, offset, len, offset + len);
             return candidate;
@@ -1433,11 +1433,11 @@ static StreamingBufferRegion *BufferInsertAtRegion(StreamingBuffer *sb,
             data_offset + data_len);
     ListRegions(sb);
 
-    if (RegionsIntersect(sb, cfg, &sb->region, data_offset, data_offset + data_len)) {
+    if (RegionsIntersect(cfg, &sb->region, data_offset, data_offset + data_len)) {
         SCLogDebug("data_offset %" PRIu64 ", data_len %u intersects with main region (next %p)",
                 data_offset, data_len, sb->region.next);
         if (sb->region.next == NULL ||
-                !RegionsIntersect(sb, cfg, sb->region.next, data_offset, data_offset + data_len)) {
+                !RegionsIntersect(cfg, sb->region.next, data_offset, data_offset + data_len)) {
             SCLogDebug(
                     "data_offset %" PRIu64
                     ", data_len %u intersects with main region, no next or way before next region",

From f6e1a202159720adc70e90bc413d3d3ae40cff6e Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 30 Nov 2023 14:32:08 +0100
Subject: [PATCH 396/462] detect: dns.opcode as first-class integer

Ticket: 5446

That means it can accept ranges
---
 doc/userguide/rules/dns-keywords.rst |  11 ++
 rust/src/detect/uint.rs              |   2 +-
 rust/src/dns/detect.rs               | 167 +++++++++------------------
 src/detect-dns-opcode.c              |   5 +-
 4 files changed, 67 insertions(+), 118 deletions(-)

diff --git a/doc/userguide/rules/dns-keywords.rst b/doc/userguide/rules/dns-keywords.rst
index a514ae25195b..005164dfd159 100644
--- a/doc/userguide/rules/dns-keywords.rst
+++ b/doc/userguide/rules/dns-keywords.rst
@@ -28,12 +28,15 @@ dns.opcode
 
 This keyword matches on the **opcode** found in the DNS header flags.
 
+dns.opcode uses an :ref:`unsigned 8-bit integer <rules-integer-keywords>`.
+
 Syntax
 ~~~~~~
 
 ::
 
    dns.opcode:[!]<number>
+   dns.opcode:[!]<number1>-<number2>
 
 Examples
 ~~~~~~~~
@@ -46,6 +49,14 @@ Match on DNS requests where the **opcode** is NOT 0::
 
   dns.opcode:!0;
 
+Match on DNS requests where the **opcode** is between 7 and 15, exclusively:
+
+  dns.opcode:7-15;
+
+Match on DNS requests where the **opcode** is not between 7 and 15:
+
+  dns.opcode:!7-15;
+
 dns.query
 ---------
 
diff --git a/rust/src/detect/uint.rs b/rust/src/detect/uint.rs
index 7ce86d57f3e2..5b28830cea6a 100644
--- a/rust/src/detect/uint.rs
+++ b/rust/src/detect/uint.rs
@@ -42,7 +42,7 @@ pub enum DetectUintMode {
     DetectUintModeNegBitmask,
 }
 
-#[derive(Debug)]
+#[derive(Debug, PartialEq)]
 #[repr(C)]
 pub struct DetectUintData<T> {
     pub arg1: T,
diff --git a/rust/src/dns/detect.rs b/rust/src/dns/detect.rs
index 5d9d945be0ce..452d4e8380e1 100644
--- a/rust/src/dns/detect.rs
+++ b/rust/src/dns/detect.rs
@@ -16,49 +16,15 @@
  */
 
 use super::dns::DNSTransaction;
-use crate::core::*;
-use std::ffi::CStr;
-use std::os::raw::{c_char, c_void};
-
-#[derive(Debug, PartialEq, Eq)]
-pub struct DetectDnsOpcode {
-    negate: bool,
-    opcode: u8,
-}
-
-/// Parse a DNS opcode argument returning the code and if it is to be
-/// negated or not.
-///
-/// For now only an indication that an error occurred is returned, not
-/// the details of the error.
-fn parse_opcode(opcode: &str) -> Result<DetectDnsOpcode, ()> {
-    let mut negated = false;
-    for (i, c) in opcode.chars().enumerate() {
-        match c {
-            ' ' | '\t' => {
-                continue;
-            }
-            '!' => {
-                negated = true;
-            }
-            _ => {
-                let code: u8 = opcode[i..].parse().map_err(|_| ())?;
-                return Ok(DetectDnsOpcode {
-                    negate: negated,
-                    opcode: code,
-                });
-            }
-        }
-    }
-    Err(())
-}
+use crate::core::Direction;
+use crate::detect::uint::{detect_match_uint, DetectUintData};
 
 /// Perform the DNS opcode match.
 ///
 /// 1 will be returned on match, otherwise 0 will be returned.
 #[no_mangle]
 pub extern "C" fn rs_dns_opcode_match(
-    tx: &mut DNSTransaction, detect: &mut DetectDnsOpcode, flags: u8,
+    tx: &mut DNSTransaction, detect: &mut DetectUintData<u8>, flags: u8,
 ) -> u8 {
     let header_flags = if flags & Direction::ToServer as u8 != 0 {
         if let Some(request) = &tx.request {
@@ -76,116 +42,87 @@ pub extern "C" fn rs_dns_opcode_match(
         // Not to server or to client??
         return 0;
     };
+    let opcode = ((header_flags >> 11) & 0xf) as u8;
 
-    match_opcode(detect, header_flags).into()
-}
-
-fn match_opcode(detect: &DetectDnsOpcode, flags: u16) -> bool {
-    let opcode = ((flags >> 11) & 0xf) as u8;
-    if detect.negate {
-        detect.opcode != opcode
-    } else {
-        detect.opcode == opcode
-    }
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn rs_detect_dns_opcode_parse(carg: *const c_char) -> *mut c_void {
-    if carg.is_null() {
-        return std::ptr::null_mut();
-    }
-    let arg = match CStr::from_ptr(carg).to_str() {
-        Ok(arg) => arg,
-        _ => {
-            return std::ptr::null_mut();
-        }
-    };
-
-    match parse_opcode(arg) {
-        Ok(detect) => Box::into_raw(Box::new(detect)) as *mut _,
-        Err(_) => std::ptr::null_mut(),
-    }
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn rs_dns_detect_opcode_free(ptr: *mut c_void) {
-    if !ptr.is_null() {
-        std::mem::drop(Box::from_raw(ptr as *mut DetectDnsOpcode));
+    if detect_match_uint(detect, opcode) {
+        return 1;
     }
+    return 0;
 }
 
 #[cfg(test)]
 mod test {
     use super::*;
+    use crate::detect::uint::{detect_parse_uint, DetectUintMode};
 
     #[test]
     fn parse_opcode_good() {
         assert_eq!(
-            parse_opcode("1"),
-            Ok(DetectDnsOpcode {
-                negate: false,
-                opcode: 1
-            })
-        );
-        assert_eq!(
-            parse_opcode("123"),
-            Ok(DetectDnsOpcode {
-                negate: false,
-                opcode: 123
-            })
+            detect_parse_uint::<u8>("1").unwrap().1,
+            DetectUintData {
+                mode: DetectUintMode::DetectUintModeEqual,
+                arg1: 1,
+                arg2: 0,
+            }
         );
         assert_eq!(
-            parse_opcode("!123"),
-            Ok(DetectDnsOpcode {
-                negate: true,
-                opcode: 123
-            })
+            detect_parse_uint::<u8>("123").unwrap().1,
+            DetectUintData {
+                mode: DetectUintMode::DetectUintModeEqual,
+                arg1: 123,
+                arg2: 0,
+            }
         );
         assert_eq!(
-            parse_opcode("!123"),
-            Ok(DetectDnsOpcode {
-                negate: true,
-                opcode: 123
-            })
+            detect_parse_uint::<u8>("!123").unwrap().1,
+            DetectUintData {
+                mode: DetectUintMode::DetectUintModeNe,
+                arg1: 123,
+                arg2: 0,
+            }
         );
-        assert_eq!(parse_opcode(""), Err(()));
-        assert_eq!(parse_opcode("!"), Err(()));
-        assert_eq!(parse_opcode("!   "), Err(()));
-        assert_eq!(parse_opcode("!asdf"), Err(()));
+        assert!(detect_parse_uint::<u8>("").is_err());
+        assert!(detect_parse_uint::<u8>("!").is_err());
+        assert!(detect_parse_uint::<u8>("!   ").is_err());
+        assert!(detect_parse_uint::<u8>("!asdf").is_err());
     }
 
     #[test]
     fn test_match_opcode() {
-        assert!(match_opcode(
-            &DetectDnsOpcode {
-                negate: false,
-                opcode: 0,
+        assert!(detect_match_uint(
+            &DetectUintData {
+                mode: DetectUintMode::DetectUintModeEqual,
+                arg1: 0,
+                arg2: 0,
             },
             0b0000_0000_0000_0000,
         ));
 
-        assert!(!match_opcode(
-            &DetectDnsOpcode {
-                negate: true,
-                opcode: 0,
+        assert!(!detect_match_uint(
+            &DetectUintData {
+                mode: DetectUintMode::DetectUintModeNe,
+                arg1: 0,
+                arg2: 0,
             },
             0b0000_0000_0000_0000,
         ));
 
-        assert!(match_opcode(
-            &DetectDnsOpcode {
-                negate: false,
-                opcode: 4,
+        assert!(detect_match_uint(
+            &DetectUintData {
+                mode: DetectUintMode::DetectUintModeEqual,
+                arg1: 4,
+                arg2: 0,
             },
-            0b0010_0000_0000_0000,
+            ((0b0010_0000_0000_0000 >> 11) & 0xf) as u8,
         ));
 
-        assert!(!match_opcode(
-            &DetectDnsOpcode {
-                negate: true,
-                opcode: 4,
+        assert!(!detect_match_uint(
+            &DetectUintData {
+                mode: DetectUintMode::DetectUintModeNe,
+                arg1: 4,
+                arg2: 0,
             },
-            0b0010_0000_0000_0000,
+            ((0b0010_0000_0000_0000 >> 11) & 0xf) as u8,
         ));
     }
 }
diff --git a/src/detect-dns-opcode.c b/src/detect-dns-opcode.c
index 4baee19b8cd3..f5dcab700f27 100644
--- a/src/detect-dns-opcode.c
+++ b/src/detect-dns-opcode.c
@@ -19,6 +19,7 @@
 
 #include "detect-parse.h"
 #include "detect-engine.h"
+#include "detect-engine-uint.h"
 #include "detect-dns-opcode.h"
 #include "rust.h"
 
@@ -35,7 +36,7 @@ static int DetectDnsOpcodeSetup(DetectEngineCtx *de_ctx, Signature *s,
         return -1;
     }
 
-    void *detect = rs_detect_dns_opcode_parse(str);
+    void *detect = DetectU8Parse(str);
     if (detect == NULL) {
         SCLogError("failed to parse dns.opcode: %s", str);
         return -1;
@@ -57,7 +58,7 @@ static void DetectDnsOpcodeFree(DetectEngineCtx *de_ctx, void *ptr)
 {
     SCEnter();
     if (ptr != NULL) {
-        rs_dns_detect_opcode_free(ptr);
+        rs_detect_u8_free(ptr);
     }
     SCReturn;
 }

From 6de885c60379225e0b636a5a8df07fd2cfe6193d Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 30 Jan 2024 20:42:16 +0100
Subject: [PATCH 397/462] ci: update scorecard analysis workflow

---
 .github/workflows/scorecards-analysis.yml | 29 ++++++++++-------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/.github/workflows/scorecards-analysis.yml b/.github/workflows/scorecards-analysis.yml
index 3b6612849206..07d4eda8121f 100644
--- a/.github/workflows/scorecards-analysis.yml
+++ b/.github/workflows/scorecards-analysis.yml
@@ -17,39 +17,36 @@ jobs:
     permissions:
       # Needed to upload the results to code-scanning dashboard.
       security-events: write
-      actions: read
-      contents: read
+      id-token: write
 
     steps:
       - name: "Checkout code"
-        uses: actions/checkout@v3.5.3
-        with:
-          persist-credentials: false
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
 
       - name: "Run analysis"
-        uses: ossf/scorecard-action@e38b1902ae4f44df626f11ba0734b14fb91f8f86 # v1.0.1
+        uses: ossf/scorecard-action@0864cf19026789058feabb7e87baa5f140aac736 # v2.3.1
         with:
           results_file: results.sarif
           results_format: sarif
-          # Read-only PAT token. To create it,
-          # follow the steps in https://github.com/ossf/scorecard-action#pat-token-creation.
           repo_token: ${{ secrets.SCORECARD_READ_TOKEN }}
-          # Publish the results to enable scorecard badges. For more details, see
-          # https://github.com/ossf/scorecard-action#publishing-results.
-          # For private repositories, `publish_results` will automatically be set to `false`,
-          # regardless of the value entered here.
+          # Scorecard team runs a weekly scan of public GitHub repos,
+          # see https://github.com/ossf/scorecard#public-data.
+          # Setting `publish_results: true` helps us scale by leveraging your workflow to
+          # extract the results instead of relying on our own infrastructure to run scans.
+          # And it's free for you!
           publish_results: true
 
-      # Upload the results as artifacts (optional).
+      # https://docs.github.com/en/actions/advanced-guides/storing-workflow-data-as-artifacts
+      # Optional.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+        uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v3
         with:
           name: SARIF file
           path: results.sarif
           retention-days: 5
 
       # Upload the results to GitHub's code scanning dashboard.
-      - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@v2 # v1.0.26
+      - name: "Upload SARIF results"
+        uses: github/codeql-action/upload-sarif@cdcdbb579706841c47f7063dda365e292e5cad7a # v1
         with:
           sarif_file: results.sarif

From 264101ba22a3e28ffb2293e15e324412f97d5126 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 1 Feb 2024 15:20:15 +0530
Subject: [PATCH 398/462] detect: remove unused port in SigGroupHeadInitData

port is not used and logically makes sense to not be in this struct as
this struct is already referenced by DetectPort itself as a part of
SigGroupHead.
---
 src/detect.h | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/src/detect.h b/src/detect.h
index 0fc5d21fb1d8..e7cc0dfe935a 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1434,9 +1434,6 @@ typedef struct SigGroupHeadInitData_ {
 
     /** Array with sig ptrs... size is sig_cnt * sizeof(Signature *) */
     Signature **match_array;
-
-    /* port ptr */
-    struct DetectPort_ *port;
 } SigGroupHeadInitData;
 
 /** \brief Container for matching data for a signature group */

From 395c74d81eec59b07181c0015c6760845a9494e7 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 1 Feb 2024 16:44:33 +0530
Subject: [PATCH 399/462] detect/engine: set max sig ID per SGH

Present scenario
----------------
Currently, as a part of setting signature count per SGH, a max_idx is
passed which could be as high as the highest signature number (internal
ID).

Issue
-----
Not every SGH needs to evaluate all the signatures while setting
the signature count or while creating the match_array.
In a nonideal scenario, when say, there are 2 SGHs and one SGH has 2
signatures and the other one has 60k, given the current scheme of
evaluating max_idx, the max_idx will be set to 60k, and this shall
later be passed on to SigGroupHeadSetSigCnt or
SigGroupHeadBuildMatchArra which shall traverse over all the 60k sigs
for either SGHs.

Other info
----------
This is a very fast operation as the internal arithmetic is done
bitwise.

Patch
-----
The functions SigGroupHeadSetSigCnt and SigGroupHeadBuildMatchArray can
be optimized by storing the max signature id (internal) per SGH (which
also seemed to be the initial intention as per fn comments).
As a result of this, the sig_array is only walked up until the max sig
id of that respective SGH.
---
 src/detect-engine-build.c    | 23 ++++++++++++-----------
 src/detect-engine-siggroup.c | 11 +++++++----
 src/detect.h                 |  1 +
 3 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index 604164b21c15..b99776bd13f3 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -1009,8 +1009,8 @@ static int RulesGroupByProto(DetectEngineCtx *de_ctx)
         if (lookup_sgh == NULL) {
             SCLogDebug("proto group %d sgh %p is the original", p, sgh_ts[p]);
 
-            SigGroupHeadSetSigCnt(sgh_ts[p], max_idx);
-            SigGroupHeadBuildMatchArray(de_ctx, sgh_ts[p], max_idx);
+            SigGroupHeadSetSigCnt(sgh_ts[p], 0);
+            SigGroupHeadBuildMatchArray(de_ctx, sgh_ts[p], 0);
 
             SigGroupHeadHashAdd(de_ctx, sgh_ts[p]);
             SigGroupHeadStore(de_ctx, sgh_ts[p]);
@@ -1041,8 +1041,8 @@ static int RulesGroupByProto(DetectEngineCtx *de_ctx)
         if (lookup_sgh == NULL) {
             SCLogDebug("proto group %d sgh %p is the original", p, sgh_tc[p]);
 
-            SigGroupHeadSetSigCnt(sgh_tc[p], max_idx);
-            SigGroupHeadBuildMatchArray(de_ctx, sgh_tc[p], max_idx);
+            SigGroupHeadSetSigCnt(sgh_tc[p], 0);
+            SigGroupHeadBuildMatchArray(de_ctx, sgh_tc[p], 0);
 
             SigGroupHeadHashAdd(de_ctx, sgh_tc[p]);
             SigGroupHeadStore(de_ctx, sgh_tc[p]);
@@ -1129,7 +1129,8 @@ static int RuleSetWhitelist(Signature *s)
     return wl;
 }
 
-int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, DetectPort **newhead, uint32_t unique_groups, int (*CompareFunc)(DetectPort *, DetectPort *), uint32_t max_idx);
+int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, DetectPort **newhead,
+        uint32_t unique_groups, int (*CompareFunc)(DetectPort *, DetectPort *));
 int CreateGroupedPortListCmpCnt(DetectPort *a, DetectPort *b);
 
 static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, uint32_t direction)
@@ -1223,7 +1224,7 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
     DetectPort *newlist = NULL;
     uint16_t groupmax = (direction == SIG_FLAG_TOCLIENT) ? de_ctx->max_uniq_toclient_groups :
                                                            de_ctx->max_uniq_toserver_groups;
-    CreateGroupedPortList(de_ctx, list, &newlist, groupmax, CreateGroupedPortListCmpCnt, max_idx);
+    CreateGroupedPortList(de_ctx, list, &newlist, groupmax, CreateGroupedPortListCmpCnt);
     list = newlist;
 
     /* step 4: deduplicate the SGH's */
@@ -1243,8 +1244,8 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
         if (lookup_sgh == NULL) {
             SCLogDebug("port group %p sgh %p is the original", iter, iter->sh);
 
-            SigGroupHeadSetSigCnt(iter->sh, max_idx);
-            SigGroupHeadBuildMatchArray(de_ctx, iter->sh, max_idx);
+            SigGroupHeadSetSigCnt(iter->sh, 0);
+            SigGroupHeadBuildMatchArray(de_ctx, iter->sh, 0);
             SigGroupHeadSetProtoAndDirection(iter->sh, ipproto, direction);
             SigGroupHeadHashAdd(de_ctx, iter->sh);
             SigGroupHeadStore(de_ctx, iter->sh);
@@ -1541,7 +1542,8 @@ int CreateGroupedPortListCmpCnt(DetectPort *a, DetectPort *b)
  *  The joingr is meant to be a catch all.
  *
  */
-int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, DetectPort **newhead, uint32_t unique_groups, int (*CompareFunc)(DetectPort *, DetectPort *), uint32_t max_idx)
+int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, DetectPort **newhead,
+        uint32_t unique_groups, int (*CompareFunc)(DetectPort *, DetectPort *))
 {
     DetectPort *tmplist = NULL, *joingr = NULL;
     char insert = 0;
@@ -1560,8 +1562,7 @@ int CreateGroupedPortList(DetectEngineCtx *de_ctx, DetectPort *port_list, Detect
         list->next = NULL;
 
         groups++;
-
-        SigGroupHeadSetSigCnt(list->sh, max_idx);
+        SigGroupHeadSetSigCnt(list->sh, 0);
 
         /* insert it */
         DetectPort *tmpgr = tmplist, *prevtmpgr = NULL;
diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index 4fadaac5c013..9bc992cb894a 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -347,7 +347,7 @@ int SigGroupHeadAppendSig(const DetectEngineCtx *de_ctx, SigGroupHead **sgh,
 
     /* enable the sig in the bitarray */
     (*sgh)->init->sig_array[s->num / 8] |= 1 << (s->num % 8);
-
+    (*sgh)->init->max_sig_id = MAX(s->num, (*sgh)->init->max_sig_id);
     return 0;
 
 error:
@@ -405,6 +405,8 @@ int SigGroupHeadCopySigs(DetectEngineCtx *de_ctx, SigGroupHead *src, SigGroupHea
     if (src->init->score)
         (*dst)->init->score = MAX((*dst)->init->score, src->init->score);
 
+    if (src->init->max_sig_id)
+        (*dst)->init->max_sig_id = MAX((*dst)->init->max_sig_id, src->init->max_sig_id);
     return 0;
 
 error:
@@ -422,9 +424,9 @@ int SigGroupHeadCopySigs(DetectEngineCtx *de_ctx, SigGroupHead *src, SigGroupHea
 void SigGroupHeadSetSigCnt(SigGroupHead *sgh, uint32_t max_idx)
 {
     uint32_t sig;
-
+    sgh->init->max_sig_id = MAX(max_idx, sgh->init->max_sig_id);
     sgh->init->sig_cnt = 0;
-    for (sig = 0; sig < max_idx + 1; sig++) {
+    for (sig = 0; sig < sgh->init->max_sig_id + 1; sig++) {
         if (sgh->init->sig_array[sig / 8] & (1 << (sig % 8)))
             sgh->init->sig_cnt++;
     }
@@ -492,12 +494,13 @@ int SigGroupHeadBuildMatchArray(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
         return 0;
 
     BUG_ON(sgh->init->match_array != NULL);
+    sgh->init->max_sig_id = MAX(sgh->init->max_sig_id, max_idx);
 
     sgh->init->match_array = SCCalloc(sgh->init->sig_cnt, sizeof(Signature *));
     if (sgh->init->match_array == NULL)
         return -1;
 
-    for (sig = 0; sig < max_idx + 1; sig++) {
+    for (sig = 0; sig < sgh->init->max_sig_id + 1; sig++) {
         if (!(sgh->init->sig_array[(sig / 8)] & (1 << (sig % 8))) )
             continue;
 
diff --git a/src/detect.h b/src/detect.h
index e7cc0dfe935a..0707d8a5b211 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1419,6 +1419,7 @@ typedef struct SigGroupHeadInitData_ {
     uint8_t protos[256];    /**< proto(s) this sgh is for */
     uint32_t direction;     /**< set to SIG_FLAG_TOSERVER, SIG_FLAG_TOCLIENT or both */
     int score;              /**< try to make this group a unique one */
+    uint32_t max_sig_id;    /**< max signature idx for this sgh */
 
     MpmCtx **app_mpms;
     MpmCtx **pkt_mpms;

From 7f89aaf772e65bc289528b59a7aee166c7c84f43 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 1 Feb 2024 16:58:48 +0530
Subject: [PATCH 400/462] detect: remove unneeded max_idx

---
 src/detect-engine-build.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/src/detect-engine-build.c b/src/detect-engine-build.c
index b99776bd13f3..0c07150e696e 100644
--- a/src/detect-engine-build.c
+++ b/src/detect-engine-build.c
@@ -960,7 +960,6 @@ static int RulesGroupByProto(DetectEngineCtx *de_ctx)
 {
     Signature *s = de_ctx->sig_list;
 
-    uint32_t max_idx = 0;
     SigGroupHead *sgh_ts[256] = {NULL};
     SigGroupHead *sgh_tc[256] = {NULL};
 
@@ -979,15 +978,12 @@ static int RulesGroupByProto(DetectEngineCtx *de_ctx)
 
             if (s->flags & SIG_FLAG_TOCLIENT) {
                 SigGroupHeadAppendSig(de_ctx, &sgh_tc[p], s);
-                max_idx = s->num;
             }
             if (s->flags & SIG_FLAG_TOSERVER) {
                 SigGroupHeadAppendSig(de_ctx, &sgh_ts[p], s);
-                max_idx = s->num;
             }
         }
     }
-    SCLogDebug("max_idx %u", max_idx);
 
     /* lets look at deduplicating this list */
     SigGroupHeadHashFree(de_ctx);
@@ -1140,7 +1136,6 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
      *         that belong to the SGH. */
     DetectPortHashInit(de_ctx);
 
-    uint32_t max_idx = 0;
     const Signature *s = de_ctx->sig_list;
     DetectPort *list = NULL;
     while (s) {
@@ -1199,7 +1194,6 @@ static DetectPort *RulesGroupByPorts(DetectEngineCtx *de_ctx, uint8_t ipproto, u
 
             p = p->next;
         }
-        max_idx = s->num;
     next:
         s = s->next;
     }

From db99c45d239d5ca6e805094195f7ae39d3051e44 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <contact@catenacyber.fr>
Date: Mon, 9 Oct 2023 14:49:54 +0200
Subject: [PATCH 401/462] detect: errors on 65k filestore signatures

Errors when a detection engine gets 65k filestore signatures to
avoid the hard limit to have 65k filestore per signature group
head

Ticket: #6393
---
 src/detect-engine-siggroup.c | 3 +++
 src/detect-filestore.c       | 6 ++++++
 src/detect.h                 | 3 +++
 3 files changed, 12 insertions(+)

diff --git a/src/detect-engine-siggroup.c b/src/detect-engine-siggroup.c
index 9bc992cb894a..52073cf0bfda 100644
--- a/src/detect-engine-siggroup.c
+++ b/src/detect-engine-siggroup.c
@@ -48,6 +48,7 @@
 
 #include "util-error.h"
 #include "util-debug.h"
+#include "util-validate.h"
 #include "util-cidr.h"
 #include "util-unittest.h"
 #include "util-unittest-helper.h"
@@ -552,6 +553,8 @@ void SigGroupHeadSetupFiles(const DetectEngineCtx *de_ctx, SigGroupHead *sgh)
         }
 #endif
         if (SignatureIsFilestoring(s)) {
+            // should be insured by caller that we do not overflow
+            DEBUG_VALIDATE_BUG_ON(sgh->filestore_cnt == UINT16_MAX);
             sgh->filestore_cnt++;
         }
     }
diff --git a/src/detect-filestore.c b/src/detect-filestore.c
index 07bbd91ff199..c510544469aa 100644
--- a/src/detect-filestore.c
+++ b/src/detect-filestore.c
@@ -333,6 +333,11 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
     static bool warn_not_configured = false;
     static uint32_t de_version = 0;
 
+    if (de_ctx->filestore_cnt == UINT16_MAX) {
+        SCLogError("Cannot have more than 65535 filestore signatures");
+        return -1;
+    }
+
     /* Check on first-time loads (includes following a reload) */
     if (!warn_not_configured || (de_ctx->version != de_version)) {
         if (de_version != de_ctx->version) {
@@ -466,6 +471,7 @@ static int DetectFilestoreSetup (DetectEngineCtx *de_ctx, Signature *s, const ch
     }
 
     s->flags |= SIG_FLAG_FILESTORE;
+    de_ctx->filestore_cnt++;
 
     if (match)
         pcre2_match_data_free(match);
diff --git a/src/detect.h b/src/detect.h
index 0707d8a5b211..76c6d2b66f03 100644
--- a/src/detect.h
+++ b/src/detect.h
@@ -1040,6 +1040,9 @@ typedef struct DetectEngineCtx_ {
 
     /* Track rule requirements for reporting after loading rules. */
     SCDetectRequiresStatus *requirements;
+
+    /* number of signatures using filestore, limited as u16 */
+    uint16_t filestore_cnt;
 } DetectEngineCtx;
 
 /* Engine groups profiles (low, medium, high, custom) */

From aff54f29f8c3f583ae0524a661aa90dc7a2d3f92 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 25 Jan 2024 16:01:14 +0100
Subject: [PATCH 402/462] http2: handle reassembly for continuation frames

Ticket: 5926

HTTP2 continuation frames are defined in RFC 9113.
They allow header blocks to be split over multiple HTTP2 frames.
For Suricata to process correctly these header blocks, it
must do the reassembly of the payload of these HTTP2 frames.
Otherwise, we get incomplete decoding for headers names and/or
values while decoding a single frame.

Design is to add a field to the HTTP2 state, as the RFC states that
these continuation frames form a discrete unit :
> Field blocks MUST be transmitted as a contiguous sequence of frames,
> with no interleaved frames of any other type or from any other stream.
So, we do not have to duplicate this reassembly field per stream id.

Another design choice is to wait for the reassembly to be complete
before doing any decoding, to avoid quadratic complexity on partially
decoding of the data.
---
 rules/http2-events.rules |  1 +
 rust/src/http2/http2.rs  | 86 ++++++++++++++++++++++++++++++++++++++--
 suricata.yaml.in         |  2 +
 3 files changed, 85 insertions(+), 4 deletions(-)

diff --git a/rules/http2-events.rules b/rules/http2-events.rules
index 7cceaf24c307..413fdd652cad 100644
--- a/rules/http2-events.rules
+++ b/rules/http2-events.rules
@@ -20,3 +20,4 @@ alert http2 any any -> any any (msg:"SURICATA HTTP2 variable-length integer over
 alert http2 any any -> any any (msg:"SURICATA HTTP2 too many streams"; flow:established; app-layer-event:http2.too_many_streams; classtype:protocol-command-decode; sid:2290012; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 authority host mismatch"; flow:established,to_server; app-layer-event:http2.authority_host_mismatch; classtype:protocol-command-decode; sid:2290013; rev:1;)
 alert http2 any any -> any any (msg:"SURICATA HTTP2 user info in uri"; flow:established,to_server; app-layer-event:http2.userinfo_in_uri; classtype:protocol-command-decode; sid:2290014; rev:1;)
+alert http2 any any -> any any (msg:"SURICATA HTTP2 reassembly limit reached"; flow:established; app-layer-event:http2.reassembly_limit_reached; classtype:protocol-command-decode; sid:2290015; rev:1;)
diff --git a/rust/src/http2/http2.rs b/rust/src/http2/http2.rs
index 14d7b47dfb03..047b41402e12 100644
--- a/rust/src/http2/http2.rs
+++ b/rust/src/http2/http2.rs
@@ -61,6 +61,8 @@ const HTTP2_FRAME_RSTSTREAM_LEN: usize = 4;
 const HTTP2_FRAME_PRIORITY_LEN: usize = 5;
 const HTTP2_FRAME_WINDOWUPDATE_LEN: usize = 4;
 pub static mut HTTP2_MAX_TABLESIZE: u32 = 65536; // 0x10000
+// maximum size of reassembly for header + continuation
+static mut HTTP2_MAX_REASS: usize = 102400;
 static mut HTTP2_MAX_STREAMS: usize = 4096; // 0x1000
 
 #[repr(u8)]
@@ -406,6 +408,7 @@ pub enum HTTP2Event {
     TooManyStreams,
     AuthorityHostMismatch,
     UserinfoInUri,
+    ReassemblyLimitReached,
 }
 
 pub struct HTTP2DynTable {
@@ -432,6 +435,12 @@ impl HTTP2DynTable {
     }
 }
 
+#[derive(Default)]
+struct HTTP2HeaderReassemblyBuffer {
+    data: Vec<u8>,
+    stream_id: u32,
+}
+
 pub struct HTTP2State {
     state_data: AppLayerStateData,
     tx_id: u64,
@@ -441,6 +450,9 @@ pub struct HTTP2State {
     dynamic_headers_tc: HTTP2DynTable,
     transactions: VecDeque<HTTP2Transaction>,
     progress: HTTP2ConnectionState,
+
+    c2s_buf: HTTP2HeaderReassemblyBuffer,
+    s2c_buf: HTTP2HeaderReassemblyBuffer,
 }
 
 impl State<HTTP2Transaction> for HTTP2State {
@@ -473,6 +485,8 @@ impl HTTP2State {
             dynamic_headers_tc: HTTP2DynTable::new(),
             transactions: VecDeque::new(),
             progress: HTTP2ConnectionState::Http2StateInit,
+            c2s_buf: HTTP2HeaderReassemblyBuffer::default(),
+            s2c_buf: HTTP2HeaderReassemblyBuffer::default(),
         }
     }
 
@@ -686,8 +700,11 @@ impl HTTP2State {
     }
 
     fn parse_frame_data(
-        &mut self, ftype: u8, input: &[u8], complete: bool, hflags: u8, dir: Direction,
+        &mut self, head: &parser::HTTP2FrameHeader, input: &[u8], complete: bool, dir: Direction,
+        reass_limit_reached: &mut bool,
     ) -> HTTP2FrameTypeData {
+        let ftype = head.ftype;
+        let hflags = head.flags;
         match num::FromPrimitive::from_u8(ftype) {
             Some(parser::HTTP2FrameType::GoAway) => {
                 if input.len() < HTTP2_FRAME_GOAWAY_LEN {
@@ -847,17 +864,47 @@ impl HTTP2State {
                 return HTTP2FrameTypeData::DATA;
             }
             Some(parser::HTTP2FrameType::Continuation) => {
+                let buf = if dir == Direction::ToClient {
+                    &mut self.s2c_buf
+                } else {
+                    &mut self.c2s_buf
+                };
+                if head.stream_id == buf.stream_id {
+                    let max_reass = unsafe { HTTP2_MAX_REASS };
+                    if buf.data.len() + input.len() < max_reass {
+                        buf.data.extend(input);
+                    } else if buf.data.len() < max_reass {
+                        buf.data.extend(&input[..max_reass - buf.data.len()]);
+                        *reass_limit_reached = true;
+                    }
+                    if head.flags & parser::HTTP2_FLAG_HEADER_END_HEADERS == 0 {
+                        let hs = parser::HTTP2FrameContinuation {
+                            blocks: Vec::new(),
+                        };
+                        return HTTP2FrameTypeData::CONTINUATION(hs);
+                    }
+                } // else try to parse anyways
+                let input_reass = if head.stream_id == buf.stream_id { &buf.data } else { input };
+
                 let dyn_headers = if dir == Direction::ToClient {
                     &mut self.dynamic_headers_tc
                 } else {
                     &mut self.dynamic_headers_ts
                 };
-                match parser::http2_parse_frame_continuation(input, dyn_headers) {
+                match parser::http2_parse_frame_continuation(input_reass, dyn_headers) {
                     Ok((_, hs)) => {
+                        if head.stream_id == buf.stream_id {
+                            buf.stream_id = 0;
+                            buf.data.clear();
+                        }
                         self.process_headers(&hs.blocks, dir);
                         return HTTP2FrameTypeData::CONTINUATION(hs);
                     }
                     Err(Err::Incomplete(_)) => {
+                        if head.stream_id == buf.stream_id {
+                            buf.stream_id = 0;
+                            buf.data.clear();
+                        }
                         if complete {
                             self.set_event(HTTP2Event::InvalidFrameData);
                             return HTTP2FrameTypeData::UNHANDLED(HTTP2FrameUnhandled {
@@ -870,6 +917,10 @@ impl HTTP2State {
                         }
                     }
                     Err(_) => {
+                        if head.stream_id == buf.stream_id {
+                            buf.stream_id = 0;
+                            buf.data.clear();
+                        }
                         self.set_event(HTTP2Event::InvalidFrameData);
                         return HTTP2FrameTypeData::UNHANDLED(HTTP2FrameUnhandled {
                             reason: HTTP2FrameUnhandledReason::ParsingError,
@@ -878,6 +929,22 @@ impl HTTP2State {
                 }
             }
             Some(parser::HTTP2FrameType::Headers) => {
+                if head.flags & parser::HTTP2_FLAG_HEADER_END_HEADERS == 0 {
+                    let buf = if dir == Direction::ToClient {
+                        &mut self.s2c_buf
+                    } else {
+                        &mut self.c2s_buf
+                    };
+                    buf.data.clear();
+                    buf.data.extend(input);
+                    buf.stream_id = head.stream_id;
+                    let hs = parser::HTTP2FrameHeaders {
+                        padlength: None,
+                        priority: None,
+                        blocks: Vec::new(),
+                    };
+                    return HTTP2FrameTypeData::HEADERS(hs);
+                }
                 let dyn_headers = if dir == Direction::ToClient {
                     &mut self.dynamic_headers_tc
                 } else {
@@ -961,15 +1028,19 @@ impl HTTP2State {
                         input = &rem[hlsafe..];
                         continue;
                     }
+                    let mut reass_limit_reached = false;
                     let txdata = self.parse_frame_data(
-                        head.ftype,
+                        &head,
                         &rem[..hlsafe],
                         complete,
-                        head.flags,
                         dir,
+                        &mut reass_limit_reached,
                     );
 
                     let tx = self.find_or_create_tx(&head, &txdata, dir);
+                    if reass_limit_reached {
+                        tx.tx_data.set_event(HTTP2Event::ReassemblyLimitReached as u8);
+                    }
                     tx.handle_frame(&head, &txdata, dir);
                     let over = head.flags & parser::HTTP2_FLAG_HEADER_EOS != 0;
                     let ftype = head.ftype;
@@ -1306,6 +1377,13 @@ pub unsafe extern "C" fn rs_http2_register_parser() {
                 SCLogError!("Invalid value for http2.max-table-size");
             }
         }
+        if let Some(val) = conf_get("app-layer.protocols.http2.max-reassembly-size") {
+            if let Ok(v) = val.parse::<u32>() {
+                HTTP2_MAX_REASS = v as usize;
+            } else {
+                SCLogError!("Invalid value for http2.max-reassembly-size");
+            }
+        }
         SCLogDebug!("Rust http2 parser registered.");
     } else {
         SCLogNotice!("Protocol detector and parser disabled for HTTP2.");
diff --git a/suricata.yaml.in b/suricata.yaml.in
index 412ab1aea850..1a2774986bd8 100644
--- a/suricata.yaml.in
+++ b/suricata.yaml.in
@@ -929,6 +929,8 @@ app-layer:
       #max-streams: 4096
       # Maximum headers table size
       #max-table-size: 65536
+      # Maximum reassembly size for header + continuation frames
+      #max-reassembly-size: 102400
     smtp:
       enabled: yes
       raw-extraction: no

From 8f63a8f3bffbbaf8fae4985ee5f974ab326b08c0 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Fri, 7 Apr 2023 16:02:41 +0200
Subject: [PATCH 403/462] http1: remove transactions from their list

instead of keeping a NULL pointer in an array

Ticket: #5921
---
 src/app-layer-htp-file.c |  3 ++-
 src/app-layer-htp.c      | 16 +++++++++-------
 src/app-layer-htp.h      |  7 +++++++
 3 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/src/app-layer-htp-file.c b/src/app-layer-htp-file.c
index f96b37016061..7b3ba62edcee 100644
--- a/src/app-layer-htp-file.c
+++ b/src/app-layer-htp-file.c
@@ -179,7 +179,8 @@ int HTPFileOpenWithRange(HtpState *s, HtpTxUserData *txud, const uint8_t *filena
     }
 
     // Then, we will try to handle reassembly of different ranges of the same file
-    htp_tx_t *tx = htp_list_get(s->conn->transactions, txid);
+    // TODO have the caller pass directly the tx
+    htp_tx_t *tx = htp_list_get(s->conn->transactions, txid - s->tx_freed);
     if (!tx) {
         SCReturnInt(-1);
     }
diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index 1e8c0b8ea689..babb87d28320 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -401,7 +401,7 @@ void HTPStateFree(void *state)
         uint64_t total_txs = HTPStateGetTxCnt(state);
         /* free the list of body chunks */
         if (s->conn != NULL) {
-            for (tx_id = 0; tx_id < total_txs; tx_id++) {
+            for (tx_id = s->tx_freed; tx_id < total_txs; tx_id++) {
                 htp_tx_t *tx = HTPStateGetTx(s, tx_id);
                 if (tx != NULL) {
                     HtpTxUserData *htud = (HtpTxUserData *) htp_tx_get_user_data(tx);
@@ -458,8 +458,10 @@ static void HTPStateTransactionFree(void *state, uint64_t id)
             tx->request_progress = HTP_REQUEST_COMPLETE;
             tx->response_progress = HTP_RESPONSE_COMPLETE;
         }
+        // replaces tx in the s->conn->transactions list by NULL
         htp_tx_destroy(tx);
     }
+    s->tx_freed += htp_connp_tx_freed(s->connp);
 }
 
 /**
@@ -3076,7 +3078,7 @@ static uint64_t HTPStateGetTxCnt(void *alstate)
         if (size < 0)
             return 0ULL;
         SCLogDebug("size %"PRIu64, size);
-        return (uint64_t)size;
+        return (uint64_t)size + http_state->tx_freed;
     } else {
         return 0ULL;
     }
@@ -3086,8 +3088,8 @@ static void *HTPStateGetTx(void *alstate, uint64_t tx_id)
 {
     HtpState *http_state = (HtpState *)alstate;
 
-    if (http_state != NULL && http_state->conn != NULL)
-        return htp_list_get(http_state->conn->transactions, tx_id);
+    if (http_state != NULL && http_state->conn != NULL && tx_id >= http_state->tx_freed)
+        return htp_list_get(http_state->conn->transactions, tx_id - http_state->tx_freed);
     else
         return NULL;
 }
@@ -3097,9 +3099,9 @@ void *HtpGetTxForH2(void *alstate)
     // gets last transaction
     HtpState *http_state = (HtpState *)alstate;
     if (http_state != NULL && http_state->conn != NULL) {
-        size_t txid = htp_list_array_size(http_state->conn->transactions);
-        if (txid > 0) {
-            return htp_list_get(http_state->conn->transactions, txid - 1);
+        size_t txid = HTPStateGetTxCnt(http_state);
+        if (txid > http_state->tx_freed) {
+            return htp_list_get(http_state->conn->transactions, txid - http_state->tx_freed - 1);
         }
     }
     return NULL;
diff --git a/src/app-layer-htp.h b/src/app-layer-htp.h
index dee5c17e833e..5972bdaf5001 100644
--- a/src/app-layer-htp.h
+++ b/src/app-layer-htp.h
@@ -247,6 +247,13 @@ typedef struct HtpState_ {
     htp_conn_t *conn;
     Flow *f;                /**< Needed to retrieve the original flow when using HTPLib callbacks */
     uint64_t transaction_cnt;
+    // tx_freed is the number of already freed transactions
+    // This is needed as libhtp only keeps the live transactions :
+    // To get the total number of transactions, we need to add
+    // the number of transactions tracked by libhtp to this number.
+    // It is also needed as an offset to translate between suricata
+    // transaction id to libhtp offset in its list/array
+    uint64_t tx_freed;
     const struct HTPCfgRec_ *cfg;
     uint16_t flags;
     uint16_t events;

From 4175680a8a1c0dfaa491ee63d6e36c011d498473 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 17 Oct 2023 15:28:53 +0200
Subject: [PATCH 404/462] http1: configurable max number of live tx per flow

Ticket: #5921

Co-authored-by: Jason Ish <jason.ish@oisf.net>
---
 configure.ac                                  |  2 ++
 doc/userguide/configuration/suricata-yaml.rst |  2 +-
 src/app-layer-htp.c                           | 16 ++++++++++++++++
 suricata.yaml.in                              |  2 ++
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 3acab5b3acfd..95613b674182 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1620,6 +1620,7 @@
         AC_CHECK_LIB([htp], [htp_config_set_lzma_layers],AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_LZMA_LAYERS],[1],[Found htp_config_set_lzma_layers function in libhtp]) ,,[-lhtp])
         AC_CHECK_LIB([htp], [htp_config_set_compression_bomb_limit],AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_COMPRESSION_BOMB_LIMIT],[1],[Found htp_config_set_compression_bomb_limit function in libhtp]) ,,[-lhtp])
         AC_CHECK_LIB([htp], [htp_config_set_compression_time_limit],AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_COMPRESSION_TIME_LIMIT],[1],[Found htp_config_set_compression_time_limit function in libhtp]) ,,[-lhtp])
+        AC_CHECK_LIB([htp], [htp_config_set_max_tx],AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_MAX_TX],[1],[Found htp_config_set_max_tx function in libhtp]) ,,[-lhtp])
     ])
 
     if test "x$enable_non_bundled_htp" = "xno"; then
@@ -1644,6 +1645,7 @@
             AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_LZMA_LAYERS],[1],[Assuming htp_config_set_lzma_layers function in bundled libhtp])
             AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_COMPRESSION_BOMB_LIMIT],[1],[Assuming htp_config_set_compression_bomb_limit function in bundled libhtp])
             AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_COMPRESSION_TIME_LIMIT],[1],[Assuming htp_config_set_compression_time_limit function in bundled libhtp])
+            AC_DEFINE_UNQUOTED([HAVE_HTP_CONFIG_SET_MAX_TX],[1],[Assuming htp_config_set_max_tx function in bundled libhtp])
         else
             echo
             echo "  ERROR: Libhtp is not bundled. Get libhtp by doing:"
diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index ba103a1f38de..ebae0bc479f9 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -1748,7 +1748,7 @@ incompatible with ``decode-mime``. If both are enabled,
 Maximum transactions
 ~~~~~~~~~~~~~~~~~~~~
 
-MQTT, FTP, PostgreSQL, SMB, DCERPC and NFS have each a `max-tx` parameter that can be customized.
+MQTT, FTP, PostgreSQL, SMB, DCERPC, HTTP1 and NFS have each a `max-tx` parameter that can be customized.
 `max-tx` refers to the maximum number of live transactions for each flow.
 An app-layer event `protocol.too_many_transactions` is triggered when this value is reached.
 The point of this parameter is to find a balance between the completeness of analysis
diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index babb87d28320..1d654c2c7c5b 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -2517,6 +2517,10 @@ static void HTPConfigSetDefaultsPhase1(HTPCfgRec *cfg_prec)
 #endif
 #ifdef HAVE_HTP_CONFIG_SET_COMPRESSION_TIME_LIMIT
     htp_config_set_compression_time_limit(cfg_prec->cfg, HTP_CONFIG_DEFAULT_COMPRESSION_TIME_LIMIT);
+#endif
+#ifdef HAVE_HTP_CONFIG_SET_MAX_TX
+#define HTP_CONFIG_DEFAULT_MAX_TX_LIMIT 512
+    htp_config_set_max_tx(cfg_prec->cfg, HTP_CONFIG_DEFAULT_MAX_TX_LIMIT);
 #endif
     /* libhtp <= 0.5.9 doesn't use soft limit, but it's impossible to set
      * only the hard limit. So we set both here to the (current) htp defaults.
@@ -2868,6 +2872,18 @@ static void HTPConfigParseParameters(HTPCfgRec *cfg_prec, ConfNode *s,
             }
             SCLogConfig("Setting HTTP decompression time limit to %" PRIu32 " usec", limit);
             htp_config_set_compression_time_limit(cfg_prec->cfg, (size_t)limit);
+#endif
+#ifdef HAVE_HTP_CONFIG_SET_MAX_TX
+        } else if (strcasecmp("max-tx", p->name) == 0) {
+            uint32_t limit = 0;
+            if (ParseSizeStringU32(p->val, &limit) < 0) {
+                FatalError("failed to parse 'max-tx' "
+                           "from conf file - %s.",
+                        p->val);
+            }
+            /* set default soft-limit with our new hard limit */
+            SCLogConfig("Setting HTTP max-tx limit to %" PRIu32 " bytes", limit);
+            htp_config_set_max_tx(cfg_prec->cfg, (size_t)limit);
 #endif
         } else if (strcasecmp("randomize-inspection-sizes", p->name) == 0) {
             if (!g_disable_randomness) {
diff --git a/suricata.yaml.in b/suricata.yaml.in
index 1a2774986bd8..1d3542f059ee 100644
--- a/suricata.yaml.in
+++ b/suricata.yaml.in
@@ -1081,6 +1081,8 @@ app-layer:
            #compression-bomb-limit: 1mb
            # Maximum time spent decompressing a single transaction in usec
            #decompression-time-limit: 100000
+           # Maximum number of live transactions per flow
+           #max-tx: 512
 
          server-config:
 

From 8f73a0ac5588cb5e5c501b3c7a07cb5d35b99d92 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 9 Nov 2023 12:27:59 +0100
Subject: [PATCH 405/462] smtp: config limit maximum number of live
 transactions

Ticket: #6477
---
 doc/userguide/configuration/suricata-yaml.rst |  2 +-
 src/app-layer-smtp.c                          | 27 ++++++++++++++++---
 src/app-layer-smtp.h                          |  1 +
 suricata.yaml.in                              |  2 ++
 4 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index ebae0bc479f9..c04573778b03 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -1748,7 +1748,7 @@ incompatible with ``decode-mime``. If both are enabled,
 Maximum transactions
 ~~~~~~~~~~~~~~~~~~~~
 
-MQTT, FTP, PostgreSQL, SMB, DCERPC, HTTP1 and NFS have each a `max-tx` parameter that can be customized.
+SMTP, MQTT, FTP, PostgreSQL, SMB, DCERPC, HTTP1 and NFS have each a `max-tx` parameter that can be customized.
 `max-tx` refers to the maximum number of live transactions for each flow.
 An app-layer event `protocol.too_many_transactions` is triggered when this value is reached.
 The point of this parameter is to find a balance between the completeness of analysis
diff --git a/src/app-layer-smtp.c b/src/app-layer-smtp.c
index a4d94a94ded2..20eb8f526af8 100644
--- a/src/app-layer-smtp.c
+++ b/src/app-layer-smtp.c
@@ -112,6 +112,8 @@
 #define SMTP_EHLO_EXTENSION_STARTTLS
 #define SMTP_EHLO_EXTENSION_8BITMIME
 
+#define SMTP_DEFAULT_MAX_TX 256
+
 typedef struct SMTPInput_ {
     /* current input that is being parsed */
     const uint8_t *buf;
@@ -421,6 +423,18 @@ static void SMTPConfigure(void) {
         smtp_config.raw_extraction = 0;
     }
 
+    uint64_t value = SMTP_DEFAULT_MAX_TX;
+    smtp_config.max_tx = SMTP_DEFAULT_MAX_TX;
+    const char *str = NULL;
+    if (ConfGet("app-layer.protocols.smtp.max-tx", &str) == 1) {
+        if (ParseSizeStringU64(str, &value) < 0) {
+            SCLogWarning("max-tx value cannot be deduced: %s,"
+                         " keeping default",
+                    str);
+        }
+        smtp_config.max_tx = value;
+    }
+
     SCReturn;
 }
 
@@ -436,8 +450,11 @@ static void SMTPSetEvent(SMTPState *s, uint8_t e)
     SCLogDebug("couldn't set event %u", e);
 }
 
-static SMTPTransaction *SMTPTransactionCreate(void)
+static SMTPTransaction *SMTPTransactionCreate(SMTPState *state)
 {
+    if (state->tx_cnt > smtp_config.max_tx) {
+        return NULL;
+    }
     SMTPTransaction *tx = SCCalloc(1, sizeof(*tx));
     if (tx == NULL) {
         return NULL;
@@ -1170,7 +1187,7 @@ static int SMTPProcessRequest(SMTPState *state, Flow *f, AppLayerParserState *ps
         return 0;
     }
     if (state->curr_tx == NULL || (state->curr_tx->done && !NoNewTx(state, line))) {
-        tx = SMTPTransactionCreate();
+        tx = SMTPTransactionCreate(state);
         if (tx == NULL)
             return -1;
         state->curr_tx = tx;
@@ -1203,7 +1220,7 @@ static int SMTPProcessRequest(SMTPState *state, Flow *f, AppLayerParserState *ps
                     // we did not close the previous tx, set error
                     SMTPSetEvent(state, SMTP_DECODER_EVENT_UNPARSABLE_CONTENT);
                     FileCloseFile(&tx->files_ts, &smtp_config.sbcfg, NULL, 0, FILE_TRUNCATED);
-                    tx = SMTPTransactionCreate();
+                    tx = SMTPTransactionCreate(state);
                     if (tx == NULL)
                         return -1;
                     state->curr_tx = tx;
@@ -1221,7 +1238,7 @@ static int SMTPProcessRequest(SMTPState *state, Flow *f, AppLayerParserState *ps
                      * of first one. So we start a new transaction. */
                     tx->mime_state->state_flag = PARSE_ERROR;
                     SMTPSetEvent(state, SMTP_DECODER_EVENT_UNPARSABLE_CONTENT);
-                    tx = SMTPTransactionCreate();
+                    tx = SMTPTransactionCreate(state);
                     if (tx == NULL)
                         return -1;
                     state->curr_tx = tx;
@@ -1932,6 +1949,8 @@ static void SMTPTestInitConfig(void)
     smtp_config.content_inspect_window = FILEDATA_CONTENT_INSPECT_WINDOW;
     smtp_config.content_inspect_min_size = FILEDATA_CONTENT_INSPECT_MIN_SIZE;
 
+    smtp_config.max_tx = SMTP_DEFAULT_MAX_TX;
+
     smtp_config.sbcfg.buf_size = FILEDATA_CONTENT_INSPECT_WINDOW;
 }
 
diff --git a/src/app-layer-smtp.h b/src/app-layer-smtp.h
index 9fc1d506bbbb..7977922ebac9 100644
--- a/src/app-layer-smtp.h
+++ b/src/app-layer-smtp.h
@@ -101,6 +101,7 @@ typedef struct SMTPConfig {
     uint32_t content_limit;
     uint32_t content_inspect_min_size;
     uint32_t content_inspect_window;
+    uint64_t max_tx;
 
     bool raw_extraction;
 
diff --git a/suricata.yaml.in b/suricata.yaml.in
index 1d3542f059ee..97a7e1318edd 100644
--- a/suricata.yaml.in
+++ b/suricata.yaml.in
@@ -934,6 +934,8 @@ app-layer:
     smtp:
       enabled: yes
       raw-extraction: no
+      # Maximum number of live SMTP transactions per flow
+      # max-tx: 256
       # Configure SMTP-MIME Decoder
       mime:
         # Decode MIME messages from SMTP transactions

From 61f2e4e1e5b34dfd8ae44d1c15253e2da65f6e6a Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 14 Nov 2023 21:51:37 +0100
Subject: [PATCH 406/462] smtp: avoid creating empty transaction

Ticket: 6477

So as to avoid ending up with too many empty transactions.

This happens when Suricata sees a DATA command in the current
transaction but did not have a confirmation response for it.
Then, if Suricata receives another DATA command, it will
create another new transaction, even if the previous one
is empty. And so, a malicious client can create many empty
transactions by just sending a repeated amount of DATA commands
without having a confirmation code for them.

Suricata cannot use state->current_command == SMTP_COMMAND_DATA
to prevent this attack and needs to resort to a new boolean
is_data because the malicious client may send another dummy command
after each DATA command.

This patch leaves only one call to SMTPTransactionCreate
---
 src/app-layer-smtp.c | 34 +++++++++-------------------------
 src/app-layer-smtp.h |  7 ++++++-
 2 files changed, 15 insertions(+), 26 deletions(-)

diff --git a/src/app-layer-smtp.c b/src/app-layer-smtp.c
index 20eb8f526af8..c61421cf0169 100644
--- a/src/app-layer-smtp.c
+++ b/src/app-layer-smtp.c
@@ -819,7 +819,7 @@ static inline void SMTPTransactionComplete(SMTPState *state)
 {
     DEBUG_VALIDATE_BUG_ON(state->curr_tx == NULL);
     if (state->curr_tx)
-        state->curr_tx->done = 1;
+        state->curr_tx->done = true;
 }
 
 /**
@@ -1215,36 +1215,19 @@ static int SMTPProcessRequest(SMTPState *state, Flow *f, AppLayerParserState *ps
             state->current_command = SMTP_COMMAND_STARTTLS;
         } else if (line->len >= 4 && SCMemcmpLowercase("data", line->buf, 4) == 0) {
             state->current_command = SMTP_COMMAND_DATA;
-            if (smtp_config.raw_extraction) {
-                if (state->tx_cnt > 1 && !state->curr_tx->done) {
-                    // we did not close the previous tx, set error
-                    SMTPSetEvent(state, SMTP_DECODER_EVENT_UNPARSABLE_CONTENT);
-                    FileCloseFile(&tx->files_ts, &smtp_config.sbcfg, NULL, 0, FILE_TRUNCATED);
-                    tx = SMTPTransactionCreate(state);
-                    if (tx == NULL)
-                        return -1;
-                    state->curr_tx = tx;
-                    TAILQ_INSERT_TAIL(&state->tx_list, tx, next);
-                    tx->tx_id = state->tx_cnt++;
-                }
+            if (state->curr_tx->is_data) {
+                // We did not receive a confirmation from server
+                // And now client sends a next DATA
+                SMTPSetEvent(state, SMTP_DECODER_EVENT_UNPARSABLE_CONTENT);
+                SCReturnInt(0);
+            } else if (smtp_config.raw_extraction) {
                 if (FileOpenFileWithId(&tx->files_ts, &smtp_config.sbcfg, state->file_track_id++,
                             (uint8_t *)rawmsgname, strlen(rawmsgname), NULL, 0,
                             FILE_NOMD5 | FILE_NOMAGIC) == 0) {
                     SMTPNewFile(tx, tx->files_ts.tail);
                 }
             } else if (smtp_config.decode_mime) {
-                if (tx->mime_state) {
-                    /* We have 2 chained mails and did not detect the end
-                     * of first one. So we start a new transaction. */
-                    tx->mime_state->state_flag = PARSE_ERROR;
-                    SMTPSetEvent(state, SMTP_DECODER_EVENT_UNPARSABLE_CONTENT);
-                    tx = SMTPTransactionCreate(state);
-                    if (tx == NULL)
-                        return -1;
-                    state->curr_tx = tx;
-                    TAILQ_INSERT_TAIL(&state->tx_list, tx, next);
-                    tx->tx_id = state->tx_cnt++;
-                }
+                DEBUG_VALIDATE_BUG_ON(tx->mime_state);
                 tx->mime_state = MimeDecInitParser(f, SMTPProcessDataChunk);
                 if (tx->mime_state == NULL) {
                     return MIME_DEC_ERR_MEM;
@@ -1260,6 +1243,7 @@ static int SMTPProcessRequest(SMTPState *state, Flow *f, AppLayerParserState *ps
                     tx->msg_tail = tx->mime_state->msg;
                 }
             }
+            state->curr_tx->is_data = true;
             /* Enter immediately data mode without waiting for server reply */
             if (state->parser_state & SMTP_PARSER_STATE_PIPELINING_SERVER) {
                 state->parser_state |= SMTP_PARSER_STATE_COMMAND_DATA_MODE;
diff --git a/src/app-layer-smtp.h b/src/app-layer-smtp.h
index 7977922ebac9..33b81d026a49 100644
--- a/src/app-layer-smtp.h
+++ b/src/app-layer-smtp.h
@@ -75,7 +75,12 @@ typedef struct SMTPTransaction_ {
 
     AppLayerTxData tx_data;
 
-    int done;
+    /** the tx is complete and can be logged and cleaned */
+    bool done;
+    /** the tx has seen a DATA command */
+    // another DATA command within the same context
+    // will trigger an app-layer event.
+    bool is_data;
     /** the first message contained in the session */
     MimeDecEntity *msg_head;
     /** the last message contained in the session */

From bc422c17d6961f03f673f2999a949913e89fc2d0 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 7 Nov 2023 10:33:21 +0100
Subject: [PATCH 407/462] detect: fixes use-after-free with http.request_header

Ticket: #6441

This keyword and the response one use a multiple inspection buffer.
But the different instances point to the same memory address
that comes from HttpHeaderGetBufferSpace and is not owned
by the transaction, and is rebuilt, which is a functional
bug in itself.

As it gets crafted, it can get reallocated if one header
is over 1024 bytes, while the previous freed pointer will still get
used for the previous headers.
---
 src/detect-http-header.c | 114 ++++++++++++++++++++++++++++++---------
 1 file changed, 88 insertions(+), 26 deletions(-)

diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index 93942c06fa7a..98e438c21111 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -48,6 +48,7 @@
 #include "util-print.h"
 #include "util-memcmp.h"
 #include "util-profiling.h"
+#include "util-validate.h"
 
 #include "app-layer.h"
 #include "app-layer-parser.h"
@@ -462,6 +463,8 @@ void DetectHttpHeaderRegister(void)
 
 static int g_http_request_header_buffer_id = 0;
 static int g_http_response_header_buffer_id = 0;
+static int g_request_header_thread_id = 0;
+static int g_response_header_thread_id = 0;
 
 static InspectionBuffer *GetHttp2HeaderData(DetectEngineThreadCtx *det_ctx, const uint8_t flags,
         const DetectEngineTransforms *transforms, Flow *_f, const struct MpmListIdDataArgs *cbdata,
@@ -570,6 +573,36 @@ static int PrefilterMpmHttp2HeaderRegister(DetectEngineCtx *de_ctx, SigGroupHead
             mpm_reg->app_v2.tx_min_progress, pectx, PrefilterMpmHttpHeaderFree, mpm_reg->name);
 }
 
+typedef struct HttpMultiBufItem {
+    uint8_t *buffer;
+    size_t len;
+} HttpMultiBufItem;
+
+typedef struct HttpMultiBufHeaderThreadData {
+    // array of items, being defined as a buffer with its length just above
+    HttpMultiBufItem *items;
+    // capacity of items (size of allocation)
+    size_t cap;
+    // length of items (number in use)
+    size_t len;
+} HttpMultiBufHeaderThreadData;
+
+static void *HttpMultiBufHeaderThreadDataInit(void *data)
+{
+    HttpMultiBufHeaderThreadData *td = SCCalloc(1, sizeof(*td));
+    return td;
+}
+
+static void HttpMultiBufHeaderThreadDataFree(void *data)
+{
+    HttpMultiBufHeaderThreadData *td = data;
+    for (size_t i = 0; i < td->cap; i++) {
+        SCFree(td->items[i].buffer);
+    }
+    SCFree(td->items);
+    SCFree(td);
+}
+
 static InspectionBuffer *GetHttp1HeaderData(DetectEngineThreadCtx *det_ctx, const uint8_t flags,
         const DetectEngineTransforms *transforms, Flow *f, const struct MpmListIdDataArgs *cbdata,
         int list_id)
@@ -583,10 +616,15 @@ static InspectionBuffer *GetHttp1HeaderData(DetectEngineThreadCtx *det_ctx, cons
     if (buffer->initialized)
         return buffer;
 
-    HttpHeaderThreadData *hdr_td = NULL;
-    HttpHeaderBuffer *buf =
-            HttpHeaderGetBufferSpace(det_ctx, f, flags, g_keyword_thread_id, &hdr_td);
-    if (unlikely(buf == NULL)) {
+    int kw_thread_id;
+    if (flags & STREAM_TOSERVER) {
+        kw_thread_id = g_request_header_thread_id;
+    } else {
+        kw_thread_id = g_response_header_thread_id;
+    }
+    HttpMultiBufHeaderThreadData *hdr_td =
+            DetectThreadCtxGetGlobalKeywordThreadCtx(det_ctx, kw_thread_id);
+    if (unlikely(hdr_td == NULL)) {
         return NULL;
     }
 
@@ -597,33 +635,53 @@ static InspectionBuffer *GetHttp1HeaderData(DetectEngineThreadCtx *det_ctx, cons
     } else {
         headers = tx->response_headers;
     }
-    if (cbdata->local_id < htp_table_size(headers)) {
-        htp_header_t *h = htp_table_get_index(headers, cbdata->local_id, NULL);
-        size_t size1 = bstr_size(h->name);
-        size_t size2 = bstr_size(h->value);
-        size_t b_len = size1 + 2 + size2;
-        if (b_len > buf->size) {
-            if (HttpHeaderExpandBuffer(hdr_td, buf, b_len) != 0) {
+    size_t no_of_headers = htp_table_size(headers);
+    if (cbdata->local_id == 0) {
+        // We initialize a big buffer on first item
+        // Then, we will just use parts of it
+        hdr_td->len = 0;
+        if (hdr_td->cap < no_of_headers) {
+            void *new_buffer = SCRealloc(hdr_td->items, no_of_headers * sizeof(HttpMultiBufItem));
+            if (unlikely(new_buffer == NULL)) {
                 return NULL;
             }
+            hdr_td->items = new_buffer;
+            // zeroes the new part of the items
+            memset(hdr_td->items + hdr_td->cap, 0,
+                    (no_of_headers - hdr_td->cap) * sizeof(HttpMultiBufItem));
+            hdr_td->cap = no_of_headers;
         }
-        memcpy(buf->buffer, bstr_ptr(h->name), bstr_size(h->name));
-        buf->buffer[size1] = ':';
-        buf->buffer[size1 + 1] = ' ';
-        memcpy(buf->buffer + size1 + 2, bstr_ptr(h->value), bstr_size(h->value));
-        buf->len = b_len;
-    } else {
-        InspectionBufferSetupMultiEmpty(buffer);
-        return NULL;
-    }
-    if (buf->len == 0) {
-        InspectionBufferSetupMultiEmpty(buffer);
-        return NULL;
+        for (size_t i = 0; i < no_of_headers; i++) {
+            htp_header_t *h = htp_table_get_index(headers, i, NULL);
+            size_t size1 = bstr_size(h->name);
+            size_t size2 = bstr_size(h->value);
+            size_t size = size1 + size2 + 2;
+            if (hdr_td->items[i].len < size) {
+                // Use realloc, as this pointer is not freed until HttpMultiBufHeaderThreadDataFree
+                hdr_td->items[i].buffer = SCRealloc(hdr_td->items[i].buffer, size);
+                if (unlikely(hdr_td->items[i].buffer == NULL)) {
+                    return NULL;
+                }
+            }
+            memcpy(hdr_td->items[i].buffer, bstr_ptr(h->name), size1);
+            hdr_td->items[i].buffer[size1] = ':';
+            hdr_td->items[i].buffer[size1 + 1] = ' ';
+            memcpy(hdr_td->items[i].buffer + size1 + 2, bstr_ptr(h->value), size2);
+            hdr_td->items[i].len = size;
+        }
+        hdr_td->len = no_of_headers;
     }
 
-    InspectionBufferSetupMulti(buffer, transforms, buf->buffer, buf->len);
-
-    SCReturnPtr(buffer, "InspectionBuffer");
+    // cbdata->local_id is the index of the requested header buffer
+    // hdr_td->len is the number of header buffers
+    if (cbdata->local_id < hdr_td->len) {
+        // we have one valid header buffer
+        InspectionBufferSetupMulti(buffer, transforms, hdr_td->items[cbdata->local_id].buffer,
+                hdr_td->items[cbdata->local_id].len);
+        SCReturnPtr(buffer, "InspectionBuffer");
+    } // else there are no more header buffer to get
+    InspectionBufferSetupMultiEmpty(buffer);
+    return NULL;
 }
 
 static void PrefilterTxHttp1Header(DetectEngineThreadCtx *det_ctx, const void *pectx, Packet *p,
@@ -736,6 +794,8 @@ void DetectHttpRequestHeaderRegister(void)
     DetectBufferTypeSetDescriptionByName("http_request_header", "HTTP header name and value");
     g_http_request_header_buffer_id = DetectBufferTypeGetByName("http_request_header");
     DetectBufferTypeSupportsMultiInstance("http_request_header");
+    g_request_header_thread_id = DetectRegisterThreadCtxGlobalFuncs("http_request_header",
+            HttpMultiBufHeaderThreadDataInit, NULL, HttpMultiBufHeaderThreadDataFree);
 }
 
 static int DetectHTTPResponseHeaderSetup(DetectEngineCtx *de_ctx, Signature *s, const char *arg)
@@ -771,6 +831,8 @@ void DetectHttpResponseHeaderRegister(void)
     DetectBufferTypeSetDescriptionByName("http_response_header", "HTTP header name and value");
     g_http_response_header_buffer_id = DetectBufferTypeGetByName("http_response_header");
     DetectBufferTypeSupportsMultiInstance("http_response_header");
+    g_response_header_thread_id = DetectRegisterThreadCtxGlobalFuncs("http_response_header",
+            HttpMultiBufHeaderThreadDataInit, NULL, HttpMultiBufHeaderThreadDataFree);
 }
 
 /************************************Unittests*********************************/

From f52c033e566beafb4480c139eb18662a2870464f Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 17 Oct 2023 22:01:27 +0200
Subject: [PATCH 408/462] pgsql: parse auth message within its bound

If the next PDU is already in the slice next, do not use it and
restrict ourselves to the length of this PDU.
Avoids overconsumption of memory by quadratic complexity, when
having many small PDUS in one big chunk being parsed

Ticket: #6411
---
 rust/src/pgsql/parser.rs | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 792fb23a130b..4dbb2915c236 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -719,7 +719,6 @@ fn pgsql_parse_authentication_message<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pgsq
     let (i, identifier) = verify(be_u8, |&x| x == b'R')(i)?;
     let (i, length) = verify(be_u32, |&x| x >= 8)(i)?;
     let (i, auth_type) = be_u32(i)?;
-    let (i, payload) = peek(rest)(i)?;
     let (i, message) = map_parser(
         take(length - 8),
         |b: &'a [u8]| {
@@ -729,14 +728,14 @@ fn pgsql_parse_authentication_message<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pgsq
                                 identifier,
                                 length,
                                 auth_type,
-                                payload: payload.to_vec(),
+                                payload: b.to_vec(),
                             }))),
                 3 => Ok((b, PgsqlBEMessage::AuthenticationCleartextPassword(
                             AuthenticationMessage {
                                 identifier,
                                 length,
                                 auth_type,
-                                payload: payload.to_vec(),
+                                payload: b.to_vec(),
                             }))),
                 5 => {
                     let (b, salt) = all_consuming(take(4_usize))(b)?;
@@ -753,7 +752,7 @@ fn pgsql_parse_authentication_message<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pgsq
                                 identifier,
                                 length,
                                 auth_type,
-                                payload: payload.to_vec(),
+                                payload: b.to_vec(),
                             }))),
                 // TODO - For SASL, should we parse specific details of the challenge itself? (as seen in: https://github.com/launchbadge/sqlx/blob/master/sqlx-core/src/postgres/message/authentication.rs )
                 10 => {
@@ -767,23 +766,21 @@ fn pgsql_parse_authentication_message<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pgsq
                                 })))
                 }
                 11 => {
-                    let (b, sasl_challenge) = rest(i)?;
                     Ok((b, PgsqlBEMessage::AuthenticationSASLContinue(
                                 AuthenticationMessage {
                                     identifier,
                                     length,
                                     auth_type,
-                                    payload: sasl_challenge.to_vec(),
+                                    payload: b.to_vec(),
                                 })))
                 },
                 12 => {
-                    let (i, signature) = take(length - 8)(i)?;
-                    Ok((i, PgsqlBEMessage::AuthenticationSASLFinal(
+                    Ok((b, PgsqlBEMessage::AuthenticationSASLFinal(
                                 AuthenticationMessage {
                                     identifier,
                                     length,
                                     auth_type,
-                                    payload: signature.to_vec(),
+                                    payload: b.to_vec(),
                                 }
                                 )))
                 }

From 86de7cffa7e8f06fe9d600127e7dabe89c7e81dd Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 17 Oct 2023 22:04:57 +0200
Subject: [PATCH 409/462] pgsql: parse only PDU when type is unknown

A next PDU may already be in the slice to parse.
Do not skip its parsing, ie do not use rest, but take just
the length of the pdu
---
 rust/src/pgsql/parser.rs | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/rust/src/pgsql/parser.rs b/rust/src/pgsql/parser.rs
index 4dbb2915c236..886ee4c5dca8 100644
--- a/rust/src/pgsql/parser.rs
+++ b/rust/src/pgsql/parser.rs
@@ -23,7 +23,7 @@ use crate::common::nom7::take_until_and_consume;
 use nom7::branch::alt;
 use nom7::bytes::streaming::{tag, take, take_until, take_until1};
 use nom7::character::streaming::{alphanumeric1, char};
-use nom7::combinator::{all_consuming, cond, eof, map_parser, opt, peek, rest, verify};
+use nom7::combinator::{all_consuming, cond, eof, map_parser, opt, peek, verify};
 use nom7::error::{make_error, ErrorKind};
 use nom7::multi::{many1, many_m_n, many_till};
 use nom7::number::streaming::{be_i16, be_i32};
@@ -1078,10 +1078,12 @@ pub fn pgsql_parse_response(i: &[u8]) -> IResult<&[u8], PgsqlBEMessage> {
                 b'A' => parse_notification_response(i)?,
                 b'D' => parse_consolidated_data_row(i)?,
                 _ => {
-                    let (i, payload) = rest(i)?;
+                    let (i, identifier) = be_u8(i)?;
+                    let (i, length) = verify(be_u32, |&x| x > PGSQL_LENGTH_FIELD)(i)?;
+                    let (i, payload) = take(length - PGSQL_LENGTH_FIELD)(i)?;
                     let unknown = PgsqlBEMessage::UnknownMessageType (RegularPacket{
-                        identifier: pseudo_header.0,
-                        length: pseudo_header.1,
+                        identifier,
+                        length,
                         payload: payload.to_vec(),
                     });
                     (i, unknown)
@@ -1918,7 +1920,7 @@ mod tests {
         let res = PgsqlBEMessage::UnknownMessageType(RegularPacket {
             identifier: b'`',
             length: 54,
-            payload: bad_buf.to_vec(),
+            payload: bad_buf[5..].to_vec(),
         });
         assert_eq!(result, res);
         assert!(remainder.is_empty());

From 80abc22f6475b6a87a33166729a871203f34d578 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 9 Nov 2023 16:15:36 +0100
Subject: [PATCH 410/462] http2: limit number of concurrent transactions

Ticket: 6481

Instead of just setting the old transactions to a drop state so
that they get later cleaned up by Suricata, fail creating new ones.

This is because one call to app-layer parsing can create many
transactions, and quadratic complexity could happen in one
single app-layer parsing because of find_or_create_tx
---
 rust/src/http2/detect.rs |  4 ++--
 rust/src/http2/http2.rs  | 47 +++++++++++++++++++++++++++-------------
 2 files changed, 34 insertions(+), 17 deletions(-)

diff --git a/rust/src/http2/detect.rs b/rust/src/http2/detect.rs
index 53258b3aa7d8..52b41190555b 100644
--- a/rust/src/http2/detect.rs
+++ b/rust/src/http2/detect.rs
@@ -932,7 +932,7 @@ fn http2_tx_set_header(state: &mut HTTP2State, name: &[u8], input: &[u8]) {
         blocks,
     };
     let txdata = HTTP2FrameTypeData::HEADERS(hs);
-    let tx = state.find_or_create_tx(&head, &txdata, Direction::ToServer);
+    let tx = state.find_or_create_tx(&head, &txdata, Direction::ToServer).unwrap();
     tx.frames_ts.push(HTTP2Frame {
         header: head,
         data: txdata,
@@ -975,7 +975,7 @@ fn http2_tx_set_settings(state: &mut HTTP2State, input: &[u8]) {
             match parser::http2_parse_frame_settings(&dec) {
                 Ok((_, set)) => {
                     let txdata = HTTP2FrameTypeData::SETTINGS(set);
-                    let tx = state.find_or_create_tx(&head, &txdata, Direction::ToServer);
+                    let tx = state.find_or_create_tx(&head, &txdata, Direction::ToServer).unwrap();
                     tx.frames_ts.push(HTTP2Frame {
                         header: head,
                         data: txdata,
diff --git a/rust/src/http2/http2.rs b/rust/src/http2/http2.rs
index 047b41402e12..b62ccb985034 100644
--- a/rust/src/http2/http2.rs
+++ b/rust/src/http2/http2.rs
@@ -611,9 +611,21 @@ impl HTTP2State {
 
     pub fn find_or_create_tx(
         &mut self, header: &parser::HTTP2FrameHeader, data: &HTTP2FrameTypeData, dir: Direction,
-    ) -> &mut HTTP2Transaction {
+    ) -> Option<&mut HTTP2Transaction> {
         if header.stream_id == 0 {
-            return self.create_global_tx();
+            if self.transactions.len() >= unsafe { HTTP2_MAX_STREAMS } {
+                for tx_old in &mut self.transactions {
+                    if tx_old.state == HTTP2TransactionState::HTTP2StateTodrop {
+                        // loop was already run
+                        break;
+                    }
+                    tx_old.set_event(HTTP2Event::TooManyStreams);
+                    // use a distinct state, even if we do not log it
+                    tx_old.state = HTTP2TransactionState::HTTP2StateTodrop;
+                }
+                return None;
+            }
+            return Some(self.create_global_tx());
         }
         let sid = match data {
             //yes, the right stream_id for Suricata is not the header one
@@ -643,30 +655,31 @@ impl HTTP2State {
             let tx = &mut self.transactions[index - 1];
             tx.tx_data.update_file_flags(self.state_data.file_flags);
             tx.update_file_flags(tx.tx_data.file_flags);
-            return tx;
+            return Some(tx);
         } else {
-            let mut tx = HTTP2Transaction::new();
-            self.tx_id += 1;
-            tx.tx_id = self.tx_id;
-            tx.stream_id = sid;
-            tx.state = HTTP2TransactionState::HTTP2StateOpen;
             // do not use SETTINGS_MAX_CONCURRENT_STREAMS as it can grow too much
-            if self.transactions.len() > unsafe { HTTP2_MAX_STREAMS } {
-                // set at least one another transaction to the drop state
+            if self.transactions.len() >= unsafe { HTTP2_MAX_STREAMS } {
                 for tx_old in &mut self.transactions {
-                    if tx_old.state != HTTP2TransactionState::HTTP2StateTodrop {
-                        // use a distinct state, even if we do not log it
-                        tx_old.set_event(HTTP2Event::TooManyStreams);
-                        tx_old.state = HTTP2TransactionState::HTTP2StateTodrop;
+                    if tx_old.state == HTTP2TransactionState::HTTP2StateTodrop {
+                        // loop was already run
                         break;
                     }
+                    tx_old.set_event(HTTP2Event::TooManyStreams);
+                    // use a distinct state, even if we do not log it
+                    tx_old.state = HTTP2TransactionState::HTTP2StateTodrop;
                 }
+                return None;
             }
+            let mut tx = HTTP2Transaction::new();
+            self.tx_id += 1;
+            tx.tx_id = self.tx_id;
+            tx.stream_id = sid;
+            tx.state = HTTP2TransactionState::HTTP2StateOpen;
             tx.tx_data.update_file_flags(self.state_data.file_flags);
             tx.update_file_flags(tx.tx_data.file_flags);
             tx.tx_data.file_tx = STREAM_TOSERVER|STREAM_TOCLIENT; // might hold files in both directions
             self.transactions.push_back(tx);
-            return self.transactions.back_mut().unwrap();
+            return Some(self.transactions.back_mut().unwrap());
         }
     }
 
@@ -1038,6 +1051,10 @@ impl HTTP2State {
                     );
 
                     let tx = self.find_or_create_tx(&head, &txdata, dir);
+                    if tx.is_none() {
+                        return AppLayerResult::err();
+                    }
+                    let tx = tx.unwrap();
                     if reass_limit_reached {
                         tx.tx_data.set_event(HTTP2Event::ReassemblyLimitReached as u8);
                     }

From 7f5e98e6df279be39a47c18ac0553a1cfebb906c Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Wed, 7 Feb 2024 21:54:28 +0100
Subject: [PATCH 411/462] ci: authors check using OISF repo

As flagged critical by codescan
---
 .github/workflows/authors.yml | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index 3fee4c5a2fde..5c4702a13858 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -8,20 +8,16 @@ jobs:
     name: New Author Check
     runs-on: ubuntu-latest
     steps:
+      - name: Checkout PR code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          fetch-depth: 0
       - run: sudo apt -y install git
-      - run: git clone https://github.com/${{ github.repository }}
-      - run: git remote add author ${{ github.event.pull_request.head.repo.html_url }}
-        working-directory: suricata
-      - run: git fetch author
-        working-directory: suricata
-      - run: git checkout author/${{ github.event.pull_request.head.ref }}
-        working-directory: suricata
       - name: Export known authors from master branch
-        run: git log --format="%an <%ae>" origin/master | sort | uniq > ../authors.txt
-        working-directory: suricata
+        run: git log --format="%an <%ae>" origin/master | sort | uniq > authors.txt
       - name: Export authors from new commits
-        run: git log --format="%an <%ae>" origin/${GITHUB_BASE_REF}... | sort | uniq > ../commit-authors.txt
-        working-directory: suricata
+        run: git log --format="%an <%ae>" origin/${GITHUB_BASE_REF}... | sort | uniq > commit-authors.txt
       - name: Check new authors
         run: |
           touch new-authors.txt

From 68b0052018079adc11ea1e35ab686c30716a8aad Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 8 Feb 2024 15:47:23 +0100
Subject: [PATCH 412/462] rust: fix clippy ptr_arg warnings

error: writing `&Vec` instead of `&[_]` involves a new object where a slice will do
   --> src/dns/log.rs:371:29
    |
371 | pub fn dns_print_addr(addr: &Vec<u8>) -> std::string::String {
    |                             ^^^^^^^^ help: change this to: `&[u8]`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
---
 rust/src/dhcp/logger.rs | 4 ++--
 rust/src/dns/log.rs     | 2 +-
 rust/src/ike/ikev1.rs   | 2 +-
 rust/src/nfs/nfs.rs     | 2 +-
 rust/src/smb/log.rs     | 4 ++--
 5 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/rust/src/dhcp/logger.rs b/rust/src/dhcp/logger.rs
index b29e2158ef95..3c86b1b7c567 100644
--- a/rust/src/dhcp/logger.rs
+++ b/rust/src/dhcp/logger.rs
@@ -229,7 +229,7 @@ impl DHCPLogger {
     fn log_opt_dns_server(&self, js: &mut JsonBuilder, option: &DHCPOptGeneric) -> Result<(), JsonError> {
         js.open_array("dns_servers")?;
         for i in 0..(option.data.len() / 4) {
-            let val = dns_print_addr(&option.data[(i * 4)..(i * 4) + 4].to_vec());
+            let val = dns_print_addr(&option.data[(i * 4)..(i * 4) + 4]);
             js.append_string(&val)?;
         }
         js.close()?;
@@ -239,7 +239,7 @@ impl DHCPLogger {
     fn log_opt_routers(&self, js: &mut JsonBuilder, option: &DHCPOptGeneric) -> Result<(), JsonError> {
         js.open_array("routers")?;
         for i in 0..(option.data.len() / 4) {
-            let val = dns_print_addr(&option.data[(i * 4)..(i * 4) + 4].to_vec());
+            let val = dns_print_addr(&option.data[(i * 4)..(i * 4) + 4]);
             js.append_string(&val)?;
         }
         js.close()?;
diff --git a/rust/src/dns/log.rs b/rust/src/dns/log.rs
index 4c0d4fc065b4..b2bf72ba1e46 100644
--- a/rust/src/dns/log.rs
+++ b/rust/src/dns/log.rs
@@ -368,7 +368,7 @@ pub fn dns_rcode_string(flags: u16) -> String {
 }
 
 /// Format bytes as an IP address string.
-pub fn dns_print_addr(addr: &Vec<u8>) -> std::string::String {
+pub fn dns_print_addr(addr: &[u8]) -> std::string::String {
     if addr.len() == 4 {
         return format!("{}.{}.{}.{}", addr[0], addr[1], addr[2], addr[3]);
     } else if addr.len() == 16 {
diff --git a/rust/src/ike/ikev1.rs b/rust/src/ike/ikev1.rs
index 1e79c293cdea..6f598f9806f6 100644
--- a/rust/src/ike/ikev1.rs
+++ b/rust/src/ike/ikev1.rs
@@ -53,7 +53,7 @@ impl Ikev1ParticipantData {
     }
 
     pub fn update(
-        &mut self, key_exchange: &str, nonce: &str, transforms: &Vec<Vec<SaAttribute>>,
+        &mut self, key_exchange: &str, nonce: &str, transforms: &[Vec<SaAttribute>],
     ) {
         self.key_exchange = key_exchange.to_string();
         self.nonce = nonce.to_string();
diff --git a/rust/src/nfs/nfs.rs b/rust/src/nfs/nfs.rs
index e14b0114eac9..c1d257d22963 100644
--- a/rust/src/nfs/nfs.rs
+++ b/rust/src/nfs/nfs.rs
@@ -481,7 +481,7 @@ impl NFSState {
     }
 
     // TODO maybe not enough users to justify a func
-    pub fn mark_response_tx_done(&mut self, xid: u32, rpc_status: u32, nfs_status: u32, resp_handle: &Vec<u8>)
+    pub fn mark_response_tx_done(&mut self, xid: u32, rpc_status: u32, nfs_status: u32, resp_handle: &[u8])
     {
         if let Some(mytx) = self.get_tx_by_xid(xid) {
             mytx.response_done = true;
diff --git a/rust/src/smb/log.rs b/rust/src/smb/log.rs
index 84965749ba17..e242d02e486b 100644
--- a/rust/src/smb/log.rs
+++ b/rust/src/smb/log.rs
@@ -38,7 +38,7 @@ fn debug_add_progress(jsb: &mut JsonBuilder, tx: &SMBTransaction) -> Result<(),
 
 /// take in a file GUID (16 bytes) or FID (2 bytes). Also deal
 /// with our frankenFID (2 bytes + 4 user_id)
-fn fuid_to_string(fuid: &Vec<u8>) -> String {
+fn fuid_to_string(fuid: &[u8]) -> String {
     let fuid_len = fuid.len();
     if fuid_len == 16 {
         guid_to_string(fuid)
@@ -52,7 +52,7 @@ fn fuid_to_string(fuid: &Vec<u8>) -> String {
     }
 }
 
-fn guid_to_string(guid: &Vec<u8>) -> String {
+fn guid_to_string(guid: &[u8]) -> String {
     if guid.len() == 16 {
         let output = format!("{:02x}{:02x}{:02x}{:02x}-{:02x}{:02x}-{:02x}{:02x}-{:02x}{:02x}-{:02x}{:02x}{:02x}{:02x}{:02x}{:02x}",
                 guid[3],  guid[2],  guid[1],  guid[0],

From f800ed0f901488d19fccf858ba0711792bf407ff Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 8 Feb 2024 13:21:11 -0600
Subject: [PATCH 413/462] detect-http: add superfluous alloc check for cocci

Add not-needed SCCalloc return check to satisfy our Cocci malloc
checks as it can't see that the caller immediately checks the return
value of this simple wrapper around SCCalloc.
---
 src/detect-http-header.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index 98e438c21111..7c15bf094a76 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -590,6 +590,11 @@ typedef struct HttpMultiBufHeaderThreadData {
 static void *HttpMultiBufHeaderThreadDataInit(void *data)
 {
     HttpMultiBufHeaderThreadData *td = SCCalloc(1, sizeof(*td));
+
+    /* This return value check to satisfy our Cocci malloc checks. */
+    if (td == NULL) {
+        return NULL;
+    }
     return td;
 }
 

From b48ec8a03922f36e76f2d6d942f2963afc2a3345 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 8 Feb 2024 20:23:59 +0100
Subject: [PATCH 414/462] detect/http_header: fix leak on realloc failure

---
 src/detect-http-header.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index 7c15bf094a76..e32220576ba1 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -663,10 +663,11 @@ static InspectionBuffer *GetHttp1HeaderData(DetectEngineThreadCtx *det_ctx, cons
             size_t size = size1 + size2 + 2;
             if (hdr_td->items[i].len < size) {
                 // Use realloc, as this pointer is not freed until HttpMultiBufHeaderThreadDataFree
-                hdr_td->items[i].buffer = SCRealloc(hdr_td->items[i].buffer, size);
-                if (unlikely(hdr_td->items[i].buffer == NULL)) {
+                void *tmp = SCRealloc(hdr_td->items[i].buffer, size);
+                if (unlikely(tmp == NULL)) {
                     return NULL;
                 }
+                hdr_td->items[i].buffer = tmp;
             }
             memcpy(hdr_td->items[i].buffer, bstr_ptr(h->name), size1);
             hdr_td->items[i].buffer[size1] = ':';

From 7e4dba7dfbd063be554b6145b747d7879eaa1f84 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Fri, 9 Feb 2024 08:09:38 +0100
Subject: [PATCH 415/462] detect/http: report error on alloc failure

---
 src/detect-http-header.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/detect-http-header.c b/src/detect-http-header.c
index e32220576ba1..f433d9fc9f07 100644
--- a/src/detect-http-header.c
+++ b/src/detect-http-header.c
@@ -593,6 +593,8 @@ static void *HttpMultiBufHeaderThreadDataInit(void *data)
 
     /* This return value check to satisfy our Cocci malloc checks. */
     if (td == NULL) {
+        SCLogError("failed to allocate %" PRIuMAX " bytes: %s", (uintmax_t)sizeof(*td),
+                strerror(errno));
         return NULL;
     }
     return td;

From 231c892befe25c23d5d3b54e48077c8446940eb5 Mon Sep 17 00:00:00 2001
From: Alexey Simakov <bigalex934@gmail.com>
Date: Wed, 7 Feb 2024 17:51:00 +0300
Subject: [PATCH 416/462] util/mime: fix memory leak

Fix memory leak at util-decode-mime:MimeDecInitParser, which
root cause is not-freeing allocated memory for mimeMsg

Bug: #6745
---
 src/util-decode-mime.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/util-decode-mime.c b/src/util-decode-mime.c
index d9941cd986b9..73f7335e5d0c 100644
--- a/src/util-decode-mime.c
+++ b/src/util-decode-mime.c
@@ -2432,6 +2432,7 @@ MimeDecParseState * MimeDecInitParser(void *data,
     PushStack(state->stack);
     if (state->stack->top == NULL) {
         SCFree(state->stack);
+        SCFree(state->msg);
         SCFree(state);
         return NULL;
     }

From 364adeeb0442592ceea4d329622ca5fe34bf31fd Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jeff.lucovsky@corelight.com>
Date: Thu, 8 Feb 2024 09:12:47 -0500
Subject: [PATCH 417/462] netmap: Release lock to avoid deadlock

Issue: 6755

When NetmapOpen encounters an error opening the netmap device, it'll
retry a bit. When the retry limit is reached, it'll shutdown Suricata.

This commit ensures that the device list lock is not held when before
closing all open devices before terminating Suricata.
---
 src/source-netmap.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/source-netmap.c b/src/source-netmap.c
index 0b04b41b52d6..8e409ea00e6f 100644
--- a/src/source-netmap.c
+++ b/src/source-netmap.c
@@ -453,6 +453,7 @@ static int NetmapOpen(NetmapIfaceSettings *ns, NetmapDevice **pdevice, int verbo
             }
         }
 
+        SCMutexUnlock(&netmap_devlist_lock);
         NetmapCloseAll();
         FatalError("opening devname %s failed: %s", devname, strerror(errno));
     }

From ee6208be9dfcaa9594823713e69169364c8b2c48 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 30 Jan 2024 10:19:05 -0500
Subject: [PATCH 418/462] config/nss: Remove libnspr/libnss traces

Issue: 6712
---
 .github/workflows/builds.yml | 16 ----------------
 .github/workflows/rust.yml   |  1 -
 doc/userguide/install.rst    |  2 +-
 qa/valgrind.suppress         | 16 ----------------
 4 files changed, 1 insertion(+), 34 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index eb02c0c21775..16f6ca2fad92 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -235,7 +235,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
@@ -363,7 +362,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
@@ -467,7 +465,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
@@ -539,7 +536,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
@@ -627,7 +623,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
@@ -709,7 +704,6 @@ jobs:
                 libpcap-devel \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python36-PyYAML \
@@ -796,7 +790,6 @@ jobs:
                 llvm-devel \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -892,7 +885,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python \
@@ -987,7 +979,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -1084,7 +1075,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -1173,7 +1163,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -1255,7 +1244,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -1347,7 +1335,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-softokn-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-yaml \
@@ -2101,7 +2088,6 @@ jobs:
               libjansson-dev \
               libjansson4 \
               liblua5.1-dev \
-              libnspr4-dev \
               libnuma-dev \
               liblz4-dev \
               libssl-dev \
@@ -2185,7 +2171,6 @@ jobs:
               libmagic-dev \
               libjansson-dev \
               libjansson4 \
-              libnspr4-dev \
               liblz4-dev \
               libssl-dev \
               liblzma-dev \
@@ -2264,7 +2249,6 @@ jobs:
               libjansson-dev \
               libjansson4 \
               liblua5.1-dev \
-              libnspr4-dev \
               libnuma-dev \
               liblz4-dev \
               libssl-dev \
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index e6e8460733a5..e6bf4f4211f4 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -53,7 +53,6 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
-                nss-devel \
                 pcre2-devel \
                 pkgconfig \
                 python3-devel \
diff --git a/doc/userguide/install.rst b/doc/userguide/install.rst
index b3d39d216a0b..cfb3c1bbbd56 100644
--- a/doc/userguide/install.rst
+++ b/doc/userguide/install.rst
@@ -152,7 +152,7 @@ Recommended::
                    jansson-devel jq libcap-ng-devel libevent-devel \
                    libmaxminddb-devel libnet-devel libnetfilter_queue-devel \
                    libnfnetlink-devel libpcap-devel libtool libyaml-devel \
-                   lua-devel lz4-devel make nss-devel pcre2-devel pkgconfig \
+                   lua-devel lz4-devel make pcre2-devel pkgconfig \
                    python3-devel python3-sphinx python3-yaml sudo which \
                    zlib-devel
     cargo install --force cbindgen
diff --git a/qa/valgrind.suppress b/qa/valgrind.suppress
index ffe057180021..8f94ec5b15e2 100644
--- a/qa/valgrind.suppress
+++ b/qa/valgrind.suppress
@@ -49,22 +49,6 @@
    fun:start_thread
    fun:clone
 }
-{
-   Warning on ARM, not Suricata related
-   Memcheck:Addr4
-   fun:strlen
-   fun:_dl_open
-   fun:do_dlopen
-   fun:_dl_catch_error
-   fun:dlerror_run
-   fun:__libc_dlopen_mode
-   fun:__nss_lookup_function
-   fun:__nss_lookup
-   fun:getprotobyname_r@@GLIBC_2.4
-   fun:getprotobyname
-   fun:DetectIPProtoParse
-   fun:DetectIPProtoTestParse02
-}
 {
    Known warning, NIC offloading - Param, see Bug 2230
    Memcheck:Param

From 9fe00ff71077caf5d13ac442ef2d729a88b00c49 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Tue, 30 Jan 2024 10:21:15 -0500
Subject: [PATCH 419/462] config/jansson: Remove excess libjansson mentions

Issue: 6712

Remove multiple occurrences of libjansson installation packages.
---
 .github/workflows/builds.yml     | 8 --------
 .github/workflows/commits.yml    | 1 -
 .github/workflows/scan-build.yml | 1 -
 3 files changed, 10 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 16f6ca2fad92..7dab8ed84c0a 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -1398,7 +1398,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 llvm-14-dev \
                 make \
@@ -1516,7 +1515,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 make \
                 parallel \
@@ -1609,7 +1607,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 libpcre2-dev \
                 make \
@@ -1693,7 +1690,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 libpcre2-dev \
                 make \
@@ -1758,7 +1754,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 make \
                 parallel \
@@ -1835,7 +1830,6 @@ jobs:
                 libnfnetlink0 \
                 libhiredis-dev \
                 libjansson-dev \
-                libjansson-dev \
                 libpython2.7 \
                 make \
                 rustc \
@@ -1898,7 +1892,6 @@ jobs:
                   libjansson-dev \
                   libevent-dev \
                   libevent-pthreads-2.1-7 \
-                  libjansson-dev \
                   libpython2.7 \
                   make \
                   parallel \
@@ -1991,7 +1984,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 make \
                 parallel \
diff --git a/.github/workflows/commits.yml b/.github/workflows/commits.yml
index 04bbb3fdf96b..6729472747b8 100644
--- a/.github/workflows/commits.yml
+++ b/.github/workflows/commits.yml
@@ -48,7 +48,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 libpython2.7 \
                 libssl-dev \
                 make \
diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml
index ef9c10bf1df9..fc7cf4d3f962 100644
--- a/.github/workflows/scan-build.yml
+++ b/.github/workflows/scan-build.yml
@@ -54,7 +54,6 @@ jobs:
                 libjansson-dev \
                 libevent-dev \
                 libevent-pthreads-2.1-7 \
-                libjansson-dev \
                 liblz4-dev \
                 llvm-16-dev \
                 make \

From 7956fa52423f80858643b25080efba2f64d78512 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sun, 11 Feb 2024 09:29:38 +0100
Subject: [PATCH 420/462] multi-tenant: fix loader dead lock

A dead lock could occur at start up, where a loader thread would
get stuck on it's condition variable, while the main thread was
polling the loaders task results.

The vector to the dead lock is as follows:

main	                        loader
DetectEngineMultiTenantSetup
-DetectLoaderSetupLoadTenant
--DetectLoaderQueueTask
---lock loader
---add task
---unlock loader
	                        lock loader
	                        check/exec tasks
	                        unlock loader
---wake up threads
	                        lock ctrl mutx
	                        cond wait ctrl
	                        unlock ctrl
-DetectLoadersSync
--lock loader
--check tasks
--unlock loader

Between the main thread unlocking the loader and waking up the
threads, it is possible that the loader has already moved ahead
but not yet entered its conditional wait. The main thread sends
its condition signal, but since the loader isn't yet waiting on
it the signal is ignored. Then when the loader does enter its
conditional wait, the signal is not sent again.

This patch updates the logic to send signals much more often.
It also makes sure that the signal is sent under lock, as the
API requires.

Bug: #6766.
---
 src/detect-engine-loader.c | 13 +++++++++++++
 src/detect-engine-loader.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 1f5363c6f80c..9073c1e9c29d 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -469,6 +469,12 @@ int DetectLoadersSync(void)
                 done = true;
             }
             SCMutexUnlock(&loader->m);
+            if (!done) {
+                /* nudge thread in case it's sleeping */
+                SCCtrlMutexLock(loader->tv->ctrl_mutex);
+                pthread_cond_broadcast(loader->tv->ctrl_cond);
+                SCCtrlMutexUnlock(loader->tv->ctrl_mutex);
+            }
         }
         SCMutexLock(&loader->m);
         if (loader->result != 0) {
@@ -524,7 +530,9 @@ static void TmThreadWakeupDetectLoaderThreads(void)
         while (tv != NULL) {
             if (strncmp(tv->name,"DL#",3) == 0) {
                 BUG_ON(tv->ctrl_cond == NULL);
+                SCCtrlMutexLock(tv->ctrl_mutex);
                 pthread_cond_broadcast(tv->ctrl_cond);
+                SCCtrlMutexUnlock(tv->ctrl_mutex);
             }
             tv = tv->next;
         }
@@ -568,6 +576,11 @@ static TmEcode DetectLoaderThreadInit(ThreadVars *t, const void *initdata, void
     /* pass thread data back to caller */
     *data = ftd;
 
+    DetectLoaderControl *loader = &loaders[ftd->instance];
+    SCMutexLock(&loader->m);
+    loader->tv = t;
+    SCMutexUnlock(&loader->m);
+
     return TM_ECODE_OK;
 }
 
diff --git a/src/detect-engine-loader.h b/src/detect-engine-loader.h
index 7ffb8c8648a0..8a6f7b8f17be 100644
--- a/src/detect-engine-loader.h
+++ b/src/detect-engine-loader.h
@@ -44,6 +44,7 @@ typedef struct DetectLoaderTask_ {
 typedef struct DetectLoaderControl_ {
     int id;
     int result;     /* 0 for ok, error otherwise */
+    ThreadVars *tv; /* loader threads threadvars - for waking them up */
     SCMutex m;
     TAILQ_HEAD(, DetectLoaderTask_) task_list;
 } DetectLoaderControl;

From edfda9f69fb2f095a195860b382eed94655238be Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 13:10:31 -0600
Subject: [PATCH 421/462] rust: weekly cargo audit and update

Add GitHub actions to perform:

- cargo audit: catch new warnings in dependendent packages
- cargo update: catch updated dependencies that depend on a new MSRV
    than we use
---
 .github/workflows/rust-checks.yml | 179 ++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)
 create mode 100644 .github/workflows/rust-checks.yml

diff --git a/.github/workflows/rust-checks.yml b/.github/workflows/rust-checks.yml
new file mode 100644
index 000000000000..5ac2739bf3d7
--- /dev/null
+++ b/.github/workflows/rust-checks.yml
@@ -0,0 +1,179 @@
+name: Cargo Audit and Update
+
+on:
+  schedule:
+    # Run on Monday mornings, 11AM UTC.
+    - cron: '0 11 * * 1'
+  # Enable push for testing when working on this file.
+  #push:
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions: read-all
+
+env:
+  RUST_VERSION_MIN: "1.63.0"
+
+jobs:
+
+  # This job runs `cargo audit` and will exit with a failure code if
+  # any warnings are raised.
+  audit:
+    name: Cargo Audit
+    runs-on: ubuntu-latest
+    container: almalinux:9
+    steps:
+      - name: Cache cargo registry
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
+        with:
+          path: ~/.cargo
+          key: ${{ github.job }}-cargo
+
+      - name: Cache RPMs
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
+        with:
+          path: /var/cache/dnf
+          key: ${{ github.job }}-dnf
+      - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
+
+      - name: Install system packages
+        run: |
+          dnf -y install dnf-plugins-core epel-release
+          dnf config-manager --set-enabled crb
+          dnf -y install \
+                autoconf \
+                automake \
+                cbindgen \
+                diffutils \
+                numactl-devel \
+                dpdk-devel \
+                file-devel \
+                gcc \
+                gcc-c++ \
+                git \
+                jansson-devel \
+                jq \
+                lua-devel \
+                libtool \
+                libyaml-devel \
+                libnfnetlink-devel \
+                libnetfilter_queue-devel \
+                libnet-devel \
+                libcap-ng-devel \
+                libevent-devel \
+                libmaxminddb-devel \
+                libpcap-devel \
+                libtool \
+                lz4-devel \
+                make \
+                pcre2-devel \
+                pkgconfig \
+                python3-devel \
+                python3-sphinx \
+                python3-yaml \
+                sudo \
+                which \
+                zlib-devel
+      - name: Install Rust
+        run: |
+          curl https://sh.rustup.rs -sSf | sh -s -- -y
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+      - name: Install Cargo Audit
+        run: cargo install cargo-audit
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      - name: Configure Suricata
+        run: |
+          ./scripts/bundle.sh libhtp
+          ./autogen.sh
+          ./configure
+      - name: Run Cargo Audit
+        working-directory: rust
+        run: |
+          IGNORES=()
+
+          # failure, via bendy
+          IGNORES+=(--ignore RUSTSEC-2020-0036)
+          # failure, via bendy
+          IGNORES+=(--ignore RUSTSEC-2019-0036)
+
+          cargo audit -D warnings "${IGNORES[@]}"
+
+  # This job uses our MSRV and does a `cargo update` with the idea
+  # that it should catch early any dependencies that have done a patch
+  # update pulling in a new MSRV. This would be an indicator that we
+  # have to more tightly pin the dependency, or even attempt to pin a
+  # transitive dependency.
+  update:
+    name: Cargo Update
+    runs-on: ubuntu-latest
+    container: almalinux:9
+    steps:
+      - name: Cache cargo registry
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
+        with:
+          path: ~/.cargo
+          key: ${{ github.job }}-cargo
+
+      - name: Cache RPMs
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
+        with:
+          path: /var/cache/dnf
+          key: ${{ github.job }}-dnf
+      - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
+
+      - name: Install system packages
+        run: |
+          dnf -y install dnf-plugins-core epel-release
+          dnf config-manager --set-enabled crb
+          dnf -y install \
+                autoconf \
+                automake \
+                cbindgen \
+                diffutils \
+                numactl-devel \
+                dpdk-devel \
+                file-devel \
+                gcc \
+                gcc-c++ \
+                git \
+                jansson-devel \
+                jq \
+                lua-devel \
+                libtool \
+                libyaml-devel \
+                libnfnetlink-devel \
+                libnetfilter_queue-devel \
+                libnet-devel \
+                libcap-ng-devel \
+                libevent-devel \
+                libmaxminddb-devel \
+                libpcap-devel \
+                libtool \
+                lz4-devel \
+                make \
+                pcre2-devel \
+                pkgconfig \
+                python3-devel \
+                python3-sphinx \
+                python3-yaml \
+                sudo \
+                which \
+                zlib-devel
+      - name: Install Minimum Supported Rust Version
+        run: |
+          curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain ${RUST_VERSION_MIN} -y
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      - name: Configure Suricata
+        run: |
+          ./scripts/bundle.sh libhtp
+          ./autogen.sh
+          ./configure
+      - name: Cargo Update and Build
+        working-directory: rust
+        run: |
+          cargo update
+          cargo build --all-features --all-targets

From 6922fef4ab0ba8ecc19204c661d173037ad6d94c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 00:02:32 -0600
Subject: [PATCH 422/462] github-ci: move centos-7 build to its own workflow

CentOS 7 requires older actions due to newer GitHub actions depending
on a newer glibc. So move to its own workflow file so the main builds
can move forward to newer versions of actions.
---
 .github/workflows/build-centos-7.yml | 174 +++++++++++++++++++++++++++
 .github/workflows/builds.yml         |  78 ------------
 2 files changed, 174 insertions(+), 78 deletions(-)
 create mode 100644 .github/workflows/build-centos-7.yml

diff --git a/.github/workflows/build-centos-7.yml b/.github/workflows/build-centos-7.yml
new file mode 100644
index 000000000000..83f853306669
--- /dev/null
+++ b/.github/workflows/build-centos-7.yml
@@ -0,0 +1,174 @@
+name: build-centos-7
+
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+    inputs:
+      LIBHTP_REPO:
+      LIBHTP_BRANCH:
+      SU_REPO:
+      SU_BRANCH:
+      SV_REPO:
+      SV_BRANCH:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions: read-all
+
+env:
+  DEFAULT_SV_REPO: https://github.com/OISF/suricata-verify
+  DEFAULT_SV_BRANCH: master
+  DEFAULT_CFLAGS: "-Wall -Wextra -Werror -Wno-unused-parameter -Wno-unused-function"
+
+jobs:
+  centos-7:
+    runs-on: ubuntu-latest
+    container: centos:7
+    steps:
+      - name: Cache cargo registry
+        uses: actions/cache@v3.3.3
+        with:
+          path: ~/.cargo
+          key: ${{ github.job }}-cargo
+
+      - name: Cache RPMs
+        uses: actions/cache@v3.3.3
+        with:
+          path: /var/cache/yum
+          key: ${{ github.job }}-yum
+
+      - run: |
+          yum -y install epel-release
+          yum -y install \
+                autoconf \
+                automake \
+                cargo \
+                curl \
+                diffutils \
+                file-devel \
+                gcc \
+                gcc-c++ \
+                git \
+                jansson-devel \
+                jq \
+                lua-devel \
+                libtool \
+                libyaml-devel \
+                libnfnetlink-devel \
+                libnetfilter_queue-devel \
+                libnet-devel \
+                libcap-ng-devel \
+                libevent-devel \
+                libmaxminddb-devel \
+                libpcap-devel \
+                lz4-devel \
+                make \
+                nss-devel \
+                pcre2-devel \
+                pkgconfig \
+                python36-PyYAML \
+                rust \
+                sudo \
+                which \
+                zlib-devel
+      - name: Parse repo and branch information
+        env:
+          # We fetch the actual pull request to get the latest body as
+          # github.event.pull_request.body has the body from the
+          # initial pull request.
+          PR_HREF: ${{ github.event.pull_request._links.self.href }}
+        run: |
+          if test "${PR_HREF}"; then
+              body=$(curl -s "${PR_HREF}" | jq -r .body | tr -d '\r')
+
+              echo "Parsing branch and PR info from:"
+              echo "${body}"
+
+              LIBHTP_REPO=$(echo "${body}" | awk -F = '/^LIBHTP_REPO=/ { print $2 }')
+              LIBHTP_BRANCH=$(echo "${body}" | awk -F = '/^LIBHTP_BRANCH=/ { print $2 }')
+
+              SU_REPO=$(echo "${body}" | awk -F = '/^SU_REPO=/ { print $2 }')
+              SU_BRANCH=$(echo "${body}" | awk -F = '/^SU_BRANCH=/ { print $2 }')
+
+              SV_REPO=$(echo "${body}" | awk -F = '/^SV_REPO=/ { print $2 }')
+              SV_BRANCH=$(echo "${body}" | awk -F = '/^SV_BRANCH=/ { print $2 }')
+          else
+              echo "No pull request body, will use inputs or defaults."
+              LIBHTP_REPO=${{ inputs.LIBHTP_REPO }}
+              LIBHTP_BRANCH=${{ inputs.LIBHTP_BRANCH }}
+              SU_REPO=${{ inputs.SU_REPO }}
+              SU_BRANCH=${{ inputs.SU_BRANCH }}
+              SV_REPO=${{ inputs.SV_REPO }}
+              SV_BRANCH=${{ inputs.SV_BRANCH }}
+          fi
+
+          # If the _REPO variables don't contain a full URL, add GitHub.
+          if [ "${LIBHTP_REPO}" ] && ! echo "${LIBHTP_REPO}" | grep -q '^https://'; then
+              LIBHTP_REPO="https://github.com/${LIBHTP_REPO}"
+          fi
+          if [ "${SU_REPO}" ] && ! echo "${SU_REPO}" | grep -q '^https://'; then
+              SU_REPO="https://github.com/${SU_REPO}"
+          fi
+          if [ "${SV_REPO}" ] && ! echo "${SV_REPO}" | grep -q '^https://'; then
+              SV_REPO="https://github.com/${SV_REPO}"
+          fi
+
+          echo LIBHTP_REPO=${LIBHTP_REPO} | tee -a ${GITHUB_ENV}
+          echo LIBHTP_BRANCH=${LIBHTP_BRANCH} | tee -a ${GITHUB_ENV}
+
+          echo SU_REPO=${SU_REPO} | tee -a ${GITHUB_ENV}
+          echo SU_BRANCH=${SU_BRANCH} | tee -a ${GITHUB_ENV}
+
+          echo SV_REPO=${SV_REPO:-${DEFAULT_SV_REPO}} | tee -a ${GITHUB_ENV}
+          echo SV_BRANCH=${SV_BRANCH:-${DEFAULT_SV_BRANCH}} | tee -a ${GITHUB_ENV}
+
+      - name: Annotate output
+        run: |
+          echo "::notice:: LIBHTP_REPO=${LIBHTP_REPO}"
+          echo "::notice:: LIBHTP_BRANCH=${LIBHTP_BRANCH}"
+          echo "::notice:: SU_REPO=${SU_REPO}"
+          echo "::notice:: SU_BRANCH=${SU_BRANCH}"
+          echo "::notice:: SV_REPO=${SV_REPO}"
+          echo "::notice:: SV_BRANCH=${SV_BRANCH}"
+
+      - name: Install cbindgen
+        run: |
+          cargo install --debug cbindgen
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+
+      # Now checkout Suricata for the bundle script.
+      - name: Checking out Suricata
+        uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744
+
+      - run: ./scripts/bundle.sh
+
+      - name: Fetching suricata-verify
+        run: |
+          # Looking for a pull request number. in the SV_BRANCH
+          # value. This could be "pr/NNN", "pull/NNN" or a link to an
+          # OISF/suricata-verify pull request.
+          pr=$(echo "${SV_BRANCH}" | sed -n \
+              -e 's/^https:\/\/github.com\/OISF\/suricata-verify\/pull\/\([0-9]*\)$/\1/p' \
+              -e 's/^pull\/\([0-9]*\)$/\1/p' \
+              -e 's/^pr\/\([0-9]*\)$/\1/p')
+          if [ "${pr}" ]; then
+              SV_BRANCH="refs/pull/${pr}/head"
+              echo "Using suricata-verify pull-request ${SV_BRANCH}"
+          else
+              echo "Using suricata-verify branch ${SV_BRANCH}"
+          fi
+          git clone --depth 1 ${SV_REPO} suricata-verify
+          cd suricata-verify
+          git fetch --depth 1 origin ${SV_BRANCH}
+          git -c advice.detachedHead=false checkout FETCH_HEAD
+
+      - run: ./autogen.sh
+      - run: ./configure
+      - run: make -j ${{ env.CPUS }}
+      - run: python3 ./suricata-verify/run.py -q --debug-failed
+      - run: make install-full
+      - run: suricata-update -V
+      - run: suricatasc -h
diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 7dab8ed84c0a..df679b1bbdec 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -660,84 +660,6 @@ jobs:
       - run: suricata-update -V
       - run: suricatasc -h
 
-  centos-7:
-    name: CentOS 7
-    runs-on: ubuntu-latest
-    container: centos:7
-    needs: [prepare-deps, debian-12-dist]
-    steps:
-      - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
-        with:
-          path: ~/.cargo
-          key: ${{ github.job }}-cargo
-
-      - name: Cache RPMs
-        uses: actions/cache@v3.3.1
-        with:
-          path: /var/cache/yum
-          key: ${{ github.job }}-yum
-      - run: echo "keepcache=1" >> /etc/yum.conf
-
-      - name: Install system dependencies
-        run: |
-          yum -y install epel-release
-          yum -y install \
-                autoconf \
-                automake \
-                cargo \
-                diffutils \
-                file-devel \
-                gcc \
-                gcc-c++ \
-                jansson-devel \
-                jq \
-                lua-devel \
-                libtool \
-                libyaml-devel \
-                libnfnetlink-devel \
-                libnetfilter_queue-devel \
-                libnet-devel \
-                libcap-ng-devel \
-                libevent-devel \
-                libmaxminddb-devel \
-                libpcap-devel \
-                lz4-devel \
-                make \
-                pcre2-devel \
-                pkgconfig \
-                python36-PyYAML \
-                rust \
-                sudo \
-                which \
-                zlib-devel
-      - name: Download suricata.tar.gz
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
-        with:
-          name: dist
-      - run: tar zxvf suricata-*.tar.gz --strip-components=1
-      # This isn't really needed as we are building from a prepared
-      # package, but some package managers like RPM and Debian like to
-      # run this command even on prepared packages, so make sure it
-      # works.
-      - name: Test autoreconf
-        run: autoreconf -fv --install
-      - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure
-      - run: make -j2
-      - run: make install
-      - run: make install-conf
-      - run: make distcheck
-      - run: make clean
-      - run: make -j2
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
-        with:
-          name: prep
-          path: prep
-      - run: tar xf prep/suricata-verify.tar.gz
-      - run: python3 ./suricata-verify/run.py -q --debug-failed
-      - run: suricata-update -V
-      - run: suricatasc -h
-
   fedora-38-sv-codecov:
     name: Fedora 38 (Suricata Verify codecov)
     runs-on: ubuntu-latest

From 8522256aaa1c1df8c013d1e37f55fdb9db403a29 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Thu, 8 Feb 2024 18:19:41 -0600
Subject: [PATCH 423/462] github-ci: use all cores available

GitHub action Linux runners now have 4 cores, instead of hardcoding
the number, use nproc to determine how many cores are available and
use them.
---
 .github/workflows/build-centos-7.yml |   3 +
 .github/workflows/builds.yml         | 126 +++++++++++++++++++++------
 2 files changed, 104 insertions(+), 25 deletions(-)

diff --git a/.github/workflows/build-centos-7.yml b/.github/workflows/build-centos-7.yml
index 83f853306669..49744252de25 100644
--- a/.github/workflows/build-centos-7.yml
+++ b/.github/workflows/build-centos-7.yml
@@ -40,6 +40,9 @@ jobs:
           path: /var/cache/yum
           key: ${{ github.job }}-yum
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           yum -y install epel-release
           yum -y install \
diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index df679b1bbdec..fd294bb2e95d 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -193,6 +193,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - uses: actions/checkout@v3.5.3
 
       # Download and extract dependency archives created during prep
@@ -266,9 +269,10 @@ jobs:
         run: |
           ./autogen.sh
           CFLAGS="${DEFAULT_CFLAGS}" ./configure
-      - run: make -j2 distcheck
+      - run: make -j ${{ env.CPUS }} distcheck
         env:
           DISTCHECK_CONFIGURE_FLAGS: "--enable-unittests --enable-debug --enable-lua --enable-geoip --enable-profiling --enable-profiling-locks --enable-dpdk"
+          MAKEFLAGS: "-j ${{ env.CPUS }}"
       - run: test -e doc/userguide/suricata.1
       - name: Checking includes
         run: |
@@ -321,6 +325,9 @@ jobs:
           path: ~/.cargo/registry
           key: cargo-registry
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - uses: actions/checkout@v3.5.3
 
       # Download and extract dependency archives created during prep
@@ -378,9 +385,9 @@ jobs:
         run: |
           ./autogen.sh
           CFLAGS="${DEFAULT_CFLAGS}" ./configure
-          make -j2
+          make -j ${{ env.CPUS }}
       - run: ./scripts/setup-app-layer.py --parser --logger --detect FooBar payload
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ./src/suricata --list-app-layer-protos | grep foobar
       - name: Verify rustfmt
         run: rustfmt -v --check src/applayerfoobar/*.rs
@@ -409,6 +416,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - uses: actions/checkout@v3.5.3
 
       # Prebuild check for duplicate SIDs
@@ -477,7 +487,7 @@ jobs:
         run: |
           ./autogen.sh
           CFLAGS="${DEFAULT_CFLAGS}" ./configure
-      - run: make -j2 check
+      - run: make -j ${{ env.CPUS }} check
       - name: Checking includes
         run: |
           cppclean src/*.h | grep "does not need to be #included" | python3 scripts/cppclean_check.py
@@ -505,6 +515,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install system packages
         run: |
           dnf -y install dnf-plugins-core epel-release
@@ -552,7 +565,7 @@ jobs:
       - run: tar zxvf suricata-*.tar.gz --strip-components=1
       - name: ./configure
         run: CFLAGS="${DEFAULT_CFLAGS}" ./configure
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make install
       - run: make install-conf
       - run: suricatasc -h
@@ -594,6 +607,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install system packages
         run: |
           dnf -y install dnf-plugins-core epel-release
@@ -638,7 +654,7 @@ jobs:
       - run: tar zxvf suricata-*.tar.gz --strip-components=1
       - name: ./configure
         run: CFLAGS="${DEFAULT_CFLAGS}" ./configure
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make install
       - run: make install-conf
       - run: suricatasc -h
@@ -681,6 +697,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -735,7 +754,7 @@ jobs:
           CC: "clang"
           RUSTFLAGS: "-C instrument-coverage"
           CFLAGS: "-fprofile-instr-generate -fcoverage-mapping -O0"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
         env:
           CC: "clang"
           RUSTFLAGS: "-C instrument-coverage"
@@ -774,6 +793,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -833,7 +855,7 @@ jobs:
           LDFLAGS: "-fsanitize=address"
           ac_cv_func_realloc_0_nonnull: "yes"
           ac_cv_func_malloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ASAN_OPTIONS="detect_leaks=0" ./src/suricata -u -l .
       - name: Extracting suricata-verify
         run: tar xf prep/suricata-verify.tar.gz
@@ -872,6 +894,9 @@ jobs:
           path: ~/.cargo/registry
           key: cargo-registry
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -921,7 +946,7 @@ jobs:
           LDFLAGS: "-fsanitize=address"
           ac_cv_func_realloc_0_nonnull: "yes"
           ac_cv_func_malloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ASAN_OPTIONS="detect_leaks=0" ./src/suricata -u -l .
       - name: Extracting suricata-verify
         run: tar xf prep/suricata-verify.tar.gz
@@ -965,6 +990,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -1017,7 +1045,7 @@ jobs:
           LDFLAGS: "-fsanitize=address"
           ac_cv_func_realloc_0_nonnull: "yes"
           ac_cv_func_malloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ASAN_OPTIONS="detect_leaks=0" ./src/suricata -u -l .
       - name: Extracting suricata-verify
         run: tar xf prep/suricata-verify.tar.gz
@@ -1056,6 +1084,9 @@ jobs:
           path: ~/.cargo/registry
           key: cargo-registry
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -1105,7 +1136,7 @@ jobs:
           LDFLAGS: "-fsanitize=address"
           ac_cv_func_realloc_0_nonnull: "yes"
           ac_cv_func_malloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ASAN_OPTIONS="detect_leaks=0" ./src/suricata -u -l .
       - name: Extracting suricata-verify
         run: tar xf prep/suricata-verify.tar.gz
@@ -1136,6 +1167,9 @@ jobs:
     container: fedora:39
     needs: [prepare-deps]
     steps:
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install \
                 autoconf \
@@ -1198,7 +1232,7 @@ jobs:
           CC: "clang"
           CFLAGS: "${{ env.DEFAULT_CFLAGS }} -Wshadow -fsanitize=address -fno-omit-frame-pointer"
 
-      - run: sudo -u suricata -s env PATH="/home/suricata/.cargo/bin:$PATH" make -j2
+      - run: sudo -u suricata -s env PATH="/home/suricata/.cargo/bin:$PATH" make -j ${{ env.CPUS }}
         working-directory: /home/suricata/suricata
 
       - run: sudo -u suricata -s make check
@@ -1229,6 +1263,9 @@ jobs:
           key: ${{ github.job }}-dnf
       - run: echo "keepcache=1" >> /etc/dnf/dnf.conf
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           dnf -y install dnf-plugins-core epel-release
           dnf config-manager --set-enable crb
@@ -1289,6 +1326,9 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1354,7 +1394,7 @@ jobs:
           RUSTFLAGS: "-C instrument-coverage"
           CFLAGS: "-fprofile-instr-generate -fcoverage-mapping -O0"
           CXXFLAGS: "-fprofile-instr-generate -fcoverage-mapping -O0"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
         env:
           CC: "clang-14"
           CXX: "clang++-14"
@@ -1405,6 +1445,9 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1476,7 +1519,7 @@ jobs:
           CXXFLAGS: "-fprofile-instr-generate -fcoverage-mapping -O0 -g -fno-strict-aliasing -fsanitize=address -fno-omit-frame-pointer -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION=1 -stdlib=libc++ -Wimplicit-int-float-conversion -Wimplicit-int-conversion"
           ac_cv_func_malloc_0_nonnull: "yes"
           ac_cv_func_realloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
         env:
           CC: "clang-14"
           CXX: "clang++-14"
@@ -1502,6 +1545,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1555,7 +1601,7 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: CFLAGS="$DEFAULT_CFLAGS -DNDEBUG" ./configure --enable-unittests
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
       - run: make dist
       - name: Extracting suricata-verify
@@ -1589,6 +1635,9 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1649,6 +1698,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1704,7 +1756,7 @@ jobs:
           LDFLAGS: "-fsanitize=address" 
           ac_cv_func_malloc_0_nonnull: "yes"
           ac_cv_func_realloc_0_nonnull: "yes"
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
       - name: Extracting suricata-verify
         run: tar xf prep/suricata-verify.tar.gz
@@ -1726,6 +1778,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1773,7 +1828,7 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: AFL_HARDEN=1 ac_cv_func_realloc_0_nonnull=yes ac_cv_func_malloc_0_nonnull=yes CFLAGS="-fsanitize=address -fno-omit-frame-pointer" CXXFLAGS=$CFLAGS CC=afl-clang-fast CXX=afl-clang-fast++ LDFLAGS="-fsanitize=address" ./configure --enable-fuzztargets --disable-shared
-      - run: AFL_HARDEN=1 make -j2
+      - run: AFL_HARDEN=1 make -j ${{ env.CPUS }}
 
   ubuntu-22-04-netmap-build:
     name: Ubuntu 22.04 (Netmap build)
@@ -1787,6 +1842,9 @@ jobs:
           path: ~/.cargo/registry
           key: cargo-registry
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
             sudo apt update
@@ -1842,7 +1900,7 @@ jobs:
         run: |
           cd $GITHUB_WORKSPACE/netmap/LINUX
           ./configure --no-drivers
-          make -j2
+          make -j ${{ env.CPUS }}
           sudo make install
 
       - uses: actions/checkout@v3.5.3
@@ -1859,7 +1917,7 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-netmap
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: ./src/suricata --build-info | grep -E "Netmap support:\s+yes"
 
   ubuntu-22-04-dpdk-build:
@@ -1879,6 +1937,9 @@ jobs:
           path: ~/.cargo/registry
           key: cargo-registry
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - name: Install dependencies
         run: |
           apt update
@@ -1956,7 +2017,7 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-dpdk
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
 
   debian-12:
@@ -1972,6 +2033,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: apt update
       - run: |
           apt -y install \
@@ -2028,7 +2092,7 @@ jobs:
       - run: tar xf prep/suricata-verify.tar.gz
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
         # -j2 caused random failures during cargo vendor
       - run: make distcheck
@@ -2058,6 +2122,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: apt update
       - run: |
           apt -y install \
@@ -2135,6 +2202,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: apt update
       - run: |
           apt -y install \
@@ -2196,7 +2266,7 @@ jobs:
       - run: tar xf prep/suricata-verify.tar.gz
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests --enable-debug --enable-lua --enable-geoip --enable-profiling --enable-profiling-locks --enable-dpdk
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
       - name: Building Rust documentation
         run: make doc
@@ -2220,6 +2290,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           echo "deb http://deb.debian.org/debian bullseye-backports main" >> /etc/apt/sources.list
           apt update
@@ -2274,7 +2347,7 @@ jobs:
           chmod 755 $HOME/.cargo/bin/cbindgen
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests --enable-fuzztargets --enable-ebpf --enable-ebpf-build
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
       - run: tar xf prep/suricata-verify.tar.gz
       - name: Running suricata-verify
@@ -2296,6 +2369,9 @@ jobs:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
+      - name: Determine number of CPUs
+        run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
+
       - run: |
           apt update
           apt -y install \
@@ -2347,7 +2423,7 @@ jobs:
           chmod 755 $HOME/.cargo/bin/cbindgen
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests --enable-fuzztargets
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       - run: make check
       - run: tar xf prep/suricata-verify.tar.gz
       - name: Running suricata-verify
@@ -2398,7 +2474,7 @@ jobs:
       - run: tar xvf prep/suricata-update.tar.gz
       - run: ./autogen.sh
       - run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-unittests
-      - run: make -j2
+      - run: make -j ${{ env.CPUS }}
       # somehow it gets included by some C++ stdlib header (case unsensitive)
       - run: rm libhtp/VERSION && make check
       - run: tar xf prep/suricata-verify.tar.gz

From 5bfaeb3bf5b8cd3cda27f6d7f5fec54160c0f79b Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 00:12:58 -0600
Subject: [PATCH 424/462] github-ci: update {download,upload} artifact actions

Multiple uploads can no longer use the same name, so give the cbindgen
artifact its own name of "cbindgen". Requires an additional download
for each build depending on this cbindgen artifact.
---
 .github/workflows/authors.yml             |   2 +-
 .github/workflows/builds.yml              | 114 +++++++++++++++-------
 .github/workflows/cifuzz.yml              |   2 +-
 .github/workflows/commits.yml             |   2 +-
 .github/workflows/scorecards-analysis.yml |   2 +-
 5 files changed, 83 insertions(+), 39 deletions(-)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index 5c4702a13858..6331ed4c12d5 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -35,7 +35,7 @@ jobs:
       - run: echo ${{ github.event.number }} > new-authors/pr-number.txt
       - run: ls -l
       - name: Upload new authors
-        uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
         with:
           name: new-authors
           path: new-authors
diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index fd294bb2e95d..ee53953101e6 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -141,7 +141,7 @@ jobs:
           cd ..
           tar zcf suricata-verify.tar.gz suricata-verify
       - name: Uploading prep archive
-        uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
         with:
           name: prep
           path: |
@@ -168,9 +168,9 @@ jobs:
           cargo install --target x86_64-unknown-linux-musl --debug cbindgen
           cp $HOME/.cargo/bin/cbindgen .
       - name: Uploading prep archive
-        uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
         with:
-          name: prep
+          name: cbindgen
           path: .
 
   almalinux-9:
@@ -200,7 +200,7 @@ jobs:
 
       # Download and extract dependency archives created during prep
       # job.
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -332,7 +332,7 @@ jobs:
 
       # Download and extract dependency archives created during prep
       # job.
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -432,13 +432,17 @@ jobs:
 
       # Download and extract dependency archives created during prep
       # job.
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xvf prep/libhtp.tar.gz
       - run: tar xvf prep/suricata-update.tar.gz
       - run: tar xvf prep/suricata-verify.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -559,7 +563,7 @@ jobs:
                 which \
                 zlib-devel
       - name: Download suricata.tar.gz
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: dist
       - run: tar zxvf suricata-*.tar.gz --strip-components=1
@@ -578,7 +582,7 @@ jobs:
           test -e /usr/local/lib/suricata/python/suricata/update/configs/modify.conf
           test -e /usr/local/lib/suricata/python/suricata/update/configs/threshold.in
           test -e /usr/local/lib/suricata/python/suricata/update/configs/update.yaml
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -648,7 +652,7 @@ jobs:
                 which \
                 zlib-devel
       - name: Download suricata.tar.gz
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: dist
       - run: tar zxvf suricata-*.tar.gz --strip-components=1
@@ -667,7 +671,7 @@ jobs:
           test -e /usr/local/lib/suricata/python/suricata/update/configs/modify.conf
           test -e /usr/local/lib/suricata/python/suricata/update/configs/threshold.in
           test -e /usr/local/lib/suricata/python/suricata/update/configs/update.yaml
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -742,7 +746,7 @@ jobs:
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -838,7 +842,7 @@ jobs:
                 which \
                 zlib-devel
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -933,7 +937,7 @@ jobs:
                 which \
                 zlib-devel
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -1033,7 +1037,7 @@ jobs:
                 which \
                 zlib-devel
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -1123,7 +1127,7 @@ jobs:
                 which \
                 zlib-devel
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -1209,7 +1213,7 @@ jobs:
                 zlib-devel
       - run: adduser suricata
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -1301,7 +1305,7 @@ jobs:
                 which \
                 zlib-devel
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -1375,11 +1379,15 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -1497,11 +1505,15 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -1587,12 +1599,16 @@ jobs:
                 exuberant-ctags \
                 dpdk-dev
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
       - run: tar xf prep/suricata-update.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -1672,7 +1688,7 @@ jobs:
       - run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.62.0 -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - name: Download suricata.tar.gz
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: dist
       - run: tar zxvf suricata-*.tar.gz --strip-components=1
@@ -1738,11 +1754,15 @@ jobs:
                 zlib1g-dev \
                 exuberant-ctags
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -1815,11 +1835,15 @@ jobs:
                 zlib1g-dev
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -1904,11 +1928,15 @@ jobs:
           sudo make install
 
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -2004,11 +2032,15 @@ jobs:
           ldconfig
           cd $HOME
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -2083,7 +2115,7 @@ jobs:
               zlib1g \
               zlib1g-dev
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -2168,7 +2200,7 @@ jobs:
               zlib1g \
               zlib1g-dev
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -2183,7 +2215,7 @@ jobs:
         run: |
           mkdir dist
           mv suricata-*.tar.gz dist
-      - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+      - uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
         name: Uploading distribution
         with:
           name: dist
@@ -2252,10 +2284,14 @@ jobs:
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain ${RUST_VERSION_MIN} -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -2334,12 +2370,16 @@ jobs:
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain $RUST_VERSION_KNOWN -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
       - run: tar xf prep/suricata-update.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -2410,12 +2450,16 @@ jobs:
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain $RUST_VERSION_KNOWN -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
       - run: tar xf prep/libhtp.tar.gz
       - run: tar xf prep/suricata-update.tar.gz
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
+        with:
+          name: cbindgen
+          path: prep
       - name: Setup cbindgen
         run: |
           mkdir -p $HOME/.cargo/bin
@@ -2466,7 +2510,7 @@ jobs:
       - run: pip3 install PyYAML
       - uses: actions/checkout@v3.5.3
       - name: Downloading prep archive
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -2508,7 +2552,7 @@ jobs:
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -2564,7 +2608,7 @@ jobs:
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
@@ -2608,7 +2652,7 @@ jobs:
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
       - uses: actions/checkout@v3.5.3
-      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
           path: prep
diff --git a/.github/workflows/cifuzz.yml b/.github/workflows/cifuzz.yml
index 8a55272cccbd..c9816e1e5497 100644
--- a/.github/workflows/cifuzz.yml
+++ b/.github/workflows/cifuzz.yml
@@ -28,7 +28,7 @@ jobs:
        dry-run: false
        sanitizer: ${{ matrix.sanitizer }}
    - name: Upload Crash
-     uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+     uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
      if: failure()
      with:
        name: ${{ matrix.sanitizer }}-artifacts
diff --git a/.github/workflows/commits.yml b/.github/workflows/commits.yml
index 6729472747b8..9f4437bfe95d 100644
--- a/.github/workflows/commits.yml
+++ b/.github/workflows/commits.yml
@@ -93,7 +93,7 @@ jobs:
               make -ik distclean > /dev/null
           done
       - run: sccache -s
-      - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
+      - uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
         name: Uploading build log
         if: always()
         with:
diff --git a/.github/workflows/scorecards-analysis.yml b/.github/workflows/scorecards-analysis.yml
index 07d4eda8121f..66647b239e21 100644
--- a/.github/workflows/scorecards-analysis.yml
+++ b/.github/workflows/scorecards-analysis.yml
@@ -39,7 +39,7 @@ jobs:
       # https://docs.github.com/en/actions/advanced-guides/storing-workflow-data-as-artifacts
       # Optional.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v3
+        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.1.2
         with:
           name: SARIF file
           path: results.sarif

From 32d55febedc0862c7e6ee589700ca549b34b32c5 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 00:52:11 -0600
Subject: [PATCH 425/462] github-ci: update actions/cache

---
 .github/workflows/builds.yml     | 76 ++++++++++++++++----------------
 .github/workflows/commits.yml    |  2 +-
 .github/workflows/formatting.yml |  2 +-
 .github/workflows/rust.yml       |  2 +-
 .github/workflows/scan-build.yml |  2 +-
 5 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index ee53953101e6..fdc282e14e1d 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -154,7 +154,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -181,13 +181,13 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -311,7 +311,7 @@ jobs:
     needs: [prepare-deps]
     steps:
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           # TODO: Find some variable that matches the job name.
@@ -320,7 +320,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
@@ -404,13 +404,13 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -507,13 +507,13 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -599,13 +599,13 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -689,13 +689,13 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -785,13 +785,13 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -893,7 +893,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
@@ -982,13 +982,13 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -1083,7 +1083,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
@@ -1255,13 +1255,13 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
 
       - name: Cache RPMs
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: /var/cache/dnf
           key: ${{ github.job }}-dnf
@@ -1326,7 +1326,7 @@ jobs:
     needs: [prepare-deps, prepare-cbindgen]
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1449,7 +1449,7 @@ jobs:
     needs: [prepare-deps, prepare-cbindgen]
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1552,7 +1552,7 @@ jobs:
     needs: [prepare-deps, prepare-cbindgen]
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1647,7 +1647,7 @@ jobs:
     needs: debian-12-dist
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1709,7 +1709,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1793,7 +1793,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -1861,7 +1861,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
@@ -1960,7 +1960,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
@@ -2060,7 +2060,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2149,7 +2149,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2229,7 +2229,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2321,7 +2321,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2404,7 +2404,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2483,7 +2483,7 @@ jobs:
     steps:
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2537,7 +2537,7 @@ jobs:
         shell: msys2 {0}
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2593,7 +2593,7 @@ jobs:
         shell: msys2 {0}
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
@@ -2637,7 +2637,7 @@ jobs:
         shell: msys2 {0}
     steps:
       - name: Cache ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
diff --git a/.github/workflows/commits.yml b/.github/workflows/commits.yml
index 9f4437bfe95d..e80375643b40 100644
--- a/.github/workflows/commits.yml
+++ b/.github/workflows/commits.yml
@@ -16,7 +16,7 @@ jobs:
     container: ubuntu:20.04
     steps:
       - name: Caching ~/.cargo
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: commit-check-cargo
diff --git a/.github/workflows/formatting.yml b/.github/workflows/formatting.yml
index 03a8e81169f5..164f365807cc 100644
--- a/.github/workflows/formatting.yml
+++ b/.github/workflows/formatting.yml
@@ -29,7 +29,7 @@ jobs:
 
       # Cache Rust stuff.
       - name: Cache cargo registry
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo/registry
           key: cargo-registry
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index e6bf4f4211f4..c8fd6797bf87 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -18,7 +18,7 @@ jobs:
     container: almalinux:9
     steps:
       - name: Cache rust
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: check-rust
diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml
index fc7cf4d3f962..608cffb6c3d9 100644
--- a/.github/workflows/scan-build.yml
+++ b/.github/workflows/scan-build.yml
@@ -15,7 +15,7 @@ jobs:
     container: ubuntu:23.04
     steps:
       - name: Cache scan-build
-        uses: actions/cache@v3.3.1
+        uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
         with:
           path: ~/.cargo
           key: scan-build

From e78629749789ca656136f699bccb7a7a2fad0d6c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 00:53:43 -0600
Subject: [PATCH 426/462] github-ci: update actions/checkout

---
 .github/workflows/authors.yml    |  2 +-
 .github/workflows/builds.yml     | 62 ++++++++++++++++----------------
 .github/workflows/codeql.yml     |  2 +-
 .github/workflows/commits.yml    |  2 +-
 .github/workflows/formatting.yml |  2 +-
 .github/workflows/rust.yml       |  2 +-
 .github/workflows/scan-build.yml |  2 +-
 7 files changed, 37 insertions(+), 37 deletions(-)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index 6331ed4c12d5..b7efe1186446 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -9,7 +9,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout PR code
-        uses: actions/checkout@v3
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
         with:
           ref: ${{ github.event.pull_request.head.sha }}
           fetch-depth: 0
diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index fdc282e14e1d..6d8b7379a321 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -108,7 +108,7 @@ jobs:
 
       # Now checkout Suricata for the bundle script.
       - name: Checking out Suricata
-        uses: actions/checkout@v3.5.3
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
 
       - name: Fetching libhtp
         run: |
@@ -196,7 +196,7 @@ jobs:
       - name: Determine number of CPUs
         run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
 
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
 
       # Download and extract dependency archives created during prep
       # job.
@@ -328,7 +328,7 @@ jobs:
       - name: Determine number of CPUs
         run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
 
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
 
       # Download and extract dependency archives created during prep
       # job.
@@ -419,7 +419,7 @@ jobs:
       - name: Determine number of CPUs
         run: echo CPUS=$(nproc --all) >> $GITHUB_ENV
 
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
 
       # Prebuild check for duplicate SIDs
       - name: Check for duplicate SIDs
@@ -745,7 +745,7 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -841,7 +841,7 @@ jobs:
                 systemd-devel \
                 which \
                 zlib-devel
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -936,7 +936,7 @@ jobs:
                 sudo \
                 which \
                 zlib-devel
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1036,7 +1036,7 @@ jobs:
                 systemd-devel \
                 which \
                 zlib-devel
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1126,7 +1126,7 @@ jobs:
                 sudo \
                 which \
                 zlib-devel
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1212,7 +1212,7 @@ jobs:
                 which \
                 zlib-devel
       - run: adduser suricata
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1304,7 +1304,7 @@ jobs:
                 sudo \
                 which \
                 zlib-devel
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1378,7 +1378,7 @@ jobs:
       # packaged Rust version is too old for coverage, so get from rustup
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1504,7 +1504,7 @@ jobs:
       # packaged Rust version is too old for coverage, so get from rustup
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.63.0 -y
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1598,7 +1598,7 @@ jobs:
                 zlib1g-dev \
                 exuberant-ctags \
                 dpdk-dev
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1753,7 +1753,7 @@ jobs:
                 zlib1g \
                 zlib1g-dev \
                 exuberant-ctags
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1834,7 +1834,7 @@ jobs:
                 zlib1g \
                 zlib1g-dev
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -1914,7 +1914,7 @@ jobs:
                 linux-headers-$(uname -r)
 
       - name: Checkout Netmap repository
-        uses: actions/checkout@v3.5.3
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
         with:
           repository: luigirizzo/netmap
           # gets cloned to $GITHUB_WORKSPACE/netmap/
@@ -1927,7 +1927,7 @@ jobs:
           make -j ${{ env.CPUS }}
           sudo make install
 
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2031,7 +2031,7 @@ jobs:
           ninja -C build install
           ldconfig
           cd $HOME
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2114,7 +2114,7 @@ jobs:
               texlive-latex-extra \
               zlib1g \
               zlib1g-dev
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2199,7 +2199,7 @@ jobs:
               texlive-latex-extra \
               zlib1g \
               zlib1g-dev
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2283,7 +2283,7 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain ${RUST_VERSION_MIN} -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2369,7 +2369,7 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain $RUST_VERSION_KNOWN -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2449,7 +2449,7 @@ jobs:
       - name: Install Rust
         run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain $RUST_VERSION_KNOWN -y
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2508,7 +2508,7 @@ jobs:
         run: cargo install --debug --version 0.24.3 cbindgen
       - run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - run: pip3 install PyYAML
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - name: Downloading prep archive
         uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
@@ -2541,7 +2541,7 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: msys2/setup-msys2@v2
         with:
           msystem: MINGW64
@@ -2551,7 +2551,7 @@ jobs:
       # preinstalled one to be picked up by configure
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2597,7 +2597,7 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: msys2/setup-msys2@v2
         with:
           msystem: MINGW64
@@ -2607,7 +2607,7 @@ jobs:
       # preinstalled one to be picked up by configure
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
@@ -2641,7 +2641,7 @@ jobs:
         with:
           path: ~/.cargo
           key: ${{ github.job }}-cargo
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: msys2/setup-msys2@v2
         with:
           msystem: MINGW64
@@ -2651,7 +2651,7 @@ jobs:
       # preinstalled one to be picked up by configure
       - name: cbindgen
         run: cargo install --root /usr --force --debug --version 0.24.3 cbindgen
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe
         with:
           name: prep
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index 3d13d276b02a..c59b71212851 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -31,7 +31,7 @@ jobs:
         # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3.5.3
+      uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
diff --git a/.github/workflows/commits.yml b/.github/workflows/commits.yml
index e80375643b40..1ca2a48c5875 100644
--- a/.github/workflows/commits.yml
+++ b/.github/workflows/commits.yml
@@ -70,7 +70,7 @@ jobs:
           cd $HOME/.cargo/bin
           curl -OL https://github.com/eqrion/cbindgen/releases/download/v0.24.3/cbindgen
           chmod 755 cbindgen
-      - uses: actions/checkout@v3.3.0
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
         with:
           fetch-depth: 0
       # The action above is supposed to do this for us, but it doesn't appear to stick.
diff --git a/.github/workflows/formatting.yml b/.github/workflows/formatting.yml
index 164f365807cc..ca3ddc87c894 100644
--- a/.github/workflows/formatting.yml
+++ b/.github/workflows/formatting.yml
@@ -85,7 +85,7 @@ jobs:
       # My patience simply ran too short to keep on looking. See follow-on
       # action to manually fix this up.
       - name: Checkout - might be merge commit!
-        uses: actions/checkout@v3.5.3
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
         with:
           fetch-depth: 0
         # Use last commit of branch, not potential merge commit!
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index c8fd6797bf87..0b38f36ae426 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -68,7 +68,7 @@ jobs:
           echo "$HOME/.cargo/bin" >> $GITHUB_PATH
       - name: Install cbindgen
         run: cargo install --debug cbindgen
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - run: git config --global --add safe.directory /__w/suricata/suricata
       - run: ./scripts/bundle.sh
       - run: ./autogen.sh
diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml
index 608cffb6c3d9..480970658205 100644
--- a/.github/workflows/scan-build.yml
+++ b/.github/workflows/scan-build.yml
@@ -62,7 +62,7 @@ jobs:
                 software-properties-common \
                 zlib1g \
                 zlib1g-dev
-      - uses: actions/checkout@v3.5.3
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - run: ./scripts/bundle.sh
       - run: ./autogen.sh
       - run: scan-build-16 ./configure --enable-dpdk --enable-nfqueue --enable-nflog

From 49834eabf1ca0bf36e8249d2caf37e6d2172dac9 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 10:57:40 -0600
Subject: [PATCH 427/462] github-ci: update actions/github-script

---
 .github/workflows/authors-done.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/authors-done.yml b/.github/workflows/authors-done.yml
index d0fdf4491c45..fa449745f987 100644
--- a/.github/workflows/authors-done.yml
+++ b/.github/workflows/authors-done.yml
@@ -12,7 +12,7 @@ jobs:
       - run: echo "Author check is complete"
 
       - name: Download artifact new authors
-        uses: actions/github-script@v6
+        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea
         with:
           script: |
             let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
@@ -38,7 +38,7 @@ jobs:
           fi
       - name: Comment on PR
         if: ${{ env.new_authors == 'yes' }}
-        uses: actions/github-script@v6
+        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |

From d5a3bfcab6c9b3d9174bc96a281c21237aaf774c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 11:05:17 -0600
Subject: [PATCH 428/462] github-ci: don't depend on cbindgen when installed
 from package

---
 .github/workflows/builds.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 6d8b7379a321..095998042792 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -977,7 +977,7 @@ jobs:
     name: Fedora 39 (clang, debug, asan, wshadow, rust-strict, systemd)
     runs-on: ubuntu-latest
     container: fedora:39
-    needs: [prepare-deps, prepare-cbindgen]
+    needs: [prepare-deps]
     steps:
 
       # Cache Rust stuff.
@@ -1078,7 +1078,7 @@ jobs:
     name: Fedora 39 (gcc, debug, asan, wshadow, rust-strict)
     runs-on: ubuntu-latest
     container: fedora:39
-    needs: [prepare-deps, prepare-cbindgen]
+    needs: [prepare-deps]
     steps:
 
       # Cache Rust stuff.

From 7c981346247cd560ccbab3a5708f05b97db62f8c Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Mon, 12 Feb 2024 15:13:30 -0600
Subject: [PATCH 429/462] github-ci: cancel previous job for all workflows

Previously only enabled in build.yml, apply cancen-in-progress to all
workflow files.
---
 .github/workflows/authors.yml             | 4 ++++
 .github/workflows/cifuzz.yml              | 4 ++++
 .github/workflows/codeql.yml              | 4 ++++
 .github/workflows/commits.yml             | 4 ++++
 .github/workflows/formatting.yml          | 4 ++++
 .github/workflows/rust.yml                | 4 ++++
 .github/workflows/scan-build.yml          | 4 ++++
 .github/workflows/scorecards-analysis.yml | 4 ++++
 8 files changed, 32 insertions(+)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index b7efe1186446..77bb2614de24 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -3,6 +3,10 @@ name: New Authors Check
 on:
   pull_request:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check-id:
     name: New Author Check
diff --git a/.github/workflows/cifuzz.yml b/.github/workflows/cifuzz.yml
index c9816e1e5497..5ebcb6d1cf56 100644
--- a/.github/workflows/cifuzz.yml
+++ b/.github/workflows/cifuzz.yml
@@ -5,6 +5,10 @@ on:
     paths-ignore:
       - "doc/**"
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 permissions: read-all
 jobs:
  Fuzzing:
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index c59b71212851..31be87c39da0 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -13,6 +13,10 @@ on:
   schedule:
     - cron: '18 21 * * 1'
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   analyze:
     name: Analyze
diff --git a/.github/workflows/commits.yml b/.github/workflows/commits.yml
index 1ca2a48c5875..7d1d87350bf8 100644
--- a/.github/workflows/commits.yml
+++ b/.github/workflows/commits.yml
@@ -5,6 +5,10 @@ on:
 
 permissions: read-all
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 env:
   DEBIAN_FRONTEND: "noninteractive"
 
diff --git a/.github/workflows/formatting.yml b/.github/workflows/formatting.yml
index ca3ddc87c894..5adf1dd86f2e 100644
--- a/.github/workflows/formatting.yml
+++ b/.github/workflows/formatting.yml
@@ -12,6 +12,10 @@ on:
     paths-ignore:
       - "doc/**"
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 permissions: read-all
 
 env:
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 0b38f36ae426..eebf43c13eb8 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -11,6 +11,10 @@ on:
 permissions:
   contents: read #  to fetch code (actions/checkout)
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check-rust:
     name: Check Rust
diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml
index 480970658205..f0df97e7c80e 100644
--- a/.github/workflows/scan-build.yml
+++ b/.github/workflows/scan-build.yml
@@ -8,6 +8,10 @@ on:
     paths-ignore:
       - "doc/**"
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   scan-build:
     name: Scan-build
diff --git a/.github/workflows/scorecards-analysis.yml b/.github/workflows/scorecards-analysis.yml
index 66647b239e21..f63517641bb0 100644
--- a/.github/workflows/scorecards-analysis.yml
+++ b/.github/workflows/scorecards-analysis.yml
@@ -7,6 +7,10 @@ on:
   push:
     branches: [ master ]
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 # Declare default permissions as read only.
 permissions: read-all
 

From be07d96c3df84b42dbc0435abcee99f43a26ceab Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 12 Feb 2024 14:44:15 +0000
Subject: [PATCH 430/462] github-actions: bump codecov/codecov-action from
 3.1.1 to 4.0.1

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3.1.1 to 4.0.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70...e0b68c6749509c5f83f984dd99a76a1c1a231044)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
---
 .github/workflows/builds.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index 095998042792..b3527e94ab47 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -770,7 +770,7 @@ jobs:
       - run: llvm-profdata merge -o default.profdata $(find suricata-verify/tests/ -name '*.profraw')
       - run: llvm-cov show ./src/suricata -instr-profile=default.profdata --show-instantiations --ignore-filename-regex="^/root/.*" > coverage.txt
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70
+        uses: codecov/codecov-action@e0b68c6749509c5f83f984dd99a76a1c1a231044
         with:
           fail_ci_if_error: false
           flags: suricata-verify
@@ -1437,7 +1437,7 @@ jobs:
       - run: llvm-profdata-14 merge -o htp-test.profdata /tmp/htp-test.profraw
       - run: llvm-cov-14 show libhtp/test/test_all -instr-profile=htp-test.profdata --show-instantiations --ignore-filename-regex="^/root/.*" >> coverage.txt
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70
+        uses: codecov/codecov-action@e0b68c6749509c5f83f984dd99a76a1c1a231044
         with:
           fail_ci_if_error: false
           flags: unittests
@@ -1540,7 +1540,7 @@ jobs:
       - run: llvm-profdata-14 merge -o default.profdata $(find /tmp/ -name '*.profraw')
       - run: llvm-cov-14 show ./src/suricata -instr-profile=default.profdata --show-instantiations --ignore-filename-regex="^/root/.*" > coverage.txt
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70
+        uses: codecov/codecov-action@e0b68c6749509c5f83f984dd99a76a1c1a231044
         with:
           fail_ci_if_error: false
           flags: fuzzcorpus

From 7881e850883114929e661a3122f52bd5e80b475b Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue, 13 Feb 2024 11:15:09 +0000
Subject: [PATCH 431/462] github-actions: bump github/codeql-action from 2 to 3

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Commits](https://github.com/github/codeql-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
---
 .github/workflows/codeql.yml              | 4 ++--
 .github/workflows/scorecards-analysis.yml | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index 31be87c39da0..e467241adbf5 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -39,7 +39,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v2
+      uses: github/codeql-action/init@v3
       with:
         languages: ${{ matrix.language }}
 
@@ -59,4 +59,4 @@ jobs:
        ./configure
        make
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v2
+      uses: github/codeql-action/analyze@v3
diff --git a/.github/workflows/scorecards-analysis.yml b/.github/workflows/scorecards-analysis.yml
index f63517641bb0..1fad04559e40 100644
--- a/.github/workflows/scorecards-analysis.yml
+++ b/.github/workflows/scorecards-analysis.yml
@@ -51,6 +51,6 @@ jobs:
 
       # Upload the results to GitHub's code scanning dashboard.
       - name: "Upload SARIF results"
-        uses: github/codeql-action/upload-sarif@cdcdbb579706841c47f7063dda365e292e5cad7a # v1
+        uses: github/codeql-action/upload-sarif@dc021d495cb77b369e4d9d04a501700fd83b8c51 # v1
         with:
           sarif_file: results.sarif

From f9a4e9c588588f5fbb9c056f219a18470032f2d1 Mon Sep 17 00:00:00 2001
From: Daniel Olatunji <danielolatunji20@outlook.com>
Date: Fri, 26 Jan 2024 09:27:39 +0100
Subject: [PATCH 432/462] codeql: add security-extended query suite

Add the CodeQL security-extended suite to
the CodeQL workflow configuration.
---
 .github/workflows/codeql.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index e467241adbf5..e7fc1e43c06a 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -42,6 +42,7 @@ jobs:
       uses: github/codeql-action/init@v3
       with:
         languages: ${{ matrix.language }}
+        queries: security-extended
 
     - run: |
        sudo apt-get update

From e891ef3d4ed414987b485d88ac144e5d6fb981f0 Mon Sep 17 00:00:00 2001
From: jason taylor <jtfas90@gmail.com>
Date: Wed, 31 Jan 2024 14:51:58 +0000
Subject: [PATCH 433/462] doc: add pcap file logging variable details

Signed-off-by: jason taylor <jtfas90@gmail.com>
---
 doc/userguide/configuration/suricata-yaml.rst | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index c04573778b03..920be735302d 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -505,6 +505,27 @@ the alert.
       mode: normal # "normal" or multi
       conditional: alerts
 
+In ``normal`` mode a pcap file "filename" is created in the default-log-dir or as
+specified by "dir". ``normal`` mode is generally not as performant as ``multi``
+mode.
+
+In multi mode, multiple pcap files are created (per thread) which performs
+better than ``normal`` mode.
+
+In multi mode the filename takes a few special variables:
+  - %n representing the thread number
+  - %i representing the thread id
+  - %t representing the timestamp (secs or secs.usecs based on 'ts-format')
+  
+  Example: filename: pcap.%n.%t
+
+.. note:: It is possible to use directories but the directories are not
+  created by Suricata. For example ``filename: pcaps/%n/log.%s`` will log into
+  the pre-existing ``pcaps`` directory and per thread sub directories.
+  
+.. note:: that the limit and max-files settings are enforced per thread. So the
+  size limit using 8 threads with 1000mb files and 2000 files is about 16TiB.
+
 Verbose Alerts Log (alert-debug.log)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

From 2a1a70b3089751b30f623871063ce155451d4cbc Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Sun, 4 Feb 2024 09:44:44 -0500
Subject: [PATCH 434/462] threads/mutex: Ensure mutex held before signaling

Ensure that the mutex protecting the condition variable is held before
signaling it. This ensures that the thread(s) awaiting the signal are
notified.

Issue: 6569
---
 src/tm-threads.c  | 10 ++++++++++
 src/tmqh-simple.c |  5 ++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/tm-threads.c b/src/tm-threads.c
index 10ef45da278f..e1eee3b6412c 100644
--- a/src/tm-threads.c
+++ b/src/tm-threads.c
@@ -1239,13 +1239,17 @@ static int TmThreadKillThread(ThreadVars *tv)
         }
         if (tv->inq != NULL) {
             for (int i = 0; i < (tv->inq->reader_cnt + tv->inq->writer_cnt); i++) {
+                SCMutexLock(&tv->inq->pq->mutex_q);
                 SCCondSignal(&tv->inq->pq->cond_q);
+                SCMutexUnlock(&tv->inq->pq->mutex_q);
             }
             SCLogDebug("signalled tv->inq->id %" PRIu32 "", tv->inq->id);
         }
 
         if (tv->ctrl_cond != NULL ) {
+            SCCtrlMutexLock(tv->ctrl_mutex);
             pthread_cond_broadcast(tv->ctrl_cond);
+            SCCtrlMutexUnlock(tv->ctrl_mutex);
         }
         return 0;
     }
@@ -1425,7 +1429,9 @@ void TmThreadDisableReceiveThreads(void)
 
             if (tv->inq != NULL) {
                 for (int i = 0; i < (tv->inq->reader_cnt + tv->inq->writer_cnt); i++) {
+                    SCMutexLock(&tv->inq->pq->mutex_q);
                     SCCondSignal(&tv->inq->pq->cond_q);
+                    SCMutexUnlock(&tv->inq->pq->mutex_q);
                 }
                 SCLogDebug("signalled tv->inq->id %" PRIu32 "", tv->inq->id);
             }
@@ -1505,7 +1511,9 @@ void TmThreadDisablePacketThreads(void)
          * THV_KILL flag. */
         if (tv->inq != NULL) {
             for (int i = 0; i < (tv->inq->reader_cnt + tv->inq->writer_cnt); i++) {
+                SCMutexLock(&tv->inq->pq->mutex_q);
                 SCCondSignal(&tv->inq->pq->cond_q);
+                SCMutexUnlock(&tv->inq->pq->mutex_q);
             }
             SCLogDebug("signalled tv->inq->id %" PRIu32 "", tv->inq->id);
         }
@@ -2296,7 +2304,9 @@ void TmThreadsInjectFlowById(Flow *f, const int id)
 
     /* wake up listening thread(s) if necessary */
     if (tv->inq != NULL) {
+        SCMutexLock(&tv->inq->pq->mutex_q);
         SCCondSignal(&tv->inq->pq->cond_q);
+        SCMutexUnlock(&tv->inq->pq->mutex_q);
     } else if (tv->break_loop) {
         TmThreadsCaptureBreakLoop(tv);
     }
diff --git a/src/tmqh-simple.c b/src/tmqh-simple.c
index 47faed5702c5..0bfa173e5009 100644
--- a/src/tmqh-simple.c
+++ b/src/tmqh-simple.c
@@ -76,8 +76,11 @@ void TmqhInputSimpleShutdownHandler(ThreadVars *tv)
         return;
     }
 
-    for (i = 0; i < (tv->inq->reader_cnt + tv->inq->writer_cnt); i++)
+    for (i = 0; i < (tv->inq->reader_cnt + tv->inq->writer_cnt); i++) {
+        SCMutexLock(&tv->inq->pq->mutex_q);
         SCCondSignal(&tv->inq->pq->cond_q);
+        SCMutexUnlock(&tv->inq->pq->mutex_q);
+    }
 }
 
 void TmqhOutputSimple(ThreadVars *t, Packet *p)

From c99d93c2574cfa0fe2b94c1ab3a2cacd5a15035c Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Mon, 12 Feb 2024 13:42:14 +0100
Subject: [PATCH 435/462] app-layer/template: use a max number of txs

Ticket: 6773
---
 rust/src/applayertemplate/template.rs | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/rust/src/applayertemplate/template.rs b/rust/src/applayertemplate/template.rs
index acc6c26c37fa..dbbc7841fad5 100644
--- a/rust/src/applayertemplate/template.rs
+++ b/rust/src/applayertemplate/template.rs
@@ -17,6 +17,7 @@
 
 use super::parser;
 use crate::applayer::{self, *};
+use crate::conf::conf_get;
 use crate::core::{AppProto, Flow, ALPROTO_UNKNOWN, IPPROTO_TCP};
 use nom7 as nom;
 use std;
@@ -24,10 +25,14 @@ use std::collections::VecDeque;
 use std::ffi::CString;
 use std::os::raw::{c_char, c_int, c_void};
 
+static mut TEMPLATE_MAX_TX: usize = 256;
+
 static mut ALPROTO_TEMPLATE: AppProto = ALPROTO_UNKNOWN;
 
 #[derive(AppLayerEvent)]
-enum TemplateEvent {}
+enum TemplateEvent {
+    TooManyTransactions,
+}
 
 pub struct TemplateTransaction {
     tx_id: u64,
@@ -145,7 +150,13 @@ impl TemplateState {
                     SCLogNotice!("Request: {}", request);
                     let mut tx = self.new_tx();
                     tx.request = Some(request);
+                    if self.transactions.len() >= unsafe {TEMPLATE_MAX_TX} {
+                        tx.tx_data.set_event(TemplateEvent::TooManyTransactions as u8);
+                    }
                     self.transactions.push_back(tx);
+                    if self.transactions.len() >= unsafe {TEMPLATE_MAX_TX} {
+                        return AppLayerResult::err();
+                    }
                 }
                 Err(nom::Err::Incomplete(_)) => {
                     // Not enough data. This parser doesn't give us a good indication
@@ -429,6 +440,13 @@ pub unsafe extern "C" fn rs_template_register_parser() {
         if AppLayerParserConfParserEnabled(ip_proto_str.as_ptr(), parser.name) != 0 {
             let _ = AppLayerRegisterParser(&parser, alproto);
         }
+        if let Some(val) = conf_get("app-layer.protocols.template.max-tx") {
+            if let Ok(v) = val.parse::<usize>() {
+                TEMPLATE_MAX_TX = v;
+            } else {
+                SCLogError!("Invalid value for template.max-tx");
+            }
+        }
         SCLogNotice!("Rust template parser registered.");
     } else {
         SCLogNotice!("Protocol detector and parser disabled for TEMPLATE.");

From 3a7a4cd581b8140fa4c5f2e41d4b6ae5fdc81b4a Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Sat, 27 Jan 2024 22:13:37 +0100
Subject: [PATCH 436/462] http: code simplification

removing function unused parameter tx_id in HTPFileOpen
And using directly tx instead of its id in HTPFileOpenWithRange
---
 src/app-layer-htp-file.c | 13 ++++---------
 src/app-layer-htp-file.h |  6 +++---
 src/app-layer-htp.c      | 12 ++++++------
 3 files changed, 13 insertions(+), 18 deletions(-)

diff --git a/src/app-layer-htp-file.c b/src/app-layer-htp-file.c
index 7b3ba62edcee..b2c8776a7192 100644
--- a/src/app-layer-htp-file.c
+++ b/src/app-layer-htp-file.c
@@ -48,7 +48,7 @@ extern StreamingBufferConfig htp_sbcfg;
  *  \retval -2 not handling files on this flow
  */
 int HTPFileOpen(HtpState *s, HtpTxUserData *tx, const uint8_t *filename, uint16_t filename_len,
-        const uint8_t *data, uint32_t data_len, uint64_t txid, uint8_t direction)
+        const uint8_t *data, uint32_t data_len, uint8_t direction)
 {
     int retval = 0;
     uint16_t flags = 0;
@@ -147,8 +147,8 @@ static int HTPParseAndCheckContentRange(
  *  \retval -1 error
  */
 int HTPFileOpenWithRange(HtpState *s, HtpTxUserData *txud, const uint8_t *filename,
-        uint16_t filename_len, const uint8_t *data, uint32_t data_len, uint64_t txid,
-        bstr *rawvalue, HtpTxUserData *htud)
+        uint16_t filename_len, const uint8_t *data, uint32_t data_len, htp_tx_t *tx, bstr *rawvalue,
+        HtpTxUserData *htud)
 {
     SCEnter();
     uint16_t flags;
@@ -159,7 +159,7 @@ int HTPFileOpenWithRange(HtpState *s, HtpTxUserData *txud, const uint8_t *filena
     HTTPContentRange crparsed;
     if (HTPParseAndCheckContentRange(rawvalue, &crparsed, s, htud) != 0) {
         // range is invalid, fall back to classic open
-        return HTPFileOpen(s, txud, filename, filename_len, data, data_len, txid, STREAM_TOCLIENT);
+        return HTPFileOpen(s, txud, filename, filename_len, data, data_len, STREAM_TOCLIENT);
     }
     flags = FileFlowToFlags(s->f, STREAM_TOCLIENT);
     FileContainer *files = &txud->files_tc;
@@ -179,11 +179,6 @@ int HTPFileOpenWithRange(HtpState *s, HtpTxUserData *txud, const uint8_t *filena
     }
 
     // Then, we will try to handle reassembly of different ranges of the same file
-    // TODO have the caller pass directly the tx
-    htp_tx_t *tx = htp_list_get(s->conn->transactions, txid - s->tx_freed);
-    if (!tx) {
-        SCReturnInt(-1);
-    }
     uint8_t *keyurl;
     uint32_t keylen;
     if (tx->request_hostname != NULL) {
diff --git a/src/app-layer-htp-file.h b/src/app-layer-htp-file.h
index 4b682bc03781..b0436df22f54 100644
--- a/src/app-layer-htp-file.h
+++ b/src/app-layer-htp-file.h
@@ -27,10 +27,10 @@
 
 #include "app-layer-htp.h"
 
-int HTPFileOpen(HtpState *, HtpTxUserData *, const uint8_t *, uint16_t, const uint8_t *, uint32_t,
-        uint64_t, uint8_t);
+int HTPFileOpen(
+        HtpState *, HtpTxUserData *, const uint8_t *, uint16_t, const uint8_t *, uint32_t, uint8_t);
 int HTPFileOpenWithRange(HtpState *, HtpTxUserData *, const uint8_t *, uint16_t, const uint8_t *,
-        uint32_t, uint64_t, bstr *rawvalue, HtpTxUserData *htud);
+        uint32_t, htp_tx_t *, bstr *rawvalue, HtpTxUserData *htud);
 bool HTPFileCloseHandleRange(const StreamingBufferConfig *sbcfg, FileContainer *, const uint16_t,
         HttpRangeContainerBlock *, const uint8_t *, uint32_t);
 int HTPFileStoreChunk(HtpState *, HtpTxUserData *, const uint8_t *, uint32_t, uint8_t);
diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index 1d654c2c7c5b..f8e6e9e8de06 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -1571,7 +1571,7 @@ static int HtpRequestBodyHandleMultipart(HtpState *hstate, HtpTxUserData *htud,
 #endif
 
                 result = HTPFileOpen(hstate, htud, filename, filename_len, filedata, filedata_len,
-                        HtpGetActiveRequestTxID(hstate), STREAM_TOSERVER);
+                        STREAM_TOSERVER);
                 if (result == -1) {
                     goto end;
                 } else if (result == -2) {
@@ -1633,7 +1633,7 @@ static int HtpRequestBodyHandleMultipart(HtpState *hstate, HtpTxUserData *htud,
                         filedata_len = 0;
                     }
                     result = HTPFileOpen(hstate, htud, filename, filename_len, filedata,
-                            filedata_len, HtpGetActiveRequestTxID(hstate), STREAM_TOSERVER);
+                            filedata_len, STREAM_TOSERVER);
                     if (result == -1) {
                         goto end;
                     } else if (result == -2) {
@@ -1648,7 +1648,7 @@ static int HtpRequestBodyHandleMultipart(HtpState *hstate, HtpTxUserData *htud,
                     SCLogDebug("filedata_len %u", filedata_len);
 
                     result = HTPFileOpen(hstate, htud, filename, filename_len, filedata,
-                            filedata_len, HtpGetActiveRequestTxID(hstate), STREAM_TOSERVER);
+                            filedata_len, STREAM_TOSERVER);
                     if (result == -1) {
                         goto end;
                     } else if (result == -2) {
@@ -1725,7 +1725,7 @@ static int HtpRequestBodyHandlePOSTorPUT(HtpState *hstate, HtpTxUserData *htud,
                 HTPSetEvent(hstate, htud, STREAM_TOSERVER, HTTP_DECODER_EVENT_FILE_NAME_TOO_LONG);
             }
             result = HTPFileOpen(hstate, htud, filename, (uint16_t)filename_len, data, data_len,
-                    HtpGetActiveRequestTxID(hstate), STREAM_TOSERVER);
+                    STREAM_TOSERVER);
             if (result == -1) {
                 goto end;
             } else if (result == -2) {
@@ -1802,10 +1802,10 @@ static int HtpResponseBodyHandle(HtpState *hstate, HtpTxUserData *htud,
             }
             if (h_content_range != NULL) {
                 result = HTPFileOpenWithRange(hstate, htud, filename, (uint16_t)filename_len, data,
-                        data_len, HtpGetActiveResponseTxID(hstate), h_content_range->value, htud);
+                        data_len, tx, h_content_range->value, htud);
             } else {
                 result = HTPFileOpen(hstate, htud, filename, (uint16_t)filename_len, data, data_len,
-                        HtpGetActiveResponseTxID(hstate), STREAM_TOCLIENT);
+                        STREAM_TOCLIENT);
             }
             SCLogDebug("result %d", result);
             if (result == -1) {

From cc2eb2d8b77e96586a607f661c7eed9ab41076fc Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Sun, 11 Feb 2024 13:42:20 +0100
Subject: [PATCH 437/462] dpdk: sanitize integer overflow in the configuration

Ticket: #6737
---
 src/runmode-dpdk.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/runmode-dpdk.c b/src/runmode-dpdk.c
index 8a7643b250e6..67e1e0050ea1 100644
--- a/src/runmode-dpdk.c
+++ b/src/runmode-dpdk.c
@@ -475,6 +475,9 @@ static int ConfigSetMempoolSize(DPDKIfaceConfig *iconf, intmax_t entry_int)
     if (entry_int <= 0) {
         SCLogError("%s: positive memory pool size is required", iconf->iface);
         SCReturnInt(-ERANGE);
+    } else if (entry_int > UINT32_MAX) {
+        SCLogError("%s: memory pool size cannot exceed %" PRIu32, iconf->iface, UINT32_MAX);
+        SCReturnInt(-ERANGE);
     }
 
     iconf->mempool_size = entry_int;
@@ -521,6 +524,9 @@ static int ConfigSetRxDescriptors(DPDKIfaceConfig *iconf, intmax_t entry_int)
     if (entry_int <= 0) {
         SCLogError("%s: positive number of RX descriptors is required", iconf->iface);
         SCReturnInt(-ERANGE);
+    } else if (entry_int > UINT16_MAX) {
+        SCLogError("%s: number of RX descriptors cannot exceed %" PRIu16, iconf->iface, UINT16_MAX);
+        SCReturnInt(-ERANGE);
     }
 
     iconf->nb_rx_desc = entry_int;
@@ -533,6 +539,9 @@ static int ConfigSetTxDescriptors(DPDKIfaceConfig *iconf, intmax_t entry_int)
     if (entry_int <= 0) {
         SCLogError("%s: positive number of TX descriptors is required", iconf->iface);
         SCReturnInt(-ERANGE);
+    } else if (entry_int > UINT16_MAX) {
+        SCLogError("%s: number of TX descriptors cannot exceed %" PRIu16, iconf->iface, UINT16_MAX);
+        SCReturnInt(-ERANGE);
     }
 
     iconf->nb_tx_desc = entry_int;

From c65ff35819845a3f42c75f79d54f9ab91c5c2ec9 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lsismis@oisf.net>
Date: Sun, 11 Feb 2024 20:43:37 +0100
Subject: [PATCH 438/462] dpdk: max cache size should be lower than one of the
 constraints

Ticket: 6741
---
 src/runmode-dpdk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/runmode-dpdk.c b/src/runmode-dpdk.c
index 67e1e0050ea1..5bb071f7f1d7 100644
--- a/src/runmode-dpdk.c
+++ b/src/runmode-dpdk.c
@@ -498,7 +498,7 @@ static int ConfigSetMempoolCacheSize(DPDKIfaceConfig *iconf, const char *entry_s
             SCReturnInt(-EINVAL);
         }
 
-        uint32_t max_cache_size = MAX(RTE_MEMPOOL_CACHE_MAX_SIZE, iconf->mempool_size / 1.5);
+        uint32_t max_cache_size = MIN(RTE_MEMPOOL_CACHE_MAX_SIZE, iconf->mempool_size / 1.5);
         iconf->mempool_cache_size = GreatestDivisorUpTo(iconf->mempool_size, max_cache_size);
         SCReturnInt(0);
     }

From 356f9ffa130fbaaf82e7e28de98bdc24fe32f945 Mon Sep 17 00:00:00 2001
From: Lukas Sismis <lukas.sismis@gmail.com>
Date: Sun, 11 Feb 2024 21:14:08 +0100
Subject: [PATCH 439/462] doc: mention the limited number of RX/TX descriptors
 on Intel NICs

Ticket: 6748
---
 doc/userguide/configuration/suricata-yaml.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index 920be735302d..db9040aedc60 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -2138,7 +2138,11 @@ size of the cache is covered in the YAML file.
 To be able to run DPDK on Intel cards, it is required to change the default
 Intel driver to either `vfio-pci` or `igb_uio` driver. The process is
 described in `DPDK manual page regarding Linux drivers
-<https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html>`_.
+<https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html>`_. 
+The Intel NICs have the amount of RX/TX descriptors capped at 4096.
+This should be possible to change by manually compiling the DPDK while
+changing the value of respective macros for the desired drivers
+(e.g. IXGBE_MAX_RING_DESC/I40E_MAX_RING_DESC).
 DPDK is natively supported by Mellanox and thus their NICs should work
 "out of the box".
 

From abbd507b5ca82692efa4da854cf4cf04d9b8695f Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 10 Feb 2024 10:41:05 +0100
Subject: [PATCH 440/462] security: update policy wrt CVE ID's

To match that we'll now request CVE ID's ourselves as well,
and we can do it for reported issues as well.

See also:
https://forum.suricata.io/t/security-new-cve-policy/4473
---
 SECURITY.md | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/SECURITY.md b/SECURITY.md
index 70d57aec09e1..9eb94f32ce73 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -45,6 +45,15 @@ releases.
 Note that we'll be refining the levels based on our experiences with applying them
 to actual issues.
 
+## CVE ID's and Github Security Advisories (GHSA)
+
+We will request a CVE ID for an issue if appropriate. Note that multiple
+issues may share the same CVE ID.
+
+We work with the Github CNA, through the Github Security Advisory (GHSA) facility.
+
+The GHSA's will be published at least 2 weeks after the public release addressing
+the issue, together with the redmine security tickets.
 
 ## Support Status of affected code
 
@@ -63,13 +72,14 @@ other data, please clearly state if these can (eventually) enter our public CI/Q
 
 We will assign a severity and will share our assessment with you.
 
-We will create a security ticket, which will be private until a few weeks after
+We will create a security ticket, which will be private until at least 2 weeks after
 a public release addressing the issue.
 
-We will acknowledge you in the release notes and the release announcement. If you
-do not want this, please clearly state this.
+We will acknowledge you in the release notes, release announcement and GHSA. If you
+do not want this, please clearly state this. For the GHSA credits, please give us
+your github handle.
 
-We will not request a CVE, but if you do please let us know the CVE ID.
+Please let us know if you've requested a CVE ID. If you haven't, we can do it.
 
 OISF does not participate in bug bounty programs, or offer any other rewards
 for reporting issues.

From a87943d9bfb47687a40763774b9972c9a00d33dd Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 09:42:55 -0600
Subject: [PATCH 441/462] github-ci: apply read-only permissions to more
 workflows

- authors.yml
- codeql.yml
- scan-build.yml
---
 .github/workflows/authors.yml    | 2 ++
 .github/workflows/codeql.yml     | 2 ++
 .github/workflows/scan-build.yml | 2 ++
 3 files changed, 6 insertions(+)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index 77bb2614de24..e4b0c563c7c3 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -3,6 +3,8 @@ name: New Authors Check
 on:
   pull_request:
 
+permissions: read-all
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index e7fc1e43c06a..d62339a48399 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -13,6 +13,8 @@ on:
   schedule:
     - cron: '18 21 * * 1'
 
+permissions: read-all
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml
index f0df97e7c80e..966139d10a57 100644
--- a/.github/workflows/scan-build.yml
+++ b/.github/workflows/scan-build.yml
@@ -8,6 +8,8 @@ on:
     paths-ignore:
       - "doc/**"
 
+permissions: read-all
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true

From c7cb3e92a60e73c3ef225282bb46eb25e2db9358 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 10:05:02 -0600
Subject: [PATCH 442/462] dependabot: ignore actions/{cache,checkout} v3

The CentOS 7 build requires older GitHub actions, try to make
dependabot ignore these older versions.
---
 .github/dependabot.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index b10ccce16cf1..c063687ed180 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -12,3 +12,8 @@ updates:
       interval: "daily"
     commit-message:
       prefix: "github-actions:"
+    ignore:
+      - dependency-name: "actions/cache"
+        versions: ["3.x"]
+      - dependency-name: "actions/checkout"
+        versions: ["3.x"]

From 5c686af149a02f415221556a6c72f6e5f99c5230 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 10:08:37 -0600
Subject: [PATCH 443/462] dependabot: disable rust checks

As we don't have a Cargo.toml and a Cargo.lock, dependabot for Rust
hasn't been working correctly. Disable, as we now have our own cargo
audit and update workflows.
---
 .github/dependabot.yml | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index c063687ed180..46cc10a4f8ce 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -1,11 +1,5 @@
 version: 2
 updates:
-  - package-ecosystem: "cargo"
-    directory: "/rust"
-    schedule:
-      interval: "daily"
-    commit-message:
-      prefix: "rust:"
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:

From 2242d10fa0ce503ce03a2a99edc21c71925b34bf Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 11:57:02 -0600
Subject: [PATCH 444/462] github-ci: fix authors check with special characters

Dependabot is always getting flagged as a new author even tho it uses
a consistent author of:

dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

But this doesn't work with plain grep. Fix by telling grep to treat
the value as a fixed string instead of a regular expression.
---
 .github/workflows/authors.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index e4b0c563c7c3..242cadd181a4 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -29,7 +29,7 @@ jobs:
           touch new-authors.txt
           while read -r author; do
              echo "Checking author: ${author}"
-             if ! grep -q "^${author}\$" authors.txt; then
+             if ! grep -qFx "${author}" authors.txt; then
                  echo "ERROR: ${author} NOT FOUND"
                  echo "::warning ::New author found: ${author}"
                  echo "${author}" >> new-authors.txt

From 3c06457b74db5a2fd070fe1675c72f53423786de Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Sat, 27 Jan 2024 09:59:55 +0100
Subject: [PATCH 445/462] detect/tls.certs: fix direction handling

Direction flag was checked against wrong field, leading to undefined behavior.

Bug: #6778.
---
 src/detect-tls-certs.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/detect-tls-certs.c b/src/detect-tls-certs.c
index a082c345df4d..f1adb040d033 100644
--- a/src/detect-tls-certs.c
+++ b/src/detect-tls-certs.c
@@ -70,6 +70,7 @@ static int g_tls_certs_buffer_id = 0;
 struct TlsCertsGetDataArgs {
     uint32_t local_id; /**< used as index into thread inspect array */
     SSLCertsChain *cert;
+    const uint8_t flags;
 };
 
 typedef struct PrefilterMpmTlsCerts {
@@ -148,7 +149,7 @@ static InspectionBuffer *TlsCertsGetData(DetectEngineThreadCtx *det_ctx,
     const SSLState *ssl_state = (SSLState *)f->alstate;
     const SSLStateConnp *connp;
 
-    if (f->flags & STREAM_TOSERVER) {
+    if (cbdata->flags & STREAM_TOSERVER) {
         connp = &ssl_state->client_connp;
     } else {
         connp = &ssl_state->server_connp;
@@ -183,7 +184,7 @@ static uint8_t DetectEngineInspectTlsCerts(DetectEngineCtx *de_ctx, DetectEngine
         transforms = engine->v2.transforms;
     }
 
-    struct TlsCertsGetDataArgs cbdata = { 0, NULL };
+    struct TlsCertsGetDataArgs cbdata = { .local_id = 0, .cert = NULL, .flags = flags };
 
     while (1)
     {
@@ -214,7 +215,7 @@ static void PrefilterTxTlsCerts(DetectEngineThreadCtx *det_ctx, const void *pect
     const MpmCtx *mpm_ctx = ctx->mpm_ctx;
     const int list_id = ctx->list_id;
 
-    struct TlsCertsGetDataArgs cbdata = { 0, NULL };
+    struct TlsCertsGetDataArgs cbdata = { .local_id = 0, .cert = NULL, .flags = flags };
 
     while (1)
     {

From fa98c48e65a05de7135285018954cfc17bb862a7 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue, 13 Feb 2024 19:27:24 +0000
Subject: [PATCH 446/462] github-actions: bump github/codeql-action from 2.24.0
 to 3.24.1

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.24.0 to 3.24.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Commits](https://github.com/github/codeql-action/compare/v2.24.0...v3.24.1)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
---
 .github/workflows/codeql.yml              | 4 ++--
 .github/workflows/scorecards-analysis.yml | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index d62339a48399..9e0f3a7099f3 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -41,7 +41,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@v3.24.1
       with:
         languages: ${{ matrix.language }}
         queries: security-extended
@@ -62,4 +62,4 @@ jobs:
        ./configure
        make
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@v3.24.1
diff --git a/.github/workflows/scorecards-analysis.yml b/.github/workflows/scorecards-analysis.yml
index 1fad04559e40..782488514a32 100644
--- a/.github/workflows/scorecards-analysis.yml
+++ b/.github/workflows/scorecards-analysis.yml
@@ -51,6 +51,6 @@ jobs:
 
       # Upload the results to GitHub's code scanning dashboard.
       - name: "Upload SARIF results"
-        uses: github/codeql-action/upload-sarif@dc021d495cb77b369e4d9d04a501700fd83b8c51 # v1
+        uses: github/codeql-action/upload-sarif@bc64d12bb9f349435efba65d373bac054665b85f # v1
         with:
           sarif_file: results.sarif

From f9a20dafc6abe2c03dea98286c86430b447d8196 Mon Sep 17 00:00:00 2001
From: Jeff Lucovsky <jlucovsky@oisf.net>
Date: Wed, 3 Jan 2024 09:28:41 -0500
Subject: [PATCH 447/462] mqtt: Improve frame parsing w/mult. PDUs

This commit improves the mqtt parsing of frames to handle multiple PDUs.

Issue: 6592
---
 rust/src/mqtt/mqtt.rs | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/rust/src/mqtt/mqtt.rs b/rust/src/mqtt/mqtt.rs
index fbf03e19af69..3b09e4423cf8 100644
--- a/rust/src/mqtt/mqtt.rs
+++ b/rust/src/mqtt/mqtt.rs
@@ -433,8 +433,8 @@ impl MQTTState {
                     let _pdu = Frame::new(
                         flow,
                         &stream_slice,
-                        input,
-                        current.len() as i64,
+                        current,
+                        (current.len() - rem.len()) as i64,
                         MQTTFrameType::Pdu as u8,
                     );
                     SCLogDebug!("request msg {:?}", msg);
@@ -518,8 +518,8 @@ impl MQTTState {
                     let _pdu = Frame::new(
                         flow,
                         &stream_slice,
-                        input,
-                        input.len() as i64,
+                        current,
+                        (current.len() - rem.len()) as i64,
                         MQTTFrameType::Pdu as u8,
                     );
 

From 2d7c3d8d5970c0629f2358ab91ed895328efa112 Mon Sep 17 00:00:00 2001
From: Victor Julien <vjulien@oisf.net>
Date: Tue, 13 Feb 2024 09:51:15 +0100
Subject: [PATCH 448/462] multi-tenant: fix coverity warning

Rework locking logic to avoid the following coverity warning.

** CID 1591966:  Concurrent data access violations  (MISSING_LOCK)
/src/detect-engine-loader.c: 475 in DetectLoadersSync()

    474                     SCCtrlMutexLock(loader->tv->ctrl_mutex);
    >>>     CID 1591966:  Concurrent data access violations  (MISSING_LOCK)
    >>>     Accessing "loader->tv" without holding lock "DetectLoaderControl_.m". Elsewhere, "DetectLoaderControl_.tv" is written to with "DetectLoaderControl_.m" held 1 out of 1 times (1 of these accesses strongly imply that it is necessary).
    475                     pthread_cond_broadcast(loader->tv->ctrl_cond);
    476                     SCCtrlMutexUnlock(loader->tv->ctrl_mutex);

The warning itself is harmless.
---
 src/detect-engine-loader.c |  2 +-
 src/detect-engine-loader.h | 12 ++++++++----
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 9073c1e9c29d..153c056a85b4 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -578,8 +578,8 @@ static TmEcode DetectLoaderThreadInit(ThreadVars *t, const void *initdata, void
 
     DetectLoaderControl *loader = &loaders[ftd->instance];
     SCMutexLock(&loader->m);
-    loader->tv = t;
     SCMutexUnlock(&loader->m);
+    loader->tv = t;
 
     return TM_ECODE_OK;
 }
diff --git a/src/detect-engine-loader.h b/src/detect-engine-loader.h
index 8a6f7b8f17be..f43ff9a5491c 100644
--- a/src/detect-engine-loader.h
+++ b/src/detect-engine-loader.h
@@ -43,10 +43,14 @@ typedef struct DetectLoaderTask_ {
 
 typedef struct DetectLoaderControl_ {
     int id;
-    int result;     /* 0 for ok, error otherwise */
-    ThreadVars *tv; /* loader threads threadvars - for waking them up */
-    SCMutex m;
-    TAILQ_HEAD(, DetectLoaderTask_) task_list;
+    ThreadVars *tv; /**< loader threads threadvars - for waking them up */
+
+    /** struct to group members and mutex */
+    struct {
+        SCMutex m;  /**< mutex protects result and task_list */
+        int result; /**< 0 for ok, error otherwise */
+        TAILQ_HEAD(, DetectLoaderTask_) task_list;
+    };
 } DetectLoaderControl;
 
 int DetectLoaderQueueTask(int loader_id, LoaderFunc Func, void *func_ctx, LoaderFreeFunc FreeFunc);

From 41a621178fd2d177866cf9cadcd5d38fc6eb2afd Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Tue, 13 Feb 2024 21:11:32 +0100
Subject: [PATCH 449/462] ci: right sha for authors check

---
 .github/workflows/authors.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/authors.yml b/.github/workflows/authors.yml
index 242cadd181a4..633d4b73ec04 100644
--- a/.github/workflows/authors.yml
+++ b/.github/workflows/authors.yml
@@ -23,7 +23,7 @@ jobs:
       - name: Export known authors from master branch
         run: git log --format="%an <%ae>" origin/master | sort | uniq > authors.txt
       - name: Export authors from new commits
-        run: git log --format="%an <%ae>" origin/${GITHUB_BASE_REF}... | sort | uniq > commit-authors.txt
+        run: git log --format="%an <%ae>" ${{ github.event.pull_request.base.sha }}... | sort | uniq > commit-authors.txt
       - name: Check new authors
         run: |
           touch new-authors.txt

From 63caa0b40a66ecf1a34bbb6d942d4a044b7728a5 Mon Sep 17 00:00:00 2001
From: Philippe Antoine <pantoine@oisf.net>
Date: Thu, 25 Jan 2024 14:26:09 +0100
Subject: [PATCH 450/462] detect: respect directionality for filestore

Ticket: 6617

So that rules with keyword like `filestore:to_server,flow`
only store the files to server and not the ones to client...

Directionality only worked with the default scope, ie the
current file, and not the scope tx or scope flow.
For non-default scope, tx or flow, both directions were stored
whatever the directionality specified.

For these non-default scopes, this commit keeps a default
of both directions, but use only one direction if specified.

Need to split flag FLOWFILE_STORE per direction, so that Suricata
can retain this (optional) directional info from the filestore
keyword.

Fixes: 79499e476979 ("app-layer: move files into transactions")
---
 src/detect-filestore.c | 25 +++++++++++++++++++------
 src/flow.h             |  5 +++--
 src/util-file.c        | 13 ++++++++-----
 3 files changed, 30 insertions(+), 13 deletions(-)

diff --git a/src/detect-filestore.c b/src/detect-filestore.c
index c510544469aa..fa8492161e61 100644
--- a/src/detect-filestore.c
+++ b/src/detect-filestore.c
@@ -118,7 +118,8 @@ static int FilestorePostMatchWithOptions(Packet *p, Flow *f, const DetectFilesto
     switch (filestore->direction) {
         case FILESTORE_DIR_DEFAULT:
             rule_dir = 1;
-            break;
+            // will use both sides if scope is not default
+            // fallthrough
         case FILESTORE_DIR_BOTH:
             toserver_dir = 1;
             toclient_dir = 1;
@@ -160,16 +161,28 @@ static int FilestorePostMatchWithOptions(Packet *p, Flow *f, const DetectFilesto
             AppLayerTxData *txd = AppLayerParserGetTxData(f->proto, f->alproto, txv);
             DEBUG_VALIDATE_BUG_ON(txd == NULL);
             if (txd != NULL) {
-                txd->file_flags |= FLOWFILE_STORE;
+                if (toclient_dir) {
+                    txd->file_flags |= FLOWFILE_STORE_TC;
+                }
+                if (toserver_dir) {
+                    txd->file_flags |= FLOWFILE_STORE_TS;
+                }
             }
         }
     } else if (this_flow) {
         /* set in flow and AppLayerStateData */
-        f->file_flags |= FLOWFILE_STORE;
-
         AppLayerStateData *sd = AppLayerParserGetStateData(f->proto, f->alproto, f->alstate);
-        if (sd != NULL) {
-            sd->file_flags |= FLOWFILE_STORE;
+        if (toclient_dir) {
+            f->file_flags |= FLOWFILE_STORE_TC;
+            if (sd != NULL) {
+                sd->file_flags |= FLOWFILE_STORE_TC;
+            }
+        }
+        if (toserver_dir) {
+            f->file_flags |= FLOWFILE_STORE_TS;
+            if (sd != NULL) {
+                sd->file_flags |= FLOWFILE_STORE_TS;
+            }
         }
     } else {
         FileStoreFileById(fc, file_id);
diff --git a/src/flow.h b/src/flow.h
index c7b5867ea896..418becfc2963 100644
--- a/src/flow.h
+++ b/src/flow.h
@@ -142,8 +142,9 @@ typedef struct AppLayerParserState_ AppLayerParserState;
 #define FLOWFILE_NO_SIZE_TS             BIT_U16(10)
 #define FLOWFILE_NO_SIZE_TC             BIT_U16(11)
 
-/** store all files in the flow */
-#define FLOWFILE_STORE BIT_U16(12)
+/** store files in the flow */
+#define FLOWFILE_STORE_TS BIT_U16(12)
+#define FLOWFILE_STORE_TC BIT_U16(13)
 
 #define FLOWFILE_NONE_TS (FLOWFILE_NO_MAGIC_TS | \
                           FLOWFILE_NO_STORE_TS | \
diff --git a/src/util-file.c b/src/util-file.c
index 3221d116870d..a1c30d89ecde 100644
--- a/src/util-file.c
+++ b/src/util-file.c
@@ -235,8 +235,11 @@ uint16_t FileFlowFlagsToFlags(const uint16_t flow_file_flags, uint8_t direction)
     uint16_t flags = 0;
 
     if (direction == STREAM_TOSERVER) {
-        if ((flow_file_flags & (FLOWFILE_NO_STORE_TS | FLOWFILE_STORE)) == FLOWFILE_NO_STORE_TS) {
+        if ((flow_file_flags & (FLOWFILE_NO_STORE_TS | FLOWFILE_STORE_TS)) ==
+                FLOWFILE_NO_STORE_TS) {
             flags |= FILE_NOSTORE;
+        } else if (flow_file_flags & FLOWFILE_STORE_TS) {
+            flags |= FILE_STORE;
         }
 
         if (flow_file_flags & FLOWFILE_NO_MAGIC_TS) {
@@ -255,8 +258,11 @@ uint16_t FileFlowFlagsToFlags(const uint16_t flow_file_flags, uint8_t direction)
             flags |= FILE_NOSHA256;
         }
     } else {
-        if ((flow_file_flags & (FLOWFILE_NO_STORE_TC | FLOWFILE_STORE)) == FLOWFILE_NO_STORE_TC) {
+        if ((flow_file_flags & (FLOWFILE_NO_STORE_TC | FLOWFILE_STORE_TC)) ==
+                FLOWFILE_NO_STORE_TC) {
             flags |= FILE_NOSTORE;
+        } else if (flow_file_flags & FLOWFILE_STORE_TC) {
+            flags |= FILE_STORE;
         }
 
         if (flow_file_flags & FLOWFILE_NO_MAGIC_TC) {
@@ -275,9 +281,6 @@ uint16_t FileFlowFlagsToFlags(const uint16_t flow_file_flags, uint8_t direction)
             flags |= FILE_NOSHA256;
         }
     }
-    if (flow_file_flags & FLOWFILE_STORE) {
-        flags |= FILE_STORE;
-    }
     DEBUG_VALIDATE_BUG_ON((flags & (FILE_STORE | FILE_NOSTORE)) == (FILE_STORE | FILE_NOSTORE));
 
     SCLogDebug("direction %02x flags %02x", direction, flags);

From 17f9d7aeccd5a69ef2e3344386d1585d231e2933 Mon Sep 17 00:00:00 2001
From: Simon Dugas <Simon.Dugas@cyber.gc.ca>
Date: Fri, 29 Dec 2023 11:58:50 -0500
Subject: [PATCH 451/462] detect-engine-iponly: improve ip list performance

The runtime complexity of insertion sort is approx. O(h*n)^2 where
h is the size of the HOME_NET and n is the number of ip only rules
that use the HOME_NET.

Replacing this with qsort significantly improves rule load time when
a large HOME_NET is used in combination with a moderate amount of ip
only rules.
---
 src/detect-engine-iponly.c | 262 +++++++++++++++++++++++++------------
 1 file changed, 180 insertions(+), 82 deletions(-)

diff --git a/src/detect-engine-iponly.c b/src/detect-engine-iponly.c
index 63261ee716d5..3bd1eb04978e 100644
--- a/src/detect-engine-iponly.c
+++ b/src/detect-engine-iponly.c
@@ -81,16 +81,78 @@ static IPOnlyCIDRItem *IPOnlyCIDRItemNew(void)
     SCReturnPtr(item, "IPOnlyCIDRItem");
 }
 
-static uint8_t IPOnlyCIDRItemCompare(IPOnlyCIDRItem *head,
-                                         IPOnlyCIDRItem *item)
+/**
+ * \brief Compares two list items
+ *
+ * \retval An integer less than, equal to, or greater than zero if lhs is
+ *         considered to be respectively less than, equal to, or greater than
+ *         rhs.
+ */
+static int IPOnlyCIDRItemCompareReal(const IPOnlyCIDRItem *lhs, const IPOnlyCIDRItem *rhs)
 {
-    uint8_t i = 0;
-    for (; i < head->netmask / 32 || i < 1; i++) {
-        if (item->ip[i] < head->ip[i])
-        //if (*(uint8_t *)(item->ip + i) < *(uint8_t *)(head->ip + i))
-            return 1;
+    if (lhs->netmask == rhs->netmask) {
+        uint8_t i = 0;
+        for (; i < lhs->netmask / 32 || i < 1; i++) {
+            if (lhs->ip[i] < rhs->ip[i])
+                return -1;
+            if (lhs->ip[i] > rhs->ip[i])
+                return 1;
+        }
+        return 0;
     }
-    return 0;
+
+    return lhs->netmask < rhs->netmask ? -1 : 1;
+}
+
+static int IPOnlyCIDRItemCompare(const void *lhsv, const void *rhsv)
+{
+    const IPOnlyCIDRItem *lhs = *(const IPOnlyCIDRItem **)lhsv;
+    const IPOnlyCIDRItem *rhs = *(const IPOnlyCIDRItem **)rhsv;
+
+    return IPOnlyCIDRItemCompareReal(lhs, rhs);
+}
+
+static void IPOnlyCIDRListQSort(IPOnlyCIDRItem **head)
+{
+    if (unlikely(head == NULL || *head == NULL))
+        return;
+
+    // First count the number of elements in the list
+    size_t len = 0;
+    IPOnlyCIDRItem *curr = *head;
+
+    while (curr) {
+        curr = curr->next;
+        len++;
+    }
+
+    // Place a pointer to the list item in an array for sorting
+    IPOnlyCIDRItem **tmp = SCMalloc(len * sizeof(IPOnlyCIDRItem *));
+
+    if (unlikely(tmp == NULL)) {
+        SCLogError("Failed to allocate enough memory to sort IP-only CIDR items.");
+        return;
+    }
+
+    curr = *head;
+    for (size_t i = 0; i < len; i++) {
+        tmp[i] = curr;
+        curr = curr->next;
+    }
+
+    // Perform the sort using the qsort algorithm
+    qsort(tmp, len, sizeof(IPOnlyCIDRItem *), IPOnlyCIDRItemCompare);
+
+    // Update the links to the next element
+    *head = tmp[0];
+
+    for (size_t i = 0; i + 1 < len; i++) {
+        tmp[i]->next = tmp[i + 1];
+    }
+
+    tmp[len - 1]->next = NULL;
+
+    SCFree(tmp);
 }
 
 //declaration for using it already
@@ -348,11 +410,9 @@ static int IPOnlyCIDRItemSetup(IPOnlyCIDRItem **gh, char *s)
     return -1;
 }
 
-
 /**
  * \brief This function insert a IPOnlyCIDRItem
- *        to a list of IPOnlyCIDRItems sorted by netmask
- *        ascending
+ *        to a list of IPOnlyCIDRItems
  * \param head Pointer to the head of IPOnlyCIDRItems list
  * \param item Pointer to the item to insert in the list
  *
@@ -361,37 +421,12 @@ static int IPOnlyCIDRItemSetup(IPOnlyCIDRItem **gh, char *s)
 static IPOnlyCIDRItem *IPOnlyCIDRItemInsertReal(IPOnlyCIDRItem *head,
                                          IPOnlyCIDRItem *item)
 {
-    IPOnlyCIDRItem *it, *prev = NULL;
-
     if (item == NULL)
         return head;
 
-    /* Compare with the head */
-    if (item->netmask < head->netmask || (item->netmask == head->netmask && IPOnlyCIDRItemCompare(head, item))) {
-        item->next = head;
-        return item;
-    }
-
-    if (item->netmask == head->netmask && !IPOnlyCIDRItemCompare(head, item)) {
-        item->next = head->next;
-        head->next = item;
-        return head;
-    }
-
-    for (prev = it = head;
-         it != NULL && it->netmask < item->netmask;
-         it = it->next)
-        prev = it;
-
-    if (it == NULL) {
-        prev->next = item;
-        item->next = NULL;
-    } else {
-        item->next = it;
-        prev->next = item;
-    }
-
-    return head;
+    /* Always insert item as head */
+    item->next = head;
+    return item;
 }
 
 /**
@@ -1108,6 +1143,9 @@ void IPOnlyPrepare(DetectEngineCtx *de_ctx)
        IPOnlyCIDRListPrint((de_ctx->io_ctx).ip_dst);
      */
 
+    IPOnlyCIDRListQSort(&(de_ctx->io_ctx).ip_src);
+    IPOnlyCIDRListQSort(&(de_ctx->io_ctx).ip_dst);
+
     IPOnlyCIDRItem *src, *dst;
     SCRadixNode *node = NULL;
 
@@ -1725,64 +1763,124 @@ static int IPOnlyTestSig03 (void)
 static int IPOnlyTestSig04 (void)
 {
     int result = 1;
-
     IPOnlyCIDRItem *head = NULL;
-    IPOnlyCIDRItem *new;
 
-    new = IPOnlyCIDRItemNew();
-    new->netmask= 10;
+    // Test a linked list of size 0, 1, 2, ..., 5
+    for (int size = 0; size < 6; size++) {
+        IPOnlyCIDRItem *new = NULL;
 
-    head = IPOnlyCIDRItemInsert(head, new);
+        if (size > 0) {
+            new = IPOnlyCIDRItemNew();
+            new->netmask = 10;
+            new->ip[0] = 3;
 
-    new = IPOnlyCIDRItemNew();
-    new->netmask= 11;
+            head = IPOnlyCIDRItemInsert(head, new);
+        }
 
-    head = IPOnlyCIDRItemInsert(head, new);
+        if (size > 1) {
+            new = IPOnlyCIDRItemNew();
+            new->netmask = 11;
 
-    new = IPOnlyCIDRItemNew();
-    new->netmask= 9;
+            head = IPOnlyCIDRItemInsert(head, new);
+        }
 
-    head = IPOnlyCIDRItemInsert(head, new);
+        if (size > 2) {
+            new = IPOnlyCIDRItemNew();
+            new->netmask = 9;
 
-    new = IPOnlyCIDRItemNew();
-    new->netmask= 10;
+            head = IPOnlyCIDRItemInsert(head, new);
+        }
 
-    head = IPOnlyCIDRItemInsert(head, new);
+        if (size > 3) {
+            new = IPOnlyCIDRItemNew();
+            new->netmask = 10;
+            new->ip[0] = 1;
 
-    new = IPOnlyCIDRItemNew();
-    new->netmask= 10;
+            head = IPOnlyCIDRItemInsert(head, new);
+        }
 
-    head = IPOnlyCIDRItemInsert(head, new);
+        if (size > 4) {
+            new = IPOnlyCIDRItemNew();
+            new->netmask = 10;
+            new->ip[0] = 2;
 
-    IPOnlyCIDRListPrint(head);
-    new = head;
-    if (new->netmask != 9) {
-        result = 0;
-        goto end;
-    }
-    new = new->next;
-    if (new->netmask != 10) {
-        result = 0;
-        goto end;
-    }
-    new = new->next;
-    if (new->netmask != 10) {
-        result = 0;
-        goto end;
-    }
-    new = new->next;
-    if (new->netmask != 10) {
-        result = 0;
-        goto end;
-    }
-    new = new->next;
-    if (new->netmask != 11) {
-        result = 0;
-        goto end;
+            head = IPOnlyCIDRItemInsert(head, new);
+        }
+
+        IPOnlyCIDRListPrint(head);
+
+        IPOnlyCIDRListQSort(&head);
+
+        if (size == 0) {
+            if (head != NULL) {
+                result = 0;
+                goto end;
+            }
+        }
+
+        /**
+         * Validate the following list entries for each size
+         * 1 - 10
+         * 2 - 10<3> 11
+         * 3 - 9     10<3> 11
+         * 4 - 9     10<1> 10<3> 11
+         * 5 - 9     10<1> 10<2> 10<3> 11
+         */
+        new = head;
+        if (size >= 3) {
+            if (new->netmask != 9) {
+                result = 0;
+                goto end;
+            }
+            new = new->next;
+        }
+
+        if (size >= 4) {
+            if (new->netmask != 10 || new->ip[0] != 1) {
+                result = 0;
+                goto end;
+            }
+            new = new->next;
+        }
+
+        if (size >= 5) {
+            if (new->netmask != 10 || new->ip[0] != 2) {
+                result = 0;
+                goto end;
+            }
+            new = new->next;
+        }
+
+        if (size >= 1) {
+            if (new->netmask != 10 || new->ip[0] != 3) {
+                result = 0;
+                goto end;
+            }
+            new = new->next;
+        }
+
+        if (size >= 2) {
+            if (new->netmask != 11) {
+                result = 0;
+                goto end;
+            }
+            new = new->next;
+        }
+
+        if (new != NULL) {
+            result = 0;
+            goto end;
+        }
+
+        IPOnlyCIDRListFree(head);
+        head = NULL;
     }
 
 end:
-    IPOnlyCIDRListFree(head);
+    if (head) {
+        IPOnlyCIDRListFree(head);
+        head = NULL;
+    }
     return result;
 }
 

From b8b8aa69b49ac0dd222446c28d00a50f9fd7d716 Mon Sep 17 00:00:00 2001
From: Arne Welzel <arne.welzel@corelight.com>
Date: Mon, 5 Feb 2024 17:45:30 +0100
Subject: [PATCH 452/462] stats: Do not expand dots of tm_name

When an interface with dots is used, per worker stats are nested by the
dot-separated-components of the interface due to the usage of
OutputStats2Json().

Prevent this by using OutputStats2Json() on a per-thread specific object
and setting this object into the threads object using the
json_object_set_new() which won't do the dot expansion.

This was tested by creating an interface with dots in the name
and checking the stats.

    ip link add name a.b.c type dummy

With Suricata 7.0.2, sniffing on the a.b.c interface results in the
following worker stats format:

    "threads": {
      "W#01-a": {
        "b": {
          "c": {
            "capture": {
              "kernel_packets": 0,

After this fix, the output looks as follows:

    "threads": {
      "W#01-a.b.c": {
        "capture": {
          "kernel_packets": 0,

Ticket: #6732
---
 src/output-json-stats.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/src/output-json-stats.c b/src/output-json-stats.c
index 718298e48592..0e0f7f7680c4 100644
--- a/src/output-json-stats.c
+++ b/src/output-json-stats.c
@@ -36,6 +36,7 @@
 #include "util-print.h"
 #include "util-time.h"
 #include "util-unittest.h"
+#include "util-validate.h"
 
 #include "util-debug.h"
 #include "output.h"
@@ -265,20 +266,30 @@ json_t *StatsToJSON(const StatsTable *st, uint8_t flags)
         for (x = 0; x < st->ntstats; x++) {
             uint32_t offset = x * st->nstats;
 
+            // Stats for for this thread.
+            json_t *thread = json_object();
+            if (unlikely(thread == NULL)) {
+                json_decref(js_stats);
+                json_decref(threads);
+                return NULL;
+            }
+
             /* for each counter */
             for (u = offset; u < (offset + st->nstats); u++) {
                 if (st->tstats[u].name == NULL)
                     continue;
 
+                // Seems this holds, but assert in debug builds.
+                DEBUG_VALIDATE_BUG_ON(
+                        strcmp(st->tstats[offset].tm_name, st->tstats[u].tm_name) != 0);
+
                 json_t *js_type = NULL;
                 const char *stat_name = st->tstats[u].short_name;
                 if (st->tstats[u].short_name == NULL) {
                     stat_name = st->tstats[u].name;
                     js_type = threads;
                 } else {
-                    char str[256];
-                    snprintf(str, sizeof(str), "%s.%s", st->tstats[u].tm_name, st->tstats[u].name);
-                    js_type = OutputStats2Json(threads, str);
+                    js_type = OutputStats2Json(thread, st->tstats[u].name);
                 }
 
                 if (js_type != NULL) {
@@ -292,6 +303,7 @@ json_t *StatsToJSON(const StatsTable *st, uint8_t flags)
                     }
                 }
             }
+            json_object_set_new(threads, st->tstats[offset].tm_name, thread);
         }
         json_object_set_new(js_stats, "threads", threads);
     }

From 08db0f302b3b2817d0fdfb1fe836fd16cc5e8775 Mon Sep 17 00:00:00 2001
From: Arne Welzel <arne.welzel@corelight.com>
Date: Thu, 8 Feb 2024 18:23:38 +0100
Subject: [PATCH 453/462] stats: Add unittest for basic stats serialization

Main purpose is to validate that the 30 of bond0.30 isn't expanded into
a nested object during serialization.
---
 src/Makefile.am               |  3 +-
 src/output-json-stats.c       |  4 ++
 src/output-json-stats.h       |  2 +
 src/runmode-unittests.c       |  3 ++
 src/tests/output-json-stats.c | 70 +++++++++++++++++++++++++++++++++++
 5 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 src/tests/output-json-stats.c

diff --git a/src/Makefile.am b/src/Makefile.am
index b8c28dcf6372..166555e8c0b8 100755
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1289,7 +1289,8 @@ EXTRA_DIST = \
 	tests/detect-tls-version.c \
 	tests/detect-ipaddr.c \
 	tests/detect.c \
-	tests/stream-tcp.c
+	tests/stream-tcp.c \
+	tests/output-json-stats.c
 
 install-headers:
 	mkdir -p $(DESTDIR)${includedir}/suricata
diff --git a/src/output-json-stats.c b/src/output-json-stats.c
index 0e0f7f7680c4..33f98afa2dda 100644
--- a/src/output-json-stats.c
+++ b/src/output-json-stats.c
@@ -483,3 +483,7 @@ void JsonStatsLogRegister(void) {
         "eve-log.stats", OutputStatsLogInitSub, JsonStatsLogger,
         JsonStatsLogThreadInit, JsonStatsLogThreadDeinit, NULL);
 }
+
+#ifdef UNITTESTS
+#include "tests/output-json-stats.c"
+#endif
diff --git a/src/output-json-stats.h b/src/output-json-stats.h
index 9b96d5001298..b569e30b6429 100644
--- a/src/output-json-stats.h
+++ b/src/output-json-stats.h
@@ -35,4 +35,6 @@ TmEcode OutputEngineStatsReloadTime(json_t **jdata);
 TmEcode OutputEngineStatsRuleset(json_t **jdata);
 void JsonStatsLogRegister(void);
 
+void OutputJsonStatsRegisterTests(void);
+
 #endif /* __OUTPUT_JSON_COUNTERS_H__ */
diff --git a/src/runmode-unittests.c b/src/runmode-unittests.c
index 1150bad89580..8ce0244146a3 100644
--- a/src/runmode-unittests.c
+++ b/src/runmode-unittests.c
@@ -114,6 +114,8 @@
 #include "decode-vntag.h"
 #include "decode-vxlan.h"
 
+#include "output-json-stats.h"
+
 #ifdef OS_WIN32
 #include "win32-syscall.h"
 #endif
@@ -215,6 +217,7 @@ static void RegisterUnittests(void)
 #endif
     SCProtoNameRegisterTests();
     UtilCIDRTests();
+    OutputJsonStatsRegisterTests();
 }
 #endif
 
diff --git a/src/tests/output-json-stats.c b/src/tests/output-json-stats.c
new file mode 100644
index 000000000000..ac1336eff898
--- /dev/null
+++ b/src/tests/output-json-stats.c
@@ -0,0 +1,70 @@
+/* Copyright (C) 2024 Open Information Security Foundation
+ *
+ * You can copy, redistribute or modify this Program under the terms of
+ * the GNU General Public License version 2 as published by the Free
+ * Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301, USA.
+ */
+
+#include "../suricata-common.h"
+
+#include "../output-json-stats.h"
+
+#include "../util-unittest.h"
+
+static int OutputJsonStatsTest01(void)
+{
+    StatsRecord global_records[] = { { 0 }, { 0 } };
+    StatsRecord thread_records[2];
+    thread_records[0].name = "capture.kernel_packets";
+    thread_records[0].short_name = "kernel_packets";
+    thread_records[0].tm_name = "W#01-bond0.30";
+    thread_records[0].value = 42;
+    thread_records[1].name = "capture.kernel_drops";
+    thread_records[1].short_name = "kernel_drops";
+    thread_records[1].tm_name = "W#01-bond0.30";
+    thread_records[1].value = 4711;
+
+    StatsTable table = {
+        .nstats = 2,
+        .stats = &global_records[0],
+        .ntstats = 1,
+        .tstats = &thread_records[0],
+    };
+
+    json_t *r = StatsToJSON(&table, JSON_STATS_TOTALS | JSON_STATS_THREADS);
+    if (!r)
+        return 0;
+
+    // Remove variable content
+    json_object_del(r, "uptime");
+
+    char *serialized = json_dumps(r, 0);
+
+    // Cheesy comparison
+    const char *expected = "{\"threads\": {\"W#01-bond0.30\": {\"capture\": {\"kernel_packets\": "
+                           "42, \"kernel_drops\": 4711}}}}";
+
+    int cmp_result = strcmp(expected, serialized);
+    if (cmp_result != 0)
+        printf("unexpected result\nexpected=%s\ngot=%s\n", expected, serialized);
+
+    free(serialized);
+    json_decref(r);
+
+    return cmp_result == 0;
+}
+
+void OutputJsonStatsRegisterTests(void)
+{
+    UtRegisterTest("OutputJsonStatsTest01", OutputJsonStatsTest01);
+}

From 747730718161a0df46f93063063bef3325357395 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Thu, 15 Feb 2024 10:17:38 +0530
Subject: [PATCH 454/462] multi-tenant: remove futile mutex lock

No shared resource is being changed when the lock is held, it is
immediately unlocked. So, remove it.
---
 src/detect-engine-loader.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/detect-engine-loader.c b/src/detect-engine-loader.c
index 153c056a85b4..8a02ae148e2f 100644
--- a/src/detect-engine-loader.c
+++ b/src/detect-engine-loader.c
@@ -577,8 +577,6 @@ static TmEcode DetectLoaderThreadInit(ThreadVars *t, const void *initdata, void
     *data = ftd;
 
     DetectLoaderControl *loader = &loaders[ftd->instance];
-    SCMutexLock(&loader->m);
-    SCMutexUnlock(&loader->m);
     loader->tv = t;
 
     return TM_ECODE_OK;

From 5a1a32ba5b6f90e38dba382cf371bccd9ca645be Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 5 Feb 2024 16:19:24 +0530
Subject: [PATCH 455/462] eve/stats: add description for common fields

Ticket 6434
---
 etc/schema.json | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index b06dbd4b29fe..5832a61cc36d 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -3669,12 +3669,15 @@
             "optional": true,
             "properties": {
                 "uptime": {
+                    "description": "Suricata engine's uptime",
                     "type": "integer"
                 },
                 "memcap_pressure": {
+                    "description": "Percentage of memcaps used by flow, stream, stream-reassembly and app-layer-http",
                     "type": "integer"
                 },
                 "memcap_pressure_max": {
+                    "description": "Maximum memcap_pressure seen by the engine",
                     "type": "integer"
                 },
                 "app_layer": {

From 1816e98ef04a25f97a77ad94fb304c192e7ad235 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 5 Feb 2024 16:21:34 +0530
Subject: [PATCH 456/462] eve/stats: add description for applayer errors

Ticket 6434
---
 etc/schema.json | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index 5832a61cc36d..c43747b389aa 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -3690,108 +3690,143 @@
                             "type": "object",
                             "properties": {
                                 "bittorrent-dht": {
+                                    "description": "Errors encountered parsing BitTorrent DHT protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dcerpc_tcp": {
+                                    "description": "Errors encountered parsing DCERPC/TCP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dcerpc_udp": {
+                                    "description": "Errors encountered parsing DCERPC/UDP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dhcp": {
+                                    "description": "Errors encountered parsing DHCP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dnp3": {
+                                    "description": "Errors encountered parsing DNP3",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dns_tcp": {
+                                    "description": "Errors encountered parsing DNS/TCP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "dns_udp": {
+                                    "description": "Errors encountered parsing DNS/UDP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "enip_tcp": {
+                                    "description": "Errors encounterd parsing ENIP/TCP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "enip_udp": {
+                                    "description": "Errors encountered parsing ENIP/UDP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "failed_tcp": {
+                                    "description": "Errors encountered parsing TCP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "ftp": {
+                                    "description": "Errors encountered parsing FTP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "ftp-data": {
+                                    "description": "Errors encountered parsing FTP data",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "http": {
+                                    "description": "Errors encountered parsing HTTP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "http2": {
+                                    "description": "Errors encountered parsing HTTP/2",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "ike": {
+                                    "description": "Errors encountered parsing IKE protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "imap": {
+                                    "description": "Errors encountered parsing IMAP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "krb5_tcp": {
+                                    "description": "Errors encountered parsing Kerberos v5/TCP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "krb5_udp": {
+                                    "description": "Errors encountered parsing Kerberos v5/UDP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "modbus": {
+                                    "description": "Errors encountered parsing Modbus protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "mqtt": {
+                                    "description": "Errors encountered parsing MQTT protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "nfs_tcp": {
+                                    "description": "Errors encountered parsing NFS/TCP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "nfs_udp": {
+                                    "description": "Errors encountered parsing NFS/UDP protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "ntp": {
+                                    "description": "Errors encountered parsing NTP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "pgsql": {
+                                    "description": "Errors encountered parsing PostgreSQL protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "quic": {
+                                    "description": "Errors encountered parsing QUIC protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "rdp": {
+                                    "description": "Errors encountered parsing RDP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "rfb": {
+                                    "description": "Errors encountered parsing RFB protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "sip": {
+                                    "description": "Errors encountered parsing SIP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "smb": {
+                                    "description": "Errors encountered parsing SMB protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "smtp": {
+                                    "description": "Errors encountered parsing SMTP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "snmp": {
+                                    "description": "Errors encountered parsing SNMP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "ssh": {
+                                    "description": "Errors encountered parsing SSH protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "telnet": {
+                                    "description": "Errors encountered parsing Telnet protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "tftp": {
+                                    "description": "Errors encountered parsing TFTP",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 },
                                 "tls": {
+                                    "description": "Errors encountered parsing TLS protocol",
                                     "$ref": "#/$defs/stats_applayer_error"
                                 }
                             },
@@ -5517,15 +5552,19 @@
             "type": "object",
             "properties": {
                 "gap": {
+                    "description": "Number of errors processing gaps",
                     "type": "integer"
                 },
                 "alloc": {
+                    "description": "Number of errors allocating memory",
                     "type": "integer"
                 },
                 "parser": {
+                    "description": "Number of errors reported by parser",
                     "type": "integer"
                 },
                 "internal": {
+                    "description": "Number of internal parser errors",
                     "type": "integer"
                 }
             },

From 8817514bea9c3b611fb4ea0479786cdcac06d628 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Mon, 5 Feb 2024 16:22:18 +0530
Subject: [PATCH 457/462] eve/stats: add description for expectations

Ticket 6434
---
 etc/schema.json | 1 +
 1 file changed, 1 insertion(+)

diff --git a/etc/schema.json b/etc/schema.json
index c43747b389aa..c9f818c836b9 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -3684,6 +3684,7 @@
                     "type": "object",
                     "properties": {
                         "expectations": {
+                            "description": "Expectation (dynamic parallel flow) counter",
                             "type": "integer"
                         },
                         "error": {

From 487ba82fb910b9dd7f2863e5cc6229ca9d2194d4 Mon Sep 17 00:00:00 2001
From: Shivani Bhardwaj <shivani@oisf.net>
Date: Fri, 9 Feb 2024 15:42:42 +0530
Subject: [PATCH 458/462] eve/stats: add description for applayer flows

Ticket 6434
---
 etc/schema.json | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/etc/schema.json b/etc/schema.json
index c9f818c836b9..502888feb025 100644
--- a/etc/schema.json
+++ b/etc/schema.json
@@ -3837,114 +3837,151 @@
                             "type": "object",
                             "properties": {
                                 "bittorrent-dht": {
+                                    "description": "Number of flows for BitTorrent DHT protocol",
                                     "type": "integer"
                                 },
                                 "dcerpc_tcp": {
+                                    "description": "Number of flows for DCERPC/TCP protocol",
                                     "type": "integer"
                                 },
                                 "dcerpc_udp": {
+                                    "description": "Number of flows for DCERPC/UDP protocol",
                                     "type": "integer"
                                 },
                                 "dhcp": {
+                                    "description": "Number of flows for DHCP",
                                     "type": "integer"
                                 },
                                 "dnp3": {
+                                    "description": "Number of flows for DNP3",
                                     "type": "integer"
                                 },
                                 "dns_tcp": {
+                                    "description": "Number of flows for DNS/TCP protocol",
                                     "type": "integer"
                                 },
                                 "dns_udp": {
+                                    "description": "Number of flows for DNS/UDP protocol",
                                     "type": "integer"
                                 },
                                 "enip_tcp": {
+                                    "description": "Number of flows for ENIP/TCP",
                                     "type": "integer"
                                 },
                                 "enip_udp": {
+                                    "description": "Number of flows for ENIP/UDP",
                                     "type": "integer"
                                 },
                                 "failed_tcp": {
+                                    "description": "Number of failed flows for TCP",
                                     "type": "integer"
                                 },
                                 "failed_udp": {
+                                    "description": "Number of failed flows for UDP",
                                     "type": "integer"
                                 },
                                 "ftp": {
+                                    "description": "Number of flows for FTP",
                                     "type": "integer"
                                 },
                                 "ftp-data": {
+                                    "description": "Number of flows for FTP data protocol",
                                     "type": "integer"
                                 },
                                 "http": {
+                                    "description": "Number of flows for HTTP",
                                     "type": "integer"
                                 },
                                 "http2": {
+                                    "description": "Number of flows for HTTP/2",
                                     "type": "integer"
                                 },
                                 "ike": {
+                                    "description": "Number of flows for IKE protocol",
                                     "type": "integer"
                                 },
                                 "ikev2": {
+                                    "description": "Number of flows for IKE v2 protocol",
                                     "type": "integer"
                                 },
                                 "imap": {
+                                    "description": "Number of flows for IMAP",
                                     "type": "integer"
                                 },
                                 "krb5_tcp": {
+                                    "description": "Number of flows for Kerberos v5/TCP protocol",
                                     "type": "integer"
                                 },
                                 "krb5_udp": {
+                                    "description": "Number of flows for Kerberos v5/UDP protocol",
                                     "type": "integer"
                                 },
                                 "modbus": {
+                                    "description": "Number of flows for Modbus protocol",
                                     "type": "integer"
                                 },
                                 "mqtt": {
+                                    "description": "Number of flows for MQTT protocol",
                                     "type": "integer"
                                 },
                                 "nfs_tcp": {
+                                    "description": "Number of flows for NFS/TCP protocol",
                                     "type": "integer"
                                 },
                                 "nfs_udp": {
+                                    "description": "Number of flows for NFS/UDP protocol",
                                     "type": "integer"
                                 },
                                 "ntp": {
+                                    "description": "Number of flows for NTP",
                                     "type": "integer"
                                 },
                                 "pgsql": {
+                                    "description": "Number of flows for PostgreSQL protocol",
                                     "type": "integer"
                                 },
                                 "quic": {
+                                    "description": "Number of flows for QUIC protocol",
                                     "type": "integer"
                                 },
                                 "rdp": {
+                                    "description": "Number of flows for RDP",
                                     "type": "integer"
                                 },
                                 "rfb": {
+                                    "description": "Number of flows for RFB protocol",
                                     "type": "integer"
                                 },
                                 "sip": {
+                                    "description": "Number of flows for SIP",
                                     "type": "integer"
                                 },
                                 "smb": {
+                                    "description": "Number of flows for SMB protocol",
                                     "type": "integer"
                                 },
                                 "smtp": {
+                                    "description": "Number of flows for SMTP",
                                     "type": "integer"
                                 },
                                 "snmp": {
+                                    "description": "Number of flows for SNMP",
                                     "type": "integer"
                                 },
                                 "ssh": {
+                                    "description": "Number of flows for SSH protocol",
                                     "type": "integer"
                                 },
                                 "telnet": {
+                                    "description": "Number of flows for Telnet protocol",
                                     "type": "integer"
                                 },
                                 "tftp": {
+                                    "description": "Number of flows for TFTP",
                                     "type": "integer"
                                 },
                                 "tls": {
+                                    "description": "Number of flows for TLS protocol",
                                     "type": "integer"
                                 }
                             },

From 6198ea5a91cf62997c9562417ba9da7fee7f30b0 Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 13:21:26 -0600
Subject: [PATCH 459/462] github-ci: use all cpus for coccinelle checks

Also put "cocci" in the job name and install parallel so the script can
actually run with concurrency.
---
 .github/workflows/builds.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/builds.yml b/.github/workflows/builds.yml
index b3527e94ab47..c40a92788b51 100644
--- a/.github/workflows/builds.yml
+++ b/.github/workflows/builds.yml
@@ -777,7 +777,7 @@ jobs:
 
   # Fedora 38 build using Clang.
   fedora-38-clang:
-    name: Fedora 38 (clang, debug, asan, wshadow, rust-strict, systemd)
+    name: Fedora 38 (clang, cocci, debug, asan, wshadow, rust-strict, systemd)
     runs-on: ubuntu-latest
     container: fedora:38
     needs: [prepare-deps]
@@ -833,6 +833,7 @@ jobs:
                 libtool \
                 lz4-devel \
                 make \
+                parallel \
                 pcre2-devel \
                 pkgconfig \
                 python \
@@ -852,7 +853,7 @@ jobs:
       - run: CC="clang" CFLAGS="$DEFAULT_CFLAGS -Wshadow" ./configure --disable-shared  --enable-coccinelle
       - name: Running unit tests and cocci checks
         # Set the concurrency level for cocci.
-        run: CONCURRENCY_LEVEL=2 make check
+        run: CONCURRENCY_LEVEL=${{ env.CPUS }} make check
       - run: make distclean
       - run: CC="clang" CFLAGS="$DEFAULT_CFLAGS -Wshadow -fsanitize=address -fno-omit-frame-pointer" ./configure --enable-debug --enable-unittests --disable-shared --enable-rust-strict --enable-hiredis --enable-nfqueue --enable-lua
         env:

From f7114b7fe38861f1dc618586158617f9b1c14ddd Mon Sep 17 00:00:00 2001
From: Jason Ish <jason.ish@oisf.net>
Date: Tue, 13 Feb 2024 13:38:57 -0600
Subject: [PATCH 460/462] cocci/run-check: log if parallel command is not found

If CONCURRENCY_LEVEL was set, the script would log a concurrency level
even if the parallel command was not available. Not log if parallel is
not available and set concurrency to 1.
---
 qa/coccinelle/run_check.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/qa/coccinelle/run_check.sh b/qa/coccinelle/run_check.sh
index 325e1732fa5a..d9a451661b93 100755
--- a/qa/coccinelle/run_check.sh
+++ b/qa/coccinelle/run_check.sh
@@ -29,9 +29,12 @@ else
 	BUILT_COCCI_FILES=""
 fi
 
-if [ -z "$CONCURRENCY_LEVEL" ]; then
+if ! command -v parallel > /dev/null; then
+	echo "Concurrency disabled, command 'parallel' not available"
+	CONCURRENCY_LEVEL=1
+elif [ -z "$CONCURRENCY_LEVEL" ]; then
+        echo "No concurrency"
 	CONCURRENCY_LEVEL=1
-	echo "No concurrency"
 else
 	echo "Using concurrency level $CONCURRENCY_LEVEL"
 fi

From 1c11a1995ce0ccb91da6c975ed3e307aaa83a617 Mon Sep 17 00:00:00 2001
From: Dean Balandin <dean.metoh@gmail.com>
Date: Tue, 27 Jun 2023 12:40:37 +0000
Subject: [PATCH 461/462] stream: decouple stream.bypass dependency from tls
 bypass

Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.
---
 src/stream-tcp.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/src/stream-tcp.c b/src/stream-tcp.c
index b77423161800..6536b92ae373 100644
--- a/src/stream-tcp.c
+++ b/src/stream-tcp.c
@@ -5472,17 +5472,13 @@ int StreamTcpPacket (ThreadVars *tv, Packet *p, StreamTcpThread *stt,
         }
 
         if (ssn->flags & STREAMTCP_FLAG_BYPASS) {
-            /* we can call bypass callback, if enabled */
-            if (StreamTcpBypassEnabled()) {
-                PacketBypassCallback(p);
-            }
-
-        /* if stream is dead and we have no detect engine at all, bypass. */
+            PacketBypassCallback(p);
         } else if (g_detect_disabled &&
                 (ssn->client.flags & STREAMTCP_STREAM_FLAG_NOREASSEMBLY) &&
                 (ssn->server.flags & STREAMTCP_STREAM_FLAG_NOREASSEMBLY) &&
                 StreamTcpBypassEnabled())
         {
+            /* if stream is dead and we have no detect engine at all, bypass. */
             SCLogDebug("bypass as stream is dead and we have no rules");
             PacketBypassCallback(p);
         }

From 6bddaefb4d32f39c1d5b227594e55eec7d5ff570 Mon Sep 17 00:00:00 2001
From: Dean Balandin <dean.metoh@gmail.com>
Date: Tue, 27 Jun 2023 12:53:36 +0000
Subject: [PATCH 462/462] userguide: update encrypted traffic bypass

Update documentation to reflect the new features and changes.
---
 doc/userguide/configuration/suricata-yaml.rst | 24 +++++++++----------
 .../performance/ignoring-traffic.rst          |  7 +++---
 2 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst
index db9040aedc60..372b69000b53 100644
--- a/doc/userguide/configuration/suricata-yaml.rst
+++ b/doc/userguide/configuration/suricata-yaml.rst
@@ -1700,13 +1700,13 @@ as raw ``content`` inspection will still be disabled. There is no point in doing
 pattern matching on traffic known to be encrypted. Inspection for (encrypted)
 Heartbleed and other protocol anomalies still happens.
 
-When ``encryption-handling`` is set to ``bypass``, all processing of this session is
-stopped. No further parsing and inspection happens. If ``stream.bypass`` is enabled
-this will lead to the flow being bypassed, either inside Suricata or by the
-capture method if it supports it and is configured for it.
+When ``encryption-handling`` is set to ``bypass``, all processing of this
+session is stopped. No further parsing and inspection happens. This will also
+lead to the flow being bypassed, either inside Suricata or by the capture method
+if it supports it and is configured for it.
 
-Finally, if ``encryption-handling`` is set to ``full``, Suricata will process the
-flow as normal, without inspection limitations or bypass.
+Finally, if ``encryption-handling`` is set to ``full``, Suricata will process
+the flow as normal, without inspection limitations or bypass.
 
 The option has replaced the ``no-reassemble`` option. If ``no-reassemble`` is
 present, and ``encryption-handling`` is not, ``false`` is interpreted as
@@ -2033,12 +2033,12 @@ are typically provided through the command line, are contained in the node
 parameters. There are two ways to specify arguments: lengthy and short.
 Dashes are omitted when describing the arguments. This setup node can be
 used to set up the memory configuration, accessible NICs, and other EAL-related
-parameters, among other things. The node `dpdk.eal-params` also supports 
-multiple arguments of the same type. This can be useful for EAL arguments 
-such as `--vdev`, `--allow`, or `--block`. Values for these EAL arguments 
-are specified as a comma-separated list. 
-An example of such usage can be found in the example above where the `allow` 
-argument only makes `0000:3b:00.0` and `0000:3b:00.1` accessible to Suricata. 
+parameters, among other things. The node `dpdk.eal-params` also supports
+multiple arguments of the same type. This can be useful for EAL arguments
+such as `--vdev`, `--allow`, or `--block`. Values for these EAL arguments
+are specified as a comma-separated list.
+An example of such usage can be found in the example above where the `allow`
+argument only makes `0000:3b:00.0` and `0000:3b:00.1` accessible to Suricata.
 arguments with list node. such as --vdev, --allow, --block eal options.
 The definition of lcore affinity as an EAL
 parameter is a standard practice. However, lcore parameters like `-l`, `-c`,
diff --git a/doc/userguide/performance/ignoring-traffic.rst b/doc/userguide/performance/ignoring-traffic.rst
index a2c7a8825528..712d1ff0b1e1 100644
--- a/doc/userguide/performance/ignoring-traffic.rst
+++ b/doc/userguide/performance/ignoring-traffic.rst
@@ -74,9 +74,10 @@ encrypted traffic
 -----------------
 
 The TLS app layer parser has the ability to stop processing encrypted traffic
-after the initial handshake. By setting the `app-layer.protocols.tls.encryption-handling`
-option to `bypass` the rest of this flow is ignored. If flow bypass is enabled,
-the bypass is done in the kernel or in hardware.
+after the initial handshake. By setting the
+`app-layer.protocols.tls.encryption-handling` option to `bypass` the rest of
+this flow is ignored. The bypass is done in the kernel or in hardware, similar
+to how flow bypass is done.
 
 .. _bypass: