exception policies batch backports - v3

doc/userguide/configuration/exception-policies.rst

@@ -16,6 +16,49 @@ simulate failures or errors and understand Suricata behavior under such conditio
 Exception Policies
 ------------------
 
+.. _master-switch:
+
+Master Switch
+~~~~~~~~~~~~~
+
+It is possible to set all configuration policies via what we call "master
+switch". This offers a quick way to define what the engine should do in case of
+traffic exceptions, while still allowing for the flexibility of indicating a
+different behavior for specific exception policies your setup/environment may
+have the need to.
+
+::
+
+   # The default is ``ignore``.
+   #
+   # All values available for exception policies can be used, and there is one
+   # extra option: auto - same as ``ignore``.
+   # Exception policy values are: drop-packet, drop-flow, reject, bypass,
+   # pass-packet, pass-flow, ignore (disable).
+   exception-policy: auto
+
+.. note::
+
+    The default/``auto`` value changes to ``drop-flow`` in Suricata 7.0, for IPS mode.
+
+This value will be overwritten by specific exception policies whose settings are
+also defined in the yaml file.
+
+Auto
+''''
+
+The default behavior is to ``ignore`` exception policies. This behavior changes
+with Suricata 7.0, where **in IPS mode**, the default for most of the exception
+policies is to fail close, that is, ``drop-flow``, or ``drop-packet`` if the
+flow action is not supported. For the midstream exception policy, the default
+will be ``ignore`` if midstream flows are accepted.
+
+**In IDS mode**, setting auto mode actually means disabling the
+``master-swtich``, or ignoring the exception policies.
+
+Specific settings
+~~~~~~~~~~~~~~~~~
+
 Exception policies are implemented for:
 
 .. list-table:: Exception Policy configuration variables
@@ -27,28 +70,26 @@ Exception policies are implemented for:
      - Expected behavior
    * - stream.memcap
      - memcap-policy
-     - If a stream memcap limit is reached, call the memcap policy on the packet
-       and flow.
+     - If a stream memcap limit is reached, apply the memcap policy to the packet and/or
+       flow.
    * - stream.midstream
      - midstream-policy
-     - If a session is picked up midstream, call the memcap policy on the packet
-       and flow.
+     - If a session is picked up midstream, apply the midstream policy to the flow.
    * - stream.reassembly.memcap
      - memcap-policy
-     - If stream reassembly reaches memcap limit, call the memcap policy on the
-       packet and flow.
+     - If stream reassembly reaches memcap limit, apply memcap policy to the
+       packet and/or flow.
    * - flow.memcap
      - memcap-policy
      - Apply policy when the memcap limit for flows is reached and no flow could
-       be freed up. Apply policy to the packet.
+       be freed up. **Policy can only be applied to the packet.**
    * - defrag.memcap
      - memcap-policy
      - Apply policy when the memcap limit for defrag is reached and no tracker
-       could be picked up. Apply policy to the packet.
+       could be picked up. **Policy can only be applied to the packet.**
    * - app-layer
      - error-policy
-     - Apply policy if a parser reaches an error state. Apply policy to the
-       packet and flow.
+     - Apply policy if a parser reaches an error state. Policy can be applied to packet and/or flow.
 
 To change any of these, go to the specific section in the suricata.yaml file
 (for more configuration details, check the :doc:`suricata.yaml's<suricata-yaml>`
@@ -74,6 +115,91 @@ are:
 The *Drop*, *pass* and *reject* are similar to the rule actions described in :ref:`rule
 actions<suricata-yaml-action-order>`.
 
+Exception Policies and Midstream Pick-up Sessions
+-------------------------------------------------
+
+Suricata behavior can be difficult to track in case of midstream session
+pick-ups. Consider this matrix illustrating the different interactions for
+midstream pick-ups enabled or not and the various exception policy values:
+
+.. list-table:: **Exception Policy Behaviors - IDS Mode**
+   :widths: auto
+   :header-rows: 1
+   :stub-columns: 1
+
+   * - Exception Policy
+     - Midstream pick-up sessions ENABLED (stream.midstream=true)
+     - Midstream pick-up sessions DISABLED (stream.midstream=false)
+   * - Ignore
+     - Session tracket and parsed.
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+   * - Drop-flow
+     - Not valid.*
+     - Not valid.*
+   * - Drop-packet
+     - Not valid.*
+     - Not valid.*
+   * - Reject
+     - Not valid.*
+     - Session not tracked, flow REJECTED.
+   * - Pass-flow
+     - Track session, inspect and log app-layer traffic, no detection.
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+   * - Pass-packet
+     - Not valid.*
+     - Not valid.*
+   * - Bypass
+     - Not valid.*
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+   * - Auto
+     - Midstream policy applied: "ignore". Same behavior.
+     - Midstream policy applied: "ignore". Same behavior.
+
+The main difference between IDS and IPS scenarios is that in IPS mode flows can
+be allowed or blocked (as in with the PASS and DROP rule actions). Packet
+actions are not valid, as midstream pick-up is a configuration that affects the
+whole flow.
+
+.. list-table:: **Exception Policy Behaviors - IPS Mode**
+   :widths: 15 42 43
+   :header-rows: 1
+   :stub-columns: 1
+
+   * - Exception Policy
+     - Midstream pick-up sessions ENABLED (stream.midstream=true)
+     - Midstream pick-up sessions DISABLED (stream.midstream=false)
+   * - Ignore
+     - Session tracket and parsed.
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+   * - Drop-flow
+     - Not valid.*
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+       Flow DROPPED.
+   * - Drop-packet
+     - Not valid.*
+     - Not valid.*
+   * - Reject
+     - Not valid.*
+     - Session not tracked, flow DROPPED and REJECTED.
+   * - Pass-flow
+     - Track session, inspect and log app-layer traffic, no detection.
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+   * - Pass-packet
+     - Not valid.*
+     - Not valid.*
+   * - Bypass
+     - Not valid.*
+     - Session not tracked. No app-layer inspection or logging. No detection. No stream reassembly.
+       Packets ALLOWED.
+   * - Auto
+     - Midstream policy applied: "ignore". Same behavior.
+     - Midstream policy applied: "drop-flow". Same behavior.
+
+Notes:
+
+   * Not valid means that Suricata will error out and won't start.
+   * ``REJECT`` will make Suricata send a Reset-packet unreach error to the sender of the matching packet.
+
 Command-line Options for Simulating Exceptions
 ----------------------------------------------
 

src/app-layer-parser.c

@@ -169,7 +169,7 @@ struct AppLayerParserState_ {
     AppLayerDecoderEvents *decoder_events;
 };
 
-enum ExceptionPolicy g_applayerparser_error_policy = EXCEPTION_POLICY_IGNORE;
+enum ExceptionPolicy g_applayerparser_error_policy = EXCEPTION_POLICY_NOT_SET;
 
 static void AppLayerConfg(void)
 {

src/app-layer.c

@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2021 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free

src/decode.c

@@ -584,16 +584,10 @@ void DecodeRegisterPerfCounters(DecodeThreadVars *dtv, ThreadVars *tv)
 
     dtv->counter_defrag_ipv4_fragments =
         StatsRegisterCounter("defrag.ipv4.fragments", tv);
-    dtv->counter_defrag_ipv4_reassembled =
-        StatsRegisterCounter("defrag.ipv4.reassembled", tv);
-    dtv->counter_defrag_ipv4_timeouts =
-        StatsRegisterCounter("defrag.ipv4.timeouts", tv);
+    dtv->counter_defrag_ipv4_reassembled = StatsRegisterCounter("defrag.ipv4.reassembled", tv);
     dtv->counter_defrag_ipv6_fragments =
         StatsRegisterCounter("defrag.ipv6.fragments", tv);
-    dtv->counter_defrag_ipv6_reassembled =
-        StatsRegisterCounter("defrag.ipv6.reassembled", tv);
-    dtv->counter_defrag_ipv6_timeouts =
-        StatsRegisterCounter("defrag.ipv6.timeouts", tv);
+    dtv->counter_defrag_ipv6_reassembled = StatsRegisterCounter("defrag.ipv6.reassembled", tv);
     dtv->counter_defrag_max_hit =
         StatsRegisterCounter("defrag.max_frag_hits", tv);
 

src/decode.h

@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2022 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -706,10 +706,8 @@ typedef struct DecodeThreadVars_
     /** frag stats - defrag runs in the context of the decoder. */
     uint16_t counter_defrag_ipv4_fragments;
     uint16_t counter_defrag_ipv4_reassembled;
-    uint16_t counter_defrag_ipv4_timeouts;
     uint16_t counter_defrag_ipv6_fragments;
     uint16_t counter_defrag_ipv6_reassembled;
-    uint16_t counter_defrag_ipv6_timeouts;
     uint16_t counter_defrag_max_hit;
 
     uint16_t counter_flow_memcap;

src/defrag-hash.h

@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2012 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free

src/defrag.c

@@ -1041,8 +1041,12 @@ Defrag(ThreadVars *tv, DecodeThreadVars *dtv, Packet *p)
 
     /* return a locked tracker or NULL */
     tracker = DefragGetTracker(tv, dtv, p);
-    if (tracker == NULL)
+    if (tracker == NULL) {
+        if (tv != NULL && dtv != NULL) {
+            StatsIncr(tv, dtv->counter_defrag_max_hit);
+        }
         return NULL;
+    }
 
     Packet *rp = DefragInsertFrag(tv, dtv, tracker, p);
     DefragTrackerRelease(tracker);

src/flow-hash.c

@@ -1,4 +1,4 @@
-/* Copyright (C) 2007-2020 Open Information Security Foundation
+/* Copyright (C) 2007-2023 Open Information Security Foundation
  *
  * You can copy, redistribute or modify this Program under the terms of
  * the GNU General Public License version 2 as published by the Free
@@ -534,7 +534,8 @@ static inline void NoFlowHandleIPS(Packet *p)
  *  \param tv thread vars
  *  \param fls lookup support vars
  *
- *  \retval f *LOCKED* flow on succes, NULL on error.
+ *  \retval f *LOCKED* flow on success, NULL on error or if we should not create
+ *  a new flow.
  */
 static Flow *FlowGetNew(ThreadVars *tv, FlowLookupStruct *fls, Packet *p)
 {

src/stream-tcp.c

@@ -478,15 +478,7 @@ void StreamTcpInitConfig(char quiet)
     stream_config.ssn_memcap_policy = ExceptionPolicyParse("stream.memcap-policy", true);
     stream_config.reassembly_memcap_policy =
             ExceptionPolicyParse("stream.reassembly.memcap-policy", true);
-    SCLogConfig("memcap-policy: %u/%u", stream_config.ssn_memcap_policy,
-            stream_config.reassembly_memcap_policy);
-    stream_config.midstream_policy = ExceptionPolicyParse("stream.midstream-policy", true);
-    if (stream_config.midstream && stream_config.midstream_policy != EXCEPTION_POLICY_IGNORE) {
-        SCLogWarning(SC_WARN_COMPATIBILITY,
-                "stream.midstream_policy setting conflicting with stream.midstream enabled. "
-                "Ignoring stream.midstream_policy.");
-        stream_config.midstream_policy = EXCEPTION_POLICY_IGNORE;
-    }
+    stream_config.midstream_policy = ExceptionPolicyMidstreamParse(stream_config.midstream);
 
     if (!quiet) {
         SCLogConfig("stream.\"inline\": %s",
@@ -961,9 +953,8 @@ static int StreamTcpPacketStateNone(ThreadVars *tv, Packet *p,
             SCLogDebug("Midstream not enabled, so won't pick up a session");
             return 0;
         }
-        if (!(stream_config.midstream_policy == EXCEPTION_POLICY_IGNORE ||
-                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_FLOW ||
-                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_PACKET)) {
+        if (!(stream_config.midstream_policy == EXCEPTION_POLICY_NOT_SET ||
+                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_FLOW)) {
             SCLogDebug("Midstream policy not permissive, so won't pick up a session");
             return 0;
         }
@@ -1132,9 +1123,8 @@ static int StreamTcpPacketStateNone(ThreadVars *tv, Packet *p,
             SCLogDebug("Midstream not enabled, so won't pick up a session");
             return 0;
         }
-        if (!(stream_config.midstream_policy == EXCEPTION_POLICY_IGNORE ||
-                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_FLOW ||
-                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_PACKET)) {
+        if (!(stream_config.midstream_policy == EXCEPTION_POLICY_NOT_SET ||
+                    stream_config.midstream_policy == EXCEPTION_POLICY_PASS_FLOW)) {
             SCLogDebug("Midstream policy not permissive, so won't pick up a session");
             return 0;
         }

src/suricata.c

@@ -2607,6 +2607,8 @@ int PostConfLoadedSetup(SCInstance *suri)
         EngineModeSetIDS();
     }
 
+    SetMasterExceptionPolicy();
+
     AppLayerSetup();
 
     /* Suricata will use this umask if provided. By default it will use the