From d5810a42e19dc7b522d84724aa97481e88b2d8b1 Mon Sep 17 00:00:00 2001 From: Juliana Fajardini Date: Thu, 21 Jul 2022 17:44:59 -0300 Subject: [PATCH] userguide: document how suricata processes rules Added a page that explains how rules are prioritized by Suri, as well as what main different types of inspection happen and what elements are involved when ordering rules. Task #5449 --- doc/userguide/configuration/suricata-yaml.rst | 4 + doc/userguide/rules/flow-keywords.rst | 3 + doc/userguide/rules/http-keywords.rst | 2 +- doc/userguide/rules/index.rst | 1 + doc/userguide/rules/meta.rst | 2 + doc/userguide/rules/rules-internals.rst | 195 ++++++++++++++++++ 6 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 doc/userguide/rules/rules-internals.rst diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst index 3f9bff13b7..b76ba20e66 100644 --- a/doc/userguide/configuration/suricata-yaml.rst +++ b/doc/userguide/configuration/suricata-yaml.rst @@ -164,6 +164,8 @@ back to the default. We recommend that you use the default value for this setting unless you are seeing a high number of discarded alerts (``alert_queue_overflow``) - see the `Discarded and Suppressed Alerts Stats`_ section for more details. +.. _alert queue overflow impact: + Impact on engine behavior ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -649,6 +651,8 @@ The following shows the configuration options for version 2 of the # file naming scheme. #force-hash: [sha1, md5] +.. _detection-engine: + Detection engine ---------------- diff --git a/doc/userguide/rules/flow-keywords.rst b/doc/userguide/rules/flow-keywords.rst index 3ba6bbe06a..6f3995e66d 100644 --- a/doc/userguide/rules/flow-keywords.rst +++ b/doc/userguide/rules/flow-keywords.rst @@ -6,6 +6,8 @@ Flow Keywords .. role:: example-rule-options .. role:: example-rule-emphasis +.. _flowbits: + flowbits -------- @@ -121,6 +123,7 @@ The determination of *established* depends on the protocol: .. image:: flow-keywords/Flow2.png +.. _flowint: flowint ------- diff --git a/doc/userguide/rules/http-keywords.rst b/doc/userguide/rules/http-keywords.rst index 6d51e39b71..12caf75046 100644 --- a/doc/userguide/rules/http-keywords.rst +++ b/doc/userguide/rules/http-keywords.rst @@ -1284,4 +1284,4 @@ Example HTTP Response:: classtype:bad-unknown; sid:102; rev:1;) .. note:: ``http.start`` contains the normalized headers and is terminated by - an extra \\r\\n to indicate the end of the headers. \ No newline at end of file + an extra \\r\\n to indicate the end of the headers. diff --git a/doc/userguide/rules/index.rst b/doc/userguide/rules/index.rst index b34c5519c5..59c8c23e99 100644 --- a/doc/userguide/rules/index.rst +++ b/doc/userguide/rules/index.rst @@ -57,3 +57,4 @@ Suricata Rules pgsql-keywords rule-types email-keywords + rules-internals diff --git a/doc/userguide/rules/meta.rst b/doc/userguide/rules/meta.rst index 8312c91de0..fd892c1c5d 100644 --- a/doc/userguide/rules/meta.rst +++ b/doc/userguide/rules/meta.rst @@ -174,6 +174,8 @@ This would make a reference to http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE All the reference types are defined in the reference.config configuration file. +.. _priority: + priority -------- diff --git a/doc/userguide/rules/rules-internals.rst b/doc/userguide/rules/rules-internals.rst new file mode 100644 index 0000000000..4ed0486849 --- /dev/null +++ b/doc/userguide/rules/rules-internals.rst @@ -0,0 +1,195 @@ +Rule Processing +=============== + +.. toctree:: + +Suricata rules have many elements that influence how they are processed by +Suricata and matched against network traffic. + +This section explains some key aspects of how Suricata handles rules internally, +so it can be easier to understand/predict how different rules may interact in +specific scenarios. + +**This material is intended for:** rule writers; developers. + +Possible questions one should be better equipped to answer after reading this +document: + +- What happens if two rules have the same priority value (the keyword)? +- What type of rules will be evaluated first, given a set of rules? +- How does Suricata decide what rule is "more important" when matching traffic? + +.. important:: Rules processing is also heavily affected by rule types, as mentioned + in this chapter. You may want to read more on :doc:`rule-types`. + +.. note:: + + Throughout this documentation and for Suricata, the terms "rule" and + "signature" are mainly interchangeable, unless context indicates otherwise. + +Overview +-------- + +Rules are provided to Suricata via rules files. Starting from those, the +Detection Engine loader will: + +#. Load all the signatures; +#. Check for validity (non-existing keywords; duplicated ``sid`` etc); +#. Report stats on loaded, good and bad signatures; +#. Sort all valid signatures and store them in a list in the Detect Engine + Context (``DetectEngineCtx``) structure, taking into consideration several + rule aspects according to their order of relevance to the Detection Engine; +#. Attribute internal rule IDs that reflect this `rule prioritization`_; +#. During inspection, match rules against the inspected traffic, according to rule + and traffic type. + +.. _rule prioritization: + +Rule Prioritization +------------------- + +Suricata registers several different ordering functions (with +``SCSigRegisterSignatureOrderingFuncs()``), which are then used to compare +the rules, sort them, and define their priority. The elements taken into +consideration for such are the signature's: + +#. :ref:`Action ` +#. Usage of :ref:`flowbits` +#. Usage of :ref:`flowint` +#. Usage of flowvar +#. Usage of pktvar +#. Usage of hostbits +#. Usage of ippair +#. :ref:`"Priority" keyword` + +In this order. Once signatures are ordered, they are attributed a unique +internal ID (``Signature::iid``) which symbolizes their priority (the lower the +``iid``, the higher the priority). This could mean that a rule with a +keyword-defined priority of 1 could have lower priority than another rule that +had flowbits set and a rule action with higher priority, for instance. + +.. note:: this list isn't fully comprehensive, in the sense that each item has + extra logic for prioritization. For example, considering flowbits, the + priority is write (highest) > write + read > read (lowest) > no flowbits. + +Another important element when considering rule parsing, processing and matching +is that the ruleset is optimized into signature group heads based on the signature +elements (thus, for instance, a TCP rule and an UDP rule would be loaded into +different groups, and their internal ids will not interfere between one another, +as they're matched against different traffic). For more on this, see +:ref:`detection-engine`. + +Inspection Process +------------------ + +Once it is time to inspect network traffic against the loaded rules, the +Detect Engine will match against - if applicable: + +#. IP Only rules; +#. Packet/payload-related rules; +#. Frame keywords; +#. Application layer protocol transaction rules. + +During packet inspection, if the signature uses the last two in this list, +inspection is left to those steps. + +.. tip:: + + With the introduction of :doc:`Firewall mode <../firewall/firewall-design>`, + it is possible to explicitly control to which step of the detection engine flow + a rule will be hooked. This is done with :ref:`Explicit rule hooks `. + +For each rule that is matched, a ``PacketAlert`` is created. After all matches +for a packet have been processed, and the :ref:`alert queue limit ` is taken into account, the remaining ``PacketAlerts`` become +the alerts in Suricata logs. + +Considerations on Inspection Steps +---------------------------------- + +IP Only rules +~~~~~~~~~~~~~ + +Without optimization, IP Only signatures would match on every packet on a flow. +To improve performance, what Suricata does is to evaluate rules that are +``ip-only`` only once per flow direction, for the first packet in each direction. + +Application layer protocol transactions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each parser has its own state machine, and uses a per-direction parsing state +"progress". Keywords can be registered for each progress value. So, for +instance, ``http`` has a value "request line available" for which there are +keywords like :ref:`http.uri `, :ref:`http.method +` etc. registered. While parsing the traffic, if the engine reaches +this state, the signatures with those keywords may be already evaluated, even if +they have a lower priority than an http body inspecting signature. + +Relatedly, a rule with two keywords matching at two different progress stages may +be evaluated against two different packets. + +Implications +------------ + +Action precedence and interaction with ``ip-only`` rules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To illustrate what may be counter-intuitive implications of how inspection +steps, action prioritization and rule keywords interact and affect the engine +behavior, we will use a real case example. Consider these three rules: + +.. container:: example-rule + + pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 443 (msg:"Allow TCP in port 443"; flow: + not_established; sid:1; rev:1;) + +.. container:: example-rule + + pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 80 (msg:"Allow TCP in port 80"; flow: + not_established; sid:2; rev:1;) + +.. container:: example-rule + + drop ip 0.0.0.0/0 any -> 0.0.0.0/0 any (msg:"No outbound internet access + from host"; sid:3; rev:1;) + +The first two are signatures that analyze individual packets and match only if +the flow has not been established (``flow:not_established``): the rules grant +``PASS`` to the matched packet - but not to its flow. + +The third signature is considered ``ip-only``. This means it will be evaluated +for the *first* packet in both directions of a flow, in addition to rules 1 and +2. By extension, **the other packets in the same flow will not be evaluated +against this rule.** + +With an action order configuration that prioritizes ``PASS`` over ``DROP``, this +means that rules 1 and 2 will have a higher internal priority over rule 3, +therefore nullifying the ``DROP`` outcome. The result: a flow for outbound +internet traffic from the host, expected to be dropped, wouldn't be. + +If the expected behavior with those three signatures was to allow traffic on +ports 80 and 443 only, while dropping everything else, the simplest way to +achieve this would be to remove the ``flow:not_established`` portion from rules +sid:1 and sid:2. This ensures that the ``PASS`` action would be applied to the +whole flow following the match on the first packet and that all other traffic +would be dropped. + +Following that, all three rules will be evaluated on the same step, and if a +flow isn't flagged with ``pass``, it will be dropped with the third rule. + +.. Tip:: + A more straightforward way to achieve that in Suricata 8 is using the firewall + more. See :doc:`../firewall/firewall-design`. + +Alerts not seen +~~~~~~~~~~~~~~~ + +Another aspect of rule prioritization combined with the alerts queue size is +that, in corner case scenarios, if a packet matches against too many rules, +signatures with lower priority could be discarded from the ``PacketAlert`` queue +(see the section on :ref:`alert queue overflow impact ` +for more). + +The stats counter ``detect.alert_queue_overflow`` will be higher than zero if +an alert was discarded due to ``Alert Queue`` overflow (cf. :ref:`alerts stats`). +