Rule Processing =============== .. toctree:: Suricata rules have many elements that influence how they are processed by Suricata and matched against network traffic. This section explains some key aspects of how Suricata handles rules internally, so it can be easier to understand/predict how different rules may interact in specific scenarios. **This material is intended for:** rule writers; developers. Possible questions one should be better equipped to answer after reading this document: - What happens if two rules have the same priority value (the keyword)? - What type of rules will be evaluated first, given a set of rules? - How does Suricata decide what rule is "more important" when matching traffic? .. important:: Rules processing is also heavily affected by rule types, as mentioned in this chapter. You may want to read more on :doc:`rule-types`. .. note:: Throughout this documentation and for Suricata, the terms "rule" and "signature" are mainly interchangeable, unless context indicates otherwise. Overview -------- Rules are provided to Suricata via rules files. Starting from those, the Detection Engine loader will: #. Load all the signatures; #. Check for validity (non-existing keywords; duplicated ``sid`` etc); #. Report stats on loaded, good and bad signatures; #. Sort all valid signatures and store them in a list in the Detect Engine Context (``DetectEngineCtx``) structure, taking into consideration several rule aspects according to their order of relevance to the Detection Engine; #. Attribute internal rule IDs that reflect this `rule prioritization`_; #. During inspection, match rules against the inspected traffic, according to rule and traffic type. .. _rule prioritization: Rule Prioritization ------------------- Suricata registers several different ordering functions (with ``SCSigRegisterSignatureOrderingFuncs()``), which are then used to compare the rules, sort them, and define their priority. The elements taken into consideration for such are the signature's: #. :ref:`Action ` #. Usage of :ref:`flowbits` #. Usage of :ref:`flowint` #. Usage of flowvar #. Usage of pktvar #. Usage of hostbits #. Usage of ippair #. :ref:`"Priority" keyword` In this order. Once signatures are ordered, they are attributed a unique internal ID (``Signature::iid``) which symbolizes their priority (the lower the ``iid``, the higher the priority). This could mean that a rule with a keyword-defined priority of 1 could have lower priority than another rule that had flowbits set and a rule action with higher priority, for instance. .. note:: this list isn't fully comprehensive, in the sense that each item has extra logic for prioritization. For example, considering flowbits, the priority is write (highest) > write + read > read (lowest) > no flowbits. Another important element when considering rule parsing, processing and matching is that the ruleset is optimized into signature group heads based on the signature elements (thus, for instance, a TCP rule and an UDP rule would be loaded into different groups, and their internal ids will not interfere between one another, as they're matched against different traffic). For more on this, see :ref:`detection-engine`. Inspection Process ------------------ Once it is time to inspect network traffic against the loaded rules, the Detect Engine will match against - if applicable: #. IP Only rules; #. Packet/payload-related rules; #. Frame keywords; #. Application layer protocol transaction rules. During packet inspection, if the signature uses the last two in this list, inspection is left to those steps. .. tip:: With the introduction of :doc:`Firewall mode <../firewall/firewall-design>`, it is possible to explicitly control to which step of the detection engine flow a rule will be hooked. This is done with :ref:`Explicit rule hooks `. For each rule that is matched, a ``PacketAlert`` is created. After all matches for a packet have been processed, and the :ref:`alert queue limit ` is taken into account, the remaining ``PacketAlerts`` become the alerts in Suricata logs. Considerations on Inspection Steps ---------------------------------- IP Only rules ~~~~~~~~~~~~~ Without optimization, IP Only signatures would match on every packet on a flow. To improve performance, what Suricata does is to evaluate rules that are ``ip-only`` only once per flow direction, for the first packet in each direction. Application layer protocol transactions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each parser has its own state machine, and uses a per-direction parsing state "progress". Keywords can be registered for each progress value. So, for instance, ``http`` has a value "request line available" for which there are keywords like :ref:`http.uri `, :ref:`http.method ` etc. registered. While parsing the traffic, if the engine reaches this state, the signatures with those keywords may be already evaluated, even if they have a lower priority than an http body inspecting signature. Relatedly, a rule with two keywords matching at two different progress stages may be evaluated against two different packets. Implications ------------ Action precedence and interaction with ``ip-only`` rules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To illustrate what may be counter-intuitive implications of how inspection steps, action prioritization and rule keywords interact and affect the engine behavior, we will use a real case example. Consider these three rules: .. container:: example-rule pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 443 (msg:"Allow TCP in port 443"; flow: not_established; sid:1; rev:1;) .. container:: example-rule pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 80 (msg:"Allow TCP in port 80"; flow: not_established; sid:2; rev:1;) .. container:: example-rule drop ip 0.0.0.0/0 any -> 0.0.0.0/0 any (msg:"No outbound internet access from host"; sid:3; rev:1;) The first two are signatures that analyze individual packets and match only if the flow has not been established (``flow:not_established``): the rules grant ``PASS`` to the matched packet - but not to its flow. The third signature is considered ``ip-only``. This means it will be evaluated for the *first* packet in both directions of a flow, in addition to rules 1 and 2. By extension, **the other packets in the same flow will not be evaluated against this rule.** With an action order configuration that prioritizes ``PASS`` over ``DROP``, this means that rules 1 and 2 will have a higher internal priority over rule 3, therefore nullifying the ``DROP`` outcome. The result: a flow for outbound internet traffic from the host, expected to be dropped, wouldn't be. If the expected behavior with those three signatures was to allow traffic on ports 80 and 443 only, while dropping everything else, the simplest way to achieve this would be to remove the ``flow:not_established`` portion from rules sid:1 and sid:2. This ensures that the ``PASS`` action would be applied to the whole flow following the match on the first packet and that all other traffic would be dropped. Following that, all three rules will be evaluated on the same step, and if a flow isn't flagged with ``pass``, it will be dropped with the third rule. .. Tip:: A more straightforward way to achieve that in Suricata 8 is using the firewall more. See :doc:`../firewall/firewall-design`. Alerts not seen ~~~~~~~~~~~~~~~ Another aspect of rule prioritization combined with the alerts queue size is that, in corner case scenarios, if a packet matches against too many rules, signatures with lower priority could be discarded from the ``PacketAlert`` queue (see the section on :ref:`alert queue overflow impact ` for more). The stats counter ``detect.alert_queue_overflow`` will be higher than zero if an alert was discarded due to ``Alert Queue`` overflow (cf. :ref:`alerts stats`).