Commit Graph

6814 Commits (371113e21e808b74cfbb0878ed14e08a3b926d5c)
 

Author SHA1 Message Date
Victor Julien ba9d43cce5 mpm: improve negated mpm
The idea is: if mpm is negated, it's both on mpm and nonmpm sid lists
and we can kick it out in that case during the merge sort.

It only works for patterns that are 'independent'. This means that the
rule doesn't need to only match if the negated mpm pattern is limited
to the first 10 bytes for example.

Or more generally, an negated mpm pattern that has depth, offset,
distance or within settings can't be handled this way. These patterns
are not added to the mpm at all, but just to to non-mpm list. This
makes sense as they will *always* need manual inspection.

Similarly, a pattern that is 'chopped' always needs validation. This
is because in this case we only inspect a part of the final pattern.
9 years ago
Victor Julien 9e71ef4c3b detect: remove signature pattern id reference 9 years ago
Victor Julien 46734ec41b mpm: remove unused pmq merge function 9 years ago
Victor Julien c1ad08d11e detect: remove stream pmq array 9 years ago
Victor Julien 4e8e591715 detect mpm: mpm store cleanup
Move all rule modification to the fast_pattern assigment.
9 years ago
Victor Julien c87fcb29ff detect mpm: fast_pattern assignment cleanup 9 years ago
Victor Julien 7c94077892 detect mpm: remove unused mpm flags 9 years ago
Victor Julien a7d126738a detect address: remove unused features 9 years ago
Victor Julien cbf80de6fe detect-port: cleanup 9 years ago
Victor Julien a96fa0fc2f detect: remove unused dport sgh hash 9 years ago
Victor Julien e6248b0dbe detect: clean up sgh's at detect engine free 9 years ago
Victor Julien 1f7e33a4b0 detect: remove unused flag 9 years ago
Victor Julien e8c9598035 detect sgh: remove unused field 9 years ago
Victor Julien 2ab20d0b9b detect: free lookup structures 9 years ago
Victor Julien 4ba1ac55f0 detect: output sgh stats
Output stats for the rule groups into a json format.
9 years ago
Victor Julien 810d2d3ec6 detect: add list id to string funcs 9 years ago
Victor Julien fa885e1d85 mpm: remove pattern id logic 9 years ago
Victor Julien 69d38a3222 mpm: ac-bs use internal pattern id tracking 9 years ago
Victor Julien 4edb03ab9d mpm: ac use internal pattern id tracking 9 years ago
Victor Julien cd8283bb72 smtp: use rule_id mpm support instead of pattern id 9 years ago
Victor Julien b2fcb17859 proto detect: in mpm switch to rule id
Use the rule id API instead of pattern id API.
9 years ago
Victor Julien a34be23002 detect: simplify negated mpm handling 9 years ago
Victor Julien b84d6d402f detect grouping: multiple whitelist conditions
Instead of the binary yes/no whitelisting used so far, use different
values for different sorts of whitelist reasons. The port list will
be sorted by whitelist value first, then by rule count.

The goal is to whitelist groups that have weak sigs:

 - 1 byte pattern groups

 - SYN sigs

    Rules that check for SYN packets are mostly scan detection rules.
    They will be checked often as SYN packets are very common.

    e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;)

    This patch adds whitelisting for SYN-sigs, so that the sigs end up
    in as unique groups as possible.

 - negated mpm sigs

    Currently negated mpm sigs are inspected often, so they are quite
    expensive. For this reason, try to whitelist them.

These values are set during 'stage 1', rule preprocessing.
9 years ago
Victor Julien 3c184c19cd detect grouping: port based group whitelisting
Whitelist some ports in grouping to make sure they get their own group.
9 years ago
Victor Julien 5772f526dc detect grouping: warn on and fix up bad sigs
Only inspect directionless SYN scan sigs toserver. Issue a warning for
those rules.
9 years ago
Victor Julien 2ce03fbabb detect: split non-mpm list into syn/nosyn
Since SYN inspecting rules are expensive, this patch splits the
'non-mpm' list (i.e. the rules that are always considered) into
a 'syn' and 'non-syn' list. The SYN list is only inspected if the
packet has the SYN flag set, otherwise the non-syn list is used.

The syn-list contains _all_ rules. The non-syn list contains all
minus the rules requiring the SYN bit in a packet.
9 years ago
Victor Julien e48d745ed7 mpm: constify search func args 9 years ago
Victor Julien 26517b8b61 detect: mpm store frees mpm_ctx' it owns 9 years ago
Victor Julien 102a82fc7b detect: use mpm store for app layer mpms
Rework app-layer mpm setup and registration to make this possible.
9 years ago
Victor Julien fac2cc0560 detect: mpm deduplication
Create hash for mpm's that we can reuse. Have packet/stream mpms
use this.
9 years ago
Victor Julien f0ba00e51d detect: remove old unused code 9 years ago
Victor Julien 10b049304f detect: set new defaults for grouping 9 years ago
Victor Julien 2ee9bf2aef detect: rename groupings vars 9 years ago
Victor Julien 1f70ccfc23 detect: remove unused grouping settings 9 years ago
Victor Julien ae80ed5964 detect: make port grouping use config limits 9 years ago
Victor Julien df529b13ce detect: change port grouping
Update port grouping logic. Previously it would create one consistent
list w/o overlap. It largely still does this, except for the 'catch
all' port group at the end of the list. This port group contains all
the sigs that didn't fit into the other groups.
9 years ago
Victor Julien a392812302 detect: sort/group port sigs 9 years ago
Victor Julien e570b10abe detect: display unique sgh count 9 years ago
Victor Julien eda9552e95 detect: group proto sghs 9 years ago
Victor Julien d82df4eb8b detect-mpm: make sgh setup proto aware
Allow multi-proto, multi-direction sgh's.
9 years ago
Victor Julien 4223ce9aba detect: remove obsolete grouping code 9 years ago
Victor Julien 9ae4cb9e02 detect: debug output 9 years ago
Victor Julien fd5a06017d detect: per port and proto rule grouping
Replace tree based approach for rule grouping with a per port (tcp/udp)
and per protocol approach.

Grouping now looks like:

               +----+
               |icmp+--->
               +----+
               |gre +--->
               +----+
               |esp +--->
               +----+
          other|... |
        +----->-----+
        |      |N   +--->
        |      +----+
        |
        | tcp  +----+   +----+
        +----->+ 80 +-->+ 139+-->
        |      +----+   +----+
        |
        | udp  +----+   +----+
    +---+----->+ 53 +-->+ 135+-->
    |          +----+   +----+
    |toserver
+--->
    |toclient
    |
    +--->

So the first 'split' in the rules is the direction: toserver or toclient.
Rules that don't have a direction, are in both branches.

Then the split is between tcp/udp and the other protocols. For tcp and
udp port lists are used. For the other protocols, grouping is simply per
protocol.

The ports used are the destination ports for toserver sigs and source
ports for toclient sigs.
9 years ago
Victor Julien 27e63a1e11 detect: track direction and ipproto of sgh
Each SGH has a unique ipproto and direction.
9 years ago
Victor Julien e75a93b125 detect: pass ipproto to rule grouping funcs 9 years ago
Victor Julien c71c991669 detect: delay sgh cleanup 9 years ago
Victor Julien bb662a65f8 detect: delay sgh mpm setup 9 years ago
Victor Julien 18dd54dfa7 Start rule inspect with mask check 9 years ago
Victor Julien bfe49b60f7 rule analyzer: add no/both direction warning 9 years ago
Victor Julien b7d81fc3b0 detect: SYN flags
Add funcs to see if a rule needs a SYN flag in the packet.
9 years ago