Instead of a Host and IPPair table thresholding layer, use a dedicated
THash to store both. This allows hashing on host+sid+tracker or
ippair+sid+tracker, to create more unique hash keys.
This allows for fewer hash collisions.
The per rule tracking also uses this, so that the single big lock is no
longer a single point of contention.
Reimplement storage for flow thresholds to reuse as much logic as
possible from the host/ippair/rule thresholds.
Ticket: #426.
Thresholding often has 2 stages:
1. recording matches
2. appling an action, like suppress
E.g. with something like:
threshold:type limit, count 10, seconds 3600, track by_src;
the recording state is about counting 10 first hits for an IP,
then followed by the "suppress" state that might last an hour.
By_src/by_dst are expensive, as they do a host table lookup and lock
the host. If many threads require this access, lock contention becomes
a serious problem.
This patch adds a thread local cache to avoid the synchronization
overhead. When the threshold for a host enters the "apply" stage,
a thread local hash entry is added. This entry knows the expiry
time and the action to apply. This way the action can be applied
w/o the synchronization overhead.
A rbtree is used to handle expiration.
Implemented for IPv4.
Limits propegation checked for DETECT_DEPTH as a content flag,
which appears to have worked by chance. After reshuffling the
keyword id's it no longer worked. This patch uses the proper
flag DETECT_CONTENT_DEPTH.
Traffic variables (flowvars, flowbits, xbits, etc) use a smaller int for
their type than detection types. As a workaround make sure the values fit
in a uint8_t.
Add support for 'by_flow' track option. This allows using the various
threshold options in the context of a single flow.
Example:
alert tcp ... stream-event:pkt_broken_ack; \
threshold:type limit, track by_flow, count 1, seconds 3600;
The example would limit the number of alerts to once per hour for
packets triggering the 'pkt_broken_ack' stream event.
Implemented as a special "flowvar" holding the threshold entries. This
means no synchronization is required, making this a cheaper option
compared to the other trackers.
Ticket: #6822.
Issue: 6487
To avoid ambiguity, a single definition for base 64 decoding modes will
be used. The Rust base64 transform contains the definitions for the
existing mode types: Strict, RFC2045, RFC4648
Issue: 6487
Implement the from_base64 transform:
[bytes value] [offset value] [mode strict|rfc4648|rfc2045]
The value for bytes and offset may be a byte_ variable or an
unsigned integer.
Ticket: 6390
This can happen with keyword filestore:both,flow
If one direction does not have a signature group with a filestore,
the file is set to nostore on opening, until a signature in
the other direction tries to set it to store.
Subsequent files will be stored in both directions as flow flags
are now set.