Commit Graph

16230 Commits (1345c6d1cb5a9ac825f4bafc0e2a26d5e4a98e1e)
 

Author SHA1 Message Date
Victor Julien 1e9fdc4005 detect/threshold: consider tenant id in tracking
Ticket: #6967.
8 months ago
Victor Julien 2be998fbcd detect/threshold: include rev in threshold tracking 8 months ago
Victor Julien 3471c0f6ad detect/threshold: improve hash function 8 months ago
Victor Julien b8028bf386 thresholds: use dedicated storage
Instead of a Host and IPPair table thresholding layer, use a dedicated
THash to store both. This allows hashing on host+sid+tracker or
ippair+sid+tracker, to create more unique hash keys.

This allows for fewer hash collisions.

The per rule tracking also uses this, so that the single big lock is no
longer a single point of contention.

Reimplement storage for flow thresholds to reuse as much logic as
possible from the host/ippair/rule thresholds.

Ticket: #426.
8 months ago
Victor Julien ac400af8f4 range: use thash expiry API for timeout 8 months ago
Victor Julien 00e1e89449 thash: add expiration logic
Add a callback and helper function to handle data expiration.

Update datasets to explicitly not use expiration.
8 months ago
Victor Julien 114fc37294 detect/address: constify ipv6 cmp funcs 8 months ago
Victor Julien 3a7247b1ed detect/threshold: minor rate filter cleanup 8 months ago
Victor Julien ab5e04525f detect/threshold: minor code cleanup
Packet pointer is not used during allocation.
8 months ago
Victor Julien 6622dc7444 detect/threshold: minor cleanup 8 months ago
Victor Julien c08c81cacf detect/threshold: implement per thread cache
Thresholding often has 2 stages:

1. recording matches
2. appling an action, like suppress

E.g. with something like:
threshold:type limit, count 10, seconds 3600, track by_src;
the recording state is about counting 10 first hits for an IP,
then followed by the "suppress" state that might last an hour.

By_src/by_dst are expensive, as they do a host table lookup and lock
the host. If many threads require this access, lock contention becomes
a serious problem.

This patch adds a thread local cache to avoid the synchronization
overhead. When the threshold for a host enters the "apply" stage,
a thread local hash entry is added. This entry knows the expiry
time and the action to apply. This way the action can be applied
w/o the synchronization overhead.

A rbtree is used to handle expiration.

Implemented for IPv4.
8 months ago
Victor Julien c963158443 detect: add ticket id to var related todos 8 months ago
Victor Julien 405491c3fc detect/detection_filter: add support for track by_flow 8 months ago
Victor Julien 3f04af7c7f doc: add thresholding by_flow 8 months ago
Victor Julien f028648750 detect/content: fix wrong value for depth check
Limits propegation checked for DETECT_DEPTH as a content flag,
which appears to have worked by chance. After reshuffling the
keyword id's it no longer worked. This patch uses the proper
flag DETECT_CONTENT_DEPTH.
8 months ago
Victor Julien d0f3f2d462 detect: group content inspect keyword id's 8 months ago
Victor Julien 022173d7ab detect: group types used in traffic variables
Traffic variables (flowvars, flowbits, xbits, etc) use a smaller int for
their type than detection types. As a workaround make sure the values fit
in a uint8_t.
8 months ago
Victor Julien cfd55ead74 threshold: add by_flow support for global thresholds
Allow rate_filter and thresholds from the global config to specify
tracking "by_flow".
8 months ago
Victor Julien 1552f0953a detect/threshold: implement tracking 'by_flow'
Add support for 'by_flow' track option. This allows using the various
threshold options in the context of a single flow.

Example:

    alert tcp ... stream-event:pkt_broken_ack; \
        threshold:type limit, track by_flow, count 1, seconds 3600;

The example would limit the number of alerts to once per hour for
packets triggering the 'pkt_broken_ack' stream event.

Implemented as a special "flowvar" holding the threshold entries. This
means no synchronization is required, making this a cheaper option
compared to the other trackers.

Ticket: #6822.
8 months ago
Victor Julien a81b23254c util/var: add comments explaining types 8 months ago
Victor Julien 1fa13e4b81 util/var: remove printf; add assert 8 months ago
Philippe Antoine 5bd17934df http2: do not expand duplicate headers
Ticket: 7104

As this can cause a big mamory allocation due to the quadratic
nature of the HPACK compression.
8 months ago
Philippe Antoine 37509e8e0e modbus: abort flow parsing on flood
Ticket: 6987

Let's not spend more resources for a flow which is trying to
make us do it...
8 months ago
Victor Julien ce727cf4b1 detect: remove unnecessary detect thread flags stores 8 months ago
Philippe Antoine b34d4b1314 detect/nfs: do not free a null pointer
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69840
8 months ago
Jeff Lucovsky 5b97f4040c detect/base64: Use Rust defined modes everywhere
Issue: 6487

To avoid ambiguity, a single definition for base 64 decoding modes will
be used. The Rust base64 transform contains the definitions for the
existing mode types: Strict, RFC2045, RFC4648
8 months ago
Jeff Lucovsky 01e20c91fb doc/transform: Correct typo 8 months ago
Jeff Lucovsky d205ff82d0 doc/transform: Describe the from_base64 transform
Issue: 6487

Document the new transform and indicate that it's the preferred way to
perform base64 decoding (preferred over base64_decode)
8 months ago
Jeff Lucovsky f042e9034b detect/transform: Add from_base64 transform
Issue: 6487

Implement the from_base64 transform:
    [bytes value] [offset value] [mode strict|rfc4648|rfc2045]

    The value for bytes and offset may be a byte_ variable or an
    unsigned integer.
8 months ago
Jeff Lucovsky 1823681709 detect/transform: from_base64 option parsing
Issue: 6487

Implement from_base64 option parsing in Rust. The Rust module also
contains unit tests.
8 months ago
Jeff Lucovsky ab0cb960a1 detect/parser: Refactor utility routines
Refactor utility functions/definitions from the byte_math module into
the parser module. This includes parse_var and ResultValue

Issue: 6487
8 months ago
Shivani Bhardwaj 903283d76e flow: declare and use constansts where possible 8 months ago
Shivani Bhardwaj 00a644c5c2 flow/manager: make fn calls only when necessary 8 months ago
Shivani Bhardwaj eb95d2bf66 flow/timeout: cleanup fn names and comments 8 months ago
Shivani Bhardwaj 8818b9cbe0 flow: remove unneeded args to fn 8 months ago
Shivani Bhardwaj f97b4ec1e8 flow/manager: add fn docs 8 months ago
Shivani Bhardwaj 14cd594d3c flow: add defensive check 8 months ago
Shivani Bhardwaj a87c8eb46f packetpool: use DEBUG_VALIDATE statement 8 months ago
Shivani Bhardwaj 87fa7f10ef flow: use bool wherever possible 8 months ago
Philippe Antoine 8b831e6751 detect/icmp: require real packet in signature
Fixes: 956c8bebd1 ("detect/prefilter: use sig mask to exclude pkt engines")
8 months ago
Philippe Antoine 0a953fe1ce detect: add to signature mask for decode events
Ticket: 6291
8 months ago
Philippe Antoine 4e584ed201 detect: fix check for app_layer events
Ticket: 7106
8 months ago
Jeff Lucovsky 834cd6fbdb af-packet: Remove unused preprocessor define
Remove unused preprocessor value; exposed by compilation warning
8 months ago
Philippe Antoine c9ce43b31e output: configurable payload_length field for alerts
Ticket: 7098
8 months ago
Philippe Antoine a21232828e dcerpc: add app-layer metadata in alerts
Ticket: 6090
8 months ago
Philippe Antoine 5f35035928 filestore: do not try to store a file set to nostore
Ticket: 6390

This can happen with keyword filestore:both,flow
If one direction does not have a signature group with a filestore,
the file is set to nostore on opening, until a signature in
the other direction tries to set it to store.
Subsequent files will be stored in both directions as flow flags
are now set.
8 months ago
Philippe Antoine 0d4efe0c0f app-layer: fix -Wshorten-64-to-32 warnings
Ticket: #6186

Warnings about downcast from 64 to 32 bits
8 months ago
Philippe Antoine 1790aa49a4 util: fix -Wshorten-64-to-32 warnings
Ticket: 6186

Warnings about downcast from 64 to 32 bits

Generic fixes required to get app-layer clean
8 months ago
Philippe Antoine dc043d0297 detect: remove unused field
content_inspect_window is used in app-layer-smtp, but
not directly in detect-file-data
8 months ago
Victor Julien 3d059611c3 detect: add tls.alpn keyword
Ticket: #7108.
8 months ago