Commit Graph

17165 Commits (suricata-8.0.0-beta1)
 

Author SHA1 Message Date
Shivani Bhardwaj c965c5687d release: 8.0.0-beta1; update changelog 4 months ago
Victor Julien 57c73880db lua: enable lua rules by default
Now that sandboxing is in place, lua rule support is enabled by default.
4 months ago
Alice Akaki d4ec5b9765 detect: add ldap.responses.attribute_type
ldap.responses.attribute_type matches on LDAP attribute type/description
This keyword maps the eve field ldap.responses[].search_result_entry.attributes[].type
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering

Ticket: #7533
4 months ago
Alice Akaki 75fb352bde detect: add ldap.request.attribute_type
ldap.request.attribute_type matches on LDAP attribute type/description
This keyword maps the following eve fields:
ldap.request.search_request.attributes[]
ldap.request.modify_request.changes[].modification.attribute_type
ldap.request.add_request.attributes[].name
ldap.request.compare_request.attribute_value_assertion.description
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering

Ticket: #7533
4 months ago
Alice Akaki 6d48acdd63 rustfmt: rust/src/ldap/detect.rs 4 months ago
Jeff Lucovsky fdc43e5ac4 app/ftp: Eliminate scan-build warning
Scan-build reports that FTPRealloc could be called with size=0. Modify
the logic so it's never passed 0.
4 months ago
Jeff Lucovsky 88c38fc4a0 doc/ftp: Document the ftp.reply keyword
Issue: 7508
4 months ago
Jeff Lucovsky e0131f6098 detect/ftp: Add ftp.reply rule keyword
Issue: 7508

Add the rule detection keyword "ftp.reply". This is a multi-buffer
command.
4 months ago
Jeff Lucovsky d674ce2510 app/ftp: Use Rust FTP response line handling
Use the Rust logic to parse FTP response lines with the goal to support
multi-buffer matches better.

A side effect is that the completion codes are no longer strings; the
schema update reflects this.

Issue: 4082
4 months ago
Jeff Lucovsky dfc896e2a7 app/ftp: Move FTP response handling to rust
Move handling of FTP responses to Rust to improve support for FTP
keyword matching. Parsing the response line when encountered
simplifies multi-buffer matching and metadata output.

Issue: 4082
4 months ago
Juliana Fajardini 63adfc6b1e userguide/pop3: minor typo and formatting fixing 4 months ago
Alex Savage 778053876b pop3: app-layer parser using sawp-pop3
This module uses the sawp-pop3 crate to parse POP3 requests and responses
Features:
- eve logging
- events for parsable but non-RFC-compliant messages

Ticket: 3243
4 months ago
Eric Leblond ed20e7cfe4 doc/userguide: doc domain and tld transforms 4 months ago
Eric Leblond 585c624482 tld: add new transform
Extract the TLD from a buffer.

Ticket: #5639
4 months ago
Eric Leblond 0b031a33c6 domain: add new transformation
Ticket: #5639
4 months ago
Philippe Antoine 8cb6a4b82b plugin: document app-layer plugins
Ticket: 7149
Ticket: 7150
Ticket: 7153
4 months ago
Philippe Antoine 5e87b6bd51 plugin: add in-tree app-layer template plugin for testing
Ticket: 7151
Ticket: 7152
Ticket: 7154
4 months ago
Philippe Antoine 51859050cb rust: export jsonbuilder API in sys crate
using cbindgen + bindgen
4 months ago
Victor Julien 5c7173c6df doc/firewall: start of documentation 4 months ago
Victor Julien 98372b6599 firewall: beginning of ruleset analyzer
Output a `firewall.json` with a per table list of rules.

Also output a sorted list of the threat detection rules.
4 months ago
Victor Julien 31d048ed4b firewall: start of firewall rules support
Config:

Firewall rules are like normal rule, with some key differences.

They are loaded separate, and first, from:

```yaml
firewall-rule-path: /etc/suricata/firewall/
firewall-rule-files:
  - fw.rules
```

Can also be loaded with --firewall-rules-exclusive: Mostly for QA purposes.

Allow -S with --firewall-rules-exclusive, so that firewall and threat detection
rules can be tested together.

Rules:

Differences with regular "threat detection" rules:

1. these rules are evaluated before threat detection rules

2. these rules are evaluated in the order as they appear in the rule file

3. currently only rules specifying an explicit hook at supported

   a. as a consequence, no rules will be treated as (like) IP-only, PD-only or
      DE-only

Require explicit action scope for firewall rules. Default policy is
drop for the firewall tables.

Actions:

New action "accept" is added to allow traffic in the filter tables.

New scope "accept:tx" is added to allow accepting a transaction.

Tables:

Rulesets are per table.

Table processing order: `packet:filter` -> `packet:td` -> `app:*:*` -> `app:td`.

Each of the tables has some unique properties:

`packet:filter`:
- default policy is `drop:packet`
- rules are process in order
- action scopes are explicit
- `drop` or `accept` is immediate
- `accept:hook` continues to `packet:td`

`packet:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `app:*:*` or `alert/action finalize`

`app:*:*`:
- default policy is `drop:flow`
- rules are process in order
- action scopes are explicit
- `drop` is immediate
- `accept` is conditional on possible `drop` from `packet:td`
- `accept:hook` continues to `app:td`, `accept:packet` or `accept:flow`
  continues to `alert/action finalize`

`app:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `alert/action finalize`

Implementation:

During sigorder, split into packet:filter, app:*:* and general td.

Allow fw rules to work when in pass:flow mode. When firewall mode is enabled,
`pass:flow` will not skip the detection engine anymore, but instead
process the firewall rules and then apply the pass before inspecting threat
detect rules.
4 months ago
Victor Julien 10dcc383ae detect/xbits: implement tx bits
Implement `xbits:set,mybit,track tx;` and `xbits:isset,mybit,track tx;`.

Store these in the AppLayerTxData.

Ticket: #6455.
4 months ago
Victor Julien 7c8a55de54 eve/alert: log ts_progress/tc_progress
This is mostly to help with debugging firewall rules, but
can be useful in other places.
4 months ago
Victor Julien 6f5fd77cb9 detect/app-layer-state: keyword for protocol state
Allow matching on the app-layer protocol state.
4 months ago
Victor Julien fff65c5e40 detect/prefilter: allow special engine for each app update
Allow registering the progress as -1, which means it will be invoked
each time the app prefilters are called.
4 months ago
Victor Julien f60e1efc8a detect: start of pkt hooks
New facility to allow a user to specify where to hook a rule
into the engine. This patch adds this for packets, adding two
hooks:

- `all`: to let a rule be evaluated by all rules
- `flow_start`: to have a rule be evaluated only for the first
  packet in both directions

Implemented by adding a hook flags field in the packet.
4 months ago
Victor Julien 95ed975cca detect/tls.version: allow using rule hook
tls.version isn't hooked to a specific state by default. Allow it
to register at the rule hook.
4 months ago
Victor Julien 4071acd659 detect: explicit action scopes
For drop and pass allow setting the scope explicitly:

        drop:flow
        drop:packet

        pass:flow
        pass:packet

Ticket: #7481.
4 months ago
Victor Julien 9539002b39 detect/tls: more precise state registration for keywords 4 months ago
Victor Julien 3485e57953 detect: reuse hook based generic lists 4 months ago
Victor Julien 592444cdc4 detect/lua: use rule hook instead of init() 'needs' table
For registration of app-layer inspection, no longer use the 'needs'
table from the script, but instead use the rule hook setting.

Ticket: #4783.
4 months ago
Victor Julien 67df6ad94e detect: introduce explicit hooks
Generic:
        <app_proto>:request_started and <app_proto>:response_started
        <app_proto>:request_complete and <app_proto>:response_complete

Per protocol, it uses the registered progress (state) values. E.g.

        tls:client_hello_done

A rule ruleset could be:

        pass tls:client_hello_done any any -> any any (tls.sni; content:"www.google.com"; sid:21; alert;)
        drop tls:client_hello_done any any -> any any (sid:22;)

The pass rule is evaluated when the client hello is parsed, and if it
doesn't match the drop rule will be evaluated.

Registers each generic lists as "<alproto>:<progress state>:generic"
(e.g. "tls:client_hello_done:generic").

Ticket: #7485.
4 months ago
Victor Julien 8c674c8647 http1: register progress state names
Use `request_started` for HTP_REQUEST_NOT_STARTED as when data is
received, the request parsing has started. The request line isn't
complete yet.

Similar for `response_started`.
4 months ago
Victor Julien 6795b4bbb0 tls: expose progress by name 4 months ago
Victor Julien add7d3fcf5 app-layer: API for mapping progress name vs id 4 months ago
Victor Julien 780aeee55c tls: introduce per direction progress tracking
Per direction track progress to be able to have more fine grained
control over where the detection engines and logging hooks in.
4 months ago
Victor Julien b5bc00382d detect: allow longer buffer names
To support hook based buffer names.
4 months ago
Victor Julien 94644ac960 detect: move non-pf rules into special prefilter engines
Instead of having a per detection engine list of rule that couldn't be
prefiltered, put those into special "prefilter" engines.

For packet and frame rules this doesn't change much, it just removes
some hard coded logic from the detect engine.

For the packet non-prefilter rules in the "non-prefilter" special prefilter
engine, add additional filtering for the packet variant. It can prefilter on
alproto, dsize and dest port.

The frame non-prefilter rules are added to a single engine, that per
rule checks the alproto and the type.

For app-layer, there is an engine per progress value, per app-layer
protocol and per direction. This hooks app-layer non-prefilter rules
into the app inspect logic at the correct "progress" hook.

e.g. a rule like
        dns.query; bsize:1;

Negated MPM rules will also fall into this category:
        dns.query; content:!"abc";

Are part of a special "generic list" app engine for dns, at the
same progress hook as `dns.query`.

This all results in a lot fewer checks:

previous:

  --------------------------------------------------------------------------
  Date: 1/29/2025 -- 10:22:25. Sorted by: number of checks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        20           1        0        181919672    11.85  588808   221      60454       308.96      2691.46     308.07
  2        50           1        0        223455914    14.56  453104   418      61634       493.17      3902.59     490.02
  3        60           1        0        185990683    12.12  453104   418      60950       410.48      1795.40     409.20
  4        51           1        0        192436011    12.54  427028   6084     61223       450.64      2749.12     417.42
  5        61           1        0        180401533    11.75  427028   6084     61093       422.46      2177.04     397.10
  6        70           1        0        153899099    10.03  369836   0        61282       416.13      0.00        416.13
  7        71           1        0        123389405    8.04   369836   12833    44921       333.63      2430.23     258.27
  8        41           1        0        63889876     4.16   155824   12568    39138       410.01      1981.97     272.10
  9        40           1        0        64149724     4.18   155818   210      39792       411.70      4349.57     406.38
  10       10           1        0        70848850     4.62   65558    0        39544       1080.70     0.00        1080.70
  11       11           1        0        94743878     6.17   65558    32214    60547       1445.19     2616.14     313.92

this commit:

  --------------------------------------------------------------------------
  Date: 1/29/2025 -- 10:15:46. Sorted by: number of checks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        50           1        0        138776766    19.23  95920    418      167584      1446.80     3953.11     1435.83
  2        60           1        0        97988084     13.58  95920    418      182817      1021.56     1953.63     1017.48
  3        51           1        0        105318318    14.60  69838    6084     65649       1508.04     2873.38     1377.74
  4        61           1        0        89571260     12.41  69838    6084     164632      1282.56     2208.41     1194.20
  5        11           1        0        91132809     12.63  32779    32214    373569      2780.22     2785.58     2474.45
  6        10           1        0        66095303     9.16   32779    0        56704       2016.39     0.00        2016.39
  7        70           1        0        48107573     6.67   12928    0        42832       3721.19     0.00        3721.19
  8        71           1        0        32308792     4.48   12928    12833    39565       2499.13     2510.05     1025.09
  9        41           1        0        25546837     3.54   12886    12470    41479       1982.53     1980.84     2033.05
  10       40           1        0        26069992     3.61   12886    210      38495       2023.13     4330.05     1984.91
  11       20           1        0        639025       0.09   221      221      14750       2891.52     2891.52     0.00
4 months ago
Victor Julien 187f71de24 detect: check for a flow drop first 4 months ago
Victor Julien 84a9673af6 flow: remove FLOW_NOPACKET_INSPECTION flag
It was not used separately from FLOW_ACTION_PASS.
4 months ago
Victor Julien a4ae92319f detect: skip duplicate nopacket inspect check 4 months ago
Philippe Antoine 706a0bd99b output: option to add suricata version in eve logs
Ticket: 4853
4 months ago
Jason Ish 0b551641b4 pcap-log: define PCAP_NETMASK_UNKNOWN if not set
This is defined in by newer versions of libpcap,
but older versions are still in use.
4 months ago
Jason Ish cddd3a261f pcap-log: only allow one instance of pcap-log
As a global is used to store state, only one instance can be used.

See https://redmine.openinfosecfoundation.org/issues/7640.
4 months ago
Jason Ish fdc17b2125 pcap-log: add stats for written and filtered
Filtered is the number of packets not written as they were filtered out.
4 months ago
Jason Ish 2fce106aec pcap-log: add bpf filter for packets that are logged
Add an optional bpf filter to pcap-log. If set, packets must match the
filter to be logged, otherwise they will be ignored.

This allows a user to limit what is logged to disk if they have pcap-log
enabled, but still inspect all data captured.

Ticket: #6832
4 months ago
Philippe Antoine c203ff774d http: restore behavior for event http.uri_delim_non_compliant
If we see a space-like character that is not space 0x20 in uri,
we set this event, even it the request line finished with a normal
space and protocol

Fixes: 9c324b796e ("http: Use libhtp-rs.)
4 months ago
Lukas Sismis fbed416e26 doc: stylistic and grammar fixes 4 months ago
Lukas Sismis 7574be1dc6 ssh: sync the hassh setting with the defaults 4 months ago
Dean Balandin 3cdb0ceb7e stream: decouple stream.bypass dependency from tls bypass
Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.

Ticket: 6788
4 months ago