Commit Graph

17142 Commits (6f5fd77cb9271dd9bebe304d73a433e17f891f93)
 

Author SHA1 Message Date
Victor Julien 6f5fd77cb9 detect/app-layer-state: keyword for protocol state
Allow matching on the app-layer protocol state.
1 year ago
Victor Julien fff65c5e40 detect/prefilter: allow special engine for each app update
Allow registering the progress as -1, which means it will be invoked
each time the app prefilters are called.
1 year ago
Victor Julien f60e1efc8a detect: start of pkt hooks
New facility to allow a user to specify where to hook a rule
into the engine. This patch adds this for packets, adding two
hooks:

- `all`: to let a rule be evaluated by all rules
- `flow_start`: to have a rule be evaluated only for the first
  packet in both directions

Implemented by adding a hook flags field in the packet.
1 year ago
Victor Julien 95ed975cca detect/tls.version: allow using rule hook
tls.version isn't hooked to a specific state by default. Allow it
to register at the rule hook.
1 year ago
Victor Julien 4071acd659 detect: explicit action scopes
For drop and pass allow setting the scope explicitly:

        drop:flow
        drop:packet

        pass:flow
        pass:packet

Ticket: #7481.
1 year ago
Victor Julien 9539002b39 detect/tls: more precise state registration for keywords 1 year ago
Victor Julien 3485e57953 detect: reuse hook based generic lists 1 year ago
Victor Julien 592444cdc4 detect/lua: use rule hook instead of init() 'needs' table
For registration of app-layer inspection, no longer use the 'needs'
table from the script, but instead use the rule hook setting.

Ticket: #4783.
1 year ago
Victor Julien 67df6ad94e detect: introduce explicit hooks
Generic:
        <app_proto>:request_started and <app_proto>:response_started
        <app_proto>:request_complete and <app_proto>:response_complete

Per protocol, it uses the registered progress (state) values. E.g.

        tls:client_hello_done

A rule ruleset could be:

        pass tls:client_hello_done any any -> any any (tls.sni; content:"www.google.com"; sid:21; alert;)
        drop tls:client_hello_done any any -> any any (sid:22;)

The pass rule is evaluated when the client hello is parsed, and if it
doesn't match the drop rule will be evaluated.

Registers each generic lists as "<alproto>:<progress state>:generic"
(e.g. "tls:client_hello_done:generic").

Ticket: #7485.
1 year ago
Victor Julien 8c674c8647 http1: register progress state names
Use `request_started` for HTP_REQUEST_NOT_STARTED as when data is
received, the request parsing has started. The request line isn't
complete yet.

Similar for `response_started`.
1 year ago
Victor Julien 6795b4bbb0 tls: expose progress by name 1 year ago
Victor Julien add7d3fcf5 app-layer: API for mapping progress name vs id 1 year ago
Victor Julien 780aeee55c tls: introduce per direction progress tracking
Per direction track progress to be able to have more fine grained
control over where the detection engines and logging hooks in.
1 year ago
Victor Julien b5bc00382d detect: allow longer buffer names
To support hook based buffer names.
1 year ago
Victor Julien 94644ac960 detect: move non-pf rules into special prefilter engines
Instead of having a per detection engine list of rule that couldn't be
prefiltered, put those into special "prefilter" engines.

For packet and frame rules this doesn't change much, it just removes
some hard coded logic from the detect engine.

For the packet non-prefilter rules in the "non-prefilter" special prefilter
engine, add additional filtering for the packet variant. It can prefilter on
alproto, dsize and dest port.

The frame non-prefilter rules are added to a single engine, that per
rule checks the alproto and the type.

For app-layer, there is an engine per progress value, per app-layer
protocol and per direction. This hooks app-layer non-prefilter rules
into the app inspect logic at the correct "progress" hook.

e.g. a rule like
        dns.query; bsize:1;

Negated MPM rules will also fall into this category:
        dns.query; content:!"abc";

Are part of a special "generic list" app engine for dns, at the
same progress hook as `dns.query`.

This all results in a lot fewer checks:

previous:

  --------------------------------------------------------------------------
  Date: 1/29/2025 -- 10:22:25. Sorted by: number of checks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        20           1        0        181919672    11.85  588808   221      60454       308.96      2691.46     308.07
  2        50           1        0        223455914    14.56  453104   418      61634       493.17      3902.59     490.02
  3        60           1        0        185990683    12.12  453104   418      60950       410.48      1795.40     409.20
  4        51           1        0        192436011    12.54  427028   6084     61223       450.64      2749.12     417.42
  5        61           1        0        180401533    11.75  427028   6084     61093       422.46      2177.04     397.10
  6        70           1        0        153899099    10.03  369836   0        61282       416.13      0.00        416.13
  7        71           1        0        123389405    8.04   369836   12833    44921       333.63      2430.23     258.27
  8        41           1        0        63889876     4.16   155824   12568    39138       410.01      1981.97     272.10
  9        40           1        0        64149724     4.18   155818   210      39792       411.70      4349.57     406.38
  10       10           1        0        70848850     4.62   65558    0        39544       1080.70     0.00        1080.70
  11       11           1        0        94743878     6.17   65558    32214    60547       1445.19     2616.14     313.92

this commit:

  --------------------------------------------------------------------------
  Date: 1/29/2025 -- 10:15:46. Sorted by: number of checks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        50           1        0        138776766    19.23  95920    418      167584      1446.80     3953.11     1435.83
  2        60           1        0        97988084     13.58  95920    418      182817      1021.56     1953.63     1017.48
  3        51           1        0        105318318    14.60  69838    6084     65649       1508.04     2873.38     1377.74
  4        61           1        0        89571260     12.41  69838    6084     164632      1282.56     2208.41     1194.20
  5        11           1        0        91132809     12.63  32779    32214    373569      2780.22     2785.58     2474.45
  6        10           1        0        66095303     9.16   32779    0        56704       2016.39     0.00        2016.39
  7        70           1        0        48107573     6.67   12928    0        42832       3721.19     0.00        3721.19
  8        71           1        0        32308792     4.48   12928    12833    39565       2499.13     2510.05     1025.09
  9        41           1        0        25546837     3.54   12886    12470    41479       1982.53     1980.84     2033.05
  10       40           1        0        26069992     3.61   12886    210      38495       2023.13     4330.05     1984.91
  11       20           1        0        639025       0.09   221      221      14750       2891.52     2891.52     0.00
1 year ago
Victor Julien 187f71de24 detect: check for a flow drop first 1 year ago
Victor Julien 84a9673af6 flow: remove FLOW_NOPACKET_INSPECTION flag
It was not used separately from FLOW_ACTION_PASS.
1 year ago
Victor Julien a4ae92319f detect: skip duplicate nopacket inspect check 1 year ago
Philippe Antoine 706a0bd99b output: option to add suricata version in eve logs
Ticket: 4853
1 year ago
Jason Ish 0b551641b4 pcap-log: define PCAP_NETMASK_UNKNOWN if not set
This is defined in by newer versions of libpcap,
but older versions are still in use.
1 year ago
Jason Ish cddd3a261f pcap-log: only allow one instance of pcap-log
As a global is used to store state, only one instance can be used.

See https://redmine.openinfosecfoundation.org/issues/7640.
1 year ago
Jason Ish fdc17b2125 pcap-log: add stats for written and filtered
Filtered is the number of packets not written as they were filtered out.
1 year ago
Jason Ish 2fce106aec pcap-log: add bpf filter for packets that are logged
Add an optional bpf filter to pcap-log. If set, packets must match the
filter to be logged, otherwise they will be ignored.

This allows a user to limit what is logged to disk if they have pcap-log
enabled, but still inspect all data captured.

Ticket: #6832
1 year ago
Philippe Antoine c203ff774d http: restore behavior for event http.uri_delim_non_compliant
If we see a space-like character that is not space 0x20 in uri,
we set this event, even it the request line finished with a normal
space and protocol

Fixes: 9c324b796e ("http: Use libhtp-rs.)
1 year ago
Lukas Sismis fbed416e26 doc: stylistic and grammar fixes 1 year ago
Lukas Sismis 7574be1dc6 ssh: sync the hassh setting with the defaults 1 year ago
Dean Balandin 3cdb0ceb7e stream: decouple stream.bypass dependency from tls bypass
Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.

Ticket: 6788
1 year ago
Lukas Sismis 5e8c775d77 ssh: add option to select behavior of encrypted parts
Ticket: 6788
1 year ago
Lukas Sismis 185123a130 tls: deprecate "default" as a encryption-handling option
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
1 year ago
Jason Ish 92d5a3fd0a lua/detect: remove rule sid, rev and gid globals
For some rules the values are always 0, for others they get set, so not
consistent. Also not documented and not tested and should be replaced by
a "suricata.rule" lib.

For example, these globals would be set for this rule:

alert tcp any any -> any any (flow:to_server;
flowbits:isnotset,dataset_added; lua:dataset.lua;
flowbits:set,dataset_added;sid:1;rev:99;)

But not this rule:

alert dns any any -> any any (msg:"TEST DNS LUA dns.rrname"; \
      content:"org"; lua:test-rrname.lua; sid:1; rev:1;)

Related to ticket
https://redmine.openinfosecfoundation.org/issues/7490 (but not a fix)

This does not affect the functions with similar names for getting rule
details in output context.
1 year ago
Philippe Antoine 3b271b3f5b detect: rename with prefix SCSigTablePreRegister
to make it available to rust via bindgen
1 year ago
Philippe Antoine e3ee922fbf snmp: restrict rust visibility 1 year ago
Philippe Antoine d24a3eb5f6 snmp: register protocol dynamically
Ticket: 4103
1 year ago
Philippe Antoine 640a4c8b11 snmp: make log function use the generic prototype
and cast

and also remove unneeded mut

and rustfmt
1 year ago
Philippe Antoine 924129b185 output: public header for use in rust
Ticket: 5053

Move enum OutputJsonLogDirection and struct
EveJsonTxLoggerRegistrationData to a public header user by rust
thanks to bindgen

Rename to use SC prefix on the way

And make EveJsonSimpleTxLogFunc use a const pointer to transaction
1 year ago
Philippe Antoine c32f2dee70 scripts: remove unused code in setup-app-layer
and fix typo
1 year ago
Philippe Antoine 243d14cda2 output: plugins can log flow or packet direction 1 year ago
Philippe Antoine 850bd72399 app-layer: helper function AppProtoNewProtoFromString
So that we do not have to know g_alproto_max to register
dynamically a new protocol from its name
1 year ago
Philippe Antoine 6750274d48 app-layer: make some arrays even more dynamic
Ticket: 5053

Do not asume that we know the number of alprotos at the end
of AppLayerNamesSetup, but make arrays allocated by later
AppLayerProtoDetectSetup dynamic so that it can be reallocated
from AppLayerParserRegisterProtocolParsers

This helps have a single entry point for a protocol like SNMP
1 year ago
Philippe Antoine 58a8559440 detect: fix overflow for files protocol
CID 1640392

Would happen only if we reached 15 protocols handling files
1 year ago
Philippe Antoine 6afad1af51 smb/log: configuration option for types logging
suricata.yaml output section for smb now parses a types list
and will restrict logging of transactions to these types.

By default, everything still gets logged

Remove unused rs_smb_log_json_request on the way

Ticket: 7620
1 year ago
Philippe Antoine 30af626a92 conf: extend API for rust
Will allow rust to list the values of an array such as
dns.types for outputs
1 year ago
Juliana Fajardini ef8b13c1ef pgsql: use generic name for TerminationMessage msg
Since this structure can be used for different messages with other
meanings.

Related to
Task #4854
1 year ago
Juliana Fajardini f06ec5336e doc/pgsql: document CopyOut-related response msgs
Related to
Task #4854
1 year ago
Juliana Fajardini e75fcffa29 pgsql: add initial support to copy-out subproto
This sub-protocol inspects messages exchanged between postgresql backend
and frontend after a 'COPY TO STDOUT' has been processed.

Parses new messages:
- CopyOutResponse -- initiates copy-out mode/sub-protocol
- CopyData -- data transfer messages
- CopyDone -- signals that no more CopyData messages will be seen from
  the sender for the current transaction

Task #4854
1 year ago
Jeff Lucovsky 22ea5ddbb7 detect/transforms: Add engine detect thread ctx to signature
Modify the transform function signature to include the detect engine
thread ctx.
1 year ago
Jeff Lucovsky 590c8f6131 doc/ftp: Add ftp.command_data
This commits adds documentation for the ftp.command_data rule keyword
that includes usage examples.
1 year ago
Jeff Lucovsky 559cdc358d detect/ftp: ftp.command_data keyword
Issue: 7503

Add the rule keyword ftp.command_data; a sticky buffer that matches on
the values send with FTP commands, e.g., USER someuser
1 year ago
Philippe Antoine 0f3932afb7 rust: fix clippy warning manual_contains
warning: using `contains()` instead of `iter().any()` is more efficient
   --> src/http2/http2.rs:267:20
    |
267 |                 if block.value.iter().any(|&x| x == b'@') {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `block.value.contains(&b'@')`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_contains
    = note: `#[warn(clippy::manual_contains)]` on by default
1 year ago
Philippe Antoine 857efdde62 rust: fix clippy warnings doc_overindented_list_items
warning: doc list item overindented
   --> src/nfs/rpc_records.rs:284:5
    |
284 | ///           type: bool
    |     ^^^^^^^^^^ help: try using `  ` (2 spaces)
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_overindented_list_items
    = note: `#[warn(clippy::doc_overindented_list_items)]` on by default
1 year ago