Commit Graph

15587 Commits (suricata-7.0.11)
 

Author SHA1 Message Date
Shivani Bhardwaj 98b12d1c30 release: 7.0.11; update changelog 3 weeks ago
Philippe Antoine 7fa88ea9e7 http2: do not set file flags for global txs
Global txs means here txs with stream id 0, used for connection control
messages.

(cherry picked from commit fa8d3a4ccb)
3 weeks ago
Philippe Antoine de2bdfec99 http2: mark old txs as updated
As is done in the other case a few lines below

(cherry picked from commit 349c21af2c)
3 weeks ago
Philippe Antoine 97eee2cada http2: forbid data on stream 0
Ticket: 7658

Suricata will not handle well if we open a file for this tx,
do not close it, but set the transaction state to completed.

RFC 9113 section 6.1 states:

If a DATA frame is received whose Stream Identifier field is 0x00,
the recipient MUST respond with a connection error (Section 5.4.1)
 of type PROTOCOL_ERROR.

(cherry picked from commit 1d6d331752)
3 weeks ago
Philippe Antoine 805ac10fad rust/smb: fix manual_unwrap_or_default
warning: match can be simplified with `.unwrap_or_default()`
   --> src/smb/smb2.rs:682:41
    |
682 |                           let _guid_vec = match state.ssn2vec_map.remove(&guid_key) {
    |  _________________________________________^
683 | |                             Some(p) => p,
684 | |                             None => {
685 | |                                 SCLogDebug!("SMBv2 response: GUID NOT FOUND");
686 | |                                 Vec::new()
687 | |                             },
688 | |                         };
    | |_________________________^ help: replace it with: `state.ssn2vec_map.remove(&guid_key).unwrap_or_default()`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
    = note: `#[warn(clippy::manual_unwrap_or_default)]` on by default
1 month ago
Philippe Antoine 8d45e8c95a rust: allow collapsible_else_if for debug logs
see https://github.com/rust-lang/rust-clippy/issues/15158
1 month ago
Philippe Antoine 30be40a483 rust: make cargo clippy clean
Fixing single_match and manual_find intertwined with SCLogDebug

(cherry picked from commit 38db51b878)
1 month ago
Philippe Antoine 9a0edd0ce5 rust/dns: fix clippy char_indices_as_byte_indices
error: indexing into a string with a character position where a byte index is expected
  --> src/dns/detect.rs:45:39
   |
45 |                 let code: u8 = opcode[i..].parse().map_err(|_| ())?;
   |                                       ^
   |
   = note: a character can take up more than one byte, so they are not interchangeable
note: position comes from the enumerate iterator
  --> src/dns/detect.rs:36:10
   |
36 |     for (i, c) in opcode.chars().enumerate() {
   |          ^                       ^^^^^^^^^^^
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#char_indices_as_byte_indices
   = note: `#[deny(clippy::char_indices_as_byte_indices)]` on by default
help: consider using `.char_indices()` instead
   |
36 -     for (i, c) in opcode.chars().enumerate() {
36 +     for (i, c) in opcode.char_indices() {
1 month ago
Victor Julien 3528625f04 threading: fix shutdown of IPS autofp modes
For IPS modes with a verdict thread in autofp there was an issue with
the verdict thread not shutting down, leading to a long shutdown time
until an error condition was reached.

The problem was that when the packet threads, of which the verdict
thread is one, were told to enter their flow timeout loop the verdict
thread got stuck as it immediately progressed to THV_RUNNING_DONE
instead of the expected THV_FLOW_LOOP.

This patch updates the shutdown logic to only apply the flow timeout
logic to the relevant threads, and skip the verdict thread(s).

Add TM_FLAG_VERDICT_TM to indicate a thread has a verdict module to more
explicitly shut it down.

Fixes: 12f8f03532 ("threads: fix autofp shutdown race condition")

Bug: #7681.
(cherry picked from commit bdac028fc7)
1 month ago
Victor Julien d8d44648eb flow: fix unittests for ThreadVars requirement
(cherry picked from commit ee59d9a894)
1 month ago
Victor Julien eeb0b29000 flow: fix time handling for non-TCP
Track per flow thread id for UDP and other non-TCP protocols. This
improves the timeout handling as the per thread timestamp is used in
offline mode.

Fixes: ada2bfe009 ("flow/worker: improve flow timeout time accuracy")
Fixes: ef396f7509 ("flow/manager: in offline mode, use owning threads time")

Bug #7687.

(cherry picked from commit c648abad0d)
1 month ago
Victor Julien 136b43713a threads: fix autofp shutdown race condition
Sometimes a single flow pcap would log 2 flows. It turns out FlowWorkToDoCleanup
ran before all the packet threads had processed their "wire" packets. It then
removed a flow that a wire packet would still have needed, leading to the worker
thread creating a new flow for it.

This could happen due to the logic in TmThreadDisableReceiveThreads which calls
TmThreadDrainPacketThreads to made sure it only returns when all autofp-workers
have processed all the packets the autofp-capture thread fed to them.

However, the way it checked this is by checking the size of the autofp-worker's
input queue. If 0, it assumes it is done.

What this missed, is that a worker thread could have just taken the last packet
from the input queue, but it is not yet done processing it. If then the
FlowWorkToDoCleanup is ran as well, it would race the worker thread to the flow
handling logic. When it won, the flow was evicted and the packet thread
created a new flow.

This patch improves the shutdown logic to force the worker threads to
enter a "flow loop" (THV_FLOW_LOOP) state before moving on to the
FlowWorkToDoCleanup step. This makes sure that any in progress packets
in the worker threads have been processed.

Bug: #7681.
(cherry picked from commit 12f8f03532)
1 month ago
Victor Julien fe1f846e50 threads: pktacq loop cleanup
Manual backport of relevant bits from:
35d7d77ddb ("threads: refactor TmThreadsSlotPktAcqLoop for user threads")
1 month ago
Victor Julien 75b32ac418 threads: remove unused flag
(cherry picked from commit b42eea67d5)
1 month ago
Lukas Sismis fce8336c93 pcap-file: document capture method options
(cherry picked from commit eb52e337da)
1 month ago
Lukas Sismis 96b3bc4d9b doc: update available options in the example config
(cherry picked from commit e780a20f82)
1 month ago
Lukas Sismis 2764695596 flow-manager: move time check after RUNNNING state change
When running in pcap-file mode and with a continous directory
reading, the provided directory can be empty.
By having no packets and being in offline mode, the initial packet timestamp
is never set. However, Flow Manager waited until the timestamp was set
to proceed with transferring its state to "RUNNING".
Other pcap-related threads (RX / workers) are set in RUNNING state while
waiting for the PCAP to appear in the directory.

As a result, the main Suricata-Main thread timed out after the default
60 seconds budget for threads to turn from INIT_DONE to RUNNING state.

Ticket: 7661
(cherry picked from commit 58df970391)
1 month ago
Lukas Sismis 2ae33c6c4c dpdk: use default iface-copy value if not specified
Ticket: 7375
(cherry picked from commit 31fbfc322c)
1 month ago
Philippe Antoine 290e317150 util/mpm: grow state queue on demand
Ticket: 7678
(cherry picked from commit 9f83662f20)
1 month ago
Philippe Antoine 51b3254dfa util/mpm: factorize code
(cherry picked from commit 679bd23cb7)
1 month ago
Philippe Antoine e4cf0b22cd mpm: allocate StateQueue on the heap for ks
Completes commit 92fce2fdc0

Ticket: 6264
(cherry picked from commit 330cff94e8)
1 month ago
Jeff Lucovsky f3b544eec8 detect/content: account for distance variables
Under some cases (below), the depth and offset values are used
twice. This commit disregards the distance variable (if any), when
computing the final depth.

These rules are logically equivalent::
1. alert tcp any any -> any 8080 (msg:"distance name"; flow:to_server; content:"Authorization:"; content:"5f71ycy"; distance:0; byte_extract:1,0,option_len,string,relative; content:!"|38|"; distance:option_len; within:1; content:"|37|"; distance:-1; within:1; content:"|49|"; distance:option_len; within:1; sid:1;)
2. alert tcp any any -> any 8080 (msg:"distance number"; flow:to_server; content:"Authorization:"; content:"5f71ycy"; distance:0; byte_extract:1,0,option_len,string,relative; content:!"|38|"; distance:7; within:1; content:"|37|"; distance:-1; within:1; content:"|49|"; distance:option_len; within:1; sid:2;)

The differences:
Rule 1: content:!"|38|"; distance:option_len; within:1; //option_len == 7

Rule 2: content:!"|38|"; distance:7; within:1;

Without this commit, rule 2 triggers an alert but rule 1 doesn't.

Issue: 7390
(cherry picked from commit ace0d37636)
2 months ago
Philippe Antoine b027350efc rust: update brotli crate to latest version
Ticket: 7735

New version has a fix for an integer underflow

(cherry picked from commit 97591230a9)
2 months ago
Philippe Antoine dd4687486b dcerpc: use wrapping to prevent u16 overflow
Otherwise, rust with debug assertion may trigger a panic

Ticket: 7730

(cherry picked from commit 261d2ad63b)
2 months ago
Philippe Antoine 0e9d05b8e5 snmp: probing parser returns unknown if not enough data
Ticket: 7019
(cherry picked from commit 54a3a18a9e)
2 months ago
Philippe Antoine a348f36cd8 scripts: clang-format can use a different base than master
useful for git hook running on main-7.0.x branches so that
not every commit gets its format checked again.

Ticket: 7292
(cherry picked from commit cca169f307)
2 months ago
Eric Leblond 5ef11ae022 datasets: fix set with ip sets
It can get an IPv6 or an IPv4 so we need to handle both length.

Ticket: #7689
(cherry picked from commit e499a98ba9)
2 months ago
Jason Ish cc5511c094 github-ci: use current directory for unit test logging
/tmp appears to exist when you make it, but doesn't appear to actually
exist after msys translation, so just use "."

(cherry picked from commit 11cef2980b)
3 months ago
Victor Julien 8a32965609 detect: don't set conflicting packet/flow actions
If for the same a packet a drop rule and a pass rule would match,
the applying of actions could be contradictionary:

- the drop would be applied to the packet
- the pass rule would also be considered, not overriding the drop,
  but still setting the flow pass flag.

This would lead to the packet being dropped, but the rest of the
flow getting passed, including retransmissions of the dropped
packet.

This patch only sets drop/pass actions if no conflicting action
has been set on the packet before. It respects the action-order.

Bug: #7653.

Fix based on:
57b17fb3b2 ("detect: don't set conflicting packet/flow actions")
3 months ago
Philippe Antoine 155487e79d detect: do not bug on tx data being NULL
Ticket: 7610

As this can happen for HTTP1 in Suricata 7

This was fixed in Suricata 8 by f301cd3702
and 833a738dd1 from ticket 5739
3 months ago
Philippe Antoine e23aa06196 dnp3: mark tx as updated when creating it
Ticket: 7668

We should set updated_tx when allocating a dnp3 tx

(cherry picked from commit e41c28f7c9)
3 months ago
Philippe Antoine 529bec83b2 ftp: mark tx as updated when creating it
Ticket: 7668

We should set updated_tx when allocating a ftp tx

Was already done right for updated_tc

(cherry picked from commit f24d3ffb74)
3 months ago
Philippe Antoine a954066812 http1: always mark tx as updated on request/response start
Ticket: 7668

We should set updated_tx when allocating HtpTxUserData

(cherry picked from commit a5b987266b)
3 months ago
Philippe Antoine 858739519d rust: fix clippy warning manual_contains
warning: using `contains()` instead of `iter().any()` is more efficient
   --> src/http2/http2.rs:267:20
    |
267 |                 if block.value.iter().any(|&x| x == b'@') {
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `block.value.contains(&b'@')`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_contains
    = note: `#[warn(clippy::manual_contains)]` on by default

(cherry picked from commit 0f3932afb7)
4 months ago
Jason Ish 2826d670d6 rust: fix rustdoc indentation in lists
Ticket: #7652
4 months ago
Jason Ish 1504dcea4a rust: fix clippy warnings for unspecified extern ABI
Fix done by clippy --fix.

Ticket: #7652
4 months ago
Jason Ish f19e28f121 version: start development towards 7.0.11 4 months ago
Philippe Antoine cb38f509ce fuzz: set flow flags as in Suricata
Fixes: d8ddef4c14 ("detect: delay tx cleanup in some edge case")
(cherry picked from commit 09aed7e243)
4 months ago
Jason Ish 4a9d081490 github-ci: don't run builds on PR if only docs changed
(cherry picked from commit 3658d502ff)
4 months ago
Jason Ish cd5a961b36 github-ci: update Fedora non-root build to Fedora 41
(cherry picked from commit 65b863b087)
4 months ago
Jason Ish 8674befdc7 github-ci: remove fedora 40 builds where 41 exists
Remove Fedora 40 builds where there is a Fedora 41 equivalent.

(cherry picked from commit 70d5bae160)
4 months ago
Victor Julien 7816a51f98 github-ci: update Fedora 39 jobs to 41
(cherry picked from commit c56b741088)
4 months ago
Victor Julien 9d89bbd2dd github-ci: bump known rust ver to 1.85.1 4 months ago
Shivani Bhardwaj 9378707700 release: 7.0.10; update changelog 4 months ago
Victor Julien 9b02cd42d0 detect: add padding to suppress scan-build warning
Add to DetectEngineCtx to avoid:

./detect.h:840:16: warning: Excessive padding in 'struct DetectEngineCtx_' (32 padding bytes, where 0 is optimal). Optimal fields order: sig_list, srepCIDR_ctx, sig_array, sc_sig_order_funcs, sgh_hash_table, mpm_hash_table, pattern_hash_table, dup_sig_hash_table, spm_global_thread_ctx, mpm_ctx_factory_container, sgh_array, decoder_event_sgh, rule_file, sigerror, keyword_hash, next, dport_hash_table, tcp_whitelist, udp_whitelist, address_table, metadata_table, buffer_type_hash_name, buffer_type_hash_id, app_mpms_list, app_inspect_engines, pkt_inspect_engines, pkt_mpms_list, frame_inspect_engines, frame_mpms_list, prefilter_hash_table, fp_support_smlist_list, class_conf_ht, class_conf_regex, class_conf_regex_match, reference_conf_ht, reference_conf_regex, reference_conf_regex_match, ea, tenant_path, requirements, last_reload, sig_stat, ths_ctx, io_ctx, flow_gh, tenant_id, sig_cnt, srep_version, sig_array_size, sig_array_len, signum, non_pf_store_cnt_max, inspection_recursion_limit, filemagic_thread_ctx_id, max_fb_id, sgh_array_cnt, sgh_array_size, sgh_mpm_context_proto_tcp_packet, sgh_mpm_context_proto_udp_packet, sgh_mpm_context_proto_other_packet, sgh_mpm_context_stream, byte_extract_max_local_id, version, rule_line, keyword_id, type, ref_cnt, loader_id, prefilter_setting, buffer_type_id, app_mpms_list_cnt, pkt_mpms_list_cnt, frame_mpms_list_cnt, prefilter_id, filedata_config, max_uniq_toclient_groups, max_uniq_toserver_groups, base64_decode_max_len, filestore_cnt, failure_fatal, flags, mpm_matcher, spm_matcher, guess_applayer, sigerror_silent, sigerror_ok, sigerror_requires, filedata_config_initialized, sgh_mpm_ctx_cnf, config_prefix, sm_types_prefilter, sm_types_silent_error, consider reordering the fields or adding explicit padding members [optin.performance.Padding]
  840 | typedef struct DetectEngineCtx_ {
      |         ~~~~~~~^~~~~~~~~~~~~~~~~~
  841 |     bool failure_fatal;
      |     ~~~~~~~~~~~~~~~~~~~
  842 |     uint8_t flags;       /**< only DE_QUIET */
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  843 |     uint8_t mpm_matcher; /**< mpm matcher this ctx uses */
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  844 |     uint8_t spm_matcher; /**< spm matcher this ctx uses */
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  845 |
  846 |     uint32_t tenant_id;
      |     ~~~~~~~~~~~~~~~~~~~
  847 |
  848 |     Signature *sig_list;
      |     ~~~~~~~~~~~~~~~~~~~~
  849 |     uint32_t sig_cnt;
      |     ~~~~~~~~~~~~~~~~~
  850 |
  851 |     /* version of the srep data */
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  852 |     uint32_t srep_version;
      |     ~~~~~~~~~~~~~~~~~~~~~~
  853 |
  854 |     /* reputation for netblocks */
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  855 |     SRepCIDRTree *srepCIDR_ctx;
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
4 months ago
Victor Julien dba022d150 af-packet: use actual snaplen in bpf
Avoids setting a 0 snaplen in BPF, leading to an error.

Fixes: b8b6ed550a ("af-packet: delay setting default-packet-size for af-packet")

Ticket: #7618.
(cherry picked from commit 749ffbd06a)
4 months ago
Victor Julien c6fd3e69bc datasets: work around scan-build warning
datasets.c:493:27: warning: Dereference of null pointer [core.NullDereference]
  493 |     DEBUG_VALIDATE_BUG_ON(set->hash->config.hash_size != hashsize);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-validate.h:95:44: note: expanded from macro 'DEBUG_VALIDATE_BUG_ON'
   95 | #define DEBUG_VALIDATE_BUG_ON(exp) BUG_ON((exp))
      |                                            ^~~
./suricata-common.h:307:36: note: expanded from macro 'BUG_ON'
  307 |         #define BUG_ON(x) assert(!(x))
      |                                    ^
/usr/include/assert.h:109:7: note: expanded from macro 'assert'
  109 |     ((expr)                                                             \
      |       ^~~~
1 warning generated.

(cherry picked from commit c6fdf99cec)
4 months ago
Shivani Bhardwaj 77652d0f7a version: start development towards 7.0.10 4 months ago
Shivani Bhardwaj 76729e4266 release: 7.0.9; update changelog 4 months ago
Juliana Fajardini f6140df708 upgrade: list inspection recursion default limit
As the yaml indicated before that if no value was specified there were
no limits, and now there will be one.

(cherry picked from commit 3985b24e1b)
4 months ago