To avoid possible upgrade issues, allow higher defaults than in the
master branch. Add some upgrade guidance and a note that defaults will
probably be further reduced.
Ticket: 7615
Avoids signatures setting extreme hash sizes, which would lead to very
high memory use.
Default to allowing:
- 65536 per dataset
- 16777216 total
To override these built-in defaults:
```yaml
datasets:
# Limits for per rule dataset instances to avoid rules using too many
# resources.
limits:
# Max value for per dataset `hashsize` setting
#single-hashsize: 65536
# Max combined hashsize values for all datasets.
#total-hashsizes: 16777216
```
(cherry picked from commit a7713db709)
Make hashsize default local to dataset code, instead of relying on the
thash code.
Use the same default value as before.
(cherry picked from commit d32a39ca4b)
AF_PACKET needs more information about its configuration before we can
set the default packet size, so on startup, leave unset in suricata.c
if in AF_PACKET mode.
If defrag is enabled, use a default packet size of 9k for tpacket-v2.
This can still lead to truncation events, then the user can increase
their 'default-packet-size'.
Tpacket-v3 does not need an increased packet size as it will handle
any size of packet that is smaller than the configured block size
which now has a default of 128k.
9k for the snap is somewhat arbitrary but is large enough for the
common 9000 jumbo frame plus some extra headers including tpacket
headers.
Ticket: #7458
(cherry picked from commit b8b6ed550a)
If using tpacket-v3 and defrag, warn if the block size is not large
enough for a fully defragmented packet.
Ticket: #7458
(cherry picked from commit 9f96975d55)
If using tpacket-v2, defrag and a user provided v2-block-size, warn if
the block size is not large enough to hold one fully defragmented
packet.
Ticket: #7458
(cherry picked from commit 320ef7b617)
With the change of the default tpacket-v2 block size from 32k to 128k,
allow it to be configurable for users who may want to make it larger,
or revert it back to the pre 7.0.9 default of 32k.
Ticket: #7458
(cherry picked from commit 5871c6458c)
Increase the default block size from 32k to 128k. This allows for a
fully defragmented packet to fit in the buffer.
Ticket: #7458
(cherry picked from commit c342b054f4)
AF_PACKET defrag should not be used for inline modes. Its possible that
a packet received could be larger than can be set when defrag is
enabled, so warn if disabled for inline use.
Likewise, warn if defrag is disabled for IDS use, or non-inline mode.
Ticket: #7458
(cherry picked from commit 808502d5ca)
If cluster-type was not set we default to "cluster_flow" with defrag
always on. Instead check for defrag value and disable defrag if disabled
by the user.
Ticket: #7458
(cherry picked from commit 25d0fba912)
Ticket: 7613
Avoids potential large per-thread memory allocation. A buffer with the
size of the largest decode_base64 buffer size setting would be allocated
per thread. As this was a u32, it could mean a per-thread 4GiB memory
allocation.
64KiB was already the built-in default for cases where bytes size wasn't
specified.
(cherry picked from commit 32d0bd2bbb)
Ticket: 7526
The usage of negated pcre, followed by other relative payload
content keywords could lead to an infinite loop.
This is because regular (not negated) pcre can test multiple
occurences, but negated pcre should be tried only once.
(cherry picked from commit b14c67cbdf)
Since forever (1578ef1e3e) a valid RST
would update the internal `last_ack` representation to include all
unack'd data. This was originally done to make sure the unACK'd data was
inspected/processed at flow timeout.
It was observed however, that if GAPs existed in this unACK'd data, a
GAP could be reported in the stats and a GAP event would be raised. This
doesn't make sense, as missing segments in the unACK'd part of the
stream are completely normal. Segments simply do not all arrive in
order.
It turns out that the original behavior of updating `last_ack` to
include all unACK'd data is no longer needed.
For raw stream inspection, the detection engine will already include the
unACK'd data on flow end.
For app-layer updates the unACK'd data is often harmful, as the data
often has GAPs. Parser like the http parser would report these GAPs and
could also get confused about the post-GAP data being a new transaction
including a file. This lead to many reported errors and fantom txs and
files.
Since the GAP detection uses `last_ack` to determine GAPs, not moving
`last_ack` addresses the GAP false positives.
Ticket: #7422.
(cherry picked from commit bd1b9f6229)
Ticket: 7552
f->sgh_toserver may be NULL but because FLOW_SGH_TOSERVER is unset
and thus, we want to delay cleanup until detection has really been
run with the right signature group head.
This may happen for a rule using
`alert tcp any any -> any any` and
a app-layer keyword to client
with a app-layer supporting both udp and tcp
with stream.midstream=true
and with the first packet of a flow being a server response
In this case, we swap the flow and reset its signature group heads
(cherry picked from commit d8ddef4c14)
Additional fix in rfb unit test which moved to SV in suricata 8
Ticket: 7552
When we use midstream, and the first packet we see of a flow is
a response from server, and we want to match on some signature
to client :
- we had first set sgh_toserver/FLOW_SGH_TOSERVER as we first
thought this was a packet to server
- we then swap/reverse the flow, so sgh_toclient becomes sgh_toserver
but it contains signatures to server and cannot match our
to_client signature
The detect engine with DetectRunSetup will set again the
signatures group heads properly
(cherry picked from commit d74bc774b7)
Ticket: 7577
When HTTP1 post multipart handles a small file, it will call
HTPFileClose with some data
This data needs to be appended to the streaming buffer for usage
by file.data keyword even if we do not end up storing the file
(cherry picked from commit f68e2f5537)
To optimize detection, and logging, to avoid going through
all the live transactions when only a few were modified.
Two boolean fields are added to the tx data: updated_tc and ts
The app-layer parsers are now responsible to set these when
needed, and the logging and detection uses them to skip
transactions that were not updated.
There may some more optimization remaining by when we set
both updated_tc and updated_ts in functions returning
a mutable transaction, by checking if all the callers
are called in one direction only (request or response)
Ticket: 7087
(cherry picked from commit b02557ac7d)
Ticket: 7556
See RFC 9000 section 17.2.5.2 :
After the client has received and processed an Initial
or Retry packet from the server,
it MUST discard any subsequent Retry packets that it receives.
(cherry picked from commit 726de5520f)
Ticket: 7556
To do so, we need to add 2 buffers (one for each direction)
to the QuicState structure, so that on parsing the second packet
with hello/crypto fragment, we still have the data of the first
hello/crypto fragment.
Use a hardcoded limit so that these buffers cannot grow indefinitely
and set an event when reaching the limit
(cherry picked from commit f295cc059d)
If not all data is ACK'd during the FIN session shutdown, the last calls
to the parser can be with a non-NULL data pointer, but a input length of
0. This wasn't considered by the EOF check, which then lead to it being
seen as an error. No event was raised, but the tls error stats were
incremented.
Bug: #7554.
(cherry picked from commit 471bde4426)
Describe the change of behavior between the stream.checksum-validation
setting and checksum-based rule keywords.
(cherry picked from commit cfbf8fda94)
Issue: 7467
Stream checksum validation no longer has a side effect of setting
PKT_IGNORE_CHECKSUM and thus, no longer affects csum keyword checks.
(cherry picked from commit 758da982f0)
The `is_version_char` function incorrectly allowed characters that are not
part of the valid SIP version "SIP/2.0".
For instance, 'HTTP/1.1' was mistakenly accepted as a valid SIP version,
although it's not.
This commit fixes the issue by updating the condition to strictly
check for the correct version string.
cherry-picked from commit 69f841c998
Ticket: 7495
We want to finish also if we tested all the expected protocols
in mask, or if we tested even more.
There can be one more protocol coming from pe0, which can be
the protocol already found in the other direction.
(cherry picked from commit b5094b00b6)
Issue: 7466
This commit releases the memory for the flow variable "key" when
the flow variable is of type string. The key is allocated in the Lua
extension logic.
(cherry picked from commit 2d9df5a1ae)
The fix for issue 7447 introduced an error with threaded eve output.
The changes that were committed for that issue mishandled the return
value when a file is being opened for the 2nd or higher time.
Instead of returning the existing file context, null was returned.
(cherry picked from commit 1d996c5aed)
Issue: 7447
When the output file can't be opened, 2 error messages are displayed
for the same problem. The second message doesn't add value and lacks
context (error reason, e.g., "Permission denied").
Retaining the second message as a debug message.
Without this commit:
Error: logopenfile: Error opening file: "/home/jlucovsky/src/jal/suricata-verify/tests/bug-5198/output/noperms/eve.1.json": Permission denied [SCLogOpenFileFp:util-logopenfile.c:428]
Error: logopenfile: Unable to open slot 1 for file /home/jlucovsky/src/jal/suricata-verify/tests/bug-5198/output/noperms/eve.json [LogFileEnsureExists:util-logopenfile.c:737]
Error: runmodes: unable to initialize sub-module eve-log.stats [RunModeInitializeEveOutput:runmodes.c:692]
With commit:
Error: logopenfile: Error opening file: "/home/jlucovsky/src/jal/suricata-verify/tests/bug-5198/output/noperms/eve.1.json": Permission denied [SCLogOpenFileFp:util-logopenfile.c:428]
Error: runmodes: unable to initialize sub-module eve-log.stats [RunModeInitializeEveOutput:runmodes.c:692]
(cherry picked from commit d853972c74)
Shivani Bhardwaj
Juliana Fajardini
Victor Julien
Philippe Antoine
Jason Ish
Jeff Lucovsky
Giuseppe Longo