Hyperscan MPM can cache the compiled contexts to files.
This however grows as rulesets change and leads to bloating
the system. This addition prunes the stale cache files based
on their modified file timestamp.
Part of this work incorporates new model for MPM cache stats
to split it out from the cache save function and aggregate
cache-related stats in one place (newly added pruning).
Ticket: 7893
(cherry picked from commit 15c83be61a)
hs: suppress TOCTOU stat use
To explain a bit more the TOCTOU issue found, we can consider
a case where Suricata starts to prune, yet externally somebody also
starts erasing cache files.
Right after Suricata checks the file age with the stat function,
somebody may delete or update the file of our interest.
Suricata aging decision doesn't reflect the actual state of the file.
This commit additionally adds a check for noent failure of the unlink operation
(considered as a success). The code can still delete a file that is recently
updated but was considered stale.
In the documentation-following deployments this should not happen anyway as
one cache folder should only be used by a single Suricata instance (and within
Suricata instance only one thread handles cache eviction).
Additionally, the `stat` and `unlink` command are immediatelly followed, making
this scenario extra unlikely.
Additional comment in the code explains problems of using fstat and potential
issues on Windows.
Ticket: 8244
(cherry picked from commit 0fe0390a2f)
hs/cache: cleaner and more detailed output
Reduce logging level of a minor informational message.
Split tracking of pruning by age and by version and log those
separately, where the logging only appears if something has been
removed.
Ticket: 8323
(cherry picked from commit 569ba3d26f)
hs: remove redundant file handle in HSLoadCache
HSLoadCache opened the cache file but never used the resulting handle
for reading. The actual read was done by HSReadStream which opened
the same file independently.
Removed the unused fopen/fclose pair and flattened the control flow.
Ticket: 8326
(cherry picked from commit d754b28717)
hs: use binary mode for cache file I/O
HSSaveCache wrote serialized Hyperscan databases using text mode ("w")
while HSReadStream already read them with binary mode ("rb").
Matched file reading modes to the binary format and simplified
write-size check.
Ticket: 8326
(cherry picked from commit 0cdc77b707)
hs: warn about the same cache directory
This is especially relevant for multi-instance simultaneous setups
as we might risk read/write races.
(cherry picked from commit 56c1552c3e)
hs: validate cached database against current HS installation
After deserializing a cached Hyperscan database, verify that its
version, CPU features, and mode match the current Hyperscan
installation by comparing hs_database_info output against a
reference database. Reject loading incompatible caches.
Ticket: 8326
(cherry picked from commit 2e7b12dda4)
hs: include HS platform info in cache file hash
Hash Hyperscan installation info (version, CPU features, mode)
into the cache filename. A Hyperscan upgrade or platform change
would now produce a different filename, so stale caches from an
older installation are never opened.
Ticket: 8326
(cherry picked from commit d640719413)
hs: address coverity warning in a reference string
Move the locking mechanism outside of the getter function and hold the
lock until the reference string is no longer reused.
** CID 1682023: Concurrent data access violations (MISSING_LOCK)
/src/util-mpm-hs-cache.c: 139 in HSGetReferenceDbInfo()
(cherry picked from commit 6ec9e5c957)
Docs were claiming greater than, less than, etc. support even though this was never supported.
Fixes: 51be8f0238 ("doc/dcerpc: add proto keywords")
Ticket: #8330.
(cherry picked from commit 2480f8c038)
The Deprecation note on Syslog was partly conveying the opposite
message, and we were missing a deprecation notice on the Syslog output
section.
(cherry picked from commit 926fde859e)
Adding the directory "install" to EXTRA_DIST, actually triggers make
to run "make install", which is not what we want. Instead, avoid this
magic keyword and list the files in the install directory
individually.
If the user doesn't have permission to install files to the prefix,
like "/usr", then "make dist" can fail. Worse, even they do have
permission to write into the prefix, a "make dist" will install files
there when it shouldn't.
Ticket: #8279
(cherry picked from commit 2defd5ae5a)
Adds a chapter indicating what are the main steps when adding exception
policies, how is it possible to extend them, as well as main aspects and
files to consider when doing so.
Task #5612
(cherry picked from commit 48b15ac152)
Redmine ticket: #8261
According to [1], the within pointer (if combined with distance)
includes the distance pointer, which is not clearly visible in the
graphic.
Fixed this in a new graphic by some GIMP arts.
PS: Special thanks to one of our team members Annika C. for initially
spotting this!
[1] https://forum.suricata.io/t/is-within-affected-by-distance/1688
(cherry picked from commit 69eb567dea)
16384 is used as the max, but a configuration parameter has been
provided. The reason for setting an upper bound is that bit flags can
create a memory amplification as we parse them into individual data
structures.
Ticket: #8181
(cherry picked from commit 3a32bb5743)
Lower the number of unreplied requests from 500 to 32 to consider a
flood. At the very least this is an anomaly given the DNP3 spec mentions
that DNP3 should only have one outstanding request at a time, with an
exception for unsolicited responses, so in practice no more than 2
should be seen.
Additionally make this value configurable by introducing the max-tx
parameter.
Ticket: #8181
(cherry picked from commit a16f087b93)
In corner cases, we assume that a midstream exception policy could be
triggered by a prior exception policy in effect. Explain this in the
docs.
Task #5830
(cherry picked from commit 0ca874b678)
Ticket: 3220
DetectSslVersionMatch did not handle properly negation.
It could never match on a signatrue with ssl_version: !tls1.3
That is because, if we had such a signature and network traffic
with tls1.1, we were looking into DetectSslVersionData field
for tls1.1, which was not set, instead of looking at field
for tls1.3 which was set with negated flag.
Previous DetectSslVersionData was holding redundant information.
It did not need to have it for each ssl version, but just globally.
Also, it did not need to hold the version as a value in the array,
as it was redundant with the index of the array.
(cherry picked from commit c93e69830a)
Issue: 8035
The `luaxform` transform doesn't support the `init` function. This
commit removes that from the example and clarifies how functions in the
Lua script are used.
(cherry picked from commit b02d9bb4f1)
Issue: 8007
Highlight the change to how ether_type values are displayed. Previously,
they were displayed in network order as a decimal value.
They are now displayed in host order as a decimal value.
(cherry picked from commit 16d124cfda)
Following commit 5379b52af2
rules that use multiple times the keyword tls.cert_subject
will result in
Warning: detect: duplicate instance for tls.cert_subject
These rules likely meant to use a multi-buffer which is not the
case for tls.cert_subject (even if it was documented so).
Ticket: 7890
This is put in a new section of upgrade notes for
upgrading to 8.0.1
Added a page that explains how rules are prioritized by Suri, as well
as what main different types of inspection happen and what elements are
involved when ordering rules.
Task #5449
Explain briefly the internals of inspection of raw data in the following order:
- Stream Engine
- Stream reassembly
- Role of Detection Engine and Applayer Parsers
- High level communication between Stream and Detection Engine
- Relevant suricata.yaml settings
alongwith some diagrams.
Ticket 4351
Lukas Sismis
Philippe Antoine
Jason Ish
Victor Julien
Sven Cuyt
Juliana Fajardini
Andreas Dolp
Shivani Bhardwaj
Jeff Lucovsky
Shivani Bhardwaj