Unlike the original commit, this commit just enforces a maximum limit,
but does not expose the configuration of it.
Ticket: #5921
(cherry picked from commit 4175680a8a)
Ticket: 5926
HTTP2 continuation frames are defined in RFC 9113.
They allow header blocks to be split over multiple HTTP2 frames.
For Suricata to process correctly these header blocks, it
must do the reassembly of the payload of these HTTP2 frames.
Otherwise, we get incomplete decoding for headers names and/or
values while decoding a single frame.
Design is to add a field to the HTTP2 state, as the RFC states that
these continuation frames form a discrete unit :
> Field blocks MUST be transmitted as a contiguous sequence of frames,
> with no interleaved frames of any other type or from any other stream.
So, we do not have to duplicate this reassembly field per stream id.
Another design choice is to wait for the reassembly to be complete
before doing any decoding, to avoid quadratic complexity on partially
decoding of the data.
(cherry picked from commit aff54f29f8)
When reloading rules, respect `--set default-rule-path=...` from the
command line if set.
Previously the rule reload would always take the default-rule-path from
the configuration file, even if overrided on the command line.
Issue: #1911
(cherry picked from commit 3ea6572e22)
So that we can have multi-tenant on MacOS without stack
overflows because of the size of the structure...
Ticket: #6263.
(cherry picked from commit 92fce2fdc0)
Where clang --version was returning:
clang version 9.0.1-15+b1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Newer version like clang-10 on Debian are returning:
Debian clang version 10.0.1-8+b1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
As a result the parsing was failing to determine which llc was
available on the system.
Ticket: #6598
(cherry picked from commit 37b1595c20)
Issue: 6239
This commit moves the global variables associated with engine analysis
into the detect engine context. Doing so provides encapsulation of the
analysis components as well as thread-safe operation in a multi-tenant
(context) deployment.
(cherry picked from commit c8615bcd47)
Our tls fields not_after and not_before are actually logged as
`notafter` and `notbefore`, but were documented with the underscore.
Update the documentation, since updating the log format itself would be
a breaking change.
Task #5494
(cherry picked from commit a649a92afd)
Ticket: #6426
as per RFC 9113
":authority" MUST NOT include the deprecated userinfo subcomponent
for "http" or "https" schemed URIs.
(cherry picked from commit e3cd0d073f)
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Bug: #6420
(cherry picked from commit a240a93b69)
Issue: 4623
This commit addresses the issues reported in issue 4623 when the jump
value points at the last byte in the buffer.
(cherry picked from commit f363b99fd7)
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Bug 6380
(cherry picked from commit 7406ac0fa5)
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Bug: #6402.
(cherry picked from commit 15947f2173)
Pseudo packets originating in the flow worker do not need to leave the
flow worker. Putting those in the ThreadVars::decode_pq will make them
be evaluated by the next steps in the pipeline, but those will all
ignore pseudo packets.
Instead, this patch returns them to the packet pool, while still honoring
the IPS verdict logic.
(cherry picked from commit 3247e39f0c)
Issue: 4624
Handle negative post_offset values that jump before the buffer as though
they refer to the buffer start.
(cherry picked from commit 2bf9d0fdf9)
Ticket: #6279
If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
(cherry picked from commit c6afee64d5)
Adapted to use hash functions from master6
A bad pattern in a rule that hyperscan would fail to compile would
exit Suricata. This could happen during a rule reload as well.
In case of a untrusted ruleset, this could potentially be used to
shut down the sensor.
Commit 7d0851b0c2 already blocks the only know case, but this patch
is more defensive.
Ticket: #6195.
(cherry picked from commit 00e00254ea)
Make sure thread ctx registration happens and id remains correct
in case of reloads.
To do so, move id var into the detect ctx.
(cherry picked from commit 2cac440f7d)
Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Bug: #6044.
Bug: #6201.
(cherry picked from commit b130234b26)
When comparing IPv6 addresses based on uint32_t chunks, one needs to
apply ntohl() conversion to the individual parts, otherwise on little
endian systems individual bytes are compared in the wrong order.
Avoid this all and leverage memcmp(), it'll short circuit on the first
differing byte and its return values tells us which address sorts lower.
Bug: #6276
(cherry picked from commit 0ddc44f4c1)
If the interface and copy-iface are same for an af-packet IPS device
setting then fataly exit else it leads to a segfault in later stages.
Bug 5870
(cherry picked from commit d4dd53c95f)
Register file.name and file.magic at correct progress values.
In HTTP1, the files are (part of) the body, so make sure the file
detection logic only runs when the parser has started processing
the body.
Philippe Antoine
Jason Ish
Philippe Antoine
Victor Julien
Lukas Sismis
Eric Leblond
Jeff Lucovsky
jason taylor
Juliana Fajardini
Travis Green
Shivani Bhardwaj
Arne Welzel
Haleema Khan