The HTTP bodies (http_client_body and http_server_body/file_data) use
settings to control how much data we have before doing first inspection:
request-body-minimal-inspect-size
response-body-minimal-inspect-size
These settings default to 32k as quite some existing rules need this.
At the same time, the 'raw stream' inspection uses its own limits. By
default it inspects the data in blocks of about 2.5k. This could lead
to a situation where rules would not match.
For example, with 2 rules like this:
content:"abc"; content:"data="; http_client_body; depth:5; sid:1;
content:"xyz"; sid:2;
Sid 1 would only be inspected when the POST body reached the 32k limit
or when it was complete. Observed case shows the POST body to be 18k.
Sid 2 is inspected as soon as the 2.5k limit is reached, and then again
for each 2.5k increment. This moves the raw stream tracker forward.
So by the time sid 1 is inspected, some 18/19k into the stream, the
raw stream tracker is actually already moved forward for approximately
17.5k, this leads to the stream match of sid 1 possibly not matching.
Since the body match is at the start of the buffer, it makes sense
that the body and stream are inspected together.
The body inspection uses a tracker 'body_inspected', that keeps track
of how far into the body both MPM and per signature inspection has
moved.
This patch updates the logic in 2 ways:
1. it triggers earlier HTTP body inspection, which is matched to the
stream inspection. When the detection engine finds it has stream
data available for inspection, it passes the new 'STREAM_FLUSH'
flag to the HTTP body inspection code. Which will then do an
early inspection, even if still before the min inspect size.
2. to still somewhat adhere to the min inspect size, the body
tracker is not updated until the min inspect size is reached.
This will lead to some re-evaluation of the same body data.
If raw stream reassembly is disabled, this 'STREAM_FLUSH' flag is
never set, and the old behavior is used.
Bug #2522.
Trigger app layer reassembly in both directions as soon as we've set
the TCP state to closed.
In IDS mode, if a toserver packet would close the state, the app layer
would not get updated until the next toclient packet. However, in
detection, the raw stream inspection would already use all available
stream data in detection and move the 'raw stream progress' tracker
forward. When in later (a) packet(s) the app layer was updated and
inspection ran on the app layer, the stream progress was already
moved too far forward. This would lead to signatures that matched
on both stream and app layer to not match.
By triggering the app layer reassembly as soon as the TCP state is
set to closed, the inspection as both the stream and app layer data
available at the same time so these rules can match.
Bug: #2570
Bug: #2554
If a stream is async we see only on side of the traffic. This would
lead to the flow engine not flagging packets as 'established' even
if the flow state was in fact established. The flow was tagged as
such by the TCP engine.
This patch considers the flow state for setting the packet flag.
Bug #2491.
When a file in TOSERVER direction is being stored and
libhtp or stream depth limit is reached,
it will be closed by HTPCallbackRequest without setting
any flags so the file state will be set to CLOSED
instead of TRUNCATED.
When an app-layer parser is enabled, it could set its
own stream_depth value calling the API AppLayerParserSetStreamDepth.
Then, the function AppLayerParserPostStreamSetup will replace
the stream_depth value already set with stream_config.reassembly_depth.
To avoid overwriting, in AppLayerParserSetStreamDepth API a flag
will be set internally to specify that a value is already set.
Add device to tenant mapping support:
mappings:
- device: ens5f0
tenant-id: 1
- device: ens5f1
tenant-id: 23
Implemented by assigning the tenant id to the 'livedev', which means
it's only supported for capture methods that use the livedev API.
It's also currently not supported for IPS. In a case like 'eth0 -> eth1'
it's unclear which tenant should be used for the return traffic in a
flow, where the incoming device is 'eth1'.
Change the decode handler signature to increase the size of its decode
handler, from uint16 to uint32. This is necessary to let suricata use
interfaces with mtu > 65535 (ex: lo interface has default size 65536).
It's necessary to change several primitive for Packet manipulation, to
unify the parameter "packet length" whenever we are before IP decoding.
Add tests before calling DecodeIPVX function to avoid a possible
integer overflow over the len parameter.
fix bug in fileext and filename preventing negated match to work
correctly. Previously, negated fileext (such as !"php") would cause a
match anyway on files that have extension php, as the last if would not
be accessed.
Using the same workflow as detect-filemagic we remove the final
isolated if and set it as a branch of the previous if.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
4474889667 did break the logic of IPS
mode detection for layer 2 IPS mode like Netmap and AF_PACKET. This
pach fixes the issue by calling the IPS detection mode after
complete registration of the interfaces.
Reported-by: Alexander Gozman <a.gozman@securitycode.ru>
If a signature didn't explicitly specified 'dcerpc' or 'smb' as the
app proto, false positives on other traffic could happen. This was
caused by the sig not having a app_proto set. This isn't set as the
rule is supposed to match against either ALPROTO_DCERPC or ALPROTO_SMB.
To avoid adding runtime costs for checking for both protocols, this
patch adds a new flag for DCERPC in the 'mask' logic. The flag is set
on the sig if dce_* keywords are present and set on the packet if the
flow's app proto is either ALPROTO_DCERPC or ALPROTO_SMB.
Bug #2559
Reported-by: Jason Taylor
Add TX creation for NFS4 transactions. Start with the 'REMOVE' procedure.
Start on logging all procs. In NFS4 COMPOUND records there are multiple
procedures. One of them can be considered the 'main' procedure, with others
as supporting utility. This patch adds the first step in supporting to
track those in the TX for logging and inspection.
Victor Julien
Eric Leblond
Giuseppe Longo
Maurizio Abba