Update HTTP parser to set the min inspect depth per transaction. This
allows for signatures to have their fast_pattern in the HTTP body,
while still being able to inspect the raw stream reliably with it.
The inspect depth is set per transaction as it:
- depends on the per personality config for min inspect size
- is set to the size of the actual body if it is smaller
After the initial inspection is done, it is set to 0 which disables
the feature for the rest of the transaction.
This removes the rescanning flush logic in commit
7e004f52c6 and provides an alternative
fix for bug #2522. The old approach caused too much rescanning of
HTTP body data leading to a performance degradation.
Bug #2522
Some rules need to inspect both raw stream data and higher level
buffers together. When this higher level buffer is a streaming
buffer itself, the risk of mismatch exists.
This patch allows an app-layer parser to set a 'min inspect depth'.
The value is used by the stream engine to keep at least this
depth worth of data, so that the detection engine can request
all of it for inspection.
For rules that have the SIG_FLAG_FLUSH flag set, data is inspected
not from offset raw_progress, but from raw_progress minus
min_inspect_depth.
At this time this is only used for sigs that have their fast_pattern
in a HTTP body and have raw stream match as well.
In the case we exceed the number of simultaneously open
file we can reach a state were we will not close the file
after writing.
Thanks to Steve Grubb <sgrubb@redhat.com> for the analysis.
If the global detect.prefilter.default setting is not "auto", it is
wasteful to run each prefilter setup routine. This patch tracks which
of the engines have been explicitly enabled in the rules and only
runs those.
When prefilter is not enabled globally, it is still possible to
enable it per signature. This was broken however, as the setup
code would never be called.
This commit always call the setup code and lets that sort out
which signatures (if any) to enable prefiltering for.
If prefilter is used on a content keyword, it acts as a simple
fast_pattern statement. This was broken because the SIG_FLAG_PREFILTER
flag bypasses MPM for a sig. This commits fixes this by not setting
the flag when it should act as fast_pattern.
The HTTP bodies (http_client_body and http_server_body/file_data) use
settings to control how much data we have before doing first inspection:
request-body-minimal-inspect-size
response-body-minimal-inspect-size
These settings default to 32k as quite some existing rules need this.
At the same time, the 'raw stream' inspection uses its own limits. By
default it inspects the data in blocks of about 2.5k. This could lead
to a situation where rules would not match.
For example, with 2 rules like this:
content:"abc"; content:"data="; http_client_body; depth:5; sid:1;
content:"xyz"; sid:2;
Sid 1 would only be inspected when the POST body reached the 32k limit
or when it was complete. Observed case shows the POST body to be 18k.
Sid 2 is inspected as soon as the 2.5k limit is reached, and then again
for each 2.5k increment. This moves the raw stream tracker forward.
So by the time sid 1 is inspected, some 18/19k into the stream, the
raw stream tracker is actually already moved forward for approximately
17.5k, this leads to the stream match of sid 1 possibly not matching.
Since the body match is at the start of the buffer, it makes sense
that the body and stream are inspected together.
The body inspection uses a tracker 'body_inspected', that keeps track
of how far into the body both MPM and per signature inspection has
moved.
This patch updates the logic in 2 ways:
1. it triggers earlier HTTP body inspection, which is matched to the
stream inspection. When the detection engine finds it has stream
data available for inspection, it passes the new 'STREAM_FLUSH'
flag to the HTTP body inspection code. Which will then do an
early inspection, even if still before the min inspect size.
2. to still somewhat adhere to the min inspect size, the body
tracker is not updated until the min inspect size is reached.
This will lead to some re-evaluation of the same body data.
If raw stream reassembly is disabled, this 'STREAM_FLUSH' flag is
never set, and the old behavior is used.
Bug #2522.
Trigger app layer reassembly in both directions as soon as we've set
the TCP state to closed.
In IDS mode, if a toserver packet would close the state, the app layer
would not get updated until the next toclient packet. However, in
detection, the raw stream inspection would already use all available
stream data in detection and move the 'raw stream progress' tracker
forward. When in later (a) packet(s) the app layer was updated and
inspection ran on the app layer, the stream progress was already
moved too far forward. This would lead to signatures that matched
on both stream and app layer to not match.
By triggering the app layer reassembly as soon as the TCP state is
set to closed, the inspection as both the stream and app layer data
available at the same time so these rules can match.
Bug: #2570
Bug: #2554
A bad packet (rejected by stream engine) could still trigger a file
prune, even though (most of the) detection wouldn't happen for the
packet. The next valid packet would then not be able to match on the
file, as it was already freed.
This patch uses the same logic before file pruning as in the detect
engine.
Bug: 2576
If the data file can't be found in the default location, which
normally is 'default-rule-path', try to see if it can be found
in the path of the rule file that references it.
This makes QA much easier.
If a stream is async we see only on side of the traffic. This would
lead to the flow engine not flagging packets as 'established' even
if the flow state was in fact established. The flow was tagged as
such by the TCP engine.
This patch considers the flow state for setting the packet flag.
Bug #2491.
The stream depth computation was partly done with the stream_config
depth instead of using the value in the TCP session. As a result,
some configuration were resulting in abnormal behavior.
In particular, when stream depth was 0 and the file store depth was
not 0, Suricata was stopping the streaming on the flow as soon as
the filestore was started.
Reported-by: Pascal Delalande <pdl35@free.fr>
fix bug in fileext and filename preventing negated match to work
correctly. Previously, negated fileext (such as !"php") would cause a
match anyway on files that have extension php, as the last if would not
be accessed.
Using the same workflow as detect-filemagic we remove the final
isolated if and set it as a branch of the previous if.
CID 1428797 (#1 of 1): Unchecked return value (CHECKED_RETURN)
check_return: Calling HashTableAdd without checking return value
(as is done elsewhere 5 out of 6 times).
In case of a valid RST on a SYN, the state is switched to 'TCP_CLOSED'.
However, the target of the RST may not have received it, or may not
have accepted it. Also, the RST may have been injected, so the supposed
sender may not actually be aware of the RST that was sent in it's name.
In this case the previous behavior was to switch the state to CLOSED and
accept no further TCP updates or stream reassembly.
This patch changes this. It still switches the state to CLOSED, as this
is by far the most likely to be correct. However, it will reconsider
the state if the receiver continues to talk.
To do this on each state change the previous state will be recorded in
TcpSession::pstate. If a non-RST packet is received after a RST, this
TcpSession::pstate is used to try to continue the conversation.
If the (supposed) sender of the RST is also continueing the conversation
as normal, it's highly likely it didn't send the RST. In this case
a stream event is generated.
Ticket: #2501
Reported-By: Kirill Shipulin
Victor Julien
Eric Leblond
Maurizio Abba
Jason Ish