In the case we exceed the number of simultaneously open
file we can reach a state were we will not close the file
after writing.
Thanks to Steve Grubb <sgrubb@redhat.com> for the analysis.
Fix mpm progress being updated by irrelevant engines. Esp in the
case of file_data engines, signature can contain multiple versions
of the same engine, registered for different 'progress' values.
This would lead to signatures being considered 'can't match' even
in cases where they clearly could still match.
Only consider those progress values that apply to the protocol in
use.
If the data file can't be found in the default location, which
normally is 'default-rule-path', try to see if it can be found
in the path of the rule file that references it.
This makes QA much easier.
If the global detect.prefilter.default setting is not "auto", it is
wasteful to run each prefilter setup routine. This patch tracks which
of the engines have been explicitly enabled in the rules and only
runs those.
When prefilter is not enabled globally, it is still possible to
enable it per signature. This was broken however, as the setup
code would never be called.
This commit always call the setup code and lets that sort out
which signatures (if any) to enable prefiltering for.
If prefilter is used on a content keyword, it acts as a simple
fast_pattern statement. This was broken because the SIG_FLAG_PREFILTER
flag bypasses MPM for a sig. This commits fixes this by not setting
the flag when it should act as fast_pattern.
The HTTP bodies (http_client_body and http_server_body/file_data) use
settings to control how much data we have before doing first inspection:
request-body-minimal-inspect-size
response-body-minimal-inspect-size
These settings default to 32k as quite some existing rules need this.
At the same time, the 'raw stream' inspection uses its own limits. By
default it inspects the data in blocks of about 2.5k. This could lead
to a situation where rules would not match.
For example, with 2 rules like this:
content:"abc"; content:"data="; http_client_body; depth:5; sid:1;
content:"xyz"; sid:2;
Sid 1 would only be inspected when the POST body reached the 32k limit
or when it was complete. Observed case shows the POST body to be 18k.
Sid 2 is inspected as soon as the 2.5k limit is reached, and then again
for each 2.5k increment. This moves the raw stream tracker forward.
So by the time sid 1 is inspected, some 18/19k into the stream, the
raw stream tracker is actually already moved forward for approximately
17.5k, this leads to the stream match of sid 1 possibly not matching.
Since the body match is at the start of the buffer, it makes sense
that the body and stream are inspected together.
The body inspection uses a tracker 'body_inspected', that keeps track
of how far into the body both MPM and per signature inspection has
moved.
This patch updates the logic in 2 ways:
1. it triggers earlier HTTP body inspection, which is matched to the
stream inspection. When the detection engine finds it has stream
data available for inspection, it passes the new 'STREAM_FLUSH'
flag to the HTTP body inspection code. Which will then do an
early inspection, even if still before the min inspect size.
2. to still somewhat adhere to the min inspect size, the body
tracker is not updated until the min inspect size is reached.
This will lead to some re-evaluation of the same body data.
If raw stream reassembly is disabled, this 'STREAM_FLUSH' flag is
never set, and the old behavior is used.
Bug #2522.
Trigger app layer reassembly in both directions as soon as we've set
the TCP state to closed.
In IDS mode, if a toserver packet would close the state, the app layer
would not get updated until the next toclient packet. However, in
detection, the raw stream inspection would already use all available
stream data in detection and move the 'raw stream progress' tracker
forward. When in later (a) packet(s) the app layer was updated and
inspection ran on the app layer, the stream progress was already
moved too far forward. This would lead to signatures that matched
on both stream and app layer to not match.
By triggering the app layer reassembly as soon as the TCP state is
set to closed, the inspection as both the stream and app layer data
available at the same time so these rules can match.
Bug: #2570
Bug: #2554
If a stream is async we see only on side of the traffic. This would
lead to the flow engine not flagging packets as 'established' even
if the flow state was in fact established. The flow was tagged as
such by the TCP engine.
This patch considers the flow state for setting the packet flag.
Bug #2491.
Victor Julien
Mats Klepsland
Eric Leblond
Jason Ish
jason taylor
Maurizio Abba
Konstantin Klinger