Commit Graph

68 Commits (suricata-3.2.1)

Author SHA1 Message Date
Sascha Steinbiss 5e96977983 mpm-ac: fix integer overflow on allocation
The size of a memory buffer to be allocated was kept in a signed int
instead of a size_t, leading to an overflow when large lists of long
and diverse patterns cause the amount of AC states to blow up (>2GB).
Fixes Redmine issues #1827 and #1843.

Signed-off-by: Sascha Steinbiss <sascha@steinbiss.name>
9 years ago
Victor Julien bb0cd0e883 prefilter: rename PatternMatcherQueue datatype
In preparation of the introduction of more general purpose prefilter
engines, rename PatternMatcherQueue to PrefilterRuleStore. The new
engines will fill this structure a similar way to the current mpm
prefilters.
9 years ago
Victor Julien 4c0ab681f2 mpm: remove Cleanup API call
It's unused by all of the implementations.
9 years ago
Victor Julien 6286e70555 ac: allow use of 31bits of pid space instead of 16 9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
10 years ago
Victor Julien 9b6e292a28 mpm: remove unused max pattern len field 10 years ago
Victor Julien 9b3d4f7e24 mpm: unify & localize mpm pattern (id) handling
So far, the patterns as passed to the mpm's would use global id's that
were shared among all buffers, directions. This would lead to a fairly
large pattern id space. As the mpm algo's use the pattern id's to
prevent duplicate matching through a pattern id based bitarray,
shrinking this space will optimize performance.

This patch implements this. It sets a flag before adding the pattern
to the mpm ctx, instructing the mpm to ignore the provided pid and
handle pids management itself. This leads to a shrinking of the
bitarray size.

This is made possible by the previous work that removes the pid logic
from the code.

Next to this, this patch moves the pattern setup stage to common util
functions. This avoids code duplication.

Update ac, ac-bs and ac-ks to use this.
10 years ago
Victor Julien fa885e1d85 mpm: remove pattern id logic 10 years ago
Victor Julien 4edb03ab9d mpm: ac use internal pattern id tracking 10 years ago
Victor Julien e48d745ed7 mpm: constify search func args 10 years ago
Victor Julien 102a82fc7b detect: use mpm store for app layer mpms
Rework app-layer mpm setup and registration to make this possible.
10 years ago
Victor Julien 14d9ce7b2e detect/mpm: remove unused max_id param from API 10 years ago
Victor Julien c0b16fa2bb detect: allow for more than 64k mpm rules 10 years ago
Victor Julien 4e98a3e530 AC: fix memory leak 11 years ago
Ken Steele 736ac6a459 Use SigIntId as the type for storing signature IDs (Internal)
Previously using uint32_t, but SigIntId is currently uint16_t, so arrays
will take less memory.
11 years ago
Ken Steele 77269fbb2c Fix missing use of MpmAddPid()
Found by Victor using ASAN. One place was not checking to resize the
pid array before adding a new PID.
11 years ago
Ken Steele ab8b1158b0 Dynamically resize pattern id array as needed
Rather than creating the array of size maxpatid, dynamically resize as needed.
This also handles the case where duplicate pid are added to the array.

Also fix error in bitarray allocation (local version) to always use bitarray_size.
11 years ago
Ken Steele 104a903478 Dynamically resize pmq->rule_id_array
Rather than statically allocate 64K entries in every rule_id_array,
increase the size only when needed. Created a new function MpmAddSids()
to check the size before adding the new sids. If the array is not large
enough, it calls MpmAddSidsResize() that calls realloc and does error
checking. If the realloc fails, it prints an error and drops the new sids
on the floor, which seems better than exiting Suricata.

The size is increased to (current_size + new_count) * 2. This handles the
case where new_count > current_size, which would not be handled by simply
using current_size * 2. It should also be faster than simply reallocing to
current_size + new_count, which would then require another realloc for each
new addition.
11 years ago
Ken Steele d03f124445 Implement MPM opt for b2g, b3g, wumanber
Found problems in b2gm and b2gc, so those are removed.
11 years ago
Victor Julien 29074af9a6 AC: use local bit array
Use a local pattern bit array to making sure we don't match more than
once, in addition to the pmq bitarray that is still used for results
validation higher up in the rule matching process.

Why: pmq->pattern_id_bitarray is currently sometimes used in a
'stateful' way, meaning that for a single packet we run multiple
MPM's on the same pmq w/o resetting it.

The new bitarray is used to determine wherther we need to append the
patterns associated 'sids' list to the pmq rule_id_array.

It has been observed that MPM1 matches for PAT1, and MPM2 matches for
PAT1 as well. However, in MPM1 PAT1 doesn't have the same sids list.
In this case MPM2 would not add it's sids to the list, leading to missed
detection.
11 years ago
Victor Julien e49d0a5924 MPM: build sid list from MPM matches
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
11 years ago
Anoop Saldanha b334b8a6e9 CUDA: Update the inspection engine to inform the cuda module that it
doesn't need the gpu results and to release the packet for the next run.

Previously the inspection engine wouldn't inform the cuda module, if it
didn't need the results.  As a consequence, when the packet is next taken
for re-use, and if the packet is still being processed by the cuda module,
the engine would wait till the cuda module frees the packet.

This commits updates this functionality to inform the cuda module to
release the packet for the afore-mentioned case.
11 years ago
Victor Julien 94571c5dd2 AC: shrink output table after initialization 11 years ago
Victor Julien 04c9db398e AC: reduce realloc for new states
Don't realloc per state add, but grow by larger blocks per realloc.
11 years ago
Ken Steele f2e777e3a5 Fix lowercase table initialization (bug 1221)
The for loop needed to check for < 256, not < 255.
11 years ago
Ken Steele 970f22c752 Move memcpy_lower() into new util-memcpy.h
Remove local copies from each MPM file and use include file instead.
Might be better to also add util-memcpy.c rather than inlining it each time,
to get smaller code, since only seems to be used at initialization.
12 years ago
Ken Steele 6b1517c0b8 Remove case_state usage
The case_state in MPMs was just to track when a pid could have no-case and
case-sensitive matches for the same PID. Now that can't happen after fixing
bug 1110, so remove the code and storage for case_state.
12 years ago
Ken Steele c41041a9c7 When assigning Pattern IDs pids, check Case flags
This fixes bug 1110. When assigning PIDs, use the NO_CASE flag when comparing
for duplicates. The state of the flag must be the same, but also use the same
type of comparisons when checking for duplicates.

Previously, "foo":CS would match with "foo":CI when it should not.
and "foo":CI would not match "FoO":CI when it should. Both of those
cases are fixed with this change.

This then allows simplifying the use of pid in MPMs because now if they
pids match, then so do the flags, so checking the flags is not required.
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Eric Leblond 1f07d1521e Fix realloc error handling
This patch is fixing realloc error handling. In case of a realloc
failure, it free the initial memory and continue existing error
handling.

The patch has been obtained via the following semantic patch and
a bit oh hand editing:

@@
expression x, E;
identifier f;
@@

f(...)
{
+ void *ptmp;
<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (x == NULL)
+ if (ptmp == NULL)
{
+ SCFree(x);
+ x = NULL;
...
- }
+ } else {
+     x = ptmp;
+ }
...+>
}

@@
expression x, E;
identifier f;
statement ES;
@@

f(...) {
+ void *ptmp;

<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (x == NULL) ES
+ if (ptmp == NULL) {
+ SCFree(x);
+ x = NULL;
+ ES
+ } else {
+     x = ptmp;
+ }
...+>

}

@@
expression x, E;
identifier f;
@@

f(...)
{
+ void *ptmp;
<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (unlikely(x == NULL))
+ if (unlikely(ptmp == NULL))
{
+ SCFree(x);
+ x = NULL;
...
- }
+ } else {
+     x = ptmp;
+ }
...+>
}

@@
expression x, E;
identifier f;
statement ES;
@@

f(...) {
+ void *ptmp;

<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (unlikely(x == NULL)) ES
+ if (unlikely(ptmp == NULL)) {
+ SCFree(x);
+ x = NULL;
+ ES
+ } else {
+     x = ptmp;
+ }
...+>

}
12 years ago
Anoop Saldanha a49cbf8a49 Code cleanup.
Use the MpmAddPattern[CS|CI] wrapper to add patterns to the mpm context.

Also use MpmInitCtx() to init the mpm context.
12 years ago
Anoop Saldanha cfa2cda42b fix for bug #973.
An alternative solution for bug #970.

For chopped patterns, which in it's whole is a duplicate of another
pattern we assign an unique content id.
12 years ago
Anoop Saldanha d2ea799d38 fix for bug #970.
Content strings that are a duplicate of a pattern from another sig, but
have a fast_pattern chop being applied, would end up being assigned the
same pattern id as the duplicate string.  But the string supplied to the
mpm would be the chopped string, which might result in the state_table
output_state content entry being over-riden by the the fuller string at
the final state of the smaller content length, because of which during a
match we might end up inspecting the search buffer against the fuller
content pattern, instead of the chopped pattern, which would end up being
an inspection beyond the buffer bounds.
12 years ago
Anoop Saldanha da75db9330 Unittest to display bug #970. 12 years ago
Nelson Escobar ef4d11aeb5 Use the Async versions of SCCudaMemcpy* to improve gpu performance. 12 years ago
Anoop Saldanha 8516ba24c9 Rearrange ac state.
Notice a minor speed bump of around 2% on runs.  More updates to follow.
12 years ago
Anoop Saldanha cdaa13012a fix for #882.
Refactor the code that initializes the cuda mpm environment.
12 years ago
Anoop Saldanha 3c2ddf04c1 Update mpm init ctx to not accept the final cuda_rc_module argument.
It was a part of our older architecture and is no longer used.
12 years ago
Anoop Saldanha 602c91ed41 Minor cosmetic changes to the cuda code.
Moved a couple of functions to more cuda relevant files;
Re-structured some data types.
12 years ago
Anoop Saldanha c9f076def3 Modified CudaBufferCullCompletedSlices.
Allow readers specify max size of data they want to read.
12 years ago
Anoop Saldanha 17c763f855 Version 1 of AC Cuda. 12 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Anoop Saldanha 0eaf0b0129 mpm engine and ac mem free fixes 13 years ago
Eileen Donlon 0bb4ff34b8 added null checks for init_hash to all ac mpms 14 years ago
Anoop Saldanha 420befb180 Changed my email address to anoopsaldanha at gmail dot com from my current one 14 years ago
Anoop Saldanha d034b10180 remove debug prints added to ac code 14 years ago
Anoop Saldanha 781e7c776f fix indentation in ac code 14 years ago
Anoop Saldanha 5c56053a33 Reintroduced optimized support for < 32k states for ac 14 years ago
Anoop Saldanha b7b58074de fix ac unittest 14 years ago
Anoop Saldanha d6f9e06bbb update ac to behave the same way irrespective of the state count. Should improve performance. Also fix unittests to accomodate these changes 14 years ago