When reaching the end of either list, merging is no longer required,
simply walk down the other list.
If the non-MPM list can't have duplicates, it would be worth removing
the duplicate check for the non-MPM list when it is the only non-empty list
remaining.
Create a copy of the SigMatch data in the sig_lists linked-lists and store
it in an array for faster access and not next and previous pointers. The
array is then used when calling the Match() functions.
Gives a 7.7% speed up on one test.
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.
Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.
The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
Read one signature pointer ahead to prefetch the value.
Use a variable, sflags, for s->flags, since it is used many times and the
compiles doesn't know that the signatures structure doesn't change, so it
will reload s->flags.
Use an MPM specific pattern index, which is simply an index starting
at zero and incremented for each pattern added to the MPM, rather than
the externally provided Pattern ID (pid), since that can be much
larger than the number of patterns. The Pattern ID is shared across at
MPMs. For example, an MPM with one pattern with pid=8000 would result
in a max_pid of 8000, so the pid_pat_list would have 8000 entries.
The pid_pat_list[] is replaced by a array of pattern indexes. The PID is
moved to the SCACTilePatternList as a single value. The PatternList is
also indexed by the Pattern Index.
max_pat_id is no longer needed and mpm_ctx->pattern_cnt is used instead.
The local bitarray is then also indexed by pattern index instead of PID, making
it much smaller. The local bit array sets a bit for each pattern found
for this MPM. It is only kept during one MPM search (stack allocated).
One note, the local bit array is checked first and if the pattern has already
been found, it will stop checking, but count a match. This could result in
over counting matches of case-sensitve matches, since following case-insensitive
matches will also be counted. For example, finding "Foo" in "foo Foo foo" would
report finding "Foo" 2 times, mis-counting the third word as "Foo".
In PmqMerge() use MpmAddSids() instead of blindly copying the src
rule list onto the end of the dst rule list, since there might not
be enough room in the dst list. MpmAddSids() will resize the dst array
if needed.
Also add code to MpmAddSids() MpmAddPid() to better handle the case
that realloc fails to get more space. It first tries 2x the needed
space, but if that fails, it tries for just 1x. If that fails resize
returns 0. For MpmAddPid(), if resize fails, the new pid is lost. For
MpmAddSids(), as many SIDs as will fit are added, but some will be
lost.
Rather than creating the array of size maxpatid, dynamically resize as needed.
This also handles the case where duplicate pid are added to the array.
Also fix error in bitarray allocation (local version) to always use bitarray_size.
Rather than statically allocate 64K entries in every rule_id_array,
increase the size only when needed. Created a new function MpmAddSids()
to check the size before adding the new sids. If the array is not large
enough, it calls MpmAddSidsResize() that calls realloc and does error
checking. If the realloc fails, it prints an error and drops the new sids
on the floor, which seems better than exiting Suricata.
The size is increased to (current_size + new_count) * 2. This handles the
case where new_count > current_size, which would not be handled by simply
using current_size * 2. It should also be faster than simply reallocing to
current_size + new_count, which would then require another realloc for each
new addition.
Use a local pattern bit array to making sure we don't match more than
once, in addition to the pmq bitarray that is still used for results
validation higher up in the rule matching process.
Why: pmq->pattern_id_bitarray is currently sometimes used in a
'stateful' way, meaning that for a single packet we run multiple
MPM's on the same pmq w/o resetting it.
The new bitarray is used to determine wherther we need to append the
patterns associated 'sids' list to the pmq rule_id_array.
It has been observed that MPM1 matches for PAT1, and MPM2 matches for
PAT1 as well. However, in MPM1 PAT1 doesn't have the same sids list.
In this case MPM2 would not add it's sids to the list, leading to missed
detection.
Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.
This disables the old match array building code and thus also bypasses
the mask checking.
Treat negated MPM sigs as if non-MPM, so we consider them always.
As MPM results and non-MPM rules lists are now merged and considered
for further inspection, rules that need to be considerd when a pattern
is absent are caught in the middle.
As a HACK/workaround this patch adds them to the non-MPM list. This
causes them to be inspected each time.
Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.
AC: store sids in the pattern list, append to Pmq::rule_id_array on match.
Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
Creates an inline wrapper to check for flowvarlist == NULL before calling
DetectFlowvarProcessList() to remove the overhead of checking since the
list is usually empty.
** CID 1257764: Dereference after null check (FORWARD_NULL)
/src/defrag.c: 291 in Defrag4Reassemble()
** CID 1257763: Dereference after null check (FORWARD_NULL)
/src/defrag.c: 409 in Defrag6Reassemble()
In the error case 'rp' can be both NULL or non-NULL.
MemcmpLowercase would not compare the first byte of both input buffers
leading to two non-identical buffers to be considered the same.
Affects SSE_4_1 and SSE_4_2 implementations of SCMemcmpLowercase, as well
as the non-SIMD implementation. SSE_3 and Tile version are not affected.
Work around OS X 10.10 Yosemite returning EDEADLK on a rwlock wrlocked
then tested by wrtrylock. All other OS' (and versions of OS X that I
tested) seem to return EBUSY instead.
Updated reference.config to match ET Open reference.config found here:
https://rules.emergingthreats.net/open/suricata/reference.config
Due to startup error shown here:
root@xxxxxxx01:/etc/suricata/rules# /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet
23/12/2014 -- 22:07:56 - <Error> - [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "osvdb". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"
<...>
Killed
Fix issue where SMTPStateGetTxCnt would return the actual active tx'.
The 'GetCnt' API call is not named correctly. It should be 'GetMaxId',
as this is actually the expected behavior.
Victor Julien
Ken Steele
Travis Green
Jason Ish