When replaying a pcap file, it is not possible to get rules
profiling because it has to be activated from the unix socket.
This patch adds a new option to be able to activate profiling
collection at start so a pcap run can get rules profiling
information.
(cherry picked from commit eecb3440e2)
When splitting the field over commas, skip the commas
which are innside quotes
Ticket: 7060
Not a direct backport, but heavily inspired by new rust code
handling it
This commit allows ja4 hashes to be logged iff enabled in the tls/quic
section of the outputs.
With the default setting ("off"), ja4 hashes will only be logged in
alerts when the signatures uses the ja4.hash keyword.
When enabled, ja4 hashes will be inclued in quic and tls logs.
- tls:
ja4: on
- quic:
ja4: on
Issue: 7010
EVE logging has a direction parameter that can cause the logging
of an application layer to be done in a direction that is not linked
to the packet. As a result the source IP addres could be assigned the
MAC address of the destination IP and reverse.
This patch addresses this by propagating the direction to the ethernet
logging function and using it there to define the correct mapping.
Issue #6405
(cherry picked from commit 21916b9743)
In certain conditions, it can take a long time for threads to start up.
For example in af-packet, setting up the socket, rings, etc has been
observed to take close to half a second per thread, and since the
threads go one by one in a preset order, this means the start up can
take a lot of time if there are many threads. The old logic would just
allow a hard coded 60s. This was not always enough when the number of
threads was high.
This patch makes the wait time take the number of threads into account.
It adds a second of time budget to the base 60s for each thread.
So as an example, if a system has 112 af-packet threads, it would wait
172 seconds (60 + 112) for the threads to get ready.
Ticket: #7048.
(cherry picked from commit 41b9836b11)
When starting a large amount of threads, the loop was inefficient. It
would loop over the threads and if one wasn't yet ready it would sleep a
bit and then reevaluate all the threads. This reevaluation of threads
already checked was inefficient, and could lead to the time budget
running out.
This patch splits the check, and keeps track of the threads that have
already passed. This avoids the rescanning of already checked threads.
(cherry picked from commit 85fd4b2ec7)
The functions in detect-engine-port.c are only being used at the time of
parsing the ports from rules initially. Since there are no SGHs at that
point, remove the ops related to them too.
(cherry picked from commit e14fefe24a)
So far, if a port was found to be single which was earlier a part of the
range, port + 1 was added to the list to honor the range that it was a
part of. But, this is incorrect in case the port is 65535 or if the port
was found to be of range when it was earlier a single port.
Bug 6896
(cherry picked from commit 4227e52c4b)
If a single port happens before a range port, the port groups created
were incorrect. Fix it to use smarter range check.
For example, given,
80:80 - SGH1
100:120 - SGH2
Range created should be
80:80 - SGH1
100:120 - SGH2
Bug 6881
(cherry picked from commit 7d937db5cb)
If a port point is single but later on also a part of a range, it ends
up only creating the port groups for single points and not the range.
Fix it by adding the port next to current single one to unique points
and marking it a range port.
Bug 6843
(cherry picked from commit 632ca75dd3)
Fix Coverity warning
** CID 1592992: Incorrect expression (COPY_PASTE_ERROR)
/src/util-port-interval-tree.c: 255 in SCPortIntervalFindOverlaps()
________________________________________________________________________________________________________
*** CID 1592992: Incorrect expression (COPY_PASTE_ERROR)
/src/util-port-interval-tree.c: 255 in SCPortIntervalFindOverlaps()
249 * will be sorted, insert any new ports to the end of the list
250 * and avoid walking the entire list */
251 if (*list == NULL) {
252 *list = new_port;
253 (*list)->last = new_port;
254 } else if (((*list)->last->port != new_port->port) &&
>>> CID 1592992: Incorrect expression (COPY_PASTE_ERROR)
>>> "port" in "(*list)->last->port2 != new_port->port" looks like a copy-paste error.
255 ((*list)->last->port2 != new_port->port)) {
256 DEBUG_VALIDATE_BUG_ON(new_port->port < (*list)->last->port);
257 (*list)->last->next = new_port;
258 new_port->prev = (*list)->last;
259 (*list)->last = new_port;
260 } else {
The code does not generate two port ranges that are same other than the
cases where port == port2 which is why it worked so far. Fix it.
Bug 6839
(cherry picked from commit 2d6708f1ff)
During startup large rulesets use a lot of large bitarrays, that
are frequently merged (OR'd).
Optimize this using SSE2 _mm_or_si128.
(cherry picked from commit 94b4619)
Make rule group head bitarray 16 bytes aligned and padded to 16 bytes
boundaries to assist SIMD operations in follow up commits.
(cherry picked from commit 4ba1f44e0d)
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.
Ticket #6795
(cherry picked from commit e7e4305d91)
To avoid getting multiple entries in the final port list and to also
make the next step more efficient by reducing the size of the items to
traverse over.
Ticket 6792
Bug 6414
(cherry picked from commit 643ae85b5f)
As this is already taken care of and a list of ports is available for
use by the next stage.
Ticket 6792
Bug 6414
(cherry picked from commit 83aba93f40)
Using the unique port points, create a list of small port ranges which
contain the DetectPort objects and the designated SGHs found by finding
the overlaps with the existing ports and copying the SGHs accordingly.
Ticket 6792
Bug 6414
(cherry picked from commit 4ac2382f26)
After all the SGHs have been appropriately copied to the designated
ports, create an interval tree out of it for a faster lookup when later
a search for overlaps is made.
Ticket 6792
Bug 6414
(cherry picked from commit a02c44a3a4)
In order to create the smallest possible port ranges, it is convenient
to first have a list of unique ports. Then, the work becomes simple. See
below:
Given, a port range P1 = [1, 8]; SGH1
and another, P2 = [3, 94]; SGH2
right now, the code will follow a logic of recursively cutting port
ranges until we create the small ranges. But, with the help of unique
port points, we get, unique_port_points = [1, 3, 8, 94]
So, now, in a later stage, we can create the ranges as
[1, 2], [3, 7], [8, 8], [9, 94] and copy the designated SGHs where they
belong. Note that the intervals are closed which means that the range
is inclusive of both the points.
The final result becomes:
1. [1, 2]; SGH1
2. [3, 7]; SGH1 + SGH2
3. [8, 8]; SGH1 + SGH2
4. [9, 94]; SGH2
There would be 3 unique rule groups made for the case above.
Group 1: [1, 2]
Group 2: [3, 7], [8, 8]
Group 3: [9, 94]
Ticket 6792
Bug 6414
(cherry picked from commit c9a911b6f8)
Warning was:
src/util-port-interval-tree.c:50:1: warning: Either the condition 'tmp!=NULL' is redundant or there is possible null pointer dereference: tmp. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'tmp!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'oleft!=NULL' is redundant or there is possible null pointer dereference: oleft. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'oleft!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'oright!=NULL' is redundant or there is possible null pointer dereference: oright. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'oright!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'left!=NULL' is redundant or there is possible null pointer dereference: left. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'left!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
(cherry picked from commit 86f89e0966)
Add new utility files to deal with the interval trees. These cover the
basic ops:
1. Creation/Destruction of the tree
2. Creation/Destruction of the nodes
It also adds the support for finding overlaps for a given set of ports.
This function is used by the detection engine is the Stage 2 of
signature preparation.
Ticket 6792
Bug 6414
Co-authored-by: Victor Julien <vjulien@oisf.net>
(cherry picked from commit 54558f1b4a)
An interval tree uses red-black tree as its base data structure and
follows all the properties of a usual red-black tree. The additional
params are:
1. An interval such as [low, high] per node.
2. A max attribute per node. This attribute stores the maximum high
value of any subtree rooted at this node.
At any point in time, an inorder traversal of an interval tree should
give the port ranges sorted by the low key in ascending order.
This commit modifies the IRB_AUGMENT macro and it's call sites to make
sure that on every insertion, the max attribute of the tree is properly
updated.
Ticket 6792
Bug 6414
(cherry picked from commit d36d03a428)
In the commit 4a00ae607, the whitelisting check was updated in a quest
to make use of the conditional better but it made things worse as every
range would be whitelisted as long as it had any of the default
whitelisted port which is very common.
(cherry picked from commit fb9680bb7b)
When we only have stream matches.
Ticket: 6846
This solves the case where another transaction was created
by parsing data in the other direction, before running the
detection.
Like
1. get data in direction 1
2. acked data: parse it, but do not run detection in dir 1
3. other data in direction 2
4. other data acked : parse it and create new tx,
then run detection for direction 1 with data from first packet
(cherry picked from commit 7274ad58aa)
Recognize PPP_CCP, PPP_CBCP and PPP_COMP_DGRAM.
Does not implement decoders for these record types, so these
are logged as unsupported types. Was "wrong_type" before.
(cherry picked from commit 516441b600)
Eric Leblond
Philippe Antoine
Jeff Lucovsky
Sascha Steinbiss
jason taylor
Victor Julien
Alexey Simakov
Shivani Bhardwaj