When fuzzing, AFL will create lots of malformed rules. We don't want
to error out on those. As we're fuzzing the parser any non-crash
should return 0. Crashes (ASAN or not) will return a non-0 code.
Add regex setup and free util functions. Keywords often use a regex
to parse rule input. Introduce a common function to do this setup.
Also create a list of registered regexes to free at engine shutdown.
Unit testing support macros for failing on expressions,
as well as passing tests on expressions.
If fatal unittests are enabled BUG_ON will be triggered for
an assertion providing the line number of the failure, otherwise
the test will simply fail.
Moved the fatal flag to a global var instead of a configuration
parameter for ease of access from a macro.
Instead of detect-engine which used a list for no good reason, use a
simple map now.
detect:
profile: medium
custom-values:
toclient-groups: 3
toserver-groups: 25
sgh-mpm-context: auto
inspection-recursion-limit: 3000
# If set to yes, the loading of signatures will be made after the capture
# is started. This will limit the downtime in IPS mode.
#delayed-detect: yes
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.
Logs SGH id so it can be compared with the rule_group.json output.
Implemented both in a human readable text format and a JSON format.
Simplify handling of USR2 signal. The SCLogInfo usage could lead to
dead locks as the SCLog API can do many complicated things including
memory allocations, syslog calls, libjansson message construction.
If an existing malloc call was interupted, it could lead to the
following dead lock:
0 __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
1 0x0000003140c7d2df in _L_lock_10176 () from /lib64/libc.so.6
2 0x0000003140c7ab83 in __libc_malloc (bytes=211543457408) at malloc.c:3655
3 0x0000003140c80ec2 in __strdup (s=0x259ca40 "[%i] %t - (%f:%l) <%d> (%n) -- ") at strdup.c:43
4 0x000000000059dd4a in SCLogMessageGetBuffer (tval=0x7fff52b47360, color=1, type=SC_LOG_OP_TYPE_REGULAR, buffer=0x7fff52b47370 "", buffer_size=2048,
log_format=0x259ca40 "[%i] %t - (%f:%l) <%d> (%n) -- ", log_level=SC_LOG_INFO, file=0x63dd00 "suricata.c", line=287, function=0x640f50 "SignalHandlerSigusr2StartingUp", error_code=SC_OK,
message=0x7fff52b47bb0 "Live rule reload only possible after engine completely started.") at util-debug.c:307
5 0x000000000059e940 in SCLogMessage (log_level=SC_LOG_INFO, file=0x63dd00 "suricata.c", line=287, function=0x640f50 "SignalHandlerSigusr2StartingUp", error_code=SC_OK,
message=0x7fff52b47bb0 "Live rule reload only possible after engine completely started.") at util-debug.c:549
6 0x000000000057e374 in SignalHandlerSigusr2StartingUp (sig=12) at suricata.c:287
7 <signal handler called>
8 _int_malloc (av=0x3140f8fe80, bytes=<value optimized out>) at malloc.c:4751
9 0x0000003140c7ab1c in __libc_malloc (bytes=296) at malloc.c:3657
10 0x0000000000504d55 in FlowAlloc () at flow-util.c:60
11 0x00000000004fd909 in FlowInitConfig (quiet=0 '\000') at flow.c:454
12 0x0000000000584c8e in main (argc=6, argv=0x7fff52b4a3b8) at suricata.c:2300
This patch simply sets a variable and lets the main loop act on that.
On systems like Gentoo where _FORTIFY_SOURCE is already defined like
FORTIFY_SOURCE=((defined __OPTIMIZE && OPTIMIZE > 0) ? 2 : 0) the use
within the printf function (%d) won't result in the correct value and
we end up with 'defined' undeclared compile error. This workaround makes
sure that just the resolved value is checked and then printed.
It is not possible to use simultaneously -l and unix socket
runmode because setting the log directory make it final so
not modifable by other call.
It is a implementation limitation but it does not make sense
to set logging directory to have it overwritten by the first
directory specified 'by pcap-file'. So it seems correct to
only trigger an error if this both options are used at the same
time.
When user asks for a configuration dump, it is useless to display
the version and CPU info. Also initializing the log system conduct
to overwrite the some log files and in particular suricata.log and
this is annoying as a command should not interfere with a running
daemon.
When there are many threads and/or the packet pool (max-pending-packets) is
small, a potential dead lock exists between the packet pool return pool
logic and the capture threads. The autofp workers together can have all the
packets in their return pools, while the capture thread(s) are waiting at an
empty pool. A race between the worker threads and the capture thread, where
the latter signals the former, is lost by the capture thread. Now everyone
is waiting.
To avoid this scenario, this patch makes the previously hardcoded 'return
pool' threshold dynamic based on the number of threads and the packet pool
size.
It sets the threshold to the max pending packets value, divided by the number
of lister threads. The max value hasn't changed. Normally, in the autofp
runmode these are the stream/detect/log worker threads.
The max_pending_return_packets value needs to stay below the packet pool size
of the 'producers' (normally pkt capture threads but also flow timeout
injection) to avoid the deadlock.
As it's quite impossible at this time to learn how many threads will be
created before starting the runmodes, and thus spawning the threads and
already initializing the packet pools, this code sets a global variable
after runmode setup, but before the threads are 'unpaused'.
An design error was made when doing the TLS storage module which
has been made dependant of the TLS logging. At the time there was
only one TLS logging module but there is now two different ones.
By putting the TLS store module in a separate module, we can now
use EVE output and TLS store at the same time.
Due to an error at initialization, the stream engine would not disable
'raw' reassembly automatically when --disable-detection was used.
This lead to segments not getting cleared from the segment lists.
Don't kill flow manager and recyclers before the rest of the threads. The
packet threads may still have packets from their pools. As the flow threads
would destroy their pools the packets would be lost.
This patch doesn't kill the threads, it just pulls them out of their run
loop and into a wait loop. The packet pools won't be cleared until all
threads are killed.
Wait for flow management threads to close before moving on to the
next steps in the shutdown process.
Don't destroy flow force reassembly packet pool too early. Worker
threads may still want to return packets to it.
Load the YAML into a prefix "detect-engine-reloads.N" where N is the
reload counter. This way we can load the updated config w/o overwriting
the current one.
Use new DetectEngineReload() function. It's called from the main loop
instead of it being spawned into it's own temporary thread. This greatly
simplifies the signal handling.
An added advantage is that this seems to improve the memory usage.
Related to bug #1358
The minimal detect engine has only the minimal memory use and setup
time. It's to be used for 'delayed' detect where the first detection
engine is essentially empty.
The threads setup are also minimal.
Instead of threading logic with dummy slots and all, use the regular
reload logic for delayed detect.
This means we pass a empty detect engine to the threads and then
reload (live swap) it as soon as the engine is running.
Victor Julien
Jason Ish
Justin Viiret
Eric Leblond
Andreas Herz
Alexander Gozman
Tom DeCanio