Commit e7c0f0ad91 removed uses of atoi with a new number parsing
functions. This broke parsing ip-reputation data files that contained
trailing carriage returns as it was being included in the number
string to convert.
Bug: #6243.
Multi-tenancy uses loader threads that initialize detection engines. During
this, esp the AC family of MPM implementations, there is significant stack
usage. In most OS' threads have a lower stack size by default. In Linux, when
using the Musl C library, a thread by default gets 128KiB.
This patch does 2 things:
1. it centralizes the handling of the `threading.stack-size`. It it is not
longer handled by the runmodes, but called from the global initialization
logic.
2. it sets a minimum per thread stack size of 512k, unless `threading.stack-size`
is set.
Ticket: #6265.
Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Bug: #6044.
Bug: #6201.
Make sure to first close all ports before freeing device mempools.
Thread 1 "Suricata-Main" received signal SIGSEGV, Segmentation fault.
0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
(gdb) bt
#0 0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#1 0x00007ffff469a948 in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#2 0x00007ffff45606aa in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#3 0x00007ffff6d4ed8d in rte_eth_dev_close () from /usr/lib/x86_64-linux-gnu/librte_ethdev.so.20.0
#4 0x000000000055fc4c in DPDKCloseDevice (ldev=ldev@entry=0xe3a400) at util-dpdk.c:53
#5 0x000000000055f4eb in LiveDeviceListClean () at util-device.c:331
#6 0x00000000005511c8 in GlobalsDestroy (suri=<optimized out>) at suricata.c:381
#7 0x0000000000550a76 in SuricataMain (argc=<optimized out>, argv=<optimized out>) at suricata.c:3059
#8 0x00007ffff6a24083 in __libc_start_main (main=0x54cca0 <main>, argc=8, argv=0x7fffffffe4c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4b8) at ../csu/libc-start.c:308
#9 0x000000000054cbde in _start ()
Bug: #5619.
Issue: 6239
This commit moves the global variables associated with engine analysis
into the detect engine context. Doing so provides encapsulation of the
analysis components as well as thread-safe operation in a multi-tenant
(context) deployment.
When viewing the docs online at Readthedocs, or similar it might be
immediately apparent what version of the documentation is being
displayed. Display the version on the first line before the table of
contents to make it clear.
Data on RST packets is not invalid, but also shouldn't be used
in reassembly.
RFC 1122:
4.2.2.12 RST Segment: RFC-793 Section 3.4
A TCP SHOULD allow a received RST segment to include data.
DISCUSSION
It has been suggested that a RST segment could contain
ASCII text that encoded and explained the cause of the
RST. No standard has yet been established for such
data.
RST data will be presented to the detection engine per packet,
but will not be part of stream reassembly.
Bug: #6244.
Thomas Winter
Victor Julien
Philippe Antoine
Shivani Bhardwaj
Jason Ish
jason taylor
Alexandre Iooss
Juliana Fajardini
Jeff Lucovsky
Andreas Herz