DetectEngineReloadThreads does not work for the fuzz targets
as there is no_of_detect_tvs = 0 as we did not register
real threads and slots.
So, we force the flow worker module to use the newly detect engine
conetxt with all it needs
The flows exceeding the spare pools config setting would be freed
per at max 100 flows a second. After a high speed test this would
lead to excessive memory use for a long time.
This patch updates the logic to free 10% of the excess flows per
run, freeing multiple blocks of flows as needed.
Bug: #4731.
In a scenario where there was suddenly no more traffic flowing, flows
in a threads `flow_queue` would not be processed. The easiest way to
see this would be in a traffic replay scenario. After the replay is done
no more packets come in and these evicted flows got stuck.
In workers mode, the capture part handles timeout this was updated to
take the `ThreadVars::flow_queue` into account.
The autofp mode the logic that puts a flow into a threads `flow_queue`
would already wake a thread up, but the `flow_queue` was then ignored.
This has been updated to take the `flow_queue` into account.
In both cases a "capture timeout" packet is pushed through the pipeline
to "flush" the queues.
Bug: #4722.
When testing for fanout support a cluster-id of 1 was always being
used instead of the configured cluster-id. This limited fanout
support to only one Suricata instance.
Instead of hardcoding an ID of 1, use the configured cluster-id.
Also make cluster_id a uint16_t instead of an int in AFPThreadVars.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3419
A block is determined out of order on opening.
But on closing, the gap before it may have been filled.
So, we must post-process it, ie iterate over the red and black
tree so see what blocks we can get.
Better structure design to ensure that one flow maximum
is owning and appending into the file, adding fileOwning field.
Adds also a gap field in a range buffer, so that we can
feed the gap on closing, when we are protected from concurrency
by a lock, (lock which got removed in the append path)
Fixes memcap when encountering a duplicate while inserting
in red and black tree
Adds many comments
Simplify locking by using the THashData lock instead of a separate
range lock.
Avoid size_t in function arguments.
Clean up file handling functions.
Implement handling of alloc errors.
Rename yaml entry to byterange
Unify public api naming
adds a container, ie a thread safe hash table whose
key is the filename
keep a tree of unordered ranges, up to a memcap limit
adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
Moves Lua includes to util-lua.h instead of suricata-common
so plugins (or library users) don't need to be aware of the
Lua that Suricata was built with. Instead only source files
that need to be Lua aware can include util-lua.h.