Flow timeout code keeps track of thread module running detect, and
fails (hard) if it doesn't find it.
This changeset retrieves the global g_detect_disabled and passes
it to the timeout handling code during setup.
If FlowForceReassemblyForFlowV2 can't get packets to inject into the
engine, until now it would bail and retry later. In case of resource
starvation issues, this would cause a lot of lock contention, as the
flow manager would try over and over again.
This patch limits FlowForceReassemblyForFlowV2 to one try per flow,
if it fails... bad luck. It will only fail in serious conditions,
which means we must prefer the health of the engine over the proper
inspection of the flow in question.
StreamMsgs would be stored in a per thread queue before being
attached to the tcp ssn. This is unnecessary, so this patch
removes this queue and puts the smsgs into the ssn directly.
Large patch as it affects a lot of tests.
Preparation for removing flow pointer from StreamMsg. Instead of
getting the ssn indirectly through StreamMsg->flow, we pass it
directly as all callers have it already.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
Flow timeout code worked by luck when checking if a flow still needed
reassembly for app layer inspection or logging. It would check for a
part of raw reassembly (smsg list) to determine if detection was
needed. In this case it would also process app layer cleanup,
including logging.
Introduced AppLayerTransactionGetActive which returns the lowest tx_id
in a direction that still needs some work.
FlowForceReassemblyNeedReassmbly now uses it to determine if the
applayer still needs work.
Converted FlowForceReassemblyForHash to use the checking function
FlowForceReassemblyNeedReassmbly as well, so that checking if a flow
needs work is now unified.
In FFRv2, dereference flow from a packet using the new reference/dereference
util macros. This allows the decr use_cnt for flow and reseting the flow
pointer to NULL for the pseudo pkt to happen simultaneously, in case there we
fail to retrieve a pseudo_packet and have to return the already obtained
pseudo packets, back to the packetpool.
This patch modifies the init of Detect threads. They are now started
with a dummy function and their initialisation is done after the
signatures are loaded. Just after this, the dummy function is switched
to normal one.
In IPS mode, this permit to route packets without waiting for the
signature to start and should fix#488.
Offline mode such as pcap file don't use this mode to be sure to
analyse all packets in the file.
The patch introduces a "delayed-detect" configuration variable
under detect-engine. It can be used to activate the feature
(set to "yes" to have signature loaded after capture is started).
Major redesign of the flow engine. Remove the flow queues that turned
out to be major choke points when using many threads. Flow manager now
walks the hash table directly. Simplify the way we get a new flow in
case of emergency.
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.
Add macro's to check the family, copy the address, etc.
Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.
On 64 bit this shrinks the flow with 8 bytes.
Victor Julien
Anoop Saldanha
Last G
Eric Leblond
Anoop Saldanha