Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
Avoid looping in transaction output.
Update app-layer API to store the bits in one step
and retrieve the bits in a single step as well.
Update users of the API.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
Prevents the case where the logged id is incremented if a newer
transaction is complete and an older one is still outstanding.
For example, dns request0, unsolicited dns response, dns response0
would result in the valid response0 never being logged.
Similarily this could happen for:
request0, request1, response1, response0
which would end up having request0, request1 and response1 logged,
but response0 would not be logged.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
Introduces a new thread module, TMM_LOGGER, which is the
root most logger.
Only handles loggers in the packet path, stats and flow
logging are not included.
The loggers are made up of a hierarchy of loggers. At the top we
have the root logger which is the main entry point to
logging. Under the root there exists parent loggers that are the
entry point for specific types of loggers such as packet logger,
transaction loggers, etc. Each parent logger may have 0 or more
loggers that actual handle the job of producing output to something
like a file.
When setting up a configured logger, do so for all registered
loggers of that name instead of just the first registered one.
This allows a logger to register itself more than once, which
can allow for independent logging of requests and replies without
touching the core transaction handling logic.
We do this so just having "dns" in the eve-log can configured
multiple "dns" loggers instead of having something like "dns-tc"
and "dns-ts" in the configuration file.
Sometimes we want to log when we reach a specified state instead of
waiting for the session to end. E.g for TLS we want to log as soon
as the handshake is done.
To do this, a new logger is added, where it is possible to specify
a custom "ProgressCompletionStatus".
Change AppLayerParserRegisterGetStateProgressCompletionStatus to
only store one ProgressCompletionStatus callback function for each
alproto, instead of storing one for each ipproto.
This enables us to use AppLayerParserGetStateProgressCompletionStatus
in functions where we do not know the ipproto used.
This fixes:
Direct leak of 31752 byte(s) in 3969 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0xe385b9 in OutputTxLogThreadInit /home/pmanev/sandnet-qa/stage/oisf/src/output-tx.c:193:34
#2 0x106c255 in TmThreadsSlotPktAcqLoop /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:295:17
#3 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
This is a beginning of implementation for bug #1660:
https://redmine.openinfosecfoundation.org/issues/1160
This patch adds a cleaning function for each logger of new type
(packet, tx and file). These functions are called in RunModeShutDown().
The state of this patch is that it is crashing suricata when sending
pcap to analyse:
- At first pcap if tx and file cleaning function are called
- At second pcap if only packet cleaning function is called
The cause in first case is unknown. In second case this is due to
the necessity of cleaning the list of logger registered to a logging
type.
The log API calls thread modules directly, so the TMM profiling logic
can be applied to it. This patch does so.
The "Thread Module" out now again lists the individual loggers. As the
module are normally called much less frequently the numbers are hard to
compare to pre-log-api numbers.
This patch introduces a new API for logging transactions from
tx-aware app layer protocols. It runs all the registered loggers
from a single thread module. This thread module takes care of the
transaction handling and flow locking. The logger just gets a
transaction to log out.
All loggers for a protocol will be run at the same time, so there
will not be any timing differences.
Loggers will no longer act as Thread Modules in the strictest sense.
The Func is NULL, and SetupOuputs no longer attaches them to the
thread module chain individually. Instead, after registering through
OutputRegisterTxModule, the setup data is used in the single logging
module.
The logger (LogFunc) is called for each transaction once, at the end
of the transaction.
Victor Julien
Jason Ish
Mats Klepsland
Eric Leblond
Ken Steele