45 Commits (8e655cf107db515d6bcc601ccf9c45481cc56a09)

Author	SHA1	Message	Date
Jason Ish	55f2704a25	logging: remove the packetqueue's from the logging path ... They are not referenced by any loggers, and they probably shouldn't be either.	9 years ago
Jason Ish	00b6e628d1	logging: hook into flow worker thread	9 years ago
Jason Ish	fc35a78ba1	logging: use a single entry point for all loggers ... Introduces a new thread module, TMM_LOGGER, which is the root most logger. Only handles loggers in the packet path, stats and flow logging are not included. The loggers are made up of a hierarchy of loggers. At the top we have the root logger which is the main entry point to logging. Under the root there exists parent loggers that are the entry point for specific types of loggers such as packet logger, transaction loggers, etc. Each parent logger may have 0 or more loggers that actual handle the job of producing output to something like a file.	9 years ago
Jason Ish	8b38b9d728	output.[ch]: consistent style ... - Clean up function declaration. - Consistenly use typedefs for function points. No functional changes.	9 years ago
Jason Ish	fa27a76462	logging: add profiling back for non-tmm loggers ... The loggers moved away from a TMM required a new profiling support.	9 years ago
Jason Ish	7a8e8343e5	logging: convert tcp data logging to non-thread module	9 years ago
Jason Ish	4d8b8ca046	logging: convert tls store logging to non-thread module	9 years ago
Jason Ish	60b6ccc3c4	logging: convert file data logging to non-thread module	9 years ago
Jason Ish	f9bb9029c5	logging: convert file logging to non-thread module	9 years ago
Jason Ish	b580016c80	logging: convert stats loggers to non-thread module	9 years ago
Jason Ish	983a619ff0	logging: convert json flow output to non-thread module	9 years ago
Jason Ish	ad15ac8297	logging: convert json alert output to non-thread module	9 years ago
Jason Ish	aaa65f3d16	logging: convert json tls output to non-thread module	9 years ago
Jason Ish	687602c0ca	logging: convert eve dns logging to non-thread module	9 years ago
Jason Ish	b1200dba54	logging: convert fast log to a non-thread module	9 years ago
Jason Ish	637aa34610	logging: convert dns log to a non-thread module	9 years ago
Jason Ish	fcad270d96	logging: setup all registered loggers for a name ... When setting up a configured logger, do so for all registered loggers of that name instead of just the first registered one. This allows a logger to register itself more than once, which can allow for independent logging of requests and replies without touching the core transaction handling logic. We do this so just having "dns" in the eve-log can configured multiple "dns" loggers instead of having something like "dns-tc" and "dns-ts" in the configuration file.	9 years ago
Mats Klepsland	77cc03505b	output: add new tx logger to log at certain condition ... Some loggers needs certain conditions to be met before logging. This enables us to use conditions on the tx logger.	10 years ago
Mats Klepsland	47a4daeeb8	output: allow more than one TLS logger	10 years ago
Mats Klepsland	fa6257b66a	output: add new logger to log at specified state ... Sometimes we want to log when we reach a specified state instead of waiting for the session to end. E.g for TLS we want to log as soon as the handshake is done. To do this, a new logger is added, where it is possible to specify a custom "ProgressCompletionStatus".	10 years ago
Victor Julien	e98346b555	Introduce stats log API, convert existing output ... Convert regular 'stats.log' output to this new API. In addition to the current stats value, also give the last value. This makes it easy to display the difference.	11 years ago
Victor Julien	bac6c3ab02	streaming logger: support Http Body logging ... Add an argument to the registration to indicate which iterator needs to be used: Stream or HttpBody Add HttpBody Iterator, calling the logger(s) for each Http body chunk.	11 years ago
Victor Julien	9d9ef983dd	output-streaming: a Log API for streaming data ... This patch adds a new Log API for streaming data such as TCP reassembled data and HTTP body data. It could also replace Filedata API. Each time a new chunk of data is available, the callback will be called.	11 years ago
Victor Julien	c7ebfd1b68	flow: flow log threading setup ... Set up threading for the flow logger.	11 years ago
Jason Ish	fc2014ab40	Unregister for file rotation notification when a context is ... de-initialized. Required for unix-socket mode where contexts come and go.	12 years ago
Jason Ish	698a0f7f48	Registration for SIGHUP notification - for loggers interested ... in file rotation on SIGHUP.	12 years ago
Victor Julien	9df045d086	output: add Disable funcs to mirror Enable ... For the loggers that we allow only one instance for: tls, ssh, drop, we track active loggers through Output*Enable functions. Add Disable functions to mirror this. They are to be called from the shutdown funcs those loggers use.	12 years ago
Victor Julien	b877cf6158	ssh: add json logger ... Sub module of eve-log, but can also run separately as ssh-json-log. Only one at a time though.	12 years ago
Victor Julien	9950427466	output: check for multiple instances of drop and tls ... Both the drop and tls logs are currently not designed to have multiple instances running. So until that is changed, error out if more than one instance is started.	12 years ago
Victor Julien	4bd37cc46a	log api: use AppProto instead of uint16_t	12 years ago
Victor Julien	52c3d3ad7c	log api: convert all names to const ... Instead of strdupping all names w/o a need, use const ptrs.	12 years ago
Victor Julien	79771ff570	output: sub-module support for other log api's ... Packets: void OutputRegisterPacketSubModule(const char *parent_name, char *name, char *conf_name, OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *), PacketLogger LogFunc, PacketLogCondition ConditionFunc); Files: void OutputRegisterFileSubModule(const char *parent_name, char *name, char *conf_name, OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *), FileLogger FileLogFunc); Filedata: void OutputRegisterFiledataSubModule(const char *parent_name, char *name, char *conf_name, OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *), FiledataLogger FiledataLogFunc);	12 years ago
Victor Julien	f830cb8026	output: introduce concept of sub-modules ... To support the 'eve-log' idea, we need to be able to force all log modules to be enabled by the master eve-log module, and need to be able to make all logs go into a single file. This didn't fit the API so far, so added the sub-module concept. A sub-module is a regular module, that registers itself as a sub- module of another module: OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "http", OutputHttpLogInitSub, ALPROTO_HTTP, JsonHttpLogger); The first argument is the name of the parent. The 4th argument is the OutputCtx init function. It differs slightly from the non-sub one. The different is that in addition to it's ConfNode, it gets the OutputCtx from the parent. This way it can set the parents LogFileCtx in it's own OutputCtx. The runmode setup code will take care of all the extra setup. It's possible to register a module both as a normal module and as a sub- module, which can operate at the same time. Only the TxLogger API is handled in this patch, the rest will be updated later.	12 years ago
Victor Julien	9ff6608668	Introduce Filedata Logger API ... A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (*FiledataLogger)(ThreadVars *, void *thread_data, const Packet *, const File *, const FileData *, uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work.	12 years ago
Victor Julien	ee2a8a9cda	Introduce 'file' logging API ... This patch introduces a new logging API for logging extracted file info. It allows for registration of a callback that is called once per file: when it's considered 'closed'. Users of this API register their Log Function through: OutputRegisterFileModule() The API uses a magic settings globally. This might be changed later.	12 years ago
Victor Julien	ad70793f78	Introduce TX logging API ... This patch introduces a new API for logging transactions from tx-aware app layer protocols. It runs all the registered loggers from a single thread module. This thread module takes care of the transaction handling and flow locking. The logger just gets a transaction to log out. All loggers for a protocol will be run at the same time, so there will not be any timing differences. Loggers will no longer act as Thread Modules in the strictest sense. The Func is NULL, and SetupOuputs no longer attaches them to the thread module chain individually. Instead, after registering through OutputRegisterTxModule, the setup data is used in the single logging module. The logger (LogFunc) is called for each transaction once, at the end of the transaction.	12 years ago
Victor Julien	d43ac9ae98	Introduce packet logging output API ... This patch introduces a new API for outputs that log based on the packet, such as alert outputs. In converts fast-log to the new API. The API gets rid of the concept of each logger being a thread module, but instead there is one thread module that runs all packet loggers. Through the registration function OutputRegisterPacketModule a log module can register itself to be considered for each packet. Each logger registers itself to this new API with 2 functions and the OutputCtx object that was already used in the old implementation. The function pointers are: LogFunc: the log function ConditionFunc: this function is called before the LogFunc and only if this returns TRUE the LogFunc is called. For a simple alert logger like fast-log, the condition function will simply return TRUE if p->alerts.cnt > 0.	12 years ago
Mike Pomraning	dec34afa40	SCConfLogOpenGeneric() abstraction for regular and AF_UNIX logs. ... util-logopenfile.[ch] implements the abstraction; util-error.[ch] modified to include a socket-specific error code; output.h adds a default filetype for logs ("regular").	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Gurvinder Singh	f4392e1dcc	added support for appending the log files	15 years ago
William Metcalf	ce01927515	Import of GPLv2 Header 050410	16 years ago
Jason Ish	40f9653c06	Have output plugs use an OutputCtx which is a little more generic than LogFileCtx. The OutputCtx provides a place for module private data to avoi overriding the LogFileCtx.	16 years ago
Jason Ish	fbdf1baf1c	- rebase ... Provide limits to the unified outputs.	16 years ago
Pablo Rincon	cef12d30b5	Unified output fixes: alert count per module (not per thread), fix timestamps on pcap mode, write *all* the alerts of a packet, write the log header once also on unified alert	16 years ago
Jason Ish	e204d07717	Have output modules register themselves so run mode configurator becomes aware of them for purposes of being configured from the config file.	16 years ago