58 Commits (eb09118d64003b06edaf455bd2bee0cab1fa467b)

Author	SHA1	Message	Date
Victor Julien	eb09118d64	detect: create loader threads ... To speed up startup with many tenants, tenant loading will be parallelized. As no tempary threads should be used for these memory allocation heavy tasks, this patch adds new type of 'command' thread that can be used to load and reload tenants. This patch hardcodes the number of loaders to 4. Future work will make it dynamic. The loader thread essentially sleeps constantly. When a tasks is sent to it, it will wake up and execute it.	11 years ago
Eric Leblond	4db0a35f25	tls-store: now a separate module ... An design error was made when doing the TLS storage module which has been made dependant of the TLS logging. At the time there was only one TLS logging module but there is now two different ones. By putting the TLS store module in a separate module, we can now use EVE output and TLS store at the same time.	11 years ago
Tom DeCanio	117eed0385	eve-log: add JSON stats logging ... Support for counters in stats.log in eve output JSON stream.	11 years ago
Victor Julien	df5e9d44ca	unix-manager: convert to thread module ... Sync command thread for unix manager with other managers and make it a full thread module.	11 years ago
gureedo	10104066e1	netmap support	11 years ago
Victor Julien	e98346b555	Introduce stats log API, convert existing output ... Convert regular 'stats.log' output to this new API. In addition to the current stats value, also give the last value. This makes it easy to display the difference.	11 years ago
Tom DeCanio	31f8f5cf20	eve-log: SMTP JSON logger	11 years ago
Victor Julien	db30ed8c3e	output: Lua HTTP log initial implementation ... Initial version of a HTTP LUA logger. Execute lua scripts from the Tx-log API.	11 years ago
Victor Julien	3dec0e96f8	tcp-data: new streaming logger ... tcp-data logs out reassembled stream data in a streaming fashion. Records type to log into different directories.	11 years ago
Victor Julien	9d9ef983dd	output-streaming: a Log API for streaming data ... This patch adds a new Log API for streaming data such as TCP reassembled data and HTTP body data. It could also replace Filedata API. Each time a new chunk of data is available, the callback will be called.	11 years ago
Ken Steele	8f1d75039a	Enforce function coding standard ... Functions should be defined as: int foo(void) { } Rather than: int food(void) { } All functions where changed by a script to match this standard.	11 years ago
Victor Julien	e0841218f0	flow-manager: support multiple instances ... Use new management API to run the flow manager. Support multiple flow managers, where each of them works with it's own part of the flow hash. Make number of threads configurable: flow: memcap: 64mb hash-size: 65536 prealloc: 10000 emergency-recovery: 30 managers: 2 This sets up 2 flow managers. Handle misc tasks only in instance 1: Handle defrag hash timeout handing, host hash timeout handling and flow spare queue updating only from the first instance.	12 years ago
Victor Julien	db15339f47	netflow-json: initial version ... Initial version of netflow module, a flow logger that logs each direction in a completely separate record (line).	12 years ago
Victor Julien	3c7af02067	flow-json-log: stub ... Stub for JSON flow logger.	12 years ago
Giuseppe Longo	d213d89981	Updating the Tmm Id for declaration of nflog capture mode	12 years ago
Victor Julien	b877cf6158	ssh: add json logger ... Sub module of eve-log, but can also run separately as ssh-json-log. Only one at a time though.	12 years ago
Ken Steele	497575d38e	Add option on Tile-Gx for logging for fast.log alerts over PCIe ... When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log to pcie, will open a connection over PCIe to a host application caleld tile-pcie-logd, that receives the alert strings and writes them to a file on the host. The file name to open is also passed over the PCIe link. This allows running Suricata on the TILEncore-Gx PCIe card, but have the alerts logged to the host system's file system efficiently. The PCIe API that is used is the Tilera Packet Queue (PQ) API which can access PCIe from User Space, thus avoiding system calls. Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE specific PCIe logging functionality. Using Write() and Close() function pointers in LogFileCtx, which default to standard write and close for files and sockets, but are changed to PCIe write and close functions when a PCIe channel is openned for logging. Moved Logging contex out of tm-modules.h into util-logopenfile.h, where it makes more sense. This required including util-logopenfile.h into a couple of alert-*.c files, which previously were getting the definitions from tm-modules.h. The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd. By default, the file name for fast.log specified in suricata.yaml is used as the filename on the host. An optional argument to tile-pcie-logd, --prefix=, can be added to prepend the supplied file path. For example, is the file in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp", then the file will be written to "/tmp/var/log/fast.log". Check for TILERA_ROOT environment variable before building tile_pcie_logd Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries and API header files. Configure now checs for TILERA_ROOT before enabling builing tile_pcie_logd in contrib/tile_pcie_logd	12 years ago
Victor Julien	f902c9e6c7	threading: add missing json types to TmModuleTmmIdToString ... Also, remove 'default' case from switch statement, so missing entries will lead to a compiler warning.	12 years ago
Victor Julien	c36a8d30e5	threading: add utility to get module id from name ... Add TmModuleGetIdByName to go from module name to id directly.	12 years ago
Victor Julien	52c3d3ad7c	log api: convert all names to const ... Instead of strdupping all names w/o a need, use const ptrs.	12 years ago
Victor Julien	9ff6608668	Introduce Filedata Logger API ... A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (*FiledataLogger)(ThreadVars *, void *thread_data, const Packet *, const File *, const FileData *, uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work.	12 years ago
Victor Julien	4c024f9658	profiling: add logger api labels	12 years ago
Victor Julien	9faa4b740d	Add --unittests-coverage option to list how many code modules have tests	13 years ago
Victor Julien	8e01cba85d	DNS TCP and UDP parser and DNS response logger	13 years ago
Anoop Saldanha	b787da5643	Remove all cuda related code in the engine except for the cuda api wrappers	13 years ago
Victor Julien	82769a1b37	profiling: fix missing profile names	13 years ago
Matt Keeler	37e3de8425	Refactor Napatech 3GD to just Napatech as Suricata is only going to support 3GD. ... Signed-off-by: Matt Keeler <mk@npulsetech.com>	13 years ago
Matt Keeler	5786a32d0f	Remove Napatech 2GD support ... Removed the Napatech 2GD support runmode-napatech-3gd.c had an include from runmode-napatech.h which was erroneous and has been removed as well. Signed-off-by: Matt Keeler <mk@npulsetech.com>	13 years ago
Matt Keeler	844e4dba11	Napatech 3GD Support ... For use with Network Cards from Napatech utilizing the 3GD driver/api. - Implemented new run modes in runmode-napatech-3gd.* - Implemented capture/decode threads in source-napatech-3gd.* - Integrated the new run modes and source into the build infrastructure. New configure switches --enabled-napatech-3gd : Turns on the NT 3GD support --with-napatech-3gd-includes : The directory containing the NT 3GD header files --with-napatech-3gd-libraries : The directory containing the NT 3GD libraries to link against. New CLI switch --napatech-3gd : Uses the Napatech 3GD run mode Runmodes Supported: - auto - autofp - workers Notes: - tested with 1 Gbps sustained traffic (no drops) Signed-off-by: Matt Keeler <mk@npulsetech.com>	13 years ago
Victor Julien	979edf0b97	Add way to profile mutex/spin locks per thread module.	14 years ago
Victor Julien	1d9f6ff8f2	Initial Napatech support by Randy Caldejon / nPulse.	14 years ago
Victor Julien	effe01ae7b	Add Init and DeInit calls to the thread module API.	14 years ago
Eric Leblond	3944357058	Remove unified related enum. ... This patch removes the enum related to unified1 output.	14 years ago
Anoop Saldanha	a844eecb0e	- Updated all runmodes to use synchronization points, right before each thread(slot function) tries to de-init the thread. - Main thread now first disables receive thread(s) before it kills receive and rest of the threads.	14 years ago
Eric Leblond	834c91eece	af-packet: add AFP to per packet performance system.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
William Metcalf	2eef905c07	GPL and Copyright header updates.	16 years ago
Gerardo Iglesias Galvan	9f4fae5b1a	Fix inconsistent use of dynamic memory allocation	16 years ago
William Metcalf	ce01927515	Import of GPLv2 Header 050410	16 years ago
Victor Julien	655b6acad9	Flush fp to be sure before closing.	16 years ago
Victor Julien	3c7027687f	Fix logging api not cleaning up LogFileCtx mutex.	16 years ago
Victor Julien	8f0fe76346	Minor dbg output formatting fix.	16 years ago
Gurvinder Singh	4879045c6f	bug 102 patch	16 years ago
Pablo Rincon	25a3a5c6d8	Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks.	16 years ago
Gurvinder Singh	fea277b2aa	memory leak fixes	16 years ago
Jason Ish	844c444af1	Use the configuration file to setup alert logging (and http logging). ... Only setup for the live pcap modes at the moment.	16 years ago
Victor Julien	b7bac14040	Fixup code to compile with -Wall -Werror -Wextra -Wno-unused-parameter compiler options.	16 years ago
Victor Julien	ecf86f9c23	Rename to Suricata.	16 years ago
Pablo Rincon	e26833be3f	Changing mutex/spinlocks/conditions naming types	16 years ago
Pablo Rincon	769022f4be	Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests	16 years ago