126 Commits (6608e7f523f67da927c6ed1c56d7f2dd1d04b4e5)

Author	SHA1	Message	Date
Victor Julien	7edcc13514	NFQ: fix packets not getting freed	13 years ago
Victor Julien	9f3e2f7a92	NFQ: adapt to ReleasePacket API	13 years ago
Eric Leblond	0e92469222	nfq: be sure to always verdict packets ... To be sure to always verdict packets (bug #769), this patch adds a ReleaseData function to NFQ packets. The release function simply drop the packet if it has not been verdicted before.	13 years ago
Eric Leblond	3f107fa130	decode: Packet action start with PACKET ... Rename all Packet action macro to have them prefixed by PACKET.	13 years ago
Eric Leblond	75cd1f6096	Coverity 1038515: check function return ... This is more cosmetic than useful but it is cleaner anyway.	13 years ago
Eric Leblond	a35c367942	action handling: use macro for test. ... Use test macro instead of direct access to action field. This patch has been obtained by using the following spatch file: @@ Packet *p; expression E; @@ - p->action & E + TEST_PACKET_ACTION(p, E)	13 years ago
Victor Julien	53fe756798	NFQ: convert batchcount related yaml errors to warnings.	13 years ago
Eric Leblond	703e5848e4	nfq: add errno display when verdict fail ... In case of error, errno is set by sendmsg which is called by nfnetlink and which is called by libnetfilter_queue. This patch displays the string expression of errno if verdict has failed.	13 years ago
Florian Westphal	8da02115c9	nfq: add support for batch verdicts ... Normally, there is one verdict per packet, i.e., we receive a packet, process it, and then tell the kernel what to do with that packet (eg. DROP or ACCEPT). recv(), packet id x send verdict v, packet id x recv(), packet id x+1 send verdict v, packet id x+1 [..] recv(), packet id x+n send verdict v, packet id x+n An alternative is to process several packets from the queue, and then send a batch-verdict. recv(), packet id x recv(), packet id x+1 [..] recv(), packet id x+n send batch verdict v, packet id x+n A batch verdict affects all previous packets (packet_id <= x+n), we thus only need to remember the last packet_id seen. Caveats: - can't modify payload - verdict is applied to all packets - nfmark (if set) will be set for all packets - increases latency (packets remain queued by the kernel until batch verdict is sent). To solve this, we only defer verdict for up to 20 packets and send pending batch-verdict immediately if: - no packets are currently queue - current packet should be dropped - current packet has different nfmark - payload of packet was modified This patch adds a configurable batch verdict support for workers runmode. The batch verdicts are turned off by default. Problem is that batch verdicts only work with kernels >= 3.1, i.e. using newer libnetfilter_queue with an old kernel means non-working suricata. So the functionnality has to be disabled by default.	13 years ago
Florian Westphal	6678c9feb9	nfq: avoid extra copy when running in workers mode ... currently, the packet payload recv()d from the nfqueue netlink socket is copied into a new packet buffer. This is required because the recv-buffer space used is tied to the current thread, but a packet may be handed off to other threads, and the recv-buffer can be re-used while the packet is handled by another thread. However, in worker runmode, the packet will always be handled by the current thread, and the recv-buffer will only be reused after the entire packet processing stack is done with the packet. Thus, in worker runmode, we can avoid the copy and assign the packet data area directly.	13 years ago
Victor Julien	472e061c6d	build: more checking for includes	13 years ago
Eric Leblond	e3a38810b6	nfq: suppress non loop receive function	13 years ago
Anoop Saldanha	b33986c887	Add a packet src for every packet generated inside suricata.	13 years ago
Eric Leblond	19756488ab	nfq: close the queue when leaving acquisition. ... This patch adds a call to close the queue when the acquisition loop is ending. This way the incoming packets will be accepted during all the shutdown phase (if the queue-bypass option of NFQUEUE is used). At the same time the currently processed packets will be dropped but the time scale are different: suricata will drop 20 ms of packets and the shutdown can take 0.5 seconds. Patch based on an idea of Victor Julien.	13 years ago
Eric Leblond	6303b5d987	SC_LOG_ERROR is not an error.	13 years ago
Eric Leblond	2c57275921	nfq: implement "fail-open" support. ... On linux >= 3.6, you can use the fail-open option on a NFQ queue to have the kernel accept the packet if userspace is not able to keep pace. Please note that the kernel will not trigger an error if the feature is activated in userspace libraries but not available in kernel. This patch implements the option for suricata by adding a nfq.fail-open configuration variable which is desactivated by default.	13 years ago
Eric Leblond	a3465fb971	Rename 'worker' running mode to 'workers' ... This patch renamed the 'worker' running mode into 'workers'. Thus, there is only one name in Suricata for the same thing. Backward compatibility is ensured by replacing "worker" by "workers" when the old name is used. A warning is printed in the log when the old name is used.	14 years ago
Anoop Saldanha	bc6cf43840	#482 - use decode_flag for all decode TMs. Use the flag as a way to retrieve decode TMs from ThreadVars	14 years ago
Anoop Saldanha	cd4705e699	flag recieve acq tms that previously missed the receive_tm flag	14 years ago
Victor Julien	4ebb6b7fae	nfq: switch locking code to macro's to lock profiling can track the exact lock locations.	14 years ago
Nikolay Denev	139768dd58	Do not use underscored config vars internally.	14 years ago
Eric Leblond	7fb78a0ff6	Fix compilation warning.	14 years ago
Eric Leblond	58855494c1	nfq: do not use mutex in 'worker' mode ... Using a mutex on the queue handle is not necessary in 'worker' mode as there is no concurrent access to it.	14 years ago
Eric Leblond	c1ad64b333	ips: update copyright date and author list.	14 years ago
Eric Leblond	acc9634106	nfq: add some comments about possible evolution	14 years ago
Eric Leblond	9ca7257279	nfq: suppress unused functions.	14 years ago
Eric Leblond	aee2e3ddd6	nfq: Add autofp mode support	14 years ago
Eric Leblond	115c3499d2	nfq: factorize auto mode	14 years ago
Eric Leblond	830ca7c2c8	source-nfq: suppress insecable space. ... This patch supresses an insecable space and fixes an indentation.	14 years ago
Eric Leblond	bbd04fde30	NFQ: fix race condition at exit. ... A race condition was observed when leaving NFQ. This was caused by the queue handle being accessed after been nullified. This patch uses the handle mutex to protect the destruction and adds tests on nullity to avoid crashed.	14 years ago
Anoop Saldanha	3f1c4efceb	Add new flags var to tm module. TMs can now set flags to identify special properties. Also use these to identify receive TMs	14 years ago
Victor Julien	c590bba4a4	Undo tunnel reference counting using atomic operations. Revert to mutex based code.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Eric Leblond	a354034cfc	nfq: Fix deinit phase ... If receive thread is failling, we need to restart it but the code was not restarting the queue (this was done in verdict thread).	14 years ago
Eric Leblond	eddcedba0a	nfq: make thread abort if NFQ verdict fail	14 years ago
Eric Leblond	2ffcef0a8e	nfq: Add iterator on nfq_set_verdict ... This patch adds retry to nfq_set_verdict in case of error.	14 years ago
Eric Leblond	bc68c108a7	NFQ: use per thread allocated data for recv buffer.	15 years ago
Victor Julien	52eb8d2be0	Convert mutex protected tunnel counters to lockless atomic counters.	15 years ago
Eric Leblond	ee6552f25e	nfq_set_mark: handle feature in NFQ. ... This patch implements the nfq_set_mark related modification of verdict handling.	15 years ago
Eric Leblond	04f2afa81b	nfq: fix exit function ... Exit function was trying to close the nfq handler even if it was null. This was causing a crash.	15 years ago
Victor Julien	477bc1d050	Set DROP flag on a packet in addition to the REJECT flags. This makes sure we not only send a reject, but also drop the offending packet. Closes #248.	15 years ago
Victor Julien	d424ac7c61	Fix nfq lockup due to improper handling of PKT_PSEUDO_STREAM_END packets.	15 years ago
Victor Julien	c9f9e3f9a4	Add configure check for signed or unsigned nfq_get_payload, adapt code.	15 years ago
Eric Leblond	c74116949c	source-nfq: improve nfq option system ... This patch modifies the NFQ option system to avoid implicit choice. 'nfq.mode' is now a string which can take a value in the 'accept', 'repeat' and 'route' set. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	94596ff282	source-nfq: add queue redirect support ... This patch adds the support of queue redirect. If 'next_queue' variable is set, the verdict sent to kernel is modified to contain the indication of a queue number (equal to 'next_queue') which will receive the packet after the verdict. This feature can be used to chain easily tools using NFQUEUE. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	aded7b4fae	source-nfq: add detection of already treated packet. ... This patch adds detection of already treated packet. If a packet is coming with an already set mark, it will be accepted and the processing of the packet is aborted. The patch display a message when the problem occurs but the number of message is limited to a fix counter in a way to avoid flooding the log. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	e399e74fc1	source-nfq: Factorize buffer usage ... A big sized buffer was allocated at each packet parsing. This patch uses a per-thread variable to have a persistent memory usage. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	1e600c1054	source-nfq: add simulated non-terminal NFQUEUE verdict ... This patch adds a new mode for NFQ inline mode. The idea is to simulate a non final NFQUEUE rules. This permit to do send all needed packets to suricata via a simple FORWARD rule: iptables -I FORWARD -m mark ! --mark $MARK/$MASK -j NFQUEUE And below, we have a standard filtering ruleset. To do so, suricata issues a NF_REPEAT instead of a NF_ACCEPT verdict and put a mark ($MARK) with respect to a mask ($MASK) on the handled packet. NF_REPEAT verdict has for effect to have the packet reinjected at start of the hook after the verdict. As it has been marked by suricata during the verdict it will not rematch the initial rules and make his way to the following classical ruleset. Mode, mark and mask can be configured via suricata.yaml file with the following syntax: nfq: repeat_mode: (false|true) mark: $MARK mask: $MASK Default is false to preserve backward compatibility. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	72ec56ab23	source-nfq: autodetection of queue max length function ... Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Eric Leblond	1375e90030	Prepare multi queue support in NFQ ... This patch prepare support for multiqueue in the source file. The NFQ vars contained in Packet structure has a new member. It is a reference to the NFQ thread var it comes from. The behaviour is modified as a single verdict thread treat packet for all Netfilter queues. Locking is done in the verdict function to ensure that simultaneous modifications of counters can not occur. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago