This patch handles the end of AF_PACKET socket support work. It
provides conditional compilation, autofp and single runmode.
It also adds a 'defrag' option which is used to activate defrag
support in kernel to avoid rx_hash computation in flow mode to fail
due to fragmentation.
This patch contains some fixes by Anoop Saldanha, and incorporate
change following review by Anoop Saldanha and Victor Julien.
AF_PACKET support is only build if the --enable-af-packet flag is
given to the configure command line. Detection of code availability
is also done: a check of the existence of AF_PACKET in standard
header is done. It seems this variable is Linux specific and it
should be enough to avoid compilation of AF_PACKET support on other
OSes.
Compilation does not depend on up-to-date headers on the system. If
none are present, wemake our own declaration of FANOUT variables. This
will permit compilation of the feature for system where only the kernel
has been updated to a version superior to 3.1.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.
Stats per packet include:
1) total ticks spent
2) ticks spent per individual thread module
3) "threading overhead" which is simply calculated by subtracting (2) of (1).
A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.
Code depends on --enable-profiling just like the rule profiling code.
New yaml parameters:
profiling:
# packet profiling
packets:
# Profiling can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: yes
filename: packet_stats.log
append: yes
# per packet csv output
csv:
# Output can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: no
filename: packet_stats.csv
Example output of summary stats:
IP ver Proto cnt min max avg
------ ----- ------ ------ ---------- -------
IPv4 6 19436 11448 5404365 32993
IPv4 256 4 11511 49968 30575
Per Thread module stats:
Thread Module IP ver Proto cnt min max avg
------------------------ ------ ----- ------ ------ ---------- -------
TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770
TMM_DETECT IPv4 6 19436 1107 137241 1504
TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155
TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138
TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154
TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944
TMM_STREAMTCP IPv4 6 19434 828 1299159 19438
The proto 256 is a counter for handling of pseudo/tunnel packets.
Example output of csv:
pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337
First line of the file contains labels.
2 example gnuplot scripts added to plot the data.
This patch adds a new mode for NFQ inline mode. The idea is to
simulate a non final NFQUEUE rules.
This permit to do send all needed packets to suricata via a simple
FORWARD rule:
iptables -I FORWARD -m mark ! --mark $MARK/$MASK -j NFQUEUE
And below, we have a standard filtering ruleset.
To do so, suricata issues a NF_REPEAT instead of a NF_ACCEPT verdict and
put a mark ($MARK) with respect to a mask ($MASK) on the handled packet.
NF_REPEAT verdict has for effect to have the packet reinjected at start
of the hook after the verdict. As it has been marked by suricata during
the verdict it will not rematch the initial rules and make his way to
the following classical ruleset.
Mode, mark and mask can be configured via suricata.yaml file with the
following syntax:
nfq:
repeat_mode: (false|true)
mark: $MARK
mask: $MASK
Default is false to preserve backward compatibility.
Signed-off-by: Eric Leblond <eric@regit.org>
This patch adds support for multiple Netfilter queue
in the NFQ run mode. Suricata can now be started on
multiple queue by using a comma separated list of
queue identifier on the command line. The following syntax:
suricata -q 0 -q 1 -c /opt/suricata/etc/suricata.yaml
will start a suricata listening to Netfilter queue 0 and 1.
Signed-off-by: Eric Leblond <eric@regit.org>
Timer in the main loop was of 100 usec. This patch increases it
to 10 ms which should be a reasonnable delay to declare some threads
dead.
Signed-off-by: Eric Leblond <eric@regit.org>
If default-packet-size is not set, it is possible in some case to
guess a correct value.
If PCAP or PF_RING are used we are linked to a "physical" interface.
Thus, it is possible to get information about the link MTU and
hardware header size. This give us the ability to auto discover a
decent default-packet-size.
If suricata is running under a different running-mode, it will
default to 1514.
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
This patch modifies decode.c and decode.h to avoid the usage
by default of a bigger than 65535 bytes array in Packet structure.
The idea is that the packet are mainly under 1514 bytes size and
a bigger size must be supported but should not be the default.
If the packet length is bigger than DFLT_PACKET_SIZE then the
data are stored in a dynamically allocated part of the memory.
To ease the modification of the rest of the code, functions to
access and set the payload/length in a Packet have been introduced.
The default packet size can be set at runtime via the default-packet-size
configuration variable.
Use the --dag <dagname> cmd line option to specify from which DAG card to read pkts
from.
Issue at the moment with pkts being ejected during shutdown -- at the moment we
ignore any packets that are not of link type Ethernet.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:
- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.
This commit fixes bug #124.
Add support for reporting alerts to the Prelude SIEM system, using
libprelude to send IDMEF (RFC4765) messages.
Each message contains the alert description and reference (using
the SID/GID), and a normalized description (assessment, impact,
sources etc.)
libprelude handles the connection with the manager (collecting component),
spooling and sending the event asynchronously. It also offers transport
security (using TLS and trusted certificates) and reliability (events
are retransmitted if not sent successfully).
This modules requires a Prelude profile to work (see man prelude-admin
and the Prelude Handbook for help).
Signed-off-by: Pierre Chifflier <chifflier@edenwall.com>
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.