To reload ruleset during engine runtime, send the USR2 signal to the engine, and the ruleset would be reloaded from the same yaml file supplied at engine startup
Simply added the -T to be printed out when suricata is run without any
arguments. The capability to test a configuration file has been in
suricata for some time, just doesn't show up as an option right now.
Add profiling per lock location in the code. Accounts how often a
lock is requested, how often it was contended, the max number of
ticks spent waiting for it, avg number of ticks waiting for it and
the total ticks for that location.
Added a new configure flag --enable-profiling-locks to enable this
feature.
Add a host table similar to the flow table. A hash using fine grained
locking. Flow manager for now takes care of book keeping / garbage
collecting.
Tag subsystem now uses this for host based tagging instead of the
global tag hash table. Because the latter used a global lock and the
new code uses very fine grained locking this patch should improve
scalability.
Major redesign of the flow engine. Remove the flow queues that turned
out to be major choke points when using many threads. Flow manager now
walks the hash table directly. Simplify the way we get a new flow in
case of emergency.
The http_server_body content modifier modifies the previous content to inspect
the normalized (dechunked, unzipped) http_server_body. The workings are similar
to http_client_body. Additionally, a new pcre flag was introduced "/S".
To facilitate this change the signature flags field was changed to be 64 bit.
This patch modifies LiveBuildQueueList name to LiveBuildDeviceList
to have a consistent naming accross function. It also adds a
doxygen comment to add author and description of util-device.c
file.
This patch adds a new alert format called pcap-info. It aims at
providing an easy to parse one-line per-alert format containing
the packet id in the parsed pcap for each alert. This permit to
add information inside the pcap parser.
This format is made to be used with suriwire which is a plugin for
wireshark. Its target is to enable the display of suricata results
inside wireshark.
This format doesn't use append mode per default because a clean file
is needed to operate with wireshark.
The format is a list of values separated by ':':
Packet number:GID of matching signature:SID of signature:REV of signature:Flow:To Server:To Client:0:0:Message of signature
The two zero are not yet used values. Candidate for usage is the
part of the packet that matched the signature.
This patch adds a --pcap option which can be used to select or
an interface if an argument is provided or the interfaces defined
in the configuration file.
This patch convert pfring to pktacqloop and use the new factorisation
function. This also fixes commmand line parsing of pfring which is now
able to work like af-packet:
- 'suricata -c s.yaml --pfring' start suricata with all interfaces in
conf
- 'suricata -c s.yaml --pfring=eth2' start suricata on eth2
This patch changes the option name. af-packet long option is
now used instead of -a to mimic pfring behaviour.
This patch improves the standard parsing of the command line.
Running
suricata -c suricata.yaml --af-packet
will start a suricata running in AF_PACKET mode listening on all
interfaces defined in the suricata.yaml configuration file. The
traditionnal syntax:
suricata -c suricata.yaml --af-packet=ppp0
will start a suricata listening on ppp0 only.
This patch adds multi interface support to AF_PACKET. A structure
is used at thread creation to give all needed information to the
input module. Parsing of the options is done in runmode preparation
through a dedicated function which return the configuration in a
structure usable by thread creation.
This patch handles the end of AF_PACKET socket support work. It
provides conditional compilation, autofp and single runmode.
It also adds a 'defrag' option which is used to activate defrag
support in kernel to avoid rx_hash computation in flow mode to fail
due to fragmentation.
This patch contains some fixes by Anoop Saldanha, and incorporate
change following review by Anoop Saldanha and Victor Julien.
AF_PACKET support is only build if the --enable-af-packet flag is
given to the configure command line. Detection of code availability
is also done: a check of the existence of AF_PACKET in standard
header is done. It seems this variable is Linux specific and it
should be enough to avoid compilation of AF_PACKET support on other
OSes.
Compilation does not depend on up-to-date headers on the system. If
none are present, wemake our own declaration of FANOUT variables. This
will permit compilation of the feature for system where only the kernel
has been updated to a version superior to 3.1.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.
Stats per packet include:
1) total ticks spent
2) ticks spent per individual thread module
3) "threading overhead" which is simply calculated by subtracting (2) of (1).
A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.
Code depends on --enable-profiling just like the rule profiling code.
New yaml parameters:
profiling:
# packet profiling
packets:
# Profiling can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: yes
filename: packet_stats.log
append: yes
# per packet csv output
csv:
# Output can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: no
filename: packet_stats.csv
Example output of summary stats:
IP ver Proto cnt min max avg
------ ----- ------ ------ ---------- -------
IPv4 6 19436 11448 5404365 32993
IPv4 256 4 11511 49968 30575
Per Thread module stats:
Thread Module IP ver Proto cnt min max avg
------------------------ ------ ----- ------ ------ ---------- -------
TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770
TMM_DETECT IPv4 6 19436 1107 137241 1504
TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155
TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138
TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154
TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944
TMM_STREAMTCP IPv4 6 19434 828 1299159 19438
The proto 256 is a counter for handling of pseudo/tunnel packets.
Example output of csv:
pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337
First line of the file contains labels.
2 example gnuplot scripts added to plot the data.
This patch adds a new mode for NFQ inline mode. The idea is to
simulate a non final NFQUEUE rules.
This permit to do send all needed packets to suricata via a simple
FORWARD rule:
iptables -I FORWARD -m mark ! --mark $MARK/$MASK -j NFQUEUE
And below, we have a standard filtering ruleset.
To do so, suricata issues a NF_REPEAT instead of a NF_ACCEPT verdict and
put a mark ($MARK) with respect to a mask ($MASK) on the handled packet.
NF_REPEAT verdict has for effect to have the packet reinjected at start
of the hook after the verdict. As it has been marked by suricata during
the verdict it will not rematch the initial rules and make his way to
the following classical ruleset.
Mode, mark and mask can be configured via suricata.yaml file with the
following syntax:
nfq:
repeat_mode: (false|true)
mark: $MARK
mask: $MASK
Default is false to preserve backward compatibility.
Signed-off-by: Eric Leblond <eric@regit.org>
This patch adds support for multiple Netfilter queue
in the NFQ run mode. Suricata can now be started on
multiple queue by using a comma separated list of
queue identifier on the command line. The following syntax:
suricata -q 0 -q 1 -c /opt/suricata/etc/suricata.yaml
will start a suricata listening to Netfilter queue 0 and 1.
Signed-off-by: Eric Leblond <eric@regit.org>
Timer in the main loop was of 100 usec. This patch increases it
to 10 ms which should be a reasonnable delay to declare some threads
dead.
Signed-off-by: Eric Leblond <eric@regit.org>
If default-packet-size is not set, it is possible in some case to
guess a correct value.
If PCAP or PF_RING are used we are linked to a "physical" interface.
Thus, it is possible to get information about the link MTU and
hardware header size. This give us the ability to auto discover a
decent default-packet-size.
If suricata is running under a different running-mode, it will
default to 1514.
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
This patch modifies decode.c and decode.h to avoid the usage
by default of a bigger than 65535 bytes array in Packet structure.
The idea is that the packet are mainly under 1514 bytes size and
a bigger size must be supported but should not be the default.
If the packet length is bigger than DFLT_PACKET_SIZE then the
data are stored in a dynamically allocated part of the memory.
To ease the modification of the rest of the code, functions to
access and set the payload/length in a Packet have been introduced.
The default packet size can be set at runtime via the default-packet-size
configuration variable.
Use the --dag <dagname> cmd line option to specify from which DAG card to read pkts
from.
Issue at the moment with pkts being ejected during shutdown -- at the moment we
ignore any packets that are not of link type Ethernet.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:
- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.
This commit fixes bug #124.
Add support for reporting alerts to the Prelude SIEM system, using
libprelude to send IDMEF (RFC4765) messages.
Each message contains the alert description and reference (using
the SID/GID), and a normalized description (assessment, impact,
sources etc.)
libprelude handles the connection with the manager (collecting component),
spooling and sending the event asynchronously. It also offers transport
security (using TLS and trusted certificates) and reliability (events
are retransmitted if not sent successfully).
This modules requires a Prelude profile to work (see man prelude-admin
and the Prelude Handbook for help).
Signed-off-by: Pierre Chifflier <chifflier@edenwall.com>
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.