Suricata was inconditionaly dropping packets that are invalid with
respect to the streaming engine. In some corner case like asymetric
trafic capture, this was leading to dropping some legitimate trafic.
The async-oneside option did help but this was not perfect in some
real life case. So this patch introduces an option that allow the
user to tell Suricata not to drop packet that are invalid with
respect to streaming.
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
We assume session resumption has occurred if the Client Hello message
included a session id, we have not seen the server certificate, but
we have seen a Change Cipher Spec message from the server.
Previously, these transactions were not logged at all because the
server cert was never seen.
Ticket: https://redmine.openinfosecfoundation.org/issues/1969
eve: detects libevent for async redis at configure
eve: moves redis output code to new file - util-log-redis.{c,h}
eve: redis ECHO and QUIT commands for async mode
eve: redis output defaults if conf is missing
Give unified2 a nostamp option which will create the file
without the timestamp suffix (like Snort's nostamp option).
Also register for rotation notification on SIGHUP so the file
will be recreated if it is removed by an external rotation
program (only when nostamp is used).
If segments section in the yaml is ommitted (default) or when the
pool size is set to 'from_mtu', the size of the pool will be MTU
minus 40. If the MTU couldn't be determined, it's assumed to be
1500, so the segment size for the bool will be 1460.
The code already looks for "dir" first instead of
"squil-base-dir", and already respects this configuration
parameter in other modes than the "sguil" mode.
Coda will still access "sguil-base-dir".
When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.
This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
Some protocol like modbus requires
a infinite stream depth because session
are kept open and we want to analyze everything.
Since we have a stream reassembly depth per stream,
we can also set a stream reassembly depth per proto.
Add support for the ENIP/CIP Industrial protocol
This is an app layer implementation which uses the "enip" protocol
and "cip_service" and "enip_command" keywords
Implements AFL entry points
This adds a new timeout value for local bypassed state. For user
simplication it is called only `bypassed`. The patch also adds
a emergency value so we can clean bypassed flows a bit faster.
Add a generic 'capture' section to the YAML:
# general settings affecting packet capture
capture:
# disable NIC offloading. It's restored when Suricata exists.
# Enabled by default
#disable-offloading: false
#
# disable checksum validation. Same as setting '-k none' on the
# commandline
#checksum-validation: none
When running in live mode, the new default 'auto' value of
unix-command.enabled causes unix-command to be activated. This
will allow users of live capture to benefit from the feature and
result in no side effect for user running in offline capture.
- drop:
alerts: yes # log alerts that caused drops
flows: all # start or all: 'start' logs only a single drop
# per flow direction. All logs each dropped pkt.
Update the code to use mmap capture by default even in unset in
configuration file. mmap capture is now be turned off by using
explicitely 'use-mmap: no' in configuration.
It is used to set the block size in tpacket_v3. It will allow user
to tune the capture depending on his bandwidth.
Default block size value has been updated to a bigger value to
allow more efficient wlak on block.
This patch adds a basic implementation of AF_PACKET tpacket v3. It
is basic in the way it is only working for 'workers' runnning mode.
If not in 'workers' mode there is a fallback to tpacket_v2. Feature
is activated via tpacket-v3 option in the af-packet section of
Suricata YAML.
Make the rule grouping dump to rule_group.json configurable.
detect:
profiling:
grouping:
dump-to-disk: false
include-rules: false # very verbose
include-mpm-stats: false
Make the port grouping whitelisting configurable. A whitelisted port
ends up in it's own port group.
detect:
grouping:
tcp-whitelist: 80, 443
udp-whitelist: 53, 5060
No portranges are allowed at this point.
Instead of detect-engine which used a list for no good reason, use a
simple map now.
detect:
profile: medium
custom-values:
toclient-groups: 3
toserver-groups: 25
sgh-mpm-context: auto
inspection-recursion-limit: 3000
# If set to yes, the loading of signatures will be made after the capture
# is started. This will limit the downtime in IPS mode.
#delayed-detect: yes
Dump a json record containing all sigs that need to be inspected after
prefilter. Part of profiling. Only dump if threshold is met, which is
currently set by:
--set detect.profiling.inspect-logging-threshold=200
A file called packet_inspected_rules.json is created in the default
log dir.
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.
Logs SGH id so it can be compared with the rule_group.json output.
Implemented both in a human readable text format and a JSON format.
It is better to use a transformation to define the default
directory of output message instead of using an hardcoded value.
Same apply to the directory for the pid file.
This patch implements redis pipelining. This consist in contacting
the redis server every N events to minimize the number of TCP
exchange. This is optional and setup via the configuration file.
When using redis output, we are loosing the host key (added by
logstash or logstash-forwarder) and we can't find anymore what
Suricata did cause the alert.
This patch is adding this key during message generation using the
'sensor-name' variable or the hostname is 'sensor-name' is not
defined.