Add a raw-extraction option for smtp. When enabled, this feature will
store the raw e-mail inside a file, including headers, e-mail content,
attachments (base64 encoded). This content is stored in a normal File *,
allowing for normal file detection.
It'd also allow for all-emails extraction if a rule has
detect-filename:"rawmsg" matcher (and filestore).
Note that this feature is in contrast with decode-mime.
This feature is disabled by default, and will be disabled automatically
if decode-mime is enabled.
Add support for community flow id, meant to give a records a
predictable flow id that can be used to match records to
output of other tools.
Takes a 'seed' that needs to be same across sensors and tools
to make the id less predictable.
Remove 'experimental' label for Rust, and enable it by default if
rustc and cargo (and libjansson) are available.
Add rustc and cargo versions to the build-info.
Move the rule file configuration down near the bottom of the
configuration file under advanced settings. With the bundling
of Suricata-Update, any rule file configuration within
suricata.yaml could be considered advanced.
Add extra comments to the yaml to make it more clear which was
enabled at installation time.
This is a DHCP decoder and logger written in Rust. Unlike most
parsers, this one is stateless so responses are not matched
up to requests by Suricata. However, the output does contain
enough fields to match them up in post-processing.
Rules are included to alert of malformed or truncated options.
XFF configuration is already set in app-layer-htp-xff, and in
output-json-alert. Extending XFF configuration to files and HTTP allow
to get the same behavior as for alerts.
Extend the configuration of filestore json to let filestore metafile
dump be aware of xff. This is available only if write-fileinfo is set
to yes and file-store version is 2.
Introduces the option 'outputs.pcap-log.compression' which can be set
to 'none' or 'lz4', plus options to set the compression level and to
enable checksums. SCFmemopen is used to make pcap_dump() write to a
buffer which is then compressed using liblz4.
If suricata-update is bundled, set the default-rule-dir
to lib/suricata/rules under the $localstatedir
For now use 2 rule-files section that are renamed depending
on if suricata-update is bundled or not.
Don't treat 'external' parsers as more experimental. All parsers
depend on crates to some extend, and all have C glue code. So the
distinction doesn't really make sense.
Add a new parser for Internet Key Exchange version (IKEv2), defined in
RFC 7296.
The IKEv2 parser itself is external. The embedded code includes the
parser state and associated variables, the state machine, and the
detection code.
The parser looks the first two messages of a connection, and analyzes
the client and server proposals to check the cryptographic parameters.
This patch adds a new configuration for dns,
introducing a "version" that permits to switch
between the new and old format to provide
backward compatibility.
The new configuration is made up of these new fields:
- version
- requests (query)
- response (answer)
- types (custom)
Implement SMB app-layer parser for SMB1/2/3. Features:
- file extraction
- eve logging
- existing dce keyword support
- smb_share/smb_named_pipe keyword support (stickybuffers)
- auth meta data extraction (ntlmssp, kerberos5)
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
This patch introduces the ebpf cluster mode. This mode is using
an extended BPF function that is loaded into the kernel and
provide the load balancing.
An example of cluster function is provided in the ebpf
subdirectory and provide ippair load balancing function.
This is a function which uses the same method as
the one used in autofp ippair to provide a symetrical
load balancing based on IP addresses.
A simple filter example allowing to drop IPv6 is added to the
source.
This patch also prepares the infrastructure to be able to load
and use map inside eBPF files. This will be used later for flow
bypass.
Under eve/alert, introduce a new metadata configuration
section. If no provided, or simply yes defaults will be used.
Otherwise this a map with fields that can be toggled on and
off. The defaults are:
outputs:
- eve-log:
types:
- alert:
metadata:
app-layer: true
flow: true
rule:
raw: false
metadata: true
To enable something that is disabled by default, or to disable
something that is enabled by default, only that key need to
be changed, everything else will keep its default value.
For SIEM analysis it is often useful to refer to the actual rules to
find out why a specific alert has been triggered when the signature
message does not convey enough information.
Turn on the new rule flag to include the rule text in eve alert output.
The feature is turned off by default.
With a rule like this:
alert dns $HOME_NET any -> 8.8.8.8 any (msg:"Google DNS server contacted"; sid:42;)
The eve alert output might look something like this (pretty-printed for
readability):
{
"timestamp": "2017-08-14T12:35:05.830812+0200",
"flow_id": 1919856770919772,
"in_iface": "eth0",
"event_type": "alert",
"src_ip": "10.20.30.40",
"src_port": 50968,
"dest_ip": "8.8.8.8",
"dest_port": 53,
"proto": "UDP",
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 42,
"rev": 0,
"signature": "Google DNS server contacted",
"category": "",
"severity": 3,
"rule": "alert dns $HOME_NET any -> 8.8.8.8 any (msg:\"Google DNS server contacted\"; sid:43;)"
},
"app_proto": "dns",
"flow": {
"pkts_toserver": 1,
"pkts_toclient": 0,
"bytes_toserver": 81,
"bytes_toclient": 0,
"start": "2017-08-14T12:35:05.830812+0200"
}
}
Feature #2020
Metadata of the signature can now conditionaly put in the alert
events. This will allow user to get more context about the events
generated by the alert.
detect-metadata: conditional parsing
Only parses metadata if an output module will use the information.
Patch also adds a unittest to check metadata is not parsed if not
asked to.
output-json-alert: optional output keys as array
Update rule metadata configuration to have an option to output
value as array. Also adds an option to log only a series of keys
as array. This is useful in the case of some ruleset where from
instance the `tag` key is used multiple time.
(Jason Ish) rule metadata: always log as lists
After review of rule metadata, we can't make assumptions
on what should be a list or not. So log everything as a list.
By default log metadata.
Remove toggles for individual protocol types and just use a
single toggle to control including the app-layer with the
alert.
The metadata (currently app-layer and flow) can be disabled
by setting metadata to a falsey value, but its removed
from the default configuration (but wil be in docs)
As fileinfo records are logged to the main eve log, disable
metadata by default. But when enabled, just use the fileinfo
record.
Metadata is stored in a file named:
<sha256>.<seconds>.<file_id>.json
where the sha256 is the same as the file logged, the seconds
is the unix timestamp in seconds for the fileinfo record,
and the file_id is an atomically incremented integer per
Suricata instance.
This should allow for each occurrence of the same file to have
its own metadata file. But a collision is expected when running
Suricata repeatedly over the same pcap, as that would be the
exact same occurrence of a file.
Filestore v2 is starts as a copy of log-filestore with the
following changes.
- NSS is required as file names as based on the SHA256.
- Work/tmp files are stored in a temp. directory, then
moved into a directory tree where the directory names
are the first 2 characters of the hex SHA256.
- Removes the need for a waldo file or pid in the filenames.
This takes the form of an option to add the pid of the process to file
names. Additionally, it adds a suffix to the file name to indicate it is
not finalized.
Adding the pid to the file name reduces the likelihood that a file is
overwritten when suricata is unexpectedly killed. The number in the
waldo file is only written out during a clean shutdown. In the event
of an improper shutdown, extracted files will be written using the old
number and existing files with the same name will be overwritten.
Writes extracted files and their metadata to a temporary file suffixed
with '.tmp'. Renames the files when they are completely done being
written. As-is there is no way to know that a file on disk is still
being written to by suricata.
This patch adds support for hw bypass by enabling flow offload in the network
card (when supported) and implementing the BypassPacketsFlow callback.
Hw bypass support is disabled by default, and can be enabled by setting
"bypass: yes" in the pfring interface configuration section in suricata.yaml.
This adds a new redis mode rpush. Also more consistent config keywords orientated at the redis command: lpush and publish.
Keeping list and channel config keywords for backwards compatibility
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
Suricata was inconditionaly dropping packets that are invalid with
respect to the streaming engine. In some corner case like asymetric
trafic capture, this was leading to dropping some legitimate trafic.
The async-oneside option did help but this was not perfect in some
real life case. So this patch introduces an option that allow the
user to tell Suricata not to drop packet that are invalid with
respect to streaming.
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
We assume session resumption has occurred if the Client Hello message
included a session id, we have not seen the server certificate, but
we have seen a Change Cipher Spec message from the server.
Previously, these transactions were not logged at all because the
server cert was never seen.
Ticket: https://redmine.openinfosecfoundation.org/issues/1969
eve: detects libevent for async redis at configure
eve: moves redis output code to new file - util-log-redis.{c,h}
eve: redis ECHO and QUIT commands for async mode
eve: redis output defaults if conf is missing
Give unified2 a nostamp option which will create the file
without the timestamp suffix (like Snort's nostamp option).
Also register for rotation notification on SIGHUP so the file
will be recreated if it is removed by an external rotation
program (only when nostamp is used).
If segments section in the yaml is ommitted (default) or when the
pool size is set to 'from_mtu', the size of the pool will be MTU
minus 40. If the MTU couldn't be determined, it's assumed to be
1500, so the segment size for the bool will be 1460.
Maurizio Abba
Victor Julien
Jason Ish
Travis Green
Eric Leblond
jason taylor
Konstantin Klinger
Pierre Chifflier
Max Fillinger
Mats Klepsland
Giuseppe Longo
Andreas Herz
Martin Natano
Pascal Delalande
Gaurav Singh
Alfredo Cardigliano
Julian
Phil Young
Ray Ruvinskiy
fooinha