As fileinfo records are logged to the main eve log, disable
metadata by default. But when enabled, just use the fileinfo
record.
Metadata is stored in a file named:
<sha256>.<seconds>.<file_id>.json
where the sha256 is the same as the file logged, the seconds
is the unix timestamp in seconds for the fileinfo record,
and the file_id is an atomically incremented integer per
Suricata instance.
This should allow for each occurrence of the same file to have
its own metadata file. But a collision is expected when running
Suricata repeatedly over the same pcap, as that would be the
exact same occurrence of a file.
Filestore v2 is starts as a copy of log-filestore with the
following changes.
- NSS is required as file names as based on the SHA256.
- Work/tmp files are stored in a temp. directory, then
moved into a directory tree where the directory names
are the first 2 characters of the hex SHA256.
- Removes the need for a waldo file or pid in the filenames.
This takes the form of an option to add the pid of the process to file
names. Additionally, it adds a suffix to the file name to indicate it is
not finalized.
Adding the pid to the file name reduces the likelihood that a file is
overwritten when suricata is unexpectedly killed. The number in the
waldo file is only written out during a clean shutdown. In the event
of an improper shutdown, extracted files will be written using the old
number and existing files with the same name will be overwritten.
Writes extracted files and their metadata to a temporary file suffixed
with '.tmp'. Renames the files when they are completely done being
written. As-is there is no way to know that a file on disk is still
being written to by suricata.
This patch adds support for hw bypass by enabling flow offload in the network
card (when supported) and implementing the BypassPacketsFlow callback.
Hw bypass support is disabled by default, and can be enabled by setting
"bypass: yes" in the pfring interface configuration section in suricata.yaml.
This adds a new redis mode rpush. Also more consistent config keywords orientated at the redis command: lpush and publish.
Keeping list and channel config keywords for backwards compatibility
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
Suricata was inconditionaly dropping packets that are invalid with
respect to the streaming engine. In some corner case like asymetric
trafic capture, this was leading to dropping some legitimate trafic.
The async-oneside option did help but this was not perfect in some
real life case. So this patch introduces an option that allow the
user to tell Suricata not to drop packet that are invalid with
respect to streaming.
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
We assume session resumption has occurred if the Client Hello message
included a session id, we have not seen the server certificate, but
we have seen a Change Cipher Spec message from the server.
Previously, these transactions were not logged at all because the
server cert was never seen.
Ticket: https://redmine.openinfosecfoundation.org/issues/1969
eve: detects libevent for async redis at configure
eve: moves redis output code to new file - util-log-redis.{c,h}
eve: redis ECHO and QUIT commands for async mode
eve: redis output defaults if conf is missing
Give unified2 a nostamp option which will create the file
without the timestamp suffix (like Snort's nostamp option).
Also register for rotation notification on SIGHUP so the file
will be recreated if it is removed by an external rotation
program (only when nostamp is used).
If segments section in the yaml is ommitted (default) or when the
pool size is set to 'from_mtu', the size of the pool will be MTU
minus 40. If the MTU couldn't be determined, it's assumed to be
1500, so the segment size for the bool will be 1460.
The code already looks for "dir" first instead of
"squil-base-dir", and already respects this configuration
parameter in other modes than the "sguil" mode.
Coda will still access "sguil-base-dir".
When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.
This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
Jason Ish
Victor Julien
Eric Leblond
Gaurav Singh
Alfredo Cardigliano
Julian
Phil Young
Pierre Chifflier
Ray Ruvinskiy
fooinha
Mats Klepsland
Travis Green
Giuseppe Longo