By adding the key in the root of *flow and fileinfo events it
will be possible to get all events for one application layer by
using a 'event_type:proto OR app_proto:proto' filter. This will
permit to the analyst to get a good view of events related to
one protocol.
This patch also fixes a regression in file logging where app_proto
was available before 94dbd303e4 create
the regression.
The datalen variable is declared unsigned. If txtlen and datalen are equal,
datalen will first be reduced to 0, and then the datalen-- line will cause its
value to wrap to 65535. This will cause the loop to continue much longer than
intended, and eventually may crash on an out-of-bounds *tdata dereference.
Signed-off-by: Aaron Campbell <aaron@monkey.org>
instead
mappings:
- vlan:
vlan-id: 1
tenant-id: 2
we'll now use:
mappings:
- vlan-id: 1
tenant-id: 2
For YAML it pretty much means the same thing.
Ticket: 1517
Introduce a strict mode to base64 decode. If strict,
the function will fail when invalid input data is seen.
If not strict, what has been decoded will be returned.
This is in support of adding a Snort compatible base64_decode
rule option that uses whatever data can be decoded as a length
of data to decode is optional.
Even though the json output callback is called with a null terminated
string, it's not useable directly. The size parameter to the callback
might be a lot smaller than the string size. Libjansson gives the size
up to the first point that needs escaping.
When user asks for a configuration dump, it is useless to display
the version and CPU info. Also initializing the log system conduct
to overwrite the some log files and in particular suricata.log and
this is annoying as a command should not interfere with a running
daemon.
Suricata creates a pfring cluster with a default ID = 1 when not explicitly configured,
unless the device has prefix 'dna' or 'zc'. Since pf_ring also supports other cards
implementing kernel-bypass (cluster not supported), this is preventing those cards from
running on top of this module. This patch stops suricata on 'pfring_set_cluster' failure
only when error code != PF_RING_ERROR_NOT_SUPPORTED or cluster ID has not been explicitly
configured.
When there are many threads and/or the packet pool (max-pending-packets) is
small, a potential dead lock exists between the packet pool return pool
logic and the capture threads. The autofp workers together can have all the
packets in their return pools, while the capture thread(s) are waiting at an
empty pool. A race between the worker threads and the capture thread, where
the latter signals the former, is lost by the capture thread. Now everyone
is waiting.
To avoid this scenario, this patch makes the previously hardcoded 'return
pool' threshold dynamic based on the number of threads and the packet pool
size.
It sets the threshold to the max pending packets value, divided by the number
of lister threads. The max value hasn't changed. Normally, in the autofp
runmode these are the stream/detect/log worker threads.
The max_pending_return_packets value needs to stay below the packet pool size
of the 'producers' (normally pkt capture threads but also flow timeout
injection) to avoid the deadlock.
As it's quite impossible at this time to learn how many threads will be
created before starting the runmodes, and thus spawning the threads and
already initializing the packet pools, this code sets a global variable
after runmode setup, but before the threads are 'unpaused'.
For each packet the capture module checks whether it is time to dump stats calling
TimeGet(). TimeGet() is an expensive function using gettimeofday() or SCSpinLock()
which affect performance. Since gettimeofday() is already called for setting packet
timestamp, it is more efficient to use the packet timestamp directly.
This patch removes packet copy when suricata is running in workers runmode,
packet copy is not needed in this case since packets are processed in sequence.
We will now output the sensor name independantly of the output
method if it is set in the YAML file. In the case of redis we are
using the hostname value if unset.
This patch implements reconnection handling for the redis output.
A reconnect limitation has been implemented with a limitation of
one connection per second.
This patch implements redis pipelining. This consist in contacting
the redis server every N events to minimize the number of TCP
exchange. This is optional and setup via the configuration file.
Introduce a function to realize the parsing and config file and
opening of connection to the database. Only used by output-json
for now it will be usable by other logging modules.
Introduce a function LogFileWrite that will handle the writing with
respect of the type defined in the configuration. This is used in
this patch to remove the write complexity from output-json.
When using redis output, we are loosing the host key (added by
logstash or logstash-forwarder) and we can't find anymore what
Suricata did cause the alert.
This patch is adding this key during message generation using the
'sensor-name' variable or the hostname is 'sensor-name' is not
defined.
This adds a counter indicating how many times
the flow max memcap has been reached
Since there is no always a reference to FlowManagerThreadData,
the counter is put in DecodeThreadVars.
Currently when there is no counter increase in one call of FlowGetNew
because we don't have tv or dtv at the time of the call.
The following is a snippet of the generated EVE entry:
"flow":{"memcap":0,"spare":10000,"emerg_mode_entered":0,"emerg_mode_over":0,"tcp_reuse":0,"memuse":7085248}
The code was not correct and coverity did detect a potential
overflow problem that should not happen because of the structure
of md5 string and of format.
Unlike other app-layer protocols which are enabled by default,
disable this one by default as it likely shouldn't be enabled
in production use of Suricata.
Two structure fields were uninitialized and used has a bit field.
Weird behavior were seen in list of logged fields due to that with
some build options.
Multiple events can be applied on a transaction so we may need to
log the same header twice.
The HDR_IS_LOGGED flag was making it impossible. And this system
is usless as email application layer is transaction based.
Add a boolean option named "payload" to the unified2-alert output type.
Such options makes suricata omit the payload in the resulting unified2
file. The default value is true in order to preserve the current behaviour.
- Some tests always passed due to the result value being
initialized to 1.
- Fix tests that now fail. Looks like just the test were wrong
and that the code does the right thing.
Remove the old tracker reset macro which is no longer being used.
Clear last_seen and remove flags on initialization.
Remove extra call to DefragTrackerInit as it was being called 2x
for each new tracker.
Now that DefragTrackerNew is just a wrapper for DefragTrackerAlloc,
remove it and just call DefragTrackerAlloc directly.
These trackers are likely for completed fragments, but have
not been cleaned up. If a packet on the same flow with an
already seen IP ID is seen, it could be reused prior to
being properly reinitialized.
Previous code was wrongly supposing the lines are ending with CRLF.
But this is not the case so we must propagate the length of end of
line delimiter to be able to compute the md5 of the mail body.
To be able to correlate between events, it is better to have the
tx_id information in the root object. This function adds a new
function to automate the addition of the field.
Victor Julien
Jason Ish
Eric Leblond
Alexander Gozman
Maurizio Abba
Aaron Campbell
Mats Klepsland
cardigliano
Giuseppe Longo
Torgeir Natvig
Alessandro Guido