When the config is missing, DefragPolicyGetHostTimeout will default
to returning -1. This will effectively set no timeout at all, leading
to defrag trackers being freed too early.
This patch is fixing an issue in defragmentation code. The
insertion of a fragment in the list of fragments is done with
respect to the offset of the fragment. But the code was using
the original offset of the fragment and not the one of the
new reconstructed fragment (which can be different in the
case of overlapping segment where the left part is trimmed).
This case could lead to some evasion techniques by causing
Suricata to analyse a different payload.
This patch fixes the following issue reported by valgrind:
31 errors in context 1 of 1:
Conditional jump or move depends on uninitialised value(s)
at 0x8AB2F8: UnixSocketPcapFilesCheck (runmode-unix-socket.c:279)
by 0x97725D: UnixCommandBackgroundTasks (unix-manager.c:368)
by 0x97BC52: UnixManagerThread (unix-manager.c:884)
by 0x6155F6D: start_thread (pthread_create.c:311)
by 0x6E3A9CC: clone (clone.S:113)
The running field in PcapCommand was not initialized.
This patch fixes an issue in unix socket handling. It is possible
that a socket did disconnect when analysing a command and because
the data treatment is done in a loop on clients this was leading
to a update of the list of clients during the loop. So we need
in fact to use TAILQ_FOREACH_SAFE instead of TAILQ_FOREACH.
Reported-by: Luigi Sandon <luigi.sandon@gmail.com>
Fix-suggested-by: Luigi Sandon <luigi.sandon@gmail.com>
It is possible to have a non-contiguous CPU set, which was not being
handled correctly on the TILE architecture.
Added a "rank" field in the ThreadVar to store the worker's rank separately
from the cpu for this case.
When applying wildcard thresholds (with sid = 0 and/or gid = 0) it's wrong
to exit on the first signature already having an event filter. Indeed,
doing so results in the theshold not being applied to all subsequent
signatures. Change the code in order to skip signatures with event
filters instead of breaking out of the loop.
If a live reload signal was given before the engine was fully started
up (e.g. pcap file thread waiting for a disk to spin up), a segv could
occur.
This patch only enables live reloads after the threads have been
started up completely.
*** CID 1211009: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 265 in JsonHttpLogJSON()
259 /* log custom fields if configured */
260 if (http_ctx->fields != 0)
261 {
262 HttpField f;
263 for (f = HTTP_FIELD_ACCEPT; f < HTTP_FIELD_SIZE; f++)
264 {
>>> CID 1211009: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
265 if ((http_ctx->fields & (1<<f)) != 0)
266 {
267 /* prevent logging a field twice if extended logging is
268 enabled */
269 if (((http_ctx->flags & LOG_HTTP_EXTENDED) == 0) ||
270 ((http_ctx->flags & LOG_HTTP_EXTENDED) !=
________________________________________________________________________________________________________
*** CID 1211010: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 492 in OutputHttpLogInitSub()
486 {
487 if ((strcmp(http_fields[f].config_field,
488 field->val) == 0) ||
489 (strcasecmp(http_fields[f].htp_field,
490 field->val) == 0))
491 {
>>> CID 1211010: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
492 http_ctx->fields |= (1<<f);
493 break;
494 }
495 }
496 }
497 }
StreamTcpSetDisableRawReassemblyFlag() has the same effect as
AppLayerParserTriggerRawStreamReassembly in that it will force the
raw reassembly to flush out asap. So it is redundant to call both.
Implement StreamTcpSetDisableRawReassemblyFlag() which stops raw
reassembly for _NEW_ segments in a stream direction.
It is used only by TLS/SSL now, to flag the streams as encrypted.
Existing segments will still be reassembled and inspected, while
new segments won't be. This allows for pattern based inspection
of the TLS handshake.
Like is the case with completely disabled 'raw' reassembly, the
logic is that the segments are flagged as completed for 'raw' right
away. So they are not considered in raw reassembly anymore.
As no new segments will be considered, the chunk limit check will
return true on the next call.
Have a single function StreamTcpReturnSegmentCheck determine if a
segment is ready to be removed from the stream.
Handle FLOW_NOPAYLOAD_INSPECT in raw reassembly.
httplog_ctx->fields would not be initialized before setting flags in
it:
Scanbuild:
output-json-http.c:491:46: warning: The left expression of the compound assignment is an uninitialized value. The computed value will also be garbage
http_ctx->fields |= (1<<f);
~~~~~~~~~~~~~~~~ ^
1 warning generated.
Drmemory:
~~27874~~ Error #1: UNINITIALIZED READ: reading register eax
~~27874~~ # 0 JsonHttpLogJSON [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:260]
~~27874~~ # 1 JsonHttpLogger [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:375]
Just memset the whole structure right after initialition.
Fix issue detected byCoverity:
*** CID 1197756: Bad bit shift operation (BAD_SHIFT)
/src/util-rohash.c: 74 in ROHashInit()
68 }
69 if (hash_bits < 4 || hash_bits > 32) {
70 SCLogError(SC_ERR_HASH_TABLE_INIT, "invalid hash_bits setting, valid range is 4-32");
71 return NULL;
72 }
73
>>> CID 1197756: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1U << hash_bits", left shifting by more than 31 bits has undefined behavior. The shift amount, "hash_bits", is as much as 32.
74 uint32_t size = hashsize(hash_bits) * sizeof(ROHashTableOffsets);
75
76 ROHashTable *table = SCMalloc(sizeof(ROHashTable) + size);
77 if (unlikely(table == NULL)) {
78 SCLogError(SC_ERR_HASH_TABLE_INIT, "failed to alloc memory");
79 return NULL;
This was only a potential issue as ROHashInit was only called with
hash_bits 16 in the code.
Bug #1170.
During socket creation all error cases were leading to suricata to
retry the opening of capture. This patch updates this behavior to
have fatal and recoverable error case. In case of a fatal error,
suricata is leaving cleanly.
Change GAP detection logic. If we encounter missing data before
last_ack, we know we have missed data. The receiving host has ack'd
it already, so a retransmission of the missing data is highly
unlikely.
AppLayer reassembly correctly only flags a segment as done when it's
completely used in reassembly. Raw reassembly could flag a partially
used segment as complete as well. In this case the segment could be
discarded early. Further reassembly would miss data, leading to a
gap. Due to this, up to 'window size' bytes worth of segments could
remain in the session for a long time, leading to memory resource
pressure.
This patch sets the flag correctly.
Some traffic is very unbalanced. For example a 4 bytes request
followed by 12mb of response data. If the 4 bytes don't lead to
the protocol being detected, then app layer processing won't
start, but it will not give up either. It will just wait for more
data. This leads to piling up data on the opposite side.
Observed:
TS: 4 bytes. PP failed (bit set), PM has not given up (bit not set).
This makes sense as max_depth is not yet reached.
TC: 12mb. PP and PM failed (bits set).
As ts-PM never gives up, we never consider proto detect complete,
so all segments in either direction are still kept in the session.
This patch adds a cutoff for this case:
- IF for TS we have PP bit set, PM not set, AND
- we have TC both bits set, AND
- proto is unknown, AND
- TC data is 100k already, THEN
- give up on protocol detection.
The same for the opposite direction.
The reassembly gap detection makes use of the window. However, in
midstream mode the window size we use is unreliable, as we have to
assume window scaling is in place. This patch improves midstream
handling of those cases.
In midstream mode we may encounter a case where the data we is beyond
the isn, but below last_ack. This means we're missing some data, that
is already acked so it won't be retransmitted. Therefore, we can
conclude it's a data gap.
If we're getting a lot of data in one direction and the proto for this
direction is unknown, proto detect will hold up segments in the segment
list in the stream. They are held so that if we detect the protocol on
the opposing stream, we can still parse this side of the stream as well.
However, some sessions are very unbalanced. FTP data channels, large
PUT/POST request and many others, can lead to cases where we would have
to store many megabytes worth of segments before we see the opposing
stream. This leads to risks of resource starvation.
In this patch, a cutoff point is enforced. If we've stored 100k in one
direction and we've seen no data in the other direction, we give up.
If we've given up, the applayer_proto_detection_skipped event is set.
app-layer-event: applayer_proto_detection_skipped;
If a TCP session is midstream, we may end up with a case where the
start of an HTTP request is missing. We won't detect HTTP based on
the request. However, the reply is fine, so we detect HTTP anyway.
This leads to passing the incomplete request to the htp parser.
This has been observed, where the http parser then saw many bogus
requests in the incomplete data. This is not limited to HTTP.
To counter this case, a midstream session MUST find it's protocol
in the toserver direction. If not, we assume the start of the
request/toserver is incomplete and no reliable detection and parsing
is possible. So we give up.
If midstream is enabled and the first packet is the syn/ack packet from
the 3whs, initialized server.last_ack to the packets seq.
This fixes tracking the session.
Extended data were freed before the release function was called.
The result was that, in AF_PACKET IPS mode, the release function
was only sending void data because it the content of the extended
data is the content of the packet.
This patch updates the code to have the freeing of extended data
done in the cleaning function for a packet which is called by the
release function. This improves consistency of the code and fixes
the bug.
The direction specific masks were not used correctly. The toserver ones
were only used for 'dp' registrations, the toclient ones only for 'sp'.
The patch merges them.
If a flow matches both an 'sp' based PP registration and a 'dp' based,
until now we would only check the 'dp' one. This patch changes that. It
will inspect both.
Instead of the notion of toserver and toclient protocol detection, use
destination port and source port.
Independent of the data direction, the flow's port settings will be used
to find the correct probing parser, where we first try the dest port,
and if that fails the source port.
Update the configuration file format, where toserver is replaced by 'dp'
and toclient by 'sp'. Toserver is intrepreted as 'dp' and toclient as
'sp' for backwards compatibility.
Example for dns:
dns:
# memcaps. Globally and per flow/state.
#global-memcap: 16mb
#state-memcap: 512kb
# How many unreplied DNS requests are considered a flood.
# If the limit is reached, app-layer-event:dns.flooded; will match.
#request-flood: 500
tcp:
enabled: yes
detection-ports:
dp: 53
udp:
enabled: yes
detection-ports:
dp: 53
Like before, progress of protocol detection is tracked per flow direction.
Bug #1142.
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.
- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode
To set the modes:
- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode
Bug #1177.
The SLOAD define using __insn_ld2s_L2 is used to provide a compiler
hint that the load will come from the L2 cache instead of the L1. It
also specifies that it is a 2 byte signed load. For the Tiny MPM, that
needs to be a 1-byte load, which is what is specified in util-ac-mpm-tile.c,
but the #undef was removing that definition.
Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.
Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
This is an intrusive change. This patch modifies the way AMATCH
inspection uses locking.
So far, each keyword did it's own locking. This lead to a situation
where a 'alstate' pointer was passed around that was not always
protected by a lock.
This patch moves the locking to the Stateful detection functions.
This patch fixes a warning message when suricata is started without
giving an interface name on the command line. The code was trying
to get the MTU even if pcap_dev was not set.
FreeBSD 10 32-bit with clang 3.3:
log-tlslog.c:172:14: error: format specifies type 'long' but the argument has type 'time_t' (aka 'int') [-Werror,-Wformat]
p->ts.tv_sec,
^~~~~~~~~~~~
1 error generated.
detect-engine-payload.c:508:27: warning: format specifies type 'long' but the argument has type 'time_t' (aka 'int') [-Wformat]
printf("%ld.%06ld\n", tv_diff.tv_sec, (long int)tv_diff.tv_usec);
~~~ ^~~~~~~~~~~~~~
%d
1 warning generated.
Add output 'free list' that contains all the output ctx' that need
cleanup at shutdown. It differs from the runmode output list in that
it will also contain a 'parent' for the submodules that share the
context of it's parent.
For the loggers that we allow only one instance for: tls, ssh, drop, we
track active loggers through Output*Enable functions. Add Disable
functions to mirror this. They are to be called from the shutdown funcs
those loggers use.
This is a beginning of implementation for bug #1660:
https://redmine.openinfosecfoundation.org/issues/1160
This patch adds a cleaning function for each logger of new type
(packet, tx and file). These functions are called in RunModeShutDown().
The state of this patch is that it is crashing suricata when sending
pcap to analyse:
- At first pcap if tx and file cleaning function are called
- At second pcap if only packet cleaning function is called
The cause in first case is unknown. In second case this is due to
the necessity of cleaning the list of logger registered to a logging
type.
The OpenSSL implementation of RFC 6520 (Heartbeat extension) does not
check the payload length correctly, resulting in a copy of at most 64k
of memory from the server (ref: CVE-2014-0160).
This patch adds support for decoding heartbeat messages (if not
encrypted), and checking several parts (type, length and padding).
When an anomaly is detected, a TLS event is raised.
In case of multiple transactions, the stored AMATCH list would not have
been reset, but it would still be reconsidered. Even though none would
match, the engine would still conclude that the rule matched.
Pierre Chifflier pointed out that a rule like:
alert tls any any -> any any (msg:"TLS store"; tls.issuerdn:!"C=FR"; tls.store;)
was alerting but not storing the certificate. If the filter was
removed:
alert tls any any -> any any (msg:"TLS store"; tls.store;)
then tls.store is working as expected.
This was linked with fact that logging is only done once for a SSL
state. So without filter, once we have the info we can log and we
run the storage. But when there is a filter, we log and then there
is a filter analysis and alerting. And as logging as already be done
we don't enter in the logging function and there is no storage.
This patch forces the entrance in the log function when there is a
request for TLS storage. And it adds an exit in the logging function
to only do the storage part if the TLS state has already being logged.
Previously the sync code would depend on traffic to complete. This
patch adds poll support and can complete the setup if the poll timeout
is reached as well.
Part of bug #1130.
This patch is updating af-packet to discard packets that have been
sent to a socket before all socket in a fanout group have been setup.
Without this, there is no way to assure that all packets for a single
flow will be treated by the same thread.
Tests have been done on a system with an ixgbe network card. When using
'cluster_flow' load balancing and disactivating receive hash on the iface:
ethtool -K IFACE rxhash off
then suricata is behaving as expected and all packets for a single flow
are treated by the same thread.
For some unknown reason, this is not the case when using cluster_cpu. It
seems that in that case the load balancing is not perfect on the card side.
The rxhash offloading has a direct impact on the cluster_flow load balancing
because load balancing is done by using a generic hash key attached to
each skb. This hash can be computed by the network card or can be
computed by the kernel. In the xase of a ixgbe network card, it seems there
is some issue with the hash key for TCP. This explains why it is necessary to
remove the rxhash offloading to have a correct behavior. This could also
explain why cluster_cpu is currently failing because the card is using the
same hash key computation to do the RSS queues load balancing.
This allows for registering a keyword under another name while keeping
the old name active and supported.
Do this for 'luajit', which can now also be used as just 'lua'.
The HTTP module of Eve didn't register itself with the app-layer
for HTTP. This meant that if no other HTTP logger was active, the
HTTP logging in Eve wouldn't work.
This patch makes the HTTP Eve module register itself correctly.
Bug #1133.
To remain constistent with the other logs, set the event type to
the same name as the structure containing the defails. In this
case fileinfo.
Part of bug #1127.
Due to what appears to be an issue in logstash, the 'file' part of
the file event types was masked by a field that logstash-forwarder
added itself.
Since logstash-forwarder is an important part of the logstash stack,
this patch works around the issue by renaming our 'file' structure
to 'fileinfo', thus resolving the naming conflict.
Bug #1127
Move pfring_enable_ring to the start of ReceivePfringLoop() so that
it's guaranteed to be called after all threads have called
pfring_set_cluster first.
This is necessary because pfring will already make packets available
to thread N, while thread N+1 is still registering itself. This leads
to cases where the first packet(s) of a flow are processed by a
different thread in Suricata than the later ones.
This is a race condition only at start up. New flows after the pfring
initialization is complete will not be influenced by this.
Bug #1129.
This patch updates the timestamp format used in eve loggin.
It uses a ISO 8601 comptatible string. This allow tools parsing
the output to easily detect adn/or use the timestamp.
In the EVE JSON output, the value of the timestamp key has been
changed to 'timestamp' (instead of 'time'). This allows tools
like Splunk to detect the timestamp and use it without configuration.
Logstash configuration is simple:
input {
file {
path => [ "/usr/local/var/log/suricata/eve.json" ]
codec => json
type => "suricata-log"
}
}
filter {
if [type] == "suricata-log" {
date {
match => [ "timestamp", "ISO8601" ]
}
}
}
In splunk, auto detection of the fle format is failling and it seems
you need to define a type to parse JSON in
$SPLUNK_DIR/etc/system/local/props.conf:
[suricata]
KV_MODE = json
NO_BINARY_CHECK = 1
TRUNCATE = 0
Then you can simply declare the log file in
$SPLUNK_DIR/etc/system/local/inputs.conf:
[monitor:///usr/local/var/log/suricata/eve.json]
sourcetype = suricata
In both cases the timestamp are correctly imported by
the tools.
PF_RING is delivering the packet with VLAN header stripped. This
patch updates the code to get the information from PF_RING extended
header information.
This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.