In case of transform issues (transform not consumed before pkt_data
for example), the code would hit an ugly BUG_ON.
Address this by a more graceful error message, that will still
invalidate the sig but not crash the engine.
Implement support for limiting Teredo detection and decoding to specific
UDP ports, with 3544 as the default.
If no ports are specified, the old behaviour of detecting/decoding on any
port is still in place. This can also be forced by specifying 'any' as the
port setting.
StringParse* functions would perform a stricter check compared to
ByteExtractString* functions. These new functions shall also check if
any extra characters follow the extracted numeric value in addition to
the checks performed by ByteExtractString* and return -1 in that case.
This is particularly important in parser, configuration and setup functions.
This commit adds support for the Remote Framebuffer Protocol (RFB) as
used, for example, by various VNC implementations. It targets the
official versions 3.3, 3.7 and 3.8 of the protocol and provides logging
for the RFB handshake communication for now. Logged events include
endpoint versions, details of the security (i.e. authentication)
exchange as well as metadata about the image transfer parameters.
Detection is enabled using keywords for:
- rfb.name: Session name as sticky buffer
- rfb.sectype: Security type, e.g. VNC-style challenge-response
- rfb.secresult: Result of the security exchange, e.g. OK, FAIL, ...
The latter could be used, for example, to detect brute-force attempts
on open VNC servers, while the name could be used to map unwanted VNC
sessions to the desktop owners or machines.
We also ship example EVE-JSON output and keyword docs as part of the
Sphinx source for Suricata's RTD documentation.
This command causes `pcre_jit_exec` to be used when available. If it's
available and there are allocation errors preparing for it, things
fallback to `pcre_exec`.
This commit adds a warning used by the PCRE detect logic when it fails
to register initialization and free functions for per-thread JIT stack
handling.
This error code is only used when the platform has PCRE JIT exec
functionality.
When a TCP DNS flow would start with a GAP on the TS side, the successful
protocol detection on the TC side would trigger 'opposing side' reassembly
and app-layer processing. In this case the stream flags would indicate the
wrong direction and the wrong parser would be called.
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.
Introduce two macros:
APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)
Update all parsers to use this.
The idea of an OK signature parsing error is an error that is
allowed to occur, but still lets test mode pass, unlike
silent errors which will still fail testing.
This is introduced to allow for app-layer event keywords to be
removed, but not have old rules fail out on this case. For example
the Rust DNS parser removes from DNS app-layer events that are
not used anymore.
To signal that an error is OK, -3 is returned. This also implies
silent.
On an unknown app-layer event, return -3 for "silent OK fail". A
warning will still be emitted, but its not considered a rule parse
error. This is to handle app-layer events being removed in a more
graceful manner for the user.
This allows -T to pass with an old app-layer events rule file
that may used removed app-layer event keywords.
This commit places restrictions on the length of the file name specified
in attachments (`name=` or `filename=`) to `NAME_MAX`. Names exceeding
these limits will be truncated and processing will continue with the
truncated name.
This patch addresses two problems.
First, various parts of the engine, but most notably the flow manager (FM),
use a minimum of the time notion of the packet threads. This did not
however, take into account the scenario where one or more of these
threads would be inactive for prolonged times. This could lead to the
time used by the FM could get stale.
This is addressed by keeping track of the last time the per thread packet
timestamp was updated, and only considering it for the 'minimum' when it
is reasonably current.
Second, there was a minor race condition at start up, where the FM would
already inspect the hash table(s) while the packet threads weren't active
yet. Since FM gets the time from the packet threads, it would use a bogus
time of 0.
This is addressed by adding a wait loop to the start of the FM that waits
for 'time' to get ready.
A race condition during the start of pcap file processing could cause
missed alerts and logged events. This race happens between the packet
threads and the flow manager. It was observed on slower hardware, but in
theory could happen on any machine. It required the 'autofp' runmode.
In commit 6f560144c1 ("time: improve offline time handling") the logic
was added to make the flow manager use a minimum of all the packet threads
perception of time.
The race condition was that the flow manager may become active _before_
all of the packet threads have started processing packets and thus setting
their timestamp. The threads that had not yet initialized their timestamp
would not be considered when calculating the minimum.
As a result of this, older packets timestamps would not yet be registered.
This would give the Flow Manager a timestamp too far in the future. While
the FM was running, the packet processing would start and a flow would
be created. This flow would then immediately be considered 'timed out' by
the FM, due to the timestamp too far in the future.
In the observed case, the thread processing packet 1 from the pcap had not
yet started processing while other threads had already started. The FM was
also already active. Due to the timestamps in the pcap this meant that the
time the FM used was about 500 seconds in the future compared to packet 1.
This patch fixes the issue by initializing all of the threads timestamps
with the timestamp value of the first packet. This way the minimum will
always consider this timestamp.
This commit adds logic checking if the sticky buffer in effect provides
the required content.
If the sticky buffer doesn't, the rule will not load and a diagnostic
message with follow-on steps is displayed.
Improve flow file flags and file flags updates. Introduce a mask
that is set at start up to avoid lots of runtime checks.
Disable cocci flags check as it doesn't support the more dynamic
nature of the flag updates.
Using the run-as configuration option with the nflog capture method
results in the following error during the startup of suricata:
[ERRCODE: SC_ERR_NFLOG_BIND(248)] - nflog_bind_pf() for AF_INET failed
This is because SCDropMainThreadCaps does not have any capabilities
defined for the nflog runmode (unlike other runmodes). Therefore, apply
the same capabilities to the nflog runmode that are already defined for
the nfqueue runmode. This has been confirmed to allow suricata start
and drop its privileges in the nflog runmode.
Fixes redmine issue #3265.
Signed-off-by: Timo Sigurdsson <public_timo.s@silentcreek.de>
Previously each 'TmSlot' had it's own packet queue that was passed
to the registered SlotFunc as an argument. This was used mostly for
tunnel packets by the decoders and by defrag.
This patch removes that in favor of a single queue in the ThreadVars:
decode_pq. This is the non-locked version of the queue as this is
only a temporary store for handling packets within a thread.
This patch removes the PacketQueue pointer argument from the API.
The new queue can be accessed directly through the ThreadVars
pointer.
Shrink ThreadVars by removing the queue handler names that are only
used at shutdown. Since this is not performance critical, we can use
the id's to look up the queue handler.
When TmThreadsSlotProcessPkt fails it will return the packet that was
passed to it to the packetpool.
Some of the packet sources were doing this manually as well. This patch
fixes those sources.
Use a single packet queue per thread for flow timeout packet
injection. The per slot queue was unused except for this use
case. Having a single queue makes the logic and implementation
simpler.
In case of 'autofp', the per thread packet queue will actually
use the threads input queue. For workers/single a dedicated
queue will be set up.
Rename TmThreadsSlotHandlePostPQs to TmThreadsHandleInjectedPackets
to reflect the changed logic.
Set the livedev on reassembled packets to that of the parent
packet. Fixes issues with multidetect, specifically a segfault
as reported in issue 3380.
Bug #3380.
This patch splits the list of loggers the tx logging walks into lists per
alproto. The list was getting longer with each eve addition. The result
was that for each tx we would have to loop through multiple loggers that
did not apply to this tx as it was for the wrong protocol.
In some cases a SBB could be seen as overlapping with the requested
offset, when it was in fact precisely before it. In some special cases
this could lead to the stream engine not progressing the 'raw' progress.
Don't assume that the next block after the sbb head is after the
requested offset.
If the next block was before the offset, the returned data_len
would underflow and return a nonsense value to the app-layer.
Bug #2993.
This commit causes Suricata to exit when a buffer from the analyzer
table is not recognized.
Since the table must match what's registered, exiting will bring noticed
to the condition.
Currently all failures in IPv4 option decode are ignore with
respect to continuing to handle the packet.
Change this to fail, and abort handling the packet if the
option length is invalid.
Ticket 3328:
https://redmine.openinfosecfoundation.org/issues/3328
Fix evasion posibility by rejecting packets with a broken ACK field.
These packets have a non-0 ACK field, but do not have a ACK flag set.
Bug #3324.
Reported-by: Nicolas Adba
RST injection during the SYN_SENT state could trick Suricata into marking
a session as CLOSED. The way this was done is: using invalid TSECR value
in RST+ACK packet. The ACK was needed to force Linux into considering the
TSECR value and compare it to the TSVAL from the SYN packet.
The second works only against Windows. The client would not use a TSVAL
but the RST packet would. Windows will reject this, but Suricata considered
the RST valid and triggered the CLOSED logic.
This patch addresses both. When the SYN packet used timestamp support
the timestamp of incoming packet is validated. Otherwise, packet responding
should not have a timestamp.
Bug #3286
Reported-by: Nicolas Adba
Fix address negation detection not resolving variables when
looking for the negation.
This patch makes use of the actual parsing routines to relay this
information to the signature parser.
Bug #3389.
Fixes: 92f08d85aa ("detect/iponly: improve negation handling in parsing")
If file prune is called inspect has already run. So if file is closed
we can just prune. No need to consider a window anymore.
When still in progress, fix the left_edge calculation.
Now that we call the file prune loop very regularly, we can move the
SMTP specific inspection pruning logic into this loop. Helps with
cases there we don't (often) update a files inspection trackers.
If we already know that the boundary exists, we can start looking
there. Otherwise, we can skip trying as the boundary is a subset
of the form end marker.
This commit ensures direction warnings for ICMP v4 and v6
are suppressed and corrects check so that both protocols
are checked (instead of the same protocol being checked twice).
Update the default packet size computation to use LiveDeviceName
instead of LiveDevice as the LiveDevice list is not built when
the default packet size is built.
Coccinelle test was doing a false positive on the function
AppLayerParserStateSetFlag and AppLayerParserStateIssetFlag.
To address that, this patch adds a new coccinelle markup:
/* coccinelle: AppLayerParserStateSetFlag():2,2:APP_LAYER_PARSER_ */
It indicates that AppLayerParserStateSetFlag is a setter and getter
and that the checks should be disabled inside the function.
Currently this markup is only used for that but following patch will
add some checks on option value.
When transforms are part of a rule, improve information displayed with
fast patterns to include the original buffer name and whether any
transform(s) are applied.
When registing a detection engine, check that the app-layer
protocol supports tx detect flags. Exit with a fatal
error if it does not as this is a code implementation
error that should be resolved during development.
Add method to check if a parser for an app-layer protocol
supports tx detect flags.
This is a bit of a hack for now as where we need to run
this check from we do not have the IP protocol.
When HTTP pipelining was in use, the transaction id used for events
and files could be off. If the request side was several requests ahead
of the responses, it would use the HtpState::transaction_cnt for events
and files, even though that is only incremented on complete requests.
Split request and response tx id tracking. The response is still handled
by the HtpState::transaction_cnt, but the request side is now handled by
its own logic.
Since ebcc4db84a the flow worker runs
file pruning after parsing, detection and loging. This means we can
simplify the pruning logic. If a file is in state >= CLOSED, we can
prune it. Detection and outputs will have had a final chance to
process it.
Remove the calls to the pruning code from Rust. They are no longer
needed.
If a protocol does not support TxDetectFlags, don't try to use them.
The consequence of trying to use them was that a TX would never be
considered done, and it would never be freed. This would lead to excessive
memory use and performance problems due to walking an ever increasing
list.
When a BPF filter is given on the command line when reading a
pcap file, the BPF filter is not honored.
The regression has been introduced in:
commit 3ab9120821
Author: Dana Helwig <dana.helwig@protectwise.com>
Date: Thu Apr 27 11:17:16 2017 -0600
source-pcap-file: Pcap Directory Mode (Feature #2222)
Reported-By: Tim Colin <tcolin@et.esiea.fr>
A BUG_ON statement would seemingly randomly trigger during the threading
shutdown logic. After a packet thread reached the THV_RUNNING_DONE state,
it would sometimes still receive flow timeout packets which would then
remain unprocessed.
1 main: TmThreadDisableReceiveThreads(); <- stop capturing packets
2 worker: -> TmThreadTimeoutLoop (THV_FLOW_LOOP) phase starts
3 main: FlowForceReassembly(); <- inject packets from flow engine
4 main: TmThreadDisablePacketThreads(); <- then disable packet threads
5 main: -> checks if 'worker' is ready processing packets
6 main: -> sends THV_KILL to worker
7 worker: breaks out of TmThreadTimeoutLoop and changes to THV_RUNNING_DONE.
Part of the problem was with (5) above. When checking if the worker was
already done with its work, TmThreadDisablePacketThreads would not consider
the injected flow timeout packets. The second part of the problem was with (7),
where the worker checked if it was ready with the TmThreadTimeoutLoop in a
thread unsafe way.
As a result TmThreadDisablePacketThreads would not wait long enough for the
worker(s) to finish its work and move the threads to the THV_RUNNING_DONE
phase by issuing the THV_KILL command.
When waiting for packet processing threads to process all in-flight packets,
also consider the 'stream_pq'. This will have received the flow timeout
packets.
Bug #1871.
This commit adds `ARRAY_SIZE` as an helper for determining the number of
elements in an initialized array. The calculation is the same but the
macro provides a convenient shortcut. The implementation was borrowed
from the kernel sources.
Create a single function to return the version string, to avoid lots
of ifdefs in multiple places.
Make the version determine the 'release' status. If the version from
autoconf has '-dev' in the name, it is not a release. If it hasn't
it is considered a release version.
Avoids using uninitialized memory. Show showed itself
in nonsense values in counters, and in nfq_handle_packet
errors that were likely the result of passing uninitialized
memory to the nfq API.
Bug 3263.
Bug 3120.
Fixes: b2a6c60dee ("source-nfq: increase maximum queues number to 65535")
NFQ can generate warnings/errors with a delay. After Suricata has
succesfully passed a verdict to the kernel, there are still things
that can go wrong for that verdict. This is then passed to the
queue through a netlink error message, which leads to nfq_handle_packet
returning an error code.
Suppress the warning. Also remove the errno/strerror use as
nfq_handle_packet does not set the errno.
Thanks to Florian Westphal.
Bug 3120.
TCP_OPT_INVALID_LEN was set if the opt len was 2. While useless
an empty SACK is not uncommon.
Seen on an iOS device talking to an Apple server.
Bug #3254.