So far, the suricata socket suricata-command.socket has the rights
rw-r----- suricata:user.
When suricata is used with restricted access, an other application
(suricatasc like) that needs to access to the command socket also
with restricted access can not write to the socket since it is not
the owner (e.g suricata within container, with an hardened value
for umask and hardened rights for users).
The socket should be set as rw-rw----. Use chmod instead of fchmod
and set it after the socket creation.
https://redmine.openinfosecfoundation.org/issues/2412
Suricatasc is not supporting pcap-file processing in continuous mode.
Register a new command pcap-file-continuous in the unix manager to work
with suricatasc. Add defaulted arguments for pcap-file to support
backwards compatibility.
RAND_MAX is not guaranteed to be a divisor of ULONG_MAX, so take the
necessary precautions to get unbiased random numbers. Although the
bias might be negligible, it's not advisable to rely on it.
according to its man page, sigprocmask has undefined behavior in
multithreaded environments. Instead of explictly blocking the handling
of SIGUSR2 in every thread, direct block handling SIGUSR2 before
creating the threads and enable again the handling of this signal
afterwards. In this way, only the main thread will be able to manage
this signal properly.
Add startswith modifier to simplify matching patterns at the start
of a buffer.
Instead of:
content:"abc"; depth:3;
This enables:
content:"abc"; startswith;
Especially with longer patterns this makes the intention of the rule
more clear and eases writing the rules.
Internally it's simply a shorthand for 'depth:<pattern len>;'.
Ticket https://redmine.openinfosecfoundation.org/issues/742
Fix the inspection of multiple files in a single TX, where new files
may be added to the TX after inspection started.
Assign the hard coded id DE_STATE_FLAG_FILE_INSPECT to the file
inspect engine.
Make sure that sigs that do file inspection and don't match on the
current file always store a detailed state. This state will include
the DE_STATE_FLAG_FILE_INSPECT flag.
When the app-layer indicates a new file is available, for each sig
that has the DE_STATE_FLAG_FILE_INSPECT flag set, reset part of the
state so that the sig is evaluated again.
Free txs that are done out of order if we can. Some protocol
implementations have transactions running in parallel, where it is
possible that a tx that started later finishes earlier than other
transactions. Support freeing those.
Also improve handling on asynchronious transactions. If transactions
are unreplied, e.g. in the dns flood case, the parser may at some
point free transactions on it's own. Handle this case in
the app-layer engine so that the various tracking id's (inspect, log,
and 'min') are updated accordingly.
Next, free txs much more aggressively. Instead of freeing old txs
at the app-layer parsing stage, free all complete txs at the end
of the flow-worker. This frees txs much sooner in many cases.
When stateless rules are depending on a flowbit being set by a stateful
rule, the inspection order is almost certainly wrong.
Switch stateless rules depending on stateful rules to being stateful.
This is used to turn 'TCP stream' inspecting rules (which are stateless
unless mixed with stateful keywords) into stateful rules.
Use per tx detect_flags to track prefilter. Detect flags are used for 2
things:
1. marking tx as fully inspected
2. tracking already run prefilter (incl mpm) engines
This supercedes the MpmIDs API for directionless tracking
of the prefilter engines.
When we have no SGH we have to flag the txs that are 'complete'
as inspected as well.
Special handling for the stream engine:
If a rule mixes TX inspection and STREAM inspection, we can encounter
the case where the rule is evaluated against multiple transactions
during a single inspection run. As the stream data is exactly the same
for each of those runs, it's wasteful to rerun inspection of the stream
portion of the rule.
This patch enables caching of the stream 'inspect engine' result in
the local 'RuleMatchCandidateTx' array. This is valid only during the
live of a single inspection run.
Remove stateful inspection from 'mask' (SignatureMask). The mask wasn't
used in most cases for those rules anyway, as there we rely on the
prefilter. Add a alproto check to catch the remaining cases.
When building the active non-mpm/non-prefilter list check not just
the mask, but also the alproto. This especially helps stateful rules
with negated mpm.
Simplify AppLayerParserHasDecoderEvents usage in detection to only
return true if protocol detection events are set. Other detection is done
in inspect engines.
Move rule group lookup and handling into it's own function. Handle
'post lookup' tasks immediately, instead of after the first detect
run. The tasks were independent of the initial detection.
Many cleanups and much refactoring.
Add API meant to replace the MpmIDs API. It uses a u64 for each direction
in a tx to keep track of 2 things:
1. is inspection done?
2. which prefilter engines (like mpm) are already completed
Analyze flowbits to find which bits are only checked.
Track whether they are set and checked on the same level of 'statefulness'
for later used.
Dump flowbits to json including the sids that set/check etc the bit.
There is probably not too much bad about enabling both, but
open file counts can get messy with both enabled. And v1
should be schedule for deprecation soon enough.
This doesn't need to be registered from suricata.c. And moving
it to the init function makes sure its only registered if
the logger is actually enabled.
Track each type of error warning and only log it once. Also create
a new stat, file_store.fs_errors to count each file system type
error (open, rename, unlink).
Also remove exit stats, they are of limited value.
As fileinfo records are logged to the main eve log, disable
metadata by default. But when enabled, just use the fileinfo
record.
Metadata is stored in a file named:
<sha256>.<seconds>.<file_id>.json
where the sha256 is the same as the file logged, the seconds
is the unix timestamp in seconds for the fileinfo record,
and the file_id is an atomically incremented integer per
Suricata instance.
This should allow for each occurrence of the same file to have
its own metadata file. But a collision is expected when running
Suricata repeatedly over the same pcap, as that would be the
exact same occurrence of a file.
Even if a file is truncated, force the SHA256 if force sha256
is set to yes.
The new file store requires the sha256 regardless of the file
state if it is to be logged, as the filename is based on the
sha256.
Filestore v2 is starts as a copy of log-filestore with the
following changes.
- NSS is required as file names as based on the SHA256.
- Work/tmp files are stored in a temp. directory, then
moved into a directory tree where the directory names
are the first 2 characters of the hex SHA256.
- Removes the need for a waldo file or pid in the filenames.
Split the building of the fileinfo record from the writing
of the record so the building can be called from other code.
Specifically the new filestore output which uses fileinfo
records as the metadata.
Give SCCreateDirectoryTree a new argument, final. If true the
full path will be created as a directory. If false, the last
component will not be created as a directory (current
behaviour).
Renames SCLogCreateDirectoryTree to SCCreateDirectoryTree
and move into a util module for re-use.
Also moves SCMkDir from suricata-common.h to the more
appropriately names util-path.h.
I would have prefered to use util-file for file related options
but that is already used by file store utilities. util-path
is close enough for file related operations.
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
The flow manager thread (that also runs the host cleanup code) would
sometimes free a host before it's thresholds are timed out. This would
lead to misdetection or too many alerts.
This was mostly (only?) visible on slower systems. And was caused by a
mismatch between time concepts of the async flow manager thread and the
packet threads, resulting in the flow manager using a timestamp that
was before the threshold entry creation ts. This would lead to an
integer underflow in the timeout check, leading to a incorrect conclusion
that the threshold entry was timed out.
To address this, check if the 'check' timestamp is not before the creation
timestamp.
If TmThreadDrainPacketThreads would take more than 60 seconds, the wait
loop that follows it would reach 'timeout' condition immediately. This
would lead to a null ptr deref of 'tv'.
Fix by not counting the TmThreadDrainPacketThreads and also not doing
the null ptr deref in any case.
In offline mode, if the starting timestamp is 0 suricata will never
initialize cached_minute_start array. This cause the timestamp to be
ignored when needed (e.g., in fast.log).
This commit will force the initialization of this array.
PrintRawUriFp does not properly escape backslash. This causes confusion
between a \ character and an hex-encoded character. PrintRawUriBuffer,
instead, correctly does backslash-encoding.
Adding proper escaping of backslash to PrintRawUriFp.
For loggers that register once per direction, use unique id's per
direction.
Reshuffle id's to keep tx log id's low so we can use u32 for tracking
logged loggers.
Avoid looping in transaction output.
Update app-layer API to store the bits in one step
and retrieve the bits in a single step as well.
Update users of the API.
Create a bitmap of the loggers per protocol. This is done at runtime
based on the loggers that are enabled. Take the logger_id for each
logger and store it as a bitmap in the app-layer protcol storage.
Goal is to be able to use it as an expectation later.
usleep on MinGW doesn't behave as expected. Added replacement
wrapper around 'Sleep(msec)'. As that has msec resolution and
not a usec resolution, change the various thread init and stop
functions to test for the actual time waited instead of counting
the usecs passed to usleep.
On MinGW the result of ntohl needs to be casted to uint32_t and
the result of ntohs to uint16_t. To avoid doing this everywhere
add SCNtohl and SCNtohs macros.
Many were using AlertJsonThread instead of OutputJsonCtx,
but as the datatypes were similar enough no harm was done.
Now that they are using their proper datatype, removed
AlertJsonThread from output.h as its no longer used.
According to PF_RING upstream the vlan header should never be stripped
from the packet PF_RING feeds to Suricata. But upstream also indicated
keeping the check would be a good "safety check".
So in addition to the check, add a warning that warns once (per thread
for implementation simplicity) if the vlan hdr does appear to be stripped
after all.
When Suricata was monitoring traffic with a single vlan layer, the stats
and output instead showed 2. This was caused by the raw packets PF_RING
feeds Suricata would hold the vlan header, but the code assumed that
the header was stripped and the vlan_id passed to Suricata through
PF_RING's extended_hdr.parsed_pkt.
This patch adds the following logic: Check vlan id from the parser packet
PF_RING prepared. PF_RING sets the vlan_id based on its own parsing or
based on the hardware offload. It gives no indication on where the vlan_id
came from, so we rely on the vlan_offset field. If it's 0, we assume the
PF_RING parser did not see the vlan header and got it from the hardware
offload. In this case we will use this information directly, as we won't
get a raw vlan header later. If PF_RING did set the offset, we do the
parsing in the Suricata decoder so that we have full control.
PF_RING *should* put back the vlan header in all cases, and also set the
vlan_offset field, but as a extra precaution keep the check described
above.
Bug #2355.
This keyword mathes on ftp operation STOR and RETR. It will allow
rules writer to select if the alert has to be on a put or a fetch
operation.
It is now possible to write a signature like:
alert ftp-data any any -> any any (msg:"FTP data get firwmare"; ftdata_command:retr; sid:2; rev:1;)
to alert when a file is retrieved from a FTP server.
Use expectation to be able to identify connections that are
ftp data. It parses the PASV response, STOR message and the
RETR message to provide extraction of files.
Implementation in Rust of FTP messages parsing is available.
Also this patch changes some var name prefixed by ssh to ftp.
This patch provides a working expectation system. This will allow
suricata to have a way to identify parallel connections opened by
a protocol such as FTP.
Expectation are a chained list and there is a cleaning by timeout
of the entries.
This patch also defined a counter of expectations that is also
used to check if we need to query IPPairs. This way we only query
the IPPairs store if we have an expectation.
This patch adds a parent_id field to the Flow structure that
contain the flow ID of the parent connection for protocol with
dynamic parallel connection opening like FTP.
This permits to handle memcap values through
unix socket for:
- stream
- stream-reassembly
- flow
- applayer-proto-http
- defrag
- ippair
- host
It will be possible to show or change a memcap value
for a specified configuration and list all memcap values
available.
The following commands are registered for unix-socket:
- memcap-set
- memcap-show
- memcap-list
Output:
>>> memcap-show flow
Success:
{
"value": "64mb"
}
>>> memcap-set flow 64mb
Success:
"memcap value for 'flow' updated: 67108864"
Command with invalid memcap key:
>>> memcap-set udp 32mb
Error:
"Available config: stream stream-reassembly flow applayer-proto-http defrag ippair host"
Command with an invalid memcap value:
>>> memcap-set http 32mmb
Error:
"error parsing memcap specified, value not changed"
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
Another function is added to gets
the memuse value since it will be shown
through unix-socket.
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
Another function is added to gets
the memuse value since it will be shown
through unix-socket.
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
Another function is added to gets
the memuse value since it will be shown
through unix-socket.
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
FlowGetMemuse() function is made as public
because the memuse value will be shown
through unix-socket.
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
According to all checking memcap functions,
the size passed as argument is declared as uint64_t
except for StreamTcpReassembleCheckMemcap where it's
defined as uint32_t.
If the hash function returns an index greater than the array size of the
hash table, the index is not checked. Even if this is the responsibility
of the caller, add a safety check to avoid errors.
We did had a race condition with running logrotate with multiple
EVE Json files. Consequence was one of the file not being reopen
by suricata that did continue to write to the rotated one.
Trying fix on signal handler did fail so this patch implements
log rotation support by adding a dedicated command to unix socket
to reopen the log files.
Multiple NULL-pointer dereferences after ConfGet in PostConfLoadedSetup can cause suricata to terminate with segfaults. The ASAN-output:
ASAN:DEADLYSIGNAL =================================================================
5734ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f1a9a3967cc bp 0x7ffdff033ad0 sp 0x7ffdff033250 T0)
0 0x7f1a9a3967cb (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb)
1 0x55ba65f66f27 in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2652
2 0x55ba65f6870e in main /root/suricata-1/src/suricata.c:2898
3 0x7f1a96aeb2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
4 0x55ba65af9039 in _start (/usr/local/bin/suricata+0xc8039)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb)
This commit fixes Bug #2370 by replacing ConfGet by ConfGetValue
There are multiple NULL-pointer dereferences after calling ConfGetBool in StreamTcpInitConfig. ConfGetBool calls ConfGet which doesn't check the vptr-argument. This is a sample ASAN-output:
1453ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f2969b83a28 bp 0x7ffdbf613a90 sp 0x7ffdbf613210 T0)
0 0x7f2969b83a27 in strcasecmp (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x51a27)
1 0x564185accefd in ConfValIsTrue /root/suricata-1/src/conf.c:559
2 0x564185accb4f in ConfGetBool /root/suricata-1/src/conf.c:512
3 0x564185dcbe05 in StreamTcpInitConfig /root/suricata-1/src/stream-tcp.c:381
4 0x564185e21a88 in PreRunInit /root/suricata-1/src/suricata.c:2264
5 0x564185e24d2c in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2763
6 0x564185e2570e in main /root/suricata-1/src/suricata.c:2898
7 0x7f29662cb2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
8 0x5641859b6039 in _start (/usr/local/bin/suricata+0xc8039)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x51a27) in strcasecmp
1453ABORTING
This commit replaces ConfGet by ConfGetValue in ConfGetBool. This does not only fix Bug #2368 but might also fix others too.
Multiple NULL-pointer dereferences after ConfGet in HostInitConfig can cause suricata to terminate with segfaults. The ASAN-output:
==29747==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7ff937904646 bp
0 0x7ff937904645 in strlen (/lib/x86_64-linux-gnu/libc.so.6+0x80645)
1 0x7ff93b146eec (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x3beec)
2 0x5618387c86a3 in HostInitConfig /root/suricata-1/src/host.c:174
3 0x56183893eccb in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2752
4 0x56183893f70e in main /root/suricata-1/src/suricata.c:2898
5 0x7ff9378a42b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
6 0x5618384d0039 in _start (/usr/local/bin/suricata+0xc8039)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x80645) in strlen
This commit fixes Bug #2367
In some cases the rule reload could hang. The pending USR2 signals will
be recognized even with the <2 check. Also the SCLogWarning shouldn't be
used in the handler (see Warning about SCLog* API above in the code).
Check counter id before updating a counter. In case of a disabled
parser with the protocol detection enable, the id can be 0. In
debug mode this would lead to a BUG_ON.
https://redmine.openinfosecfoundation.org/issues/2353
Command line option pcap-file-continuous was ignoring command line options
passed after its usage. Fixed bug, fixed formatting of help command
regarding pcap-file-continuous.
There are several NULL-pointer derefs in StreamTCPInitConfig. All of them happen because ConfGet returns 1 even if the value is NULL(due to misconfiguration for example).
This commit introduces a new function "ConfGetValue". It adds return values for NULL-pointer to ConfGet and could be used as a replacement for ConfGet.
Note: Simply modify ConfGet might not be a good idea, because there are some places where ConfGet should return 1 even if "value" is NULL. For example if ConfGet should get a Config-Leave in the yaml-hierarchy.
Bug: 2354
This commit fixes multiple NULL-pointer dereferences in FlowInitConfig after reading in config-values(flow.hash-size, flow.prealloc and flow.memcap) for flow. Here is a sample ASAN-output:
=================================================================
ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fea73456646 bp 0x7fffd70e1ba0 sp 0x7fffd70e1328 T0)
0 0x7fea73456645 in strlen (/lib/x86_64-linux-gnu/libc.so.6+0x80645)
1 0x7fea76c98eec (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x3beec)
2 0x5643efb4c205 in FlowInitConfig /root/suricata-1/src/flow.c:455
3 0x5643efcd1751 in PreRunInit /root/suricata-1/src/suricata.c:2247
4 0x5643efcd49f4 in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2748
5 0x5643efcd5402 in main /root/suricata-1/src/suricata.c:2884
6 0x7fea733f62b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
7 0x5643ef8761a9 in _start (/usr/local/bin/suricata+0xc51a9)
Ticketno: Bug #2349
The "sig_file" argument of DetectLoadCompleteSigPath() is not checked for NULL-values. If this argument is NULL a SEGV occurs because of a dereferenced NULL-pointer in strlen in PathIsAbsolute. This commit fixes bug #2347. Here is the ASAN-output:
==17170==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fd1afa00646 bp 0x7ffe8398e6d0 sp 0x7ffe8398de58 T0)
0 0x7fd1afa00645 in strlen (/lib/x86_64-linux-gnu/libc.so.6+0x80645)
1 0x7fd1b3242eec (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x3beec)
2 0x5561c8cddf7f in PathIsAbsolute /root/suricata-1/src/util-path.c:40
3 0x5561c8cddfea in PathIsRelative /root/suricata-1/src/util-path.c:65
4 0x5561c89275e4 in DetectLoadCompleteSigPath /root/suricata-1/src/detect.c:264
5 0x5561c8929e75 in SigLoadSignatures /root/suricata-1/src/detect.c:486
6 0x5561c8c0f2b3 in LoadSignatures /root/suricata-1/src/suricata.c:2419
7 0x5561c8c1051d in PostConfLoadedDetectSetup /root/suricata-1/src/suricata.c:2550
8 0x5561c8c12424 in main /root/suricata-1/src/suricata.c:2887
9 0x7fd1af9a02b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
10 0x5561c87b31a9 in _start (/usr/local/bin/suricata+0xc51a9)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x80645) in strlen
The value for the configuration-option "unix-command.enabled" is not properly checked in ConfUnixSocketIsEnable. This causes a NULL-pointer dereference in strcmp. This commit fixes bug #2346. The ASAN-output looks like:
ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f03b69737cc bp 0x7ffcef322c10 sp 0x7ffcef322390 T0)
0 0x7f03b69737cb (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb)
1 0x5617a76d3f55 in ConfUnixSocketIsEnable /root/suricata-1/src/util-conf.c:104
2 0x5617a741b6e7 in DetectEngineMultiTenantSetup /root/suricata-1/src/detect-engine.c:2447
3 0x5617a769e0c3 in PostConfLoadedDetectSetup /root/suricata-1/src/suricata.c:2527
4 0x5617a76a0424 in main /root/suricata-1/src/suricata.c:2887
5 0x7f03b30c82b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
6 0x5617a72411a9 in _start (/usr/local/bin/suricata+0xc51a9)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb
There is a memory-leak in DetectAddressTestConfVars. If the programm takes the "goto error"-path, the pointers gh and ghn will not be freed. This commit fixes bug #2345. Here is the ASAN-output:
=================================================================
ERROR: LeakSanitizer: detected memory leaks
Direct leak of 24 byte(s) in 1 object(s) allocated from:
0 0x7f4347cb1d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
1 0x55fe1fc8dcfc in DetectAddressHeadInit /root/suricata-1/src/detect-engine-address.c:1534
2 0x55fe1fc8c50a in DetectAddressTestConfVars /root/suricata-1/src/detect-engine-address.c:1306
3 0x55fe1ff356bd in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2696
4 0x55fe1ff365eb in main /root/suricata-1/src/suricata.c:2884
5 0x7f43443892b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
Direct leak of 24 byte(s) in 1 object(s) allocated from:
0 0x7f4347cb1d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
1 0x55fe1fc8dcfc in DetectAddressHeadInit /root/suricata-1/src/detect-engine-address.c:1534
2 0x55fe1fc8c524 in DetectAddressTestConfVars /root/suricata-1/src/detect-engine-address.c:1310
3 0x55fe1ff356bd in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2696
4 0x55fe1ff365eb in main /root/suricata-1/src/suricata.c:2884
5 0x7f43443892b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
SUMMARY: AddressSanitizer: 48 byte(s) leaked in 2 allocation(s).
If output log reopen fails, don't try to output the error. This would
lead to a deadlock as reopen was called from a SCLogMessage call. This
call already held the output lock.
Bug #2306.
Propagate inspection limits from anchered keywords to the rest of
a rule.
Examples:
content:"A"; depth:1; is anchored, it can only match in the first byte
content:"A"; depth:1; content:"BC"; distance:0; within:2;
"BC" can only be in the 2nd and 3rd byte of the payload. So effectively
it has an implicite offset of 1 and an implicit depth of 3.
content:"A"; depth:1; content:"BC"; distance:0; can assume offset:1; for
the 2nd content.
content:"A"; depth:1; pcre:"/B/R"; content:"C"; distance:0; can assume
at least offset:1; for content "C". We can't analyzer the pcre pattern
(yet), so we assume it matches with 0 bytes.
Add lots of test cases.
Currently, when live reload is executed through
unix-socket, suri prints in the console the following
error message:
"Live rule reload not possible if -s or -S option used at runtime."
Instead, prints "done" in unix socket,
when the live reload is not executed.
This permits to print the invalid rules through
unix socket.
An example output is the following:
>>> show-failed-rules
Success:
[
{
"filename": "/home/eric/git/oisf/benches/tls-store.rules",
"line": 2,
"rule": "alert ts any any -> any 334 (msg:\"Store TLS\"; tls.store; sid:2; rev:1;)"
},
{
"filename": "/home/eric/git/oisf/benches/tls-store.rules",
"line": 3,
"rule": "alert ls any any -> any 334 (msg:\"Store TLS\"; tls.store; sid:3; rev:1;)"
}
]
The dump is limited to 20 entries to avoid to send a too big
message to the client that don't support it by default.
Add a non blocking function to reload rules. It will be useful
for remote system management to avoid to block them waiting the
reload. And as we now have a last-reload command we can get the
status of the current reload.
Remove the DONE state to fix a problem with state not being
changed correctly when multiple reload were done. As DONE was
not really useful, we can remove it.
This adds the engine stats in stats event.
If multi-tenancy is enabled, it will add
stats for each tenant
The following is a snippet of the generated EVE entry:
"detect":{"engines":[{"last_reload":"2015-10-13T09:59:48.044996+0200","rules_loaded":17184,"rules_failed":0}],"alert":28}
Multi-tenancy enabled:
"detect":{"engines":[{"id":1,"last_reload":"2015-10-13T09:56:46.447153+0200","rules_loaded":17084,"rules_failed":0},
{"id":2,"last_reload":"2015-10-13T09:56:36.504877+0200","rules_loaded":3268,"rules_failed":0}],
"alert":28}
This takes the form of an option to add the pid of the process to file
names. Additionally, it adds a suffix to the file name to indicate it is
not finalized.
Adding the pid to the file name reduces the likelihood that a file is
overwritten when suricata is unexpectedly killed. The number in the
waldo file is only written out during a clean shutdown. In the event
of an improper shutdown, extracted files will be written using the old
number and existing files with the same name will be overwritten.
Writes extracted files and their metadata to a temporary file suffixed
with '.tmp'. Renames the files when they are completely done being
written. As-is there is no way to know that a file on disk is still
being written to by suricata.
Don't use pcre for the high level rule parsing, instead
using a tokenizing parser for breaking out the rule
into keywords and options.
Much faster, especially on older CPUs. Should also allow
us to provide better context where a rule parse error
occurs.
Converting the NTP parser to the new registration method is a simple,
3-steps process:
- change the extern functions to use generic input parameters (functions
in all parsers must share common types to be generic) and cast them
- declare the Parser structure
- remove the C code and call the registration function
Add a common structure definition to contain all required elements
required for registering a parser.
This also reduces the risk of incorrectly registering a parser: the
compiler will type-check functions.
The registration function allows factorization of the code. It can be
used to register parsers, but is not mandatory.
If extra registration code (for functions not in the structure)
it is still possible by calling the C functions after the registration.
Add StringToAppProto to map a protocol name to a AppProto.
Exposing this function is required to let parsers discover their
AppProto identifier constant dynamically.
For example, a parser can request this value, and use it for
registration without knowing the value.
pfring.h brings a different version of likely/unlikely that gives
warnings. So make sure we include our own before.
Make sure pfring.h isn't included globally due to apparent redefinition
of pthread_rwlock_t.
This patch adds support for hw bypass by enabling flow offload in the network
card (when supported) and implementing the BypassPacketsFlow callback.
Hw bypass support is disabled by default, and can be enabled by setting
"bypass: yes" in the pfring interface configuration section in suricata.yaml.
Netflow entry collects the minimum and maximum
time to live during the life of the incoming flow.
This adds those field to a netflow event.
Signed-off-by: Eric Leblond <eric@regit.org>
Reimplement keyword to match on SHA-1 fingerprint of TLS
certificate as a mpm keyword.
alert tls any any -> any (msg:"TLS cert fingerprint test";
tls_cert_fingerprint;
content:"4a:a3:66:76:82:cb:6b:23:bb:c3:58:47:23:a4:63:a7:78:a4:a1:18";
sid:12345;)
https://redmine.openinfosecfoundation.org/issues/2215
Fixes issue with events being dropped since socket was non-blocking for
offline run modes.
Add a method for determining offline from run mode. Make sure SCInstance
offline is set correctly. Use current run mode to set socket flags.
SCLogMapLogLevelToSyslogLevel(): treat SC_LOG_PERF messages as LOG_DEBUG
Previously, when logging to syslog, perf events had a default EMERG priority,
which could be a bit confusing.
An empty value for coredump.max-dump in the config-file leads to a segfault because of a NULL-pointer dereference in CoredumpLoadConfig().
Here is a configuration example:
coredump.max-dump: []
This lets suricata crash with a segfault:
ASAN-output:
==9412==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f22e851aa28 bp 0x7ffd90006fc0 sp 0x7ffd90006740 T0)
0 0x7f22e851aa27 in strcasecmp (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x51a27)
1 0x5608a7ec0108 in CoredumpLoadConfig /root/suricata-1/src/util-coredump-config.c:52
2 0x5608a7e8bb22 in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2752
3 0x5608a7e8c577 in main /root/suricata-1/src/suricata.c:2892
4 0x7f22e4c622b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
5 0x5608a7a30c59 in _start (/usr/local/bin/suricata+0xc4c59)
Bug #2276
If there are empty values in the config-file where integer values are expected, strtoimax in the ConfGetInt-function will segfault because of NULL-pointer dereference.
Here is a configuration example:
pcre.match-limit: []
This will let suricata crash with a segfault.
ASAN-output:
ASAN:DEADLYSIGNAL =================================================================
16951ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fa690e3ccc5 bp 0x000000000000 sp 0x7ffd0d770ad0 T0)
0 0x7fa690e3ccc4 (/lib/x86_64-linux-gnu/libc.so.6+0x36cc4)
1 0x7fa6946a6534 in strtoimax (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x44534)
2 0x55e0aeba6499 in ConfGetInt /root/suricata-1/src/conf.c:390
3 0x55e0aed2545d in DetectPcreRegister /root/suricata-1/src/detect-pcre.c:99
4 0x55e0aec1b4ce in SigTableSetup /root/suricata-1/src/detect.c:3783
5 0x55e0aeeed58d in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2690
6 0x55e0aeeee4f2 in main /root/suricata-1/src/suricata.c:2892
7 0x7fa690e262b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
8 0x55e0aea92d39 in _start (/usr/local/bin/suricata+0xc7d39)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x36cc4)
This commit fixes Ticket #2275
If someone accidently writes invalid characters in some parts of the suricata.yaml-configfile, the size-parameter of the ParseSizeString-function becomes NULL and gets dereferenced. Suricata crashes with SEGV. This commit fixes Ticket #2274
The following config value leads to a Segfault:
app-layer.protocols.smtp.inspected-tracker.content-inspect-window: *4096
If 'raw' content patterns were used in a dns_query rule, the raw
patterns would only be evaluated for TCP, but not for UDP.
This patch adds the inspection for UDP as well.
Bug #2263.
Leak in error path as seen by scan-build:
CC detect-engine-port.o
detect-engine-port.c:1083:13: warning: Potential leak of memory pointed to by 'temp_rule_var_port'
return -1;
^
The older random functions returned random values in the range of
0 - RAND_MAX. This is what the http randomize code was expecting.
Newer methods, based on getrandom (or probably Windows too), return
a much large range of values, including negative values and >RAND_MAX.
This patch adds a wrapper to turn the returned value into the expected
range before using it in the http code.
The same is true for the stream engine.
Interface name shortening introduces double periods ('..') as spacers,
which cause issues during JSON stats serialization as there '.'
characters are also used as separators to define nesting of the JSON
output. This commit makes sure that '..' are skipped during tokenizing.
Fixes Redmine bug #2208.
With large number of threads the default memcaps lead to pool setup
failures. Make sure these are reported properly so that the user
knows what is going on.
Bug: #2226
Fixes incorrect variable in ticket #2207
In app-layer-dns-tcp.c in the DNSTCPResponseParse function
a variable is set to last_req when it should be last_resp.
This makes it consistent with UDP DNS response parsing.
There is no need to try to set erroneous BPF filter again and again. Such attempts
lead to constant mmap() calls without corresponding munmap() when 'use-mmap' is enabled.
This adds a new redis mode rpush. Also more consistent config keywords orientated at the redis command: lpush and publish.
Keeping list and channel config keywords for backwards compatibility
Decoding client certificate overwrites the validity dates from the
server certificate, so we therefore don't decode it, since we don't
do anything with it (right now) anyway.
Fixes Bug #2050
added util-napatech module which contains implementation threads
for processing statistics. And modified source-napatech and
runmode-napatech to instantiate the threads.
napatech: Implementation of packet counters
napatech: implementation of statistics counters
napatech: Implementation of packet counters.
napatech: added util-napatech module
napatech: added utils-napatech module.
added include declaration and napatech specific structure when HAVE_NAPATECH
is defined.
Added util-napatech module to project.
Stub data buffer could grow without limit depending on traffic.
This patch improves the handling. It honors the 'last frag' setting
and implements a hard limit of 1MB per buffer.
Bug #2186
The reason the stream engine can't easily decide to bypass streams
is that there can be non-stream dependent rules that wouldn't match
if bypassing is done too aggressively.
However, if there is no detection engine, there is no reason to hold
back. In this case we can bypass as soon as the stream engine is done
with a session.
This patch turn on code optimization on BPF filter building by
libpcap. This allow to reduce the size of the BPF bytecode and
thus increase the size of BPF filter supported by Suricata.
Reported-by: Martijn van Oosterhout
Rules can contain conflicting statements and lead to a unmatchable rule.
2 examples are rejected by this patch:
1. dsize < content
2. dsize < content@offset
Bug #2187
GCC7 said:
CC util-radix-tree.o
In file included from util-debug-filters.h:29:0,
from util-debug.h:34,
from suricata-common.h:421,
from util-radix-tree.c:26:
util-radix-tree.c: In function ‘SCRadixAddKey’:
util-mem.h:177:12: error: argument 1 range [18446744071562067968, 18446744073709551615] exceeds maximum object size 9223372036854775807 [-Werror=alloc-size-larger-than=]
ptrmem = malloc((a)); \
~~~~~~~^~~~~~~~~~~~~
util-radix-tree.c:749:42: note: in expansion of macro ‘SCMalloc’
if ( (inter_node->netmasks = SCMalloc((node->netmask_cnt - i) *
^~~~~~~~
In file included from suricata-common.h:69:0,
from util-radix-tree.c:26:
/usr/include/stdlib.h:443:14: note: in a call to allocation function ‘malloc’ declared here
extern void *malloc (size_t __size) __THROW __attribute_malloc__ __wur;
^~~~~~
scan-build said:
util-radix-tree.c:749:42: warning: Call to 'malloc' has an allocation size of 0 bytes
if ( (inter_node->netmasks = SCMalloc((node->netmask_cnt - i) *
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-mem.h:177:14: note: expanded from macro 'SCMalloc'
ptrmem = malloc((a)); \
^~~~~~~~~~~
1 warning generated.
Example:
util-runmodes.c: In function ‘RunModeSetIPSAutoFp’:
util-runmodes.c:496:40: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=]
snprintf(qname, sizeof(qname), "pickup%d", thread+1);
^~~~~~~~~~
util-runmodes.c:496:9: note: ‘snprintf’ output between 8 and 17 bytes into a destination of size16
snprintf(qname, sizeof(qname), "pickup%d", thread+1);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Solved by reducing 'thread' to a uint16_t and limiting the max
thread count to 1024.
The http_host keyword checks if the regex contains uppercase characters.
This check was rejecting valid syntax in the following format:
content:"|2E|suricata"; http_host; pcre:"/\x2Esuricata$/W";
This patch addresses this case.
Bug #1957.
The application log is subject to rotation, so the check for
rotation, the actual rotation and write needs to be done under
lock to ensure the file pointer is in a consisten state
at the time of write().
Fixes issue:
https://redmine.openinfosecfoundation.org/issues/2155
On OpenBSD 6.0 and 6.1 the following pcap gets a datalink type of
101 instead of our defined DLT_RAW.
File type: Wireshark/tcpdump/... - pcap
File encapsulation: Raw IP
File timestamp precision: microseconds (6)
Packet size limit: file hdr: 262144 bytes
Number of packets: 23
File size: 11 kB
Data size: 11 kB
Capture duration: 7,424945 seconds
First packet time: 2017-05-25 21:59:31,957953
Last packet time: 2017-05-25 21:59:39,382898
Data byte rate: 1536 bytes/s
Data bit rate: 12 kbps
Average packet size: 496,00 bytes
Average packet rate: 3 packets/s
SHA1: 120cff9878b93ac74b68fb9216027bef3b3c018f
RIPEMD160: 35fa287bf30d8be8b8654abfe26e8d3883262e8e
MD5: 13fe4bc50fe09bdd38f07739bd1ff0f0
Strict time order: True
Number of interfaces in file: 1
Interface #0 info:
Encapsulation = Raw IP (7/101 - rawip)
Capture length = 262144
Time precision = microseconds (6)
Time ticks per second = 1000000
Number of stat entries = 0
Number of packets = 23
On Linux it is 12.
On the tcpdump/libpcap site the DLT_RAW is defined as 101:
http://www.tcpdump.org/linktypes.html
Strangely, on OpenBSD the DLT_RAW macro is defined as 14 as expected.
So for some reason, libpcap on OpenBSD uses 101 which seems to match
the tcpdump/libpcap documentation.
So this patch adds support for datalink 101 as RAW.
Certain rules can apply to both TCP and UDP. For example 'alert dns'
rules are inspected against both TCP and UDP. This lead to the
stream inspect engine being called on a UDP packet.
This patch fixes the issue by exiting early from the stream inspect
engine if a) proto is not TCP or b) ssn is not available
Bug #2158.
Clean up option parsing. Allow options to be disabled as well as
enabled.
E.g.
metadata: true
flow: false
The metadata setting will enable all. Then flow is disabled.
This patch adds a partial flow entry in the alert event
(if applayer or flow is selected) or simply app_proto if
it is not.
app_proto is useful as filter and aggregation field. And
the partial flow entry contains more information about the
proto as well as some volumetry info.
Observed:
STARTTLS creates 2 pseudo packets which are tied to a real packet.
TPR (tunnel packet ref) counter increased to 2.
Pseudo 1: goes through 'verdict', increments 'ready to verdict' to 1.
Packet pool return code frees this packet and decrements TPR in root
to 1. RTV counter not changed. So both are now 1.
Pseudo 2: verdict code sees RTV == TPR, so verdict is set based on
pseudo packet. This is too soon. Packet pool return code frees this
packet and decrements TPR in root to 0.
Real packet: TRP is 0 so set verdict on this packet. As verdict was
already set, NFQ reports an issue.
The decrementing of TPR doesn't seem to make sense as RTV is not
updated.
Solution:
This patch refactors the ref count and verdict count logic. The beef
is now handled in the generic function TmqhOutputPacketpool(). NFQ
and IPFW call a utility function VerdictTunnelPacket to see if they
need to verdict a packet.
Remove some unused macro's for managing these counters.
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
In normal records it will try to continue parsing.
GAP 'data' will be passed to file api as '0's. New call is used
so that the file API does know it is dealing with a GAP. Such
files are flagged as truncated at the end of the file and no
checksums are calculated.
TCP reassembly is now deactivated more frequently and triggering a
bypass on it is resulting in missing some alerts due forgetting
about packet based signature.
So this patch is introducing a dedicated flag that can be set in
the app layer and transmitted in the streaming to trigger bypass.
It is currently used by the SSL app layer to trigger bypass when
the stream becomes encrypted.
Only the tests that make sense were copied over, those testing
correlation of responses to requests were not.
Also, remove compiler warning when not built with
unit tests.
IDMEF alert contains two entities named Source and Target that are
defined using common language:
* "The Source class contains information about the possible source(s) of
the event(s) that generated an alert."
* "The Target class contains information about the possible target(s) of
the event(s) that generated an alert."
Previous alerts event were not following that so we can updated the code
when we know the direction thanks to the metadata field.
Use metadata provided information to output the Source and Target
in the definition of IDMEF.
The output is now the following:
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 1,
"rev": 1,
"signature": "connection to home",
"category": "",
"severity": 3,
"source": {
"ip": "2001:31d0:000a:f68a:0000:0000:0000:0001",
"port": 80
},
"target": {
"ip": "2a01:0e34:ee97:b130:c685:08ff:dab3:c9c8",
"port": 48390
}
The target keyword allows rules writer to specify information about
target of the attack. Using this keyword in a signature causes
some fields to be added in the EVE output. It also fixes ambiguity
in the Prelude output.
Pascal Delalande
Danny Browning
Martin Natano
Victor Julien
Maurizio Abba
Ruslan Usmanov
Jason Ish
Eric Leblond
Giuseppe Longo
Pierre Chifflier
Wolfgang Hotwagner
Andreas Herz
Dana Helwig
Clément Galland
Gaurav Singh
Alfredo Cardigliano
Mats Klepsland
Alexander Gozman
Sascha Steinbiss
qiangbei
Derek
Alexander Gozman
Julian
Phil Young