If a signature didn't explicitly specified 'dcerpc' or 'smb' as the
app proto, false positives on other traffic could happen. This was
caused by the sig not having a app_proto set. This isn't set as the
rule is supposed to match against either ALPROTO_DCERPC or ALPROTO_SMB.
To avoid adding runtime costs for checking for both protocols, this
patch adds a new flag for DCERPC in the 'mask' logic. The flag is set
on the sig if dce_* keywords are present and set on the packet if the
flow's app proto is either ALPROTO_DCERPC or ALPROTO_SMB.
Bug #2559
Reported-by: Jason Taylor
In case of a valid RST on a SYN, the state is switched to 'TCP_CLOSED'.
However, the target of the RST may not have received it, or may not
have accepted it. Also, the RST may have been injected, so the supposed
sender may not actually be aware of the RST that was sent in it's name.
In this case the previous behavior was to switch the state to CLOSED and
accept no further TCP updates or stream reassembly.
This patch changes this. It still switches the state to CLOSED, as this
is by far the most likely to be correct. However, it will reconsider
the state if the receiver continues to talk.
To do this on each state change the previous state will be recorded in
TcpSession::pstate. If a non-RST packet is received after a RST, this
TcpSession::pstate is used to try to continue the conversation.
If the (supposed) sender of the RST is also continueing the conversation
as normal, it's highly likely it didn't send the RST. In this case
a stream event is generated.
Ticket: #2501
Reported-By: Kirill Shipulin
https://redmine.openinfosecfoundation.org/issues/2417
Add option to have pcap files deleted after they have been processed.
This option combines well with pcap file continuous and streaming
files to a directory being processed.
Calls with both START and MIDSTREAM mean the record might be cut and the
start of it could be missing. For this case, enable the same logic as is
used when catching up after a GAP. Search for the start of the record
instead of assuming it sits exactly at the start of the input data.
Last multi-detect changes broken delayed-detect by refusing to reload
a 'stub' detect engine. This patch distinguishes between a stub for
multi-tenancy and for delayed detect.
There are 3 types of detect engine objects:
1. normal
The normal detection engine if no multi-tenancy is in use
2. tenant
A per tenant detection engine
3. stub
A stub (or minimal as it was called before) detect engine
that is needed to have something in place when there are
only tenants.
A stub is also used in case of 'delayed detect', where we
need a minimal detect engine to start up which is replaced
by a full (normal type) detect engine after startup.
This patch adds a new field 'type' to the DetectEngineCtx object
to distinguish between the types. This replaces the boolean 'minimal'.
The global keyword registration and per thread init handling used
the lock from the DetectEngineMasterCtx. This lead to a dead lock
situation at multi-tenancy tenant reloads.
The lock was unnecessary however, as the only time the registration
list is updated is at engine initialization. At that time Suricata
is still running in a single thread. After this, the data structure
doesn't change anymore.
Bug #2516.
Enables IPS functionality on Windows using the open-source
(LGPLv3/GPLv2) WinDivert driver and API.
From https://www.reqrypt.org/windivert-doc.html : "WinDivert is a
user-mode capture/sniffing/modification/blocking/re-injection package
for Windows Vista, Windows Server 2008, Windows 7, and Windows 8.
WinDivert can be used to implement user-mode packet filters, packet
sniffers, firewalls, NAT, VPNs, tunneling applications, etc., without
the need to write kernel-mode code."
- adds `--windivert [filter string]` and `--windivert-forward [filter
string]` command-line options to enable WinDivert IPS mode.
`--windivert[-forward] true` will open a filter for all traffic. See
https://www.reqrypt.org/windivert-doc.html#filter_language for more
information.
Limitation: currently limited to `autofp` runmode.
Additionally:
- `tmm_modules` now zeroed during `RegisterAllModules`
- fixed Windows Vista+ `inet_ntop` call in `PrintInet`
- fixed `GetRandom` bug (nonexistent keys) on fresh Windows installs
- fixed `RandomGetClock` building on Windows builds
- Added WMI queries for MTU
Much of encrypted traffic is uninteresting to Suricata. Once encrypted
communication starts, inspecting the packet payloads is generally
not interesting anymore. The default behavior is to disable the parts
of the detection engine and stream reassembly that relate to raw content
inspection.
The tls app-layer parser also had a crude option to affect this behavior:
set 'no-reassemble' to true went much further than the default behavior.
It disabled the TCP reassembly on the flow completely, disabled all
inspection on the flow and enabled bypass if available.
This patch adds a new option: full inspection. This continues to treat
a TLS session as any other, so without any limits to inspection.
The new option is implemented in a new config option 'encrypt-handling',
that replaces 'no-reassemble'. The new option has 3 values:
'default', 'full' and 'bypass'. Default is the current default behavior,
'bypass' is the current 'no-reassemble = true' behavior and 'full'
is the new full inspection mode.
The use of stream_size in combination with raw content matches is an
indication that the rule needs to be evaluated per packet, not just
per reassembled stream chunk.
This is a DHCP decoder and logger written in Rust. Unlike most
parsers, this one is stateless so responses are not matched
up to requests by Suricata. However, the output does contain
enough fields to match them up in post-processing.
Rules are included to alert of malformed or truncated options.
Before setting up a sub eve-logger, check that it is enabled. This
allows us to set "enabled: no" for loggers that are not registered
with the system without generating an error. An example of this
is loggers that are only available with Rust.
getrandom syscall availability is detected at runtime. So it is
possible that the build is done on a box that supports it but
the run is done on a system with no availability. So a workaround
solution is needed to fix this case.
Also we have seen some issue in docker environment where the build
is detecting getrandom but where it does not work at runtime.
For both reasons, the code is updated to have a call to a fallback
function if ever the getrandom call returns that the syscall is
not available.
Fix Coverity issue:
** CID 1435535: Null pointer dereferences (REVERSE_INULL)
/src/output-json-file.c: 212 in JsonBuildFileInfoRecord()
Where we check a variable for being NULL, when all paths to the
code show that it can't be NULL.
XFF configuration is already set in app-layer-htp-xff, and in
output-json-alert. Extending XFF configuration to files and HTTP allow
to get the same behavior as for alerts.
Extend the configuration of filestore json to let filestore metafile
dump be aware of xff. This is available only if write-fileinfo is set
to yes and file-store version is 2.
When a file is transferred over anything other than HTTP, the previously hard-coded HTTP protocol would trigger a non-existent index into htp_list_array_get(), causing a segfault. This patch mimics the logic in detect-lua-extensions.c.
Validate that the content that follows the 'tls_cert_serial' keyword
is on the correct form. If it's longer than two bytes it should be
separated by colons.
Introduces the option 'outputs.pcap-log.compression' which can be set
to 'none' or 'lz4', plus options to set the compression level and to
enable checksums. SCFmemopen is used to make pcap_dump() write to a
buffer which is then compressed using liblz4.
When traffic is becoming null (mainly seen in tests) we reach the
situation where there is timeouts in the poll on the socket and
only that. Existing code is then just looping on the poll and
the result is that the packet iface counters are not updated.
This patch calls the dump counter function to be sure to get
the counter right faster (and not only right at exit).
The stream depth computation was partly done with the stream_config
depth instead of using the value in the TCP session. As a result,
some configuration were resulting in abnormal behavior.
In particular, when stream depth was 0 and the file store depth was
not 0, Suricata was stopping the streaming on the flow as soon as
the filestore was started.
Reported-by: Pascal Delalande <pdl35@free.fr>
There is a difference in the size of the buffer length as passed from
the content buffers (cfr HttpReassembledBody.buffer_len) and the buflen
variable passed to mpm primitives. This can cause a misdetection
whenever the bufferlen is multiple of 65536 (as uint16(X*65536) == 0).
Increasing the buflen variable type to uint32 solves the issue (this
does not cause any issue with primitives, they all accept uint32).
If a file transfer stops on flow timeout, it won't be closed or
truncated. This patch makes sure that in such cases the files
are indeed truncated. This fixes the filestore-v2 output module,
as that requires a sha256 for storing the partial file correctly.
CID 1428797 (#1 of 1): Unchecked return value (CHECKED_RETURN)
check_return: Calling HashTableAdd without checking return value
(as is done elsewhere 5 out of 6 times).
Don't treat 'external' parsers as more experimental. All parsers
depend on crates to some extend, and all have C glue code. So the
distinction doesn't really make sense.
Add a new parser for Internet Key Exchange version (IKEv2), defined in
RFC 7296.
The IKEv2 parser itself is external. The embedded code includes the
parser state and associated variables, the state machine, and the
detection code.
The parser looks the first two messages of a connection, and analyzes
the client and server proposals to check the cryptographic parameters.
https://redmine.openinfosecfoundation.org/issues/2451
When a missing (or empty named) file is passed to source-pcap-file while
using unix socket, the pcap processing thread will incorrectly be stopped,
and no longer available for subsequent files.
As an optimization, reset bidirectional flag for rules with same src and dst.
If one created bidirectional rule like 'alert tcp any any <> any any ...',
the rule was checked twice (for each packet in every direction). This is
suboptimal and may give duplicated alerts. To avoid this, bidirectional
rules are now checked for the same src and dst (addresses and ports) and
if it's the case, the rule is treated as unidirectional and a corresponding
message is logged.
Add Ja3GetString() to return the content of the JA3 string buffer from the
TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3_string.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3_string = Ja3GetString()
if ja3_string == nil then
return
end
file:write(ja3_string .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
Add Ja3GetHash() to return the content of the JA3 hash buffer from the
TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3_hash.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3_hash = Ja3GetHash()
if ja3_hash == nil then
return
end
file:write(ja3_hash .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
In the (useless) example above, each JA3 hash is logged to a log file.
Match on JA3 hash using ja3_hash keyword, e.g:
alert tls any any -> any any (msg:"JA3 hash test";
ja3_hash;
content:"e7eca2baf4458d095b7f45da28c16c34";
sid:1;)
This changes LogQuery and LogAnswer functions
returning a json object instead of writing it in a log file.
In this way it's possible to reuse them to add dns info
into an alert.
The following is an alert record with dns:
{
"timestamp": "2017-07-31T15:01:17.885281+0200",
"event_type": "alert",
"src_ip": "8.8.8.8",
...
"dns": {
"query": [
{
"type": "query",
"id": 25394,
"rrname": "notifications.google.com",
"rrtype": "A",
"tx_id": 0
}
],
"answer": {
"type": "answer",
"id": 25394,
"rcode": "NOERROR",
"answers": [
{
"rrname": "notifications.google.com",
"rrtype": "CNAME",
"ttl": 3599,
"rdata": "plus.l.google.com"
},
{
"rrname": "plus.l.google.com",
"rrtype": "A",
"ttl": 299,
"rdata": "216.58.205.174"
}
]
}
}
}
This adds two new output formats that permits to reduce
the number of line logged for a dns answer because
actually an event is logged for each answer.
With this patch, only an event that contains all the answers
is logged.
The formats are named 'detailed' and 'grouped'.
The first format provides a list of answers with
the following fields:
- rrname
- rrdata
- ttl
- rdata
The second format provides a list of record data grouped
by their type.
The output below is an example of the formats:
{
"timestamp": "2017-11-29T10:27:18.148282+0100",
"flow_id": 268864910185905,
"in_iface": "wlp2s0",
"event_type": "dns",
"src_ip": "192.168.1.254",
"src_port": 53,
"dest_ip": "192.168.1.176",
"dest_port": 52609,
"proto": "UDP",
"dns": {
"type": "answer",
"id": 3654,
"rcode": "NOERROR",
"answers": [
{
"rrname": "wordpress.org",
"rrtype": "A",
"ttl": 544,
"rdata": "66.155.40.249"
},
{
"rrname": "wordpress.org",
"rrtype": "A",
"ttl": 544,
"rdata": "66.155.40.250"
}
],
"grouped": {
"A": [
"66.155.40.249",
"66.155.40.250"
]
}
}
}
This patch adds a new configuration for dns,
introducing a "version" that permits to switch
between the new and old format to provide
backward compatibility.
The new configuration is made up of these new fields:
- version
- requests (query)
- response (answer)
- types (custom)
When destination IP address does not suffice to uniquely identify
the Modbus/TCP device.
Some Modbus/TCP devices act as gateways to other Modbus/TCP devices
that are behind this gateways.
Implement SMB app-layer parser for SMB1/2/3. Features:
- file extraction
- eve logging
- existing dce keyword support
- smb_share/smb_named_pipe keyword support (stickybuffers)
- auth meta data extraction (ntlmssp, kerberos5)
The 'tx_id' variable was used to be passed into the IterFunc as a
minumum tx to return. The IterFunc could then return either the tx
for that id, or a later one if that turned out to be the first available
tx.
The tx_id however, was still used for some things as if it was the
current tx id. Most importantly for setting the tx id for alert
ammending. So this could lead to alerts with missing or wrong
applayer records.
If suricata was started with --init-errors-fatal and an error occured
during setup of lua output (like if lua scripts configured in the conf file
don't exist or are not readable) suricata continued, which did not reflect
"init errors fatal" very well.
This fix makes the suricata initialization abort and send an error message
in such cases.
For details see:
https://redmine.openinfosecfoundation.org/issues/1503
output-lua.c contained an include of detect.h.
Since we don't (and shouldn't) call any functions from detect.c in output-lua.c
and such coupling is generally unwanted this patch removes that include.
In extra-data mode, suricata does not output xff data without
undocumented conditions (including enabling packet output). This
behaviour has been fixed to remove the hidden requirements. Fix
included removing previous xff data output implementation and adding a
new function for outputting xff that is called after outputting each
event.
IPv6 XFF entries were also being recorded incorrectly as if they were
IPv4 and this has been fixed.
When upgrading to TLS from HTTP logging of the final HTTP tx could
have the wrong direction. This was due to the original packet triggering/
finalizing the upgrade would be used as the base for both the toserver
and toclient pseudo packet meaning it was wrong in one direction.
This patch creates a pseudo packet in the same way as the flow timeout
code does, so it no longer takes the raw original packet in.
Bug #2430
https://redmine.openinfosecfoundation.org/issues/2394
Certain parameters of delay and poll interval could cause newly added
files in a directory to be missed. Cleaned up how time is handled for
files in a directory and fix which time is used for future directory
traversals. Add a mutex to make sure processing time is not optimized
away.
When the filedata logger is enabled (file extraction), but a file is not
stored due to no rules matching to force this, the file would never be
freed.
This was caused by a check in the file pruning logic that only freed a
file when the FILE_STORED flag was set. However files can also have the
FILE_NOSTORE flag set which indicates that a file won't be stored.
This patch makes sure that both conditions lead to file pruning.
Allows matching on stickybuffers. Like dsize, it allows matching on
exact values, greater than and less than, and ranges.
For streaming buffers, such as HTTP bodies, the final size of the
body is only known at the end of the transaction.
As we can have multiple files per TX we use the multi inspect
buffer support.
By using this API file_data supports transforms.
Redo part of the flash decompression as a hard coded built-in sort
of transform.
Make sure content is applied to the transformed version of a buffer.
Support content with its modifiers, and also isdataat, pcre, bytetest
and bytejump.
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.
Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.
At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.
Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.
The transforms are called from within the prefilter engines themselves.
Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.
Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.
Add new registration calls for mpm/prefilters and inspect engines.
Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.
The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.
The v2 API assumes that registered MPM implements transformations.
Add util func to set new transform in rule and add util funcs for rule
parsing.
Remove DetectAppLayerInspectEngineRegister for TOCLIENT direction
because Modbus inspection engine is only performing in request (TOSERVER).
Detect Value keyword in read access rule. In read access, match on value
is not possible.
Update Modbus keyword documentation.
When loading an empty file, libyaml will fire a single scalar
event causing us to create a key that contains an empty string.
We're not interested in this, so skip an empty scalar value
when expecting a key.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2418
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
Main objective of the function is to be able to bypass a flow on
other interfaces. This is necessary in AF_PACKET case as the flow
table are per interface.
This patch adds a boolean option "xdp-cpu-redirect" to af-packet
interface configuration. If set, then the XDP filter will load
balance the skb creation on specified CPUs instead of doing the
creation on the CPU handling the packet. In the case of a card
with asymetric hashing this will allow to avoid saturating the
single CPU handling the trafic.
The XDP filter must contains a set of map allowing load balancing.
This is the case of xdp_filter.bpf.
Fixed-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
Device storage requires the devices to be created after storage
is finalized so we need to first get the list of devices then
create them when the storage is finalized.
This patch introduces the LiveDeviceName structure that is a list
of device name used during registration.
Code uses LiveRegisterDeviceName for pre registration and keep
using the LiveRegisterDevice function for part of the code that
create the interface during the runmode creation.
eBPF has a data type which is a per CPU array. By adding one element
to the array it is in fact added to all per CPU arrays in the kernel.
This allows to have a lockless structure in the kernel even when doing
counter update.
In userspace, we need to update the flow bypass code to fetch all
elements of the per CPU arrays.
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
This patch introduces the ebpf cluster mode. This mode is using
an extended BPF function that is loaded into the kernel and
provide the load balancing.
An example of cluster function is provided in the ebpf
subdirectory and provide ippair load balancing function.
This is a function which uses the same method as
the one used in autofp ippair to provide a symetrical
load balancing based on IP addresses.
A simple filter example allowing to drop IPv6 is added to the
source.
This patch also prepares the infrastructure to be able to load
and use map inside eBPF files. This will be used later for flow
bypass.
This adds a new module that permits to decompress
swf file compressed with zlib or lzma algorithms.
The API that performs decompression will take a compressed
buffer and build a new decompressed buffer following the
FWS format which represents an uncompressed file.
The maximum buffer that can be created is up to 50mb.
During the inspection phase actually is not possible to catch
an error if it occurs.
This patch permits to store events in the detection engine
such that we can match on events and catch them.
Victor Julien
Danny Browning
Jason Ish
Eric Leblond
Jacob Masen-Smith
fooinha
Pierre Chifflier
Maurizio Abba
Elazar Broad
Mats Klepsland
Max Fillinger
Giuseppe Longo
Alexander Gozman
Antoine LUONG
Antti Tönkyrä
David DIALLO
Richard Sailer
Thomas Andrejak
Daniel Humphries
Andreas Herz