CID 1428797 (#1 of 1): Unchecked return value (CHECKED_RETURN)
check_return: Calling HashTableAdd without checking return value
(as is done elsewhere 5 out of 6 times).
Don't treat 'external' parsers as more experimental. All parsers
depend on crates to some extend, and all have C glue code. So the
distinction doesn't really make sense.
Add a new parser for Internet Key Exchange version (IKEv2), defined in
RFC 7296.
The IKEv2 parser itself is external. The embedded code includes the
parser state and associated variables, the state machine, and the
detection code.
The parser looks the first two messages of a connection, and analyzes
the client and server proposals to check the cryptographic parameters.
https://redmine.openinfosecfoundation.org/issues/2451
When a missing (or empty named) file is passed to source-pcap-file while
using unix socket, the pcap processing thread will incorrectly be stopped,
and no longer available for subsequent files.
As an optimization, reset bidirectional flag for rules with same src and dst.
If one created bidirectional rule like 'alert tcp any any <> any any ...',
the rule was checked twice (for each packet in every direction). This is
suboptimal and may give duplicated alerts. To avoid this, bidirectional
rules are now checked for the same src and dst (addresses and ports) and
if it's the case, the rule is treated as unidirectional and a corresponding
message is logged.
Add Ja3GetString() to return the content of the JA3 string buffer from the
TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3_string.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3_string = Ja3GetString()
if ja3_string == nil then
return
end
file:write(ja3_string .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
Add Ja3GetHash() to return the content of the JA3 hash buffer from the
TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3_hash.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3_hash = Ja3GetHash()
if ja3_hash == nil then
return
end
file:write(ja3_hash .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
In the (useless) example above, each JA3 hash is logged to a log file.
Match on JA3 hash using ja3_hash keyword, e.g:
alert tls any any -> any any (msg:"JA3 hash test";
ja3_hash;
content:"e7eca2baf4458d095b7f45da28c16c34";
sid:1;)
This changes LogQuery and LogAnswer functions
returning a json object instead of writing it in a log file.
In this way it's possible to reuse them to add dns info
into an alert.
The following is an alert record with dns:
{
"timestamp": "2017-07-31T15:01:17.885281+0200",
"event_type": "alert",
"src_ip": "8.8.8.8",
...
"dns": {
"query": [
{
"type": "query",
"id": 25394,
"rrname": "notifications.google.com",
"rrtype": "A",
"tx_id": 0
}
],
"answer": {
"type": "answer",
"id": 25394,
"rcode": "NOERROR",
"answers": [
{
"rrname": "notifications.google.com",
"rrtype": "CNAME",
"ttl": 3599,
"rdata": "plus.l.google.com"
},
{
"rrname": "plus.l.google.com",
"rrtype": "A",
"ttl": 299,
"rdata": "216.58.205.174"
}
]
}
}
}
This adds two new output formats that permits to reduce
the number of line logged for a dns answer because
actually an event is logged for each answer.
With this patch, only an event that contains all the answers
is logged.
The formats are named 'detailed' and 'grouped'.
The first format provides a list of answers with
the following fields:
- rrname
- rrdata
- ttl
- rdata
The second format provides a list of record data grouped
by their type.
The output below is an example of the formats:
{
"timestamp": "2017-11-29T10:27:18.148282+0100",
"flow_id": 268864910185905,
"in_iface": "wlp2s0",
"event_type": "dns",
"src_ip": "192.168.1.254",
"src_port": 53,
"dest_ip": "192.168.1.176",
"dest_port": 52609,
"proto": "UDP",
"dns": {
"type": "answer",
"id": 3654,
"rcode": "NOERROR",
"answers": [
{
"rrname": "wordpress.org",
"rrtype": "A",
"ttl": 544,
"rdata": "66.155.40.249"
},
{
"rrname": "wordpress.org",
"rrtype": "A",
"ttl": 544,
"rdata": "66.155.40.250"
}
],
"grouped": {
"A": [
"66.155.40.249",
"66.155.40.250"
]
}
}
}
This patch adds a new configuration for dns,
introducing a "version" that permits to switch
between the new and old format to provide
backward compatibility.
The new configuration is made up of these new fields:
- version
- requests (query)
- response (answer)
- types (custom)
When destination IP address does not suffice to uniquely identify
the Modbus/TCP device.
Some Modbus/TCP devices act as gateways to other Modbus/TCP devices
that are behind this gateways.
Implement SMB app-layer parser for SMB1/2/3. Features:
- file extraction
- eve logging
- existing dce keyword support
- smb_share/smb_named_pipe keyword support (stickybuffers)
- auth meta data extraction (ntlmssp, kerberos5)
The 'tx_id' variable was used to be passed into the IterFunc as a
minumum tx to return. The IterFunc could then return either the tx
for that id, or a later one if that turned out to be the first available
tx.
The tx_id however, was still used for some things as if it was the
current tx id. Most importantly for setting the tx id for alert
ammending. So this could lead to alerts with missing or wrong
applayer records.
If suricata was started with --init-errors-fatal and an error occured
during setup of lua output (like if lua scripts configured in the conf file
don't exist or are not readable) suricata continued, which did not reflect
"init errors fatal" very well.
This fix makes the suricata initialization abort and send an error message
in such cases.
For details see:
https://redmine.openinfosecfoundation.org/issues/1503
output-lua.c contained an include of detect.h.
Since we don't (and shouldn't) call any functions from detect.c in output-lua.c
and such coupling is generally unwanted this patch removes that include.
In extra-data mode, suricata does not output xff data without
undocumented conditions (including enabling packet output). This
behaviour has been fixed to remove the hidden requirements. Fix
included removing previous xff data output implementation and adding a
new function for outputting xff that is called after outputting each
event.
IPv6 XFF entries were also being recorded incorrectly as if they were
IPv4 and this has been fixed.
When upgrading to TLS from HTTP logging of the final HTTP tx could
have the wrong direction. This was due to the original packet triggering/
finalizing the upgrade would be used as the base for both the toserver
and toclient pseudo packet meaning it was wrong in one direction.
This patch creates a pseudo packet in the same way as the flow timeout
code does, so it no longer takes the raw original packet in.
Bug #2430
https://redmine.openinfosecfoundation.org/issues/2394
Certain parameters of delay and poll interval could cause newly added
files in a directory to be missed. Cleaned up how time is handled for
files in a directory and fix which time is used for future directory
traversals. Add a mutex to make sure processing time is not optimized
away.
When the filedata logger is enabled (file extraction), but a file is not
stored due to no rules matching to force this, the file would never be
freed.
This was caused by a check in the file pruning logic that only freed a
file when the FILE_STORED flag was set. However files can also have the
FILE_NOSTORE flag set which indicates that a file won't be stored.
This patch makes sure that both conditions lead to file pruning.
Allows matching on stickybuffers. Like dsize, it allows matching on
exact values, greater than and less than, and ranges.
For streaming buffers, such as HTTP bodies, the final size of the
body is only known at the end of the transaction.
As we can have multiple files per TX we use the multi inspect
buffer support.
By using this API file_data supports transforms.
Redo part of the flash decompression as a hard coded built-in sort
of transform.
Make sure content is applied to the transformed version of a buffer.
Support content with its modifiers, and also isdataat, pcre, bytetest
and bytejump.
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.
Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.
At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.
Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.
The transforms are called from within the prefilter engines themselves.
Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.
Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.
Add new registration calls for mpm/prefilters and inspect engines.
Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.
The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.
The v2 API assumes that registered MPM implements transformations.
Add util func to set new transform in rule and add util funcs for rule
parsing.
Remove DetectAppLayerInspectEngineRegister for TOCLIENT direction
because Modbus inspection engine is only performing in request (TOSERVER).
Detect Value keyword in read access rule. In read access, match on value
is not possible.
Update Modbus keyword documentation.
When loading an empty file, libyaml will fire a single scalar
event causing us to create a key that contains an empty string.
We're not interested in this, so skip an empty scalar value
when expecting a key.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2418
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
Main objective of the function is to be able to bypass a flow on
other interfaces. This is necessary in AF_PACKET case as the flow
table are per interface.
This patch adds a boolean option "xdp-cpu-redirect" to af-packet
interface configuration. If set, then the XDP filter will load
balance the skb creation on specified CPUs instead of doing the
creation on the CPU handling the packet. In the case of a card
with asymetric hashing this will allow to avoid saturating the
single CPU handling the trafic.
The XDP filter must contains a set of map allowing load balancing.
This is the case of xdp_filter.bpf.
Fixed-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
Device storage requires the devices to be created after storage
is finalized so we need to first get the list of devices then
create them when the storage is finalized.
This patch introduces the LiveDeviceName structure that is a list
of device name used during registration.
Code uses LiveRegisterDeviceName for pre registration and keep
using the LiveRegisterDevice function for part of the code that
create the interface during the runmode creation.
eBPF has a data type which is a per CPU array. By adding one element
to the array it is in fact added to all per CPU arrays in the kernel.
This allows to have a lockless structure in the kernel even when doing
counter update.
In userspace, we need to update the flow bypass code to fetch all
elements of the per CPU arrays.
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
This patch introduces the ebpf cluster mode. This mode is using
an extended BPF function that is loaded into the kernel and
provide the load balancing.
An example of cluster function is provided in the ebpf
subdirectory and provide ippair load balancing function.
This is a function which uses the same method as
the one used in autofp ippair to provide a symetrical
load balancing based on IP addresses.
A simple filter example allowing to drop IPv6 is added to the
source.
This patch also prepares the infrastructure to be able to load
and use map inside eBPF files. This will be used later for flow
bypass.
This adds a new module that permits to decompress
swf file compressed with zlib or lzma algorithms.
The API that performs decompression will take a compressed
buffer and build a new decompressed buffer following the
FWS format which represents an uncompressed file.
The maximum buffer that can be created is up to 50mb.
During the inspection phase actually is not possible to catch
an error if it occurs.
This patch permits to store events in the detection engine
such that we can match on events and catch them.
This patch updates the unix socket protocol. Messages send from
the server and the client have now a '\n' at the end. This allows
both sides to detect easily the end of a command.
As a side effect, this fixes the problem of long answer in
suricatasc. There is now a limit at the arbitrary value of 65536.
Backward compatility is preserved as a client with the older
version of the protocol can still connect to a Suricata with
version 2 of the protocol.
Giuseppe Longo
Victor Julien
Alexander Gozman
Mats Klepsland
Pierre Chifflier
Antoine LUONG
Danny Browning
Antti Tönkyrä
Jason Ish
David DIALLO
Richard Sailer
Thomas Andrejak
Daniel Humphries
Andreas Herz
Eric Leblond
Maurizio Abba