Added a napatech section in the yaml configuration.
hba - host buffer allowance
use-all-streams - whether all streams should be used
streams - list of stream numbers to use when use-all-streams is no
The source-napatech.* files were modified to support the host buffer allowance configuration.
The runmode-napatech.c file was modified to support both the host buffer allowance configuration and stream configuration
Signed-off-by: Matt Keeler <mk@npulsetech.com>
This patch introduces 'snaplen' a new YAML variable in the pcap section.
It can be set per-interface to force pcap capture snaplen. If not set
it defaults to interface MTU if MTU can be known via a ioctl call and to
full capture if not.
Main objective of this patch is to use a dynamic snaplen to avoid
to truncate packet at the currently fixed snaplen.
It set snaplen to MTU length if the MTU can be retrieved. If not, it
does not set the snaplen which results in using a 65535 snaplen.
libpcap is trying to use mmaped capture and setup the ring by using buffer_size
as the total memory. It also use "rounded" snaplen as frame size. So if we set
snaplen to MTU when available we are optimal regarding the building of the ring.
This patch fixes an error in pointer arythmetic and add some
comments to increase maintanability of the code. It also
simplify the decoding code as a careful RFC reading indicate
that if we discard packet containing an authentication field,
it is only possible to have a single origin indication field.
Adds support for match-on conditions (src, dst, any, both)
Uses GEOIP_MEMORY_CACHE for performance reasons
Adds support for negation and multiple countries in the same rule
Bug fixes
Changed to take flow direction from rule, if present
Comments addressed. Unit tests added.
This patch introduces a new set of functions to the ConfGetChildValue
family. They permit to look under a default node if looking under
base node as failed. This will be used to access to default parameters
for a data type (for instance, first usage will be interface).
TAILQ_FOREACH macro was not safe for element removal as it was
accessing the next element in case of a free. This patch is inspired
by Linux list handling and provide a new macro TAILQ_FOREACH_SAFE.
This macro is removal safe and only differs by a last argument being
a temporaty pointer to an element.
prelude_string_set_ref don't like when it is called with a NULL
parameter. This patch adds check for NULL value. This is formally
good as there is no use of a NULL description.
Insert pseudo packet under low load conditions to complete rule swap.
This is necessary when we use autofp active packets where most packets
would be sent to the first queue under low load conditions.
This patch should fix the bug #637. Between pcap files, it uses a
new function HostCleanup() to clear tag and threshold on host with
an IP regputation. An other consequence of this modification is
that Host init and shutdown are now init and shutdown unconditionaly.
Treat sigs with negated addresses as non ip-only.
This fix exposes bug #608, which results in 2 failed unittest which
have now been disabled by this commit. Would be reenabled when we
have #608 fix in.
This patch adds two commands to unix-command. 'iface-list' displays
the list of interface which are sniffed by Suricata and 'iface-stat'
display the available statistics for a single interface. For now,
this is the number of packets and the number of invalid checksums.
The affinity setting code was using the old API. This patch updates
to the new API and also adds a call to RunModeInitiaze() which was
missing in Single running mode.
This patch transforms the unix socket into a flexible system to
add commands (triggered by user) and taks (run periodically).
It introduces two functions UnixManagerRegisterCommand and
UnixManagerRegisterBackroundTask to registed commands and tasks.
Other part of Suricata can then declare a new command via a simple
call of the function. In the case of a command the caller is
responsible of building the answer message using Jansson API. The
sending of the message is made by unix manager code.
This patch introduces a unix command socket. JSON formatted messages
can be exchanged between suricata and a program connecting to a
dedicated socket.
The protocol is the following:
* Client connects to the socket
* It sends a version message: { "version": "$VERSION_ID" }
* Server answers with { "return": "OK|NOK" }
If server returns OK, the client is now allowed to send command.
The format of command is the following:
{
"command": "pcap-file",
"arguments": { "filename": "smtp-clean.pcap", "output-dir": "/tmp/out" }
}
The server will try to execute the "command" specified with the
(optional) provided "arguments".
The answer by server is the following:
{
"return": "OK|NOK",
"message": JSON_OBJECT or information string
}
A simple script is provided and is available under scripts/suricatasc. It
is not intended to be enterprise-grade tool but it is more a proof of
concept/example code. The first command line argument of suricatasc is
used to specify the socket to connect to.
Configuration of the feature is made in the YAML under the 'unix-command'
section:
unix-command:
enabled: yes
filename: custom.socket
The path specified in 'filename' is not absolute and is relative to the
state directory.
A new running mode called 'unix-socket' is also added.
When starting in this mode, only a unix socket manager
is started. When it receives a 'pcap-file' command, the manager
start a 'pcap-file' running mode which does not really leave at
the end of file but simply exit. The manager is then able to start
a new running mode with a new file.
To start this mode, Suricata must be started with the --unix-socket
option which has an optional argument which fix the file name of the
socket. The path is not absolute and is relative to the state directory.
THe 'pcap-file' command adds a file to the list of files to treat.
For each pcap file, a pcap file running mode is started and the output
directory is changed to what specified in the command. The running
mode specified in the 'runmode' YAML setting is used to select which
running mode must be use for the pcap file treatment.
This requires modification in suricata.c file where initialisation code
is now conditional to the fact 'unix-socket' mode is not used.
Two other commands exists to get info on the remaining tasks:
* pcap-file-number: return the number of files in the waiting queue
* pcap-file-list: return the list of waiting files
'pcap-file-list' returns a structured object as message. The
structure is the following:
{
'count': 2,
'files': ['file1.pcap', 'file2.pcap']
}
This patch modifies the file store system to have it create the
file store directory if needed. It dos not create the full
directory tree as the parent directory must have already been
created.
This patch update the glafs list to be able to indicate that a
flag is not supported. This information is used by list-keyword to
display information to the user.
The output of the list-keyword is modified to include the url to
the keyword documentation when this is available. All documented
keywords should have their link set.
list-keyword can be used with an optional value:
no option or short: display list of keywords
csv: display a csv output on info an all keywords
all: display a human readable output of keywords info
$KWD: display the info about one keyword.
In list-keywords and list-app-layer mode, suricata now only
displays the messages linked with the feature. This allow users
to redirect the output and easily work on it. For exemple, the
csv output will be easily imported into a spreadsheet.
This patch update the list-keyword command. Without any option,
the previous behavior is conserved. If 'all' is used as option,
suricata print a csv formatted output of keyword information:
name;features;description
If a keyword name is used as argument, suricata print a readable
message:
tls.subject
Features: state inspecting
Description: Match TLS/SSL certificate Subject field
As we don't parse the YAML file when listing of keywords is asked,
suricata make a test on existence of the build-default directory.
So with a non standard (working) install (even a single configure
without option lead to a failure), the keyword listing fails
because the default logging directory does not exist.
It is now possible to use the 'daemon-directory' configuration
variable to specify the working directory of suricata in daemon
mode. This will permit to specify the place for core and other
related files.
This patch creates a pid file per default and use it to avoid to be
able to run two Suricata. Separate pid file have to be provided to
be able to do it.
Removed the Napatech 2GD support
runmode-napatech-3gd.c had an include from runmode-napatech.h which was erroneous and has been removed as well.
Signed-off-by: Matt Keeler <mk@npulsetech.com>
For use with Network Cards from Napatech utilizing the 3GD driver/api.
- Implemented new run modes in runmode-napatech-3gd.*
- Implemented capture/decode threads in source-napatech-3gd.*
- Integrated the new run modes and source into the build infrastructure.
New configure switches
--enabled-napatech-3gd : Turns on the NT 3GD support
--with-napatech-3gd-includes : The directory containing the NT 3GD header files
--with-napatech-3gd-libraries : The directory containing the NT 3GD libraries to link against.
New CLI switch
--napatech-3gd : Uses the Napatech 3GD run mode
Runmodes Supported:
- auto
- autofp
- workers
Notes:
- tested with 1 Gbps sustained traffic (no drops)
Signed-off-by: Matt Keeler <mk@npulsetech.com>
stream.inline YAML configuration variable now support the 'auto' value.
In this case, inline mode is activated for IPS running mode (NFQ and
IPFW) and is deactivated for IDS mode. This patch should fix bug #592.
If no packet arrives to a capture thread, it is possible that the
AFPReadLoop() function goes into an infinite loop. This could cause
suricata to hang at exit on non busy system.
This patch adds a counter to detect when Suricata start looping in
the ring to stop when it reaches this point.
Set event on overlapping data segments that have different data.
Add stream-events option stream-event:reassembly_overlap_different_data and
add an example rule.
Issue 603.
This patch adds a call to close the queue when the acquisition
loop is ending. This way the incoming packets will be accepted
during all the shutdown phase (if the queue-bypass option of
NFQUEUE is used). At the same time the currently processed packets
will be dropped but the time scale are different: suricata will
drop 20 ms of packets and the shutdown can take 0.5 seconds.
Patch based on an idea of Victor Julien.
If a pattern has matched on mpm, don't re-inspect it later, subject to certain
conditions met by the pattern - namely, not negated, right chop, no replacet
attached to it.
The power of libhtp customisation now available to users.
Options available -
path-backslash-separators: yes
path-compress-separators: yes
path-control-char-handling: none
path-convert-utf8: yes
path-decode-separators: yes
path-decode-u-encoding: yes
path-invalid-encoding-handling: preserve_percent
path-invalid-utf8-handling: none
path-nul-encoded-handling: none
path-nul-raw-handling: none
set-path-replacement-char: ?
set-path-unicode-mapping: bestfit
You can use this for your libhtp customisation. Options explained in our
wiki.
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Advanced_libhtp_customization
In FFRv2, dereference flow from a packet using the new reference/dereference
util macros. This allows the decr use_cnt for flow and reseting the flow
pointer to NULL for the pseudo pkt to happen simultaneously, in case there we
fail to retrieve a pseudo_packet and have to return the already obtained
pseudo packets, back to the packetpool.
When a thread is dead at init the THV_INIT_DONE flag is not set
and the spawn function can freeze (see bug #553 for an example).
In this case THV_RUNNING_DONE is set and we can also check on this
state for leaving the function. This should fix #bug553
Allow threshold.conf to override rule thresholds in the following
cases:
- threshold.config rule uses threshold or event_filter AND
- threshold.config rule applies to a single signature (so no
gid 0 or sid 0)
Confirmed to work with both threshold and detection_filter rule
keywords.
Part of bug #425.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
This patch resets the AFPPacketVar linked to a Packet in the release
function to avoid any side effect when the packet is reused. To do
so a new AFPV_CLEANUP macro has been introduced.
This patch cleans the code were two almost identical treatment on
the packet we're made. It may be linked by a merge error I've done
or to a simple mistake on my side.
There was an inversion in code resulting as all sockets being seen
as non IPS mode when doing the peering. This resulted in a crash at
first packet because it has no peer.
This patch changes the policy of the timeout function by cleaning
every timeouted trackers.
Previous code was only freeing the first tracker and this was resulting
in calling the timeout function continuously. One of my previous patch
has modified the function to avoid to run it more than twice a second.
But as it was not taken into account the fact only the first tracker was
freed, the result was that a lot of tracker could not be allocated.
When nothing can be fetch from the pool, this can repeat frequently.
Thus displaying a message in the log will not help. This patch
uses a counter instead of a log message. As this is a sort of memcap
this is conformed to what is done for other issues of the same type.
In some setup, suricata may receive broadcast packets and the call
to sendto may fail if the wrong interface is choosen by kernel.
This patch change the error treatment to avoid to leave when
this problem occurs.
A crash can occurs in the following conditions:
* Suricata running in other mode than "workers"
* Kernel fill in the ring buffer
Under this conditions, it is possible that the capture thread reads
a packet that has not yet released by one of the treatment threads
because there is no modification done on the ring buffer entry when
a packet is read. Doing, this it access to memory which can be
released to the kernel and modified. This results in a kind of memory
corruption.
This bug has only been seen recently and this has to be linked with the
read speed improvement recently made in AF_PACKET support.
The patch fixes the issue by modifying the tp_status bitmask in the
ring buffer. It sets the TP_STATUS_USER_BUSY flag when it is confirmed
that the packet will be treated. And at the start of the read, it exits
from the reading loop (returning to poll) when it reaches a packet with
the flag set. As tp_status is set to 0 during packet release the flag
is destroyed when releasing the packet.
Regarding concurrency, we've got a sequence of modification. The
capture thread read the packet and set the flag, then it passes the
queue and the packet get processed by other threads. The change on
tp_status are thus made at different time.
Regarding the value of the flag, the patch uses the last bit of
tp_status to avoid be impacting by a change in kernel. I will
propose a patch to have TP_STATUS_USER_BUSY included in kernel
as this is a generic issue for multithreading application using
AF_PACKET mechanism.
Coverity pointed out that PoolReturn is almost like free and detected
a use after free when accessing to tracker->af (issue 720339).
This patch fixes this by storing the value in a local variable.
If a capture loop does exit, the thread needs to start without
synchronization with the other threads. This patch fixes this
by resetting the turn count on the peerslist structure and
adding a test on this condition in the wait function.
It seems that, in some case, there is a read waiting but the
offset in the ring buffer is not correct and Suricata need to
walk the ring to find the correct place and make the read.
This patch fixes emergency mode by setting the variable even if we
have a non kernel checksum check. It also does a call to
AFPDUmpCounters() as it seems to improve thing to do it ASAP.
This patch implements "late open". On high performance system, it
is needed to create the AF_PACKET just before reading to avoid
overflow. Socket creation has to be done with respect to the order
of thread creation to respect affinity settings.
This patch adds a counter to AFPPeer to be ale to synchronize the
initial socket creation.
Suricata was not able to start cleanly in AF_PACKET with default
suricata.yaml file if there was no eth1 on the system. This patch
fixes this issue and rework the socket transition phase to fix
some serious issues (file descriptor leak) found when fixing this
problem.
Every 20 seconds it displays a message to the user to warn him about
the interface not being accessible:
[ERRCODE: SC_ERR_AFP_CREATE(196)] - Can not open iface 'eth1'
DecodeTeredo, DecodeIPv6InIPv6 and DecodeIPv4inIPv6 were calling
DecodeTunnel with packet being a pseudo packet and data being
data from initial packet:
DecodeTunnel(tv, dtv, tp, start, blen,
pq, IPPROTO_IPV6);
In decoding functions, arithmetic was done on pkt to set some values?
It was resulting in field of packet pointing outside of the scope of
packet data.
This patch switch to what has been done in DecodeGre(), I mean:
DecodeTunnel(tv, dtv, tp, GET_PKT_DATA(tp),
GET_PKT_LEN(tp), pq, IPPROTO_IP);
Data buffer is then relative to the packet and the arithmetic is
correct.
Some detection keywords need thread local ctx storage. Example is the
filemagic keyword that has a ctx that is modified with each call. That
is not thread safe. This functionality allows registration of thread
local ctxs so that each detect thread works on it's own copy.
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
This patch adds a series of unit tests. First two check test the keyword
by checking packet on signatures using it. Last one adds is here to check
that there is no interaction of l3_proto and ip_proto.
This patch adds a l3_proto keyword to the signature language. It
can be used to specify if the signature has to match on IPv4, IPv6
or both. For example, one can write:
alert http any any -> any 22 (msg: "HTTP v6"; l3_proto:ip6; sid:14;)
This should close#494.
If the MTU on the reception interface and the one on the transmission
interface are different, this will result in an error at transmission
when sending packet to the wire.
Flush all waiting packets to be in sync with kernel when drop
occurs. This mode can be activated by setting use-emergency-flush
to yes in the interface configuration.
This patch moves raw socket binding at the end of init code to
avoid to have a flow of packets reaching the socket before we
start to read them.
The socket creation is now made in the loop function to avoid
any timing issue between init function and the call of the loop.
This patch adds a new feature to AF_PACKET capture mode. It is now
possible to use AF_PACKET in IPS and TAP mode: all traffic received
on a interface will be forwarded (at the Ethernet level) to an other
interface. To do so, Suricata create a raw socket and sends the receive
packets to a interface designed in the configuration file.
This patch adds two variables to the configuration of af-packet
interface:
copy-mode: ips or tap
copy-iface: eth1 #the interface where packet are copied
If copy-mode is set to ips then the packet wth action DROP are not
copied to the destination interface. If copy-mode is set to tap,
all packets are copied to the destination interface.
Any other value of copy-mode results in the feature to be unused.
There is no default interface for copy-iface and the variable has
to be set for the ids or tap mode to work.
For now, this feature depends of the release data system. This
implies you need to activate the ring mode and zero copy. Basically
use-mmap has to be set to yes.
This patch adds a peering of AF_PACKET sockets from the thread on
one interface to the threads on another interface. Peering is
necessary as if we use an other socket the capture socket receives
all emitted packets. This is made using a new AFPPeer structure to
avoid direct interaction between AFPTreadVars.
There is currently a bug in Linux kernel (prior to 3.6) and it is
not possible to use multiple threads.
You need to setup two interfaces with equality on the threads
variable. copy-mode variable must be set on the two interfaces
and use-mmap must be set to activated.
A valid configuration for an IPS using eth0 and vboxnet1 interfaces
will look like:
af-packet:
- interface: eth0
threads: 1
defrag: yes
cluster-type: cluster_flow
cluster-id: 98
copy-mode: ips
copy-iface: vboxnet1
buffer-size: 64535
use-mmap: yes
- interface: vboxnet1
threads: 1
cluster-id: 97
defrag: yes
cluster-type: cluster_flow
copy-mode: ips
copy-iface: eth0
buffer-size: 64535
use-mmap: yes
This patch adds a data release mechanism. If the capture module
has a call to indicate that userland has finished with the data,
it is possible to use this system. The data will then be released
when the treatment of the packet is finished.
To do so the Packet structure has been modified:
+ TmEcode (*ReleaseData)(ThreadVars *, struct Packet_ *);
If ReleaseData is null, the function is called when the treatment
of the Packet is finished.
Thus it is sufficient for the capture module to code a function
wrapping the data release mechanism and to assign it to ReleaseData
field.
This patch also includes an implementation of this mechanism for
AF_PACKET.
The mmaped mode was using a too small ring buffer size which was
not able to handle burst of packets coming from the network. This
may explain the important packet loss rate observed by Edward
Fjellskål.
This patch increases the default value and adds a ring-size
variable which can be used to manually tune the value.
This patch converts the series of variable to an atomic.
Furthermore, as the callbacks are now always run, it is not
necessary anymore to refuse a ruleswap if HTP parameters are
changing.
This patch modifies the init of Detect threads. They are now started
with a dummy function and their initialisation is done after the
signatures are loaded. Just after this, the dummy function is switched
to normal one.
In IPS mode, this permit to route packets without waiting for the
signature to start and should fix#488.
Offline mode such as pcap file don't use this mode to be sure to
analyse all packets in the file.
The patch introduces a "delayed-detect" configuration variable
under detect-engine. It can be used to activate the feature
(set to "yes" to have signature loaded after capture is started).
When multiple certificates forming a chain are sent. A pointer to
the start of each certificate is kept. This will allow treatment
on certificates chains.
This patch adds a TLS store option to save certificate in PEM format.
Each time the store action is met, a file and a metafile are created.
Reworked-by: Eric Leblond <eric@regit.org>
Creation of the log-tlslog file in order to log tls message.
Need to add some information into suricata.yaml to work.
- tls-log:
enabled: yes # Log TLS connections.
filename: tls.log # File to store TLS logs.
This patch should fix#480 by adding the support of Teredo tunnel.
The IPv6 content of the tunnel will be parsed in a similar way as
what is done the GRE tunnel. Signatures will then be matched on the
IPv6 content.
On linux >= 3.6, you can use the fail-open option on a NFQ queue
to have the kernel accept the packet if userspace is not able to keep
pace.
Please note that the kernel will not trigger an error if the feature is activated
in userspace libraries but not available in kernel.
This patch implements the option for suricata by adding a nfq.fail-open
configuration variable which is desactivated by default.
On some setup you want to run each thread init function sequentially.
For example, if I use flow_cpu load balancing on AF_PACKET, my target
is to have CPU 0 (first socket in the group) to be link with the
thread 0 in detect cpu set (first thread to be initialised). A good
way to achieve this is to run only one thread init function at a time
to avoid any possible race condition.
This patch disables checksum alert if checksum-validation is set
to no in the configuration file. Without this patch, when parsing
a pcap which checksum offloading, it was not possible to get rid
of event caused by checksum validation.
This isn't a perfect solution. More like we have patched this for the case we
are in tcp's established state. The right solution would be to accept states
based on the presence(using operator OR) of certain flags in the tcp header,
rather than list out all possible flag combinations.
This patch should bring some improvements by looping on the
ring when there is some data available instead of getting back
to the poll. It also fix recovery in case of drops on the ring
because the poll command will not return correctly in this case.
This patch fixes the collision issue observed on an intensive network
trafic. When there is fragmentation it is the case for all data
exchanged between two hosts. Thus using a hash func only involving
IP addresses (and protocol) was leading to a collision for all
exchanges between the hosts. At a larger scale, it was resulting in
a packet loss. By using the IP ID instead of the protocol family, we
introduce a real difference between the trackers.
This patch renamed the 'worker' running mode into 'workers'. Thus,
there is only one name in Suricata for the same thing. Backward
compatibility is ensured by replacing "worker" by "workers" when
the old name is used. A warning is printed in the log when the old
name is used.
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
This patch adds counters for kernel drops and accepts to af-packet
capture module. This information are periodically displayed in
stats.log:
capture.kernel_packets | RxAFP1 | 1792
capture.kernel_drops | RxAFP1 | 0
The statistic is fetch via a setsockopt call every 255 packets.
This patch adds support for BPF in AF_PACKET running
mode. The command line syntax is the same as the one
used of PF_RING.
The method is the same too: The pcap_compile__nopcap()
function is used to build the BPF filter. It is then
injected into the kernel with a setsockopt() call. If
the adding of the BPF fail, suricata exit.
This patch will allow us to use the datalink when computing the filter.
It also fixes a potential issue where an interface data type change
after the interface if going down/up.
To reload ruleset during engine runtime, send the USR2 signal to the engine, and the ruleset would be reloaded from the same yaml file supplied at engine startup
- suricata treates sigs with offset/depth without any packet keywords as stream sigs
- as a consequence suricata will FN on such sigs
The tests introduced here will fail, displaying the issues. The
next patch in the series would fix the said issues.
Some cards like Napatech or Myricom support libpcap wrappers that allow for
multiple streams, queues, ringbuffers. The workers mode can be of use in
those cases.
Simply added the -T to be printed out when suricata is run without any
arguments. The capability to test a configuration file has been in
suricata for some time, just doesn't show up as an option right now.
This function is a wrapper to localtime_r. It is needed to avoid
a compilation warning on OpenBSD. I'm forced to type the function
to a non pointer first parameter. If not we will have to use two
differents functions in OpenBSD where tv->tv_sec is a long
(different from time_t).
gcc on OpenBSD does not support C99 inline functions. This patch
modify the build system to handle this. It also change the order
of declaration of some functions to avoid to use them before
declaring them as inline.
Found here:
http://burtleburtle.net/bob/hash/doobs.htmlhttp://burtleburtle.net/bob/c/lookup3.c
From the file header:
lookup3.c, by Bob Jenkins, May 2006, Public Domain.
These are functions for producing 32-bit hashes for hash table lookup.
hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and final()
are externally useful functions. Routines to test the hash are included
if SELF_TEST is defined. You can use this free for any purpose. It's in
the public domain. It has no warranty.