Add support for AFL PERSISTANT_MODE when Suricata is compiled with
a supported compiler (only afl-clang-fast for now).
This gives a ~10x performance boost when fuzzing.
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.
Per protocol, 2 commandline options are added:
--afl-http-request=<filename>
--afl-http=<filename>
In the former case, the contents of the file are passed directly to
the HTTP parser as request data.
In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
When --with-libpcre-libraries is used we skip the libpcre 8.35 check
since pkg-config might still point to the 8.35 version installed
although newer version was passed with --with-libpcre-libraries.
It is the small things that count. This is an example of the fix
Before
--disable-threading-tls Disable TLS (thread local storage)]
After
--disable-threading-tls Disable TLS (thread local storage)
It was not possible to simply specify PKG_CONFIG_PATH to build
with an non bundled libhtp. With this patch we don't need anymore
the htp lib and include configure options.
Remove most of the CFLAGS updates from configure. Flags are now (mostly)
set in AM_CLFLAGS.
Update all -DBLAH additions to CFLAGS to use AC_DEFINE([BLAH], ...)
Improve Lua vs LuaJIT checking.
Improve the configure output a bit.
Lots of smaller cleanups.
Sync the replacement define with the latest Linux code.
This patch also updates the detection part in configure.ac
to do a declaration of all fields if the newest features are
not present.
Don't link suricata with libnfnetlink when we don't have support
for NFQUEUE or NFLOG. Previously, suricata was linked with this
library without reason.
Check for the minimal pkg-config 0.21 version. Without it, CentOS'
pkg-config will fail with the warning:
configure: error: The pkg-config script could not be found or is too old.
Add --disable-threading-tls switch to force the posix thread local
storage code paths even if __thread is available.
Goal is to make it easier to QA the posix code path.
Moved the libpcap section in configure.ac before libpfring to
enable libpfring to use the specified libpcap includes and
libraries when testing for libpfring support.
Bug #1294
The base 'lua' library has different names on different OS' and even
Linux distro's. Instead of selecting the proper one, we now just try
all. This way no OS/distro specific knowledge about the name is needed.
AC_TRY_COMPILE puts the code in a function already, and Clang didn't like
the function within the function declaration. This lead to test failure.
Clang now properly detects __thread support.
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.
This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.
Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.
Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.
Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.
The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.
By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".
Check for TILERA_ROOT environment variable before building tile_pcie_logd
Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
Fixes configure enabling of prelude. CFLAGS is reset, so the previous
adding of -DPRELUDE was nixed. Using AC_DEFINE now.
Cleanups:
- make functions static
- simplify handling of no prelude support
- move registration to the bottom
GCC typically generates better code without the -no-strict-aliasing flag.
It is only required if code makes assumptiosn that break strict aliasing.
The unit tests pass on x86 and Tile without the flag.
Check all compilers to see if they support the -march=native flags, rather
than assuming gcc 4.2 or later does. Tile GCC doesn't currently support it,
so not checking break Tile compiles.
When generating src/build-info.h the use of ac_builddir was problematic.
There were several cases where it was undefined leading to the absolute
path /src/build-info.h. ./configure should be called from the parent dir
to the actual src dir, so this should work. Make distcheck and normal builds
complete fine.
In preparation of the libhtp upgrade, move all libhtp related conditionals
to configure. This allows for one set of build scripts that works regardless
of the presence of a local libhtp dir.
Normally, there is one verdict per packet, i.e., we receive a packet,
process it, and then tell the kernel what to do with that packet (eg.
DROP or ACCEPT).
recv(), packet id x
send verdict v, packet id x
recv(), packet id x+1
send verdict v, packet id x+1
[..]
recv(), packet id x+n
send verdict v, packet id x+n
An alternative is to process several packets from the queue, and then send
a batch-verdict.
recv(), packet id x
recv(), packet id x+1
[..]
recv(), packet id x+n
send batch verdict v, packet id x+n
A batch verdict affects all previous packets (packet_id <= x+n),
we thus only need to remember the last packet_id seen.
Caveats:
- can't modify payload
- verdict is applied to all packets
- nfmark (if set) will be set for all packets
- increases latency (packets remain queued by the kernel
until batch verdict is sent).
To solve this, we only defer verdict for up to 20 packets and
send pending batch-verdict immediately if:
- no packets are currently queue
- current packet should be dropped
- current packet has different nfmark
- payload of packet was modified
This patch adds a configurable batch verdict support for workers runmode.
The batch verdicts are turned off by default.
Problem is that batch verdicts only work with kernels >= 3.1, i.e.
using newer libnetfilter_queue with an old kernel means non-working
suricata. So the functionnality has to be disabled by default.
'make install' install now suricatasc script and Python module to
the system. The suricatasc client module can now be used in other
Python projects by using 'import suricatasc'.
A transformation was needed for distribution of a module and a script.
Module in src directory is now containing most of the code and the
script only handle argument parsing and the creation of a unix socket
client through 'suricatasc' module.
As reported in bug #688, htp_config_set_path_decode_u_encoding
function is not included in libhtp header before 0.3.0. Result
is that suricata compilation fail with an external htp library.
The following patch detect the issue and adds the missing
declaration.
Adds support for match-on conditions (src, dst, any, both)
Uses GEOIP_MEMORY_CACHE for performance reasons
Adds support for negation and multiple countries in the same rule
Bug fixes
Changed to take flow direction from rule, if present
Comments addressed. Unit tests added.
This patch improve the error message when luajit libraries are not
found. It displays information about the possibility to use
PKG_CONFIG_PATH or the dedicated configure options.
This patch introduces a unix command socket. JSON formatted messages
can be exchanged between suricata and a program connecting to a
dedicated socket.
The protocol is the following:
* Client connects to the socket
* It sends a version message: { "version": "$VERSION_ID" }
* Server answers with { "return": "OK|NOK" }
If server returns OK, the client is now allowed to send command.
The format of command is the following:
{
"command": "pcap-file",
"arguments": { "filename": "smtp-clean.pcap", "output-dir": "/tmp/out" }
}
The server will try to execute the "command" specified with the
(optional) provided "arguments".
The answer by server is the following:
{
"return": "OK|NOK",
"message": JSON_OBJECT or information string
}
A simple script is provided and is available under scripts/suricatasc. It
is not intended to be enterprise-grade tool but it is more a proof of
concept/example code. The first command line argument of suricatasc is
used to specify the socket to connect to.
Configuration of the feature is made in the YAML under the 'unix-command'
section:
unix-command:
enabled: yes
filename: custom.socket
The path specified in 'filename' is not absolute and is relative to the
state directory.
A new running mode called 'unix-socket' is also added.
When starting in this mode, only a unix socket manager
is started. When it receives a 'pcap-file' command, the manager
start a 'pcap-file' running mode which does not really leave at
the end of file but simply exit. The manager is then able to start
a new running mode with a new file.
To start this mode, Suricata must be started with the --unix-socket
option which has an optional argument which fix the file name of the
socket. The path is not absolute and is relative to the state directory.
THe 'pcap-file' command adds a file to the list of files to treat.
For each pcap file, a pcap file running mode is started and the output
directory is changed to what specified in the command. The running
mode specified in the 'runmode' YAML setting is used to select which
running mode must be use for the pcap file treatment.
This requires modification in suricata.c file where initialisation code
is now conditional to the fact 'unix-socket' mode is not used.
Two other commands exists to get info on the remaining tasks:
* pcap-file-number: return the number of files in the waiting queue
* pcap-file-list: return the list of waiting files
'pcap-file-list' returns a structured object as message. The
structure is the following:
{
'count': 2,
'files': ['file1.pcap', 'file2.pcap']
}
Removed the Napatech 2GD support
runmode-napatech-3gd.c had an include from runmode-napatech.h which was erroneous and has been removed as well.
Signed-off-by: Matt Keeler <mk@npulsetech.com>
clang is supposed to support march=native but if the CPU is too
recent for clang, this can cause an invalid detection of arch and
result in a incapability of clang to compile any binary.
This patch updates the test to try a compilation with march=native
when clang is the used compiler.
The version checking was made similarly for clang and gcc. This
patch modifies this to check on compiler name. This way we can
avoid to set march=native which is not supported by clang on
some system.
At the same time, this fix the annoying warning about no-fp-tree
being unsupported by clang.
For use with Network Cards from Napatech utilizing the 3GD driver/api.
- Implemented new run modes in runmode-napatech-3gd.*
- Implemented capture/decode threads in source-napatech-3gd.*
- Integrated the new run modes and source into the build infrastructure.
New configure switches
--enabled-napatech-3gd : Turns on the NT 3GD support
--with-napatech-3gd-includes : The directory containing the NT 3GD header files
--with-napatech-3gd-libraries : The directory containing the NT 3GD libraries to link against.
New CLI switch
--napatech-3gd : Uses the Napatech 3GD run mode
Runmodes Supported:
- auto
- autofp
- workers
Notes:
- tested with 1 Gbps sustained traffic (no drops)
Signed-off-by: Matt Keeler <mk@npulsetech.com>
On linux >= 3.6, you can use the fail-open option on a NFQ queue
to have the kernel accept the packet if userspace is not able to keep
pace.
Please note that the kernel will not trigger an error if the feature is activated
in userspace libraries but not available in kernel.
This patch implements the option for suricata by adding a nfq.fail-open
configuration variable which is desactivated by default.
Jason Ish
Victor Julien
Mats Klepsland
Alexander Gozman
Justin Viiret
Andreas Herz
bladeswords
Eric Leblond
Helmut Schaa
Jason Ish
gureedo
Giuseppe Longo
Ken Steele
Victor Julien
Anoop Saldanha
Florian Westphal
Christian Kreibich
Ignacio Sanchez
Matt Keeler