In extra-data mode, suricata does not output xff data without
undocumented conditions (including enabling packet output). This
behaviour has been fixed to remove the hidden requirements. Fix
included removing previous xff data output implementation and adding a
new function for outputting xff that is called after outputting each
event.
IPv6 XFF entries were also being recorded incorrectly as if they were
IPv4 and this has been fixed.
When upgrading to TLS from HTTP logging of the final HTTP tx could
have the wrong direction. This was due to the original packet triggering/
finalizing the upgrade would be used as the base for both the toserver
and toclient pseudo packet meaning it was wrong in one direction.
This patch creates a pseudo packet in the same way as the flow timeout
code does, so it no longer takes the raw original packet in.
Bug #2430
On error, print the last 500 lines of output then exit 1.
Shoud allow us to see why a build fail on the debug tests,
when the error was burried in an output file we weren't
making visible.
https://redmine.openinfosecfoundation.org/issues/2394
Certain parameters of delay and poll interval could cause newly added
files in a directory to be missed. Cleaned up how time is handled for
files in a directory and fix which time is used for future directory
traversals. Add a mutex to make sure processing time is not optimized
away.
The current ebpf/Makefile.am have the problem that clang compile
errors still result in an ELF .bpf output file. This is obviously
problematic as the problem is first seen runtime when loading
the bpf-prog. This is caused by the uses of a pipe from
clang to llc.
To address this problem, split up the clang and llc invocations
up into two separate commands, to get proper reaction based on
the compiler exit code. The clang compiler is used as a
frontend (+ optimizer) and instructed (via -S -emit-llvm) to
generate LLVM IR (Intermediate Representation) with suffix .ll.
The LLVM llc command is used as a compiler backend taking IR and
producing BPF machine bytecode, and storing this into a ELF
object. In the last step the IR .ll suffix code it removed.
The official documentation of the IR language:
http://llvm.org/docs/LangRef.html
Also fix the previous make portability warning:
'%-style pattern rules are a GNU make extension'
I instead use some static pattern rules:
https://www.gnu.org/software/make/manual/html_node/Static-Usage.html
Signed-off-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
Enable compiling eBPF programs with clang -target bpf.
This is mostly to workaround a bug in libbpf, where clang > ver 4.0.0
generates some ELF sections (.eh_frame) when -target bpf is NOT specified,
and libbpf fails loading such files.
Notice libbpf is provided by the kernel, and in kernel v4.16 the library
will contain the needed function for attaching to the XDP hook.
Kernel commit 949abbe88436 ("libbpf: add function to setup XDP")
https://git.kernel.org/torvalds/c/949abbe88436
The library fix has reached kernel v4.16 but the workaround for Suricata
is interesting anyway in case people use a kernel v4.15.
Signed-off-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
This patch prepares code before enabling the clang -target bpf.
The clang compiler does not like #include <stdint.h> when
using '-target bpf' it will fail with:
fatal error: 'gnu/stubs-32.h' file not found
This is because using clang -target bpf, then clang will have '__bpf__'
defined instead of '__x86_64__' hence the gnu/stubs-32.h include
attempt as /usr/include/gnu/stubs.h contains, on x86_64:
#if !defined __x86_64__
# include <gnu/stubs-32.h>
#endif
#if defined __x86_64__ && defined __LP64__
# include <gnu/stubs-64.h>
#endif
#if defined __x86_64__ && defined __ILP32__
# include <gnu/stubs-x32.h>
#endif
This can be worked around by installing the 32-bit version of
glibc-devel.i686 on your distribution.
But the BPF programs does not really need to include stdint.h,
if converting:
uint64_t -> __u64
uint32_t -> __u32
uint16_t -> __u16
uint8_t -> __u8
This patch does this type syntax conversion.
The build of a ebpf files had an issue for system like Debian
because they don't have a asm/types.h in the include path if the
architecture is not defined which is the case due to target bpf.
This results in:
clang-5.0 -Wall -Iinclude -O2 \
-D__KERNEL__ -D__ASM_SYSREG_H \
-target bpf -S -emit-llvm vlan_filter.c -o vlan_filter.ll
In file included from vlan_filter.c:19:
In file included from include/linux/bpf.h:11:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not
found
#include <asm/types.h>
^~~~~~~~~~~~~
1 error generated.
Makefile:523: recipe for target 'vlan_filter.bpf' failed
This patch fixes the issue by adding a include path setting the
architecture to the one of the builder.
Signed-off-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
Sidned-off-by: Eric Leblond <eric@regit.org>
When the filedata logger is enabled (file extraction), but a file is not
stored due to no rules matching to force this, the file would never be
freed.
This was caused by a check in the file pruning logic that only freed a
file when the FILE_STORED flag was set. However files can also have the
FILE_NOSTORE flag set which indicates that a file won't be stored.
This patch makes sure that both conditions lead to file pruning.
Allows matching on stickybuffers. Like dsize, it allows matching on
exact values, greater than and less than, and ranges.
For streaming buffers, such as HTTP bodies, the final size of the
body is only known at the end of the transaction.
As we can have multiple files per TX we use the multi inspect
buffer support.
By using this API file_data supports transforms.
Redo part of the flash decompression as a hard coded built-in sort
of transform.
Make sure content is applied to the transformed version of a buffer.
Support content with its modifiers, and also isdataat, pcre, bytetest
and bytejump.
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.
Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.
At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.
Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.
The transforms are called from within the prefilter engines themselves.
Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.
Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.
Add new registration calls for mpm/prefilters and inspect engines.
Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.
The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.
The v2 API assumes that registered MPM implements transformations.
Add util func to set new transform in rule and add util funcs for rule
parsing.
Victor Julien
Thomas Andrejak
Daniel Humphries
Pascal Delalande
Jason Ish
Brandon Sterne
Danny Browning
Eric Leblond
Jesper Dangaard Brouer
Andreas Herz
Philippe Antoine