Move initialization of filestore global counter to PreRunInit,
so they get registered during program initialization, or as
required in unix-socket mode, initialized for each file run.
Fixes Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4216
Fix handling of file progress tracking for regular http.request_body
along with transform combinations.
This is done by implementing the 'base id' logic.
Related tickets: #4361#4199#3616
Fix handling of file progress tracking for regular file.data along
with transform combinations for the part of the implementation that
uses the HTTP inspection logic.
This is done by implementing the 'base id' logic.
Related tickets: #4361#4199#3616
Fix handling of file progress tracking for regular file.data along
with transform combinations for the part of the implementation that
uses the File API.
This is done by implementing the 'base id' logic.
Related tickets: #4361#4199#3616
Buffers with transforms are based on the non-transformed "base"
buffer, with a new ID assigned and the transform callbacks added.
This patch stores the id of the original buffer in the new buffer
inspect and prefilter structures. This way the buffers with and
without transforms can share some of the logic are progression
of file and body inspection trackers.
Related tickets: #4361#4199#3616
The function THashConsolidateMemcap is used to allow to load a
dataset even when the memcap is not set. But the implementation
was in fact resetting the memcap value to the max of memory
usaga after loading and default memcap. As a result, the
function was resetting memcap to the default memcap even if
a huge memcap was set in the dataset definition. In the case
of dataset where we add to the set it was leading to memcap
limit hitting despite the settings of memcap by the user.
This patch udpates the code to set the final memcap value to
the max of memory usage after loading and set memcap.
Set "done flag" only if parsers for both directions are not found in a
case of midstream parsers from other direction are tried if nothing is found
for the initial one. "done flag" must be set if nothing is found in both
directions. Otherwise processing of incomplete data is terminated at the very
first try.
Split the headers and source into 2 variables. Headers are
marked noinst so they don't get automatically installed on
"make install". Instead they will be installed by a custom
Makefile target, "make install-headers".
Fix another issue with library ordering when breaking apart
LDFLAGS from LIBS for outputting usable command lines for
users of a Suricata library.
RUST_LDADD should just contain the extra libs required by
Rust, not the actual Suricata Rust library.
As we don't install the libraries by default, provide a make target,
"install-library" to install the libsuricata library files.
If shared library support exists, both the static and shared
libraries will be installed, otherwise only the static libraries
will be installed.
Building the shared library on Linux is not something by default.
Instead a user must opt-in to building by running the
"make libsuricata.so" target in the src/ directory.
Currently shared library support is only available on Linux. More
OSs will be supported as we can test them.
With the circular reference gone, we can now make use
of a convenience library for the Suricata program
as well as any other programs that depend on the same
source such as the fuzzer.
While its not a libtool convenience library, it serves
the same purpose and is a common idiom in Make and CMake
projects whereas the COMMON_SOURCES approach was more
of a hack we had to resort to until the circular
reference was resolved.
The old compare macro would compare all bytes of an address, even
when for IPv4 addresses the additional bytes were not in use. This
made the logic vulnerable to mistakes like in issue #4280.
AppLayerRegisterParser was creating a link error when attempting
to use a convenience library for the Suricata C code, then linking
the library of C code with the library of Rust code into a final
Suricata executable, or use with fuzz targets.
By moving AppLayerRegisterParser to the context structure and
calling it like a callback the circular reference is removed
allowing the convenience libraries to work again.
This is also a stepping block to proving a Suricata library
as a single .a or .so file.
Commit 93642a0d1d did prevent to
specify the logging directory on command line and use the unix
socket.
It looks like the implementation has evolved and the arbitrary
limitation can be removed allowing a user to start unix socket
without editing the configuration file.
In some cases, the InspectionBufferGet function would be followed by
a failure to set the buffer up, for example due to a HTTP body limit
not yet being reached. Yet each call to InspectionBufferGet would lead
to the matching list_id to be added to the
DetectEngineThreadCtx::inspect.to_clear_queue. This array is sized to
add each list only once, but in this case the same id could be added
multiple times, potentially overflowing the array.
Sleep 250 microseconds instead of 100 as running in KVM cause the
old value to use 100% CPU for these threads.
Perf testing suggests no measurable impact for the non-KVM case.
Ticket: #4096
Fix a 'skipped' transaction early in the list leading to all further
transactions getting skipped, even if they were fully processed and
ready to be cleaned up.
When Suricata was build with ebpf support and when it was started
in NFQ mode, it was crashing at exit because it was trying to free
the device extension.
This patch fixes the issue by only trigger the eBPF related code
when Suricata is running in AFP_PACKET mode.
Fix usage of out-of-scope variables. Introduced with the hashing
and adding the guard of g_disable_hashing.
To fix, just remove the guard so all variables are in scope. Hashes
are not initialized here so there is no need for the guard.
This commit adds additional Lua API interfaces to bring consistency to
functions such that the `SC` prefix is available consistently across
flow int and flow var functions.
The idea with a flag to disable hashing is to provide a way to
get the effect of building Suricata without libnss, which is
sometimes done for performance reasons.
clang-format will reformat this to have multiple options on a
single line which is hard to read and I don't see a specific
clang-format option to format this better. So for now,
disable clang-format on this section of code.
Required including NSS header in places that depended on
util-file.h including it.
All filestore suricata-verify tests now pass without libnss.
Also enabled detect-file{md5,sha1,sha256} without NSS support.
The cbindgen generated header should not include rust.h as
rust.h already includes the generated binding.
Fixup C source code that only pulled the generated include, it
should instead pull in "rust.h" which includes the generated
binding plus other misc. stuff.
Currently, unit tests use integer values 1 and 0 to denote pass
and fail status of tests respectively. Modify the unit test
detect-filesha1 to use the PASS/FAIL API instead.
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.
Update all parsers to take advantage of this.
Base64Encode function requires the maximum length of the output string
as its last parameter. Use the macro BASE64_BUFFER_SIZE to calculate it
correctly.
The size of encoded_data array and the maximum output length parameter
to Base64Encode function were incorrect leading to buffer overflow for
certain cases. The algorithm requires at least 5 bytes of space to even
convert a string of length 1.
Use BASE64_BUFFER_SIZE macro to correctly calculate this output length.
Set size of encoded_data array to the calculated output length.
Maximum length of a base64 encoded string can be 33% over the actual
length of the input string. The formula to best cover all the edge cases
is mathematically
(4 * (input_length + 2) / 3) + 1
Add a macro to calculate this for a given input length.
Removes the use of PCRE for performance
Forbids empty negations after a valid pattern
Forbids mixing negative and positive forms as it is irrelevant
Forbids useless repetition of a version
Even when the rules are only applied on traffic with the protocol
the structure for the protocol header can be set to NULL if there
was an error parsing the header
When stateful detection rules, for which detection has already started
for a previous packet, are added to the candidates array, the array
is sorted to mantain the correct inspection order. However, due to a
trivial error in the sort helper the array was sorted in descending
instead of ascending order.
Implement missing transaction handling.
Fix logging wrongly casting 'state' to DCERPCState instead of
DCERPCUDPState leading to crashes and malformed output.
Remove unused fields from DCERPCUDPState.
DNP3 uses little endian on the wire, for the most part this
is handled as the messages are deserialize. However, the link
header is a cast over raw data, so swap these bytes as they
are being logged.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4173
So that protocol detection does not run for too long because
TCPProtoDetectCheckBailConditions somehow relies on its TCP stream
to start from zero, which is not the case on protocol change
Adds also debug validation checks, such as
both sides are known on protocol change
And only sets once alproto_orig
This commit improves handling of ICMPv4 messages, especially those with
variable sized headers.
This commit also adds a header length variable for use by the new
sticky buffer for the header.
Removed the unnecessary restriction on the use of inline mode only when
bypass is enabled. Now, Inline can be used independent of bypass
functionality.
This addresses readmine issue #4018. There was the potential for a packet
buffer to be released twice in response to an error condition. This
addresses this by only calling NT_NetRxRelease() when the p->ReleasePacket
is called.
We must make sure not to access the flow storage (e.g. keeping a
MacSet) before making sure we have a flow to begin with, We can,
for example, run into an alert without a flow with `ip` rules,
in which case the flow might be NULL. See Redmine issue #4109.
This commit improves support for shared buffer usage, i.e., when
multiple rules share the file data (http) buffer and apply different
combinations of transforms and fast_patterns (or none).
This changseset fixes a bug causing a segmentation fault.
When rules are reloaded and a rule using libmagic matches, suricata
crashes due to an improper reinitialization of the thread contexts.
Bug: #3726
This changeset fixes a bug that was preventing suricata to dump
alert metadata info when running in unix-socket mode.
When running in unix-socket mode, suricata was skipping the
initialization of the output modules and, as a consequence,
the metadata output module was never invoked.
This commit adds a buffer validator for compress whitespace. Buffers
containing two or more consecutive whitespace characters are invalid
with this transform.
This commit changes the name of the file used with threaded eve logging
to better support log rotation
Instead of using "eve.json.N" and creating potential issues with log
rotation (which also uses a ".N" suffix), the eve logs will be named
"eve.N.json" when threaded.
This commit changes the size of reporting variables to be dynamic based
on the buffer ids in use instead of a fixed value to address a SEGV when
the fixed value was less than the max buffer/type id in use.
If a pattern matches in the other direction, after
probing parser finished without finding a protocol,
we will rerun the probing parser, which will include
the newly protocol found by its pattern
It a protocol is found in a first direction, we should run the
probing parser, even if it is not in the known ports.
That can happen for HTTP2, where client magic is detected,
then server probe can be run
TCPProtoDetect can either set f->alproto, change f->alstate
and return error.
When the original alstate gets freed, we shall set the pointer
to NULL, as it can get reused.
When one side of the connection reaches the STREAM_DEPTH condition the
parser should be aware of this. Otherwise transactions will forever be
waiting for data in that direction.
This commit changes the name of the "fileinfo" array in the alert object
to "files" to better support legacy use of "fileinfo" in reporting and
elsewhere.
The "fileinfo" event type is not an array while the alert "fileinfo"
member was.
Datasets can now have a global defaults setting in suricata.yaml. In
case the settings for memcap and hashsize are not find in the yaml or
rule, this shall be the fallback.
Example:
datasets:
defaults:
memcap: 100mb
hashsize: 2048
ua-seen:
type: string
load: datasets.csv
It is now possible to set the memcap and hashsize via suricata.yaml and
rules.
Rule example:
alert http any any -> any any (http.user_agent; dataset:isset,ua-seen,type string,load datasets.csv,memcap 100mb,hashsize 2048; sid:1;)
suricata.yaml example:
datasets:
ua-seen:
type: string
load: datasets.csv
memcap: 20mb
hashsize: 2048
This commit adds an interface to declare and initialize an atomic with a
specific value. This can help with situations where there's no defined
initialization path to set things up.
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2
This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
Test the full path instead of just the filename provided in the
rule to see if it exists.
Fixes the case where a rule file is loaded from a directory
other than the default-rule-directory.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3916
The JsonBuilder set_string_from_bytes will escape unprintable
characters in the output stream. Using BytesToStringBuffer
can generate invalid UTF-8 which prevents the conversion from
a C string to a Rust string.
Instead of looking for a symbol, "PluginSpec" look for a function
named SCPluginRegister that returns a SCPlugin.
This makes it much easier to create Rust plugins without having
to deal with dlopen constructors and such, which is rather
straight forward in C, but a bit of advanced boilerplate in Rust
that can be eliminated by simply calling a registration function.
Expose the "SuricataContext" required by Rust as a function. During
normal startup we register this context with the Rust code, but
plugins written in Rust will need to get the same registration
done, but to do this in a plugin, the plugin code must
call and set the context within its address space.