with setrlimit NPROC.
So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.
Ticket: #5373
This patch updates the NT status code definition to use the status
definition used on Microsoft documentation website. A first python
script is building JSON object with code definition.
```
import json
from bs4 import BeautifulSoup
import requests
ntstatus = requests.get('https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55')
ntstatus_parsed = BeautifulSoup(ntstatus.text, 'html.parser')
ntstatus_parsed = ntstatus_parsed.find('tbody')
ntstatus_dict = {}
for item in ntstatus_parsed.find_all('tr'):
cell = item.find_all('td')
if len(cell) == 0:
continue
code = cell[0].find_all('p')
description_ps = cell[1].find_all('p')
description_list = []
if len(description_ps):
for desc in description_ps:
if not desc.string is None:
description_list.append(desc.string.replace('\n ', ''))
else:
description_list = ['Description not available']
if not code[0].string.lower() in ntstatus_dict:
ntstatus_dict[code[0].string.lower()] = {"text": code[1].string, "desc": ' '.join(description_list)}
print(json.dumps(ntstatus_dict))
```
The second one is generating the code that is ready to be inserted into the
source file:
```
import json
ntstatus_file = open('ntstatus.json', 'r')
ntstatus = json.loads(ntstatus_file.read())
declaration_format = 'pub const SMB_NT%s:%su32 = %s;\n'
resolution_format = ' SMB_NT%s%s=> "%s",\n'
declaration = ""
resolution = ""
text_max = len(max([ntstatus[x]['text'] for x in ntstatus.keys()], key=len))
for code in ntstatus.keys():
text = ntstatus[code]['text']
text_spaces = ' ' * (4 + text_max - len(text))
declaration += declaration_format % (text, text_spaces, code)
resolution += resolution_format % (text, text_spaces, text)
print(declaration)
print('\n')
print('''
pub fn smb_ntstatus_string(c: u32) -> String {
match c {
''')
print(resolution)
print('''
_ => { return (c).to_string(); },
}.to_string()
}
''')
```
Bug #5412.
This describes briefly what the exception policies are, what is the
engine's behavior, what options are available and to which parts are
they implemented.
Task #5475
Task #5515
Some of these were recently introduced, some were highlited after the
applayer sections got merged. Some paragraphs seem to have been changed
due to trying to respect character limits for lines. Also includes a
typo pointed out by one of our community members via Discord.
This enables the usage of 'reject' as an exception policy. As for both
IPS and IDS modes the intended result of sending a reject packet is to
reject the related flow, this will effectively mean setting the reject
action to the packet that triggered the exception condition, and then
dropping the associated flow.
Task #5503
We had two sections under the suricata.yaml configuration section
describing settings for application layer parsers. This merges them into
one and also fixes a few subsection title levels.
Task #5364
sphinx-build 5.1.1 and above throws a warning which is treated as an
error while building.
Invalid configuration value found: 'language = None'. Update your configuration to a valid language code. Falling back to 'en' (English).
While Suri will throw an error if two signatures have the same `sid`
and no `gid`, or same `sid` and same `gid`, it will just accept same
`sid` for different `gid`s.
Related to
Task #5441
The image describing Suricata's sliding window had two of the "packets"
with the same text. Now they actually give the sense of a sliding
window.
This was found by Zhiyuan-liao.
That page existed only in our redmine. Updated and added a few things,
like a paragraph about our expectations for feature contributors.
Also updated links, contacts and some other processes that may have
changed since last edition.
Added some section labels in related documents, for ease of referencing.
Task #4929
Added sections along packet-alert-max config section explaining
packet alert queue overflow (when Suri reaches packet alert max), when
alerts are discarded etc.
Since from the user perspective it shouldn't matter how we process the
alert queue, the term "replace" is used, even though there's not exactly
a replacing action happening, with the queue bein pre-processed before
being appended to the Packet.
Also described the associated stats and added an explanation on when to
change packet-alert-max.
Task #5178
Use it to explain how to go about the sequence diagram images
(generation, updating, what is mscgen etc).
Also remove portion that referred to Sphinx builds, as these don't make
sense now.
Currently, it seems easier to upload the diagram images to git than to
try to make the image generation script work with out of the tree builds
and other corner cases.
This means, however, that one must activelly remember to update msc
diagram files, run the script and re-add new png files, if those ever
need to be updated. To raise awareness to that, a watermark was added
to the diagram images.
Also removed configuration steps that added mscgen as dependency
(locally and for workflow builds and readthedocs).
Moved devguide dir into userguide dir.
Since the devguide is now incorporated as the last chapter of the
userguide, removed build and configuration files from the devguide
dir, as these are no longer needed.
Task #4909
Issue: 4550
This commit documents the new per-thread stack-size setting. Some
systems have a small default value that is not suitable for Suricata's
multi-threaded architecture and adjustment may be required.
This explains the frame support from the perspective of a developer,
with introductory explanation on how to add frame support to an
applayer proto.
Doc #4697
While ReadTheDocs can't execute arbitrary scripts, we can install
mscgen in the container that builds the docs (see .readthedocs.yaml).
Then instead of calling generate-images.sh from the Makefile, move this
into the setup function defined on conf.yaml, which will generate the
scripts as part of a normal Sphinx workflow.
This should give us an image generation solution that works on
ReadTheDocs, and locally within anyones build system provided they have
mscgen installed.
- add nom parsers for decoding most messages from StartupPhase and
SimpleQuery subprotocols
- add unittests
- tests/fuzz: add pgsql to confyaml
Feature: #4241
Many places were still referencing the old Suricata page.
Used git grep with replace to update them. Checked that new links work.
Left old references when they were only documentation examples (for
output or unittests).
Task#4915
The differences on how the `need` key works, depending on script
usage (output or detection) confuses users, sometimes (cf doc#4725).
While we don't fix that, just explain this behavior.
Part of the task to offer better guidance on how and when to write
unit tests or suricata-verify tests
Also updated linking and index files, as well as testing page to refer
to the unit tests pages
Doc: #4590
Part of ongoing task to add more guidance on how to create unittests
and suricata-verify tests for suri. There will also be a unittests-rust
page.
Doc: #4590
This page offers guidance about when to use unittests or s-v tests,
and how to create input for those. Also lists other common ways to test
Suri, such as fuzzing and the CI checks.
Doc: #4590
Semantically speaking it makes more sense, because it stores `msc`
files for dynamic image generation.
Updated files that refered to `img` accordingly, too.
- DNS sequence diagram was incorrect (transactions should be
unidirectional). After changing it, it made sense to rename the file.
Adjusted spacing, too. Updated transactions.rst accordingly.
- TLS sequence diagram was refined to illustrate how Suricata actually
implements the protocol.
A guide on what is a transaction for Suricata engine, focusing on
developers.
- What's the purpose of a transaction;
- transaction states and API callbacks;
- Examples and sequence diagrams.
- doc/devguide: add transactions.rst
- doc/devguide/extending/app-layer/index.rst: add transactions.rst
This commit adds a one-liner to the upgrade document for 7.0 stating
that protocol names/values are now builtin to Suricata and that names
and their casing may change.
Renaming was done with shell commands, git mv for moving the files and content like
find -iname '*.c' | xargs sed -i 's/ikev1/ike/g' respecting the different mixes of upper/lower case.
Fix a mistake in Makefile.am where the man page was only being
added to the distribution if the PDF was also created. It should
be included even if the PDF cannot be included.
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
Fixes https://redmine.openinfosecfoundation.org/issues/2689
Adds a new source file to handle this keyword.
And modifies documentation, Makefile, and registration accordingly.
url_decode decodes url-encoded data, ie replacing '+' with space
and '%HH' with its value.
Elastic search didn't accept the 'hassh' and 'hassh.string'. It would
see the first 'hassh' as a string and split the second key into a
object 'hassh' with a string member 'string'. So two different types
for 'hassh', so it rejected it.
This patch mimics the ja3(s) logging by creating a 'hassh' object
with 2 members: 'hash', which holds the md5 representation, and
'string' which holds the string representation.
This commit improves the description of the `tos` keyword by emphasizing
that the value used should adhere to the guidelines in RFC2474. Instead
of specifying the DSCP value directly, right shift the DSCP value and
use that.
This commit adds code submission and coding style guidelines to the
devguide. Most of the material is a straight port from the wiki but
there have been some content modifications and additions.
Implement support for limiting Teredo detection and decoding to specific
UDP ports, with 3544 as the default.
If no ports are specified, the old behaviour of detecting/decoding on any
port is still in place. This can also be forced by specifying 'any' as the
port setting.
This commit adds support for the Remote Framebuffer Protocol (RFB) as
used, for example, by various VNC implementations. It targets the
official versions 3.3, 3.7 and 3.8 of the protocol and provides logging
for the RFB handshake communication for now. Logged events include
endpoint versions, details of the security (i.e. authentication)
exchange as well as metadata about the image transfer parameters.
Detection is enabled using keywords for:
- rfb.name: Session name as sticky buffer
- rfb.sectype: Security type, e.g. VNC-style challenge-response
- rfb.secresult: Result of the security exchange, e.g. OK, FAIL, ...
The latter could be used, for example, to detect brute-force attempts
on open VNC servers, while the name could be used to map unwanted VNC
sessions to the desktop owners or machines.
We also ship example EVE-JSON output and keyword docs as part of the
Sphinx source for Suricata's RTD documentation.
Update -v documentation to reflect the new behaviour discussed
in bug #1851 where -v changes the log level to fixed levels
instead of an offset of the default log level configured
in suricata.yaml.
This is the loggers such as alert-json-log, dns-json-log, etc.
They are not even referenced in the default configuration file,
and are easily replaced with multiple eve instances.
Older versions of Sphinx will generate duplicate IDs when you have
options like:
.. option:: some-option
.. option:: some-other-option
The version of Sphinx provided on CentOS 7 has this issue, newer
versions of Sphinx do not. As CentOS 7 is still a popular
distribution, change ".. option" to ".. describe" which has the
same visual output, but does not generate links.
Modified transaction logic to create a new transaction with each
request; replies location transactions by using the oldest "open"
(unmatched) transaction or the last transaction if none are open.
Kernel time is not available (and/or costly) on NIC such as
Netronome so we update the logic to detect dead flows based on a
lack of update of packets counters. This way, the XDP filter will
be usable by network card.
This patch also updates the ebpf code to support per CPU and
regular mapping. Netronome is not supporting it and the structure
is using atomic for counter so the cost of simultaneous update
is really low.
This patch also updates the xdp_filter to be able to select if the
flow table is per CPU on shared. Second option will be used for
hardward offload. To deactivate the per cpu hash, you need to set
USE_PERCPU_HASH to 0.
This patch also adds an new option to af-packet named no-percpu-hash
If this option is set to yes then the Flow bypassed manager thread
will use one CPU instead of the number of cores. By doing that
we are able to handle the case where USE_PERCPU_HASH is unset (so
hardware offload for Netronome).
This patch also remove aligment indications in the eBPF filter. This
was not really needed and it seems it is causing problem with
some recent version of LLVM toolchain.
Add the missing manpages and the corresponding Sphinx configuration
for the command line tools `suricatasc` and `suricatactl`.
Closes redmine ticket #884.
Process multiple packets at nm_dispatch. Use zero copy for workers
recv mode.
Add configure check netmap check for API 11+ and find netmap api version.
Add netmap guide to the userguide.
Add a keyword configuration dump-all-headers, with allowed values
{both, request, response}, dumping all HTTP headers in the eve-log http
object. Each header is a single object in the list request_headers
(response_headers) with the following notation:
{
"name": <header name>,
"value": <header value>
}
To avoid forged malicious headers, the header name size is capped at 256
bytes, the header value size at 2048.
By default, dump-all-headers is disabled.
Add a raw-extraction option for smtp. When enabled, this feature will
store the raw e-mail inside a file, including headers, e-mail content,
attachments (base64 encoded). This content is stored in a normal File *,
allowing for normal file detection.
It'd also allow for all-emails extraction if a rule has
detect-filename:"rawmsg" matcher (and filestore).
Note that this feature is in contrast with decode-mime.
This feature is disabled by default, and will be disabled automatically
if decode-mime is enabled.