It was pointed out by a contributor that our workflow mentioned
rewrite-branch as the preferred way, while in fact our policy is to add
said changes to a different commit. Updating documentation to prevent
other situations like that.
This commit adds brief discussion for additional cluster types for use
with the pf-ring packet source.
Newly added:
- cluster_inner_flow
- cluster_inner_flow_2_tuple
- cluster_inner_flow_4_tuple
- cluster_inner_flow_5_tuple
Issue: 5975
As flow.memcap-policy and defrag.memcap-policy do not support flow
actions, clarify that in the documentation. Also fix some typos, and
add missing values in some places where the exception policies were
explained.
Related to
Bug #5940
This allows ftp-data and ftp flows to be processed by the same
thread. Otherwise, there may be a concurrency issue where the
would-be ftp-data flow is first processed, and thus not recognized
as such. And the ftp flow gets processed later and the expectation
coming from it is never found.
To do so, the flow hash gets used as usual, except for flows that
may be either ftp or ftp-data, that is either one port is 21, or
both ports are high ones.
Ticket: #5205
This allows all traffic Exception Policies to be set from one
configuration point. All exception policy options are available in IPS
mode. Bypass, pass and auto (disabled) are also available in iDS mode
Exception Policies set up individually will overwrite this setup for the
given traffic exception.
Task #5219
As part of the process of moving documentation from redmine
to "Read the Docs", this commit moves installing Suricata using git
page from redmine wiki into Suricata Developer Guide section.
It also updates the necessary steps.
Ticket: #5585
This commit updates the bsize documentation
1. Describe what happens when "content" immediately precedes "bsize"
2. Include the operators and
3. Include examples using the operators.
with setrlimit NPROC.
So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.
Ticket: #5373
This patch updates the NT status code definition to use the status
definition used on Microsoft documentation website. A first python
script is building JSON object with code definition.
```
import json
from bs4 import BeautifulSoup
import requests
ntstatus = requests.get('https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55')
ntstatus_parsed = BeautifulSoup(ntstatus.text, 'html.parser')
ntstatus_parsed = ntstatus_parsed.find('tbody')
ntstatus_dict = {}
for item in ntstatus_parsed.find_all('tr'):
cell = item.find_all('td')
if len(cell) == 0:
continue
code = cell[0].find_all('p')
description_ps = cell[1].find_all('p')
description_list = []
if len(description_ps):
for desc in description_ps:
if not desc.string is None:
description_list.append(desc.string.replace('\n ', ''))
else:
description_list = ['Description not available']
if not code[0].string.lower() in ntstatus_dict:
ntstatus_dict[code[0].string.lower()] = {"text": code[1].string, "desc": ' '.join(description_list)}
print(json.dumps(ntstatus_dict))
```
The second one is generating the code that is ready to be inserted into the
source file:
```
import json
ntstatus_file = open('ntstatus.json', 'r')
ntstatus = json.loads(ntstatus_file.read())
declaration_format = 'pub const SMB_NT%s:%su32 = %s;\n'
resolution_format = ' SMB_NT%s%s=> "%s",\n'
declaration = ""
resolution = ""
text_max = len(max([ntstatus[x]['text'] for x in ntstatus.keys()], key=len))
for code in ntstatus.keys():
text = ntstatus[code]['text']
text_spaces = ' ' * (4 + text_max - len(text))
declaration += declaration_format % (text, text_spaces, code)
resolution += resolution_format % (text, text_spaces, text)
print(declaration)
print('\n')
print('''
pub fn smb_ntstatus_string(c: u32) -> String {
match c {
''')
print(resolution)
print('''
_ => { return (c).to_string(); },
}.to_string()
}
''')
```
Bug #5412.
This describes briefly what the exception policies are, what is the
engine's behavior, what options are available and to which parts are
they implemented.
Task #5475
Task #5515
Some of these were recently introduced, some were highlited after the
applayer sections got merged. Some paragraphs seem to have been changed
due to trying to respect character limits for lines. Also includes a
typo pointed out by one of our community members via Discord.
This enables the usage of 'reject' as an exception policy. As for both
IPS and IDS modes the intended result of sending a reject packet is to
reject the related flow, this will effectively mean setting the reject
action to the packet that triggered the exception condition, and then
dropping the associated flow.
Task #5503
We had two sections under the suricata.yaml configuration section
describing settings for application layer parsers. This merges them into
one and also fixes a few subsection title levels.
Task #5364
sphinx-build 5.1.1 and above throws a warning which is treated as an
error while building.
Invalid configuration value found: 'language = None'. Update your configuration to a valid language code. Falling back to 'en' (English).
While Suri will throw an error if two signatures have the same `sid`
and no `gid`, or same `sid` and same `gid`, it will just accept same
`sid` for different `gid`s.
Related to
Task #5441
The image describing Suricata's sliding window had two of the "packets"
with the same text. Now they actually give the sense of a sliding
window.
This was found by Zhiyuan-liao.
That page existed only in our redmine. Updated and added a few things,
like a paragraph about our expectations for feature contributors.
Also updated links, contacts and some other processes that may have
changed since last edition.
Added some section labels in related documents, for ease of referencing.
Task #4929
Added sections along packet-alert-max config section explaining
packet alert queue overflow (when Suri reaches packet alert max), when
alerts are discarded etc.
Since from the user perspective it shouldn't matter how we process the
alert queue, the term "replace" is used, even though there's not exactly
a replacing action happening, with the queue bein pre-processed before
being appended to the Packet.
Also described the associated stats and added an explanation on when to
change packet-alert-max.
Task #5178
Use it to explain how to go about the sequence diagram images
(generation, updating, what is mscgen etc).
Also remove portion that referred to Sphinx builds, as these don't make
sense now.
Currently, it seems easier to upload the diagram images to git than to
try to make the image generation script work with out of the tree builds
and other corner cases.
This means, however, that one must activelly remember to update msc
diagram files, run the script and re-add new png files, if those ever
need to be updated. To raise awareness to that, a watermark was added
to the diagram images.
Also removed configuration steps that added mscgen as dependency
(locally and for workflow builds and readthedocs).
Moved devguide dir into userguide dir.
Since the devguide is now incorporated as the last chapter of the
userguide, removed build and configuration files from the devguide
dir, as these are no longer needed.
Task #4909
Issue: 4550
This commit documents the new per-thread stack-size setting. Some
systems have a small default value that is not suitable for Suricata's
multi-threaded architecture and adjustment may be required.
This explains the frame support from the perspective of a developer,
with introductory explanation on how to add frame support to an
applayer proto.
Doc #4697
While ReadTheDocs can't execute arbitrary scripts, we can install
mscgen in the container that builds the docs (see .readthedocs.yaml).
Then instead of calling generate-images.sh from the Makefile, move this
into the setup function defined on conf.yaml, which will generate the
scripts as part of a normal Sphinx workflow.
This should give us an image generation solution that works on
ReadTheDocs, and locally within anyones build system provided they have
mscgen installed.
- add nom parsers for decoding most messages from StartupPhase and
SimpleQuery subprotocols
- add unittests
- tests/fuzz: add pgsql to confyaml
Feature: #4241
Many places were still referencing the old Suricata page.
Used git grep with replace to update them. Checked that new links work.
Left old references when they were only documentation examples (for
output or unittests).
Task#4915
The differences on how the `need` key works, depending on script
usage (output or detection) confuses users, sometimes (cf doc#4725).
While we don't fix that, just explain this behavior.
Part of the task to offer better guidance on how and when to write
unit tests or suricata-verify tests
Also updated linking and index files, as well as testing page to refer
to the unit tests pages
Doc: #4590
Part of ongoing task to add more guidance on how to create unittests
and suricata-verify tests for suri. There will also be a unittests-rust
page.
Doc: #4590
This page offers guidance about when to use unittests or s-v tests,
and how to create input for those. Also lists other common ways to test
Suri, such as fuzzing and the CI checks.
Doc: #4590
Semantically speaking it makes more sense, because it stores `msc`
files for dynamic image generation.
Updated files that refered to `img` accordingly, too.
- DNS sequence diagram was incorrect (transactions should be
unidirectional). After changing it, it made sense to rename the file.
Adjusted spacing, too. Updated transactions.rst accordingly.
- TLS sequence diagram was refined to illustrate how Suricata actually
implements the protocol.
A guide on what is a transaction for Suricata engine, focusing on
developers.
- What's the purpose of a transaction;
- transaction states and API callbacks;
- Examples and sequence diagrams.
- doc/devguide: add transactions.rst
- doc/devguide/extending/app-layer/index.rst: add transactions.rst
This commit adds a one-liner to the upgrade document for 7.0 stating
that protocol names/values are now builtin to Suricata and that names
and their casing may change.
Renaming was done with shell commands, git mv for moving the files and content like
find -iname '*.c' | xargs sed -i 's/ikev1/ike/g' respecting the different mixes of upper/lower case.
Fix a mistake in Makefile.am where the man page was only being
added to the distribution if the PDF was also created. It should
be included even if the PDF cannot be included.
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
Fixes https://redmine.openinfosecfoundation.org/issues/2689
Adds a new source file to handle this keyword.
And modifies documentation, Makefile, and registration accordingly.
url_decode decodes url-encoded data, ie replacing '+' with space
and '%HH' with its value.
Elastic search didn't accept the 'hassh' and 'hassh.string'. It would
see the first 'hassh' as a string and split the second key into a
object 'hassh' with a string member 'string'. So two different types
for 'hassh', so it rejected it.
This patch mimics the ja3(s) logging by creating a 'hassh' object
with 2 members: 'hash', which holds the md5 representation, and
'string' which holds the string representation.
This commit improves the description of the `tos` keyword by emphasizing
that the value used should adhere to the guidelines in RFC2474. Instead
of specifying the DSCP value directly, right shift the DSCP value and
use that.
This commit adds code submission and coding style guidelines to the
devguide. Most of the material is a straight port from the wiki but
there have been some content modifications and additions.
Implement support for limiting Teredo detection and decoding to specific
UDP ports, with 3544 as the default.
If no ports are specified, the old behaviour of detecting/decoding on any
port is still in place. This can also be forced by specifying 'any' as the
port setting.
This commit adds support for the Remote Framebuffer Protocol (RFB) as
used, for example, by various VNC implementations. It targets the
official versions 3.3, 3.7 and 3.8 of the protocol and provides logging
for the RFB handshake communication for now. Logged events include
endpoint versions, details of the security (i.e. authentication)
exchange as well as metadata about the image transfer parameters.
Detection is enabled using keywords for:
- rfb.name: Session name as sticky buffer
- rfb.sectype: Security type, e.g. VNC-style challenge-response
- rfb.secresult: Result of the security exchange, e.g. OK, FAIL, ...
The latter could be used, for example, to detect brute-force attempts
on open VNC servers, while the name could be used to map unwanted VNC
sessions to the desktop owners or machines.
We also ship example EVE-JSON output and keyword docs as part of the
Sphinx source for Suricata's RTD documentation.
Update -v documentation to reflect the new behaviour discussed
in bug #1851 where -v changes the log level to fixed levels
instead of an offset of the default log level configured
in suricata.yaml.
This is the loggers such as alert-json-log, dns-json-log, etc.
They are not even referenced in the default configuration file,
and are easily replaced with multiple eve instances.