One of the lessons of the XZ backdoor story was that just linking to
libsystemd to call sd_notify is discouraged by the systemd project:
Lennart Poettering:
"PSA: In context of the xzpocalypse we now added an example reimplementation
of sd_notify() to our man page:
https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
It's pretty comprehensive (i.e. uses it for reload notification too), but
still relatively short.
In the past, I have been telling anyone who wanted to listen that if all you
want is sd_notify() then don't bother linking to libsystemd, since the
protocol is stable and should be considered the API, not our C wrapper
around it. After all, the protocol is so trivial"
From: https://mastodon.social/@pid_eins/112202687764571433
This commit takes the example code and uses it to reimplement the notify
logic.
The code is enabled if Linux is detected in configure. Since the code
won't do anything if the NOTIFY_SOCKET env var isn't set, this should
also work fine on systems w/o systemd.
Ticket: #6913.
By default, when Sphinx generates the man pages, the current date will
be embedded in them. This can be set to a specific date with the
"today" variable. Typically the date embedded in manpages in the
release date.
To achieve this, attempt to use the environment variable, RELEASE_DATE
to set the "today" variable, reverting back to the empty string if not
set. It is up to our build system to properly set this date.
Ticket: #6911
dns.rcode matches the rcode header field in DNS messages
It's an unsigned integer
valid ranges = [0-15]
Does not support prefilter
Supports matches in both flow directions
Task #6621
It matches the rrtype field in DNS
It's an unsigned integer match
valid ranges = [0-65535]
Does not support prefilter
Supports flow in both directions
Feature #6666
Our code handles Uint ranges as exclusive, but for bsize, our
documentation stated that they're inclusive.
Cf. from uint.rs:
DetectUintMode::DetectUintModeRange => {
if val > x.arg1 && val < x.arg2 {
return true;
}
}
Task #6708
Remove references that are mentioning Suricata 3 or less
As a note - only one Suricata 4 reference found:
(suricata-yaml.rst:"In 4.1.x")
Fast pattern selection criteria can be internally found by inspecting
SupportFastPatternForSigMatchList and SigTableSetup functions.
Ticket: #6570
When the packet load is low, Suricata can run in interrupt
mode. This more resembles the classic approach of processing
packets - CPU cores run low and only fetch packets
on interrupt.
Ticket: #5839
Previous integration of hugepage analysis only fetched data
from /proc/meminfo. However this proved to be often
deceiving mainly for providing only global information and
not taking into account different hugepage sizes (e.g. 1GB
hugepages) and different NUMA nodes.
Ticket: #6419
If a commit introduces code that changes Suricata behavior, the related
documentation changes should go in a separate commit, but refer to the
same ticket number.
This reduces the chances of said changes being lost if there are backports
while still keeping the backporting process a bit less bulky, for each
commit.
Related to
Task #6568
This section seemed to aim both at PR reviewers and PR authors at the
same time, even though some info is probably of low value for
contributors.
Created new section for PR reviewers and maintainers, and kept the info
for PR authors separated. Also highlighted information on requested
changes and stale PRs.
This could be justified from a semantic point of view, and also can help
in bringing more attention to where this information is, as it is less
hidden, now.
Also add Dev Guide as one of our resources in our Readme.
Add a new rule keyword "requires" that allows a rule to require specific
Suricata versions and/or Suricata features to be enabled.
Example:
requires: feature geoip, version >= 7.0.0, version < 8;
requires: version >= 7.0.3 < 8
requires: version >= 7.0.3 < 8 | >= 8.0.3
Feature: #5972
Co-authored-by: Philippe Antoine <pantoine@oisf.net>
Add a more visible explanation of that requests, responses, frontend and
and backend are, in Pgsql context, to avoid having to repeat that over
different portions of the docs.
A CanceldRequest can occur after any query request, and is sent over a
new connection, leading to a new flow. It won't take any reply, but, if
processed by the backend, will lead to an ErrorResponse.
Task #6577
Rename DetectAppLayerInspectEngineRegister2 to
DetectAppLayerInspectEngineRegister as there is no other variant of
this function, and the versioning with lack of supporting
documentation can lead to confusion.
The old DetectAppLayerMpmRegister has not been around since 4.1.x.
Rename the v2 of this function to a versionless function as there is no
documentation referring to what the 2 means.
The term "whitelist" is actually used to store a list of DetectPort type
items for tcp and udp in detect.h. Using the same term for also keeping
the score that affects the grouping of rules is confusing. So, rename
the variable to "score".
Our tls fields not_after and not_before are actually logged as
`notafter` and `notbefore`, but were documented with the underscore.
Update the documentation, since updating the log format itself would be
a breaking change.
Task #5494
Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.
Kept that reference, since it might still be useful for non-Ubuntu
cases.
- Use SPHINX_BUILD instead of HAVE_SPHINX_BUILD, as here we're
actually using the path of the program.
- Wrap some elements in [] as is done in modern idiomatic autoconf
When viewing the docs online at Readthedocs, or similar it might be
immediately apparent what version of the documentation is being
displayed. Display the version on the first line before the table of
contents to make it clear.
With the release of 7, people are starting to have issues with traffic
being blocked. While we don't add a more expansive documentation for
this, add a link to the FAQ covering possible fixes for drops caused by
the fail closed default behavior of the exception policies.
Add CSS to avoid horizontal scroll in tables on ReadTheDocs. This will
wrap the text instead.
Also, vertically align to top so if a cell does wrap, other cells that
do not wrap don't place the text in the middle of the cell.
The `field action` portion seemed to be comprised of a more generic
section that followed it. Also formatted the section for lines to be
within the character limit.
If an exception policy wasn't set up individually, use the GetDefault
function to pick one. This will check for the master switch option and
handle 'auto' cases.
Instead of deciding what the auto value should be when we are parsing
the master switch, leave that for when some of the other policies is to
be set via the master switch, when since this can change for specific
exception policies - like for midstream, for instance.
Update exceptions policies documentation to clarify that the default
configuration in IPS when midstream is enabled is `ignore`, not
`drop-flow`.
Bug #6169
DPDK apps can specify multiple arguments of the same
type. YAML format only allows unique keys within a single
node. This commit adds support for multiple EAL arguments
of the same type to be used within suricata.yaml.
Ticket: #5964
Some exception policies can only be applied to the triggering packet or
only make sense considering the whole flow. Highlight such cases in the
table showing each exception policy.
Related to
Bug #5825
The different interactions between midstream pick-up sessions and the
exception policy can be quite difficult to visualize. Add a section for
that in the userguide.
Related to
Bug #5825
It was pointed out by a contributor that our workflow mentioned
rewrite-branch as the preferred way, while in fact our policy is to add
said changes to a different commit. Updating documentation to prevent
other situations like that.
This commit adds brief discussion for additional cluster types for use
with the pf-ring packet source.
Newly added:
- cluster_inner_flow
- cluster_inner_flow_2_tuple
- cluster_inner_flow_4_tuple
- cluster_inner_flow_5_tuple
Issue: 5975
As flow.memcap-policy and defrag.memcap-policy do not support flow
actions, clarify that in the documentation. Also fix some typos, and
add missing values in some places where the exception policies were
explained.
Related to
Bug #5940
This allows ftp-data and ftp flows to be processed by the same
thread. Otherwise, there may be a concurrency issue where the
would-be ftp-data flow is first processed, and thus not recognized
as such. And the ftp flow gets processed later and the expectation
coming from it is never found.
To do so, the flow hash gets used as usual, except for flows that
may be either ftp or ftp-data, that is either one port is 21, or
both ports are high ones.
Ticket: #5205
This allows all traffic Exception Policies to be set from one
configuration point. All exception policy options are available in IPS
mode. Bypass, pass and auto (disabled) are also available in iDS mode
Exception Policies set up individually will overwrite this setup for the
given traffic exception.
Task #5219
As part of the process of moving documentation from redmine
to "Read the Docs", this commit moves installing Suricata using git
page from redmine wiki into Suricata Developer Guide section.
It also updates the necessary steps.
Ticket: #5585
This commit updates the bsize documentation
1. Describe what happens when "content" immediately precedes "bsize"
2. Include the operators and
3. Include examples using the operators.
with setrlimit NPROC.
So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.
Ticket: #5373
This patch updates the NT status code definition to use the status
definition used on Microsoft documentation website. A first python
script is building JSON object with code definition.
```
import json
from bs4 import BeautifulSoup
import requests
ntstatus = requests.get('https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55')
ntstatus_parsed = BeautifulSoup(ntstatus.text, 'html.parser')
ntstatus_parsed = ntstatus_parsed.find('tbody')
ntstatus_dict = {}
for item in ntstatus_parsed.find_all('tr'):
cell = item.find_all('td')
if len(cell) == 0:
continue
code = cell[0].find_all('p')
description_ps = cell[1].find_all('p')
description_list = []
if len(description_ps):
for desc in description_ps:
if not desc.string is None:
description_list.append(desc.string.replace('\n ', ''))
else:
description_list = ['Description not available']
if not code[0].string.lower() in ntstatus_dict:
ntstatus_dict[code[0].string.lower()] = {"text": code[1].string, "desc": ' '.join(description_list)}
print(json.dumps(ntstatus_dict))
```
The second one is generating the code that is ready to be inserted into the
source file:
```
import json
ntstatus_file = open('ntstatus.json', 'r')
ntstatus = json.loads(ntstatus_file.read())
declaration_format = 'pub const SMB_NT%s:%su32 = %s;\n'
resolution_format = ' SMB_NT%s%s=> "%s",\n'
declaration = ""
resolution = ""
text_max = len(max([ntstatus[x]['text'] for x in ntstatus.keys()], key=len))
for code in ntstatus.keys():
text = ntstatus[code]['text']
text_spaces = ' ' * (4 + text_max - len(text))
declaration += declaration_format % (text, text_spaces, code)
resolution += resolution_format % (text, text_spaces, text)
print(declaration)
print('\n')
print('''
pub fn smb_ntstatus_string(c: u32) -> String {
match c {
''')
print(resolution)
print('''
_ => { return (c).to_string(); },
}.to_string()
}
''')
```
Bug #5412.
This describes briefly what the exception policies are, what is the
engine's behavior, what options are available and to which parts are
they implemented.
Task #5475
Task #5515
Some of these were recently introduced, some were highlited after the
applayer sections got merged. Some paragraphs seem to have been changed
due to trying to respect character limits for lines. Also includes a
typo pointed out by one of our community members via Discord.
This enables the usage of 'reject' as an exception policy. As for both
IPS and IDS modes the intended result of sending a reject packet is to
reject the related flow, this will effectively mean setting the reject
action to the packet that triggered the exception condition, and then
dropping the associated flow.
Task #5503
We had two sections under the suricata.yaml configuration section
describing settings for application layer parsers. This merges them into
one and also fixes a few subsection title levels.
Task #5364
sphinx-build 5.1.1 and above throws a warning which is treated as an
error while building.
Invalid configuration value found: 'language = None'. Update your configuration to a valid language code. Falling back to 'en' (English).
While Suri will throw an error if two signatures have the same `sid`
and no `gid`, or same `sid` and same `gid`, it will just accept same
`sid` for different `gid`s.
Related to
Task #5441
The image describing Suricata's sliding window had two of the "packets"
with the same text. Now they actually give the sense of a sliding
window.
This was found by Zhiyuan-liao.
That page existed only in our redmine. Updated and added a few things,
like a paragraph about our expectations for feature contributors.
Also updated links, contacts and some other processes that may have
changed since last edition.
Added some section labels in related documents, for ease of referencing.
Task #4929
Added sections along packet-alert-max config section explaining
packet alert queue overflow (when Suri reaches packet alert max), when
alerts are discarded etc.
Since from the user perspective it shouldn't matter how we process the
alert queue, the term "replace" is used, even though there's not exactly
a replacing action happening, with the queue bein pre-processed before
being appended to the Packet.
Also described the associated stats and added an explanation on when to
change packet-alert-max.
Task #5178
Use it to explain how to go about the sequence diagram images
(generation, updating, what is mscgen etc).
Also remove portion that referred to Sphinx builds, as these don't make
sense now.
Currently, it seems easier to upload the diagram images to git than to
try to make the image generation script work with out of the tree builds
and other corner cases.
This means, however, that one must activelly remember to update msc
diagram files, run the script and re-add new png files, if those ever
need to be updated. To raise awareness to that, a watermark was added
to the diagram images.
Also removed configuration steps that added mscgen as dependency
(locally and for workflow builds and readthedocs).
Moved devguide dir into userguide dir.
Since the devguide is now incorporated as the last chapter of the
userguide, removed build and configuration files from the devguide
dir, as these are no longer needed.
Task #4909
Issue: 4550
This commit documents the new per-thread stack-size setting. Some
systems have a small default value that is not suitable for Suricata's
multi-threaded architecture and adjustment may be required.
This explains the frame support from the perspective of a developer,
with introductory explanation on how to add frame support to an
applayer proto.
Doc #4697
While ReadTheDocs can't execute arbitrary scripts, we can install
mscgen in the container that builds the docs (see .readthedocs.yaml).
Then instead of calling generate-images.sh from the Makefile, move this
into the setup function defined on conf.yaml, which will generate the
scripts as part of a normal Sphinx workflow.
This should give us an image generation solution that works on
ReadTheDocs, and locally within anyones build system provided they have
mscgen installed.
- add nom parsers for decoding most messages from StartupPhase and
SimpleQuery subprotocols
- add unittests
- tests/fuzz: add pgsql to confyaml
Feature: #4241
Many places were still referencing the old Suricata page.
Used git grep with replace to update them. Checked that new links work.
Left old references when they were only documentation examples (for
output or unittests).
Task#4915
The differences on how the `need` key works, depending on script
usage (output or detection) confuses users, sometimes (cf doc#4725).
While we don't fix that, just explain this behavior.
Part of the task to offer better guidance on how and when to write
unit tests or suricata-verify tests
Also updated linking and index files, as well as testing page to refer
to the unit tests pages
Doc: #4590
Part of ongoing task to add more guidance on how to create unittests
and suricata-verify tests for suri. There will also be a unittests-rust
page.
Doc: #4590
This page offers guidance about when to use unittests or s-v tests,
and how to create input for those. Also lists other common ways to test
Suri, such as fuzzing and the CI checks.
Doc: #4590
Semantically speaking it makes more sense, because it stores `msc`
files for dynamic image generation.
Updated files that refered to `img` accordingly, too.
- DNS sequence diagram was incorrect (transactions should be
unidirectional). After changing it, it made sense to rename the file.
Adjusted spacing, too. Updated transactions.rst accordingly.
- TLS sequence diagram was refined to illustrate how Suricata actually
implements the protocol.
A guide on what is a transaction for Suricata engine, focusing on
developers.
- What's the purpose of a transaction;
- transaction states and API callbacks;
- Examples and sequence diagrams.
- doc/devguide: add transactions.rst
- doc/devguide/extending/app-layer/index.rst: add transactions.rst