Commit Graph

1757 Commits (master)

Author SHA1 Message Date
Jason Ish 1206c1c5af lua: convert dns function into suricata.dns lib
Notable changes from the previous API:
- rcode will return the rcode as an integer
- rcode_string will return the string representation

Also fixes an issue where an rcode of 0 was returned as nil.

Ticket: #7602
4 months ago
Alice Akaki 90aab0d62f detect: add email.from
email.from matches on MIME EMAIL FROM
This keyword maps to the EVE field email.from
It is a sticky buffer
Supports prefiltering

Ticket: #7592
4 months ago
Alice Akaki 90cf59ac71 rustfmt: rust/src/mime/smtp.rs 4 months ago
Jason Ish 870bf73380 dns: refactor function to get rrname to be safe
Make the function safe by returning a reference to the DNSName object,
the unsafe C wrapper can do the conversion to pointers.
5 months ago
Nathan Scrivens 07632fdf4e dns: add dns.response sticky buffer
Feature: 7012
Add dns.response sticky buffer to match on dns response fields.
Add rust functions to return dns response packet data.
Unit tests verifying signature matching.
5 months ago
Alice Akaki 137f7fe652 detect: add ldap.responses.message
ldap.responses.message matches on LDAPResult error message
This keyword maps the following eve fields:
ldap.responses[].bind_response.message
ldap.responses[].search_result_done.message
ldap.responses[].modify_response.message
ldap.responses[].add_response.message
ldap.responses[].del_response.message
ldap.responses[].mod_dn_response.message
ldap.responses[].compare_response.message
ldap.responses[].extended_response.message
It is a sticky buffer
Supports prefiltering

Ticket: #7532
5 months ago
Alice Akaki 84605db01d detect: add ldap.responses.result_code
ldap.responses.result_code matches on LDAP result code
This keyword maps the following eve fields:
ldap.responses[].bind_response.result_code
ldap.responses[].search_result_done.result_code
ldap.responses[].modify_response.result_code
ldap.responses[].add_response.result_code
ldap.responses[].del_response.result_code
ldap.responses[].mod_dn_response.result_code
ldap.responses[].compare_response.result_code
ldap.responses[].extended_response.result_code
It is an unsigned 32-bit integer
Doesn't support prefiltering

Ticket: #7532
5 months ago
Alice Akaki d827728661 ldap: create a generic funtion to match integer responses 5 months ago
Alice Akaki bfa3558cf0 ldap: refactor function aux_ldap_parse_protocol_resp_op
Split code to create a generic function that parses LdapIndex
5 months ago
Alice Akaki caffde9428 ldap: rename DetectLdapRespData struct to be more specific 5 months ago
Alice Akaki 599d33c5bf ldap: return empty buffer in ldap_tx_get_responses_dn
Funciton ldap_tx_get_responses_dn returns empty buffer in case
the response doesn't contain the distinguished name field

Fixes: 73ae6e997f ("detect: add ldap.responses.dn")
5 months ago
Alice Akaki 82ca3e667b ldap: fix LDAPDN nits
Change variable name 'req' to 'resp' in function ldap_tx_get_responses_dn and documentation nits

Fixes:
73ae6e997f ("detect: add ldap.responses.dn")
16dcee46fc ("detect: add ldap.request.dn")
5 months ago
Alice Akaki 3b6106e8f4 ldap: apply rustfmt
Fixes: 4554c4778d ("rust: use AppProto from generated bindings instead of duplicating")
5 months ago
Jason Ish 2e52e9501f mqtt: naming and visibility cleanups
- remove rs_prefix, replace with SC if needed
- remove pub and no_mangle where not needed
- remove some unused functions and fields

Related to ticket: #7498
5 months ago
Jason Ish 0fe11cdfe5 enip: remove rs_ prefix
Related to ticket: #7498
5 months ago
Jason Ish c726d67bb0 dcerpc: visibility and naming cleanups
- replace rs_ prefixed names with SC
- remove no_mangle and pub where not needed
- remove some unused functions

Related to ticket: #7498
5 months ago
Jason Ish f0116c3a6b bittorrent: no_mangle, pub and naming cleanups
- Remove rs_prefix
- Remove no_mangle and pub when not needed

Related to ticket: #7498
5 months ago
Jeff Lucovsky 9b088ed018 applayer/ftp: Misc cleanup
Issue: 4082

Small fixups.
5 months ago
Jeff Lucovsky b7d240fb14 applayer/ftp: Move MPM declaration
This commit moves the MPM fn declaration into core.rs making it
available for other Rust modules.

Issue: 4082
5 months ago
Jeff Lucovsky 4d0cf8a8fa app-layer/ftp: Fixup state values
Issue: 4082

Fixup the incorrect state values -- they should be the default enum
values to match the pre-Rust implementation.
5 months ago
Jason Ish c33bebd630 rust: remove allow of static mutables
As references to static mutables are highly discouraged, remove the
global suppressing of the compiler warning. Each use case can be
suppressed as needed.

Ticket: #7417
5 months ago
Jason Ish 4a2f10d235 smb: wrap read access to static mutables in function
Simply wrapping in a function removes static_mut_refs compiler
warning.

Ticket: #7417
5 months ago
Jason Ish d8c080b268 smtp: suppress static_mut_refs where needed
Allows us to get rid of the global supression.

Ticket: #7417
5 months ago
Jason Ish 9ed5b4c002 smtp: remove SCMimeSmtpConfigExtractUrlsSchemeReset
It doesn't appear to be needed. The vec being cleared is only set once
per run, so never needs to be cleared.

Removes one point where we have to supress the static_mut_refs compiler
warning.

Ticket: #7417
5 months ago
Jeff Lucovsky dd344bd07c ftp: Move config file handling to Rust
Issue: 4082

Move the configuration file handling to Rust.

These changes will no longer terminate Suricata when there's an invalid
value for ftp.memcap. Like earlier Suricata releases, an error message
is logged "Invalid value <value> for ftp.memcap" but Suricata will no
longer terminate execution. It will use a default value of "0" instead.
5 months ago
Jeff Lucovsky f0410c93d0 ftp: convert enumerations to Rust
As part of the effort to convert the FTP/FTPDATA parser to rust, move
the enums from C to rust.

Issue: 4082
5 months ago
Philippe Antoine 8c3bd3e8a0 dcerpc: consume bytes after gap resync
Ticket: 7567

After a gap, we search a new record that may start later than
the beginning of current stream slice.
If so, consume the first bytes before the start of the record,
so that AppLayerResult::incomplete can be consistent and not
trigger assertion !((res.needed + res.consumed < input_len))
5 months ago
Philippe Antoine 3bc2a14fbf rust: fix clippy 1.85 precedence warnings
warning: operator precedence can trip the unwary
   --> src/jsonbuilder.rs:781:36
    |
781 |                 buf[offset] = HEX[(x >> 4 & 0xf) as usize];
    |                                    ^^^^^^^^^^^^ help: consider parenthesizing your expression: `(x >> 4) & 0xf`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
    = note: `#[warn(clippy::precedence)]` on by default
5 months ago
Philippe Antoine 726de5520f quic: discard late retry packets
Ticket: 7556

See RFC 9000 section 17.2.5.2 :
After the client has received and processed an Initial
or Retry packet from the server,
it MUST discard any subsequent Retry packets that it receives.
5 months ago
Shivani Bhardwaj a551674eae dcerpc: add iface to dcerpc request event
so as to avoid extra steps for correlation among events to find
this information.

Feature 7565
5 months ago
Alice Akaki 73ae6e997f detect: add ldap.responses.dn
ldap.responses.dn matches on LDAPDN from responses operations
This keyword maps the following eve fields:
ldap.responses[].search_result_entry.base_object
ldap.responses[].bind_response.matched_dn
ldap.responses[].search_result_done.matched_dn
ldap.responses[].modify_response.matched_dn
ldap.responses[].add_response.matched_dn
ldap.responses[].del_response.matched_dn
ldap.responses[].mod_dn_response.matched_dn
ldap.responses[].compare_response.matched_dn
ldap.responses[].extended_response.matched_dn
It is a sticky buffer
Supports prefiltering

Ticket: #7471
5 months ago
Alice Akaki 16dcee46fc detect: add ldap.request.dn
ldap.request.dn matches on LDAPDN from request operations
This keyword maps the following eve fields:
ldap.request.bind_request.name
ldap.request.add_request.entry
ldap.request.search_request.base_object
ldap.request.modify_request.object
ldap.request.del_request.dn
ldap.request.mod_dn_request.entry
ldap.request.compare_request.entry
It is a sticky buffer
Supports prefiltering

Ticket: #7471
5 months ago
Alice Akaki 7b350e9933 misc: fix name prefix in detect register functions 5 months ago
Philippe Antoine d61f36c66f quic: decrypt only initial packets
Ticket: 7556

Avoids failed_decrypt events when the first packet seen is not
a Quic Initial packet
5 months ago
Philippe Antoine 6d8910d245 quic: handle retry packets
Ticket: 7556
5 months ago
Philippe Antoine f295cc059d quic: handle fragmented hello over multiple packets
Ticket: 7556

To do so, we need to add 2 buffers (one for each direction)
to the QuicState structure, so that on parsing the second packet
with hello/crypto fragment, we still have the data of the first
hello/crypto fragment.

Use a hardcoded limit so that these buffers cannot grow indefinitely
and set an event when reaching the limit
5 months ago
Philippe Antoine 68adc87bd2 quic: parse ack frame number 3
cf rfc9000 section 19.3. ACK Frames

Ticket: 7556
5 months ago
Philippe Antoine ee04d667b5 quic: move all_consuming check to callee
Will alow to have decode_frames accept one additional parameter
with past fragment data
5 months ago
Philippe Antoine 294a33a890 quic: rustfmt 5 months ago
Juliana Fajardini ae00181736 pgsql/parser: fix type complexity clippy warning
Cf https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity
5 months ago
Juliana Fajardini a4ce9cfe15 pgsql/logger: don't log empty transactions
This may happen in some situations if the app-layer parser only sees
unknown messages and sets an event: there will be an empty transaction,
but nothing to log.

Related to
Task #5566
5 months ago
Juliana Fajardini d676d711ee pgsql: add case for Unknown response message state
No state change, but since we added Unknown responses, we should handle
that case -- should we have a specific state for such cases?

Related to
Bug #5524
Task #5566
5 months ago
Juliana Fajardini 1b6f4da23b pgsql: add events
Events for:
- parsing error when parsing pgsql packet length
- parsing error for pgsql requests (post length parsing)
- parsing error for pgsql responses (post length parsing)
- too many transactions

Include `pgsql-events.rules` file, and PGSQL events SID range definition

Task #5566
5 months ago
Juliana Fajardini 6eadb752ec pgsql/parser: add more debug statements 5 months ago
Juliana Fajardini 38f8508bf5 pgsql: apply rustfmt changes 5 months ago
Juliana Fajardini ff8d4e972c pgsql: don't always return error for parsing errors
This allows the app-proto to continue onto parsing next PDUs, if
possible.

Bug #5524
5 months ago
Juliana Fajardini 4fed424d74 pgsql: don't fail if proto parses unknown message
Even if unknown, if the message is properly parsed, allow the parser to
proceed.

Related to
Bug #5524
5 months ago
Juliana Fajardini cc841e66db pgsql/parser: always use fn for parsing PDU length
Some inner parsers were using it, some weren't. Better to standardize
this. Also take the time to avoid magic numbers for representing the
expected lengths for pgsql PDUs.
Also throwing PgsqlParseError and allowing for incomplete results.

Related to
Task #5566
Bug #5524
5 months ago
Juliana Fajardini 29d3aa7a6a pgsql: add PgsqlParserError
Building on top of work done by Jason Ish.

Related to
Bug #5524
5 months ago
Juliana Fajardini 737fea751f pgsql/parser: simplify response parsing
The initial parsing for message type checking was more complex than
needed be.

Related to
Bug #5524
5 months ago
Juliana Fajardini 24375a1fef pgsql/parser: fix response message length
Some backend messages can be the shortest pgsql length possible,
4 bytes, but the parser expectd all messages to be longer than that.

Related to
Bug #5524
5 months ago
Philippe Antoine b514b25165 detect/flow: move keyword parsing code to rust
for flow.pkts and flow.bytes keywords

Ticket: 7562

Avoid null deref when parsing flow.bytes:toserver;
5 months ago
Philippe Antoine 8ae5665767 detect/krb5: avoid integer underflow with krb5.ticket_encryption
Ticket: 7560

When passing INT32_MIN aka 0x80000000, we cannot compute -vali
as it does not fit into a i32
5 months ago
Jason Ish 4554c4778d rust: use AppProto from generated bindings instead of duplicating
Have bindgen generate bindings for app-layer-protos.h, then use the
generated definitions of AppProto/AppProtoEnum instead if defining
them ourselves.

This header was chosen as its used by Rust, and its a simple header
with no circular dependencies.

Ticket: #7341
5 months ago
Shivani Bhardwaj bc79300dfa datasets: move initial file reading to rust
In a recent warning reported by scan-build, datasets were found to be
using a blocking call in a critical section.

datasets.c:187:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
  187 |     while (fgets(line, (int)sizeof(line), fp) != NULL) {
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:292:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
  292 |     while (fgets(line, (int)sizeof(line), fp) != NULL) {
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:368:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
  368 |     while (fgets(line, (int)sizeof(line), fp) != NULL) {
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:442:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
  442 |     while (fgets(line, (int)sizeof(line), fp) != NULL) {
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:512:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
  512 |     while (fgets(line, (int)sizeof(line), fp) != NULL) {
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5 warnings generated.

These calls are blocking in the multi tenant mode where several tenants
may be trying to load the same dataset in parallel.
In a single tenant mode, this operation is performed as a part of a
single thread before the engine startup.

In order to evade the warning and simplify the code, the initial file
reading is moved to Rust with this commit with a much simpler handling
of dataset and datarep.

Bug 7398
5 months ago
Shivani Bhardwaj 015da2c7ed rust: add C callback for FatalErrorOnInit 5 months ago
Shivani Bhardwaj 0ce8b15ca8 rust: add macro to return val if unwrap fails 5 months ago
Jason Ish 7a936cb486 base64: expose no padding and padding optional variants
A no padding option is provided as a mode, as its a variant suitable
for encoding and decoding.

A padding optional function is added that is indifferent to padding
when decoding. This can be useful when you're not sure if padding
exists, and don't really care.
5 months ago
Jason Ish ad6d9e0d93 base64: prefix base64 C API with "SC" 5 months ago
Alice Akaki 73455179d7 detect/integers: add support for negated strings when enum is used
function detect_parse_uint_enum can parse strings like !bind_request

Ticket: #7513
5 months ago
Philippe Antoine ef044b208c dcerpc: prevent integer underflow
in case a fragment has a length lesser than DCERPC_HDR_LEN

Fixes: 9daf8528b7 ("dcerpc: tidy up code")

Ticket: 7548
5 months ago
Ilya Bakhtin 27f0db7526 protodetect/dcerpc: improve DCERPC UDP probing parser
Several additional checks are added to the probing parser to avoid false
detection of DNS as DCERPC

Ticket - 7111
6 months ago
Shivani Bhardwaj c73299a298 dcerpc/tcp: add frames support
Frames of the following types have been added for toserver direction:
1. Pdu: The entire Protocol Data Unit
2. Hdr: Header of the request
3. Data: PDU data

Feature 4904
6 months ago
Shivani Bhardwaj 9daf8528b7 dcerpc: tidy up code
- remove unneeded variables
- remove unnecessary tracking of bytes in state
- modify calculations as indicated by failing tests
6 months ago
Shivani Bhardwaj 4790da1825 dcerpc: remove fragmented data tests
With the introduction of AppLayerResult::incomplete API, fragmented data
is no longer handled fully in the dcerpc code. Given that these code
paths are already covered by the following s-v tests, these tests can now be
safely removed.
- dce-gap-handling
- dcerpc-dce-iface-*

Ticket 5699
6 months ago
Shivani Bhardwaj 74de1042a9 dcerpc: use AppLayerResult::incomplete API
Instead of own internal mechanism of buffering in case of fragmented
data, use AppLayerResult::incomplete API to let the AppLayer Parser take
care of it. This makes the memory use more efficient.
Remove any unneeded variables and code with the introduction of this
API.

Ticket 5699
6 months ago
Shivani Bhardwaj fc88e61c7f dcerpc: save version info in tx
to make it available for logging.
6 months ago
Shivani Bhardwaj 0d6017d174 dcerpc: do not assume an upper bound on data
TCP data can be presented to the protocol parser in any way e.g. one
byte at a time, single complete PDU, fragmented PDU, multiple PDUs at
once. A limit of 1MB can be easily reached in some of such scenarios.
Remove the check that rejects data that is more than 1MB.
6 months ago
Jason Ish 99096699c0 rust/logging: fix logging from plugins
Commit 2bcc66da58 broke logging from
plugins:

- debug visibility was reduced making it unusable from an external crate
- the plugins view of the log level was broken

To fix:
- make debug pub
- minor change to initialization of the log LEVEL as seen by the plugin
  so its seen by the plugin. I'm not really sure why the previous
  version wasn't working though, but this one does
6 months ago
Jason Ish c4fc2aac3b rust/hashing: add more hex variants
Make all the hasher's have the same variants:
- add hex digest for sha256
- add finalize to hex for sha1
- add hex digest for sha1
6 months ago
Philippe Antoine b11381d952 rust: AppLayerRegisterParser out of SuricataContext
Just use a regular compile time rust export, instead of having
a runtime definition through the SuricataContext structure
6 months ago
Alice Akaki 8416289752 detect: add ldap.responses.count
ldap.responses.count matches on the number of LDAP responses
This keyword maps to the eve field len(ldap.responses[])
It is an unsigned 32-bit integer
Doesn't support prefiltering

Ticket: #7453
6 months ago
Alice Akaki da593abd99 detect: add ldap.responses.operation
ldap.responses.operation matches on Lightweight Directory Access Protocol response operations
This keyword maps to the eve field ldap.responses[].operation
It is an unsigned 8-bit integer
Doesn't support prefiltering

Ticket: #7453
6 months ago
Alice Akaki cdb043810f detect: add ldap.request.operation
ldap.request.operation matches on Lightweight Directory Access Protocol request operations
This keyword maps to the eve field ldap.request.operation
It is an unsigned 8-bit integer
Doesn't support prefiltering

Ticket: #7453
6 months ago
Alice Akaki 824c5d4f06 rustfmt: rust/src/ldap/types.rs 6 months ago
Jason Ish 5d9fad5edf rust/ssh: rustfmt 6 months ago
Jason Ish 9c87ac1b9b ssh: cleanup rust visibility and naming
Remove pub and no_mangle from non-exported functions and rename to
Rust as-needed.

Ticket: 7498
6 months ago
Jason Ish f8d27992f7 rust/ntp: rustfmt 6 months ago
Jason Ish f023b3a6f1 ntp: cleanup visibility and naming
Remove pub and no_mangle from non-exported functions and rename to
Rust as-needed.

Ticket: 7498
6 months ago
Jason Ish 4cca135793 pgsql: cleanup visibility and naming
Remove pub and no_mangle from non-exported functions and rename to
Rust style as needed.

Ticket: 7498
6 months ago
Jason Ish a7f7dc5296 rust/dhcp: rustfmt 6 months ago
Jason Ish c5089ac5f4 dhcp: cleanup visibility and naming
- remove "rs_" prefix from functions that are not exported
- prefix exported functions with "SC"
- don't export functions that are only used by pointer

Ticket: 7498
6 months ago
Jason Ish c86b2608aa rust/applayer: clean visibility of export macros
Both the macros export_tx_data_get and export_state_data_get can
generate non-pub functions as the function they generate is only used
as a pointer during registration.

Remove "pub" and "no_mangle" from the generated functions and update
the names of the generated functions to follow Rust rules as they are
no longer exported into the global C namespace.

Ticket: 7498
6 months ago
Jason Ish c93bba8e2c rust/transforms: remove no_mangle from non-pub functions
Ticket: 7498
6 months ago
Jason Ish f370afb5bd rust/ldap: cleanup no_mangle and extern C pub functions
If a function is extern "C" and not pub, remove no_mangle so its not
exposed as-is into the global name space.

Ticket: 7498
6 months ago
Jason Ish c207cfc67e rust/detect: remove nested unsafe around keyword registration
Just a cleanup, I'm not sure why Rust wasn't warning on these
nested/unneeded unsafe's.
6 months ago
Jason Ish 2bcc66da58 rust/log: move rust log and debug utils to debug module
Move Rust logging, and debug_validation to a debug module to mirrow
the C side.
6 months ago
Jason Ish f62be374ea rust/direction: move direction to own file (cleanup)
Move the implementation of Direction to its own file, direction.rs.
6 months ago
Jason Ish 7ef4caf90e rust/flow: move flow support to its own file (cleanup)
Move the Rust Flow support from core.rs to flow.rs.
6 months ago
Philippe Antoine 8f6795d049 rust: remove unneeded mut for tx in detect
Detection does not change transactions, it reads only.
6 months ago
Philippe Antoine af4e1e8888 rust: remove unneeded mut for tx in loggers
Loggers do not change transactions, they read only.
6 months ago
Philippe Antoine edfe0ccbe1 ldap: implement abandon request
Ticket: #7477
6 months ago
Alice Akaki b1c2643c87 detect: add vlan.id keyword
vlan.id matches on Virtual Local Area Network IDs
It is an unsigned 16-bit integer
Valid range = [0-4095]
Supports prefiltering

Ticket: #1065
6 months ago
Philippe Antoine e6be049c5d app-layer: move ALPROTO_FAILED definition
Because some alprotos will remain static and defined as a constant,
such as ALPROTO_UNKNOWN=0, or ALPROTO_FAILED.

The regular already used protocols keep for now their static
identifier such as ALPROTO_SNMP, but this could be made more
dynamic in a later commit.

ALPROTO_FAILED was used in comparison and these needed to change to use
either ALPROTO_MAX or use standard function AppProtoIsValid
6 months ago
Philippe Antoine 6291e220e4 dns: improved handling of corrupt additionals
Ticket: 7228

That means log the rest of queries and answers, even if the
final field additionals is corrupt.
Set an event in this case.
6 months ago
Philippe Antoine 494d7bfe99 ldap: truly enforce max-tx
Ticket: 7465

If a bug chunk of data is parsed in one go, we could create many
transactions even if marking them as complete, and have
quadratic complexity calling find_request.

Proposed solution is to fail on creating a new transaction if too
many already exist.
6 months ago
Philippe Antoine fd7e2e5e9a doh2: really enforce 65K dns message limit
Ticket: #7464
7 months ago
Jason Ish 604e3c7422 rust/jsonbuilder: make set_uint generic over Into<u64>
Allow `set_uint` to accept any number value that can be converted to a
u64. Prevents callers from having to do `as u64`.

This required fixing up any callers that used `.into()` to just pass in
their value without the into conversion.

Most calls using `as u64` can have that cast removed, with the exception
of `usize` values which must still be cast is conversion can't be
guaranteed to be non-fallible.
7 months ago
Philippe Antoine 5da2b5d7a4 dns/probe: check that a request has at least one query
Ticket: 7279
7 months ago
Philippe Antoine 9b40446bea dns/probe: adds check for 0 records and big size
Ticket: 7279

Make dns probing function stricter to avoid matching on non-DNS
on port 53 and later returning a app-layer error.
7 months ago
Shivani Bhardwaj 2c0d3b83c4 dns: fix clippy lint warnings
Fix vector lint issues:
- same_item_push
- vec_init_then_push
7 months ago
Jason Ish 19cf0f8133 dns: provide events for recoverable parse errors
Add events for the following resource name parsing issues:

- name truncated as its too long
- maximum number of labels reached
- infinite loop

Currently these events are only registered when encountered, but
recoverable. That is where we are able to return some of the name,
usually in a truncated state.

As name parsing has many code paths, we pass in a pointer to a flag
field that can be updated by the name parser, this is done in
addition to the flags being set on a specific name as when logging we
want to designate which fields are truncated, etc. But for alerts, we
just care that something happened during the parse. It also reduces
errors as it won't be forgotten to check for the flags and set the
event if some new parser is written that also parses names.

Ticket: #7280
7 months ago
Jason Ish 37f4c52b22 eve/dns: add truncation flags for fields that are truncated
If rrname, rdata or mname are truncated, set a flag field like
'rrname_truncated: true' to indicate that the name is truncated.

Ticket: #7280
7 months ago
Jason Ish 3a5671739f dns: truncate names larger than 1025 characters
Once a name has gone over 1025 chars it will be truncated to 1025
chars and no more labels will be added to it, however the name will
continue to be parsed up to the label limit in attempt to find the end
so parsing can continue.

This introduces a new struct, DNSName which contains the name and any
flags which indicate any name parsing errors which should not error
out parsing the complete message, for example, infinite recursion
after some labels are parsed can continue, or truncation of name where
compression was used so we know the start of the next data to be
parsed.

This limits the logged DNS messages from being over our maximum size
of 10Mb in the case of really long names.

Ticket: #7280
7 months ago
Philippe Antoine f426ee3ee2 detect: rename stream_log variables
to better reflect their true meaning
7 months ago
Philippe Antoine b02557ac7d app-layer: track modified/processed txs
To optimize detection, and logging, to avoid going through
all the live transactions when only a few were modified.

Two boolean fields are added to the tx data: updated_tc and ts
The app-layer parsers are now responsible to set these when
needed, and the logging and detection uses them to skip
transactions that were not updated.

There may some more optimization remaining by when we set
both updated_tc and updated_ts in functions returning
a mutable transaction, by checking if all the callers
are called in one direction only (request or response)

Ticket: 7087
7 months ago
Philippe Antoine e62c7d733b rust/ftp: handle NULL inputs
Completes Ticket 7013
7 months ago
Philippe Antoine 38d7900fa9 sip: remove UPDATE method for detection
As it is also used for HTTP/1
Remove it only for TCP and keep it for UDP.

Ticket: 7436
7 months ago
Philippe Antoine b58b886db7 detect/transforms: in place modifications of buffers
As is the case when chaining multiple transforms.
Avoids using memcpy in these cases.

Add tests for these cases.

Ticket: 7409
8 months ago
Jason Ish 289ff25f5b requires: support requires check for keyword
For example:

    requires: keyword foo;

Will require that Suricata supports the "foo" keyword.

Ticket: #7403
8 months ago
Jason Ish 820a3e51b7 requires: treat unknown requires keywords as unmet requirements
For example, "requires: foo bar" is an unknown requirement, however
its not tracked, nor an error as it follows the syntax. Instead,
record these unknown keywords, and fail the requirements check if any
are present.

A future version of Suricata may have new requires keywords, for
example a check for keywords.

Ticket: #7418
8 months ago
Jason Ish 4c12165816 rust: allow static_mut_refs for now
But we should fix all these soon.
8 months ago
Jason Ish aa6e94fc73 rust/smb: fix rustdoc line
'///' style rust comments/documentation come before the item being
documented.

Spotted by clippy.
8 months ago
Jason Ish 7bdbe7ed32 rust: remove unnecessary lifetimes
Fix provided by cargo clippy --fix.
8 months ago
Pierre Chifflier bd7d38e91e ldap: add support for STARTTLS to make certificate information available
Ticket: #7394.
8 months ago
Pierre Chifflier 3341907125 ldap: add port 3268 (used by Active Directory) 8 months ago
Philippe Antoine c26cb9abd3 detect/transform: fix leak in xor transform parse
Fixes: 8984bc6801 ("transforms: move xor to rust")
8 months ago
Victor Julien 543429f176 smb: use lru for ssn2vec_map
Generic ssn2vec_map was a HashMap used for mapping session key to
different types of vector data:
- GUID
- filename
- share name

Turn this into a bounded LruCache. Rename to ssn2vec_cache.

Size of the cache is 512 by default, and can be configured using:

`app-layer.protocols.smb.max-session-cache-size`

Ticket: #5672.
8 months ago
Victor Julien ba7a4ece76 smb: use lru for ssnguid2vec_map; rename
Reimplement the ssnguid2vec_map HashMap as a LruCache.

Since this is a DCERPC record cache, name it as such.

Default size is 128. Can be controlled by
`app-layer.protocols.smb.max-dcerpc-frag-cache-size`.

Ticket: #5672.
8 months ago
Victor Julien 0f23557ef7 smb: use lru for ssn2tree; rename
Turn the map mapping the smb session key to smb tree into a lru cache,
limited to 1024 by default.

Add `app-layer.protocols.smb.max-tree-cache-size` option to control the
limit.

Ticket: #5672.
8 months ago
Victor Julien ce44d38ca8 smb: use lru for ssn2vecoffset_map; rename
Rename to read_offset_cache.

Add `app-layer.protocols.smb.max-read-offset-cache-size` option to
control the limit.

Ticket: #5672.
8 months ago
Victor Julien 91828ec00b smb: use lru for guid2name map; rename
Use `lru` crate. Rename to reflect this.

Add `app-layer.protocols.smb.max-guid-cache-size` to control the max
size of the LRU cache.

Ticket: #5672.
8 months ago
Victor Julien 23f2317c6a smb1: remove name on close
Ticket: #5672.
8 months ago
Victor Julien 85987aaad6 smb2: remove filename on close
Ticket: #5672.
8 months ago
Victor Julien c2124f0b9e smb: update to GAP handling
Don't tag the session as gap'd when the GAP is in a precise location:

1. in "skip" data, where the GAP just fits the skip data

2. in file data, where we pass the GAP on to the file

This reduces load of GAP post-processing that is unnecessary in these
case.
8 months ago
Victor Julien d535d7f768 smb2: use if let for read/write parsing 8 months ago
Philippe Antoine 63324b7368 transforms: move urldecode to rust
Ticket: 7229
8 months ago
Philippe Antoine 8984bc6801 transforms: move xor to rust
Ticket: 7229
8 months ago
Philippe Antoine 45e0acf772 transforms: move http headers transforms to rust
Ticket: 7229
8 months ago
Philippe Antoine f0414570d2 transforms: move casechange to rust
Ticket: 7229
8 months ago
Philippe Antoine 0e5b49d20f transforms: move hash transforms to rust
md5, sha1 and sha256

Ticket: 7229
8 months ago
Philippe Antoine 71da38e702 transforms: move dotprefix to rust
Ticket: 7229
8 months ago
Philippe Antoine 966f659201 transforms: move compress_whitespace to rust
Ticket: 7229
8 months ago
Philippe Antoine 4985ebc0e0 transforms: move strip_whitespace to rust
Ticket: 7229
8 months ago
Jason Ish b1c26dccf3 http2: rename event variant to match rule
Rename InvalidHTTP1Settings to InvalidHttp1Settings so it gets the
expected name transformation of "invalid_http1_settings".

Ticket: #7361
9 months ago
Sascha Steinbiss 5d8252117f mqtt: double-check detection directions
Ticket: #7323
9 months ago
Sascha Steinbiss 377d4705e1 mqtt: add reason code support for SUBACK
Ticket: #7323
9 months ago
Jason Ish ab6dcb3fc1 app-layer: use uint8_t consistent for event IDs
Introduce a common function for mapping names to IDs that performs
bounds checking.

Note: For event IDs in the enum that are larger than a uint8_t, -1
will be returned instead of -4. -4 has special meaning during
signature parsin that means requirements were not met. -4 has no
special handling prior to requirements, or the meaning has been lost.
9 months ago
Alice Akaki 5e563b0c8b detect/analyzer: add more details for flow_age
Ticket: #6312
9 months ago
Jason Ish 45384ef969 rust/applayer: use c_int as return type for get_info_by_id
Rust was using i8 as the return type, while C uses int. As of Rust
1.82, the return value is turned to garbage over the FFI boundary.

Ticket: #7338
9 months ago
Philippe Antoine b5fcf16d97 enip/detect: remove double registration of enip_command keyword
As found with
./src/suricata --list-keywords | sort | uniq -c | awk '$1 > 1'
9 months ago
Philippe Antoine 2fb89f1b79 ldap: improve some rust style 9 months ago
Philippe Antoine 7d854bd97f template/detect: allow empty buffers 9 months ago
Philippe Antoine 96c8470cdd template: move detect keywords to pure rust
Ticket: 3195

Also remove unused src/tests/detect-template-buffer.c

Completes commit 4a7567b3f0
to remove references to template-rust
9 months ago
Philippe Antoine 1e152d1f10 ja4: handles non alphanumeric alpn
Ticket: 7267

Follows more closely the specification :
https://github.com/FoxIO-LLC/ja4/blob/main/technical_details/JA4.md#alpn-extension-value

Also fixes the case with a single-char alpn.
10 months ago
Philippe Antoine 521928e2a3 sip: use pure rust function
For internal function that is not part of C FFI
10 months ago
Shivani Bhardwaj cbb571a61c rust/base64: add decoder
Add a pure rust base64 decoder. This supports 3 modes of operation just
like the C decoder as follows.
1. RFC 2045
2. RFC 4648
3. Strict

One notable change is that "strict" mode is carried out by the rust
base64 crate instead of native Rust. This crate was already used for
encoding in a few places like datasets of string type. As a part of this
mode, now, only the strings that can be reliably converted back are
decoded.

The decoder fn is available to C via FFI.

Bug 6280
Ticket 7065
Ticket 7058
10 months ago
Giuseppe Longo d43154adbe sip: add sip.content_length sticky buffer
This adds a sticky (multi) buffer to match the "Content-Length" header field in
both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo 6f727ab970 sip: add sip.content_type sticky buffer
This adds a sticky (multi) buffer to match the "Content-Type" header field in
both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo 5e8e61bb3b sip: add sip.user_agent sticky buffer
This adds a sticky (multi) buffer to match the "User-Agent" header field in
both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo 6ddc7d6223 sip: add sip.via sticky buffer
This adds a sticky (multi) buffer to match the "Via" header field in
both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo c205e87911 sip: add sip.to sticky buffer
This adds a sticky (multi) buffer to match the 'To' header field in
both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo e3fefcf55c sip: add sip.from sticky buffer
This adds a sticky (multi) buffer to match the "From" header field
in both requests and responses.

Ticket #6374
10 months ago
Giuseppe Longo fbc3cd1b3f rust/sip: store response headers
To match on response SIP headers, those headers must be stored.

Ticket #6374
10 months ago
Giuseppe Longo cfb793ce28 rust/sip: store multiple header values
According to RFC 3261, a single header can be repeated one or more times,
and its name can also be specified using the 'compact form.'

This patch updates the hashmap used for storing headers to accommodate multiple
values instead of just one.

Additionally, if a header name is defined in the compact form, it is expanded
into its long form (i.e., the standard name).

This conversion simplifies the logic for matching a given header
and ensures 1:1 parity with keywords.

Ticket #6374
10 months ago
Giuseppe Longo 969f4d131f sip: rustify sticky buffers
Ticket #7204
10 months ago
Juliana Fajardini 2b1ad81cf5 pgsql: trigger raw stream reassembly at tx completion
Once we are tracking tx progress per-direction for PGSQL, we can trigger
the raw stream reassembly, for detection purposes, as soon as the
transactions are completed in the given direction.

Task #7000
10 months ago
Juliana Fajardini dcccbb1196 pgsql: track transaction progress per direction
PGSQL's current implementation tracks the transaction progress without
taking into consideration flow direction, and also has indirections
that make it harder to understand how the progress is tracked, as well
as when a request or response is actually complete.

This patch introduces tracking such progress per direction and adds
completion status per direction, too. This will help when triggering
raw stream reassembly or for unidirectional transactions, and may be
useful when we implement sub-protocols that can have multiple requests
per transaction, as well.

CancelRequests and TerminationRequests are examples of unidirectional
transactions. There won't be any responses to those requests, so we can
also mark the response side as done, and set their transactions as
completed.

Bug #7113
10 months ago
Juliana Fajardini 2c7824a41f pgsql: use new API style for extern C functions 10 months ago
Juliana Fajardini 3ba179422d pgsql: order StateProgress enum per direction
Related to
Bug #7113
10 months ago
Juliana Fajardini 7aeb718dd7 pgsql: apply rust fmt changes 10 months ago
Philippe Antoine de9413c654 detect: safety for app-layer logging of stream-only rules
If a stream-only rule matches, and we find a tx where we
want to log the app-layer data, store into the tx data that
we already logged, so that we do not log again the app-layer metadata

Ticket: 7085
10 months ago
Shivani Bhardwaj fbb97c51e4 dcerpc: return error on invalid header
DCERPC/TCP tends to return the same values for invalid and incomplete
headers. As a result of this, invalid headers and any traffic following
it is buffered and processed later on assumed to be valid DCERPC traffic.
Fix this by clearly defining error and incomplete data and taking
appropriate actions.

Bug 7230
10 months ago
Philippe Antoine dc3c048b49 rust/detect: fix too_long_first_doc_paragraph clippy warning
warning: first doc comment paragraph is too long
  --> src/detect/iprep.rs:57:1
   |
57 | / /// value matching is done use `DetectUintData` logic.
58 | | /// isset matching is done using special `DetectUintData` value ">= 0"
59 | | /// isnotset matching bypasses `DetectUintData` and is handled directly
60 | | /// in the match function (in C).
   | |_
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_long_first_doc_paragraph
   = note: `#[warn(clippy::too_long_first_doc_paragraph)]` on by default
help: add an empty line
10 months ago
Philippe Antoine 2a984e3b13 rust/dcerpc: fix single_match clippy warning
warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
  --> src/dcerpc/log.rs:36:33
   |
36 |               DCERPC_TYPE_BIND => match &state.bind {
   |  _________________________________^
37 | |                 Some(bind) => {
38 | |                     jsb.open_array("interfaces")?;
39 | |                     for uuid in &bind.uuid_list {
...  |
51 | |                 None => {}
52 | |             },
   | |_____________^
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_match
   = note: `#[warn(clippy::single_match)]` on by default
10 months ago
Philippe Antoine 0ebb84538e http2: add frames support
Ticket: 5743

Why ? To add detection capabilities
10 months ago
Victor Julien 65392c02f5 dcerpc: don't reuse completed tx
In the DCERPC over TCP pcap, logging and rule matching is disrupted by adding a simple rule:

        alert tcp any any -> any any (flow:to_server,established; \
                dce_iface:5d2b62aa-ee0a-4a95-91ae-b064fdb471fc; dce_opnum:1; \
                dce_stub_data; content:"|42 77 4E 6F 64 65 49 50 2E 65 78 65 20|"; \
                content:!"|00|"; within:100; distance:97; sid:1; rev:1; )

Works: alert + 3 dcerpc records.

But when adding a trivial rule:

        alert tcp any any -> any any (flow:to_server,established; \
                dce_iface:5d2b62aa-ee0a-4a95-91ae-b064fdb471fc; dce_opnum:1; \
                dce_stub_data; content:"|42 77 4E 6F 64 65 49 50 2E 65 78 65 20|"; \
                content:!"|00|"; within:100; distance:97; sid:1; rev:1; )
        alert tcp any any -> any any (dsize:3; sid:2; rev:1; )

The alert for sid:1 disappears and also there is one dcerpc event less.

In the single rule case we can aggressively free the transactions, as there
is only an sgh in the toserver direction.

This means that when we encounter the 2nd REQUEST, the first 2 transactions
have already been processed and freed. So for the 2nd REQUEST we open a new
TX and run inspection and logging on it.

When the 2nd rule is added, it adds toclient sgh as well. This means that we
will now slightly delay the freeing of the transactions.

As a consequence we still have the TX for the first REQUEST when the 2nd REQUEST
is parsed. This leads to the 2nd REQUEST re-using the TX. Since the TX is
already marked as inspected, it means the toserver rule now no longer matches.
Also we're not logging this TX correctly now.

This commit fixes the issue by not "finding" a TX that as already been
marked complete in the search direction.

Bug #7187.
10 months ago
Shivani Bhardwaj e93743a094 rust/base64: upgrade crate to latest
base64 crate is updated to the latest version 0.22.1. This came with
several API changes which are applied to the code. The old calls have
been replaced with the newer calls.

This was done following the availability of better fns to directly
decode into slices/vectors as needed and also that previous version was
too old.
Along with this change, update the Cargo.lock.in to reflect all changes
in the package versions.

Task 7219
10 months ago
Jason Ish 080681aff5 pgsql: don't expose PgsqlTransactionState to C
PgsqlTransactionState has a variant named "Init" which is a little too
generic to export to C. Fortunately this method doesn't need to be
exposed to C, instead remove it as it was only called by
rs_pgsql_tx_get_alstate_progress which also doesn't need to be public
or expose to C.

Ticket: #7227
11 months ago
Philippe Antoine 304271e63a rust: compatibility with cbindgen 0.27
Ticket: 7206

Cbindgen 0.27 now handles extern blocks as extern "C" blocks.
The way to differentiate them is to use a special comment
before the block.
11 months ago
Giuseppe Longo 564a6c9a20 rust/ldap: handle GAPs
Following the same logic as for PGSQL, if there is a gap in an LDAP request or
response, the parser tries to sync up again by checking if the message can be
parsed and effectively parses it on the next call.

Ticket #7176
12 months ago
Giuseppe Longo 6a606ff21e rust/ldap: add pdu frames
This adds a pdu frame for both request and response, and removes invalid
returns in SCLdapParseRequest and SCLdapParseResponse.

Ticket #7202
12 months ago
Giuseppe Longo edf70276d6 rust/ldap: enable parser for udp
This introduces a new parser registration function for LDAP/UDP, and update
ldap configuration in order to be able to enable/disable a single parser
independently (such as dns).
Also, GAPs are accepted only for TCP parser and not for UDP.

Ticket #7203
12 months ago
Philippe Antoine ede77bc4db rfb: move app-layer registration code to rust
Ticket: 7178
12 months ago
Philippe Antoine 62a186ceef detect/rfb: move keywords to rust
Ticket: 7178

On the way, convert rfb.secresult to a generic integer with enumeration
cf ticket 6723
12 months ago
Philippe Antoine a673e1913b ssh/frames: avoid unsigned integer overflow
Fixes: 0b2ed97f36 ("ssh: frames support")
12 months ago
Philippe Antoine 42e5e556e5 rust/ike: fix collapsible_match clippy warning
warning: this `match` can be collapsed into the outer `match`
help: the outer pattern can be modified to include the inner pattern
12 months ago
Philippe Antoine 564f685eea rust: fix byte_char_slices clippy warnings
warning: can be more succinctly written as a byte str
   --> src/mime/smtp.rs:762:37
    |
762 |     mime_smtp_find_url_strings(ctx, &[b'\n']);
    |                                     ^^^^^^^^ help: try: `b"\n"`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#byte_char_slices
    = note: `#[warn(clippy::byte_char_slices)]` on by default
12 months ago
Philippe Antoine 089d2b11fd frames: remove unneeded comments
Used by documentation with the SIP frames only
12 months ago
Philippe Antoine ef42f835eb ssh: avoid panic in packet path
use debug_validate_bug_on instead
12 months ago
Philippe Antoine 0b2ed97f36 ssh: frames support
Ticket: 5734

Adds frames for SSH records, that come after banner, and before
the data is encrypted.
These records may contain cipher lists for instance.
12 months ago
Victor Julien a0bf282963 rust: address clippy errors 12 months ago
Victor Julien 5bda7b5017 ssh/hassh: fix clippy warning 12 months ago
Philippe Antoine 7617fe5ab0 ldap: reset tx_index_completed on tx removal
So, that this index does not overflow
12 months ago
Philippe Antoine 7f6c963ac4 doh2: log like dns v3 1 year ago
Philippe Antoine 8aa2964e73 doh: move fields into dedicated Optional struct
So as to consume less memory for HTTP2Transaction
1 year ago
Philippe Antoine 6e12475f48 doh2: handle dns message in POST requests
Ticket: 5773

Handles both directions the same way for data if content type is
application/dns-message
1 year ago
Philippe Antoine 0ccad8fd88 doh: make dns and http keywords for doh2
Ticket: 5773
1 year ago
Philippe Antoine 1e82e20c65 doh: implement dns over http2 app-proto
Ticket: 5773
1 year ago
Philippe Antoine 46d98ae81c http2: log dns if DoH is recognized
Ticket: 5773
1 year ago
Philippe Antoine 29d9dc2729 http2: rustfmt 1 year ago
Philippe Antoine b5f55b5b1f dns: prepare for dns over http2 support
by making tx parsing and creation more easily available,
without needing a dns state.

Dns event NotResponse is now set on the right tx, and not the one
before.

Also debug log for Z-flag on request says "request" instead of
"response"

Also rustfmt dns.rs
1 year ago
Giuseppe Longo 910a5b226c rust/ldap: implement logger 1 year ago
Giuseppe Longo 93da339975 rust/ldap: implement app-layer 1 year ago
Giuseppe Longo ce7e190501 rust/ldap: implement types and filters
This implementation adds types and filters specified in the LDAP RFC to
work with the ldap_parser.
Although using the parser directly would be
best, strange behavior has been observed during transaction logging.
It appears that C pointers are being overwritten, leading to incorrect
output when LDAP fields are logged.
1 year ago
Philippe Antoine cc3dde8ada smtp: adds server side detection
Ticket: #1125
1 year ago
Philippe Antoine 0a1062fad2 detect/mqtt: move keywords to rust
Ticket: 4863

On the way, convert some keywords to use the first-class integer
support.
And helpers for pure rust the support for multi-buffer.

Move the C unit tests about keyword mqtt.protocol_version
to unit tests for generic integer parsing, and test version 5
instead of testing twice version 3.

Also iterate all tx's messages for reason code as is done for other
keywords.

And allow detection on empty topics.
1 year ago
Philippe Antoine ad08309c75 mqtt: parse and store raw connect flags
for easier later matching
1 year ago
Jason Ish fcc1b1067b eve/dns: make version required
The "eve.version" field is not always logged. Update the schema to
enforce that it is, and fix it for records that don't log it.

Ticket: #7167
1 year ago
Shivani Bhardwaj f2de3e01cb src: remove truncate fn and glue code
truncate fn is only active and used by dcerpc and smb parsers. In case
stream depth is reached for any side, truncate fn is supposed to set the
tx entity (request/response) in the same direction as complete so the
other side is not forever waiting for data.

However, whether the stream depth is reached is already checked by
AppLayerParserGetStateProgress fn which is called by:
- DetectTx
- DetectEngineInspectBufferGeneric
- AppLayerParserSetTransactionInspectId
- OutputTxLog
- AppLayerParserTransactionsCleanup

and, in such a case, StateGetProgressCompletionStatus is returned for
the respective direction. This fn following efc9a7a, always returns 1
as long as the direction is valid meaning that the progress for the
current direction is marked complete. So, there is no need for the additional
callback to mark the entities as done in case of depth or a gap.
Remove all such glue code and callbacks for truncate fns.

Bug 7044
1 year ago
Shivani Bhardwaj 80159eb519 applayer: remove truncation logic
as its functionality is already covered by the generic code.
This removes APP_LAYER_PARSER_TRUNC_TC and APP_LAYER_PARSER_TRUNC_TS
flags as well as FlowGetDisruptionFlags sets STREAM_DEPTH flag in case
the respective stream depth was reached. This flag tells that whether
all the open files should be truncated or not.

Bug 7044
1 year ago
Jason Ish 575e5b471f dns: add v3 dns logging
DNS v3 logging fixes the discrepancies between request and response
logging with the main difference being queries always being placed in an
array.

Bug: #6281
1 year ago
Jason Ish df656324ba dns: new v3 style logging for alerts
V3 style DNS logging fixes the discrepancies between request and
response logging better dns records and alert records.

The main change is that queries and answers are always logged as
arrays, and header fields are not logged in array items.

For alerts this means that answers are now logged as arrays, queries
already were.

DNS records will get this new format as well, but with a configuration
parameter.

Bug: #6281
1 year ago
Nathan Scrivens 9ecc3573a7 dns: parse and populate OPT rdata struct
Feature: 7017
Add DNSRDataOPT struct and DNSRData enum type OPT.
Add OPT parsing function and test function.
Add DNSRData OPT type to lua.rs match.
Log OPT rdata.
1 year ago
Nathan Scrivens 4598ca164d dns log: add additional section
Feature: 7011
dns_log_json_answer: log additional section records.
update schema.json with new "additionals" section.
1 year ago
Nathan Scrivens 1cd89640ef dns parsing: add additional section
Feature: 7011
Add additionals to DNSMessage struct.
Add parsing logic to populate additional section data.
Patch dns tests to account for additional section parsing.
1 year ago
Sascha Steinbiss e047ad25e2 mqtt: run rustfmt 1 year ago
Sascha Steinbiss ad02040860 mqtt: enable limiting of logged message length
Ticket: #6984
1 year ago
Sascha Steinbiss dd972f72dd rust: add JsonBuilder::set_string_limited() 1 year ago
Jeff Lucovsky 99f9451be3 detect: Use Option where appropriate
This commit uses Option instead of Result.

Issue: 6873
1 year ago
Jeff Lucovsky 70bdc37f96 detect/byte_extract: Move keyword parser to Rust
Implement the keyword parser in Rust.

Issue: 6873
1 year ago
Jeff Lucovsky 73dfc58772 detect/byte: Refactor endian, base
Issue: 6873

Refactor the enums for endian and base handling for broader use.
1 year ago
Philippe Antoine 7dfddab9ed detect: parse units for integer for every cases
Ticket: #6423

Not just equality, but also >3MB should work
For example flow.bytes_toserver>3MB
1 year ago
Victor Julien f59c43b1c7 smb/ntlmssp: improve version check
Don't assume the ntlmssp version field is always present if the flag is
set. Instead keep track of the offsets of the data of the various blobs
and see if there is space for the version.

Inspired by how Wireshark does the parsing.

Bug: #7121.
1 year ago
Philippe Antoine 647e878f7c detect: helper function for multibuffer 1 year ago
Philippe Antoine 5bd17934df http2: do not expand duplicate headers
Ticket: 7104

As this can cause a big mamory allocation due to the quadratic
nature of the HPACK compression.
1 year ago
Philippe Antoine 37509e8e0e modbus: abort flow parsing on flood
Ticket: 6987

Let's not spend more resources for a flow which is trying to
make us do it...
1 year ago
Jeff Lucovsky 5b97f4040c detect/base64: Use Rust defined modes everywhere
Issue: 6487

To avoid ambiguity, a single definition for base 64 decoding modes will
be used. The Rust base64 transform contains the definitions for the
existing mode types: Strict, RFC2045, RFC4648
1 year ago
Jeff Lucovsky 1823681709 detect/transform: from_base64 option parsing
Issue: 6487

Implement from_base64 option parsing in Rust. The Rust module also
contains unit tests.
1 year ago
Jeff Lucovsky ab0cb960a1 detect/parser: Refactor utility routines
Refactor utility functions/definitions from the byte_math module into
the parser module. This includes parse_var and ResultValue

Issue: 6487
1 year ago
Philippe Antoine 5f35035928 filestore: do not try to store a file set to nostore
Ticket: 6390

This can happen with keyword filestore:both,flow
If one direction does not have a signature group with a filestore,
the file is set to nostore on opening, until a signature in
the other direction tries to set it to store.
Subsequent files will be stored in both directions as flow flags
are now set.
1 year ago
Philippe Antoine 5bb5b4f46f rust: remove unnecessary nested unsafe 1 year ago
Philippe Antoine 4ccbcc4684 sip: use right slice to take line from
We iterate over input, but we are now at start.
Avois quadratic complexity turning to OOM.

Ticket: 7093
1 year ago
Jason Ish 49ecf37126 rust/ike: prefix never read field names with _
New warning from rustc.

The other option is to allow dead code, however this is more explicit,
and when they are read, its obvious they should be renamed.
1 year ago
Jason Ish 29d7ff026a rust: simply matches with unwrap_or_default
New default clippy warning:
https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
1 year ago
Jason Ish ee2175cdb6 rust: fix clippy lint for legacy_numeric_constants
https://rust-lang.github.io/rust-clippy/master/index.html#legacy_numeric_constants
1 year ago
Philippe Antoine 4fe3f04fa3 detect/enip: move keywords to rust
Ticket: 4863
1 year ago
Philippe Antoine ce1eea4ad6 detect/websocket: move keywords to rust
Ticket: 4863
1 year ago
Philippe Antoine 16952d67e7 detect/dhcp: move keywords to rust
Ticket: 4863
1 year ago
Philippe Antoine ae72376ebe detect/snmp: move keywords to rust
Ticket: 4863

On the way, convert unit test DetectSNMPCommunityTest to a SV test.

And also, make snmp.pdu_type use a generic uint32 for detection,
allowing operators, instead of just equality.
1 year ago
Philippe Antoine 4bbe7d92dc detect: helper to have pure rust keywords
detect: make number of keywords dynamic

Ticket: 4683
1 year ago
Philippe Antoine 08c511f1bf enip: remove unnecessary unsafe
As the function SCEnipRegisterParsers is already marked as unsafe
1 year ago
Victor Julien 37be66eef9 detect/iprep: update function naming
Bring in line with new Rust code naming for FFI functions.
1 year ago
Victor Julien 83976a4cd4 detect/iprep: implement isset and isnotset
Implement special "isset" and "isnotset" modes.

"isset" matches if an IP address is part of an iprep category with any
value.

It is internally implemented as ">=,0", which should always be true if
there is a value to evaluate, as valid reputation values are 0-127.

"isnotset" matches if an IP address is not part of an iprep category.

Internally it is implemented outside the uint support.

Ticket: #6857.
1 year ago
Victor Julien 539ab3a404 detect/iprep: update keyword parser for extendibility 1 year ago
Jason Ish f0dbfe863d misc: prefix functions with SC not Sc 1 year ago
Philippe Antoine d9d5170ec0 websocket: add data frame
Ticket: 7051
1 year ago
Juliana Fajardini bb45ac71ef dns: allow triggering raw stream reassembly
For TCP streams, app proto stream reassembly can start earlier, instead
of waiting and queueing up data before doing so.

Task #7018
Related to
Bug #7004
1 year ago
Philippe Antoine 82c03f72c3 enip: convert to rust
Ticket: 3958

- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
    enip.product_name, enip.protocol_version, enip.revision,
    enip.identity_status, enip.state, enip.serial, enip.product_code,
    enip.device_type, enip.vendor_id, enip.capabilities,
    enip.cip_attribute, enip.cip_class, enip.cip_instance,
    enip.cip_status, enip.cip_extendedstatus
1 year ago
Philippe Antoine 0d267e29a5 files: remove the need for state in callbacks
As files now belong to transactions
1 year ago
Philippe Antoine 5167ff6411 smtp/mime: look for urls in base64 message
Ticket: 5185

Previously, it was looked for message in plain text, and base64
encoding was only handled for attachments.

This commit also fixes the buffering got such base64 data streamed
into urls finding, by buffering a beginning non-empty line,
and by ensuring that we run extraction on the last line,
even if it had no EOL.
1 year ago
Juliana Fajardini 0946c213cd pgsql: trigger raw stream reassembly
Expose the raw stream earlier to the detection engine, as Pgsql can have
multiple messages per transaction and usually will have a message
complete within one TCP packet.

Bug #7000

Related to
Bug #7026
1 year ago
Juliana Fajardini 69e26de197 pgsql/logger: open json object from logger function
Before, the JsonBuilder object for the pgsql event was being created
from the C-side function that actually called the Rust logger.

This resulted that if another module - such as the Json Alert called the
PGSQL logger, we wouldn't have the `pgsql` key present in the log output
- only its inner fields.

Bug #6983
1 year ago
Philippe Antoine 7fb10676e7 dns: remove unneeded mut in logger 1 year ago
Philippe Antoine a10c1f1dde smtp: use rust for mime parsing
Ticket: #3487
1 year ago
Philippe Antoine 5f75b9a6e3 http: use rust for mime parsing
Ticket: #3487
1 year ago
Philippe Antoine 5555aa6788 mime: improved token parsing
Accepts escaped quote in escaped string
1 year ago
Jason Ish 2e440169d6 lua: remove lua as a compile time feature
Its always built-in. However, can be disabled at runtime.
1 year ago
Jason Ish 1fd2c1a379 rust/lua: remove lua_int8 feature
Now that we're fixed to Lua 5.4, the integer size is always 8.
1 year ago
Jason Ish bc011f2205 lua: use rust crate to vendor (bundle) lua
Remove lua-dev(el) from all CI tests.
1 year ago