Commit Graph

1757 Commits (master)

Author SHA1 Message Date
Shivani Bhardwaj 3a1c12414a tls: store list of subject alternative names
So far, the SANs were available as a part of IssuerDN via x509_parser
crate but SANs were not available to the SSLState* to be directly used
to setup and match against a sticky buffer.
Expose it to SSLStateConnp.

Feature 5234
1 year ago
Philippe Antoine 37a9003736 rust/probing: safety check for null input
Ticket: 7013

Done consistently for all protocols

This may change some protocols behaviors which failed early
if they found there was not enough data...
1 year ago
Philippe Antoine 5dc8dea869 rust: return empty slice without using from_raw_parts
As this triggers rustc 1.78
unsafe precondition(s) violated: slice::from_raw_parts requires
the pointer to be aligned and non-null,
and the total size of the slice not to exceed `isize::MAX`

Ticket: 7013
1 year ago
Philippe Antoine 806052d762 websocket: fix opcodes values for ping/pong
And also set close

Ticket: 7025
1 year ago
Philippe Antoine 8b103ae755 dns: set tx id for frames 1 year ago
Philippe Antoine 715bf048ee frames: rust API makes tx_id explicit
And set it right for SIP and websocket,
so that relevant tx app-layer metadata gets logged.

Ticket: 6973
1 year ago
Jeff Lucovsky cb56752bf7 config/ja3: Eliminate warnings when JA3 is disabled
This commit eliminates warnings when either ja3, ja4 or both are
disabled.
1 year ago
Philippe Antoine c53e9ac0dd sdp: fix logging medias
As introduced by bff790b6ac

Also handles errors in the caller

Ticket: 6994
1 year ago
Jason Ish df8568ee30 rust/dns: visibility cleanups
Remove pub from functions that don't require it.
1 year ago
Jason Ish 556cfe56bf rust/dns: ffi naming and visibility cleanups
- Remove no_mangle and pub from FFI functions that are only accessed
  with a function pointer.
- Rename all no_mangle FFI functions to our C naming scheme.
1 year ago
Giuseppe Longo 868493529b rust/sip: parse and log sdp
If SDP payload is found within a SIP message, it will be parsed and then
logged.

Ticket #6627
1 year ago
Giuseppe Longo bff790b6ac rust/sdp: implement logger
This implements a logger for the SDP protocol.
Given that SDP is encapsulated within other protocols (such as SIP),
enabling it separately is not necessary.

Ticket #6627
1 year ago
Giuseppe Longo 1ccfc35214 rust/sdp: implement protocol parser
This implements a parser for the SDP protocol.
Given that SDP is encapsulated within other protocols (such as SIP),
enabling it separately is not necessary.

Ticket #6627.
1 year ago
Philippe Antoine 03442c9071 http2: do not log duplicate headers
Ticket: 6900

And thus avoid DOS by logging a request using a compressed
header block repeated many times and having a long value...
1 year ago
Philippe Antoine 390f09692e http2: use a reference counter for headers
Ticket: 6892

As HTTP hpack header compression allows one single byte to
express a previously seen arbitrary-size header block (name+value)
we should avoid to copy the vectors data, but just point
to the same data, while reamining memory safe, even in the case
of later headers eviction from the dybnamic table.

Rust std solution is Rc, and the use of clone, so long as the
data is accessed by only one thread.
1 year ago
Philippe Antoine 0291d37009 websocket: configurable logging of payload in alerts 1 year ago
Philippe Antoine 44b6aa5e4b app-layer: websockets protocol support
Ticket: 2695
1 year ago
Philippe Antoine f83ec543e3 http2: add settings from newer RFCs
Including the one for websocket over HTTP/2
1 year ago
Juliana Fajardini ce1556cefd pgsql: check for eol when parsing response
It was brought to my attention by GLongo that Pgsql parser handled eof
diffrently for requests and responses, and apparently there isn't a good
reason for such a difference therefore, apply same logic used for
rs_pgsql_parse_request for checking for eof when parsing a response.
1 year ago
Sascha Steinbiss 120313f4da ja4: implement for TLS and QUIC
Ticket: OISF#6379
1 year ago
Sascha Steinbiss 9d0db71ebf ja3: make feature compile time configurable 1 year ago
Philippe Antoine 49caf005a4 detect/analyzer: create tojson function for generic integers
As will be needed for tcp.mss
1 year ago
Philippe Antoine 3643b6ed4b output: generic simple tx json logger
Ticket: 3827
1 year ago
Jason Ish 71f59e529c jsonbuilder: fix serialization of nan and inf
When outputting a float, check if its infinity, or not a number and
output a null instead.

Using a null was chosen as this is what serde_yaml, Firefox, Chrome,
Node, etc. do.

Ticket: #6921
1 year ago
Philippe Antoine ee50fe4c30 sip: convert transaction list to vecdeque
Ticket: 6891

So as to avoid quadratic complexity on tx cleanup with SIP/TCP
that can create many transactions in one go.
1 year ago
Philippe Antoine f7cde8f00e rust/smb: fix clippy nightly warning
error: unnecessary use of `to_vec`
    --> src/smb/smb.rs:1048:62
     |
1048 |         let (name, is_dcerpc) = match self.guid2name_map.get(&guid.to_vec()) {
     |                                                              ^^^^^^^^^^^^^^ help: replace it with: `guid`
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_to_owned
     = note: `#[deny(clippy::unnecessary_to_owned)]` implied by `#[deny(warnings)]`

And also other uses of to_vec() on already Vec
1 year ago
Philippe Antoine 02f2fb8833 rust: fix clippy 1.77 warning
Ticket: 6883

error: field `0` is never read
  --> src/asn1/mod.rs:36:14
   |
36 |     BerError(Err<der_parser::error::BerError>),
   |     -------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |     |
   |     field in this variant
   |
1 year ago
Philippe Antoine c4b8fb7aca ssh: limit length for banner logs
Ticket: 6770
1 year ago
Philippe Antoine 271ed2008b ssh: avoid quadratic complexity from long banner
Ticket: 6799

When we find an overlong banner, we get into the state just
waiting for end of line, and we just want to skip the bytes
until then.
Returning AppLayerResult::incomplete made TCP engine retain
the bytes and grow the buffer that we parsed again and again...
1 year ago
Hadiqa Alamdar Bukhari 3aa313d0c5 dns: add dns.rcode keyword
dns.rcode matches the rcode header field in DNS messages
It's an unsigned integer
valid ranges = [0-15]
Does not support prefilter
Supports matches in both flow directions

Task #6621
1 year ago
Victor Julien 1d3a156179 rust: update parser dependencies
Time locked to 0.3.20 to guarantee MSRV of 1.63.
Update snmp-parser to 0.10.0.
Update asn1-rs to 0.6.1.
Update kerberos-parser to 0.8.0.
Update x509-parser 0.16.0.
Update der-parser to 9.0.0.
Remove specific use of der-parser 6.

Ticket: #6817.
Ticket: #6818.
1 year ago
Hadiqa Alamdar Bukhari 4b81851097 dns: add dns.rrtype keyword
It matches the rrtype field in DNS
It's an unsigned integer match
valid ranges = [0-65535]
Does not support prefilter
Supports flow in both directions
Feature #6666
1 year ago
Giuseppe Longo fe77def816 rust/sip: register pattern matching
This permits to detect the SIP protocol using pattern matching instead of
probing parser.

Since it is no longer used, the respective probing functions have been removed.
1 year ago
Giuseppe Longo 9c9b1a4230 rust/sip: add direction to transaction
This patch permits to set a direction when a new transaction is created in order
to avoid 'signature shadowing' as reported by Eric Leblond in commit
5aaf50760f
1 year ago
Giuseppe Longo c9d309219e rust/sip: register parser for tcp
This patch lets the parser to work over tcp protocol, taking care of handling
data before calling the request/response parsers.

Ticket #3351.
1 year ago
Giuseppe Longo 69f841c998 sip/parser: enforce valid chars for sip version
The `is_version_char` function incorrectly allowed characters that are not
part of the valid SIP version "SIP/2.0".

For instance, 'HTTP/1.1' was mistakenly accepted as a valid SIP version,
although it's not.

This commit fixes the issue by updating the condition to strictly
check for the correct version string.
1 year ago
Giuseppe Longo 7e993d5081 sip/parser: accept valid chars
Accepts valid characters as defined in RFC3261.
1 year ago
Giuseppe Longo 8ff80cb84d rust/sip: rustfmt sip module 1 year ago
Jeff Lucovsky f9a20dafc6 mqtt: Improve frame parsing w/mult. PDUs
This commit improves the mqtt parsing of frames to handle multiple PDUs.

Issue: 6592
1 year ago
Philippe Antoine c99d93c257 app-layer/template: use a max number of txs
Ticket: 6773
1 year ago
Philippe Antoine 68b0052018 rust: fix clippy ptr_arg warnings
error: writing `&Vec` instead of `&[_]` involves a new object where a slice will do
   --> src/dns/log.rs:371:29
    |
371 | pub fn dns_print_addr(addr: &Vec<u8>) -> std::string::String {
    |                             ^^^^^^^^ help: change this to: `&[u8]`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
1 year ago
Philippe Antoine 80abc22f64 http2: limit number of concurrent transactions
Ticket: 6481

Instead of just setting the old transactions to a drop state so
that they get later cleaned up by Suricata, fail creating new ones.

This is because one call to app-layer parsing can create many
transactions, and quadratic complexity could happen in one
single app-layer parsing because of find_or_create_tx
1 year ago
Philippe Antoine 86de7cffa7 pgsql: parse only PDU when type is unknown
A next PDU may already be in the slice to parse.
Do not skip its parsing, ie do not use rest, but take just
the length of the pdu
1 year ago
Philippe Antoine f52c033e56 pgsql: parse auth message within its bound
If the next PDU is already in the slice next, do not use it and
restrict ourselves to the length of this PDU.
Avoids overconsumption of memory by quadratic complexity, when
having many small PDUS in one big chunk being parsed

Ticket: #6411
1 year ago
Philippe Antoine aff54f29f8 http2: handle reassembly for continuation frames
Ticket: 5926

HTTP2 continuation frames are defined in RFC 9113.
They allow header blocks to be split over multiple HTTP2 frames.
For Suricata to process correctly these header blocks, it
must do the reassembly of the payload of these HTTP2 frames.
Otherwise, we get incomplete decoding for headers names and/or
values while decoding a single frame.

Design is to add a field to the HTTP2 state, as the RFC states that
these continuation frames form a discrete unit :
> Field blocks MUST be transmitted as a contiguous sequence of frames,
> with no interleaved frames of any other type or from any other stream.
So, we do not have to duplicate this reassembly field per stream id.

Another design choice is to wait for the reassembly to be complete
before doing any decoding, to avoid quadratic complexity on partially
decoding of the data.
1 year ago
Philippe Antoine f6e1a20215 detect: dns.opcode as first-class integer
Ticket: 5446

That means it can accept ranges
1 year ago
Philippe Antoine d05f3ac791 detect: integer keywords now accept bitmasks
Ticket: 6648

Like &0x40=0x40 to test for a specific bit set
2 years ago
Philippe Antoine 370ac05419 detect/integer: rust derive for enumerations
Ticket: 6647

Allows keywords using integers to use strings in signature
parsing based on a rust enumeration with a derive.
2 years ago
Philippe Antoine 06c5dd3133 detect: integer keywords now accept negated ranges
Ticket: 6646
2 years ago
Philippe Antoine 3b65a2bb61 detect: integer keywords now support hexadecimal
So that we can write enip.revision: 0x203

Ticket: 6645
2 years ago
Philippe Antoine 38db51b878 rust: make cargo clippy clean
Fixing single_match and manual_find intertwined with SCLogDebug
2 years ago
Philippe Antoine 89936b6530 mqtt: fix logic when setting event
Especially sets transactions to complete when we get a response
without having seen the request, so that the transactions
end up getting cleaned (instead of living/leaking in the state).

Also try to set the event on the relevant transaction, instead
of creating a new transaction just for the purpose of having
the event.

Ticket: #6299
2 years ago
jason taylor 3cb7112aa5 detect: update smb.version keyword
Signed-off-by: jason taylor <jtfas90@gmail.com>
2 years ago
jason taylor bfc0790d87 rust: fix rustfmt warnings for smb detect
Signed-off-by: jason taylor <jtfas90@gmail.com>
2 years ago
Eloy Pérez González 415722dab2 smb: add smb.version keyword
Ticket: #5075

Signed-off-by: jason taylor <jtfas90@gmail.com>
2 years ago
Philippe Antoine 259cdf169e rust: fix single_binding
error: this match could be written as a `let` statement
   --> src/nfs/nfs3_records.rs:747:9
    |
747 | /         match result {
748 | |             (r, request) => {
749 | |                 assert_eq!(r.len(), 0);
750 | |                 assert_eq!(request.handle, expected_handle);
751 | |                 assert_eq!(request.name_vec, br#"bln"#);
752 | |             }
753 | |         }
    | |_________^
2 years ago
Philippe Antoine b141eb9f11 rust: fix single_match
warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
   --> src/http2/parser.rs:882:17
    |
882 | /                 match ctx.value {
883 | |                     Some(_) => {
884 | |                         panic!("Unexpected value");
885 | |                     }
886 | |                     None => {}
887 | |                 }
    | |_________________^
2 years ago
Philippe Antoine 9a84681bd9 rust: fix vec_init_then_push
warning: calls to `push` immediately after creation
    --> src/pgsql/parser.rs:1179:9
     |
1179 | /         let mut database_param: Vec<PgsqlParameter> = Vec::new();
1180 | |         database_param.push(database);
     | |______________________________________^
help: consider using the `vec![]` macro: `let database_param: Vec<PgsqlParameter> = vec![..];`
2 years ago
Philippe Antoine 85329f5351 rust: fix zero_prefixed_literal
warning: this is a decimal constant
   --> src/mqtt/parser.rs:888:19
    |
888 |             0x00, 06, /* Topic Length: 6 */
    |                   ^^
    |
2 years ago
Philippe Antoine a8199bf2ca rust: fix assertions_on_constants for assert!(false)
using panic! instead with a string message
2 years ago
Philippe Antoine c49463c86f rust: fix assertions_on_constants for assert!(true)
Which will be optimized away by the compiler
2 years ago
Juliana Fajardini 8d3de85edd pgsql: fix u16 overflow in query data_row
Found by oss-fuzz with quadfuzz.

Cf https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63113

According to PostgreSQL documentation the maximum number of rows can be
the maximum of tuples that can fit onto max u32 pages - 4,294,967,295 (cf
https://www.postgresql.org/docs/current/limits.html). Some rough
calculations for that indicate that this could go over max u32, so
updating the data_row data type to u64.

Bug #6389
2 years ago
Philippe Antoine 673d13d445 rust: allow clippy::items_after_test_module
As clippy began to complain about jsonbuilder.rs
2 years ago
Jeff Lucovsky f12e026696 mqtt: Move conf code to rust
Issue: 6387

This commit moves the configuration logic to Rust.
2 years ago
Jason Ish 5d5b0509a5 requires: add requires keyword
Add a new rule keyword "requires" that allows a rule to require specific
Suricata versions and/or Suricata features to be enabled.

Example:

  requires: feature geoip, version >= 7.0.0, version < 8;
  requires: version >= 7.0.3 < 8
  requires: version >= 7.0.3 < 8 | >= 8.0.3

Feature: #5972

Co-authored-by: Philippe Antoine <pantoine@oisf.net>
2 years ago
Jason Ish 15ed51f9b8 feature: provide a Rust binding to the feature API
As the feature module is not available for Rust unit tests, a mock
version is also provided.
2 years ago
Juliana Fajardini 1afb485dfa pgsql: remove unused msg field
The `ConsolidatedDataRow` struct had a `length` field that wasn't truly
used.

Related to
Bug #6389
2 years ago
Juliana Fajardini 30ac77ce65 pgsql: add cancel request message
A CanceldRequest can occur after any query request, and is sent over a
new connection, leading to a new flow. It won't take any reply, but, if
processed by the backend, will lead to an ErrorResponse.

Task #6577
2 years ago
Juliana Fajardini 7fa8bbfe43 pgsql: extract length validation into function
This is called so many times that it seems to make sense that we use a
function for this.
2 years ago
Victor Julien b8440a0917 jsonbuilder: add set_int for signed ints
Bug: #6615
2 years ago
Jason Ish f91122e0e8 dns: replace usage of rs_dns_tx_get_query_name with SCDnsTxGetQueryName
SCDnsTxGetQueryName was introduced to allow for getting the query name
in responses as well as requests, so covers the functionality of
rs_dns_tx_get_query_name.
2 years ago
Jason Ish 482325e28b dns: add dns.query.name sticky buffer
This buffer is much like dns.query_name but allows for detection in both
directions.

Feature: #6497
2 years ago
Jason Ish 5f99abb0cb dns: add dns.answer.name keyword
This sticky buffer will allow content matching on the answer names.
While ansers typically only occur in DNS responses, we allow the buffer
to be used in request context as well as the request message format
allows it.

Feature: #6496
2 years ago
Jason Ish 9464d0b14a dns: consolidate DNSRequest and DNSResponse to DNSMessage
DNS request and response messages follow the same format so there is
no reason not to use the same data structure for each. While its
unlikely to see fields like answers in a request, the message format
does not disallow them, so it might be interesting data to have the
ability to log.
2 years ago
Jason Ish e2d7a7f877 dns: rustfmt with latest stable 2 years ago
Philippe Antoine 1b5e04bee3 http2: do not have leading space for response line
Ticket: 6547
2 years ago
Juliana Fajardini bdec2d8ea8 pgsql: don't log password msg if password disabled
If the logging of the password is disabled, there isn't much point in
logging the password message itself.
2 years ago
Juliana Fajardini 9aeeac532e pgsql: remove probe_ts function
With the changes in the probing_ts function, this other one could become
obsolete. Remove it, and directly call `parser::parse_request` when
checking for gaps, instead.
2 years ago
Juliana Fajardini 53d29f652a pgsql: remove unused error handling call 2 years ago
Juliana Fajardini afd6e4dc41 pgsql: don't log unknown message type 2 years ago
Juliana Fajardini 4f85d06192 pgsql: fix probing functions
Some non-pgsql traffic seen by Suricata is mistankenly identified as
pgsql, as the probing function is too generic. Now, if the parser sees
an unknown message type, even if it looks like pgsql, it will fail.

Bug #6080
2 years ago
Juliana Fajardini 1ac5d97259 pgsql: add unknonwn frontend message type
We had unkonwn message type for the backend, but not the frontend
messages. It's important to better identify those to improve pgsql
probing functions.

Related to
Bug #6080
2 years ago
Victor Julien d3ccff5822 detect/asn1: handle in PMATCH
Since the asn1 keyword is processing payload data, move the handling of
the keyword into the PMATCH with content inspection.

Use u32 as buffer length in the Rust FFI
2 years ago
Victor Julien 132fe57ac6 rust: add copyright header to common.rs 2 years ago
Philippe Antoine e38b9de6a2 output/krb5: have krb5 properties in alerts
Ticket: 5977
2 years ago
Philippe Antoine 8a09bff0aa output/tftp: have tftp properties in alerts
Ticket: 6501
2 years ago
Philippe Antoine 0b6b015e26 output/alert: rewrite code for app-layer properties
Especially fix setup-app-layer script to not forget this part

This allows, for simple loggers, to have a unique definition
of the actual logging function with the jsonbuilder.
This way, alerts, files, and app-layer event can share the code
to output the same data.

Ticket: #3827
2 years ago
Philippe Antoine 90c17652a3 rust: remove unused
Ticket: #4083
2 years ago
Sascha Steinbiss 0c55fe3515 detect: add mqtt.connect.protocolstring
Ticket:  OISF#6396
2 years ago
Philippe Antoine e3cd0d073f http2: app-layer event for userinfo in uri
Ticket: #6426

as per RFC 9113
":authority" MUST NOT include the deprecated userinfo subcomponent
for "http" or "https" schemed URIs.
2 years ago
Philippe Antoine 6249722589 http2: normalize host when there is user info
Ticket: 6479
2 years ago
Philippe Antoine 46a46e5b1f http2: event on mismatch between authority and host
Ticket: #6425
2 years ago
Philippe Antoine ae72ce77fa detect: parse units for integers
Ticket: #6423

Especially for filesize, instead of just a number, a signature
can use a number and a unit such as kb, mb or Gb
2 years ago
Daniel Olatunji 54de0450f4 rust: remove cbindgen:ignore on frames module
This directive is no longer required, and does
mess up the rustdoc description of the module.
2 years ago
Daniel Olatunji 5c0af0b203 rust/doc: add docstring to rust module files.
Issue: #4584
2 years ago
Philippe Antoine 9157070907 quic: v2 support per rfc 9369
Ticket: #4968
2 years ago
Yatin Kanetkar b67ff4badf dhcp: Log Vendor Client Identifier (dhcp option 60)
* Log vendor client identifier (dhcp option 60) if extended dhcp
logging is turned on. This required the `vendor_client_identifier` to
be added to the json schema. Validation done using an SV Test
* Added `requested_ip` to the json schema as well, since it was
missed. My SV test failed without it.

Feature #4587
2 years ago
Philippe Antoine 5bdbc1a313 rdp: do not use zero-bit bitflag
cf https://docs.rs/bitflags/latest/bitflags/#zero-bit-flags

As warned by clippy 1.72.0
2 years ago
Philippe Antoine b235e85c68 rust: fix clippy warnings for version 1.72.0
Includes using the right prototype for C SRepCatGetByShortname
2 years ago
Shivani Bhardwaj 8770431986 dcerpc: accept ALTER_CONTEXT as a valid request
So far, if only the starting request was a DCERPC request, it would be
considered DCERPC traffic. Since ALTER_CONTEXT is a valid request type,
it should be accepted too.

Reported and patch proposed in the following Redmine ticket by
InterNALXz.

Bug 6191
2 years ago
Victor Julien 389f166d78 file: remove FILE_USE_DETECT flag
All implementations were converted to use the logic, so the flag itself
can be removed.
2 years ago
Shivani Bhardwaj d4e674b390 rust: fix clippy warnings 2 years ago
Philippe Antoine 60db5e981c http2: do not append data after closing file
Ticket: #6211

Completes commit 02dece5db5

Once a http2 stream has end of stream flag, we close the file.
If we see new data frames with this stream id, the new_chunk
function should ignore them as the file was already closed.
2 years ago
Jeff Lucovsky 690b65ae88 detect/byte_math: Permit var name for bytes value
Issue: 6145

Modifications to permit a variable name to be used for the byte_math
bytes value.
2 years ago
Philippe Antoine 02dece5db5 http2: file tracker is initialized when file is closed
Ticket: #6130

This avoids quadratic complexity by having http2_range_key_get
looking in a growing number of frames
2 years ago
Sascha Steinbiss 1521b77edd rfb: also set unimplemented auth types 2 years ago
Sascha Steinbiss 1606aca881 rfb: ensure logging of incompletely parsed txs 2 years ago
Sascha Steinbiss 1f8a5874fb rfb: never return error on unknown traffic
We only try to parse a small subset of what is possible in
RFB. Currently we only understand some standard auth schemes
and stop parsing when the server-client handshake is complete.
Since in IPS mode returning an error from the parser causes
drops that are likely uncalled for, we do not want to return
errors when we simply do not understand what happens in the
traffic. This addresses Redmine #5912.

Bug: #5912.
2 years ago
Sascha Steinbiss 836fff3679 rfb: add myself as contributor 2 years ago
Sascha Steinbiss bd1fbf392e rfb: be more strict parsing the version 2 years ago
Philippe Antoine d40dca5e55 dcerpc: maximum number of live transactions also for UDP
Ticket: #6129

Avoids that quadratic complexity gets too bad
2 years ago
Jason Ish 68d0d6ca24 rust: fix unit test link error on Rust 1.70
Rust 1.70 appears to now link code on both branches of `if cfg!(test)`
now causing Rust unit tests to fail as that pattern was used to
disable functions only available when linked with the Suricata C code.

To work-around this issue, provide two versions of the `new` function,
one for unit tests and one when running as an application.
2 years ago
Philippe Antoine 7256ec8a6e detect/http2: do not escape ':' in header name or value
for keywords http.request_header and http.response_header

Ticket: #5780
2 years ago
Philippe Antoine 4c466ec5f4 rust/pgsql: remove unused/unconstructed enum variants 2 years ago
Philippe Antoine f2a18e91c4 rust: define AppLayerEventType only in rust
And detect.h does no longer depend on app-layer-events.h
2 years ago
Philippe Antoine 668501c225 rust: remove unused 2 years ago
Philippe Antoine 7ca43e7e1f output/snmp: log version from tx
and not the one from state

If a SNMP flow starts with a V2 version transaction,
then there is a V3i version transaction,
we will now log V3 for the second transaction
2 years ago
Philippe Antoine 0ec0d8de67 output/rfb: remove unused function parameters 2 years ago
Philippe Antoine 24c2702a05 output/mqtt: remove unused function parameters 2 years ago
Philippe Antoine 09d364b32f output/krb5: remove unused function parameters 2 years ago
Lancer Cheng abc76e27de smb: fix data padding logic in writeAndX parser
Bug: #6008
2 years ago
Lancer Cheng 000eb91078 smb: fix wrong data offset when wct = 12
Bug: #6008
2 years ago
Philippe Antoine 6350736882 http2: avoid quadratic complexity in headers
When adding an element to the dynamic headers table, the oldest
ones may get evicted. When multiple elements get evicted, they
should get evicted all at once with drain, instead of one by one
as there will be a massive move each time.

Ticket: #6103
2 years ago
Philippe Antoine 7d3aa91bf4 mqtt: fix quadratic complexity
get_tx_by_pkt_id loops only over the last transactions
in case there is a transaction flood

Ticket: #6100
2 years ago
Haleema Khan 8e19906afa mqtt: rustfmt mqtt.rs 2 years ago
Haleema Khan e474858e25 mqtt: add mqtt frames
Adds PDU, Header and Data frame to the MQTT parser.
Ticket: 5731
2 years ago
Jason Ish 33827beae5 jsonbuilder: check buffer growth
Use try_reserve before growing the internal buffer, and the internal
state vector. This allows allocation errors to be caught and an error
returned instead of just aborting the process.

Ticket: #6057
2 years ago
Jason Ish 95cfc2b34f jsonbuilder: rustfmt
Some very minor changes to formatting.
2 years ago
Jason Ish c30fff8bcb rust/doc: restore comment with code example, but ignore
Use backticks for proper markdown processing. As Rust code in
backticks is compiled, and this is a non-complete example, tag the
code sample to be ignored.
2 years ago
Philippe Antoine 5391f0a8a0 detect: http_response_line for HTTP2
Ticket: #4067

Synthetized as HTTP/2 <STAT>\r\n
2 years ago
Philippe Antoine 0dca8cc796 detect: http_request_line support for HTTP2
Ticket: #4067

Synthetized as <METHOD> <URI> HTTP/2\r\n
2 years ago
Jason Ish 13fe957b7e rust/doc: wrap some code examples in backticks 2 years ago
Victor Julien d4c60924f1 rust/doc: fix doc compile issues 2 years ago
Eloy Pérez González ed91d689f2 krb5: use req_type instead of msg_type to get request type 2 years ago
Eloy Pérez González a9b7241417 krb5: set msg_type for KRB-ERROR messages to MessageType::KRB_ERROR 2 years ago
Eloy Pérez González 511dbfe171 krb5: add AS-REQ and TGS-REQ transactions
Fix bug in ticket #4529
2 years ago
Victor Julien f9276fdf00 rust: spelling fixes
Thanks to Josh Soref.
2 years ago
Victor Julien cdd3251982 snmp: fix spelling
Thanks to Josh Soref.
2 years ago
Victor Julien ee7ed99b6f rust: spelling 2 years ago
Victor Julien d630f0fa34 rust: rustfmt files with recent new tests 2 years ago
Victor Julien 77f1658c2a rust: fix new clippy warnings 2 years ago
Lancer Cheng 0cf742a9ca smb: add unit tests
Issue: 4865
2 years ago
tianjinshan 2c0c6cb0a5 smb/ntlmssp: fix parsing of negotiate flags
Ticket: #5783
2 years ago
Jason Ish 0e55307c1d app-layer: remove APP_LAYER_PARSER_OPT_UNIDIR_TXS
This flag is no longer needed as a parser can now create a transaction
as unidirectional.

Setting this flag also doesn't make sense on parsers that may have
request/reply and some unidirectional messaging.
2 years ago
William Correia e378aa8d15 modbus: bump crate version
sawp 0.12 is available and addresses future compilation failures in
dependent crates.
Updated modbus test case to expect 12 bytes needed instead of 15. This
aligns with expectations as the test case slices 3 bytes off the end of
a 12 byte message so needing 12 bytes is correct.

Ticket #5989
2 years ago
Jason Ish b4f0d3c741 rust: update der-parser to 8.2.0
Minimal modifications required on the Suricata side, mainly for fields
becoming private and needing an accessor instead.

Note: As the kerberos parser still depends on der-parser 6.0, we still
have to depend on that so it is depended on, but renamed to
der-parser6. There is not an udpated kerberos-parser yet that uses
der-parser 8.2.0.

Ticket: #5991
2 years ago
Jason Ish 3d9fc3bf1d rust: update snmp-parser to 0.9.0
Updating snmp-parser required directly depending on the asn1-rs crate
for the Oid type, as snmp-parser does not re-export this type anymore.

Ticket: #5992
2 years ago
Jason Ish d2fb958e28 rust: fix clippy lint for assert
Fix done automatically by clippy --fix
2 years ago
Haleema Khan 3531a4abaa rfb: rustfmt rfb.rs 2 years ago
Haleema Khan 3eee311350 rfb: add rfb frames, update tests
Adds a PDU frame to the RFB parser.
Update function signature in tests to reflect frames

Ticket: 5717
2 years ago
Victor Julien 0bbc411743 nfs: fix newline in debug messages 2 years ago
Philippe Antoine 9adb59bcdb http2: faster when reducing dynamic headers size
avoid quadratic complexity from removing the first element
and copying all the contents a big number fo times.

Ticket: #5909
2 years ago
Jason Ish 8ef410e284 app-layer: add direction to transaction creation where needed
Build on Eric's but set the direction on transaction creation when
needed. I think this makes it a little more clear, and easier to
document when creating single direction transactions.

This also somewhat abstracts the inner-workings of a directional
transaction from the implementation.

Ticket: #4759
2 years ago
Eric Leblond 9f4ca26962 sip: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 8b0d56c414 nfs: TX are not unidirectional
NFS transactions are not unidirectional so we should not declare
them as such.
2 years ago
Eric Leblond 19174de4f3 quic: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 44482a68b0 ntp: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond c64e4526cd krb: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 322f3896ba mqtt: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 7ce557a44c ike: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 8926d82465 dns: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 744fccee17 bittorrent_dht: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond a82a5aa84b snmp: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
2 years ago
Eric Leblond 5aaf50760f app-layer: add flag to skip detection on TX
Stamus team did discover a problem were a signature can shadow
other signatures.

For example, on a PCAP only containing Kerberos protocol and where the
following signature is matching:

alert krb5 $HOME_NET any -> any any (msg:"krb match"; krb5_cname; content:"marlo"; sid:3; rev:1;)

If we add the following signature to the list of signature

alert ssh $HOME_NET any -> any any (msg:"rr"; content:"rr"; flow:established,to_server; sid:4; rev:2;)

Then the Kerberos signature is not matching anymore.

To understand this case, we need some information:

- The krb5_cname is a to_client keyword
- The signal on ssh is to_server
- Kerberos has unidirectional transaction
- kerberos application state progress is a function always returning 1

As the two signatures are in opposite side, they end up in separate
sig group head.

Another fact is that, in the PCAP, the to_server side of the session
is sent first to the detection. It thus hit the sig group head of
the SSH signature. When Suricata runs detection in this direction
the Kerberos application layer send the transaction as it is existing
and because the alstate progress function just return 1 if the transaction
exists. So Suricata runs DetectRunTx() and stops when it sees that
sgh->tx_engines is NULL.

But the transaction is consumed by the engine as it has been evaluated
in one direction and the kerberos transaction are unidirectional so
there is no need to continue looking at it.

This results in no matching of the kerberos signature as the match
should occur in the evaluation of the other side but the transaction
with the data is already seen has been handled.

This problem was discovered on this Kerberos signature but all
the application layer with unidirectional transaction are impacted.

This patch introduces a flag that can be used by application layer
to signal that the TX should not be inspected. By using this flag
on the directional detect_flags_[ts|tc] the application layer can
prevent the TX to be consumed in the wrong direction.

Application layers with unidirectional TX will be updated
in separate commits to set the flag on the direction opposite
to the one they are.

Ticket: #5799
2 years ago
Eric Leblond 236869bc58 detect: remove STREAM_FLUSH
It is unused in the code so can be removed.

Ticket: #5799
2 years ago
Jason Ish d4418034d1 rust: update aes and aes-gcm crates
Addresses RUSTSEC-2021-0059, RUSTSEC-2021-0060.
2 years ago
Jason Ish 159b72c101 rust/clippy: allow derivable impls
The latest Rust will automatically "fix" derivable default
implementation, which is nice, but makes changes that don't meet our
current MSRV, so allow derivable impls for now.
2 years ago
Lancer Cheng 9207012e4b smb: fix parser of ntlmssp negotiateflags
Fix endian-conversion bug in function parse_ntlm_auth_nego_flags

Bug OISF#5783
2 years ago
Haleema Khan 6b55e53ff5 rfb: add unittests to rfb.rs
Task: #5741
2 years ago
Philippe Antoine 233ab11148 smb: handles records with trailing nbss data
If a file (read/write) SMB record has padding/trailing data
after the buffer being read or written, and that Suricata falls
in one case where it skips the data, it should skip until
the very end of the NBSS record, meaning it should also skip the
padding/trailing data.

Otherwise, an attacker may smuggle some NBSS/SMB record in this
trailing data, that will be interpreted by Suricata, but not
by the SMB client/server, leading to evasions.

Ticket: #5786
2 years ago
Philippe Antoine c1b7befb18 smb: checks against nbss records length
When Suricata handles files over SMB, it does not wait for the
NBSS record to be complete, and can stream the payload to the
file... But it did not check the consistency of the SMB record
length being read or written against the NBSS record length.

This could lead to an evasion where an attacker crafts a SMB
write with a too big Length field, and then sends its evil
payload, even if the server returned an error for the write request.

Ticket: #5770
2 years ago
Jason Ish f15f092a69 rfb: remove duplicate logging of depth
The "depth" field in the "pixel_format" object was being logged twice.

Issue: 5813
3 years ago
Jason Ish 717e2b0248 smb: fix duplicate interface logging
An array of interfaces was being logged without creating an array,
resulting in duplicate "interface" objects being logged. Instead put
these interfaces into an array like already done elsewhere.

Issue: 5814
3 years ago
Jason Ish 67baab573b smb: remove duplicate tree_id logging
Remove the second occurrence of tree_id logging which appears to
always be a duplicate of the first tree_id logged, even though they
come from different data structures.

Issue: 5811
3 years ago
Jason Ish e21ae88e05 rust: utility function to copy Rust strings to C strings
As there are a few places where a Rust string is copied into a provided
C string buffer, create a utility function to take care of these
details.
3 years ago
Jason Ish 6344501dba tls: fix date logging for dates before 1970
The Rust time crate used by the x509-parser crate represents dates
before 1970 as negative numbers which do not survive the conversion to
SCTime_t and formatting with the current time formatting functions.

Instead of fixing our formatting functions to handle such dates,
create a Rust function for logging TLS dates directly to JSON using
the time crate that handles such dates properly.

Also add a FFI function for formatting to a provided C buffer for the
legacy tls-log.

Issue: 5817
3 years ago
Jason Ish 64cb687a65 rust: suppress specific manual_flatten list
In this case of debug code, the explicit iterator seems to make more
sense.
3 years ago
Jason Ish 7080ecbb76 rust: remove explicit lifetimes where not needed 3 years ago
Jason Ish e7f5bd047d rust: fix needless borrows of references
Fixed automatically by cargo clippy --fix.
3 years ago
Jason Ish 29f345af1a rust: allow uninlined_format_args
Newer versions of Rust/clippy are getting picky about format strings.
We should allow and use the new style, but also not prevent the old
style.
3 years ago
Jason Ish 3f4dad8676 ftp: add events for command too long
Issue: 5235
3 years ago
Philippe Antoine b52293b609 dcerpc: config limit maximum number of live transactions
As is done for other protocols

Ticket: #5779
3 years ago
Philippe Antoine ba99241957 http2: fix leak with range files
Ticket: #5808

May have been introduced by a24d7dc45c

Function http2_range_open expects to be called only when
tx.file_range is nil. One condition to ensure this is to check
that we are beginning the files contents. The filetracker field
file_open is not fit for this, as it may be reset to false.
3 years ago
Victor Julien 37f13a4fc7 smb: set defaults for file transfer limits
Ticket: #5782.
3 years ago
Jason Ish fab3f36b8c dns: never return error on UDP DNS
UDP parsers should never return error as it should indicate to Suricata
that an unrecoverable error has occurred.  UDP being record based for
the most part is almost always recoverable, at least for protocols like
DNS.
3 years ago
Jason Ish d720ead470 dns: split header and body parsing
As part of extra header validation, split out DNS body parsing to
avoid the overhead of parsing the header twice.
3 years ago
Jason Ish 595700ab7e dns: validate header on every incoming message
As UDP streams getting probed, a stream that does not appear to be DNS
at first, may have a single packet that does look close enough to DNS
to be picked up as DNS causing every subsequent packet to result in a
parser error.

To mitigate this, probe every incoming DNS message header for validity
before continuing onto the body.  If the header doesn't validate as
DNS, just ignore the packet so no parse error is registered.
3 years ago
Jason Ish c98c49d4ba dns: parse and alert on invalid opcodes
Accept DNS messages with an invalid opcode that are otherwise
valid. Such DNS message will create a parser event.

This is a change of behavior, previously an invalid opcode would cause
the DNS message to not be detected or parsed as DNS.

Issue: #5444
3 years ago
Jason Ish 7afc2e3aed dns: rustfmt 3 years ago
Jason Ish 39d2524bf6 dns: mark test buffers with rustfmt::skip 3 years ago
Victor Julien 6cc9811edd files: move FileContainer into FileTransferTracker
Update SMB, NFS, HTTP2.
3 years ago
Victor Julien e3e55406a7 files: update API and callers to take stream config
This is to allow not storing the stream buffer config in each file.
3 years ago
Victor Julien 71bc9e75f5 app-layer: get sbconfg with files 3 years ago
Victor Julien a1a221066f files: remove filecontainer drop trait
In preparation of it becoming impossible to use due to the free
function getting an cfg argument.
3 years ago
Victor Julien 0320c03f8c http2: explicity free files
In preparation of adding an argument to the free functions which
means the drop trait can't be used anymore.
3 years ago
Victor Julien 4b1e9f7c21 smb: explicity free files
In preparation of adding an argument to the free functions which
means the drop trait can't be used anymore.
3 years ago
Victor Julien 3a24cce289 nfs: explicity free files
In preparation of adding an argument to the free functions which
means the drop trait can't be used anymore.
3 years ago
Victor Julien 4bfeac6591 nfs: file handling cleanups 3 years ago
Victor Julien 33f6a16290 smb: file handling cleanups 3 years ago
Victor Julien d57510a10f files: remove unused Rust binding for file pruning 3 years ago
Victor Julien a24d7dc45c smb: fix post-trunc chunk behavior
After a gap in a file transaction, the file tracker is truncated. However
this did not clear any stored out of order chunks from memory or stop more
chunks to be stored, leading to accumulation of a large number of chunks.

This patches fixes this be clearing the stored chunks on trunc. It also
makes sure no more chunks are stored in the tracker after the trunc.

Bug: #5781.
3 years ago
Philippe Antoine 55c4834e4e smb: configurable max number of transactions per flow
Ticket: #5753
3 years ago
Philippe Antoine 1d9183638f smb: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5753
3 years ago
Philippe Antoine cb89192ec3 smb: fix typo in comment 3 years ago
Haleema Khan cfcb7df9dc mqtt: rustfmt parser.rs 3 years ago
Haleema Khan 23acb89653 mqtt: add unittests for nom7 parsers
Ticket: #5742
3 years ago
Haleema Khan cdc5ccd7f7 rfb: rustfmt parser.rs 3 years ago
Haleema Khan b95d7efbd0 rfb: add unittests for nom7 parsers
Task: #5741
3 years ago
Philippe Antoine 3979acb5ed smb: set event for ntlmssp unusual order 3 years ago
Philippe Antoine e41c01a483 smb: rustfmt ntlmssp_records.rs 3 years ago
Philippe Antoine 1db8685848 smb/ntlmssp: parse fields independently of order
Instead of relying on the usual ordering...

Ticket: #5258
3 years ago
Jason Ish ae192ebae7 rust: sync log levels with C 3 years ago
Jeff Lucovsky f8474344cd log: Add module and subsystem identifiers to log
Issue: 2497

This changeset provides subsystem and module identifiers in the log when
the log format string contains "%S". By convention, the log format
surrounds "%S" with brackets.

The subsystem name is generally the same as the thread name. The module
name is derived from the source code module name and usually consists of
the first one or 2 segments of the name using the dash character as the
segment delimiter.
3 years ago
Victor Julien b31ffde6f4 output: remove error codes from output 3 years ago
Jason Ish bd9adac3ac rust/clippy: comments on why we have specific allows 3 years ago
Jason Ish dfd7abe185 rust/clippy: fix lint: type_complexity
Convert a DNS sub-parser to use a return type rather than a large
tuple. For mqtt, allow the lint for now, but remove the global allow.
3 years ago
Jason Ish e49ce49471 rust/clippy: allow result_unit_err in http2 only
Its the only module making use of this pattern, but we shouldn't let
new modules use this pattern.
3 years ago
Jason Ish 7ba2dadc7f rust/clippy: fix lint: upper_case_acronyms 3 years ago
Jason Ish 029ac650d7 rust/clippy: fix lint: manual_find
These get_tx methods look like ideal candidates for generic and/or
derived methods.
3 years ago
Jason Ish 4940dfb3bd rust/clippy: fix lint: len_without_is_empty 3 years ago
Jason Ish e1cffd348f rust/clippy: fix lint: field_reassign_with_default 3 years ago
Jason Ish 9df7c326b9 rust/clippy: remove allow: collapsible_else_if 3 years ago
Jason Ish 30ee5fc835 rust/clippy: remove allow: collapsible_if
Already clean.
3 years ago
Jason Ish da12b77f18 rust/clippy: fix lint: new_without_default 3 years ago
Jason Ish c4cf062a6f rust/clippy: fix lint: redundant_pattern_matching 3 years ago
Jason Ish 7c293ff68f rust/clippy: fix lint: never_loop 3 years ago
Jason Ish e8823644ec rust/clippy: fix lint: nonminimal_bool 3 years ago
Jason Ish 53ae0c8a06 rust/clippy: fix lint: derive_partial_eq_without_eq 3 years ago
Jason Ish 5d62995e26 rust/clippy: fix lint: explicit_counter_loop 3 years ago
Jason Ish f250b92180 rust/clippy: fix lint: extra_unused_lifetimes 3 years ago
Jason Ish 3044565cf4 rust/clippy: fix lint: needless_range_loop 3 years ago
Jason Ish 2ac52d0610 rust/clippy: remove lint: for_loops_over_fallibles
Already clean.
3 years ago
Jason Ish c026d8531b rust/clippy: fix lint: match_ref_pats 3 years ago
Jason Ish 359d5fcb7e rust/clippy: fix lint: needless_lifetimes 3 years ago
Jason Ish 4e001688de rust/clippy: remove lint: bool_comparison
Already clean.
3 years ago
Jason Ish f15ffbc869 rust/clippy: fix lint: single_match
Allow this lint in some cases where a match statement adds clarity.
3 years ago
Jason Ish 925bc74c1f rust/clippy: fix lint: while_let_loop 3 years ago
Jason Ish cf20fa1e67 template: import c_void, c_char, c_int
These are ffi types that are commonly used, import them so they can be
used by their short names instead of a fully qualified name.
3 years ago
Jason Ish 4220f18258 template: remove no_mangle and pub where not needed
Extern functions that are only used as a function pointer do not
require "pub" or "no_mangle".
3 years ago
Jason Ish 4a7567b3f0 template: rename template-rust to template
Remove the distinction between the C template protocol "template" and
the Rust template protocol "template-rust" and make the Rust parser
simply template now that we no longer have support to generate a C
protocol template.
3 years ago
Jason Ish 38321a213f rust/app-layer-template: rustfmt 3 years ago
Jason Ish 50a787a9a3 app-layer-template-rust: remove C app-layer stub
Remove the app-layer-PROTO stub for Rust based parsers.  It is no longer
needed as Rust parsers now contain the registration function in Rust.

Ticket: 4939
3 years ago
Jason Ish baa7021ee6 rust/conf: add fn conf_get_node
A wrapper around ConfGetNode to get a configuration node by name.
3 years ago
Victor Julien 64c0459d2d rust/lzma: clippy fixup 3 years ago
Jason Ish 35f99d1af7 rust/http2: fix clippy lint for is_empty()
This snuck through as "cargo clippy" check wasn't finding lints that
were fixed by the previous test for fixable lints.
3 years ago
Todd Mortimer 7d1a8cc335 file/swf: Use lzma-rs decompression instead of libhtp.
Use the lzma-rs crate for decompressing swf/lzma files instead of
the lzma decompressor in libhtp. This decouples suricata from libhtp
except for actual http parsing, and means libhtp no longer has to
export a lzma decompression interface.

Ticket: #5638
3 years ago
Victor Julien 45eb038e63 smb: fix file reopening issue
Fuzzing highlighted an issue where a command sequence on the same file
id triggered a logging issue:

file data for id N
close id N
file data for id N

If this happened in a single blob of data passed to the parser, the
existing file tx would be reused, the file "reopened", confusing the
file logging logic. This would trigger a debug assert.

This patch makes sure a new file tx is created for the file data
coming in after the first file tx is closed.

Bug: #5567.
3 years ago
Philippe Antoine 29f40c9e07 dcerpc: fix integer underflow
as input.len() can be 65536, it cannot be directly cast to u16

Ticket: #5557
3 years ago
Philippe Antoine af44504550 smb: do not use tree id to match request and response
Completes commit e94920b49f

This must be true for access to state ssn2vecoffset_map

Ticket: #5161
3 years ago
Victor Julien cade6046c5 rust/files: open file without trackid as pointer 3 years ago
Victor Julien ad869e1c52 rust/filecontainer: remove unused declaration 3 years ago
Philippe Antoine 086b28da3d http2: fix decompression buffering
It was not enough to set Cursor position to 0,
also its inner Vec should be cleared.

This way, a new input gets written at the beginning of the
Cursor and its inner Vec...

Ticket: #5691
3 years ago
Philippe Antoine c6349d3cfc http2: support padded data frames
Ticket: #5691
3 years ago
Philippe Antoine e1ee401a12 quic: use VecDeque
Ticket: #5637
3 years ago
Philippe Antoine 286bd2a7ed rust: fix cargo clippy --all-features 3 years ago
Philippe Antoine bc287018e5 rust: cargo clippy --all-features --fix --allow-no-vcs 3 years ago
Philippe Antoine cd4bf518f3 rust: fix warnings on rustc 1.67.0-nightly
warning: for loop over an `Option`. This is more readably written
as an `if let` statement
3 years ago
Juliana Fajardini a654ef50de pgsql: add test for parameter status parser
Since we've done some changes to how the parameters are parsed, add one
more test case to check that.

Bug #5579
3 years ago
Juliana Fajardini c4fbd78770 pgsql: move database into opt parameters list
For StartupMessages, the database parameter is optional. This moves the
parameter into the optional_parameters list.

Bug #5579
3 years ago
Philippe Antoine cc68898532 pgsql: support empty parameter values
Bug #5579
3 years ago
Philippe Antoine 1e0190bc6b pgsql: support out of order parameters for startup message
As user can be not the first parameter

Bug #5579
3 years ago
Jason Ish 1f056f9974 bittorrent-dht: parse and log node6 lists
Node6 lists are just like node lists, but for IPv6 addresses.
3 years ago
Jason Ish 86d5ab8644 bittorrent-dht: remove tests that are no longer valid 3 years ago
Jason Ish 2f9eb5d1dd bittorrent-dht: fix values decoding, as a list of peers
The "values" field is not a string, but instead peer information in
compact format. Decode this properly and then properly format in the
log.
3 years ago
Jason Ish 4a0859beeb jsonbuilder: add append_hex - add hex to array
New method, append_hex to add a byte array to a JSON array in hex
encoding.
3 years ago
Jason Ish 4bc9cf3986 bittorrent-dht: parse token and target as byte values 3 years ago
Jason Ish 5a30ee77a1 bittorrent-dht: only attempt to parse dht messages
The bittorrent flow is shared with transport messages as well as dht
messages. Only attempt to parse dht message as dht, ignore the rest.
3 years ago
Jason Ish 98a9391210 bittorrent-dht: decode node data structures
Instead of decoding the nodes field into a blog of bytes, decode it into
an array of node info objects, each with a node id, IP address and port.
3 years ago
Jason Ish 3cb50592ed bittorrent-dht: convert some fields to byte arrays
Some fields that were previously strings are not always value UTF-8
data, instead the protocol specification refers to them as strings of
bytes, so in other words byte arrays.

Currently fields converted are:
- client_version
- info_hash
- response.id
- request.id
- nodes
- token
3 years ago
Jason Ish 78ba17caa8 bittorrent-dht: register a pattern for protocol detection
Removes the port based probing which takes a long time to register for
ports 1024-65535 and instead use pattern based protocol detection.
3 years ago
Jason Ish 350c0723d7 bittorrent-dht: set parser to unidirectional
This parser does not match up responses with requests so flag it as
unidirectional.
3 years ago
Jason Ish 06eaec67ac bittorrent: updates for new event handling
Fixes anomaly events.
3 years ago
Aaron Bungay 86037885a9 bittorrent-dht: add bittorrent-dht app layer
Parses and logs the bittorrent-dht protocol.

Note: Includes some compilation fixups after rebase by Jason Ish.

Feature: #3086
3 years ago
Haleema Khan 8d5c5f24a1 dns/eve: add 'HTTPS' type logging
Add a new DNS record type to represent HTTPS
Ticket: #4751
3 years ago
Alice Akaki ccdc992a71 rust: fix lint warnings about mixed case hex literals
Ticket: #4593
3 years ago
Gabriel Lima Luz 4e90d17fd9 rust: fix lint warnings about if same then else
Ticket: 4609
3 years ago
Kristina Jefferson 9cd00424c3 rust: fix lint warnings about ptr_arg
Ticket: #4599
3 years ago
Jason Ish 21bb697bc9 rust: fix clippy lint for unneeded late initialization 3 years ago
Jason Ish 7cca238128 rust: fix clippy lint for cmp_null is debug code
Ticket: 5577
3 years ago
Jason Ish f0952aef0d rust: fix clippy lints for unneeded reference in debug code 3 years ago
Jason Ish 6a7439a26b rust: fix clippy lints for is_empty in debug code 3 years ago
Jason Ish 560c4ea125 rust: don't allow fixed up clippy lints 3 years ago
Jason Ish 6db85d6f89 rust: clippy fix for bitwise or with 0 3 years ago
Jason Ish 04f0ee0151 rust: fix clippy lints for clippy::unnecessary_cast 3 years ago
Jason Ish b6cc0e25b1 rust: fix clippy lints for clippy::redundant_static_lifetimes 3 years ago
Jason Ish 13db83274b rust: fix clippy lints for clippy::redundant_pattern_matching 3 years ago
Jason Ish 7ba1d3e300 rust: fix clippy lints for clippy::nonminimal_bool 3 years ago
Jason Ish 6ba0a67143 rust: fix clippy lints for clippy::map_flatten 3 years ago
Jason Ish 7ebdfa539a rust: fix clippy lints for clippy::manual_find 3 years ago
Jason Ish 119e02cf81 rust: fix clippy lints for clippy::collapsible_if 3 years ago
Jason Ish 572505870a rust: fix clippy lints for clippy::collapsible_else_if 3 years ago
Jason Ish 6b71d69356 rust: fix clippy lints for clippy::bool_comparison 3 years ago
Jason Ish e373d9f5e0 rust: fix clippy lints for clippy::crate_in_macro_def 3 years ago
Jason Ish 565da0d0af rust: fix clippy lints for clippy::redundant_field_names 3 years ago
Jason Ish 5f7ba03e63 rust: fix clippy lints for clippy::needless_bool 3 years ago
Jason Ish 3ec435a703 rust: fix clippy lints for clippy::manual_range_contains 3 years ago
Jason Ish f342d4aacd rust: fix clippy lints for clippy::len_zero 3 years ago
Jason Ish 5e5401d3e9 rust: fix clippy lints for clippy::char_lit_as_u8 3 years ago
Jason Ish 29a4a7fddc rust: fix clippy lints for clippy::assign_op_pattern 3 years ago
Jason Ish c4034dafa1 rust: fix clippy lints for clippy::derive_partial_eq_without_eq 3 years ago
Jason Ish 5a10fcd2d8 rust: suppress large enum variant lint at location
And disable the global lint.
3 years ago
Jason Ish 74b7522b6a rust/http2: box decompressor variants
These variants, in particular the Brotli one can be large at over 2500
bytes which is allocated no matter which decompressor is being used.

Gzip comes in at over 500 bytes.  Box deflate for consistency.
3 years ago
Jason Ish 36f8ada305 rust: remove clippy lints that no longer warn 3 years ago
Jason Ish e8c00dd980 rust: sort clippy allow statements 3 years ago
Haleema Khan 6c922e0b98 rust: fix lint warning for clippy::enum's name
Ticket: #4597
3 years ago
Jason Ish 2a42386c28 rust: fix clippy lint for null comparison
Use .is_null() instead of checking for equality against
std::ptr::null().
3 years ago
Jason Ish 45dfea2497 rust/modbus: derive default instead of manual impl
Cleans up a clippy lint for a trivial default impl that can be derived.
3 years ago
Jason Ish 9218da0eb8 rust/frames: cleanup clippy lint for unsafe
Where possible mark the relevant functions unsafe.  Otherwise suppress
the warning for now as this pattern is supposed to be a safe API around
an unsafe one. Might need some further investigation, but in general the
"guarantee" here is provided from the C side.
3 years ago
Jason Ish 105d9a5f02 rust: fix clippy lint for unnecessary_unwrap
Avoid check if not none followed by unwrap.
3 years ago
Jason Ish 85cfa7254b rust: fix clippy lint for single_char_add_str
Idiomatic cleanup and a fix automatically done by `cargo clippy --fix`.
3 years ago
Jason Ish f3e4bcfe23 rust: fix clippy lint for bool_assert_comparison
Checking for is_empty is faster than checking for equality.
3 years ago
Jason Ish f60e1b30f6 rust: fix clippy lint for partialeq_to_none
Use .is_some() and .is_none() instead of comparing against None.
Comparing against None requires a value to impl PartialEq, is_none() and
is_some() do not and are more idiomatic.
3 years ago
Jason Ish 7d623f0854 rust: fix clippy lint for explicit_auto_deref
This adds unnecessary complexity to code.
3 years ago
Jason Ish c503ca62e2 rust: fix clippy lint for needless_late_init 3 years ago
Jason Ish 94dd85baed rust: fix clippy lint for borrow_deref_ref
This type of borrow then reference has no effect.
3 years ago
Jason Ish e9597f3d0c rust: fix clippy lint for redundant_closure
Removes a closure where the function can be directly provided.
3 years ago
Jason Ish c5b26e2043 rust: fix clippy ling for needless borrows
Cleanup needless borrows found by clippy. This fix done automatically by
`cargo clippy --fix`.
3 years ago
Jason Ish 63b3d73ccc rust: allow some more clippy lints
Allow these lints for now until some more investigation can be done, as
--fix attempts to fix these.
3 years ago
Eric Leblond a9519778de rust/smb: avoid allocation in smb status function
Avoid an allocation by returning a static string.
3 years ago
Eric Leblond 9cb06d4376 detect/smb: add smb.ntlmssp_domain keyword
Feature #5411.
3 years ago
Eric Leblond 5debb86cd5 rust/smb1: add a missing command 3 years ago
Eric Leblond 69ef1bc194 detect/smb: add smb.ntlmssp_user keyword
Feature #5411.
3 years ago
Eric Leblond f46f895e8d rust/smb: import NT status code for Microsoft doc
This patch updates the NT status code definition to use the status
definition used on Microsoft documentation website. A first python
script is building JSON object with code definition.

```
import json
from bs4 import BeautifulSoup
import requests

ntstatus = requests.get('https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55')

ntstatus_parsed = BeautifulSoup(ntstatus.text, 'html.parser')

ntstatus_parsed = ntstatus_parsed.find('tbody')

ntstatus_dict = {}

for item in ntstatus_parsed.find_all('tr'):
    cell = item.find_all('td')
    if len(cell) == 0:
        continue
    code = cell[0].find_all('p')
    description_ps = cell[1].find_all('p')
    description_list = []
    if len(description_ps):
        for desc in description_ps:
            if not desc.string is None:
                description_list.append(desc.string.replace('\n ', ''))
    else:
        description_list = ['Description not available']
    if not code[0].string.lower() in ntstatus_dict:
        ntstatus_dict[code[0].string.lower()] = {"text": code[1].string, "desc": ' '.join(description_list)}

print(json.dumps(ntstatus_dict))
```

The second one is generating the code that is ready to be inserted into the
source file:

```
import json

ntstatus_file = open('ntstatus.json', 'r')

ntstatus = json.loads(ntstatus_file.read())

declaration_format = 'pub const SMB_NT%s:%su32 = %s;\n'
resolution_format = '        SMB_NT%s%s=> "%s",\n'

declaration = ""
resolution = ""

text_max = len(max([ntstatus[x]['text'] for x in ntstatus.keys()], key=len))

for code in ntstatus.keys():
    text = ntstatus[code]['text']
    text_spaces = ' ' * (4 + text_max - len(text))
    declaration += declaration_format % (text, text_spaces, code)
    resolution += resolution_format % (text, text_spaces, text)

print(declaration)
print('\n')
print('''
pub fn smb_ntstatus_string(c: u32) -> String {
    match c {
''')
print(resolution)
print('''
        _ => { return (c).to_string(); },
    }.to_string()
}
''')
```

Bug #5412.
3 years ago
Victor Julien db0f9ddc69 files/tx: inspection, logging and loop optimizations
Introduce AppLayerTxData::file_tx as direction(s) indicator for transactions.
When set to 0, its not a file tx and it will not be considered for file
inspection, logging and housekeeping tasks.

Various tx loop optimizations in housekeeping and output.

Update the "file capable" app-layers to set the fields based on their
directional file support as well as on the traffic.
3 years ago
Victor Julien 79499e4769 app-layer: move files into transactions
Update APIs to store files in transactions instead of the per flow state.

Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.

Update all protocol implementations to support this.

Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.

Update the filestore keyword to consider a match if filestore output not
enabled.
3 years ago
Victor Julien 01e64d80da app-layer: trunc parser per direction 3 years ago
Victor Julien ff9d1807f9 app-layer: parser flags to u16 3 years ago
Victor Julien c27df6304d app-layer: introduce common AppLayerStateData API
Add per state structure for storing flags and other variables.
3 years ago
Pierre Chifflier 16db04c1a7 rust: remove nom 5 dependency 3 years ago
Pierre Chifflier 0acf75bff7 rust/applayertemplate: convert to nom7 3 years ago
Pierre Chifflier 378e915846 rust/asn1: convert parsers to nom7 3 years ago
Pierre Chifflier 0ba0572c4a rust/x509: finish transition to nom7 3 years ago
Pierre Chifflier 3ef5121ab0 rust/telnet: convert parsers to nom7 3 years ago
Pierre Chifflier d98b386f36 rust/conf: convert parser to nom7 3 years ago
Pierre Chifflier db9a1e17b6 rust/ssh: finish transition to nom7 3 years ago
Pierre Chifflier b31c72c06a rust/rdp: convert parsers to nom7 3 years ago
Pierre Chifflier 49520b2143 rust/rdp: upgrade dependency on tls-parser 3 years ago
Pierre Chifflier beadd090b8 rust: upgrade versions of BER/DER, Kerberos and SNMP parsers 3 years ago
Pierre Chifflier 3aace49649 rust/x509: update dependency on x509-parser 3 years ago
Jeff Lucovsky ab4d0f7f4a detect/stream_size: Rename detect.rs to stream_size.rs
This commit renames detect.rs to stream_size.rs to reflect its content.
3 years ago
Jeff Lucovsky 2386f2614f detect/iprep: Move iprep logic into a separate module 3 years ago
Jeff Lucovsky c957882d1c detect/uri: Move uri logic into a separate module 3 years ago
Jeff Lucovsky 484c34bc60 detect/uint: Move uint logic into a separate module
This commit moves the uint logic into its own module.
3 years ago
Jeff Lucovsky 8bfe427a74 rust/detect: Create detect module for rule parsing
This commit creates a module named "detect" for rule parsing logic. As
part of this commit, detect.rs is moved from its toplevel position into
the new module. Thus, use crate::detect::detect to refer to items within
detect.rs (instead of create::detect).

Ticket: 5077
3 years ago
Jeff Lucovsky ccd1063e43 detect/bytemath: convert parser to Rust
Issue: 5077

This commit
- Converts the PCRE based parser to Rust.
- Adds unit tests to the new Rust modules
- Removes the PCRE parser from detect-bytemath.c
- Adjusts the C source modules to refer to the Rust definitions
- Includes the multiply operator (missing from the C parser)
3 years ago
Philippe Antoine af40873127 pgsql: config limit maximum number of live transactions
As is done for other protocols

Ticket: #5527
3 years ago
Philippe Antoine e160917bcf mqtt: remove quadratic time complexity
When having many transactions in a single parsing call...

Fix has overhead of having one more field in the mqtt state.

Completes commit a8079dc978

Ticket: #5399
3 years ago
Philippe Antoine 5ef259722b dhcp: adds renewal-time keyword
Ticket: #5507
3 years ago
Philippe Antoine 6faf6299e0 dhcp: adds rebinding-time keyword
Ticket: #5506
3 years ago
Philippe Antoine 95f0424423 nfs4: fix write record handling
Ticket: #5280
3 years ago
Philippe Antoine bf43011a43 dcerpc: convert transaction list to vecdeque for UDP
As was done for TCP in dfe76bb90 and d745d28d4

Ticket: #5518
3 years ago
Eric Leblond 2cc9152fc9 rust/smb: log uuid of interface in dcerpc
When doing a DCERPC request, we can use the context id to log the
interface that is used. Doing that we can see in one single event
what is the DCERPC interface and opnum that are used. This allows
to have all the information needed to resolve the request to a
function call.

Feature #5413.
3 years ago
Eric Leblond b6f1cf255c rust/smb/dcerpc: parse context id
As context id is used to know to which variant of the endpoint the
request is done, it is interesting to parse it.

Feature #5413.
3 years ago
Philippe Antoine 11f849c3ee protocol-change: sets event in case of failure
Protocol change can fail if one protocol change is already
occuring.

Ticket: #5509
3 years ago
Philippe Antoine e94920b49f smb: do not use tree id to match create request and response
As an SMB2 async response does not have a tree id, even if
the request has it.

Per spec, MessageId should be enough to identifiy a message request
and response uniquely across all messages that are sent on the same
SMB2 Protocol transport connection.
So, the tree id is redundant anyways.

Ticket: #5508
3 years ago
Philippe Antoine 461725a9bf dhcp: adds leasetime keyword
As it is logged

Ticket: #5435
3 years ago
Philippe Antoine 9b4a133777 http2: remove to_vec for comparisons
Ticket: #5454
3 years ago
Philippe Antoine d011b468da http2: fix clippy warning about &Vec<u8>
Using &[u8] instead in function prototype
3 years ago
Philippe Antoine 3de735ae70 ike: log ikev1 tx fields instead of state ones
As state fields can grow abitrarily, and this can lead to DOS
by quadratic complexity (CPU time and disk space)

Adds a direction field to retain all the information in the
transaction.

Also checks array vendor_ids had at least one element before
logging it.

Ticket: #5455
3 years ago
Philippe Antoine d0171d7418 ike: rustfmt 3 years ago
Philippe Antoine 5c7b5c5fb5 krb: detection for ticket encryption
As is done for logging.

Ticket: #5442
3 years ago
Philippe Antoine 64b2385c64 krb: log for ticket encryption
Also logs if the ticket encryption is weak.
It is different from the encryption used for the rest of the
packet, and this allows to detect kerberoasting attack.

Ticket: #5442
3 years ago
Philippe Antoine 7fcc6696cb krb: rustfmt kerberos.rs 3 years ago
Philippe Antoine 675de33405 krb: bump up crate version
kerberos parser crate is also used by other procotols : nfs and
smb. These protocols use an older der_parser crate version.
Upgrading der_parser will simplify the code further.
3 years ago
Philippe Antoine 783dff2c38 krb: rustfmt detect.rs 3 years ago
Jason Ish c862e84c01 rust/frames: cleanups
- Implement the Display trait on Direction to print "toserver" or
  "toclient" which used in a format string.

- Use Direction struct inside Frame instead of a u32.  Requires a helper
  method as there are two representation in C for direction, and the C
  methods for frames don't use the internal representation of the
  Direction enum (some sweeping changes could help here)
3 years ago
Jason Ish f92708b8ca rust/frames: derive direction from StreamSlice
On the Rust side, a Frame requires a StreamSlice to be created. We can
derive the direction from the StreamSlice removing the need for callers
to provide the direction when operating on the frame.
3 years ago
Jason Ish b39d7f46e7 dns/tests: fix StreamSlice to satisfy debug validation 3 years ago
Philippe Antoine f3b6fd3329 quic: update to nom7 3 years ago
Philippe Antoine 95125811b8 quic: reassemble crypto frames and parse it 3 years ago
Philippe Antoine f242fb7f22 quic: events and rules on them 3 years ago
Philippe Antoine b9c1d9e86b quic: parse gquic version Q039
Ticket: #5166
3 years ago
Philippe Antoine 018fef5ef8 quic: ja3 computation and logging and detection
Logging as is done in TLS.

Detection using the generic generic ja3.string keyword

Ticket: #5143
3 years ago
Philippe Antoine c6cf61a39b quic: complete parsing of initial for non gquic
The format of initial packet for quic ietf, ie quic v1,
is described in rfc 9000, section 17.2.2

Parse more frames and logs interesting extensions from crypto frame

Do not try to parse encrypted data, ie after we have seen
a crypto frame in each direction.

Use sni from crypto frame with tls for detection already implemented

Ticket: #4967
3 years ago
Philippe Antoine 7044131c39 quic: rustfmt 3 years ago
Philippe Antoine 0c346af4a9 rust: bump up digest crates
so that we can use hkdf crate for quic
3 years ago
Philippe Antoine 2294e9cdbc rdp: bump up tls-parser crate version
so that we can use new functions in quic parser
3 years ago
Philippe Antoine 11e0eb9c89 quic: do not log empty cyu array
Ticket: #5167
3 years ago
Philippe Antoine 632581ac95 ike: do not log empty notify array
Ticket: #5167
3 years ago
Philippe Antoine 262a93ce18 mqtt: do not log reason_codes if there is none
Ticket: #5167
3 years ago
Philippe Antoine 1621f5e453 detect/nfs: use inclusive ranges 3 years ago
Philippe Antoine ed6955ee98 detect: use generic integer functions for iprep
Ticket: #4112
3 years ago
Philippe Antoine cfb60d0fce detect: use generic integer functions for urilen
Ticket: #4112
3 years ago
Philippe Antoine c57052181c snmp: rustfmt detect.rs 3 years ago
Philippe Antoine c7214be99b snmp: adds usm keyword
as is logged

Ticker: #5416
3 years ago
Philippe Antoine eb1c2a6083 smb: use default stream-depth 0 by default
As broken by commit e5c948df87

Ticket: #5390
3 years ago
Philippe Antoine c585be338c nfs: fix arbitrary allocation
Bug introduced by https://github.com/OISF/suricata/pull/7111

Nom's count begins by allocating a Vector, which leads to arbitrary
allocation due to flavors_cnt coming from network, and not even
being checked against i.len()

Ticket: #5237
3 years ago
Philippe Antoine 26dc70648c dns: remove unused events field from state
found overflowing by oss-fuzz
3 years ago
Philippe Antoine d1a4dae36b detect: use generic integer functions for streamsize
By the way, adds the prefilter feature

Ticket: #2697
Ticket: #4112
3 years ago
Philippe Antoine 35b6dcec7e detect: use generic integer functions for filesize
Ticket: #4112
3 years ago
Philippe Antoine f29b43defd detect: rust generic functions for integers
Move it away from http2 to generic core crate.
And use it for DCERPC (and SMB)

And remove the C version.
Main change in API is the free function is not free itself, but
a rust wrapper around unbox.

Ticket: #4112
3 years ago
Philippe Antoine c4d9cb02ec util: better hex print function
Without dangerous snprintf pattern identified by CodeQL
even if this pattern is not a problem in those precise cases,
it may easily get copy pasted in a dangerous place, so better
get rid of it and make CodeQL happy
3 years ago
Philippe Antoine 6058792bee rust: make suricata context const
So that it is read only and its pointers do not get modified
3 years ago
Philippe Antoine 6224e283fa modbus: bump up rust crate version
So that probing parser is more strict and does not accept unknown
function code as valid modbus.

Ticket: #5377
3 years ago
Juliana Fajardini 6ccc01a79c rust: fix doc comments that trigger rust warnings
Rust generates warnings that are treated as errors for documentation
blocks before `extern` blocks.
3 years ago
Philippe Antoine d745d28d4a dcerpc: use vecdeque tx iterator
Ticket: #5321
3 years ago
Jason Ish dfe76bb905 dcerpc: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5271
3 years ago
Jason Ish 8790968281 mqtt, rdp: fix copyright dates 3 years ago
Philippe Antoine c78722a671 rust: RustParser same fields as AppLayerParser
So that there is no problem when crossing FFI
3 years ago
Sam Muhammed 323fe1c1ac nfs3/records: Fix typo
Fix response_lookup unittest name
3 years ago
Jason Ish b8b6a17a5b dns: add pdu frame
Adds a PDU frame to the DNS parser. For UDP this is the DNS payload
portion of the DNS packet, for TCP this is the payload minus the leading
legth field.

Ticket: 4984
3 years ago
Jason Ish 8d1840f595 frames(rust): don't call into C if running Rust unit tests
Wrap the calls behind frames to C code if a `cfg!(not(test))` so they
don't get compiled when running Rust unit tests.  Linkage to C functions
is not yet available for Rust unit tests, and this will keep the check
out of individual parsers.

Ticket: 4984
3 years ago
Jason Ish c74ea3840d frames (rust): method to create StreamSlice from slice
Useful in unit test for function that require a StreamSlice.
3 years ago
Jason Ish d712a8b29d eve/dns: remove dns v1 logging
Removal of DNS v1 logging was scheduled to be removed in May 2022.

Ticket: #4157
3 years ago
Jason Ish e319d31c14 template(rust): convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5298
3 years ago
Jason Ish 9b0b2beac1 pgsql: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5297
3 years ago
Jason Ish 2db84726ad http2: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5296
3 years ago
Jason Ish 4e0ad5e0bd rdp: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5295
3 years ago
Jason Ish 3b422e25f3 mqtt: convert transaction list to vecdeque
Allows for more efficient removal from front.

Ticket: #5294
3 years ago
Jason Ish 3189414788 dns: convert transaction list to vecdeque
Allows for more efficient removal from front of the list.

Ticket: #5277
3 years ago
Jason Ish 7b11b4d3a1 app-layer: more generic state trait
Instead of a method that is required to return a slice of transactions,
use 2 methods, one to return the number of transactions in the
collection, and another to get a transaction by its index in the
collection.

This allows for the transaction collection to not be a contiguous array
and instead can be a VecDeque, or possibly another collection type that
supports retrieval by index.

Ticket #5278
3 years ago
Juliana Fajardini bbd9a2ff1a pgsql: apply clippy fixes 3 years ago
Juliana Fajardini 0fc9ade7a9 pgsql: fix uint overflow and optimizations
Fuzzers found a possible integer overflow bug when parsing response
messages. To fix that, removed the case where we incremented the parsed
field length and created a new message type for situations where Suri
parsers an Unknown message. This is good because there may happen that
an unknown message to Suri is valid, and in this case, we would still be
able to log it.

Philippe Antoine found the bug while fuzzing with rust debug assertions.

Bug #5016
3 years ago
Juliana Fajardini 8daa79513a pgsql: fix clippy is_null usage warning 3 years ago
Philippe Antoine d2f00ac824 dcerpc: use wrappingadd for padding parsing
As we compute a modulo, we can safely wrap around even if there
is an overflow

Ticket: #5301
3 years ago
Philippe Antoine 464ff80c6a smb: protocol detection on pattern without midstream
To recognize a protocol, Suricata first looks for
patterns, which can be confirmed by a probing parser.
If this does not work, Suricata can try to run
some probing parsers on some ports.

This is the case for SMB.

This commit makes handling the confirming and the probing
paser differently even if they share much code.

The confirmation parser knows that a pattern has been found.
So, it must not do the midstream case of looking for this
pattern in the whole buffer, but only check it at the beginning.
But it must reverse direction if needed.
3 years ago
Victor Julien dc57460427 smb: fix event types for limit exceeded rules 3 years ago
Jason Ish 0623ada24d dns: better error handling when parsing names
The DNS name parser will error out with an error even if the
error is incomplete. Instead of manually generating errors,
use '?' to let the nom error ripple up the error handling chain.

The reason this wasn't done in the first place is this code
predates the ? operator, or we were not aware of it at the time.

This prevents the case where probing fails when there is enough data to
parse the header, but not enough to complete name parser. In such a case
a parse error is returned (instead of incomplete) resulting in the
payload not being detected as DNS.

Ticket #5034
3 years ago
Jason Ish 27679a12aa dns: don't parse a full request during probe if not enough data
If there is more data than a header, but not enough for a complete DNS
message, the hostname parser could return an error causing the probe to
fail on valid DNS messages.

So only parse the complete message if we have enough input data. This is
reliable for TCP as DNS messages are prefixed, but for UDP its just
going to be the size of the input buffer presented to the parser, so
incomplete could still happen.

Ticket #5034
3 years ago
Victor Julien fc9b65d8d3 smb2: validate negotiate read/write max sizes
Raise event if they exceed the configured limit.
3 years ago
Victor Julien 4be8334c9e smb2: allow limiting in-flight data size/cnt
Allow limiting in-flight out or order data chunks per size or count.

Implemented for read and writes separately:

app-layer.protocols.smb.max-write-queue-size
app-layer.protocols.smb.max-write-queue-cnt
app-layer.protocols.smb.max-read-queue-size
app-layer.protocols.smb.max-read-queue-cnt
3 years ago
Victor Julien 2c5ad8858e filetracker: track total queued data (in_flight)
As well as expose number of chunks.
3 years ago
Victor Julien 90d4b8e438 smb: log max read/write sizes 3 years ago
Victor Julien 5bcc4162f7 smb2: add options for max read/write size
Add options for the max read/write size accepted by the parser.
3 years ago
Victor Julien f28888513a smb2: track max read/write size and enforce its values 3 years ago
Victor Julien 594acec5dc smb: minor function cleanup
Remove used argument from `filetracker_newchunk()`. We're not
using fill_bytes with smb.
3 years ago
Victor Julien c7a474c725 filetracker: make FileChunk private 3 years ago
Philippe Antoine 3b13008c1b mqtt: fix consumed bytes computation for truncated msg
Ticket: 5268
3 years ago
Philippe Antoine 704bc878ea dcerpc: store consumed_bytes as i32
As it can grow bigger than u16
3 years ago
Philippe Antoine dfd17e9acc ike: fix integer underflow in parse_proposal
By not restricting a usize to i16
3 years ago
Victor Julien 053a9d2e68 smb/ntlmssp: add stricter len/offset validation 3 years ago
Philippe Antoine 3e48881b78 smb: prevents integer underflow
Ticket: 5246

If msg_id is 0, we cannot find the previous request
3 years ago
Philippe Antoine e72036f12f smb: ntlmssp domain_blob_offset underflow check
Ticket: 5246
3 years ago
Philippe Antoine 817a5001a5 smb: check on param parsing
Ticket: 5246

so as not to overflow u16
3 years ago
Sascha Steinbiss 7eb279ac53 mqtt: remove redundant "where" keyword 3 years ago
Sascha Steinbiss d63e5b8c51 mqtt: make some functions non-public 3 years ago
Sascha Steinbiss 2a3ed9a6ae mqtt: rustfmt 3 years ago
Sascha Steinbiss 1ba62993d5 mqtt: raise event on parse error 3 years ago
Sascha Steinbiss 5618273ef4 mqtt: ensure we do not request extra data after buffering
This addresses Redmine bug #5018 by ensuring that the parser
never requests additional data via the Incomplete error, but to
raise an actual parse error, since it is supposed to have all
the data as specified by the message length in the header already.
3 years ago
Victor Julien 6d30f4442c http2: fix file accounting for ranged files
Increment files_opened for tx that 'gets' reassembled ranged file
3 years ago
Victor Julien b336882008 smb1: apply close to direction
Instead of closing files in both direction when receiving a close request,
close only toserver files for the request and close toclient on receiving
a response.
3 years ago
Victor Julien b9cd502249 smb: convert 'close' parser to function 3 years ago
Sam Muhammed 3a490fb16c nfs: Implement frames
Feature #4872

Frames:
  - RPC Frames: Generic over TCP/UDP
     - rpc.pdu
     - rpc.hdr
     - rpc.data
     - rpc.creds -- for rpc calls

  - NFSv2, NFSv3
     - nfs.pdu
     - nfs.status -- for nfs responses

  - NFSv4 Only Frames
     - nfs4.pdu
     - nfs4.hdr
     - nfs4.ops -- for compound request/response operations
     - nfs4.status -- for nfs4 responses

RPC tcp/udp frames created with separate registeration functions e.g:
add_rpc_tcp_tc_frames()
add_rpc_udp_tc_frames()
3 years ago
Sam Muhammed d090dcbce9 rpc: Improve rpc_record struct
Add creds_len field to rpc_record
needed for rpc.creds frame length calculation
3 years ago
Sam Muhammed 8064a5348d rust/nfs4: Add NFSPROC4_DESTROY_CLIENTID op parsers 3 years ago
Sam Muhammed 9d1fad28a7 rust/nfs4: Add NFSPROC4_DESTROY_SESSION op parsers
Also add respective request unittest
test_nfs4_request_destroy_session()
3 years ago
Sam Muhammed ff81cad4f1 rust/nfs4: Add NFSPROC4_LAYOUTRETURN op parsers
Also add respective request unittest
test_nfs4_request_layoutreturn()
3 years ago
Sam Muhammed 073244a0b8 rust/nfs4: Add NFSPROC4_GETDEVINFO op parsers
Also add respective response/request unittests
test_nfs4_response_getdevinfo()
test_nfs4_request_getdevinfo()
3 years ago
Sam Muhammed ff54a6d9d5 rust/nfs4: Add NFSPROC4_LAYOUTGET op parsers
Also add respective response/request unittests
test_nfs4_response_layoutget()
test_nfs4_request_layoutget()
3 years ago
Sam Muhammed 3d542fcc67 rust/nfs4: Add NFSPROC4_SECINFO_NO_NAME op parsers 3 years ago
Sam Muhammed b35d635ac7 rust/nfs4: Add NFSPROC4_RECLAIM_COMPLETE op parsers 3 years ago
Sam Muhammed 2a41b46eca rust/nfs4: Add NFSPROC4_CREATE_SESSION op parsers
Also add respective response/request unittests
test_nfs4_request_create_session()
test_nfs4_response_create_session()
3 years ago
Sam Muhammed 0a69c66153 rust/nfs4: Add NFSPROC4_EXCHANGEID response parser
Also add test_nfs4_response_exchangeid() unittest
3 years ago
Sam Muhammed fe7a49b737 rust/nfs4: improve NFSPROC4_OPEN op parser
Improve nfs4_res_open() parser to reflect other file-delegation types
Reflect the changes on test_nfs4_response_open() unittest
3 years ago
Jason Ish 2341f47755 smb: handle records in the wrong direction
If an SMB record is seen in the wrong direction, set an event on the PDU
frame and don't process the record in the state.

No error is returned, so the next record will be processed.
3 years ago
Jason Ish 09e2d3b216 smb: expose smb1 request/reply flags with a method
Adds `.is_request()` and `.is_reply()` to check if a SMB record flags
say the message is a request or a reply.
3 years ago
Jason Ish 7b659489c8 smb: fix smb2 header flag parsing
The bits were being parsed in the order they're displayed in Wireshark,
rather than the order they were being seen on the wire, resulting in
direction and async being 0 more often than they should be.

Instead of bits, take the 4 bytes as an le_u32 and just use bit masks to
extract what we need into a struct, I think its easier to reason about
this way when comparing to the Microsoft documentation.
3 years ago
Philippe Antoine bfcd6cb46a range: validity check when end is bigger than size
Ticket: 5132

Down the line, HttpRangeOpenFileAux assumes the range has a
valid value when doing buflen = end - start + 1;
3 years ago
Victor Julien 07b1100713 nfs: clean up partial record handling
There should be no remaining data after parsing the partial
RPC record, so don't handle it but instead add a debug validation
bug on.

Successful processing for NFSv3 read/write records returns
AppLayerResult::ok() directly as all data is consumed.
3 years ago
Victor Julien d85b77cad0 nfs3: improve read validation; fix partial handling 3 years ago
Victor Julien 4418fc1b02 nfs3: fix partial write record handling 3 years ago
Victor Julien 5baf94e40d nfs3: enforce more values
Enforce values of a number of u32's that are used as bools or for
really low values.
3 years ago
Victor Julien 1c57e3c18d rpc: enforce various field values
Minimal frag_len. Correct msgtype and others.
3 years ago
Victor Julien 64d8a1e16e nfs/rpc: update full record parsers to be more exact
Instead of 'take'ing all data for the RPC prog_data and then
letting the higher level parsers figure out which part to use
take the exact amount.
3 years ago
Victor Julien bfb5ae867e nfs: break out partial record handling 3 years ago
Victor Julien fe76ab1803 nfs/rpc: enforce length field limits
Limits based on the Linux kernel limits. Then multiplied a few times
to allow for other implementations to have higher limits.
3 years ago
Victor Julien 5ecb626e50 nfs4: verify bool fields 3 years ago
Pierre Chifflier b8f767d84c rust/mime: convert parser to nom7 3 years ago
Juliana Fajardini 5a7645fac1 rust: add comment tags to support documentation
With these, the portion of code within the tags should be included
in the related code-snippets (for frame support documentation) w/o
errors, even if the code within changes. The tags can also work as
a reminder that the existing code is being shown elsewhere, so folks
know documentation might need updates, in case of major changes.
3 years ago
Juliana Fajardini e0dd1820c2 sip: apply rustfmt to a few functions
Our current rust code isn't always documentation friendly when it
comes to using code snippets. Used rustfmt to apply rust default
formatting on functions that we wanted to show in our documentation
for Frame support
3 years ago
Juliana Fajardini 71cbd2bf0e telnet: apply rustfmt to parse_request
When we want to share our code in our documentation pages, the current
rust formatting isn't so nice to read. Formatted just the portion of
the code that will be shown, for now.
3 years ago
Philippe Antoine 8adf172ab8 nfs: limits the number of active transactions per flow
Ticket: 4530
3 years ago
Philippe Antoine 0e85dea3ff nfs: remove unused events variable 3 years ago
Philippe Antoine e4f2f8f78d nfs: derive AppLayerEvent for NFSEvent 3 years ago
Pierre Chifflier 8dc3431d86 rust/dcerpc: convert parser to nom7 functions 3 years ago
Pierre Chifflier b5166bdb93 rust/ntp: upgrade dependency on ntp-parser 4 years ago
Pierre Chifflier fa63945bdc rust/ike: convert parser to nom7 functions and upgrade dependency 4 years ago
Pierre Chifflier 3493537ec3 rust/rfb: convert parser to nom7 functions 4 years ago
Victor Julien 474e0e3644 sip: enable for 5061/udp 4 years ago
Victor Julien 1203750388 sip: add frames support
Frames:
- sip.pdu
- sip.request_line
- sip.response_line
- sip.request_headers
- sip.response_headers
- sip.request_body
- sip.response_body

The `sip.pdu` frame is always created, the rest only if the record
parser succeeded.

Ticket: #5036.
4 years ago
Philippe Antoine e42094f238 mqtt: make max transactions configurable
Allows users to find balance between completeness of decoding
and increases resource consumption, which can DOS suricata.
4 years ago
Philippe Antoine 4f90d4254e http2: makes some settings configurable
max-streams and max-table-size

Allows users to find balance between completeness of decoding
and increases resource consumption, which can DOS suricata.
4 years ago
Philippe Antoine a8079dc978 mqtt: limits the number of active transactions per flow
Ticket: 4530

So, that we do not get DOS by quadratic complexity, while
looking for a new pkt_id over the ever growing list
of active transactions
4 years ago
Philippe Antoine 5475212f21 http2: limits the number of active transactions per flow
Ticket: 4530

So, that we do not get DOS by quadratic complexity, while
looking for a new stream id over the ever growing list
of active streams
4 years ago
Philippe Antoine f0e869b26b mqtt: parse properties with the right buffer's length 4 years ago
Philippe Antoine df2cbd6517 http2: event for variable-length integer overflow
http2_parse_var_uint can overflow the variable-length
integer it is decoding. In this case, it now returns an error
of kind LengthValue.

The new function http2_parse_headers_blocks, which factorizes
the code loop for headers, push promise, and continuation, will
check for this specific error, and instead of erroring itself,
will return the list of so far parsed headers, plus another one
with HTTP2HeaderDecodeStatus::HTTP2HeaderDecodeIntegerOverflow

This status is then checked by process_headers to create an
app-layer event.
4 years ago
Philippe Antoine b86beb9b68 http2: check overflow before it happens
instead of checking afterwards if value got smaller
4 years ago
Eloy Pérez González bff0774767 smb/dce_iface: avoid deleting current ifaces from state
The smb dce_iface keyword must match for all those dcerpc requests
and responses sent in the context of the given interface. They are
not matching as the current bind interfaces are deleted by any
non bind message.

Ticket: 4767
4 years ago
Eloy Pérez González 1ae22fd5de smb/dce_iface: use DCERPC_TYPE_REQUEST
The smb dce_iface keyword must match for all those dcerpc requests and
responses sent in the context of the given interface. They are not
matching because in rs_smb_tx_get_dce_iface, x.req_cmd is erroneously
compared with 1. Fix this by comparing with DCERPC_TYPE_REQUEST instead.

Ticket: 4767
4 years ago
Eloy Pérez González 333db3b385 smb/dce_opnum: move range if to outer context
The smb dce_opnum matches all the opnums that are higher that the
indicated opnum. This is due the range comparison if was put in the
exact comparison context, and in case the opnum doesn't match exactly,
then the range comparison is triggered (the upper limit is always true).

Move the erroneus if to the outer context, as else option of the block
checks if comparison should be exact or range.

Ticket: 4767
4 years ago
Eloy Pérez González 8dca3d0416 smb/dce_opnum: use DCERPC_TYPE_REQUEST
The smb dce_opnum keyword doesn't match the dcerpc requests/responses.
This occurs because in the rs_smb_tx_match_dce_opnum function, the
x.req_cmd is matched against the erroneous code 1. Fix this by using
DCERPC_TYPE_REQUEST for the comparison instead.

Ticket: 4767
4 years ago
Eloy Pérez González 15f493f516 dcerpc: remove prev_tx_call_id and clear_bind_cache from DCERPCState
remove those fields since are not used because of the removal of
handle_bind_cache.
4 years ago
Eloy Pérez González 1aca2676a6 dcerpc: avoid delete the rpc state interface context
The bug:
The dcerpc dce_iface keyword just match the packet following the bind. Only the
next request after the rpc is sent will match. However the expected behaviour it
that all the rpc requests/responses sent under the context of the given
interface would match.

In the Open Group c706 the following is indicated:

In 2.2.1 Binding-related Operations, indicates that one category of binding
operations are those that "operations that establish internal call routing
information for the server." (The other are to establish the protocol which is
not relevant here). And the following statement can be found:

Operations in the second category establish a set of mappings that the server
can use to route calls internally to the appropriate manager routine. This
routing is based on the interface and version, operation and any object
requested by the call.

It indicates that server routes (to call methods) are based on the operation,
interface and object.

- Operation: To indicate the method to call, and operation number is
             specified as indicated in the second step of 2.3.3.2 (Client
             Binding Steps).
- Interface: An interface is a set of remotely callable operations offered by a
             server and invokable by clients. (2.1.1.1)
- Object: Is the manager that implements the interface, as stated in section
          Interface and Manager Selection of 2.3.3.3. It is not mandatory, can
          be nil.

To call a method, a client must send a request message as defined in 2.6.4.9,
that contains these identifiers:

- opnum: The opnum field identifies the operation being invoked within the
         interface.
- p_cont_id (Context ID in Wireshark): The p_cont_id field holds a presentation
                                       context identifier that identifies the
                                       data representation and interface, as
                                       defined in 12.6.3.4 (Context Identifiers).
- object: The object field is contained if the PFC_OBJECT_UUID is set. (Could be
          interesting to create a keyword dce_object for matching this UUID)

Therefore, to get the correct method to invoke, the server must map the context
to the correct interface. This is negotiated by the bind request

Interfaces are first negotiated using the bind message (12.6.4.3), contained in
the p_context_elem array. Then they are accepted or rejected using the bind_ack
message (12.6.4.4).

Once these contexts are established, both client and server can use the context
id, which is the index of the p_context_elem array, to refer the interface they
are using.

Moreover, in the middle of the connection, the context can be changed with the
alter_context message.

This is way suricata shouldn't delete the bindack attribute, that contains
the contexts, used by match_backuuid. This is the only way to know the interface
a request message is referring to.

ticket: 4769
https://redmine.openinfosecfoundation.org/issues/4769
4 years ago
Juliana Fajardini 0bf1227f0f pgsql: fix defect found by coverity
Pgsql was using bitwise operations to assign password output config to
its context flags, but mixing that with logic negation of the default
value, resulting in the expressions having a constant value as result.

Bug: #5007
4 years ago
Pierre Chifflier ce9efc34c7 rust/pgsql: convert parsers to nom7 functions 4 years ago
Juliana Fajardini 579d7dcc01 pgsql: add initial support
- add nom parsers for decoding most messages from StartupPhase and
SimpleQuery subprotocols
- add unittests
- tests/fuzz: add pgsql to confyaml

Feature: #4241
4 years ago
Juliana Fajardini 4c743b809c rust/applayer: add function for upgrading to TLS 4 years ago
Victor Julien e02b52c895 quic: add quic.ua for matching user agent 4 years ago
Victor Julien 4c13b73c4d quic: log user agent when available 4 years ago
Victor Julien da8b024b99 detect/quic: add quic.sni sticky buffer 4 years ago