Commit Graph

573 Commits (9f9c29a14a91f86c1bda92285e876e026dc9e3a4)

Author SHA1 Message Date
Jason Ish f26d6eaf98 http2: log as http to abstract http and http2 a little
This commit logs http2 as an http event. The idea is to somewhat
normalize http/http2 so common info can be version agnostic.

This puts the http2 specific fields in an "http2" object inside
the "http" object.

HTTP2 headers/values that are in common with HTTP1 are logged
under the "http" object to be compatible with HTTP1 logging.
6 years ago
Jason Ish 6a55606adb http2: log headers in the same format as http (1)
Log the headers in request_headers, and response_headers like
http1 to remain compatible.
6 years ago
Philippe Antoine 1422b18a99 http2: initial support 6 years ago
Jeff Lucovsky be57f6d3cc rust/rdp: Fix use of incorrect buffer
This commit updates the connection confirmation parsing function to use
the correct buffer when determining the packet type.
6 years ago
Shivani Bhardwaj 2529eb35e1 smb: use uuid crate 6 years ago
Jason Ish e6668560e0 rust: only run cbindgen if needed
Only run cbindgen when necessary. This is a bit tricky. When
building a dist we want to unconditionally build the headers.

When going through a "make; sudo make install" type process,
cbindgen should not be run as the headers already exist, are
valid, and the environment under sudo is more often than
not suitable to pick up the Rust toolchains when installed
with rustup.

For the normal "make" case we have the gen/rust-bindings.h file
depend on library file, this will cause it to only be rebuilt
if the code was modified.

For "make dist" we unconditionally create "dist/rust-bindings.h".
This means the generated file could be in 2 locations, so update
configure.ac, and the library search find to find it.

The "gen/rust-bindings.h" should be picked up first if it exists,
for those who develop from a dist archive where "dist/rust-bindings.h"
also exists.

Not completely happy having the same file in 2 locations, but not
sure how else to get the dependency tracking correct.
6 years ago
Shivani Bhardwaj 4c7f55e636 dcerpc: handle gap for TCP streams 6 years ago
Jason Ish 6cff558663 dns: add tailing data to split tcp unit test
Add trailing data to the complete payload to test the case
where data is consumed, but still incomplete.
6 years ago
Jason Ish cc154ce152 dns: fix subtraction overflow in debug message
The math for the return value wasn't updated in the debug log
to match the logic and could lead to a subtraction overflow
6 years ago
Philippe Antoine 80ccbff28a dns: return status from parsing tcp request 6 years ago
Philippe Antoine 1296740471 dns: fix incomplete results
and messages parsing whose length is >= 32768
6 years ago
Philippe Antoine 83567e8732 files: rust closes files even on 0 length chunk 6 years ago
Sascha Steinbiss c31360070b rust/mqtt: add MQTT parser 6 years ago
Victor Julien b99ffd9ece eve: remove unused jansson code 6 years ago
Jeff Lucovsky 961b314b86 dns: Remove parser buffering code 6 years ago
Victor Julien 00cc3c7374 eve/ssh: change hassh logging format
Elastic search didn't accept the 'hassh' and 'hassh.string'. It would
see the first 'hassh' as a string and split the second key into a
object 'hassh' with a string member 'string'. So two different types
for 'hassh', so it rejected it.

This patch mimics the ja3(s) logging by creating a 'hassh' object
with 2 members: 'hash', which holds the md5 representation, and
'string' which holds the string representation.
6 years ago
Philippe Antoine eb5b927787 ssh: fixing incomplete kex parsing
We use the record length from the ssh record header,
and not the size of the parsed data, as is done in other places.
6 years ago
Victor Julien bcd416e6ba nfs: fix 'dangling' files in lossy sessions
In case of lossy connections the NFS state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.

This would lead these files staying in the NFS's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.

This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.

Based on 65e9a7c31c
6 years ago
Victor Julien fc7cecb4ae nfs: check post-gap timeouts once a second at most
Based on 25f2efe977
6 years ago
Victor Julien 5d65e5a748 nfs: update ts only if it changed
Based on 8aa380600d
6 years ago
Zach Kelly b25de4d99a rdp: remove parser buffering code 6 years ago
Zach Kelly 22a2bee614 rdp/eve: convert to jsonbuilder 6 years ago
Zach Kelly ef397daba3 rdp: rustfmt (update) 6 years ago
Shivani Bhardwaj 35362b7bfa jsonbuilder: run test if not debug-validate 6 years ago
Shivani Bhardwaj 2ce7d98af1 dcerpc: fix tests to have a valid header 6 years ago
Shivani Bhardwaj 9f9670ebdc logging: Add DCERPC logger 6 years ago
Shivani Bhardwaj bab497ab2c dcerpc: Add multi transaction support
DCERPC parser so far provided support for single transactions only.
Extend that to support multiple transactions.

In order for multiple transactions to work, there is always a
transaction identifier for any protocol in its header that lets a
response match the request. In DCERPC, for TCP, that param is call_id in
the header which is a 32 bit field. For UDP, however since it uses
different version of RPC (4.x), this is defined by serial number field
defined in the header. This field however is not contiguous and needs to
be assembled by the provided serial_low and serial_hi fields.
6 years ago
Shivani Bhardwaj 67e7be633c krb: convert to jsonbuilder
Closes redmine ticket 3754.
6 years ago
Shivani Bhardwaj 72dab0a8b7 snmp: convert to jsonbuilder
Closes redmine ticket 3756.
6 years ago
Jason Ish 7ca24041b6 rust: add doc target to build rust docs
Uses "cargo doc --no-deps" to build the documentation just for
our Suricata package. Without --no-deps, documentation will be
build for all our dependencies as well.

The generated documentation will end up in target/doc as HTML.
6 years ago
Jason Ish abc71dc4a5 applayer template (rust): better gap handling example
In the request parser, show checking if a gap was received
and what one example of trying to continue might look like.
6 years ago
Jason Ish b91bb92b7d applayer template (rust): incomplete support
Show how to use the incomplete AppLayerResult type within the
limits of what the template protocol parser can provide.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3541
6 years ago
Jason Ish 3f615f751b rust app-layer template: add stubs for gap handling 6 years ago
Jason Ish d60671d855 rust/dns: use new flags field to set parser option flags 6 years ago
Jason Ish 53aa967e0b applayer: add flags to parser registration struct
This will allow Rust parsers to register for gap handing from
Rust (some Rust parsers do handle gaps, but they set the flag
from C).
6 years ago
Jason Ish 9d9a901b68 dns: conditional logging
Apply config to newly created response TX.
6 years ago
Victor Julien ac3cf6ff75 detect/config: set config for special cases
Allow app-layer to declare the txs are uni-directional and special
care is needed for applying config.
6 years ago
Victor Julien 5dd4d948d9 app-layer: remove unused detect flags API 6 years ago
Victor Julien c94a5e6392 app-layer/rust: don't use option for GetTxDataFn anymore 6 years ago
Victor Julien 8fe9faecb2 app-layer: remove DetectFlags API. Replaced by AppLayerTxData 6 years ago
Victor Julien 88dd0abb38 rdp: support AppLayerTxData 6 years ago
Victor Julien 9664f73f75 app-layer: remove logged API calls 6 years ago
Victor Julien e0debed0b4 tftp: support AppLayerTxData 6 years ago
Victor Julien 4ff51a0e07 sip: support AppLayerTxData 6 years ago
Victor Julien cc1210c956 ntp: support AppLayerTxData 6 years ago
Victor Julien e0f75157a0 ikev2: support AppLayerTxData 6 years ago
Victor Julien 64e2a27512 applayer/template: support AppLayerTxData 6 years ago
Victor Julien a484bbbe1b dhcp: support AppLayerTxData 6 years ago
Victor Julien 11e2434526 snmp: support AppLayerTxData 6 years ago
Victor Julien 5afe4835ad rfb: support AppLayerTxData 6 years ago
Victor Julien 9f29366c7c krb5: support AppLayerTxData 6 years ago
Victor Julien 2aab1938d7 ssh: support AppLayerTxData 6 years ago
Victor Julien a1e06247a6 dcerpc/udp: support AppLayerTxData 6 years ago
Victor Julien 3202d29325 dcerpc: support AppLayerTxData 6 years ago
Victor Julien 77a95eddd9 smb: support AppLayerTxData 6 years ago
Victor Julien 7a7805cde6 nfs: support AppLayerTxData 6 years ago
Victor Julien fb3bdd8cf3 dns: remove detect_flags and logged now that we use AppLayerTxData 6 years ago
Victor Julien 5665fc8301 app-layer: add ApplyTxConfig API
Optional callback a parser can register for applying configuration
to the 'transaction'. Most parsers have a bidirectional tx. For those
parsers that have different types of transaction handling, this new
callback can be used to properly apply the config.
6 years ago
Victor Julien e15995e2d2 detect: store detect flags in AppLayerTxData 6 years ago
Victor Julien c797c9f09c app-layer: add logger flags to AppLayerTxData 6 years ago
Victor Julien 411f428a38 app-layer: define AppLayerTxData and AppLayerTxConfig
AppLayerTxData is a structure each tx should include that will contain
the common fields the engine needs for tracking logging, detection and
possibly other things.

AppLayerTxConfig will be used by the detection engine to configure
the transaction.
6 years ago
Jason Ish 03efbccfe6 jsonbuilder: set_float, append_float methods
New methods for setting and appending float values.
6 years ago
Philippe Antoine ece29c4210 ssh: fix incomplete return for ssh kex
In the case where we already parsed some records
6 years ago
Philippe Antoine ca6d072297 dcerpc: detect right parsing of empty op version 6 years ago
Emmanuel Thompson 6e5d64f102 detect/asn1: Simplify errors and checks 6 years ago
Emmanuel Thompson 4fc45b5c60 detect/asn1: Update ASN1 struct lifetime
- 'static is only realistic when allocating and leaking it over the
FFI boundary
6 years ago
Emmanuel Thompson 627e90a4bd detect/asn1: Log out errors
- Failure to parse asn1-max-frames
- Failure on asn1 detection checks
6 years ago
Emmanuel Thompson 88601b1993 detect/asn1: Update relative_offset keyword
- To be consistent with recent C version changes
- Add checks for over/underflows
6 years ago
Emmanuel Thompson 7af6cdb7ec detect/asn1: Update asn1 C files to use rust code
Mark rust extern "C" functions as pub in asn1 module to expose via cbindgen
Update detect-asn1.c/h to use rust functions
6 years ago
Emmanuel Thompson 63704fdf13 rust/asn1: Introduce ASN1 rust module
This module uses the `der-parser` crate to parse ASN1 objects in order to replace src/util-decode-asn1.c
It also handles the parsing of the asn1 keyword rules and detection checks performed in src/detect-asn1.c
6 years ago
Emmanuel Thompson 6b8517dc12 rust: Update der, kerberos and snmp parser dependencies
- The update to der-parser allows us to use the latest API changes
6 years ago
Jason Ish 43b9bfaed4 applayer template (rust): convert to JsonBuilder 6 years ago
Vadym Malakhatko 126597144c eve: add Hassh fields to SSH JSON logger and add ssh log condition 6 years ago
Vadym Malakhatko 536cee3ba9 rust/ssh: add hassh generation
Add generation of hassh fingerprints based on fields in the kexinit record
6 years ago
Jeff Lucovsky d5bb41011c output/ikev2: Convert to JsonBuilder
Convert the IKEV2 Json logging to use JsonBuilder.
6 years ago
Victor Julien 65e9a7c31c smb: fix 'dangling' files in lossy sessions
In case of lossy connections the SMB state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.

This would lead these files staying in the SMB's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.

This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.
6 years ago
Victor Julien 25f2efe977 smb: check post-gap timeouts once a second at most 6 years ago
Victor Julien 8aa380600d smb: update ts only if it changed 6 years ago
Jeff Lucovsky 8c5c949cfa output/tftp: Convert to JsonBuilder
This commit converts the TFTP logging mechanisms to JsonBuilder.
6 years ago
Jason Ish 07e88a7479 jsonbuilder: add debug_validate to state
If debug validation is enabled, panic on invalid state errors.

For example, calling close on an already closed jsonbuilder
object.
6 years ago
Jason Ish ca6b70ea1b rust: macro debug_validate_fail to fail with message
Add a new debug_validate macro that unconditionally panics
with a message. Useful in Rust pattern matching.
6 years ago
Shivani Bhardwaj a7535099b4 smb/eve: convert to jsonbuilder
Closes redmine ticket 3712.
6 years ago
Jason Ish 639f3d265e rust: lock to nom 5.1.1
5.1.2 pulls in dependencies that don't build on Rust 1.34.
6 years ago
Victor Julien 79681bf655 app-layer: remove old MPMId API calls
Had been deprecated and non-functional since 2017.
6 years ago
Jason Ish a545cdef6a jsonbuilder: setter for formatted data
Create a method to set preformatted data that contains the key
and the value already formatted.

This is an optimization for static data.
6 years ago
Jason Ish e3b7c58218 jsonbuilder: export {set,append}_string_from_bytes to C 6 years ago
Jason Ish f184bcc10e jsonbuilder: use Box::from_raw instead of transmute to free
I think this is a bad use of transmute, while the end result
is the same, Box::from_raw is more correct as we created this
pointer with Box::into_raw.
6 years ago
Philippe Antoine baf5f52f22 ssh/eve: convert to jsonbuilder 6 years ago
Victor Julien b3b5802c85 eve/nfs: switch output to jsonbuilder 6 years ago
Jason Ish 6ce9b2972b rdp: enable by default
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3255
6 years ago
Jason Ish 5a7ba62493 sip: enable by default
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3256
6 years ago
Jason Ish 36d687580a rfb/eve: convert to jsonbuilder 6 years ago
Jason Ish 60bfbd43fd jsonbuilder: add reset marks
Add methods to get the state of a JsonBuilder (called a mark),
then allow restoring to the mark.
6 years ago
Jason Ish ddb22549be rust: allow some clippy lints without warning
Suppresses some clippy lints that have more to do with style
than anything else, to reduce the amount of noise in the
clippy output.
6 years ago
Jason Ish 03cf3dcd6d dns/eve: convert to jsonbuilder 6 years ago
Jason Ish 6a70d6bb6e sip/eve: convert to jsonbuilder 6 years ago
Jason Ish deed0541bb dhcp/eve: convert to jsonbuilder 6 years ago
Jason Ish 942dd08654 jsonbuilder: new module for generating json
JsonBuilder is a Rust module for creating JSON output. Unlike
Jansson, the final JSON string is built up as items are added,
instead of building up an object tree and rendering it when
done.

The idea is to create a more efficient JSON serializer instead
of a flexible one.
6 years ago
Jason Ish c1f4edc434 rust: bring back libc as a dependency
Its already pulled in by some of other dependencies so adds zero
extra weight, and provides handy definitions for basic functions
like free().
6 years ago
Jason Ish 5513b4ed0b rust/json: expose libjansson json_dumps
This will be temporarily used by JsonBuilder to add the ability
to extend JsonBuilder with Jansson's json_t types.
6 years ago