JsonBuilder is a Rust module for creating JSON output. Unlike
Jansson, the final JSON string is built up as items are added,
instead of building up an object tree and rendering it when
done.
The idea is to create a more efficient JSON serializer instead
of a flexible one.
To signal incomplete data, we must return the number of
consumed bytes. When we get a banner and some records, we have
to take into account the number of bytes already consumed by
the banner parsing before reaching an incomplete record.
All the dead code in C after the Rust implementation is hereby removed.
Invalid/migrated tests have also been deleted.
All the function calls in C have been replaced with appropriate calls to
Rust functions. Same has been done for smb/detect.rs as a part of this
migration.
This parser rewrites the DCE/RPC protocol implementation of Suricata
in Rust. More tests have been added to improve the coverage and some
fixes have been made to the tests already written in C. Most of the
valid tests from C have been imported to Rust.
File anatomy
src/dcerpc.rs
This file contains the implementation of single transactions in DCE/RPC
over TCP. It takes care of REQUEST, RESPONSE, BIND and BINDACK business
logic before and after the data parsing. DCERPCState holds the state
corresponding to a particular transaction and handles all important
aspects. It also defines any common structures and constants required
for DCE/RPC parsing irrespective of the carrier protocol.
src/dcerpc_udp.rs
This file contains the implementation of single transactions in DCE/RPC
over UDP. It takes care of REQUEST and RESPONSE parsing. It borrows the
Request and Response structs from src/dcerpc.rs.
src/detect.rs
This file contains the implementation of dce_iface and opnum detect
keywords. Both the parsing and the matching is taken care of by
functions in this file. Tests have been rewritten with the test data
from C.
src/parser.rs
This file contains all the nom parsers written for DCERPCRequest,
DCERPCResponse, DCERPCBind, DCERPCBindAck, DCERPCHeader, DCERPCHdrUdp.
It also implements functions to assemble and convert UUIDs. All the
fields have their endianness defined unless its an 8bit field or an
unusable one, then it's little endian but it won't make any difference.
src/mod.rs
This file contains all the modules of dcerpc folder which should be
taken into account during compilation.
Function calls
This is a State-wise implementation of the protocol for single
transaction only i.e. a valid state object is required to parse any
record. Function calls start with the app layer parser in C which
detects the application layer protocol to be DCE/RPC and calls the
appropriate functions in C which in turn make a call to these functions
in Rust using FFI. All the necessary information is passed from C to the
parsers and handlers in Rust.
Implementation
When a batch of input comes in, there is an analysis of whether the
input header and the direction is appropriate. Next check is about the
size of fragment. If it is as defined by the header, process goes
through else the data is buffered and more data is awaited. After this,
type of record as indicated by the header is checked. A call to the
appropriate handler is made. After the handling, State is updated with
the latest information about whatever record came in.
AppLayerResult::ok() is returned in case all went well else
AppLayerResult::err() is returned indicating something went wrong.
This commit adds an updated incomplete handling for the RFB-Parser. If
incomplete data is processed, the successfully consumed position and
length of remainder + 1 is returned. If the next packet is not empty
suricata will call the parser again.
This commit is a result of discussion on https://github.com/OISF/suricata/pull/4792.
Addresses #3570 by extra checking of calculated size requests.
With the given input, the parser eventually arrived at
parser::parse_failure_reason() which parsed from the remaining four
bytes (describing the string length) that the failure string to follow
would be 4294967295 bytes long. While calculating the total size of the
data to request via AppLayerResult::incomplete(), adding the four bytes
for the parsed but not consumed string length caused the u32 length to
overflow, resulting in a much smaller value triggering the bug condition.
This problem was addressed by more careful checking of values in each step
that could overflow: one subtraction, one addition (which could overflow
the usize length values), and a final check to determine whether the result
still fit into the u32 values required by AppLayerResult::incomplete().
If so, we would safely convert the values and pass them to the result type.
If not, we simply return AppLayerResult::err() but do not erroneously and
silently request the wrong amount.
This commit adds support for the Remote Framebuffer Protocol (RFB) as
used, for example, by various VNC implementations. It targets the
official versions 3.3, 3.7 and 3.8 of the protocol and provides logging
for the RFB handshake communication for now. Logged events include
endpoint versions, details of the security (i.e. authentication)
exchange as well as metadata about the image transfer parameters.
Detection is enabled using keywords for:
- rfb.name: Session name as sticky buffer
- rfb.sectype: Security type, e.g. VNC-style challenge-response
- rfb.secresult: Result of the security exchange, e.g. OK, FAIL, ...
The latter could be used, for example, to detect brute-force attempts
on open VNC servers, while the name could be used to map unwanted VNC
sessions to the desktop owners or machines.
We also ship example EVE-JSON output and keyword docs as part of the
Sphinx source for Suricata's RTD documentation.
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.
Introduce two macros:
APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)
Update all parsers to use this.
Unfortunately, the transition to nom 5 (and functions instead of macros)
has side-effects, one of them being requiring lots of types annotations
when using a parsing, for ex in a match instruction.
Close all prior transactions in the direction of the GAP, except the
file xfers. Those use their own logic described below.
After a GAP all normal transactions are closed. File transactions
are left open as they can handle GAPs in principle. However, the
GAP might have contained the closing of a file and therefore it
may remain active until the end of the flow.
This patch introduces a time based heuristic for these transactions.
After the GAP all file transactions are stamped with the current
timestamp. If 60 seconds later a file has seen no update, its marked
as closed.
This is meant to fix resource starvation issues observed in long
running SMB sessions where packet loss was causing GAPs. Due to the
similarity of the NFS and SMB parsers, this issue is fixed for NFS
as well in this patch.
Bug #3424.
Bug #3425.
After a GAP all normal transactions are closed. File transactions
are left open as they can handle GAPs in principle. However, the
GAP might have contained the closing of a file and therefore it
may remain active until the end of the flow.
This patch introduces a time based heuristic for these transactions.
After the GAP all file transactions are stamped with the current
timestamp. If 60 seconds later a file has seen no update, its marked
as closed.
This is meant to fix resource starvation issues observed in long
running SMB sessions where packet loss was causing GAPs.
For make clean, only remove gen/ if cbindgen is available.
This prevents make clean from remove gen when the headers
were bundled, but cbindgen is not available to remove them.
Unconditionally remove gen and vendor in maintainerclean.
The modifications as part of the cbindgen commit caused issues
with distcheck, revert the Makefile to how it was with the Python
generator, but still using cbindgen.
Also always assume we'll include the generated headers in the
distribution archive to fix make distcheck from distribution
archives with headers included, but no cbindgen.
If sources are vendored, we get the same effect of using frozen
with a lock file, and the Cargo.lock is generated based
on the vendored sources.
This also removes the need to ship a Cargo.lock.
Fixed out of source builds with vendored sources.
Rust 1.40 in strict mode will now fail the build on the
presence of unnecessary parentheses.
warning: unnecessary parentheses around type
--> src/smb/smb2_ioctl.rs:41:12
|
41 | -> (&mut SMBTransaction)
| ^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
|
= note: `#[warn(unused_parens)]` on by default
Since ebcc4db84a the flow worker runs
file pruning after parsing, detection and loging. This means we can
simplify the pruning logic. If a file is in state >= CLOSED, we can
prune it. Detection and outputs will have had a final chance to
process it.
Remove the calls to the pruning code from Rust. They are no longer
needed.
If rustup is in use, and a user uses sudo or su for the make
install, the install may fail with a "no default toolchain"
error.
To prevent this, detect at configure if rustup is being used,
then set RUSTUP_HOME for all calls to cargo.
Add a rule keyword, dns.opcode to match on the opcode flag
found in the DNS request and response headers.
Only exact matches are allowed with negation.
Examples:
- dns.opcode:4;
- dns.opcode:!1;
Get rid of enable-rust-debug flag and use enable-debug for acheiving the
desired functionality. From now, adding `--enable-debug` to `configure`
shall create an [unoptimitized + debuginfo] target. Rest behavior stays
the same.
Closes redmine ticket #3054
Fix the following warnings by compiler,
(1) warning: use of deprecated item 'take_until_s': Please use `take_until` instead
(2) warning: `...` range patterns are deprecated
For the second warning, the builtin lint
"ellipsis_inclusive_range_pattern" has been added which causes the
following warning to show up with rustc 1.24.
warning: unknown lint: `ellipsis_inclusive_range_patterns`
--> /home/travis/build/OISF/suricata/suricata-5.0.0-dev/rust/src/lib.rs:18:10
|
18 | #![allow(ellipsis_inclusive_range_patterns)]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: #[warn(unknown_lints)] on by default
Since there is no other way to fix this, the above warning shall stay.
We need to take care of modifying this if and when the support for 1.24
as MSRV is dropped.
rustc 1.36 introduced:
error: variable does not need to be mutable
--> src/dhcp/parser.rs:202:17
|
202 | let mut malformed_options = false;
| ----^^^^^^^^^^^^^^^^^
| |
| help: remove this `mut`
|
note: lint level defined here
--> src/lib.rs:18:38
|
18 | #![cfg_attr(feature = "strict", deny(warnings))]
| ^^^^^^^^
= note: #[deny(unused_mut)] implied by #[deny(warnings)]
error: aborting due to previous error
error: Could not compile `suricata`.
Ticket #3072.
Extend the Rust parsing infrastructure with the "get event info by id"
calls. This changeset extends the parser structure, the C-based
registration handlers and the template parser.
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
Make sure they are valid 8 bit integers before combining the
two parts into a u16 to prevent an overflow of the u16
return value.
Add unit tests to check parsing of invalid ports.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2904
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
The `set_event` function requires that the transaction is already
inserted, or the event set is silently lost.
When parsing first IKEv2 message, first insert transaction, prepare
values, and borrow back inserted transaction to update it.
The C header generation script was failing with a unicode error
in Python 3 on FreeBSD. Fix the reading of files to properly
handle unicode in all Python 3 environments.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2794
Only use ssn_id and msg_id for mapping a response to a request.
By not using the tree_id it can always be included in the tx.hdr which
means it can be logged properly in case of IOCTL and DCERPC.
Also, when requesting the query, if the request doesn't exist,
return the query from the response. This makes it behave
more like C implementation.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2730
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
In case of packet loss during an in-progress chunk the file tracker
could loose track of a file because it couldn't map the XID to a
file handle.
The file tracker would then panic if a new file was opened, as
it noticed the last chunk wasn't yet complete.
This patch tracks the file handle for a in-progress chunk in the
state, just like the tracking of the size that is left.
Bug #2717
As the generated Cargo.toml is shipped as part of a release
tarball, build from the source directory but set the cargo
CARGO_TARGET_DIR to the build directory.
If we missed the tree connect we can't know for sure if we're
reading from a (DCERPC) PIPE or not. In this case probe the data
to see if it looks like DCERPC.
If the detection succeeds, use a special 'suricata::dcerpc' service
in the TX.
Simplify handling of DCERPC records that cross records
Update logging for the response only TXs.
The mismatch between the types would randomly lead to the return code
of the Rust parser to be not correctly handled over the C/Rust
boundary. This would lead to the API considering a parser to be in
error state when it was not.
The protocol is a simple request/reply based protocol that can
be hand driven with netcat.
Request -> 12:Hello World!
Response -> 3:Byte
Its of the format <length>:<message> where length is the length
of the message, not including the length or the delimiter.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
Jason Ish
Victor Julien
Philippe Antoine
Shivani Bhardwaj
frank honza
Sascha Steinbiss
Pierre Chifflier
Danny Browning
Ciprian
Jeff Lucovsky
Giuseppe Longo
Zach Kelly
Jason Ish
Wesley van der Ree
Maurizio Abba