When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
The `set_event` function requires that the transaction is already
inserted, or the event set is silently lost.
When parsing first IKEv2 message, first insert transaction, prepare
values, and borrow back inserted transaction to update it.
The C header generation script was failing with a unicode error
in Python 3 on FreeBSD. Fix the reading of files to properly
handle unicode in all Python 3 environments.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2794
Only use ssn_id and msg_id for mapping a response to a request.
By not using the tree_id it can always be included in the tx.hdr which
means it can be logged properly in case of IOCTL and DCERPC.
Also, when requesting the query, if the request doesn't exist,
return the query from the response. This makes it behave
more like C implementation.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2730
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
In case of packet loss during an in-progress chunk the file tracker
could loose track of a file because it couldn't map the XID to a
file handle.
The file tracker would then panic if a new file was opened, as
it noticed the last chunk wasn't yet complete.
This patch tracks the file handle for a in-progress chunk in the
state, just like the tracking of the size that is left.
Bug #2717
As the generated Cargo.toml is shipped as part of a release
tarball, build from the source directory but set the cargo
CARGO_TARGET_DIR to the build directory.
If we missed the tree connect we can't know for sure if we're
reading from a (DCERPC) PIPE or not. In this case probe the data
to see if it looks like DCERPC.
If the detection succeeds, use a special 'suricata::dcerpc' service
in the TX.
Simplify handling of DCERPC records that cross records
Update logging for the response only TXs.
The mismatch between the types would randomly lead to the return code
of the Rust parser to be not correctly handled over the C/Rust
boundary. This would lead to the API considering a parser to be in
error state when it was not.
The protocol is a simple request/reply based protocol that can
be hand driven with netcat.
Request -> 12:Hello World!
Response -> 3:Byte
Its of the format <length>:<message> where length is the length
of the message, not including the length or the delimiter.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
Victor Julien
Wesley van der Ree
Pierre Chifflier
Jason Ish
Maurizio Abba