After a file has been closed (CLOSE, COMMIT command or EOF/SYNC part of
READ/WRITE data block) mark it as such so that new file commands on that
file do not reuse the transaction.
When a file transfer is completed it will be flagged as such and not be
found anymore by the NFSState::get_file_tx_by_handle() method. This forces
a new transaction to be created.
Based on the Rust clippy lint that recommends that any public
function that dereferences a raw pointer, mark all FFI functions
that reference raw pointers with build_slice and cast_pointer
as unsafe.
This commits starts by removing the unsafe wrapper inside
the build_slice and cast_pointer macros then marks all
functions that use these macros as unsafe.
Then fix all not_unsafe_ptr_arg_deref warnings from clippy.
Fixes clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#not_unsafe_ptr_arg_deref
All cases of our transmute can be replaced with more idiomatic
solutions and do no require the power of transmute.
When returning an object to C for life-time management, use
Box::into_raw to convert the boxed object to pointer and use
Box::from_raw to convert back.
For cases where we're just returning a pointer to Rust managed
data, use a cast.
Registering parsers in Rust requires signatures to be a certain way and
compatible with C. Change signatures of all the functions.
Probe fn has also been changed to return AppProto as required by the new
fn signature.
Before, even if there were no outputs, all the arguments
were evaluated, which could turn expensive
All variables which are used only in certain build configurations
are now prefixed by underscore to avoid warnings
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.
Update all parsers to take advantage of this.
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2
This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
In case of lossy connections the NFS state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.
This would lead these files staying in the NFS's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.
This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.
Based on 65e9a7c31c
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.
Introduce two macros:
APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)
Update all parsers to use this.
Close all prior transactions in the direction of the GAP, except the
file xfers. Those use their own logic described below.
After a GAP all normal transactions are closed. File transactions
are left open as they can handle GAPs in principle. However, the
GAP might have contained the closing of a file and therefore it
may remain active until the end of the flow.
This patch introduces a time based heuristic for these transactions.
After the GAP all file transactions are stamped with the current
timestamp. If 60 seconds later a file has seen no update, its marked
as closed.
This is meant to fix resource starvation issues observed in long
running SMB sessions where packet loss was causing GAPs. Due to the
similarity of the NFS and SMB parsers, this issue is fixed for NFS
as well in this patch.
Bug #3424.
Bug #3425.
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
In case of packet loss during an in-progress chunk the file tracker
could loose track of a file because it couldn't map the XID to a
file handle.
The file tracker would then panic if a new file was opened, as
it noticed the last chunk wasn't yet complete.
This patch tracks the file handle for a in-progress chunk in the
state, just like the tracking of the size that is left.
Bug #2717
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
READ replies with large data chunks are processed partially to avoid
queuing too much data. When the final chunk was received however, the
start of the chunk would already tag the transaction as 'done'. The
more aggressive tx freeing that was recently merged would cause this
tx to be freed before the rest of the in-progress chunk was done.
This patch delays the tagging of the tx until the final data has been
received.
Avoid looping in transaction output.
Update app-layer API to store the bits in one step
and retrieve the bits in a single step as well.
Update users of the API.
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
Victor Julien
Jason Ish
Shivani Bhardwaj
Philippe Antoine
Jeff Lucovsky
Juliana Fajardini
Pierre Chifflier
Jason Ish
Nick Price