Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
Some protocols transfer multiple files in parallel. To support this add
a 'track id' to the API. This track id is set by the protocol parser. It
will use this id to indicate what file in the FileContainer it wants to
act on.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
This patch introduces the FileDataSize and FileTrackedSize functions.
The first one is just a renaming of the initial FilSize function
whereas the other one is using the newly introduced size field as
value.
The file size returned by FileSize is invalid if file store is not
used so we introduce a new size field in File structure that is used
to store the size.
When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.
This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
Moved and adapted code from detect-filemd5 to util-detect-file-hash,
generalised code to work with SHA-1 and SHA-256 and added necessary
flags and other constants.
Make the file storage use the streaming buffer API.
As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
When the file API is used to do content inspection (currently only
smtp does this), the detection should be considered while pruning
the file chunks.
This patch introduces a new flag for the file API: FILE_USE_DETECT
When it is used, 'FilePrune' will not remove chunks that are (partly)
beyond the File::content_inspected tracker.
When using this flag, it's important to realize that when the detect
engine is disabled or rules are not matching, content_inspected
might not get updated.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
The filestore keyword until now flagged a file, tx or ssn for storage as soon
as the keyword was inspected. This happens before flowbits and some other
keywords, so files were stored that weren't supposed to.
This patch makes the filestore keyword fill an array in the detect engine
thread ctx. Then if the full signature matches, a post-match filestore
function makes the store final.
Filestore keyword by default (... filestore; ... ) marks only the file in the
same direction as the rule match for storing. This makes sense when inspecting
individual files (filemagic, filename, etc) but not so much when looking at
suspicious file requests, where the actual file is in the response.
The filestore keyword now takes 2 optional options:
filestore:<direction>,<scope>;
By default the direction is "same as rule match", and scope is "currently
inspected file".
For direction the following values are possible: "request" and "to_server",
"response" and "to_client", "both".
For scope the following values are possible: "tx" for all files in the current
HTTP/1.1 transation, "ssn" and "flow" for all files in the session/flow.
For the above case, where a suspious request should lead to a response file
download, this would work:
alert http ... content:"/suspicious/"; http_uri; filestore:response; ...
Eric Leblond
Victor Julien
Giuseppe Longo
Duarte Silva
Tom DeCanio
Giuseppe Longo
Anoop Saldanha