Add API calls to upgrade to TLS or to request a protocol change
without a specific protocol expectation.
If the HTTP CONNECT session includes a port on the url, use that to
look up the probing parser during protocol detection. Solves a
missed detection of a SSLv2 session that upgrades to TLSv1. SSLv2
relies on the probing parser which is limited to certain ports.
In case of STARTTLS in SMTP and FTP, the port is hardcoded to 443.
A new event APPLAYER_UNEXPECTED_PROTOCOL is set if there was a
mismatch.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.
For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.
This was obviously inefficent, so this patch changes the logic.
The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.
Implemented for HTTP, TLS and SSH.
In HTTP detection registered patterns were upper case only. Since the
detection is based on both sides this would still work for sessions
where one of the talkers misbehaved. If both sides misbehave this
would fail however, so this patch introduces case insensive matching.
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.
In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.
Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
Change AppLayerParserRegisterGetStateProgressCompletionStatus to
only store one ProgressCompletionStatus callback function for each
alproto, instead of storing one for each ipproto.
This enables us to use AppLayerParserGetStateProgressCompletionStatus
in functions where we do not know the ipproto used.
The stateful detection engine needs some assistance when inspecting
transactions with multiple files. This patch flags the detect state
(if any) about the availability of new files in http. For http it
should only apply to multipart bodies although the flag is set for
all files.
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.
Per protocol, 2 commandline options are added:
--afl-http-request=<filename>
--afl-http=<filename>
In the former case, the contents of the file are passed directly to
the HTTP parser as request data.
In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
In some conditions, if stream.reassembly.depth is greater than
request/response-body-limit size, the logging output is wrong
if filestore keyword is used with http.
For example, we get:
{... "app_proto":"http","fileinfo":{"filename":"\/file.pdf","state":"CLOSED","stored":false,"size":1049292,"tx_id":0}}
"state":"CLOSED","stored":false should be "state":"TRUNCATED","stored":true.
This happens because the file state and file flags,
which is the information that determine a correct output,
are not set properly since a file is logged before and then closed (HTPFileClose).
The logic of this patch is to close a file when we are above
the limits, such that the proper state and flags can be set
and the file will be logged correctly.
Optimize HTTP multipart body parsing. Big records that were not files
could slow down Suricata. The reason was that the body tracker was not
moved forward. This lead to growing body buffers, which were expensive
wrt memory and inspection.
This patch add logic to move the tracker forward in this case.
In case the body wasn't inspected the body_inspected variable wouldn't
get updated leading to the body not getting pruned at all.
This patch adds support for this case.
It's not uncommon to see an header like:
X-Forwarded-For: 1.2.3.4:56789
This patch recognizes this case and ignores the port. It also supports
this for IPv6 if the address has the following notation:
X-Forwarded-For: [12::34]:1234
This patch also adds unittests.
On receiving TCP end of stream packets (e.g. RST, but also sometimes FIN
packets), in some cases the AppLayer parser would not be notified. This
could happen in IDS mode, but would especially be an issue in IPS mode.
This patch changes the logic of the AppLayer API to handle this. When no
new data is available, and the stream ends, the AppLayer API now gets
called with a NULL/0 input, but with the EOF flag set.
This allows the AppLayer parser to call it's final routines still in the
context of a real packet.
Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.
The comments have also been changed to reflect the desired style.
1) Reworked pattern registration for http methods and versions.
Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.
In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.
2) Modified pattern registration after feedback.
Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.
Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
Victor Julien
Mats Klepsland
Giuseppe Longo
Jason Ish
Giuseppe Longo
maxtors