The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.
This patch simply removes the offending code.
Reported-by: The Yahoo pentest team
By assuming that HTPCallbackRequestLine would always be run first,
an memory leak was introduced. It would not check if user data already
existed in the tx, causing it to overwrite the user data pointer is
it already existed.
Bug #1092.
The HTP config tree is a radix. The lookups are updated to the new API.
The return of user_data is treated as a succesful lookup, instead of
the node itself.
This patch introduces wrapper functions around allocation functions
to be able to have a global HTP memcap. A simple subsitution of
function was not enough because allocated size needed to be known
during freeing and reallocation.
The value of the memcap can be set in the YAML and is left by default
to unlimited (0) to avoid any surprise to users.
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.
AppLayerParserState is declared in flow.h as it's part of the Flow
structure.
AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
[src/app-layer-htp.c:1967] -> [src/app-layer-htp.c:1978]: (warning) \
Possible null pointer dereference: tx - otherwise it is redundant \
to check it against null.
The meta-field-option allows for setting the hard limit of request
and response fields in HTTP. In requests this applies to the request
line and headers, not the body. In responses, this applies to the
response line and headers, not the body.
Libhtp uses a default limit of 18k. If this is reached an event is
raised.
Ticket 986.
Strip the 'proxy' parts from the normalized uri as inspected by http_uri,
urilen, pcre /U and others.
In a request line like:
GET http://suricata-ids.org/blah/ HTTP/1.1
the normalized URI will now be:
/blah/
This doesn't affect http_raw_uri. So matching the hostname, etc is still
possible through this keyword.
Additionally, a new per HTTP 'personality' option was added to change
this behavior: "uri-include-all":
uri-include-all: <true|false>
Include all parts of the URI. By default the
'scheme', username/password, hostname and port
are excluded. Setting this option to true adds
all of them to the normalized uri as inspected
by http_uri, urilen, pcre with /U and the other
keywords that inspect the normalized uri.
Note that this does not affect http_raw_uri.
So adding uri-include-all:true to all personalities in the yaml will
restore the old default behavior.
Ticket 1008.
Libhtp decodes the + character in the query string to a space by default.
Suricata rules (e.g. etpro sid 2806767) are expecting to see the space in
the http_uri buffer.
Added an option per htp config to reenable this default behavior:
query-plusspace-decode: yes
Bug #1035.
This patch is a result of applying the following coccinelle
transformation to suricata sources:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc|SCMallocAligned|SCRealloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
In case of recursive call to protocol detection from within protocol
detection, and the recursively invoked stream still hasn't been ack'ed
yet, protocol detection doesn't take place. In such cases we will end up
still calling the app layer with the wrong direction data. Introduce a
check to not call app layer with wrong direction data.
When sockets are re-used reset all relevant vars correctly.
This commit fixes a bug where we were not reseting app proto detection
vars.
While fixing #989, we discovered some other bugs which have also been
fixed, or rather some features which are now updated. One of the feature
update being if we recieve wrong direction data first, we don't reset the
protocol values for the flow. We let the flow retain the detected
values.
Unittests have been modified to accomodate the above change.
Some events we retrieved from error messages are flag now, so check
those. Not all can be converted though. These are no longer set:
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE
HTTP_DECODER_EVENT_INVALID_AUTHORITY_PORT
Part of Bug #982.
1. Proto detection
2. Parsers
For app layer protocols.
libhtp has now been moved to the section under app-layer.protocols.http,
but we still provide backward compatibility with older conf files.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.
To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.
HTP, DNS, SMB, DCERPC parsers updated.
Now depth is kept in mind when we inspect chunks in client/server body.
This takes care of FPs originating from inspecting subsequent chunks that
match with depth, but shouldn't.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
As reported in bug #688, htp_config_set_path_decode_u_encoding
function is not included in libhtp header before 0.3.0. Result
is that suricata compilation fail with an external htp library.
The following patch detect the issue and adds the missing
declaration.
The power of libhtp customisation now available to users.
Options available -
path-backslash-separators: yes
path-compress-separators: yes
path-control-char-handling: none
path-convert-utf8: yes
path-decode-separators: yes
path-decode-u-encoding: yes
path-invalid-encoding-handling: preserve_percent
path-invalid-utf8-handling: none
path-nul-encoded-handling: none
path-nul-raw-handling: none
set-path-replacement-char: ?
set-path-unicode-mapping: bestfit
You can use this for your libhtp customisation. Options explained in our
wiki.
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Advanced_libhtp_customization
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
This patch converts the series of variable to an atomic.
Furthermore, as the callbacks are now always run, it is not
necessary anymore to refuse a ruleswap if HTP parameters are
changing.
maxtors
Victor Julien
Ken Steele
sxhlinux
Eric Leblond
Anoop Saldanha
Last G
Anoop Saldanha