warning: extern declarations without an explicit ABI are deprecated
--> src/core.rs:72:1
|
72 | extern {
| ^^^^^^ help: explicitly specify the "C" ABI: `extern "C"`
|
= note: `#[warn(missing_abi)]` on by default
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Feature 5647
email.x_mailer matches on MIME EMAIL X-Mailer
This keyword maps to the EVE field email.x_mailer
It is a sticky buffer
Supports prefiltering
Ticket: #7598
email.message_id matches on MIME EMAIL Message-Id
This keyword maps to the EVE field email.message_id
It is a sticky buffer
Supports prefiltering
Ticket: #7593
Allow for more efficient rules that 'prefilter' on flowbits with 'isset' logic.
This prefilter is enabled by default, which means that if no mpm is present or
no explicit prefilter is used, the flowbits prefilter will be set up for a rule.
flowbits 'isset' prefilter
For rules that have a 'flowbits:isset,<bit>' statement, a "regular" prefilter
facility is created. It means that the rules are removed from the normal
match list(s) and added to a prefilter engine that runs prior to the individual
rule inspection stage.
Implementation: the prefilter is implemented as an RB_TREE of flowbits, with the
rule id's they "enable" stored per tree node. The matching logic is walking the
list of bits set in the flow and looking each of them up in the RB_TREE, adding
the rule ids of each of the matching bits to the list of rule candidates.
The 'isset' prefilter has one important corner case, which is that bits can in
fact be set during the rule evaluation stage. This is different from all other
prefilter engines, that evaluate an immutable state (for the lifetime of the
packet inspection).
flowbits 'set' post-match prefilter
For flowbits 'set' action, special post-match 'prefilter' facilities deal with
this corner case. The high level logic is that these track which 'isset' sigs
depend on them, and add these dependencies to the candidates list when a 'set'
action occurs.
This is implemented in a few steps:
1. flowbits 'set' is flagged
2. when 'set' action occurs the flowbit is added to a "post rule
match work queue"
3. when the rule evaluation ends, the post-match "prefilter" engine is run
on each of the flowbits in the "post rule match work queue"
4. these engines ammend the candidates list with the rule id dependencies
for the flowbit
5. the candidates list is sorted to make sure within the execution for that
packet the inspection order is maintained
Ticket: #2486.
Add support for special post-match engines. This allows a rule to enable
other rules when it matches.
Implementation is similar to prefilter engines, however prefilter
engines run before individual rules while this post-match engine runs
after and individual rule match. It will then add the new rules to the
existing rule list.
Ticket: 5634
Allows to share the same validator functions when only the buffer
id is changing like for urilen, while still accessing the buffer
name for error logs
This adds a sticky (multi) buffer to match the "Connection data"
subfield of the "Media description" field in both requests and
responses.
Ticket #7291
This adds a stick (multi) buffer to match the "Session information"
subfield of the "Media description" field in both requests and
responses.
Ticket #7291
The current parser implementations take a field, such as connection data, and
split it into subfields for a specific structure (e.g., struct ConnectionData).
However, following this approach requires several sticky buffers to match the
whole field, which can make a rule a bit verbose and doesn't offer any advantage
for matching specific parts of a field.
With this patch, a single line is still split into pieces if it makes sense for
parsing purposes, but these pieces are then reassembled into a single string.
This way, only one sticky buffer is needed to match the entire field.
Ticket #7291
As spotted by cppcheck
fd is set only once, and DetectFlowSetup bails early if it is NULL
Fixes: c272a646c5 ("detect: SigMatchAppendSMToList can fail")
In thread startup, return error of TmThreadsWaitForUnpause() fails.
Fixed coverity check as in all other places the return value of this
function is checked and acted on.
This commits adds documentation for the entropy keyword.
The entropy keyword calculates the Shannon entropy value for content
with the calculated value used to determine whether an alert occurs.
Philippe Antoine
Todd Mortimer
Shivani Bhardwaj
Jason Ish
Alice Akaki
Victor Julien
Giuseppe Longo
dependabot[bot]
Jeff Lucovsky