Commit Graph

8196 Commits (e62c75335ee760dcaadd496e95284b5f954d0727)

Author SHA1 Message Date
Victor Julien 4ae85f16c5 detect/replace: fix mem leak in error path 7 years ago
Victor Julien 0d0da7880e isdataat: fix mem leak in error path 7 years ago
Victor Julien 2ef2febc7c bits: avoid memory leak in case of adding types 7 years ago
Victor Julien 5c69bbb236 ipproto: fix memleak in error case 7 years ago
Victor Julien 5283796018 bytetest: don't leak memory in error condition 7 years ago
Victor Julien 9d54a8361d yaml: fix potential memleak and suppress coverity issue 7 years ago
Victor Julien 15e0f7f5bb outputs: fix memleaks in the error paths reported by coverity 7 years ago
Victor Julien 575fb69a06 coverity: suppress warning for intentional code 7 years ago
Victor Julien 335df629a2 rust/dns: don't compile unused C code if Rust is enabled 7 years ago
Victor Julien f97bf298b2 coverity: fix filestore v2 memleak 7 years ago
Victor Julien 00111499b0 log-pcap: fix coverity memleak warning 7 years ago
Victor Julien a683279137 coverity: don't warn on fall back random 7 years ago
Mats Klepsland a8347e1bc2 app-layer-ssl: fix flow and inspection bypass for TLSv1.3 7 years ago
Mats Klepsland 3b73b7d542 app-layer-ssl: add 0-RTT support for TLSv1.3 7 years ago
Mats Klepsland 23993c18cd app-layer-ssl: decode early data extension in ClientHello record
Decode early data extension used by 0-RTT that is used to indicate that
application data will be sent right after the ClientHello record.
7 years ago
Mats Klepsland 7556004a51 app-layer-ssl: use extension length when decoding extensions
Pass extension length to functions decoding extensions, instead of
passing the length left in the record. This enables us to also
decode empty extensions.
7 years ago
Mats Klepsland ee1de4c812 app-layer-ssl: handle all versions above TLSv1.2 as TLSv1.3
This makes it more likely to log custom versions of TLSv1.3 that
doesn't comply with the draft version numbering.
7 years ago
Victor Julien 31b87d5f8f tls: remove debug printfs 7 years ago
Mats Klepsland 16643befe7 detect-tls-ja3-hash: add another unit test
Add unit test that covers the JA3 bug in TLS extensions decoding.
7 years ago
Mats Klepsland fc53b2ecd5 app-layer-ssl: fix JA3 bug in TLS extension decoding 7 years ago
Mats Klepsland 89bd274f44 app-layer-ssl: fix JA3 bug in TLS version decoding 7 years ago
Victor Julien 215e37a9c0 capture: multidev is not experimental 7 years ago
Victor Julien 631ee383bb flow/stream: 'wrong thread' as stream event & counter
Set event at most once per flow, for the first 'wrong' packet.

Add 'tcp.pkt_on_wrong_thread' counter. This is incremented for each
'wrong' packet. Note that the first packet for a flow determines
what thread is 'correct'.
7 years ago
Victor Julien 588a56c8ba smtp: fix clang -Wunreachable-code warning
app-layer-smtp.c:756:12: error: will never be executed [-Werror,-Wunreachable-code]
    return 0;
           ^
1 error generated.
7 years ago
Victor Julien 17e7d179d0 profiling/csv: update output format
Update output to be:

pcap_cnt,total,receive,decode,flowworker,threading,proto detect,flow,
stream,app-layer,detect,tcp-prune,loggers,<detect stages>,<loggers>

For TCP, the app-layer cost is not part of stream anymore.
7 years ago
Victor Julien 6781146556 test mode: parse interface list in test mode 7 years ago
Victor Julien 77c7cf0211 windivert: fix whitespace issue 7 years ago
Victor Julien b86e176262 af-packet: suppress noisy info message 7 years ago
Victor Julien d6460392c5 detect/transforms: fix doc urls 7 years ago
Victor Julien 57921d95d6 eve: improve error handling for Rust loggers
Give useful warning message if Rust is not compiled in.
7 years ago
Victor Julien 24b18e47bd pfring: minor cleanups 7 years ago
Victor Julien 4f84672d7c stats: decoder/stream events as stats 7 years ago
Victor Julien 014056f686 detect/analyzer: add detection for sigs that could use http keywords 7 years ago
Victor Julien 127937b2dd detect/analyzer: add debug statements 7 years ago
Victor Julien c05459ce89 detect/analyzer: fix json analyzer being called on incomplete rules 7 years ago
Victor Julien 6c97909a92 stream/events: log as stats 7 years ago
Victor Julien fa06879563 detect/events: cleanup keyword 7 years ago
Victor Julien 2ae8d1a208 cocci/detect: add flags check to SigTableElmt 7 years ago
Victor Julien ecb5d6419b rules/transform: add to list-keywords 7 years ago
Mats Klepsland 81cdcd315b detect-ssh-software: fix url for keyword 7 years ago
Mats Klepsland 08efbdc632 detect-ssh-software-version: add description and url to keyword 7 years ago
Mats Klepsland f4da3050f2 detect-ssh-proto-version: add description and url to keyword 7 years ago
Mats Klepsland c58252bb3b detect-ssh-proto: fix url for keyword 7 years ago
Victor Julien daaa90d515 rust/smb: suppress noisy messages 7 years ago
Victor Julien c4d8508f51 eve/json: introduce community flow id
Add support for community flow id, meant to give a records a
predictable flow id that can be used to match records to
output of other tools.

Takes a 'seed' that needs to be same across sensors and tools
to make the id less predictable.
7 years ago
Victor Julien e956b484c5 eve/json: handle common options in central function 7 years ago
Victor Julien df1ec82b55 eve/json: move common settings into it's own struct 7 years ago
Victor Julien 116c03cf17 nfs: use common json output structures 7 years ago
Victor Julien 04edc7cb6c smb: use common json output structures 7 years ago
Victor Julien 8b8270e732 eve/json: add common helper funcs
Add simple helper funcs for option-less loggers
7 years ago
Victor Julien f357ad1df2 eve/flow: minor cleanups 7 years ago
Victor Julien 7bf71805b8 hash/sha1: optimize by avoiding mem alloc
Don't allocate an output buffer for each call. These buffers
would have the exact same size every time.
7 years ago
Victor Julien efbb5ce0fe afpacket: fix formatting of errors 7 years ago
Victor Julien 8d5da9e00f dns: shrink per flow state by improving layout 7 years ago
Victor Julien 275cf9b029 detect/ttl: major clean up of ttl code
Redo unittests using FAIL/PASS macros
Switch parsing to pcre_copy_substring.
Misc cleanups.
7 years ago
Victor Julien 13ea30ef23 spelling: fixing minor spelling mistakes 7 years ago
Danny Browning a307e637c6 suricata: file existence check (bug #2615)
Files and directories passed via command line option -r should be checked for
existence during command line parsing and not start additional suricata
functionality.
7 years ago
Mats Klepsland 8c3f1aa7a5 tlslog: don't log as "resumed" without ServerHello
Don't log a session as "resumed" if a ServerHello record has not been
seen. This makes sure that incomplete TLS sessions where the ClientHello
contains a session ticket, is not logged as a session resumption.
7 years ago
Mats Klepsland 814e1624c2 output-json-tls: don't log as "resumed" without ServerHello
Don't log a session as "resumed" if a ServerHello record has not been
seen. This makes sure that incomplete TLS sessions where the ClientHello
contains a session ticket, is not logged as a session resumption.
7 years ago
Mats Klepsland 4470b05ae4 app-layer-ssl: remove unnecessary length check
We already check that empty extensions are not decoded, so this length
check is not needed.
7 years ago
Jason Ish 35fd10bc2e rust: app-layer detect template for rust parsers 7 years ago
Jason Ish c3f1a35e28 rust: app-layer template parser and logger
The protocol is a simple request/reply based protocol that can
be hand driven with netcat.

Request  -> 12:Hello World!
Response -> 3:Byte

Its of the format <length>:<message> where length is the length
of the message, not including the length or the delimiter.
7 years ago
Jason Ish ee3aba9008 templates: C stub output for Rust logger 7 years ago
Jason Ish 96dc20abb1 templates: C stub template for Rust parser 7 years ago
Victor Julien 486054595a detect/template2: template with prefilter (copy of ttl) 7 years ago
Victor Julien 4d0fc67560 decode/template: minor updates 7 years ago
Victor Julien a013cece69 app-layer/template: code cleanups 7 years ago
Victor Julien 33914c2f2f detect/template: clean up packet keyword 7 years ago
Victor Julien d3e5c15995 detect/template: move test to own file in src/tests/ 7 years ago
Victor Julien 1bb8fcecec detect/template: switch to v2 API, add MPM 7 years ago
Victor Julien 234d113838 detect/template: clean up unittest 7 years ago
Jacob Masen-Smith b1b45a54c5 detect/analyzer: disable automatic json output
EngineAnalysisRules2 was in a strange location where it did not respect
the --engine-analysis flag. It has been moved to the same call location
as EngineAnalysisRules.
7 years ago
Victor Julien 64d75496b8 detect/analyzer: add notes (and warnings) 7 years ago
Victor Julien e02b74dee7 http: implement min size stream logic
Update HTTP parser to set the min inspect depth per transaction. This
allows for signatures to have their fast_pattern in the HTTP body,
while still being able to inspect the raw stream reliably with it.

The inspect depth is set per transaction as it:
- depends on the per personality config for min inspect size
- is set to the size of the actual body if it is smaller

After the initial inspection is done, it is set to 0 which disables
the feature for the rest of the transaction.

This removes the rescanning flush logic in commit
7e004f52c6 and provides an alternative
fix for bug #2522. The old approach caused too much rescanning of
HTTP body data leading to a performance degradation.

Bug #2522
7 years ago
Victor Julien 7186ce7b99 stream: introduce min inspect depth logic
Some rules need to inspect both raw stream data and higher level
buffers together. When this higher level buffer is a streaming
buffer itself, the risk of mismatch exists.

This patch allows an app-layer parser to set a 'min inspect depth'.
The value is used by the stream engine to keep at least this
depth worth of data, so that the detection engine can request
all of it for inspection.

For rules that have the SIG_FLAG_FLUSH flag set, data is inspected
not from offset raw_progress, but from raw_progress minus
min_inspect_depth.

At this time this is only used for sigs that have their fast_pattern
in a HTTP body and have raw stream match as well.
7 years ago
Jason Ish 9b86c7c5c0 defrag: break out of loop in linux profile when able to 7 years ago
Jason Ish aa98678662 defrag: remove fragments that have complete overlap
Instead of just marking fragments that have been completely
overlapped and won't be part of the assembled packet, remove
them from the fragment tree when detected.
7 years ago
Jason Ish fe6e96a8c1 defrag: use rb tree to store fragments 7 years ago
Victor Julien 023a2fe9ab unittests: fix format-truncation warning 7 years ago
Victor Julien 269313a53e stream/segments: change packing to reduce size
Change the way fields are ordered to reduce TcpSegment structure
with 8 bytes.
7 years ago
Victor Julien b6b9b56e45 stream/segments: keep track of tree right edge
Use this in places where we need to use the outer right
edge of our sequence space.

This way we can avoid walking the tree to find this, which
is a potentially expensive operation.
7 years ago
Victor Julien ea771c69af streaming/sbb: convert RB_MIN to 'head' 7 years ago
Victor Julien bbf1f78ffe streaming: keep track of tree 'head' 7 years ago
Victor Julien 450500e667 streaming: use rbtree for stream blocks
Switch StreamBufferBlocks implementation to use RBTREE instead of
a list. This makes inserts/removals and lookups a lot cheaper if
the number of data gaps is large.

Use separate compare functions for inserts and regular lookups.
Inserts care about the offset, while lookups care about the blocks
right edge as well.
7 years ago
Victor Julien 9bda558c59 stream/sack: optimize SACK size handling
Optimize by keeping count during insert/remove instead of
walking the tree per check.
7 years ago
Victor Julien 7ec7d234cc stream/sack: turn SACK record list into rbtree
Convert to rbtree from linked list. These ranges, of which there can
be multiple per packet, are fully controlled by an attacked. The
attacker could craft a stream of packet in such a way that the list
would grow very large. This would make inserts/removals very expensive,
as well as the list walk that is done and size calculation and pruning
operations.

The RBTREE makes inserts/removals much cheaper, at a slight overhead
for 'normal' operations and slightly higher per record memory use.
7 years ago
Victor Julien 51ce03e76a stream/segments: speed up inserts
Don't try to do a 'fast path' by checking RB_MAX. RB_MAX walks the
tree which means it can be quite expensive. This cost would be paid
for virtually every data segment. The actual insert that follows would
walk the tree again.

Instead, simply insert it. There is a slight cost of the unnecessary
overlap check, but this is much less than the tree walk in a full
tree.
7 years ago
Victor Julien f4ff33969e stream/segments: remove RB_MIN/RB_MAX 7 years ago
Victor Julien 00e65e3cfa stream/segments: optimize overlap tree operations
Now that with the RBTREE we have a properly sorted Segment tree,
where with exact SEQ matches the tree is sorted by payload_len
smallest to largest, we can avoid walking backwards when checking
for overlaps. Our direct RB_PREV either overlaps or not and that
is a reliable verdict for the rest of the tree.
7 years ago
Victor Julien 26b5e1ed13 stream/segments: turn linked list into rbtree
To improve worst case performance turn the segments list into a rbtree.
This greatly improves inserts, lookups and removals if the number of
segments gets very large.

The tree is sorted by the segment sequence number as its primary key.
If 2 segments have the same seq, the payload_len (segment length) is
used. Then the larger segment will be places after the smaller segment.
Exact matches are not added to the tree.
7 years ago
Victor Julien 6a0cf0dd74 tree: add scan-build assertions to suppress FPs 7 years ago
Victor Julien 555fb15ab4 tree: add 2-clause BSD licensed tree.h 7 years ago
Victor Julien 9266334430 stream: expand GAP detection 7 years ago
Victor Julien 3a0eca9fde detect/mpm: minor code cleanup 7 years ago
Victor Julien 164252e381 detect/file: fix minor scan-build warnings 7 years ago
Victor Julien 11f213fb80 stream/tcp: be more liberal in last_ack
Don't set even if seq is before next_seq, as this could still be
a valid packet that was sent before the state was reached.
7 years ago
Victor Julien d7dae87e8b stream/tcp: add debug statements to state dispatcher 7 years ago
Mats Klepsland eba0d04171 app-layer-ssl: don't decode empty extensions 7 years ago
Mats Klepsland 04e78ace0a lua: add function 'TlsGetVersion'
Add another function to get TLS version, since 'TlsGetCertInfo' only
works when a TLS session contains a clear text certificate, which is
not the case in TLSv1.3 or when a session is resumed.
7 years ago
Mats Klepsland df9853b75c detect-tls-version: add support for 'raw' matching
Add support for matching a 'raw' TLS version using a hex string, e.g:

  tls.version:0x7f12;

The above example matches TLSv1.3 draft 16.
7 years ago
Mats Klepsland 4323e7840f detect-tls-version: add support for TLSv1.3 7 years ago
Mats Klepsland e813842731 detect-ssl-version: add support for TLSv1.3 7 years ago
Mats Klepsland db2fc9208a lua: use 'SSLVersionToString' in TlsGetCertInfo() 7 years ago
Mats Klepsland 13918be589 tlslog: add support for TLSv1.3 7 years ago
Mats Klepsland 25fb02da9a output-json-tls: add support for TLSv1.3 7 years ago
Mats Klepsland 97cc3475bf app-layer-ssl: add function to get string from version
Add 'SSLVersionToString' to get string from version.
7 years ago
Mats Klepsland 91acd3831f app-layer-ssl: add support for earlier TLSv1.3 drafts
Add support for TLSv1.3 draft 1 to draft 21.
7 years ago
Mats Klepsland 831ddb62d2 app-layer-ssl: add support for TLSv1.3 from draft 22
Add support for draft 22 to draft 28 and for the final
version (RFC8446) of TLSv1.3.
7 years ago
Mats Klepsland e0ef578c46 app-layer-ssl: add support for session tickets
Add support for logging a session as 'resumed' when using a non-empty
session ticket extension in the client hello record.
7 years ago
Mats Klepsland 21897a4d7a app-layer-ssl: add better session id support
Verify that the session id from both the client hello record and the
server hello record matches before marking the session as 'resumed'.
7 years ago
Mats Klepsland f22bd5a75b app-layer-ssl: decode server hello record
Decoding server hello is needed to do a better implementation of
session resumption.
7 years ago
Victor Julien 93364b9175 flow/timeout: code simplification and cleanup 7 years ago
Victor Julien c8ecca59f8 stream: minor code cleanup 7 years ago
Victor Julien af6f52cc09 rules: hide 'template' from --list-keywords 7 years ago
Victor Julien b0577402b6 rules: hide internal keywords from --list-keywords 7 years ago
Victor Julien 8c7aee92eb flow-manager: fix unittest initialization 7 years ago
Mats Klepsland 68cc53d188 app-layer-ssl: make sure that JA3 stuff is only initialized once
Avoid possible memory leaks by making sure that JA3 buffer and
string is only initialized once.
7 years ago
Mats Klepsland 5ec2f6e7b3 app-layer-ssl: fix memleak/coredump (Bug #2603) 7 years ago
Eric Leblond fcd5e138b9 af-packet: close the socket in case of early fail 7 years ago
Eric Leblond 7e8a749227 log-filestore: fix file descriptor leak
In the case we exceed the number of simultaneously open
file we can reach a state were we will not close the file
after writing.

Thanks to Steve Grubb <sgrubb@redhat.com> for the analysis.
7 years ago
Victor Julien 876156d3a1 profiling/app-layer: fix TCP parsers showing UDP stats 7 years ago
Victor Julien 1f16b42d78 profiling: add missing logger labels 7 years ago
Victor Julien 1f4cd75f05 detect: clean up sgh flags and add cocci check 7 years ago
Victor Julien e6b74f8ee0 stream: minor code cleanups 7 years ago
Victor Julien 7abb8745bf detect/mpm: clean up setup code 7 years ago
Victor Julien 88277d0402 detect: fix file_data detect issue with alert ip
Fix mpm progress being updated by irrelevant engines. Esp in the
case of file_data engines, signature can contain multiple versions
of the same engine, registered for different 'progress' values.

This would lead to signatures being considered 'can't match' even
in cases where they clearly could still match.

Only consider those progress values that apply to the protocol in
use.
7 years ago
Victor Julien a68eec630f detect/parse: try to set flow direction for sigs w/o explict app proto as well 7 years ago
Maurizio Abba bce7c2dd87 eve/http: add tx->request_port_number as http_port
Add the port specified in the hostname (if any) to the http object in
eve. The port may be different from the dest_port used by the TCP flow.
7 years ago
Eric Leblond c9b9f7fd1b util-unittest: fix typo 7 years ago
Victor Julien 28e74abcc5 detect/files: fix inspection issues with 'alert ip'
Don't track the 'skipped' engines as matches.
7 years ago
Victor Julien 6ffa0507d2 detect/filehash: try to open data file from rulefile dir
If the data file can't be found in the default location, which
normally is 'default-rule-path', try to see if it can be found
in the path of the rule file that references it.

This makes QA much easier.
7 years ago
Victor Julien 39ca1db8e8 files: only prune in own direction
Only prune files in own direction. The opposite direction may still
require inspection.
7 years ago
Victor Julien 1df38c3b97 cocci: add more flag checks 7 years ago
Victor Julien 67c90954c0 detect: use BIT_* macros
Also add notes that when adding flags they should be added to the
analyzer as well.
7 years ago
Victor Julien 25a87cbbed stream: use BIT_U8 for stream flags 7 years ago
Victor Julien 7fca17639d detect/prefilter: speed up setup
If the global detect.prefilter.default setting is not "auto", it is
wasteful to run each prefilter setup routine. This patch tracks which
of the engines have been explicitly enabled in the rules and only
runs those.
7 years ago
Victor Julien 4f1befd217 detect/prefilter: fix prefilter when setting is 'mpm'
When prefilter is not enabled globally, it is still possible to
enable it per signature. This was broken however, as the setup
code would never be called.

This commit always call the setup code and lets that sort out
which signatures (if any) to enable prefiltering for.
7 years ago
Victor Julien 085521b218 detect: include keyword types in detect.h 7 years ago
Victor Julien 38b698c50f detect/analyzer: show pattern that is used by mpm
Set a new DETECT_CONTENT_MPM flag on the pattern that is selected
during setup.
7 years ago
Victor Julien 0b5d8a1d75 detect/prefilter: fix alias for fast_pattern
If prefilter is used on a content keyword, it acts as a simple
fast_pattern statement. This was broken because the SIG_FLAG_PREFILTER
flag bypasses MPM for a sig. This commits fixes this by not setting
the flag when it should act as fast_pattern.
7 years ago
Victor Julien 35c5ae3458 detect: limit flush logic to sigs that need it
Limit the early 'flush' logic to sigs that actually need to match
on both stream and http bodies.
7 years ago
Victor Julien f35a3bbae0 detect/analyzer: add built-in lists 7 years ago
Victor Julien 28a0291d07 detect: don't setup PMATCH if it will be unused
Safes a bit of memory and makes it easier to debug.
7 years ago
Victor Julien 6694593cc0 detect/analyzer: add Signature::flags 7 years ago
Victor Julien c0adff3770 detect: remove STATE_MATCH flag use at runtime
Instead, use it only at init time and use Signature::app_inspect
directly at runtime.
7 years ago
Victor Julien 5879dafe55 detect: cleanup direct SIG_FLAG_STATE_MATCH use
This flag should normally not be set manually. It will be set by the
code registering the app engines in a signature.
7 years ago
Victor Julien cbd5ca3f69 detect/dnp3: cleanup list and proto registration 7 years ago
Victor Julien c279a801e5 detect: remove SIG_FLAG_STATE_MATCH flag check
It could not fail as before it the flag was already checked as a bail
out condition.
7 years ago
Victor Julien 83d2d7bb4b detect: minor formatting fixups 7 years ago
Victor Julien adfcb1071b flow: reduce structure size by removing gap 7 years ago