Commit Graph

12281 Commits (e1035fd3aed7e546f3ef18fceba8ae7f1441cd24)
 

Author SHA1 Message Date
Victor Julien e1035fd3ae detect/prefilter: bail early if possible 4 years ago
Victor Julien 88bb23b7cf detect/prefilter: update tx_min_progress to uint8_t
Now that our make progress value is 47, we don't need an int.
4 years ago
Victor Julien ed87784907 detect: enforce max app-layer progress
Allow progress values in the range 0-47 so we have 48 bits to track
prefilter engines.

Mark bits 48-62 as reserved explicitly.

Add debug validation checks to make sure the reserved space isn't used.
4 years ago
Victor Julien 932cf0b6a6 detect: track prefilter by progress, not engine
Fix FNs in case of too many prefilter engines. A transaction was tracking
which engines have run using a u64 bit array. The engines 'local_id' was
used to set and check this bit. However the bit checking code didn't
handle int types correctly, leading to an incorrect left shift result of
a u32 to a u64 bit value.

This commit addresses that by fixing the int handling, but also by
changing how the engines are tracked.

To avoid wasting prefilter engine tracking bit space, track what
ran by the progress they are registered at, instead of the individual
engine id's. While we can have many engines, the protocols use far
fewer unique progress values. So instead of tracking for dozens of
prefilter id's, we track for the handful of progress values.

To allow for this the engine array is sorted by tx_min_progress, then
app_proto and finally local_id. A new field is added to "know" when
the last relevant engine for a progress value is reached, so that we
can set the prefilter bit then.

A consquence is that the progress values have a ceiling now that
needs to fit in a 64 bit bitarray. The values used by parsers currently
does not exceed 5, so that seems to be ok.

Bug: #4685.
4 years ago
Victor Julien 9a09fe454b flow: log action applied to all packets
Log if action applied to whole flow is drop or pass.
4 years ago
Victor Julien 3874d08015 tests: fix drop test; cleanup
SigTestDropFlow04 was incorrectly expecting an alert in the packet
following a "drop" packet. The first drop is applied to the flow, so
it should lead to the 2nd packet being dropped before inspection is
run.

Clean up the test as well.
4 years ago
Victor Julien e36b9b89a1 detect/tests: improve detection entry
Lots of tests still use SigMatchSignatures as their main detection
entry function, which bypassed some logic. Make it match main logic
more closely.
4 years ago
Victor Julien 3f4110af32 tests: clean up drop test 4 years ago
Victor Julien 802c1ffee3 detect: enforce flow drops earlier
Enforcing flow drops is now done earlier in the detection engine and
moved out of the IP-only engine where it didn't belong.
4 years ago
Victor Julien aa93984b7e detect: unify alert handling; fix bugs
Unify handling of signature matches between various rule types and
between noalert and regular rules.

"noalert" sigs are added to the alert queue initially, but removed
from it after handling their actions. This way all actions are applied
from a single place.

Make sure flow drop and pass are mutually exclusive.

The above addresses issue with pass and drops not getting applied
correctly in various cases.

Bug: #4663
Bug: #4670
4 years ago
Victor Julien ae89874b06 detect: remove dead code 4 years ago
Victor Julien 33c8fda795 detect/lua: use BIT_U32 for flags 4 years ago
Victor Julien dc6755bf8e detect/lua: minor cleanup 4 years ago
Victor Julien 093ed6f9bc output/tx: check flags using BIT_U32 4 years ago
Victor Julien 29d5eb969e packet: use BIT_U32 for flags 4 years ago
Victor Julien ce18f4b8e2 detect/mpm: micro optimization for initialization
Do less expensive check first.
4 years ago
Victor Julien dfe71bb773 detect: remove ticker
Last consumer of it has been converted.
4 years ago
Victor Julien 9a5c666b26 detect/http: clean up header buffer logic
Simplify and clean up header buffer management. The code was designed
to track buffers for several transactions in parallel, from when the
detection engine wasn't aware of transactions.

For http.start and http.header_names use generic mpm and inspect
functions.
4 years ago
Philippe Antoine ca760e305c ipv6: decoder event on invalid length
From RFC 2460, section 4.5,
each fragment, except the last one, must have a length
which is a multiple of 8
4 years ago
Philippe Antoine 596a4a9d6e http2: better rust style 4 years ago
Philippe Antoine 48ed874dda http2: concatenate one headers multiple values
For detection, as is done with HTTP1
4 years ago
Philippe Antoine e3ff0e7731 http2: generic http2_header_blocks
so as not to forget continuation and push promise
when iterating over headers
4 years ago
Philippe Antoine 0b0649d98e http2: http.header keyword now works for HTTP2
As well as http.header.raw
4 years ago
Philippe Antoine 9b9f909d7d http2: http.header_names keyword now works for HTTP2 4 years ago
Philippe Antoine 547e9f4ab4 http2: http.host normalized keyword now works for HTTP2 4 years ago
Philippe Antoine 75f75e1eb0 http2: turn Host header into authority during upgrade
HTTP1 uses Host, but HTTP2 uses rather :authority cf HPACK
4 years ago
Philippe Antoine bb98a18b3d http2: better file tracking
If an HTTP2 file was within only ont DATA frame, the filetracker
would open it and close it in the same call, preventing the
firther call to incr_files_opened

Also includes rustfmt again for all HTTP2 files
4 years ago
Philippe Antoine 1378b2f451 http2: support deflate decompression
cf #4556
4 years ago
Victor Julien 04ba6dc138 ftp: support per-tx file accounting 4 years ago
Victor Julien 0867b0dbcd smtp: support per-tx file accounting 4 years ago
Victor Julien c9cee7af49 smb: add debug validation on file counts 4 years ago
Victor Julien 114d3ba730 smb: count files in tx 4 years ago
Victor Julien c1dfb619c4 http2: support per-tx file accounting 4 years ago
Victor Julien 1b3c3225cd nfs: add debug validation on file counts 4 years ago
Victor Julien 1d48601c25 nfs: support per-tx file accounting 4 years ago
Victor Julien 67759795c6 nfs: don't reuse file transactions
After a file has been closed (CLOSE, COMMIT command or EOF/SYNC part of
READ/WRITE data block) mark it as such so that new file commands on that
file do not reuse the transaction.

When a file transfer is completed it will be flagged as such and not be
found anymore by the NFSState::get_file_tx_by_handle() method. This forces
a new transaction to be created.
4 years ago
Victor Julien d74c18ee28 http: support per-tx file accounting 4 years ago
Victor Julien 56d3e28a3a filestore: track files getting stored per tx
Avoid evicting a tx before the filedata logger has decided it is
done.
4 years ago
Victor Julien ca124b033e filestore: store chunks in packet direction
Storing too early can lead to files being considered TRUNCATED if the
TCP state is not yet CLOSED when logging is triggered. This has been
observed with FTP-DATA and might also be an issue with simple HTTP.
4 years ago
Victor Julien c78f5ac316 app-layer/transactions: track files opens and logs
To make sure a transaction is not evicted before all file logging is complete.
4 years ago
Victor Julien 45dc4cdeec eve/files: log in packet direction only
Bug: #3703.

Don't log files too soon.
4 years ago
Vladimir Ivchenko e89e563eb4 GRE: Handling pptp without payload
If one of the ppp peers sends a packet with an acknowledge flag,
the ppp payload will be empty and DecodePPP will return TM_ECODE_FAILED.
To handle this case, the packet_length field in the GRE extended header (https://tools.ietf.org/html/rfc2637#section-4.1) is used.
DecodeGRE no longer tries to parse PPP payload if packet_length is zero.
4 years ago
Jason Ish 16a21d7839 scripts: bundle script for requirements
Add a bundle.sh script to bundle the requirements of libhtp
and suricata-update. This uses a Python like requirements.txt
file to specify the URL to download for libhtp and suricata-update.
4 years ago
Lukas Sismis 71196098a1 doc: Update public-data-sets.rst
Replace dead link

Dataset on ll.mit.edu returns 404. Link updated with a search result of more datasets.
4 years ago
Joshua Lumb cf9b2b5fd1 detect-dsize: Add ! operator for dsize matching 4 years ago
Philippe Antoine 9b8be5a650 smb: get file name in case of chained commands 4 years ago
Philippe Antoine 3e5f59e2cb smb: fix parsing of file deletion over SMB1 4 years ago
Philippe Antoine fde753d9d2 smb: recognizes file deletion over SMB2
using set_info_level == SMB2_FILE_DISPOSITION_INFO
4 years ago
Jason Ish 71679c6ad0 ike: use derive macro from app-layer events 4 years ago
Jason Ish eb55297876 modbus: use derive macro from app-layer events 4 years ago