Commit Graph

9451 Commits (0701d828900ff6dfd9fcf81cc0082502e15c1617)

Author SHA1 Message Date
Victor Julien 1e71eecf47 fuzz/siginit: fix leak in case of bidir sig 6 years ago
Victor Julien 5430141f7a fuzz/siginit: minor improvements
Enable detect engine 'quiet' mode to generate less output.

Set a fake filename so that datarep doesn't hit a reachable assert.
6 years ago
Victor Julien 13c9d0ca7e detect/pkt_data: error on unconsumed transforms
If a rule has transforms w/o consuming them (e.g. a content keyword),
don't consider 'pkt_data' valid.
6 years ago
Victor Julien e1c474a1b0 detect/pkt_data: code and test cleanup 6 years ago
Victor Julien 7f19da1cc0 detect: more robust against transform issues
In case of transform issues (transform not consumed before pkt_data
for example), the code would hit an ugly BUG_ON.

Address this by a more graceful error message, that will still
invalidate the sig but not crash the engine.
6 years ago
Jeff Lucovsky 2823bc5aed detect/tls: Use pcre_copy_substring to avoid leak
This commit eliminates a memory leak while parsing TLS version
information. The leak was identified through fuzzing.
6 years ago
Victor Julien 3d969a1c7d build: wrap fuzz targets in guard to fix 'make tags' 6 years ago
Victor Julien 8cbae1371f fuzz/sigpcap: fix FPs due to missing pkt cleanup 6 years ago
Victor Julien e97cdb48f3 decode/teredo: implement port support
Implement support for limiting Teredo detection and decoding to specific
UDP ports, with 3544 as the default.

If no ports are specified, the old behaviour of detecting/decoding on any
port is still in place. This can also be forced by specifying 'any' as the
port setting.
6 years ago
Shivani Bhardwaj 0e4f261224 Use StringParse* for all parsers and configurations 6 years ago
Shivani Bhardwaj c4c734541a Use appropriate ByteExtractString* functions 6 years ago
Shivani Bhardwaj 6b2c7d5be8 util: Add StringParse* functions
StringParse* functions would perform a stricter check compared to
ByteExtractString* functions. These new functions shall also check if
any extra characters follow the extracted numeric value in addition to
the checks performed by ByteExtractString* and return -1 in that case.
This is particularly important in parser, configuration and setup functions.
6 years ago
Philippe Antoine 293eebd999 fuzz: remove obsolete AFL code 6 years ago
Philippe Antoine bf60959d84 fuzz: simpler way to force usage of CXX linker 6 years ago
Philippe Antoine 440bb4d600 fuzz: remove decodeder fuzz target
As we removed decodeder function
6 years ago
Victor Julien e500c59b99 stream/tcp: fix STREAM_HAS_SEEN_DATA macro
The macro would not return true for smaller TCP streams, leading to
cases where the app-layer was not notified of EOF.
6 years ago
Victor Julien 1618fb1b97 stream/tcp: clean up stream flags 6 years ago
Pierre Chifflier 01aef49cbd rust/x509: map decoding errors to decoder events 6 years ago
Pierre Chifflier 333fcc43e7 ssl/tls: call rs_cstring_free for strings allocated in Rust 6 years ago
Pierre Chifflier 1d9f37a60e DER: remove the C parser for DER 6 years ago
Pierre Chifflier d92321d8b1 ssl/tls: use the rust decoder to decode X.509 certificates 6 years ago
Jeff Lucovsky e0bd79670c detect: byte-test convert neg_op flag to a bool
Only 8 flags are permitted so convert one of them to a struct member. I
choose neg_op
6 years ago
Jeff Lucovsky 313c23a26b detect: Add unittests to exercise bitmask 6 years ago
Jeff Lucovsky d12950c9e4 detect: fixup incorrect comments, indentation 6 years ago
Jeff Lucovsky 31ed9786f6 detect: byte_test impl for bitmask
This commit implements byte_test's bitmask feature.
6 years ago
Victor Julien b85539b2ab stream/tcp: fix fast open off by one
With data on SYN the sequence number used for the first data
was off by one, leading to the next segments to appear to come
after a one byte gap.
6 years ago
Philippe Antoine f51d7d8947 fuzz: check tcp splitting evasions in protocol detection 6 years ago
Philippe Antoine 9eddaa038e fuzz: enable AFLFUZZ_PERSISTANT_MODE for libfuzzer targets 6 years ago
Philippe Antoine ac35118ebe fuzz: use env variable to restrict app layer 6 years ago
Philippe Antoine 600b0d7c55 fuzz: adds eight fuzz targets
And ways to compile them with enable-fuzztargets at configure time
Adds utility function in util-unittest-helper
6 years ago
Frank Honza 1c8943dedd add RFB parser
This commit adds support for the Remote Framebuffer Protocol (RFB) as
used, for example, by various VNC implementations. It targets the
official versions 3.3, 3.7 and 3.8 of the protocol and provides logging
for the RFB handshake communication for now. Logged events include
endpoint versions, details of the security (i.e. authentication)
exchange as well as metadata about the image transfer parameters.
Detection is enabled using keywords for:

 - rfb.name: Session name as sticky buffer
 - rfb.sectype: Security type, e.g. VNC-style challenge-response
 - rfb.secresult: Result of the security exchange, e.g. OK, FAIL, ...

The latter could be used, for example, to detect brute-force attempts
on open VNC servers, while the name could be used to map unwanted VNC
sessions to the desktop owners or machines.

We also ship example EVE-JSON output and keyword docs as part of the
Sphinx source for Suricata's RTD documentation.
6 years ago
Victor Julien b4d75b7448 output/anomaly: minor code cleanups 6 years ago
Victor Julien 4d21b03575 detect/app-layer-event: code cleanups 6 years ago
Jeff Lucovsky f0bd69e7e3 detect/pcre: Correct spelling typos 6 years ago
Jeff Lucovsky 7f6af10fed general: copyright bump 6 years ago
Jeff Lucovsky 4b0085b03c detect: Update to take advantage of PCRE refactor
This commit changes the keyword detectors to use the refactored PCRE
modifications from detect-parse.[ch]
6 years ago
Jeff Lucovsky abe0cdc4ad detect/pcre: Changes to support pcre_jit_exec
This command causes `pcre_jit_exec` to be used when available. If it's
available and there are allocation errors preparing for it, things
fallback to `pcre_exec`.
6 years ago
Jeff Lucovsky aa67a0a236 detect/pcre: Add warning for failed registrations
This commit adds a warning used by the PCRE detect logic when it fails
to register initialization and free functions for per-thread JIT stack
handling.

This error code is only used when the platform has PCRE JIT exec
functionality.
6 years ago
Jeff Lucovsky d19429f7e5 detect/parse: Refactor interfaces/definitions
This commit refactors existing code patterns to reduce code duplication
and to be a base for supporting additional PCRE jit-related actions.
6 years ago
Philippe Antoine 1cd314c500 detect: adds icmpv6.mtu keyword 6 years ago
Philippe Antoine 75ec528384 detect: adds utility file for uint keywords 6 years ago
Philippe Antoine 0355b70f5a detect: define generic PrefilterIsPrefilterableById 6 years ago
Philippe Antoine bb7abbd6ec detect: extend PrefilterPacketHeaderValue to 16 bytes 6 years ago
Philippe Antoine d04d69acbb decode: fix endianness for ICMPv6 MTU 6 years ago
Philippe Antoine 9d2fabcd99 doc: fix typo in DetectTcpmssSetup description
And use lowercase true for boolean
6 years ago
Victor Julien efee458af8 stream: fix direction flags in corner case
When a TCP DNS flow would start with a GAP on the TS side, the successful
protocol detection on the TC side would trigger 'opposing side' reassembly
and app-layer processing. In this case the stream flags would indicate the
wrong direction and the wrong parser would be called.
6 years ago
Victor Julien 960045323d app-layer: minor code cleanup 6 years ago
Victor Julien 9b1f0656d0 app-layer: fix progress tracking
Esp in combination with GAPs and proto detection.
6 years ago
Victor Julien 21e6f1f063 app-layer: document return macros 6 years ago
Victor Julien 66598f9ce7 nfs: switch to AppLayerResult 6 years ago
Victor Julien 5cd9cfb5a0 smb: update return type of GAP handling 6 years ago
Victor Julien 4bf87d30e4 smb: convert to return AppLayerResult
Support returning 'incomplete' and remove the buffering
code from the parser.
6 years ago
Victor Julien 674b8dc0fb app-layer: add 'incomplete' return logic
Allow app-layer parsers to indicate how much data they need
before being called again.
6 years ago
Victor Julien 44d3f264bf app-layer: update API to return more details
Add AppLayerResult struct as the Parser return type in
preparation of allowing returning 'Incomplete(size)' similar
to what nom in Rust allows.
6 years ago
Victor Julien 3bcf948a75 app-layer: change return codes
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.

Introduce two macros:

APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)

Update all parsers to use this.
6 years ago
Victor Julien 54b2ec0e4d app-layer: minor optimization 6 years ago
Victor Julien fee7b97ec0 stream: fix function style 6 years ago
Victor Julien 157d01e87e ftp: minor code cleanups 6 years ago
Stephen Donnelly 47082dd5df dag: Skip over ERF_TYPE_META records
Suricata generates an error on unrecognised ERF types.
Suricata should ignore ERF 'Provenance' records with ERF_TYPE_META.
6 years ago
Jason Ish 8a643c893c detect/parse: allow for OK signature parsing errors
The idea of an OK signature parsing error is an error that is
allowed to occur, but still lets test mode pass, unlike
silent errors which will still fail testing.

This is introduced to allow for app-layer event keywords to be
removed, but not have old rules fail out on this case. For example
the Rust DNS parser removes from DNS app-layer events that are
not used anymore.

To signal that an error is OK, -3 is returned. This also implies
silent.
6 years ago
Jason Ish 947cfac62e detect/parse: softer error on unknown app-layer event
On an unknown app-layer event, return -3 for "silent OK fail". A
warning will still be emitted, but its not considered a rule parse
error. This is to handle app-layer events being removed in a more
graceful manner for the user.

This allows -T to pass with an old app-layer events rule file
that may used removed app-layer event keywords.
6 years ago
Jason Ish 688874582a dhcp: remove C app-layer-dhcp wrapper
This just wrapped the Rust function to register the parse,
so instead just call the Rust function directly to remove
the C wrapper, and the 2 C files.
6 years ago
Jason Ish a0e3e2d7b4 dns: register parsers from Rust
And port the C DNS tests to Rust.
6 years ago
Jason Ish 0af9a3a5f7 dns: remove C wrapper functions to Rust
Remove registration of C wrapper functions and register
the Rust functions directly for UDP.
6 years ago
Jason Ish 1b44f839e6 dns: cleanup: move event callbacks into Rust
Remove app-layer-dns-common.c as its no longer needed.
6 years ago
Jason Ish ca5a3f0f04 dns: cleanup: remove unused events
Removed events that are no longer used since the Rust
implementation of DNS:
- UnsolicitedResponse
- StateMemCapReached
- Flooded
6 years ago
Jason Ish 4d312b6813 dns: cleanup: remove C DNS type definitions (dead code) 6 years ago
Jason Ish d809b0959b dns: cleanup: move DnsGetRcode (Lua) to rust
Move the implementation of Lua DnsGetRcode to Rust.
6 years ago
Jason Ish d6b3b33f4a dns: cleanup: remove unused function DNSCreateTypeString 6 years ago
Philippe Antoine 8396333493 detect: adds icmpv6.hdr keyword 6 years ago
Philippe Antoine 02d94f5cf2 detect: fix typo for ipv6.hdr description 6 years ago
Philippe Antoine fc48a7cebf detect: use SC macros for IPV6 header 6 years ago
Philippe Antoine 5f95822f8b doc: fix typo in ByteExtractUint32 description 6 years ago
Jeff Lucovsky c20ab53eae detect/threshold: Correct typos 6 years ago
Jeff Lucovsky ff9a01ee1b detect/threshold: Don't allow duplicates
This commit detects duplicate threshold rule options. When duplicates
are found in a rule, an error message is displayed and the rule is
rejected.
6 years ago
Victor Julien f6c77dcdd9 pcap/file: minor code cleanups 6 years ago
Victor Julien faf5beed2f threads/time: minor code cleanup 6 years ago
Jeff Lucovsky c92975e22b mime: Test cases for filename length limit 6 years ago
Jeff Lucovsky 9a33b5d5de smtp/mime: Fix typos 6 years ago
Jeff Lucovsky 130b8d26e7 smtp/mime: Set event when name exceeds limit 6 years ago
Jeff Lucovsky d0d20bd874 smtp/mime: Restrict file name lengths
This commit places restrictions on the length of the file name specified
in attachments (`name=` or `filename=`) to `NAME_MAX`. Names exceeding
these limits will be truncated and processing will continue with the
truncated name.
6 years ago
Victor Julien 072c421e46 pcap/file: improve time handling
This patch addresses two problems.

First, various parts of the engine, but most notably the flow manager (FM),
use a minimum of the time notion of the packet threads. This did not
however, take into account the scenario where one or more of these
threads would be inactive for prolonged times. This could lead to the
time used by the FM could get stale.

This is addressed by keeping track of the last time the per thread packet
timestamp was updated, and only considering it for the 'minimum' when it
is reasonably current.

Second, there was a minor race condition at start up, where the FM would
already inspect the hash table(s) while the packet threads weren't active
yet. Since FM gets the time from the packet threads, it would use a bogus
time of 0.

This is addressed by adding a wait loop to the start of the FM that waits
for 'time' to get ready.
6 years ago
Victor Julien 4977daec32 threads/time: rename ts to pktts to make purpose clear 6 years ago
Victor Julien ae4137937e pcap/file: fix race during pcap processing start
A race condition during the start of pcap file processing could cause
missed alerts and logged events. This race happens between the packet
threads and the flow manager. It was observed on slower hardware, but in
theory could happen on any machine. It required the 'autofp' runmode.

In commit 6f560144c1 ("time: improve offline time handling") the logic
was added to make the flow manager use a minimum of all the packet threads
perception of time.

The race condition was that the flow manager may become active _before_
all of the packet threads have started processing packets and thus setting
their timestamp. The threads that had not yet initialized their timestamp
would not be considered when calculating the minimum.

As a result of this, older packets timestamps would not yet be registered.
This would give the Flow Manager a timestamp too far in the future. While
the FM was running, the packet processing would start and a flow would
be created. This flow would then immediately be considered 'timed out' by
the FM, due to the timestamp too far in the future.

In the observed case, the thread processing packet 1 from the pcap had not
yet started processing while other threads had already started. The FM was
also already active. Due to the timestamps in the pcap this meant that the
time the FM used was about 500 seconds in the future compared to packet 1.

This patch fixes the issue by initializing all of the threads timestamps
with the timestamp value of the first packet. This way the minimum will
always consider this timestamp.
6 years ago
Victor Julien 1b1855c3bd time: remove unused time structure 6 years ago
Victor Julien 13b4ece11d time: minor code cleanup 6 years ago
Victor Julien 6176338d0b time: fix function name typo 6 years ago
Victor Julien 227e2812f6 pcap/file: fix function ptr naming 6 years ago
Jason Ish a2d91d9bf5 log-pcap: fix log message: unified2 -> pcap
Likely a result of copy and pasted code.
6 years ago
Jason Ish 13635dba57 unified2: not supported message if configured 6 years ago
Jason Ish d86973b386 unified2: remove deprecated output unified2
Ticket 2385:
https://redmine.openinfosecfoundation.org/issues/2385
6 years ago
Jeff Lucovsky e73aec150b pcre: Sticky buffer check
This commit adds logic checking if the sticky buffer in effect provides
the required content.

If the sticky buffer doesn't, the rule will not load and a diagnostic
message with follow-on steps is displayed.
6 years ago
Jeff Lucovsky 8279bab8dc general: Wordsmith "no rules loaded" message 6 years ago
Sascha Steinbiss efaa1755c6 snmp-version: make comment more clear 6 years ago
Sascha Steinbiss f02a18e557 snmp: do not set SIGMATCH_NOOPT 6 years ago
Jeff Lucovsky 6d94b096a9 spelling: Fix spelling error 6 years ago
Jeff Lucovsky 6c3503932f detect/ssl_state: Fix memory leaks from pcre_get_substring
This commit replaces usages of pcre_get_substring with
pcre_copy_substring to avoid leaking memory on error conditions.
6 years ago
Jeff Lucovsky 9fe51a8bd2 detect/flowvar: Fix memory leaks from pcre_get_substring
This commit replaces usages of pcre_get_substring with
pcre_copy_substring to avoid leaking memory on error conditions.
6 years ago
Jeff Lucovsky c2071e1c4e detect/filestore: Fix memory leaks from pcre_get_substring
This commit replaces usages of pcre_get_substring with
pcre_copy_substring to avoid leaking memory on error conditions.
6 years ago
Jeff Lucovsky bcea730266 util-error: define SC_ERR_PCRE_COPY_SUBSTRING 6 years ago
Jeff Lucovsky be4c6b85de general: Convert _Bool to bool
This commit addresses task 3167 and changes usages of '_Bool` to `bool`.
The latter is included from `suricata-common.h`
6 years ago
Jeff Lucovsky 1519c1f006 detect/feature: Support --dump-features 6 years ago
Jeff Lucovsky 5e08e6bccf detect: update version comment 6 years ago
Jeff Lucovsky 1cddbc085b detect: Fix typos/spelling errors. 6 years ago
Jeff Lucovsky 8f6e133497 main: feature init and release 6 years ago
Jeff Lucovsky 9631808b04 output/filestore: announce provided features 6 years ago
Jeff Lucovsky cf840c37b3 detect/filestore: Warn if req'd feature missing 6 years ago
Jeff Lucovsky 31e2cff9cb detect/analysis: Warn if required feature missing 6 years ago
Jeff Lucovsky f83b199f2a feature: feature tracking services 6 years ago
Philippe Antoine 5597801ccb perf: optimizes alert fast log
In the case when we have multiple alerts for one packet
We suppose this happens more often than having decoder_event != 0
6 years ago
Phil Young 1c99536945 napatech: add hardware based bypass support
Napatech hardware bypass support enables Suricata to utilize
capabilities of Napatech SmartNICs to selectively bypass flow-based
traffic.
6 years ago
Philippe Antoine 52970d8508 detect/parse: move spaces skip up the stack
Switch to isspace() as well.
6 years ago
Philippe Antoine 6663246563 parser: make rust probing parsers optional 6 years ago
Jeff Lucovsky 427ec4e739 decode: Fix typos/spelling 6 years ago
Jeff Lucovsky aec4e9a032 decode: Handle ERSPAN Type I 6 years ago
Philippe Antoine af2a689a99 init: move main to a separate file
so as to reuse code from suricata.c with fuzz targets
6 years ago
Philippe Antoine bb42be7e08 fuzz: disable signal handlers while fuzzing 6 years ago
Philippe Antoine 47eba33110 init: make PostConfLoadedSetup global
to be used by fuzz targets
6 years ago
Philippe Antoine 913082941c init: fix PostConfLoadedSetup to use local suri ptr 6 years ago
Philippe Antoine d2b3668a68 init: InitGlobal function
To be reused by fuzz targets
6 years ago
Philippe Antoine f5190da67e util: UTHmemsearch to use memmem if defined 6 years ago
Philippe Antoine ce55d06569 check: Using const keyword for some arguments
For ConfigSetLogDirectory and PrintRawLineHexBuf
6 years ago
Philippe Antoine 12a3a24906 log: can use a file set from env variable
Enables the redirection of log to a file set by an environment
variable SC_LOG_FILE when SC_LOG_OP_IFACE=file
6 years ago
vanlink 2456f27d08 stream/reassembly: fix data overlap check
Fix function CheckOverlap bug.
6 years ago
Victor Julien aeefc82eb9 tls: fix missing extern logic for cert_id tracking 6 years ago
Victor Julien 62c0f3d2b4 stats: fix missing extern keyword 6 years ago
Victor Julien 903291f88a defrag: fix use of globals 6 years ago
Victor Julien 5c3c6c609c threading: fix queue handlers globals use 6 years ago
Victor Julien 3ae1854d2f htp: fix globals use for flags 6 years ago
Victor Julien 85289f3283 proto: fix globals use 6 years ago
Victor Julien 5e583f3a12 flow: fix global variable use 6 years ago
Victor Julien 29f54a34ae stream: fix global declaration of the config 6 years ago
Victor Julien 2436daccd9 threading/modules: fix global declarations 6 years ago
Victor Julien c5f4b41881 ippair: fix global declarations 6 years ago
Victor Julien 0a006d2258 host: fix global declarations 6 years ago
Victor Julien 29780d6164 mpm: fix global declarations 6 years ago
Victor Julien b89059bda7 detect: fix global declaration of sigmatch_table 6 years ago
Victor Julien 0118e07d57 spm: fix global declaration of spm_table 6 years ago
Victor Julien a12c0b499d threading: fix global declaration of threading_set_cpu_affinity 6 years ago
Victor Julien 45955d2e58 unix-socket: avoid using global variable w/o extern 6 years ago
Victor Julien a9a522fac3 decode: fix default-packet-size global variable 6 years ago
Victor Julien 7709b90c16 detect/file-data: remove debug abort that wasn't reachable 6 years ago
Victor Julien ac8ceae9bf detect/file-data: fix function doc 6 years ago
Victor Julien 500e8da63a files: tracking flag update
Improve flow file flags and file flags updates. Introduce a mask
that is set at start up to avoid lots of runtime checks.

Disable cocci flags check as it doesn't support the more dynamic
nature of the flag updates.
6 years ago
Victor Julien a4a4d17ad0 app-layer/files: optimize GetFiles calls
Remove FlowGetProtoMapping calls from the GetFiles wrapper and
get the alstate from the flow directly.
6 years ago
Victor Julien d369e54f1d app-layer: all protocols are tx aware now
So remove the runtime check for it.
6 years ago
Timo Sigurdsson 1262ecbde0 init: Fix dropping privileges in nflog runmode
Using the run-as configuration option with the nflog capture method
results in the following error during the startup of suricata:
[ERRCODE: SC_ERR_NFLOG_BIND(248)] - nflog_bind_pf() for AF_INET failed

This is because SCDropMainThreadCaps does not have any capabilities
defined for the nflog runmode (unlike other runmodes). Therefore, apply
the same capabilities to the nflog runmode that are already defined for
the nfqueue runmode. This has been confirmed to allow suricata start
and drop its privileges in the nflog runmode.

Fixes redmine issue #3265.

Signed-off-by: Timo Sigurdsson <public_timo.s@silentcreek.de>
6 years ago
Victor Julien 7810f22413 decode: remove pseudo packet checks
Bug 1107 checks/hacks should not longer be needed, so remove them.
6 years ago
Victor Julien 272a5f526b threading/queues: simplify error handling 6 years ago
Victor Julien dce227ec88 threading/queues: remove 256 queue limit
Convert fixed size array to a dynamic TAILQ so we can
grow as needed.
6 years ago
Victor Julien 0e3f27a87e threading: remove 'trans_q' array of packet queues
Let the queues code set up PacketQueues on demand.
6 years ago
Victor Julien efa8a69923 packet-queue: create alloc and free functions 6 years ago
Victor Julien 550cfdd98d threading: hide 'trans_q' from queue handlers 6 years ago
Victor Julien 45e5e19e6e threading/threadvars: optimize layout
Make sure StatsPublicThreadContext is on its own cache line.
6 years ago
Victor Julien e3fbdf1948 flowworker/stream: use no-lock packet queue
Use smaller structure for temporary packet queues.
6 years ago
Victor Julien f8aed4ce2d threading: change local packet queue logic
Previously each 'TmSlot' had it's own packet queue that was passed
to the registered SlotFunc as an argument. This was used mostly for
tunnel packets by the decoders and by defrag.

This patch removes that in favor of a single queue in the ThreadVars:
decode_pq. This is the non-locked version of the queue as this is
only a temporary store for handling packets within a thread.

This patch removes the PacketQueue pointer argument from the API.
The new queue can be accessed directly through the ThreadVars
pointer.
6 years ago
Victor Julien b8c2b66d33 packet-queue: introduce a non-locked version
Works exactly like PacketQueue, just does not contain a mutex
and cond var, leading to much reduced memory size.
6 years ago
Victor Julien 9ed260c489 threading: more efficient TmSlot layout 6 years ago
Victor Julien 18e652309f threading: remove 'id' field from TmSlot
Field was now unused.
6 years ago
Victor Julien d7cb0774dd detect: cleanup reload thread handling 6 years ago
Victor Julien 786e697590 threading: simplify flow timeout loop 6 years ago
Victor Julien 261b77742e threading: shrink and reorganize TmSlot 6 years ago
Victor Julien 87c9b11d8c threading/threadvars: rearrange for better cache behavior 6 years ago
Victor Julien 071b753e84 threading/threadvars: remove unused 'prev' field 6 years ago
Victor Julien f53f004917 threading: remove unused 'TmThreadRemove' function 6 years ago
Victor Julien 569a5d985b threading: remove handler names to shink struct
Shrink ThreadVars by removing the queue handler names that are only
used at shutdown. Since this is not performance critical, we can use
the id's to look up the queue handler.
6 years ago
Victor Julien 74a6f8d4dd threading/queues: add way to lookup by ID
In preparation of doing runtime operations by ID instead of by name,
add functions to look up by ID and to convert name to ID.
6 years ago
Victor Julien d0218696ba threading: shrink threadvars struct size 6 years ago
Victor Julien c029599515 threading: remove unused threadvars field 6 years ago
Victor Julien f1ee176111 threading: clarify threadvars fields 6 years ago
Victor Julien d50492cb20 threading: cleanup packet thread shutdown loop 6 years ago
Victor Julien 8e762f5190 source-pcap: remove unused function 6 years ago
Victor Julien 3a703c84ad threading/modules: declare prototypes static
Declare registered threading API funcs static where appropriate.
6 years ago
Victor Julien 7c83cb585e sources: fix pipeline failure handling
When TmThreadsSlotProcessPkt fails it will return the packet that was
passed to it to the packetpool.

Some of the packet sources were doing this manually as well. This patch
fixes those sources.
6 years ago
Victor Julien 49599dfe89 threading: use tm_flowworker for pseudo packets
Pseudo packets don't need to be processed by the decoding layer.
6 years ago
Victor Julien 9df8e1c984 threading: add shortcut to flowworker 6 years ago
Victor Julien 02004fa547 threading: remove per slot post_pq
Use a single packet queue per thread for flow timeout packet
injection. The per slot queue was unused except for this use
case. Having a single queue makes the logic and implementation
simpler.

In case of 'autofp', the per thread packet queue will actually
use the threads input queue. For workers/single a dedicated
queue will be set up.

Rename TmThreadsSlotHandlePostPQs to TmThreadsHandleInjectedPackets
to reflect the changed logic.
6 years ago
Victor Julien 15e3bdb7b8 af-packet: prototypes cleanup
Remove unused prototype.

Declare other prototypes static.
6 years ago
Victor Julien 44d7f636f2 threading: remove post_pq argument from 'SlotFunc'
This was not in use anywhere.
6 years ago
Victor Julien f5045af3e3 runmodes: code cleanups 6 years ago
Victor Julien 1a8562b3c6 detect: clean up threads handling
Clean up reload and break loop thread handling.
6 years ago
Victor Julien e5010d7704 detect: inject packet cleanup 6 years ago
Victor Julien abea227cfc flow-manager: code cleanups 6 years ago
Victor Julien 6fd35fb786 flow-manager: avoid doubly signaling threads
Don't try to wake up the threads we just flagged and validated that
they changed their state.
6 years ago
Victor Julien 539c1a275f threading: remove commented out function 6 years ago
Victor Julien b55f617c2f threading: optimize and unify post_pq checks
TmThreadsSlotProcessPkt did not need to look all 'slots' as only the first
slots post_pq can have been used.

Unify post_pq cleanup handling.
6 years ago
Victor Julien 2a1ed3ba1b threading: remove wrong unlikely statement; minor cleanups 6 years ago
Victor Julien e5192ae20a threading: TmSlot::SlotFunc does not need to be atomic 6 years ago
Victor Julien 89048d71ad threading: fix flags handling by using uint32_t everywhere 6 years ago
Victor Julien 85cf341189 threading: optimize error handling in main packet loop 6 years ago
Victor Julien 9d8ea3b4fe threading: minor code style cleanups 6 years ago
Victor Julien b1056b3836 threading: simplify packetpool checks 6 years ago
Victor Julien 603b2ced47 threading/queues: add shortcut for packetpool check
Allows code simplifycation in the threading loops.
6 years ago
Victor Julien 29cb9d1d52 threading/queues: minor code cleanups 6 years ago
Victor Julien f05c12b70f afl: fix compilation 6 years ago
Victor Julien 01862eae78 afl/decoder: make file dumps optional 6 years ago
Victor Julien 42d112e7b6 detect/address: dead code removal and style cleanups 6 years ago
Jason Ish 80cafb2979 flow: expose last time as a function
This function returns the individual components
of the timeval in output pointers making it suitable
for use over Rust FFI.
6 years ago
Jason Ish d1eab5aa46 defrag: set livedev on the reassembled packet (issue-3380)
Set the livedev on reassembled packets to that of the parent
packet. Fixes issues with multidetect, specifically a segfault
as reported in issue 3380.

Bug #3380.
6 years ago
Victor Julien 9d0976ea8a output/tx: split list of loggers per alproto
This patch splits the list of loggers the tx logging walks into lists per
alproto. The list was getting longer with each eve addition. The result
was that for each tx we would have to loop through multiple loggers that
did not apply to this tx as it was for the wrong protocol.
6 years ago
Victor Julien 5b7aa506c1 output: micro optimization
LogFunc is always set, so don't check for it at runtime.
6 years ago
Victor Julien 4f9e4d41e0 output: optimize root logging loop
Instead of unconditionally looping all the 'root' loggers, loop only
those that are in use.

Root loggers are: packet, tx, file, filedata, streaming.
6 years ago
Victor Julien 07df1ce6af output: clarify registration 6 years ago
Victor Julien 284c3cf68a output/tx: bail early if no flow 6 years ago
Victor Julien ed99e9204f output: fatal error if root logger alloc fails 6 years ago
Danny Browning b573c16dd5 build: cbindgen
Rust headers are now generated using cbindgen. If cbindgen is present, they can
be generated during dist, otherwise they will be available for builds.
6 years ago
Victor Julien 2c050187a3 streaming/api: fix overlap check
In some cases a SBB could be seen as overlapping with the requested
offset, when it was in fact precisely before it. In some special cases
this could lead to the stream engine not progressing the 'raw' progress.
6 years ago
Victor Julien 0f41cf3d74 debug/validation: check tcp/app-layer data lengths 6 years ago
Victor Julien a742c86741 stream: improve app-layer data retrieval with GAPs
Don't assume that the next block after the sbb head is after the
requested offset.

If the next block was before the offset, the returned data_len
would underflow and return a nonsense value to the app-layer.

Bug #2993.
6 years ago
Jeff Lucovsky ed2f6ac64b modbus: Correct typo 6 years ago
Jeff Lucovsky d4428d94de modbus: Update correct TX flags 6 years ago
Jeff Lucovsky 6c2cdbb5f0 analysis: exit if table entries are stale
This commit causes Suricata to exit when a buffer from the analyzer
table is not recognized.

Since the table must match what's registered, exiting will bring noticed
to the condition.
6 years ago
Victor Julien 627cc23769 detect/asn1: fix offset bounds checking 6 years ago
Jason Ish 8609939e60 ipv4: continue parsing options after invalid option
As long as an option has a valid length, we can continue
parsing the options after an invalid one.
6 years ago
Jason Ish df8db1ddb0 ipv4: fail packet decoding on bad ipv4 option length
Currently all failures in IPv4 option decode are ignore with
respect to continuing to handle the packet.

Change this to fail, and abort handling the packet if the
option length is invalid.

Ticket 3328:
https://redmine.openinfosecfoundation.org/issues/3328
6 years ago
Victor Julien fa692df37a stream: reject broken ACK packets
Fix evasion posibility by rejecting packets with a broken ACK field.
These packets have a non-0 ACK field, but do not have a ACK flag set.

Bug #3324.

Reported-by: Nicolas Adba
6 years ago
Victor Julien 9f0294fadc stream: fix SYN_SENT RST/FIN injection
RST injection during the SYN_SENT state could trick Suricata into marking
a session as CLOSED. The way this was done is: using invalid TSECR value
in RST+ACK packet. The ACK was needed to force Linux into considering the
TSECR value and compare it to the TSVAL from the SYN packet.

The second works only against Windows. The client would not use a TSVAL
but the RST packet would. Windows will reject this, but Suricata considered
the RST valid and triggered the CLOSED logic.

This patch addresses both. When the SYN packet used timestamp support
the timestamp of incoming packet is validated. Otherwise, packet responding
should not have a timestamp.

Bug #3286

Reported-by: Nicolas Adba
6 years ago
Victor Julien df74f34a62 decode/tcp: accept TCP fast open cookie request 6 years ago
Victor Julien 040aff5197 htp: close request only from request side
This allows the response side to keep going for just
a bit longer.
6 years ago
Victor Julien 77539e08fc stream: in IDS mode, call app-layer at EOF
On stream end call app-layer with empty message in IDS mode.
6 years ago
Victor Julien eceb7dcba4 eve: support pcap_filename for unix socket mode
Bug #3390.
6 years ago
Philippe Antoine 4a2918e6b5 yaml: clarify comment about dump-all-headers
Logs a warning if the value is unknown
Fixes #2810
6 years ago
Victor Julien 007a461d69 detect/parse: track negation during address parsing
Fix address negation detection not resolving variables when
looking for the negation.

This patch makes use of the actual parsing routines to relay this
information to the signature parser.

Bug #3389.

Fixes: 92f08d85aa ("detect/iponly: improve negation handling in parsing")
6 years ago
Victor Julien 34b7035a0d detect/iponly: debug output improvements 6 years ago
Victor Julien 618ad0d92f app-layer: optimize inspection id tracking
Increase the inspect id for a completely inspected tx in any case.
This avoids re-evaluating transactions.

Reported-by: Ilya Bakhtin
6 years ago
Victor Julien f302f3543f files: add call for setting inspect sizes
The inspect sizes are currently only used during file prune
house keeping for SMTP.
6 years ago
Victor Julien f9f958d66e smtp: fix and clean up new file handling
Set tx id on files that were just opened.

Move logic to a small util func.
6 years ago
Victor Julien 683b22d114 smtp: use FILE_USE_DETECT for raw-extract 6 years ago
Victor Julien 21760bfc76 files: change pruning behavior
If file prune is called inspect has already run. So if file is closed
we can just prune. No need to consider a window anymore.

When still in progress, fix the left_edge calculation.
6 years ago
Victor Julien 682014619f files: fix FILE_USE_DETECT with --disable-detection
Don't set FILE_USE_DETECT flag if detect is disabled.
6 years ago
Victor Julien 4ac9cd2c70 files: move smtp prune logic to main
Now that we call the file prune loop very regularly, we can move the
SMTP specific inspection pruning logic into this loop. Helps with
cases there we don't (often) update a files inspection trackers.
6 years ago
Victor Julien 4b7599af90 http/file: modernize unittests
Part of ticket #2975.
6 years ago
Victor Julien 1cdb2182e4 fastlog: apply icmp type logic to icmpv6 too 6 years ago
Victor Julien 5ef05ffad1 http/multipart: small cleanup 6 years ago
Victor Julien aae00df4df http/multipart: use wider type for boundary lengths
Use uint32_t for a local type instead of uint8_t to avoid casts.

Length should always stay under this regardless.
6 years ago
Victor Julien 4d0db9cb4a http/multipart: optimize form end search
If we already know that the boundary exists, we can start looking
there. Otherwise, we can skip trying as the boundary is a subset
of the form end marker.
6 years ago
Victor Julien 54d93e1eb9 http/multipart: process incomplete file data
Start processing multipart data as soon as it is available to
allow inspection sooner.
6 years ago
Jeff Lucovsky fcfb679893 detect/analyzer: Suppress direction warnings
This commit ensures direction warnings for ICMP v4 and v6
are suppressed and corrects check so that both protocols
are checked (instead of the same protocol being checked twice).
6 years ago
Eric Leblond 1b9009ea0e suricata: fix computing of default packet size
Update the default packet size computation to use LiveDeviceName
instead of LiveDevice as the LiveDevice list is not built when
the default packet size is built.
6 years ago
Victor Julien c010f092e0 detect/replace: fix debug print issue
Don't print field that will likely not be 0 terminated.
6 years ago
Victor Julien c3ea5e71e5 detect/file.data: fix buffer reusing id 0 6 years ago
Victor Julien cd66c37711 http/multipart: use proper progress value to test eof 6 years ago
Philippe Antoine 08b84e060b fastlog: use icmp type and code instead of port
Fixes #3266
6 years ago
Philippe Antoine 75a7d9641c fastlog: move code to reduce variable scope 6 years ago
Philippe Antoine c2fdd7c969 transform: fixes comment about compress_whitespace 6 years ago
Philippe Antoine 9126fc25c1 transform: updates doc about compress_whitespace
And removes duplicate test from strip_whitespace
6 years ago
Eric Leblond 3ded7f1170 qa/coccinelle: fix false positive in setter getter
Coccinelle test was doing a false positive on the function
AppLayerParserStateSetFlag and AppLayerParserStateIssetFlag.
To address that, this patch adds a new coccinelle markup:

 /* coccinelle: AppLayerParserStateSetFlag():2,2:APP_LAYER_PARSER_ */

It indicates that AppLayerParserStateSetFlag is a setter and getter
and that the checks should be disabled inside the function.

Currently this markup is only used for that but following patch will
add some checks on option value.
6 years ago
Jeff Lucovsky 8f4f1cb633 detect/analyzer: Improved fast pattern display
When transforms are part of a rule, improve information displayed with
fast patterns to include the original buffer name and whether any
transform(s) are applied.
6 years ago
Jeff Lucovsky c88c1f1e14 detect/analyzer: Suppress direction warnings
This commit suppresses direction warnings by the rules analyzer for ICMP
and ICMPV6 since it's not actionable.
6 years ago
Victor Julien 83bbe287e7 stats: fix stats not always syncing in flow timeout 6 years ago
Jason Ish ba3a2c31bf app-layer: validate TX detect flag callbacks
Check that both are set or unset.
6 years ago
Jason Ish 706558d4d5 enip: add tx detect flags 6 years ago
Jason Ish cb62c8dacf dcerpc: add tx detect flags 6 years ago
Jason Ish 21f014f5c3 modbus: add tx detect flags 6 years ago
Jason Ish 20bc08a722 app-layer: add tx detect functions to register struct 6 years ago
Jason Ish fdb587d2fc detect-engine: check for tx detect flag support
When registing a detection engine, check that the app-layer
protocol supports tx detect flags. Exit with a fatal
error if it does not as this is a code implementation
error that should be resolved during development.
6 years ago
Jason Ish b1beb76fd7 ftpdata: add tx detect flags 6 years ago
Jason Ish 62e4211f04 debug: add SCReturnBool function exit macro 6 years ago
Jason Ish 739df21e2d app-layer: method to see if parser supports tx detect flags
Add method to check if a parser for an app-layer protocol
supports tx detect flags.

This is a bit of a hack for now as where we need to run
this check from we do not have the IP protocol.
6 years ago
Jeff Lucovsky 218a5c4345 mpm: Fix typos and spelling errors 6 years ago
Jeff Lucovsky aef24bee96 detect: Fix spelling errors 6 years ago
Jeff Lucovsky f318a46d34 detect: Improve handling of variable values
When one of offset/depth/distance is from a variable, adjust the depth
by the offset as is done with scalar values at parse time.
6 years ago
Jeff Lucovsky db8527e7b3 detect/mpm: Improved handling of variable values
This commit removes the offset and depth if either of these values are
dependent upon a byte-extract operation.
6 years ago
Victor Julien 94982ae690 http: split request/response tx id handling
When HTTP pipelining was in use, the transaction id used for events
and files could be off. If the request side was several requests ahead
of the responses, it would use the HtpState::transaction_cnt for events
and files, even though that is only incremented on complete requests.

Split request and response tx id tracking. The response is still handled
by the HtpState::transaction_cnt, but the request side is now handled by
its own logic.
6 years ago
Victor Julien b82e71b95e files: remove FILE_USE_TRACKID flag
Once it was optional but as it no longer is it is no longer useful.

Remove it.
6 years ago
Victor Julien f9155aa121 files: simplify pruning logic
Since ebcc4db84a the flow worker runs
file pruning after parsing, detection and loging. This means we can
simplify the pruning logic. If a file is in state >= CLOSED, we can
prune it. Detection and outputs will have had a final chance to
process it.

Remove the calls to the pruning code from Rust. They are no longer
needed.
6 years ago
Victor Julien ab471c3054 app-layer: don't consider tx flags if not registered
If a protocol does not support TxDetectFlags, don't try to use them.

The consequence of trying to use them was that a TX would never be
considered done, and it would never be freed. This would lead to excessive
memory use and performance problems due to walking an ever increasing
list.
6 years ago
Eric Leblond 54d3620662 source-pcap-file: honor bpf filter on command line
When a BPF filter is given on the command line when reading a
pcap file, the BPF filter is not honored.

The regression has been introduced in:

commit 3ab9120821
Author: Dana Helwig <dana.helwig@protectwise.com>
Date:   Thu Apr 27 11:17:16 2017 -0600

    source-pcap-file: Pcap Directory Mode (Feature #2222)

Reported-By: Tim Colin <tcolin@et.esiea.fr>
6 years ago
Eric Leblond 860f43753c source-pcap-file: fix memory leak on pcap filter 6 years ago
Philippe Antoine 20e06f45c0 util: removes warning about double conversion
From clang 10 :
implicit conversion from 'unsigned long' to 'double' changes value
from 18446744073709551615 to 18446744073709551616
6 years ago
Victor Julien 502a8b5fb3 detect: fix inspection buffer for packet engines
Fix buffers not being reset per inspection round for packet engines.

Bug #3341.
6 years ago
Victor Julien 58b9a2dc21 threading: add debug validation for stale packets 6 years ago
Victor Julien fe9aeed0f0 threading: fix shutdown race condition
A BUG_ON statement would seemingly randomly trigger during the threading
shutdown logic. After a packet thread reached the THV_RUNNING_DONE state,
it would sometimes still receive flow timeout packets which would then
remain unprocessed.

1 main:   TmThreadDisableReceiveThreads(); <- stop capturing packets
2 worker: -> TmThreadTimeoutLoop (THV_FLOW_LOOP) phase starts
3 main:   FlowForceReassembly();           <- inject packets from flow engine
4 main:   TmThreadDisablePacketThreads();  <- then disable packet threads
5 main:   -> checks if 'worker' is ready processing packets
6 main:   -> sends THV_KILL to worker
7 worker: breaks out of TmThreadTimeoutLoop and changes to THV_RUNNING_DONE.

Part of the problem was with (5) above. When checking if the worker was
already done with its work, TmThreadDisablePacketThreads would not consider
the injected flow timeout packets. The second part of the problem was with (7),
where the worker checked if it was ready with the TmThreadTimeoutLoop in a
thread unsafe way.

As a result TmThreadDisablePacketThreads would not wait long enough for the
worker(s) to finish its work and move the threads to the THV_RUNNING_DONE
phase by issuing the THV_KILL command.

When waiting for packet processing threads to process all in-flight packets,
also consider the 'stream_pq'. This will have received the flow timeout
packets.

Bug #1871.
6 years ago
Victor Julien 825173a2ba threading: fix flow timeout loop race 6 years ago
Victor Julien 56354afd41 threading: improve thread queues checking by dumping more info 6 years ago
Victor Julien 0a809bf577 packet: set unique pkt_src 'flush' packets
Set unique type for capture timeout and for detect reload flush
to assist in debugging.
6 years ago
Victor Julien 6bc7636826 stream: remove unused code
Remove now unused 'pkt_src' type as well.

Remove related unittests.
6 years ago
Victor Julien 1633744fcb nfq: remove unused queue handler type 6 years ago
Victor Julien ab01cbe345 log-pcap: remove stale comments 6 years ago
Victor Julien 6de025bb12 alert-syslog: remove stale comments 6 years ago
Jeff Lucovsky 90c2e3561c Add general purpose `ARRAY_SIZE` macro
This commit adds `ARRAY_SIZE` as an helper for determining the number of
elements in an initialized array. The calculation is the same but the
macro provides a convenient shortcut. The implementation was borrowed
from the kernel sources.
6 years ago
Jeff Lucovsky ae198add6d detect/analyzer: Refactor engine analysis code
This commit changes the analysis code to be table driven to better
identify the rule elements covered by the analysis.
6 years ago
Philippe Antoine 6e63c957ff signature: Fixes memory leak in parsing app layer event 6 years ago
Jason Ish fccbd36d37 dns: log addresses in flow direction, not packet
Ticket #3340.
https://redmine.openinfosecfoundation.org/issues/3340
6 years ago
Victor Julien 0824b04134 filestore: don't assume flow is TCP
Filestore can be used by UDP based protocols as well. NFSv2 is one
that Suricata supports.

Bug #3277.
6 years ago
Victor Julien 2a55afbd89 decode/pppoe: fix potential crash in debug statement 6 years ago
Victor Julien 2ab7fb4b41 version: automate and cleanup ver handling
Create a single function to return the version string, to avoid lots
of ifdefs in multiple places.

Make the version determine the 'release' status. If the version from
autoconf has '-dev' in the name, it is not a release. If it hasn't
it is considered a release version.
6 years ago
Victor Julien 51ec980e80 dataset: fix string length handling in hash 6 years ago
Eric Leblond 1721da91ef dataset: fix hash computation 6 years ago
Victor Julien 64a789bbf6 nfq: clear memory of queue before using it
Avoids using uninitialized memory. Show showed itself
in nonsense values in counters, and in nfq_handle_packet
errors that were likely the result of passing uninitialized
memory to the nfq API.

Bug 3263.
Bug 3120.

Fixes: b2a6c60dee ("source-nfq: increase maximum queues number to 65535")
6 years ago
Victor Julien 2fd1174a56 nfq: micro optimization 6 years ago
Victor Julien 9d6087f7d6 nfq: don't warn on 'handle_packet' error
NFQ can generate warnings/errors with a delay. After Suricata has
succesfully passed a verdict to the kernel, there are still things
that can go wrong for that verdict. This is then passed to the
queue through a netlink error message, which leads to nfq_handle_packet
returning an error code.

Suppress the warning. Also remove the errno/strerror use as
nfq_handle_packet does not set the errno.

Thanks to Florian Westphal.

Bug 3120.
6 years ago
Victor Julien f8acad7fca nfq: code cleanups 6 years ago
Alexander Gozman f280e66f84 nfq: check for EAGAIN after recv() call in NFQRecvPkt() 6 years ago
Victor Julien 4cc90e9a4c nfq: minor code cleanups 6 years ago
Victor Julien 01cea2ec89 datasets: suppress noisy debug statement 6 years ago
Victor Julien 505b2dd256 log-pcap: don't print (null) for compression method 6 years ago
Victor Julien fb26268c6b tcp: don't set event on empty SACK opt
TCP_OPT_INVALID_LEN was set if the opt len was 2. While useless
an empty SACK is not uncommon.

Seen on an iOS device talking to an Apple server.

Bug #3254.
6 years ago
Victor Julien aae76a84cd suricata: use version from autoconf 6 years ago
Eric Leblond 2d11e9394c detect-base64: fix url in list keywords commands 6 years ago