Commit Graph

11122 Commits (7bfb63210e3a128c999343a4805b1d972ae64b73)

Author SHA1 Message Date
Haleema Khan df45fc9e97 detect-l3proto: Convert unittests to new FAIL/PASS API
Ticket: #4050
3 years ago
Alice Akaki d46ce170ba detect-icmp-seq: clean-up unittests
Task: #5597
3 years ago
Eric Leblond 4e1ff9a7ff flow: fix FlowSwapFileFlags function
Ticket: #5625
3 years ago
Haleema Khan 160c778a1f detect-filesize: Convert unittests to new FAIL/PASS API
Ticket: 4039
3 years ago
Haleema Khan f25d4cc546 detect-distance: Convert unittests to new FAIL/PASS API
Ticket : #4020
3 years ago
Alice Akaki f251b6af65 detect-icmp-id: convert unittests to FAIL/PASS APIs
Task: #4042
3 years ago
Haleema Khan 1fdd56a61d detect-itype: Convert unittests to new FAIL/PASS API
Bug: #5589
3 years ago
Simeon Miteff 6673846481 profiling: Make rule_perf.log JSON output newline-delimited
Adding a newline makes the output valid JSON when multiple sortings of
the rules profile object are written to the log.
3 years ago
Jason Ish 0d3cfbbe3f bittorrent-dht/eve: log as bittorrent_dht 3 years ago
Aaron Bungay 86037885a9 bittorrent-dht: add bittorrent-dht app layer
Parses and logs the bittorrent-dht protocol.

Note: Includes some compilation fixups after rebase by Jason Ish.

Feature: #3086
3 years ago
Jason Ish 27672c950c dnp3: fixups to work with unified json tx logger
Update DNP3 to work with a single TX logger, and just register one
logger instead of 2.

This primarily creates a TX per message instead of correlating replies
to requests, which fits the DNP3 model better, but we didn't really have
this concept nailed down when DNP3 was written.
3 years ago
Victor Julien 2f9ca8bb34 dpdk: set new running flag at thread startup 3 years ago
Victor Julien 91b7b63ed2 threads: count thread types after all initialized
To avoid double counting in case a thread took longer than
expected to start up.
3 years ago
Alice Akaki 6621b0ec93 detect-icmp-seq: convert unittests to FAIL/PASS APIs
Task: #4043
3 years ago
Haleema Khan 8d5c5f24a1 dns/eve: add 'HTTPS' type logging
Add a new DNS record type to represent HTTPS
Ticket: #4751
3 years ago
Haleema Khan b1972a5c61 detect-itype: remove unittests
Ticket: #5590
3 years ago
Gabriel Lima Luz 4b009eb907 detect-replace: Convert unittests to FAIL/PASS API
Ticket: 4054
3 years ago
Philippe Antoine a003640ecf security: prevents process creation
with setrlimit NPROC.

So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.

Ticket: #5373
3 years ago
Jason Ish 2ab3646fad profiling: sort LoggerId's in same order as defined
Sort the LoggerId's in the order they are define in suricata-common.h.
3 years ago
Jason Ish 99cb8c666a loggers: all json tx loggers can share the same loggerid
This is to avoid the tx logging code that doesn't support LoggerId
values over 31 at this time. The simplest fix for now is to just have
all JSON (eve) loggers use the same ID.

DNP3 is left as-is for now as it needs some extra support in the parser.
3 years ago
Richard McConnell 13beba141c source: add THV_RUNNING flag to notify of running state
Each module (thread) updates its status to indicate running.
Main thread awaits for all threads to be in a running state
before continuing the initialisation process

Implements feature 5384
(https://redmine.openinfosecfoundation.org/issues/5384)
3 years ago
Eric Leblond 94664ef565 datasets: don't exit on invalid data 3 years ago
Eric Leblond 3bd48d9336 detect: doc link for ip.src and ip.dst 3 years ago
Eric Leblond 7518204ad4 datasets: introduce new IPv6 type
This patch also simplifies IPv6 parsing.

Feature: #5383
3 years ago
Eric Leblond 6fe9d510cf detect: add ip.dst keyword as sticky buffer
Feature: #5383
3 years ago
Eric Leblond 7e516aad94 detect: add ip.src keyword
It is a sticky buffer matching on src_ip.

Feature: #5383
3 years ago
Eric Leblond b2cdc6c899 datasets: introduce ipv4 type
This patch introduce the IPv4 type for dataset so Suricata commandmatch
on a set of IPv4 addresses. This is meant to complement iprep feature
for people that needs more flexibility such as settings the IP on
the packet path.

Feature: #5383
3 years ago
Eric Leblond a9c05c7d96 datasets: factorize serialised operations
Ticket: #5184
3 years ago
Eric Leblond 843dba0a28 datasets: add dataset-lookup command
Ticket: #5184
3 years ago
Eric Leblond 2f25e48897 datasets: add dataset-clear command
Ticket: #5184
3 years ago
Eric Leblond a480abcdd0 datasets: add dump via unix socket
This patch adds a dataset-dump command to the list of unix socket
commands. Implementation is not optimal as we are locking the
datasets when doing the dump. But if we consider that the current
alternative from an implementation point of view is to stop Suricata
then this is far better than current state.

Ticket: #5184
3 years ago
Victor Julien 1fafb83fed packet: turn tunnel lock into spinlock
Lock is only held to update/check ints, so spin lock will be more
efficient.

Place the member of Packet in a new "persistent" area to make it
clear this is not touched by the PacketReinit logic.

Ticket: #5592.
3 years ago
Victor Julien 57e70841c4 stream/tcp: remove obsolete and commented out tests 3 years ago
Victor Julien e72770c1b2 decode/vxlan/tests: don't memset new packet 3 years ago
Victor Julien edf93ae5b6 decode/mpls/tests: improve pkt handling; cleanups 3 years ago
Victor Julien 0f7fe2a4c3 app-layer/tests: don't memset new packet 3 years ago
Victor Julien 6dc53447f1 decode/geneve/tests: don't memset packet
Packet is already initialized.
3 years ago
Victor Julien b07c7ad14c threading: improve/add thread queues explanations 3 years ago
Victor Julien 951bcde0b2 eve/alert: remove tunnel locking
Tunnel lock is only used to sync verdict logic.
3 years ago
Victor Julien 0e7adc21a6 decode: alloc packets using calloc 3 years ago
Victor Julien 68a9da52ad packetpool: remove PKT_ALLOC flag
Use Packet::pool instead. If Packet::pool is non-NULL the packet is
owned by a pool. Otherwise it is allocated and should be freed after
use.
3 years ago
Victor Julien 3ed7b4473e runmodes: remove dead error check 3 years ago
Victor Julien 9d3c60bde3 smtp/mime: no error logging in packet path 3 years ago
Victor Julien 256f0f2c5c defrag: no error logging in packet path 3 years ago
Victor Julien 8f02a3e415 applayer: make sure to use correct ipproto 3 years ago
Victor Julien dba7103a96 eve/alert: use flow proto to avoid crash on ICMP 3 years ago
Alice Akaki 51c0714d71 detect-filemd5: convert unittests to FAIL/PASS APIs
Task: #4035
3 years ago
Alice Akaki fe31d51c30 detect-filesha256: convert unittests to FAIL/PASS APIs
Task: #4038
3 years ago
Haleema Khan 6988168114 src: Use WARN_UNUSED for ByteExtract* functions
Add WARN_UNUSED macro for ByteExtract* functions
Fix warning raised in code related to WARN_UNUSED for ByteExtract*

Ticket: #3658
3 years ago
Alice Akaki 76024f7571 detect-filename: convert unittests to FAIL/PASS APIs
Task: #4036
3 years ago
Alice Akaki 8614bff017 detect-filemagic: convert unittests to FAIL/PASS APIs
Task: #4034
3 years ago
Victor Julien 55cf11fdc2 flow/storage: use const for getter 3 years ago
Victor Julien 99fd69ee8c eve: mac logging code cleanup 3 years ago
Victor Julien 2d79a5206b eve/drop: log direction if we have a flow 3 years ago
Victor Julien 90f3823cad eve: log mac addresses in packet direction 3 years ago
Victor Julien f1068bbb08 dpdk: fix timestamp issues
Each thread had its own version of the `machine_start_time`, which
lead to slight time differences. This became apparent mostly in IPS,
where 2 threads each process a side of the flow.

This patch makes the `machine_start_time` global.
3 years ago
Victor Julien cd2a5ec84f packet: move action functions to packet files 3 years ago
Victor Julien 0977f40d1c detect/tag: improve time handling on windows
Bug: #5584.
3 years ago
Victor Julien 410acf226f detect: unittest guards for ut only function 3 years ago
Victor Julien 60285a629f detect: remove unused function 3 years ago
Victor Julien 5d6212183b eve/alert: minor cleanups 3 years ago
Victor Julien 6c200c7793 detect: issue drop to root packet in all cases
Update DROP action handling in tunnel packets. DROP/REJECT action is set
to outer (root) and inner packet.

Check action flags both against outer (root) and inner packet.

Remove PACKET_SET_ACTION macro. Replace with RESET for the one reset usecase.
The reason to remove is to make the logic easier to understand.

Reduce scope of RESET macros.

Rename PacketTestAction to PacketCheckAction except in unittests. Keep
PacketTestAction as a wrapper around PacketCheckAction. This makes it
easier to trace the action handling in the real code.

Fix rate_filter setting actions directly.

General code cleanups.

Bug: #5571.
3 years ago
Victor Julien 79fc8e74cb packetpool: debug message to assist drop checks 3 years ago
Victor Julien d1009e295c packetpool: remove debug validation check
Current packet might be the root or a child. Root would have set
drop action set, but Packet::pkt_src might be set in either.
3 years ago
Victor Julien fe5a8beb50 decode: minor code cleanup 3 years ago
Victor Julien ba3e0b3155 nfq: set drop reason on verdict error 3 years ago
Victor Julien a7333a3ea5 napatech: reduce size of Packet structure
Put napatech packet vars in the union that is meant for this type of
data.
3 years ago
Haleema Khan b31a286952 detect-fileext: convert unittests to FAIL/PASS APIs
Fixes Bug: #4033
3 years ago
Lukas Sismis 5365fdccf7 dpdk: fix mempool cache error message 3 years ago
Shivani Bhardwaj 7005443b8b base64: add and clean tests 3 years ago
Shivani Bhardwaj dad52f133d base64: add new mode as per RFC 4648
As per RFC 4648,
Implementations MUST reject the encoded data if it contains characters
outside the base alphabet when interpreting base-encoded data, unless
the specification referring to this document explicitly states
otherwise.

Add a new mode BASE64_MODE_RFC4648, and handle input strictly as per the
specification.

Bug 5223
3 years ago
Lukas Sismis e101384e7b transversal: remove suricata-ids.org references 3 years ago
Lukas Sismis aeb690317a dpdk: allow specifying RSS hash function flags in the config
Ticket: #5400
3 years ago
Eric Leblond e46a0bd46a eve: explicit default when setting port 3 years ago
Eric Leblond 00c419a6f8 eve: micro simplification 3 years ago
Eric Leblond a0065f4368 eve/alert: add direction field to log data way
Add a key in the event to specify if the data that did
trigger the alert are in to_client or to_server direction.
3 years ago
Eric Leblond f1300e68c9 eve/alert: add src and dest info to flow in alert
When looking at an alert event, it was impossible to determine which
side from src or dest IP in the alert was the client and wich side
was the server with regards to the underlying flow. This was a problem
when you try to known who belongs a metadata property such as a HTTP
hostname or a TLS JA3.

This patch updates the code to add src and dest IP in the flow
subobject as well as src and dst port. This way, we can now which
side is the client and which side is the server.

The result is looking like:

{
  "event_type": "alert",
  "src_ip": "22.47.184.196",
  "src_port": 81,
  "dest_ip": "192.168.1.47",
  "dest_port": 1063,
  "proto": "TCP",
  "tx_id": 0,
  "alert": {
    "signature_id": 2018959,
    "rev": 3,
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 22,
    "pkts_toclient": 35,
    "bytes_toserver": 1370,
    "bytes_toclient": 48852,
    "start": "2009-10-28T10:01:46.755232+0100",
    "src_ip": "192.168.1.47",
    "dest_ip": "22.47.184.196",
    "src_port": 1063,
    "dest_port": 81
  }
}
3 years ago
Eric Leblond bb93d67ddd unix-socket: add command to get flow stats
Add a command to extract the accounting data from a live
flow using the unix socket. It takes the flow_id as param
and return the volume of data seen on the flow as well as
its age.
3 years ago
Eric Leblond 19400a7d69 flow: add function to get flow using flow_id 3 years ago
Eric Leblond 06756314d6 flow: change flow id computation method
Previous method was truncating the flow hash value when building
the flow_id. It is interesting not to loose the flow hash value
as it can be used in other tools or to interact with a flow that
is still active.
3 years ago
Eric Leblond 06b6f85c1f json/flow: log if flow had gap in TCP 3 years ago
Eric Leblond f9faff5c4c flow: add function to say if there is gap 3 years ago
Eric Leblond e6768118da stream: flag TCP streams with gap 3 years ago
Eric Leblond 9cb06d4376 detect/smb: add smb.ntlmssp_domain keyword
Feature #5411.
3 years ago
Eric Leblond 69ef1bc194 detect/smb: add smb.ntlmssp_user keyword
Feature #5411.
3 years ago
Victor Julien a6e86fef04 detect: remove unused data struct 3 years ago
Victor Julien f837146321 flow/worker: process injected flows more gradually
Worker threads are responsible for final processing of timed out flows.
These are selected by the Flow Manager and inserted into a per thread
queue. The Flow Worker then checks this queue after each packet. Due to
the burstiness of this process, the packet threads would sometimes process
a lot of these flows in the context of a single packet, leading to spike
in latency which might cause packet loss.

This patch changes the behavior to only process at max 2 flows per packet.
This way added processing cost is amortized over many packets.
3 years ago
Victor Julien ce1bdcb474 flow: count max number of injected flows in workers 3 years ago
Victor Julien 235f369ab9 stream: reduce pool locking overhead
Add thread local cache to avoid locking overhead for ssns and segments.

A thread will return segments/ssns to a local cache first, and if that
is full, to a return queue where the actual return to the pool returns
a batch, to amortize locking overhead.

Adds segment and session pool/cache counters to see where how effective
the cache is.
3 years ago
Victor Julien 536d66e344 stream: minor test cleanup 3 years ago
Victor Julien 839a4f5ab4 files: only call loggers is there is work to do 3 years ago
Victor Julien fa6d41ca66 filestore: remove obsolete checks 3 years ago
Victor Julien 0b3053a376 filestore: fix empty file not opening, but trying to close 3 years ago
Victor Julien 23323a961f app-layer: reduce app cleanup and output-tx calls
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.

Do the same of end of flow packets.

This skips needless iterations over the transaction stores.
3 years ago
Victor Julien 4bb7f827e0 output/tx: minor cleanups/optimizations 3 years ago
Victor Julien 39cf5b151a src: includes cleanup
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.

Update files to compile after this.

Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.

Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
3 years ago
Victor Julien 3e2295a963 detect: clean up detect-engine-state.h
Remove prototypes that are not about purely the data structures.
3 years ago
Victor Julien ad0a29cabc detect: remove wrapper func 3 years ago
Victor Julien 1dd0a2fed6 detect: move DetectTransaction to header its used in 3 years ago
Victor Julien 41265a859e detect/files: optimize file.data by skipping non-file txs
As well as 'file' txs not in our direction.

Implement the same logic for file.name and file.magic prefilter engines.
3 years ago