Commit Graph

355 Commits (35bc73e4e2cb4865c06e0da7a05ea60f2966ee08)

Author SHA1 Message Date
Eric Leblond ea9b9b5063 stream-tcp: add option to accept invalid packets
Suricata was inconditionaly dropping packets that are invalid with
respect to the streaming engine. In some corner case like asymetric
trafic capture, this was leading to dropping some legitimate trafic.

The async-oneside option did help but this was not perfect in some
real life case. So this patch introduces an option that allow the
user to tell Suricata not to drop packet that are invalid with
respect to streaming.
8 years ago
Eric Leblond 775e67459c filestore: avoid open write close sequence
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.

Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.

This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.

This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.

In EVE, the entries are:
 stats {file_store: {"open_files_max_hit":0,"open_files":5}}
8 years ago
Eric Leblond 54718b306e filestore: add option to disable meta file writing
As the fileinfo entry is containing the file_id it is enough to
have this entry to link the extracted file with metadata.
8 years ago
Victor Julien d6592211d0 rust/nfs: NFSv3 parser, logger and detection 8 years ago
Victor Julien a995734b3a yaml: sync with new stream engine 8 years ago
Victor Julien 807312320f stream-tcp: implement thread pool for segments
Config option:

stream:
  reassembly:
    segment-prealloc: 2048
8 years ago
Victor Julien 91f57200c7 stream: add stream.reassembly.check-overlap-different-data option 8 years ago
Victor Julien b3e9d39771 stream: remove unused zero copy setting 8 years ago
Ray Ruvinskiy 7539973109 tls: logging for session resumption
We assume session resumption has occurred if the Client Hello message
included a session id, we have not seen the server certificate, but
we have seen a Change Cipher Spec message from the server.

Previously, these transactions were not logged at all because the
server cert was never seen.

Ticket: https://redmine.openinfosecfoundation.org/issues/1969
8 years ago
fooinha a64e5e77c7 eve: async mode for redis output
eve: detects libevent for async redis at configure
eve: moves redis output code to new file - util-log-redis.{c,h}
eve: redis ECHO and QUIT commands for async mode
eve: redis output defaults if conf is missing
8 years ago
fooinha 20d4d40051 log: tls custom format log 8 years ago
Mats Klepsland 2b460b8d06 output-json-lua: log certificate serial number 8 years ago
Mats Klepsland 115b3138cc output-json-tls: log certificate and chain
Log entire certificate and certificate chain Base64 encoded.
8 years ago
Mats Klepsland 0716199acb output-json-tls: custom tls logging 8 years ago
Jason Ish 82f6103149 unified2: nostamp and file rotation
Give unified2 a nostamp option which will create the file
without the timestamp suffix (like Snort's nostamp option).

Also register for rotation notification on SIGHUP so the file
will be recreated if it is removed by an external rotation
program (only when nostamp is used).
8 years ago
Victor Julien 5ca4a2e6fe outputs: vars log
EVE addition called 'vars' that logs pkt/flow vars for each packet/flow.
8 years ago
Victor Julien 1a2ad059a1 eve: log pktvars/flowvars/bits/ints
Optionally logs 'vars' into alerts
8 years ago
Victor Julien 75907fce06 profiling: output all sort options for rules
Limit the default number of sids to 10.
8 years ago
Victor Julien 7d8a5a75ef profiling: honor limit in json rule output 8 years ago
Victor Julien a0580d8805 stream: initialize stream segment pool from mtu
If segments section in the yaml is ommitted (default) or when the
pool size is set to 'from_mtu', the size of the pool will be MTU
minus 40. If the MTU couldn't be determined, it's assumed to be
1500, so the segment size for the bool will be 1460.
8 years ago
Victor Julien 15f4144eda smb: add tcp/445 to proto detect fallback 8 years ago
Travis Green f08cc1f3db yaml: update commented rule files
Disabled scada.rules, added commented rule file names to help
administrators find informational rule files.
8 years ago
Victor Julien 3012edae1c luajit: update default yaml and doc for 'states' 8 years ago
Victor Julien 3973363164 yaml: group ICS protocols together 8 years ago
Victor Julien b231558957 ENIP: add default ports to yaml 8 years ago
Jason Ish bbaa79b80e DNP3: Application layer decoder.
Decodes TCP DNP3 and raises some DNP3 decoder alerts.
8 years ago
Victor Julien b789d2ae3d tls: change 'no-reassemble' option to default off
This option was broken so there should be no visible change to
actual deployments.
9 years ago
Jason Ish a6854147be pcap-log config: sguil-base-dir -> dir and update comment
The code already looks for "dir" first instead of
"squil-base-dir", and already respects this configuration
parameter in other modes than the "sguil" mode.

Coda will still access "sguil-base-dir".
9 years ago
Victor Julien e6cf7ae8fa yaml: improve stream-depth comments 9 years ago
Giuseppe Longo 3f214b506a file-store: add depth setting
When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.

This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
9 years ago
Giuseppe Longo 9ab1194f68 modbus: set stream depth
Some protocol like modbus requires
a infinite stream depth because session
are kept open and we want to analyze everything.

Since we have a stream reassembly depth per stream,
we can also set a stream reassembly depth per proto.
9 years ago
Victor Julien 050f36eaa5 enip: improve yaml 9 years ago
kwong a3ffebd835 Adding SCADA EtherNet/IP and CIP protocol support
Add support for the ENIP/CIP Industrial protocol

This is an app layer implementation which uses the "enip" protocol
and "cip_service" and "enip_command" keywords

Implements AFL entry points
9 years ago
Victor Julien 125603871b detect: config opt to enable keyword prefilters 9 years ago
Giuseppe Longo e6bac998d9 flow: add timeout for local bypass
This adds a new timeout value for local bypassed state. For user
simplication it is called only `bypassed`. The patch also adds
a emergency value so we can clean bypassed flows a bit faster.
9 years ago
Giuseppe Longo 177df305d4 stream-tcp: enable bypass setting
This permits to enable/disable in suricata.yaml
and the bypass function will be called
when stream.depth is reached.
9 years ago
Giuseppe Longo 97783f8142 nfq: introduce bypass function 9 years ago
Victor Julien da8f3c987b offloading: make disabling offloading configurable
Add a generic 'capture' section to the YAML:

  # general settings affecting packet capture
  capture:
    # disable NIC offloading. It's restored when Suricata exists.
    # Enabled by default
    #disable-offloading: false
    #
    # disable checksum validation. Same as setting '-k none' on the
    # commandline
    #checksum-validation: none
9 years ago
Duarte Silva 53ebe4c538 file-hashing: added configuration options and common parsing code 9 years ago
Eric Leblond f2d1e93e65 unix-socket: add auto mode
When running in live mode, the new default 'auto' value of
unix-command.enabled causes unix-command to be activated. This
will allow users of live capture to benefit from the feature and
result in no side effect for user running in offline capture.
9 years ago
Victor Julien 2997d086be eve-drop: allow logging all drops
- drop:
    alerts: yes      # log alerts that caused drops
    flows: all       # start or all: 'start' logs only a single drop
                     # per flow direction. All logs each dropped pkt.
9 years ago
Tom DeCanio 0f6c8806a0 output-json-dns: dns output filtering. 9 years ago
Jason Ish 1691c10681 eve: make logging of tagged packets optional
But it is enabled in the default configuration.
9 years ago
Victor Julien f7124b1149 afpacket: disable tpacket-v3 by default
It's still considered experimental at this point.
9 years ago
Victor Julien 5ec885e451 http: set of response body decompress limit
This is a per personality setting.
9 years ago
Victor Julien 0b6171854d yaml: improve affinity defaults 9 years ago
Victor Julien 723e90a174 affinity: rename detect-cpu-set to worker-cpu-set
Add fallback for existing configs.
9 years ago
Victor Julien 45b72d61c9 affinity: improve suricata.yaml doc 9 years ago
Victor Julien 570b9d06e0 affinity: remove unused settings
These were never referenced to in the code so they can be removed.

Add bypass to config parser in case the settings are still in old
yamls.
9 years ago
Victor Julien 1c0f20f0e5 yaml: profiling 'json' depend on jansson availability 9 years ago
Victor Julien d58d02fed5 netmap: handle missing config with better defaults
Default to 'threads: auto' which uses RSS RX count when no config
has been created for a interface.
9 years ago
Victor Julien be9cd0fd84 yaml: replace ac-tile by ac-ks 9 years ago
Victor Julien f55dbca57b yaml: make eve log in yaml depend on libjansson 9 years ago
Victor Julien df6f9269ec yaml: improve capture comments 9 years ago
Victor Julien 766bc95e3c yaml: move classification etc below the rules 9 years ago
Victor Julien 1b4e1ea389 yaml: new defaults for outputs
Enable eve.flow, disable plain http.log.
9 years ago
Victor Julien 4d056912d3 yaml: file logging at info level 9 years ago
Victor Julien cb47c2f682 yaml: improved defaults and misc cleanups 9 years ago
Victor Julien ea7923cc81 yaml: add performance tuning section 9 years ago
Victor Julien 6d7b4c81e3 yaml: more reshuffling 9 years ago
Victor Julien a6a69f0099 yaml: create advancted sections
Sections for advancted detection settings and traffic tracking and
reconstruction.
9 years ago
Victor Julien d79c95dded yaml: add hw accel section, move cuda there 9 years ago
Victor Julien 8fae138d3b yaml: add netfilter section 9 years ago
Victor Julien 056f88b458 yaml: move outputs to the logging step 9 years ago
Victor Julien 11e6809d55 yaml: introduce 'advanced settings' 9 years ago
Victor Julien c5ca642a28 yaml: move app layer up 9 years ago
Victor Julien c160f78758 yaml: move afpacket, pcap, pcap-file up 9 years ago
Victor Julien d48098f189 yaml: move logging up 9 years ago
Victor Julien c949668863 yaml: move rules up in the file
Also disable decoder and stream events by default, as they are too noisy
in a untuned environment.
9 years ago
Victor Julien a9cea53e62 yaml: move vars to the top 9 years ago
Justin Viiret c9d0d6f698 mpm: add "auto" default for mpm-algo
Setting mpm-algo to "auto" will use "hs" if Suricata was built against
Hyperscan, and "ac" otherwise (or "ac-tile" on Tilera platforms).
9 years ago
Eric Leblond ff05fb760b af-packet: fix some typos in yaml 9 years ago
Eric Leblond 876b356bbe af-packet: use mmap capture by default
Update the code to use mmap capture by default even in unset in
configuration file. mmap capture is now be turned off by using
explicitely 'use-mmap: no' in configuration.
9 years ago
Eric Leblond f5c2019167 af-packet: add option to use memory locked mmap 9 years ago
Eric Leblond 234aefdff9 af-packet: configurable tpacket_v3 block timeout
Block timeout defines the maximum filling duration of a block.
9 years ago
Eric Leblond fa902abedf af-packet: configurable tpacket_v3 block size
It is used to set the block size in tpacket_v3. It will allow user
to tune the capture depending on his bandwidth.

Default block size value has been updated to a bigger value to
allow more efficient wlak on block.
9 years ago
Eric Leblond bae1b03cf5 af-packet: tpacket_v3 implementation
This patch adds a basic implementation of AF_PACKET tpacket v3. It
is basic in the way it is only working for 'workers' runnning mode.
If not in 'workers' mode there is a fallback to tpacket_v2. Feature
is activated via tpacket-v3 option in the af-packet section of
Suricata YAML.
9 years ago
Justin Viiret 91011b30a6 spm: add "spm-algo: auto" setting
This will default to Hyperscan when Suricata is built with Hyperscan
support. Otherwise, Boyer-Moore is used by default.
9 years ago
Justin Viiret 7ba9dbe36a suricata.yaml: document spm-algo option 9 years ago
maxtors c6bbd89251 Added payload-buffer-size option to yaml configuration 9 years ago
Victor Julien 5f676167a3 detect grouping: make json dump configurable
Make the rule grouping dump to rule_group.json configurable.

detect:
  profiling:
    grouping:
      dump-to-disk: false
      include-rules: false      # very verbose
      include-mpm-stats: false
9 years ago
Victor Julien d6ba01b1b7 detect: make port whitelisting configurable
Make the port grouping whitelisting configurable. A whitelisted port
ends up in it's own port group.

detect:
  grouping:
    tcp-whitelist: 80, 443
    udp-whitelist: 53, 5060

No portranges are allowed at this point.
9 years ago
Victor Julien 725d6c3739 yaml: convert detect-engine to just detect
Instead of detect-engine which used a list for no good reason, use a
simple map now.

detect:
  profile: medium
  custom-values:
    toclient-groups: 3
    toserver-groups: 25
  sgh-mpm-context: auto
  inspection-recursion-limit: 3000
  # If set to yes, the loading of signatures will be made after the capture
  # is started. This will limit the downtime in IPS mode.
  #delayed-detect: yes
9 years ago
Victor Julien caea596ce5 profiling: output post-prefilter matches
Dump a json record containing all sigs that need to be inspected after
prefilter. Part of profiling. Only dump if threshold is met, which is
currently set by:

 --set detect.profiling.inspect-logging-threshold=200

A file called packet_inspected_rules.json is created in the default
log dir.
9 years ago
Victor Julien 722e2dbf7c profiling: initial rulegroup tracking
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.

Logs SGH id so it can be compared with the rule_group.json output.

Implemented both in a human readable text format and a JSON format.
9 years ago
Victor Julien 4f8e1f59a6 mpm: remove obsolete mpm algos
Remove: ac-gfbs, wumanber, b2g, b3g.
9 years ago
Victor Julien 4526aed2b1 smtp: fix config parsing and config defaults 9 years ago
Travis Green 72c9debbd6 yaml: disable rules by default
Change to "disable by default" rulefiles
9 years ago
Tom DeCanio 559747e325 file-store: add force-filestore configuration option to enable writing all
extracted files to filesystem.
9 years ago
Andreas Herz 5cee70f9ae Fix the comment and explanation for random-chunk-size 9 years ago
Andreas Herz 15c98c6085 file-magic: improve libmagic handling on *nix systems 9 years ago
Victor Julien fae2836039 http: more sane body inspection/tracking defaults 9 years ago
Victor Julien b4dad91e26 unified2: disable by default 9 years ago
Victor Julien 36fde7df42 stats log: suppress 0 counters by default 9 years ago
Jason Ish d87a60f3cc modbus: disable by default 9 years ago
Victor Julien c1bf0e1b07 rule profiling: json output 9 years ago
Eric Leblond affb399cd9 config: don't use hardcoded path
It is better to use a transformation to define the default
directory of output message instead of using an hardcoded value.
Same apply to the directory for the pid file.
9 years ago
Eric Leblond b834e2d19a util-logopenfile: implement redis pipelining
This patch implements redis pipelining. This consist in contacting
the redis server every N events to minimize the number of TCP
exchange. This is optional and setup via the configuration file.
9 years ago
Eric Leblond 60ea49c777 output-json: add sensor-name config variable
When using redis output, we are loosing the host key (added by
logstash or logstash-forwarder) and we can't find anymore what
Suricata did cause the alert.

This patch is adding this key during message generation using the
'sensor-name' variable or the hostname is 'sensor-name' is not
defined.
9 years ago
Eric Leblond eef5678e5e output-json: add redis support
This patch adds redis support to JSON output.
9 years ago
Alessandro Guido dcbbda505f Describe new unified2-alert "payload" option 10 years ago
Eric Leblond f03a7a032f json-alert: add smtp elements in alert 10 years ago
Eric Leblond 946f2a6acc email-json: add bcc to extended fields 10 years ago
Eric Leblond 8fd88f543d yaml: add comment describing smtp extended 10 years ago
Eric Leblond f81f353d1f email-json: add 'date' field extraction 10 years ago
Eric Leblond d1b0a5aa6d yaml: document new MIME features 10 years ago
Victor Julien 7281ae6e80 yaml: add missing ippair section 10 years ago
Eric Leblond 3054af7900 af-packet: don't activate rollover by default
Rollover option is causing issue with TCP streaming code because
packets from the same flow to be treated out of order. As long as
the situation is not fixed in the streaming engine, it is a bad idea
to enable it by default.
10 years ago
gureedo a7a902a071 netmap: extended comments for options in configuration file.
Added extended description of the use of OS endpoint with copy mode.
10 years ago
Eric Leblond 8fde842f97 af-packet: implement rollover option
This patch implements the rollover option in af_packet capture.
This should heavily minimize the packet drops as well as the
maximum bandwidth treated for a single flow.

The option has been deactivated by default but it is activated in
the af_packet default section. This ensure there is no change for
old users using an existing YAML. And new users will benefit from
the change.

This option is available since Linux 3.10. An analysis of af_packet
kernel code shows that setting the flag in all cases should not
cause any trouble for older kernel.
10 years ago
Eric Leblond dc306f3bad af-packet: implement new load balancing modes
This patch implements the fanout load balancing modes available
in kernel 4.0. The more interesting is cluster_qm that does the
load balancing based on the RSS queues. So if the network card
is doing a flow based load balancing then a given socket will
receive all packets of a flow indepently of the CPU affinity.
10 years ago
Aleksey Katargin caa2438b98 netmap: support SW rings
Netmap uses SW rings to send and receive packets from OS.
10 years ago
Eric Leblond 4db0a35f25 tls-store: now a separate module
An design error was made when doing the TLS storage module which
has been made dependant of the TLS logging. At the time there was
only one TLS logging module but there is now two different ones.

By putting the TLS store module in a separate module, we can now
use EVE output and TLS store at the same time.
10 years ago
Victor Julien f43767ba44 config: update yaml to show json logging option 10 years ago
Giuseppe Longo a459376d2e app-layer-htp: add http_body_inline setting 10 years ago
Jason Ish e3ce29f694 json-stats: log deltas
If "deltas" is yes, log delta values as the name of the value
suffixed with _delta.
10 years ago
Zachary Rasmor 0edf28a4f8 Add Feature #1454. Generic eve-log prefix support. 10 years ago
Victor Julien 3fab736539 log-stats: make global/threads logging configurable 10 years ago
Victor Julien 2c9a2c8327 stats: support per thread stats in json output
Default is only to output totals. Optionally per thread can be added.

Both can be enabled together.
10 years ago
Tom DeCanio 117eed0385 eve-log: add JSON stats logging
Support for counters in stats.log in eve output JSON stream.
10 years ago
Eric Leblond 7d73db9b80 suricata.yaml: fix the name of EVE module
It is netflow and not newflow.
10 years ago
Giuseppe Longo 26ba647d58 filedata: read inspected tracker settings from suricata.yaml 10 years ago
Eric Leblond bd0041470f rules: add app layer events rules
Some application layer events are defined but the corresponding
rules were not available in the rules directory.
10 years ago
Eric Leblond 9fb82390ab suricata.yaml: add missing mpm-algo 10 years ago
Victor Julien 0704ece4d7 detect-reload: enable unconditionally
Reloading is available unconditionally now.
10 years ago
Victor Julien 2e754ca6fa drop json: make alerts logging optional
Make logging out alerts that caused the drop optional.
10 years ago
Eric Leblond 881f32cc02 json-alert: add SSH fields in alert logging 10 years ago
Eric Leblond 180faece7c json-alert: log tls info in alert
This patch adds the capabilities to log the TLS information the
same way it is currently possible to do with HTTP. As it is
quite hard to read ASN.1 directly in the stream, this will help
people to understand why suricata is firing on alert relative
to TLS.
10 years ago
gureedo 10104066e1 netmap support 10 years ago
Victor Julien 9327b08ab1 tcp: add stream.reassembly.zero-copy-size option
The option sets in bytes the value at which segment data is passed to
the app layer API directly. Data sizes equal to and higher than the
value set are passed on directly.

Default is 128.
10 years ago
Ken Steele 5008d0a58b Remove the b2gm and b2gc MPMs
These MPMs have code that looks like it won't work and updating them to
for the new MPM optimization wasn't working.
10 years ago
Duarte Silva 496200dd08 Prepared everything for the proxy deployment configuration
- Added the suricata.yaml configurations and updated the comments
- Renamed the field in the configuration structure to something generic
- Added two new constants and the warning codes
10 years ago
Duarte Silva 4e04cd2d1b Adding XFF support to EVE alert output
- Created app-layer-htp-xff.c and app-layer-htp-xff.h
- Added entries in the Makefile.am
- Added the necessary configuration options to EVE alert section
- Updated Unified2 XFF configuration comments and removed unnecessary whitespace
- Created a generic function to parse the configuration
- Release the flow locks sooner and remove debug logging
- Added XFF support to EVE alert output
10 years ago
DIALLO David bacbe113d0 Add a warning in Modbus section of YAML file to remind user to modify stream depth (unlimited) 10 years ago
Victor Julien c174c9d779 af-packet: threads: auto, default to workers
Add a new default value for the 'threads:' setting in af-packet: "auto".
This will create as many capture threads as there are cores.

Default runmode of af-packet to workers.
10 years ago
Victor Julien a95c95f74c stats: introduce global config
As the stats api calls the loggers at a global interval, the global
interval should be configured globally.

 # global stats configuration
 stats:
   enabled: yes
   # The interval field (in seconds) controls at what interval
   # the loggers are invoked.
   interval: 8

If this config isn't found, the old config will be supported.
10 years ago
DIALLO David 5a0409959f App-layer: Add Modbus protocol parser
Decode Modbus request and response messages, and extracts
MODBUS Application Protocol header and the code function.

In case of read/write function, extracts message contents
(read/write address, quantity, count, data to write).

Links request and response messages in a transaction according to
Transaction Identifier (transaction management based on DNS source code).

MODBUS Messaging on TCP/IP Implementation Guide V1.0b
(http://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf)
MODBUS Application Protocol Specification V1.1b3
(http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b3.pdf)

Based on DNS source code.

Signed-off-by: David DIALLO <diallo@et.esia.fr>
10 years ago
Tom DeCanio e5c36952d6 app-layer-smtp: move old smtp-mime section in suricata.yaml into
app-layer-protocols.smtp.mine section and update code to accomodate.
10 years ago
Tom DeCanio f1c160ed22 smtp: turn on smtp mime decoding and enable smtp eve logging. 10 years ago
David Abarbanel c2dc686742 SMTP MIME Email Message decoder 10 years ago
Giuseppe Longo 07fffa6a7d Fixes comments for pfring section in suricata.yaml
Bug #1301
11 years ago
Victor Julien d44cb3f6fe pcap-log: add option to honor pass rules
Add option (disabled by default) to honor pass rules. This means that
when a pass rule matches in a flow, it's packets are no longer stored
by the pcap-log module.
11 years ago
Jason Ish 5b9c6d4774 Comment out in the action-order section, as its not needed if
the default configuration is used.
11 years ago
Victor Julien 936db9c02a output-lua: add config to yaml
Disabled by default.
11 years ago
Victor Julien 47cd497447 yaml: add eve flow and netflow entries
Added, commented out by default.

Bug #1257.
11 years ago
Victor Julien cd78705e3a streaming-loggers: add configuration
Add a (disabled by default) config to the yaml
11 years ago
Giuseppe Longo b188d93630 json-alert: include HTTP info on the alerts
Extends the JSON alert output to include the HTTP data
at the time of the alert.
11 years ago
Victor Julien 2bcd48bc12 pcap-log: yaml comment update 11 years ago
Victor Julien 0ac94ef777 flow-recycler: support multiple instances
Use new management API to run the flow recycler.

Make number of threads configurable:

flow:
  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  managers: 2
  recyclers: 2

This sets up 2 flow recyclers.
11 years ago
Victor Julien e0841218f0 flow-manager: support multiple instances
Use new management API to run the flow manager.

Support multiple flow managers, where each of them works with it's
own part of the flow hash.

Make number of threads configurable:

flow:
  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  managers: 2

This sets up 2 flow managers.

Handle misc tasks only in instance 1: Handle defrag hash timeout
handing, host hash timeout handling and flow spare queue updating
only from the first instance.
11 years ago