diff --git a/doc/userguide/output/eve/eve-json-output.rst b/doc/userguide/output/eve/eve-json-output.rst index 580504d1f7..7700725105 100644 --- a/doc/userguide/output/eve/eve-json-output.rst +++ b/doc/userguide/output/eve/eve-json-output.rst @@ -17,6 +17,32 @@ can then be processed by 3rd party tools like Logstash (ELK) or jq. If ``ethernet`` is set to yes, then ethernet headers will be added to events if available. +Output Buffering +~~~~~~~~~~~~~~~~ + +Output flushing is controlled by values in the configuration section ``heartbeat``. By default, Suricata's +output is synchronous with little possibility that written data will not be persisted. However, if ``output.buffer-size`` +has a non-zero value, then some data may be written for the output, but not actually flushed. ``buffer-size`` bytes +may be held in memory and written a short time later opening the possibility -- but limited -- for output data +loss. + +Hence, a heartbeat mechanism is introduced to limit the amount of time buffered data may exist before being +flushed. Control is provided to instruct Suricata's detection threads to flush their EVE output. With default +values, there is no change in output buffering and flushing behavior. ``output-flush-interval`` controls +how often Suricata's detect threads will flush output in a heartbeat fashion. A value of ``0`` means +"never"; non-zero values must be in ``[1-60]`` seconds. + +Flushing should be considered when ``outputs.buffer-size`` is greater than 0 to limit the amount and +age of buffered, but not persisted, output data. Flushing is never needed when ``buffer-size`` is ``0``. + + +:: + + heartbeat: + #output-flush-interval: 0 + + + Output types ~~~~~~~~~~~~ @@ -30,6 +56,10 @@ Output types:: # Enable for multi-threaded eve.json output; output files are amended # with an identifier, e.g., eve.9.json. Default: off #threaded: off + # Specify the amount of buffering, in bytes, for + # this output type. The default value 0 means "no + # buffering". + #buffer-size: 0 #prefix: "@cee: " # prefix to prepend to each log entry # the following are valid when type: syslog above #identity: "suricata" @@ -245,7 +275,7 @@ In the ``custom`` option values from both columns can be used. The DNS ~~~ -.. note:: +.. note:: As of Suricata 7.0 the v1 EVE DNS format has been removed. diff --git a/doc/userguide/partials/eve-log.yaml b/doc/userguide/partials/eve-log.yaml index 4897ac5685..845d7e1157 100644 --- a/doc/userguide/partials/eve-log.yaml +++ b/doc/userguide/partials/eve-log.yaml @@ -7,6 +7,10 @@ outputs: # Enable for multi-threaded eve.json output; output files are amended with # an identifier, e.g., eve.9.json #threaded: false + # Specify the amount of buffering, in bytes, for + # this output type. The default value 0 means "no + # buffering". + #buffer-size: 0 #prefix: "@cee: " # prefix to prepend to each log entry # the following are valid when type: syslog above #identity: "suricata" @@ -280,3 +284,14 @@ outputs: # event-set: false # log packets that have a decoder/stream event # state-update: false # log packets triggering a TCP state update # spurious-retransmission: false # log spurious retransmission packets + # +heartbeat: + # The output-flush-interval value governs how often Suricata will instruct the + # detection threads to flush their EVE output. Specify the value in seconds [1-60] + # and Suricata will initiate EVE log output flushes at that interval. A value + # of 0 means no EVE log output flushes are initiated. When the EVE output + # buffer-size value is non-zero, some EVE output that was written may remain + # buffered. The output-flush-interval governs how much buffered data exists. + # + # The default value is: 0 (never instruct detection threads to flush output) + #output-flush-interval: 0