Commit Graph

2251 Commits (a7182353e7ec801146e25d54255a05a649453a32)
 

Author SHA1 Message Date
Victor Julien a7182353e7 Remove vim .swp file from repo. 13 years ago
Anoop Saldanha 58b595cc21 fastlog print updates for ipv6. combine the io write 13 years ago
Anoop Saldanha e8f9557664 fastlog print updates. combine the io write 13 years ago
Victor Julien d45e01e129 Add a few more example gnuplot scripts. 13 years ago
Victor Julien fca541f40e Add per app layer parser profiling
Per packet per app layer parser profiling. Example summary output:

Per App layer parser stats:

App Layer              IP ver   Proto   cnt        min      max          avg
--------------------   ------   -----   ------     ------   ----------   -------
ALPROTO_HTTP            IPv4       6    163394        126     38560320     42814
ALPROTO_FTP             IPv4       6       644        117        26100      2566
ALPROTO_TLS             IPv4       6       670        117         7137       799
ALPROTO_SMB             IPv4       6    114794        126       225270       957
ALPROTO_DCERPC          IPv4       6      5207        126        25596      1266

Also added to the csv out.

In the csv out there is a new column "stream (no app)" that removes the
app layer parsers from the stream tracking. So raw stream engine performance
becomes visible.
13 years ago
Victor Julien 0cc9f39200 Move TlsConfig structure out of app-layer-protos.h and rename it to SslConfig. 13 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
13 years ago
Victor Julien 1bd1a62526 Rename profile macro's and variables to reflect that they are for rule profiling. 13 years ago
Eric Leblond 88559901d4 pcap-file: Allocated packet must be free if there's error 13 years ago
Eric Leblond f6628f140d detect: fix regular expression used for check. 13 years ago
Eric Leblond a354034cfc nfq: Fix deinit phase
If receive thread is failling, we need to restart it but the code was
not restarting the queue (this was done in verdict thread).
13 years ago
Eric Leblond eddcedba0a nfq: make thread abort if NFQ verdict fail 13 years ago
Eric Leblond 2ffcef0a8e nfq: Add iterator on nfq_set_verdict
This patch adds retry to nfq_set_verdict in case of error.
13 years ago
Eric Leblond a8b21066df tm-thread: fix documentation string 13 years ago
Eric Leblond a8ae1c42c3 Fix macro about default packet size
Being pessimistic about packet default size has side effect in
some module. Falling back to the sane correct value.
13 years ago
Eric Leblond 685e0e1a63 Rename rule_type_t to ThresholdRuleType. 13 years ago
Eric Leblond 8787e6f6d0 suppress: use DetectAddress instead of DetectAddressHead 13 years ago
Eric Leblond 8ff8ec4f82 Export some DetectAddress related function. 13 years ago
Eric Leblond 7938344e1b threshold: refactoring of parsing code
This patch factorize the regular expression to be ease the parsing
process. It also adds a missing free and factorize exit code.
13 years ago
Eric Leblond 03c185a3ad threshold: add suppress keyword
This patch adds the suppress keyword to the threshold.config file.
The alerts are suppressed but the other elements like flowbits are
maintained.
13 years ago
Eric Leblond 85e8d8e200 Add sanity check to DetectAdressParse.
The function is only used at parsing time, this is not costly to add
a simple sanity check.
13 years ago
Eric Leblond 7168e0aafc threshold: fix trivial typo in parsing. 13 years ago
Eric Leblond a56f8dd6b2 doc: introduce doxygen group "threshold"
This patch introduces a doxygen group to put together the documentation
relative to threshold. Group appear in a separate page and they can have
their own documentation. This is useful when a feature is splitted into
different files.
13 years ago
Victor Julien dc218388e5 Fix flowint keyword pcre_get_substring issue. 13 years ago
Victor Julien 1740c3a7c7 Fix urilen keyword pcre_get_substring issue. 13 years ago
Victor Julien f52b54f63e Fix ssl keyword pcre_get_substring issue. 13 years ago
Eric Leblond 6b9d1012ff Transform inet_ntop call into PrintInet one. 13 years ago
Eric Leblond 2fa07780c2 Introduce PrintInet function
This function has the same signature than inet_ntop() and it
will be used as substitution in the code. For IPv4 this is a simple
wrapper. For IPv6, it display addresses with fixed length.
13 years ago
Victor Julien 7e1d911215 Small optimizations to pkt acq loop code. 13 years ago
Victor Julien b753ecce50 Implement a pkt acq loop infra with support for pcap-file. 13 years ago
Anoop Saldanha 975ebf2e4f Minor changes to move function calls that kills threads + frees resources to the clean up phase right to the end of main thread 13 years ago
Anoop Saldanha ff7284e7b7 Fix code that allows the engine to restart threads that have exited on failure 13 years ago
Anoop Saldanha 524af82b1a code cleanup in tm-threads.c 13 years ago
Anoop Saldanha 4f7df1029d Unify the use of slots to a single struct for threading API. Remove separate slot append functions for 1slot and varslot 13 years ago
William 6730c3ace1 Actually limit recursion and backtracking and stack usage by PCRE. Logic was broken, no example was provided in suricata.yaml even though it could be set from there. 13 years ago
William 61fe05b220 Fix for silly pcap counters mistake made by me. ps_recv includes dropped packets. 13 years ago
William 6408febaec Only check for PF_RING if we enable PF_RING. 13 years ago
William b3f7e6a2fc Only set PF_RING cluster if we have more than one receive thread. Gives us accurate drop stats. 13 years ago
Anoop Saldanha d3bc3f0fe5 coverity fix for counters api 13 years ago
Anoop Saldanha be3996ac02 coverity fix - 1.1beta branch - add some comments to indicate false positives by coverity for future reference - mainly comments for switch statement fall through 13 years ago
Victor Julien df3ca322a4 Fixes for out of bounds pcre_get_substring calls no longer silently accepted by modern pcre. 13 years ago
William 1099093e0f Support for PF_RING versions where packet passed as a reference and version 4.7.1 where pfring_enable_ring now seems to be required. 13 years ago
Eric Leblond a0b4068041 autotools: fix duplicate check command in Makefile.
It seems that check target can not be used in Makefile.am. Using
check-am fix a make failure.
13 years ago
Eric Leblond 586aae0ff3 Indentation fix on source-pcap. 13 years ago
Anoop Saldanha c8701cf8d1 fix var name parsing in byte_extract 13 years ago
Anoop Saldanha 7e5c52c80b add flowbits:set; only sigs to be treated as ip only 13 years ago
William bca8fbc79e Add Num, Rev, and Gid columns to rule perf output 13 years ago
Victor Julien 0625d54267 Improve HTPParserTest07 test to be more helpful if it fails. 13 years ago
Victor Julien 862b708a70 Fix stream unittest. 13 years ago
Anoop Saldanha 88115902b0 Have separate parser vars in smtp to hold dynamic buffers for parsing fragmented lines 13 years ago