Commit Graph

102 Commits (8a5710307d3b2dd144ea7d1ba884441695721cf9)

Author SHA1 Message Date
Victor Julien 84f14438c3 Bug 980: fix HTTP memory cleanup at shutdown
Buffers in per thread HTTP header, client body and server body storage
would be freed based on the usage indicator instead of the size
indicator.

As the usage indicator (e.g. hsbd_buffers_list_len) could be reset
while leaving the memory untouched for later reuse, the free function
would not iterate over all memory blocks.

Removed DrMemory suppressions as well.

Bug #980.
11 years ago
Eric Leblond 55108167e5 prscript: add --norebase option
If --norebase option is provided, the prscript will start a build
that can be used to check if an out-of-sync branch pass the test.
11 years ago
Eric Leblond 7cc87bc02a coccinelle: protecting regexp operator is not needed
It seems there was an evolution of coccinelle and the protection
of regexp is not necessary anymore. And doing it causing the
expression not to match.
11 years ago
Eric Leblond 1fbaebad63 coccinelle: add test on realloc
If we use SCRealloc like:
 x = SCRealloc(x, ...)
then in case of failure we are loosing the original pointer value
and the memory is lost and can not be free.

This test just check for this construction and output an error if
it finds it.
11 years ago
Victor Julien 01c440bf28 Add DrMemory suppression for Bug #980. Suppress useless (likely) buggy leak message too 11 years ago
Victor Julien 1509c9a2e6 Remove DrMemory suppressions for Bug #979, it is fixed. 11 years ago
Eric Leblond 5a7ad5b382 qa: prscript now output pastable line for PR. 11 years ago
Eric Leblond a597237aed coccinelle: fix malloc test
We can have more than an identifier to be assigned the result of
a malloc function.
11 years ago
Eric Leblond 6378db89f6 coccinelle: add option to continue on errors
When a script has been updated or introduced, it is interesting to
detect all errors at once. With this patch it is now possible to
do so by using:
   NOT_TERMINAL=1 CONCURRENCY_LEVEL=12  qa/coccinelle/run_check.sh
11 years ago
Eric Leblond a8fde0112e prscript: add support for pcap build
Now also start a pcap test build.
11 years ago
Victor Julien 614133b4ca valgrind: add suppression file 11 years ago
Eric Leblond 0a1ca02b3b coccinelle: implement parallel check
This patch is an implementation of parallel check of files. It uses
GNU parallel to run multiple spatch at once.
The concurrency level is set via the CONCURRENCY_LEVEL environment
variable.
11 years ago
Victor Julien 1180f633e2 Add DrMemory suppress file
The suppress file currently suppresses:
- bug #978
- bug #979

Plus a seemingly harmeless warning that happens during libmagic init.

DrMemory is a valgrind like memory checker: http://www.drmemory.org/
11 years ago
Eric Leblond 013e7a0573 prscript: update code following buildbot upgrade
The authentication scheme did change on the buildbot due to a
software upgrade. This patch update prscript.py to fix the build
submission.
11 years ago
Eric Leblond fa861b09ac prscript: support bigger PR
The script now looks for originan HEAD in 100 commits instead of 30.
It should be enough becasue a sane PR should not have 100 commits.
11 years ago
Eric Leblond ad1546d59a prscript: display url where user can watch build 11 years ago
Eric Leblond 8a96296b4a prscript: add verbose option 11 years ago
Eric Leblond f23556dcdb prscript: exit when no build exists 11 years ago
Eric Leblond c151b218f1 prscript: check if branch is synced with master
The script now check if the tested branch is in sync with current
inliniac's master.
11 years ago
Eric Leblond c390006aee script: add script to start personal builder
This script HAS to be used by developer having an account on Suricata
buildbot. It MUST be run before doing a PR. It will trigger a build on
the branch and this will check the validity of the proposed branch.
The cinematic is simple:
 - Push branch XXX to github
 - Run 'prscript.py -u USER -p PASSWORD XXX'
 - Wait for the result
 - If successful, PR can be done
11 years ago
Eric Leblond 867a44f378 autotools: all target are conditional 11 years ago
Eric Leblond 9212ff7e59 coccinelle: dynamic testing through make check
This patch modify build system to have make to create the
struct-flags.cocci file by running struct-flags.py.

This way 'make check' is running the test defined from source
code.
11 years ago
Eric Leblond e05fd7f1d0 coccinelle: add script to generate flags test
This patch adds a script which can be used to generate a test
on coherence of flag usage.

By adding comment in the code, it is possible to declare that we
link a flag in a structure to a specific family of constant:

For example:
	/* coccinelle: Packet:flowflags:FLOW_PKT_ */
will trigger the generation on a test which verifies that the
flowflags field in Packet structure is only used with constant
starting by FLOW_PKT_.
11 years ago
Eric Leblond ce95fbdda0 action handling: add test to avoid direct access
Direct access to the action field of Packet structure is not
allowed.
11 years ago
Eric Leblond 41ebfa77af coccinelle: update pkt not set test
This patch updates the test to add the support of initialization
of a Packet via the INITIALIZE macro.
12 years ago
Eric Leblond c9d90e6596 coccinelle: add tcp flag check
The different TCP related structures have all a flags field and its
value must match the type of structure. This patch adds a check
alerting on invalid value usage.
12 years ago
Eric Leblond aa9f795800 cocci test: add sizeof test
This patch adds a new semantic patch taken from
http://coccinellery.org/. This patch tests if a sizeof take size
of pointer and not of pointed value.
12 years ago
Eric Leblond 7293040ed8 coccinelle: fix distcheck
distcheck is running run_check.sh from another directory and
run_check.sh was not ready for this.
12 years ago
Eric Leblond 8199832d71 coccinelle: improve run_check
This patch adds two features to run_check.sh, it is now posssible
to specify a list of files to check:
 ./run_check.sh ../../src/suricata.c ../../src/detect.c
It is also possible to ask a review of the files modified by a commit.
To so simply put the SHA1 as argument
 ./run_check.sh HEAD
 ./run_check 6af7d5f
It is also possible to check all the files for an arbitrary range:
 ./run_check.sh origin/master..buildbot-fixes

Last improvement of this patch is to get a real error message in case
of problem as 2 is not redirected anymore to /dev/null.
12 years ago
Eric Leblond 6e5caf6838 coccinelle: add new correct case to error treatment 12 years ago
Eric Leblond 28ca36acf7 coccinelle: add test on malloc error check.
This patch adds a coccinelle code check on SCMalloc, SCCalloc and
SCStrdup and other memory handling functions. It verifies that the
error checking is made.
12 years ago
Eric Leblond 3d2998a9cf coccinelle: don't test UNITTEST code 12 years ago
Eric Leblond c36aa041f3 Update coccinelle script to match syntax evolution. 12 years ago
William 7d07b5375e Add simple socket to gzip file PoC. 13 years ago
Eric Leblond 238cad77e2 coccinelle: test for invalid size_t printing. 13 years ago
Victor Julien 3693a7a9ee Profiling: add accounting for several detection phases. 13 years ago
Victor Julien 5c8feb3851 profiling / qa: make plot-csv-large-all.sh much more flexible. 13 years ago
Victor Julien a7182353e7 Remove vim .swp file from repo. 13 years ago
Victor Julien d45e01e129 Add a few more example gnuplot scripts. 13 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
13 years ago
Victor Julien c3c03b5d77 Add qa/wirefuzz.pl to release tarball. 14 years ago
Eric Leblond 7227f93032 Add coccinelle files
This patch adds coccinelle related files to EXTRA_DIST. This fixes
make distcheck.
14 years ago
William Metcalf 663d03c0e9 Add -z option for excluding pcaps from fuzzing.. What you don't want to fuzz a 750G pcap? 14 years ago
Victor Julien 076d77cd80 Add strncpy and strncat to banned function list as we have better replacements: strlcpy and strlcat. 14 years ago
Eric Leblond 7f1a0d1ed1 coccinelle: add test for banned function
The added semantic patch will trigger an error if banned functions
are used.
14 years ago
Eric Leblond 91213d5ec8 Add option to run_check script
If given an argument run_check.sh will test this file against
the cocci patches.
14 years ago
Eric Leblond 12369b4393 Coccinelle: test invalid Packet usage
This coccinelle patches is checking that there is no direct
use of p->pkt or p->pktlen in the code. This variable must be
acceded via GET_PKT_* macros.
14 years ago
Eric Leblond 7c841e1d7c Add coccinelle check to 'make check'
This patch adds coccinelle checking to the autotools
'make check'.
14 years ago
Eric Leblond d151314b4d Import coccinelle test
This is a import of two coccinelle patches that detect problem
on Packet handling. They are run on all commited C files in src
by the script run_check.sh.
14 years ago
William Metcalf 6e8f572724 small operator fixes to qa script 15 years ago
William Metcalf c6bf08eec8 Updates to the fuzzer script. Some clean up but you can now also: 1. Keep log files. 2. Exclude files based on user supplied regex. 15 years ago
William Metcalf 6817d41555 Import of fuzzer script qa/wirefuzz.pl 15 years ago