Remove all strtok uses and replace them by strtok_r.
Do the same for Windows builds. Cygwin builds fine with strtok_r.
Add strtok to banned function list.
Handle cleanly the case where user start docker handling command
without having the dependency installed. The help message does
now say:
```
-d, --docker use docker based testing
-C, --create create docker container (disabled)
-s, --start start docker container (disabled)
-S, --stop stop docker container (disabled)
You need to install Python docker module to enable docker container handling
options.
```
And running a disabled options says:
```
$ qa/prscript.py -s
You need to install python docker to use docker handling features.
```
You can now directly access to docker related command:
qa/prscript.py -C # create container
qa/prscript.py -s # start container
qa/prscript.py -S # stop container
The build option does not change:
qa/prscript.pu -l -d MY_BRANCH # build test of branch MY_BRANCH
This patch is changing the logic when using docker mode. We are
iterating over each builds so we know when one build is over in
the 5 seconds following the event instead of getting the result
when the builds that are checked before are others.
On OISF's build system, the two builds comes out in order so there
is no problem.
This patch change the logic of the Docker buildbot system. The
buildbot configuration is now the one available in qa/docker
directory.
This way, developers can test features in docker buildbot that
could require some specific flags to be set. They just need to
edit the buildbot.cfg for instance to enable a new feature.
In the same way, the tested pcap files are the one which are in
the qa/docker/pcaps/ directory. So to test some private ones it is
enough to put them in that directory.
To take into consideration a buildbot.cfg modification or a new
pcap, it is enough to stop and restart the container:
sudo qa/prscript.py -l -d -S master
sudo qa/prscript.py -l -d -s master
This patch also fixes the container update issue. A local modification
to the buildbot will be kept. It is also fixing the issue when working
on old code that could possibly not support the same build flag as
the one of buildbot. Here the configuration will remains in sync.
You can now create the docker image necessary to run a
suricata builbot in docker. To do that you need to have
docker and python-docker installed on the system.
Then you can go to the qa directory.
You need to run once the creattion procedure:
sudo ./prscript.py -C -d master -l
This will create a container named 'suri-buildbot'.
You can start it with:
sudo ./prscript.py -s -d master -l
And stop it with:
sudo ./prscript.py -S -d master -l
To start a test, you can do:
./prscript.py -d my_branch -l
With the current work in progress on docker we need to update the
script to handle this case.
This patch adds two options:
- -d to run in docker mode. In that case the build is started in
the local dockerized buildbot instance
- -l to not test the tested branch synchronization with github.
The -l option allows user to run a complete test without publishing
the code on github and without Internet access.
coccinelle 1.0rc21 has a problem with regular expression handling.
This result in a Fatal Error when test system detects an coding
error.
This patch fixes the problem by using a simple blob inside
semantic patch instead of using a regular expression to define
the function.
It also fixes add an optimization on matching suppressing a
useless <.. ..> construction.
Fixes have been suggested by Julia Lawall.
Buffers in per thread HTTP header, client body and server body storage
would be freed based on the usage indicator instead of the size
indicator.
As the usage indicator (e.g. hsbd_buffers_list_len) could be reset
while leaving the memory untouched for later reuse, the free function
would not iterate over all memory blocks.
Removed DrMemory suppressions as well.
Bug #980.
If we use SCRealloc like:
x = SCRealloc(x, ...)
then in case of failure we are loosing the original pointer value
and the memory is lost and can not be free.
This test just check for this construction and output an error if
it finds it.
When a script has been updated or introduced, it is interesting to
detect all errors at once. With this patch it is now possible to
do so by using:
NOT_TERMINAL=1 CONCURRENCY_LEVEL=12 qa/coccinelle/run_check.sh
This patch is an implementation of parallel check of files. It uses
GNU parallel to run multiple spatch at once.
The concurrency level is set via the CONCURRENCY_LEVEL environment
variable.
The suppress file currently suppresses:
- bug #978
- bug #979
Plus a seemingly harmeless warning that happens during libmagic init.
DrMemory is a valgrind like memory checker: http://www.drmemory.org/
This script HAS to be used by developer having an account on Suricata
buildbot. It MUST be run before doing a PR. It will trigger a build on
the branch and this will check the validity of the proposed branch.
The cinematic is simple:
- Push branch XXX to github
- Run 'prscript.py -u USER -p PASSWORD XXX'
- Wait for the result
- If successful, PR can be done
This patch modify build system to have make to create the
struct-flags.cocci file by running struct-flags.py.
This way 'make check' is running the test defined from source
code.
This patch adds a script which can be used to generate a test
on coherence of flag usage.
By adding comment in the code, it is possible to declare that we
link a flag in a structure to a specific family of constant:
For example:
/* coccinelle: Packet:flowflags:FLOW_PKT_ */
will trigger the generation on a test which verifies that the
flowflags field in Packet structure is only used with constant
starting by FLOW_PKT_.
The different TCP related structures have all a flags field and its
value must match the type of structure. This patch adds a check
alerting on invalid value usage.
This patch adds two features to run_check.sh, it is now posssible
to specify a list of files to check:
./run_check.sh ../../src/suricata.c ../../src/detect.c
It is also possible to ask a review of the files modified by a commit.
To so simply put the SHA1 as argument
./run_check.sh HEAD
./run_check 6af7d5f
It is also possible to check all the files for an arbitrary range:
./run_check.sh origin/master..buildbot-fixes
Last improvement of this patch is to get a real error message in case
of problem as 2 is not redirected anymore to /dev/null.
This patch adds a coccinelle code check on SCMalloc, SCCalloc and
SCStrdup and other memory handling functions. It verifies that the
error checking is made.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.
Stats per packet include:
1) total ticks spent
2) ticks spent per individual thread module
3) "threading overhead" which is simply calculated by subtracting (2) of (1).
A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.
Code depends on --enable-profiling just like the rule profiling code.
New yaml parameters:
profiling:
# packet profiling
packets:
# Profiling can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: yes
filename: packet_stats.log
append: yes
# per packet csv output
csv:
# Output can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: no
filename: packet_stats.csv
Example output of summary stats:
IP ver Proto cnt min max avg
------ ----- ------ ------ ---------- -------
IPv4 6 19436 11448 5404365 32993
IPv4 256 4 11511 49968 30575
Per Thread module stats:
Thread Module IP ver Proto cnt min max avg
------------------------ ------ ----- ------ ------ ---------- -------
TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770
TMM_DETECT IPv4 6 19436 1107 137241 1504
TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155
TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138
TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154
TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944
TMM_STREAMTCP IPv4 6 19434 828 1299159 19438
The proto 256 is a counter for handling of pseudo/tunnel packets.
Example output of csv:
pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337
First line of the file contains labels.
2 example gnuplot scripts added to plot the data.
This coccinelle patches is checking that there is no direct
use of p->pkt or p->pktlen in the code. This variable must be
acceded via GET_PKT_* macros.
This is a import of two coccinelle patches that detect problem
on Packet handling. They are run on all commited C files in src
by the script run_check.sh.
Victor Julien
Andreas Herz
Eric Leblond
Jason Ish
William
William Metcalf