You can now directly access to docker related command:
qa/prscript.py -C # create container
qa/prscript.py -s # start container
qa/prscript.py -S # stop container
The build option does not change:
qa/prscript.pu -l -d MY_BRANCH # build test of branch MY_BRANCH
This patch is changing the logic when using docker mode. We are
iterating over each builds so we know when one build is over in
the 5 seconds following the event instead of getting the result
when the builds that are checked before are others.
On OISF's build system, the two builds comes out in order so there
is no problem.
This patch change the logic of the Docker buildbot system. The
buildbot configuration is now the one available in qa/docker
directory.
This way, developers can test features in docker buildbot that
could require some specific flags to be set. They just need to
edit the buildbot.cfg for instance to enable a new feature.
In the same way, the tested pcap files are the one which are in
the qa/docker/pcaps/ directory. So to test some private ones it is
enough to put them in that directory.
To take into consideration a buildbot.cfg modification or a new
pcap, it is enough to stop and restart the container:
sudo qa/prscript.py -l -d -S master
sudo qa/prscript.py -l -d -s master
This patch also fixes the container update issue. A local modification
to the buildbot will be kept. It is also fixing the issue when working
on old code that could possibly not support the same build flag as
the one of buildbot. Here the configuration will remains in sync.
You can now create the docker image necessary to run a
suricata builbot in docker. To do that you need to have
docker and python-docker installed on the system.
Then you can go to the qa directory.
You need to run once the creattion procedure:
sudo ./prscript.py -C -d master -l
This will create a container named 'suri-buildbot'.
You can start it with:
sudo ./prscript.py -s -d master -l
And stop it with:
sudo ./prscript.py -S -d master -l
To start a test, you can do:
./prscript.py -d my_branch -l
With the current work in progress on docker we need to update the
script to handle this case.
This patch adds two options:
- -d to run in docker mode. In that case the build is started in
the local dockerized buildbot instance
- -l to not test the tested branch synchronization with github.
The -l option allows user to run a complete test without publishing
the code on github and without Internet access.
A bad last_ack update where it would be set beyond next_seq could
lead to rejection of valid segments and thus stream gaps.
Update tests to reflect new last_ack/next_seq behaviour.
The post match list was called with an unlocked flow until now.
However, recent de_state handling updates changed this. The stateful
detection code can now call the post match functions while keeping
the flow locked. The normal detection code still calls it with an
unlocked flow.
This patch adds a hint to the DetectEngineThreadCtx called
'flow_locked' that is set to true if the caller has already locked
the flow.
The detection engine and log engines can walk the tx list indirectly,
by looping AppLayerParserGetTx. This would lead to new list walks in
the DNS tx list though. Leading to bad performance.
This patch stores the last returned tx and uses that to determine if
the next tx is what we need next. If so, we can return that w/o list
walk.
Load the YAML into a prefix "detect-engine-reloads.N" where N is the
reload counter. This way we can load the updated config w/o overwriting
the current one.
Initalize detection engine by configuration prefix.
DetectEngineCtxInitWithPrefix(const char *prefix)
Takes the detection engine configuration from:
<prefix>.<config>
If prefix is NULL the regular config will be used.
Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
Add function to load a yaml file and insert it into the conf tree at
a specific prefix.
Example YAML:
somefile: myfile.txt
If loaded using ConfYamlLoadFileWithPrefix with prefix "myprefix", it
can be retrieved by the name of "myprefix.somefile".
Rename the thread init function DetectEngineThreadCtxInitForLiveRuleSwap
to DetectEngineThreadCtxInitForReload and change it's logic to take the
new detection engine as argument and let it return the
DetectEngineThreadCtx or NULL on error.
The old approach used the thread init API format, but it wasn't used in
that way.
Add DetectEngineReference, which takes a reference to a detect engine,
and make DetectEngineThreadCtxInitForLiveRuleSwap use it. This way
reload will not depend on master staying the same. This allows master
to be updated in between w/o affecting the reload that is in progress.
Use new DetectEngineReload() function. It's called from the main loop
instead of it being spawned into it's own temporary thread. This greatly
simplifies the signal handling.
An added advantage is that this seems to improve the memory usage.
Related to bug #1358
The minimal detect engine has only the minimal memory use and setup
time. It's to be used for 'delayed' detect where the first detection
engine is essentially empty.
The threads setup are also minimal.
Instead of threading logic with dummy slots and all, use the regular
reload logic for delayed detect.
This means we pass a empty detect engine to the threads and then
reload (live swap) it as soon as the engine is running.
Add code to allow for unittests not following the complete api.
Update replace tests as they don't use the unittests runmode that
powers the workaround based on RunmodeIsUnittests().
Update detect engine management to make it easier to reload the detect
engine.
Core of the new approach is a 'master' ctx, that keeps a list of one or
more detect engines. The detect engines will not be passed to any thread
directly, but instead will only be accessed through the detect engine
thread contexts. As we can replace those atomically, replacing a detect
engine becomes easier.
Each thread keeps a reference to its detect context. When a detect engine
is replaced or removed, it's added to a free list. Once its reference
count reaches 0, it is freed.
Eric Leblond
Jason Ish
Alexander Gozman
Victor Julien