The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
The log API calls thread modules directly, so the TMM profiling logic
can be applied to it. This patch does so.
The "Thread Module" out now again lists the individual loggers. As the
module are normally called much less frequently the numbers are hard to
compare to pre-log-api numbers.
Add option "profiling.sample-rate":
# Run profiling for every xth packet. The default is 1, which means we
# profile every packet. If set to 1000, one packet is profiled for every
# 1000 received.
#sample-rate: 1000
This allows for configuration of the sample rate.
Instead of a large (6k+) structure in the Packet, make the profiling
storage dynamic. To do this the Packet->profile is now a pointer.
Initial support for selective sampling, e.g. only profile every
1000th packet.
A negated match is matching if the tested field is NULL. But as it
is not set, nor negated nor normal test must match.
Without this patch, a rule like:
alert tls any any -> any any (msg:"negated match"; tls.subject:!"CN=home.regit.org"; sid:1; rev:1;)
is alerting for all connections. Event if they are done on a certificate
with matching subject. This was due to the fact that tls protocol
is discovered before the handshake is complete. Thus the condition
on tls is true with a NULL tls.subject. And code was returning a
positive match in the case of a NULL subject and a signature with
a negated match.
Introduce a utility function to convert an array of bytes into a
null-terminated string:
char *BytesToString(const uint8_t *bytes, size_t nbytes);
All non-printables are copied over, except for '\0', which is
turned into literal '\' '0' in the string. So the resulting string
may be bigger than the input.
In various places SCStrndup was used to 'dup' a bstr string, however
libhtp provides bstr_util_strdup_to_c for this. As this is a cleaner
interface, it's preferred.
This patch separates http keys from file to have a different value
list:
{
"time":"01\/31\/2014-12:04:52.837245","event_type":"file","src_ip":"5.3.1.1","src_port":80,"dest_ip":"1.8.1.9","dest_port":9539,"proto":"TCP",
"http":{"url":"/foo/","hostname":"bar.com","http_refer":"http:\/\/bar.org","http_user_agent":"Mozilla\/5.0"},
"file":{"filename":"bar","magic":"unknown","state":"CLOSED","stored":false,"size":21}
}
One interest of this modification is that it is possible to use the
same key as the one used in http events. Thus correlating both type
of events is trivial. On code side, this will permit to factorize
the code by simply asking the underlying protocol to output its
info in a json object.
Second interest is that adding file extraction for a new protocol
will result in only changing the protocol specific json list.
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
Without this patch DNS answers for a single query are stored in a
single json event. The result is an array in the object like this one:
{"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com","rrtype":"CNAME","ttl":734},
{"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com.edgekey.net","rrtype":"CNAME","ttl":1710},
This type of output is not well supported in logstash. It is
displayed as it is written above and it is not possible to
query the fields.
I think the reason is that this is not logical if we consider search
query. For example if we search for "rrname" equal "s-static.ak.facebook.com"
we got one entry with two values in it. That's against the logic
of event. Furthermore, if we want to get a complete query, we can
used the id.
This patch splits the answer part in mulitple message. The result
is then accepted by logstash and fields can be queried easily.
This patch updates DNS field name to be in sync with RFC 2629:
https://github.com/adulau/pdns-qof
This will allow to easily use Suricata with other passive DNS tools.
This patch is synchronizing key name with Common Information Model.
It updates key name following what is proposed in:
http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
The interest of these modifications is that using the same key name
as other software will provide an easy to correlate and improve
data. For example, geoip setting in logstash can be applied on
all src_ip fields allowing geoip tagging of data.
Both the drop and tls logs are currently not designed to have multiple
instances running. So until that is changed, error out if more than one
instance is started.
To support the 'eve-log' idea, we need to be able to force all log
modules to be enabled by the master eve-log module, and need to be
able to make all logs go into a single file. This didn't fit the
API so far, so added the sub-module concept.
A sub-module is a regular module, that registers itself as a sub-
module of another module:
OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "http",
OutputHttpLogInitSub, ALPROTO_HTTP, JsonHttpLogger);
The first argument is the name of the parent. The 4th argument is
the OutputCtx init function. It differs slightly from the non-sub
one. The different is that in addition to it's ConfNode, it gets
the OutputCtx from the parent. This way it can set the parents
LogFileCtx in it's own OutputCtx.
The runmode setup code will take care of all the extra setup. It's
possible to register a module both as a normal module and as a sub-
module, which can operate at the same time.
Only the TxLogger API is handled in this patch, the rest will be
updated later.
- restore json output-file.[ch] as output-json-file.[ch] after rebase conflict
- fix Makefile.am after merge conflict
- some dev-log-api-v4.0 rebase json fallout cleanup
The stream chunk pool contains preallocating stream chunks (StreamMsg).
These are used for raw reassembly, used in raw content inspection by
the detection engine. The default setting so far has been 250, which
was hardcoded. This meant that in setups that needed more, allocs and
frees would be happen constantly.
This patch introduces a yaml option to set the 'prealloc' value in the
pool. The default is still 250.
stream.reassembly.chunk-prealloc
Related to feature #1093.
The stream reassembly engine uses a set of pools in which preallocated
segments are stored. There are various pools each with different packet
sizes. The goal is to lower memory presure. Until now, these pools were
hardcoded.
This patch introduces the ability to configure them fully from the yaml.
There can be at max 256 of these pools.
Yaml layout is as follows:
stream:
reassemble:
segments:
- size: 2048
prealloc: 3000
- size: 4
prealloc: 1000
- size: 1024
prealloc: 2000
The size is the packet size. The prealloc value indicates how many
segments are set up at startup.
The pools have no limit wrt how many segments can be used of a certain
size. If the engine needs more than the prealloc size, segments are
malloc'd and free'd. The only limit here is the stream.reassemble.memcap.
If the yaml part if omitted, the default values are the same as before.
Feature #1093
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.
The logger call in the API is as follows:
typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
const Packet *, const File *, const FileData *, uint8_t flags);
All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.
Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
The new API call:
int AppLayerParserProtocolHasLogger(uint8_t ipproto,
AppProto alproto)
Returns TRUE if a logger is registered on the ip/alproto pair, and
FALSE otherwise.
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.
Users of this API register their Log Function through:
OutputRegisterFileModule()
The API uses a magic settings globally. This might be changed later.
Fixes configure enabling of prelude. CFLAGS is reset, so the previous
adding of -DPRELUDE was nixed. Using AC_DEFINE now.
Cleanups:
- make functions static
- simplify handling of no prelude support
- move registration to the bottom
Remove separate ipv4 and ipv6 registration functions.
Make all functions static.
Move registration function to the bottom.
Simplify OS_WIN32 wrappers usage.
This patch converts log-tls to use the packet logger API. The packet
logger API was choosen as the TLS parser is not transaction aware.
To make sure the state is only logged once, the flag
SSL_AL_FLAG_STATE_LOGGED was added to the parser. This flag is checked
by the condition function, and set at the end of the Logger function.
Make all functions static. Remove separate ipv4 and ipv6 registration
functions. Move register function to the bottom so that we no longer
need function prototype declarations.
This patch introduces a new API for logging transactions from
tx-aware app layer protocols. It runs all the registered loggers
from a single thread module. This thread module takes care of the
transaction handling and flow locking. The logger just gets a
transaction to log out.
All loggers for a protocol will be run at the same time, so there
will not be any timing differences.
Loggers will no longer act as Thread Modules in the strictest sense.
The Func is NULL, and SetupOuputs no longer attaches them to the
thread module chain individually. Instead, after registering through
OutputRegisterTxModule, the setup data is used in the single logging
module.
The logger (LogFunc) is called for each transaction once, at the end
of the transaction.
This patch introduces a new API for outputs that log based on the
packet, such as alert outputs. In converts fast-log to the new API.
The API gets rid of the concept of each logger being a thread module,
but instead there is one thread module that runs all packet loggers.
Through the registration function OutputRegisterPacketModule a log
module can register itself to be considered for each packet.
Each logger registers itself to this new API with 2 functions and the
OutputCtx object that was already used in the old implementation.
The function pointers are:
LogFunc: the log function
ConditionFunc: this function is called before the LogFunc and only
if this returns TRUE the LogFunc is called.
For a simple alert logger like fast-log, the condition function will
simply return TRUE if p->alerts.cnt > 0.
If a protocol parser is active without a logger when detection is
disabled, the transaction handling logic would fail. Now it will
return the proper tx id so we can clean up the complete transactions.
When running w/o detect, TX cleanup handling needs to ignore the
inspect_id as it's only updated by detect.
This patch introduces a new ActiveTx handler for logging only:
AppLayerTransactionGetActiveLogOnly
If --disable-detection is passed on the commandline, this function
is registered.
Flow timeout code keeps track of thread module running detect, and
fails (hard) if it doesn't find it.
This changeset retrieves the global g_detect_disabled and passes
it to the timeout handling code during setup.
This global will be set to TRUE if detect is disabled. The reason for
adding a global is that there currently is no clean way to pass
configuration options to management threads.
If FlowForceReassemblyForFlowV2 can't get packets to inject into the
engine, until now it would bail and retry later. In case of resource
starvation issues, this would cause a lot of lock contention, as the
flow manager would try over and over again.
This patch limits FlowForceReassemblyForFlowV2 to one try per flow,
if it fails... bad luck. It will only fail in serious conditions,
which means we must prefer the health of the engine over the proper
inspection of the flow in question.
AppLayer Proto detection code uses a mix of pattern search and
"probing parsers". The pattern search validates potential matches
using a single pattern search algo. The code was using SpmSearch
for this, but this made it inefficient as it builds a BoyerMoore
context for each search. This lead to significant memory pressure,
especially on high speed/bandwidth boxes.
This patch switches the search calls to BoyerMoore and BoyerMoore-
Nocase directly. This can be done as the ctx' were available already.
Move app layer event handling into app-layer-event.[ch].
Convert 'Set' macro's to functions.
Get rid of duplication in Set and SetRaw. Set now calls SetRaw.
Fix potentential int overflow condition in the event storage.
Update callers.
Don't use FlowGetProtoMapping at runtime, use f->protomap instead.
Add safety check to make sure its value is within range, as it's
used to index an array.
Update unittests to initialized flows (somewhat).
Remove unnecessay inlining.
Rename functions with wrong naming scheme. E.g. AllocAppLayer.. instead
of AppLayer..Alloc.
Use AppProto instead of uint16_t.
Convert u16 ipproto cases to u8.
This patch updates the DNS counters from the main AppLayer entry
functions. Due to the limited scope of AppLayerThreadCtx some of
the logic had to be implemented in app-layer.c, where it doesn't
belong.
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:
- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
Add memuse tracking and memcap checking to the DNS parsers. Memuse
is tracked globally and per flow (state).
Memcaps are also checked per flow and globally before memory allocs
are done.
In some cases the TLS state pointers to subject and issuerdn could
be overwritten by a new memory allocation, causing us to loose
track of the old.
This has been observed in the case of improper VLAN handling, where
it was suspected that multiple unrelated TLS streams were mangled
together.
The loop is freeing elements so we need to use the safe version
of TIALQ_FOREACH.
This fixes a valgrind error:
Thread 1 Suricata-Main:
Invalid read of size 8
at 0x8E129C: LiveDeviceListClean (util-device.c:167)
by 0x89B742: main (suricata.c:2284)
Address 0x8382988 is 24 bytes inside a block of size 40 free'd
at 0x4C2A70C: free (vg_replace_malloc.c:468)
by 0x8E1297: LiveDeviceListClean (util-device.c:179)
by 0x89B742: main (suricata.c:2284)
Buffers in per thread HTTP header, client body and server body storage
would be freed based on the usage indicator instead of the size
indicator.
As the usage indicator (e.g. hsbd_buffers_list_len) could be reset
while leaving the memory untouched for later reuse, the free function
would not iterate over all memory blocks.
Removed DrMemory suppressions as well.
Bug #980.
Logic of patch 98e4a14f6d was correct
but implementation is wrong because TP_STATUS_KERNEL is equal to
zero and thus can not be evaluated in a binary operation. This patch
updates the logic by doing two tests.
Reported-by: Alessandro Guido
In preparation of a patchset that will allow for disabling the detect
module, this patch introduces a way to register a function for getting
the lowest active tx id. This is used by the app layer for cleaning up
transactions that already fully inspected, and by the flow timeout code
to determine if a flow is fully inspected and logged at timeout.
The registration function RegisterAppLayerGetActiveTxIdFunc allows for
registration of a custom function of type:
uint64_t (*GetActiveTxIdFunc)(Flow *f, uint8_t flags);
If no function is called, AppLayerTransactionGetActiveDetectLog is used,
which implements the existing behaviour of considering both the
inspect_id's and the log_id.
In AppLayerTransactionsCleanup instead of figuring out 'done' tx id's
itself, now call AppLayerTransactionGetActive for both directions to
figure out the completed TX id's.
StreamMsgs would be stored in a per thread queue before being
attached to the tcp ssn. This is unnecessary, so this patch
removes this queue and puts the smsgs into the ssn directly.
Large patch as it affects a lot of tests.
StreamMsg' flow reference was used mostly to make sure a flow would
not get removed from the hash before inspection. For this it needed
to reference the flow use_cnt reference counter. Nowadays we have
more advanced flow timeout handling. This will make sure that if
there still are pending smsgs' in a flow, these will still be
processed.
Preparation for removing flow pointer from StreamMsg. Instead of
getting the ssn indirectly through StreamMsg->flow, we pass it
directly as all callers have it already.
StreamSmgs are used for raw stream reassembly only. They could also
be used to tell the rest of the engine about sequence gaps. This was
a left over from the older implementation, where the app layer used
the smsgs as well.
This patch updates the logic of the packet acquisition loop. When
the reader loop function is called and when the data to read
at offset is a without data (kernel) or still used by suricata. We
try to iter for a loop on the ring to try to find kernel put by
data.
As we are entering the function because the poll said there was some
data. This allow us to jump to the data added to the ring by the
kernel.
When using suricata in autofp mode, with multiple detect threads and
packet acquisition threads attached to a dedicated CPU, the reader
loop function was looping really fast because poll call was returning
immediatly because we did read the data available.
This patch adds a memory counter for HTP memory usage. As
there is no thread variables available in application layer
the counter has been added to the TCP reassembly thread.
This patch introduces wrapper functions around allocation functions
to be able to have a global HTP memcap. A simple subsitution of
function was not enough because allocated size needed to be known
during freeing and reallocation.
The value of the memcap can be set in the YAML and is left by default
to unlimited (0) to avoid any surprise to users.
When running with sgh-mpm-context: full, many more MPMs are created
(16K) and many are small. If they have less than 128 states, they only
need 1 byte for the next state instead of 2 bytes, cutting the size of
the next-state table in half. This reduces total memory usage.
Since that makes 3 different state sizes (1, 2 and 4 bytes), rather
than going from 2 copies of the code to create the MPM to 3, I
factored out the code that fills the next-state table into three
functions so that all the other code could be the same.
The search function is now parameterize for 8-bit and 16-bit state
sizes and alphabet sizes 8, 16, 32, 64, 128 and 256.
Live device counter was in fact the number of packets seen by suricata
and not the total number of packet reported by pfring. This patch fixes
this by using counter provided by kernel instead.
Pfring kernel counter is per socket and is not cleared after read.
So to get the number of packet on the interface we can add the new
value for this thread and add it to the interface counter.
Live device counter was in fact the number of packets seen by suricata
and not the total number of packet reported by kernel. This patch fixes
this by using counter provided by kernel instead.
The counter is Clear On Read, so by adding the value fetch at each call
and earch sockets we get the number of packets and drops for the
interface.
- Seems to be a regression introduced in the commit
796bfab231 (fix was already done in commit
ee0b21652b)
- Doesn't happen with htplib v0.5.6, but it does in the latest, v0.5.9
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.
AppLayerParserState is declared in flow.h as it's part of the Flow
structure.
AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
A number of unittests would lead to clang build errors because
of unsafe det_ctx ptr usage. This patch fixes these and inits
det_ctx to NULL in the other detect tests.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
of the archaic features we use in the app layer. We will reintroduce this
parser shortly. Also do note that keywords that rely on the ssh parser
would now be disabled.
Coverity 1139544
If strdup would fail, 'node' was freed but it wasn't set to NULL. The
code then returned node. The caller would not detect there was an error
and use the freed pointer.
By moving FlowReference() out of FlowGetFlowFromHash() and into the one
function that calls it, all the flow functions take const Packet * instead
of Packet *.
A ptr to local var is stored in the radix tree currently,
this patch permits to alloc space to store host timeout
and thus also free it when data is removed.
defrag-config.c: In function 'DefragParseParameters':
defrag-config.c:105: warning: passing argument 2 of 'DefragPolicyAddHostInfo' from incompatible pointer type
make[3]: *** [defrag-config.o] Error 1
This would only happen in memory failure conditions.
util-decode-der.c:634:27: warning: Potential leak of memory pointed to by 'child'
return (Asn1Generic *)node;
stream-tcp-reassemble.c:2569:17: warning: Value stored to 'seg' is never read
seg = seg->next;
^ ~~~~~~~~~
stream-tcp-reassemble.c:2587:17: warning: Value stored to 'seg' is never read
seg = seg->next;
These were only used if debug is enabled.
app-layer-dns-tcp.c:407:13: warning: Value stored to 'length' is never read
length = *data;
app-layer-dns-udp.c:236:13: warning: Value stored to 'length' is never read
length = *data;
Detect when default_packet_size is zero, which enables zero-copy mode for
pfring and in that case, do what AF Packet does and set pkt_ext pointer to
the data and set PKT_ZERO_COPY flag.
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
Some of the fields in the SCACTileCtx struct are only used to create the MPM,
but are not needed to search the MPM. Create a new structure to contain just
the data needed by AC Search. After creating the MPM, copy the data into the
new structure and then free the memory only needed during initialization.
This reduces the size of the AC-Tile MPM context from 1360 bytes down to 296
bytes.
Add two new mPIPE load-balancing configuration options in suricata.yaml.
1) "sticky" which keep sending flows to one CPU, but if that queue is full,
don't drop the packet, move the flow to the least loaded queue.
2) Round-robin, which always picks the least full input queue for each
packet.
Allow configuring the number of packets in the input queue (iqueue) in
suricata.yaml.
For the mPipe.buckets configuration, which must be a power of 2, round
up to the next power of two, rather than report an error.
Added mpipe.min-buckets, which defaults to 256, so if the requested number
of buckets can't be allocated, Suricata will keep dividing by 2 until either
it succeeds in allocating buckets, or reaches the minimum number of buckets
and fails.
In SigMatchSignatures, the value p->flow doens't change, but GCC can't
figure that out, so it reloads p->flow many times during the function.
When p->flow is loaded into the variable pflow once at the start of the
function, the compile then doesn't need to reload it.
[src/app-layer-htp.c:1967] -> [src/app-layer-htp.c:1978]: (warning) \
Possible null pointer dereference: tx - otherwise it is redundant \
to check it against null.
pcap has a callback function that is called for each packet. Once a
second, it's meant to 'dump stats'. However, the timing logic was
broken, so it would actually dump stats for each packet.
By moving the stats second timer into the thread vars, next calls of
the callback will be able to use the stored time.
Flow timeout code worked by luck when checking if a flow still needed
reassembly for app layer inspection or logging. It would check for a
part of raw reassembly (smsg list) to determine if detection was
needed. In this case it would also process app layer cleanup,
including logging.
Introduced AppLayerTransactionGetActive which returns the lowest tx_id
in a direction that still needs some work.
FlowForceReassemblyNeedReassmbly now uses it to determine if the
applayer still needs work.
Converted FlowForceReassemblyForHash to use the checking function
FlowForceReassemblyNeedReassmbly as well, so that checking if a flow
needs work is now unified.
Raw reassembly is used only by the detection engine. For users only
caring about logging it's a significant overhead, both in cpu and
memory usage.
The option is called 'raw' and lives under the stream.reassembly
options.
stream:
memcap: 32mb
checksum-validation: yes # reject wrong csums
inline: auto # auto will use inline mode in IPS mode, yes or no set it statically
reassembly:
memcap: 64mb
depth: 1mb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
#randomize-chunk-range: 10
raw: false # <- new option
Spotted out by clang:
source-erf-dag.h|25 col 9| warning: '__SOURCE_ERR_DAG_H__'
is used as a header guard here, followed by #define of a different macro
[-Wheader-guard]
Prevents benign log message of parent nodes of final values being
redefined (which ends up having no affect as the final nodes
are protected from being removed).
If we have multiple layer of tunnel, the decoding of initial
Packet will recurse in DecodeTunnel function called in
PacketTunnelPktSetup. If we are not setting the pseudo
packet root before calling DecodeTunnel (as done in previous
code), then the tunnel root will no be correct for the lower
layer packets. This result in an counter problem and a suricata
failure after some time.
If a session only has data in one direction, like ftp data sessions,
protocol detection will only run in one direction. This led to a
situation where reassembly would hold all the segments as proto
detection was never flagged as complete.
This patch introduces a limit for protocol detection in this case.
If the limit is reached, detection will give up.
This patch adds a '-k' option to suricata to be able to specify
the checksum validation to use. If '-k all' is used, checksum
validation is forced. If '-k none' is used, no checksum validation
is made.
Message output in case of detection of a pcap file with a probable
cheksum issue has been updated to indicate that '-k' is a solution.
This patch adds support for checksum-checks in the pcap-file running
mode. This is the same functionnality as the one already existing for
live interface.
It can be setup in the YAML:
pcap-file:
checksum-checks: auto
A message is displayed for small pcap to warn that invalid checksum
rate is big on the pcap file and that checksum-check could
be set to no.
This patch set a new value in pkt->flag to signal that a packet is
invalid during decoding. The patch has been obtained via a coccinelle
transformation.
This patch adds and increments a invalid packet counter. It
does this by introducing PacketDecodeFinalize function
This function is incrementing the invalid counter and is also
signalling the packet to CUDA.
Remove all flags indicating the buffer type. They were only used
at parse time.
Because of this the DetectPcreData_ structure could shrink to 32
bytes.
This patch replaces PacketPseudoPktSetup by a better named
PacketTunnelPktSetup function which is also in charge of doing
the decoding of the tunneled packet.
This allow to clean the code. But it also fixes an issue.
Previously, if the DecodeTunnel function was failling (cause of
an invalid packet mainly), the result was that the original packet
to be considered as a tunnel packet (and not inspected by payload
detection).
In some cases, the decoding is not possible and some really invalid
packet can be created. This is in particular the case of tunnel. In
that case, it is more interesting to forget about the tunneled
packet and only consider the original packet.
DecodeTunnel function is maked as warn_unused_result because it is
meaningful for the decoder to know if the underlying data were not
correct. And in this case, only focus detection on the content.
Fix sigatures not supporting [10.0.0.0/24, !10.1.1.1] notation when
used directly in a rule instead of through a variable.
Add tests for Bugs #815 and #920.
When inspecting HTTP bodies there are several limits involved.
In this patch the reaching of the body limit will trigger body
inspection.
Without this, the body would only be inspected when inspection
limits "request-body-minimal-inspect-size" or
"response-body-minimal-inspect-size" were reached. If the body
limit was smaller than this value, the body would only be
inspected at the end of the tx or stream.
Use the new GetIfaceOffloading function to display a warning message
if pcap capture is used on Linux with GRO or LRO activated. This is
helpful for kernel after 2.6.31 were pcap will use mmaped capture.
TPACKET_V2 is used and this limit the size of the packet resulting
in truncated packets when merged packets are received.
When the PKT_NOPAYLOAD_INSPECTION flag is set, don't apply it to smsgs.
This way we can still inspect the outstanding smsgs.
The PKT_NOPAYLOAD_INSPECTION is set for encrypted traffic, and is combined
with disabling stream reassembly. So we only inspect the smsgs up to the
point of the disable detection point.
When checking the reassembly limit for raw reassembly, consider the
STREAMTCP_STREAM_FLAG_NOREASSEMBLY a trigger immediately. We won't
process any more segments in the reassembly engine anyway.
The meta-field-option allows for setting the hard limit of request
and response fields in HTTP. In requests this applies to the request
line and headers, not the body. In responses, this applies to the
response line and headers, not the body.
Libhtp uses a default limit of 18k. If this is reached an event is
raised.
Ticket 986.
In the SSE 4.2 SCMemcmpLowercase implementation, there would be a
_mm_load_si128 of a 2 byte array. However, _mm_load_si128 loads
16 bytes, causing it to read beyond the var. I don't think this lead
to crashes, as it was a static var, but clangs ASAN complained about
it.
Share memory space for IPV4Vars and (IPV6Vars, IPV6ExtHdrs), since a
packet can only be either IPv4 or IPv6, but not both.
Share memory for TCPVars, UDPVars, ICMPV4Vars and ICMPV6Vars, since a
packet can only be only of these.
Then move other structure members around to remove holes reported by pahole.
This reduces the size of the Packet structure from 2944 bytes (46 cachelines)
down to 1976 (31 cachelines), a 33% reduction.
Strip the 'proxy' parts from the normalized uri as inspected by http_uri,
urilen, pcre /U and others.
In a request line like:
GET http://suricata-ids.org/blah/ HTTP/1.1
the normalized URI will now be:
/blah/
This doesn't affect http_raw_uri. So matching the hostname, etc is still
possible through this keyword.
Additionally, a new per HTTP 'personality' option was added to change
this behavior: "uri-include-all":
uri-include-all: <true|false>
Include all parts of the URI. By default the
'scheme', username/password, hostname and port
are excluded. Setting this option to true adds
all of them to the normalized uri as inspected
by http_uri, urilen, pcre with /U and the other
keywords that inspect the normalized uri.
Note that this does not affect http_raw_uri.
So adding uri-include-all:true to all personalities in the yaml will
restore the old default behavior.
Ticket 1008.
Move the allocation of the mPipe ingress queue from a loop over
the number of workers in the main init function to being done inside
each worker thread. This allows allocating the memory locally on the
worker's CPU without needing to figure out ahead of time where that thread
will be running. This fixes one case of static mapping of workers to CPUs.
Use __thread to hold the queue rather than a global tables of queues.
pcre_study returning NULL is not necessarily an error, from the man page
pcre_study(3):
"If the function returns NULL, either it could not find any additional
information, or there was an error. You can tell the difference by
looking at the error value. It is NULL in first case."
Older libpcre versions would return NULL, causing errors.
Keep a separate checksum for IPV4, since a packet can have both an IPV4
checksum and a TCPV4 checksum, or IPV4 and UDPV4 checksum.
This will allow future sharing of more values.
Use PACKET_RESET_CHECKSUMS() in Unit Tests in place of setting the
individual checksum values.
When http and/or tls logging is disabled, the app layer would still
be flagged as logging. This caused transactions not to be freed until
the end of the flow as the logged tx id would never increment.
This fix postpones the setting of the app layer parser "logger"
flag to the point where we know the logger is enabled.
When logging is disabled, the app layer would still be flagged
as logging. This caused transactions not to be freed until the
end of the flow as the logged tx id would never increment.
This fix postpones the setting of the app layer parser "logger"
flag to the point where we know the logger is enabled.
In the case where DNS requests are sent over the same flow w/o a
reply being received, we now set an event in the flow and refuse
to add more transactions to the state. This protects the DNS
handling from getting overloaded slowing down everything.
A new option to configure this behaviour was added:
app-layer:
protocols:
dnsudp:
enabled: yes
detection-ports:
udp:
toserver: 53
request-flood: 750
The request-flood parameter can be 0 (disabling this feature) or a
positive integer. It defaults to 500.
This means that if 500 unreplied requests are seen in a row an event
is set. Rule 2240007 was added to dns-events.rules to match on this.
Packets that are rejected by the stream engine are not considered
part of an established tcp session. By allowing them to inspect
an smsg, some smsgs would not be properly inspected.
Copied SigTest26TCPV4Keyword and added check for invalid IPV4 checksums.
Created new SigTest26TCPV4AndIPV4Keyword test with a new packet with valid
IPV4 checksums.
When multiple segments were put into a smsg, the seq would be updated
each time a segment was added. Because of this, the seq wasn't pointing
to the start of the data.
This caused some false negatives when the fast_pattern was in the raw
stream, but another part of the inspection was in the state. Because of
the wrong seq, the inspection of the smsg could be delayed. This in turn,
could make the inspection engine consider a TX inspected, even if it wasn't
fully yet.
When installing the rules to tell mPIPE to send packet to Suricata,
give it a higher priority than the default used by Linux. This way if
Linux also tells mPIPE to send it packets, Suricata will get them
instead, as long as Suricata is running.
Libhtp decodes the + character in the query string to a space by default.
Suricata rules (e.g. etpro sid 2806767) are expecting to see the space in
the http_uri buffer.
Added an option per htp config to reenable this default behavior:
query-plusspace-decode: yes
Bug #1035.
Now that we call stream reassembly directly from proto detection, we will
need to check if reassembly has been disabled inside the stream reassembly
callback.
This prevents any calls to bypass and re-enter proto detection, despite
having reassembly disabled.
Make sure we register the detect.alerts counter before packet runtime starts
even in delayed detect mode. The registration of new counters at packet
runtime is not supported by the counters api and might lead to crashes as there
is no proper locking to allow for this operation.
This changes how delayed detect works a bit. Now we call the ThreadInit
callback twice. The first call will only register the counter. The 2nd call
will do all the other setup. This way the counter is registered before the
counters api starts operating in the packet runtime.
Fixes the segv reported in ticket #1018.
This patch is a result of applying the following coccinelle
transformation to suricata sources:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc|SCMallocAligned|SCRealloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
Victor Julien
Eric Leblond
Tom DeCanio
Anoop Saldanha
Ken Steele
Jason Ish
Duarte Silva
Giuseppe Longo