Commit Graph

4783 Commits (suricata-2.0.8)

Author SHA1 Message Date
Victor Julien 3a6be9772f http-json: add missing cleanup functions
Add cleanup functions.
12 years ago
Victor Julien 7ffd227133 file-json: cleanup at shutdown
Fix a memory leak at shutdown. Module didn't have a cleanup function.
12 years ago
Victor Julien 7ee3b456a3 dns-json: fix cleanup
Use specialized cleanup function for sub-module case. Freeing the
LogFileCtx is not the responsibility of a sub-module.
12 years ago
Victor Julien 1f2310bb34 alert-json: fix cleanup
Call specialized clean up function when running as a sub-module.
12 years ago
Victor Julien f62185c207 log-tls: run Disable at shutdown
Call OutputTlsLoggerDisable at cleanup.
12 years ago
Victor Julien f96a54535c drop loggers: call disable func
Call OutputDropLoggerDisable() on cleanup.
12 years ago
Victor Julien 9df045d086 output: add Disable funcs to mirror Enable
For the loggers that we allow only one instance for: tls, ssh, drop, we
track active loggers through Output*Enable functions. Add Disable
functions to mirror this. They are to be called from the shutdown funcs
those loggers use.
12 years ago
Victor Julien 4a104ae315 unified2: fix memory leak at shutdown
Module didn't properly free output context at shutdown. Led to a leak
in Unix Socket mode.
12 years ago
Eric Leblond 9961520316 output: clean file desc at exit.
This is a beginning of implementation for bug #1660:
 https://redmine.openinfosecfoundation.org/issues/1160

This patch adds a cleaning function for each logger of new type
(packet, tx and file). These functions are called in RunModeShutDown().

The state of this patch is that it is crashing suricata when sending
pcap to analyse:
 - At first pcap if tx and file cleaning function are called
 - At second pcap if only packet cleaning function is called

The cause in first case is unknown. In second case this is due to
the necessity of cleaning the list of logger registered to a logging
type.
12 years ago
Pierre Chifflier d476c654ee TLS: add detection for malicious heartbeats (AKA heartbleed)
The OpenSSL implementation of RFC 6520 (Heartbeat extension) does not
check the payload length correctly, resulting in a copy of at most 64k
of memory from the server (ref: CVE-2014-0160).
This patch adds support for decoding heartbeat messages (if not
encrypted), and checking several parts (type, length and padding).
When an anomaly is detected, a TLS event is raised.
12 years ago
Victor Julien ab503873ca erf-file: clean up decode thread local storage
Clean up the thread local data the decode portion of ERF users.

Bug #978
12 years ago
Victor Julien 6da8652a77 endace-dag: clean up decode thread local storage
Clean up the thread local data the decode portion of DAG uses.

Bug #978
12 years ago
Victor Julien 09ebbe08df mpipe: clean up decode thread local storage
Clean up the thread local data the decode portion of mpipe uses.

Bug #978
12 years ago
Victor Julien 8c16fede08 ipfw: clean up decode thread local storage
Clean up the thread local data the decode portion of ipfw uses.

Bug #978
12 years ago
Victor Julien 10c791c937 napatech: clean up decode thread local storage
Clean up the thread local data the decode portion of napatech uses.

Bug #978
12 years ago
Victor Julien c3e193e786 pcap: clean up decode thread local storage
Clean up the thread local data the decode portion of pcap uses.

Bug #978
12 years ago
Victor Julien 900fc6fdc7 pfring: clean up decode thread local storage
Clean up the thread local data the decode portion of pfring uses.

Bug #978
12 years ago
Victor Julien a8b1af3369 nfq: clean up decode thread local storage
Clean up the thread local data the decode portion of nfq uses.

Bug #978
12 years ago
Victor Julien 2864f9eef9 af-packet: clean up decode thread local storage
Clean up the thread local data of the decode part of afpacket.

Bug #978
12 years ago
Victor Julien bb2e9af40f pcap-file: clean up decode thread local storage
Clean up the thread local data the decode portion of pcap-file use.

Bug #978.
12 years ago
Victor Julien d26ceb2356 decode: introduce DecodeThreadVarsFree
As a mirror of DecodeThreadVarsAlloc, DecodeThreadVarsFree is used
to free the memory that DecodeThreadVarsAlloc alloc'd, including
AppLayer storage.
12 years ago
Victor Julien 552558894c app-layer: cleanups
Clean up AppLayerParserThreadCtxAlloc and AppLayerParserThreadCtxFree.
Both used confusing variables in loops, with the wrong types.
12 years ago
Jason Ish 7e268bd4d4 Force pidfile creation of --pidfile.
A pidfile can be useful when not in daemon mode, for example
when running under a process supervisor.
12 years ago
Tom DeCanio 4085f08602 json: somewhere along the way IP/port pairs had gotten swapped in
http and ssh eve logs
12 years ago
Victor Julien b9227ad20c tls: no event on 'new session ticket' in handshake
Don't set an event on encountering a 'new session ticket' (4) record
in the TLS handshake.
12 years ago
Victor Julien 1195f882b9 ipv6: add support for PAD1
Support PAD1 in IPv6 HOP options header and DST options header.
12 years ago
Victor Julien 7539372db7 icmpv6: add multicast types
Only add them to check if the code is 0 and to make sure the default
case doesn't set an 'unknown type' event.
12 years ago
Victor Julien eb3a9d3076 TLS: register patterns for tls-alerts
Register patterns for when server has an alert as the first message.
12 years ago
Victor Julien a96446d39e detect state: fix indent
AMATCH block was indented too far.
12 years ago
Victor Julien 6c8ca76942 detect state: remove alproto check for AMATCH
Not all AMATCHes set a alproto.
12 years ago
Victor Julien c9436a6aef Fix app-layer-protocol FP on multi TX flow
In case of multiple transactions, the stored AMATCH list would not have
been reset, but it would still be reconsidered. Even though none would
match, the engine would still conclude that the rule matched.
12 years ago
Eric Leblond 9e03550230 tls: fix problem with tls.store keyword
Pierre Chifflier pointed out that a rule like:
 alert tls any any -> any any (msg:"TLS store"; tls.issuerdn:!"C=FR"; tls.store;)
was alerting but not storing the certificate. If the filter was
removed:
 alert tls any any -> any any (msg:"TLS store"; tls.store;)
then tls.store is working as expected.

This was linked with fact that logging is only done once for a SSL
state. So without filter, once we have the info we can log and we
run the storage. But when there is a filter, we log and then there
is a filter analysis and alerting. And as logging as already be done
we don't enter in the logging function and there is no storage.

This patch forces the entrance in the log function when there is a
request for TLS storage. And it adds an exit in the logging function
to only do the storage part if the TLS state has already being logged.
12 years ago
Victor Julien 806844d852 af-packet: fix init sync with no traffic
Previously the sync code would depend on traffic to complete. This
patch adds poll support and can complete the setup if the poll timeout
is reached as well.

Part of bug #1130.
12 years ago
Eric Leblond 238ff23111 af-packet: move packet fanout code
The sooner is the better for that caode as these means we will get
all sockets binded to fanout group as fast as possible.
12 years ago
Eric Leblond 919377d4a5 af-packet: synchronize reading start
This patch is updating af-packet to discard packets that have been
sent to a socket before all socket in a fanout group have been setup.
Without this, there is no way to assure that all packets for a single
flow will be treated by the same thread.

Tests have been done on a system with an ixgbe network card. When using
'cluster_flow' load balancing and disactivating receive hash on the iface:
 ethtool -K IFACE rxhash off
then suricata is behaving as expected and all packets for a single flow
are treated by the same thread.

For some unknown reason, this is not the case when using cluster_cpu. It
seems that in that case the load balancing is not perfect on the card side.

The rxhash offloading has a direct impact on the cluster_flow load balancing
because load balancing is done by using a generic hash key attached to
each skb. This hash can be computed by the network card or can be
computed by the kernel. In the xase of a ixgbe network card, it seems there
is some issue with the hash key for TCP. This explains why it is necessary to
remove the rxhash offloading to have a correct behavior. This could also
explain why cluster_cpu is currently failing because the card is using the
same hash key computation to do the RSS queues load balancing.
12 years ago
Victor Julien 70efc66e33 fast-log: restore logging of Drop/wDrop
Restore drop print logic. Probably got lost in large refactoring round
introducing log APIs.

Bug #1138.
12 years ago
Victor Julien 1af39d55cf detect: allow alias registration for rule keywords
This allows for registering a keyword under another name while keeping
the old name active and supported.

Do this for 'luajit', which can now also be used as just 'lua'.
12 years ago
Victor Julien 010a334e1a json alert: fix action
It would give 'Pass' as an action when the rule is set to 'alert'.
12 years ago
Victor Julien e04b5f0dca eve-http: register with app-layer api
The HTTP module of Eve didn't register itself with the app-layer
for HTTP. This meant that if no other HTTP logger was active, the
HTTP logging in Eve wouldn't work.

This patch makes the HTTP Eve module register itself correctly.

Bug #1133.
12 years ago
Victor Julien 5d96ea570f eve-file: set event_type to fileinfo
To remain constistent with the other logs, set the event type to
the same name as the structure containing the defails. In this
case fileinfo.

Part of bug #1127.
12 years ago
Victor Julien 305da0248d eve-files: file -> fileinfo
Due to what appears to be an issue in logstash, the 'file' part of
the file event types was masked by a field that logstash-forwarder
added itself.

Since logstash-forwarder is an important part of the logstash stack,
this patch works around the issue by renaming our 'file' structure
to 'fileinfo', thus resolving the naming conflict.

Bug #1127
12 years ago
Victor Julien 3e7714aca2 eve-http: print <unknown> like in eve-files
When UA or Host are unknown, print <unknown> instead of <useragent
unknown> or <hostname unknown>.

Bug #1131.
12 years ago
Victor Julien c5041d35d5 Fix live reload
Fix memsets clearing out of bounds memory on live reload, causing
crashes and corrupted backtraces.

Bug #1128.
12 years ago
Victor Julien fdb1bd9668 pfring: call enable_ring after set_cluster
Move pfring_enable_ring to the start of ReceivePfringLoop() so that
it's guaranteed to be called after all threads have called
pfring_set_cluster first.

This is necessary because pfring will already make packets available
to thread N, while thread N+1 is still registering itself. This leads
to cases where the first packet(s) of a flow are processed by a
different thread in Suricata than the later ones.

This is a race condition only at start up. New flows after the pfring
initialization is complete will not be influenced by this.

Bug #1129.
12 years ago
Victor Julien a3020b5306 eve-log: output cleanup
Suppress debug messages and print in a nicer way which modules are
being enabled.
12 years ago
Victor Julien ed877c64d1 Bug 611: fix for iponly
Fix Bug 611 for ip-only rules as well. If 'alert ip' rule has ports,
don't match on protocols that don't have ports. Like ICMP.

Bug #611.
12 years ago
Eric Leblond 6c3c234ca5 output-json: update timestamp format
This patch updates the timestamp format used in eve loggin.
It uses a ISO 8601 comptatible string. This allow tools parsing
the output to easily detect adn/or use the timestamp.

In the EVE JSON output, the value of the timestamp key has been
changed to 'timestamp' (instead of 'time'). This allows tools
like Splunk to detect the timestamp and use it without configuration.

Logstash configuration is simple:

input {
   file {
      path => [ "/usr/local/var/log/suricata/eve.json" ]
      codec =>   json
      type => "suricata-log"
   }
}

filter {
   if [type] == "suricata-log" {
      date {
        match => [ "timestamp", "ISO8601" ]
      }
   }
}

In splunk, auto detection of the fle format is failling and it seems
you need to define a type to parse JSON in
$SPLUNK_DIR/etc/system/local/props.conf:

[suricata]
KV_MODE = json
NO_BINARY_CHECK = 1
TRUNCATE = 0

Then you can simply declare the log file in
$SPLUNK_DIR/etc/system/local/inputs.conf:

[monitor:///usr/local/var/log/suricata/eve.json]
sourcetype = suricata

In both cases the timestamp are correctly imported by
the tools.
12 years ago
Eric Leblond 1fa4233d67 pfring: get vlan id from header
PF_RING is delivering the packet with VLAN header stripped. This
patch updates the code to get the information from PF_RING extended
header information.

This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.
12 years ago
Eric Leblond cf30adcedc ipfw: fix build
Buildbot reported:
 runmode-ipfw.c: In function 'RunModeIpsIPFWAuto':
 runmode-ipfw.c:85: error: implicit declaration of function 'LiveDeviceHasNoStats'
12 years ago
Eric Leblond 606e19124b http: add info message about memcap
Display a message about http memcap when it is set in config file.
12 years ago
Victor Julien f00703acc1 Fix False Positive of rules with ports on portless protocols
In case of 'alert ip' rules that have ports, the port checks would
be bypassed for non-port protocols, such as ICMP. This would lead to
a rule matching: a false positive.

This patch adds a check. If the rule has a port setting other than
'any' and the protocol is not TCP, UDP or SCTP, then we rule won't
match.

Rules with 'alert ip' and ports are rare, so the impact should be
minimal.

Bug #611.
12 years ago
Victor Julien 00d2f2d627 Fix BytesToString indexing array using wrong index
This would lead to reading past the end of the buffer and also writing
past the end of the newly allocated buffer.

Bug #1121
12 years ago
Victor Julien 88e9c85e36 json output: fix vlan byte order in output
VLAN functions/macros return vlan id in host byte order, so no need
to convert them in output functions.
12 years ago
Victor Julien 872bb5664e Fix null dereference in eve-log
Eve-log would call GET_VLAN_ID on the packets vlan header if p->vlan_idx
was bigger than 0. GET_VLAN_ID would then unconditionally dereference
p->vlanh[0] or [1]. However, there are a number of cases in which these
pointers are not set. Defrag pseudo packets, AF_PACKET and in the future
PF_RING, do set the id's, but not the header pointers.

This patch adds 2 new macro's which are wrappers around a function:

VLAN_GET_ID1 and VLAN_GET_ID2 get the id's by calling DecodeVLANGetId.

This function will return the correct id.

Bug #1120.
12 years ago
Victor Julien 684d787567 Coverity fix
Coverity 992695, fix potential array index with negative int. Very unlikely
case at rule keyword parsing stage.
12 years ago
Victor Julien f6bb867df8 ssh: fix scan-build warnings
app-layer-ssh.c:165:5: warning: Value stored to 'input_len' is never read
    input_len -= 1;
    ^            ~
1 warning generated.

app-layer-ssh.c:160:5: warning: Value stored to 'input_len' is never read
    input_len -= 4;
    ^            ~
1 warning generated.
12 years ago
Victor Julien 0967f0777c ssh: improve banner checking
Don't use input_len as banner length. Instead, look for banner end
to calculate banner length.

Add test for banner buffering corner case.
12 years ago
Victor Julien 669b351dad ssh: fixes for minor scan-build warnings
/usr/share/clang/scan-build/ccc-analyzer -DHAVE_CONFIG_H -I. -I..   -I./../libhtp/  -I/usr/include/nspr   -I/usr/include/nss -I/usr/include/nspr   -DLOCAL_STATE_DIR=\"/usr/local/var\" -g -O2 -Wall -Wno-unused-parameter -std=gnu99 -DHAVE_LIBNET11 -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -DHAVE_LIBNET_ICMPV6_UNREACH  -I/usr/include -DLIBPCAP_VERSION_MAJOR=1 -DHAVE_PCAP_SET_BUFF -DHAVE_LIBCAP_NG -DREVISION="51e0dee" -MT app-layer-ssh.o -MD -MP -MF .deps/app-layer-ssh.Tpo -c -o app-layer-ssh.o app-layer-ssh.c
app-layer-ssh.c:164:5: warning: Value stored to 'input' is never read
    input += 1;
    ^        ~
app-layer-ssh.c:165:5: warning: Value stored to 'input_len' is never read
    input_len -= 1;
    ^            ~
app-layer-ssh.c:212:13: warning: Value stored to 'ret' is never read
            ret = 0;
            ^     ~
3 warnings generated.
12 years ago
Victor Julien b877cf6158 ssh: add json logger
Sub module of eve-log, but can also run separately as ssh-json-log. Only
one at a time though.
12 years ago
Victor Julien 65b228ccfd ssh: improve large and fragmented banner handling
Including tests.
12 years ago
Victor Julien b4aeb43af1 ssh: disable inspection in encrypted phase
When both sides of the session have completed the encryption setup,
flag the stream to disable detection.
12 years ago
Victor Julien 294ff49f6d ssh: allow for space characters in the software version
Previously the software version would only contain up to the first
space.

E.g. in SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu3

It would contain "OpenSSH_4.7p1".

This patch changes the behavior to:

"OpenSSH_4.7p1 Debian-8ubuntu3"
12 years ago
Victor Julien b792234dd0 ssh: clean up flags 12 years ago
Victor Julien b5afe2b51f ssh: reenable ssh.protoversion keyword 12 years ago
Victor Julien 6c0162bf26 ssh: reenable ssh.softwareversion keyword 12 years ago
Victor Julien 884cecd9af ssh: handle fragmented banner
Cleanups.
12 years ago
Victor Julien 32fcdfe6eb ssh: server support, cleanups 12 years ago
Victor Julien 3648adb533 ssh: record parser 12 years ago
Victor Julien a3c9832b90 ssh: reenable parser as stub
Reenable the SSH parser. It now compiles, however the actual parsing
code is still disabled (commented out).
12 years ago
Eric Leblond 79de8c8f4b runmode: remove unused variable.
default_mode_auto is not used anymore and can be removed.
12 years ago
Eric Leblond a6bb86a9e0 Exit if BPF filter file is specified in IPS mode
This patch adds a check that was missing when specifying BPF filter
from a file. Suricata behavior should have been the same as when
BPF filter is specified on command line.
12 years ago
Eric Leblond 941cfe1641 ipfw: fix operator error in test
Fix warning spotted by clang on FreeBSD:

source-ipfw.c:241:49: warning: use of logical '||' with constant operand [-Wconstant-logical-operand]
        if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
                                                ^  ~~~~~~~~~~~~~
source-ipfw.c:241:49: note: use '|' for a bitwise operation
        if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
                                                ^~
                                                |

Use same logic as the one used in other capture mode.
12 years ago
Eric Leblond 9f6527dc16 ipfw: improve exit message
This patch synchronizes the exit message with what is done in
NFQ capture mode.
12 years ago
Eric Leblond 1c48a81f91 ipfw: update running modes to hide device stats
This patch adds call to the function used to disable the display
of live device stats at exit.
12 years ago
Eric Leblond d8a305356e nfq: update message displayed at exit
This patch updates the message displayed at exit to have something
more readable.
12 years ago
Eric Leblond 00c8408c55 nfq: update running modes to hide device stats
This patch adds call to the function used to disable the display
of live device stats at exit.
12 years ago
Eric Leblond 6b2ca63d9d util-device: add function to avoid stat display
In the case of running mode like NFQ there is no need possibility
to compute the statistics as it is done in LiveDevice (drop and
checksum count are meaningless).

This patch adds a function that allow running mode to disable the
display of the counters at exit.
12 years ago
Eric Leblond a12c46c700 util-device: fix debug message
Reference to Pcap was not correct.
12 years ago
Ken Steele 970f22c752 Move memcpy_lower() into new util-memcpy.h
Remove local copies from each MPM file and use include file instead.
Might be better to also add util-memcpy.c rather than inlining it each time,
to get smaller code, since only seems to be used at initialization.
12 years ago
Ken Steele cd1c18d981 Store case-insensitive patterns as lowercase.
This is required because SCMemcmpLowercase() expects it first argument
to be already lowercase for the comparison. This is done by using
memcpy_tolower() for NO_CASE patterns.

This addresses code review comments from Victor.
12 years ago
Ken Steele 6b1517c0b8 Remove case_state usage
The case_state in MPMs was just to track when a pid could have no-case and
case-sensitive matches for the same PID. Now that can't happen after fixing
bug 1110, so remove the code and storage for case_state.
12 years ago
Ken Steele c41041a9c7 When assigning Pattern IDs pids, check Case flags
This fixes bug 1110. When assigning PIDs, use the NO_CASE flag when comparing
for duplicates. The state of the flag must be the same, but also use the same
type of comparisons when checking for duplicates.

Previously, "foo":CS would match with "foo":CI when it should not.
and "foo":CI would not match "FoO":CI when it should. Both of those
cases are fixed with this change.

This then allows simplifying the use of pid in MPMs because now if they
pids match, then so do the flags, so checking the flags is not required.
12 years ago
Ken Steele b7baa561c0 Cleanup in ac-tile MPM
Remove return from void functions.
Add some commments
Remove inline on functions where it doesn't make sense.
Rewrote if statement to be more clear.
12 years ago
Victor Julien df927f7ea8 unittest: fix mutex unlock w/o a lock
Fixes an error in a test. SCMutexUnlock was called w/o a prior
SCMutexLock.
12 years ago
Victor Julien 82ae41d320 pool: on Init() error, properly clean up
In the stream engine, Init() can fail if the memcap is reached. In this
case the segment was not freed by PoolGet:

==8600== Thread 1:
==8600== 70,480 bytes in 1,762 blocks are definitely lost in loss record 611 of 612
==8600==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8600==    by 0x914CC8: TcpSegmentPoolAlloc (stream-tcp-reassemble.c:166)
==8600==    by 0xA0D315: PoolGet (util-pool.c:297)
==8600==    by 0x9302CD: StreamTcpGetSegment (stream-tcp-reassemble.c:3768)
==8600==    by 0x921FE8: StreamTcpReassembleHandleSegmentHandleData (stream-tcp-reassemble.c:1873)
==8600==    by 0x92EEDA: StreamTcpReassembleHandleSegment (stream-tcp-reassemble.c:3584)
==8600==    by 0x8D3BB1: HandleEstablishedPacketToServer (stream-tcp.c:1969)
==8600==    by 0x8D7F98: StreamTcpPacketStateEstablished (stream-tcp.c:2323)
==8600==    by 0x8F13B8: StreamTcpPacket (stream-tcp.c:4243)
==8600==    by 0x8F2537: StreamTcp (stream-tcp.c:4485)
==8600==    by 0x95DFBB: TmThreadsSlotVarRun (tm-threads.c:559)
==8600==    by 0x8BE60D: TmThreadsSlotProcessPkt (tm-threads.h:142)

tcp.segment_memcap_drop   | PcapFile                  | 1762

This patch fixes PoolGet to both Cleanup and Free the Alloc'd data in
case Init fails.
12 years ago
Victor Julien fd6fd9ce48 Fix memory leak in proto - name mapping
==15745== 3 bytes in 1 blocks are definitely lost in loss record 5 of 615
==15745==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745==    by 0x71858C1: strdup (strdup.c:42)
==15745==    by 0xA20814: SCProtoNameInit (util-proto-name.c:75)
==15745==    by 0x952D1B: PostConfLoadedSetup (suricata.c:1983)
==15745==    by 0x9537CD: main (suricata.c:2112)

Also, clean up and add a check to make sure it's initialized only once.
12 years ago
Victor Julien bb4def7949 lua: fix minor memory leak
The full path of the script names is stored in a buffer that wasn't
freed at exit.

==24195== 41 bytes in 1 blocks are definitely lost in loss record 300 of 613
==24195==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24195==    by 0x565D06: DetectLoadCompleteSigPath (detect.c:251)
==24195==    by 0x7CABE8: DetectLuajitParse (detect-luajit.c:595)
==24195==    by 0x7CD2AE: DetectLuajitSetup (detect-luajit.c:827)
==24195==    by 0x7DC273: SigParseOptions (detect-parse.c:547)
==24195==    by 0x7DDC75: SigParse (detect-parse.c:856)
==24195==    by 0x7E1C2B: SigInitHelper (detect-parse.c:1336)
==24195==    by 0x7E2968: SigInit (detect-parse.c:1559)
==24195==    by 0x7E37B1: DetectEngineAppendSig (detect-parse.c:1831)
==24195==    by 0x566D17: DetectLoadSigFile (detect.c:335)
==24195==    by 0x567636: SigLoadSignatures (detect.c:423)
==24195==    by 0x951A97: LoadSignatures (suricata.c:1816)

This patch frees the buffer.
12 years ago
Victor Julien 9eed83c62b profiling: fix memory leak
For packets that were freed, not recycled, profiling memory wasn't
freed:

==15745== 13,312 bytes in 8 blocks are definitely lost in loss record 611 of 615
==15745==    at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745==    by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745==    by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745==    by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745==    by 0x8413BF: FlowForceReassemblyForHash (flow-timeout.c:588)
==15745==    by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745==    by 0x9540F6: main (suricata.c:2296)
==15745==
==15745== 14,976 bytes in 9 blocks are definitely lost in loss record 612 of 615
==15745==    at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745==    by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745==    by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745==    by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745==    by 0x841508: FlowForceReassemblyForHash (flow-timeout.c:620)
==15745==    by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745==    by 0x9540F6: main (suricata.c:2296)

This patch addresses that.
12 years ago
Victor Julien 3f49eb843d lock profiling: fix memory leak
If lock profiling was compiled in, but disabled in the config a
serious memory leak condition was triggered.

Valgrind output:

==11169== 9,091,248 bytes in 189,401 blocks are definitely lost in loss record 564 of 564
==11169==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11169==    by 0xABC44C: LockRecordAdd (util-profiling-locks.c:112)
==11169==    by 0xABC950: SCProfilingAddPacketLocks (util-profiling-locks.c:141)
==11169==    by 0xA04CD5: TmThreadsSlotVarRun (tm-threads.c:562)
==11169==    by 0x958793: TmThreadsSlotProcessPkt (tm-threads.h:142)
==11169==    by 0x9599C3: PcapFileCallbackLoop (source-pcap-file.c:172)
==11169==    by 0x56FC130: ??? (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.4.0)
==11169==    by 0x959D24: ReceivePcapFileLoop (source-pcap-file.c:210)
==11169==    by 0xA05B9E: TmThreadsSlotPktAcqLoop (tm-threads.c:703)
==11169==    by 0x6155F6D: start_thread (pthread_create.c:311)
==11169==    by 0x6E399CC: clone (clone.S:113)
12 years ago
Eric Leblond 3ce39433dd capture: use 64 bits counters
Some of the packets counters were using a 32bit integer. Given the
bandwidth that is often seen, this is not a good idea. This patch
switches to 64bit counter.
12 years ago
Eric Leblond 6630456a67 af-packet: fix livedev packets counter
Packets counter is incremented in AFPDumpCounters and it was
also incremented during packet reading. The result was a value
that is twice the expected result.

Spotted-by: Victor Julien <victor@inliniac.net>
12 years ago
Victor Julien f9213d7cc6 stream-tcp: proper error if segment pool init fails
Until now a PoolInit failure for the segment pools would result in
an abort() through BUG_ON(). This patch adds a proper error message,
then exits.

Bug #1108.
12 years ago
Victor Julien 7e38347d99 stream-tcp: fix error handling in segment pool
When TcpSegmentPoolInit fails (e.g. because of a too low memcap),
it would free the segment. However, the segment memory is managed
by the Pool API, which would also free the same memory location.
This patch fixes that.

Also, memset the structure before any checks are done, as the segment
memory is passed to TcpSegmentPoolCleanup in case of error as well.

Bug #1108
12 years ago
Victor Julien 9442dce451 commandline parsing: check optarg ptr before using it
Fixes:
** CID 1075221:  Dereference after null check  (FORWARD_NULL)
/src/suricata.c: 1344 in ParseCommandLine()

The reason it gave this warning is that in other paths using optarg
there was a check, so the checker assumed optarg can be NULL.
12 years ago
Victor Julien 9f2ce16ef3 icmpv6: Fix Coverity warnings on ND_* types
This patch fixes:
** CID 1187544:  Missing break in switch  (MISSING_BREAK)
/src/decode-icmpv6.c: 268 in DecodeICMPV6()

** CID 1187545:  Missing break in switch  (MISSING_BREAK)
/src/decode-icmpv6.c: 270 in DecodeICMPV6()

** CID 1187546:  Missing break in switch  (MISSING_BREAK)
/src/decode-icmpv6.c: 272 in DecodeICMPV6()

** CID 1187547:  Missing break in switch  (MISSING_BREAK)
/src/decode-icmpv6.c: 274 in DecodeICMPV6()

It duplicates the logic instead of adding 'fall through' statements
as the debug statements were wrong and confusing. For ND_REDIRECT
all 5 ND_* types would have been printed.
12 years ago
Ken Steele e381af98dd Only update mPipe stats occasionally around the packet loop.
Check for termination or stats update only once every 10,000 times
around the mPipe packet processing loop, to reduce locking.
12 years ago
Victor Julien 3c9a135c0f isdataat: fix coverity issue
During keyword setup there was a path that in theory could lead to
indexing an array with a negative int.

Coverity 400608
12 years ago
Victor Julien aa8918abd1 fast-pattern: fix error check in keyword setup
Check the right variable. Coverity 1038096
12 years ago
Victor Julien 3fa958b9a7 byte_extract/test/jump: fix coverity issues
During keyword setup there was a path that in theory could lead to
indexing an array with a negative int.

Coverity 992695, 400605, 400604
12 years ago
Victor Julien e82c6efaca Minor cleanups
Remove useless checks: coverity 1038130, 1038131, 1038132
Small other fixes: coverity 1164817
12 years ago
Victor Julien 2b734b8d8a htp: fix pointer check logic
Don't check pointer after it has already been used.

Coverity 1047545
12 years ago
Victor Julien abdc0072aa rule reload: fix unitialized memory access on error
Coverity 709220
12 years ago
Victor Julien d8c486231c output-json: fix minor memory leak on error
If the json file couldn't be opened, a minor memory leak would occur.

Coverity 1166039
12 years ago
Victor Julien b4ab9a0a3c ftp: fix memory leak
db pointers in both directions were not freed. This patch frees them
in the state free function.

Bug #1090
12 years ago
Victor Julien d63df6937c tls: fix uninitialized var use
errcode wasn't initialized and in some code paths it remained
uninitialized, leading to us evaluating this uninitialized value.

Bug #1091
12 years ago
Victor Julien 95fa5ae1d2 smtp: don't read uninitialized value
If a reply would be seen before a command, a read of a uninitialized
value could happen.

This patch adds a check for this.

Bug #1089.
12 years ago
Victor Julien 0416a8428d htp: don't assume HTPCallbackRequestLine is the first callback
By assuming that HTPCallbackRequestLine would always be run first,
an memory leak was introduced. It would not check if user data already
existed in the tx, causing it to overwrite the user data pointer is
it already existed.

Bug #1092.
12 years ago
Victor Julien 4e561d6b20 pcap/afpacket: update counters at exit
In really short Suricata runtimes the capture counters would not
be updated. This patch does a force update at the end of the
capture loops in pcap and af-packet.
12 years ago
Victor Julien a97662e08c dns: if no (valid) config is found, use defaults
This patch will set up probing parsers if no (valid) config is
found.

Add a warning in those cases.
12 years ago
Victor Julien ab10c0a099 app-layer: tell pp registrar if config was found
The probing parser registration function
AppLayerProtoDetectPPParseConfPorts was a void, meaning it would
give no feedback to the registering protocol implementation. If a
config was missing, it would just give up.

This patch changes it to return a bool. 0 if no config was found,
1 if a config was found.

This allows the caller to setup a default case.
12 years ago
Victor Julien ec4288f7c2 dns yaml: fix detect ports of tcp relying on udp
The probing parser detection ports yaml settings of the TCP part
of the DNS parser accidentally used udp as protocol string, causing
the wrong part of the YAML to be evaluated.
12 years ago
Victor Julien ad51a57b91 app-layer-event: make error reporting more clear
If the protocol is disabled, app-layer-event would print a cryptic
error message. This patch makes sure we inform the user the protocol
is in fact disabled.
12 years ago
Victor Julien f791d0f5c5 vlan/QinQ: add vlan_qinq counter
This patch introduces a new counter "decoder.vlan_qinq". It counts
packets that have more than two stacked vlan layers.

Packets with 2 vlan layers will both increment "decoder.vlan" and
"decoder.vlan_qinq".
12 years ago
Victor Julien c95df281e3 eve-log: add warning if enabled but not supported
If we're not compiled against libjansson, the eve-log output is not
available. This patch adds a warning.
12 years ago
Jason Ish 33e4cc065f Promote nodes set with --set to sequence nodes as needed.
A node isn't known to be a sequence node until the YAML is parsed.
If a node sequence node was set on the command line, promote
it to a sequence node when it is discovered by YAML to be
a sequence node.

Fixes comment #18 in issue 921.
12 years ago
Victor Julien 18edec8769 vlan: support QinQ ethernet types
Add support for 802.1AD and 802.1QinQ in ethernet and vlan parsing.

Tickets: #814, #1103, #1104
12 years ago
Victor Julien 91f8c33b0a defrag: fix reassembly with vlan
When creating a pseudo packet with the reassembled IP packet, the
parent's vlan id or id's are also needed. The defrag packet is run
through decode and the flow engine, where the vlan id is necessary
for connecting the packet to the correct flow.
12 years ago
Eric Leblond b603ad62e5 af-packet: declare TP_STATUS_VLAN_VALID if needed
Some old distribution don't ship recent enough linux header. This
result in TP_STATUS_VLAN_VALID being undefined. This patch defines
the constant and use it as it is used in backward compatible method
in the code: the flag is not set by kernel and a test on vci value
will be made.

This should fix https://redmine.openinfosecfoundation.org/issues/1106
12 years ago
Victor Julien f7b1aefaf4 Bug 1107: decoders: bail out on pseudo packets
Flow-timeout code injects pseudo packets into the decoders, leading
to various issues. For a full explanation, see:
  https://redmine.openinfosecfoundation.org/issues/1107

This patch works around the issues with a hack. It adds a check to
each of the decoder entry points to bail out as soon as a pseudo
packet from the flow timeout is encountered.

Ticket #1107.
12 years ago
Victor Julien 85760a7044 Flow: fix flow reference cnt issues
FlowReference stores the flow in the destination pointer and increases
the flow reference counter (use_cnt). This should only be called once
per destination pointer. The reference counter is decremented when
FlowDereference is called. Multiple FlowReference calls would lead to
multiple use_cnt bumps, while there would be only one FlowRereference.
This lead to a use_cnt that would never become 0, meaning the flow
would stay in the hash for the entire lifetime of the process.

The fix here is to check if the destination pointer is already set to
the flow. If so, we don't increase the reference counter.

As this is really a bug, this condition will lead to a BUG_ON if the
DEBUG_VALIDATION checking is enabled.
12 years ago
Victor Julien fdca557e01 ipv4 decoder: set 'invalid' event on icmpv6
ICMPv6 on IPv4 is invalid, so if we encounter this we set an event
and flag the packet as invalid.

Ticket #1105.
12 years ago
Victor Julien 43aa74d711 debug-validation: fix packet check
On fragments and invalid packets we can have p->proto set, while the
matching protocol header pointer is null.
12 years ago
Victor Julien 01b80e2d8f detect ip-only: update radix usage
Update IP-only lookups to the changed radix API.

The return of user_data is treated as a succesful lookup, instead of
the node.
12 years ago
Victor Julien b75eb77295 host-os-info: update radix lookups
Update host OS lookups to use the new API.

The return of user_data is treated as a succesful lookup, instead of
the node.
12 years ago
Victor Julien fd193107de unused reputation: radix update
Update the unused reputation code to compile after radix update.
12 years ago
Victor Julien cd91738a4b defrag: update radix usage
Update defrag timeout lookup to use the updated radix API.

The return of user_data is treated as a succesful lookup, instead of
the node.
12 years ago
Victor Julien d0a26c6a17 radix: update HTP config lookup logic
The HTP config tree is a radix. The lookups are updated to the new API.
The return of user_data is treated as a succesful lookup, instead of
the node itself.
12 years ago
Victor Julien 3b914eb7ba radix: update tests after API change
This patch updates all the radix tests to the new API. In most cases
it just passes a NULL user data return pointer.

It also removes the tests related to SC_RADIX_NODE_USERDATA, as this
macro is removed.

Bug #1073
12 years ago
Victor Julien 7b4be598c4 radix: don't modify node prefix on lookup
The radix tree stores user data. However, it had no function to return
this data to the consumers of the API. Instead, on lookup, it would
set a field "user_data_result" in the nodes prefix structure which
could then be read by the caller.

Apart for this not being a very nice design as it exposes API internals
to the caller, it is not thread safe. By updating the global data
structure without any form (or suggestion) of locking, threads could
overwrite the same field unexpectedly.

This patch modifies the lookup logic to get rid of this stored
user_data_result. Instead, all the lookup functions how take an
addition argument: void **user_data_result.

Through this pointer the user data is returned. It's allowed to be
NULL, in this case the user data is ignored.

This is a significant API change, that affects a lot of tests and
callers. These will be updated in follow up patches.

Bug #1073.
12 years ago
Jason Ish 7a9da787f9 Don't alert on valid ICMP6 solicit/advert messages.
Handles ND_ROUTER_SOLICIT, ND_ROUTER_ADVERT, ND_NEIGHBOUR_ADMIN,
ND_NEIGHBOUR_SOLICIT and ND_REDIRECT.  Don't set ICMPV6_UNKONWN_CODE
if code is the expected value of 0.
12 years ago
Victor Julien 7726cecbaa Fix CUDA test warning. 12 years ago
Anoop Saldanha 29fb9b099a CUDA: Fix header file resolution issues. 12 years ago
Eric Leblond 2cd6e1287f af-packet: no VLAN id from msg header for old kernel
This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.
12 years ago
Eric Leblond 1ccb93ab50 Add new function to work on kernel version.
This patch adds a new file containing a function that can be used
to compare the version number of the running kernel with a specific
version.
12 years ago
Eric Leblond e871f7132b af-packet: improve VLAN detection
Since commit in kernel
  commit a3bcc23e890a6d49d6763d9eb073d711de2e0469
  Author: Ben Greear <greearb@candelatech.com>
  Date:   Wed Jun 1 06:49:10 2011 +0000

      af-packet: Add flag to distinguish VID 0 from no-vlan.
a flag is set to indicate VLAN has been set in packet header.

As suggested in commit message, using a test of the flag followed
by a check on vci value ensure backward compatibility of the test.
12 years ago
Eric Leblond 1fb7c0dddc af-packet: handle vlan counter
This patch adds a vlan counter increment in decodeAFP. This
is needed to take into account vlan_id set by the packet
reading function.
12 years ago
Eric Leblond 71e47868bf af-packet: get vlan id from header
Since kernel 3.8, the VLAN header are stripped from packet. So the
first VLAN id can only be fetched from the tpacket header at capture
time.
12 years ago
Victor Julien 3967bd5517 app-layer: fix AppLayerParserProtocolIsTxEventAware
AppLayerParserProtocolIsTxEventAware would check if a proto is tx
event aware by checking if it had registered a StateHasEvents function.
However, this is an optimization function. This patch changes it to
use the StateGetEvents function instead, which is a better indicator.
12 years ago
Victor Julien 3f5acc5447 http: per tx decoder events
Store HTTP decoder events per TX, so they can be inspected per TX.

Ticket: #839.
12 years ago
Victor Julien 1030cf58fa dns: fix protocol yaml setting
UDP code would accidentally depend on 'tcp' setting.
12 years ago
Ken Steele 497575d38e Add option on Tile-Gx for logging for fast.log alerts over PCIe
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.

This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.

Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.

Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.

Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.

The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.

By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".

Check for TILERA_ROOT environment variable before building tile_pcie_logd

Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
12 years ago
Ken Steele 2200dd61a1 Reduce time the file lock is held to write an alert to Fast.log
Generate the alert string into a temporary buffer before aquiring the
file lock. Only hold the file lock while writing the alert string to the
file.

In the case of multiple alerts, it would be better to generate all the
alerts, then aquire the lock once and write them all and then flush.

Changed PrintRawLineHexFp, which printed to a file, to PrintBufferRawLineHex,
that puts the same output into a string buffer. It was only used by fast.log.
12 years ago
Ken Steele 235cd0211a Alert file formatting clean up.
Put { on new line for function declarations. Remove space after function
name.

Add static to unit tests delcaration.
12 years ago
Victor Julien 1ac805f1b3 Fix address parsing issues
Fix issue where negating a range containing a negation would fail.

E.g. HOME_NET: [192.168.0.0/16,!192.168.10.0], can be used in a rule
     as !$HOME_NET.

Also, fix another parsing issue:

If the negation range would be bigger than the 'positive' range, parsing
wouldn't be correct. Now this case is rejected.

E.g. [192.168.1.3,!192.168.0.0/16] is now explicitly rejected

Ticket 1079.
12 years ago
Victor Julien 55f8672a04 profiling: end profiling tunnel packets inside lock
End profiling inside the lock for a tunnel packet as otherwise another
thread may already free the packet while the profiling code runs.

SEGV's observed and now gone.
12 years ago
Victor Julien 4e5572b8dc fileext: make case insensitive
Change keyword to be case insensitive when matching.

Ticket #597.
12 years ago
Jason Ish e8626509cc When setting a sequence value, check for an existing node.
This makes --set af-packet.0.threads=X do what you expect by
looking for the sequence node before setting.
12 years ago
Jason Ish ab562ce226 Add a --set command line option to set/override a configuration value. 12 years ago
Victor Julien 4ce1fd347e json dns: fix tx logic
The JSON DNS logger would still have some conditions in the main
Logger function. This led to some transactions not beeing logged.
12 years ago
Victor Julien 6dd10443ce profiling: output log api modules separately
Skip log api thread modules in the regular 'thread modules' list,
instead print them in a separate list.
12 years ago
Victor Julien a37a1d9de7 profiling: fix percentage of detect phases
Use proper 'total' to calculate weigth of each detection phase.
12 years ago
Victor Julien f902c9e6c7 threading: add missing json types to TmModuleTmmIdToString
Also, remove 'default' case from switch statement, so missing entries
will lead to a compiler warning.
12 years ago
Victor Julien a3b0577a1f output: add TM_FLAG_LOGAPI_TM thread module flag
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.

Set flag in all all Log API users' registration code.

Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
12 years ago
Victor Julien b166e2f0e7 profiling: support log api
The log API calls thread modules directly, so the TMM profiling logic
can be applied to it. This patch does so.

The "Thread Module" out now again lists the individual loggers. As the
module are normally called much less frequently the numbers are hard to
compare to pre-log-api numbers.
12 years ago
Victor Julien c36a8d30e5 threading: add utility to get module id from name
Add TmModuleGetIdByName to go from module name to id directly.
12 years ago
Victor Julien 8a735a9b90 profiling: add sample-rate yaml option
Add option "profiling.sample-rate":

  # Run profiling for every xth packet. The default is 1, which means we
  # profile every packet. If set to 1000, one packet is profiled for every
  # 1000 received.
  #sample-rate: 1000

This allows for configuration of the sample rate.
12 years ago
Victor Julien 2c3a92a1c9 profiling: conditional rule profiling
Add support for conditional rule profiling. Currently only simple
rate limiting is supported, but hardcoded to inspecting rules for
each packet.
12 years ago
Victor Julien 13d491f577 profiling: lower overhead when disabled
Instead of a large (6k+) structure in the Packet, make the profiling
storage dynamic. To do this the Packet->profile is now a pointer.

Initial support for selective sampling, e.g. only profile every
1000th packet.
12 years ago
Eric Leblond c2fcf329f0 tls: fix negated match
A negated match is matching if the tested field is NULL. But as it
is not set, nor negated nor normal test must match.

Without this patch, a rule like:
 alert tls any any -> any any (msg:"negated match"; tls.subject:!"CN=home.regit.org"; sid:1; rev:1;)
is alerting for all connections. Event if they are done on a certificate
with matching subject. This was due to the fact that tls protocol
is discovered before the handshake is complete. Thus the condition
on tls is true with a NULL tls.subject. And code was returning a
positive match in the case of a NULL subject and a signature with
a negated match.
12 years ago
Victor Julien 717c271e58 Replace strchrnul with strchr
And add a null check then of course. strchrnul isn't supported on
all platforms.
12 years ago
Victor Julien 1e4421a7d4 Remove SCStrndup
Removed strndup wrapper as it's not available in all plaforms.
12 years ago
Victor Julien 74fb60c010 Replace remaining SCStrndup calls
Replace them with BytesToString().
12 years ago
Victor Julien c07f5397f4 Introduce BytesToString utility
Introduce a utility function to convert an array of bytes into a
null-terminated string:

 char *BytesToString(const uint8_t *bytes, size_t nbytes);

All non-printables are copied over, except for '\0', which is
turned into literal '\' '0' in the string. So the resulting string
may be bigger than the input.
12 years ago
Victor Julien 2b60871bf1 json loggers: dup bstr with bstr_util_strdup_to_c
In various places SCStrndup was used to 'dup' a bstr string, however
libhtp provides bstr_util_strdup_to_c for this. As this is a cleaner
interface, it's preferred.
12 years ago
Victor Julien 0cf71befbb util-host-os-info: scan-build fix
util-host-os-info.c:202:13: warning: Potential leak of memory pointed to by 'user_data'
            SCLogError(SC_ERR_INVALID_IPV6_ADDR, "Invalid IPV6 address inside");
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:278:35: note: expanded from macro 'SCLogError'
 #define SCLogError(err_code, ...) SCLogErr(SC_LOG_ERROR, err_code, \
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:214:82: note: expanded from macro 'SCLogErr'
                                  char _sc_log_err_msg[SC_LOG_MAX_LOG_MSG_LEN] = ""; \
                                                                                 ^~
1 warning generated.
12 years ago
Victor Julien a6474bd6bf util-host-os-info: scan build fixes
util-host-os-info.c:200:13: warning: Potential leak of memory pointed to by 'ip_str'
            SCLogError(SC_ERR_INVALID_IPV6_ADDR, "Invalid IPV6 address inside");
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:278:35: note: expanded from macro 'SCLogError'
 #define SCLogError(err_code, ...) SCLogErr(SC_LOG_ERROR, err_code, \
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:214:82: note: expanded from macro 'SCLogErr'
                                  char _sc_log_err_msg[SC_LOG_MAX_LOG_MSG_LEN] = ""; \
                                                                                 ^~
util-host-os-info.c:200:13: warning: Potential leak of memory pointed to by 'user_data'
            SCLogError(SC_ERR_INVALID_IPV6_ADDR, "Invalid IPV6 address inside");
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:278:35: note: expanded from macro 'SCLogError'
 #define SCLogError(err_code, ...) SCLogErr(SC_LOG_ERROR, err_code, \
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-debug.h:214:82: note: expanded from macro 'SCLogErr'
                                  char _sc_log_err_msg[SC_LOG_MAX_LOG_MSG_LEN] = ""; \
                                                                                 ^~
2 warnings generated.
12 years ago
Victor Julien 9ef9a14315 Fix util-debug scan-build warnings
util-debug.c:461:12: warning: Potential leak of memory pointed to by 'substr'
    return SC_ERR_SPRINTF;
           ^~~~~~~~~~~~~~
util-debug.c:856:31: warning: Potential leak of memory pointed to by 's'
                op_ifaces_ctx = SCLogInitFileOPIface(s, NULL, SC_LOG_LEVEL_MAX);
                ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
util-debug.c:1349:9: warning: Potential leak of memory pointed to by 's'
    if (log_level >= 0 && log_level < SC_LOG_LEVEL_MAX)
        ^~~~~~~~~
3 warnings generated.
12 years ago
Eric Leblond 0189b4d1eb json file: separate http params
This patch separates http keys from file to have a different value
list:

{
 "time":"01\/31\/2014-12:04:52.837245","event_type":"file","src_ip":"5.3.1.1","src_port":80,"dest_ip":"1.8.1.9","dest_port":9539,"proto":"TCP",
 "http":{"url":"/foo/","hostname":"bar.com","http_refer":"http:\/\/bar.org","http_user_agent":"Mozilla\/5.0"},
 "file":{"filename":"bar","magic":"unknown","state":"CLOSED","stored":false,"size":21}
}

One interest of this modification is that it is possible to use the
same key as the one used in http events. Thus correlating both type
of events is trivial. On code side, this will permit to factorize
the code by simply asking the underlying protocol to output its
info in a json object.

Second interest is that adding file extraction for a new protocol
will result in only changing the protocol specific json list.
12 years ago
Eric Leblond 6fd9b4b255 json: add event_type key
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
12 years ago
Eric Leblond 93a84180dc json dns: do not use array to output answer
Without this patch DNS answers for a single query are stored in a
single json event. The result is an array in the object like this one:
 {"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com","rrtype":"CNAME","ttl":734},
 {"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com.edgekey.net","rrtype":"CNAME","ttl":1710},
This type of output is not well supported in logstash. It is
displayed as it is written above and it is not possible to
query the fields.

I think the reason is that this is not logical if we consider search
query. For example if we search for "rrname" equal "s-static.ak.facebook.com"
we got one entry with two values in it. That's against the logic
of event. Furthermore, if we want to get a complete query, we can
used the id.

This patch splits the answer part in mulitple message. The result
is then accepted by logstash and fields can be queried easily.
12 years ago
Eric Leblond eab0b7fae9 json-dns: sync field names with draft rfc2629
This patch updates DNS field name to be in sync with RFC 2629:
 https://github.com/adulau/pdns-qof
This will allow to easily use Suricata with other passive DNS tools.
12 years ago
Eric Leblond 7a9efd74e4 json: sync key name with CIM
This patch is synchronizing key name with Common Information Model.
It updates key name following what is proposed in:
 http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
The interest of these modifications is that using the same key name
as other software will provide an easy to correlate and improve
data. For example, geoip setting in logstash can be applied on
all src_ip fields allowing geoip tagging of data.
12 years ago
Victor Julien 31a024c9b5 Various fixes for scan-build warnings 12 years ago
Victor Julien cd7a5ff0ca output: cleanups
Preparation of making output type for json logs configurable.
12 years ago
Victor Julien efd4c42c0a json tls log: rename to output-json-tls 12 years ago
Victor Julien 9950427466 output: check for multiple instances of drop and tls
Both the drop and tls logs are currently not designed to have multiple
instances running. So until that is changed, error out if more than one
instance is started.
12 years ago
Victor Julien 870bb23ff6 json drop log: rename to output-json-drop 12 years ago
Victor Julien 6cecb4e4d2 json dns: rename output-dnslog -> output-json-dns 12 years ago
Victor Julien 5874f52ec6 json: rename output-httplog -> output-json-http 12 years ago
Victor Julien b5ef269b03 json outputs: cleanups
Clean up header files and improve memory handling.
12 years ago
Victor Julien 3fc63d3656 jansson file log: make file log module
Turn the libjansson based file logger into a file module, as a child
of eve-log.
12 years ago
Victor Julien 039f7b3e5f tls json: turn into packet logger
Like log-tls, turn the json tls logger into a packet logger as the
protocol parser is not tx aware.

Make it a child of eve-log as well.
12 years ago
Victor Julien a9eab06593 output: simple name space support for sub modules
To avoid module name clashes, a submode abc of parent xyz, will now
register itself as xyz.abc.
12 years ago
Victor Julien 3a794f7a63 drop-json: make child of eve-log
Make drop json child of eve-log.
12 years ago
Victor Julien f0aa2ed240 json drop log: move into packet module
Move JSON drop log into a full packet module.
12 years ago
Victor Julien 4bd37cc46a log api: use AppProto instead of uint16_t 12 years ago
Victor Julien 52c3d3ad7c log api: convert all names to const
Instead of strdupping all names w/o a need, use const ptrs.
12 years ago
Victor Julien 85335d9cbe alert json: make child of eve-log
Enable alert json for eve-log by registering the module as a sub-
module of eve-log.
12 years ago
Victor Julien 42858647e2 alert-json: make full module out of json alert
Make a full module out of the json alert code in output-json-alert.[ch].
12 years ago
Victor Julien 79771ff570 output: sub-module support for other log api's
Packets:
void OutputRegisterPacketSubModule(const char *parent_name, char *name, char *conf_name,
    OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *),
    PacketLogger LogFunc, PacketLogCondition ConditionFunc);

Files:
void OutputRegisterFileSubModule(const char *parent_name, char *name, char *conf_name,
    OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *), FileLogger FileLogFunc);

Filedata:
void OutputRegisterFiledataSubModule(const char *parent_name, char *name, char *conf_name,
    OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *), FiledataLogger FiledataLogFunc);
12 years ago
Victor Julien f830cb8026 output: introduce concept of sub-modules
To support the 'eve-log' idea, we need to be able to force all log
modules to be enabled by the master eve-log module, and need to be
able to make all logs go into a single file. This didn't fit the
API so far, so added the sub-module concept.

A sub-module is a regular module, that registers itself as a sub-
module of another module:

    OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "http",
            OutputHttpLogInitSub, ALPROTO_HTTP, JsonHttpLogger);

The first argument is the name of the parent. The 4th argument is
the OutputCtx init function. It differs slightly from the non-sub
one. The different is that in addition to it's ConfNode, it gets
the OutputCtx from the parent. This way it can set the parents
LogFileCtx in it's own OutputCtx.

The runmode setup code will take care of all the extra setup. It's
possible to register a module both as a normal module and as a sub-
module, which can operate at the same time.

Only the TxLogger API is handled in this patch, the rest will be
updated later.
12 years ago
Victor Julien 8c3e71559a dns-json: turn logger to tx api
Convert Json DNS logger into a Tx Logger API logger.
12 years ago
Victor Julien bc71a43e08 http-json: separate module using tx api
Turn HTTP json logger into a Tx Logger API logger.
12 years ago
Victor Julien 4874d5abbb Various compile fixes after rebase with master 12 years ago
Tom DeCanio 18458a14fb json: rebase fixes
- restore json output-file.[ch] as output-json-file.[ch] after rebase conflict
- fix Makefile.am after merge conflict
- some dev-log-api-v4.0 rebase json fallout cleanup
12 years ago
Tom DeCanio 6fd1b31c57 Remaining JSON output pull request comment edits 12 years ago
Tom DeCanio a3d86594dc address most initial JSON pull request comments 12 years ago
Tom DeCanio 55df2d5cdb add "united" drop JSON log 12 years ago
Tom DeCanio 0c067646a8 Add "united" JSON files output 12 years ago
Tom DeCanio 730ee3d721 First cut at "united" file log output in JSON 12 years ago
Tom DeCanio 88a04742c0 JSON output cleanup 12 years ago
Tom DeCanio b4ac0d90a4 remove unused http JSON code 12 years ago
Tom DeCanio a12fa7c4e1 more output JSON cleanup 12 years ago
Tom DeCanio 6974817f72 remove dead JSON DNS output code 12 years ago
Tom DeCanio a44b2b987b JSON output cleanup 12 years ago
Tom DeCanio 3241732e27 rename alert-json.[ch] output-json.[ch] 12 years ago
Tom DeCanio 3bc95c9258 fix compile errors w/o libjansson 12 years ago
Tom DeCanio 8adbc741ba remove unused JSON TMM_*JSON enumerations 12 years ago
Tom DeCanio 07d3b38d3b Add support for JSON output to syslog/unix_stream/unix_dgram 12 years ago
Tom DeCanio 6c1de2115c JSON output cleanup 12 years ago
Tom DeCanio c654b63f6a add united TLS JSON logging 12 years ago
Tom DeCanio 51b7cf1491 add ICMP type and code support to JSON log 12 years ago
Tom DeCanio c8beb9bf9d Support for configuration of JSON http output module 12 years ago
Tom DeCanio 8c95b085c5 Add vlan and pcap_cnt to JSON logs 12 years ago
Tom DeCanio ce6b07b1b9 First cut at united .yaml configuration 12 years ago
Tom DeCanio 11f84d4ff7 beginning of JSON config alignment 12 years ago
Tom DeCanio 280e4bcb61 move some JSON alert work outside of lock 12 years ago
Tom DeCanio 34d04c3104 JSON cleanup 12 years ago
Tom DeCanio 0df6af3a0b Alert/HTTP/DNS JSON output working with Logstash 12 years ago
Tom DeCanio 5543b6eef4 nested json alert output 12 years ago
Tom DeCanio b94b8e03bd cleanup fallout from upstream merge with alert json work 12 years ago
Tom DeCanio 07571367d3 Change JSON alert syslog level to INFO 12 years ago
Tom DeCanio 860523f5bc fix NULL string into JSON in alert-json 12 years ago
Tom DeCanio e9b192fcc0 change srcport->sp dstport->dp 12 years ago
Tom DeCanio 5498654114 Add JSON formatted alert output 12 years ago
Victor Julien 7450f32351 stream: add performance output for stream pools
Add info messages at shutdown that give an indication of pool use
for the various segment and chunk pools.
12 years ago
Victor Julien 84696ebe2a stream: configurable stream chunk prealloc
The stream chunk pool contains preallocating stream chunks (StreamMsg).
These are used for raw reassembly, used in raw content inspection by
the detection engine. The default setting so far has been 250, which
was hardcoded. This meant that in setups that needed more, allocs and
frees would be happen constantly.

This patch introduces a yaml option to set the 'prealloc' value in the
pool. The default is still 250.

stream.reassembly.chunk-prealloc

Related to feature #1093.
12 years ago
Victor Julien fe1c4951f9 stream: silence stream.reassembly.raw message 12 years ago
Victor Julien b5f8f386a3 stream: configurable segment pools
The stream reassembly engine uses a set of pools in which preallocated
segments are stored. There are various pools each with different packet
sizes. The goal is to lower memory presure. Until now, these pools were
hardcoded.

This patch introduces the ability to configure them fully from the yaml.
There can be at max 256 of these pools.

Yaml layout is as follows:

stream:
  reassemble:
    segments:
      - size: 2048
        prealloc: 3000
      - size: 4
        prealloc: 1000
      - size: 1024
        prealloc: 2000

The size is the packet size. The prealloc value indicates how many
segments are set up at startup.

The pools have no limit wrt how many segments can be used of a certain
size. If the engine needs more than the prealloc size, segments are
malloc'd and free'd. The only limit here is the stream.reassemble.memcap.

If the yaml part if omitted, the default values are the same as before.

Feature #1093
12 years ago
Victor Julien b27d03e2f9 log-filestore: convert to FiledataLog API
This patch converts the log-filestore module to use the new
FiledataLog API.
12 years ago
Victor Julien 9ff6608668 Introduce Filedata Logger API
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.

The logger call in the API is as follows:
    typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
        const Packet *, const File *, const FileData *, uint8_t flags);

All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.

Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
12 years ago
Victor Julien b31e0abffe log-filestore: cleanups
Remove unused code.
Make functions static.
Move registration to the bottom.
12 years ago
Victor Julien 3e33ab4f83 log-filestore: tag truncated files as such
Tag truncated files as truncated in the same way log-file does.
12 years ago
Victor Julien 38249398a3 tx-logger: speed up
By bailing out early in case no logger is enabled for the protocol,
a significant speed up is reached.
12 years ago
Victor Julien 078ff0c0cc app-layer: add logger check to API
The new API call:
    int AppLayerParserProtocolHasLogger(uint8_t ipproto,
                                        AppProto alproto)

Returns TRUE if a logger is registered on the ip/alproto pair, and
FALSE otherwise.
12 years ago
Victor Julien 4c024f9658 profiling: add logger api labels 12 years ago
Victor Julien 0e8ad126d7 log-file: convert to file-logger API
Use file logger API.

Also, check if the protocol is HTTP before getting the HTTP
fields.
12 years ago
Victor Julien ee2a8a9cda Introduce 'file' logging API
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.

Users of this API register their Log Function through:
    OutputRegisterFileModule()

The API uses a magic settings globally. This might be changed later.
12 years ago
Victor Julien cef2eb01c5 log-file: cleanups
Make all functions static.
Move registration to the bottom.
12 years ago
Victor Julien fb5b6dd019 prelude: convert to packet logger API
Convert prelude logger to use the packet logger API.
12 years ago
Victor Julien 8623b8f941 prelude: fix configure and cleanup
Fixes configure enabling of prelude. CFLAGS is reset, so the previous
adding of -DPRELUDE was nixed. Using AC_DEFINE now.

Cleanups:
- make functions static
- simplify handling of no prelude support
- move registration to the bottom
12 years ago
Victor Julien b0a9d08267 alert-syslog: convert to packet logger API
Convert Syslog alert logger to packet logger API.
12 years ago
Victor Julien ec20f45916 alert-syslog: cleanup
Remove separate ipv4 and ipv6 registration functions.
Make all functions static.
Move registration function to the bottom.
Simplify OS_WIN32 wrappers usage.
12 years ago
Victor Julien 6c36824d69 alert-pcapinfo: convert to packet logger API
Convert pcap-info to use the packet logger API.
12 years ago
Victor Julien a536e73695 alert-pcapinfo: clean up
Make functions static.
Move registration to the bottom.
12 years ago
Victor Julien 73377048fd alert-debuglog: minor cleanups
Clean up log functions after packet logger conversion. No more
PacketQueue arguments.
12 years ago
Victor Julien cd4796f3ca alert-debuglog: port to packet logger api
Convert AlertDebugLog to Packet logger API. Convert packet args to
const.
12 years ago
Victor Julien 4b57d0272c alert-debug log cleanups
Make all funcs but registration static.
Remove stale registation prototypes.
Move registation func to the bottom.
12 years ago
Victor Julien 504f39adef log-tls: convert to packet logger API
This patch converts log-tls to use the packet logger API. The packet
logger API was choosen as the TLS parser is not transaction aware.

To make sure the state is only logged once, the flag
SSL_AL_FLAG_STATE_LOGGED was added to the parser. This flag is checked
by the condition function, and set at the end of the Logger function.
12 years ago
Victor Julien bcf5c1f2fb log-tls: clean ups
Make all functions static. Remove separate ipv4 and ipv6 registration
functions. Move register function to the bottom so that we no longer
need function prototype declarations.
12 years ago