We want to add counters in order to track the number of times we hit a
decode event. A decode event is related to an error in the protocol
decoding over a certain packet.
This patch fist modifies the decode-event list, reordering it in order
to separate single packet events from stream-related events and adding
the prefix "decoder" to decode events.
The counters are created during the decode setup and the relative event
counter is increased every time a packet with the flag PKT_IS_INVALID is
finalized in the decode phase
If stream.inline setting was missing it would default to IDS.
This patch changes the default to 'auto', meaning that in IPS mode
the stream engine also uses IPS mode and in IDS mode it's still in
IDS mode.
Bug #1570
When defrag creates a new reassembled IP packet, it then passes this
packet to the IP decoder. If this decoder returns an error the packet
is returned back to the packet pool with a call to TmqhOutputPacketpool
This lead to the first problem. The returned packet had it's p->root
pointer set, and it's PKT_TUNNEL flag set. This could cause problems
in TmqhOutputPacketpool, as this may reference the packet referenced
in p->root.
The second and more glaring problem is that the packet that was
returned to the packetpool, was still returned by the Defrag function
and processed further. It would then at the end of it's processing
be returned to the packet pool, which at this point already had a
reference to this packet.
This patch fixes both issues by unsetting the tunnel references and
returning NULL from Defrag in this case.
When --with-libpcre-libraries is used we skip the libpcre 8.35 check
since pkg-config might still point to the 8.35 version installed
although newer version was passed with --with-libpcre-libraries.
Logic used when adding a new prefix to a node was not correct
as we were allocating a prefix that could be at the end unused.
This patch is updating the code to have a complete creation to
be done if and only if we are needing the complete object. In
the other cases, it was enough to use the function input values.
This fixes:
104 (48 direct, 56 indirect) bytes in 2 blocks are definitely lost in loss record 184 of 327
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x9C2DAD: SCRadixCreatePrefix (util-radix-tree.c:144)
by 0x9AFA5B: SCRadixAddKey (util-radix-tree.c:522)
by 0x9B1A4D: SCRadixAddKeyIPV4Netblock (util-radix-tree.c:897)
by 0x67C824: IPOnlyPrepare (detect-engine-iponly.c:1197)
by 0x55172B: SigAddressPrepareStage2 (detect.c:3534)
by 0x5486F4: SigGroupBuild (detect.c:4671)
by 0x547C87: SigLoadSignatures (detect.c:538)
by 0x8FB5D0: LoadSignatures (suricata.c:1976)
by 0x8F3B32: main (suricata.c:2342)
TmThreadCreate copy string provided as name for threads to
avoid any issue is a non allocated string is used.
This patch also introduce TmThreadSetGroupName function. This
function is used to be sure we have an allocation when
assigning the thread group name. This way we can free allocated
memory at exit.
Both code changes have required some fixes in different parts of
the code to be in sync with the new API.
Good point about these changes is that it fixes an inconsistency
were some names were not allocated and some were.
This fixes:
72 bytes in 3 blocks are definitely lost in loss record 153 of 316
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x9AF041: SCRadixCreateRadixTree (util-radix-tree.c:430)
by 0x50FF5D: DefragPolicyLoadFromConfig (defrag-config.c:138)
by 0x5129F5: DefragInit (defrag.c:962)
by 0x87ECFD: UnixSocketPcapFilesCheck (runmode-unix-socket.c:386)
by 0x90FEC0: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x913C6D: UnixManager (unix-manager.c:980)
by 0x9072F3: TmThreadsManagement (tm-threads.c:602)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
This fixes:
16 bytes in 2 blocks are definitely lost in loss record 69 of 319
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x85955D: OutputStatsLogThreadInit (output-stats.c:118)
by 0x4CAE13: StatsMgmtThread (counters.c:352)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
This fixes:
96 bytes in 4 blocks are definitely lost in loss record 178 of 320
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x8FF88C: TmqhOutputFlowSetupCtx (tmqh-flow.c:163)
by 0x908D5F: TmThreadCreate (tm-threads.c:1097)
by 0x909A62: TmThreadCreatePacketHandler (tm-threads.c:1156)
by 0x8790CF: RunModeFilePcapAutoFp (runmode-pcap-file.c:188)
by 0x8837CB: RunModeDispatch (runmodes.c:372)
by 0x87F2A5: UnixSocketPcapFilesCheck (runmode-unix-socket.c:393)
by 0x910460: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x91420D: UnixManager (unix-manager.c:980)
by 0x907853: TmThreadsManagement (tm-threads.c:602)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
Fix cleaning of threads where mutex and condition where not freed.
This fixes:
352 (192 direct, 160 indirect) bytes in 4 blocks are definitely lost in loss record 301 of 327
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x909404: TmThreadInitMC (tm-threads.c:1764)
by 0x908DE7: TmThreadCreate (tm-threads.c:1120)
by 0x90A326: TmThreadCreateMgmtThread (tm-threads.c:1183)
by 0x4CA0AD: StatsSpawnThreads (counters.c:856)
by 0x87F254: UnixSocketPcapFilesCheck (runmode-unix-socket.c:396)
by 0x910330: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x9140DD: UnixManager (unix-manager.c:980)
by 0x9077F3: TmThreadsManagement (tm-threads.c:600)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
Queue name was sometimes allocated and sometimes not. This
patch updates the behavior of creation function so it is
always allocated. This way we can free it at exit and fix
memory leak.
This fixes:
900 bytes in 110 blocks are definitely lost in loss record 322 of 329
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x803E0A9: strdup (in /lib/x86_64-linux-gnu/libc-2.21.so)
by 0x90090E: StoreQueueId (tmqh-flow.c:112)
by 0x8FFEA8: TmqhOutputFlowSetupCtx (tmqh-flow.c:180)
by 0x908C7F: TmThreadCreate (tm-threads.c:1095)
by 0x909982: TmThreadCreatePacketHandler (tm-threads.c:1154)
by 0x87906F: RunModeFilePcapAutoFp (runmode-pcap-file.c:188)
by 0x88376B: RunModeDispatch (runmodes.c:372)
by 0x87F245: UnixSocketPcapFilesCheck (runmode-unix-socket.c:393)
by 0x9102B0: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x91405D: UnixManager (unix-manager.c:980)
by 0x907773: TmThreadsManagement (tm-threads.c:600)
tm-queue: fix logic WIP
This function is causing a memleak because it is necessary to clean
up after usage.
This fixes at least:
37 (32 direct, 5 indirect) bytes in 1 blocks are definitely lost in loss record 104 of 394
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x66D0C7B: ??? (in /usr/lib/x86_64-linux-gnu/libjansson.so.4.7.0)
by 0x943584: LiveDeviceIfaceList (util-device.c:264)
by 0x910889: UnixCommandExecute (unix-manager.c:486)
by 0x910D7E: UnixCommandRun (unix-manager.c:545)
by 0x911193: UnixMain (unix-manager.c:593)
by 0x913CC7: UnixManager (unix-manager.c:961)
by 0x907753: TmThreadsManagement (tm-threads.c:600)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
This fixes:
2,595 (1,824 direct, 771 indirect) bytes in 57 blocks are definitely lost in loss record 328 of 332
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x66D0C7B: ??? (in /usr/lib/x86_64-linux-gnu/libjansson.so.4.7.0)
by 0x911A27: UnixManagerListCommand (unix-manager.c:766)
by 0x9108A9: UnixCommandExecute (unix-manager.c:486)
by 0x910D9E: UnixCommandRun (unix-manager.c:545)
by 0x9111B3: UnixMain (unix-manager.c:593)
by 0x913D27: UnixManager (unix-manager.c:961)
by 0x907773: TmThreadsManagement (tm-threads.c:600)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
This fixes:
Direct leak of 31752 byte(s) in 3969 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0xe385b9 in OutputTxLogThreadInit /home/pmanev/sandnet-qa/stage/oisf/src/output-tx.c:193:34
#2 0x106c255 in TmThreadsSlotPktAcqLoop /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:295:17
#3 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
This fixes:
Direct leak of 31792 byte(s) in 3974 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0xd86ce2 in OutputFiledataLogThreadInit /home/pmanev/sandnet-qa/stage/oisf/src/output-filedata.c:308:34
#2 0x106c255 in TmThreadsSlotPktAcqLoop /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:295:17
#3 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
This fixes:
Direct leak of 31832 byte(s) in 3979 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0xe22129 in OutputPacketLogThreadInit /home/pmanev/sandnet-qa/stage/oisf/src/output-packet.c:123:34
#2 0x106c255 in TmThreadsSlotPktAcqLoop /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:295:17
#3 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
This fixes:
Direct leak of 31856 byte(s) in 3982 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0xd7ff39 in OutputFileLogThreadInit /home/pmanev/sandnet-qa/stage/oisf/src/output-file.c:182:34
#2 0x106c255 in TmThreadsSlotPktAcqLoop /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:295:17
#3 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
Code was unused and was leaking memory.
This fixes:
Direct leak of 614240 byte(s) in 3839 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0x11bc12e in LogFileNewCtx /home/pmanev/sandnet-qa/stage/oisf/src/util-logopenfile.c:474:27
#2 0xcf7ef2 in LogFilestoreLogInitCtx /home/pmanev/sandnet-qa/stage/oisf/src/log-filestore.c:430:31
#3 0xec3275 in RunModeInitializeOutputs /home/pmanev/sandnet-qa/stage/oisf/src/runmodes.c:763:26
#4 0xeae17f in UnixSocketPcapFilesCheck /home/pmanev/sandnet-qa/stage/oisf/src/runmode-unix-socket.c:391:9
#5 0x109bc37 in UnixCommandBackgroundTasks /home/pmanev/sandnet-qa/stage/oisf/src/unix-manager.c:430:20
#6 0x10a9be2 in UnixManager /home/pmanev/sandnet-qa/stage/oisf/src/unix-manager.c:977:9
#7 0x1075643 in TmThreadsManagement /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:600:9
#8 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
This patch adds a new callback PktAcqBreakLoop() in TmModule to let
packet acquisition modules define "break-loop" functions to terminate
the capture loop. This is useful in case of blocking functions that
need special actions to take place in order to stop the execution.
Implement this for PF_RING
Those lines were included former to prevent possible null pointer
dereference but that won't happen anymore with the rest of the rework
done in the code. The code even results in a control flow issue reported
by coverity scan, so just remove it.
Suricata crashed when variable (either address or port)
referred to itself or if one created a looped chain of
variables. For instance:
HOME_NET: "!$EXTERNAL_NET"
EXTERNAL_NET: "!$HOME_NET"
Or:
Var1: "$Var2"
Var2: "$Var3"
Var3: "$Var1"
This patch fixes the following leak:
Direct leak of 9982880 byte(s) in 2902 object(s) allocated from:
#0 0x4c253b in malloc ??:?
#1 0x10c39ac in MimeDecInitParser /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/util-decode-mime.c:2379
#2 0x6a0f91 in SMTPProcessRequest /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1085
#3 0x697658 in SMTPParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1185
#4 0x68fa7a in SMTPParseClientRecord /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1208
#5 0x6561c5 in AppLayerParserParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-parser.c:908
#6 0x53dc2e in AppLayerHandleTCPData /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer.c:444
#7 0xf8e0af in DoReassemble /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:2635
#8 0xf8c3f8 in StreamTcpReassembleAppLayer /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3028
#9 0xf94267 in StreamTcpReassembleHandleSegmentUpdateACK /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3404
#10 0xf9643d in StreamTcpReassembleHandleSegment /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3432
#11 0xf578b4 in HandleEstablishedPacketToClient /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2245
#12 0xeea3c7 in StreamTcpPacketStateEstablished /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2489
#13 0xec1d38 in StreamTcpPacket /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:4568
#14 0xeb0e16 in StreamTcp /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:5064
#15 0xff52a4 in TmThreadsSlotVarRun /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:130
#16 0xffdad1 in TmThreadsSlotVar /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:474
#17 0x7f7cd678d181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 (discriminator 2)
We come to this case when a SMTP session contains at least 2 mails
and then the ending of the first is not correctly detected. In that
case, switching to a new tx seems a good solution. This way we still
have partial logging.
If SMTP session is weird then we may reach a state where a field
like MAIL FROM is seen as duplicated.
Valgrind output is:
30 bytes in 1 blocks are definitely lost in loss record 96 of 399
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x4A5803: SMTPParseCommandWithParam (app-layer-smtp.c:996)
by 0x4A4DCE: SMTPParseCommandMAILFROM (app-layer-smtp.c:1016)
by 0x4A3F55: SMTPProcessRequest (app-layer-smtp.c:1127)
by 0x4A1F8C: SMTPParse (app-layer-smtp.c:1191)
by 0x493AD7: SMTPParseClientRecord (app-layer-smtp.c:1214)
by 0x4878A6: AppLayerParserParse (app-layer-parser.c:908)
by 0x42384E: AppLayerHandleTCPData (app-layer.c:444)
by 0x8D7EAD: DoReassemble (stream-tcp-reassemble.c:2635)
by 0x8D795F: StreamTcpReassembleAppLayer (stream-tcp-reassemble.c:3028)
by 0x8D8BE0: StreamTcpReassembleHandleSegmentUpdateACK (stream-tcp-reassemble.c:3404)
by 0x8D8F6E: StreamTcpReassembleHandleSegment (stream-tcp-reassemble.c:3432)
Victor Julien
Maurizio Abba
Eric Leblond
Andreas Herz
cardigliano
Alexander Gozman
Jason Ish