180 Commits (818656524003c4501401709aecffb19c28ca588c)

Author	SHA1	Message	Date
Victor Julien	b8659daef7	Add stream engine counters ... Added stream counters: - tcp.reassembly_memuse -- current memory use by reassembly in bytes - tcp.memuse -- current memory use by stream tracking in bytes - tcp.reused_ssn -- ssn reused by new session with identical tuple - tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits	14 years ago
Eric Leblond	4071d3cf57	PACKET_INITIALIZE is enough for packet init.	14 years ago
Victor Julien	30d84ab20d	Unlock flow in StreamTcpSegmentForEach if there is no TCP session.	14 years ago
Eric Leblond	628bfcc1b9	stream: Change return of StreamSegmentForEach ... The function now returns the number of segment where the callback has ben runned successfully.	14 years ago
Eric Leblond	4f0cdf28a3	Introduce StreamSegmentForEach function ... This patch introduces a function called StreamMsgForEach which can be used to run a callback on all segments of a stream. This is currently only supported for TCP as this is the only streaming aware protocol.	14 years ago
Victor Julien	f5ef842752	Implement a counter for TCP packets with invalid checksums: tcp.invalid_checksum. Bug #311.	14 years ago
Victor Julien	8208eacd79	Convert stream memcaps to u64. Bug #332.	14 years ago
Eric Leblond	de59c9f4b1	Add and use utility functions for checksum computing.	14 years ago
pilcrow	ed69eeab14	Safer macro parenthesization and do/while use	14 years ago
Anoop Saldanha	8363533a02	support for forced stream reassembly for to be pruned flows	14 years ago
Anoop Saldanha	c365bafbf6	We now inspect timed out streams + streams not processed as yet, at engine shutdown	14 years ago
Eileen Donlon	89599d3b9b	fixed bug 288; corrected config boolean parsing problems	14 years ago
Eric Leblond	7425bf5ca6	Rename some decode event structure and macro. ... This patch renames DECODER_SET_EVENT, DECODER_ISSET_EVENT and some other structures to ENGINE equivalent to take into account the fact the event list is now related to all engines and not only to decoder.	14 years ago
Victor Julien	fca541f40e	Add per app layer parser profiling ... Per packet per app layer parser profiling. Example summary output: Per App layer parser stats: App Layer IP ver Proto cnt min max avg -------------------- ------ ----- ------ ------ ---------- ------- ALPROTO_HTTP IPv4 6 163394 126 38560320 42814 ALPROTO_FTP IPv4 6 644 117 26100 2566 ALPROTO_TLS IPv4 6 670 117 7137 799 ALPROTO_SMB IPv4 6 114794 126 225270 957 ALPROTO_DCERPC IPv4 6 5207 126 25596 1266 Also added to the csv out. In the csv out there is a new column "stream (no app)" that removes the app layer parsers from the stream tracking. So raw stream engine performance becomes visible.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Victor Julien	add02a4ef3	Fix handling of FIN/ACK packet on TCP state TCP_FIN_WAIT2.	15 years ago
Victor Julien	4025567a5a	Fix a number of unittests not properly initializing a packet causing issues on some archs.	15 years ago
Gerardo Iglesias Galvan	dd5e438d6f	Make all access to memory tracking counters in stream engine lock protected	15 years ago
Victor Julien	e1d4e16645	Simplify packet decoding macro's.	15 years ago
Victor Julien	54cd3552e1	Remove tunnel_proto field from Packet structure.	15 years ago
Victor Julien	8999de2f93	Add proper RST handling to all TCP states.	15 years ago
Victor Julien	1578ef1e3e	Make sure that the stream engine fully reassembles both sides of the session upon receiving a valid RST.	15 years ago
Victor Julien	14ad853b94	Process a stream end pseudo packet when going from TIME_WAIT to CLOSED.	15 years ago
Victor Julien	d0374ced38	Implement SACK in the stream engine.	15 years ago
Victor Julien	892a8a4985	Make stream inline use the chunk size settings.	15 years ago
Victor Julien	2dc057d1b1	Set datalink on stream pseudo packets to prevent unified2 from writing a malformed record.	15 years ago
Victor Julien	4f5aad1476	Enforce configurable minimum chunk size in raw stream reassembly. Minor stream cleanups, unittest updates.	15 years ago
Victor Julien	936b34ddf6	Remove minimum init chunk length code, set a default limit of 2560 to the minimum chunk size, allow toclient raw reassembly to start even if toserver hasn't started yet.	15 years ago
Victor Julien	bc7e21aee6	Add special sguil mode to log-pcap to support logging into date based directory structure and rotate when the day passes. Also do not log packets beyond stream reassembly depth and encrypted traffic.	15 years ago
Eric Leblond	9be1f1a31c	Use GET_PKT macros.	15 years ago
Victor Julien	174db08567	Force reassembly of unack'd data on receiving a valid RST packet.	15 years ago
Victor Julien	b9429ecec1	Fix invalid RST considered valid due to wrong returns codes. Only validate ACK from a RST packet if an ACK value was set.	15 years ago
Victor Julien	cb67d61ab5	Fix broken setup of end of stream pseudo packet.	15 years ago
Eric Leblond	a8417377e7	Don't use direct pkt access ... pkt field in Packet needs to be accessed via macro. This patch supress some direct access.	15 years ago
Victor Julien	00e4dde6a6	Fix PKT_STREAM_EOF never being set, resulting in some raw stream chunks never being inspected. Improve debug output.	15 years ago
Victor Julien	1dca88fe69	Do the actual checksum recalculation and packet replacement on modifing a packet in the stream engine.	15 years ago
Victor Julien	3a774165fa	Initial version of a inline raw reassembly function that reassembles in a sliding window. Introduce new unittest helpers for stream reassembly.	15 years ago
Victor Julien	8cacd5fe50	Fix the stream.inline config option. Set PKT_STREAM_EST flag also for packets that are part of a session in a state beyond TCP_ESTABLISHED.	15 years ago
Victor Julien	a8bb98836b	Don't handle and validate the TCP timestamp at the same time. Instead validate first, then later when all other validation has been done as well, handle.	15 years ago
Victor Julien	8d3f9c53a9	Minor cleanups.	15 years ago
Victor Julien	0f072648e6	Another iteration of the reassembly depth enforcement, now considering retransmissions.	15 years ago
Victor Julien	935958219d	Rename RST validation function to match convention	15 years ago
Victor Julien	94fe0d5fa2	Add ACK validation to Reset/RST validation code.	15 years ago
Victor Julien	7af9c58af7	Improve ACK value validation, timestamp checking code. Overall layout.	15 years ago
Victor Julien	6ffb9da9be	Better support ack/psh data packets on several states. Updates to ack validation code.	15 years ago
Victor Julien	25f5589078	First round of adding 'stream events'. Basic stream tracking events added.	15 years ago
Victor Julien	2849d2b1d3	Initial code for stream 'inline' mode: packets that are (partly) overlapping with already accepted packets (meaning in the streams seg list) are rewritten to make sure they contain the exact same data.	15 years ago
Victor Julien	3857154f4b	Fix the pseudo packet having the wrong proto set, causing massive fp's. Flag packets to be part of the established phase of a tcp session, so we won't prematurely inspect the app layer state.	15 years ago
Gurvinder Singh	55a863359c	support for pseudo packet creation from reassembled stream segments	15 years ago
Gurvinder Singh	2beb7af7f8	support for validating the ACK before updating the last_ack field and also update next_seq if we missed the last packet	15 years ago