Commit Graph

4955 Commits (6c2ae469be6a79b1405ca793f15c74c0fba0e3a8)

Author SHA1 Message Date
Victor Julien 9f55ca0057 flow: add flow_end_flags field, add logging
The flow end flags field is filled by the flow manager or the flow
hash (in case of forced timeout of a flow) to record the timeout
conditions in the flow:
- emergency mode
- state
- reason (timed out or forced)

Add logging to the flow logger.
12 years ago
Victor Julien fc6ad56944 flow: move FlowGetFlowState
Move FlowGetFlowState to flow-private.h so that all parts of the flow
engine can use it.
12 years ago
Victor Julien e6ed6731b1 flow log: log TCP state
Log the TCP state at timeout.
12 years ago
Victor Julien 8c231702d9 flow-recycler: speed up flow-recycler shutdown
Thread was killed by the generic TmThreadKillThreads instead of
the FlowKillFlowRecyclerThread. The latter wakes the thread up, so
that shutdown is quite a bit faster.
12 years ago
Victor Julien 6f9a2fcd58 flow: log individual tcp flags
Log the tcp flags.
12 years ago
Victor Julien f4dfaacff3 netflow: log individual tcp flags
Log the tcp flags.
12 years ago
Victor Julien eaf01449e3 json: add tcp flags to json utility function
Turns a flags bitfield into a set of json bools.
12 years ago
Victor Julien db15339f47 netflow-json: initial version
Initial version of netflow module, a flow logger that logs each
direction in a completely separate record (line).
12 years ago
Victor Julien 07b7f66f3c flow-log: log TCP flags per direction
In addition to flags for the entire session, also log out TCP flags
for both directions separately.
12 years ago
Victor Julien 3bb0ccba98 stream: track TCP flags per stream direction
For netflow logging track TCP flags per stream direction. As the struct
had no more space left without expanding it, the flags and wscale
fields are now compressed.
12 years ago
Victor Julien d19a15701c flow: init logger thread data for decoders
Initialize the output flow api thread data for the decoder threads.
12 years ago
Victor Julien 98c88d5170 decode: pass ThreadVars to DecodeThreadVarsFree
Flow output thread data deinit function which will be called from
DecodeThreadVarsFree will need it.
12 years ago
Victor Julien de034f1867 flow: prepare flow forced reuse logging
Most flows are marked for clean up by the flow manager, which then
passes them to the recycler. The recycler logs and cleans up. However,
under resource stress conditions, the packet threads can recycle
existing flow directly. So here the recycler has no role to play, as
the flow is immediately used.

For this reason, the packet threads need to be able to invoke the
flow logger directly.

The flow logging thread ctx will stored in the DecodeThreadVars
stucture. Therefore, this patch makes the DecodeThreadVars an argument
to FlowHandlePacket.
12 years ago
Victor Julien bd490736c2 flow: take flow pkt & byte count out of debug
Until now the flow packet and byte counters were only available in
DEBUG mode. For logging purposes they are now available always.
12 years ago
Victor Julien e6ee5feaba flow: don't BUG_ON if no loggers are enabled
API is always called, even if no loggers are enabled. Don't abort()
in this case.
12 years ago
Victor Julien 52b0ec027e flow: clean up recycle queue at shutdown
Mostly for tests that don't start the recycler thread, make sure
all flows are cleaned up.
12 years ago
Victor Julien 4aff4c650f flow unittest: update flow manager unit test
Test now tests a different queue.
12 years ago
Victor Julien 7acea2c66d flow: track lastts in struct timeval
Track full timestamp for lastts in flows to be able to log it.
12 years ago
Victor Julien c66a29b67d flow: track bytes per direction
Track bytes in both flow directions for logging purposes.
12 years ago
Victor Julien f828793f8f flow log: log start/end times
Log time of first packet (flow creation) and of the last packet.
12 years ago
Victor Julien 672f6523a7 flow-log: log TCP flags seen
Log TCP flags seen during the life time of a flow/session.
12 years ago
Victor Julien fddeca8aae tcp: track TCP packet flags per session
For logging out in flow logging.
12 years ago
Victor Julien ec7d446f16 flow-log: log pkts, bytes
Only in DEBUG currently.
12 years ago
Victor Julien 3c7af02067 flow-json-log: stub
Stub for JSON flow logger.
12 years ago
Victor Julien c7ebfd1b68 flow: flow log threading setup
Set up threading for the flow logger.
12 years ago
Victor Julien e30c083cff flow log: call logger from recycler
Call the flow logger API from the recycler thread, so that timed
out flows are logged.
12 years ago
Victor Julien 115ad1e81f flow: output api stub
Basic output API for flow logging.
12 years ago
Victor Julien a52a4ae9d4 flow recycler: unix socket support
Support starting and shutting down the flow recycler thread in the
unix socket runmode.
12 years ago
Victor Julien f476732139 flow recycler: shutdown
Only shut down when all flows in the recycle queue have been processed.
12 years ago
Victor Julien f26f82e9a6 flow: move flow cleanup to new 'recycler'
Move Flow clean up from the flow manager to the new flow recycler.
12 years ago
Victor Julien 94cb52897b flow: introduce FlowRecycler stub
FlowRecycler thread stub. Start/stop code.
12 years ago
Victor Julien e892d99827 flow: new flow queue: flow_recycle_q
This queue will be used by the FlowManager to pass timed out flows
to another thread that will do the actual cleanup.
12 years ago
Victor Julien fdd407751e Fix eve 'filetype' parsing
Now that we use 'filetype' instead of 'type', we should also
use 'regular' instead of 'file'.

Added fallback to make sure we stay compatible to old configs.
12 years ago
Alexander Gozman bfb6175bf6 Fixed memory leak 12 years ago
Alexander Gozman a0bb4477db Fix possible crash when logfile descriptor is invalid 12 years ago
Alexander Gozman 8048eebd39 Fix handling filetype for eve log 12 years ago
Alexander Gozman 54193e89d5 Fixed variables names in suricata.yaml.in Changed logging logic - now it's possible to enable different payload dumping modes separately Fixed bug in dumping packet without stream segments Fixed indents 12 years ago
Alexander Gozman 6d569013c6 Changed attribute name for printable payload 12 years ago
Alexander Gozman c770ade9c2 Changed variable name when dumping single packet 12 years ago
Alexander Gozman 2a4c7ee5dc Add ability to encode payload in Base64 12 years ago
Alexander Gozman ffac6b71e2 Fixed stream handling Fixed some coding style issues 12 years ago
Matt Carothers ab58ee2676 Add packet and payload logging to JSON alert output 12 years ago
Victor Julien c53b428079 Fix engine getting stuck because of optimizations
At -O1+ in both Gcc and Clang, PacketPoolWait would optimize the
wait loop in the wrong way. Adding a compiler barrier to prevent
this optimization issue.
12 years ago
Victor Julien c4a8e2cd14 Remove unused variables 12 years ago
Victor Julien 1d9278bef4 Fix packet pool pending stack adds
Add packets after the first as the list/stack head as well.
12 years ago
Victor Julien b5d3b7e92a Fix pcap packet acquisition methods
Fix pcap packet acquisition methods passing 0 to pcap_dispatch.
Previously they passed the packet pool size, but the packet_q_len
variable was now hardcoded at 0.

This patch sets packet_q_len to 64. If packet pool is empty, we fall
back to direct alloc. As the pcap_dispatch function is only called
when packet pool is not empty, we alloc at most 63 packets.
12 years ago
Ken Steele 0dd16461cf Update max-pending-packet comments to show it is now per-thread.
Updated suricata.yaml and comments in the code.
12 years ago
Ken Steele 28ccea51d3 Add error checking for pthread_setspecific() and pthread_key_create(). 12 years ago
Ken Steele b1a7e76ca7 Use posix_memalign instead of mm_malloc on non-Windows systems. 12 years ago
Ken Steele a38d5a0135 Implement thread specific data option when __thread is not available. 12 years ago
Ken Steele be448aef22 For PktPool add local pending freed packets list.
Better handle the autofp case where one thread allocates the majority
of the packets and other threads free those packets.

Add a list of locally pending packets. The first packet freed goes on the
pending list, then subsequent freed packets for the same Packet Pool are
added to this list until it hits a fixed number of packets, then the
entire list of packets is pushed onto the pool's return stack. If a freed
packet is not for the pending pool, it is freed immediately to its pool's
return stack, as before.

For the autofp case, since there is only one Packet Pool doing all the
allocation, every other thread will keep a list of pending packets for
that pool.

For the worker run mode, most packets are allocated and freed locally. For
the case where packets are being returned to a remote pool, a pending list
will be kept for one of those other threads, all others are returned as before.

Which remote pool for which to keep a pending list is changed each time the
pending list is returned. Since the return pending pool is cleared when it is
freed, then next packet to be freed chooses the new pending pool.
12 years ago
Ken Steele 3c6e01f653 Replace ringbuffer in Packet Pool with a stack for better cache locality
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.

The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.

Returning packets back to their "home" stack keeps the stacks from getting out
of balance.

The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.
12 years ago
Victor Julien 94571c5dd2 AC: shrink output table after initialization 12 years ago
Victor Julien 04c9db398e AC: reduce realloc for new states
Don't realloc per state add, but grow by larger blocks per realloc.
12 years ago
Ken Steele ba1e2ed69d Fix Boyer Moore Nocase bug where BoyerMooreCtxToNocase was missing.
Whenever DETECT_CONTENT_NOCASE is set for a BoyerMoore matcher, the
function BoyerMooreCtxToNocase() must be called. This call was missing
in AppLayerProtoDetectPMRegisterPattern().

Also created BoyerMooreNocaseCtxInit() that calls BoyerMooreCtxToNocase()
to make some code cleaner and safer.
12 years ago
Ken Steele 967f7aefde Store Boyer Moore no case strings in lower case.
Rather than converting the search string to lower case while searching,
convert it to lowercase during initialization.

Changes the Boyer Moore search API for take BmCtx

Change the API for BoyerMoore to take a BmCtx rather than the two parts that
are stored in the context. Which is how it is mostly used. This enforces
always calling BoyerMooreCtxToNocase() to convert to no-case.

Use CtxInit and CtxDeinit functions to create and destroy the context,
even in unit tests.
12 years ago
Ken Steele 54214d1251 Fix comment wording in Boyer Moore pattern matcher. 12 years ago
Eric Leblond de6dac0043 Remove pcapinfo output
EVE logging is a really good substitute for pcapinfo. Suriwire is
now supporting EVE output so it is not anymore necessary to have
pcapinfo in Suricata.
12 years ago
Victor Julien be1979b2f9 pcap-log: support dynamic file names in multi
When using multi mode, the filename can use a few variables:

%n -- thread number, where the 1st thread has 1, and it increments
%i -- thread id (system thread id, similar to pid)
%t -- timestamp, where seconds or seconds+usecs depends on
      the ts-format option.

Example:
filename: filename: pcaps/%n/pcap.%t
This will translate to: pcaps/3/pcap.1256792217 for the 3rd thread.

Note that while it's possible to use directories, they won't be
created. So make sure they exist.
12 years ago
Victor Julien 6cebe7ef7b pcap-log: performance optimizations
This patch adds a field 'is_private' to PcapLogData, so that the
using thread knows if it needs to lock access to it or not.

Reshuffle PcapLogData to roughly match order of access.
12 years ago
Victor Julien 923341fa05 pcap-log: implement multi mode
This patch implements a new mode in pcap-logging: 'multi'. It stores
a pcap file per logger thread, instead of just one file globally.

This removes lock contention, so it brings a lot more performance.

The trade off is that there are now mulitple files where there would
be one before.

Files have a thread id added to their name: base_name.tid.ts, so by
we have something like: "log.pcap.20057.1254500095".
12 years ago
Victor Julien 4922cd2d36 pcap-log: introduce PcapLogThreadData
PcapLog uses the global data structure PcapLogData as thread data
as well. This is possible because all operations on it are locked.

This patch introduces PcapLogThreadData. It contains a pointer to
the PcapLogData. Currently to the global instance, but in the future
it may hold a thread-local instance of PcapLogData.
12 years ago
Victor Julien bbc8c1ea05 log-pcap: multi mode yaml parsing
In preparation of the multi file mode, add 'multi' as a value to
the mode.
12 years ago
Victor Julien cf4db47931 log-pcap: lock profiling
Add lock profiling to pcap logging profiling.
12 years ago
Victor Julien adde58d2cb log-pcap: improve profiling
Add profiling to a logfile. Default is $log_dir/pcaplog_stats.log

The counters for open, close, rotate, write and handles are written
to it, as well as:
- total bytes written
- cost per MiB
- cost per GiB

Option is disabled by default.
12 years ago
Victor Julien f6c5b1715f Update log-pcap.h, add license
Clean up log-pcap.h and add the OISF license header.
12 years ago
Victor Julien 1af2f6528b log-pcap code cleanups
Code cleanups to make functions static.
12 years ago
Victor Julien fd7dd09f4c profiling: add pcap logger profiling
Tracks: file open, file close, file rotate (which includes open and
close), file write and open handles.

Open handles measures the cost of open the libpcap handles.
12 years ago
Victor Julien ed84c8795d Update version number to 2.1dev 12 years ago
Victor Julien 2646edc129 Profiling: fix compilation on CentOS5
Bug #1207
12 years ago
Victor Julien f232fdc0c9 htp: init memuse atomics
In case of the spinlocked fallback code the lock was uninitialized.
12 years ago
Victor Julien f06e5f3c73 ethtool: add missing include necessary for CentOS5 12 years ago
Alexander Gozman 405baa3cb2 Fix compile-time error on old kernels and ethtool.h 12 years ago
sxhlinux 546ae9737b Update log-file.c
test whether tx_ud is NULL
12 years ago
Ken Steele f2e777e3a5 Fix lowercase table initialization (bug 1221)
The for loop needed to check for < 256, not < 255.
12 years ago
Victor Julien 27eb0f450a defrag: fix timeout setting when config is missing
When the config is missing, DefragPolicyGetHostTimeout will default
to returning -1. This will effectively set no timeout at all, leading
to defrag trackers being freed too early.
12 years ago
Eric Leblond 97ca02f0c5 defrag: fix reconstruction
This patch is fixing an issue in defragmentation code. The
insertion of a fragment in the list of fragments is done with
respect to the offset of the fragment. But the code was using
the original offset of the fragment and not the one of the
new reconstructed fragment (which can be different in the
case of overlapping segment where the left part is trimmed).

This case could lead to some evasion techniques by causing
Suricata to analyse a different payload.
12 years ago
Eric Leblond 09fd7060ec unix socket: fix valgrind issue
This patch fixes the following issue reported by valgrind:
 31 errors in context 1 of 1:
 Conditional jump or move depends on uninitialised value(s)
    at 0x8AB2F8: UnixSocketPcapFilesCheck (runmode-unix-socket.c:279)
    by 0x97725D: UnixCommandBackgroundTasks (unix-manager.c:368)
    by 0x97BC52: UnixManagerThread (unix-manager.c:884)
    by 0x6155F6D: start_thread (pthread_create.c:311)
    by 0x6E3A9CC: clone (clone.S:113)

The running field in PcapCommand was not initialized.
12 years ago
Eric Leblond a33d1e28e9 unix-manager: fix crash when client disconnect
This patch fixes an issue in unix socket handling. It is possible
that a socket did disconnect when analysing a command and because
the data treatment is done in a loop on clients this was leading
to a update of the list of clients during the loop. So we need
in fact to use TAILQ_FOREACH_SAFE instead of TAILQ_FOREACH.

Reported-by: Luigi Sandon <luigi.sandon@gmail.com>
Fix-suggested-by: Luigi Sandon <luigi.sandon@gmail.com>
12 years ago
Ken Steele 6ebc20f6d8 Rework Tile CPU affinity setting to handle non-contiguous sets of CPUs.
It is possible to have a non-contiguous CPU set, which was not being
handled correctly on the TILE architecture.

Added a "rank" field in the ThreadVar to store the worker's rank separately
from the cpu for this case.
12 years ago
Mats Klepsland 1f3fbbc992 Fix bug #1206
PF_RING ZC uses clusters in the same way as PF_RING DNA. Therefore,
this bug can be fixed as it was fixed for DNA (bug #598).
12 years ago
Victor Julien 896b61452c htp: make htp state handling function more robust
Also, fix wrong cast that worked only by luck.
12 years ago
Victor Julien eff85aba5e http: remove BUG_ON(1) statement
Remove BUG_ON(1) statement that was a leftover from debugging.

Bug #1189
Bug #1212
12 years ago
Alessandro Guido 13448aca1c Fix issue #1214
When applying wildcard thresholds (with sid = 0 and/or gid = 0) it's wrong
to exit on the first signature already having an event filter. Indeed,
doing so results in the theshold not being applied to all subsequent
signatures. Change the code in order to skip signatures with event
filters instead of breaking out of the loop.
12 years ago
Victor Julien 9de536efdb Bug 1098: improve invalid pcre/R handling
When not using a file_data or similar 'sticky buffer', a pcre/R option
needs a content in the same buffer.
12 years ago
Giuseppe Longo 8db3f214f0 nflog: fix memory leaks
This fixes the following memory leaks:

[src/source-nflog.c:222]: (error) Memory leak: ntv
[src/source-nflog.c:236]: (error) Memory leak: ntv
[src/source-nflog.c:253]: (error) Memory leak: ntv
[src/source-nflog.c:258]: (error) Memory leak: ntv
12 years ago
Victor Julien cc54250cf9 Fix live reload segv when startup isn't complete
If a live reload signal was given before the engine was fully started
up (e.g. pcap file thread waiting for a disk to spin up), a segv could
occur.

This patch only enables live reloads after the threads have been
started up completely.
12 years ago
Victor Julien 2c20c9d409 Fix Coverity 1220098 and 1220099
*** CID 1220098:  Missing unlock  (LOCK)
/src/log-droplog.c: 195 in LogDropLogNetFilter()
189         SCMutexLock(&dlt->file_ctx->fp_mutex);
190
191         if (dlt->file_ctx->rotation_flag) {
192             dlt->file_ctx->rotation_flag  = 0;
193             if (SCConfLogReopen(dlt->file_ctx) != 0) {
194                 /* Rotation failed, error already logged. */
>>>     CID 1220098:  Missing unlock  (LOCK)
>>>     Returning without unlocking "dlt->file_ctx->fp_mutex".
195                 return TM_ECODE_FAILED;
196             }
197         }
198
199         if (dlt->file_ctx == NULL) {
200             return TM_ECODE_FAILED;

*** CID 1220099:  Dereference before null check  (REVERSE_INULL)
/src/log-droplog.c: 199 in LogDropLogNetFilter()
193             if (SCConfLogReopen(dlt->file_ctx) != 0) {
194                 /* Rotation failed, error already logged. */
195                 return TM_ECODE_FAILED;
196             }
197         }
198
>>>     CID 1220099:  Dereference before null check  (REVERSE_INULL)
>>>     Null-checking "dlt->file_ctx" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
199         if (dlt->file_ctx == NULL) {
200             return TM_ECODE_FAILED;
201         }
202
203         char srcip[46] = "";
204         char dstip[46] = "";
12 years ago
Victor Julien 8a77e6bc8e Fix Coverity 1220097
*** CID 1220097:  Missing unlock  (LOCK)
/src/log-file.c: 160 in LogFileWriteJsonRecord()
154             }
155         }
156
157         /* Bail early if no file pointer to write to (in the unlikely
158          * event file rotation failed. */
159         if (aft->file_ctx->fp == NULL) {
>>>     CID 1220097:  Missing unlock  (LOCK)
>>>     Returning without unlocking "aft->file_ctx->fp_mutex".
160             return;
161         }
162
163         FILE *fp = aft->file_ctx->fp;
164         char timebuf[64];
165         AppProto alproto = FlowGetAppProtocol(p->flow);
12 years ago
Jason Ish fc2014ab40 Unregister for file rotation notification when a context is
de-initialized.  Required for unix-socket mode where
contexts come and go.
12 years ago
Jason Ish e1b97fed70 Add signal based file rotation for:
- alert debug log
- fast log
- stats log
- dns log
- drop log
- file log
- http log
- tls log
- eve/json log
12 years ago
Jason Ish 0a33e73417 Add macros for access to the underlying buffer and offset.
Useful for using passing the buffer through to another writer
such as LogFileCtx.
12 years ago
Jason Ish c1b6894ce3 Add a rotation flag to LogFileCtx which loggers can use to register
for log rotation.  Have the LogFileCtx handle the log rotation.
12 years ago
Jason Ish 698a0f7f48 Registration for SIGHUP notification - for loggers interested
in file rotation on SIGHUP.
12 years ago
Victor Julien 25cbf36d40 lua/luajit: use HAVE_LUA mostly
Only use HAVE_LUAJIT if things are done differently from HAVE_LUA,
like in the states pool.
12 years ago
Victor Julien a7118a4ff3 profiling: use wider columns in keyword output
Use wider columns in keyword output so that even on high end sensors
the stats tables remain readable.
12 years ago
jeka dc1599e0dc bugfix in debug mode:
removed function calls from SCReturnX macros
12 years ago
Victor Julien 0765bcc73e nflog: set socket timeout
Set socket timeout so that we can exit if there is no traffic.

It would hang after the SIGINT signal, until packets arived.
12 years ago
Victor Julien 26c0915375 nflog: warn if buffer-size is larger than max-size
If buffer-size is larger than max size, give a warning and adjust
buffer-size to max-size.
12 years ago
Victor Julien 462f9de134 dns: unify type to string logging utility
Both DNS loggers had their own CreateTypeString. This patch unifies
them.
12 years ago
Victor Julien 5e87257845 dns: add names for common types
Add names for SRV, NAPTR, DS, RRSIG, NSEC, NSEC3 types.
12 years ago
Victor Julien 0bbec75764 nflog: fix typo rising->raising 12 years ago
Victor Julien 0857a60fce nflog: improve error handling on NOBUFS
Don't fall through to handle_packet on any NOBUFS condition. Make
sure we catch all NOBUFS.
12 years ago
Giuseppe Longo 4d72911e17 This patch adds the fields into PacketVars struct to setup a packet from a nflog message 12 years ago
Giuseppe Longo 4dda018ede Adds nflog option 12 years ago
Giuseppe Longo 0368d5e4a4 Declare a wrapper to parse group option for nflog 12 years ago
Giuseppe Longo c35432b265 Implements NFLOG runmode 12 years ago
Giuseppe Longo 2ad8a8e111 Bootstraping NFLOG capture mode 12 years ago
Giuseppe Longo 0162e7e809 Adds nflog error code 12 years ago
Giuseppe Longo d213d89981 Updating the Tmm Id for declaration of nflog capture mode 12 years ago
Victor Julien db563ed4b0 tls: check SSL3/TLS version per record
Set event if SSL3/TLS record isn't within the acceptable range.
12 years ago
Victor Julien 8ddcf6a816 dns: add tests for TXT response parsing
Add valid and invalid examples.
12 years ago
Victor Julien bddb2c3bdc dns json: log TXT response data
Log TXT data in the rdata field.
12 years ago
Victor Julien 683d2d64e9 dns: parse and store TXT responses
This way the TXT data can be logged by the loggers.

Ticket #1158
12 years ago
Victor Julien 8ba8c0bf6f json output: don't set 'unknown' for missing data
Instead of setting 'unknown' or '<unknown>' just pass NULL to json_*
function, which results in omitting the data.
12 years ago
Tom DeCanio 11ca25ddca eve-log: swap ip/port pairs in dns answers 12 years ago
Victor Julien d4215fca84 http-json: fix coverity warning
*** CID 1211009:  Bad bit shift operation  (BAD_SHIFT)
/src/output-json-http.c: 265 in JsonHttpLogJSON()
259         /* log custom fields if configured */
260         if (http_ctx->fields != 0)
261         {
262             HttpField f;
263             for (f = HTTP_FIELD_ACCEPT; f < HTTP_FIELD_SIZE; f++)
264             {
>>>     CID 1211009:  Bad bit shift operation  (BAD_SHIFT)
>>>     In expression "1 << f", left shifting by more than 31 bits has undefined behavior.  The shift amount, "f", is as much as 46.
265                 if ((http_ctx->fields & (1<<f)) != 0)
266                 {
267                     /* prevent logging a field twice if extended logging is
268                        enabled */
269                     if (((http_ctx->flags & LOG_HTTP_EXTENDED) == 0) ||
270                         ((http_ctx->flags & LOG_HTTP_EXTENDED) !=

________________________________________________________________________________________________________
*** CID 1211010:  Bad bit shift operation  (BAD_SHIFT)
/src/output-json-http.c: 492 in OutputHttpLogInitSub()
486                         {
487                             if ((strcmp(http_fields[f].config_field,
488                                        field->val) == 0) ||
489                                 (strcasecmp(http_fields[f].htp_field,
490                                             field->val) == 0))
491                             {
>>>     CID 1211010:  Bad bit shift operation  (BAD_SHIFT)
>>>     In expression "1 << f", left shifting by more than 31 bits has undefined behavior.  The shift amount, "f", is as much as 46.
492                                 http_ctx->fields |= (1<<f);
493                                 break;
494                             }
495                         }
496                     }
497                 }
12 years ago
Victor Julien 5cdd9b460a unix-socket: reset logging api's properly
Lack of proper reset lead to logs not being written after the first
pcap had been processed.
12 years ago
Victor Julien fd56acd4b3 stream: cleanup
StreamTcpSetDisableRawReassemblyFlag() has the same effect as
AppLayerParserTriggerRawStreamReassembly in that it will force the
raw reassembly to flush out asap. So it is redundant to call both.
12 years ago
Victor Julien 3543150f42 stream: implement raw reassembly stop api
Implement StreamTcpSetDisableRawReassemblyFlag() which stops raw
reassembly for _NEW_ segments in a stream direction.

It is used only by TLS/SSL now, to flag the streams as encrypted.
Existing segments will still be reassembled and inspected, while
new segments won't be. This allows for pattern based inspection
of the TLS handshake.

Like is the case with completely disabled 'raw' reassembly, the
logic is that the segments are flagged as completed for 'raw' right
away. So they are not considered in raw reassembly anymore.

As no new segments will be considered, the chunk limit check will
return true on the next call.
12 years ago
Victor Julien b2184f936e stream: unify segment discard handling
Have a single function StreamTcpReturnSegmentCheck determine if a
segment is ready to be removed from the stream.

Handle FLOW_NOPAYLOAD_INSPECT in raw reassembly.
12 years ago
Victor Julien ad355c3c0a app-layer: improve no payload inspect flag
If setting APP_LAYER_PARSER_NO_INSPECTION_PAYLOAD, trigger raw
reassembly.
12 years ago
Victor Julien f0bdb009ed tls/heartbleed: fix test
Now that we continue to track ssl/tls after the handshake, we need
to fix tests that checked for the cutoff flags.
12 years ago
Victor Julien 31655aef7e tls/heartbleed: improve encrypted logic
Don't assume that if the type field isn't 01 or 02 it's an encrypted
heartbeat. Instead, use our knowledge of the SSL state.
12 years ago
Victor Julien fdbd9b3f25 tls/heartbleed: formatting fixes 12 years ago
Will Metcalf 26169ad8c5 Look for Mismatched Encrypted HB request and response sizes, along with multiple in-flight HB requests from the same direction 12 years ago
Victor Julien 0564a8da3c detect: add more defensive checks for flow handling
Don't unconditionally deref f->alparser in detection through
DeStateFlowHasInspectableState(). In very rare cases it can
be NULL.
12 years ago
Victor Julien 2002067fb1 http-json: init 'fields' to 0 before setting it
httplog_ctx->fields would not be initialized before setting flags in
it:

Scanbuild:
output-json-http.c:491:46: warning: The left expression of the compound assignment is an uninitialized value. The computed value will also be garbage
                            http_ctx->fields |= (1<<f);
                            ~~~~~~~~~~~~~~~~ ^
1 warning generated.

Drmemory:
~~27874~~ Error #1: UNINITIALIZED READ: reading register eax
~~27874~~ # 0 JsonHttpLogJSON                       [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:260]
~~27874~~ # 1 JsonHttpLogger                        [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:375]

Just memset the whole structure right after initialition.
12 years ago
Tom DeCanio 7df9b283f1 json: address custom output capability to http eve log review comments 12 years ago
Tom DeCanio 4838b9bf4f json: add custom output capability to http eve log 12 years ago
Victor Julien 9abf595122 rohash: fix potential bad shift
Fix issue detected byCoverity:

*** CID 1197756:  Bad bit shift operation  (BAD_SHIFT)
/src/util-rohash.c: 74 in ROHashInit()
68         }
69         if (hash_bits < 4 || hash_bits > 32) {
70             SCLogError(SC_ERR_HASH_TABLE_INIT, "invalid hash_bits setting, valid range is 4-32");
71             return NULL;
72         }
73
>>>     CID 1197756:  Bad bit shift operation  (BAD_SHIFT)
>>>     In expression "1U << hash_bits", left shifting by more than 31 bits has undefined behavior.  The shift amount, "hash_bits", is as much as 32.
74         uint32_t size = hashsize(hash_bits) * sizeof(ROHashTableOffsets);
75
76         ROHashTable *table = SCMalloc(sizeof(ROHashTable) + size);
77         if (unlikely(table == NULL)) {
78             SCLogError(SC_ERR_HASH_TABLE_INIT, "failed to alloc memory");
79             return NULL;

This was only a potential issue as ROHashInit was only called with
hash_bits 16 in the code.

Bug #1170.
12 years ago
Eric Leblond 1992a22769 af-packet: exit in case of a fatal error
During socket creation all error cases were leading to suricata to
retry the opening of capture. This patch updates this behavior to
have fatal and recoverable error case. In case of a fatal error,
suricata is leaving cleanly.
12 years ago
Jason Ish d28879f1a2 DAG: Pull some raw values out into defines. 12 years ago
Jason Ish 8ab962df7c DAG: Consistent code style. 12 years ago
Jason Ish c00ec5f4fc DAG: Sync dag packet and drop counts to live device on exit for better exit
logging.
12 years ago
Victor Julien 4d6cc1dbc6 json-file: improve error handling
If the functions getting uri, etc fail. Return "unknown" instead of
NULL pointer. This improves consistency.
12 years ago
Victor Julien 2d25f12cda json-file: check pointer before using
A check was missing to see if ht_ud was not null before using the
pointer. This should be rare, but it can happen.

Reported-by: Will Metcalf
12 years ago
Victor Julien d8481cb7cd stream: improve retransmission handling
Improve retransmission handling in the CLOSE_WAIT state.

Bug #1180.
12 years ago
Victor Julien 4929c840cd stream: update GAP detection
Change GAP detection logic. If we encounter missing data before
last_ack, we know we have missed data. The receiving host has ack'd
it already, so a retransmission of the missing data is highly
unlikely.
12 years ago
Victor Julien 5db228b9f6 stream: fix raw reassembly flag issue
AppLayer reassembly correctly only flags a segment as done when it's
completely used in reassembly. Raw reassembly could flag a partially
used segment as complete as well. In this case the segment could be
discarded early. Further reassembly would miss data, leading to a
gap. Due to this, up to 'window size' bytes worth of segments could
remain in the session for a long time, leading to memory resource
pressure.

This patch sets the flag correctly.
12 years ago
Victor Julien 539bf57a65 stream: improve StreamTcpPruneSession
Check if a segment is done for the app-layer using
StreamTcpAppLayerSegmentProcessed instead of the flag directly so the
gap case works better.
12 years ago
Victor Julien 3fa818d087 stream: flags cleanup
Stream flags are 16bit, but notation is still 8bit. Clean this up to
avoid confusion.
12 years ago
Victor Julien 1bd189a076 protocol detection: handle very unbalanced case
Some traffic is very unbalanced. For example a 4 bytes request
followed by 12mb of response data. If the 4 bytes don't lead to
the protocol being detected, then app layer processing won't
start, but it will not give up either. It will just wait for more
data. This leads to piling up data on the opposite side.

Observed:

TS: 4 bytes. PP failed (bit set), PM has not given up (bit not set).
    This makes sense as max_depth is not yet reached.

TC: 12mb. PP and PM failed (bits set).

As ts-PM never gives up, we never consider proto detect complete,
so all segments in either direction are still kept in the session.

This patch adds a cutoff for this case:
- IF for TS we have PP bit set, PM not set, AND
- we have TC both bits set, AND
- proto is unknown, AND
- TC data is 100k already, THEN
- give up on protocol detection.

The same for the opposite direction.
12 years ago
Victor Julien ed46fd715d stream: improve midstream reassembly gap detection
The reassembly gap detection makes use of the window. However, in
midstream mode the window size we use is unreliable, as we have to
assume window scaling is in place. This patch improves midstream
handling of those cases.
12 years ago
Victor Julien 165f129c61 stream: detect data gap at stream start
In midstream mode we may encounter a case where the data we is beyond
the isn, but below last_ack. This means we're missing some data, that
is already acked so it won't be retransmitted. Therefore, we can
conclude it's a data gap.
12 years ago
Victor Julien 32aafa6a48 proto detect: add cutoff for unbalanced traffic
If we're getting a lot of data in one direction and the proto for this
direction is unknown, proto detect will hold up segments in the segment
list in the stream. They are held so that if we detect the protocol on
the opposing stream, we can still parse this side of the stream as well.
However, some sessions are very unbalanced. FTP data channels, large
PUT/POST request and many others, can lead to cases where we would have
to store many megabytes worth of segments before we see the opposing
stream. This leads to risks of resource starvation.

In this patch, a cutoff point is enforced. If we've stored 100k in one
direction and we've seen no data in the other direction, we give up.

If we've given up, the applayer_proto_detection_skipped event is set.
    app-layer-event: applayer_proto_detection_skipped;
12 years ago
Victor Julien 44b8be3732 protocol detection: midstream handling update
If a TCP session is midstream, we may end up with a case where the
start of an HTTP request is missing. We won't detect HTTP based on
the request. However, the reply is fine, so we detect HTTP anyway.
This leads to passing the incomplete request to the htp parser.

This has been observed, where the http parser then saw many bogus
requests in the incomplete data. This is not limited to HTTP.

To counter this case, a midstream session MUST find it's protocol
in the toserver direction. If not, we assume the start of the
request/toserver is incomplete and no reliable detection and parsing
is possible. So we give up.
12 years ago
Victor Julien c1dcc7a268 stream: fix midstream syn/ack setup
If midstream is enabled and the first packet is the syn/ack packet from
the 3whs, initialized server.last_ack to the packets seq.

This fixes tracking the session.
12 years ago
Victor Julien 59ccd7b62d byte-test: switch to pcre_copy_substring
Fixes cppcheck:
[src/detect-bytejump.c:360]: (error) Uninitialized variable: str_ptr
[src/detect-bytejump.c:372]: (error) Uninitialized variable: str_ptr
[src/detect-bytejump.c:375]: (error) Uninitialized variable: str_ptr

Bug #1046.
12 years ago
Victor Julien 1de9256b18 byte-extract: switch to pcre_copy_substring
Fixes memory leaks (str_ptr was never freed).
Fixes cppcheck warning:
    [src/detect-byte-extract.c:268]: (error) Uninitialized variable: str_ptr

Bug #1046.
12 years ago
Victor Julien a25f532ce7 dns: improve response name parsing
Improve parsing of names with multiple pointers following each other.
12 years ago
Eric Leblond 18c7eeefca packet handling: fix release function
Extended data were freed before the release function was called.
The result was that, in AF_PACKET IPS mode, the release function
was only sending void data because it the content of the extended
data is the content of the packet.

This patch updates the code to have the freeing of extended data
done in the cleaning function for a packet which is called by the
release function. This improves consistency of the code and fixes
the bug.
12 years ago
Victor Julien 3df904475c proto-detect: masks cleanup
The direction specific masks were not used correctly. The toserver ones
were only used for 'dp' registrations, the toclient ones only for 'sp'.

The patch merges them.
12 years ago
Victor Julien 8252416c10 proto-detect: update port logic
If a flow matches both an 'sp' based PP registration and a 'dp' based,
until now we would only check the 'dp' one. This patch changes that. It
will inspect both.
12 years ago
Victor Julien eae5b1ba35 app-layer: proto detection update
Instead of the notion of toserver and toclient protocol detection, use
destination port and source port.

Independent of the data direction, the flow's port settings will be used
to find the correct probing parser, where we first try the dest port,
and if that fails the source port.

Update the configuration file format, where toserver is replaced by 'dp'
and toclient by 'sp'. Toserver is intrepreted as 'dp' and toclient as
'sp' for backwards compatibility.

Example for dns:

    dns:
      # memcaps. Globally and per flow/state.
      #global-memcap: 16mb
      #state-memcap: 512kb

      # How many unreplied DNS requests are considered a flood.
      # If the limit is reached, app-layer-event:dns.flooded; will match.
      #request-flood: 500

      tcp:
        enabled: yes
        detection-ports:
          dp: 53
      udp:
        enabled: yes
        detection-ports:
          dp: 53

Like before, progress of protocol detection is tracked per flow direction.

Bug #1142.
12 years ago
Victor Julien 7b0f1e9512 stream: improve retransmission handling
When connection are closing, don't reject retransmissions of data
packets.

Bug #1180.
12 years ago
Victor Julien 96adcf6829 refactor IDS/IPS engine mode logic
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.

- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode

To set the modes:

- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode

Bug #1177.
12 years ago
Ken Steele 354a24e2ef Fix unaligned load in AC-TILE MPM.
The SLOAD define using __insn_ld2s_L2 is used to provide a compiler
hint that the load will come from the L2 cache instead of the L1. It
also specifies that it is a 2 byte signed load. For the Tiny MPM, that
needs to be a 1-byte load, which is what is specified in util-ac-mpm-tile.c,
but the #undef was removing that definition.
12 years ago
Victor Julien fc559ce227 detect: fix alstate handling
Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.

Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
12 years ago
Victor Julien b6e2a6f525 detect: locking update continued
Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
12 years ago
Victor Julien cf31e2cc74 detect: locking update
Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
12 years ago
Victor Julien 5e1bc99e5b detect: cleanup
Remove unused alstate and app layer flags arguments from
DetectEngineInspectPacketPayload()
12 years ago
Victor Julien 6e0112d737 detect: modify AMATCH locking
This is an intrusive change. This patch modifies the way AMATCH
inspection uses locking.

So far, each keyword did it's own locking. This lead to a situation
where a 'alstate' pointer was passed around that was not always
protected by a lock.

This patch moves the locking to the Stateful detection functions.
12 years ago
Eric Leblond 43b6cbd4bc af-packet: fix error handling
Only exit from synchronization loop on poll error and not in case
of a timeout.
12 years ago
Eric Leblond 6e77c4d1b4 util-ioctl: only get MTU when iface name is set
This patch fixes a warning message when suricata is started without
giving an interface name on the command line. The code was trying
to get the MTU even if pcap_dev was not set.
12 years ago
Victor Julien 79c924af8c Fix 2 compiler warnings
FreeBSD 10 32-bit with clang 3.3:

log-tlslog.c:172:14: error: format specifies type 'long' but the argument has type 'time_t' (aka 'int') [-Werror,-Wformat]
             p->ts.tv_sec,
             ^~~~~~~~~~~~
1 error generated.

detect-engine-payload.c:508:27: warning: format specifies type 'long' but the argument has type 'time_t' (aka 'int') [-Wformat]
    printf("%ld.%06ld\n", tv_diff.tv_sec, (long int)tv_diff.tv_usec);
            ~~~           ^~~~~~~~~~~~~~
            %d
1 warning generated.
12 years ago
Victor Julien 26778b8703 output-api: cleanup handling
Add output 'free list' that contains all the output ctx' that need
cleanup at shutdown. It differs from the runmode output list in that
it will also contain a 'parent' for the submodules that share the
context of it's parent.
12 years ago
Victor Julien c27304451e output api: complete shutdown functions
Add missing function for Filedata API. Clean up list in all functions.
12 years ago
Victor Julien b2d29a85e9 tls-json: add cleanup function
Properly clean up output context when shutting down.
12 years ago
Victor Julien 34069054ce drop-json: fix cleanup
Use proper function for sub-module cleanup. LogFileCtx is not managed
by the sub-module, so don't clean it.
12 years ago
Victor Julien bc1c06b9e4 eve-log: fix mem leak at shutdown
Make sure we free all memory in the shutdown function.
12 years ago
Victor Julien 90c4834709 ssh-json: add clean up functions
Add clean up functions for the SSH json logger.
12 years ago
Victor Julien 3a6be9772f http-json: add missing cleanup functions
Add cleanup functions.
12 years ago
Victor Julien 7ffd227133 file-json: cleanup at shutdown
Fix a memory leak at shutdown. Module didn't have a cleanup function.
12 years ago
Victor Julien 7ee3b456a3 dns-json: fix cleanup
Use specialized cleanup function for sub-module case. Freeing the
LogFileCtx is not the responsibility of a sub-module.
12 years ago
Victor Julien 1f2310bb34 alert-json: fix cleanup
Call specialized clean up function when running as a sub-module.
12 years ago
Victor Julien f62185c207 log-tls: run Disable at shutdown
Call OutputTlsLoggerDisable at cleanup.
12 years ago
Victor Julien f96a54535c drop loggers: call disable func
Call OutputDropLoggerDisable() on cleanup.
12 years ago
Victor Julien 9df045d086 output: add Disable funcs to mirror Enable
For the loggers that we allow only one instance for: tls, ssh, drop, we
track active loggers through Output*Enable functions. Add Disable
functions to mirror this. They are to be called from the shutdown funcs
those loggers use.
12 years ago
Victor Julien 4a104ae315 unified2: fix memory leak at shutdown
Module didn't properly free output context at shutdown. Led to a leak
in Unix Socket mode.
12 years ago
Eric Leblond 9961520316 output: clean file desc at exit.
This is a beginning of implementation for bug #1660:
 https://redmine.openinfosecfoundation.org/issues/1160

This patch adds a cleaning function for each logger of new type
(packet, tx and file). These functions are called in RunModeShutDown().

The state of this patch is that it is crashing suricata when sending
pcap to analyse:
 - At first pcap if tx and file cleaning function are called
 - At second pcap if only packet cleaning function is called

The cause in first case is unknown. In second case this is due to
the necessity of cleaning the list of logger registered to a logging
type.
12 years ago
Pierre Chifflier d476c654ee TLS: add detection for malicious heartbeats (AKA heartbleed)
The OpenSSL implementation of RFC 6520 (Heartbeat extension) does not
check the payload length correctly, resulting in a copy of at most 64k
of memory from the server (ref: CVE-2014-0160).
This patch adds support for decoding heartbeat messages (if not
encrypted), and checking several parts (type, length and padding).
When an anomaly is detected, a TLS event is raised.
12 years ago
Victor Julien ab503873ca erf-file: clean up decode thread local storage
Clean up the thread local data the decode portion of ERF users.

Bug #978
12 years ago
Victor Julien 6da8652a77 endace-dag: clean up decode thread local storage
Clean up the thread local data the decode portion of DAG uses.

Bug #978
12 years ago
Victor Julien 09ebbe08df mpipe: clean up decode thread local storage
Clean up the thread local data the decode portion of mpipe uses.

Bug #978
12 years ago
Victor Julien 8c16fede08 ipfw: clean up decode thread local storage
Clean up the thread local data the decode portion of ipfw uses.

Bug #978
12 years ago
Victor Julien 10c791c937 napatech: clean up decode thread local storage
Clean up the thread local data the decode portion of napatech uses.

Bug #978
12 years ago
Victor Julien c3e193e786 pcap: clean up decode thread local storage
Clean up the thread local data the decode portion of pcap uses.

Bug #978
12 years ago
Victor Julien 900fc6fdc7 pfring: clean up decode thread local storage
Clean up the thread local data the decode portion of pfring uses.

Bug #978
12 years ago
Victor Julien a8b1af3369 nfq: clean up decode thread local storage
Clean up the thread local data the decode portion of nfq uses.

Bug #978
12 years ago
Victor Julien 2864f9eef9 af-packet: clean up decode thread local storage
Clean up the thread local data of the decode part of afpacket.

Bug #978
12 years ago
Victor Julien bb2e9af40f pcap-file: clean up decode thread local storage
Clean up the thread local data the decode portion of pcap-file use.

Bug #978.
12 years ago
Victor Julien d26ceb2356 decode: introduce DecodeThreadVarsFree
As a mirror of DecodeThreadVarsAlloc, DecodeThreadVarsFree is used
to free the memory that DecodeThreadVarsAlloc alloc'd, including
AppLayer storage.
12 years ago
Victor Julien 552558894c app-layer: cleanups
Clean up AppLayerParserThreadCtxAlloc and AppLayerParserThreadCtxFree.
Both used confusing variables in loops, with the wrong types.
12 years ago
Jason Ish 7e268bd4d4 Force pidfile creation of --pidfile.
A pidfile can be useful when not in daemon mode, for example
when running under a process supervisor.
12 years ago
Tom DeCanio 4085f08602 json: somewhere along the way IP/port pairs had gotten swapped in
http and ssh eve logs
12 years ago
Victor Julien b9227ad20c tls: no event on 'new session ticket' in handshake
Don't set an event on encountering a 'new session ticket' (4) record
in the TLS handshake.
12 years ago
Victor Julien 1195f882b9 ipv6: add support for PAD1
Support PAD1 in IPv6 HOP options header and DST options header.
12 years ago
Victor Julien 7539372db7 icmpv6: add multicast types
Only add them to check if the code is 0 and to make sure the default
case doesn't set an 'unknown type' event.
12 years ago
Victor Julien eb3a9d3076 TLS: register patterns for tls-alerts
Register patterns for when server has an alert as the first message.
12 years ago
Victor Julien a96446d39e detect state: fix indent
AMATCH block was indented too far.
12 years ago
Victor Julien 6c8ca76942 detect state: remove alproto check for AMATCH
Not all AMATCHes set a alproto.
12 years ago
Victor Julien c9436a6aef Fix app-layer-protocol FP on multi TX flow
In case of multiple transactions, the stored AMATCH list would not have
been reset, but it would still be reconsidered. Even though none would
match, the engine would still conclude that the rule matched.
12 years ago
Eric Leblond 9e03550230 tls: fix problem with tls.store keyword
Pierre Chifflier pointed out that a rule like:
 alert tls any any -> any any (msg:"TLS store"; tls.issuerdn:!"C=FR"; tls.store;)
was alerting but not storing the certificate. If the filter was
removed:
 alert tls any any -> any any (msg:"TLS store"; tls.store;)
then tls.store is working as expected.

This was linked with fact that logging is only done once for a SSL
state. So without filter, once we have the info we can log and we
run the storage. But when there is a filter, we log and then there
is a filter analysis and alerting. And as logging as already be done
we don't enter in the logging function and there is no storage.

This patch forces the entrance in the log function when there is a
request for TLS storage. And it adds an exit in the logging function
to only do the storage part if the TLS state has already being logged.
12 years ago
Victor Julien 806844d852 af-packet: fix init sync with no traffic
Previously the sync code would depend on traffic to complete. This
patch adds poll support and can complete the setup if the poll timeout
is reached as well.

Part of bug #1130.
12 years ago
Eric Leblond 238ff23111 af-packet: move packet fanout code
The sooner is the better for that caode as these means we will get
all sockets binded to fanout group as fast as possible.
12 years ago
Eric Leblond 919377d4a5 af-packet: synchronize reading start
This patch is updating af-packet to discard packets that have been
sent to a socket before all socket in a fanout group have been setup.
Without this, there is no way to assure that all packets for a single
flow will be treated by the same thread.

Tests have been done on a system with an ixgbe network card. When using
'cluster_flow' load balancing and disactivating receive hash on the iface:
 ethtool -K IFACE rxhash off
then suricata is behaving as expected and all packets for a single flow
are treated by the same thread.

For some unknown reason, this is not the case when using cluster_cpu. It
seems that in that case the load balancing is not perfect on the card side.

The rxhash offloading has a direct impact on the cluster_flow load balancing
because load balancing is done by using a generic hash key attached to
each skb. This hash can be computed by the network card or can be
computed by the kernel. In the xase of a ixgbe network card, it seems there
is some issue with the hash key for TCP. This explains why it is necessary to
remove the rxhash offloading to have a correct behavior. This could also
explain why cluster_cpu is currently failing because the card is using the
same hash key computation to do the RSS queues load balancing.
12 years ago
Victor Julien 70efc66e33 fast-log: restore logging of Drop/wDrop
Restore drop print logic. Probably got lost in large refactoring round
introducing log APIs.

Bug #1138.
12 years ago
Victor Julien 1af39d55cf detect: allow alias registration for rule keywords
This allows for registering a keyword under another name while keeping
the old name active and supported.

Do this for 'luajit', which can now also be used as just 'lua'.
12 years ago
Victor Julien 010a334e1a json alert: fix action
It would give 'Pass' as an action when the rule is set to 'alert'.
12 years ago
Victor Julien e04b5f0dca eve-http: register with app-layer api
The HTTP module of Eve didn't register itself with the app-layer
for HTTP. This meant that if no other HTTP logger was active, the
HTTP logging in Eve wouldn't work.

This patch makes the HTTP Eve module register itself correctly.

Bug #1133.
12 years ago
Victor Julien 5d96ea570f eve-file: set event_type to fileinfo
To remain constistent with the other logs, set the event type to
the same name as the structure containing the defails. In this
case fileinfo.

Part of bug #1127.
12 years ago
Victor Julien 305da0248d eve-files: file -> fileinfo
Due to what appears to be an issue in logstash, the 'file' part of
the file event types was masked by a field that logstash-forwarder
added itself.

Since logstash-forwarder is an important part of the logstash stack,
this patch works around the issue by renaming our 'file' structure
to 'fileinfo', thus resolving the naming conflict.

Bug #1127
12 years ago
Victor Julien 3e7714aca2 eve-http: print <unknown> like in eve-files
When UA or Host are unknown, print <unknown> instead of <useragent
unknown> or <hostname unknown>.

Bug #1131.
12 years ago
Victor Julien c5041d35d5 Fix live reload
Fix memsets clearing out of bounds memory on live reload, causing
crashes and corrupted backtraces.

Bug #1128.
12 years ago
Victor Julien fdb1bd9668 pfring: call enable_ring after set_cluster
Move pfring_enable_ring to the start of ReceivePfringLoop() so that
it's guaranteed to be called after all threads have called
pfring_set_cluster first.

This is necessary because pfring will already make packets available
to thread N, while thread N+1 is still registering itself. This leads
to cases where the first packet(s) of a flow are processed by a
different thread in Suricata than the later ones.

This is a race condition only at start up. New flows after the pfring
initialization is complete will not be influenced by this.

Bug #1129.
12 years ago
Victor Julien a3020b5306 eve-log: output cleanup
Suppress debug messages and print in a nicer way which modules are
being enabled.
12 years ago
Victor Julien ed877c64d1 Bug 611: fix for iponly
Fix Bug 611 for ip-only rules as well. If 'alert ip' rule has ports,
don't match on protocols that don't have ports. Like ICMP.

Bug #611.
12 years ago
Eric Leblond 6c3c234ca5 output-json: update timestamp format
This patch updates the timestamp format used in eve loggin.
It uses a ISO 8601 comptatible string. This allow tools parsing
the output to easily detect adn/or use the timestamp.

In the EVE JSON output, the value of the timestamp key has been
changed to 'timestamp' (instead of 'time'). This allows tools
like Splunk to detect the timestamp and use it without configuration.

Logstash configuration is simple:

input {
   file {
      path => [ "/usr/local/var/log/suricata/eve.json" ]
      codec =>   json
      type => "suricata-log"
   }
}

filter {
   if [type] == "suricata-log" {
      date {
        match => [ "timestamp", "ISO8601" ]
      }
   }
}

In splunk, auto detection of the fle format is failling and it seems
you need to define a type to parse JSON in
$SPLUNK_DIR/etc/system/local/props.conf:

[suricata]
KV_MODE = json
NO_BINARY_CHECK = 1
TRUNCATE = 0

Then you can simply declare the log file in
$SPLUNK_DIR/etc/system/local/inputs.conf:

[monitor:///usr/local/var/log/suricata/eve.json]
sourcetype = suricata

In both cases the timestamp are correctly imported by
the tools.
12 years ago
Eric Leblond 1fa4233d67 pfring: get vlan id from header
PF_RING is delivering the packet with VLAN header stripped. This
patch updates the code to get the information from PF_RING extended
header information.

This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.
12 years ago
Eric Leblond cf30adcedc ipfw: fix build
Buildbot reported:
 runmode-ipfw.c: In function 'RunModeIpsIPFWAuto':
 runmode-ipfw.c:85: error: implicit declaration of function 'LiveDeviceHasNoStats'
12 years ago
Eric Leblond 606e19124b http: add info message about memcap
Display a message about http memcap when it is set in config file.
12 years ago
Victor Julien f00703acc1 Fix False Positive of rules with ports on portless protocols
In case of 'alert ip' rules that have ports, the port checks would
be bypassed for non-port protocols, such as ICMP. This would lead to
a rule matching: a false positive.

This patch adds a check. If the rule has a port setting other than
'any' and the protocol is not TCP, UDP or SCTP, then we rule won't
match.

Rules with 'alert ip' and ports are rare, so the impact should be
minimal.

Bug #611.
12 years ago
Victor Julien 00d2f2d627 Fix BytesToString indexing array using wrong index
This would lead to reading past the end of the buffer and also writing
past the end of the newly allocated buffer.

Bug #1121
12 years ago
Victor Julien 88e9c85e36 json output: fix vlan byte order in output
VLAN functions/macros return vlan id in host byte order, so no need
to convert them in output functions.
12 years ago
Victor Julien 872bb5664e Fix null dereference in eve-log
Eve-log would call GET_VLAN_ID on the packets vlan header if p->vlan_idx
was bigger than 0. GET_VLAN_ID would then unconditionally dereference
p->vlanh[0] or [1]. However, there are a number of cases in which these
pointers are not set. Defrag pseudo packets, AF_PACKET and in the future
PF_RING, do set the id's, but not the header pointers.

This patch adds 2 new macro's which are wrappers around a function:

VLAN_GET_ID1 and VLAN_GET_ID2 get the id's by calling DecodeVLANGetId.

This function will return the correct id.

Bug #1120.
12 years ago
Victor Julien 684d787567 Coverity fix
Coverity 992695, fix potential array index with negative int. Very unlikely
case at rule keyword parsing stage.
12 years ago
Victor Julien f6bb867df8 ssh: fix scan-build warnings
app-layer-ssh.c:165:5: warning: Value stored to 'input_len' is never read
    input_len -= 1;
    ^            ~
1 warning generated.

app-layer-ssh.c:160:5: warning: Value stored to 'input_len' is never read
    input_len -= 4;
    ^            ~
1 warning generated.
12 years ago
Victor Julien 0967f0777c ssh: improve banner checking
Don't use input_len as banner length. Instead, look for banner end
to calculate banner length.

Add test for banner buffering corner case.
12 years ago
Victor Julien 669b351dad ssh: fixes for minor scan-build warnings
/usr/share/clang/scan-build/ccc-analyzer -DHAVE_CONFIG_H -I. -I..   -I./../libhtp/  -I/usr/include/nspr   -I/usr/include/nss -I/usr/include/nspr   -DLOCAL_STATE_DIR=\"/usr/local/var\" -g -O2 -Wall -Wno-unused-parameter -std=gnu99 -DHAVE_LIBNET11 -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -DHAVE_LIBNET_ICMPV6_UNREACH  -I/usr/include -DLIBPCAP_VERSION_MAJOR=1 -DHAVE_PCAP_SET_BUFF -DHAVE_LIBCAP_NG -DREVISION="51e0dee" -MT app-layer-ssh.o -MD -MP -MF .deps/app-layer-ssh.Tpo -c -o app-layer-ssh.o app-layer-ssh.c
app-layer-ssh.c:164:5: warning: Value stored to 'input' is never read
    input += 1;
    ^        ~
app-layer-ssh.c:165:5: warning: Value stored to 'input_len' is never read
    input_len -= 1;
    ^            ~
app-layer-ssh.c:212:13: warning: Value stored to 'ret' is never read
            ret = 0;
            ^     ~
3 warnings generated.
12 years ago
Victor Julien b877cf6158 ssh: add json logger
Sub module of eve-log, but can also run separately as ssh-json-log. Only
one at a time though.
12 years ago
Victor Julien 65b228ccfd ssh: improve large and fragmented banner handling
Including tests.
12 years ago
Victor Julien b4aeb43af1 ssh: disable inspection in encrypted phase
When both sides of the session have completed the encryption setup,
flag the stream to disable detection.
12 years ago
Victor Julien 294ff49f6d ssh: allow for space characters in the software version
Previously the software version would only contain up to the first
space.

E.g. in SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu3

It would contain "OpenSSH_4.7p1".

This patch changes the behavior to:

"OpenSSH_4.7p1 Debian-8ubuntu3"
12 years ago
Victor Julien b792234dd0 ssh: clean up flags 12 years ago
Victor Julien b5afe2b51f ssh: reenable ssh.protoversion keyword 12 years ago
Victor Julien 6c0162bf26 ssh: reenable ssh.softwareversion keyword 12 years ago
Victor Julien 884cecd9af ssh: handle fragmented banner
Cleanups.
12 years ago
Victor Julien 32fcdfe6eb ssh: server support, cleanups 12 years ago
Victor Julien 3648adb533 ssh: record parser 12 years ago
Victor Julien a3c9832b90 ssh: reenable parser as stub
Reenable the SSH parser. It now compiles, however the actual parsing
code is still disabled (commented out).
12 years ago
Eric Leblond 79de8c8f4b runmode: remove unused variable.
default_mode_auto is not used anymore and can be removed.
12 years ago
Eric Leblond a6bb86a9e0 Exit if BPF filter file is specified in IPS mode
This patch adds a check that was missing when specifying BPF filter
from a file. Suricata behavior should have been the same as when
BPF filter is specified on command line.
12 years ago
Eric Leblond 941cfe1641 ipfw: fix operator error in test
Fix warning spotted by clang on FreeBSD:

source-ipfw.c:241:49: warning: use of logical '||' with constant operand [-Wconstant-logical-operand]
        if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
                                                ^  ~~~~~~~~~~~~~
source-ipfw.c:241:49: note: use '|' for a bitwise operation
        if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
                                                ^~
                                                |

Use same logic as the one used in other capture mode.
12 years ago
Eric Leblond 9f6527dc16 ipfw: improve exit message
This patch synchronizes the exit message with what is done in
NFQ capture mode.
12 years ago
Eric Leblond 1c48a81f91 ipfw: update running modes to hide device stats
This patch adds call to the function used to disable the display
of live device stats at exit.
12 years ago
Eric Leblond d8a305356e nfq: update message displayed at exit
This patch updates the message displayed at exit to have something
more readable.
12 years ago
Eric Leblond 00c8408c55 nfq: update running modes to hide device stats
This patch adds call to the function used to disable the display
of live device stats at exit.
12 years ago
Eric Leblond 6b2ca63d9d util-device: add function to avoid stat display
In the case of running mode like NFQ there is no need possibility
to compute the statistics as it is done in LiveDevice (drop and
checksum count are meaningless).

This patch adds a function that allow running mode to disable the
display of the counters at exit.
12 years ago
Eric Leblond a12c46c700 util-device: fix debug message
Reference to Pcap was not correct.
12 years ago