Commit Graph

453 Commits (10104066e17a59769467cc5350eac59ee37404f1)

Author SHA1 Message Date
Tom DeCanio a3d86594dc address most initial JSON pull request comments 12 years ago
Tom DeCanio a44b2b987b JSON output cleanup 12 years ago
Tom DeCanio 3241732e27 rename alert-json.[ch] output-json.[ch] 12 years ago
Tom DeCanio 0df6af3a0b Alert/HTTP/DNS JSON output working with Logstash 12 years ago
Tom DeCanio 5498654114 Add JSON formatted alert output 12 years ago
Victor Julien 9ff6608668 Introduce Filedata Logger API
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.

The logger call in the API is as follows:
    typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
        const Packet *, const File *, const FileData *, uint8_t flags);

All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.

Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
12 years ago
Victor Julien ee2a8a9cda Introduce 'file' logging API
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.

Users of this API register their Log Function through:
    OutputRegisterFileModule()

The API uses a magic settings globally. This might be changed later.
12 years ago
Victor Julien ec20f45916 alert-syslog: cleanup
Remove separate ipv4 and ipv6 registration functions.
Make all functions static.
Move registration function to the bottom.
Simplify OS_WIN32 wrappers usage.
12 years ago
Victor Julien bcf5c1f2fb log-tls: clean ups
Make all functions static. Remove separate ipv4 and ipv6 registration
functions. Move register function to the bottom so that we no longer
need function prototype declarations.
12 years ago
Victor Julien 15eb4b292d TX logging API: convert HTTP log
Convert the HTTP log to the new TX logging API.
12 years ago
Victor Julien ad70793f78 Introduce TX logging API
This patch introduces a new API for logging transactions from
tx-aware app layer protocols. It runs all the registered loggers
from a single thread module. This thread module takes care of the
transaction handling and flow locking. The logger just gets a
transaction to log out.

All loggers for a protocol will be run at the same time, so there
will not be any timing differences.

Loggers will no longer act as Thread Modules in the strictest sense.
The Func is NULL, and SetupOuputs no longer attaches them to the
thread module chain individually. Instead, after registering through
OutputRegisterTxModule, the setup data is used in the single logging
module.

The logger (LogFunc) is called for each transaction once, at the end
of the transaction.
12 years ago
Victor Julien d43ac9ae98 Introduce packet logging output API
This patch introduces a new API for outputs that log based on the
packet, such as alert outputs. In converts fast-log to the new API.

The API gets rid of the concept of each logger being a thread module,
but instead there is one thread module that runs all packet loggers.
Through the registration function OutputRegisterPacketModule a log
module can register itself to be considered for each packet.

Each logger registers itself to this new API with 2 functions and the
OutputCtx object that was already used in the old implementation.
The function pointers are:

LogFunc:       the log function

ConditionFunc: this function is called before the LogFunc and only
               if this returns TRUE the LogFunc is called.

For a simple alert logger like fast-log, the condition function will
simply return TRUE if p->alerts.cnt > 0.
12 years ago
Victor Julien f4872a2f08 Add --disable-detection configure option
Add --disable-detection configure option to compile Suricata with
detection disabled.
12 years ago
Victor Julien 6a3621f2fe Error out if -s/-S are used with disable detect
When Suricata is started with --disabled-detection, the -s and -S
options make no sense. So error out.
12 years ago
Victor Julien 3e2205d08f detect-less: disable raw reassembly
When using --disable-detection automatically disable raw stream
reassembly by forcing stream.reassembly.raw to be false.
12 years ago
Victor Julien 5cc880c5c1 detect-less: add log only TX handling function
When running w/o detect, TX cleanup handling needs to ignore the
inspect_id as it's only updated by detect.

This patch introduces a new ActiveTx handler for logging only:
AppLayerTransactionGetActiveLogOnly

If --disable-detection is passed on the commandline, this function
is registered.
12 years ago
Victor Julien 50f5b246db Update main loop to handle detect-less
Update main startup and shutdown loops to work will when detect
is disabled. In this case de_ctx remains NULL.
12 years ago
Victor Julien 258778cde4 Introduce g_detect_disabled global
This global will be set to TRUE if detect is disabled. The reason for
adding a global is that there currently is no clean way to pass
configuration options to management threads.
12 years ago
Victor Julien b44ec80590 Add --disable-detection commandline option
Flags the SuriInstance that the detection engine should be disabled.
Actual disabling is not yet implemented.
12 years ago
Jason Ish 5850607fea Remove the old include support. 12 years ago
Eric Leblond a8c787a265 device list: clean and display stat at exit
This patch adds a cleaning function to device list. This also
permits to display per-interface statistics during the exit.
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Eric Leblond 58eb6428d1 suricata: ignore SIGHUP signal
This patch ignores the SIGHUP signal instead of having the default
behavior.
12 years ago
Jason Ish e9a4871077 Fix alignment in usage. 12 years ago
Victor Julien 0a24ac0855 Fix Conf api usage after rebase 12 years ago
Jason Ish b8e13d4bd6 More concise API for setting config values that
can be overrided or not (final values).
12 years ago
Eric Leblond 1bdc39fe9b cmdline: add -k to specify checksum validation
This patch adds a '-k' option to suricata to be able to specify
the checksum validation to use. If '-k all' is used, checksum
validation is forced. If '-k none' is used, no checksum validation
is made.

Message output in case of detection of a pcap file with a probable
cheksum issue has been updated to indicate that '-k' is a solution.
12 years ago
Victor Julien 480fddd189 build-info: add a nicer way of printing atomics support 12 years ago
Eric Leblond 3dceca70ee suricata: move some code into PostConfLoadedSetup
All functions before daemonization are initialisation functions and thus the
call can be moved in PostConfLoadedSetup.
12 years ago
Victor Julien a84c502e50 Add SSE support to --build-info 12 years ago
Victor Julien 25636597af Fix live rule reload confusing delayed detect
Fixes bug 1023 and the previous attempt to fix it.
12 years ago
Victor Julien 5906eeb8c8 detect: don't do rule reload during delayed detect
When both rule reloads and delayed detect are enabled, make sure we don't
trigger a reload during delayed detect initialization.

Bug #1023.
12 years ago
Victor Julien 97bfcac444 profiling: introduce per keyword profiling
Initial version of per keyword profiling. Prints stats about
how ofter a keyword was checked and what the costs were.
12 years ago
Victor Julien 7ebd1e6433 Counters: fix delayed-detect counter registration
Make sure we register the detect.alerts counter before packet runtime starts
even in delayed detect mode. The registration of new counters at packet
runtime is not supported by the counters api and might lead to crashes as there
is no proper locking to allow for this operation.

This changes how delayed detect works a bit. Now we call the ThreadInit
callback twice. The first call will only register the counter. The 2nd call
will do all the other setup. This way the counter is registered before the
counters api starts operating in the packet runtime.

Fixes the segv reported in ticket #1018.
12 years ago
Anoop Saldanha 619414c59e Add a /* fall through */ comment for all switch case fall throughs.
This should server as a message to coverity that the fall through is
intentional.
12 years ago
Victor Julien 37669bfdd2 threshold: register threshold host storage. Related to bug #991 12 years ago
Victor Julien 74d8d95f83 Don't initialize threshold before rules on delayed detect. Bug #999. 12 years ago
Eric Leblond 2be194d03f suricata: add -v[v] option to increase verbosity
This patch adds a -v option to suricata. It increases the log level
defined in the YAML.
12 years ago
Eric Leblond 4a4600539d suricata: info message after log init
This patch moves version display after log init so we can have an
homogeneous display.
12 years ago
Eric Leblond fdc1757e34 suricata: reorder start
Initalizing output just after configuration file parsing allow to
log almost all messages accordingly to configuration.
12 years ago
Eric Leblond 7bcacc712a log: change default log level to notice
This patch updates the log level of meaningful start messages to
notice. It also sets the default log level to notice.
12 years ago
Victor Julien 8d6bca72f7 Improve 'host-mode' info message 12 years ago
Eric Leblond 6cf7da30e2 Introduce host-mode.
This variable can be used to indicate to suricata that the host
running is running as a router or is in sniffing only mode.
This will used at least to determine which interfaces are used to
send reject message.
12 years ago
Victor Julien 468a8e1ca3 Properly cleanup NSS ctx 12 years ago
Victor Julien eedd4329da Change ParseSize api to not leak memory and only setup pcre once. 12 years ago
Victor Julien 397a55457d Add sanity checks for command line argument handling
Coverity 1075221.

Normally getopt_long should cover this case, but can't hurt to
add in some extra checks.
12 years ago
Victor Julien 38aaae1fd7 IsRuleReloadSet() shouldn't return an uninitialized value 12 years ago
Victor Julien ff668c2030 Fix Tile compile 12 years ago
Eric Leblond 20ca270dc3 fix pf_ring build 12 years ago
Eric Leblond 2a46f0dae4 suricata: rename SuriInstance to SCInstance. 12 years ago
Eric Leblond 9b422f3a8c suricata: suppress Suri prefix
Suppress Suri prefix in internal function name.
12 years ago
Eric Leblond 18ced653c3 Use a typedef for SuriInstance. 12 years ago
Eric Leblond 2d77e53f2c Add offline flag to SuriInstance and some refactoring 12 years ago
Eric Leblond 34abd818dd Prefix util-conf function with Config 12 years ago
Eric Leblond 7242cb30e7 Move CreateLowercaseTable to GLobalInits 12 years ago
Eric Leblond 02e9851315 Generic code don't need ifdef 12 years ago
Eric Leblond 8c00a963aa Use function for delayed detect setup. 12 years ago
Eric Leblond 4296e5f29e Add functions for elapsed time computation. 12 years ago
Eric Leblond 9d1d08c7a4 Factorize Signature loading 12 years ago
Eric Leblond 20c5683b60 Use function for daemonification and signal handler 12 years ago
Eric Leblond 90aaf55201 set rule_reload as part of SuriInstance 12 years ago
Eric Leblond bb19ce1847 SetBPfString is part of command line parsing 12 years ago
Eric Leblond 1a6983ee19 suricata: use function to print version 12 years ago
Eric Leblond 4f789dbe84 Add function for internal running mode 12 years ago
Eric Leblond d3cb043001 suricata: windows specific in one function 12 years ago
Eric Leblond 4401c048ba Running mode is set earlier so out earlier 12 years ago
Eric Leblond 40a25112a0 kill remaining run_mode usage 12 years ago
Eric Leblond 75fa1e20d7 engine analysis is a running mode 12 years ago
Eric Leblond c0d5ee77f9 get (almost) rid of run_mode variable. 12 years ago
Eric Leblond 80542816cd add internal running mode 12 years ago
Eric Leblond e07fdb20a8 Add SuriInstance structure
To be able to split code in functions in main, we need to pass
information about the current running Suricata to functions.
For that we create a structure to store suricata run parameters.

In this patch it allows to separate command line parsing and to
treat internal running mode in a switch just after command line
parsing.
12 years ago
Eric Leblond 325462d396 Export IsRuleReloadSet and use it. 12 years ago
Eric Leblond 8e68b357c7 Suppress Suri prefix. 12 years ago
Eric Leblond 42011e2d32 suricata: function for lowercase table creation 12 years ago
Eric Leblond 132bebb2b2 Simplify code by removing comment 12 years ago
Eric Leblond 07ef1f9837 suricata: add wrapper for interface listing 12 years ago
Eric Leblond 2be7c8aea8 Add util-conf for config util 12 years ago
Eric Leblond 27752818c2 suricata: add some wrapper for config file handling 12 years ago
Eric Leblond b2fa4edd36 move unittest out of suricata.c 12 years ago
Eric Leblond 9a0bf0956b suricata: list cuda cards in separate function 12 years ago
Eric Leblond bed48e3a54 suricata: separate keyword and app layer listing code
The list-keyword and app-layer listing code was spread over all the
init code. This patch introduces a separate file to store non standard
running mode like these ones.
12 years ago
Victor Julien c08b395c2c Init storage api at start up 12 years ago
Victor Julien 0d2a6e515e Add Host specific wrapper to StorageRegister() 12 years ago
Victor Julien 022c0e466e Initial storage api work 12 years ago
Ken Steele 316190c6b9 Add TILE-Gx mPIPE packet processing support.
The TILE-Gx processor includes a packet processing engine, called
mPIPE, that can deliver packets directly into user space memory. It
handles buffer allocation and load balancing (either static 5-tuple
hashing, or dynamic flow affinity hashing are used here). The new
packet source code is in source-mpipe.c and source-mpipe.h

A new Tile runmode is added that configures the Suricata pipelines in
worker mode, where each thread does the entire packet processing
pipeline.  It scales across all the Gx chips sizes of 9, 16, 36 or 72
cores. The new runmode is in runmode-tile.c and runmode-tile.h

The configure script detects the TILE-Gx architecture and defines
HAVE_MPIPE, which is then used to conditionally enable the code to
support mPIPE packet processing. Suricata runs on TILE-Gx even without
mPIPE support enabled.

The Suricata Packet structures are allocated by the mPIPE hardware by
allocating the Suricata Packet structure immediatley before the mPIPE
packet buffer and then pushing the mPIPE packet buffer pointer onto
the mPIPE buffer stack.  This way, mPIPE writes the packet data into
the buffer, returns the mPIPE packet buffer pointer, which is then
converted into a Suricata Packet pointer for processing inside
Suricata. When the Packet is freed, the buffer is returned to mPIPE's
buffer stack, by setting ReleasePacket to an mPIPE release specific
function.

The code checks for the largest Huge page available in Linux when
Suricata is started. TILE-Gx supports Huge pages sizes of 16MB, 64MB,
256MB, 1GB and 4GB. Suricata then divides one of those page into
packet buffers for mPIPE.

The code is not yet optimized for high performance. Performance
improvements will follow shortly.

The code was originally written by Tom Decanio and then further
modified by Tilera.

This code has been tested with Tilera's Multicore Developement
Environment (MDE) version 4.1.5. The TILEncore-Gx36 (PCIe card) and
TILEmpower-Gx (1U Rack mount).
12 years ago
Victor Julien 055b422c28 Remove obsolete code: flow alert sid storage 12 years ago
Victor Julien 9faa4b740d Add --unittests-coverage option to list how many code modules have tests 12 years ago
Victor Julien fc7879322e Rename GetIfaceMaxPayloadSize to GetIfaceMaxPacketSize to reflect the actual function. 12 years ago
Anoop Saldanha cdaa13012a fix for #882.
Refactor the code that initializes the cuda mpm environment.
12 years ago
Anoop Saldanha f85a2dc84b fix for #875.
Update configure.ac to check for either 0.5.5 and 0.5.x version of libhtp.
12 years ago
Anoop Saldanha 48cf0585fb Suricata upgrade to libhtp 0.5.x.
Remove the support for now unsupported personalities from libhtp -
TOMCAT_6_0, APACHE and APACHE_2_2.  We instead use the APACHE_2
personality.
12 years ago
Victor Julien 33818c0272 DNS: fix CUDA build 13 years ago
Victor Julien 8e01cba85d DNS TCP and UDP parser and DNS response logger 13 years ago
Anoop Saldanha 602c91ed41 Minor cosmetic changes to the cuda code.
Moved a couple of functions to more cuda relevant files;
Re-structured some data types.
13 years ago
Anoop Saldanha 17c763f855 Version 1 of AC Cuda. 13 years ago
Anoop Saldanha b787da5643 Remove all cuda related code in the engine except for the cuda api wrappers 13 years ago
Victor Julien 724ad9e8e7 Detect L1 cache line size at build time. Fall back to 64 bytes if detection failed. 13 years ago
Eric Leblond 539de3f5ea bpf filter: use SCLogError instead of fprintf 13 years ago
Eric Leblond dfbb31df8a Exit if bpf is used in IPS mode 13 years ago
Eric Leblond 74a9fc4b66 Add function to display current capture mode
This patch adds a function to display the capture mode.
13 years ago