SA Network Connection Profiler v 1.6.1

**************************************************************************
**Security Analyst Network Connection Profiler [sancp] - v 1.6.1
**    A TCP/IP statistics and pcap collection tool
 * ************************************************************************
 * * Copyright (C) 2003,2004 John Curry <john.curry@metre.net>
 * *
 * * This program is distributed under the terms of version 1.0 of the
 * * Q Public License.  See LICENSE.QPL for further details.
 * *
 * * This program is distributed in the hope that it will be useful,
 * * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 * *
 * ***********************************************************************


 About SANCP:
 ------------

 Without specifying any options, SANCP (pronounced san-cee-pee) collects
 all network traffic on the default pcap interface 'any' 
 SANCP creates three types of output files (pcap, realtime, and stats)
 in the current directory. Filenames contain the interface (-i) and
 timestamp, both are optional.  All three output types are optional see: -R -P -S cmdline 
 options to disable these


	pcap:
	-----
	We record all pcap data in tcpdump format, as it arrives.
	Filename format: pcap.<interface>.<unixtimestamp>

	realtime:
	---------
	We record a 'realtime' entry upon seeing the first 
    packet of each unique connection. The 'realtime' format is changable.
	Filename format: realtime.<interface>.<unixtimestamp>

	stats:
	-----
	We record a 'stats' entry when a connection terminates or times out.
    The 'stats' format is changable	Filename format: stats.<interface>.<unixtimestamp>

	debug_pcap_raw: (Fourth output for Debug mode)
	---------------
	When the -A option is given, log all pcap data to a second pcap file
    -before- any rule or packet manipulation occurs.
    See: 'default debug_pcap_raw enable'
	Filename format: debug_pcap_raw.<interface>.<unixtimestamp>


     *Important Notes about how SANCP handles pcap files*

	SANCP may actively log to multiple files -at the same time-.
	SANCP will log all pcap data (for a given connection)
	is written to the same pcap file. SANCP does this by associating
	each connection with the 'current' pcap output filehandle. 
	Pcap files will remain open until all connections associated with it
    either terminate or timeout
	SANCP will append pcap data to 'pre-existing' pcap files (ref: rule directive 'pcap filename')
	However ,preexisting files need to have valid pcap headers since SANCP will only create 
	a pcap header for new/empty files.
	When managing SANCP pcap files, always check to see if the file 
	is in use before handling it in a destructive manner. If you destroy a SANCP output
	file, while actively in use, SANCP will not recreate the file on the disk for subsequent logging.
	SANCP assumes files that it opens are always present.
     This is done so that SANCP can write pcap data quickly.

	i.e. use a system command such as 'fuser' to check whether a process is using an output file.


   HINT:
   If write rules to define your normal traffic with 'realtimes=pass' set on each,
   then all abnormal activity will be appear in the realtime log.

   HINT:
   'Tuning' sancp consists of running it, extracting new activity from realtime files,
   creating rules to identify the normal activity and having sancp re-read the configuration file
   (Using kill -HUP <sancp pid>).

  Command Line Options: (cmdline)
  ---------------------

	-? or -h  this help screen
	-c <filename>  specify the configuration/rules filename
	-d <directory>  specify the directory for output files
	-i <device>  set the network device to listen on (default: 'any')
	-g <gid>   set a group identity
	-u <uid>   set a user identity
	-r <pcapfile>  pcap file to read (overrides -i)
	-B "<bpf expression>"  set a bpf expression (alternative to -F <filename>)
	-D (daemon) forks, prints msgs to syslog only and overrides -C option
	-K (console) enable additional printing of 'realtimes' to stdout (suppressed by option -D)
	-F <bpf filename>  file containing a bpf filter expression, overrides (alternative to -B)
	-H --human-readable  write IP addresses in dotted notation and TCPflag fields in hex 
	-R  Set default for realtime to 'pass' (default is 'log') disables realtime, but rules can override
	-S  Set default for stats to 'pass' (default is 'log') disables stats, but rules can override
	-P  Set default for pcap to 'pass' (default is 'log') disables pcap, but rules can override
	-I or --enable_icmp_mixed  record 'code' and 'type' fields for ICMP
		to the fields 's_port' and 'd_port'.
		note: affects how related icmp packets are correlated 
	-V  display version
    --shift  (debug) force interpretation of packet starting at byte[2] 
                    normally performed when reading from the 'any' interface
	--strip-80211  strip 802.1Q headers from 802.1Q packets; used to 
 	  decode 802.1Q encapsulated packets - affects -A option, 
	--log-facility <facility>  where facility can be 'LOCAL1' - 'LOCAL7'
		The default log facility used by SANCP is LOG_DAEMON 

     # Debug mode for pcap data logging
	-A  records ALL traffic frames to a pcap file named 'debug_pcap_raw'
	  (despite rules). Packets are logged here prior to decoding or handling. 
	  Use -F or -B option to restrict what is collectedi.
	  Pcap data logged using this option is affected by the --strip-80211 cmdline option
	  The configuration file equivalent to this is 'default debug_pcap_raw enable'


   Kill Signals:
   -------------
	
   	-HUP   re-read rules configuration file and open new output files
		(sets new <unixtimestamp> used for new output files)
	-USR1  print running configuration (with counters for rule matches)
	-USR2  print -all ongoing- connections to stdout

   HINT:
      Run sancp something like this to have access the kill signal output in daemon mode
	sancp -D -H >> sancp.output & 
	This way you can view it 'cat sancp.output' and then clear it '> sancp.output'
	and sancp will continue to output to this file as normal.

  Output Fields: for 'realtime' and 'stats' files
  (Some realtime fields are naturally blank, i.e. counters)
  --------------

   1:   64bit sancp id: based on timeptr.tv_sec and timeptr.tv_usec
   2:   32bit start time: unix timestamp for first packet
   3:   32bit end time: unix timestamp for last packet
   4:   32bit erased time: unix timestamp for when connection was cleared from memory
   5:   16bit hw_proto: layer 2 protocol number
   6:   8bit proto: layer 3 protocol (if IP proto is layer 2)
   7:   32bit source address: dotted notation IP address
   8:   16bit source port: i.e. udp, tcp 
          also used for icmp 'type' (see: --enable_icmp_mixed)
   9:   32bit destination address: dotted notation IP address
   10:  16bit destination port: i.e. udp, tcp 
   	      also used for icmp 'code' (see: --enable_icmp_mixed)
   11:  32bit duration: seconds the connection remained active 
   	      (difference between start and end times)
   12:  16bit timeout: applicable timeout value for the connection
   13:  64bit source packets: packets received from source
   14:  64bit destination packets: packets received from destination
   15:  64bit source bytes: bytes received from source
   16:  64bit destination bytes: bytes received from destination
     The next two fields contain 8bit values representing 8 possible TCP flags
     cumulativeily seen from source and destination throughout the connection
              8Bit order is 12UAPRSF, where:
                 1: Reserved bit 1 from source
                 2: Reserved bit 2 from source
                 U: Urgent Pointer bit from source
                 A: ACK bit from source
                 P: Push bit from source
                 R: Reset bit from source
                 S: SYN bit from source
                 F: FIN bit from source

   17:  8bit sflags: cumlative tcp flags from source (bit order: 12UAPRSF)
   18:  8bit dflags: cumlative tcp flags from dest (bit order: 12UAPRSF)

     The next field contains an 8bit value representing 6 possible TCP close session flags
     from the source and destination.  The first 2 significant bits are unused.)
              8Bit order is 00AARRFF/00DSDSDS, where:
                  DA: Close ACK seen from destination
                  SA: Close ACK seen from source
                  DR: Close Reset seen from destination
                  SR: Close Reset seen from source
                  DF: Close FIN seen from destination
                  SF: Close FIN seen from source
   19:  8bit closed flags (bit order: 00AARRFF/00DSDSDS)

    The next 8 fields contain p0F information gathered from initial TCP packet
   20:  16bit wss: window segment size (initial packet, tcp only)
   21:  8bit ttl: time to live (initial packet, tcp only)
   22:  16bit mss: maximum segment size (initial packet, tcp only)
   23:  Y/N df: don't fragment bit was set (initial packet, tcp only)
   24:  8bit wscale: window scale (initial packet, tcp only)
   25:  Y/N sack_ok: sack_ok flag was set (initial packet, tcp only)
   26:  Y/N nop: 'no op' was seen (initial packet, tcp only)
   27:  16bit len: ip length (initial packet, tcp only)

    The next 8 fields contain p0F information gathered from second TCP packet
   28:  16bit wss2: window segment size (second packet, tcp only)
   29:  8bit ttl2: time to live (second packet, tcp only)
   30:  16bit mss2: maximum segment size (second packet, tcp only)
   31:  Y/N df2: don't fragment bit was set (second packet, tcp only)
   32:  8bit wscale2: window scale (second packet, tcp only)
   33:  Y/N sack_ok2: sack_ok flag was set (second packet, tcp only)
   34:  Y/N nop2: 'no op' was seen (second packet, tcp only)
   35:  16bit len2: ip length (second packet, tcp only)

    The last 7 fields contain information about how we handled the connection
   36:  8bit reversed: did we reverse the ip addresses seen in the 
         initial packet?  0=no, 1=yes, 2=no(both ports were known),
         3=no(both ports were unknown)
   37:  8bit collect: what mode was used for collecting: 
         none, both, from_src, from_dst (0,1,2,3 respectively)
   38:  64bit collected: how much data did we collect 
   39:  64bit limit: how much data were we limited to collecting
   40:  16bit tcplag value: seconds to wait for straggler packets, after the connection 'ends'
   41:  Y/N pcap enabled: did we record data to a pcap file 
         (does not apply to data recorded using the -A option)
   42:  Y/N realtime enabled: did we record the connection to a realtime file
   43:  Y/N stats enabled: did we record the connection to a stats file 
   44:  16bit hash value: used for tuning (developer's choice)
   45:  64bit total_bytes: useful for overall statistics
   46:  32bit rid: rule id  assoc. w/ the network profile rule that this connection matched on ('0' is default)
   47:  8bit status: status assigned to this connection i.e. assigned by rule
   48:  16bit node: node/network interface/sancp instance associated this connection
        i.e. assigned globally as a 'default' or, specifically, by a rule
   49:  17byte src-mac: source ethernet address in ascii format i.e. xx:xx:xx:xx:xx:xx 
   50:  17byte dst_mac: destination ethernet address in ascii format i.e. xx:xx:xx:xx:xx:xx 



   HINT:
   Check fields 41-43 to see what kind of logging was performed on the connection at a glance
   i.e. 'Was a realtime logged' (a.k.a 'have we seen this traffic before')
   i.e. 'Did we collect any data' (a.k.a '')




  Configuration and Rule Syntax:  (one rule per line)
  -----------------------------


  The configuration file designates the characters: <tab> <space> ',' and '=' as word separators
  These four characters may be used liberally as rule-beautifying delimiters; they are treated spaces.


  var syntax:
  -----------------------:
  Use vars to avoid having to use protocol numbers in rules i.e.   var icmp 1
  Vars are used to define 4 kinds of values: ethernet protocols, ip addresses, ip protocols, and ports
  These values are present in the connection rules and the known_ports definition
  These vars remain present when sancp prints the running configuration
  Var 'names' should be unique nnd represent only one kind of value, else rule behavior is undefined 
  (generally, you may get parse errors or the running configuration output will appear incorrect.)
  			
  Vars have valid value ranges depending on the kind of value they are to represent;
      ethernet protocols:  0x0-0xFFFF  (0-65535)
      ip addresses:        0.0.0.0/255.255.255.255 (0.0.0.0/32)
      ip protocols:        0x0-0xFF (0-255)
      ports:               0x0-0xFFFF (0-65535)
 	Values outside these ranges may be trunicated or otherwise result in a rule error
  	You can represent all but 'dotted-ip' values in decimal, hex or octal.
  	One single range should be specified in a var. In the case of an IP address, you 
    will want to use a normal ipaddress/mask to represent a 'network range'
  			
  var <varname> <ip address{/[<CIDR>|<dotted>]}>
  	Define <varname> for use in place of IP addresses in proceding rules

  'default' syntax:
  -----------------------:
  default <keyword> <option(s)> (defaults specified here override command line options
  		keywords:
  		 pcap     {log|pass}
  		 realtime {log|pass}
  		 stats    {log|pass}
  		 limit   <bytes>
  		 timeout <seconds>
  		 tcplag  <seconds>
  		 status  <value 0-255>
  		 pcapfilter [ bpf expression ] (read only once at start-up)
  		 strip-80211  { disable|enable }
  		 node <number>
  		 debug_pcap_raw  { disable|enable }

  known_port syntax:
  -----------------------:
  known_ports [<protocol>] [<port>{-<end_port_range>}{,<another range>}{,...}] 
     Define a list of 'known tcp and/or udp server ports'
     SANCP will use these lists to help 'resolve/guess' the direction of ambiguous tcp/udp connections
     Lists should only be provided to help reduce the occurrance of logging 'reversed' connections.
     And were going to make this one hurt... you have to specify '6' or '17' for the <protocol> 
     Or just create and use vars for them i.e. 'var tcp 6','var udp 17'

	Short Example of using vars in conjuntion with known_ports:

		var tcp 6
		var udp 17
		var http 80
		var https 443
		var dns 53
		known_ports udp dns
		known_ports tcp dns,http,https


  connection rule syntax:
  -----------------------:

	A connection rule consists of two central parts:
 		1) network connection profile 
			i.e. ether proto, ip address, ip proto and ports
		2) options
			a) collection options
				i.e. stats=pass, pcap=pass, realtime=pass, timeout=120 or limit=1500 
			b) tagging options 
				i.e. status=16 rid=1112 node=2 

  [<ether protocol>[-<end_range>] [<src_ip{/<CIDR>|<dotted>}>] [<dst_ip{/<CIDR>|<dotted>}>] [{tcp|udp|icmp|<proto number>[-<end_range>] }]
  [<src_port>{-[<end_port_range>]}] [<dst_port>{-[<end_port_range>]}] 
  { ignore | stats [{log|pass}] | realtime [{log|pass}] |
  pcap [ {log|pass|rule|connection|{filename|tsfilename} [<outputfilename>]}
  { logdst|logsrc } { timeout [<seconds>]|limit [<bytes>]|tcplag [<secs>]|retro|status <0-255>|rid <number>|node <number> }


  Description for connection options:
  --------------------------------------

    timeout <secs> - set delay after last packet before expiring the connection 
    limit <bytes> - set max bytes of pcap data to record per connection 
    realtime (option):
            pass - do not log realtime for this traffic
            log - log realtime for this traffic
    stats (option): 
            pass - do not log statistics for this traffic
            log - log statistics for this traffic
    pcap (option): 
		pass - do not record pcap data
		log - record pcap data to the default 'pcap' output file
		rule - record pcap data to output file; filename derived from rule
		connection - record pcap data to a output file; filename derived 
			 from the connection
		filename <outputfilename> - record pcap data to a specific output 
			 filename (names starting with '/' are considered absolute).
   logsrc - only record pcap data from the source (default is both)
   logdst - only record pcap data from the destination (default is both)
   ignore - set realtime, stats, and pcap to 'pass' (ignores any logdst or logsrc options)
   retro - apply this rule to -all- ongoing connections, not just new ones

  Description of 'tagging' options:
  --------------------------------------

   status <number> - status to be assigned to matching connections 
   rid <number> - rule id (32bit) for this rule (assign to matching connections)
   node <number> - node id (8bit) number to assign to matching connections
   	    the node id is formed from the notion that more than one network could be monitored
   	    by one or more instances of sancp on the same system.  Node id can be handy in rules
        to help tag traffic as belonging to a certain network interface; i.e. consider '-i any'


   	NOTE: Malformed rules are reported to syslog and simply ignored



  Basic Examples:
  ---------------


    Notes:

	Below is a matrix outlining how the three different output types are used for four different modes of operation.

	Output Type 	Mode 1	Mode 2	Mode 3  Mode 4  Mode 5
	----------------------------------------------------- 
	pcap         	log	log	pass	pass    *     
	realtime	log	pass	pass	pass    *     
	stats		log	log	log	pass    *     
	debug_pcap_raw	disable	disable	disable	disable enable

     These modes can be obtained by setting their 'defaults' in the sancp.conf
     or by providing the command line option: -P -S and -R to disable pcap, stats
	      and realtime, respectively 

	IMPORTANT NOTE: the configuration file overides the cmdline options to ensure SANCP
	can be controlled through configuration file changes (use: kill -HUP <pid>  to re-read the config)

	Mode 1: Default Monitoring Mode: allow full access to 'realtime', 'stats' and 'pcap' data
		Use a set of rules which define your network. Disable realtime for uninteresting traffic.
		Use collection options to reduce collection effort on certain traffic
		Use realtime entries as 'alerts' to notify you of new and interesting traffic
		Modify rules real-time so sancp can stay current with your changing collection requirements
		Use rule identifiers (rid)'s to mark connections as matching a different kinds (profiles) of traffic
		Storing rules with rule id's in the same database allows for quick access to connections
		of certain kinds of traffic

	Mode 2: Batch Analysis Mode: for (re)processing pcap files - realtime disabled
		Use a set of rules to extract interesting traffic from large tcpdump files;
		Create a 'pcap' file containing only traffic of interest.
		Use the 'stats' file as an index to the data available in 'pcap' file.

	Mode 3: Connection Profiling Mode: (output only a stats log from a pcap file)
		Rules may be needed to exclude certain IP traffic you don't care about.

	Mode 4: Pcap Split Mode: turns off all default output modes, uses rules to
		control which files matching traffic should be written to.
		  Use the 'pcap filename <filename>' rule option to specify an output file.
		  The 'pcap rule' option will create a filename based on/derived from the rule itself
	i.e. <sip>-<smask>:<dip>-<dmask>_<sportl>-<sporth>:<dportl>-<dporth>_<protoh>-<protol>.<sancprestarttime>
		  The 'pcap uniq' option will write to a pcap file whose filename is
		based on/derived from the connection itself:
		i.e. <sip>:<sport>_<dip>:<dport>-<proto>.<sancpstarttime>

	Mode 5: Debug Pcap Raw Mode: additionally, records all traffic to a 'debug_pcap_raw' file
	        regardless of any rules.  80211 headers are still stripped, if configured to do so
	        This is enabled via command line (-A) or via config file (default debug_pcap_raw enable)
	        It can subsequently be disabled via config file (default debug_pcap_raw disable)

	 **To use the configuration file to dynamically (re)configure sancp while running**
		see: 'kill signals'

	-----------------------------
	#
	#  Example sancp.conf file
	#
	# Define known_ports to help sancp determine connection direction
	# for pre-existing udp and tcp connections (i.e. at startup)
	# We set these only as we need them. They are used for half-open TCP connections
	# (ie. if we missed the syn or syn-ack), and for all udp connections 
	# The 'reversed' field in the connection (profile) output will tell you if 
	# SANCP recorded the direction opposite that of the initial packet (i.e. '1').
	#
	# known_ports tcp 80,443
	# known_ports udp 53

	# Override default logging for stats, pcap, and realtime 
	#
	# ** The sancp configuration file can be re-loaded dynamically while running **
	#
	# Configure default mode for 'stats' logging
	#
	#default stats log	# sets default mode to 'log stats' (*default mode)
	#			# use 'pass' to set default mode to 'do not log stats'
	#
	# Configure default mode for 'pcap' logging 
	#
	#default pcap log	# sets default mode to 'record pcap data' (*default mode)
	#			# use 'pass' to set default mode to 'do not record pcap data' 
	#
	# Configure default mode for 'realtime' logging 
	#
	#default realtime log 	# create a realtime when we record pcap data (*default mode)
	#		 	# use 'pass' set default to 'not create' realtime 
	# Note: You can add the 'realtime log' option to a rule to 'force' all matches to log a 
	# realtime regardless of whether we record pcap data
	#
	#default debug_pcap_raw disable	# enable|disable debug pcap logging mode for online debugging
	#	# if set to 'enable' we will record packets to a 'debug_pcap_raw' file, regardless of rules 
	#
	#default status 0	# sets default 8bit status (0-255) for all connections which do not match a rule, or where a status is not specified for a rule(default = 0)
	#
	# Define local vars (used for IP/MASK combinations only)
	var HOME_NET 192.168.1.0/24
	var ip 8

	#
	# The following rule syntax is supported:
	# 	Rule format:
	# 	sip dip  proto sp   dp   options
	#	any any  any   any  any  pcap none

	# Ignore outbound HTTP (ignore both pcap and stats)
	ip HOME_NET any tcp any 80 pcap pass stats pass

	# Do not record ssh data
	ip HOME_NET any tcp any 22 pcap pass

	# Streaming media can kill your logging so
	# we ignore UDP > 1024 with few a exceptions
	ip any HOME_NET udp any 1025-32769 pcap pass
	ip any HOME_NET udp any 32781- pcap pass

	# Don't log ICMP at all (no stats, pcap, or realtime) 
	ip any any icmp any any ignore

	# Ignore incoming blaster scans
	ip any HOME_NET tcp any 135 ignore


